Add files using upload-large-folder tool

.gitattributes

@@ -212,3 +212,4 @@ tuning-competition-baseline/.venv/lib/python3.11/site-packages/torch/_inductor/_
 .venv/lib/python3.11/site-packages/frozenlist/_frozenlist.cpython-311-x86_64-linux-gnu.so filter=lfs diff=lfs merge=lfs -text
 .venv/lib/python3.11/site-packages/uvloop/loop.cpython-311-x86_64-linux-gnu.so filter=lfs diff=lfs merge=lfs -text
 .venv/lib/python3.11/site-packages/functorch/_C.cpython-311-x86_64-linux-gnu.so filter=lfs diff=lfs merge=lfs -text
+.venv/lib/python3.11/site-packages/fsspec/__pycache__/spec.cpython-311.pyc filter=lfs diff=lfs merge=lfs -text

.venv/lib/python3.11/site-packages/einops-0.8.0.dist-info/INSTALLER

@@ -0,0 +1 @@
+pip

.venv/lib/python3.11/site-packages/einops-0.8.0.dist-info/METADATA

@@ -0,0 +1,360 @@
+Metadata-Version: 2.3
+Name: einops
+Version: 0.8.0
+Summary: A new flavour of deep learning operations
+Project-URL: Homepage, https://github.com/arogozhnikov/einops
+Author: Alex Rogozhnikov
+License: MIT
+License-File: LICENSE
+Keywords: deep learning,einops,machine learning,neural networks,scientific computations,tensor manipulation
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+
+
+<!--
+<a href='http://arogozhnikov.github.io/images/einops/einops_video.mp4' >
+<div align="center">
+  <img src="http://arogozhnikov.github.io/images/einops/einops_video.gif" alt="einops package examples" />
+  <br>
+  <small><a href='http://arogozhnikov.github.io/images/einops/einops_video.mp4'>This video in high quality (mp4)</a></small>
+  <br><br>
+</div>
+</a>
+-->
+
+<!-- this link magically rendered as video, unfortunately not in docs -->
+
+https://user-images.githubusercontent.com/6318811/177030658-66f0eb5d-e136-44d8-99c9-86ae298ead5b.mp4
+
+
+
+
+# einops 
+[![Run tests](https://github.com/arogozhnikov/einops/actions/workflows/run_tests.yml/badge.svg)](https://github.com/arogozhnikov/einops/actions/workflows/run_tests.yml)
+[![PyPI version](https://badge.fury.io/py/einops.svg)](https://badge.fury.io/py/einops)
+[![Documentation](https://img.shields.io/badge/documentation-link-blue.svg)](https://einops.rocks/)
+![Supported python versions](https://raw.githubusercontent.com/arogozhnikov/einops/master/docs/resources/python_badge.svg)
+
+
+Flexible and powerful tensor operations for readable and reliable code. <br />
+Supports numpy, pytorch, tensorflow, jax, and [others](#supported-frameworks).
+
+## Recent updates:
+
+- 0.7.0: no-hassle `torch.compile`, support of [array api standard](https://data-apis.org/array-api/latest/API_specification/index.html) and more
+- 10'000🎉: github reports that more than 10k project use einops
+- einops 0.6.1: paddle backend added
+- einops 0.6 introduces [packing and unpacking](https://github.com/arogozhnikov/einops/blob/master/docs/4-pack-and-unpack.ipynb)
+- einops 0.5: einsum is now a part of einops
+- [Einops paper](https://openreview.net/pdf?id=oapKSVM2bcj) is accepted for oral presentation at ICLR 2022 (yes, it worth reading).
+  Talk recordings are [available](https://iclr.cc/virtual/2022/oral/6603)
+
+
+<details markdown="1">
+<summary>Previous updates</summary>
+- flax and oneflow backend added
+- torch.jit.script is supported for pytorch layers
+- powerful EinMix added to einops. [Einmix tutorial notebook](https://github.com/arogozhnikov/einops/blob/master/docs/3-einmix-layer.ipynb) 
+</details>
+
+<!--<div align="center">
+  <img src="http://arogozhnikov.github.io/images/einops/einops_logo_350x350.png" 
+  alt="einops package logo" width="250" height="250" />
+  <br><br>
+</div> -->
+
+
+## Tweets 
+
+> In case you need convincing arguments for setting aside time to learn about einsum and einops...
+[Tim Rocktäschel](https://twitter.com/_rockt/status/1230818967205425152)
+
+> Writing better code with PyTorch and einops 👌
+[Andrej Karpathy](https://twitter.com/karpathy/status/1290826075916779520)
+
+> Slowly but surely, einops is seeping in to every nook and cranny of my code. If you find yourself shuffling around bazillion dimensional tensors, this might change your life
+[Nasim Rahaman](https://twitter.com/nasim_rahaman/status/1216022614755463169)
+
+[More testimonials](https://einops.rocks/pages/testimonials/)
+
+<!--
+## Recordings of talk at ICLR 2022
+
+<a href='https://iclr.cc/virtual/2022/oral/6603'>
+<img width="922" alt="Screen Shot 2022-07-03 at 1 00 15 AM" src="https://user-images.githubusercontent.com/6318811/177030789-89d349bf-ef75-4af5-a71f-609896d1c8d9.png">
+</a>
+
+Watch [a 15-minute talk](https://iclr.cc/virtual/2022/oral/6603) focused on main problems of standard tensor manipulation methods, and how einops improves this process.
+-->
+
+## Contents
+
+- [Installation](#Installation)
+- [Documentation](https://einops.rocks/)
+- [Tutorial](#Tutorials) 
+- [API micro-reference](#API)
+- [Why using einops](#Why-using-einops-notation)
+- [Supported frameworks](#Supported-frameworks)
+- [Citing](#Citing)
+- [Repository](https://github.com/arogozhnikov/einops) and [discussions](https://github.com/arogozhnikov/einops/discussions)
+
+## Installation  <a name="Installation"></a>
+
+Plain and simple:
+```bash
+pip install einops
+```
+
+<!--
+`einops` has no mandatory dependencies (code examples also require jupyter, pillow + backends). 
+To obtain the latest github version 
+
+```bash
+pip install https://github.com/arogozhnikov/einops/archive/master.zip
+```
+-->
+
+## Tutorials <a name="Tutorials"></a>
+
+Tutorials are the most convenient way to see `einops` in action
+
+- part 1: [einops fundamentals](https://github.com/arogozhnikov/einops/blob/master/docs/1-einops-basics.ipynb) 
+- part 2: [einops for deep learning](https://github.com/arogozhnikov/einops/blob/master/docs/2-einops-for-deep-learning.ipynb)
+- part 3: [packing and unpacking](https://github.com/arogozhnikov/einops/blob/master/docs/4-pack-and-unpack.ipynb)
+- part 4: [improve pytorch code with einops](http://einops.rocks/pytorch-examples.html)   
+
+Kapil Sachdeva recorded a small [intro to einops](https://www.youtube.com/watch?v=xGy75Pjsqzo).
+
+## API <a name="API"></a>
+
+`einops` has a minimalistic yet powerful API.
+
+Three core operations provided ([einops tutorial](https://github.com/arogozhnikov/einops/blob/master/docs/) 
+shows those cover stacking, reshape, transposition, squeeze/unsqueeze, repeat, tile, concatenate, view and numerous reductions)
+
+```python
+from einops import rearrange, reduce, repeat
+# rearrange elements according to the pattern
+output_tensor = rearrange(input_tensor, 't b c -> b c t')
+# combine rearrangement and reduction
+output_tensor = reduce(input_tensor, 'b c (h h2) (w w2) -> b h w c', 'mean', h2=2, w2=2)
+# copy along a new axis
+output_tensor = repeat(input_tensor, 'h w -> h w c', c=3)
+```
+
+Later additions to the family are `pack` and `unpack` functions (better than stack/split/concatenate):
+
+```python
+from einops import pack, unpack
+# pack and unpack allow reversibly 'packing' multiple tensors into one.
+# Packed tensors may be of different dimensionality:
+packed,  ps = pack([class_token_bc, image_tokens_bhwc, text_tokens_btc], 'b * c')
+class_emb_bc, image_emb_bhwc, text_emb_btc = unpack(transformer(packed), ps, 'b * c')
+```
+
+Finally, einops provides einsum with a support of multi-lettered names: 
+
+```python
+from einops import einsum, pack, unpack
+# einsum is like ... einsum, generic and flexible dot-product 
+# but 1) axes can be multi-lettered  2) pattern goes last 3) works with multiple frameworks
+C = einsum(A, B, 'b t1 head c, b t2 head c -> b head t1 t2')
+```
+
+### EinMix
+
+`EinMix` is a generic linear layer, perfect for MLP Mixers and similar architectures.
+
+### Layers
+
+Einops provides layers (`einops` keeps a separate version for each framework) that reflect corresponding functions
+
+```python
+from einops.layers.torch      import Rearrange, Reduce
+from einops.layers.tensorflow import Rearrange, Reduce
+from einops.layers.flax       import Rearrange, Reduce
+from einops.layers.paddle     import Rearrange, Reduce
+from einops.layers.chainer    import Rearrange, Reduce
+```
+
+<details markdown="1">
+<summary>Example of using layers within a pytorch model</summary>
+Example given for pytorch, but code in other frameworks is almost identical
+
+```python 
+from torch.nn import Sequential, Conv2d, MaxPool2d, Linear, ReLU
+from einops.layers.torch import Rearrange
+
+model = Sequential(
+    ...,
+    Conv2d(6, 16, kernel_size=5),
+    MaxPool2d(kernel_size=2),
+    # flattening without need to write forward
+    Rearrange('b c h w -> b (c h w)'),  
+    Linear(16*5*5, 120), 
+    ReLU(),
+    Linear(120, 10), 
+)
+```
+
+No more flatten needed! 
+
+Additionally, torch users will benefit from layers as those are script-able and compile-able.
+</details>
+
+
+
+
+## Naming <a name="Naming"></a>
+
+`einops` stands for Einstein-Inspired Notation for operations 
+(though "Einstein operations" is more attractive and easier to remember).
+
+Notation was loosely inspired by Einstein summation (in particular by `numpy.einsum` operation).
+
+## Why use `einops` notation?! <a name="Why-using-einops-notation"></a>
+
+
+### Semantic information (being verbose in expectations)
+
+```python
+y = x.view(x.shape[0], -1)
+y = rearrange(x, 'b c h w -> b (c h w)')
+```
+While these two lines are doing the same job in *some* context,
+the second one provides information about the input and output.
+In other words, `einops` focuses on interface: *what is the input and output*, not *how* the output is computed.
+
+The next operation looks similar:
+
+```python
+y = rearrange(x, 'time c h w -> time (c h w)')
+```
+but it gives the reader a hint: 
+this is not an independent batch of images we are processing, 
+but rather a sequence (video). 
+
+Semantic information makes the code easier to read and maintain. 
+
+### Convenient checks
+
+Reconsider the same example:
+
+```python
+y = x.view(x.shape[0], -1) # x: (batch, 256, 19, 19)
+y = rearrange(x, 'b c h w -> b (c h w)')
+```
+The second line checks that the input has four dimensions, 
+but you can also specify particular dimensions. 
+That's opposed to just writing comments about shapes since comments don't prevent mistakes, not tested, and without code review tend to be outdated   
+```python
+y = x.view(x.shape[0], -1) # x: (batch, 256, 19, 19)
+y = rearrange(x, 'b c h w -> b (c h w)', c=256, h=19, w=19)
+```
+
+### Result is strictly determined
+
+Below we have at least two ways to define the depth-to-space operation
+```python
+# depth-to-space
+rearrange(x, 'b c (h h2) (w w2) -> b (c h2 w2) h w', h2=2, w2=2)
+rearrange(x, 'b c (h h2) (w w2) -> b (h2 w2 c) h w', h2=2, w2=2)
+```
+There are at least four more ways to do it. Which one is used by the framework?
+
+These details are ignored, since *usually* it makes no difference, 
+but it can make a big difference (e.g. if you use grouped convolutions in the next stage), 
+and you'd like to specify this in your code.
+
+
+### Uniformity
+
+```python
+reduce(x, 'b c (x dx) -> b c x', 'max', dx=2)
+reduce(x, 'b c (x dx) (y dy) -> b c x y', 'max', dx=2, dy=3)
+reduce(x, 'b c (x dx) (y dy) (z dz) -> b c x y z', 'max', dx=2, dy=3, dz=4)
+```
+These examples demonstrated that we don't use separate operations for 1d/2d/3d pooling, 
+those are all defined in a uniform way. 
+
+Space-to-depth and depth-to space are defined in many frameworks but how about width-to-height? Here you go:
+
+```python
+rearrange(x, 'b c h (w w2) -> b c (h w2) w', w2=2)
+```
+
+### Framework independent behavior
+
+Even simple functions are defined differently by different frameworks
+
+```python
+y = x.flatten() # or flatten(x)
+```
+
+Suppose `x`'s shape was `(3, 4, 5)`, then `y` has shape ...
+
+- numpy, pytorch, cupy, chainer: `(60,)`
+- keras, tensorflow.layers, gluon: `(3, 20)`
+
+`einops` works the same way in all frameworks.
+
+### Independence of framework terminology
+
+Example: `tile` vs `repeat` causes lots of confusion. To copy image along width:
+```python
+np.tile(image, (1, 2))    # in numpy
+image.repeat(1, 2)        # pytorch's repeat ~ numpy's tile
+```
+
+With einops you don't need to decipher which axis was repeated:
+```python
+repeat(image, 'h w -> h (tile w)', tile=2)  # in numpy
+repeat(image, 'h w -> h (tile w)', tile=2)  # in pytorch
+repeat(image, 'h w -> h (tile w)', tile=2)  # in tf
+repeat(image, 'h w -> h (tile w)', tile=2)  # in jax
+repeat(image, 'h w -> h (tile w)', tile=2)  # in cupy
+... (etc.)
+```
+
+[Testimonials](https://einops.rocks/pages/testimonials/) provide users' perspective on the same question. 
+
+## Supported frameworks <a name="Supported-frameworks"></a>
+
+Einops works with ...
+
+- [numpy](http://www.numpy.org/)
+- [pytorch](https://pytorch.org/)
+- [tensorflow](https://www.tensorflow.org/)
+- [jax](https://github.com/google/jax)
+- [cupy](https://cupy.chainer.org/)
+- [chainer](https://chainer.org/)
+- [tf.keras](https://www.tensorflow.org/guide/keras)
+- [flax](https://github.com/google/flax) (experimental)
+- [paddle](https://github.com/PaddlePaddle/Paddle) (experimental)
+- [oneflow](https://github.com/Oneflow-Inc/oneflow) (community)
+- [tinygrad](https://github.com/tinygrad/tinygrad) (community)
+
+Additionally, starting from einops 0.7.0 einops can be used with any framework that supports [Python array API standard](https://data-apis.org/array-api/latest/API_specification/index.html)
+
+## Citing einops <a name="Citing"></a>
+
+Please use the following bibtex record
+
+```text
+@inproceedings{
+    rogozhnikov2022einops,
+    title={Einops: Clear and Reliable Tensor Manipulations with Einstein-like Notation},
+    author={Alex Rogozhnikov},
+    booktitle={International Conference on Learning Representations},
+    year={2022},
+    url={https://openreview.net/forum?id=oapKSVM2bcj}
+}
+```
+
+
+## Supported python versions
+
+`einops` works with python 3.8 or later.

.venv/lib/python3.11/site-packages/einops-0.8.0.dist-info/RECORD

@@ -0,0 +1,42 @@
+einops-0.8.0.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
+einops-0.8.0.dist-info/METADATA,sha256=5hTpaWnwYNe3QvhbXYTpA_LUJ2lSlyspSc0gRGni7sY,12926
+einops-0.8.0.dist-info/RECORD,,
+einops-0.8.0.dist-info/WHEEL,sha256=zEMcRr9Kr03x1ozGwg5v9NQBKn3kndp6LSoSlVg-jhU,87
+einops-0.8.0.dist-info/licenses/LICENSE,sha256=MNmENkKW9R_67K1LAe4SfpUlDFBokY1LZvyWIGcj5DQ,1073
+einops/__init__.py,sha256=UdixJ9CShlEOQfw0xcU6zYtrAn6Durgh6jCQWdcaQK4,422
+einops/__pycache__/__init__.cpython-311.pyc,,
+einops/__pycache__/_backends.cpython-311.pyc,,
+einops/__pycache__/_torch_specific.cpython-311.pyc,,
+einops/__pycache__/array_api.cpython-311.pyc,,
+einops/__pycache__/einops.cpython-311.pyc,,
+einops/__pycache__/packing.cpython-311.pyc,,
+einops/__pycache__/parsing.cpython-311.pyc,,
+einops/_backends.py,sha256=VHPPrL1mf0PDTvyFPZvmZeTqGJoWflqv7b-eoJUHudo,21081
+einops/_torch_specific.py,sha256=yMaQeqAZhBLWR1Q-Jv6uRINJfzROhLb-rzKKevpefUU,4138
+einops/array_api.py,sha256=FcKZSo7l8jC5HL8qudutz1K5x9cFpwACMDcjfbvEKmQ,5251
+einops/einops.py,sha256=AYZe5yMlH-EXO0MWFv27ajyPdVTFpYloaSCRM9jw5sA,37252
+einops/experimental/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+einops/experimental/__pycache__/__init__.cpython-311.pyc,,
+einops/experimental/__pycache__/indexing.cpython-311.pyc,,
+einops/experimental/indexing.py,sha256=4NtRNmSOrpUURvwhrbbGNK3NeTxHI4EW8R6ct3JZyLw,14868
+einops/layers/__init__.py,sha256=vBtnAt2afs4QlqpeFU4dlZNxBuC9IXl3fmilk-2OzHM,3747
+einops/layers/__pycache__/__init__.cpython-311.pyc,,
+einops/layers/__pycache__/_einmix.cpython-311.pyc,,
+einops/layers/__pycache__/chainer.cpython-311.pyc,,
+einops/layers/__pycache__/flax.cpython-311.pyc,,
+einops/layers/__pycache__/keras.cpython-311.pyc,,
+einops/layers/__pycache__/oneflow.cpython-311.pyc,,
+einops/layers/__pycache__/paddle.cpython-311.pyc,,
+einops/layers/__pycache__/tensorflow.cpython-311.pyc,,
+einops/layers/__pycache__/torch.cpython-311.pyc,,
+einops/layers/_einmix.py,sha256=0cl3r4Xp44S2HO-tx0MHa4cMFD2KJXpG5O-4gJM5AtU,8464
+einops/layers/chainer.py,sha256=hUB-XSjN5CP8zALZtalL3n2lQkq7vymftRI8okEMO2Q,1861
+einops/layers/flax.py,sha256=zFy83gSLRm31cLuKFRvZ82_HsefnXPbRvkKZh1KkC1I,2536
+einops/layers/keras.py,sha256=-7So0w94phvf9HdW0xi2mSeBg02qVPvAyfp_1XR02NM,212
+einops/layers/oneflow.py,sha256=YEPzz4xc7BDRQfb8ulD3teqQJdbO6qQg7Z4KIPVTLz8,1864
+einops/layers/paddle.py,sha256=8cRZQ8BT9vYEczh7pNProuTM_3XjLty2ht2sdvXNFiI,1907
+einops/layers/tensorflow.py,sha256=T9uhSVwbXREahc31ARAHoN5K-7zsuS8NRNPdY6Zk1Bc,3324
+einops/layers/torch.py,sha256=504G99kEgy7dk1UPBbj9hzJmZkAHwVhMDFN_8J-p3C8,2399
+einops/packing.py,sha256=Ln2lAMko9hobi_qd-4dPtQY0Ks5hRK7x-5FthL2gunk,7654
+einops/parsing.py,sha256=xbqcvwReLiROEucoegZ20WQiEHlLg0uxo_vYoezKB_4,6746
+einops/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0

.venv/lib/python3.11/site-packages/einops-0.8.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.24.2
+Root-Is-Purelib: true
+Tag: py3-none-any

.venv/lib/python3.11/site-packages/einops-0.8.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Alex Rogozhnikov
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

.venv/lib/python3.11/site-packages/fsspec/__pycache__/spec.cpython-311.pyc

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35393347e555c64ec1f982848d600d95a4cba3d655679aa08a140b06534d80c6
+size 100192

.venv/lib/python3.11/site-packages/pycountry/COPYRIGHT.txt

@@ -0,0 +1,46 @@
+COPYRIGHT (c) 2008 - 2023, pycountry
+
+Pycountry is free software; you can redistribute it and/or modify
+it under the terms of the GNU Lesser General Public License as published by
+the Free Software Foundation; either version 2.1 of the License, or any later version.
+
+This project is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Lesser General Public License for more details.
+
+Contributors:
+- Christian Theune (2008-2020, 2022)
+- Nate Schimmoller (2022-2023)
+- Zachary Ware (2016, 2023)
+- Alan Orth (2023)
+- Ashok Argent-Katwala (2020)
+- Bastien Vallet (2020)
+- Chris R Bunney 2020
+- Christian Zagrodnick (2012-2013)
+- Christoph Zwerschke (2013)
+- Jakub Wilk (2020)
+- Janis Kirsteins (2019)
+- Justin Ryan Wagner 2014
+- Kevin Deldycke (2014, 2016)
+- Louis Sautier (2020)
+- Lucas Wiman (2015)
+- Mario Vilas (2014)
+- Michael Howitz (2020)
+- Michał Bielawski (2021, 2023)
+- Michał Górny (2020)
+- Mike Taves (2023)
+- Pedro Ferreira (2013)
+- Stuart Prescott (2021)
+- Victor Mireyev (2016)
+- simon klemenc (2016)
+
+Additional Acknowledgements and Licensing Information:
+- Data in the /src/databases/ and /src/locales/ folders is sourced from the Debian iso-codes project, available at: https://salsa.debian.org/iso-codes-team/iso-codes. This data is used under the terms of the GNU Lesser General Public License Version 2.1 (February 1999).
+
+The Debian iso-codes project is a collection of code lists for different standards, maintained and made available under the GNU Lesser General Public License Version 2.1. We gratefully acknowledge the Debian iso-codes team and contributors for their work and for making this resource freely available.
+
+The full text of the GNU Lesser General Public License Version 2.1 can be found at: https://salsa.debian.org/iso-codes-team/iso-codes/-/blob/main/COPYING.
+
+For the full text of the GNU Lesser General Public License,
+see https://github.com/pycountry/pycountry/blob/main/LICENSE.txt.

.venv/lib/python3.11/site-packages/pycountry/__init__.py

@@ -0,0 +1,303 @@
+"""pycountry"""
+
+import os.path
+import unicodedata
+from importlib import metadata as _importlib_metadata
+from typing import Dict, List, Optional, Type
+
+import pycountry.db
+
+# We prioritise importing the backported `importlib_resources`
+# because the function we use (`importlib.resources.files`) is only
+# available from Python 3.9, but the module itself exists since 3.7.
+# We install `importlib_resources` on Python < 3.9.
+# TODO: Remove usage of importlib_resources once support for Python 3.8 is dropped
+try:
+    import importlib_resources  # type: ignore
+except ModuleNotFoundError:
+    from importlib import resources as importlib_resources  # type: ignore
+
+
+def resource_filename(package_or_requirement: str, resource_name: str) -> str:
+    return str(
+        importlib_resources.files(package_or_requirement) / resource_name
+    )
+
+
+def get_version(distribution_name: str) -> Optional[str]:
+    try:
+        return _importlib_metadata.version(distribution_name)
+    except _importlib_metadata.PackageNotFoundError:
+        return "n/a"
+
+
+# Variable annotations
+LOCALES_DIR: str = resource_filename("pycountry", "locales")
+DATABASE_DIR: str = resource_filename("pycountry", "databases")
+__version__: Optional[str] = get_version("pycountry")
+
+
+def remove_accents(input_str: str) -> str:
+    output_str = input_str
+    if not input_str.isascii():
+        # Borrowed from https://stackoverflow.com/a/517974/1509718
+        nfkd_form = unicodedata.normalize("NFKD", input_str)
+        output_str = "".join(
+            [c for c in nfkd_form if not unicodedata.combining(c)]
+        )
+    return output_str
+
+
+class ExistingCountries(pycountry.db.Database):
+    """Provides access to an ISO 3166 database (Countries)."""
+
+    data_class = pycountry.db.Country
+    root_key = "3166-1"
+
+    def search_fuzzy(self, query: str) -> List[Type["ExistingCountries"]]:
+        query = remove_accents(query.strip().lower())
+
+        # A country-code to points mapping for later sorting countries
+        # based on the query's matching incidence.
+        results: dict[str, int] = {}
+
+        def add_result(country: "pycountry.db.Country", points: int) -> None:
+            results.setdefault(country.alpha_2, 0)
+            results[country.alpha_2] += points
+
+        # Prio 1: exact matches on country names
+        try:
+            add_result(self.lookup(query), 50)
+        except LookupError:
+            pass
+
+        # Prio 2: exact matches on subdivision names
+        match_subdivions = pycountry.Subdivisions.match(
+            self=subdivisions, query=query
+        )
+        for candidate in match_subdivions:
+            add_result(candidate.country, 49)
+
+        # Prio 3: partial matches on country names
+        for candidate in self:
+            # Higher priority for a match on the common name
+            for v in [
+                candidate._fields.get("name"),
+                candidate._fields.get("official_name"),
+                candidate._fields.get("comment"),
+            ]:
+                if v is not None:
+                    v = remove_accents(v.lower())
+                    if query in v:
+                        # This prefers countries with a match early in their name
+                        # and also balances against countries with a number of
+                        # partial matches and their name containing 'new' in the
+                        # middle
+                        add_result(
+                            candidate, max([5, 30 - (2 * v.find(query))])
+                        )
+                        break
+
+        # Prio 4: partial matches on subdivision names
+        partial_match_subdivisions = pycountry.Subdivisions.partial_match(
+            self=subdivisions, query=query
+        )
+        for candidate in partial_match_subdivisions:
+            v = candidate._fields.get("name")
+            v = remove_accents(v.lower())
+            if query in v:
+                add_result(candidate.country, max([1, 5 - v.find(query)]))
+
+        if not results:
+            raise LookupError(query)
+
+        sorted_results = [
+            self.get(alpha_2=x[0])
+            # sort by points first, by alpha2 code second, and to ensure stable
+            # results the negative value allows us to sort reversely on the
+            # points but ascending on the country code.
+            for x in sorted(results.items(), key=lambda x: (-x[1], x[0]))
+        ]
+        return sorted_results
+
+
+class HistoricCountries(ExistingCountries):
+    """Provides access to an ISO 3166-3 database
+    (Countries that have been removed from the standard)."""
+
+    data_class = pycountry.db.Country
+    root_key = "3166-3"
+
+
+class Scripts(pycountry.db.Database):
+    """Provides access to an ISO 15924 database (Scripts)."""
+
+    data_class = "Script"
+    root_key = "15924"
+
+
+class Currencies(pycountry.db.Database):
+    """Provides access to an ISO 4217 database (Currencies)."""
+
+    data_class = "Currency"
+    root_key = "4217"
+
+
+class Languages(pycountry.db.Database):
+    """Provides access to an ISO 639-1/2T/3 database (Languages)."""
+
+    no_index = ["status", "scope", "type", "inverted_name", "common_name"]
+
+    data_class = "Language"
+    root_key = "639-3"
+
+
+class LanguageFamilies(pycountry.db.Database):
+    """Provides access to an ISO 639-5 database
+    (Language Families and Groups)."""
+
+    data_class = "LanguageFamily"
+    root_key = "639-5"
+
+
+class SubdivisionHierarchy(pycountry.db.Data):
+    def __init__(self, **kw):
+        if "parent" in kw:
+            kw["parent_code"] = kw["parent"]
+        else:
+            kw["parent_code"] = None
+        super().__init__(**kw)
+        self.country_code = self.code.split("-")[0]
+        if self.parent_code is not None:
+            # Split the parent_code to check if the country_code is already present
+            parts = self.parent_code.split("-")
+            if parts[0] != self.country_code:
+                self.parent_code = f"{self.country_code}-{self.parent_code}"
+
+    @property
+    def country(self):
+        return countries.get(alpha_2=self.country_code)
+
+    @property
+    def parent(self):
+        if not self.parent_code:
+            return None
+        return subdivisions.get(code=self.parent_code)
+
+
+class Subdivisions(pycountry.db.Database):
+    # Note: subdivisions can be hierarchical to other subdivisions. The
+    # parent_code attribute is related to other subdivisions, *not*
+    # the country!
+
+    data_class = SubdivisionHierarchy
+    no_index = ["name", "parent_code", "parent", "type"]
+    root_key = "3166-2"
+
+    def _load(self, *args, **kw):
+        super()._load(*args, **kw)
+
+        # Add index for the country code.
+        self.indices["country_code"] = {}
+        for subdivision in self:
+            divs = self.indices["country_code"].setdefault(
+                subdivision.country_code.lower(), set()
+            )
+            divs.add(subdivision)
+
+    def get(self, **kw):
+        default = kw.setdefault("default", None)
+        subdivisions = super().get(**kw)
+        if subdivisions is default and "country_code" in kw:
+            # This handles the case where we know about a country but there
+            # are no subdivisions: we return an empty list in this case
+            # (sticking to the expected type here) instead of None.
+            if countries.get(alpha_2=kw["country_code"]) is not None:
+                return []
+        return subdivisions
+
+    def match(self, query):
+        query = remove_accents(query.strip().lower())
+        matching_candidates = []
+        for candidate in subdivisions:
+            for v in candidate._fields.values():
+                if v is not None:
+                    v = remove_accents(v.lower())
+                    # Some names include alternative versions which we want to
+                    # match exactly.
+                    for w in v.split(";"):
+                        if w == query:
+                            matching_candidates.append(candidate)
+                            break
+
+        return matching_candidates
+
+    def partial_match(self, query):
+        query = remove_accents(query.strip().lower())
+        matching_candidates = []
+        for candidate in subdivisions:
+            v = candidate._fields.get("name")
+            v = remove_accents(v.lower())
+            if query in v:
+                matching_candidates.append(candidate)
+
+        return matching_candidates
+
+    def search_fuzzy(self, query: str) -> List[Type["Subdivisions"]]:
+        query = remove_accents(query.strip().lower())
+
+        # A Subdivision's code to points mapping for later sorting subdivisions
+        # based on the query's matching incidence.
+        results: dict[str, int] = {}
+
+        def add_result(
+            subdivision: "pycountry.db.Subdivision", points: int
+        ) -> None:
+            results.setdefault(subdivision.code, 0)
+            results[subdivision.code] += points
+
+        # Prio 1: exact matches on subdivision names
+        match_subdivisions = self.match(query)
+        for candidate in match_subdivisions:
+            add_result(candidate, 50)
+
+        # Prio 2: partial matches on subdivision names
+        partial_match_subdivisions = self.partial_match(query)
+        for candidate in partial_match_subdivisions:
+            v = candidate._fields.get("name")
+            v = remove_accents(v.lower())
+            if query in v:
+                add_result(candidate, max([1, 5 - v.find(query)]))
+
+        if not results:
+            raise LookupError(query)
+
+        sorted_results = [
+            self.get(code=x[0])
+            # sort by points first, by alpha2 code second, and to ensure stable
+            # results the negative value allows us to sort reversely on the
+            # points but ascending on the country code.
+            for x in sorted(results.items(), key=lambda x: (-x[1], x[0]))
+        ]
+        return sorted_results
+
+
+# Initialize instances with type hints
+countries: ExistingCountries = ExistingCountries(
+    os.path.join(DATABASE_DIR, "iso3166-1.json")
+)
+subdivisions: Subdivisions = Subdivisions(
+    os.path.join(DATABASE_DIR, "iso3166-2.json")
+)
+historic_countries: HistoricCountries = HistoricCountries(
+    os.path.join(DATABASE_DIR, "iso3166-3.json")
+)
+
+currencies: Currencies = Currencies(os.path.join(DATABASE_DIR, "iso4217.json"))
+
+languages: Languages = Languages(os.path.join(DATABASE_DIR, "iso639-3.json"))
+language_families: LanguageFamilies = LanguageFamilies(
+    os.path.join(DATABASE_DIR, "iso639-5.json")
+)
+
+scripts: Scripts = Scripts(os.path.join(DATABASE_DIR, "iso15924.json"))

.venv/lib/python3.11/site-packages/pycountry/db.py

@@ -0,0 +1,200 @@
+import json
+import logging
+import threading
+from typing import Any, Iterator, List, Optional, Type, Union
+
+logger = logging.getLogger("pycountry.db")
+
+
+class Data:
+    def __init__(self, **fields: str):
+        self._fields = fields
+
+    def __getattr__(self, key):
+        if key in self._fields:
+            return self._fields[key]
+        raise AttributeError(key)
+
+    def __setattr__(self, key: str, value: str) -> None:
+        if key != "_fields":
+            self._fields[key] = value
+        super().__setattr__(key, value)
+
+    def __repr__(self) -> str:
+        cls_name = self.__class__.__name__
+        fields = ", ".join("%s=%r" % i for i in sorted(self._fields.items()))
+        return f"{cls_name}({fields})"
+
+    def __dir__(self) -> List[str]:
+        return dir(self.__class__) + list(self._fields)
+
+    def __iter__(self):
+        # allow casting into a dict
+        for field in self._fields:
+            yield field, getattr(self, field)
+
+
+class Country(Data):
+    pass
+
+
+class Subdivision(Data):
+    pass
+
+
+def lazy_load(f):
+    def load_if_needed(self, *args, **kw):
+        if not self._is_loaded:
+            with self._load_lock:
+                self._load()
+        return f(self, *args, **kw)
+
+    return load_if_needed
+
+
+class Database:
+    data_class: Union[Type, str]
+    root_key: Optional[str] = None
+    no_index: List[str] = []
+
+    def __init__(self, filename: str) -> None:
+        self.filename = filename
+        self._is_loaded = False
+        self._load_lock = threading.Lock()
+
+        if isinstance(self.data_class, str):
+            self.factory = type(self.data_class, (Data,), {})
+        else:
+            self.factory = self.data_class
+
+    def _clear(self):
+        self._is_loaded = False
+        self.objects = []
+        self.index_names = set()
+        self.indices = {}
+
+    def _load(self) -> None:
+        if self._is_loaded:
+            # Help keeping the _load_if_needed code easier
+            # to read.
+            return
+        self._clear()
+
+        with open(self.filename, encoding="utf-8") as f:
+            tree = json.load(f)
+
+        for entry in tree[self.root_key]:
+            obj = self.factory(**entry)
+            self.objects.append(obj)
+            # Inject into index.
+            for key, value in entry.items():
+                if key in self.no_index:
+                    continue
+                # Lookups and searches are case insensitive. Normalize
+                # here.
+                index = self.indices.setdefault(key, {})
+                value = value.lower()
+                if value in index:
+                    logger.debug(
+                        "%s %r already taken in index %r and will be "
+                        "ignored. This is an error in the databases."
+                        % (self.factory.__name__, value, key)
+                    )
+                index[value] = obj
+
+        self._is_loaded = True
+
+    # Public API
+
+    @lazy_load
+    def add_entry(self, **kw):
+        # create the object with the correct dynamic type
+        obj = self.factory(**kw)
+
+        # append object
+        self.objects.append(obj)
+
+        # update indices
+        for key, value in kw.items():
+            if key in self.no_index:
+                continue
+            value = value.lower()
+            index = self.indices.setdefault(key, {})
+            index[value] = obj
+
+    @lazy_load
+    def remove_entry(self, **kw):
+        # make sure that we receive None if no entry found
+        if "default" in kw:
+            del kw["default"]
+        obj = self.get(**kw)
+        if not obj:
+            raise KeyError(
+                f"{self.factory.__name__} not found and cannot be removed: {kw}"
+            )
+
+        # remove object
+        self.objects.remove(obj)
+
+        # update indices
+        for key, value in obj:
+            if key in self.no_index:
+                continue
+            value = value.lower()
+            index = self.indices.setdefault(key, {})
+            if value in index:
+                del index[value]
+
+    @lazy_load
+    def __iter__(self) -> Iterator["Database"]:
+        return iter(self.objects)
+
+    @lazy_load
+    def __len__(self) -> int:
+        return len(self.objects)
+
+    @lazy_load
+    def get(
+        self, *, default: Optional[Any] = None, **kw: Optional[str]
+    ) -> Optional[Any]:
+        if len(kw) != 1:
+            raise TypeError("Only one criteria may be given")
+        field, value = kw.popitem()
+        if not isinstance(value, str):
+            raise LookupError()
+        # Normalize for case-insensitivity
+        value = value.lower()
+        index = self.indices[field]
+        try:
+            return index[value]
+        except KeyError:
+            # Pythonic APIs implementing     get() shouldn't raise KeyErrors.
+            # Those are a bit unexpected and they should rather support
+            # returning `None` by default and allow customization.
+            return default
+
+    @lazy_load
+    def lookup(self, value: str) -> Type:
+        if not isinstance(value, str):
+            raise LookupError()
+
+        # Normalize for case-insensitivity
+        value = value.lower()
+
+        # Use indexes first
+        for key in self.indices:
+            try:
+                return self.indices[key][value]
+            except LookupError:
+                pass
+
+        # Use non-indexed values now. Avoid going through indexed values.
+        for candidate in self:
+            for k in self.no_index:
+                v = candidate._fields.get(k)
+                if v is None:
+                    continue
+                if v.lower() == value:
+                    return candidate
+
+        raise LookupError("Could not find a record for %r" % value)