diff --git a/bert-master/bert-master.gitignore b/bert-master/bert-master.gitignore
new file mode 100644
index 0000000000000000000000000000000000000000..df9efad4590305cbe4a014ed0c7b179bb337800c
--- /dev/null
+++ b/bert-master/bert-master.gitignore
@@ -0,0 +1,116 @@
+# Initially taken from Github's Python gitignore file
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
diff --git a/bert-master/bert-master/CONTRIBUTING.md b/bert-master/bert-master/CONTRIBUTING.md
new file mode 100644
index 0000000000000000000000000000000000000000..124b4b32c2546f076b2b15768c86607b0448235a
--- /dev/null
+++ b/bert-master/bert-master/CONTRIBUTING.md
@@ -0,0 +1,31 @@
+# How to Contribute
+
+BERT needs to maintain permanent compatibility with the pre-trained model files,
+so we do not plan to make any major changes to this library (other than what was
+promised in the README). However, we can accept small patches related to
+re-factoring and documentation. To submit contributes, there are just a few
+small guidelines you need to follow.
+
+## Contributor License Agreement
+
+Contributions to this project must be accompanied by a Contributor License
+Agreement. You (or your employer) retain the copyright to your contribution;
+this simply gives us permission to use and redistribute your contributions as
+part of the project. Head over to to see
+your current agreements on file or to sign a new one.
+
+You generally only need to submit a CLA once, so if you've already submitted one
+(even if it was for a different project), you probably don't need to do it
+again.
+
+## Code reviews
+
+All submissions, including submissions by project members, require review. We
+use GitHub pull requests for this purpose. Consult
+[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more
+information on using pull requests.
+
+## Community Guidelines
+
+This project follows
+[Google's Open Source Community Guidelines](https://opensource.google.com/conduct/).
diff --git a/bert-master/bert-master/LICENSE b/bert-master/bert-master/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..d645695673349e3947e8e5ae42332d0ac3164cd7
--- /dev/null
+++ b/bert-master/bert-master/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/bert-master/bert-master/README.md b/bert-master/bert-master/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..da8f792a04cbdcc3a6c5baaee5c579a749b4f4c7
--- /dev/null
+++ b/bert-master/bert-master/README.md
@@ -0,0 +1,1117 @@
+# BERT
+
+**\*\*\*\*\* New March 11th, 2020: Smaller BERT Models \*\*\*\*\***
+
+This is a release of 24 smaller BERT models (English only, uncased, trained with WordPiece masking) referenced in [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://arxiv.org/abs/1908.08962).
+
+We have shown that the standard BERT recipe (including model architecture and training objective) is effective on a wide range of model sizes, beyond BERT-Base and BERT-Large. The smaller BERT models are intended for environments with restricted computational resources. They can be fine-tuned in the same manner as the original BERT models. However, they are most effective in the context of knowledge distillation, where the fine-tuning labels are produced by a larger and more accurate teacher.
+
+Our goal is to enable research in institutions with fewer computational resources and encourage the community to seek directions of innovation alternative to increasing model capacity.
+
+You can download all 24 from [here][all], or individually from the table below:
+
+| |H=128|H=256|H=512|H=768|
+|---|:---:|:---:|:---:|:---:|
+| **L=2** |[**2/128 (BERT-Tiny)**][2_128]|[2/256][2_256]|[2/512][2_512]|[2/768][2_768]|
+| **L=4** |[4/128][4_128]|[**4/256 (BERT-Mini)**][4_256]|[**4/512 (BERT-Small)**][4_512]|[4/768][4_768]|
+| **L=6** |[6/128][6_128]|[6/256][6_256]|[6/512][6_512]|[6/768][6_768]|
+| **L=8** |[8/128][8_128]|[8/256][8_256]|[**8/512 (BERT-Medium)**][8_512]|[8/768][8_768]|
+| **L=10** |[10/128][10_128]|[10/256][10_256]|[10/512][10_512]|[10/768][10_768]|
+| **L=12** |[12/128][12_128]|[12/256][12_256]|[12/512][12_512]|[**12/768 (BERT-Base)**][12_768]|
+
+Note that the BERT-Base model in this release is included for completeness only; it was re-trained under the same regime as the original model.
+
+Here are the corresponding GLUE scores on the test set:
+
+|Model|Score|CoLA|SST-2|MRPC|STS-B|QQP|MNLI-m|MNLI-mm|QNLI(v2)|RTE|WNLI|AX|
+|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+|BERT-Tiny|64.2|0.0|83.2|81.1/71.1|74.3/73.6|62.2/83.4|70.2|70.3|81.5|57.2|62.3|21.0|
+|BERT-Mini|65.8|0.0|85.9|81.1/71.8|75.4/73.3|66.4/86.2|74.8|74.3|84.1|57.9|62.3|26.1|
+|BERT-Small|71.2|27.8|89.7|83.4/76.2|78.8/77.0|68.1/87.0|77.6|77.0|86.4|61.8|62.3|28.6|
+|BERT-Medium|73.5|38.0|89.6|86.6/81.6|80.4/78.4|69.6/87.9|80.0|79.1|87.7|62.2|62.3|30.5|
+
+For each task, we selected the best fine-tuning hyperparameters from the lists below, and trained for 4 epochs:
+- batch sizes: 8, 16, 32, 64, 128
+- learning rates: 3e-4, 1e-4, 5e-5, 3e-5
+
+If you use these models, please cite the following paper:
+
+```
+@article{turc2019,
+ title={Well-Read Students Learn Better: On the Importance of Pre-training Compact Models},
+ author={Turc, Iulia and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
+ journal={arXiv preprint arXiv:1908.08962v2 },
+ year={2019}
+}
+```
+
+[2_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-128_A-2.zip
+[2_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-256_A-4.zip
+[2_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-512_A-8.zip
+[2_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-2_H-768_A-12.zip
+[4_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-128_A-2.zip
+[4_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-256_A-4.zip
+[4_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-512_A-8.zip
+[4_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-4_H-768_A-12.zip
+[6_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-128_A-2.zip
+[6_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-256_A-4.zip
+[6_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-512_A-8.zip
+[6_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-6_H-768_A-12.zip
+[8_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-128_A-2.zip
+[8_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-256_A-4.zip
+[8_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-512_A-8.zip
+[8_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-8_H-768_A-12.zip
+[10_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-128_A-2.zip
+[10_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-256_A-4.zip
+[10_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-512_A-8.zip
+[10_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-10_H-768_A-12.zip
+[12_128]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-128_A-2.zip
+[12_256]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-256_A-4.zip
+[12_512]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-512_A-8.zip
+[12_768]: https://storage.googleapis.com/bert_models/2020_02_20/uncased_L-12_H-768_A-12.zip
+[all]: https://storage.googleapis.com/bert_models/2020_02_20/all_bert_models.zip
+
+**\*\*\*\*\* New May 31st, 2019: Whole Word Masking Models \*\*\*\*\***
+
+This is a release of several new models which were the result of an improvement
+the pre-processing code.
+
+In the original pre-processing code, we randomly select WordPiece tokens to
+mask. For example:
+
+`Input Text: the man jumped up , put his basket on phil ##am ##mon ' s head`
+`Original Masked Input: [MASK] man [MASK] up , put his [MASK] on phil
+[MASK] ##mon ' s head`
+
+The new technique is called Whole Word Masking. In this case, we always mask
+*all* of the the tokens corresponding to a word at once. The overall masking
+rate remains the same.
+
+`Whole Word Masked Input: the man [MASK] up , put his basket on [MASK] [MASK]
+[MASK] ' s head`
+
+The training is identical -- we still predict each masked WordPiece token
+independently. The improvement comes from the fact that the original prediction
+task was too 'easy' for words that had been split into multiple WordPieces.
+
+This can be enabled during data generation by passing the flag
+`--do_whole_word_mask=True` to `create_pretraining_data.py`.
+
+Pre-trained models with Whole Word Masking are linked below. The data and
+training were otherwise identical, and the models have identical structure and
+vocab to the original models. We only include BERT-Large models. When using
+these models, please make it clear in the paper that you are using the Whole
+Word Masking variant of BERT-Large.
+
+* **[`BERT-Large, Uncased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+
+* **[`BERT-Large, Cased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+
+Model | SQUAD 1.1 F1/EM | Multi NLI Accuracy
+---------------------------------------- | :-------------: | :----------------:
+BERT-Large, Uncased (Original) | 91.0/84.3 | 86.05
+BERT-Large, Uncased (Whole Word Masking) | 92.8/86.7 | 87.07
+BERT-Large, Cased (Original) | 91.5/84.8 | 86.09
+BERT-Large, Cased (Whole Word Masking) | 92.9/86.7 | 86.46
+
+**\*\*\*\*\* New February 7th, 2019: TfHub Module \*\*\*\*\***
+
+BERT has been uploaded to [TensorFlow Hub](https://tfhub.dev). See
+`run_classifier_with_tfhub.py` for an example of how to use the TF Hub module,
+or run an example in the browser on
+[Colab](https://colab.sandbox.google.com/github/google-research/bert/blob/master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb).
+
+**\*\*\*\*\* New November 23rd, 2018: Un-normalized multilingual model + Thai +
+Mongolian \*\*\*\*\***
+
+We uploaded a new multilingual model which does *not* perform any normalization
+on the input (no lower casing, accent stripping, or Unicode normalization), and
+additionally inclues Thai and Mongolian.
+
+**It is recommended to use this version for developing multilingual models,
+especially on languages with non-Latin alphabets.**
+
+This does not require any code changes, and can be downloaded here:
+
+* **[`BERT-Base, Multilingual Cased`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:
+ 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
+
+**\*\*\*\*\* New November 15th, 2018: SOTA SQuAD 2.0 System \*\*\*\*\***
+
+We released code changes to reproduce our 83% F1 SQuAD 2.0 system, which is
+currently 1st place on the leaderboard by 3%. See the SQuAD 2.0 section of the
+README for details.
+
+**\*\*\*\*\* New November 5th, 2018: Third-party PyTorch and Chainer versions of
+BERT available \*\*\*\*\***
+
+NLP researchers from HuggingFace made a
+[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)
+which is compatible with our pre-trained checkpoints and is able to reproduce
+our results. Sosuke Kobayashi also made a
+[Chainer version of BERT available](https://github.com/soskek/bert-chainer)
+(Thanks!) We were not involved in the creation or maintenance of the PyTorch
+implementation so please direct any questions towards the authors of that
+repository.
+
+**\*\*\*\*\* New November 3rd, 2018: Multilingual and Chinese models available
+\*\*\*\*\***
+
+We have made two new BERT models available:
+
+* **[`BERT-Base, Multilingual`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)
+ (Not recommended, use `Multilingual Cased` instead)**: 102 languages,
+ 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:
+ Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
+ parameters
+
+We use character-based tokenization for Chinese, and WordPiece tokenization for
+all other languages. Both models should work out-of-the-box without any code
+changes. We did update the implementation of `BasicTokenizer` in
+`tokenization.py` to support Chinese character tokenization, so please update if
+you forked it. However, we did not change the tokenization API.
+
+For more, see the
+[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).
+
+**\*\*\*\*\* End new information \*\*\*\*\***
+
+## Introduction
+
+**BERT**, or **B**idirectional **E**ncoder **R**epresentations from
+**T**ransformers, is a new method of pre-training language representations which
+obtains state-of-the-art results on a wide array of Natural Language Processing
+(NLP) tasks.
+
+Our academic paper which describes BERT in detail and provides full results on a
+number of tasks can be found here:
+[https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805).
+
+To give a few numbers, here are the results on the
+[SQuAD v1.1](https://rajpurkar.github.io/SQuAD-explorer/) question answering
+task:
+
+SQuAD v1.1 Leaderboard (Oct 8th 2018) | Test EM | Test F1
+------------------------------------- | :------: | :------:
+1st Place Ensemble - BERT | **87.4** | **93.2**
+2nd Place Ensemble - nlnet | 86.0 | 91.7
+1st Place Single Model - BERT | **85.1** | **91.8**
+2nd Place Single Model - nlnet | 83.5 | 90.1
+
+And several natural language inference tasks:
+
+System | MultiNLI | Question NLI | SWAG
+----------------------- | :------: | :----------: | :------:
+BERT | **86.7** | **91.1** | **86.3**
+OpenAI GPT (Prev. SOTA) | 82.2 | 88.1 | 75.0
+
+Plus many other tasks.
+
+Moreover, these results were all obtained with almost no task-specific neural
+network architecture design.
+
+If you already know what BERT is and you just want to get started, you can
+[download the pre-trained models](#pre-trained-models) and
+[run a state-of-the-art fine-tuning](#fine-tuning-with-bert) in only a few
+minutes.
+
+## What is BERT?
+
+BERT is a method of pre-training language representations, meaning that we train
+a general-purpose "language understanding" model on a large text corpus (like
+Wikipedia), and then use that model for downstream NLP tasks that we care about
+(like question answering). BERT outperforms previous methods because it is the
+first *unsupervised*, *deeply bidirectional* system for pre-training NLP.
+
+*Unsupervised* means that BERT was trained using only a plain text corpus, which
+is important because an enormous amount of plain text data is publicly available
+on the web in many languages.
+
+Pre-trained representations can also either be *context-free* or *contextual*,
+and contextual representations can further be *unidirectional* or
+*bidirectional*. Context-free models such as
+[word2vec](https://www.tensorflow.org/tutorials/representation/word2vec) or
+[GloVe](https://nlp.stanford.edu/projects/glove/) generate a single "word
+embedding" representation for each word in the vocabulary, so `bank` would have
+the same representation in `bank deposit` and `river bank`. Contextual models
+instead generate a representation of each word that is based on the other words
+in the sentence.
+
+BERT was built upon recent work in pre-training contextual representations —
+including [Semi-supervised Sequence Learning](https://arxiv.org/abs/1511.01432),
+[Generative Pre-Training](https://blog.openai.com/language-unsupervised/),
+[ELMo](https://allennlp.org/elmo), and
+[ULMFit](http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html)
+— but crucially these models are all *unidirectional* or *shallowly
+bidirectional*. This means that each word is only contextualized using the words
+to its left (or right). For example, in the sentence `I made a bank deposit` the
+unidirectional representation of `bank` is only based on `I made a` but not
+`deposit`. Some previous work does combine the representations from separate
+left-context and right-context models, but only in a "shallow" manner. BERT
+represents "bank" using both its left and right context — `I made a ... deposit`
+— starting from the very bottom of a deep neural network, so it is *deeply
+bidirectional*.
+
+BERT uses a simple approach for this: We mask out 15% of the words in the input,
+run the entire sequence through a deep bidirectional
+[Transformer](https://arxiv.org/abs/1706.03762) encoder, and then predict only
+the masked words. For example:
+
+```
+Input: the man went to the [MASK1] . he bought a [MASK2] of milk.
+Labels: [MASK1] = store; [MASK2] = gallon
+```
+
+In order to learn relationships between sentences, we also train on a simple
+task which can be generated from any monolingual corpus: Given two sentences `A`
+and `B`, is `B` the actual next sentence that comes after `A`, or just a random
+sentence from the corpus?
+
+```
+Sentence A: the man went to the store .
+Sentence B: he bought a gallon of milk .
+Label: IsNextSentence
+```
+
+```
+Sentence A: the man went to the store .
+Sentence B: penguins are flightless .
+Label: NotNextSentence
+```
+
+We then train a large model (12-layer to 24-layer Transformer) on a large corpus
+(Wikipedia + [BookCorpus](http://yknzhu.wixsite.com/mbweb)) for a long time (1M
+update steps), and that's BERT.
+
+Using BERT has two stages: *Pre-training* and *fine-tuning*.
+
+**Pre-training** is fairly expensive (four days on 4 to 16 Cloud TPUs), but is a
+one-time procedure for each language (current models are English-only, but
+multilingual models will be released in the near future). We are releasing a
+number of pre-trained models from the paper which were pre-trained at Google.
+Most NLP researchers will never need to pre-train their own model from scratch.
+
+**Fine-tuning** is inexpensive. All of the results in the paper can be
+replicated in at most 1 hour on a single Cloud TPU, or a few hours on a GPU,
+starting from the exact same pre-trained model. SQuAD, for example, can be
+trained in around 30 minutes on a single Cloud TPU to achieve a Dev F1 score of
+91.0%, which is the single system state-of-the-art.
+
+The other important aspect of BERT is that it can be adapted to many types of
+NLP tasks very easily. In the paper, we demonstrate state-of-the-art results on
+sentence-level (e.g., SST-2), sentence-pair-level (e.g., MultiNLI), word-level
+(e.g., NER), and span-level (e.g., SQuAD) tasks with almost no task-specific
+modifications.
+
+## What has been released in this repository?
+
+We are releasing the following:
+
+* TensorFlow code for the BERT model architecture (which is mostly a standard
+ [Transformer](https://arxiv.org/abs/1706.03762) architecture).
+* Pre-trained checkpoints for both the lowercase and cased version of
+ `BERT-Base` and `BERT-Large` from the paper.
+* TensorFlow code for push-button replication of the most important
+ fine-tuning experiments from the paper, including SQuAD, MultiNLI, and MRPC.
+
+All of the code in this repository works out-of-the-box with CPU, GPU, and Cloud
+TPU.
+
+## Pre-trained models
+
+We are releasing the `BERT-Base` and `BERT-Large` models from the paper.
+`Uncased` means that the text has been lowercased before WordPiece tokenization,
+e.g., `John Smith` becomes `john smith`. The `Uncased` model also strips out any
+accent markers. `Cased` means that the true case and accent markers are
+preserved. Typically, the `Uncased` model is better unless you know that case
+information is important for your task (e.g., Named Entity Recognition or
+Part-of-Speech tagging).
+
+These models are all released under the same license as the source code (Apache
+2.0).
+
+For information about the Multilingual and Chinese model, see the
+[Multilingual README](https://github.com/google-research/bert/blob/master/multilingual.md).
+
+**When using a cased model, make sure to pass `--do_lower=False` to the training
+scripts. (Or pass `do_lower_case=False` directly to `FullTokenizer` if you're
+using your own script.)**
+
+The links to the models are here (right-click, 'Save link as...' on the name):
+
+* **[`BERT-Large, Uncased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+* **[`BERT-Large, Cased (Whole Word Masking)`](https://storage.googleapis.com/bert_models/2019_05_30/wwm_cased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+* **[`BERT-Base, Uncased`](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-12_H-768_A-12.zip)**:
+ 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Large, Uncased`](https://storage.googleapis.com/bert_models/2018_10_18/uncased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+* **[`BERT-Base, Cased`](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-12_H-768_A-12.zip)**:
+ 12-layer, 768-hidden, 12-heads , 110M parameters
+* **[`BERT-Large, Cased`](https://storage.googleapis.com/bert_models/2018_10_18/cased_L-24_H-1024_A-16.zip)**:
+ 24-layer, 1024-hidden, 16-heads, 340M parameters
+* **[`BERT-Base, Multilingual Cased (New, recommended)`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:
+ 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Base, Multilingual Uncased (Orig, not recommended)`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)
+ (Not recommended, use `Multilingual Cased` instead)**: 102 languages,
+ 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:
+ Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
+ parameters
+
+Each .zip file contains three items:
+
+* A TensorFlow checkpoint (`bert_model.ckpt`) containing the pre-trained
+ weights (which is actually 3 files).
+* A vocab file (`vocab.txt`) to map WordPiece to word id.
+* A config file (`bert_config.json`) which specifies the hyperparameters of
+ the model.
+
+## Fine-tuning with BERT
+
+**Important**: All results on the paper were fine-tuned on a single Cloud TPU,
+which has 64GB of RAM. It is currently not possible to re-produce most of the
+`BERT-Large` results on the paper using a GPU with 12GB - 16GB of RAM, because
+the maximum batch size that can fit in memory is too small. We are working on
+adding code to this repository which allows for much larger effective batch size
+on the GPU. See the section on [out-of-memory issues](#out-of-memory-issues) for
+more details.
+
+This code was tested with TensorFlow 1.11.0. It was tested with Python2 and
+Python3 (but more thoroughly with Python2, since this is what's used internally
+in Google).
+
+The fine-tuning examples which use `BERT-Base` should be able to run on a GPU
+that has at least 12GB of RAM using the hyperparameters given.
+
+### Fine-tuning with Cloud TPUs
+
+Most of the examples below assumes that you will be running training/evaluation
+on your local machine, using a GPU like a Titan X or GTX 1080.
+
+However, if you have access to a Cloud TPU that you want to train on, just add
+the following flags to `run_classifier.py` or `run_squad.py`:
+
+```
+ --use_tpu=True \
+ --tpu_name=$TPU_NAME
+```
+
+Please see the
+[Google Cloud TPU tutorial](https://cloud.google.com/tpu/docs/tutorials/mnist)
+for how to use Cloud TPUs. Alternatively, you can use the Google Colab notebook
+"[BERT FineTuning with Cloud TPUs](https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)".
+
+On Cloud TPUs, the pretrained model and the output directory will need to be on
+Google Cloud Storage. For example, if you have a bucket named `some_bucket`, you
+might use the following flags instead:
+
+```
+ --output_dir=gs://some_bucket/my_output_dir/
+```
+
+The unzipped pre-trained model files can also be found in the Google Cloud
+Storage folder `gs://bert_models/2018_10_18`. For example:
+
+```
+export BERT_BASE_DIR=gs://bert_models/2018_10_18/uncased_L-12_H-768_A-12
+```
+
+### Sentence (and sentence-pair) classification tasks
+
+Before running this example you must download the
+[GLUE data](https://gluebenchmark.com/tasks) by running
+[this script](https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e)
+and unpack it to some directory `$GLUE_DIR`. Next, download the `BERT-Base`
+checkpoint and unzip it to some directory `$BERT_BASE_DIR`.
+
+This example code fine-tunes `BERT-Base` on the Microsoft Research Paraphrase
+Corpus (MRPC) corpus, which only contains 3,600 examples and can fine-tune in a
+few minutes on most GPUs.
+
+```shell
+export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12
+export GLUE_DIR=/path/to/glue
+
+python run_classifier.py \
+ --task_name=MRPC \
+ --do_train=true \
+ --do_eval=true \
+ --data_dir=$GLUE_DIR/MRPC \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
+ --max_seq_length=128 \
+ --train_batch_size=32 \
+ --learning_rate=2e-5 \
+ --num_train_epochs=3.0 \
+ --output_dir=/tmp/mrpc_output/
+```
+
+You should see output like this:
+
+```
+***** Eval results *****
+ eval_accuracy = 0.845588
+ eval_loss = 0.505248
+ global_step = 343
+ loss = 0.505248
+```
+
+This means that the Dev set accuracy was 84.55%. Small sets like MRPC have a
+high variance in the Dev set accuracy, even when starting from the same
+pre-training checkpoint. If you re-run multiple times (making sure to point to
+different `output_dir`), you should see results between 84% and 88%.
+
+A few other pre-trained models are implemented off-the-shelf in
+`run_classifier.py`, so it should be straightforward to follow those examples to
+use BERT for any single-sentence or sentence-pair classification task.
+
+Note: You might see a message `Running train on CPU`. This really just means
+that it's running on something other than a Cloud TPU, which includes a GPU.
+
+#### Prediction from classifier
+
+Once you have trained your classifier you can use it in inference mode by using
+the --do_predict=true command. You need to have a file named test.tsv in the
+input folder. Output will be created in file called test_results.tsv in the
+output folder. Each line will contain output for each sample, columns are the
+class probabilities.
+
+```shell
+export BERT_BASE_DIR=/path/to/bert/uncased_L-12_H-768_A-12
+export GLUE_DIR=/path/to/glue
+export TRAINED_CLASSIFIER=/path/to/fine/tuned/classifier
+
+python run_classifier.py \
+ --task_name=MRPC \
+ --do_predict=true \
+ --data_dir=$GLUE_DIR/MRPC \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$TRAINED_CLASSIFIER \
+ --max_seq_length=128 \
+ --output_dir=/tmp/mrpc_output/
+```
+
+### SQuAD 1.1
+
+The Stanford Question Answering Dataset (SQuAD) is a popular question answering
+benchmark dataset. BERT (at the time of the release) obtains state-of-the-art
+results on SQuAD with almost no task-specific network architecture modifications
+or data augmentation. However, it does require semi-complex data pre-processing
+and post-processing to deal with (a) the variable-length nature of SQuAD context
+paragraphs, and (b) the character-level answer annotations which are used for
+SQuAD training. This processing is implemented and documented in `run_squad.py`.
+
+To run on SQuAD, you will first need to download the dataset. The
+[SQuAD website](https://rajpurkar.github.io/SQuAD-explorer/) does not seem to
+link to the v1.1 datasets any longer, but the necessary files can be found here:
+
+* [train-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v1.1.json)
+* [dev-v1.1.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json)
+* [evaluate-v1.1.py](https://github.com/allenai/bi-att-flow/blob/master/squad/evaluate-v1.1.py)
+
+Download these to some directory `$SQUAD_DIR`.
+
+The state-of-the-art SQuAD results from the paper currently cannot be reproduced
+on a 12GB-16GB GPU due to memory constraints (in fact, even batch size 1 does
+not seem to fit on a 12GB GPU using `BERT-Large`). However, a reasonably strong
+`BERT-Base` model can be trained on the GPU with these hyperparameters:
+
+```shell
+python run_squad.py \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
+ --do_train=True \
+ --train_file=$SQUAD_DIR/train-v1.1.json \
+ --do_predict=True \
+ --predict_file=$SQUAD_DIR/dev-v1.1.json \
+ --train_batch_size=12 \
+ --learning_rate=3e-5 \
+ --num_train_epochs=2.0 \
+ --max_seq_length=384 \
+ --doc_stride=128 \
+ --output_dir=/tmp/squad_base/
+```
+
+The dev set predictions will be saved into a file called `predictions.json` in
+the `output_dir`:
+
+```shell
+python $SQUAD_DIR/evaluate-v1.1.py $SQUAD_DIR/dev-v1.1.json ./squad/predictions.json
+```
+
+Which should produce an output like this:
+
+```shell
+{"f1": 88.41249612335034, "exact_match": 81.2488174077578}
+```
+
+You should see a result similar to the 88.5% reported in the paper for
+`BERT-Base`.
+
+If you have access to a Cloud TPU, you can train with `BERT-Large`. Here is a
+set of hyperparameters (slightly different than the paper) which consistently
+obtain around 90.5%-91.0% F1 single-system trained only on SQuAD:
+
+```shell
+python run_squad.py \
+ --vocab_file=$BERT_LARGE_DIR/vocab.txt \
+ --bert_config_file=$BERT_LARGE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \
+ --do_train=True \
+ --train_file=$SQUAD_DIR/train-v1.1.json \
+ --do_predict=True \
+ --predict_file=$SQUAD_DIR/dev-v1.1.json \
+ --train_batch_size=24 \
+ --learning_rate=3e-5 \
+ --num_train_epochs=2.0 \
+ --max_seq_length=384 \
+ --doc_stride=128 \
+ --output_dir=gs://some_bucket/squad_large/ \
+ --use_tpu=True \
+ --tpu_name=$TPU_NAME
+```
+
+For example, one random run with these parameters produces the following Dev
+scores:
+
+```shell
+{"f1": 90.87081895814865, "exact_match": 84.38978240302744}
+```
+
+If you fine-tune for one epoch on
+[TriviaQA](http://nlp.cs.washington.edu/triviaqa/) before this the results will
+be even better, but you will need to convert TriviaQA into the SQuAD json
+format.
+
+### SQuAD 2.0
+
+This model is also implemented and documented in `run_squad.py`.
+
+To run on SQuAD 2.0, you will first need to download the dataset. The necessary
+files can be found here:
+
+* [train-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json)
+* [dev-v2.0.json](https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json)
+* [evaluate-v2.0.py](https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/)
+
+Download these to some directory `$SQUAD_DIR`.
+
+On Cloud TPU you can run with BERT-Large as follows:
+
+```shell
+python run_squad.py \
+ --vocab_file=$BERT_LARGE_DIR/vocab.txt \
+ --bert_config_file=$BERT_LARGE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \
+ --do_train=True \
+ --train_file=$SQUAD_DIR/train-v2.0.json \
+ --do_predict=True \
+ --predict_file=$SQUAD_DIR/dev-v2.0.json \
+ --train_batch_size=24 \
+ --learning_rate=3e-5 \
+ --num_train_epochs=2.0 \
+ --max_seq_length=384 \
+ --doc_stride=128 \
+ --output_dir=gs://some_bucket/squad_large/ \
+ --use_tpu=True \
+ --tpu_name=$TPU_NAME \
+ --version_2_with_negative=True
+```
+
+We assume you have copied everything from the output directory to a local
+directory called ./squad/. The initial dev set predictions will be at
+./squad/predictions.json and the differences between the score of no answer ("")
+and the best non-null answer for each question will be in the file
+./squad/null_odds.json
+
+Run this script to tune a threshold for predicting null versus non-null answers:
+
+python $SQUAD_DIR/evaluate-v2.0.py $SQUAD_DIR/dev-v2.0.json
+./squad/predictions.json --na-prob-file ./squad/null_odds.json
+
+Assume the script outputs "best_f1_thresh" THRESH. (Typical values are between
+-1.0 and -5.0). You can now re-run the model to generate predictions with the
+derived threshold or alternatively you can extract the appropriate answers from
+./squad/nbest_predictions.json.
+
+```shell
+python run_squad.py \
+ --vocab_file=$BERT_LARGE_DIR/vocab.txt \
+ --bert_config_file=$BERT_LARGE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_LARGE_DIR/bert_model.ckpt \
+ --do_train=False \
+ --train_file=$SQUAD_DIR/train-v2.0.json \
+ --do_predict=True \
+ --predict_file=$SQUAD_DIR/dev-v2.0.json \
+ --train_batch_size=24 \
+ --learning_rate=3e-5 \
+ --num_train_epochs=2.0 \
+ --max_seq_length=384 \
+ --doc_stride=128 \
+ --output_dir=gs://some_bucket/squad_large/ \
+ --use_tpu=True \
+ --tpu_name=$TPU_NAME \
+ --version_2_with_negative=True \
+ --null_score_diff_threshold=$THRESH
+```
+
+### Out-of-memory issues
+
+All experiments in the paper were fine-tuned on a Cloud TPU, which has 64GB of
+device RAM. Therefore, when using a GPU with 12GB - 16GB of RAM, you are likely
+to encounter out-of-memory issues if you use the same hyperparameters described
+in the paper.
+
+The factors that affect memory usage are:
+
+* **`max_seq_length`**: The released models were trained with sequence lengths
+ up to 512, but you can fine-tune with a shorter max sequence length to save
+ substantial memory. This is controlled by the `max_seq_length` flag in our
+ example code.
+
+* **`train_batch_size`**: The memory usage is also directly proportional to
+ the batch size.
+
+* **Model type, `BERT-Base` vs. `BERT-Large`**: The `BERT-Large` model
+ requires significantly more memory than `BERT-Base`.
+
+* **Optimizer**: The default optimizer for BERT is Adam, which requires a lot
+ of extra memory to store the `m` and `v` vectors. Switching to a more memory
+ efficient optimizer can reduce memory usage, but can also affect the
+ results. We have not experimented with other optimizers for fine-tuning.
+
+Using the default training scripts (`run_classifier.py` and `run_squad.py`), we
+benchmarked the maximum batch size on single Titan X GPU (12GB RAM) with
+TensorFlow 1.11.0:
+
+System | Seq Length | Max Batch Size
+------------ | ---------- | --------------
+`BERT-Base` | 64 | 64
+... | 128 | 32
+... | 256 | 16
+... | 320 | 14
+... | 384 | 12
+... | 512 | 6
+`BERT-Large` | 64 | 12
+... | 128 | 6
+... | 256 | 2
+... | 320 | 1
+... | 384 | 0
+... | 512 | 0
+
+Unfortunately, these max batch sizes for `BERT-Large` are so small that they
+will actually harm the model accuracy, regardless of the learning rate used. We
+are working on adding code to this repository which will allow much larger
+effective batch sizes to be used on the GPU. The code will be based on one (or
+both) of the following techniques:
+
+* **Gradient accumulation**: The samples in a minibatch are typically
+ independent with respect to gradient computation (excluding batch
+ normalization, which is not used here). This means that the gradients of
+ multiple smaller minibatches can be accumulated before performing the weight
+ update, and this will be exactly equivalent to a single larger update.
+
+* [**Gradient checkpointing**](https://github.com/openai/gradient-checkpointing):
+ The major use of GPU/TPU memory during DNN training is caching the
+ intermediate activations in the forward pass that are necessary for
+ efficient computation in the backward pass. "Gradient checkpointing" trades
+ memory for compute time by re-computing the activations in an intelligent
+ way.
+
+**However, this is not implemented in the current release.**
+
+## Using BERT to extract fixed feature vectors (like ELMo)
+
+In certain cases, rather than fine-tuning the entire pre-trained model
+end-to-end, it can be beneficial to obtained *pre-trained contextual
+embeddings*, which are fixed contextual representations of each input token
+generated from the hidden layers of the pre-trained model. This should also
+mitigate most of the out-of-memory issues.
+
+As an example, we include the script `extract_features.py` which can be used
+like this:
+
+```shell
+# Sentence A and Sentence B are separated by the ||| delimiter for sentence
+# pair tasks like question answering and entailment.
+# For single sentence inputs, put one sentence per line and DON'T use the
+# delimiter.
+echo 'Who was Jim Henson ? ||| Jim Henson was a puppeteer' > /tmp/input.txt
+
+python extract_features.py \
+ --input_file=/tmp/input.txt \
+ --output_file=/tmp/output.jsonl \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
+ --layers=-1,-2,-3,-4 \
+ --max_seq_length=128 \
+ --batch_size=8
+```
+
+This will create a JSON file (one line per line of input) containing the BERT
+activations from each Transformer layer specified by `layers` (-1 is the final
+hidden layer of the Transformer, etc.)
+
+Note that this script will produce very large output files (by default, around
+15kb for every input token).
+
+If you need to maintain alignment between the original and tokenized words (for
+projecting training labels), see the [Tokenization](#tokenization) section
+below.
+
+**Note:** You may see a message like `Could not find trained model in model_dir:
+/tmp/tmpuB5g5c, running initialization to predict.` This message is expected, it
+just means that we are using the `init_from_checkpoint()` API rather than the
+saved model API. If you don't specify a checkpoint or specify an invalid
+checkpoint, this script will complain.
+
+## Tokenization
+
+For sentence-level tasks (or sentence-pair) tasks, tokenization is very simple.
+Just follow the example code in `run_classifier.py` and `extract_features.py`.
+The basic procedure for sentence-level tasks is:
+
+1. Instantiate an instance of `tokenizer = tokenization.FullTokenizer`
+
+2. Tokenize the raw text with `tokens = tokenizer.tokenize(raw_text)`.
+
+3. Truncate to the maximum sequence length. (You can use up to 512, but you
+ probably want to use shorter if possible for memory and speed reasons.)
+
+4. Add the `[CLS]` and `[SEP]` tokens in the right place.
+
+Word-level and span-level tasks (e.g., SQuAD and NER) are more complex, since
+you need to maintain alignment between your input text and output text so that
+you can project your training labels. SQuAD is a particularly complex example
+because the input labels are *character*-based, and SQuAD paragraphs are often
+longer than our maximum sequence length. See the code in `run_squad.py` to show
+how we handle this.
+
+Before we describe the general recipe for handling word-level tasks, it's
+important to understand what exactly our tokenizer is doing. It has three main
+steps:
+
+1. **Text normalization**: Convert all whitespace characters to spaces, and
+ (for the `Uncased` model) lowercase the input and strip out accent markers.
+ E.g., `John Johanson's, → john johanson's,`.
+
+2. **Punctuation splitting**: Split *all* punctuation characters on both sides
+ (i.e., add whitespace around all punctuation characters). Punctuation
+ characters are defined as (a) Anything with a `P*` Unicode class, (b) any
+ non-letter/number/space ASCII character (e.g., characters like `$` which are
+ technically not punctuation). E.g., `john johanson's, → john johanson ' s ,`
+
+3. **WordPiece tokenization**: Apply whitespace tokenization to the output of
+ the above procedure, and apply
+ [WordPiece](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder.py)
+ tokenization to each token separately. (Our implementation is directly based
+ on the one from `tensor2tensor`, which is linked). E.g., `john johanson ' s
+ , → john johan ##son ' s ,`
+
+The advantage of this scheme is that it is "compatible" with most existing
+English tokenizers. For example, imagine that you have a part-of-speech tagging
+task which looks like this:
+
+```
+Input: John Johanson 's house
+Labels: NNP NNP POS NN
+```
+
+The tokenized output will look like this:
+
+```
+Tokens: john johan ##son ' s house
+```
+
+Crucially, this would be the same output as if the raw text were `John
+Johanson's house` (with no space before the `'s`).
+
+If you have a pre-tokenized representation with word-level annotations, you can
+simply tokenize each input word independently, and deterministically maintain an
+original-to-tokenized alignment:
+
+```python
+### Input
+orig_tokens = ["John", "Johanson", "'s", "house"]
+labels = ["NNP", "NNP", "POS", "NN"]
+
+### Output
+bert_tokens = []
+
+# Token map will be an int -> int mapping between the `orig_tokens` index and
+# the `bert_tokens` index.
+orig_to_tok_map = []
+
+tokenizer = tokenization.FullTokenizer(
+ vocab_file=vocab_file, do_lower_case=True)
+
+bert_tokens.append("[CLS]")
+for orig_token in orig_tokens:
+ orig_to_tok_map.append(len(bert_tokens))
+ bert_tokens.extend(tokenizer.tokenize(orig_token))
+bert_tokens.append("[SEP]")
+
+# bert_tokens == ["[CLS]", "john", "johan", "##son", "'", "s", "house", "[SEP]"]
+# orig_to_tok_map == [1, 2, 4, 6]
+```
+
+Now `orig_to_tok_map` can be used to project `labels` to the tokenized
+representation.
+
+There are common English tokenization schemes which will cause a slight mismatch
+between how BERT was pre-trained. For example, if your input tokenization splits
+off contractions like `do n't`, this will cause a mismatch. If it is possible to
+do so, you should pre-process your data to convert these back to raw-looking
+text, but if it's not possible, this mismatch is likely not a big deal.
+
+## Pre-training with BERT
+
+We are releasing code to do "masked LM" and "next sentence prediction" on an
+arbitrary text corpus. Note that this is *not* the exact code that was used for
+the paper (the original code was written in C++, and had some additional
+complexity), but this code does generate pre-training data as described in the
+paper.
+
+Here's how to run the data generation. The input is a plain text file, with one
+sentence per line. (It is important that these be actual sentences for the "next
+sentence prediction" task). Documents are delimited by empty lines. The output
+is a set of `tf.train.Example`s serialized into `TFRecord` file format.
+
+You can perform sentence segmentation with an off-the-shelf NLP toolkit such as
+[spaCy](https://spacy.io/). The `create_pretraining_data.py` script will
+concatenate segments until they reach the maximum sequence length to minimize
+computational waste from padding (see the script for more details). However, you
+may want to intentionally add a slight amount of noise to your input data (e.g.,
+randomly truncate 2% of input segments) to make it more robust to non-sentential
+input during fine-tuning.
+
+This script stores all of the examples for the entire input file in memory, so
+for large data files you should shard the input file and call the script
+multiple times. (You can pass in a file glob to `run_pretraining.py`, e.g.,
+`tf_examples.tf_record*`.)
+
+The `max_predictions_per_seq` is the maximum number of masked LM predictions per
+sequence. You should set this to around `max_seq_length` * `masked_lm_prob` (the
+script doesn't do that automatically because the exact value needs to be passed
+to both scripts).
+
+```shell
+python create_pretraining_data.py \
+ --input_file=./sample_text.txt \
+ --output_file=/tmp/tf_examples.tfrecord \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --do_lower_case=True \
+ --max_seq_length=128 \
+ --max_predictions_per_seq=20 \
+ --masked_lm_prob=0.15 \
+ --random_seed=12345 \
+ --dupe_factor=5
+```
+
+Here's how to run the pre-training. Do not include `init_checkpoint` if you are
+pre-training from scratch. The model configuration (including vocab size) is
+specified in `bert_config_file`. This demo code only pre-trains for a small
+number of steps (20), but in practice you will probably want to set
+`num_train_steps` to 10000 steps or more. The `max_seq_length` and
+`max_predictions_per_seq` parameters passed to `run_pretraining.py` must be the
+same as `create_pretraining_data.py`.
+
+```shell
+python run_pretraining.py \
+ --input_file=/tmp/tf_examples.tfrecord \
+ --output_dir=/tmp/pretraining_output \
+ --do_train=True \
+ --do_eval=True \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
+ --train_batch_size=32 \
+ --max_seq_length=128 \
+ --max_predictions_per_seq=20 \
+ --num_train_steps=20 \
+ --num_warmup_steps=10 \
+ --learning_rate=2e-5
+```
+
+This will produce an output like this:
+
+```
+***** Eval results *****
+ global_step = 20
+ loss = 0.0979674
+ masked_lm_accuracy = 0.985479
+ masked_lm_loss = 0.0979328
+ next_sentence_accuracy = 1.0
+ next_sentence_loss = 3.45724e-05
+```
+
+Note that since our `sample_text.txt` file is very small, this example training
+will overfit that data in only a few steps and produce unrealistically high
+accuracy numbers.
+
+### Pre-training tips and caveats
+
+* **If using your own vocabulary, make sure to change `vocab_size` in
+ `bert_config.json`. If you use a larger vocabulary without changing this,
+ you will likely get NaNs when training on GPU or TPU due to unchecked
+ out-of-bounds access.**
+* If your task has a large domain-specific corpus available (e.g., "movie
+ reviews" or "scientific papers"), it will likely be beneficial to run
+ additional steps of pre-training on your corpus, starting from the BERT
+ checkpoint.
+* The learning rate we used in the paper was 1e-4. However, if you are doing
+ additional steps of pre-training starting from an existing BERT checkpoint,
+ you should use a smaller learning rate (e.g., 2e-5).
+* Current BERT models are English-only, but we do plan to release a
+ multilingual model which has been pre-trained on a lot of languages in the
+ near future (hopefully by the end of November 2018).
+* Longer sequences are disproportionately expensive because attention is
+ quadratic to the sequence length. In other words, a batch of 64 sequences of
+ length 512 is much more expensive than a batch of 256 sequences of
+ length 128. The fully-connected/convolutional cost is the same, but the
+ attention cost is far greater for the 512-length sequences. Therefore, one
+ good recipe is to pre-train for, say, 90,000 steps with a sequence length of
+ 128 and then for 10,000 additional steps with a sequence length of 512. The
+ very long sequences are mostly needed to learn positional embeddings, which
+ can be learned fairly quickly. Note that this does require generating the
+ data twice with different values of `max_seq_length`.
+* If you are pre-training from scratch, be prepared that pre-training is
+ computationally expensive, especially on GPUs. If you are pre-training from
+ scratch, our recommended recipe is to pre-train a `BERT-Base` on a single
+ [preemptible Cloud TPU v2](https://cloud.google.com/tpu/docs/pricing), which
+ takes about 2 weeks at a cost of about $500 USD (based on the pricing in
+ October 2018). You will have to scale down the batch size when only training
+ on a single Cloud TPU, compared to what was used in the paper. It is
+ recommended to use the largest batch size that fits into TPU memory.
+
+### Pre-training data
+
+We will **not** be able to release the pre-processed datasets used in the paper.
+For Wikipedia, the recommended pre-processing is to download
+[the latest dump](https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2),
+extract the text with
+[`WikiExtractor.py`](https://github.com/attardi/wikiextractor), and then apply
+any necessary cleanup to convert it into plain text.
+
+Unfortunately the researchers who collected the
+[BookCorpus](http://yknzhu.wixsite.com/mbweb) no longer have it available for
+public download. The
+[Project Guttenberg Dataset](https://web.eecs.umich.edu/~lahiri/gutenberg_dataset.html)
+is a somewhat smaller (200M word) collection of older books that are public
+domain.
+
+[Common Crawl](http://commoncrawl.org/) is another very large collection of
+text, but you will likely have to do substantial pre-processing and cleanup to
+extract a usable corpus for pre-training BERT.
+
+### Learning a new WordPiece vocabulary
+
+This repository does not include code for *learning* a new WordPiece vocabulary.
+The reason is that the code used in the paper was implemented in C++ with
+dependencies on Google's internal libraries. For English, it is almost always
+better to just start with our vocabulary and pre-trained models. For learning
+vocabularies of other languages, there are a number of open source options
+available. However, keep in mind that these are not compatible with our
+`tokenization.py` library:
+
+* [Google's SentencePiece library](https://github.com/google/sentencepiece)
+
+* [tensor2tensor's WordPiece generation script](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/data_generators/text_encoder_build_subword.py)
+
+* [Rico Sennrich's Byte Pair Encoding library](https://github.com/rsennrich/subword-nmt)
+
+## Using BERT in Colab
+
+If you want to use BERT with [Colab](https://colab.research.google.com), you can
+get started with the notebook
+"[BERT FineTuning with Cloud TPUs](https://colab.research.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)".
+**At the time of this writing (October 31st, 2018), Colab users can access a
+Cloud TPU completely for free.** Note: One per user, availability limited,
+requires a Google Cloud Platform account with storage (although storage may be
+purchased with free credit for signing up with GCP), and this capability may not
+longer be available in the future. Click on the BERT Colab that was just linked
+for more information.
+
+## FAQ
+
+#### Is this code compatible with Cloud TPUs? What about GPUs?
+
+Yes, all of the code in this repository works out-of-the-box with CPU, GPU, and
+Cloud TPU. However, GPU training is single-GPU only.
+
+#### I am getting out-of-memory errors, what is wrong?
+
+See the section on [out-of-memory issues](#out-of-memory-issues) for more
+information.
+
+#### Is there a PyTorch version available?
+
+There is no official PyTorch implementation. However, NLP researchers from
+HuggingFace made a
+[PyTorch version of BERT available](https://github.com/huggingface/pytorch-pretrained-BERT)
+which is compatible with our pre-trained checkpoints and is able to reproduce
+our results. We were not involved in the creation or maintenance of the PyTorch
+implementation so please direct any questions towards the authors of that
+repository.
+
+#### Is there a Chainer version available?
+
+There is no official Chainer implementation. However, Sosuke Kobayashi made a
+[Chainer version of BERT available](https://github.com/soskek/bert-chainer)
+which is compatible with our pre-trained checkpoints and is able to reproduce
+our results. We were not involved in the creation or maintenance of the Chainer
+implementation so please direct any questions towards the authors of that
+repository.
+
+#### Will models in other languages be released?
+
+Yes, we plan to release a multi-lingual BERT model in the near future. We cannot
+make promises about exactly which languages will be included, but it will likely
+be a single model which includes *most* of the languages which have a
+significantly-sized Wikipedia.
+
+#### Will models larger than `BERT-Large` be released?
+
+So far we have not attempted to train anything larger than `BERT-Large`. It is
+possible that we will release larger models if we are able to obtain significant
+improvements.
+
+#### What license is this library released under?
+
+All code *and* models are released under the Apache 2.0 license. See the
+`LICENSE` file for more information.
+
+#### How do I cite BERT?
+
+For now, cite [the Arxiv paper](https://arxiv.org/abs/1810.04805):
+
+```
+@article{devlin2018bert,
+ title={BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding},
+ author={Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
+ journal={arXiv preprint arXiv:1810.04805},
+ year={2018}
+}
+```
+
+If we submit the paper to a conference or journal, we will update the BibTeX.
+
+## Disclaimer
+
+This is not an official Google product.
+
+## Contact information
+
+For help or issues using BERT, please submit a GitHub issue.
+
+For personal communication related to BERT, please contact Jacob Devlin
+(`jacobdevlin@google.com`), Ming-Wei Chang (`mingweichang@google.com`), or
+Kenton Lee (`kentonl@google.com`).
diff --git a/bert-master/bert-master/__init__.py b/bert-master/bert-master/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..effb57b1e893fc03b3782961deb060749083c696
--- /dev/null
+++ b/bert-master/bert-master/__init__.py
@@ -0,0 +1,15 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
diff --git a/bert-master/bert-master/create_pretraining_data.py b/bert-master/bert-master/create_pretraining_data.py
new file mode 100644
index 0000000000000000000000000000000000000000..5340d96ae3d5e31bd0d38f35bac785515250d98f
--- /dev/null
+++ b/bert-master/bert-master/create_pretraining_data.py
@@ -0,0 +1,469 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Create masked LM/next sentence masked_lm TF examples for BERT."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import random
+import tokenization
+import tensorflow as tf
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string("input_file", None,
+ "Input raw text file (or comma-separated list of files).")
+
+flags.DEFINE_string(
+ "output_file", None,
+ "Output TF example file (or comma-separated list of files).")
+
+flags.DEFINE_string("vocab_file", None,
+ "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_bool(
+ "do_lower_case", True,
+ "Whether to lower case the input text. Should be True for uncased "
+ "models and False for cased models.")
+
+flags.DEFINE_bool(
+ "do_whole_word_mask", False,
+ "Whether to use whole word masking rather than per-WordPiece masking.")
+
+flags.DEFINE_integer("max_seq_length", 128, "Maximum sequence length.")
+
+flags.DEFINE_integer("max_predictions_per_seq", 20,
+ "Maximum number of masked LM predictions per sequence.")
+
+flags.DEFINE_integer("random_seed", 12345, "Random seed for data generation.")
+
+flags.DEFINE_integer(
+ "dupe_factor", 10,
+ "Number of times to duplicate the input data (with different masks).")
+
+flags.DEFINE_float("masked_lm_prob", 0.15, "Masked LM probability.")
+
+flags.DEFINE_float(
+ "short_seq_prob", 0.1,
+ "Probability of creating sequences which are shorter than the "
+ "maximum length.")
+
+
+class TrainingInstance(object):
+ """A single training instance (sentence pair)."""
+
+ def __init__(self, tokens, segment_ids, masked_lm_positions, masked_lm_labels,
+ is_random_next):
+ self.tokens = tokens
+ self.segment_ids = segment_ids
+ self.is_random_next = is_random_next
+ self.masked_lm_positions = masked_lm_positions
+ self.masked_lm_labels = masked_lm_labels
+
+ def __str__(self):
+ s = ""
+ s += "tokens: %s\n" % (" ".join(
+ [tokenization.printable_text(x) for x in self.tokens]))
+ s += "segment_ids: %s\n" % (" ".join([str(x) for x in self.segment_ids]))
+ s += "is_random_next: %s\n" % self.is_random_next
+ s += "masked_lm_positions: %s\n" % (" ".join(
+ [str(x) for x in self.masked_lm_positions]))
+ s += "masked_lm_labels: %s\n" % (" ".join(
+ [tokenization.printable_text(x) for x in self.masked_lm_labels]))
+ s += "\n"
+ return s
+
+ def __repr__(self):
+ return self.__str__()
+
+
+def write_instance_to_example_files(instances, tokenizer, max_seq_length,
+ max_predictions_per_seq, output_files):
+ """Create TF example files from `TrainingInstance`s."""
+ writers = []
+ for output_file in output_files:
+ writers.append(tf.python_io.TFRecordWriter(output_file))
+
+ writer_index = 0
+
+ total_written = 0
+ for (inst_index, instance) in enumerate(instances):
+ input_ids = tokenizer.convert_tokens_to_ids(instance.tokens)
+ input_mask = [1] * len(input_ids)
+ segment_ids = list(instance.segment_ids)
+ assert len(input_ids) <= max_seq_length
+
+ while len(input_ids) < max_seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ segment_ids.append(0)
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ masked_lm_positions = list(instance.masked_lm_positions)
+ masked_lm_ids = tokenizer.convert_tokens_to_ids(instance.masked_lm_labels)
+ masked_lm_weights = [1.0] * len(masked_lm_ids)
+
+ while len(masked_lm_positions) < max_predictions_per_seq:
+ masked_lm_positions.append(0)
+ masked_lm_ids.append(0)
+ masked_lm_weights.append(0.0)
+
+ next_sentence_label = 1 if instance.is_random_next else 0
+
+ features = collections.OrderedDict()
+ features["input_ids"] = create_int_feature(input_ids)
+ features["input_mask"] = create_int_feature(input_mask)
+ features["segment_ids"] = create_int_feature(segment_ids)
+ features["masked_lm_positions"] = create_int_feature(masked_lm_positions)
+ features["masked_lm_ids"] = create_int_feature(masked_lm_ids)
+ features["masked_lm_weights"] = create_float_feature(masked_lm_weights)
+ features["next_sentence_labels"] = create_int_feature([next_sentence_label])
+
+ tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+
+ writers[writer_index].write(tf_example.SerializeToString())
+ writer_index = (writer_index + 1) % len(writers)
+
+ total_written += 1
+
+ if inst_index < 20:
+ tf.logging.info("*** Example ***")
+ tf.logging.info("tokens: %s" % " ".join(
+ [tokenization.printable_text(x) for x in instance.tokens]))
+
+ for feature_name in features.keys():
+ feature = features[feature_name]
+ values = []
+ if feature.int64_list.value:
+ values = feature.int64_list.value
+ elif feature.float_list.value:
+ values = feature.float_list.value
+ tf.logging.info(
+ "%s: %s" % (feature_name, " ".join([str(x) for x in values])))
+
+ for writer in writers:
+ writer.close()
+
+ tf.logging.info("Wrote %d total instances", total_written)
+
+
+def create_int_feature(values):
+ feature = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))
+ return feature
+
+
+def create_float_feature(values):
+ feature = tf.train.Feature(float_list=tf.train.FloatList(value=list(values)))
+ return feature
+
+
+def create_training_instances(input_files, tokenizer, max_seq_length,
+ dupe_factor, short_seq_prob, masked_lm_prob,
+ max_predictions_per_seq, rng):
+ """Create `TrainingInstance`s from raw text."""
+ all_documents = [[]]
+
+ # Input file format:
+ # (1) One sentence per line. These should ideally be actual sentences, not
+ # entire paragraphs or arbitrary spans of text. (Because we use the
+ # sentence boundaries for the "next sentence prediction" task).
+ # (2) Blank lines between documents. Document boundaries are needed so
+ # that the "next sentence prediction" task doesn't span between documents.
+ for input_file in input_files:
+ with tf.gfile.GFile(input_file, "r") as reader:
+ while True:
+ line = tokenization.convert_to_unicode(reader.readline())
+ if not line:
+ break
+ line = line.strip()
+
+ # Empty lines are used as document delimiters
+ if not line:
+ all_documents.append([])
+ tokens = tokenizer.tokenize(line)
+ if tokens:
+ all_documents[-1].append(tokens)
+
+ # Remove empty documents
+ all_documents = [x for x in all_documents if x]
+ rng.shuffle(all_documents)
+
+ vocab_words = list(tokenizer.vocab.keys())
+ instances = []
+ for _ in range(dupe_factor):
+ for document_index in range(len(all_documents)):
+ instances.extend(
+ create_instances_from_document(
+ all_documents, document_index, max_seq_length, short_seq_prob,
+ masked_lm_prob, max_predictions_per_seq, vocab_words, rng))
+
+ rng.shuffle(instances)
+ return instances
+
+
+def create_instances_from_document(
+ all_documents, document_index, max_seq_length, short_seq_prob,
+ masked_lm_prob, max_predictions_per_seq, vocab_words, rng):
+ """Creates `TrainingInstance`s for a single document."""
+ document = all_documents[document_index]
+
+ # Account for [CLS], [SEP], [SEP]
+ max_num_tokens = max_seq_length - 3
+
+ # We *usually* want to fill up the entire sequence since we are padding
+ # to `max_seq_length` anyways, so short sequences are generally wasted
+ # computation. However, we *sometimes*
+ # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
+ # sequences to minimize the mismatch between pre-training and fine-tuning.
+ # The `target_seq_length` is just a rough target however, whereas
+ # `max_seq_length` is a hard limit.
+ target_seq_length = max_num_tokens
+ if rng.random() < short_seq_prob:
+ target_seq_length = rng.randint(2, max_num_tokens)
+
+ # We DON'T just concatenate all of the tokens from a document into a long
+ # sequence and choose an arbitrary split point because this would make the
+ # next sentence prediction task too easy. Instead, we split the input into
+ # segments "A" and "B" based on the actual "sentences" provided by the user
+ # input.
+ instances = []
+ current_chunk = []
+ current_length = 0
+ i = 0
+ while i < len(document):
+ segment = document[i]
+ current_chunk.append(segment)
+ current_length += len(segment)
+ if i == len(document) - 1 or current_length >= target_seq_length:
+ if current_chunk:
+ # `a_end` is how many segments from `current_chunk` go into the `A`
+ # (first) sentence.
+ a_end = 1
+ if len(current_chunk) >= 2:
+ a_end = rng.randint(1, len(current_chunk) - 1)
+
+ tokens_a = []
+ for j in range(a_end):
+ tokens_a.extend(current_chunk[j])
+
+ tokens_b = []
+ # Random next
+ is_random_next = False
+ if len(current_chunk) == 1 or rng.random() < 0.5:
+ is_random_next = True
+ target_b_length = target_seq_length - len(tokens_a)
+
+ # This should rarely go for more than one iteration for large
+ # corpora. However, just to be careful, we try to make sure that
+ # the random document is not the same as the document
+ # we're processing.
+ for _ in range(10):
+ random_document_index = rng.randint(0, len(all_documents) - 1)
+ if random_document_index != document_index:
+ break
+
+ random_document = all_documents[random_document_index]
+ random_start = rng.randint(0, len(random_document) - 1)
+ for j in range(random_start, len(random_document)):
+ tokens_b.extend(random_document[j])
+ if len(tokens_b) >= target_b_length:
+ break
+ # We didn't actually use these segments so we "put them back" so
+ # they don't go to waste.
+ num_unused_segments = len(current_chunk) - a_end
+ i -= num_unused_segments
+ # Actual next
+ else:
+ is_random_next = False
+ for j in range(a_end, len(current_chunk)):
+ tokens_b.extend(current_chunk[j])
+ truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng)
+
+ assert len(tokens_a) >= 1
+ assert len(tokens_b) >= 1
+
+ tokens = []
+ segment_ids = []
+ tokens.append("[CLS]")
+ segment_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ segment_ids.append(0)
+
+ tokens.append("[SEP]")
+ segment_ids.append(0)
+
+ for token in tokens_b:
+ tokens.append(token)
+ segment_ids.append(1)
+ tokens.append("[SEP]")
+ segment_ids.append(1)
+
+ (tokens, masked_lm_positions,
+ masked_lm_labels) = create_masked_lm_predictions(
+ tokens, masked_lm_prob, max_predictions_per_seq, vocab_words, rng)
+ instance = TrainingInstance(
+ tokens=tokens,
+ segment_ids=segment_ids,
+ is_random_next=is_random_next,
+ masked_lm_positions=masked_lm_positions,
+ masked_lm_labels=masked_lm_labels)
+ instances.append(instance)
+ current_chunk = []
+ current_length = 0
+ i += 1
+
+ return instances
+
+
+MaskedLmInstance = collections.namedtuple("MaskedLmInstance",
+ ["index", "label"])
+
+
+def create_masked_lm_predictions(tokens, masked_lm_prob,
+ max_predictions_per_seq, vocab_words, rng):
+ """Creates the predictions for the masked LM objective."""
+
+ cand_indexes = []
+ for (i, token) in enumerate(tokens):
+ if token == "[CLS]" or token == "[SEP]":
+ continue
+ # Whole Word Masking means that if we mask all of the wordpieces
+ # corresponding to an original word. When a word has been split into
+ # WordPieces, the first token does not have any marker and any subsequence
+ # tokens are prefixed with ##. So whenever we see the ## token, we
+ # append it to the previous set of word indexes.
+ #
+ # Note that Whole Word Masking does *not* change the training code
+ # at all -- we still predict each WordPiece independently, softmaxed
+ # over the entire vocabulary.
+ if (FLAGS.do_whole_word_mask and len(cand_indexes) >= 1 and
+ token.startswith("##")):
+ cand_indexes[-1].append(i)
+ else:
+ cand_indexes.append([i])
+
+ rng.shuffle(cand_indexes)
+
+ output_tokens = list(tokens)
+
+ num_to_predict = min(max_predictions_per_seq,
+ max(1, int(round(len(tokens) * masked_lm_prob))))
+
+ masked_lms = []
+ covered_indexes = set()
+ for index_set in cand_indexes:
+ if len(masked_lms) >= num_to_predict:
+ break
+ # If adding a whole-word mask would exceed the maximum number of
+ # predictions, then just skip this candidate.
+ if len(masked_lms) + len(index_set) > num_to_predict:
+ continue
+ is_any_index_covered = False
+ for index in index_set:
+ if index in covered_indexes:
+ is_any_index_covered = True
+ break
+ if is_any_index_covered:
+ continue
+ for index in index_set:
+ covered_indexes.add(index)
+
+ masked_token = None
+ # 80% of the time, replace with [MASK]
+ if rng.random() < 0.8:
+ masked_token = "[MASK]"
+ else:
+ # 10% of the time, keep original
+ if rng.random() < 0.5:
+ masked_token = tokens[index]
+ # 10% of the time, replace with random word
+ else:
+ masked_token = vocab_words[rng.randint(0, len(vocab_words) - 1)]
+
+ output_tokens[index] = masked_token
+
+ masked_lms.append(MaskedLmInstance(index=index, label=tokens[index]))
+ assert len(masked_lms) <= num_to_predict
+ masked_lms = sorted(masked_lms, key=lambda x: x.index)
+
+ masked_lm_positions = []
+ masked_lm_labels = []
+ for p in masked_lms:
+ masked_lm_positions.append(p.index)
+ masked_lm_labels.append(p.label)
+
+ return (output_tokens, masked_lm_positions, masked_lm_labels)
+
+
+def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng):
+ """Truncates a pair of sequences to a maximum sequence length."""
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_num_tokens:
+ break
+
+ trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b
+ assert len(trunc_tokens) >= 1
+
+ # We want to sometimes truncate from the front and sometimes from the
+ # back to add more randomness and avoid biases.
+ if rng.random() < 0.5:
+ del trunc_tokens[0]
+ else:
+ trunc_tokens.pop()
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ tokenizer = tokenization.FullTokenizer(
+ vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+ input_files = []
+ for input_pattern in FLAGS.input_file.split(","):
+ input_files.extend(tf.gfile.Glob(input_pattern))
+
+ tf.logging.info("*** Reading from input files ***")
+ for input_file in input_files:
+ tf.logging.info(" %s", input_file)
+
+ rng = random.Random(FLAGS.random_seed)
+ instances = create_training_instances(
+ input_files, tokenizer, FLAGS.max_seq_length, FLAGS.dupe_factor,
+ FLAGS.short_seq_prob, FLAGS.masked_lm_prob, FLAGS.max_predictions_per_seq,
+ rng)
+
+ output_files = FLAGS.output_file.split(",")
+ tf.logging.info("*** Writing to output files ***")
+ for output_file in output_files:
+ tf.logging.info(" %s", output_file)
+
+ write_instance_to_example_files(instances, tokenizer, FLAGS.max_seq_length,
+ FLAGS.max_predictions_per_seq, output_files)
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("input_file")
+ flags.mark_flag_as_required("output_file")
+ flags.mark_flag_as_required("vocab_file")
+ tf.app.run()
diff --git a/bert-master/bert-master/extract_features.py b/bert-master/bert-master/extract_features.py
new file mode 100644
index 0000000000000000000000000000000000000000..60e3830a95b995580e209b84c31f89eceb1907c4
--- /dev/null
+++ b/bert-master/bert-master/extract_features.py
@@ -0,0 +1,419 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Extract pre-computed feature vectors from BERT."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import codecs
+import collections
+import json
+import re
+
+import modeling
+import tokenization
+import tensorflow as tf
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string("input_file", None, "")
+
+flags.DEFINE_string("output_file", None, "")
+
+flags.DEFINE_string("layers", "-1,-2,-3,-4", "")
+
+flags.DEFINE_string(
+ "bert_config_file", None,
+ "The config json file corresponding to the pre-trained BERT model. "
+ "This specifies the model architecture.")
+
+flags.DEFINE_integer(
+ "max_seq_length", 128,
+ "The maximum total input sequence length after WordPiece tokenization. "
+ "Sequences longer than this will be truncated, and sequences shorter "
+ "than this will be padded.")
+
+flags.DEFINE_string(
+ "init_checkpoint", None,
+ "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_string("vocab_file", None,
+ "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_bool(
+ "do_lower_case", True,
+ "Whether to lower case the input text. Should be True for uncased "
+ "models and False for cased models.")
+
+flags.DEFINE_integer("batch_size", 32, "Batch size for predictions.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+flags.DEFINE_string("master", None,
+ "If using a TPU, the address of the master.")
+
+flags.DEFINE_integer(
+ "num_tpu_cores", 8,
+ "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+flags.DEFINE_bool(
+ "use_one_hot_embeddings", False,
+ "If True, tf.one_hot will be used for embedding lookups, otherwise "
+ "tf.nn.embedding_lookup will be used. On TPUs, this should be True "
+ "since it is much faster.")
+
+
+class InputExample(object):
+
+ def __init__(self, unique_id, text_a, text_b):
+ self.unique_id = unique_id
+ self.text_a = text_a
+ self.text_b = text_b
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self, unique_id, tokens, input_ids, input_mask, input_type_ids):
+ self.unique_id = unique_id
+ self.tokens = tokens
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.input_type_ids = input_type_ids
+
+
+def input_fn_builder(features, seq_length):
+ """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+ all_unique_ids = []
+ all_input_ids = []
+ all_input_mask = []
+ all_input_type_ids = []
+
+ for feature in features:
+ all_unique_ids.append(feature.unique_id)
+ all_input_ids.append(feature.input_ids)
+ all_input_mask.append(feature.input_mask)
+ all_input_type_ids.append(feature.input_type_ids)
+
+ def input_fn(params):
+ """The actual input function."""
+ batch_size = params["batch_size"]
+
+ num_examples = len(features)
+
+ # This is for demo purposes and does NOT scale to large data sets. We do
+ # not use Dataset.from_generator() because that uses tf.py_func which is
+ # not TPU compatible. The right way to load data is with TFRecordReader.
+ d = tf.data.Dataset.from_tensor_slices({
+ "unique_ids":
+ tf.constant(all_unique_ids, shape=[num_examples], dtype=tf.int32),
+ "input_ids":
+ tf.constant(
+ all_input_ids, shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ "input_mask":
+ tf.constant(
+ all_input_mask,
+ shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ "input_type_ids":
+ tf.constant(
+ all_input_type_ids,
+ shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ })
+
+ d = d.batch(batch_size=batch_size, drop_remainder=False)
+ return d
+
+ return input_fn
+
+
+def model_fn_builder(bert_config, init_checkpoint, layer_indexes, use_tpu,
+ use_one_hot_embeddings):
+ """Returns `model_fn` closure for TPUEstimator."""
+
+ def model_fn(features, labels, mode, params): # pylint: disable=unused-argument
+ """The `model_fn` for TPUEstimator."""
+
+ unique_ids = features["unique_ids"]
+ input_ids = features["input_ids"]
+ input_mask = features["input_mask"]
+ input_type_ids = features["input_type_ids"]
+
+ model = modeling.BertModel(
+ config=bert_config,
+ is_training=False,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ token_type_ids=input_type_ids,
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ if mode != tf.estimator.ModeKeys.PREDICT:
+ raise ValueError("Only PREDICT modes are supported: %s" % (mode))
+
+ tvars = tf.trainable_variables()
+ scaffold_fn = None
+ (assignment_map,
+ initialized_variable_names) = modeling.get_assignment_map_from_checkpoint(
+ tvars, init_checkpoint)
+ if use_tpu:
+
+ def tpu_scaffold():
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+ return tf.train.Scaffold()
+
+ scaffold_fn = tpu_scaffold
+ else:
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+ tf.logging.info("**** Trainable Variables ****")
+ for var in tvars:
+ init_string = ""
+ if var.name in initialized_variable_names:
+ init_string = ", *INIT_FROM_CKPT*"
+ tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape,
+ init_string)
+
+ all_layers = model.get_all_encoder_layers()
+
+ predictions = {
+ "unique_id": unique_ids,
+ }
+
+ for (i, layer_index) in enumerate(layer_indexes):
+ predictions["layer_output_%d" % i] = all_layers[layer_index]
+
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)
+ return output_spec
+
+ return model_fn
+
+
+def convert_examples_to_features(examples, seq_length, tokenizer):
+ """Loads a data file into a list of `InputBatch`s."""
+
+ features = []
+ for (ex_index, example) in enumerate(examples):
+ tokens_a = tokenizer.tokenize(example.text_a)
+
+ tokens_b = None
+ if example.text_b:
+ tokens_b = tokenizer.tokenize(example.text_b)
+
+ if tokens_b:
+ # Modifies `tokens_a` and `tokens_b` in place so that the total
+ # length is less than the specified length.
+ # Account for [CLS], [SEP], [SEP] with "- 3"
+ _truncate_seq_pair(tokens_a, tokens_b, seq_length - 3)
+ else:
+ # Account for [CLS] and [SEP] with "- 2"
+ if len(tokens_a) > seq_length - 2:
+ tokens_a = tokens_a[0:(seq_length - 2)]
+
+ # The convention in BERT is:
+ # (a) For sequence pairs:
+ # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+ # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
+ # (b) For single sequences:
+ # tokens: [CLS] the dog is hairy . [SEP]
+ # type_ids: 0 0 0 0 0 0 0
+ #
+ # Where "type_ids" are used to indicate whether this is the first
+ # sequence or the second sequence. The embedding vectors for `type=0` and
+ # `type=1` were learned during pre-training and are added to the wordpiece
+ # embedding vector (and position vector). This is not *strictly* necessary
+ # since the [SEP] token unambiguously separates the sequences, but it makes
+ # it easier for the model to learn the concept of sequences.
+ #
+ # For classification tasks, the first vector (corresponding to [CLS]) is
+ # used as as the "sentence vector". Note that this only makes sense because
+ # the entire model is fine-tuned.
+ tokens = []
+ input_type_ids = []
+ tokens.append("[CLS]")
+ input_type_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ input_type_ids.append(0)
+ tokens.append("[SEP]")
+ input_type_ids.append(0)
+
+ if tokens_b:
+ for token in tokens_b:
+ tokens.append(token)
+ input_type_ids.append(1)
+ tokens.append("[SEP]")
+ input_type_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ input_type_ids.append(0)
+
+ assert len(input_ids) == seq_length
+ assert len(input_mask) == seq_length
+ assert len(input_type_ids) == seq_length
+
+ if ex_index < 5:
+ tf.logging.info("*** Example ***")
+ tf.logging.info("unique_id: %s" % (example.unique_id))
+ tf.logging.info("tokens: %s" % " ".join(
+ [tokenization.printable_text(x) for x in tokens]))
+ tf.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ tf.logging.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ tf.logging.info(
+ "input_type_ids: %s" % " ".join([str(x) for x in input_type_ids]))
+
+ features.append(
+ InputFeatures(
+ unique_id=example.unique_id,
+ tokens=tokens,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ input_type_ids=input_type_ids))
+ return features
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+
+def read_examples(input_file):
+ """Read a list of `InputExample`s from an input file."""
+ examples = []
+ unique_id = 0
+ with tf.gfile.GFile(input_file, "r") as reader:
+ while True:
+ line = tokenization.convert_to_unicode(reader.readline())
+ if not line:
+ break
+ line = line.strip()
+ text_a = None
+ text_b = None
+ m = re.match(r"^(.*) \|\|\| (.*)$", line)
+ if m is None:
+ text_a = line
+ else:
+ text_a = m.group(1)
+ text_b = m.group(2)
+ examples.append(
+ InputExample(unique_id=unique_id, text_a=text_a, text_b=text_b))
+ unique_id += 1
+ return examples
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ layer_indexes = [int(x) for x in FLAGS.layers.split(",")]
+
+ bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+
+ tokenizer = tokenization.FullTokenizer(
+ vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+ is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
+ run_config = tf.contrib.tpu.RunConfig(
+ master=FLAGS.master,
+ tpu_config=tf.contrib.tpu.TPUConfig(
+ num_shards=FLAGS.num_tpu_cores,
+ per_host_input_for_training=is_per_host))
+
+ examples = read_examples(FLAGS.input_file)
+
+ features = convert_examples_to_features(
+ examples=examples, seq_length=FLAGS.max_seq_length, tokenizer=tokenizer)
+
+ unique_id_to_feature = {}
+ for feature in features:
+ unique_id_to_feature[feature.unique_id] = feature
+
+ model_fn = model_fn_builder(
+ bert_config=bert_config,
+ init_checkpoint=FLAGS.init_checkpoint,
+ layer_indexes=layer_indexes,
+ use_tpu=FLAGS.use_tpu,
+ use_one_hot_embeddings=FLAGS.use_one_hot_embeddings)
+
+ # If TPU is not available, this will fall back to normal Estimator on CPU
+ # or GPU.
+ estimator = tf.contrib.tpu.TPUEstimator(
+ use_tpu=FLAGS.use_tpu,
+ model_fn=model_fn,
+ config=run_config,
+ predict_batch_size=FLAGS.batch_size)
+
+ input_fn = input_fn_builder(
+ features=features, seq_length=FLAGS.max_seq_length)
+
+ with codecs.getwriter("utf-8")(tf.gfile.Open(FLAGS.output_file,
+ "w")) as writer:
+ for result in estimator.predict(input_fn, yield_single_examples=True):
+ unique_id = int(result["unique_id"])
+ feature = unique_id_to_feature[unique_id]
+ output_json = collections.OrderedDict()
+ output_json["linex_index"] = unique_id
+ all_features = []
+ for (i, token) in enumerate(feature.tokens):
+ all_layers = []
+ for (j, layer_index) in enumerate(layer_indexes):
+ layer_output = result["layer_output_%d" % j]
+ layers = collections.OrderedDict()
+ layers["index"] = layer_index
+ layers["values"] = [
+ round(float(x), 6) for x in layer_output[i:(i + 1)].flat
+ ]
+ all_layers.append(layers)
+ features = collections.OrderedDict()
+ features["token"] = token
+ features["layers"] = all_layers
+ all_features.append(features)
+ output_json["features"] = all_features
+ writer.write(json.dumps(output_json) + "\n")
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("input_file")
+ flags.mark_flag_as_required("vocab_file")
+ flags.mark_flag_as_required("bert_config_file")
+ flags.mark_flag_as_required("init_checkpoint")
+ flags.mark_flag_as_required("output_file")
+ tf.app.run()
diff --git a/bert-master/bert-master/modeling.py b/bert-master/bert-master/modeling.py
new file mode 100644
index 0000000000000000000000000000000000000000..fed525971ca9ef440cbd78933d24e07896190a88
--- /dev/null
+++ b/bert-master/bert-master/modeling.py
@@ -0,0 +1,986 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""The main BERT model and related functions."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import copy
+import json
+import math
+import re
+import numpy as np
+import six
+import tensorflow as tf
+
+
+class BertConfig(object):
+ """Configuration for `BertModel`."""
+
+ def __init__(self,
+ vocab_size,
+ hidden_size=768,
+ num_hidden_layers=12,
+ num_attention_heads=12,
+ intermediate_size=3072,
+ hidden_act="gelu",
+ hidden_dropout_prob=0.1,
+ attention_probs_dropout_prob=0.1,
+ max_position_embeddings=512,
+ type_vocab_size=16,
+ initializer_range=0.02):
+ """Constructs BertConfig.
+
+ Args:
+ vocab_size: Vocabulary size of `inputs_ids` in `BertModel`.
+ hidden_size: Size of the encoder layers and the pooler layer.
+ num_hidden_layers: Number of hidden layers in the Transformer encoder.
+ num_attention_heads: Number of attention heads for each attention layer in
+ the Transformer encoder.
+ intermediate_size: The size of the "intermediate" (i.e., feed-forward)
+ layer in the Transformer encoder.
+ hidden_act: The non-linear activation function (function or string) in the
+ encoder and pooler.
+ hidden_dropout_prob: The dropout probability for all fully connected
+ layers in the embeddings, encoder, and pooler.
+ attention_probs_dropout_prob: The dropout ratio for the attention
+ probabilities.
+ max_position_embeddings: The maximum sequence length that this model might
+ ever be used with. Typically set this to something large just in case
+ (e.g., 512 or 1024 or 2048).
+ type_vocab_size: The vocabulary size of the `token_type_ids` passed into
+ `BertModel`.
+ initializer_range: The stdev of the truncated_normal_initializer for
+ initializing all weight matrices.
+ """
+ self.vocab_size = vocab_size
+ self.hidden_size = hidden_size
+ self.num_hidden_layers = num_hidden_layers
+ self.num_attention_heads = num_attention_heads
+ self.hidden_act = hidden_act
+ self.intermediate_size = intermediate_size
+ self.hidden_dropout_prob = hidden_dropout_prob
+ self.attention_probs_dropout_prob = attention_probs_dropout_prob
+ self.max_position_embeddings = max_position_embeddings
+ self.type_vocab_size = type_vocab_size
+ self.initializer_range = initializer_range
+
+ @classmethod
+ def from_dict(cls, json_object):
+ """Constructs a `BertConfig` from a Python dictionary of parameters."""
+ config = BertConfig(vocab_size=None)
+ for (key, value) in six.iteritems(json_object):
+ config.__dict__[key] = value
+ return config
+
+ @classmethod
+ def from_json_file(cls, json_file):
+ """Constructs a `BertConfig` from a json file of parameters."""
+ with tf.gfile.GFile(json_file, "r") as reader:
+ text = reader.read()
+ return cls.from_dict(json.loads(text))
+
+ def to_dict(self):
+ """Serializes this instance to a Python dictionary."""
+ output = copy.deepcopy(self.__dict__)
+ return output
+
+ def to_json_string(self):
+ """Serializes this instance to a JSON string."""
+ return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+
+class BertModel(object):
+ """BERT model ("Bidirectional Encoder Representations from Transformers").
+
+ Example usage:
+
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = tf.constant([[31, 51, 99], [15, 5, 0]])
+ input_mask = tf.constant([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = tf.constant([[0, 0, 1], [0, 2, 0]])
+
+ config = modeling.BertConfig(vocab_size=32000, hidden_size=512,
+ num_hidden_layers=8, num_attention_heads=6, intermediate_size=1024)
+
+ model = modeling.BertModel(config=config, is_training=True,
+ input_ids=input_ids, input_mask=input_mask, token_type_ids=token_type_ids)
+
+ label_embeddings = tf.get_variable(...)
+ pooled_output = model.get_pooled_output()
+ logits = tf.matmul(pooled_output, label_embeddings)
+ ...
+ ```
+ """
+
+ def __init__(self,
+ config,
+ is_training,
+ input_ids,
+ input_mask=None,
+ token_type_ids=None,
+ use_one_hot_embeddings=False,
+ scope=None):
+ """Constructor for BertModel.
+
+ Args:
+ config: `BertConfig` instance.
+ is_training: bool. true for training model, false for eval model. Controls
+ whether dropout will be applied.
+ input_ids: int32 Tensor of shape [batch_size, seq_length].
+ input_mask: (optional) int32 Tensor of shape [batch_size, seq_length].
+ token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length].
+ use_one_hot_embeddings: (optional) bool. Whether to use one-hot word
+ embeddings or tf.embedding_lookup() for the word embeddings.
+ scope: (optional) variable scope. Defaults to "bert".
+
+ Raises:
+ ValueError: The config is invalid or one of the input tensor shapes
+ is invalid.
+ """
+ config = copy.deepcopy(config)
+ if not is_training:
+ config.hidden_dropout_prob = 0.0
+ config.attention_probs_dropout_prob = 0.0
+
+ input_shape = get_shape_list(input_ids, expected_rank=2)
+ batch_size = input_shape[0]
+ seq_length = input_shape[1]
+
+ if input_mask is None:
+ input_mask = tf.ones(shape=[batch_size, seq_length], dtype=tf.int32)
+
+ if token_type_ids is None:
+ token_type_ids = tf.zeros(shape=[batch_size, seq_length], dtype=tf.int32)
+
+ with tf.variable_scope(scope, default_name="bert"):
+ with tf.variable_scope("embeddings"):
+ # Perform embedding lookup on the word ids.
+ (self.embedding_output, self.embedding_table) = embedding_lookup(
+ input_ids=input_ids,
+ vocab_size=config.vocab_size,
+ embedding_size=config.hidden_size,
+ initializer_range=config.initializer_range,
+ word_embedding_name="word_embeddings",
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ # Add positional embeddings and token type embeddings, then layer
+ # normalize and perform dropout.
+ self.embedding_output = embedding_postprocessor(
+ input_tensor=self.embedding_output,
+ use_token_type=True,
+ token_type_ids=token_type_ids,
+ token_type_vocab_size=config.type_vocab_size,
+ token_type_embedding_name="token_type_embeddings",
+ use_position_embeddings=True,
+ position_embedding_name="position_embeddings",
+ initializer_range=config.initializer_range,
+ max_position_embeddings=config.max_position_embeddings,
+ dropout_prob=config.hidden_dropout_prob)
+
+ with tf.variable_scope("encoder"):
+ # This converts a 2D mask of shape [batch_size, seq_length] to a 3D
+ # mask of shape [batch_size, seq_length, seq_length] which is used
+ # for the attention scores.
+ attention_mask = create_attention_mask_from_input_mask(
+ input_ids, input_mask)
+
+ # Run the stacked transformer.
+ # `sequence_output` shape = [batch_size, seq_length, hidden_size].
+ self.all_encoder_layers = transformer_model(
+ input_tensor=self.embedding_output,
+ attention_mask=attention_mask,
+ hidden_size=config.hidden_size,
+ num_hidden_layers=config.num_hidden_layers,
+ num_attention_heads=config.num_attention_heads,
+ intermediate_size=config.intermediate_size,
+ intermediate_act_fn=get_activation(config.hidden_act),
+ hidden_dropout_prob=config.hidden_dropout_prob,
+ attention_probs_dropout_prob=config.attention_probs_dropout_prob,
+ initializer_range=config.initializer_range,
+ do_return_all_layers=True)
+
+ self.sequence_output = self.all_encoder_layers[-1]
+ # The "pooler" converts the encoded sequence tensor of shape
+ # [batch_size, seq_length, hidden_size] to a tensor of shape
+ # [batch_size, hidden_size]. This is necessary for segment-level
+ # (or segment-pair-level) classification tasks where we need a fixed
+ # dimensional representation of the segment.
+ with tf.variable_scope("pooler"):
+ # We "pool" the model by simply taking the hidden state corresponding
+ # to the first token. We assume that this has been pre-trained
+ first_token_tensor = tf.squeeze(self.sequence_output[:, 0:1, :], axis=1)
+ self.pooled_output = tf.layers.dense(
+ first_token_tensor,
+ config.hidden_size,
+ activation=tf.tanh,
+ kernel_initializer=create_initializer(config.initializer_range))
+
+ def get_pooled_output(self):
+ return self.pooled_output
+
+ def get_sequence_output(self):
+ """Gets final hidden layer of encoder.
+
+ Returns:
+ float Tensor of shape [batch_size, seq_length, hidden_size] corresponding
+ to the final hidden of the transformer encoder.
+ """
+ return self.sequence_output
+
+ def get_all_encoder_layers(self):
+ return self.all_encoder_layers
+
+ def get_embedding_output(self):
+ """Gets output of the embedding lookup (i.e., input to the transformer).
+
+ Returns:
+ float Tensor of shape [batch_size, seq_length, hidden_size] corresponding
+ to the output of the embedding layer, after summing the word
+ embeddings with the positional embeddings and the token type embeddings,
+ then performing layer normalization. This is the input to the transformer.
+ """
+ return self.embedding_output
+
+ def get_embedding_table(self):
+ return self.embedding_table
+
+
+def gelu(x):
+ """Gaussian Error Linear Unit.
+
+ This is a smoother version of the RELU.
+ Original paper: https://arxiv.org/abs/1606.08415
+ Args:
+ x: float Tensor to perform activation.
+
+ Returns:
+ `x` with the GELU activation applied.
+ """
+ cdf = 0.5 * (1.0 + tf.tanh(
+ (np.sqrt(2 / np.pi) * (x + 0.044715 * tf.pow(x, 3)))))
+ return x * cdf
+
+
+def get_activation(activation_string):
+ """Maps a string to a Python function, e.g., "relu" => `tf.nn.relu`.
+
+ Args:
+ activation_string: String name of the activation function.
+
+ Returns:
+ A Python function corresponding to the activation function. If
+ `activation_string` is None, empty, or "linear", this will return None.
+ If `activation_string` is not a string, it will return `activation_string`.
+
+ Raises:
+ ValueError: The `activation_string` does not correspond to a known
+ activation.
+ """
+
+ # We assume that anything that"s not a string is already an activation
+ # function, so we just return it.
+ if not isinstance(activation_string, six.string_types):
+ return activation_string
+
+ if not activation_string:
+ return None
+
+ act = activation_string.lower()
+ if act == "linear":
+ return None
+ elif act == "relu":
+ return tf.nn.relu
+ elif act == "gelu":
+ return gelu
+ elif act == "tanh":
+ return tf.tanh
+ else:
+ raise ValueError("Unsupported activation: %s" % act)
+
+
+def get_assignment_map_from_checkpoint(tvars, init_checkpoint):
+ """Compute the union of the current variables and checkpoint variables."""
+ assignment_map = {}
+ initialized_variable_names = {}
+
+ name_to_variable = collections.OrderedDict()
+ for var in tvars:
+ name = var.name
+ m = re.match("^(.*):\\d+$", name)
+ if m is not None:
+ name = m.group(1)
+ name_to_variable[name] = var
+
+ init_vars = tf.train.list_variables(init_checkpoint)
+
+ assignment_map = collections.OrderedDict()
+ for x in init_vars:
+ (name, var) = (x[0], x[1])
+ if name not in name_to_variable:
+ continue
+ assignment_map[name] = name
+ initialized_variable_names[name] = 1
+ initialized_variable_names[name + ":0"] = 1
+
+ return (assignment_map, initialized_variable_names)
+
+
+def dropout(input_tensor, dropout_prob):
+ """Perform dropout.
+
+ Args:
+ input_tensor: float Tensor.
+ dropout_prob: Python float. The probability of dropping out a value (NOT of
+ *keeping* a dimension as in `tf.nn.dropout`).
+
+ Returns:
+ A version of `input_tensor` with dropout applied.
+ """
+ if dropout_prob is None or dropout_prob == 0.0:
+ return input_tensor
+
+ output = tf.nn.dropout(input_tensor, 1.0 - dropout_prob)
+ return output
+
+
+def layer_norm(input_tensor, name=None):
+ """Run layer normalization on the last dimension of the tensor."""
+ return tf.contrib.layers.layer_norm(
+ inputs=input_tensor, begin_norm_axis=-1, begin_params_axis=-1, scope=name)
+
+
+def layer_norm_and_dropout(input_tensor, dropout_prob, name=None):
+ """Runs layer normalization followed by dropout."""
+ output_tensor = layer_norm(input_tensor, name)
+ output_tensor = dropout(output_tensor, dropout_prob)
+ return output_tensor
+
+
+def create_initializer(initializer_range=0.02):
+ """Creates a `truncated_normal_initializer` with the given range."""
+ return tf.truncated_normal_initializer(stddev=initializer_range)
+
+
+def embedding_lookup(input_ids,
+ vocab_size,
+ embedding_size=128,
+ initializer_range=0.02,
+ word_embedding_name="word_embeddings",
+ use_one_hot_embeddings=False):
+ """Looks up words embeddings for id tensor.
+
+ Args:
+ input_ids: int32 Tensor of shape [batch_size, seq_length] containing word
+ ids.
+ vocab_size: int. Size of the embedding vocabulary.
+ embedding_size: int. Width of the word embeddings.
+ initializer_range: float. Embedding initialization range.
+ word_embedding_name: string. Name of the embedding table.
+ use_one_hot_embeddings: bool. If True, use one-hot method for word
+ embeddings. If False, use `tf.gather()`.
+
+ Returns:
+ float Tensor of shape [batch_size, seq_length, embedding_size].
+ """
+ # This function assumes that the input is of shape [batch_size, seq_length,
+ # num_inputs].
+ #
+ # If the input is a 2D tensor of shape [batch_size, seq_length], we
+ # reshape to [batch_size, seq_length, 1].
+ if input_ids.shape.ndims == 2:
+ input_ids = tf.expand_dims(input_ids, axis=[-1])
+
+ embedding_table = tf.get_variable(
+ name=word_embedding_name,
+ shape=[vocab_size, embedding_size],
+ initializer=create_initializer(initializer_range))
+
+ flat_input_ids = tf.reshape(input_ids, [-1])
+ if use_one_hot_embeddings:
+ one_hot_input_ids = tf.one_hot(flat_input_ids, depth=vocab_size)
+ output = tf.matmul(one_hot_input_ids, embedding_table)
+ else:
+ output = tf.gather(embedding_table, flat_input_ids)
+
+ input_shape = get_shape_list(input_ids)
+
+ output = tf.reshape(output,
+ input_shape[0:-1] + [input_shape[-1] * embedding_size])
+ return (output, embedding_table)
+
+
+def embedding_postprocessor(input_tensor,
+ use_token_type=False,
+ token_type_ids=None,
+ token_type_vocab_size=16,
+ token_type_embedding_name="token_type_embeddings",
+ use_position_embeddings=True,
+ position_embedding_name="position_embeddings",
+ initializer_range=0.02,
+ max_position_embeddings=512,
+ dropout_prob=0.1):
+ """Performs various post-processing on a word embedding tensor.
+
+ Args:
+ input_tensor: float Tensor of shape [batch_size, seq_length,
+ embedding_size].
+ use_token_type: bool. Whether to add embeddings for `token_type_ids`.
+ token_type_ids: (optional) int32 Tensor of shape [batch_size, seq_length].
+ Must be specified if `use_token_type` is True.
+ token_type_vocab_size: int. The vocabulary size of `token_type_ids`.
+ token_type_embedding_name: string. The name of the embedding table variable
+ for token type ids.
+ use_position_embeddings: bool. Whether to add position embeddings for the
+ position of each token in the sequence.
+ position_embedding_name: string. The name of the embedding table variable
+ for positional embeddings.
+ initializer_range: float. Range of the weight initialization.
+ max_position_embeddings: int. Maximum sequence length that might ever be
+ used with this model. This can be longer than the sequence length of
+ input_tensor, but cannot be shorter.
+ dropout_prob: float. Dropout probability applied to the final output tensor.
+
+ Returns:
+ float tensor with same shape as `input_tensor`.
+
+ Raises:
+ ValueError: One of the tensor shapes or input values is invalid.
+ """
+ input_shape = get_shape_list(input_tensor, expected_rank=3)
+ batch_size = input_shape[0]
+ seq_length = input_shape[1]
+ width = input_shape[2]
+
+ output = input_tensor
+
+ if use_token_type:
+ if token_type_ids is None:
+ raise ValueError("`token_type_ids` must be specified if"
+ "`use_token_type` is True.")
+ token_type_table = tf.get_variable(
+ name=token_type_embedding_name,
+ shape=[token_type_vocab_size, width],
+ initializer=create_initializer(initializer_range))
+ # This vocab will be small so we always do one-hot here, since it is always
+ # faster for a small vocabulary.
+ flat_token_type_ids = tf.reshape(token_type_ids, [-1])
+ one_hot_ids = tf.one_hot(flat_token_type_ids, depth=token_type_vocab_size)
+ token_type_embeddings = tf.matmul(one_hot_ids, token_type_table)
+ token_type_embeddings = tf.reshape(token_type_embeddings,
+ [batch_size, seq_length, width])
+ output += token_type_embeddings
+
+ if use_position_embeddings:
+ assert_op = tf.assert_less_equal(seq_length, max_position_embeddings)
+ with tf.control_dependencies([assert_op]):
+ full_position_embeddings = tf.get_variable(
+ name=position_embedding_name,
+ shape=[max_position_embeddings, width],
+ initializer=create_initializer(initializer_range))
+ # Since the position embedding table is a learned variable, we create it
+ # using a (long) sequence length `max_position_embeddings`. The actual
+ # sequence length might be shorter than this, for faster training of
+ # tasks that do not have long sequences.
+ #
+ # So `full_position_embeddings` is effectively an embedding table
+ # for position [0, 1, 2, ..., max_position_embeddings-1], and the current
+ # sequence has positions [0, 1, 2, ... seq_length-1], so we can just
+ # perform a slice.
+ position_embeddings = tf.slice(full_position_embeddings, [0, 0],
+ [seq_length, -1])
+ num_dims = len(output.shape.as_list())
+
+ # Only the last two dimensions are relevant (`seq_length` and `width`), so
+ # we broadcast among the first dimensions, which is typically just
+ # the batch size.
+ position_broadcast_shape = []
+ for _ in range(num_dims - 2):
+ position_broadcast_shape.append(1)
+ position_broadcast_shape.extend([seq_length, width])
+ position_embeddings = tf.reshape(position_embeddings,
+ position_broadcast_shape)
+ output += position_embeddings
+
+ output = layer_norm_and_dropout(output, dropout_prob)
+ return output
+
+
+def create_attention_mask_from_input_mask(from_tensor, to_mask):
+ """Create 3D attention mask from a 2D tensor mask.
+
+ Args:
+ from_tensor: 2D or 3D Tensor of shape [batch_size, from_seq_length, ...].
+ to_mask: int32 Tensor of shape [batch_size, to_seq_length].
+
+ Returns:
+ float Tensor of shape [batch_size, from_seq_length, to_seq_length].
+ """
+ from_shape = get_shape_list(from_tensor, expected_rank=[2, 3])
+ batch_size = from_shape[0]
+ from_seq_length = from_shape[1]
+
+ to_shape = get_shape_list(to_mask, expected_rank=2)
+ to_seq_length = to_shape[1]
+
+ to_mask = tf.cast(
+ tf.reshape(to_mask, [batch_size, 1, to_seq_length]), tf.float32)
+
+ # We don't assume that `from_tensor` is a mask (although it could be). We
+ # don't actually care if we attend *from* padding tokens (only *to* padding)
+ # tokens so we create a tensor of all ones.
+ #
+ # `broadcast_ones` = [batch_size, from_seq_length, 1]
+ broadcast_ones = tf.ones(
+ shape=[batch_size, from_seq_length, 1], dtype=tf.float32)
+
+ # Here we broadcast along two dimensions to create the mask.
+ mask = broadcast_ones * to_mask
+
+ return mask
+
+
+def attention_layer(from_tensor,
+ to_tensor,
+ attention_mask=None,
+ num_attention_heads=1,
+ size_per_head=512,
+ query_act=None,
+ key_act=None,
+ value_act=None,
+ attention_probs_dropout_prob=0.0,
+ initializer_range=0.02,
+ do_return_2d_tensor=False,
+ batch_size=None,
+ from_seq_length=None,
+ to_seq_length=None):
+ """Performs multi-headed attention from `from_tensor` to `to_tensor`.
+
+ This is an implementation of multi-headed attention based on "Attention
+ is all you Need". If `from_tensor` and `to_tensor` are the same, then
+ this is self-attention. Each timestep in `from_tensor` attends to the
+ corresponding sequence in `to_tensor`, and returns a fixed-with vector.
+
+ This function first projects `from_tensor` into a "query" tensor and
+ `to_tensor` into "key" and "value" tensors. These are (effectively) a list
+ of tensors of length `num_attention_heads`, where each tensor is of shape
+ [batch_size, seq_length, size_per_head].
+
+ Then, the query and key tensors are dot-producted and scaled. These are
+ softmaxed to obtain attention probabilities. The value tensors are then
+ interpolated by these probabilities, then concatenated back to a single
+ tensor and returned.
+
+ In practice, the multi-headed attention are done with transposes and
+ reshapes rather than actual separate tensors.
+
+ Args:
+ from_tensor: float Tensor of shape [batch_size, from_seq_length,
+ from_width].
+ to_tensor: float Tensor of shape [batch_size, to_seq_length, to_width].
+ attention_mask: (optional) int32 Tensor of shape [batch_size,
+ from_seq_length, to_seq_length]. The values should be 1 or 0. The
+ attention scores will effectively be set to -infinity for any positions in
+ the mask that are 0, and will be unchanged for positions that are 1.
+ num_attention_heads: int. Number of attention heads.
+ size_per_head: int. Size of each attention head.
+ query_act: (optional) Activation function for the query transform.
+ key_act: (optional) Activation function for the key transform.
+ value_act: (optional) Activation function for the value transform.
+ attention_probs_dropout_prob: (optional) float. Dropout probability of the
+ attention probabilities.
+ initializer_range: float. Range of the weight initializer.
+ do_return_2d_tensor: bool. If True, the output will be of shape [batch_size
+ * from_seq_length, num_attention_heads * size_per_head]. If False, the
+ output will be of shape [batch_size, from_seq_length, num_attention_heads
+ * size_per_head].
+ batch_size: (Optional) int. If the input is 2D, this might be the batch size
+ of the 3D version of the `from_tensor` and `to_tensor`.
+ from_seq_length: (Optional) If the input is 2D, this might be the seq length
+ of the 3D version of the `from_tensor`.
+ to_seq_length: (Optional) If the input is 2D, this might be the seq length
+ of the 3D version of the `to_tensor`.
+
+ Returns:
+ float Tensor of shape [batch_size, from_seq_length,
+ num_attention_heads * size_per_head]. (If `do_return_2d_tensor` is
+ true, this will be of shape [batch_size * from_seq_length,
+ num_attention_heads * size_per_head]).
+
+ Raises:
+ ValueError: Any of the arguments or tensor shapes are invalid.
+ """
+
+ def transpose_for_scores(input_tensor, batch_size, num_attention_heads,
+ seq_length, width):
+ output_tensor = tf.reshape(
+ input_tensor, [batch_size, seq_length, num_attention_heads, width])
+
+ output_tensor = tf.transpose(output_tensor, [0, 2, 1, 3])
+ return output_tensor
+
+ from_shape = get_shape_list(from_tensor, expected_rank=[2, 3])
+ to_shape = get_shape_list(to_tensor, expected_rank=[2, 3])
+
+ if len(from_shape) != len(to_shape):
+ raise ValueError(
+ "The rank of `from_tensor` must match the rank of `to_tensor`.")
+
+ if len(from_shape) == 3:
+ batch_size = from_shape[0]
+ from_seq_length = from_shape[1]
+ to_seq_length = to_shape[1]
+ elif len(from_shape) == 2:
+ if (batch_size is None or from_seq_length is None or to_seq_length is None):
+ raise ValueError(
+ "When passing in rank 2 tensors to attention_layer, the values "
+ "for `batch_size`, `from_seq_length`, and `to_seq_length` "
+ "must all be specified.")
+
+ # Scalar dimensions referenced here:
+ # B = batch size (number of sequences)
+ # F = `from_tensor` sequence length
+ # T = `to_tensor` sequence length
+ # N = `num_attention_heads`
+ # H = `size_per_head`
+
+ from_tensor_2d = reshape_to_matrix(from_tensor)
+ to_tensor_2d = reshape_to_matrix(to_tensor)
+
+ # `query_layer` = [B*F, N*H]
+ query_layer = tf.layers.dense(
+ from_tensor_2d,
+ num_attention_heads * size_per_head,
+ activation=query_act,
+ name="query",
+ kernel_initializer=create_initializer(initializer_range))
+
+ # `key_layer` = [B*T, N*H]
+ key_layer = tf.layers.dense(
+ to_tensor_2d,
+ num_attention_heads * size_per_head,
+ activation=key_act,
+ name="key",
+ kernel_initializer=create_initializer(initializer_range))
+
+ # `value_layer` = [B*T, N*H]
+ value_layer = tf.layers.dense(
+ to_tensor_2d,
+ num_attention_heads * size_per_head,
+ activation=value_act,
+ name="value",
+ kernel_initializer=create_initializer(initializer_range))
+
+ # `query_layer` = [B, N, F, H]
+ query_layer = transpose_for_scores(query_layer, batch_size,
+ num_attention_heads, from_seq_length,
+ size_per_head)
+
+ # `key_layer` = [B, N, T, H]
+ key_layer = transpose_for_scores(key_layer, batch_size, num_attention_heads,
+ to_seq_length, size_per_head)
+
+ # Take the dot product between "query" and "key" to get the raw
+ # attention scores.
+ # `attention_scores` = [B, N, F, T]
+ attention_scores = tf.matmul(query_layer, key_layer, transpose_b=True)
+ attention_scores = tf.multiply(attention_scores,
+ 1.0 / math.sqrt(float(size_per_head)))
+
+ if attention_mask is not None:
+ # `attention_mask` = [B, 1, F, T]
+ attention_mask = tf.expand_dims(attention_mask, axis=[1])
+
+ # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
+ # masked positions, this operation will create a tensor which is 0.0 for
+ # positions we want to attend and -10000.0 for masked positions.
+ adder = (1.0 - tf.cast(attention_mask, tf.float32)) * -10000.0
+
+ # Since we are adding it to the raw scores before the softmax, this is
+ # effectively the same as removing these entirely.
+ attention_scores += adder
+
+ # Normalize the attention scores to probabilities.
+ # `attention_probs` = [B, N, F, T]
+ attention_probs = tf.nn.softmax(attention_scores)
+
+ # This is actually dropping out entire tokens to attend to, which might
+ # seem a bit unusual, but is taken from the original Transformer paper.
+ attention_probs = dropout(attention_probs, attention_probs_dropout_prob)
+
+ # `value_layer` = [B, T, N, H]
+ value_layer = tf.reshape(
+ value_layer,
+ [batch_size, to_seq_length, num_attention_heads, size_per_head])
+
+ # `value_layer` = [B, N, T, H]
+ value_layer = tf.transpose(value_layer, [0, 2, 1, 3])
+
+ # `context_layer` = [B, N, F, H]
+ context_layer = tf.matmul(attention_probs, value_layer)
+
+ # `context_layer` = [B, F, N, H]
+ context_layer = tf.transpose(context_layer, [0, 2, 1, 3])
+
+ if do_return_2d_tensor:
+ # `context_layer` = [B*F, N*H]
+ context_layer = tf.reshape(
+ context_layer,
+ [batch_size * from_seq_length, num_attention_heads * size_per_head])
+ else:
+ # `context_layer` = [B, F, N*H]
+ context_layer = tf.reshape(
+ context_layer,
+ [batch_size, from_seq_length, num_attention_heads * size_per_head])
+
+ return context_layer
+
+
+def transformer_model(input_tensor,
+ attention_mask=None,
+ hidden_size=768,
+ num_hidden_layers=12,
+ num_attention_heads=12,
+ intermediate_size=3072,
+ intermediate_act_fn=gelu,
+ hidden_dropout_prob=0.1,
+ attention_probs_dropout_prob=0.1,
+ initializer_range=0.02,
+ do_return_all_layers=False):
+ """Multi-headed, multi-layer Transformer from "Attention is All You Need".
+
+ This is almost an exact implementation of the original Transformer encoder.
+
+ See the original paper:
+ https://arxiv.org/abs/1706.03762
+
+ Also see:
+ https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/transformer.py
+
+ Args:
+ input_tensor: float Tensor of shape [batch_size, seq_length, hidden_size].
+ attention_mask: (optional) int32 Tensor of shape [batch_size, seq_length,
+ seq_length], with 1 for positions that can be attended to and 0 in
+ positions that should not be.
+ hidden_size: int. Hidden size of the Transformer.
+ num_hidden_layers: int. Number of layers (blocks) in the Transformer.
+ num_attention_heads: int. Number of attention heads in the Transformer.
+ intermediate_size: int. The size of the "intermediate" (a.k.a., feed
+ forward) layer.
+ intermediate_act_fn: function. The non-linear activation function to apply
+ to the output of the intermediate/feed-forward layer.
+ hidden_dropout_prob: float. Dropout probability for the hidden layers.
+ attention_probs_dropout_prob: float. Dropout probability of the attention
+ probabilities.
+ initializer_range: float. Range of the initializer (stddev of truncated
+ normal).
+ do_return_all_layers: Whether to also return all layers or just the final
+ layer.
+
+ Returns:
+ float Tensor of shape [batch_size, seq_length, hidden_size], the final
+ hidden layer of the Transformer.
+
+ Raises:
+ ValueError: A Tensor shape or parameter is invalid.
+ """
+ if hidden_size % num_attention_heads != 0:
+ raise ValueError(
+ "The hidden size (%d) is not a multiple of the number of attention "
+ "heads (%d)" % (hidden_size, num_attention_heads))
+
+ attention_head_size = int(hidden_size / num_attention_heads)
+ input_shape = get_shape_list(input_tensor, expected_rank=3)
+ batch_size = input_shape[0]
+ seq_length = input_shape[1]
+ input_width = input_shape[2]
+
+ # The Transformer performs sum residuals on all layers so the input needs
+ # to be the same as the hidden size.
+ if input_width != hidden_size:
+ raise ValueError("The width of the input tensor (%d) != hidden size (%d)" %
+ (input_width, hidden_size))
+
+ # We keep the representation as a 2D tensor to avoid re-shaping it back and
+ # forth from a 3D tensor to a 2D tensor. Re-shapes are normally free on
+ # the GPU/CPU but may not be free on the TPU, so we want to minimize them to
+ # help the optimizer.
+ prev_output = reshape_to_matrix(input_tensor)
+
+ all_layer_outputs = []
+ for layer_idx in range(num_hidden_layers):
+ with tf.variable_scope("layer_%d" % layer_idx):
+ layer_input = prev_output
+
+ with tf.variable_scope("attention"):
+ attention_heads = []
+ with tf.variable_scope("self"):
+ attention_head = attention_layer(
+ from_tensor=layer_input,
+ to_tensor=layer_input,
+ attention_mask=attention_mask,
+ num_attention_heads=num_attention_heads,
+ size_per_head=attention_head_size,
+ attention_probs_dropout_prob=attention_probs_dropout_prob,
+ initializer_range=initializer_range,
+ do_return_2d_tensor=True,
+ batch_size=batch_size,
+ from_seq_length=seq_length,
+ to_seq_length=seq_length)
+ attention_heads.append(attention_head)
+
+ attention_output = None
+ if len(attention_heads) == 1:
+ attention_output = attention_heads[0]
+ else:
+ # In the case where we have other sequences, we just concatenate
+ # them to the self-attention head before the projection.
+ attention_output = tf.concat(attention_heads, axis=-1)
+
+ # Run a linear projection of `hidden_size` then add a residual
+ # with `layer_input`.
+ with tf.variable_scope("output"):
+ attention_output = tf.layers.dense(
+ attention_output,
+ hidden_size,
+ kernel_initializer=create_initializer(initializer_range))
+ attention_output = dropout(attention_output, hidden_dropout_prob)
+ attention_output = layer_norm(attention_output + layer_input)
+
+ # The activation is only applied to the "intermediate" hidden layer.
+ with tf.variable_scope("intermediate"):
+ intermediate_output = tf.layers.dense(
+ attention_output,
+ intermediate_size,
+ activation=intermediate_act_fn,
+ kernel_initializer=create_initializer(initializer_range))
+
+ # Down-project back to `hidden_size` then add the residual.
+ with tf.variable_scope("output"):
+ layer_output = tf.layers.dense(
+ intermediate_output,
+ hidden_size,
+ kernel_initializer=create_initializer(initializer_range))
+ layer_output = dropout(layer_output, hidden_dropout_prob)
+ layer_output = layer_norm(layer_output + attention_output)
+ prev_output = layer_output
+ all_layer_outputs.append(layer_output)
+
+ if do_return_all_layers:
+ final_outputs = []
+ for layer_output in all_layer_outputs:
+ final_output = reshape_from_matrix(layer_output, input_shape)
+ final_outputs.append(final_output)
+ return final_outputs
+ else:
+ final_output = reshape_from_matrix(prev_output, input_shape)
+ return final_output
+
+
+def get_shape_list(tensor, expected_rank=None, name=None):
+ """Returns a list of the shape of tensor, preferring static dimensions.
+
+ Args:
+ tensor: A tf.Tensor object to find the shape of.
+ expected_rank: (optional) int. The expected rank of `tensor`. If this is
+ specified and the `tensor` has a different rank, and exception will be
+ thrown.
+ name: Optional name of the tensor for the error message.
+
+ Returns:
+ A list of dimensions of the shape of tensor. All static dimensions will
+ be returned as python integers, and dynamic dimensions will be returned
+ as tf.Tensor scalars.
+ """
+ if name is None:
+ name = tensor.name
+
+ if expected_rank is not None:
+ assert_rank(tensor, expected_rank, name)
+
+ shape = tensor.shape.as_list()
+
+ non_static_indexes = []
+ for (index, dim) in enumerate(shape):
+ if dim is None:
+ non_static_indexes.append(index)
+
+ if not non_static_indexes:
+ return shape
+
+ dyn_shape = tf.shape(tensor)
+ for index in non_static_indexes:
+ shape[index] = dyn_shape[index]
+ return shape
+
+
+def reshape_to_matrix(input_tensor):
+ """Reshapes a >= rank 2 tensor to a rank 2 tensor (i.e., a matrix)."""
+ ndims = input_tensor.shape.ndims
+ if ndims < 2:
+ raise ValueError("Input tensor must have at least rank 2. Shape = %s" %
+ (input_tensor.shape))
+ if ndims == 2:
+ return input_tensor
+
+ width = input_tensor.shape[-1]
+ output_tensor = tf.reshape(input_tensor, [-1, width])
+ return output_tensor
+
+
+def reshape_from_matrix(output_tensor, orig_shape_list):
+ """Reshapes a rank 2 tensor back to its original rank >= 2 tensor."""
+ if len(orig_shape_list) == 2:
+ return output_tensor
+
+ output_shape = get_shape_list(output_tensor)
+
+ orig_dims = orig_shape_list[0:-1]
+ width = output_shape[-1]
+
+ return tf.reshape(output_tensor, orig_dims + [width])
+
+
+def assert_rank(tensor, expected_rank, name=None):
+ """Raises an exception if the tensor rank is not of the expected rank.
+
+ Args:
+ tensor: A tf.Tensor to check the rank of.
+ expected_rank: Python integer or list of integers, expected rank.
+ name: Optional name of the tensor for the error message.
+
+ Raises:
+ ValueError: If the expected shape doesn't match the actual shape.
+ """
+ if name is None:
+ name = tensor.name
+
+ expected_rank_dict = {}
+ if isinstance(expected_rank, six.integer_types):
+ expected_rank_dict[expected_rank] = True
+ else:
+ for x in expected_rank:
+ expected_rank_dict[x] = True
+
+ actual_rank = tensor.shape.ndims
+ if actual_rank not in expected_rank_dict:
+ scope_name = tf.get_variable_scope().name
+ raise ValueError(
+ "For the tensor `%s` in scope `%s`, the actual rank "
+ "`%d` (shape = %s) is not equal to the expected rank `%s`" %
+ (name, scope_name, actual_rank, str(tensor.shape), str(expected_rank)))
diff --git a/bert-master/bert-master/modeling_test.py b/bert-master/bert-master/modeling_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..817ad2db54068fad9ff1905d9f708a688825dff3
--- /dev/null
+++ b/bert-master/bert-master/modeling_test.py
@@ -0,0 +1,277 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import json
+import random
+import re
+
+import modeling
+import six
+import tensorflow as tf
+
+
+class BertModelTest(tf.test.TestCase):
+
+ class BertModelTester(object):
+
+ def __init__(self,
+ parent,
+ batch_size=13,
+ seq_length=7,
+ is_training=True,
+ use_input_mask=True,
+ use_token_type_ids=True,
+ vocab_size=99,
+ hidden_size=32,
+ num_hidden_layers=5,
+ num_attention_heads=4,
+ intermediate_size=37,
+ hidden_act="gelu",
+ hidden_dropout_prob=0.1,
+ attention_probs_dropout_prob=0.1,
+ max_position_embeddings=512,
+ type_vocab_size=16,
+ initializer_range=0.02,
+ scope=None):
+ self.parent = parent
+ self.batch_size = batch_size
+ self.seq_length = seq_length
+ self.is_training = is_training
+ self.use_input_mask = use_input_mask
+ self.use_token_type_ids = use_token_type_ids
+ self.vocab_size = vocab_size
+ self.hidden_size = hidden_size
+ self.num_hidden_layers = num_hidden_layers
+ self.num_attention_heads = num_attention_heads
+ self.intermediate_size = intermediate_size
+ self.hidden_act = hidden_act
+ self.hidden_dropout_prob = hidden_dropout_prob
+ self.attention_probs_dropout_prob = attention_probs_dropout_prob
+ self.max_position_embeddings = max_position_embeddings
+ self.type_vocab_size = type_vocab_size
+ self.initializer_range = initializer_range
+ self.scope = scope
+
+ def create_model(self):
+ input_ids = BertModelTest.ids_tensor([self.batch_size, self.seq_length],
+ self.vocab_size)
+
+ input_mask = None
+ if self.use_input_mask:
+ input_mask = BertModelTest.ids_tensor(
+ [self.batch_size, self.seq_length], vocab_size=2)
+
+ token_type_ids = None
+ if self.use_token_type_ids:
+ token_type_ids = BertModelTest.ids_tensor(
+ [self.batch_size, self.seq_length], self.type_vocab_size)
+
+ config = modeling.BertConfig(
+ vocab_size=self.vocab_size,
+ hidden_size=self.hidden_size,
+ num_hidden_layers=self.num_hidden_layers,
+ num_attention_heads=self.num_attention_heads,
+ intermediate_size=self.intermediate_size,
+ hidden_act=self.hidden_act,
+ hidden_dropout_prob=self.hidden_dropout_prob,
+ attention_probs_dropout_prob=self.attention_probs_dropout_prob,
+ max_position_embeddings=self.max_position_embeddings,
+ type_vocab_size=self.type_vocab_size,
+ initializer_range=self.initializer_range)
+
+ model = modeling.BertModel(
+ config=config,
+ is_training=self.is_training,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ token_type_ids=token_type_ids,
+ scope=self.scope)
+
+ outputs = {
+ "embedding_output": model.get_embedding_output(),
+ "sequence_output": model.get_sequence_output(),
+ "pooled_output": model.get_pooled_output(),
+ "all_encoder_layers": model.get_all_encoder_layers(),
+ }
+ return outputs
+
+ def check_output(self, result):
+ self.parent.assertAllEqual(
+ result["embedding_output"].shape,
+ [self.batch_size, self.seq_length, self.hidden_size])
+
+ self.parent.assertAllEqual(
+ result["sequence_output"].shape,
+ [self.batch_size, self.seq_length, self.hidden_size])
+
+ self.parent.assertAllEqual(result["pooled_output"].shape,
+ [self.batch_size, self.hidden_size])
+
+ def test_default(self):
+ self.run_tester(BertModelTest.BertModelTester(self))
+
+ def test_config_to_json_string(self):
+ config = modeling.BertConfig(vocab_size=99, hidden_size=37)
+ obj = json.loads(config.to_json_string())
+ self.assertEqual(obj["vocab_size"], 99)
+ self.assertEqual(obj["hidden_size"], 37)
+
+ def run_tester(self, tester):
+ with self.test_session() as sess:
+ ops = tester.create_model()
+ init_op = tf.group(tf.global_variables_initializer(),
+ tf.local_variables_initializer())
+ sess.run(init_op)
+ output_result = sess.run(ops)
+ tester.check_output(output_result)
+
+ self.assert_all_tensors_reachable(sess, [init_op, ops])
+
+ @classmethod
+ def ids_tensor(cls, shape, vocab_size, rng=None, name=None):
+ """Creates a random int32 tensor of the shape within the vocab size."""
+ if rng is None:
+ rng = random.Random()
+
+ total_dims = 1
+ for dim in shape:
+ total_dims *= dim
+
+ values = []
+ for _ in range(total_dims):
+ values.append(rng.randint(0, vocab_size - 1))
+
+ return tf.constant(value=values, dtype=tf.int32, shape=shape, name=name)
+
+ def assert_all_tensors_reachable(self, sess, outputs):
+ """Checks that all the tensors in the graph are reachable from outputs."""
+ graph = sess.graph
+
+ ignore_strings = [
+ "^.*/assert_less_equal/.*$",
+ "^.*/dilation_rate$",
+ "^.*/Tensordot/concat$",
+ "^.*/Tensordot/concat/axis$",
+ "^testing/.*$",
+ ]
+
+ ignore_regexes = [re.compile(x) for x in ignore_strings]
+
+ unreachable = self.get_unreachable_ops(graph, outputs)
+ filtered_unreachable = []
+ for x in unreachable:
+ do_ignore = False
+ for r in ignore_regexes:
+ m = r.match(x.name)
+ if m is not None:
+ do_ignore = True
+ if do_ignore:
+ continue
+ filtered_unreachable.append(x)
+ unreachable = filtered_unreachable
+
+ self.assertEqual(
+ len(unreachable), 0, "The following ops are unreachable: %s" %
+ (" ".join([x.name for x in unreachable])))
+
+ @classmethod
+ def get_unreachable_ops(cls, graph, outputs):
+ """Finds all of the tensors in graph that are unreachable from outputs."""
+ outputs = cls.flatten_recursive(outputs)
+ output_to_op = collections.defaultdict(list)
+ op_to_all = collections.defaultdict(list)
+ assign_out_to_in = collections.defaultdict(list)
+
+ for op in graph.get_operations():
+ for x in op.inputs:
+ op_to_all[op.name].append(x.name)
+ for y in op.outputs:
+ output_to_op[y.name].append(op.name)
+ op_to_all[op.name].append(y.name)
+ if str(op.type) == "Assign":
+ for y in op.outputs:
+ for x in op.inputs:
+ assign_out_to_in[y.name].append(x.name)
+
+ assign_groups = collections.defaultdict(list)
+ for out_name in assign_out_to_in.keys():
+ name_group = assign_out_to_in[out_name]
+ for n1 in name_group:
+ assign_groups[n1].append(out_name)
+ for n2 in name_group:
+ if n1 != n2:
+ assign_groups[n1].append(n2)
+
+ seen_tensors = {}
+ stack = [x.name for x in outputs]
+ while stack:
+ name = stack.pop()
+ if name in seen_tensors:
+ continue
+ seen_tensors[name] = True
+
+ if name in output_to_op:
+ for op_name in output_to_op[name]:
+ if op_name in op_to_all:
+ for input_name in op_to_all[op_name]:
+ if input_name not in stack:
+ stack.append(input_name)
+
+ expanded_names = []
+ if name in assign_groups:
+ for assign_name in assign_groups[name]:
+ expanded_names.append(assign_name)
+
+ for expanded_name in expanded_names:
+ if expanded_name not in stack:
+ stack.append(expanded_name)
+
+ unreachable_ops = []
+ for op in graph.get_operations():
+ is_unreachable = False
+ all_names = [x.name for x in op.inputs] + [x.name for x in op.outputs]
+ for name in all_names:
+ if name not in seen_tensors:
+ is_unreachable = True
+ if is_unreachable:
+ unreachable_ops.append(op)
+ return unreachable_ops
+
+ @classmethod
+ def flatten_recursive(cls, item):
+ """Flattens (potentially nested) a tuple/dictionary/list to a list."""
+ output = []
+ if isinstance(item, list):
+ output.extend(item)
+ elif isinstance(item, tuple):
+ output.extend(list(item))
+ elif isinstance(item, dict):
+ for (_, v) in six.iteritems(item):
+ output.append(v)
+ else:
+ return [item]
+
+ flat_output = []
+ for x in output:
+ flat_output.extend(cls.flatten_recursive(x))
+ return flat_output
+
+
+if __name__ == "__main__":
+ tf.test.main()
diff --git a/bert-master/bert-master/multilingual.md b/bert-master/bert-master/multilingual.md
new file mode 100644
index 0000000000000000000000000000000000000000..13842579f2a6eb183622f4ad0ec3b1ce8a7a2caf
--- /dev/null
+++ b/bert-master/bert-master/multilingual.md
@@ -0,0 +1,303 @@
+## Models
+
+There are two multilingual models currently available. We do not plan to release
+more single-language models, but we may release `BERT-Large` versions of these
+two in the future:
+
+* **[`BERT-Base, Multilingual Cased (New, recommended)`](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip)**:
+ 104 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Base, Multilingual Uncased (Orig, not recommended)`](https://storage.googleapis.com/bert_models/2018_11_03/multilingual_L-12_H-768_A-12.zip)**:
+ 102 languages, 12-layer, 768-hidden, 12-heads, 110M parameters
+* **[`BERT-Base, Chinese`](https://storage.googleapis.com/bert_models/2018_11_03/chinese_L-12_H-768_A-12.zip)**:
+ Chinese Simplified and Traditional, 12-layer, 768-hidden, 12-heads, 110M
+ parameters
+
+**The `Multilingual Cased (New)` model also fixes normalization issues in many
+languages, so it is recommended in languages with non-Latin alphabets (and is
+often better for most languages with Latin alphabets). When using this model,
+make sure to pass `--do_lower_case=false` to `run_pretraining.py` and other
+scripts.**
+
+See the [list of languages](#list-of-languages) that the Multilingual model
+supports. The Multilingual model does include Chinese (and English), but if your
+fine-tuning data is Chinese-only, then the Chinese model will likely produce
+better results.
+
+## Results
+
+To evaluate these systems, we use the
+[XNLI dataset](https://github.com/facebookresearch/XNLI) dataset, which is a
+version of [MultiNLI](https://www.nyu.edu/projects/bowman/multinli/) where the
+dev and test sets have been translated (by humans) into 15 languages. Note that
+the training set was *machine* translated (we used the translations provided by
+XNLI, not Google NMT). For clarity, we only report on 6 languages below:
+
+
+
+| System | English | Chinese | Spanish | German | Arabic | Urdu |
+| --------------------------------- | -------- | -------- | -------- | -------- | -------- | -------- |
+| XNLI Baseline - Translate Train | 73.7 | 67.0 | 68.8 | 66.5 | 65.8 | 56.6 |
+| XNLI Baseline - Translate Test | 73.7 | 68.3 | 70.7 | 68.7 | 66.8 | 59.3 |
+| BERT - Translate Train Cased | **81.9** | **76.6** | **77.8** | **75.9** | **70.7** | 61.6 |
+| BERT - Translate Train Uncased | 81.4 | 74.2 | 77.3 | 75.2 | 70.5 | 61.7 |
+| BERT - Translate Test Uncased | 81.4 | 70.1 | 74.9 | 74.4 | 70.4 | **62.1** |
+| BERT - Zero Shot Uncased | 81.4 | 63.8 | 74.3 | 70.5 | 62.1 | 58.3 |
+
+
+
+The first two rows are baselines from the XNLI paper and the last three rows are
+our results with BERT.
+
+**Translate Train** means that the MultiNLI training set was machine translated
+from English into the foreign language. So training and evaluation were both
+done in the foreign language. Unfortunately, training was done on
+machine-translated data, so it is impossible to quantify how much of the lower
+accuracy (compared to English) is due to the quality of the machine translation
+vs. the quality of the pre-trained model.
+
+**Translate Test** means that the XNLI test set was machine translated from the
+foreign language into English. So training and evaluation were both done on
+English. However, test evaluation was done on machine-translated English, so the
+accuracy depends on the quality of the machine translation system.
+
+**Zero Shot** means that the Multilingual BERT system was fine-tuned on English
+MultiNLI, and then evaluated on the foreign language XNLI test. In this case,
+machine translation was not involved at all in either the pre-training or
+fine-tuning.
+
+Note that the English result is worse than the 84.2 MultiNLI baseline because
+this training used Multilingual BERT rather than English-only BERT. This implies
+that for high-resource languages, the Multilingual model is somewhat worse than
+a single-language model. However, it is not feasible for us to train and
+maintain dozens of single-language models. Therefore, if your goal is to maximize
+performance with a language other than English or Chinese, you might find it
+beneficial to run pre-training for additional steps starting from our
+Multilingual model on data from your language of interest.
+
+Here is a comparison of training Chinese models with the Multilingual
+`BERT-Base` and Chinese-only `BERT-Base`:
+
+System | Chinese
+----------------------- | -------
+XNLI Baseline | 67.0
+BERT Multilingual Model | 74.2
+BERT Chinese-only Model | 77.2
+
+Similar to English, the single-language model does 3% better than the
+Multilingual model.
+
+## Fine-tuning Example
+
+The multilingual model does **not** require any special consideration or API
+changes. We did update the implementation of `BasicTokenizer` in
+`tokenization.py` to support Chinese character tokenization, so please update if
+you forked it. However, we did not change the tokenization API.
+
+To test the new models, we did modify `run_classifier.py` to add support for the
+[XNLI dataset](https://github.com/facebookresearch/XNLI). This is a 15-language
+version of MultiNLI where the dev/test sets have been human-translated, and the
+training set has been machine-translated.
+
+To run the fine-tuning code, please download the
+[XNLI dev/test set](https://www.nyu.edu/projects/bowman/xnli/XNLI-1.0.zip) and the
+[XNLI machine-translated training set](https://www.nyu.edu/projects/bowman/xnli/XNLI-MT-1.0.zip)
+and then unpack both .zip files into some directory `$XNLI_DIR`.
+
+To run fine-tuning on XNLI. The language is hard-coded into `run_classifier.py`
+(Chinese by default), so please modify `XnliProcessor` if you want to run on
+another language.
+
+This is a large dataset, so this will training will take a few hours on a GPU
+(or about 30 minutes on a Cloud TPU). To run an experiment quickly for
+debugging, just set `num_train_epochs` to a small value like `0.1`.
+
+```shell
+export BERT_BASE_DIR=/path/to/bert/chinese_L-12_H-768_A-12 # or multilingual_L-12_H-768_A-12
+export XNLI_DIR=/path/to/xnli
+
+python run_classifier.py \
+ --task_name=XNLI \
+ --do_train=true \
+ --do_eval=true \
+ --data_dir=$XNLI_DIR \
+ --vocab_file=$BERT_BASE_DIR/vocab.txt \
+ --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+ --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
+ --max_seq_length=128 \
+ --train_batch_size=32 \
+ --learning_rate=5e-5 \
+ --num_train_epochs=2.0 \
+ --output_dir=/tmp/xnli_output/
+```
+
+With the Chinese-only model, the results should look something like this:
+
+```
+ ***** Eval results *****
+eval_accuracy = 0.774116
+eval_loss = 0.83554
+global_step = 24543
+loss = 0.74603
+```
+
+## Details
+
+### Data Source and Sampling
+
+The languages chosen were the
+[top 100 languages with the largest Wikipedias](https://meta.wikimedia.org/wiki/List_of_Wikipedias).
+The entire Wikipedia dump for each language (excluding user and talk pages) was
+taken as the training data for each language
+
+However, the size of the Wikipedia for a given language varies greatly, and
+therefore low-resource languages may be "under-represented" in terms of the
+neural network model (under the assumption that languages are "competing" for
+limited model capacity to some extent). At the same time, we also don't want
+to overfit the model by performing thousands of epochs over a tiny Wikipedia
+for a particular language.
+
+To balance these two factors, we performed exponentially smoothed weighting of
+the data during pre-training data creation (and WordPiece vocab creation). In
+other words, let's say that the probability of a language is *P(L)*, e.g.,
+*P(English) = 0.21* means that after concatenating all of the Wikipedias
+together, 21% of our data is English. We exponentiate each probability by some
+factor *S* and then re-normalize, and sample from that distribution. In our case
+we use *S=0.7*. So, high-resource languages like English will be under-sampled,
+and low-resource languages like Icelandic will be over-sampled. E.g., in the
+original distribution English would be sampled 1000x more than Icelandic, but
+after smoothing it's only sampled 100x more.
+
+### Tokenization
+
+For tokenization, we use a 110k shared WordPiece vocabulary. The word counts are
+weighted the same way as the data, so low-resource languages are upweighted by
+some factor. We intentionally do *not* use any marker to denote the input
+language (so that zero-shot training can work).
+
+Because Chinese (and Japanese Kanji and Korean Hanja) does not have whitespace
+characters, we add spaces around every character in the
+[CJK Unicode range](https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_\(Unicode_block\))
+before applying WordPiece. This means that Chinese is effectively
+character-tokenized. Note that the CJK Unicode block only includes
+Chinese-origin characters and does *not* include Hangul Korean or
+Katakana/Hiragana Japanese, which are tokenized with whitespace+WordPiece like
+all other languages.
+
+For all other languages, we apply the
+[same recipe as English](https://github.com/google-research/bert#tokenization):
+(a) lower casing+accent removal, (b) punctuation splitting, (c) whitespace
+tokenization. We understand that accent markers have substantial meaning in some
+languages, but felt that the benefits of reducing the effective vocabulary make
+up for this. Generally the strong contextual models of BERT should make up for
+any ambiguity introduced by stripping accent markers.
+
+### List of Languages
+
+The multilingual model supports the following languages. These languages were
+chosen because they are the top 100 languages with the largest Wikipedias:
+
+* Afrikaans
+* Albanian
+* Arabic
+* Aragonese
+* Armenian
+* Asturian
+* Azerbaijani
+* Bashkir
+* Basque
+* Bavarian
+* Belarusian
+* Bengali
+* Bishnupriya Manipuri
+* Bosnian
+* Breton
+* Bulgarian
+* Burmese
+* Catalan
+* Cebuano
+* Chechen
+* Chinese (Simplified)
+* Chinese (Traditional)
+* Chuvash
+* Croatian
+* Czech
+* Danish
+* Dutch
+* English
+* Estonian
+* Finnish
+* French
+* Galician
+* Georgian
+* German
+* Greek
+* Gujarati
+* Haitian
+* Hebrew
+* Hindi
+* Hungarian
+* Icelandic
+* Ido
+* Indonesian
+* Irish
+* Italian
+* Japanese
+* Javanese
+* Kannada
+* Kazakh
+* Kirghiz
+* Korean
+* Latin
+* Latvian
+* Lithuanian
+* Lombard
+* Low Saxon
+* Luxembourgish
+* Macedonian
+* Malagasy
+* Malay
+* Malayalam
+* Marathi
+* Minangkabau
+* Nepali
+* Newar
+* Norwegian (Bokmal)
+* Norwegian (Nynorsk)
+* Occitan
+* Persian (Farsi)
+* Piedmontese
+* Polish
+* Portuguese
+* Punjabi
+* Romanian
+* Russian
+* Scots
+* Serbian
+* Serbo-Croatian
+* Sicilian
+* Slovak
+* Slovenian
+* South Azerbaijani
+* Spanish
+* Sundanese
+* Swahili
+* Swedish
+* Tagalog
+* Tajik
+* Tamil
+* Tatar
+* Telugu
+* Turkish
+* Ukrainian
+* Urdu
+* Uzbek
+* Vietnamese
+* Volapük
+* Waray-Waray
+* Welsh
+* West Frisian
+* Western Punjabi
+* Yoruba
+
+The **Multilingual Cased (New)** release contains additionally **Thai** and
+**Mongolian**, which were not included in the original release.
diff --git a/bert-master/bert-master/optimization.py b/bert-master/bert-master/optimization.py
new file mode 100644
index 0000000000000000000000000000000000000000..d33dabd91dd965781d3dd912e0727ba23d7c2918
--- /dev/null
+++ b/bert-master/bert-master/optimization.py
@@ -0,0 +1,174 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Functions and classes related to optimization (weight updates)."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import re
+import tensorflow as tf
+
+
+def create_optimizer(loss, init_lr, num_train_steps, num_warmup_steps, use_tpu):
+ """Creates an optimizer training op."""
+ global_step = tf.train.get_or_create_global_step()
+
+ learning_rate = tf.constant(value=init_lr, shape=[], dtype=tf.float32)
+
+ # Implements linear decay of the learning rate.
+ learning_rate = tf.train.polynomial_decay(
+ learning_rate,
+ global_step,
+ num_train_steps,
+ end_learning_rate=0.0,
+ power=1.0,
+ cycle=False)
+
+ # Implements linear warmup. I.e., if global_step < num_warmup_steps, the
+ # learning rate will be `global_step/num_warmup_steps * init_lr`.
+ if num_warmup_steps:
+ global_steps_int = tf.cast(global_step, tf.int32)
+ warmup_steps_int = tf.constant(num_warmup_steps, dtype=tf.int32)
+
+ global_steps_float = tf.cast(global_steps_int, tf.float32)
+ warmup_steps_float = tf.cast(warmup_steps_int, tf.float32)
+
+ warmup_percent_done = global_steps_float / warmup_steps_float
+ warmup_learning_rate = init_lr * warmup_percent_done
+
+ is_warmup = tf.cast(global_steps_int < warmup_steps_int, tf.float32)
+ learning_rate = (
+ (1.0 - is_warmup) * learning_rate + is_warmup * warmup_learning_rate)
+
+ # It is recommended that you use this optimizer for fine tuning, since this
+ # is how the model was trained (note that the Adam m/v variables are NOT
+ # loaded from init_checkpoint.)
+ optimizer = AdamWeightDecayOptimizer(
+ learning_rate=learning_rate,
+ weight_decay_rate=0.01,
+ beta_1=0.9,
+ beta_2=0.999,
+ epsilon=1e-6,
+ exclude_from_weight_decay=["LayerNorm", "layer_norm", "bias"])
+
+ if use_tpu:
+ optimizer = tf.contrib.tpu.CrossShardOptimizer(optimizer)
+
+ tvars = tf.trainable_variables()
+ grads = tf.gradients(loss, tvars)
+
+ # This is how the model was pre-trained.
+ (grads, _) = tf.clip_by_global_norm(grads, clip_norm=1.0)
+
+ train_op = optimizer.apply_gradients(
+ zip(grads, tvars), global_step=global_step)
+
+ # Normally the global step update is done inside of `apply_gradients`.
+ # However, `AdamWeightDecayOptimizer` doesn't do this. But if you use
+ # a different optimizer, you should probably take this line out.
+ new_global_step = global_step + 1
+ train_op = tf.group(train_op, [global_step.assign(new_global_step)])
+ return train_op
+
+
+class AdamWeightDecayOptimizer(tf.train.Optimizer):
+ """A basic Adam optimizer that includes "correct" L2 weight decay."""
+
+ def __init__(self,
+ learning_rate,
+ weight_decay_rate=0.0,
+ beta_1=0.9,
+ beta_2=0.999,
+ epsilon=1e-6,
+ exclude_from_weight_decay=None,
+ name="AdamWeightDecayOptimizer"):
+ """Constructs a AdamWeightDecayOptimizer."""
+ super(AdamWeightDecayOptimizer, self).__init__(False, name)
+
+ self.learning_rate = learning_rate
+ self.weight_decay_rate = weight_decay_rate
+ self.beta_1 = beta_1
+ self.beta_2 = beta_2
+ self.epsilon = epsilon
+ self.exclude_from_weight_decay = exclude_from_weight_decay
+
+ def apply_gradients(self, grads_and_vars, global_step=None, name=None):
+ """See base class."""
+ assignments = []
+ for (grad, param) in grads_and_vars:
+ if grad is None or param is None:
+ continue
+
+ param_name = self._get_variable_name(param.name)
+
+ m = tf.get_variable(
+ name=param_name + "/adam_m",
+ shape=param.shape.as_list(),
+ dtype=tf.float32,
+ trainable=False,
+ initializer=tf.zeros_initializer())
+ v = tf.get_variable(
+ name=param_name + "/adam_v",
+ shape=param.shape.as_list(),
+ dtype=tf.float32,
+ trainable=False,
+ initializer=tf.zeros_initializer())
+
+ # Standard Adam update.
+ next_m = (
+ tf.multiply(self.beta_1, m) + tf.multiply(1.0 - self.beta_1, grad))
+ next_v = (
+ tf.multiply(self.beta_2, v) + tf.multiply(1.0 - self.beta_2,
+ tf.square(grad)))
+
+ update = next_m / (tf.sqrt(next_v) + self.epsilon)
+
+ # Just adding the square of the weights to the loss function is *not*
+ # the correct way of using L2 regularization/weight decay with Adam,
+ # since that will interact with the m and v parameters in strange ways.
+ #
+ # Instead we want ot decay the weights in a manner that doesn't interact
+ # with the m/v parameters. This is equivalent to adding the square
+ # of the weights to the loss with plain (non-momentum) SGD.
+ if self._do_use_weight_decay(param_name):
+ update += self.weight_decay_rate * param
+
+ update_with_lr = self.learning_rate * update
+
+ next_param = param - update_with_lr
+
+ assignments.extend(
+ [param.assign(next_param),
+ m.assign(next_m),
+ v.assign(next_v)])
+ return tf.group(*assignments, name=name)
+
+ def _do_use_weight_decay(self, param_name):
+ """Whether to use L2 weight decay for `param_name`."""
+ if not self.weight_decay_rate:
+ return False
+ if self.exclude_from_weight_decay:
+ for r in self.exclude_from_weight_decay:
+ if re.search(r, param_name) is not None:
+ return False
+ return True
+
+ def _get_variable_name(self, param_name):
+ """Get the variable name from the tensor name."""
+ m = re.match("^(.*):\\d+$", param_name)
+ if m is not None:
+ param_name = m.group(1)
+ return param_name
diff --git a/bert-master/bert-master/optimization_test.py b/bert-master/bert-master/optimization_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..4f2dcf133f1bc4d4531fc9b82432d149be054b21
--- /dev/null
+++ b/bert-master/bert-master/optimization_test.py
@@ -0,0 +1,48 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import optimization
+import tensorflow as tf
+
+
+class OptimizationTest(tf.test.TestCase):
+
+ def test_adam(self):
+ with self.test_session() as sess:
+ w = tf.get_variable(
+ "w",
+ shape=[3],
+ initializer=tf.constant_initializer([0.1, -0.2, -0.1]))
+ x = tf.constant([0.4, 0.2, -0.5])
+ loss = tf.reduce_mean(tf.square(x - w))
+ tvars = tf.trainable_variables()
+ grads = tf.gradients(loss, tvars)
+ global_step = tf.train.get_or_create_global_step()
+ optimizer = optimization.AdamWeightDecayOptimizer(learning_rate=0.2)
+ train_op = optimizer.apply_gradients(zip(grads, tvars), global_step)
+ init_op = tf.group(tf.global_variables_initializer(),
+ tf.local_variables_initializer())
+ sess.run(init_op)
+ for _ in range(100):
+ sess.run(train_op)
+ w_np = sess.run(w)
+ self.assertAllClose(w_np.flat, [0.4, 0.2, -0.5], rtol=1e-2, atol=1e-2)
+
+
+if __name__ == "__main__":
+ tf.test.main()
diff --git a/bert-master/bert-master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb b/bert-master/bert-master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..466857f372fb2b7bccc73effed299ced67c164f9
--- /dev/null
+++ b/bert-master/bert-master/predicting_movie_reviews_with_bert_on_tf_hub.ipynb
@@ -0,0 +1,1231 @@
+{
+ "nbformat": 4,
+ "nbformat_minor": 0,
+ "metadata": {
+ "colab": {
+ "name": "Predicting Movie Reviews with BERT on TF Hub.ipynb",
+ "version": "0.3.2",
+ "provenance": [],
+ "collapsed_sections": []
+ },
+ "kernelspec": {
+ "name": "python3",
+ "display_name": "Python 3"
+ },
+ "accelerator": "GPU"
+ },
+ "cells": [
+ {
+ "metadata": {
+ "id": "j0a4mTk9o1Qg",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Copyright 2019 Google Inc.\n",
+ "\n",
+ "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
+ "# you may not use this file except in compliance with the License.\n",
+ "# You may obtain a copy of the License at\n",
+ "\n",
+ "# http://www.apache.org/licenses/LICENSE-2.0\n",
+ "\n",
+ "# Unless required by applicable law or agreed to in writing, software\n",
+ "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
+ "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
+ "# See the License for the specific language governing permissions and\n",
+ "# limitations under the License."
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "dCpvgG0vwXAZ",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "#Predicting Movie Review Sentiment with BERT on TF Hub"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "xiYrZKaHwV81",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "If you’ve been following Natural Language Processing over the past year, you’ve probably heard of BERT: Bidirectional Encoder Representations from Transformers. It’s a neural network architecture designed by Google researchers that’s totally transformed what’s state-of-the-art for NLP tasks, like text classification, translation, summarization, and question answering.\n",
+ "\n",
+ "Now that BERT's been added to [TF Hub](https://www.tensorflow.org/hub) as a loadable module, it's easy(ish) to add into existing Tensorflow text pipelines. In an existing pipeline, BERT can replace text embedding layers like ELMO and GloVE. Alternatively, [finetuning](http://wiki.fast.ai/index.php/Fine_tuning) BERT can provide both an accuracy boost and faster training time in many cases.\n",
+ "\n",
+ "Here, we'll train a model to predict whether an IMDB movie review is positive or negative using BERT in Tensorflow with tf hub. Some code was adapted from [this colab notebook](https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb). Let's get started!"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "hsZvic2YxnTz",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "from sklearn.model_selection import train_test_split\n",
+ "import pandas as pd\n",
+ "import tensorflow as tf\n",
+ "import tensorflow_hub as hub\n",
+ "from datetime import datetime"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "cp5wfXDx5SPH",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "In addition to the standard libraries we imported above, we'll need to install BERT's python package."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "jviywGyWyKsA",
+ "colab_type": "code",
+ "outputId": "166f3005-d219-404f-b201-2a0b75480360",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 51
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "!pip install bert-tensorflow"
+ ],
+ "execution_count": 38,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Requirement already satisfied: bert-tensorflow in /usr/local/lib/python3.6/dist-packages (1.0.1)\n",
+ "Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from bert-tensorflow) (1.11.0)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "hhbGEfwgdEtw",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "import bert\n",
+ "from bert import run_classifier\n",
+ "from bert import optimization\n",
+ "from bert import tokenization"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "KVB3eOcjxxm1",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Below, we'll set an output directory location to store our model output and checkpoints. This can be a local directory, in which case you'd set OUTPUT_DIR to the name of the directory you'd like to create. If you're running this code in Google's hosted Colab, the directory won't persist after the Colab session ends.\n",
+ "\n",
+ "Alternatively, if you're a GCP user, you can store output in a GCP bucket. To do that, set a directory name in OUTPUT_DIR and the name of the GCP bucket in the BUCKET field.\n",
+ "\n",
+ "Set DO_DELETE to rewrite the OUTPUT_DIR if it exists. Otherwise, Tensorflow will load existing model checkpoints from that directory (if they exist)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "US_EAnICvP7f",
+ "colab_type": "code",
+ "outputId": "7780a032-31d4-4794-e6aa-664a5d2ae7dd",
+ "cellView": "form",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# Set the output directory for saving model file\n",
+ "# Optionally, set a GCP bucket location\n",
+ "\n",
+ "OUTPUT_DIR = 'OUTPUT_DIR_NAME'#@param {type:\"string\"}\n",
+ "#@markdown Whether or not to clear/delete the directory and create a new one\n",
+ "DO_DELETE = False #@param {type:\"boolean\"}\n",
+ "#@markdown Set USE_BUCKET and BUCKET if you want to (optionally) store model output on GCP bucket.\n",
+ "USE_BUCKET = True #@param {type:\"boolean\"}\n",
+ "BUCKET = 'BUCKET_NAME' #@param {type:\"string\"}\n",
+ "\n",
+ "if USE_BUCKET:\n",
+ " OUTPUT_DIR = 'gs://{}/{}'.format(BUCKET, OUTPUT_DIR)\n",
+ " from google.colab import auth\n",
+ " auth.authenticate_user()\n",
+ "\n",
+ "if DO_DELETE:\n",
+ " try:\n",
+ " tf.gfile.DeleteRecursively(OUTPUT_DIR)\n",
+ " except:\n",
+ " # Doesn't matter if the directory didn't exist\n",
+ " pass\n",
+ "tf.gfile.MakeDirs(OUTPUT_DIR)\n",
+ "print('***** Model output directory: {} *****'.format(OUTPUT_DIR))\n"
+ ],
+ "execution_count": 40,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "***** Model output directory: gs://bert-tfhub/aclImdb_v1 *****\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "pmFYvkylMwXn",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "#Data"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "MC_w8SRqN0fr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "First, let's download the dataset, hosted by Stanford. The code below, which downloads, extracts, and imports the IMDB Large Movie Review Dataset, is borrowed from [this Tensorflow tutorial](https://www.tensorflow.org/hub/tutorials/text_classification_with_tf_hub)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "fom_ff20gyy6",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "from tensorflow import keras\n",
+ "import os\n",
+ "import re\n",
+ "\n",
+ "# Load all files from a directory in a DataFrame.\n",
+ "def load_directory_data(directory):\n",
+ " data = {}\n",
+ " data[\"sentence\"] = []\n",
+ " data[\"sentiment\"] = []\n",
+ " for file_path in os.listdir(directory):\n",
+ " with tf.gfile.GFile(os.path.join(directory, file_path), \"r\") as f:\n",
+ " data[\"sentence\"].append(f.read())\n",
+ " data[\"sentiment\"].append(re.match(\"\\d+_(\\d+)\\.txt\", file_path).group(1))\n",
+ " return pd.DataFrame.from_dict(data)\n",
+ "\n",
+ "# Merge positive and negative examples, add a polarity column and shuffle.\n",
+ "def load_dataset(directory):\n",
+ " pos_df = load_directory_data(os.path.join(directory, \"pos\"))\n",
+ " neg_df = load_directory_data(os.path.join(directory, \"neg\"))\n",
+ " pos_df[\"polarity\"] = 1\n",
+ " neg_df[\"polarity\"] = 0\n",
+ " return pd.concat([pos_df, neg_df]).sample(frac=1).reset_index(drop=True)\n",
+ "\n",
+ "# Download and process the dataset files.\n",
+ "def download_and_load_datasets(force_download=False):\n",
+ " dataset = tf.keras.utils.get_file(\n",
+ " fname=\"aclImdb.tar.gz\", \n",
+ " origin=\"http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\", \n",
+ " extract=True)\n",
+ " \n",
+ " train_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
+ " \"aclImdb\", \"train\"))\n",
+ " test_df = load_dataset(os.path.join(os.path.dirname(dataset), \n",
+ " \"aclImdb\", \"test\"))\n",
+ " \n",
+ " return train_df, test_df\n"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "2abfwdn-g135",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "train, test = download_and_load_datasets()"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "XA8WHJgzhIZf",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "To keep training fast, we'll take a sample of 5000 train and test examples, respectively."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "lw_F488eixTV",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "train = train.sample(5000)\n",
+ "test = test.sample(5000)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "prRQM8pDi8xI",
+ "colab_type": "code",
+ "outputId": "34445cb8-2be0-4379-fdbc-7794091f6049",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "train.columns"
+ ],
+ "execution_count": 44,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "Index(['sentence', 'sentiment', 'polarity'], dtype='object')"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 44
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "sfRnHSz3iSXz",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "For us, our input data is the 'sentence' column and our label is the 'polarity' column (0, 1 for negative and positive, respecitvely)"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "IuMOGwFui4it",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "DATA_COLUMN = 'sentence'\n",
+ "LABEL_COLUMN = 'polarity'\n",
+ "# label_list is the list of labels, i.e. True, False or 0, 1 or 'dog', 'cat'\n",
+ "label_list = [0, 1]"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "V399W0rqNJ-Z",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "#Data Preprocessing\n",
+ "We'll need to transform our data into a format BERT understands. This involves two steps. First, we create `InputExample`'s using the constructor provided in the BERT library.\n",
+ "\n",
+ "- `text_a` is the text we want to classify, which in this case, is the `Request` field in our Dataframe. \n",
+ "- `text_b` is used if we're training a model to understand the relationship between sentences (i.e. is `text_b` a translation of `text_a`? Is `text_b` an answer to the question asked by `text_a`?). This doesn't apply to our task, so we can leave `text_b` blank.\n",
+ "- `label` is the label for our example, i.e. True, False"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "p9gEt5SmM6i6",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Use the InputExample class from BERT's run_classifier code to create examples from the data\n",
+ "train_InputExamples = train.apply(lambda x: bert.run_classifier.InputExample(guid=None, # Globally unique ID for bookkeeping, unused in this example\n",
+ " text_a = x[DATA_COLUMN], \n",
+ " text_b = None, \n",
+ " label = x[LABEL_COLUMN]), axis = 1)\n",
+ "\n",
+ "test_InputExamples = test.apply(lambda x: bert.run_classifier.InputExample(guid=None, \n",
+ " text_a = x[DATA_COLUMN], \n",
+ " text_b = None, \n",
+ " label = x[LABEL_COLUMN]), axis = 1)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "SCZWZtKxObjh",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Next, we need to preprocess our data so that it matches the data BERT was trained on. For this, we'll need to do a couple of things (but don't worry--this is also included in the Python library):\n",
+ "\n",
+ "\n",
+ "1. Lowercase our text (if we're using a BERT lowercase model)\n",
+ "2. Tokenize it (i.e. \"sally says hi\" -> [\"sally\", \"says\", \"hi\"])\n",
+ "3. Break words into WordPieces (i.e. \"calling\" -> [\"call\", \"##ing\"])\n",
+ "4. Map our words to indexes using a vocab file that BERT provides\n",
+ "5. Add special \"CLS\" and \"SEP\" tokens (see the [readme](https://github.com/google-research/bert))\n",
+ "6. Append \"index\" and \"segment\" tokens to each input (see the [BERT paper](https://arxiv.org/pdf/1810.04805.pdf))\n",
+ "\n",
+ "Happily, we don't have to worry about most of these details.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "qMWiDtpyQSoU",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "To start, we'll need to load a vocabulary file and lowercasing information directly from the BERT tf hub module:"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "IhJSe0QHNG7U",
+ "colab_type": "code",
+ "outputId": "20b28cc7-3cb3-4ce6-bfff-a7847ce3bbaa",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 34
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# This is a path to an uncased (all lowercase) version of BERT\n",
+ "BERT_MODEL_HUB = \"https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/1\"\n",
+ "\n",
+ "def create_tokenizer_from_hub_module():\n",
+ " \"\"\"Get the vocab file and casing info from the Hub module.\"\"\"\n",
+ " with tf.Graph().as_default():\n",
+ " bert_module = hub.Module(BERT_MODEL_HUB)\n",
+ " tokenization_info = bert_module(signature=\"tokenization_info\", as_dict=True)\n",
+ " with tf.Session() as sess:\n",
+ " vocab_file, do_lower_case = sess.run([tokenization_info[\"vocab_file\"],\n",
+ " tokenization_info[\"do_lower_case\"]])\n",
+ " \n",
+ " return bert.tokenization.FullTokenizer(\n",
+ " vocab_file=vocab_file, do_lower_case=do_lower_case)\n",
+ "\n",
+ "tokenizer = create_tokenizer_from_hub_module()"
+ ],
+ "execution_count": 47,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "z4oFkhpZBDKm",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Great--we just learned that the BERT model we're using expects lowercase data (that's what stored in tokenization_info[\"do_lower_case\"]) and we also loaded BERT's vocab file. We also created a tokenizer, which breaks words into word pieces:"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "dsBo6RCtQmwx",
+ "colab_type": "code",
+ "outputId": "9af8c917-90ec-4fe9-897b-79dc89ca88e1",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 221
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "tokenizer.tokenize(\"This here's an example of using the BERT tokenizer\")"
+ ],
+ "execution_count": 48,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "['this',\n",
+ " 'here',\n",
+ " \"'\",\n",
+ " 's',\n",
+ " 'an',\n",
+ " 'example',\n",
+ " 'of',\n",
+ " 'using',\n",
+ " 'the',\n",
+ " 'bert',\n",
+ " 'token',\n",
+ " '##izer']"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 48
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "0OEzfFIt6GIc",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Using our tokenizer, we'll call `run_classifier.convert_examples_to_features` on our InputExamples to convert them into features BERT understands."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "LL5W8gEGRTAf",
+ "colab_type": "code",
+ "outputId": "65001dda-155b-48fc-b5fc-1e4cabc8dfbf",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 1261
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "# We'll set sequences to be at most 128 tokens long.\n",
+ "MAX_SEQ_LENGTH = 128\n",
+ "# Convert our train and test features to InputFeatures that BERT understands.\n",
+ "train_features = bert.run_classifier.convert_examples_to_features(train_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
+ "test_features = bert.run_classifier.convert_examples_to_features(test_InputExamples, label_list, MAX_SEQ_LENGTH, tokenizer)"
+ ],
+ "execution_count": 49,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Writing example 0 of 5000\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] i ' m watching this on the sci - fi channel right now . it ' s so horrible i can ' t stop watching it ! i ' m a video ##grapher and this movie makes me sad . i feel bad for anyone associated with this movie . some of the camera work is good . most is very questionable . there are a few decent actors in the flick . too bad they ' re surrounded by what must have been the director ' s relatives . that ' s the only way they could have been qualified to be in a movie ! music was a little better than the acting . if you get around to watching this i hope it [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1045 1005 1049 3666 2023 2006 1996 16596 1011 10882 3149 2157 2085 1012 2009 1005 1055 2061 9202 1045 2064 1005 1056 2644 3666 2009 999 1045 1005 1049 1037 2678 18657 1998 2023 3185 3084 2033 6517 1012 1045 2514 2919 2005 3087 3378 2007 2023 3185 1012 2070 1997 1996 4950 2147 2003 2204 1012 2087 2003 2200 21068 1012 2045 2024 1037 2261 11519 5889 1999 1996 17312 1012 2205 2919 2027 1005 2128 5129 2011 2054 2442 2031 2042 1996 2472 1005 1055 9064 1012 2008 1005 1055 1996 2069 2126 2027 2071 2031 2042 4591 2000 2022 1999 1037 3185 999 2189 2001 1037 2210 2488 2084 1996 3772 1012 2065 2017 2131 2105 2000 3666 2023 1045 3246 2009 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] i have been a fan of pushing dai ##sies since the very beginning . it is wonderful ##ly thought up , and bryan fuller has the most remarkable ideas for this show . < br / > < br / > it is unbelievable on how much tv has been needing a creative , original show like pushing dai ##sies . it is a huge relief to see a show , that is unlike the rest , where as , if you compared it to some of the newer shows , such as scrub ##s and house , you would see the similarities , and it does get ted ##ious at moments to see shows so close in identity . < br / > < br [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1045 2031 2042 1037 5470 1997 6183 18765 14625 2144 1996 2200 2927 1012 2009 2003 6919 2135 2245 2039 1010 1998 8527 12548 2038 1996 2087 9487 4784 2005 2023 2265 1012 1026 7987 1013 1028 1026 7987 1013 1028 2009 2003 23653 2006 2129 2172 2694 2038 2042 11303 1037 5541 1010 2434 2265 2066 6183 18765 14625 1012 2009 2003 1037 4121 4335 2000 2156 1037 2265 1010 2008 2003 4406 1996 2717 1010 2073 2004 1010 2065 2017 4102 2009 2000 2070 1997 1996 10947 3065 1010 2107 2004 18157 2015 1998 2160 1010 2017 2052 2156 1996 12319 1010 1998 2009 2515 2131 6945 6313 2012 5312 2000 2156 3065 2061 2485 1999 4767 1012 1026 7987 1013 1028 1026 7987 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 1 (id = 1)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] this movie starts out promising ##ly , with an early scene in which frank morgan advises against gary cooper ' s marriage to his daughter , anita louise . frank morgan , playing an una ##bas ##hed gold - digger , loudly complain ##s to cooper about his perceived pen ##ury at the hands of his family - including his daughter , anita louise . i am a fan of all 3 actors . frank morgan is ( to my mind ) a hollywood treasure , cooper a legend , and louise a very lovely , versatile and under - appreciated actress seldom seen in the leading role . i also have nothing against teresa wright , and while not blessed with great range , she [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2023 3185 4627 2041 10015 2135 1010 2007 2019 2220 3496 1999 2029 3581 5253 25453 2114 5639 6201 1005 1055 3510 2000 2010 2684 1010 12918 8227 1012 3581 5253 1010 2652 2019 14477 22083 9072 2751 1011 28661 1010 9928 17612 2015 2000 6201 2055 2010 8690 7279 13098 2012 1996 2398 1997 2010 2155 1011 2164 2010 2684 1010 12918 8227 1012 1045 2572 1037 5470 1997 2035 1017 5889 1012 3581 5253 2003 1006 2000 2026 2568 1007 1037 5365 8813 1010 6201 1037 5722 1010 1998 8227 1037 2200 8403 1010 22979 1998 2104 1011 12315 3883 15839 2464 1999 1996 2877 2535 1012 1045 2036 2031 2498 2114 12409 6119 1010 1998 2096 2025 10190 2007 2307 2846 1010 2016 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] i was over ##taken by the emotion . un ##for ##get ##table rendering of a wartime story which is unknown to most people . the performances were fault ##less and outstanding . [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1045 2001 2058 25310 2011 1996 7603 1012 4895 29278 18150 10880 14259 1997 1037 12498 2466 2029 2003 4242 2000 2087 2111 1012 1996 4616 2020 6346 3238 1998 5151 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 1 (id = 1)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] soldier blue is a movie with pre ##tension ##s : pre ##tension ##s to be some sort of profound statement on man ' s inhuman ##ity to man , on the white man ' s exploitation of and brutality towards indigenous peoples ; a biting , un ##fl ##in ##ching and sar ##don ##ic commentary on the horrors of vietnam . well , sorry , but it fails mis ##era ##bly to be any of those things . what soldier blue actually is is per ##nic ##ious , tri ##te , badly made , dish ##ones ##t rubbish . < br / > < br / > another reviewer here hit the nail on the head in saying that it appears to be a hybrid of [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 5268 2630 2003 1037 3185 2007 3653 29048 2015 1024 3653 29048 2015 2000 2022 2070 4066 1997 13769 4861 2006 2158 1005 1055 29582 3012 2000 2158 1010 2006 1996 2317 2158 1005 1055 14427 1997 1998 24083 2875 6284 7243 1025 1037 12344 1010 4895 10258 2378 8450 1998 18906 5280 2594 8570 2006 1996 22812 1997 5148 1012 2092 1010 3374 1010 2021 2009 11896 28616 6906 6321 2000 2022 2151 1997 2216 2477 1012 2054 5268 2630 2941 2003 2003 2566 8713 6313 1010 13012 2618 1010 6649 2081 1010 9841 21821 2102 29132 1012 1026 7987 1013 1028 1026 7987 1013 1028 2178 12027 2182 2718 1996 13774 2006 1996 2132 1999 3038 2008 2009 3544 2000 2022 1037 8893 1997 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:Writing example 0 of 5000\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] i just watched this today on tv . it was on abc ' s sunday afternoon movie . < br / > < br / > this wasn ' t a very good movie , but for a low budget independent film like this , it was okay . there is some suspense in it , but there are so many bad qualities that really bring the movie down . the script is pretty lame , and the plot elements aren ' t very realistic , such as the way a 911 operator would laugh and hang up when someone is reporting a murder . i don ' t know what the writer was thinking when they came up with that idea , but it isn [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1045 2074 3427 2023 2651 2006 2694 1012 2009 2001 2006 5925 1005 1055 4465 5027 3185 1012 1026 7987 1013 1028 1026 7987 1013 1028 2023 2347 1005 1056 1037 2200 2204 3185 1010 2021 2005 1037 2659 5166 2981 2143 2066 2023 1010 2009 2001 3100 1012 2045 2003 2070 23873 1999 2009 1010 2021 2045 2024 2061 2116 2919 11647 2008 2428 3288 1996 3185 2091 1012 1996 5896 2003 3492 20342 1010 1998 1996 5436 3787 4995 1005 1056 2200 12689 1010 2107 2004 1996 2126 1037 19989 6872 2052 4756 1998 6865 2039 2043 2619 2003 7316 1037 4028 1012 1045 2123 1005 1056 2113 2054 1996 3213 2001 3241 2043 2027 2234 2039 2007 2008 2801 1010 2021 2009 3475 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] from hardly alien sounding lasers , to an elementary school style shuttle crash , \" night ##be ##ast \" is better classified as a far ##cic ##al mix of fake blood and bare chest . the almost pornographic style of the film seems to be a failed attempt to recover from a lack of co ##hesive or effective story . the acting however is not nearly as beast ##ly , many of the young , aspiring , actors ad ##mir ##ably showcase a hidden talent . particularly don lei ##fer ##t and jamie ze ##mare ##l , who shed a well needed sha ##rd of light on this otherwise terrible film . night ##be ##ast would have never shown up on set had he known the [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2013 6684 7344 9391 23965 1010 2000 2019 4732 2082 2806 10382 5823 1010 1000 2305 4783 14083 1000 2003 2488 6219 2004 1037 2521 19053 2389 4666 1997 8275 2668 1998 6436 3108 1012 1996 2471 26932 2806 1997 1996 2143 3849 2000 2022 1037 3478 3535 2000 8980 2013 1037 3768 1997 2522 21579 2030 4621 2466 1012 1996 3772 2174 2003 2025 3053 2004 6841 2135 1010 2116 1997 1996 2402 1010 22344 1010 5889 4748 14503 8231 13398 1037 5023 5848 1012 3391 2123 26947 7512 2102 1998 6175 27838 24376 2140 1010 2040 8328 1037 2092 2734 21146 4103 1997 2422 2006 2023 4728 6659 2143 1012 2305 4783 14083 2052 2031 2196 3491 2039 2006 2275 2018 2002 2124 1996 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] here we have the in ##imi ##table charlie chaplin for ##sa ##king his slap ##stick past to tackle the serious subject of anti - semi ##tism , and into ##ler ##ance in general . he portrays two characters - the sweet , innocent jewish barber - a war veteran , and the ravi ##ng and ruthless dictator , aden ##oid h ##yn ##kel . the jewish ghetto in this country is not safe for long , due to the w ##him ##s of h ##yn ##kel and his armed thugs , who routinely rough up its residents , or leave them alone , dependent upon his mood that day or week . the barber is among them , but is befriended by his former commanding officer [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2182 2057 2031 1996 1999 27605 10880 4918 23331 2005 3736 6834 2010 14308 21354 2627 2000 11147 1996 3809 3395 1997 3424 1011 4100 17456 1010 1998 2046 3917 6651 1999 2236 1012 2002 17509 2048 3494 1011 1996 4086 1010 7036 3644 13362 1011 1037 2162 8003 1010 1998 1996 16806 3070 1998 18101 21237 1010 16298 9314 1044 6038 11705 1012 1996 3644 17276 1999 2023 2406 2003 2025 3647 2005 2146 1010 2349 2000 1996 1059 14341 2015 1997 1044 6038 11705 1998 2010 4273 24106 1010 2040 19974 5931 2039 2049 3901 1010 2030 2681 2068 2894 1010 7790 2588 2010 6888 2008 2154 2030 2733 1012 1996 13362 2003 2426 2068 1010 2021 2003 23386 2011 2010 2280 7991 2961 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 1 (id = 1)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] i really hated this movie and it ' s the first movie written by stephen king that i didn ' t finish . i was truly disappointed , it was the worst crap i ' ve ever seen . what were you thinking making three hours out of it ? it may have a quite good story , but actors ? no . suspense ? no . romance ? no . horror ? no . it didn ' t have anything . < br / > < br / > it ' s got this strange , crazy science man with einstein - hair , the classic thing . not real at all . and a man keep getting younger all the time . it seems [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1045 2428 6283 2023 3185 1998 2009 1005 1055 1996 2034 3185 2517 2011 4459 2332 2008 1045 2134 1005 1056 3926 1012 1045 2001 5621 9364 1010 2009 2001 1996 5409 10231 1045 1005 2310 2412 2464 1012 2054 2020 2017 3241 2437 2093 2847 2041 1997 2009 1029 2009 2089 2031 1037 3243 2204 2466 1010 2021 5889 1029 2053 1012 23873 1029 2053 1012 7472 1029 2053 1012 5469 1029 2053 1012 2009 2134 1005 1056 2031 2505 1012 1026 7987 1013 1028 1026 7987 1013 1028 2009 1005 1055 2288 2023 4326 1010 4689 2671 2158 2007 15313 1011 2606 1010 1996 4438 2518 1012 2025 2613 2012 2035 1012 1998 1037 2158 2562 2893 3920 2035 1996 2051 1012 2009 3849 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: None\n",
+ "INFO:tensorflow:tokens: [CLS] story chinese tall story tells the story of righteous monk trip ##ita ##ka , who , along with his guardians monkey , sandy and pigs ##y make their journey west on a quest to recover ancient sutra ##s , finally , they reach the final leg of their journey in sha ##che city but all is not as it seems when the city is attacked by evil tree demons . monkey tries his best to battle them but is overwhelmed , knowing his master is in grave danger , he uses his trust ##y golden staff to thrust trip ##ita ##ka to safety . < br / > < br / > the monk ends up being knocked out when he land and when he wakes [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2466 2822 4206 2466 4136 1996 2466 1997 19556 8284 4440 6590 2912 1010 2040 1010 2247 2007 2010 14240 10608 1010 7525 1998 14695 2100 2191 2037 4990 2225 2006 1037 8795 2000 8980 3418 26567 2015 1010 2633 1010 2027 3362 1996 2345 4190 1997 2037 4990 1999 21146 5403 2103 2021 2035 2003 2025 2004 2009 3849 2043 1996 2103 2003 4457 2011 4763 3392 7942 1012 10608 5363 2010 2190 2000 2645 2068 2021 2003 13394 1010 4209 2010 3040 2003 1999 6542 5473 1010 2002 3594 2010 3404 2100 3585 3095 2000 7400 4440 6590 2912 2000 3808 1012 1026 7987 1013 1028 1026 7987 1013 1028 1996 8284 4515 2039 2108 6573 2041 2043 2002 2455 1998 2043 2002 17507 102\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 1 (id = 1)\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ccp5trMwRtmr",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "#Creating a model\n",
+ "\n",
+ "Now that we've prepared our data, let's focus on building a model. `create_model` does just this below. First, it loads the BERT tf hub module again (this time to extract the computation graph). Next, it creates a single new layer that will be trained to adapt BERT to our sentiment task (i.e. classifying whether a movie review is positive or negative). This strategy of using a mostly trained model is called [fine-tuning](http://wiki.fast.ai/index.php/Fine_tuning)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "6o2a5ZIvRcJq",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def create_model(is_predicting, input_ids, input_mask, segment_ids, labels,\n",
+ " num_labels):\n",
+ " \"\"\"Creates a classification model.\"\"\"\n",
+ "\n",
+ " bert_module = hub.Module(\n",
+ " BERT_MODEL_HUB,\n",
+ " trainable=True)\n",
+ " bert_inputs = dict(\n",
+ " input_ids=input_ids,\n",
+ " input_mask=input_mask,\n",
+ " segment_ids=segment_ids)\n",
+ " bert_outputs = bert_module(\n",
+ " inputs=bert_inputs,\n",
+ " signature=\"tokens\",\n",
+ " as_dict=True)\n",
+ "\n",
+ " # Use \"pooled_output\" for classification tasks on an entire sentence.\n",
+ " # Use \"sequence_outputs\" for token-level output.\n",
+ " output_layer = bert_outputs[\"pooled_output\"]\n",
+ "\n",
+ " hidden_size = output_layer.shape[-1].value\n",
+ "\n",
+ " # Create our own layer to tune for politeness data.\n",
+ " output_weights = tf.get_variable(\n",
+ " \"output_weights\", [num_labels, hidden_size],\n",
+ " initializer=tf.truncated_normal_initializer(stddev=0.02))\n",
+ "\n",
+ " output_bias = tf.get_variable(\n",
+ " \"output_bias\", [num_labels], initializer=tf.zeros_initializer())\n",
+ "\n",
+ " with tf.variable_scope(\"loss\"):\n",
+ "\n",
+ " # Dropout helps prevent overfitting\n",
+ " output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)\n",
+ "\n",
+ " logits = tf.matmul(output_layer, output_weights, transpose_b=True)\n",
+ " logits = tf.nn.bias_add(logits, output_bias)\n",
+ " log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
+ "\n",
+ " # Convert labels into one-hot encoding\n",
+ " one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)\n",
+ "\n",
+ " predicted_labels = tf.squeeze(tf.argmax(log_probs, axis=-1, output_type=tf.int32))\n",
+ " # If we're predicting, we want predicted labels and the probabiltiies.\n",
+ " if is_predicting:\n",
+ " return (predicted_labels, log_probs)\n",
+ "\n",
+ " # If we're train/eval, compute loss between predicted and actual label\n",
+ " per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)\n",
+ " loss = tf.reduce_mean(per_example_loss)\n",
+ " return (loss, predicted_labels, log_probs)\n"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "qpE0ZIDOCQzE",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Next we'll wrap our model function in a `model_fn_builder` function that adapts our model to work for training, evaluation, and prediction."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "FnH-AnOQ9KKW",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# model_fn_builder actually creates our model function\n",
+ "# using the passed parameters for num_labels, learning_rate, etc.\n",
+ "def model_fn_builder(num_labels, learning_rate, num_train_steps,\n",
+ " num_warmup_steps):\n",
+ " \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
+ " def model_fn(features, labels, mode, params): # pylint: disable=unused-argument\n",
+ " \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
+ "\n",
+ " input_ids = features[\"input_ids\"]\n",
+ " input_mask = features[\"input_mask\"]\n",
+ " segment_ids = features[\"segment_ids\"]\n",
+ " label_ids = features[\"label_ids\"]\n",
+ "\n",
+ " is_predicting = (mode == tf.estimator.ModeKeys.PREDICT)\n",
+ " \n",
+ " # TRAIN and EVAL\n",
+ " if not is_predicting:\n",
+ "\n",
+ " (loss, predicted_labels, log_probs) = create_model(\n",
+ " is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
+ "\n",
+ " train_op = bert.optimization.create_optimizer(\n",
+ " loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu=False)\n",
+ "\n",
+ " # Calculate evaluation metrics. \n",
+ " def metric_fn(label_ids, predicted_labels):\n",
+ " accuracy = tf.metrics.accuracy(label_ids, predicted_labels)\n",
+ " f1_score = tf.contrib.metrics.f1_score(\n",
+ " label_ids,\n",
+ " predicted_labels)\n",
+ " auc = tf.metrics.auc(\n",
+ " label_ids,\n",
+ " predicted_labels)\n",
+ " recall = tf.metrics.recall(\n",
+ " label_ids,\n",
+ " predicted_labels)\n",
+ " precision = tf.metrics.precision(\n",
+ " label_ids,\n",
+ " predicted_labels) \n",
+ " true_pos = tf.metrics.true_positives(\n",
+ " label_ids,\n",
+ " predicted_labels)\n",
+ " true_neg = tf.metrics.true_negatives(\n",
+ " label_ids,\n",
+ " predicted_labels) \n",
+ " false_pos = tf.metrics.false_positives(\n",
+ " label_ids,\n",
+ " predicted_labels) \n",
+ " false_neg = tf.metrics.false_negatives(\n",
+ " label_ids,\n",
+ " predicted_labels)\n",
+ " return {\n",
+ " \"eval_accuracy\": accuracy,\n",
+ " \"f1_score\": f1_score,\n",
+ " \"auc\": auc,\n",
+ " \"precision\": precision,\n",
+ " \"recall\": recall,\n",
+ " \"true_positives\": true_pos,\n",
+ " \"true_negatives\": true_neg,\n",
+ " \"false_positives\": false_pos,\n",
+ " \"false_negatives\": false_neg\n",
+ " }\n",
+ "\n",
+ " eval_metrics = metric_fn(label_ids, predicted_labels)\n",
+ "\n",
+ " if mode == tf.estimator.ModeKeys.TRAIN:\n",
+ " return tf.estimator.EstimatorSpec(mode=mode,\n",
+ " loss=loss,\n",
+ " train_op=train_op)\n",
+ " else:\n",
+ " return tf.estimator.EstimatorSpec(mode=mode,\n",
+ " loss=loss,\n",
+ " eval_metric_ops=eval_metrics)\n",
+ " else:\n",
+ " (predicted_labels, log_probs) = create_model(\n",
+ " is_predicting, input_ids, input_mask, segment_ids, label_ids, num_labels)\n",
+ "\n",
+ " predictions = {\n",
+ " 'probabilities': log_probs,\n",
+ " 'labels': predicted_labels\n",
+ " }\n",
+ " return tf.estimator.EstimatorSpec(mode, predictions=predictions)\n",
+ "\n",
+ " # Return the actual model function in the closure\n",
+ " return model_fn\n"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "OjwJ4bTeWXD8",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Compute train and warmup steps from batch size\n",
+ "# These hyperparameters are copied from this colab notebook (https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/tools/colab/bert_finetuning_with_cloud_tpus.ipynb)\n",
+ "BATCH_SIZE = 32\n",
+ "LEARNING_RATE = 2e-5\n",
+ "NUM_TRAIN_EPOCHS = 3.0\n",
+ "# Warmup is a period of time where hte learning rate \n",
+ "# is small and gradually increases--usually helps training.\n",
+ "WARMUP_PROPORTION = 0.1\n",
+ "# Model configs\n",
+ "SAVE_CHECKPOINTS_STEPS = 500\n",
+ "SAVE_SUMMARY_STEPS = 100"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "emHf9GhfWBZ_",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Compute # train and warmup steps from batch size\n",
+ "num_train_steps = int(len(train_features) / BATCH_SIZE * NUM_TRAIN_EPOCHS)\n",
+ "num_warmup_steps = int(num_train_steps * WARMUP_PROPORTION)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "oEJldMr3WYZa",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Specify outpit directory and number of checkpoint steps to save\n",
+ "run_config = tf.estimator.RunConfig(\n",
+ " model_dir=OUTPUT_DIR,\n",
+ " save_summary_steps=SAVE_SUMMARY_STEPS,\n",
+ " save_checkpoints_steps=SAVE_CHECKPOINTS_STEPS)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "q_WebpS1X97v",
+ "colab_type": "code",
+ "outputId": "1648932a-7391-49d3-8af7-52d514e226e8",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 156
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "model_fn = model_fn_builder(\n",
+ " num_labels=len(label_list),\n",
+ " learning_rate=LEARNING_RATE,\n",
+ " num_train_steps=num_train_steps,\n",
+ " num_warmup_steps=num_warmup_steps)\n",
+ "\n",
+ "estimator = tf.estimator.Estimator(\n",
+ " model_fn=model_fn,\n",
+ " config=run_config,\n",
+ " params={\"batch_size\": BATCH_SIZE})\n"
+ ],
+ "execution_count": 55,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Using config: {'_model_dir': 'gs://bert-tfhub/aclImdb_v1', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 500, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
+ "graph_options {\n",
+ " rewrite_options {\n",
+ " meta_optimizer_iterations: ONE\n",
+ " }\n",
+ "}\n",
+ ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "NOO3RfG1DYLo",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Next we create an input builder function that takes our training feature set (`train_features`) and produces a generator. This is a pretty standard design pattern for working with Tensorflow [Estimators](https://www.tensorflow.org/guide/estimators)."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "1Pv2bAlOX_-K",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "# Create an input function for training. drop_remainder = True for using TPUs.\n",
+ "train_input_fn = bert.run_classifier.input_fn_builder(\n",
+ " features=train_features,\n",
+ " seq_length=MAX_SEQ_LENGTH,\n",
+ " is_training=True,\n",
+ " drop_remainder=False)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "t6Nukby2EB6-",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Now we train our model! For me, using a Colab notebook running on Google's GPUs, my training time was about 14 minutes."
+ ]
+ },
+ {
+ "metadata": {
+ "id": "nucD4gluYJmK",
+ "colab_type": "code",
+ "outputId": "5d728e72-4631-42bf-c48d-3f51d4b968ce",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 68
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "print(f'Beginning Training!')\n",
+ "current_time = datetime.now()\n",
+ "estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)\n",
+ "print(\"Training took time \", datetime.now() - current_time)"
+ ],
+ "execution_count": 57,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "Beginning Training!\n",
+ "INFO:tensorflow:Skipping training since max_steps has already saved.\n",
+ "Training took time 0:00:00.759709\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "CmbLTVniARy3",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Now let's use our test data to see how well our model did:"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "JIhejfpyJ8Bx",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "test_input_fn = run_classifier.input_fn_builder(\n",
+ " features=test_features,\n",
+ " seq_length=MAX_SEQ_LENGTH,\n",
+ " is_training=False,\n",
+ " drop_remainder=False)"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "PPVEXhNjYXC-",
+ "colab_type": "code",
+ "outputId": "dd5482cd-c558-465f-c854-ec11a0175316",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 445
+ }
+ },
+ "cell_type": "code",
+ "source": [
+ "estimator.evaluate(input_fn=test_input_fn, steps=None)"
+ ],
+ "execution_count": 59,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Calling model_fn.\n",
+ "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n"
+ ],
+ "name": "stdout"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
+ " \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
+ ],
+ "name": "stderr"
+ },
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Done calling model_fn.\n",
+ "INFO:tensorflow:Starting evaluation at 2019-02-12T21:04:20Z\n",
+ "INFO:tensorflow:Graph was finalized.\n",
+ "INFO:tensorflow:Restoring parameters from gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n",
+ "INFO:tensorflow:Running local_init_op.\n",
+ "INFO:tensorflow:Done running local_init_op.\n",
+ "INFO:tensorflow:Finished evaluation at 2019-02-12-21:06:05\n",
+ "INFO:tensorflow:Saving dict for global step 468: auc = 0.86659324, eval_accuracy = 0.8664, f1_score = 0.8659711, false_negatives = 375.0, false_positives = 293.0, global_step = 468, loss = 0.51870537, precision = 0.880457, recall = 0.8519542, true_negatives = 2174.0, true_positives = 2158.0\n",
+ "INFO:tensorflow:Saving 'checkpoint_path' summary for global step 468: gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n"
+ ],
+ "name": "stdout"
+ },
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "{'auc': 0.86659324,\n",
+ " 'eval_accuracy': 0.8664,\n",
+ " 'f1_score': 0.8659711,\n",
+ " 'false_negatives': 375.0,\n",
+ " 'false_positives': 293.0,\n",
+ " 'global_step': 468,\n",
+ " 'loss': 0.51870537,\n",
+ " 'precision': 0.880457,\n",
+ " 'recall': 0.8519542,\n",
+ " 'true_negatives': 2174.0,\n",
+ " 'true_positives': 2158.0}"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 59
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ueKsULteiz1B",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Now let's write code to make predictions on new sentences:"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "OsrbTD2EJTVl",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "def getPrediction(in_sentences):\n",
+ " labels = [\"Negative\", \"Positive\"]\n",
+ " input_examples = [run_classifier.InputExample(guid=\"\", text_a = x, text_b = None, label = 0) for x in in_sentences] # here, \"\" is just a dummy label\n",
+ " input_features = run_classifier.convert_examples_to_features(input_examples, label_list, MAX_SEQ_LENGTH, tokenizer)\n",
+ " predict_input_fn = run_classifier.input_fn_builder(features=input_features, seq_length=MAX_SEQ_LENGTH, is_training=False, drop_remainder=False)\n",
+ " predictions = estimator.predict(predict_input_fn)\n",
+ " return [(sentence, prediction['probabilities'], labels[prediction['labels']]) for sentence, prediction in zip(in_sentences, predictions)]"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "-thbodgih_VJ",
+ "colab_type": "code",
+ "colab": {}
+ },
+ "cell_type": "code",
+ "source": [
+ "pred_sentences = [\n",
+ " \"That movie was absolutely awful\",\n",
+ " \"The acting was a bit lacking\",\n",
+ " \"The film was creative and surprising\",\n",
+ " \"Absolutely fantastic!\"\n",
+ "]"
+ ],
+ "execution_count": 0,
+ "outputs": []
+ },
+ {
+ "metadata": {
+ "id": "QrZmvZySKQTm",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 649
+ },
+ "outputId": "3891fafb-a460-4eb8-fa6c-335a5bbc10e5"
+ },
+ "cell_type": "code",
+ "source": [
+ "predictions = getPrediction(pred_sentences)"
+ ],
+ "execution_count": 72,
+ "outputs": [
+ {
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Writing example 0 of 4\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: \n",
+ "INFO:tensorflow:tokens: [CLS] that movie was absolutely awful [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2008 3185 2001 7078 9643 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: \n",
+ "INFO:tensorflow:tokens: [CLS] the acting was a bit lacking [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1996 3772 2001 1037 2978 11158 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: \n",
+ "INFO:tensorflow:tokens: [CLS] the film was creative and surprising [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 1996 2143 2001 5541 1998 11341 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:guid: \n",
+ "INFO:tensorflow:tokens: [CLS] absolutely fantastic ! [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 7078 10392 999 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:label: 0 (id = 0)\n",
+ "INFO:tensorflow:Calling model_fn.\n",
+ "INFO:tensorflow:Saver not created because there are no variables in the graph to restore\n",
+ "INFO:tensorflow:Done calling model_fn.\n",
+ "INFO:tensorflow:Graph was finalized.\n",
+ "INFO:tensorflow:Restoring parameters from gs://bert-tfhub/aclImdb_v1/model.ckpt-468\n",
+ "INFO:tensorflow:Running local_init_op.\n",
+ "INFO:tensorflow:Done running local_init_op.\n"
+ ],
+ "name": "stdout"
+ }
+ ]
+ },
+ {
+ "metadata": {
+ "id": "MXkRiEBUqN3n",
+ "colab_type": "text"
+ },
+ "cell_type": "markdown",
+ "source": [
+ "Voila! We have a sentiment classifier!"
+ ]
+ },
+ {
+ "metadata": {
+ "id": "ERkTE8-7oQLZ",
+ "colab_type": "code",
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 221
+ },
+ "outputId": "26c33224-dc2c-4b3d-f7b4-ac3ef0a58b27"
+ },
+ "cell_type": "code",
+ "source": [
+ "predictions"
+ ],
+ "execution_count": 73,
+ "outputs": [
+ {
+ "output_type": "execute_result",
+ "data": {
+ "text/plain": [
+ "[('That movie was absolutely awful',\n",
+ " array([-4.9142293e-03, -5.3180690e+00], dtype=float32),\n",
+ " 'Negative'),\n",
+ " ('The acting was a bit lacking',\n",
+ " array([-0.03325794, -3.4200459 ], dtype=float32),\n",
+ " 'Negative'),\n",
+ " ('The film was creative and surprising',\n",
+ " array([-5.3589125e+00, -4.7171740e-03], dtype=float32),\n",
+ " 'Positive'),\n",
+ " ('Absolutely fantastic!',\n",
+ " array([-5.0434084 , -0.00647258], dtype=float32),\n",
+ " 'Positive')]"
+ ]
+ },
+ "metadata": {
+ "tags": []
+ },
+ "execution_count": 73
+ }
+ ]
+ }
+ ]
+}
\ No newline at end of file
diff --git a/bert-master/bert-master/requirements.txt b/bert-master/bert-master/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..357b5ea077b787d61c15d735d0a1b73971f22602
--- /dev/null
+++ b/bert-master/bert-master/requirements.txt
@@ -0,0 +1,2 @@
+tensorflow >= 1.11.0 # CPU Version of TensorFlow.
+# tensorflow-gpu >= 1.11.0 # GPU version of TensorFlow.
diff --git a/bert-master/bert-master/run_classifier.py b/bert-master/bert-master/run_classifier.py
new file mode 100644
index 0000000000000000000000000000000000000000..817b14720c235a3199b8419c3543bf312f544a5c
--- /dev/null
+++ b/bert-master/bert-master/run_classifier.py
@@ -0,0 +1,981 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import csv
+import os
+import modeling
+import optimization
+import tokenization
+import tensorflow as tf
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+ "data_dir", None,
+ "The input data dir. Should contain the .tsv files (or other data files) "
+ "for the task.")
+
+flags.DEFINE_string(
+ "bert_config_file", None,
+ "The config json file corresponding to the pre-trained BERT model. "
+ "This specifies the model architecture.")
+
+flags.DEFINE_string("task_name", None, "The name of the task to train.")
+
+flags.DEFINE_string("vocab_file", None,
+ "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_string(
+ "output_dir", None,
+ "The output directory where the model checkpoints will be written.")
+
+## Other parameters
+
+flags.DEFINE_string(
+ "init_checkpoint", None,
+ "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_bool(
+ "do_lower_case", True,
+ "Whether to lower case the input text. Should be True for uncased "
+ "models and False for cased models.")
+
+flags.DEFINE_integer(
+ "max_seq_length", 128,
+ "The maximum total input sequence length after WordPiece tokenization. "
+ "Sequences longer than this will be truncated, and sequences shorter "
+ "than this will be padded.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_eval", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_bool(
+ "do_predict", False,
+ "Whether to run the model in inference mode on the test set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("eval_batch_size", 8, "Total batch size for eval.")
+
+flags.DEFINE_integer("predict_batch_size", 8, "Total batch size for predict.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_float("num_train_epochs", 3.0,
+ "Total number of training epochs to perform.")
+
+flags.DEFINE_float(
+ "warmup_proportion", 0.1,
+ "Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10% of training.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+ "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+ "How many steps to make in each estimator call.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+tf.flags.DEFINE_string(
+ "tpu_name", None,
+ "The Cloud TPU to use for training. This should be either the name "
+ "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+ "url.")
+
+tf.flags.DEFINE_string(
+ "tpu_zone", None,
+ "[Optional] GCE zone where the Cloud TPU is located in. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string(
+ "gcp_project", None,
+ "[Optional] Project name for the Cloud TPU-enabled project. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+
+flags.DEFINE_integer(
+ "num_tpu_cores", 8,
+ "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+
+class InputExample(object):
+ """A single training/test example for simple sequence classification."""
+
+ def __init__(self, guid, text_a, text_b=None, label=None):
+ """Constructs a InputExample.
+
+ Args:
+ guid: Unique id for the example.
+ text_a: string. The untokenized text of the first sequence. For single
+ sequence tasks, only this sequence must be specified.
+ text_b: (Optional) string. The untokenized text of the second sequence.
+ Only must be specified for sequence pair tasks.
+ label: (Optional) string. The label of the example. This should be
+ specified for train and dev examples, but not for test examples.
+ """
+ self.guid = guid
+ self.text_a = text_a
+ self.text_b = text_b
+ self.label = label
+
+
+class PaddingInputExample(object):
+ """Fake example so the num input examples is a multiple of the batch size.
+
+ When running eval/predict on the TPU, we need to pad the number of examples
+ to be a multiple of the batch size, because the TPU requires a fixed batch
+ size. The alternative is to drop the last batch, which is bad because it means
+ the entire output data won't be generated.
+
+ We use this class instead of `None` because treating `None` as padding
+ battches could cause silent errors.
+ """
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self,
+ input_ids,
+ input_mask,
+ segment_ids,
+ label_id,
+ is_real_example=True):
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.segment_ids = segment_ids
+ self.label_id = label_id
+ self.is_real_example = is_real_example
+
+
+class DataProcessor(object):
+ """Base class for data converters for sequence classification data sets."""
+
+ def get_train_examples(self, data_dir):
+ """Gets a collection of `InputExample`s for the train set."""
+ raise NotImplementedError()
+
+ def get_dev_examples(self, data_dir):
+ """Gets a collection of `InputExample`s for the dev set."""
+ raise NotImplementedError()
+
+ def get_test_examples(self, data_dir):
+ """Gets a collection of `InputExample`s for prediction."""
+ raise NotImplementedError()
+
+ def get_labels(self):
+ """Gets the list of labels for this data set."""
+ raise NotImplementedError()
+
+ @classmethod
+ def _read_tsv(cls, input_file, quotechar=None):
+ """Reads a tab separated value file."""
+ with tf.gfile.Open(input_file, "r") as f:
+ reader = csv.reader(f, delimiter="\t", quotechar=quotechar)
+ lines = []
+ for line in reader:
+ lines.append(line)
+ return lines
+
+
+class XnliProcessor(DataProcessor):
+ """Processor for the XNLI data set."""
+
+ def __init__(self):
+ self.language = "zh"
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ lines = self._read_tsv(
+ os.path.join(data_dir, "multinli",
+ "multinli.train.%s.tsv" % self.language))
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "train-%d" % (i)
+ text_a = tokenization.convert_to_unicode(line[0])
+ text_b = tokenization.convert_to_unicode(line[1])
+ label = tokenization.convert_to_unicode(line[2])
+ if label == tokenization.convert_to_unicode("contradictory"):
+ label = tokenization.convert_to_unicode("contradiction")
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ lines = self._read_tsv(os.path.join(data_dir, "xnli.dev.tsv"))
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "dev-%d" % (i)
+ language = tokenization.convert_to_unicode(line[0])
+ if language != tokenization.convert_to_unicode(self.language):
+ continue
+ text_a = tokenization.convert_to_unicode(line[6])
+ text_b = tokenization.convert_to_unicode(line[7])
+ label = tokenization.convert_to_unicode(line[1])
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+ def get_labels(self):
+ """See base class."""
+ return ["contradiction", "entailment", "neutral"]
+
+
+class MnliProcessor(DataProcessor):
+ """Processor for the MultiNLI data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev_matched.tsv")),
+ "dev_matched")
+
+ def get_test_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "test_matched.tsv")), "test")
+
+ def get_labels(self):
+ """See base class."""
+ return ["contradiction", "entailment", "neutral"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, tokenization.convert_to_unicode(line[0]))
+ text_a = tokenization.convert_to_unicode(line[8])
+ text_b = tokenization.convert_to_unicode(line[9])
+ if set_type == "test":
+ label = "contradiction"
+ else:
+ label = tokenization.convert_to_unicode(line[-1])
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class MrpcProcessor(DataProcessor):
+ """Processor for the MRPC data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_test_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, i)
+ text_a = tokenization.convert_to_unicode(line[3])
+ text_b = tokenization.convert_to_unicode(line[4])
+ if set_type == "test":
+ label = "0"
+ else:
+ label = tokenization.convert_to_unicode(line[0])
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class ColaProcessor(DataProcessor):
+ """Processor for the CoLA data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_test_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "test.tsv")), "test")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ # Only the test set has a header
+ if set_type == "test" and i == 0:
+ continue
+ guid = "%s-%s" % (set_type, i)
+ if set_type == "test":
+ text_a = tokenization.convert_to_unicode(line[1])
+ label = "0"
+ else:
+ text_a = tokenization.convert_to_unicode(line[3])
+ label = tokenization.convert_to_unicode(line[1])
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
+ return examples
+
+
+def convert_single_example(ex_index, example, label_list, max_seq_length,
+ tokenizer):
+ """Converts a single `InputExample` into a single `InputFeatures`."""
+
+ if isinstance(example, PaddingInputExample):
+ return InputFeatures(
+ input_ids=[0] * max_seq_length,
+ input_mask=[0] * max_seq_length,
+ segment_ids=[0] * max_seq_length,
+ label_id=0,
+ is_real_example=False)
+
+ label_map = {}
+ for (i, label) in enumerate(label_list):
+ label_map[label] = i
+
+ tokens_a = tokenizer.tokenize(example.text_a)
+ tokens_b = None
+ if example.text_b:
+ tokens_b = tokenizer.tokenize(example.text_b)
+
+ if tokens_b:
+ # Modifies `tokens_a` and `tokens_b` in place so that the total
+ # length is less than the specified length.
+ # Account for [CLS], [SEP], [SEP] with "- 3"
+ _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
+ else:
+ # Account for [CLS] and [SEP] with "- 2"
+ if len(tokens_a) > max_seq_length - 2:
+ tokens_a = tokens_a[0:(max_seq_length - 2)]
+
+ # The convention in BERT is:
+ # (a) For sequence pairs:
+ # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+ # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
+ # (b) For single sequences:
+ # tokens: [CLS] the dog is hairy . [SEP]
+ # type_ids: 0 0 0 0 0 0 0
+ #
+ # Where "type_ids" are used to indicate whether this is the first
+ # sequence or the second sequence. The embedding vectors for `type=0` and
+ # `type=1` were learned during pre-training and are added to the wordpiece
+ # embedding vector (and position vector). This is not *strictly* necessary
+ # since the [SEP] token unambiguously separates the sequences, but it makes
+ # it easier for the model to learn the concept of sequences.
+ #
+ # For classification tasks, the first vector (corresponding to [CLS]) is
+ # used as the "sentence vector". Note that this only makes sense because
+ # the entire model is fine-tuned.
+ tokens = []
+ segment_ids = []
+ tokens.append("[CLS]")
+ segment_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ segment_ids.append(0)
+ tokens.append("[SEP]")
+ segment_ids.append(0)
+
+ if tokens_b:
+ for token in tokens_b:
+ tokens.append(token)
+ segment_ids.append(1)
+ tokens.append("[SEP]")
+ segment_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < max_seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ segment_ids.append(0)
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ label_id = label_map[example.label]
+ if ex_index < 5:
+ tf.logging.info("*** Example ***")
+ tf.logging.info("guid: %s" % (example.guid))
+ tf.logging.info("tokens: %s" % " ".join(
+ [tokenization.printable_text(x) for x in tokens]))
+ tf.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ tf.logging.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ tf.logging.info("segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+ tf.logging.info("label: %s (id = %d)" % (example.label, label_id))
+
+ feature = InputFeatures(
+ input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ label_id=label_id,
+ is_real_example=True)
+ return feature
+
+
+def file_based_convert_examples_to_features(
+ examples, label_list, max_seq_length, tokenizer, output_file):
+ """Convert a set of `InputExample`s to a TFRecord file."""
+
+ writer = tf.python_io.TFRecordWriter(output_file)
+
+ for (ex_index, example) in enumerate(examples):
+ if ex_index % 10000 == 0:
+ tf.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
+
+ feature = convert_single_example(ex_index, example, label_list,
+ max_seq_length, tokenizer)
+
+ def create_int_feature(values):
+ f = tf.train.Feature(int64_list=tf.train.Int64List(value=list(values)))
+ return f
+
+ features = collections.OrderedDict()
+ features["input_ids"] = create_int_feature(feature.input_ids)
+ features["input_mask"] = create_int_feature(feature.input_mask)
+ features["segment_ids"] = create_int_feature(feature.segment_ids)
+ features["label_ids"] = create_int_feature([feature.label_id])
+ features["is_real_example"] = create_int_feature(
+ [int(feature.is_real_example)])
+
+ tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+ writer.write(tf_example.SerializeToString())
+ writer.close()
+
+
+def file_based_input_fn_builder(input_file, seq_length, is_training,
+ drop_remainder):
+ """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+ name_to_features = {
+ "input_ids": tf.FixedLenFeature([seq_length], tf.int64),
+ "input_mask": tf.FixedLenFeature([seq_length], tf.int64),
+ "segment_ids": tf.FixedLenFeature([seq_length], tf.int64),
+ "label_ids": tf.FixedLenFeature([], tf.int64),
+ "is_real_example": tf.FixedLenFeature([], tf.int64),
+ }
+
+ def _decode_record(record, name_to_features):
+ """Decodes a record to a TensorFlow example."""
+ example = tf.parse_single_example(record, name_to_features)
+
+ # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+ # So cast all int64 to int32.
+ for name in list(example.keys()):
+ t = example[name]
+ if t.dtype == tf.int64:
+ t = tf.to_int32(t)
+ example[name] = t
+
+ return example
+
+ def input_fn(params):
+ """The actual input function."""
+ batch_size = params["batch_size"]
+
+ # For training, we want a lot of parallel reading and shuffling.
+ # For eval, we want no shuffling and parallel reading doesn't matter.
+ d = tf.data.TFRecordDataset(input_file)
+ if is_training:
+ d = d.repeat()
+ d = d.shuffle(buffer_size=100)
+
+ d = d.apply(
+ tf.contrib.data.map_and_batch(
+ lambda record: _decode_record(record, name_to_features),
+ batch_size=batch_size,
+ drop_remainder=drop_remainder))
+
+ return d
+
+ return input_fn
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+
+def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
+ labels, num_labels, use_one_hot_embeddings):
+ """Creates a classification model."""
+ model = modeling.BertModel(
+ config=bert_config,
+ is_training=is_training,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ token_type_ids=segment_ids,
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ # In the demo, we are doing a simple classification task on the entire
+ # segment.
+ #
+ # If you want to use the token-level output, use model.get_sequence_output()
+ # instead.
+ output_layer = model.get_pooled_output()
+
+ hidden_size = output_layer.shape[-1].value
+
+ output_weights = tf.get_variable(
+ "output_weights", [num_labels, hidden_size],
+ initializer=tf.truncated_normal_initializer(stddev=0.02))
+
+ output_bias = tf.get_variable(
+ "output_bias", [num_labels], initializer=tf.zeros_initializer())
+
+ with tf.variable_scope("loss"):
+ if is_training:
+ # I.e., 0.1 dropout
+ output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)
+
+ logits = tf.matmul(output_layer, output_weights, transpose_b=True)
+ logits = tf.nn.bias_add(logits, output_bias)
+ probabilities = tf.nn.softmax(logits, axis=-1)
+ log_probs = tf.nn.log_softmax(logits, axis=-1)
+
+ one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
+
+ per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
+ loss = tf.reduce_mean(per_example_loss)
+
+ return (loss, per_example_loss, logits, probabilities)
+
+
+def model_fn_builder(bert_config, num_labels, init_checkpoint, learning_rate,
+ num_train_steps, num_warmup_steps, use_tpu,
+ use_one_hot_embeddings):
+ """Returns `model_fn` closure for TPUEstimator."""
+
+ def model_fn(features, labels, mode, params): # pylint: disable=unused-argument
+ """The `model_fn` for TPUEstimator."""
+
+ tf.logging.info("*** Features ***")
+ for name in sorted(features.keys()):
+ tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape))
+
+ input_ids = features["input_ids"]
+ input_mask = features["input_mask"]
+ segment_ids = features["segment_ids"]
+ label_ids = features["label_ids"]
+ is_real_example = None
+ if "is_real_example" in features:
+ is_real_example = tf.cast(features["is_real_example"], dtype=tf.float32)
+ else:
+ is_real_example = tf.ones(tf.shape(label_ids), dtype=tf.float32)
+
+ is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+ (total_loss, per_example_loss, logits, probabilities) = create_model(
+ bert_config, is_training, input_ids, input_mask, segment_ids, label_ids,
+ num_labels, use_one_hot_embeddings)
+
+ tvars = tf.trainable_variables()
+ initialized_variable_names = {}
+ scaffold_fn = None
+ if init_checkpoint:
+ (assignment_map, initialized_variable_names
+ ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
+ if use_tpu:
+
+ def tpu_scaffold():
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+ return tf.train.Scaffold()
+
+ scaffold_fn = tpu_scaffold
+ else:
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+ tf.logging.info("**** Trainable Variables ****")
+ for var in tvars:
+ init_string = ""
+ if var.name in initialized_variable_names:
+ init_string = ", *INIT_FROM_CKPT*"
+ tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape,
+ init_string)
+
+ output_spec = None
+ if mode == tf.estimator.ModeKeys.TRAIN:
+
+ train_op = optimization.create_optimizer(
+ total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ train_op=train_op,
+ scaffold_fn=scaffold_fn)
+ elif mode == tf.estimator.ModeKeys.EVAL:
+
+ def metric_fn(per_example_loss, label_ids, logits, is_real_example):
+ predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)
+ accuracy = tf.metrics.accuracy(
+ labels=label_ids, predictions=predictions, weights=is_real_example)
+ loss = tf.metrics.mean(values=per_example_loss, weights=is_real_example)
+ return {
+ "eval_accuracy": accuracy,
+ "eval_loss": loss,
+ }
+
+ eval_metrics = (metric_fn,
+ [per_example_loss, label_ids, logits, is_real_example])
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ eval_metrics=eval_metrics,
+ scaffold_fn=scaffold_fn)
+ else:
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ predictions={"probabilities": probabilities},
+ scaffold_fn=scaffold_fn)
+ return output_spec
+
+ return model_fn
+
+
+# This function is not used by this file but is still used by the Colab and
+# people who depend on it.
+def input_fn_builder(features, seq_length, is_training, drop_remainder):
+ """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+ all_input_ids = []
+ all_input_mask = []
+ all_segment_ids = []
+ all_label_ids = []
+
+ for feature in features:
+ all_input_ids.append(feature.input_ids)
+ all_input_mask.append(feature.input_mask)
+ all_segment_ids.append(feature.segment_ids)
+ all_label_ids.append(feature.label_id)
+
+ def input_fn(params):
+ """The actual input function."""
+ batch_size = params["batch_size"]
+
+ num_examples = len(features)
+
+ # This is for demo purposes and does NOT scale to large data sets. We do
+ # not use Dataset.from_generator() because that uses tf.py_func which is
+ # not TPU compatible. The right way to load data is with TFRecordReader.
+ d = tf.data.Dataset.from_tensor_slices({
+ "input_ids":
+ tf.constant(
+ all_input_ids, shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ "input_mask":
+ tf.constant(
+ all_input_mask,
+ shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ "segment_ids":
+ tf.constant(
+ all_segment_ids,
+ shape=[num_examples, seq_length],
+ dtype=tf.int32),
+ "label_ids":
+ tf.constant(all_label_ids, shape=[num_examples], dtype=tf.int32),
+ })
+
+ if is_training:
+ d = d.repeat()
+ d = d.shuffle(buffer_size=100)
+
+ d = d.batch(batch_size=batch_size, drop_remainder=drop_remainder)
+ return d
+
+ return input_fn
+
+
+# This function is not used by this file but is still used by the Colab and
+# people who depend on it.
+def convert_examples_to_features(examples, label_list, max_seq_length,
+ tokenizer):
+ """Convert a set of `InputExample`s to a list of `InputFeatures`."""
+
+ features = []
+ for (ex_index, example) in enumerate(examples):
+ if ex_index % 10000 == 0:
+ tf.logging.info("Writing example %d of %d" % (ex_index, len(examples)))
+
+ feature = convert_single_example(ex_index, example, label_list,
+ max_seq_length, tokenizer)
+
+ features.append(feature)
+ return features
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ processors = {
+ "cola": ColaProcessor,
+ "mnli": MnliProcessor,
+ "mrpc": MrpcProcessor,
+ "xnli": XnliProcessor,
+ }
+
+ tokenization.validate_case_matches_checkpoint(FLAGS.do_lower_case,
+ FLAGS.init_checkpoint)
+
+ if not FLAGS.do_train and not FLAGS.do_eval and not FLAGS.do_predict:
+ raise ValueError(
+ "At least one of `do_train`, `do_eval` or `do_predict' must be True.")
+
+ bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+
+ if FLAGS.max_seq_length > bert_config.max_position_embeddings:
+ raise ValueError(
+ "Cannot use sequence length %d because the BERT model "
+ "was only trained up to sequence length %d" %
+ (FLAGS.max_seq_length, bert_config.max_position_embeddings))
+
+ tf.gfile.MakeDirs(FLAGS.output_dir)
+
+ task_name = FLAGS.task_name.lower()
+
+ if task_name not in processors:
+ raise ValueError("Task not found: %s" % (task_name))
+
+ processor = processors[task_name]()
+
+ label_list = processor.get_labels()
+
+ tokenizer = tokenization.FullTokenizer(
+ vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+ tpu_cluster_resolver = None
+ if FLAGS.use_tpu and FLAGS.tpu_name:
+ tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
+ FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+ is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
+ run_config = tf.contrib.tpu.RunConfig(
+ cluster=tpu_cluster_resolver,
+ master=FLAGS.master,
+ model_dir=FLAGS.output_dir,
+ save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+ tpu_config=tf.contrib.tpu.TPUConfig(
+ iterations_per_loop=FLAGS.iterations_per_loop,
+ num_shards=FLAGS.num_tpu_cores,
+ per_host_input_for_training=is_per_host))
+
+ train_examples = None
+ num_train_steps = None
+ num_warmup_steps = None
+ if FLAGS.do_train:
+ train_examples = processor.get_train_examples(FLAGS.data_dir)
+ num_train_steps = int(
+ len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+ num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+ model_fn = model_fn_builder(
+ bert_config=bert_config,
+ num_labels=len(label_list),
+ init_checkpoint=FLAGS.init_checkpoint,
+ learning_rate=FLAGS.learning_rate,
+ num_train_steps=num_train_steps,
+ num_warmup_steps=num_warmup_steps,
+ use_tpu=FLAGS.use_tpu,
+ use_one_hot_embeddings=FLAGS.use_tpu)
+
+ # If TPU is not available, this will fall back to normal Estimator on CPU
+ # or GPU.
+ estimator = tf.contrib.tpu.TPUEstimator(
+ use_tpu=FLAGS.use_tpu,
+ model_fn=model_fn,
+ config=run_config,
+ train_batch_size=FLAGS.train_batch_size,
+ eval_batch_size=FLAGS.eval_batch_size,
+ predict_batch_size=FLAGS.predict_batch_size)
+
+ if FLAGS.do_train:
+ train_file = os.path.join(FLAGS.output_dir, "train.tf_record")
+ file_based_convert_examples_to_features(
+ train_examples, label_list, FLAGS.max_seq_length, tokenizer, train_file)
+ tf.logging.info("***** Running training *****")
+ tf.logging.info(" Num examples = %d", len(train_examples))
+ tf.logging.info(" Batch size = %d", FLAGS.train_batch_size)
+ tf.logging.info(" Num steps = %d", num_train_steps)
+ train_input_fn = file_based_input_fn_builder(
+ input_file=train_file,
+ seq_length=FLAGS.max_seq_length,
+ is_training=True,
+ drop_remainder=True)
+ estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+ if FLAGS.do_eval:
+ eval_examples = processor.get_dev_examples(FLAGS.data_dir)
+ num_actual_eval_examples = len(eval_examples)
+ if FLAGS.use_tpu:
+ # TPU requires a fixed batch size for all batches, therefore the number
+ # of examples must be a multiple of the batch size, or else examples
+ # will get dropped. So we pad with fake examples which are ignored
+ # later on. These do NOT count towards the metric (all tf.metrics
+ # support a per-instance weight, and these get a weight of 0.0).
+ while len(eval_examples) % FLAGS.eval_batch_size != 0:
+ eval_examples.append(PaddingInputExample())
+
+ eval_file = os.path.join(FLAGS.output_dir, "eval.tf_record")
+ file_based_convert_examples_to_features(
+ eval_examples, label_list, FLAGS.max_seq_length, tokenizer, eval_file)
+
+ tf.logging.info("***** Running evaluation *****")
+ tf.logging.info(" Num examples = %d (%d actual, %d padding)",
+ len(eval_examples), num_actual_eval_examples,
+ len(eval_examples) - num_actual_eval_examples)
+ tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size)
+
+ # This tells the estimator to run through the entire set.
+ eval_steps = None
+ # However, if running eval on the TPU, you will need to specify the
+ # number of steps.
+ if FLAGS.use_tpu:
+ assert len(eval_examples) % FLAGS.eval_batch_size == 0
+ eval_steps = int(len(eval_examples) // FLAGS.eval_batch_size)
+
+ eval_drop_remainder = True if FLAGS.use_tpu else False
+ eval_input_fn = file_based_input_fn_builder(
+ input_file=eval_file,
+ seq_length=FLAGS.max_seq_length,
+ is_training=False,
+ drop_remainder=eval_drop_remainder)
+
+ result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)
+
+ output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt")
+ with tf.gfile.GFile(output_eval_file, "w") as writer:
+ tf.logging.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ tf.logging.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+ if FLAGS.do_predict:
+ predict_examples = processor.get_test_examples(FLAGS.data_dir)
+ num_actual_predict_examples = len(predict_examples)
+ if FLAGS.use_tpu:
+ # TPU requires a fixed batch size for all batches, therefore the number
+ # of examples must be a multiple of the batch size, or else examples
+ # will get dropped. So we pad with fake examples which are ignored
+ # later on.
+ while len(predict_examples) % FLAGS.predict_batch_size != 0:
+ predict_examples.append(PaddingInputExample())
+
+ predict_file = os.path.join(FLAGS.output_dir, "predict.tf_record")
+ file_based_convert_examples_to_features(predict_examples, label_list,
+ FLAGS.max_seq_length, tokenizer,
+ predict_file)
+
+ tf.logging.info("***** Running prediction*****")
+ tf.logging.info(" Num examples = %d (%d actual, %d padding)",
+ len(predict_examples), num_actual_predict_examples,
+ len(predict_examples) - num_actual_predict_examples)
+ tf.logging.info(" Batch size = %d", FLAGS.predict_batch_size)
+
+ predict_drop_remainder = True if FLAGS.use_tpu else False
+ predict_input_fn = file_based_input_fn_builder(
+ input_file=predict_file,
+ seq_length=FLAGS.max_seq_length,
+ is_training=False,
+ drop_remainder=predict_drop_remainder)
+
+ result = estimator.predict(input_fn=predict_input_fn)
+
+ output_predict_file = os.path.join(FLAGS.output_dir, "test_results.tsv")
+ with tf.gfile.GFile(output_predict_file, "w") as writer:
+ num_written_lines = 0
+ tf.logging.info("***** Predict results *****")
+ for (i, prediction) in enumerate(result):
+ probabilities = prediction["probabilities"]
+ if i >= num_actual_predict_examples:
+ break
+ output_line = "\t".join(
+ str(class_probability)
+ for class_probability in probabilities) + "\n"
+ writer.write(output_line)
+ num_written_lines += 1
+ assert num_written_lines == num_actual_predict_examples
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("data_dir")
+ flags.mark_flag_as_required("task_name")
+ flags.mark_flag_as_required("vocab_file")
+ flags.mark_flag_as_required("bert_config_file")
+ flags.mark_flag_as_required("output_dir")
+ tf.app.run()
diff --git a/bert-master/bert-master/run_classifier_with_tfhub.py b/bert-master/bert-master/run_classifier_with_tfhub.py
new file mode 100644
index 0000000000000000000000000000000000000000..9d2f80f6b4554cc1b849bddb4fff1a3e7d554035
--- /dev/null
+++ b/bert-master/bert-master/run_classifier_with_tfhub.py
@@ -0,0 +1,314 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner with TF-Hub."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import optimization
+import run_classifier
+import tokenization
+import tensorflow as tf
+import tensorflow_hub as hub
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string(
+ "bert_hub_module_handle", None,
+ "Handle for the BERT TF-Hub module.")
+
+
+def create_model(is_training, input_ids, input_mask, segment_ids, labels,
+ num_labels, bert_hub_module_handle):
+ """Creates a classification model."""
+ tags = set()
+ if is_training:
+ tags.add("train")
+ bert_module = hub.Module(bert_hub_module_handle, tags=tags, trainable=True)
+ bert_inputs = dict(
+ input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids)
+ bert_outputs = bert_module(
+ inputs=bert_inputs,
+ signature="tokens",
+ as_dict=True)
+
+ # In the demo, we are doing a simple classification task on the entire
+ # segment.
+ #
+ # If you want to use the token-level output, use
+ # bert_outputs["sequence_output"] instead.
+ output_layer = bert_outputs["pooled_output"]
+
+ hidden_size = output_layer.shape[-1].value
+
+ output_weights = tf.get_variable(
+ "output_weights", [num_labels, hidden_size],
+ initializer=tf.truncated_normal_initializer(stddev=0.02))
+
+ output_bias = tf.get_variable(
+ "output_bias", [num_labels], initializer=tf.zeros_initializer())
+
+ with tf.variable_scope("loss"):
+ if is_training:
+ # I.e., 0.1 dropout
+ output_layer = tf.nn.dropout(output_layer, keep_prob=0.9)
+
+ logits = tf.matmul(output_layer, output_weights, transpose_b=True)
+ logits = tf.nn.bias_add(logits, output_bias)
+ probabilities = tf.nn.softmax(logits, axis=-1)
+ log_probs = tf.nn.log_softmax(logits, axis=-1)
+
+ one_hot_labels = tf.one_hot(labels, depth=num_labels, dtype=tf.float32)
+
+ per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
+ loss = tf.reduce_mean(per_example_loss)
+
+ return (loss, per_example_loss, logits, probabilities)
+
+
+def model_fn_builder(num_labels, learning_rate, num_train_steps,
+ num_warmup_steps, use_tpu, bert_hub_module_handle):
+ """Returns `model_fn` closure for TPUEstimator."""
+
+ def model_fn(features, labels, mode, params): # pylint: disable=unused-argument
+ """The `model_fn` for TPUEstimator."""
+
+ tf.logging.info("*** Features ***")
+ for name in sorted(features.keys()):
+ tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape))
+
+ input_ids = features["input_ids"]
+ input_mask = features["input_mask"]
+ segment_ids = features["segment_ids"]
+ label_ids = features["label_ids"]
+
+ is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+ (total_loss, per_example_loss, logits, probabilities) = create_model(
+ is_training, input_ids, input_mask, segment_ids, label_ids, num_labels,
+ bert_hub_module_handle)
+
+ output_spec = None
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ train_op = optimization.create_optimizer(
+ total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ train_op=train_op)
+ elif mode == tf.estimator.ModeKeys.EVAL:
+
+ def metric_fn(per_example_loss, label_ids, logits):
+ predictions = tf.argmax(logits, axis=-1, output_type=tf.int32)
+ accuracy = tf.metrics.accuracy(label_ids, predictions)
+ loss = tf.metrics.mean(per_example_loss)
+ return {
+ "eval_accuracy": accuracy,
+ "eval_loss": loss,
+ }
+
+ eval_metrics = (metric_fn, [per_example_loss, label_ids, logits])
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ eval_metrics=eval_metrics)
+ elif mode == tf.estimator.ModeKeys.PREDICT:
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode, predictions={"probabilities": probabilities})
+ else:
+ raise ValueError(
+ "Only TRAIN, EVAL and PREDICT modes are supported: %s" % (mode))
+
+ return output_spec
+
+ return model_fn
+
+
+def create_tokenizer_from_hub_module(bert_hub_module_handle):
+ """Get the vocab file and casing info from the Hub module."""
+ with tf.Graph().as_default():
+ bert_module = hub.Module(bert_hub_module_handle)
+ tokenization_info = bert_module(signature="tokenization_info", as_dict=True)
+ with tf.Session() as sess:
+ vocab_file, do_lower_case = sess.run([tokenization_info["vocab_file"],
+ tokenization_info["do_lower_case"]])
+ return tokenization.FullTokenizer(
+ vocab_file=vocab_file, do_lower_case=do_lower_case)
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ processors = {
+ "cola": run_classifier.ColaProcessor,
+ "mnli": run_classifier.MnliProcessor,
+ "mrpc": run_classifier.MrpcProcessor,
+ }
+
+ if not FLAGS.do_train and not FLAGS.do_eval:
+ raise ValueError("At least one of `do_train` or `do_eval` must be True.")
+
+ tf.gfile.MakeDirs(FLAGS.output_dir)
+
+ task_name = FLAGS.task_name.lower()
+
+ if task_name not in processors:
+ raise ValueError("Task not found: %s" % (task_name))
+
+ processor = processors[task_name]()
+
+ label_list = processor.get_labels()
+
+ tokenizer = create_tokenizer_from_hub_module(FLAGS.bert_hub_module_handle)
+
+ tpu_cluster_resolver = None
+ if FLAGS.use_tpu and FLAGS.tpu_name:
+ tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
+ FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+ is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
+ run_config = tf.contrib.tpu.RunConfig(
+ cluster=tpu_cluster_resolver,
+ master=FLAGS.master,
+ model_dir=FLAGS.output_dir,
+ save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+ tpu_config=tf.contrib.tpu.TPUConfig(
+ iterations_per_loop=FLAGS.iterations_per_loop,
+ num_shards=FLAGS.num_tpu_cores,
+ per_host_input_for_training=is_per_host))
+
+ train_examples = None
+ num_train_steps = None
+ num_warmup_steps = None
+ if FLAGS.do_train:
+ train_examples = processor.get_train_examples(FLAGS.data_dir)
+ num_train_steps = int(
+ len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+ num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+ model_fn = model_fn_builder(
+ num_labels=len(label_list),
+ learning_rate=FLAGS.learning_rate,
+ num_train_steps=num_train_steps,
+ num_warmup_steps=num_warmup_steps,
+ use_tpu=FLAGS.use_tpu,
+ bert_hub_module_handle=FLAGS.bert_hub_module_handle)
+
+ # If TPU is not available, this will fall back to normal Estimator on CPU
+ # or GPU.
+ estimator = tf.contrib.tpu.TPUEstimator(
+ use_tpu=FLAGS.use_tpu,
+ model_fn=model_fn,
+ config=run_config,
+ train_batch_size=FLAGS.train_batch_size,
+ eval_batch_size=FLAGS.eval_batch_size,
+ predict_batch_size=FLAGS.predict_batch_size)
+
+ if FLAGS.do_train:
+ train_features = run_classifier.convert_examples_to_features(
+ train_examples, label_list, FLAGS.max_seq_length, tokenizer)
+ tf.logging.info("***** Running training *****")
+ tf.logging.info(" Num examples = %d", len(train_examples))
+ tf.logging.info(" Batch size = %d", FLAGS.train_batch_size)
+ tf.logging.info(" Num steps = %d", num_train_steps)
+ train_input_fn = run_classifier.input_fn_builder(
+ features=train_features,
+ seq_length=FLAGS.max_seq_length,
+ is_training=True,
+ drop_remainder=True)
+ estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+ if FLAGS.do_eval:
+ eval_examples = processor.get_dev_examples(FLAGS.data_dir)
+ eval_features = run_classifier.convert_examples_to_features(
+ eval_examples, label_list, FLAGS.max_seq_length, tokenizer)
+
+ tf.logging.info("***** Running evaluation *****")
+ tf.logging.info(" Num examples = %d", len(eval_examples))
+ tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size)
+
+ # This tells the estimator to run through the entire set.
+ eval_steps = None
+ # However, if running eval on the TPU, you will need to specify the
+ # number of steps.
+ if FLAGS.use_tpu:
+ # Eval will be slightly WRONG on the TPU because it will truncate
+ # the last batch.
+ eval_steps = int(len(eval_examples) / FLAGS.eval_batch_size)
+
+ eval_drop_remainder = True if FLAGS.use_tpu else False
+ eval_input_fn = run_classifier.input_fn_builder(
+ features=eval_features,
+ seq_length=FLAGS.max_seq_length,
+ is_training=False,
+ drop_remainder=eval_drop_remainder)
+
+ result = estimator.evaluate(input_fn=eval_input_fn, steps=eval_steps)
+
+ output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt")
+ with tf.gfile.GFile(output_eval_file, "w") as writer:
+ tf.logging.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ tf.logging.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+ if FLAGS.do_predict:
+ predict_examples = processor.get_test_examples(FLAGS.data_dir)
+ if FLAGS.use_tpu:
+ # Discard batch remainder if running on TPU
+ n = len(predict_examples)
+ predict_examples = predict_examples[:(n - n % FLAGS.predict_batch_size)]
+
+ predict_file = os.path.join(FLAGS.output_dir, "predict.tf_record")
+ run_classifier.file_based_convert_examples_to_features(
+ predict_examples, label_list, FLAGS.max_seq_length, tokenizer,
+ predict_file)
+
+ tf.logging.info("***** Running prediction*****")
+ tf.logging.info(" Num examples = %d", len(predict_examples))
+ tf.logging.info(" Batch size = %d", FLAGS.predict_batch_size)
+
+ predict_input_fn = run_classifier.file_based_input_fn_builder(
+ input_file=predict_file,
+ seq_length=FLAGS.max_seq_length,
+ is_training=False,
+ drop_remainder=FLAGS.use_tpu)
+
+ result = estimator.predict(input_fn=predict_input_fn)
+
+ output_predict_file = os.path.join(FLAGS.output_dir, "test_results.tsv")
+ with tf.gfile.GFile(output_predict_file, "w") as writer:
+ tf.logging.info("***** Predict results *****")
+ for prediction in result:
+ probabilities = prediction["probabilities"]
+ output_line = "\t".join(
+ str(class_probability)
+ for class_probability in probabilities) + "\n"
+ writer.write(output_line)
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("data_dir")
+ flags.mark_flag_as_required("task_name")
+ flags.mark_flag_as_required("bert_hub_module_handle")
+ flags.mark_flag_as_required("output_dir")
+ tf.app.run()
diff --git a/bert-master/bert-master/run_pretraining.py b/bert-master/bert-master/run_pretraining.py
new file mode 100644
index 0000000000000000000000000000000000000000..b118f62a38c04ab4c2d78c5678e1f8c758656c34
--- /dev/null
+++ b/bert-master/bert-master/run_pretraining.py
@@ -0,0 +1,493 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run masked LM/next sentence masked_lm pre-training for BERT."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import modeling
+import optimization
+import tensorflow as tf
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+ "bert_config_file", None,
+ "The config json file corresponding to the pre-trained BERT model. "
+ "This specifies the model architecture.")
+
+flags.DEFINE_string(
+ "input_file", None,
+ "Input TF example files (can be a glob or comma separated).")
+
+flags.DEFINE_string(
+ "output_dir", None,
+ "The output directory where the model checkpoints will be written.")
+
+## Other parameters
+flags.DEFINE_string(
+ "init_checkpoint", None,
+ "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_integer(
+ "max_seq_length", 128,
+ "The maximum total input sequence length after WordPiece tokenization. "
+ "Sequences longer than this will be truncated, and sequences shorter "
+ "than this will be padded. Must match data generation.")
+
+flags.DEFINE_integer(
+ "max_predictions_per_seq", 20,
+ "Maximum number of masked LM predictions per sequence. "
+ "Must match data generation.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_eval", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("eval_batch_size", 8, "Total batch size for eval.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_integer("num_train_steps", 100000, "Number of training steps.")
+
+flags.DEFINE_integer("num_warmup_steps", 10000, "Number of warmup steps.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+ "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+ "How many steps to make in each estimator call.")
+
+flags.DEFINE_integer("max_eval_steps", 100, "Maximum number of eval steps.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+tf.flags.DEFINE_string(
+ "tpu_name", None,
+ "The Cloud TPU to use for training. This should be either the name "
+ "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+ "url.")
+
+tf.flags.DEFINE_string(
+ "tpu_zone", None,
+ "[Optional] GCE zone where the Cloud TPU is located in. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string(
+ "gcp_project", None,
+ "[Optional] Project name for the Cloud TPU-enabled project. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+
+flags.DEFINE_integer(
+ "num_tpu_cores", 8,
+ "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+
+def model_fn_builder(bert_config, init_checkpoint, learning_rate,
+ num_train_steps, num_warmup_steps, use_tpu,
+ use_one_hot_embeddings):
+ """Returns `model_fn` closure for TPUEstimator."""
+
+ def model_fn(features, labels, mode, params): # pylint: disable=unused-argument
+ """The `model_fn` for TPUEstimator."""
+
+ tf.logging.info("*** Features ***")
+ for name in sorted(features.keys()):
+ tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape))
+
+ input_ids = features["input_ids"]
+ input_mask = features["input_mask"]
+ segment_ids = features["segment_ids"]
+ masked_lm_positions = features["masked_lm_positions"]
+ masked_lm_ids = features["masked_lm_ids"]
+ masked_lm_weights = features["masked_lm_weights"]
+ next_sentence_labels = features["next_sentence_labels"]
+
+ is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+ model = modeling.BertModel(
+ config=bert_config,
+ is_training=is_training,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ token_type_ids=segment_ids,
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ (masked_lm_loss,
+ masked_lm_example_loss, masked_lm_log_probs) = get_masked_lm_output(
+ bert_config, model.get_sequence_output(), model.get_embedding_table(),
+ masked_lm_positions, masked_lm_ids, masked_lm_weights)
+
+ (next_sentence_loss, next_sentence_example_loss,
+ next_sentence_log_probs) = get_next_sentence_output(
+ bert_config, model.get_pooled_output(), next_sentence_labels)
+
+ total_loss = masked_lm_loss + next_sentence_loss
+
+ tvars = tf.trainable_variables()
+
+ initialized_variable_names = {}
+ scaffold_fn = None
+ if init_checkpoint:
+ (assignment_map, initialized_variable_names
+ ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
+ if use_tpu:
+
+ def tpu_scaffold():
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+ return tf.train.Scaffold()
+
+ scaffold_fn = tpu_scaffold
+ else:
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+ tf.logging.info("**** Trainable Variables ****")
+ for var in tvars:
+ init_string = ""
+ if var.name in initialized_variable_names:
+ init_string = ", *INIT_FROM_CKPT*"
+ tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape,
+ init_string)
+
+ output_spec = None
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ train_op = optimization.create_optimizer(
+ total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ train_op=train_op,
+ scaffold_fn=scaffold_fn)
+ elif mode == tf.estimator.ModeKeys.EVAL:
+
+ def metric_fn(masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids,
+ masked_lm_weights, next_sentence_example_loss,
+ next_sentence_log_probs, next_sentence_labels):
+ """Computes the loss and accuracy of the model."""
+ masked_lm_log_probs = tf.reshape(masked_lm_log_probs,
+ [-1, masked_lm_log_probs.shape[-1]])
+ masked_lm_predictions = tf.argmax(
+ masked_lm_log_probs, axis=-1, output_type=tf.int32)
+ masked_lm_example_loss = tf.reshape(masked_lm_example_loss, [-1])
+ masked_lm_ids = tf.reshape(masked_lm_ids, [-1])
+ masked_lm_weights = tf.reshape(masked_lm_weights, [-1])
+ masked_lm_accuracy = tf.metrics.accuracy(
+ labels=masked_lm_ids,
+ predictions=masked_lm_predictions,
+ weights=masked_lm_weights)
+ masked_lm_mean_loss = tf.metrics.mean(
+ values=masked_lm_example_loss, weights=masked_lm_weights)
+
+ next_sentence_log_probs = tf.reshape(
+ next_sentence_log_probs, [-1, next_sentence_log_probs.shape[-1]])
+ next_sentence_predictions = tf.argmax(
+ next_sentence_log_probs, axis=-1, output_type=tf.int32)
+ next_sentence_labels = tf.reshape(next_sentence_labels, [-1])
+ next_sentence_accuracy = tf.metrics.accuracy(
+ labels=next_sentence_labels, predictions=next_sentence_predictions)
+ next_sentence_mean_loss = tf.metrics.mean(
+ values=next_sentence_example_loss)
+
+ return {
+ "masked_lm_accuracy": masked_lm_accuracy,
+ "masked_lm_loss": masked_lm_mean_loss,
+ "next_sentence_accuracy": next_sentence_accuracy,
+ "next_sentence_loss": next_sentence_mean_loss,
+ }
+
+ eval_metrics = (metric_fn, [
+ masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids,
+ masked_lm_weights, next_sentence_example_loss,
+ next_sentence_log_probs, next_sentence_labels
+ ])
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ eval_metrics=eval_metrics,
+ scaffold_fn=scaffold_fn)
+ else:
+ raise ValueError("Only TRAIN and EVAL modes are supported: %s" % (mode))
+
+ return output_spec
+
+ return model_fn
+
+
+def get_masked_lm_output(bert_config, input_tensor, output_weights, positions,
+ label_ids, label_weights):
+ """Get loss and log probs for the masked LM."""
+ input_tensor = gather_indexes(input_tensor, positions)
+
+ with tf.variable_scope("cls/predictions"):
+ # We apply one more non-linear transformation before the output layer.
+ # This matrix is not used after pre-training.
+ with tf.variable_scope("transform"):
+ input_tensor = tf.layers.dense(
+ input_tensor,
+ units=bert_config.hidden_size,
+ activation=modeling.get_activation(bert_config.hidden_act),
+ kernel_initializer=modeling.create_initializer(
+ bert_config.initializer_range))
+ input_tensor = modeling.layer_norm(input_tensor)
+
+ # The output weights are the same as the input embeddings, but there is
+ # an output-only bias for each token.
+ output_bias = tf.get_variable(
+ "output_bias",
+ shape=[bert_config.vocab_size],
+ initializer=tf.zeros_initializer())
+ logits = tf.matmul(input_tensor, output_weights, transpose_b=True)
+ logits = tf.nn.bias_add(logits, output_bias)
+ log_probs = tf.nn.log_softmax(logits, axis=-1)
+
+ label_ids = tf.reshape(label_ids, [-1])
+ label_weights = tf.reshape(label_weights, [-1])
+
+ one_hot_labels = tf.one_hot(
+ label_ids, depth=bert_config.vocab_size, dtype=tf.float32)
+
+ # The `positions` tensor might be zero-padded (if the sequence is too
+ # short to have the maximum number of predictions). The `label_weights`
+ # tensor has a value of 1.0 for every real prediction and 0.0 for the
+ # padding predictions.
+ per_example_loss = -tf.reduce_sum(log_probs * one_hot_labels, axis=[-1])
+ numerator = tf.reduce_sum(label_weights * per_example_loss)
+ denominator = tf.reduce_sum(label_weights) + 1e-5
+ loss = numerator / denominator
+
+ return (loss, per_example_loss, log_probs)
+
+
+def get_next_sentence_output(bert_config, input_tensor, labels):
+ """Get loss and log probs for the next sentence prediction."""
+
+ # Simple binary classification. Note that 0 is "next sentence" and 1 is
+ # "random sentence". This weight matrix is not used after pre-training.
+ with tf.variable_scope("cls/seq_relationship"):
+ output_weights = tf.get_variable(
+ "output_weights",
+ shape=[2, bert_config.hidden_size],
+ initializer=modeling.create_initializer(bert_config.initializer_range))
+ output_bias = tf.get_variable(
+ "output_bias", shape=[2], initializer=tf.zeros_initializer())
+
+ logits = tf.matmul(input_tensor, output_weights, transpose_b=True)
+ logits = tf.nn.bias_add(logits, output_bias)
+ log_probs = tf.nn.log_softmax(logits, axis=-1)
+ labels = tf.reshape(labels, [-1])
+ one_hot_labels = tf.one_hot(labels, depth=2, dtype=tf.float32)
+ per_example_loss = -tf.reduce_sum(one_hot_labels * log_probs, axis=-1)
+ loss = tf.reduce_mean(per_example_loss)
+ return (loss, per_example_loss, log_probs)
+
+
+def gather_indexes(sequence_tensor, positions):
+ """Gathers the vectors at the specific positions over a minibatch."""
+ sequence_shape = modeling.get_shape_list(sequence_tensor, expected_rank=3)
+ batch_size = sequence_shape[0]
+ seq_length = sequence_shape[1]
+ width = sequence_shape[2]
+
+ flat_offsets = tf.reshape(
+ tf.range(0, batch_size, dtype=tf.int32) * seq_length, [-1, 1])
+ flat_positions = tf.reshape(positions + flat_offsets, [-1])
+ flat_sequence_tensor = tf.reshape(sequence_tensor,
+ [batch_size * seq_length, width])
+ output_tensor = tf.gather(flat_sequence_tensor, flat_positions)
+ return output_tensor
+
+
+def input_fn_builder(input_files,
+ max_seq_length,
+ max_predictions_per_seq,
+ is_training,
+ num_cpu_threads=4):
+ """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+ def input_fn(params):
+ """The actual input function."""
+ batch_size = params["batch_size"]
+
+ name_to_features = {
+ "input_ids":
+ tf.FixedLenFeature([max_seq_length], tf.int64),
+ "input_mask":
+ tf.FixedLenFeature([max_seq_length], tf.int64),
+ "segment_ids":
+ tf.FixedLenFeature([max_seq_length], tf.int64),
+ "masked_lm_positions":
+ tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
+ "masked_lm_ids":
+ tf.FixedLenFeature([max_predictions_per_seq], tf.int64),
+ "masked_lm_weights":
+ tf.FixedLenFeature([max_predictions_per_seq], tf.float32),
+ "next_sentence_labels":
+ tf.FixedLenFeature([1], tf.int64),
+ }
+
+ # For training, we want a lot of parallel reading and shuffling.
+ # For eval, we want no shuffling and parallel reading doesn't matter.
+ if is_training:
+ d = tf.data.Dataset.from_tensor_slices(tf.constant(input_files))
+ d = d.repeat()
+ d = d.shuffle(buffer_size=len(input_files))
+
+ # `cycle_length` is the number of parallel files that get read.
+ cycle_length = min(num_cpu_threads, len(input_files))
+
+ # `sloppy` mode means that the interleaving is not exact. This adds
+ # even more randomness to the training pipeline.
+ d = d.apply(
+ tf.contrib.data.parallel_interleave(
+ tf.data.TFRecordDataset,
+ sloppy=is_training,
+ cycle_length=cycle_length))
+ d = d.shuffle(buffer_size=100)
+ else:
+ d = tf.data.TFRecordDataset(input_files)
+ # Since we evaluate for a fixed number of steps we don't want to encounter
+ # out-of-range exceptions.
+ d = d.repeat()
+
+ # We must `drop_remainder` on training because the TPU requires fixed
+ # size dimensions. For eval, we assume we are evaluating on the CPU or GPU
+ # and we *don't* want to drop the remainder, otherwise we wont cover
+ # every sample.
+ d = d.apply(
+ tf.contrib.data.map_and_batch(
+ lambda record: _decode_record(record, name_to_features),
+ batch_size=batch_size,
+ num_parallel_batches=num_cpu_threads,
+ drop_remainder=True))
+ return d
+
+ return input_fn
+
+
+def _decode_record(record, name_to_features):
+ """Decodes a record to a TensorFlow example."""
+ example = tf.parse_single_example(record, name_to_features)
+
+ # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+ # So cast all int64 to int32.
+ for name in list(example.keys()):
+ t = example[name]
+ if t.dtype == tf.int64:
+ t = tf.to_int32(t)
+ example[name] = t
+
+ return example
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ if not FLAGS.do_train and not FLAGS.do_eval:
+ raise ValueError("At least one of `do_train` or `do_eval` must be True.")
+
+ bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+
+ tf.gfile.MakeDirs(FLAGS.output_dir)
+
+ input_files = []
+ for input_pattern in FLAGS.input_file.split(","):
+ input_files.extend(tf.gfile.Glob(input_pattern))
+
+ tf.logging.info("*** Input Files ***")
+ for input_file in input_files:
+ tf.logging.info(" %s" % input_file)
+
+ tpu_cluster_resolver = None
+ if FLAGS.use_tpu and FLAGS.tpu_name:
+ tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
+ FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+ is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
+ run_config = tf.contrib.tpu.RunConfig(
+ cluster=tpu_cluster_resolver,
+ master=FLAGS.master,
+ model_dir=FLAGS.output_dir,
+ save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+ tpu_config=tf.contrib.tpu.TPUConfig(
+ iterations_per_loop=FLAGS.iterations_per_loop,
+ num_shards=FLAGS.num_tpu_cores,
+ per_host_input_for_training=is_per_host))
+
+ model_fn = model_fn_builder(
+ bert_config=bert_config,
+ init_checkpoint=FLAGS.init_checkpoint,
+ learning_rate=FLAGS.learning_rate,
+ num_train_steps=FLAGS.num_train_steps,
+ num_warmup_steps=FLAGS.num_warmup_steps,
+ use_tpu=FLAGS.use_tpu,
+ use_one_hot_embeddings=FLAGS.use_tpu)
+
+ # If TPU is not available, this will fall back to normal Estimator on CPU
+ # or GPU.
+ estimator = tf.contrib.tpu.TPUEstimator(
+ use_tpu=FLAGS.use_tpu,
+ model_fn=model_fn,
+ config=run_config,
+ train_batch_size=FLAGS.train_batch_size,
+ eval_batch_size=FLAGS.eval_batch_size)
+
+ if FLAGS.do_train:
+ tf.logging.info("***** Running training *****")
+ tf.logging.info(" Batch size = %d", FLAGS.train_batch_size)
+ train_input_fn = input_fn_builder(
+ input_files=input_files,
+ max_seq_length=FLAGS.max_seq_length,
+ max_predictions_per_seq=FLAGS.max_predictions_per_seq,
+ is_training=True)
+ estimator.train(input_fn=train_input_fn, max_steps=FLAGS.num_train_steps)
+
+ if FLAGS.do_eval:
+ tf.logging.info("***** Running evaluation *****")
+ tf.logging.info(" Batch size = %d", FLAGS.eval_batch_size)
+
+ eval_input_fn = input_fn_builder(
+ input_files=input_files,
+ max_seq_length=FLAGS.max_seq_length,
+ max_predictions_per_seq=FLAGS.max_predictions_per_seq,
+ is_training=False)
+
+ result = estimator.evaluate(
+ input_fn=eval_input_fn, steps=FLAGS.max_eval_steps)
+
+ output_eval_file = os.path.join(FLAGS.output_dir, "eval_results.txt")
+ with tf.gfile.GFile(output_eval_file, "w") as writer:
+ tf.logging.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ tf.logging.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("input_file")
+ flags.mark_flag_as_required("bert_config_file")
+ flags.mark_flag_as_required("output_dir")
+ tf.app.run()
diff --git a/bert-master/bert-master/run_squad.py b/bert-master/bert-master/run_squad.py
new file mode 100644
index 0000000000000000000000000000000000000000..edd4c3ed9863ea350dd8de330dbe9e84609a048c
--- /dev/null
+++ b/bert-master/bert-master/run_squad.py
@@ -0,0 +1,1283 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run BERT on SQuAD 1.1 and SQuAD 2.0."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import json
+import math
+import os
+import random
+import modeling
+import optimization
+import tokenization
+import six
+import tensorflow as tf
+
+flags = tf.flags
+
+FLAGS = flags.FLAGS
+
+## Required parameters
+flags.DEFINE_string(
+ "bert_config_file", None,
+ "The config json file corresponding to the pre-trained BERT model. "
+ "This specifies the model architecture.")
+
+flags.DEFINE_string("vocab_file", None,
+ "The vocabulary file that the BERT model was trained on.")
+
+flags.DEFINE_string(
+ "output_dir", None,
+ "The output directory where the model checkpoints will be written.")
+
+## Other parameters
+flags.DEFINE_string("train_file", None,
+ "SQuAD json for training. E.g., train-v1.1.json")
+
+flags.DEFINE_string(
+ "predict_file", None,
+ "SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json")
+
+flags.DEFINE_string(
+ "init_checkpoint", None,
+ "Initial checkpoint (usually from a pre-trained BERT model).")
+
+flags.DEFINE_bool(
+ "do_lower_case", True,
+ "Whether to lower case the input text. Should be True for uncased "
+ "models and False for cased models.")
+
+flags.DEFINE_integer(
+ "max_seq_length", 384,
+ "The maximum total input sequence length after WordPiece tokenization. "
+ "Sequences longer than this will be truncated, and sequences shorter "
+ "than this will be padded.")
+
+flags.DEFINE_integer(
+ "doc_stride", 128,
+ "When splitting up a long document into chunks, how much stride to "
+ "take between chunks.")
+
+flags.DEFINE_integer(
+ "max_query_length", 64,
+ "The maximum number of tokens for the question. Questions longer than "
+ "this will be truncated to this length.")
+
+flags.DEFINE_bool("do_train", False, "Whether to run training.")
+
+flags.DEFINE_bool("do_predict", False, "Whether to run eval on the dev set.")
+
+flags.DEFINE_integer("train_batch_size", 32, "Total batch size for training.")
+
+flags.DEFINE_integer("predict_batch_size", 8,
+ "Total batch size for predictions.")
+
+flags.DEFINE_float("learning_rate", 5e-5, "The initial learning rate for Adam.")
+
+flags.DEFINE_float("num_train_epochs", 3.0,
+ "Total number of training epochs to perform.")
+
+flags.DEFINE_float(
+ "warmup_proportion", 0.1,
+ "Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10% of training.")
+
+flags.DEFINE_integer("save_checkpoints_steps", 1000,
+ "How often to save the model checkpoint.")
+
+flags.DEFINE_integer("iterations_per_loop", 1000,
+ "How many steps to make in each estimator call.")
+
+flags.DEFINE_integer(
+ "n_best_size", 20,
+ "The total number of n-best predictions to generate in the "
+ "nbest_predictions.json output file.")
+
+flags.DEFINE_integer(
+ "max_answer_length", 30,
+ "The maximum length of an answer that can be generated. This is needed "
+ "because the start and end predictions are not conditioned on one another.")
+
+flags.DEFINE_bool("use_tpu", False, "Whether to use TPU or GPU/CPU.")
+
+tf.flags.DEFINE_string(
+ "tpu_name", None,
+ "The Cloud TPU to use for training. This should be either the name "
+ "used when creating the Cloud TPU, or a grpc://ip.address.of.tpu:8470 "
+ "url.")
+
+tf.flags.DEFINE_string(
+ "tpu_zone", None,
+ "[Optional] GCE zone where the Cloud TPU is located in. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string(
+ "gcp_project", None,
+ "[Optional] Project name for the Cloud TPU-enabled project. If not "
+ "specified, we will attempt to automatically detect the GCE project from "
+ "metadata.")
+
+tf.flags.DEFINE_string("master", None, "[Optional] TensorFlow master URL.")
+
+flags.DEFINE_integer(
+ "num_tpu_cores", 8,
+ "Only used if `use_tpu` is True. Total number of TPU cores to use.")
+
+flags.DEFINE_bool(
+ "verbose_logging", False,
+ "If true, all of the warnings related to data processing will be printed. "
+ "A number of warnings are expected for a normal SQuAD evaluation.")
+
+flags.DEFINE_bool(
+ "version_2_with_negative", False,
+ "If true, the SQuAD examples contain some that do not have an answer.")
+
+flags.DEFINE_float(
+ "null_score_diff_threshold", 0.0,
+ "If null_score - best_non_null is greater than the threshold predict null.")
+
+
+class SquadExample(object):
+ """A single training/test example for simple sequence classification.
+
+ For examples without an answer, the start and end position are -1.
+ """
+
+ def __init__(self,
+ qas_id,
+ question_text,
+ doc_tokens,
+ orig_answer_text=None,
+ start_position=None,
+ end_position=None,
+ is_impossible=False):
+ self.qas_id = qas_id
+ self.question_text = question_text
+ self.doc_tokens = doc_tokens
+ self.orig_answer_text = orig_answer_text
+ self.start_position = start_position
+ self.end_position = end_position
+ self.is_impossible = is_impossible
+
+ def __str__(self):
+ return self.__repr__()
+
+ def __repr__(self):
+ s = ""
+ s += "qas_id: %s" % (tokenization.printable_text(self.qas_id))
+ s += ", question_text: %s" % (
+ tokenization.printable_text(self.question_text))
+ s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
+ if self.start_position:
+ s += ", start_position: %d" % (self.start_position)
+ if self.start_position:
+ s += ", end_position: %d" % (self.end_position)
+ if self.start_position:
+ s += ", is_impossible: %r" % (self.is_impossible)
+ return s
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self,
+ unique_id,
+ example_index,
+ doc_span_index,
+ tokens,
+ token_to_orig_map,
+ token_is_max_context,
+ input_ids,
+ input_mask,
+ segment_ids,
+ start_position=None,
+ end_position=None,
+ is_impossible=None):
+ self.unique_id = unique_id
+ self.example_index = example_index
+ self.doc_span_index = doc_span_index
+ self.tokens = tokens
+ self.token_to_orig_map = token_to_orig_map
+ self.token_is_max_context = token_is_max_context
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.segment_ids = segment_ids
+ self.start_position = start_position
+ self.end_position = end_position
+ self.is_impossible = is_impossible
+
+
+def read_squad_examples(input_file, is_training):
+ """Read a SQuAD json file into a list of SquadExample."""
+ with tf.gfile.Open(input_file, "r") as reader:
+ input_data = json.load(reader)["data"]
+
+ def is_whitespace(c):
+ if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
+ return True
+ return False
+
+ examples = []
+ for entry in input_data:
+ for paragraph in entry["paragraphs"]:
+ paragraph_text = paragraph["context"]
+ doc_tokens = []
+ char_to_word_offset = []
+ prev_is_whitespace = True
+ for c in paragraph_text:
+ if is_whitespace(c):
+ prev_is_whitespace = True
+ else:
+ if prev_is_whitespace:
+ doc_tokens.append(c)
+ else:
+ doc_tokens[-1] += c
+ prev_is_whitespace = False
+ char_to_word_offset.append(len(doc_tokens) - 1)
+
+ for qa in paragraph["qas"]:
+ qas_id = qa["id"]
+ question_text = qa["question"]
+ start_position = None
+ end_position = None
+ orig_answer_text = None
+ is_impossible = False
+ if is_training:
+
+ if FLAGS.version_2_with_negative:
+ is_impossible = qa["is_impossible"]
+ if (len(qa["answers"]) != 1) and (not is_impossible):
+ raise ValueError(
+ "For training, each question should have exactly 1 answer.")
+ if not is_impossible:
+ answer = qa["answers"][0]
+ orig_answer_text = answer["text"]
+ answer_offset = answer["answer_start"]
+ answer_length = len(orig_answer_text)
+ start_position = char_to_word_offset[answer_offset]
+ end_position = char_to_word_offset[answer_offset + answer_length -
+ 1]
+ # Only add answers where the text can be exactly recovered from the
+ # document. If this CAN'T happen it's likely due to weird Unicode
+ # stuff so we will just skip the example.
+ #
+ # Note that this means for training mode, every example is NOT
+ # guaranteed to be preserved.
+ actual_text = " ".join(
+ doc_tokens[start_position:(end_position + 1)])
+ cleaned_answer_text = " ".join(
+ tokenization.whitespace_tokenize(orig_answer_text))
+ if actual_text.find(cleaned_answer_text) == -1:
+ tf.logging.warning("Could not find answer: '%s' vs. '%s'",
+ actual_text, cleaned_answer_text)
+ continue
+ else:
+ start_position = -1
+ end_position = -1
+ orig_answer_text = ""
+
+ example = SquadExample(
+ qas_id=qas_id,
+ question_text=question_text,
+ doc_tokens=doc_tokens,
+ orig_answer_text=orig_answer_text,
+ start_position=start_position,
+ end_position=end_position,
+ is_impossible=is_impossible)
+ examples.append(example)
+
+ return examples
+
+
+def convert_examples_to_features(examples, tokenizer, max_seq_length,
+ doc_stride, max_query_length, is_training,
+ output_fn):
+ """Loads a data file into a list of `InputBatch`s."""
+
+ unique_id = 1000000000
+
+ for (example_index, example) in enumerate(examples):
+ query_tokens = tokenizer.tokenize(example.question_text)
+
+ if len(query_tokens) > max_query_length:
+ query_tokens = query_tokens[0:max_query_length]
+
+ tok_to_orig_index = []
+ orig_to_tok_index = []
+ all_doc_tokens = []
+ for (i, token) in enumerate(example.doc_tokens):
+ orig_to_tok_index.append(len(all_doc_tokens))
+ sub_tokens = tokenizer.tokenize(token)
+ for sub_token in sub_tokens:
+ tok_to_orig_index.append(i)
+ all_doc_tokens.append(sub_token)
+
+ tok_start_position = None
+ tok_end_position = None
+ if is_training and example.is_impossible:
+ tok_start_position = -1
+ tok_end_position = -1
+ if is_training and not example.is_impossible:
+ tok_start_position = orig_to_tok_index[example.start_position]
+ if example.end_position < len(example.doc_tokens) - 1:
+ tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
+ else:
+ tok_end_position = len(all_doc_tokens) - 1
+ (tok_start_position, tok_end_position) = _improve_answer_span(
+ all_doc_tokens, tok_start_position, tok_end_position, tokenizer,
+ example.orig_answer_text)
+
+ # The -3 accounts for [CLS], [SEP] and [SEP]
+ max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
+
+ # We can have documents that are longer than the maximum sequence length.
+ # To deal with this we do a sliding window approach, where we take chunks
+ # of the up to our max length with a stride of `doc_stride`.
+ _DocSpan = collections.namedtuple( # pylint: disable=invalid-name
+ "DocSpan", ["start", "length"])
+ doc_spans = []
+ start_offset = 0
+ while start_offset < len(all_doc_tokens):
+ length = len(all_doc_tokens) - start_offset
+ if length > max_tokens_for_doc:
+ length = max_tokens_for_doc
+ doc_spans.append(_DocSpan(start=start_offset, length=length))
+ if start_offset + length == len(all_doc_tokens):
+ break
+ start_offset += min(length, doc_stride)
+
+ for (doc_span_index, doc_span) in enumerate(doc_spans):
+ tokens = []
+ token_to_orig_map = {}
+ token_is_max_context = {}
+ segment_ids = []
+ tokens.append("[CLS]")
+ segment_ids.append(0)
+ for token in query_tokens:
+ tokens.append(token)
+ segment_ids.append(0)
+ tokens.append("[SEP]")
+ segment_ids.append(0)
+
+ for i in range(doc_span.length):
+ split_token_index = doc_span.start + i
+ token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index]
+
+ is_max_context = _check_is_max_context(doc_spans, doc_span_index,
+ split_token_index)
+ token_is_max_context[len(tokens)] = is_max_context
+ tokens.append(all_doc_tokens[split_token_index])
+ segment_ids.append(1)
+ tokens.append("[SEP]")
+ segment_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < max_seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ segment_ids.append(0)
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ start_position = None
+ end_position = None
+ if is_training and not example.is_impossible:
+ # For training, if our document chunk does not contain an annotation
+ # we throw it out, since there is nothing to predict.
+ doc_start = doc_span.start
+ doc_end = doc_span.start + doc_span.length - 1
+ out_of_span = False
+ if not (tok_start_position >= doc_start and
+ tok_end_position <= doc_end):
+ out_of_span = True
+ if out_of_span:
+ start_position = 0
+ end_position = 0
+ else:
+ doc_offset = len(query_tokens) + 2
+ start_position = tok_start_position - doc_start + doc_offset
+ end_position = tok_end_position - doc_start + doc_offset
+
+ if is_training and example.is_impossible:
+ start_position = 0
+ end_position = 0
+
+ if example_index < 20:
+ tf.logging.info("*** Example ***")
+ tf.logging.info("unique_id: %s" % (unique_id))
+ tf.logging.info("example_index: %s" % (example_index))
+ tf.logging.info("doc_span_index: %s" % (doc_span_index))
+ tf.logging.info("tokens: %s" % " ".join(
+ [tokenization.printable_text(x) for x in tokens]))
+ tf.logging.info("token_to_orig_map: %s" % " ".join(
+ ["%d:%d" % (x, y) for (x, y) in six.iteritems(token_to_orig_map)]))
+ tf.logging.info("token_is_max_context: %s" % " ".join([
+ "%d:%s" % (x, y) for (x, y) in six.iteritems(token_is_max_context)
+ ]))
+ tf.logging.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ tf.logging.info(
+ "input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ tf.logging.info(
+ "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+ if is_training and example.is_impossible:
+ tf.logging.info("impossible example")
+ if is_training and not example.is_impossible:
+ answer_text = " ".join(tokens[start_position:(end_position + 1)])
+ tf.logging.info("start_position: %d" % (start_position))
+ tf.logging.info("end_position: %d" % (end_position))
+ tf.logging.info(
+ "answer: %s" % (tokenization.printable_text(answer_text)))
+
+ feature = InputFeatures(
+ unique_id=unique_id,
+ example_index=example_index,
+ doc_span_index=doc_span_index,
+ tokens=tokens,
+ token_to_orig_map=token_to_orig_map,
+ token_is_max_context=token_is_max_context,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ start_position=start_position,
+ end_position=end_position,
+ is_impossible=example.is_impossible)
+
+ # Run callback
+ output_fn(feature)
+
+ unique_id += 1
+
+
+def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer,
+ orig_answer_text):
+ """Returns tokenized answer spans that better match the annotated answer."""
+
+ # The SQuAD annotations are character based. We first project them to
+ # whitespace-tokenized words. But then after WordPiece tokenization, we can
+ # often find a "better match". For example:
+ #
+ # Question: What year was John Smith born?
+ # Context: The leader was John Smith (1895-1943).
+ # Answer: 1895
+ #
+ # The original whitespace-tokenized answer will be "(1895-1943).". However
+ # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match
+ # the exact answer, 1895.
+ #
+ # However, this is not always possible. Consider the following:
+ #
+ # Question: What country is the top exporter of electornics?
+ # Context: The Japanese electronics industry is the lagest in the world.
+ # Answer: Japan
+ #
+ # In this case, the annotator chose "Japan" as a character sub-span of
+ # the word "Japanese". Since our WordPiece tokenizer does not split
+ # "Japanese", we just use "Japanese" as the annotation. This is fairly rare
+ # in SQuAD, but does happen.
+ tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
+
+ for new_start in range(input_start, input_end + 1):
+ for new_end in range(input_end, new_start - 1, -1):
+ text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
+ if text_span == tok_answer_text:
+ return (new_start, new_end)
+
+ return (input_start, input_end)
+
+
+def _check_is_max_context(doc_spans, cur_span_index, position):
+ """Check if this is the 'max context' doc span for the token."""
+
+ # Because of the sliding window approach taken to scoring documents, a single
+ # token can appear in multiple documents. E.g.
+ # Doc: the man went to the store and bought a gallon of milk
+ # Span A: the man went to the
+ # Span B: to the store and bought
+ # Span C: and bought a gallon of
+ # ...
+ #
+ # Now the word 'bought' will have two scores from spans B and C. We only
+ # want to consider the score with "maximum context", which we define as
+ # the *minimum* of its left and right context (the *sum* of left and
+ # right context will always be the same, of course).
+ #
+ # In the example the maximum context for 'bought' would be span C since
+ # it has 1 left context and 3 right context, while span B has 4 left context
+ # and 0 right context.
+ best_score = None
+ best_span_index = None
+ for (span_index, doc_span) in enumerate(doc_spans):
+ end = doc_span.start + doc_span.length - 1
+ if position < doc_span.start:
+ continue
+ if position > end:
+ continue
+ num_left_context = position - doc_span.start
+ num_right_context = end - position
+ score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
+ if best_score is None or score > best_score:
+ best_score = score
+ best_span_index = span_index
+
+ return cur_span_index == best_span_index
+
+
+def create_model(bert_config, is_training, input_ids, input_mask, segment_ids,
+ use_one_hot_embeddings):
+ """Creates a classification model."""
+ model = modeling.BertModel(
+ config=bert_config,
+ is_training=is_training,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ token_type_ids=segment_ids,
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ final_hidden = model.get_sequence_output()
+
+ final_hidden_shape = modeling.get_shape_list(final_hidden, expected_rank=3)
+ batch_size = final_hidden_shape[0]
+ seq_length = final_hidden_shape[1]
+ hidden_size = final_hidden_shape[2]
+
+ output_weights = tf.get_variable(
+ "cls/squad/output_weights", [2, hidden_size],
+ initializer=tf.truncated_normal_initializer(stddev=0.02))
+
+ output_bias = tf.get_variable(
+ "cls/squad/output_bias", [2], initializer=tf.zeros_initializer())
+
+ final_hidden_matrix = tf.reshape(final_hidden,
+ [batch_size * seq_length, hidden_size])
+ logits = tf.matmul(final_hidden_matrix, output_weights, transpose_b=True)
+ logits = tf.nn.bias_add(logits, output_bias)
+
+ logits = tf.reshape(logits, [batch_size, seq_length, 2])
+ logits = tf.transpose(logits, [2, 0, 1])
+
+ unstacked_logits = tf.unstack(logits, axis=0)
+
+ (start_logits, end_logits) = (unstacked_logits[0], unstacked_logits[1])
+
+ return (start_logits, end_logits)
+
+
+def model_fn_builder(bert_config, init_checkpoint, learning_rate,
+ num_train_steps, num_warmup_steps, use_tpu,
+ use_one_hot_embeddings):
+ """Returns `model_fn` closure for TPUEstimator."""
+
+ def model_fn(features, labels, mode, params): # pylint: disable=unused-argument
+ """The `model_fn` for TPUEstimator."""
+
+ tf.logging.info("*** Features ***")
+ for name in sorted(features.keys()):
+ tf.logging.info(" name = %s, shape = %s" % (name, features[name].shape))
+
+ unique_ids = features["unique_ids"]
+ input_ids = features["input_ids"]
+ input_mask = features["input_mask"]
+ segment_ids = features["segment_ids"]
+
+ is_training = (mode == tf.estimator.ModeKeys.TRAIN)
+
+ (start_logits, end_logits) = create_model(
+ bert_config=bert_config,
+ is_training=is_training,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ use_one_hot_embeddings=use_one_hot_embeddings)
+
+ tvars = tf.trainable_variables()
+
+ initialized_variable_names = {}
+ scaffold_fn = None
+ if init_checkpoint:
+ (assignment_map, initialized_variable_names
+ ) = modeling.get_assignment_map_from_checkpoint(tvars, init_checkpoint)
+ if use_tpu:
+
+ def tpu_scaffold():
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+ return tf.train.Scaffold()
+
+ scaffold_fn = tpu_scaffold
+ else:
+ tf.train.init_from_checkpoint(init_checkpoint, assignment_map)
+
+ tf.logging.info("**** Trainable Variables ****")
+ for var in tvars:
+ init_string = ""
+ if var.name in initialized_variable_names:
+ init_string = ", *INIT_FROM_CKPT*"
+ tf.logging.info(" name = %s, shape = %s%s", var.name, var.shape,
+ init_string)
+
+ output_spec = None
+ if mode == tf.estimator.ModeKeys.TRAIN:
+ seq_length = modeling.get_shape_list(input_ids)[1]
+
+ def compute_loss(logits, positions):
+ one_hot_positions = tf.one_hot(
+ positions, depth=seq_length, dtype=tf.float32)
+ log_probs = tf.nn.log_softmax(logits, axis=-1)
+ loss = -tf.reduce_mean(
+ tf.reduce_sum(one_hot_positions * log_probs, axis=-1))
+ return loss
+
+ start_positions = features["start_positions"]
+ end_positions = features["end_positions"]
+
+ start_loss = compute_loss(start_logits, start_positions)
+ end_loss = compute_loss(end_logits, end_positions)
+
+ total_loss = (start_loss + end_loss) / 2.0
+
+ train_op = optimization.create_optimizer(
+ total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)
+
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode,
+ loss=total_loss,
+ train_op=train_op,
+ scaffold_fn=scaffold_fn)
+ elif mode == tf.estimator.ModeKeys.PREDICT:
+ predictions = {
+ "unique_ids": unique_ids,
+ "start_logits": start_logits,
+ "end_logits": end_logits,
+ }
+ output_spec = tf.contrib.tpu.TPUEstimatorSpec(
+ mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)
+ else:
+ raise ValueError(
+ "Only TRAIN and PREDICT modes are supported: %s" % (mode))
+
+ return output_spec
+
+ return model_fn
+
+
+def input_fn_builder(input_file, seq_length, is_training, drop_remainder):
+ """Creates an `input_fn` closure to be passed to TPUEstimator."""
+
+ name_to_features = {
+ "unique_ids": tf.FixedLenFeature([], tf.int64),
+ "input_ids": tf.FixedLenFeature([seq_length], tf.int64),
+ "input_mask": tf.FixedLenFeature([seq_length], tf.int64),
+ "segment_ids": tf.FixedLenFeature([seq_length], tf.int64),
+ }
+
+ if is_training:
+ name_to_features["start_positions"] = tf.FixedLenFeature([], tf.int64)
+ name_to_features["end_positions"] = tf.FixedLenFeature([], tf.int64)
+
+ def _decode_record(record, name_to_features):
+ """Decodes a record to a TensorFlow example."""
+ example = tf.parse_single_example(record, name_to_features)
+
+ # tf.Example only supports tf.int64, but the TPU only supports tf.int32.
+ # So cast all int64 to int32.
+ for name in list(example.keys()):
+ t = example[name]
+ if t.dtype == tf.int64:
+ t = tf.to_int32(t)
+ example[name] = t
+
+ return example
+
+ def input_fn(params):
+ """The actual input function."""
+ batch_size = params["batch_size"]
+
+ # For training, we want a lot of parallel reading and shuffling.
+ # For eval, we want no shuffling and parallel reading doesn't matter.
+ d = tf.data.TFRecordDataset(input_file)
+ if is_training:
+ d = d.repeat()
+ d = d.shuffle(buffer_size=100)
+
+ d = d.apply(
+ tf.contrib.data.map_and_batch(
+ lambda record: _decode_record(record, name_to_features),
+ batch_size=batch_size,
+ drop_remainder=drop_remainder))
+
+ return d
+
+ return input_fn
+
+
+RawResult = collections.namedtuple("RawResult",
+ ["unique_id", "start_logits", "end_logits"])
+
+
+def write_predictions(all_examples, all_features, all_results, n_best_size,
+ max_answer_length, do_lower_case, output_prediction_file,
+ output_nbest_file, output_null_log_odds_file):
+ """Write final predictions to the json file and log-odds of null if needed."""
+ tf.logging.info("Writing predictions to: %s" % (output_prediction_file))
+ tf.logging.info("Writing nbest to: %s" % (output_nbest_file))
+
+ example_index_to_features = collections.defaultdict(list)
+ for feature in all_features:
+ example_index_to_features[feature.example_index].append(feature)
+
+ unique_id_to_result = {}
+ for result in all_results:
+ unique_id_to_result[result.unique_id] = result
+
+ _PrelimPrediction = collections.namedtuple( # pylint: disable=invalid-name
+ "PrelimPrediction",
+ ["feature_index", "start_index", "end_index", "start_logit", "end_logit"])
+
+ all_predictions = collections.OrderedDict()
+ all_nbest_json = collections.OrderedDict()
+ scores_diff_json = collections.OrderedDict()
+
+ for (example_index, example) in enumerate(all_examples):
+ features = example_index_to_features[example_index]
+
+ prelim_predictions = []
+ # keep track of the minimum score of null start+end of position 0
+ score_null = 1000000 # large and positive
+ min_null_feature_index = 0 # the paragraph slice with min mull score
+ null_start_logit = 0 # the start logit at the slice with min null score
+ null_end_logit = 0 # the end logit at the slice with min null score
+ for (feature_index, feature) in enumerate(features):
+ result = unique_id_to_result[feature.unique_id]
+ start_indexes = _get_best_indexes(result.start_logits, n_best_size)
+ end_indexes = _get_best_indexes(result.end_logits, n_best_size)
+ # if we could have irrelevant answers, get the min score of irrelevant
+ if FLAGS.version_2_with_negative:
+ feature_null_score = result.start_logits[0] + result.end_logits[0]
+ if feature_null_score < score_null:
+ score_null = feature_null_score
+ min_null_feature_index = feature_index
+ null_start_logit = result.start_logits[0]
+ null_end_logit = result.end_logits[0]
+ for start_index in start_indexes:
+ for end_index in end_indexes:
+ # We could hypothetically create invalid predictions, e.g., predict
+ # that the start of the span is in the question. We throw out all
+ # invalid predictions.
+ if start_index >= len(feature.tokens):
+ continue
+ if end_index >= len(feature.tokens):
+ continue
+ if start_index not in feature.token_to_orig_map:
+ continue
+ if end_index not in feature.token_to_orig_map:
+ continue
+ if not feature.token_is_max_context.get(start_index, False):
+ continue
+ if end_index < start_index:
+ continue
+ length = end_index - start_index + 1
+ if length > max_answer_length:
+ continue
+ prelim_predictions.append(
+ _PrelimPrediction(
+ feature_index=feature_index,
+ start_index=start_index,
+ end_index=end_index,
+ start_logit=result.start_logits[start_index],
+ end_logit=result.end_logits[end_index]))
+
+ if FLAGS.version_2_with_negative:
+ prelim_predictions.append(
+ _PrelimPrediction(
+ feature_index=min_null_feature_index,
+ start_index=0,
+ end_index=0,
+ start_logit=null_start_logit,
+ end_logit=null_end_logit))
+ prelim_predictions = sorted(
+ prelim_predictions,
+ key=lambda x: (x.start_logit + x.end_logit),
+ reverse=True)
+
+ _NbestPrediction = collections.namedtuple( # pylint: disable=invalid-name
+ "NbestPrediction", ["text", "start_logit", "end_logit"])
+
+ seen_predictions = {}
+ nbest = []
+ for pred in prelim_predictions:
+ if len(nbest) >= n_best_size:
+ break
+ feature = features[pred.feature_index]
+ if pred.start_index > 0: # this is a non-null prediction
+ tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
+ orig_doc_start = feature.token_to_orig_map[pred.start_index]
+ orig_doc_end = feature.token_to_orig_map[pred.end_index]
+ orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
+ tok_text = " ".join(tok_tokens)
+
+ # De-tokenize WordPieces that have been split off.
+ tok_text = tok_text.replace(" ##", "")
+ tok_text = tok_text.replace("##", "")
+
+ # Clean whitespace
+ tok_text = tok_text.strip()
+ tok_text = " ".join(tok_text.split())
+ orig_text = " ".join(orig_tokens)
+
+ final_text = get_final_text(tok_text, orig_text, do_lower_case)
+ if final_text in seen_predictions:
+ continue
+
+ seen_predictions[final_text] = True
+ else:
+ final_text = ""
+ seen_predictions[final_text] = True
+
+ nbest.append(
+ _NbestPrediction(
+ text=final_text,
+ start_logit=pred.start_logit,
+ end_logit=pred.end_logit))
+
+ # if we didn't inlude the empty option in the n-best, inlcude it
+ if FLAGS.version_2_with_negative:
+ if "" not in seen_predictions:
+ nbest.append(
+ _NbestPrediction(
+ text="", start_logit=null_start_logit,
+ end_logit=null_end_logit))
+ # In very rare edge cases we could have no valid predictions. So we
+ # just create a nonce prediction in this case to avoid failure.
+ if not nbest:
+ nbest.append(
+ _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
+
+ assert len(nbest) >= 1
+
+ total_scores = []
+ best_non_null_entry = None
+ for entry in nbest:
+ total_scores.append(entry.start_logit + entry.end_logit)
+ if not best_non_null_entry:
+ if entry.text:
+ best_non_null_entry = entry
+
+ probs = _compute_softmax(total_scores)
+
+ nbest_json = []
+ for (i, entry) in enumerate(nbest):
+ output = collections.OrderedDict()
+ output["text"] = entry.text
+ output["probability"] = probs[i]
+ output["start_logit"] = entry.start_logit
+ output["end_logit"] = entry.end_logit
+ nbest_json.append(output)
+
+ assert len(nbest_json) >= 1
+
+ if not FLAGS.version_2_with_negative:
+ all_predictions[example.qas_id] = nbest_json[0]["text"]
+ else:
+ # predict "" iff the null score - the score of best non-null > threshold
+ score_diff = score_null - best_non_null_entry.start_logit - (
+ best_non_null_entry.end_logit)
+ scores_diff_json[example.qas_id] = score_diff
+ if score_diff > FLAGS.null_score_diff_threshold:
+ all_predictions[example.qas_id] = ""
+ else:
+ all_predictions[example.qas_id] = best_non_null_entry.text
+
+ all_nbest_json[example.qas_id] = nbest_json
+
+ with tf.gfile.GFile(output_prediction_file, "w") as writer:
+ writer.write(json.dumps(all_predictions, indent=4) + "\n")
+
+ with tf.gfile.GFile(output_nbest_file, "w") as writer:
+ writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
+
+ if FLAGS.version_2_with_negative:
+ with tf.gfile.GFile(output_null_log_odds_file, "w") as writer:
+ writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
+
+
+def get_final_text(pred_text, orig_text, do_lower_case):
+ """Project the tokenized prediction back to the original text."""
+
+ # When we created the data, we kept track of the alignment between original
+ # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
+ # now `orig_text` contains the span of our original text corresponding to the
+ # span that we predicted.
+ #
+ # However, `orig_text` may contain extra characters that we don't want in
+ # our prediction.
+ #
+ # For example, let's say:
+ # pred_text = steve smith
+ # orig_text = Steve Smith's
+ #
+ # We don't want to return `orig_text` because it contains the extra "'s".
+ #
+ # We don't want to return `pred_text` because it's already been normalized
+ # (the SQuAD eval script also does punctuation stripping/lower casing but
+ # our tokenizer does additional normalization like stripping accent
+ # characters).
+ #
+ # What we really want to return is "Steve Smith".
+ #
+ # Therefore, we have to apply a semi-complicated alignment heruistic between
+ # `pred_text` and `orig_text` to get a character-to-charcter alignment. This
+ # can fail in certain cases in which case we just return `orig_text`.
+
+ def _strip_spaces(text):
+ ns_chars = []
+ ns_to_s_map = collections.OrderedDict()
+ for (i, c) in enumerate(text):
+ if c == " ":
+ continue
+ ns_to_s_map[len(ns_chars)] = i
+ ns_chars.append(c)
+ ns_text = "".join(ns_chars)
+ return (ns_text, ns_to_s_map)
+
+ # We first tokenize `orig_text`, strip whitespace from the result
+ # and `pred_text`, and check if they are the same length. If they are
+ # NOT the same length, the heuristic has failed. If they are the same
+ # length, we assume the characters are one-to-one aligned.
+ tokenizer = tokenization.BasicTokenizer(do_lower_case=do_lower_case)
+
+ tok_text = " ".join(tokenizer.tokenize(orig_text))
+
+ start_position = tok_text.find(pred_text)
+ if start_position == -1:
+ if FLAGS.verbose_logging:
+ tf.logging.info(
+ "Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
+ return orig_text
+ end_position = start_position + len(pred_text) - 1
+
+ (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
+ (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
+
+ if len(orig_ns_text) != len(tok_ns_text):
+ if FLAGS.verbose_logging:
+ tf.logging.info("Length not equal after stripping spaces: '%s' vs '%s'",
+ orig_ns_text, tok_ns_text)
+ return orig_text
+
+ # We then project the characters in `pred_text` back to `orig_text` using
+ # the character-to-character alignment.
+ tok_s_to_ns_map = {}
+ for (i, tok_index) in six.iteritems(tok_ns_to_s_map):
+ tok_s_to_ns_map[tok_index] = i
+
+ orig_start_position = None
+ if start_position in tok_s_to_ns_map:
+ ns_start_position = tok_s_to_ns_map[start_position]
+ if ns_start_position in orig_ns_to_s_map:
+ orig_start_position = orig_ns_to_s_map[ns_start_position]
+
+ if orig_start_position is None:
+ if FLAGS.verbose_logging:
+ tf.logging.info("Couldn't map start position")
+ return orig_text
+
+ orig_end_position = None
+ if end_position in tok_s_to_ns_map:
+ ns_end_position = tok_s_to_ns_map[end_position]
+ if ns_end_position in orig_ns_to_s_map:
+ orig_end_position = orig_ns_to_s_map[ns_end_position]
+
+ if orig_end_position is None:
+ if FLAGS.verbose_logging:
+ tf.logging.info("Couldn't map end position")
+ return orig_text
+
+ output_text = orig_text[orig_start_position:(orig_end_position + 1)]
+ return output_text
+
+
+def _get_best_indexes(logits, n_best_size):
+ """Get the n-best logits from a list."""
+ index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)
+
+ best_indexes = []
+ for i in range(len(index_and_score)):
+ if i >= n_best_size:
+ break
+ best_indexes.append(index_and_score[i][0])
+ return best_indexes
+
+
+def _compute_softmax(scores):
+ """Compute softmax probability over raw logits."""
+ if not scores:
+ return []
+
+ max_score = None
+ for score in scores:
+ if max_score is None or score > max_score:
+ max_score = score
+
+ exp_scores = []
+ total_sum = 0.0
+ for score in scores:
+ x = math.exp(score - max_score)
+ exp_scores.append(x)
+ total_sum += x
+
+ probs = []
+ for score in exp_scores:
+ probs.append(score / total_sum)
+ return probs
+
+
+class FeatureWriter(object):
+ """Writes InputFeature to TF example file."""
+
+ def __init__(self, filename, is_training):
+ self.filename = filename
+ self.is_training = is_training
+ self.num_features = 0
+ self._writer = tf.python_io.TFRecordWriter(filename)
+
+ def process_feature(self, feature):
+ """Write a InputFeature to the TFRecordWriter as a tf.train.Example."""
+ self.num_features += 1
+
+ def create_int_feature(values):
+ feature = tf.train.Feature(
+ int64_list=tf.train.Int64List(value=list(values)))
+ return feature
+
+ features = collections.OrderedDict()
+ features["unique_ids"] = create_int_feature([feature.unique_id])
+ features["input_ids"] = create_int_feature(feature.input_ids)
+ features["input_mask"] = create_int_feature(feature.input_mask)
+ features["segment_ids"] = create_int_feature(feature.segment_ids)
+
+ if self.is_training:
+ features["start_positions"] = create_int_feature([feature.start_position])
+ features["end_positions"] = create_int_feature([feature.end_position])
+ impossible = 0
+ if feature.is_impossible:
+ impossible = 1
+ features["is_impossible"] = create_int_feature([impossible])
+
+ tf_example = tf.train.Example(features=tf.train.Features(feature=features))
+ self._writer.write(tf_example.SerializeToString())
+
+ def close(self):
+ self._writer.close()
+
+
+def validate_flags_or_throw(bert_config):
+ """Validate the input FLAGS or throw an exception."""
+ tokenization.validate_case_matches_checkpoint(FLAGS.do_lower_case,
+ FLAGS.init_checkpoint)
+
+ if not FLAGS.do_train and not FLAGS.do_predict:
+ raise ValueError("At least one of `do_train` or `do_predict` must be True.")
+
+ if FLAGS.do_train:
+ if not FLAGS.train_file:
+ raise ValueError(
+ "If `do_train` is True, then `train_file` must be specified.")
+ if FLAGS.do_predict:
+ if not FLAGS.predict_file:
+ raise ValueError(
+ "If `do_predict` is True, then `predict_file` must be specified.")
+
+ if FLAGS.max_seq_length > bert_config.max_position_embeddings:
+ raise ValueError(
+ "Cannot use sequence length %d because the BERT model "
+ "was only trained up to sequence length %d" %
+ (FLAGS.max_seq_length, bert_config.max_position_embeddings))
+
+ if FLAGS.max_seq_length <= FLAGS.max_query_length + 3:
+ raise ValueError(
+ "The max_seq_length (%d) must be greater than max_query_length "
+ "(%d) + 3" % (FLAGS.max_seq_length, FLAGS.max_query_length))
+
+
+def main(_):
+ tf.logging.set_verbosity(tf.logging.INFO)
+
+ bert_config = modeling.BertConfig.from_json_file(FLAGS.bert_config_file)
+
+ validate_flags_or_throw(bert_config)
+
+ tf.gfile.MakeDirs(FLAGS.output_dir)
+
+ tokenizer = tokenization.FullTokenizer(
+ vocab_file=FLAGS.vocab_file, do_lower_case=FLAGS.do_lower_case)
+
+ tpu_cluster_resolver = None
+ if FLAGS.use_tpu and FLAGS.tpu_name:
+ tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
+ FLAGS.tpu_name, zone=FLAGS.tpu_zone, project=FLAGS.gcp_project)
+
+ is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2
+ run_config = tf.contrib.tpu.RunConfig(
+ cluster=tpu_cluster_resolver,
+ master=FLAGS.master,
+ model_dir=FLAGS.output_dir,
+ save_checkpoints_steps=FLAGS.save_checkpoints_steps,
+ tpu_config=tf.contrib.tpu.TPUConfig(
+ iterations_per_loop=FLAGS.iterations_per_loop,
+ num_shards=FLAGS.num_tpu_cores,
+ per_host_input_for_training=is_per_host))
+
+ train_examples = None
+ num_train_steps = None
+ num_warmup_steps = None
+ if FLAGS.do_train:
+ train_examples = read_squad_examples(
+ input_file=FLAGS.train_file, is_training=True)
+ num_train_steps = int(
+ len(train_examples) / FLAGS.train_batch_size * FLAGS.num_train_epochs)
+ num_warmup_steps = int(num_train_steps * FLAGS.warmup_proportion)
+
+ # Pre-shuffle the input to avoid having to make a very large shuffle
+ # buffer in in the `input_fn`.
+ rng = random.Random(12345)
+ rng.shuffle(train_examples)
+
+ model_fn = model_fn_builder(
+ bert_config=bert_config,
+ init_checkpoint=FLAGS.init_checkpoint,
+ learning_rate=FLAGS.learning_rate,
+ num_train_steps=num_train_steps,
+ num_warmup_steps=num_warmup_steps,
+ use_tpu=FLAGS.use_tpu,
+ use_one_hot_embeddings=FLAGS.use_tpu)
+
+ # If TPU is not available, this will fall back to normal Estimator on CPU
+ # or GPU.
+ estimator = tf.contrib.tpu.TPUEstimator(
+ use_tpu=FLAGS.use_tpu,
+ model_fn=model_fn,
+ config=run_config,
+ train_batch_size=FLAGS.train_batch_size,
+ predict_batch_size=FLAGS.predict_batch_size)
+
+ if FLAGS.do_train:
+ # We write to a temporary file to avoid storing very large constant tensors
+ # in memory.
+ train_writer = FeatureWriter(
+ filename=os.path.join(FLAGS.output_dir, "train.tf_record"),
+ is_training=True)
+ convert_examples_to_features(
+ examples=train_examples,
+ tokenizer=tokenizer,
+ max_seq_length=FLAGS.max_seq_length,
+ doc_stride=FLAGS.doc_stride,
+ max_query_length=FLAGS.max_query_length,
+ is_training=True,
+ output_fn=train_writer.process_feature)
+ train_writer.close()
+
+ tf.logging.info("***** Running training *****")
+ tf.logging.info(" Num orig examples = %d", len(train_examples))
+ tf.logging.info(" Num split examples = %d", train_writer.num_features)
+ tf.logging.info(" Batch size = %d", FLAGS.train_batch_size)
+ tf.logging.info(" Num steps = %d", num_train_steps)
+ del train_examples
+
+ train_input_fn = input_fn_builder(
+ input_file=train_writer.filename,
+ seq_length=FLAGS.max_seq_length,
+ is_training=True,
+ drop_remainder=True)
+ estimator.train(input_fn=train_input_fn, max_steps=num_train_steps)
+
+ if FLAGS.do_predict:
+ eval_examples = read_squad_examples(
+ input_file=FLAGS.predict_file, is_training=False)
+
+ eval_writer = FeatureWriter(
+ filename=os.path.join(FLAGS.output_dir, "eval.tf_record"),
+ is_training=False)
+ eval_features = []
+
+ def append_feature(feature):
+ eval_features.append(feature)
+ eval_writer.process_feature(feature)
+
+ convert_examples_to_features(
+ examples=eval_examples,
+ tokenizer=tokenizer,
+ max_seq_length=FLAGS.max_seq_length,
+ doc_stride=FLAGS.doc_stride,
+ max_query_length=FLAGS.max_query_length,
+ is_training=False,
+ output_fn=append_feature)
+ eval_writer.close()
+
+ tf.logging.info("***** Running predictions *****")
+ tf.logging.info(" Num orig examples = %d", len(eval_examples))
+ tf.logging.info(" Num split examples = %d", len(eval_features))
+ tf.logging.info(" Batch size = %d", FLAGS.predict_batch_size)
+
+ all_results = []
+
+ predict_input_fn = input_fn_builder(
+ input_file=eval_writer.filename,
+ seq_length=FLAGS.max_seq_length,
+ is_training=False,
+ drop_remainder=False)
+
+ # If running eval on the TPU, you will need to specify the number of
+ # steps.
+ all_results = []
+ for result in estimator.predict(
+ predict_input_fn, yield_single_examples=True):
+ if len(all_results) % 1000 == 0:
+ tf.logging.info("Processing example: %d" % (len(all_results)))
+ unique_id = int(result["unique_ids"])
+ start_logits = [float(x) for x in result["start_logits"].flat]
+ end_logits = [float(x) for x in result["end_logits"].flat]
+ all_results.append(
+ RawResult(
+ unique_id=unique_id,
+ start_logits=start_logits,
+ end_logits=end_logits))
+
+ output_prediction_file = os.path.join(FLAGS.output_dir, "predictions.json")
+ output_nbest_file = os.path.join(FLAGS.output_dir, "nbest_predictions.json")
+ output_null_log_odds_file = os.path.join(FLAGS.output_dir, "null_odds.json")
+
+ write_predictions(eval_examples, eval_features, all_results,
+ FLAGS.n_best_size, FLAGS.max_answer_length,
+ FLAGS.do_lower_case, output_prediction_file,
+ output_nbest_file, output_null_log_odds_file)
+
+
+if __name__ == "__main__":
+ flags.mark_flag_as_required("vocab_file")
+ flags.mark_flag_as_required("bert_config_file")
+ flags.mark_flag_as_required("output_dir")
+ tf.app.run()
diff --git a/bert-master/bert-master/sample_text.txt b/bert-master/bert-master/sample_text.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a42812060c576bae870eb29b1ac083fda0d239d3
--- /dev/null
+++ b/bert-master/bert-master/sample_text.txt
@@ -0,0 +1,33 @@
+This text is included to make sure Unicode is handled properly: 力加勝北区ᴵᴺᵀᵃছজটডণত
+Text should be one-sentence-per-line, with empty lines between documents.
+This sample text is public domain and was randomly selected from Project Guttenberg.
+
+The rain had only ceased with the gray streaks of morning at Blazing Star, and the settlement awoke to a moral sense of cleanliness, and the finding of forgotten knives, tin cups, and smaller camp utensils, where the heavy showers had washed away the debris and dust heaps before the cabin doors.
+Indeed, it was recorded in Blazing Star that a fortunate early riser had once picked up on the highway a solid chunk of gold quartz which the rain had freed from its incumbering soil, and washed into immediate and glittering popularity.
+Possibly this may have been the reason why early risers in that locality, during the rainy season, adopted a thoughtful habit of body, and seldom lifted their eyes to the rifted or india-ink washed skies above them.
+"Cass" Beard had risen early that morning, but not with a view to discovery.
+A leak in his cabin roof,--quite consistent with his careless, improvident habits,--had roused him at 4 A. M., with a flooded "bunk" and wet blankets.
+The chips from his wood pile refused to kindle a fire to dry his bed-clothes, and he had recourse to a more provident neighbor's to supply the deficiency.
+This was nearly opposite.
+Mr. Cassius crossed the highway, and stopped suddenly.
+Something glittered in the nearest red pool before him.
+Gold, surely!
+But, wonderful to relate, not an irregular, shapeless fragment of crude ore, fresh from Nature's crucible, but a bit of jeweler's handicraft in the form of a plain gold ring.
+Looking at it more attentively, he saw that it bore the inscription, "May to Cass."
+Like most of his fellow gold-seekers, Cass was superstitious.
+
+The fountain of classic wisdom, Hypatia herself.
+As the ancient sage--the name is unimportant to a monk--pumped water nightly that he might study by day, so I, the guardian of cloaks and parasols, at the sacred doors of her lecture-room, imbibe celestial knowledge.
+From my youth I felt in me a soul above the matter-entangled herd.
+She revealed to me the glorious fact, that I am a spark of Divinity itself.
+A fallen star, I am, sir!' continued he, pensively, stroking his lean stomach--'a fallen star!--fallen, if the dignity of philosophy will allow of the simile, among the hogs of the lower world--indeed, even into the hog-bucket itself. Well, after all, I will show you the way to the Archbishop's.
+There is a philosophic pleasure in opening one's treasures to the modest young.
+Perhaps you will assist me by carrying this basket of fruit?' And the little man jumped up, put his basket on Philammon's head, and trotted off up a neighbouring street.
+Philammon followed, half contemptuous, half wondering at what this philosophy might be, which could feed the self-conceit of anything so abject as his ragged little apish guide;
+but the novel roar and whirl of the street, the perpetual stream of busy faces, the line of curricles, palanquins, laden asses, camels, elephants, which met and passed him, and squeezed him up steps and into doorways, as they threaded their way through the great Moon-gate into the ample street beyond, drove everything from his mind but wondering curiosity, and a vague, helpless dread of that great living wilderness, more terrible than any dead wilderness of sand which he had left behind.
+Already he longed for the repose, the silence of the Laura--for faces which knew him and smiled upon him; but it was too late to turn back now.
+His guide held on for more than a mile up the great main street, crossed in the centre of the city, at right angles, by one equally magnificent, at each end of which, miles away, appeared, dim and distant over the heads of the living stream of passengers, the yellow sand-hills of the desert;
+while at the end of the vista in front of them gleamed the blue harbour, through a network of countless masts.
+At last they reached the quay at the opposite end of the street;
+and there burst on Philammon's astonished eyes a vast semicircle of blue sea, ringed with palaces and towers.
+He stopped involuntarily; and his little guide stopped also, and looked askance at the young monk, to watch the effect which that grand panorama should produce on him.
diff --git a/bert-master/bert-master/tokenization.py b/bert-master/bert-master/tokenization.py
new file mode 100644
index 0000000000000000000000000000000000000000..0ee13595356d7a62080218202b144ce3c9f3bcfc
--- /dev/null
+++ b/bert-master/bert-master/tokenization.py
@@ -0,0 +1,399 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+import re
+import unicodedata
+import six
+import tensorflow as tf
+
+
+def validate_case_matches_checkpoint(do_lower_case, init_checkpoint):
+ """Checks whether the casing config is consistent with the checkpoint name."""
+
+ # The casing has to be passed in by the user and there is no explicit check
+ # as to whether it matches the checkpoint. The casing information probably
+ # should have been stored in the bert_config.json file, but it's not, so
+ # we have to heuristically detect it to validate.
+
+ if not init_checkpoint:
+ return
+
+ m = re.match("^.*?([A-Za-z0-9_-]+)/bert_model.ckpt", init_checkpoint)
+ if m is None:
+ return
+
+ model_name = m.group(1)
+
+ lower_models = [
+ "uncased_L-24_H-1024_A-16", "uncased_L-12_H-768_A-12",
+ "multilingual_L-12_H-768_A-12", "chinese_L-12_H-768_A-12"
+ ]
+
+ cased_models = [
+ "cased_L-12_H-768_A-12", "cased_L-24_H-1024_A-16",
+ "multi_cased_L-12_H-768_A-12"
+ ]
+
+ is_bad_config = False
+ if model_name in lower_models and not do_lower_case:
+ is_bad_config = True
+ actual_flag = "False"
+ case_name = "lowercased"
+ opposite_flag = "True"
+
+ if model_name in cased_models and do_lower_case:
+ is_bad_config = True
+ actual_flag = "True"
+ case_name = "cased"
+ opposite_flag = "False"
+
+ if is_bad_config:
+ raise ValueError(
+ "You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. "
+ "However, `%s` seems to be a %s model, so you "
+ "should pass in `--do_lower_case=%s` so that the fine-tuning matches "
+ "how the model was pre-training. If this error is wrong, please "
+ "just comment out this check." % (actual_flag, init_checkpoint,
+ model_name, case_name, opposite_flag))
+
+
+def convert_to_unicode(text):
+ """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
+ if six.PY3:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, bytes):
+ return text.decode("utf-8", "ignore")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ elif six.PY2:
+ if isinstance(text, str):
+ return text.decode("utf-8", "ignore")
+ elif isinstance(text, unicode):
+ return text
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ else:
+ raise ValueError("Not running on Python2 or Python 3?")
+
+
+def printable_text(text):
+ """Returns text encoded in a way suitable for print or `tf.logging`."""
+
+ # These functions want `str` for both Python2 and Python3, but in one case
+ # it's a Unicode string and in the other it's a byte string.
+ if six.PY3:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, bytes):
+ return text.decode("utf-8", "ignore")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ elif six.PY2:
+ if isinstance(text, str):
+ return text
+ elif isinstance(text, unicode):
+ return text.encode("utf-8")
+ else:
+ raise ValueError("Unsupported string type: %s" % (type(text)))
+ else:
+ raise ValueError("Not running on Python2 or Python 3?")
+
+
+def load_vocab(vocab_file):
+ """Loads a vocabulary file into a dictionary."""
+ vocab = collections.OrderedDict()
+ index = 0
+ with tf.gfile.GFile(vocab_file, "r") as reader:
+ while True:
+ token = convert_to_unicode(reader.readline())
+ if not token:
+ break
+ token = token.strip()
+ vocab[token] = index
+ index += 1
+ return vocab
+
+
+def convert_by_vocab(vocab, items):
+ """Converts a sequence of [tokens|ids] using the vocab."""
+ output = []
+ for item in items:
+ output.append(vocab[item])
+ return output
+
+
+def convert_tokens_to_ids(vocab, tokens):
+ return convert_by_vocab(vocab, tokens)
+
+
+def convert_ids_to_tokens(inv_vocab, ids):
+ return convert_by_vocab(inv_vocab, ids)
+
+
+def whitespace_tokenize(text):
+ """Runs basic whitespace cleaning and splitting on a piece of text."""
+ text = text.strip()
+ if not text:
+ return []
+ tokens = text.split()
+ return tokens
+
+
+class FullTokenizer(object):
+ """Runs end-to-end tokenziation."""
+
+ def __init__(self, vocab_file, do_lower_case=True):
+ self.vocab = load_vocab(vocab_file)
+ self.inv_vocab = {v: k for k, v in self.vocab.items()}
+ self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
+ self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+
+ def tokenize(self, text):
+ split_tokens = []
+ for token in self.basic_tokenizer.tokenize(text):
+ for sub_token in self.wordpiece_tokenizer.tokenize(token):
+ split_tokens.append(sub_token)
+
+ return split_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ return convert_by_vocab(self.vocab, tokens)
+
+ def convert_ids_to_tokens(self, ids):
+ return convert_by_vocab(self.inv_vocab, ids)
+
+
+class BasicTokenizer(object):
+ """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
+
+ def __init__(self, do_lower_case=True):
+ """Constructs a BasicTokenizer.
+
+ Args:
+ do_lower_case: Whether to lower case the input.
+ """
+ self.do_lower_case = do_lower_case
+
+ def tokenize(self, text):
+ """Tokenizes a piece of text."""
+ text = convert_to_unicode(text)
+ text = self._clean_text(text)
+
+ # This was added on November 1st, 2018 for the multilingual and Chinese
+ # models. This is also applied to the English models now, but it doesn't
+ # matter since the English models were not trained on any Chinese data
+ # and generally don't have any Chinese data in them (there are Chinese
+ # characters in the vocabulary because Wikipedia does have some Chinese
+ # words in the English Wikipedia.).
+ text = self._tokenize_chinese_chars(text)
+
+ orig_tokens = whitespace_tokenize(text)
+ split_tokens = []
+ for token in orig_tokens:
+ if self.do_lower_case:
+ token = token.lower()
+ token = self._run_strip_accents(token)
+ split_tokens.extend(self._run_split_on_punc(token))
+
+ output_tokens = whitespace_tokenize(" ".join(split_tokens))
+ return output_tokens
+
+ def _run_strip_accents(self, text):
+ """Strips accents from a piece of text."""
+ text = unicodedata.normalize("NFD", text)
+ output = []
+ for char in text:
+ cat = unicodedata.category(char)
+ if cat == "Mn":
+ continue
+ output.append(char)
+ return "".join(output)
+
+ def _run_split_on_punc(self, text):
+ """Splits punctuation on a piece of text."""
+ chars = list(text)
+ i = 0
+ start_new_word = True
+ output = []
+ while i < len(chars):
+ char = chars[i]
+ if _is_punctuation(char):
+ output.append([char])
+ start_new_word = True
+ else:
+ if start_new_word:
+ output.append([])
+ start_new_word = False
+ output[-1].append(char)
+ i += 1
+
+ return ["".join(x) for x in output]
+
+ def _tokenize_chinese_chars(self, text):
+ """Adds whitespace around any CJK character."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if self._is_chinese_char(cp):
+ output.append(" ")
+ output.append(char)
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+ def _is_chinese_char(self, cp):
+ """Checks whether CP is the codepoint of a CJK character."""
+ # This defines a "chinese character" as anything in the CJK Unicode block:
+ # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+ #
+ # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
+ # despite its name. The modern Korean Hangul alphabet is a different block,
+ # as is Japanese Hiragana and Katakana. Those alphabets are used to write
+ # space-separated words, so they are not treated specially and handled
+ # like the all of the other languages.
+ if ((cp >= 0x4E00 and cp <= 0x9FFF) or #
+ (cp >= 0x3400 and cp <= 0x4DBF) or #
+ (cp >= 0x20000 and cp <= 0x2A6DF) or #
+ (cp >= 0x2A700 and cp <= 0x2B73F) or #
+ (cp >= 0x2B740 and cp <= 0x2B81F) or #
+ (cp >= 0x2B820 and cp <= 0x2CEAF) or
+ (cp >= 0xF900 and cp <= 0xFAFF) or #
+ (cp >= 0x2F800 and cp <= 0x2FA1F)): #
+ return True
+
+ return False
+
+ def _clean_text(self, text):
+ """Performs invalid character removal and whitespace cleanup on text."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if cp == 0 or cp == 0xfffd or _is_control(char):
+ continue
+ if _is_whitespace(char):
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+
+class WordpieceTokenizer(object):
+ """Runs WordPiece tokenziation."""
+
+ def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200):
+ self.vocab = vocab
+ self.unk_token = unk_token
+ self.max_input_chars_per_word = max_input_chars_per_word
+
+ def tokenize(self, text):
+ """Tokenizes a piece of text into its word pieces.
+
+ This uses a greedy longest-match-first algorithm to perform tokenization
+ using the given vocabulary.
+
+ For example:
+ input = "unaffable"
+ output = ["un", "##aff", "##able"]
+
+ Args:
+ text: A single token or whitespace separated tokens. This should have
+ already been passed through `BasicTokenizer.
+
+ Returns:
+ A list of wordpiece tokens.
+ """
+
+ text = convert_to_unicode(text)
+
+ output_tokens = []
+ for token in whitespace_tokenize(text):
+ chars = list(token)
+ if len(chars) > self.max_input_chars_per_word:
+ output_tokens.append(self.unk_token)
+ continue
+
+ is_bad = False
+ start = 0
+ sub_tokens = []
+ while start < len(chars):
+ end = len(chars)
+ cur_substr = None
+ while start < end:
+ substr = "".join(chars[start:end])
+ if start > 0:
+ substr = "##" + substr
+ if substr in self.vocab:
+ cur_substr = substr
+ break
+ end -= 1
+ if cur_substr is None:
+ is_bad = True
+ break
+ sub_tokens.append(cur_substr)
+ start = end
+
+ if is_bad:
+ output_tokens.append(self.unk_token)
+ else:
+ output_tokens.extend(sub_tokens)
+ return output_tokens
+
+
+def _is_whitespace(char):
+ """Checks whether `chars` is a whitespace character."""
+ # \t, \n, and \r are technically contorl characters but we treat them
+ # as whitespace since they are generally considered as such.
+ if char == " " or char == "\t" or char == "\n" or char == "\r":
+ return True
+ cat = unicodedata.category(char)
+ if cat == "Zs":
+ return True
+ return False
+
+
+def _is_control(char):
+ """Checks whether `chars` is a control character."""
+ # These are technically control characters but we count them as whitespace
+ # characters.
+ if char == "\t" or char == "\n" or char == "\r":
+ return False
+ cat = unicodedata.category(char)
+ if cat in ("Cc", "Cf"):
+ return True
+ return False
+
+
+def _is_punctuation(char):
+ """Checks whether `chars` is a punctuation character."""
+ cp = ord(char)
+ # We treat all non-letter/number ASCII as punctuation.
+ # Characters such as "^", "$", and "`" are not in the Unicode
+ # Punctuation class but we treat them as punctuation anyways, for
+ # consistency.
+ if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
+ (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
+ return True
+ cat = unicodedata.category(char)
+ if cat.startswith("P"):
+ return True
+ return False
diff --git a/bert-master/bert-master/tokenization_test.py b/bert-master/bert-master/tokenization_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..0afaedd2eab5b655c3b4129cffed16d89872a00d
--- /dev/null
+++ b/bert-master/bert-master/tokenization_test.py
@@ -0,0 +1,137 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import tempfile
+import tokenization
+import six
+import tensorflow as tf
+
+
+class TokenizationTest(tf.test.TestCase):
+
+ def test_full_tokenizer(self):
+ vocab_tokens = [
+ "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn",
+ "##ing", ","
+ ]
+ with tempfile.NamedTemporaryFile(delete=False) as vocab_writer:
+ if six.PY2:
+ vocab_writer.write("".join([x + "\n" for x in vocab_tokens]))
+ else:
+ vocab_writer.write("".join(
+ [x + "\n" for x in vocab_tokens]).encode("utf-8"))
+
+ vocab_file = vocab_writer.name
+
+ tokenizer = tokenization.FullTokenizer(vocab_file)
+ os.unlink(vocab_file)
+
+ tokens = tokenizer.tokenize(u"UNwant\u00E9d,running")
+ self.assertAllEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"])
+
+ self.assertAllEqual(
+ tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9])
+
+ def test_chinese(self):
+ tokenizer = tokenization.BasicTokenizer()
+
+ self.assertAllEqual(
+ tokenizer.tokenize(u"ah\u535A\u63A8zz"),
+ [u"ah", u"\u535A", u"\u63A8", u"zz"])
+
+ def test_basic_tokenizer_lower(self):
+ tokenizer = tokenization.BasicTokenizer(do_lower_case=True)
+
+ self.assertAllEqual(
+ tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "),
+ ["hello", "!", "how", "are", "you", "?"])
+ self.assertAllEqual(tokenizer.tokenize(u"H\u00E9llo"), ["hello"])
+
+ def test_basic_tokenizer_no_lower(self):
+ tokenizer = tokenization.BasicTokenizer(do_lower_case=False)
+
+ self.assertAllEqual(
+ tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "),
+ ["HeLLo", "!", "how", "Are", "yoU", "?"])
+
+ def test_wordpiece_tokenizer(self):
+ vocab_tokens = [
+ "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn",
+ "##ing"
+ ]
+
+ vocab = {}
+ for (i, token) in enumerate(vocab_tokens):
+ vocab[token] = i
+ tokenizer = tokenization.WordpieceTokenizer(vocab=vocab)
+
+ self.assertAllEqual(tokenizer.tokenize(""), [])
+
+ self.assertAllEqual(
+ tokenizer.tokenize("unwanted running"),
+ ["un", "##want", "##ed", "runn", "##ing"])
+
+ self.assertAllEqual(
+ tokenizer.tokenize("unwantedX running"), ["[UNK]", "runn", "##ing"])
+
+ def test_convert_tokens_to_ids(self):
+ vocab_tokens = [
+ "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn",
+ "##ing"
+ ]
+
+ vocab = {}
+ for (i, token) in enumerate(vocab_tokens):
+ vocab[token] = i
+
+ self.assertAllEqual(
+ tokenization.convert_tokens_to_ids(
+ vocab, ["un", "##want", "##ed", "runn", "##ing"]), [7, 4, 5, 8, 9])
+
+ def test_is_whitespace(self):
+ self.assertTrue(tokenization._is_whitespace(u" "))
+ self.assertTrue(tokenization._is_whitespace(u"\t"))
+ self.assertTrue(tokenization._is_whitespace(u"\r"))
+ self.assertTrue(tokenization._is_whitespace(u"\n"))
+ self.assertTrue(tokenization._is_whitespace(u"\u00A0"))
+
+ self.assertFalse(tokenization._is_whitespace(u"A"))
+ self.assertFalse(tokenization._is_whitespace(u"-"))
+
+ def test_is_control(self):
+ self.assertTrue(tokenization._is_control(u"\u0005"))
+
+ self.assertFalse(tokenization._is_control(u"A"))
+ self.assertFalse(tokenization._is_control(u" "))
+ self.assertFalse(tokenization._is_control(u"\t"))
+ self.assertFalse(tokenization._is_control(u"\r"))
+ self.assertFalse(tokenization._is_control(u"\U0001F4A9"))
+
+ def test_is_punctuation(self):
+ self.assertTrue(tokenization._is_punctuation(u"-"))
+ self.assertTrue(tokenization._is_punctuation(u"$"))
+ self.assertTrue(tokenization._is_punctuation(u"`"))
+ self.assertTrue(tokenization._is_punctuation(u"."))
+
+ self.assertFalse(tokenization._is_punctuation(u"A"))
+ self.assertFalse(tokenization._is_punctuation(u" "))
+
+
+if __name__ == "__main__":
+ tf.test.main()
diff --git a/dark-bert-master/dark-bert-master/LICENSE b/dark-bert-master/dark-bert-master/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..261eeb9e9f8b2b4b0d119366dda99c6fd7d35c64
--- /dev/null
+++ b/dark-bert-master/dark-bert-master/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/dark-bert-master/dark-bert-master/README.md b/dark-bert-master/dark-bert-master/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..7cc342a97b5ac57e9193e41abf763a64231c8529
--- /dev/null
+++ b/dark-bert-master/dark-bert-master/README.md
@@ -0,0 +1,14 @@
+# dark-bert
+🧠 Using large language models to classify dark net documents in a zero-shot learning enviornments.
+
+Dark bert eneables you to cluster any corpus of markup documents in an entirely unsupervised way.
+
+```
+usage: darkbert.py [-h] -m {bert,albert,roberta} -i INPUT -o OUTPUT
+
+optional arguments:
+ -h, --help show this help message and exit
+ -m {bert,albert,roberta}, --model {bert,albert,roberta}
+ -i INPUT, --input INPUT
+ -o OUTPUT, --output OUTPUT
+```
\ No newline at end of file
diff --git a/dark-bert-master/dark-bert-master/darkbert.py b/dark-bert-master/dark-bert-master/darkbert.py
new file mode 100644
index 0000000000000000000000000000000000000000..d0c90bc9991129194f44c047bbd291c12e27e42e
--- /dev/null
+++ b/dark-bert-master/dark-bert-master/darkbert.py
@@ -0,0 +1,151 @@
+# Copyright 2022 Christopher K. Schmitt
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from sentence_transformers import SentenceTransformer
+from sklearn.manifold import TSNE
+from sklearn.cluster import DBSCAN
+from sklearn.metrics import silhouette_score, calinski_harabasz_score
+from pathlib import Path
+from bs4 import BeautifulSoup
+from argparse import ArgumentParser
+
+import matplotlib.pyplot as plt
+import numpy as np
+import nltk as nltk
+
+# The list of huggingface transformers with tensorflow
+# support and compatible tokenizers.
+available_models = {
+ "bert": "sentence-transformers/multi-qa-distilbert-cos-v1",
+ "albert": "sentence-transformers/paraphrase-albert-small-v2",
+ "roberta": "sentence-transformers/all-distilroberta-v1",
+}
+
+display_titles = {
+ "bert": "BERT",
+ "albert": "ALBERT",
+ "roberta": "RoBERTa",
+}
+
+# Define the CLI interface for modeling our data with
+# different transformer models. We want to control the
+# type of the tokenizer and the transformer we use, as well
+# as the input and output directories
+parser = ArgumentParser()
+parser.add_argument("-m", "--model", choices=available_models.keys(), required=True)
+parser.add_argument("-i", "--input", required=True)
+parser.add_argument("-o", "--output", required=True)
+
+args = parser.parse_args()
+input_dir = args.input
+output_dir = args.output
+model_name = available_models[args.model]
+display_name = display_titles[args.model]
+
+# To remove random glyphs and other noise, we
+# only extract words in the nltk corpus
+nltk.download("words")
+words = set(nltk.corpus.words.words())
+
+def extract_words(document):
+ cleaned = ""
+
+ for word in nltk.wordpunct_tokenize(document):
+ if word.lower() in words:
+ cleaned += word.lower() + " "
+
+ return cleaned
+
+# Iterate over all of the files in the provided data
+# directory. Parse each file with beautiful soup to parse
+# the relevant text out of the markup.
+data = Path(input_dir).iterdir()
+data = map(lambda doc: doc.read_bytes(), data)
+data = map(lambda doc: BeautifulSoup(doc, "html.parser"), data)
+data = map(lambda doc: doc.get_text(), data)
+data = filter(lambda doc: len(doc) > 0, data)
+data = map(extract_words, data)
+data = filter(lambda doc: len(doc) > 10, data)
+data = list(data)
+
+# Initilize transformer models and predict all of the
+# document embeddings as computed by bert and friends
+model = SentenceTransformer(model_name)
+embeddings = model.encode(data, show_progress_bar=True)
+
+# Fit TSNE model for embedding space. Sqush down to 2
+# dimentions for visualization purposes.
+tsne = TSNE(n_components=2, random_state=2, init="pca", learning_rate="auto", perplexity=40)
+tsne = tsne.fit_transform(embeddings)
+
+# Hyperparameter optimizations
+silhouettes = []
+outliers = []
+ch = []
+
+for eps in np.arange(0.001, 1, 0.001):
+ dbscan = DBSCAN(eps, metric="cosine", n_jobs=-1)
+ dbscan = dbscan.fit_predict(embeddings)
+
+ if len(np.unique(dbscan)) > 1:
+ silhouettes.append(silhouette_score(embeddings, dbscan, metric="cosine"))
+ ch.append(calinski_harabasz_score(embeddings, dbscan))
+ else:
+ silhouettes.append(0)
+ ch.append(0)
+
+ outliers.append(len(dbscan[dbscan == -1]))
+
+for p in range(15, 51):
+ best = np.argmax(silhouettes)
+
+ dbscan = DBSCAN(0.001 + 0.001 * best, metric="cosine", n_jobs=-1)
+ dbscan = dbscan.fit_predict(embeddings)
+
+ tsne = TSNE(n_components=2, perplexity=p, learning_rate="auto", init="pca", metric="cosine")
+ tsne = tsne.fit_transform(embeddings)
+
+ plt.figure()
+ plt.scatter(tsne[dbscan != -1][:, 0], tsne[dbscan != -1][:, 1], s=0.5, c=dbscan[dbscan != -1], cmap="hsv")
+ plt.scatter(tsne[dbscan == -1][:, 0], tsne[dbscan == -1][:, 1], s=0.5, c="#abb8c3")
+ plt.title(f"{display_name} Embeddings Visualized with T-SNE (p = {p})")
+ plt.savefig(f"{output_dir}/tnse_{p:02}.png", format="png", dpi=600)
+ plt.close()
+
+plt.figure()
+plt.plot(np.arange(0.001, 1, 0.001), silhouettes, lw=0.5, color="#dc322f")
+plt.legend()
+plt.xlabel("Epsilon")
+plt.ylabel("silhouette score")
+plt.title("Optimizing Epsilon by Silhouette Score")
+plt.savefig(f"silhouettes.png", format="png", dpi=600)
+plt.close()
+
+plt.figure()
+plt.plot(np.arange(0.001, 1, 0.001), outliers, lw=0.5, color="#dc322f")
+plt.legend()
+plt.xlabel("Epsilon")
+plt.ylabel("outliers")
+plt.title("Optimizing Epsilon by Number of Outliers")
+plt.savefig(f"outliers.png", format="png", dpi=600)
+plt.close()
+
+plt.figure()
+plt.plot(np.arange(0.001, 1, 0.001), ch, lw=0.5, color="#dc322f")
+plt.legend()
+plt.xlabel("Epsilon")
+plt.ylabel("Calinski-Harabasz score")
+plt.title("Optimizing Epsilon by Calinski-Harabasz Score")
+plt.savefig(f"calinski-harabasz.png", format="png", dpi=600)
+plt.close()
\ No newline at end of file
diff --git a/dark-bert-master/dark-bert-master/requirements.txt b/dark-bert-master/dark-bert-master/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..05997240bb2678482328835d5727633371f8c708
Binary files /dev/null and b/dark-bert-master/dark-bert-master/requirements.txt differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master.gitignore b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master.gitignore
new file mode 100644
index 0000000000000000000000000000000000000000..aeff829aa0a747bd209b435c5d2c55b07bdcdf44
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master.gitignore
@@ -0,0 +1,125 @@
+# Initially taken from Github's Python gitignore file
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# celery beat schedule file
+celerybeat-schedule
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# vscode
+.vscode
+
+# TF code
+tensorflow_code
+
+# Models
+models
\ No newline at end of file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.circleci/config.yml b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.circleci/config.yml
new file mode 100644
index 0000000000000000000000000000000000000000..7296e07ca396e7dc11e6663e0cd4eb3b91768a4d
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.circleci/config.yml
@@ -0,0 +1,29 @@
+version: 2
+jobs:
+ build_py3:
+ working_directory: ~/pytorch-pretrained-BERT
+ docker:
+ - image: circleci/python:3.5
+ steps:
+ - checkout
+ - run: sudo pip install --progress-bar off .
+ - run: sudo pip install pytest ftfy spacy
+ - run: sudo python -m spacy download en
+ - run: python -m pytest -sv tests/ --runslow
+ build_py2:
+ working_directory: ~/pytorch-pretrained-BERT
+ docker:
+ - image: circleci/python:2.7
+ steps:
+ - checkout
+ - run: sudo pip install --progress-bar off .
+ - run: sudo pip install pytest spacy
+ - run: sudo pip install ftfy==4.4.3
+ - run: sudo python -m spacy download en
+ - run: python -m pytest -sv tests/ --runslow
+workflows:
+ version: 2
+ build_and_test:
+ jobs:
+ - build_py3
+ - build_py2
\ No newline at end of file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.github/stale.yml b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.github/stale.yml
new file mode 100644
index 0000000000000000000000000000000000000000..d9f6563218bd0fe4f7b1a36170801bdf982f12c9
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/.github/stale.yml
@@ -0,0 +1,17 @@
+# Number of days of inactivity before an issue becomes stale
+daysUntilStale: 60
+# Number of days of inactivity before a stale issue is closed
+daysUntilClose: 7
+# Issues with these labels will never be considered stale
+exemptLabels:
+ - pinned
+ - security
+# Label to use when marking an issue as stale
+staleLabel: wontfix
+# Comment to post when marking an issue as stale. Set to `false` to disable
+markComment: >
+ This issue has been automatically marked as stale because it has not had
+ recent activity. It will be closed if no further activity occurs. Thank you
+ for your contributions.
+# Comment to post when closing a stale issue. Set to `false` to disable
+closeComment: false
\ No newline at end of file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/LICENSE b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/LICENSE
new file mode 100644
index 0000000000000000000000000000000000000000..d645695673349e3947e8e5ae42332d0ac3164cd7
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/LICENSE
@@ -0,0 +1,202 @@
+
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/MANIFEST.in b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/MANIFEST.in
new file mode 100644
index 0000000000000000000000000000000000000000..1aba38f67a2211cf5b09466d7b411206cb7223bf
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/MANIFEST.in
@@ -0,0 +1 @@
+include LICENSE
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/README.md b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..58c45cb56a868a1a4f4eb1b058d9755e2a7284cc
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/README.md
@@ -0,0 +1,30 @@
+# Introduction
+For installation and docs please refer to [release 0.6 of pytorch_pretrained_bert](https://github.com/huggingface/transformers/releases).
+
+The current fork adds the jupyter notebook on the attention analysis of the 12 layer BERT model. For details please refer to [the paper](https://arxiv.org/abs/1908.08593).
+
+Note that for the extraction of attention weights the source code of [](./pytorch_pretrained_bert/modeling.py) was modified (this functionality was added in later realeases of the forked repo).
+
+# Requirements and usage
+1. Install the requirements as
+```
+pip install -r requirements.txt
+```
+
+2. The current implementation assumes you have GLUE datasets downloaded and fine-tuned BERT model weights saved to a directory of your choice. You can download GLUE data as described [here](https://github.com/nyu-mll/GLUE-baselines/blob/master/download_glue_data.py). To fine-tune BERT, run [](./examples/run_classifier.py).
+
+3. The code for analysis is contained in [the jupyter notebook](./visualize_attention.ipynb).
+
+4. To repeat the results of experiments, make sure to change the `path_to_model` and `path_to_data` in the notebook.
+
+
+# References
+```
+@article{kovaleva2019revealing,
+ title={Revealing the Dark Secrets of BERT},
+ author={Kovaleva, Olga and Romanov, Alexey and Rogers, Anna and Rumshisky, Anna},
+ journal={arXiv preprint arXiv:1908.08593},
+ year={2019}
+}
+```
+
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docker/Dockerfile b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docker/Dockerfile
new file mode 100644
index 0000000000000000000000000000000000000000..e47eb548f9a13fd5ec201d5ac4e99ba3066674d8
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docker/Dockerfile
@@ -0,0 +1,7 @@
+FROM pytorch/pytorch:latest
+
+RUN git clone https://github.com/NVIDIA/apex.git && cd apex && python setup.py install --cuda_ext --cpp_ext
+
+RUN pip install pytorch-pretrained-bert
+
+WORKDIR /workspace
\ No newline at end of file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_constant_schedule.png b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_constant_schedule.png
new file mode 100644
index 0000000000000000000000000000000000000000..e2448e9f2c7999497d3e2d252a5dcb22b0ac7da5
Binary files /dev/null and b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_constant_schedule.png differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_hard_restarts_schedule.png b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_hard_restarts_schedule.png
new file mode 100644
index 0000000000000000000000000000000000000000..be73605b9c080cdc7cea8b4ff7e29de90db2d9eb
Binary files /dev/null and b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_hard_restarts_schedule.png differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_schedule.png b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_schedule.png
new file mode 100644
index 0000000000000000000000000000000000000000..6d27926ab10e9d2649ce3f28eb9656ea7cd3e9f8
Binary files /dev/null and b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_schedule.png differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_warm_restarts_schedule.png b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_warm_restarts_schedule.png
new file mode 100644
index 0000000000000000000000000000000000000000..71b39bffd3daccf7fc89cad77ef8e03df40bf0ab
Binary files /dev/null and b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_cosine_warm_restarts_schedule.png differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_linear_schedule.png b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_linear_schedule.png
new file mode 100644
index 0000000000000000000000000000000000000000..4e1af31025fafbd9c6b7c74ad6c2948ca2d3ff77
Binary files /dev/null and b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/docs/imgs/warmup_linear_schedule.png differ
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/extract_features.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/extract_features.py
new file mode 100644
index 0000000000000000000000000000000000000000..13384a9d69e42cf8a3408020e65c2aef86e2673c
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/extract_features.py
@@ -0,0 +1,297 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Extract pre-computed feature vectors from a PyTorch BERT model."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import collections
+import logging
+import json
+import re
+
+import torch
+from torch.utils.data import TensorDataset, DataLoader, SequentialSampler
+from torch.utils.data.distributed import DistributedSampler
+
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+from pytorch_pretrained_bert.modeling import BertModel
+
+logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class InputExample(object):
+
+ def __init__(self, unique_id, text_a, text_b):
+ self.unique_id = unique_id
+ self.text_a = text_a
+ self.text_b = text_b
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self, unique_id, tokens, input_ids, input_mask, input_type_ids):
+ self.unique_id = unique_id
+ self.tokens = tokens
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.input_type_ids = input_type_ids
+
+
+def convert_examples_to_features(examples, seq_length, tokenizer):
+ """Loads a data file into a list of `InputFeature`s."""
+
+ features = []
+ for (ex_index, example) in enumerate(examples):
+ tokens_a = tokenizer.tokenize(example.text_a)
+
+ tokens_b = None
+ if example.text_b:
+ tokens_b = tokenizer.tokenize(example.text_b)
+
+ if tokens_b:
+ # Modifies `tokens_a` and `tokens_b` in place so that the total
+ # length is less than the specified length.
+ # Account for [CLS], [SEP], [SEP] with "- 3"
+ _truncate_seq_pair(tokens_a, tokens_b, seq_length - 3)
+ else:
+ # Account for [CLS] and [SEP] with "- 2"
+ if len(tokens_a) > seq_length - 2:
+ tokens_a = tokens_a[0:(seq_length - 2)]
+
+ # The convention in BERT is:
+ # (a) For sequence pairs:
+ # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+ # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
+ # (b) For single sequences:
+ # tokens: [CLS] the dog is hairy . [SEP]
+ # type_ids: 0 0 0 0 0 0 0
+ #
+ # Where "type_ids" are used to indicate whether this is the first
+ # sequence or the second sequence. The embedding vectors for `type=0` and
+ # `type=1` were learned during pre-training and are added to the wordpiece
+ # embedding vector (and position vector). This is not *strictly* necessary
+ # since the [SEP] token unambigiously separates the sequences, but it makes
+ # it easier for the model to learn the concept of sequences.
+ #
+ # For classification tasks, the first vector (corresponding to [CLS]) is
+ # used as as the "sentence vector". Note that this only makes sense because
+ # the entire model is fine-tuned.
+ tokens = []
+ input_type_ids = []
+ tokens.append("[CLS]")
+ input_type_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ input_type_ids.append(0)
+ tokens.append("[SEP]")
+ input_type_ids.append(0)
+
+ if tokens_b:
+ for token in tokens_b:
+ tokens.append(token)
+ input_type_ids.append(1)
+ tokens.append("[SEP]")
+ input_type_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ input_type_ids.append(0)
+
+ assert len(input_ids) == seq_length
+ assert len(input_mask) == seq_length
+ assert len(input_type_ids) == seq_length
+
+ if ex_index < 5:
+ logger.info("*** Example ***")
+ logger.info("unique_id: %s" % (example.unique_id))
+ logger.info("tokens: %s" % " ".join([str(x) for x in tokens]))
+ logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ logger.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ logger.info(
+ "input_type_ids: %s" % " ".join([str(x) for x in input_type_ids]))
+
+ features.append(
+ InputFeatures(
+ unique_id=example.unique_id,
+ tokens=tokens,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ input_type_ids=input_type_ids))
+ return features
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+
+def read_examples(input_file):
+ """Read a list of `InputExample`s from an input file."""
+ examples = []
+ unique_id = 0
+ with open(input_file, "r", encoding='utf-8') as reader:
+ while True:
+ line = reader.readline()
+ if not line:
+ break
+ line = line.strip()
+ text_a = None
+ text_b = None
+ m = re.match(r"^(.*) \|\|\| (.*)$", line)
+ if m is None:
+ text_a = line
+ else:
+ text_a = m.group(1)
+ text_b = m.group(2)
+ examples.append(
+ InputExample(unique_id=unique_id, text_a=text_a, text_b=text_b))
+ unique_id += 1
+ return examples
+
+
+def main():
+ parser = argparse.ArgumentParser()
+
+ ## Required parameters
+ parser.add_argument("--input_file", default=None, type=str, required=True)
+ parser.add_argument("--output_file", default=None, type=str, required=True)
+ parser.add_argument("--bert_model", default=None, type=str, required=True,
+ help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")
+
+ ## Other parameters
+ parser.add_argument("--do_lower_case", action='store_true', help="Set this flag if you are using an uncased model.")
+ parser.add_argument("--layers", default="-1,-2,-3,-4", type=str)
+ parser.add_argument("--max_seq_length", default=128, type=int,
+ help="The maximum total input sequence length after WordPiece tokenization. Sequences longer "
+ "than this will be truncated, and sequences shorter than this will be padded.")
+ parser.add_argument("--batch_size", default=32, type=int, help="Batch size for predictions.")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help = "local_rank for distributed training on gpus")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+
+ args = parser.parse_args()
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+ logger.info("device: {} n_gpu: {} distributed training: {}".format(device, n_gpu, bool(args.local_rank != -1)))
+
+ layer_indexes = [int(x) for x in args.layers.split(",")]
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ examples = read_examples(args.input_file)
+
+ features = convert_examples_to_features(
+ examples=examples, seq_length=args.max_seq_length, tokenizer=tokenizer)
+
+ unique_id_to_feature = {}
+ for feature in features:
+ unique_id_to_feature[feature.unique_id] = feature
+
+ model = BertModel.from_pretrained(args.bert_model)
+ model.to(device)
+
+ if args.local_rank != -1:
+ model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank],
+ output_device=args.local_rank)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)
+ all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)
+
+ eval_data = TensorDataset(all_input_ids, all_input_mask, all_example_index)
+ if args.local_rank == -1:
+ eval_sampler = SequentialSampler(eval_data)
+ else:
+ eval_sampler = DistributedSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.batch_size)
+
+ model.eval()
+ with open(args.output_file, "w", encoding='utf-8') as writer:
+ for input_ids, input_mask, example_indices in eval_dataloader:
+ input_ids = input_ids.to(device)
+ input_mask = input_mask.to(device)
+
+ all_encoder_layers, _ = model(input_ids, token_type_ids=None, attention_mask=input_mask)
+ all_encoder_layers = all_encoder_layers
+
+ for b, example_index in enumerate(example_indices):
+ feature = features[example_index.item()]
+ unique_id = int(feature.unique_id)
+ # feature = unique_id_to_feature[unique_id]
+ output_json = collections.OrderedDict()
+ output_json["linex_index"] = unique_id
+ all_out_features = []
+ for (i, token) in enumerate(feature.tokens):
+ all_layers = []
+ for (j, layer_index) in enumerate(layer_indexes):
+ layer_output = all_encoder_layers[int(layer_index)].detach().cpu().numpy()
+ layer_output = layer_output[b]
+ layers = collections.OrderedDict()
+ layers["index"] = layer_index
+ layers["values"] = [
+ round(x.item(), 6) for x in layer_output[i]
+ ]
+ all_layers.append(layers)
+ out_features = collections.OrderedDict()
+ out_features["token"] = token
+ out_features["layers"] = all_layers
+ all_out_features.append(out_features)
+ output_json["features"] = all_out_features
+ writer.write(json.dumps(output_json) + "\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/README.md b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..f04e877ef23677ba7ed162958699a35fba6f084a
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/README.md
@@ -0,0 +1,64 @@
+# BERT Model Finetuning using Masked Language Modeling objective
+
+## Introduction
+
+The three example scripts in this folder can be used to **fine-tune** a pre-trained BERT model using the pretraining objective (combination of masked language modeling and next sentence prediction loss). In general, pretrained models like BERT are first trained with a pretraining objective (masked language modeling and next sentence prediction for BERT) on a large and general natural language corpus. A classifier head is then added on top of the pre-trained architecture and the model is quickly fine-tuned on a target task, while still (hopefully) retaining its general language understanding. This greatly reduces overfitting and yields state-of-the-art results, especially when training data for the target task are limited.
+
+The [ULMFiT paper](https://arxiv.org/abs/1801.06146) took a slightly different approach, however, and added an intermediate step in which the model is fine-tuned on text **from the same domain as the target task and using the pretraining objective** before the final stage in which the classifier head is added and the model is trained on the target task itself. This paper reported significantly improved results from this step, and found that they could get high-quality classifications even with only tiny numbers (<1000) of labelled training examples, as long as they had a lot of unlabelled data from the target domain.
+
+The BERT model has more capacity than the LSTM models used in the ULMFiT work, but the [BERT paper](https://arxiv.org/abs/1810.04805) did not test finetuning using the pretraining objective and at the present stage there aren't many examples of this approach being used for Transformer-based language models. As such, it's hard to predict what effect this step will have on final model performance, but it's reasonable to conjecture that this approach can improve the final classification performance, especially when a large unlabelled corpus from the target domain is available, labelled data is limited, or the target domain is very unusual and different from 'normal' English text. If you are aware of any literature on this subject, please feel free to add it in here, or open an issue and tag me (@Rocketknight1) and I'll include it.
+
+## Input format
+
+The scripts in this folder expect a single file as input, consisting of untokenized text, with one **sentence** per line, and one blank line between documents. The reason for the sentence splitting is that part of BERT's training involves a _next sentence_ objective in which the model must predict whether two sequences of text are contiguous text from the same document or not, and to avoid making the task _too easy_, the split point between the sequences is always at the end of a sentence. The linebreaks in the file are therefore necessary to mark the points where the text can be split.
+
+## Usage
+
+There are two ways to fine-tune a language model using these scripts. The first _quick_ approach is to use [`simple_lm_finetuning.py`](./simple_lm_finetuning.py). This script does everything in a single script, but generates training instances that consist of just two sentences. This is quite different from the BERT paper, where (confusingly) the NextSentence task concatenated sentences together from each document to form two long multi-sentences, which the paper just referred to as _sentences_. The difference between this simple approach and the original paper approach can have a significant effect for long sequences since two sentences will be much shorter than the max sequence length. In this case, most of each training example will just consist of blank padding characters, which wastes a lot of computation and results in a model that isn't really training on long sequences.
+
+As such, the preferred approach (assuming you have documents containing multiple contiguous sentences from your target domain) is to use [`pregenerate_training_data.py`](./pregenerate_training_data.py) to pre-process your data into training examples following the methodology used for LM training in the original BERT paper and repository. Since there is a significant random component to training data generation for BERT, this script includes an option to generate multiple _epochs_ of pre-processed data, to avoid training on the same random splits each epoch. Generating an epoch of data for each training epoch should result a better final model, and so we recommend doing so.
+
+You can then train on the pregenerated data using [`finetune_on_pregenerated.py`](./finetune_on_pregenerated.py), and pointing it to the folder created by [`pregenerate_training_data.py`](./pregenerate_training_data.py). Note that you should use the same `bert_model` and case options for both! Also note that `max_seq_len` does not need to be specified for the [`finetune_on_pregenerated.py`](./finetune_on_pregenerated.py) script, as it is inferred from the training examples.
+
+There are various options that can be tweaked, but they are mostly set to the values from the BERT paper/repository and default values should make sense. The most relevant ones are:
+
+- `--max_seq_len`: Controls the length of training examples (in wordpiece tokens) seen by the model. Defaults to 128 but can be set as high as 512. Higher values may yield stronger language models at the cost of slower and more memory-intensive training.
+- `--fp16`: Enables fast half-precision training on recent GPUs.
+
+In addition, if memory usage is an issue, especially when training on a single GPU, reducing `--train_batch_size` from the default 32 to a lower number (4-16) can be helpful, or leaving `--train_batch_size` at the default and increasing `--gradient_accumulation_steps` to 2-8. Changing `--gradient_accumulation_steps` may be preferable as alterations to the batch size may require corresponding changes in the learning rate to compensate. There is also a `--reduce_memory` option for both the `pregenerate_training_data.py` and `finetune_on_pregenerated.py` scripts that spills data to disc in shelf objects or numpy memmaps rather than retaining it in memory, which significantly reduces memory usage with little performance impact.
+
+## Examples
+
+### Simple fine-tuning
+
+```
+python3 simple_lm_finetuning.py
+--train_corpus my_corpus.txt
+--bert_model bert-base-uncased
+--do_lower_case
+--output_dir finetuned_lm/
+--do_train
+```
+
+### Pregenerating training data
+
+```
+python3 pregenerate_training_data.py
+--train_corpus my_corpus.txt
+--bert_model bert-base-uncased
+--do_lower_case
+--output_dir training/
+--epochs_to_generate 3
+--max_seq_len 256
+```
+
+### Training on pregenerated data
+
+```
+python3 finetune_on_pregenerated.py
+--pregenerated_data training/
+--bert_model bert-base-uncased
+--do_lower_case
+--output_dir finetuned_lm/
+--epochs 3
+```
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/finetune_on_pregenerated.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/finetune_on_pregenerated.py
new file mode 100644
index 0000000000000000000000000000000000000000..6a633245027dc884cbf29d1b4a6a0c9f836f6a3c
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/finetune_on_pregenerated.py
@@ -0,0 +1,333 @@
+from argparse import ArgumentParser
+from pathlib import Path
+import torch
+import logging
+import json
+import random
+import numpy as np
+from collections import namedtuple
+from tempfile import TemporaryDirectory
+
+from torch.utils.data import DataLoader, Dataset, RandomSampler
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm
+
+from pytorch_pretrained_bert.modeling import BertForPreTraining
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
+
+InputFeatures = namedtuple("InputFeatures", "input_ids input_mask segment_ids lm_label_ids is_next")
+
+log_format = '%(asctime)-10s: %(message)s'
+logging.basicConfig(level=logging.INFO, format=log_format)
+
+
+def convert_example_to_features(example, tokenizer, max_seq_length):
+ tokens = example["tokens"]
+ segment_ids = example["segment_ids"]
+ is_random_next = example["is_random_next"]
+ masked_lm_positions = example["masked_lm_positions"]
+ masked_lm_labels = example["masked_lm_labels"]
+
+ assert len(tokens) == len(segment_ids) <= max_seq_length # The preprocessed data should be already truncated
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+ masked_label_ids = tokenizer.convert_tokens_to_ids(masked_lm_labels)
+
+ input_array = np.zeros(max_seq_length, dtype=np.int)
+ input_array[:len(input_ids)] = input_ids
+
+ mask_array = np.zeros(max_seq_length, dtype=np.bool)
+ mask_array[:len(input_ids)] = 1
+
+ segment_array = np.zeros(max_seq_length, dtype=np.bool)
+ segment_array[:len(segment_ids)] = segment_ids
+
+ lm_label_array = np.full(max_seq_length, dtype=np.int, fill_value=-1)
+ lm_label_array[masked_lm_positions] = masked_label_ids
+
+ features = InputFeatures(input_ids=input_array,
+ input_mask=mask_array,
+ segment_ids=segment_array,
+ lm_label_ids=lm_label_array,
+ is_next=is_random_next)
+ return features
+
+
+class PregeneratedDataset(Dataset):
+ def __init__(self, training_path, epoch, tokenizer, num_data_epochs, reduce_memory=False):
+ self.vocab = tokenizer.vocab
+ self.tokenizer = tokenizer
+ self.epoch = epoch
+ self.data_epoch = epoch % num_data_epochs
+ data_file = training_path / f"epoch_{self.data_epoch}.json"
+ metrics_file = training_path / f"epoch_{self.data_epoch}_metrics.json"
+ assert data_file.is_file() and metrics_file.is_file()
+ metrics = json.loads(metrics_file.read_text())
+ num_samples = metrics['num_training_examples']
+ seq_len = metrics['max_seq_len']
+ self.temp_dir = None
+ self.working_dir = None
+ if reduce_memory:
+ self.temp_dir = TemporaryDirectory()
+ self.working_dir = Path(self.temp_dir.name)
+ input_ids = np.memmap(filename=self.working_dir/'input_ids.memmap',
+ mode='w+', dtype=np.int32, shape=(num_samples, seq_len))
+ input_masks = np.memmap(filename=self.working_dir/'input_masks.memmap',
+ shape=(num_samples, seq_len), mode='w+', dtype=np.bool)
+ segment_ids = np.memmap(filename=self.working_dir/'input_masks.memmap',
+ shape=(num_samples, seq_len), mode='w+', dtype=np.bool)
+ lm_label_ids = np.memmap(filename=self.working_dir/'lm_label_ids.memmap',
+ shape=(num_samples, seq_len), mode='w+', dtype=np.int32)
+ lm_label_ids[:] = -1
+ is_nexts = np.memmap(filename=self.working_dir/'is_nexts.memmap',
+ shape=(num_samples,), mode='w+', dtype=np.bool)
+ else:
+ input_ids = np.zeros(shape=(num_samples, seq_len), dtype=np.int32)
+ input_masks = np.zeros(shape=(num_samples, seq_len), dtype=np.bool)
+ segment_ids = np.zeros(shape=(num_samples, seq_len), dtype=np.bool)
+ lm_label_ids = np.full(shape=(num_samples, seq_len), dtype=np.int32, fill_value=-1)
+ is_nexts = np.zeros(shape=(num_samples,), dtype=np.bool)
+ logging.info(f"Loading training examples for epoch {epoch}")
+ with data_file.open() as f:
+ for i, line in enumerate(tqdm(f, total=num_samples, desc="Training examples")):
+ line = line.strip()
+ example = json.loads(line)
+ features = convert_example_to_features(example, tokenizer, seq_len)
+ input_ids[i] = features.input_ids
+ segment_ids[i] = features.segment_ids
+ input_masks[i] = features.input_mask
+ lm_label_ids[i] = features.lm_label_ids
+ is_nexts[i] = features.is_next
+ assert i == num_samples - 1 # Assert that the sample count metric was true
+ logging.info("Loading complete!")
+ self.num_samples = num_samples
+ self.seq_len = seq_len
+ self.input_ids = input_ids
+ self.input_masks = input_masks
+ self.segment_ids = segment_ids
+ self.lm_label_ids = lm_label_ids
+ self.is_nexts = is_nexts
+
+ def __len__(self):
+ return self.num_samples
+
+ def __getitem__(self, item):
+ return (torch.tensor(self.input_ids[item].astype(np.int64)),
+ torch.tensor(self.input_masks[item].astype(np.int64)),
+ torch.tensor(self.segment_ids[item].astype(np.int64)),
+ torch.tensor(self.lm_label_ids[item].astype(np.int64)),
+ torch.tensor(self.is_nexts[item].astype(np.int64)))
+
+
+def main():
+ parser = ArgumentParser()
+ parser.add_argument('--pregenerated_data', type=Path, required=True)
+ parser.add_argument('--output_dir', type=Path, required=True)
+ parser.add_argument("--bert_model", type=str, required=True, help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")
+ parser.add_argument("--do_lower_case", action="store_true")
+ parser.add_argument("--reduce_memory", action="store_true",
+ help="Store training data as on-disc memmaps to massively reduce memory usage")
+
+ parser.add_argument("--epochs", type=int, default=3, help="Number of epochs to train for")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help="local_rank for distributed training on gpus")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+ parser.add_argument('--gradient_accumulation_steps',
+ type=int,
+ default=1,
+ help="Number of updates steps to accumulate before performing a backward/update pass.")
+ parser.add_argument("--train_batch_size",
+ default=32,
+ type=int,
+ help="Total batch size for training.")
+ parser.add_argument('--fp16',
+ action='store_true',
+ help="Whether to use 16-bit float precision instead of 32-bit")
+ parser.add_argument('--loss_scale',
+ type=float, default=0,
+ help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
+ "0 (default value): dynamic loss scaling.\n"
+ "Positive power of 2: static loss scaling value.\n")
+ parser.add_argument("--warmup_proportion",
+ default=0.1,
+ type=float,
+ help="Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10%% of training.")
+ parser.add_argument("--learning_rate",
+ default=3e-5,
+ type=float,
+ help="The initial learning rate for Adam.")
+ parser.add_argument('--seed',
+ type=int,
+ default=42,
+ help="random seed for initialization")
+ args = parser.parse_args()
+
+ assert args.pregenerated_data.is_dir(), \
+ "--pregenerated_data should point to the folder of files made by pregenerate_training_data.py!"
+
+ samples_per_epoch = []
+ for i in range(args.epochs):
+ epoch_file = args.pregenerated_data / f"epoch_{i}.json"
+ metrics_file = args.pregenerated_data / f"epoch_{i}_metrics.json"
+ if epoch_file.is_file() and metrics_file.is_file():
+ metrics = json.loads(metrics_file.read_text())
+ samples_per_epoch.append(metrics['num_training_examples'])
+ else:
+ if i == 0:
+ exit("No training data was found!")
+ print(f"Warning! There are fewer epochs of pregenerated data ({i}) than training epochs ({args.epochs}).")
+ print("This script will loop over the available data, but training diversity may be negatively impacted.")
+ num_data_epochs = i
+ break
+ else:
+ num_data_epochs = args.epochs
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ torch.cuda.set_device(args.local_rank)
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+ logging.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
+ device, n_gpu, bool(args.local_rank != -1), args.fp16))
+
+ if args.gradient_accumulation_steps < 1:
+ raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+ args.gradient_accumulation_steps))
+
+ args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ if n_gpu > 0:
+ torch.cuda.manual_seed_all(args.seed)
+
+ if args.output_dir.is_dir() and list(args.output_dir.iterdir()):
+ logging.warning(f"Output directory ({args.output_dir}) already exists and is not empty!")
+ args.output_dir.mkdir(parents=True, exist_ok=True)
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ total_train_examples = 0
+ for i in range(args.epochs):
+ # The modulo takes into account the fact that we may loop over limited epochs of data
+ total_train_examples += samples_per_epoch[i % len(samples_per_epoch)]
+
+ num_train_optimization_steps = int(
+ total_train_examples / args.train_batch_size / args.gradient_accumulation_steps)
+ if args.local_rank != -1:
+ num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
+
+ # Prepare model
+ model = BertForPreTraining.from_pretrained(args.bert_model)
+ if args.fp16:
+ model.half()
+ model.to(device)
+ if args.local_rank != -1:
+ try:
+ from apex.parallel import DistributedDataParallel as DDP
+ except ImportError:
+ raise ImportError(
+ "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+ model = DDP(model)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
+ 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+
+ if args.fp16:
+ try:
+ from apex.optimizers import FP16_Optimizer
+ from apex.optimizers import FusedAdam
+ except ImportError:
+ raise ImportError(
+ "Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ optimizer = FusedAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ bias_correction=False,
+ max_grad_norm=1.0)
+ if args.loss_scale == 0:
+ optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
+ else:
+ optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)
+
+ else:
+ optimizer = BertAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ t_total=num_train_optimization_steps)
+
+ global_step = 0
+ logging.info("***** Running training *****")
+ logging.info(f" Num examples = {total_train_examples}")
+ logging.info(" Batch size = %d", args.train_batch_size)
+ logging.info(" Num steps = %d", num_train_optimization_steps)
+ model.train()
+ for epoch in range(args.epochs):
+ epoch_dataset = PregeneratedDataset(epoch=epoch, training_path=args.pregenerated_data, tokenizer=tokenizer,
+ num_data_epochs=num_data_epochs)
+ if args.local_rank == -1:
+ train_sampler = RandomSampler(epoch_dataset)
+ else:
+ train_sampler = DistributedSampler(epoch_dataset)
+ train_dataloader = DataLoader(epoch_dataset, sampler=train_sampler, batch_size=args.train_batch_size)
+ tr_loss = 0
+ nb_tr_examples, nb_tr_steps = 0, 0
+ with tqdm(total=len(train_dataloader), desc=f"Epoch {epoch}") as pbar:
+ for step, batch in enumerate(train_dataloader):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, input_mask, segment_ids, lm_label_ids, is_next = batch
+ loss = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
+ if n_gpu > 1:
+ loss = loss.mean() # mean() to average on multi-gpu.
+ if args.gradient_accumulation_steps > 1:
+ loss = loss / args.gradient_accumulation_steps
+ if args.fp16:
+ optimizer.backward(loss)
+ else:
+ loss.backward()
+ tr_loss += loss.item()
+ nb_tr_examples += input_ids.size(0)
+ nb_tr_steps += 1
+ pbar.update(1)
+ mean_loss = tr_loss * args.gradient_accumulation_steps / nb_tr_steps
+ pbar.set_postfix_str(f"Loss: {mean_loss:.5f}")
+ if (step + 1) % args.gradient_accumulation_steps == 0:
+ if args.fp16:
+ # modify learning rate with special warm up BERT uses
+ # if args.fp16 is False, BertAdam is used that handles this automatically
+ lr_this_step = args.learning_rate * warmup_linear(global_step/num_train_optimization_steps,
+ args.warmup_proportion)
+ for param_group in optimizer.param_groups:
+ param_group['lr'] = lr_this_step
+ optimizer.step()
+ optimizer.zero_grad()
+ global_step += 1
+
+ # Save a trained model
+ logging.info("** ** * Saving fine-tuned model ** ** * ")
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+ output_model_file = args.output_dir / "pytorch_model.bin"
+ torch.save(model_to_save.state_dict(), str(output_model_file))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/pregenerate_training_data.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/pregenerate_training_data.py
new file mode 100644
index 0000000000000000000000000000000000000000..e6c3598a9fecac7f221d4bbff09076218f006b8d
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/pregenerate_training_data.py
@@ -0,0 +1,302 @@
+from argparse import ArgumentParser
+from pathlib import Path
+from tqdm import tqdm, trange
+from tempfile import TemporaryDirectory
+import shelve
+
+from random import random, randrange, randint, shuffle, choice, sample
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+import numpy as np
+import json
+
+
+class DocumentDatabase:
+ def __init__(self, reduce_memory=False):
+ if reduce_memory:
+ self.temp_dir = TemporaryDirectory()
+ self.working_dir = Path(self.temp_dir.name)
+ self.document_shelf_filepath = self.working_dir / 'shelf.db'
+ self.document_shelf = shelve.open(str(self.document_shelf_filepath),
+ flag='n', protocol=-1)
+ self.documents = None
+ else:
+ self.documents = []
+ self.document_shelf = None
+ self.document_shelf_filepath = None
+ self.temp_dir = None
+ self.doc_lengths = []
+ self.doc_cumsum = None
+ self.cumsum_max = None
+ self.reduce_memory = reduce_memory
+
+ def add_document(self, document):
+ if not document:
+ return
+ if self.reduce_memory:
+ current_idx = len(self.doc_lengths)
+ self.document_shelf[str(current_idx)] = document
+ else:
+ self.documents.append(document)
+ self.doc_lengths.append(len(document))
+
+ def _precalculate_doc_weights(self):
+ self.doc_cumsum = np.cumsum(self.doc_lengths)
+ self.cumsum_max = self.doc_cumsum[-1]
+
+ def sample_doc(self, current_idx, sentence_weighted=True):
+ # Uses the current iteration counter to ensure we don't sample the same doc twice
+ if sentence_weighted:
+ # With sentence weighting, we sample docs proportionally to their sentence length
+ if self.doc_cumsum is None or len(self.doc_cumsum) != len(self.doc_lengths):
+ self._precalculate_doc_weights()
+ rand_start = self.doc_cumsum[current_idx]
+ rand_end = rand_start + self.cumsum_max - self.doc_lengths[current_idx]
+ sentence_index = randrange(rand_start, rand_end) % self.cumsum_max
+ sampled_doc_index = np.searchsorted(self.doc_cumsum, sentence_index, side='right')
+ else:
+ # If we don't use sentence weighting, then every doc has an equal chance to be chosen
+ sampled_doc_index = (current_idx + randrange(1, len(self.doc_lengths))) % len(self.doc_lengths)
+ assert sampled_doc_index != current_idx
+ if self.reduce_memory:
+ return self.document_shelf[str(sampled_doc_index)]
+ else:
+ return self.documents[sampled_doc_index]
+
+ def __len__(self):
+ return len(self.doc_lengths)
+
+ def __getitem__(self, item):
+ if self.reduce_memory:
+ return self.document_shelf[str(item)]
+ else:
+ return self.documents[item]
+
+ def __enter__(self):
+ return self
+
+ def __exit__(self, exc_type, exc_val, traceback):
+ if self.document_shelf is not None:
+ self.document_shelf.close()
+ if self.temp_dir is not None:
+ self.temp_dir.cleanup()
+
+
+def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens):
+ """Truncates a pair of sequences to a maximum sequence length. Lifted from Google's BERT repo."""
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_num_tokens:
+ break
+
+ trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b
+ assert len(trunc_tokens) >= 1
+
+ # We want to sometimes truncate from the front and sometimes from the
+ # back to add more randomness and avoid biases.
+ if random() < 0.5:
+ del trunc_tokens[0]
+ else:
+ trunc_tokens.pop()
+
+
+def create_masked_lm_predictions(tokens, masked_lm_prob, max_predictions_per_seq, vocab_list):
+ """Creates the predictions for the masked LM objective. This is mostly copied from the Google BERT repo, but
+ with several refactors to clean it up and remove a lot of unnecessary variables."""
+ cand_indices = []
+ for (i, token) in enumerate(tokens):
+ if token == "[CLS]" or token == "[SEP]":
+ continue
+ cand_indices.append(i)
+
+ num_to_mask = min(max_predictions_per_seq,
+ max(1, int(round(len(tokens) * masked_lm_prob))))
+ shuffle(cand_indices)
+ mask_indices = sorted(sample(cand_indices, num_to_mask))
+ masked_token_labels = []
+ for index in mask_indices:
+ # 80% of the time, replace with [MASK]
+ if random() < 0.8:
+ masked_token = "[MASK]"
+ else:
+ # 10% of the time, keep original
+ if random() < 0.5:
+ masked_token = tokens[index]
+ # 10% of the time, replace with random word
+ else:
+ masked_token = choice(vocab_list)
+ masked_token_labels.append(tokens[index])
+ # Once we've saved the true label for that token, we can overwrite it with the masked version
+ tokens[index] = masked_token
+
+ return tokens, mask_indices, masked_token_labels
+
+
+def create_instances_from_document(
+ doc_database, doc_idx, max_seq_length, short_seq_prob,
+ masked_lm_prob, max_predictions_per_seq, vocab_list):
+ """This code is mostly a duplicate of the equivalent function from Google BERT's repo.
+ However, we make some changes and improvements. Sampling is improved and no longer requires a loop in this function.
+ Also, documents are sampled proportionally to the number of sentences they contain, which means each sentence
+ (rather than each document) has an equal chance of being sampled as a false example for the NextSentence task."""
+ document = doc_database[doc_idx]
+ # Account for [CLS], [SEP], [SEP]
+ max_num_tokens = max_seq_length - 3
+
+ # We *usually* want to fill up the entire sequence since we are padding
+ # to `max_seq_length` anyways, so short sequences are generally wasted
+ # computation. However, we *sometimes*
+ # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
+ # sequences to minimize the mismatch between pre-training and fine-tuning.
+ # The `target_seq_length` is just a rough target however, whereas
+ # `max_seq_length` is a hard limit.
+ target_seq_length = max_num_tokens
+ if random() < short_seq_prob:
+ target_seq_length = randint(2, max_num_tokens)
+
+ # We DON'T just concatenate all of the tokens from a document into a long
+ # sequence and choose an arbitrary split point because this would make the
+ # next sentence prediction task too easy. Instead, we split the input into
+ # segments "A" and "B" based on the actual "sentences" provided by the user
+ # input.
+ instances = []
+ current_chunk = []
+ current_length = 0
+ i = 0
+ while i < len(document):
+ segment = document[i]
+ current_chunk.append(segment)
+ current_length += len(segment)
+ if i == len(document) - 1 or current_length >= target_seq_length:
+ if current_chunk:
+ # `a_end` is how many segments from `current_chunk` go into the `A`
+ # (first) sentence.
+ a_end = 1
+ if len(current_chunk) >= 2:
+ a_end = randrange(1, len(current_chunk))
+
+ tokens_a = []
+ for j in range(a_end):
+ tokens_a.extend(current_chunk[j])
+
+ tokens_b = []
+
+ # Random next
+ if len(current_chunk) == 1 or random() < 0.5:
+ is_random_next = True
+ target_b_length = target_seq_length - len(tokens_a)
+
+ # Sample a random document, with longer docs being sampled more frequently
+ random_document = doc_database.sample_doc(current_idx=doc_idx, sentence_weighted=True)
+
+ random_start = randrange(0, len(random_document))
+ for j in range(random_start, len(random_document)):
+ tokens_b.extend(random_document[j])
+ if len(tokens_b) >= target_b_length:
+ break
+ # We didn't actually use these segments so we "put them back" so
+ # they don't go to waste.
+ num_unused_segments = len(current_chunk) - a_end
+ i -= num_unused_segments
+ # Actual next
+ else:
+ is_random_next = False
+ for j in range(a_end, len(current_chunk)):
+ tokens_b.extend(current_chunk[j])
+ truncate_seq_pair(tokens_a, tokens_b, max_num_tokens)
+
+ assert len(tokens_a) >= 1
+ assert len(tokens_b) >= 1
+
+ tokens = ["[CLS]"] + tokens_a + ["[SEP]"] + tokens_b + ["[SEP]"]
+ # The segment IDs are 0 for the [CLS] token, the A tokens and the first [SEP]
+ # They are 1 for the B tokens and the final [SEP]
+ segment_ids = [0 for _ in range(len(tokens_a) + 2)] + [1 for _ in range(len(tokens_b) + 1)]
+
+ tokens, masked_lm_positions, masked_lm_labels = create_masked_lm_predictions(
+ tokens, masked_lm_prob, max_predictions_per_seq, vocab_list)
+
+ instance = {
+ "tokens": tokens,
+ "segment_ids": segment_ids,
+ "is_random_next": is_random_next,
+ "masked_lm_positions": masked_lm_positions,
+ "masked_lm_labels": masked_lm_labels}
+ instances.append(instance)
+ current_chunk = []
+ current_length = 0
+ i += 1
+
+ return instances
+
+
+def main():
+ parser = ArgumentParser()
+ parser.add_argument('--train_corpus', type=Path, required=True)
+ parser.add_argument("--output_dir", type=Path, required=True)
+ parser.add_argument("--bert_model", type=str, required=True,
+ choices=["bert-base-uncased", "bert-large-uncased", "bert-base-cased",
+ "bert-base-multilingual", "bert-base-chinese"])
+ parser.add_argument("--do_lower_case", action="store_true")
+
+ parser.add_argument("--reduce_memory", action="store_true",
+ help="Reduce memory usage for large datasets by keeping data on disc rather than in memory")
+
+ parser.add_argument("--epochs_to_generate", type=int, default=3,
+ help="Number of epochs of data to pregenerate")
+ parser.add_argument("--max_seq_len", type=int, default=128)
+ parser.add_argument("--short_seq_prob", type=float, default=0.1,
+ help="Probability of making a short sentence as a training example")
+ parser.add_argument("--masked_lm_prob", type=float, default=0.15,
+ help="Probability of masking each token for the LM task")
+ parser.add_argument("--max_predictions_per_seq", type=int, default=20,
+ help="Maximum number of tokens to mask in each sequence")
+
+ args = parser.parse_args()
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+ vocab_list = list(tokenizer.vocab.keys())
+ with DocumentDatabase(reduce_memory=args.reduce_memory) as docs:
+ with args.train_corpus.open() as f:
+ doc = []
+ for line in tqdm(f, desc="Loading Dataset", unit=" lines"):
+ line = line.strip()
+ if line == "":
+ docs.add_document(doc)
+ doc = []
+ else:
+ tokens = tokenizer.tokenize(line)
+ doc.append(tokens)
+ if doc:
+ docs.add_document(doc) # If the last doc didn't end on a newline, make sure it still gets added
+ if len(docs) <= 1:
+ exit("ERROR: No document breaks were found in the input file! These are necessary to allow the script to "
+ "ensure that random NextSentences are not sampled from the same document. Please add blank lines to "
+ "indicate breaks between documents in your input file. If your dataset does not contain multiple "
+ "documents, blank lines can be inserted at any natural boundary, such as the ends of chapters, "
+ "sections or paragraphs.")
+
+ args.output_dir.mkdir(exist_ok=True)
+ for epoch in trange(args.epochs_to_generate, desc="Epoch"):
+ epoch_filename = args.output_dir / f"epoch_{epoch}.json"
+ num_instances = 0
+ with epoch_filename.open('w') as epoch_file:
+ for doc_idx in trange(len(docs), desc="Document"):
+ doc_instances = create_instances_from_document(
+ docs, doc_idx, max_seq_length=args.max_seq_len, short_seq_prob=args.short_seq_prob,
+ masked_lm_prob=args.masked_lm_prob, max_predictions_per_seq=args.max_predictions_per_seq,
+ vocab_list=vocab_list)
+ doc_instances = [json.dumps(instance) for instance in doc_instances]
+ for instance in doc_instances:
+ epoch_file.write(instance + '\n')
+ num_instances += 1
+ metrics_file = args.output_dir / f"epoch_{epoch}_metrics.json"
+ with metrics_file.open('w') as metrics_file:
+ metrics = {
+ "num_training_examples": num_instances,
+ "max_seq_len": args.max_seq_len
+ }
+ metrics_file.write(json.dumps(metrics))
+
+
+if __name__ == '__main__':
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/simple_lm_finetuning.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/simple_lm_finetuning.py
new file mode 100644
index 0000000000000000000000000000000000000000..0f8547333082faae72755be1ae59047ea4d8a4c5
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/lm_finetuning/simple_lm_finetuning.py
@@ -0,0 +1,642 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner."""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+import logging
+import os
+import random
+from io import open
+
+import numpy as np
+import torch
+from torch.utils.data import DataLoader, Dataset, RandomSampler
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+from pytorch_pretrained_bert.modeling import BertForPreTraining
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
+
+logging.basicConfig(format='%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt='%m/%d/%Y %H:%M:%S',
+ level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class BERTDataset(Dataset):
+ def __init__(self, corpus_path, tokenizer, seq_len, encoding="utf-8", corpus_lines=None, on_memory=True):
+ self.vocab = tokenizer.vocab
+ self.tokenizer = tokenizer
+ self.seq_len = seq_len
+ self.on_memory = on_memory
+ self.corpus_lines = corpus_lines # number of non-empty lines in input corpus
+ self.corpus_path = corpus_path
+ self.encoding = encoding
+ self.current_doc = 0 # to avoid random sentence from same doc
+
+ # for loading samples directly from file
+ self.sample_counter = 0 # used to keep track of full epochs on file
+ self.line_buffer = None # keep second sentence of a pair in memory and use as first sentence in next pair
+
+ # for loading samples in memory
+ self.current_random_doc = 0
+ self.num_docs = 0
+ self.sample_to_doc = [] # map sample index to doc and line
+
+ # load samples into memory
+ if on_memory:
+ self.all_docs = []
+ doc = []
+ self.corpus_lines = 0
+ with open(corpus_path, "r", encoding=encoding) as f:
+ for line in tqdm(f, desc="Loading Dataset", total=corpus_lines):
+ line = line.strip()
+ if line == "":
+ self.all_docs.append(doc)
+ doc = []
+ #remove last added sample because there won't be a subsequent line anymore in the doc
+ self.sample_to_doc.pop()
+ else:
+ #store as one sample
+ sample = {"doc_id": len(self.all_docs),
+ "line": len(doc)}
+ self.sample_to_doc.append(sample)
+ doc.append(line)
+ self.corpus_lines = self.corpus_lines + 1
+
+ # if last row in file is not empty
+ if self.all_docs[-1] != doc:
+ self.all_docs.append(doc)
+ self.sample_to_doc.pop()
+
+ self.num_docs = len(self.all_docs)
+
+ # load samples later lazily from disk
+ else:
+ if self.corpus_lines is None:
+ with open(corpus_path, "r", encoding=encoding) as f:
+ self.corpus_lines = 0
+ for line in tqdm(f, desc="Loading Dataset", total=corpus_lines):
+ if line.strip() == "":
+ self.num_docs += 1
+ else:
+ self.corpus_lines += 1
+
+ # if doc does not end with empty line
+ if line.strip() != "":
+ self.num_docs += 1
+
+ self.file = open(corpus_path, "r", encoding=encoding)
+ self.random_file = open(corpus_path, "r", encoding=encoding)
+
+ def __len__(self):
+ # last line of doc won't be used, because there's no "nextSentence". Additionally, we start counting at 0.
+ return self.corpus_lines - self.num_docs - 1
+
+ def __getitem__(self, item):
+ cur_id = self.sample_counter
+ self.sample_counter += 1
+ if not self.on_memory:
+ # after one epoch we start again from beginning of file
+ if cur_id != 0 and (cur_id % len(self) == 0):
+ self.file.close()
+ self.file = open(self.corpus_path, "r", encoding=self.encoding)
+
+ t1, t2, is_next_label = self.random_sent(item)
+
+ # tokenize
+ tokens_a = self.tokenizer.tokenize(t1)
+ tokens_b = self.tokenizer.tokenize(t2)
+
+ # combine to one sample
+ cur_example = InputExample(guid=cur_id, tokens_a=tokens_a, tokens_b=tokens_b, is_next=is_next_label)
+
+ # transform sample to features
+ cur_features = convert_example_to_features(cur_example, self.seq_len, self.tokenizer)
+
+ cur_tensors = (torch.tensor(cur_features.input_ids),
+ torch.tensor(cur_features.input_mask),
+ torch.tensor(cur_features.segment_ids),
+ torch.tensor(cur_features.lm_label_ids),
+ torch.tensor(cur_features.is_next))
+
+ return cur_tensors
+
+ def random_sent(self, index):
+ """
+ Get one sample from corpus consisting of two sentences. With prob. 50% these are two subsequent sentences
+ from one doc. With 50% the second sentence will be a random one from another doc.
+ :param index: int, index of sample.
+ :return: (str, str, int), sentence 1, sentence 2, isNextSentence Label
+ """
+ t1, t2 = self.get_corpus_line(index)
+ if random.random() > 0.5:
+ label = 0
+ else:
+ t2 = self.get_random_line()
+ label = 1
+
+ assert len(t1) > 0
+ assert len(t2) > 0
+ return t1, t2, label
+
+ def get_corpus_line(self, item):
+ """
+ Get one sample from corpus consisting of a pair of two subsequent lines from the same doc.
+ :param item: int, index of sample.
+ :return: (str, str), two subsequent sentences from corpus
+ """
+ t1 = ""
+ t2 = ""
+ assert item < self.corpus_lines
+ if self.on_memory:
+ sample = self.sample_to_doc[item]
+ t1 = self.all_docs[sample["doc_id"]][sample["line"]]
+ t2 = self.all_docs[sample["doc_id"]][sample["line"]+1]
+ # used later to avoid random nextSentence from same doc
+ self.current_doc = sample["doc_id"]
+ return t1, t2
+ else:
+ if self.line_buffer is None:
+ # read first non-empty line of file
+ while t1 == "" :
+ t1 = next(self.file).strip()
+ t2 = next(self.file).strip()
+ else:
+ # use t2 from previous iteration as new t1
+ t1 = self.line_buffer
+ t2 = next(self.file).strip()
+ # skip empty rows that are used for separating documents and keep track of current doc id
+ while t2 == "" or t1 == "":
+ t1 = next(self.file).strip()
+ t2 = next(self.file).strip()
+ self.current_doc = self.current_doc+1
+ self.line_buffer = t2
+
+ assert t1 != ""
+ assert t2 != ""
+ return t1, t2
+
+ def get_random_line(self):
+ """
+ Get random line from another document for nextSentence task.
+ :return: str, content of one line
+ """
+ # Similar to original tf repo: This outer loop should rarely go for more than one iteration for large
+ # corpora. However, just to be careful, we try to make sure that
+ # the random document is not the same as the document we're processing.
+ for _ in range(10):
+ if self.on_memory:
+ rand_doc_idx = random.randint(0, len(self.all_docs)-1)
+ rand_doc = self.all_docs[rand_doc_idx]
+ line = rand_doc[random.randrange(len(rand_doc))]
+ else:
+ rand_index = random.randint(1, self.corpus_lines if self.corpus_lines < 1000 else 1000)
+ #pick random line
+ for _ in range(rand_index):
+ line = self.get_next_line()
+ #check if our picked random line is really from another doc like we want it to be
+ if self.current_random_doc != self.current_doc:
+ break
+ return line
+
+ def get_next_line(self):
+ """ Gets next line of random_file and starts over when reaching end of file"""
+ try:
+ line = next(self.random_file).strip()
+ #keep track of which document we are currently looking at to later avoid having the same doc as t1
+ if line == "":
+ self.current_random_doc = self.current_random_doc + 1
+ line = next(self.random_file).strip()
+ except StopIteration:
+ self.random_file.close()
+ self.random_file = open(self.corpus_path, "r", encoding=self.encoding)
+ line = next(self.random_file).strip()
+ return line
+
+
+class InputExample(object):
+ """A single training/test example for the language model."""
+
+ def __init__(self, guid, tokens_a, tokens_b=None, is_next=None, lm_labels=None):
+ """Constructs a InputExample.
+
+ Args:
+ guid: Unique id for the example.
+ tokens_a: string. The untokenized text of the first sequence. For single
+ sequence tasks, only this sequence must be specified.
+ tokens_b: (Optional) string. The untokenized text of the second sequence.
+ Only must be specified for sequence pair tasks.
+ label: (Optional) string. The label of the example. This should be
+ specified for train and dev examples, but not for test examples.
+ """
+ self.guid = guid
+ self.tokens_a = tokens_a
+ self.tokens_b = tokens_b
+ self.is_next = is_next # nextSentence
+ self.lm_labels = lm_labels # masked words for language model
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self, input_ids, input_mask, segment_ids, is_next, lm_label_ids):
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.segment_ids = segment_ids
+ self.is_next = is_next
+ self.lm_label_ids = lm_label_ids
+
+
+def random_word(tokens, tokenizer):
+ """
+ Masking some random tokens for Language Model task with probabilities as in the original BERT paper.
+ :param tokens: list of str, tokenized sentence.
+ :param tokenizer: Tokenizer, object used for tokenization (we need it's vocab here)
+ :return: (list of str, list of int), masked tokens and related labels for LM prediction
+ """
+ output_label = []
+
+ for i, token in enumerate(tokens):
+ prob = random.random()
+ # mask token with 15% probability
+ if prob < 0.15:
+ prob /= 0.15
+
+ # 80% randomly change token to mask token
+ if prob < 0.8:
+ tokens[i] = "[MASK]"
+
+ # 10% randomly change token to random token
+ elif prob < 0.9:
+ tokens[i] = random.choice(list(tokenizer.vocab.items()))[0]
+
+ # -> rest 10% randomly keep current token
+
+ # append current token to output (we will predict these later)
+ try:
+ output_label.append(tokenizer.vocab[token])
+ except KeyError:
+ # For unknown words (should not occur with BPE vocab)
+ output_label.append(tokenizer.vocab["[UNK]"])
+ logger.warning("Cannot find token '{}' in vocab. Using [UNK] insetad".format(token))
+ else:
+ # no masking token (will be ignored by loss function later)
+ output_label.append(-1)
+
+ return tokens, output_label
+
+
+def convert_example_to_features(example, max_seq_length, tokenizer):
+ """
+ Convert a raw sample (pair of sentences as tokenized strings) into a proper training sample with
+ IDs, LM labels, input_mask, CLS and SEP tokens etc.
+ :param example: InputExample, containing sentence input as strings and is_next label
+ :param max_seq_length: int, maximum length of sequence.
+ :param tokenizer: Tokenizer
+ :return: InputFeatures, containing all inputs and labels of one sample as IDs (as used for model training)
+ """
+ tokens_a = example.tokens_a
+ tokens_b = example.tokens_b
+ # Modifies `tokens_a` and `tokens_b` in place so that the total
+ # length is less than the specified length.
+ # Account for [CLS], [SEP], [SEP] with "- 3"
+ _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
+
+ tokens_a, t1_label = random_word(tokens_a, tokenizer)
+ tokens_b, t2_label = random_word(tokens_b, tokenizer)
+ # concatenate lm labels and account for CLS, SEP, SEP
+ lm_label_ids = ([-1] + t1_label + [-1] + t2_label + [-1])
+
+ # The convention in BERT is:
+ # (a) For sequence pairs:
+ # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+ # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
+ # (b) For single sequences:
+ # tokens: [CLS] the dog is hairy . [SEP]
+ # type_ids: 0 0 0 0 0 0 0
+ #
+ # Where "type_ids" are used to indicate whether this is the first
+ # sequence or the second sequence. The embedding vectors for `type=0` and
+ # `type=1` were learned during pre-training and are added to the wordpiece
+ # embedding vector (and position vector). This is not *strictly* necessary
+ # since the [SEP] token unambigiously separates the sequences, but it makes
+ # it easier for the model to learn the concept of sequences.
+ #
+ # For classification tasks, the first vector (corresponding to [CLS]) is
+ # used as as the "sentence vector". Note that this only makes sense because
+ # the entire model is fine-tuned.
+ tokens = []
+ segment_ids = []
+ tokens.append("[CLS]")
+ segment_ids.append(0)
+ for token in tokens_a:
+ tokens.append(token)
+ segment_ids.append(0)
+ tokens.append("[SEP]")
+ segment_ids.append(0)
+
+ assert len(tokens_b) > 0
+ for token in tokens_b:
+ tokens.append(token)
+ segment_ids.append(1)
+ tokens.append("[SEP]")
+ segment_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < max_seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ segment_ids.append(0)
+ lm_label_ids.append(-1)
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+ assert len(lm_label_ids) == max_seq_length
+
+ if example.guid < 5:
+ logger.info("*** Example ***")
+ logger.info("guid: %s" % (example.guid))
+ logger.info("tokens: %s" % " ".join(
+ [str(x) for x in tokens]))
+ logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ logger.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ logger.info(
+ "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+ logger.info("LM label: %s " % (lm_label_ids))
+ logger.info("Is next sentence label: %s " % (example.is_next))
+
+ features = InputFeatures(input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ lm_label_ids=lm_label_ids,
+ is_next=example.is_next)
+ return features
+
+
+def main():
+ parser = argparse.ArgumentParser()
+
+ ## Required parameters
+ parser.add_argument("--train_corpus",
+ default=None,
+ type=str,
+ required=True,
+ help="The input train corpus.")
+ parser.add_argument("--bert_model", default=None, type=str, required=True,
+ help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-base-multilingual, bert-base-chinese.")
+ parser.add_argument("--output_dir",
+ default=None,
+ type=str,
+ required=True,
+ help="The output directory where the model checkpoints will be written.")
+
+ ## Other parameters
+ parser.add_argument("--max_seq_length",
+ default=128,
+ type=int,
+ help="The maximum total input sequence length after WordPiece tokenization. \n"
+ "Sequences longer than this will be truncated, and sequences shorter \n"
+ "than this will be padded.")
+ parser.add_argument("--do_train",
+ action='store_true',
+ help="Whether to run training.")
+ parser.add_argument("--train_batch_size",
+ default=32,
+ type=int,
+ help="Total batch size for training.")
+ parser.add_argument("--learning_rate",
+ default=3e-5,
+ type=float,
+ help="The initial learning rate for Adam.")
+ parser.add_argument("--num_train_epochs",
+ default=3.0,
+ type=float,
+ help="Total number of training epochs to perform.")
+ parser.add_argument("--warmup_proportion",
+ default=0.1,
+ type=float,
+ help="Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10%% of training.")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+ parser.add_argument("--on_memory",
+ action='store_true',
+ help="Whether to load train samples into memory or use disk")
+ parser.add_argument("--do_lower_case",
+ action='store_true',
+ help="Whether to lower case the input text. True for uncased models, False for cased models.")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help="local_rank for distributed training on gpus")
+ parser.add_argument('--seed',
+ type=int,
+ default=42,
+ help="random seed for initialization")
+ parser.add_argument('--gradient_accumulation_steps',
+ type=int,
+ default=1,
+ help="Number of updates steps to accumualte before performing a backward/update pass.")
+ parser.add_argument('--fp16',
+ action='store_true',
+ help="Whether to use 16-bit float precision instead of 32-bit")
+ parser.add_argument('--loss_scale',
+ type = float, default = 0,
+ help = "Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
+ "0 (default value): dynamic loss scaling.\n"
+ "Positive power of 2: static loss scaling value.\n")
+
+ args = parser.parse_args()
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ torch.cuda.set_device(args.local_rank)
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+ logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
+ device, n_gpu, bool(args.local_rank != -1), args.fp16))
+
+ if args.gradient_accumulation_steps < 1:
+ raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+ args.gradient_accumulation_steps))
+
+ args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ if n_gpu > 0:
+ torch.cuda.manual_seed_all(args.seed)
+
+ if not args.do_train:
+ raise ValueError("Training is currently the only implemented execution option. Please set `do_train`.")
+
+ if os.path.exists(args.output_dir) and os.listdir(args.output_dir):
+ raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir))
+ if not os.path.exists(args.output_dir):
+ os.makedirs(args.output_dir)
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ #train_examples = None
+ num_train_optimization_steps = None
+ if args.do_train:
+ print("Loading Train Dataset", args.train_corpus)
+ train_dataset = BERTDataset(args.train_corpus, tokenizer, seq_len=args.max_seq_length,
+ corpus_lines=None, on_memory=args.on_memory)
+ num_train_optimization_steps = int(
+ len(train_dataset) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs
+ if args.local_rank != -1:
+ num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
+
+ # Prepare model
+ model = BertForPreTraining.from_pretrained(args.bert_model)
+ if args.fp16:
+ model.half()
+ model.to(device)
+ if args.local_rank != -1:
+ try:
+ from apex.parallel import DistributedDataParallel as DDP
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+ model = DDP(model)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+
+ if args.fp16:
+ try:
+ from apex.optimizers import FP16_Optimizer
+ from apex.optimizers import FusedAdam
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ optimizer = FusedAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ bias_correction=False,
+ max_grad_norm=1.0)
+ if args.loss_scale == 0:
+ optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
+ else:
+ optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)
+
+ else:
+ optimizer = BertAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ t_total=num_train_optimization_steps)
+
+ global_step = 0
+ if args.do_train:
+ logger.info("***** Running training *****")
+ logger.info(" Num examples = %d", len(train_dataset))
+ logger.info(" Batch size = %d", args.train_batch_size)
+ logger.info(" Num steps = %d", num_train_optimization_steps)
+
+ if args.local_rank == -1:
+ train_sampler = RandomSampler(train_dataset)
+ else:
+ #TODO: check if this works with current data generator from disk that relies on next(file)
+ # (it doesn't return item back by index)
+ train_sampler = DistributedSampler(train_dataset)
+ train_dataloader = DataLoader(train_dataset, sampler=train_sampler, batch_size=args.train_batch_size)
+
+ model.train()
+ for _ in trange(int(args.num_train_epochs), desc="Epoch"):
+ tr_loss = 0
+ nb_tr_examples, nb_tr_steps = 0, 0
+ for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, input_mask, segment_ids, lm_label_ids, is_next = batch
+ loss = model(input_ids, segment_ids, input_mask, lm_label_ids, is_next)
+ if n_gpu > 1:
+ loss = loss.mean() # mean() to average on multi-gpu.
+ if args.gradient_accumulation_steps > 1:
+ loss = loss / args.gradient_accumulation_steps
+ if args.fp16:
+ optimizer.backward(loss)
+ else:
+ loss.backward()
+ tr_loss += loss.item()
+ nb_tr_examples += input_ids.size(0)
+ nb_tr_steps += 1
+ if (step + 1) % args.gradient_accumulation_steps == 0:
+ if args.fp16:
+ # modify learning rate with special warm up BERT uses
+ # if args.fp16 is False, BertAdam is used that handles this automatically
+ lr_this_step = args.learning_rate * warmup_linear(global_step/num_train_optimization_steps, args.warmup_proportion)
+ for param_group in optimizer.param_groups:
+ param_group['lr'] = lr_this_step
+ optimizer.step()
+ optimizer.zero_grad()
+ global_step += 1
+
+ # Save a trained model
+ logger.info("** ** * Saving fine - tuned model ** ** * ")
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+ output_model_file = os.path.join(args.output_dir, "pytorch_model.bin")
+ if args.do_train:
+ torch.save(model_to_save.state_dict(), output_model_file)
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+
+def accuracy(out, labels):
+ outputs = np.argmax(out, axis=1)
+ return np.sum(outputs == labels)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_classifier.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_classifier.py
new file mode 100644
index 0000000000000000000000000000000000000000..0365f31958db45cf0a312f94f40a212af1663bde
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_classifier.py
@@ -0,0 +1,1047 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner."""
+
+from __future__ import absolute_import, division, print_function
+
+import sys
+sys.path.insert(0, "/home/okovaleva/projects/bert_attention/pretrained_bert/pytorch-pretrained-BERT")
+print(sys.path)
+
+
+import argparse
+import csv
+import logging
+import os
+import random
+import sys
+
+import numpy as np
+import torch
+from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
+ TensorDataset)
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+from torch.nn import CrossEntropyLoss, MSELoss
+from scipy.stats import pearsonr, spearmanr
+from sklearn.metrics import matthews_corrcoef, f1_score
+
+from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
+from pytorch_pretrained_bert.modeling import BertForSequenceClassification, BertConfig
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
+
+logger = logging.getLogger(__name__)
+
+
+class InputExample(object):
+ """A single training/test example for simple sequence classification."""
+
+ def __init__(self, guid, text_a, text_b=None, label=None):
+ """Constructs a InputExample.
+
+ Args:
+ guid: Unique id for the example.
+ text_a: string. The untokenized text of the first sequence. For single
+ sequence tasks, only this sequence must be specified.
+ text_b: (Optional) string. The untokenized text of the second sequence.
+ Only must be specified for sequence pair tasks.
+ label: (Optional) string. The label of the example. This should be
+ specified for train and dev examples, but not for test examples.
+ """
+ self.guid = guid
+ self.text_a = text_a
+ self.text_b = text_b
+ self.label = label
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self, input_ids, input_mask, segment_ids, label_id):
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.segment_ids = segment_ids
+ self.label_id = label_id
+
+
+class DataProcessor(object):
+ """Base class for data converters for sequence classification data sets."""
+
+ def get_train_examples(self, data_dir):
+ """Gets a collection of `InputExample`s for the train set."""
+ raise NotImplementedError()
+
+ def get_dev_examples(self, data_dir):
+ """Gets a collection of `InputExample`s for the dev set."""
+ raise NotImplementedError()
+
+ def get_labels(self):
+ """Gets the list of labels for this data set."""
+ raise NotImplementedError()
+
+ @classmethod
+ def _read_tsv(cls, input_file, quotechar=None):
+ """Reads a tab separated value file."""
+ with open(input_file, "r", encoding="utf-8") as f:
+ reader = csv.reader(f, delimiter="\t", quotechar=quotechar)
+ lines = []
+ for line in reader:
+ if sys.version_info[0] == 2:
+ line = list(unicode(cell, 'utf-8') for cell in line)
+ lines.append(line)
+ return lines
+
+
+class MrpcProcessor(DataProcessor):
+ """Processor for the MRPC data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ logger.info("LOOKING AT {}".format(os.path.join(data_dir, "train.tsv")))
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, i)
+ text_a = line[3]
+ text_b = line[4]
+ label = line[0]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class MnliProcessor(DataProcessor):
+ """Processor for the MultiNLI data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev_matched.tsv")),
+ "dev_matched")
+
+ def get_labels(self):
+ """See base class."""
+ return ["contradiction", "entailment", "neutral"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ text_a = line[8]
+ text_b = line[9]
+ label = line[-1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class MnliMismatchedProcessor(MnliProcessor):
+ """Processor for the MultiNLI Mismatched data set (GLUE version)."""
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev_mismatched.tsv")),
+ "dev_matched")
+
+
+class ColaProcessor(DataProcessor):
+ """Processor for the CoLA data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ guid = "%s-%s" % (set_type, i)
+ text_a = line[3]
+ label = line[1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
+ return examples
+
+
+class Sst2Processor(DataProcessor):
+ """Processor for the SST-2 data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, i)
+ text_a = line[0]
+ label = line[1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
+ return examples
+
+
+class StsbProcessor(DataProcessor):
+ """Processor for the STS-B data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return [None]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ text_a = line[7]
+ text_b = line[8]
+ label = line[-1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class QqpProcessor(DataProcessor):
+ """Processor for the STS-B data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ try:
+ text_a = line[3]
+ text_b = line[4]
+ label = line[5]
+ except IndexError:
+ continue
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class QnliProcessor(DataProcessor):
+ """Processor for the STS-B data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")),
+ "dev_matched")
+
+ def get_labels(self):
+ """See base class."""
+ return ["entailment", "not_entailment"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ text_a = line[1]
+ text_b = line[2]
+ label = line[-1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class RteProcessor(DataProcessor):
+ """Processor for the RTE data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["entailment", "not_entailment"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ text_a = line[1]
+ text_b = line[2]
+ label = line[-1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+class WnliProcessor(DataProcessor):
+ """Processor for the WNLI data set (GLUE version)."""
+
+ def get_train_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
+
+ def get_dev_examples(self, data_dir):
+ """See base class."""
+ return self._create_examples(
+ self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")
+
+ def get_labels(self):
+ """See base class."""
+ return ["0", "1"]
+
+ def _create_examples(self, lines, set_type):
+ """Creates examples for the training and dev sets."""
+ examples = []
+ for (i, line) in enumerate(lines):
+ if i == 0:
+ continue
+ guid = "%s-%s" % (set_type, line[0])
+ text_a = line[1]
+ text_b = line[2]
+ label = line[-1]
+ examples.append(
+ InputExample(guid=guid, text_a=text_a, text_b=text_b, label=label))
+ return examples
+
+
+def convert_examples_to_features(examples, label_list, max_seq_length,
+ tokenizer, output_mode):
+ """Loads a data file into a list of `InputBatch`s."""
+
+ label_map = {label : i for i, label in enumerate(label_list)}
+
+ features = []
+ for (ex_index, example) in enumerate(examples):
+ if ex_index % 10000 == 0:
+ logger.info("Writing example %d of %d" % (ex_index, len(examples)))
+
+ tokens_a = tokenizer.tokenize(example.text_a)
+
+ tokens_b = None
+ if example.text_b:
+ tokens_b = tokenizer.tokenize(example.text_b)
+ # Modifies `tokens_a` and `tokens_b` in place so that the total
+ # length is less than the specified length.
+ # Account for [CLS], [SEP], [SEP] with "- 3"
+ _truncate_seq_pair(tokens_a, tokens_b, max_seq_length - 3)
+ else:
+ # Account for [CLS] and [SEP] with "- 2"
+ if len(tokens_a) > max_seq_length - 2:
+ tokens_a = tokens_a[:(max_seq_length - 2)]
+
+ # The convention in BERT is:
+ # (a) For sequence pairs:
+ # tokens: [CLS] is this jack ##son ##ville ? [SEP] no it is not . [SEP]
+ # type_ids: 0 0 0 0 0 0 0 0 1 1 1 1 1 1
+ # (b) For single sequences:
+ # tokens: [CLS] the dog is hairy . [SEP]
+ # type_ids: 0 0 0 0 0 0 0
+ #
+ # Where "type_ids" are used to indicate whether this is the first
+ # sequence or the second sequence. The embedding vectors for `type=0` and
+ # `type=1` were learned during pre-training and are added to the wordpiece
+ # embedding vector (and position vector). This is not *strictly* necessary
+ # since the [SEP] token unambiguously separates the sequences, but it makes
+ # it easier for the model to learn the concept of sequences.
+ #
+ # For classification tasks, the first vector (corresponding to [CLS]) is
+ # used as as the "sentence vector". Note that this only makes sense because
+ # the entire model is fine-tuned.
+ tokens = ["[CLS]"] + tokens_a + ["[SEP]"]
+ segment_ids = [0] * len(tokens)
+
+ if tokens_b:
+ tokens += tokens_b + ["[SEP]"]
+ segment_ids += [1] * (len(tokens_b) + 1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ padding = [0] * (max_seq_length - len(input_ids))
+ input_ids += padding
+ input_mask += padding
+ segment_ids += padding
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ if output_mode == "classification":
+ label_id = label_map[example.label]
+ elif output_mode == "regression":
+ label_id = float(example.label)
+ else:
+ raise KeyError(output_mode)
+
+ if ex_index < 5:
+ logger.info("*** Example ***")
+ logger.info("guid: %s" % (example.guid))
+ logger.info("tokens: %s" % " ".join(
+ [str(x) for x in tokens]))
+ logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ logger.info("input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ logger.info(
+ "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+ logger.info("label: %s (id = %d)" % (example.label, label_id))
+
+ features.append(
+ InputFeatures(input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ label_id=label_id))
+ return features
+
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+
+def simple_accuracy(preds, labels):
+ return (preds == labels).mean()
+
+
+def acc_and_f1(preds, labels):
+ acc = simple_accuracy(preds, labels)
+ f1 = f1_score(y_true=labels, y_pred=preds)
+ return {
+ "acc": acc,
+ "f1": f1,
+ "acc_and_f1": (acc + f1) / 2,
+ }
+
+
+def pearson_and_spearman(preds, labels):
+ pearson_corr = pearsonr(preds, labels)[0]
+ spearman_corr = spearmanr(preds, labels)[0]
+ return {
+ "pearson": pearson_corr,
+ "spearmanr": spearman_corr,
+ "corr": (pearson_corr + spearman_corr) / 2,
+ }
+
+
+def compute_metrics(task_name, preds, labels):
+ assert len(preds) == len(labels)
+ if task_name == "cola":
+ return {"mcc": matthews_corrcoef(labels, preds)}
+ elif task_name == "sst-2":
+ return {"acc": simple_accuracy(preds, labels)}
+ elif task_name == "mrpc":
+ return acc_and_f1(preds, labels)
+ elif task_name == "sts-b":
+ return pearson_and_spearman(preds, labels)
+ elif task_name == "qqp":
+ return acc_and_f1(preds, labels)
+ elif task_name == "mnli":
+ return {"acc": simple_accuracy(preds, labels)}
+ elif task_name == "mnli-mm":
+ return {"acc": simple_accuracy(preds, labels)}
+ elif task_name == "qnli":
+ return {"acc": simple_accuracy(preds, labels)}
+ elif task_name == "rte":
+ return {"acc": simple_accuracy(preds, labels)}
+ elif task_name == "wnli":
+ return {"acc": simple_accuracy(preds, labels)}
+ else:
+ raise KeyError(task_name)
+
+
+def main():
+ parser = argparse.ArgumentParser()
+
+ ## Required parameters
+ parser.add_argument("--data_dir",
+ default=None,
+ type=str,
+ required=True,
+ help="The input data dir. Should contain the .tsv files (or other data files) for the task.")
+ parser.add_argument("--bert_model", default=None, type=str, required=True,
+ help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, "
+ "bert-base-multilingual-cased, bert-base-chinese.")
+ parser.add_argument("--task_name",
+ default=None,
+ type=str,
+ required=True,
+ help="The name of the task to train.")
+ parser.add_argument("--output_dir",
+ default=None,
+ type=str,
+ required=True,
+ help="The output directory where the model predictions and checkpoints will be written.")
+
+ ## Other parameters
+ parser.add_argument("--cache_dir",
+ default="",
+ type=str,
+ help="Where do you want to store the pre-trained models downloaded from s3")
+ parser.add_argument("--max_seq_length",
+ default=128,
+ type=int,
+ help="The maximum total input sequence length after WordPiece tokenization. \n"
+ "Sequences longer than this will be truncated, and sequences shorter \n"
+ "than this will be padded.")
+ parser.add_argument("--do_train",
+ action='store_true',
+ help="Whether to run training.")
+ parser.add_argument("--do_eval",
+ action='store_true',
+ help="Whether to run eval on the dev set.")
+ parser.add_argument("--do_lower_case",
+ action='store_true',
+ help="Set this flag if you are using an uncased model.")
+ parser.add_argument("--train_batch_size",
+ default=32,
+ type=int,
+ help="Total batch size for training.")
+ parser.add_argument("--eval_batch_size",
+ default=8,
+ type=int,
+ help="Total batch size for eval.")
+ parser.add_argument("--learning_rate",
+ default=5e-5,
+ type=float,
+ help="The initial learning rate for Adam.")
+ parser.add_argument("--num_train_epochs",
+ default=3.0,
+ type=float,
+ help="Total number of training epochs to perform.")
+ parser.add_argument("--warmup_proportion",
+ default=0.1,
+ type=float,
+ help="Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10%% of training.")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help="local_rank for distributed training on gpus")
+ parser.add_argument('--seed',
+ type=int,
+ default=42,
+ help="random seed for initialization")
+ parser.add_argument('--gradient_accumulation_steps',
+ type=int,
+ default=1,
+ help="Number of updates steps to accumulate before performing a backward/update pass.")
+ parser.add_argument('--fp16',
+ action='store_true',
+ help="Whether to use 16-bit float precision instead of 32-bit")
+ parser.add_argument('--loss_scale',
+ type=float, default=0,
+ help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
+ "0 (default value): dynamic loss scaling.\n"
+ "Positive power of 2: static loss scaling value.\n")
+ parser.add_argument('--server_ip', type=str, default='', help="Can be used for distant debugging.")
+ parser.add_argument('--server_port', type=str, default='', help="Can be used for distant debugging.")
+ args = parser.parse_args()
+
+ if args.server_ip and args.server_port:
+ # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
+ import ptvsd
+ print("Waiting for debugger attach")
+ ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
+ ptvsd.wait_for_attach()
+
+ processors = {
+ "cola": ColaProcessor,
+ "mnli": MnliProcessor,
+ "mnli-mm": MnliMismatchedProcessor,
+ "mrpc": MrpcProcessor,
+ "sst-2": Sst2Processor,
+ "sts-b": StsbProcessor,
+ "qqp": QqpProcessor,
+ "qnli": QnliProcessor,
+ "rte": RteProcessor,
+ "wnli": WnliProcessor,
+ }
+
+ output_modes = {
+ "cola": "classification",
+ "mnli": "classification",
+ "mrpc": "classification",
+ "sst-2": "classification",
+ "sts-b": "regression",
+ "qqp": "classification",
+ "qnli": "classification",
+ "rte": "classification",
+ "wnli": "classification",
+ }
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ torch.cuda.set_device(args.local_rank)
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+
+ logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO if args.local_rank in [-1, 0] else logging.WARN)
+
+ logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
+ device, n_gpu, bool(args.local_rank != -1), args.fp16))
+
+ if args.gradient_accumulation_steps < 1:
+ raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+ args.gradient_accumulation_steps))
+
+ args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ if n_gpu > 0:
+ torch.cuda.manual_seed_all(args.seed)
+
+ if not args.do_train and not args.do_eval:
+ raise ValueError("At least one of `do_train` or `do_eval` must be True.")
+
+ if os.path.exists(args.output_dir) and os.listdir(args.output_dir) and args.do_train:
+ raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir))
+ if not os.path.exists(args.output_dir):
+ os.makedirs(args.output_dir)
+
+ task_name = args.task_name.lower()
+
+ if task_name not in processors:
+ raise ValueError("Task not found: %s" % (task_name))
+
+ processor = processors[task_name]()
+ output_mode = output_modes[task_name]
+
+ label_list = processor.get_labels()
+ num_labels = len(label_list)
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ train_examples = None
+ num_train_optimization_steps = None
+ if args.do_train:
+ train_examples = processor.get_train_examples(args.data_dir)
+ num_train_optimization_steps = int(
+ len(train_examples) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs
+ if args.local_rank != -1:
+ num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
+
+ # Prepare model
+ cache_dir = args.cache_dir if args.cache_dir else os.path.join(str(PYTORCH_PRETRAINED_BERT_CACHE), 'distributed_{}'.format(args.local_rank))
+ model = BertForSequenceClassification.from_pretrained(args.bert_model,
+ cache_dir=cache_dir,
+ num_labels=num_labels)
+
+ ### RANDOM INITIALIZATION ####
+ # config = BertConfig.from_dict({
+ # "attention_probs_dropout_prob": 0.1,
+ # "hidden_act": "gelu",
+ # "hidden_dropout_prob": 0.1,
+ # "hidden_size": 768,
+ # "initializer_range": 0.02,
+ # "intermediate_size": 3072,
+ # "max_position_embeddings": 512,
+ # "num_attention_heads": 12,
+ # "num_hidden_layers": 12,
+ # "type_vocab_size": 2,
+ # "vocab_size": 30522
+ # })
+ # model = BertForSequenceClassification(config=config, num_labels=num_labels)
+
+
+ ###############################
+
+ if args.fp16:
+ model.half()
+ model.to(device)
+ if args.local_rank != -1:
+ try:
+ from apex.parallel import DistributedDataParallel as DDP
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ model = DDP(model)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+ if args.fp16:
+ try:
+ from apex.optimizers import FP16_Optimizer
+ from apex.optimizers import FusedAdam
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ optimizer = FusedAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ bias_correction=False,
+ max_grad_norm=1.0)
+ if args.loss_scale == 0:
+ optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
+ else:
+ optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)
+
+ else:
+ optimizer = BertAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ t_total=num_train_optimization_steps)
+
+ global_step = 0
+ nb_tr_steps = 0
+ tr_loss = 0
+ if args.do_train:
+ train_features = convert_examples_to_features(
+ train_examples, label_list, args.max_seq_length, tokenizer, output_mode)
+ logger.info("***** Running training *****")
+ logger.info(" Num examples = %d", len(train_examples))
+ logger.info(" Batch size = %d", args.train_batch_size)
+ logger.info(" Num steps = %d", num_train_optimization_steps)
+ all_input_ids = torch.tensor([f.input_ids for f in train_features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)
+ all_segment_ids = torch.tensor([f.segment_ids for f in train_features], dtype=torch.long)
+
+ if output_mode == "classification":
+ all_label_ids = torch.tensor([f.label_id for f in train_features], dtype=torch.long)
+ elif output_mode == "regression":
+ all_label_ids = torch.tensor([f.label_id for f in train_features], dtype=torch.float)
+
+ train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
+ if args.local_rank == -1:
+ train_sampler = RandomSampler(train_data)
+ else:
+ train_sampler = DistributedSampler(train_data)
+ train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size)
+
+ model.train()
+ for _ in trange(int(args.num_train_epochs), desc="Epoch"):
+ tr_loss = 0
+ nb_tr_examples, nb_tr_steps = 0, 0
+ for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, input_mask, segment_ids, label_ids = batch
+
+ # define a new function to compute loss values for both output_modes
+ logits, _ = model(input_ids, segment_ids, input_mask, labels=None)
+
+ if output_mode == "classification":
+ loss_fct = CrossEntropyLoss()
+ loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))
+ elif output_mode == "regression":
+ loss_fct = MSELoss()
+ loss = loss_fct(logits.view(-1), label_ids.view(-1))
+
+ if n_gpu > 1:
+ loss = loss.mean() # mean() to average on multi-gpu.
+ if args.gradient_accumulation_steps > 1:
+ loss = loss / args.gradient_accumulation_steps
+
+ if args.fp16:
+ optimizer.backward(loss)
+ else:
+ loss.backward()
+
+ tr_loss += loss.item()
+ print(loss.item())
+ nb_tr_examples += input_ids.size(0)
+ nb_tr_steps += 1
+ if (step + 1) % args.gradient_accumulation_steps == 0:
+ if args.fp16:
+ # modify learning rate with special warm up BERT uses
+ # if args.fp16 is False, BertAdam is used that handles this automatically
+ lr_this_step = args.learning_rate * warmup_linear(global_step/num_train_optimization_steps, args.warmup_proportion)
+ for param_group in optimizer.param_groups:
+ param_group['lr'] = lr_this_step
+ optimizer.step()
+ optimizer.zero_grad()
+ global_step += 1
+
+ if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+ # Save a trained model, configuration and tokenizer
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+
+ # If we save using the predefined names, we can load using `from_pretrained`
+ output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
+ output_config_file = os.path.join(args.output_dir, CONFIG_NAME)
+
+ torch.save(model_to_save.state_dict(), output_model_file)
+ model_to_save.config.to_json_file(output_config_file)
+ tokenizer.save_vocabulary(args.output_dir)
+
+ # Load a trained model and vocabulary that you have fine-tuned
+ model = BertForSequenceClassification.from_pretrained(args.output_dir, num_labels=num_labels)
+ tokenizer = BertTokenizer.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+ else:
+ model = BertForSequenceClassification.from_pretrained(args.bert_model, num_labels=num_labels)
+ model.to(device)
+
+ if args.do_eval and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+ eval_examples = processor.get_dev_examples(args.data_dir)
+ eval_features = convert_examples_to_features(
+ eval_examples, label_list, args.max_seq_length, tokenizer, output_mode)
+ logger.info("***** Running evaluation *****")
+ logger.info(" Num examples = %d", len(eval_examples))
+ logger.info(" Batch size = %d", args.eval_batch_size)
+ all_input_ids = torch.tensor([f.input_ids for f in eval_features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in eval_features], dtype=torch.long)
+ all_segment_ids = torch.tensor([f.segment_ids for f in eval_features], dtype=torch.long)
+
+ if output_mode == "classification":
+ all_label_ids = torch.tensor([f.label_id for f in eval_features], dtype=torch.long)
+ elif output_mode == "regression":
+ all_label_ids = torch.tensor([f.label_id for f in eval_features], dtype=torch.float)
+
+ eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
+ # Run prediction for full data
+ eval_sampler = SequentialSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size)
+
+ model.eval()
+ eval_loss = 0
+ nb_eval_steps = 0
+ preds = []
+
+ for input_ids, input_mask, segment_ids, label_ids in tqdm(eval_dataloader, desc="Evaluating"):
+ input_ids = input_ids.to(device)
+ input_mask = input_mask.to(device)
+ segment_ids = segment_ids.to(device)
+ label_ids = label_ids.to(device)
+
+ with torch.no_grad():
+ logits, attns = model(input_ids, segment_ids, input_mask, labels=None)
+
+ # create eval loss and other metric required by the task
+ if output_mode == "classification":
+ loss_fct = CrossEntropyLoss()
+ tmp_eval_loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))
+ elif output_mode == "regression":
+ loss_fct = MSELoss()
+ tmp_eval_loss = loss_fct(logits.view(-1), label_ids.view(-1))
+
+ eval_loss += tmp_eval_loss.mean().item()
+ nb_eval_steps += 1
+ if len(preds) == 0:
+ preds.append(logits.detach().cpu().numpy())
+ else:
+ preds[0] = np.append(preds[0], logits.detach().cpu().numpy(), axis=0)
+
+ eval_loss = eval_loss / nb_eval_steps
+ preds = preds[0]
+ if output_mode == "classification":
+ preds = np.argmax(preds, axis=1)
+ elif output_mode == "regression":
+ preds = np.squeeze(preds)
+ result = compute_metrics(task_name, preds, all_label_ids.numpy())
+ loss = tr_loss/nb_tr_steps if args.do_train else None
+
+ result['eval_loss'] = eval_loss
+ result['global_step'] = global_step
+ result['loss'] = loss
+
+ output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
+ with open(output_eval_file, "w") as writer:
+ logger.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ logger.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+ # hack for MNLI-MM
+ if task_name == "mnli":
+ task_name = "mnli-mm"
+ processor = processors[task_name]()
+
+ if os.path.exists(args.output_dir + '-MM') and os.listdir(args.output_dir + '-MM') and args.do_train:
+ raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir))
+ if not os.path.exists(args.output_dir + '-MM'):
+ os.makedirs(args.output_dir + '-MM')
+
+ eval_examples = processor.get_dev_examples(args.data_dir)
+ eval_features = convert_examples_to_features(
+ eval_examples, label_list, args.max_seq_length, tokenizer, output_mode)
+ logger.info("***** Running evaluation *****")
+ logger.info(" Num examples = %d", len(eval_examples))
+ logger.info(" Batch size = %d", args.eval_batch_size)
+ all_input_ids = torch.tensor([f.input_ids for f in eval_features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in eval_features], dtype=torch.long)
+ all_segment_ids = torch.tensor([f.segment_ids for f in eval_features], dtype=torch.long)
+ all_label_ids = torch.tensor([f.label_id for f in eval_features], dtype=torch.long)
+
+ eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids)
+ # Run prediction for full data
+ eval_sampler = SequentialSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size)
+
+ model.eval()
+ eval_loss = 0
+ nb_eval_steps = 0
+ preds = []
+
+ for input_ids, input_mask, segment_ids, label_ids in tqdm(eval_dataloader, desc="Evaluating"):
+ input_ids = input_ids.to(device)
+ input_mask = input_mask.to(device)
+ segment_ids = segment_ids.to(device)
+ label_ids = label_ids.to(device)
+
+ with torch.no_grad():
+ logits = model(input_ids, segment_ids, input_mask, labels=None)
+
+ loss_fct = CrossEntropyLoss()
+ tmp_eval_loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))
+
+ eval_loss += tmp_eval_loss.mean().item()
+ nb_eval_steps += 1
+ if len(preds) == 0:
+ preds.append(logits.detach().cpu().numpy())
+ else:
+ preds[0] = np.append(
+ preds[0], logits.detach().cpu().numpy(), axis=0)
+
+ eval_loss = eval_loss / nb_eval_steps
+ preds = preds[0]
+ preds = np.argmax(preds, axis=1)
+ result = compute_metrics(task_name, preds, all_label_ids.numpy())
+ loss = tr_loss/nb_tr_steps if args.do_train else None
+
+ result['eval_loss'] = eval_loss
+ result['global_step'] = global_step
+ result['loss'] = loss
+
+ output_eval_file = os.path.join(args.output_dir + '-MM', "eval_results.txt")
+ with open(output_eval_file, "w") as writer:
+ logger.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ logger.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_gpt2.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_gpt2.py
new file mode 100644
index 0000000000000000000000000000000000000000..9e9e2b255bafefb46ed8f85c98a97f219dea86ca
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_gpt2.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+
+import argparse
+import logging
+from tqdm import trange
+
+import torch
+import torch.nn.functional as F
+import numpy as np
+
+from pytorch_pretrained_bert import GPT2LMHeadModel, GPT2Tokenizer
+
+logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO)
+logger = logging.getLogger(__name__)
+
+def top_k_logits(logits, k):
+ """
+ Masks everything but the k top entries as -infinity (1e10).
+ Used to mask logits such that e^-infinity -> 0 won't contribute to the
+ sum of the denominator.
+ """
+ if k == 0:
+ return logits
+ else:
+ values = torch.topk(logits, k)[0]
+ batch_mins = values[:, -1].view(-1, 1).expand_as(logits)
+ return torch.where(logits < batch_mins, torch.ones_like(logits) * -1e10, logits)
+
+def sample_sequence(model, length, start_token=None, batch_size=None, context=None, temperature=1, top_k=0, device='cuda', sample=True):
+ if start_token is None:
+ assert context is not None, 'Specify exactly one of start_token and context!'
+ context = torch.tensor(context, device=device, dtype=torch.long).unsqueeze(0).repeat(batch_size, 1)
+ else:
+ assert context is None, 'Specify exactly one of start_token and context!'
+ context = torch.full((batch_size, 1), start_token, device=device, dtype=torch.long)
+ prev = context
+ output = context
+ past = None
+ with torch.no_grad():
+ for i in trange(length):
+ logits, past = model(prev, past=past)
+ logits = logits[:, -1, :] / temperature
+ logits = top_k_logits(logits, k=top_k)
+ log_probs = F.softmax(logits, dim=-1)
+ if sample:
+ prev = torch.multinomial(log_probs, num_samples=1)
+ else:
+ _, prev = torch.topk(log_probs, k=1, dim=-1)
+ output = torch.cat((output, prev), dim=1)
+ return output
+
+def run_model():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--model_name_or_path', type=str, default='gpt2', help='pretrained model name or path to local checkpoint')
+ parser.add_argument("--seed", type=int, default=0)
+ parser.add_argument("--nsamples", type=int, default=1)
+ parser.add_argument("--batch_size", type=int, default=-1)
+ parser.add_argument("--length", type=int, default=-1)
+ parser.add_argument("--temperature", type=float, default=1.0)
+ parser.add_argument("--top_k", type=int, default=0)
+ parser.add_argument('--unconditional', action='store_true', help='If true, unconditional generation.')
+ args = parser.parse_args()
+ print(args)
+
+ if args.batch_size == -1:
+ args.batch_size = 1
+ assert args.nsamples % args.batch_size == 0
+
+ np.random.seed(args.seed)
+ torch.random.manual_seed(args.seed)
+ torch.cuda.manual_seed(args.seed)
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+
+ enc = GPT2Tokenizer.from_pretrained(args.model_name_or_path)
+ model = GPT2LMHeadModel.from_pretrained(args.model_name_or_path)
+ model.to(device)
+ model.eval()
+
+ if args.length == -1:
+ args.length = model.config.n_ctx // 2
+ elif args.length > model.config.n_ctx:
+ raise ValueError("Can't get samples longer than window size: %s" % model.config.n_ctx)
+
+ while True:
+ context_tokens = []
+ if not args.unconditional:
+ raw_text = input("Model prompt >>> ")
+ while not raw_text:
+ print('Prompt should not be empty!')
+ raw_text = input("Model prompt >>> ")
+ context_tokens = enc.encode(raw_text)
+ generated = 0
+ for _ in range(args.nsamples // args.batch_size):
+ out = sample_sequence(
+ model=model, length=args.length,
+ context=context_tokens,
+ start_token=None,
+ batch_size=args.batch_size,
+ temperature=args.temperature, top_k=args.top_k, device=device
+ )
+ out = out[:, len(context_tokens):].tolist()
+ for i in range(args.batch_size):
+ generated += 1
+ text = enc.decode(out[i])
+ print("=" * 40 + " SAMPLE " + str(generated) + " " + "=" * 40)
+ print(text)
+ print("=" * 80)
+ if args.unconditional:
+ generated = 0
+ for _ in range(args.nsamples // args.batch_size):
+ out = sample_sequence(
+ model=model, length=args.length,
+ context=None,
+ start_token=enc.encoder['<|endoftext|>'],
+ batch_size=args.batch_size,
+ temperature=args.temperature, top_k=args.top_k, device=device
+ )
+ out = out[:,1:].tolist()
+ for i in range(args.batch_size):
+ generated += 1
+ text = enc.decode(out[i])
+ print("=" * 40 + " SAMPLE " + str(generated) + " " + "=" * 40)
+ print(text)
+ print("=" * 80)
+ if args.unconditional:
+ break
+
+if __name__ == '__main__':
+ run_model()
+
+
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_openai_gpt.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_openai_gpt.py
new file mode 100644
index 0000000000000000000000000000000000000000..cb5aa8d9cbdd3ba3ca5fdb738d0a4eaa8842286e
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_openai_gpt.py
@@ -0,0 +1,274 @@
+# coding=utf-8
+# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" OpenAI GPT model fine-tuning script.
+ Adapted from https://github.com/huggingface/pytorch-openai-transformer-lm/blob/master/train.py
+ It self adapted from https://github.com/openai/finetune-transformer-lm/blob/master/train.py
+
+ This script with default values fine-tunes and evaluate a pretrained OpenAI GPT on the RocStories dataset:
+ python run_openai_gpt.py \
+ --model_name openai-gpt \
+ --do_train \
+ --do_eval \
+ --train_dataset $ROC_STORIES_DIR/cloze_test_val__spring2016\ -\ cloze_test_ALL_val.csv \
+ --eval_dataset $ROC_STORIES_DIR/cloze_test_test__spring2016\ -\ cloze_test_ALL_test.csv \
+ --output_dir ../log \
+ --train_batch_size 16 \
+"""
+import argparse
+import os
+import csv
+import random
+import logging
+from tqdm import tqdm, trange
+
+import numpy as np
+import torch
+from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
+ TensorDataset)
+
+from pytorch_pretrained_bert import (OpenAIGPTDoubleHeadsModel, OpenAIGPTTokenizer,
+ OpenAIAdam, cached_path, WEIGHTS_NAME, CONFIG_NAME)
+
+ROCSTORIES_URL = "https://s3.amazonaws.com/datasets.huggingface.co/ROCStories.tar.gz"
+
+logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO)
+logger = logging.getLogger(__name__)
+
+def accuracy(out, labels):
+ outputs = np.argmax(out, axis=1)
+ return np.sum(outputs == labels)
+
+def load_rocstories_dataset(dataset_path):
+ """ Output a list of tuples(story, 1st continuation, 2nd continuation, label) """
+ with open(dataset_path, encoding='utf_8') as f:
+ f = csv.reader(f)
+ output = []
+ next(f) # skip the first line
+ for line in tqdm(f):
+ output.append((' '.join(line[1:5]), line[5], line[6], int(line[-1])-1))
+ return output
+
+def pre_process_datasets(encoded_datasets, input_len, cap_length, start_token, delimiter_token, clf_token):
+ """ Pre-process datasets containing lists of tuples(story, 1st continuation, 2nd continuation, label)
+
+ To Transformer inputs of shape (n_batch, n_alternative, length) comprising for each batch, continuation:
+ input_ids[batch, alternative, :] = [start_token] + story[:cap_length] + [delimiter_token] + cont1[:cap_length] + [clf_token]
+ """
+ tensor_datasets = []
+ for dataset in encoded_datasets:
+ n_batch = len(dataset)
+ input_ids = np.zeros((n_batch, 2, input_len), dtype=np.int64)
+ mc_token_ids = np.zeros((n_batch, 2), dtype=np.int64)
+ lm_labels = np.full((n_batch, 2, input_len), fill_value=-1, dtype=np.int64)
+ mc_labels = np.zeros((n_batch,), dtype=np.int64)
+ for i, (story, cont1, cont2, mc_label), in enumerate(dataset):
+ with_cont1 = [start_token] + story[:cap_length] + [delimiter_token] + cont1[:cap_length] + [clf_token]
+ with_cont2 = [start_token] + story[:cap_length] + [delimiter_token] + cont2[:cap_length] + [clf_token]
+ input_ids[i, 0, :len(with_cont1)] = with_cont1
+ input_ids[i, 1, :len(with_cont2)] = with_cont2
+ mc_token_ids[i, 0] = len(with_cont1) - 1
+ mc_token_ids[i, 1] = len(with_cont2) - 1
+ lm_labels[i, 0, :len(with_cont1)-1] = with_cont1[1:]
+ lm_labels[i, 1, :len(with_cont2)-1] = with_cont2[1:]
+ mc_labels[i] = mc_label
+ all_inputs = (input_ids, mc_token_ids, lm_labels, mc_labels)
+ tensor_datasets.append(tuple(torch.tensor(t) for t in all_inputs))
+ return tensor_datasets
+
+def main():
+ parser = argparse.ArgumentParser()
+ parser.add_argument('--model_name', type=str, default='openai-gpt',
+ help='pretrained model name')
+ parser.add_argument("--do_train", action='store_true', help="Whether to run training.")
+ parser.add_argument("--do_eval", action='store_true', help="Whether to run eval on the dev set.")
+ parser.add_argument("--output_dir", default=None, type=str, required=True,
+ help="The output directory where the model predictions and checkpoints will be written.")
+ parser.add_argument('--train_dataset', type=str, default='')
+ parser.add_argument('--eval_dataset', type=str, default='')
+ parser.add_argument('--seed', type=int, default=42)
+ parser.add_argument('--num_train_epochs', type=int, default=3)
+ parser.add_argument('--train_batch_size', type=int, default=8)
+ parser.add_argument('--eval_batch_size', type=int, default=16)
+ parser.add_argument('--max_grad_norm', type=int, default=1)
+ parser.add_argument('--learning_rate', type=float, default=6.25e-5)
+ parser.add_argument('--warmup_proportion', type=float, default=0.002)
+ parser.add_argument('--lr_schedule', type=str, default='warmup_linear')
+ parser.add_argument('--weight_decay', type=float, default=0.01)
+ parser.add_argument('--lm_coef', type=float, default=0.9)
+ parser.add_argument('--n_valid', type=int, default=374)
+
+ parser.add_argument('--server_ip', type=str, default='', help="Can be used for distant debugging.")
+ parser.add_argument('--server_port', type=str, default='', help="Can be used for distant debugging.")
+ args = parser.parse_args()
+ print(args)
+
+ if args.server_ip and args.server_port:
+ # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
+ import ptvsd
+ print("Waiting for debugger attach")
+ ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
+ ptvsd.wait_for_attach()
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ torch.cuda.manual_seed_all(args.seed)
+
+ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+ n_gpu = torch.cuda.device_count()
+ logger.info("device: {}, n_gpu {}".format(device, n_gpu))
+
+ if not args.do_train and not args.do_eval:
+ raise ValueError("At least one of `do_train` or `do_eval` must be True.")
+
+ if not os.path.exists(args.output_dir):
+ os.makedirs(args.output_dir)
+
+ # Load tokenizer and model
+ # This loading functions also add new tokens and embeddings called `special tokens`
+ # These new embeddings will be fine-tuned on the RocStories dataset
+ special_tokens = ['_start_', '_delimiter_', '_classify_']
+ tokenizer = OpenAIGPTTokenizer.from_pretrained(args.model_name, special_tokens=special_tokens)
+ special_tokens_ids = list(tokenizer.convert_tokens_to_ids(token) for token in special_tokens)
+ model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.model_name, num_special_tokens=len(special_tokens))
+ model.to(device)
+
+ # Load and encode the datasets
+ if not args.train_dataset and not args.eval_dataset:
+ roc_stories = cached_path(ROCSTORIES_URL)
+ def tokenize_and_encode(obj):
+ """ Tokenize and encode a nested object """
+ if isinstance(obj, str):
+ return tokenizer.convert_tokens_to_ids(tokenizer.tokenize(obj))
+ elif isinstance(obj, int):
+ return obj
+ return list(tokenize_and_encode(o) for o in obj)
+ logger.info("Encoding dataset...")
+ train_dataset = load_rocstories_dataset(args.train_dataset)
+ eval_dataset = load_rocstories_dataset(args.eval_dataset)
+ datasets = (train_dataset, eval_dataset)
+ encoded_datasets = tokenize_and_encode(datasets)
+
+ # Compute the max input length for the Transformer
+ max_length = model.config.n_positions // 2 - 2
+ input_length = max(len(story[:max_length]) + max(len(cont1[:max_length]), len(cont2[:max_length])) + 3 \
+ for dataset in encoded_datasets for story, cont1, cont2, _ in dataset)
+ input_length = min(input_length, model.config.n_positions) # Max size of input for the pre-trained model
+
+ # Prepare inputs tensors and dataloaders
+ tensor_datasets = pre_process_datasets(encoded_datasets, input_length, max_length, *special_tokens_ids)
+ train_tensor_dataset, eval_tensor_dataset = tensor_datasets[0], tensor_datasets[1]
+
+ train_data = TensorDataset(*train_tensor_dataset)
+ train_sampler = RandomSampler(train_data)
+ train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size)
+
+ eval_data = TensorDataset(*eval_tensor_dataset)
+ eval_sampler = SequentialSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+ num_train_optimization_steps = len(train_data) * args.num_train_epochs // args.train_batch_size
+ optimizer = OpenAIAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ max_grad_norm=args.max_grad_norm,
+ weight_decay=args.weight_decay,
+ t_total=num_train_optimization_steps)
+
+ if args.do_train:
+ nb_tr_steps, tr_loss, exp_average_loss = 0, 0, None
+ model.train()
+ for _ in trange(int(args.num_train_epochs), desc="Epoch"):
+ tr_loss = 0
+ nb_tr_steps = 0
+ tqdm_bar = tqdm(train_dataloader, desc="Training")
+ for step, batch in enumerate(tqdm_bar):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, mc_token_ids, lm_labels, mc_labels = batch
+ losses = model(input_ids, mc_token_ids, lm_labels, mc_labels)
+ loss = args.lm_coef * losses[0] + losses[1]
+ loss.backward()
+ optimizer.step()
+ optimizer.zero_grad()
+ tr_loss += loss.item()
+ exp_average_loss = loss.item() if exp_average_loss is None else 0.7*exp_average_loss+0.3*loss.item()
+ nb_tr_steps += 1
+ tqdm_bar.desc = "Training loss: {:.2e} lr: {:.2e}".format(exp_average_loss, optimizer.get_lr()[0])
+
+ # Save a trained model
+ if args.do_train:
+ # Save a trained model, configuration and tokenizer
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+
+ # If we save using the predefined names, we can load using `from_pretrained`
+ output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
+ output_config_file = os.path.join(args.output_dir, CONFIG_NAME)
+
+ torch.save(model_to_save.state_dict(), output_model_file)
+ model_to_save.config.to_json_file(output_config_file)
+ tokenizer.save_vocabulary(args.output_dir)
+
+ # Load a trained model and vocabulary that you have fine-tuned
+ model = OpenAIGPTDoubleHeadsModel.from_pretrained(args.output_dir)
+ tokenizer = OpenAIGPTTokenizer.from_pretrained(args.output_dir)
+ model.to(device)
+
+ if args.do_eval:
+ model.eval()
+ eval_loss, eval_accuracy = 0, 0
+ nb_eval_steps, nb_eval_examples = 0, 0
+ for batch in tqdm(eval_dataloader, desc="Evaluating"):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, mc_token_ids, lm_labels, mc_labels = batch
+ with torch.no_grad():
+ _, mc_loss = model(input_ids, mc_token_ids, lm_labels, mc_labels)
+ _, mc_logits = model(input_ids, mc_token_ids)
+
+ mc_logits = mc_logits.detach().cpu().numpy()
+ mc_labels = mc_labels.to('cpu').numpy()
+ tmp_eval_accuracy = accuracy(mc_logits, mc_labels)
+
+ eval_loss += mc_loss.mean().item()
+ eval_accuracy += tmp_eval_accuracy
+
+ nb_eval_examples += input_ids.size(0)
+ nb_eval_steps += 1
+
+ eval_loss = eval_loss / nb_eval_steps
+ eval_accuracy = eval_accuracy / nb_eval_examples
+ train_loss = tr_loss/nb_tr_steps if args.do_train else None
+ result = {'eval_loss': eval_loss,
+ 'eval_accuracy': eval_accuracy,
+ 'train_loss': train_loss}
+
+ output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
+ with open(output_eval_file, "w") as writer:
+ logger.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ logger.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+if __name__ == '__main__':
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_squad.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_squad.py
new file mode 100644
index 0000000000000000000000000000000000000000..410fd8529880ed3514744b44a3b0d7e9d099dc6e
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_squad.py
@@ -0,0 +1,1098 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Run BERT on SQuAD."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+import collections
+import json
+import logging
+import math
+import os
+import random
+import sys
+from io import open
+
+import numpy as np
+import torch
+from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
+ TensorDataset)
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
+from pytorch_pretrained_bert.modeling import BertForQuestionAnswering, BertConfig
+from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
+from pytorch_pretrained_bert.tokenization import (BasicTokenizer,
+ BertTokenizer,
+ whitespace_tokenize)
+
+if sys.version_info[0] == 2:
+ import cPickle as pickle
+else:
+ import pickle
+
+logger = logging.getLogger(__name__)
+
+
+class SquadExample(object):
+ """
+ A single training/test example for the Squad dataset.
+ For examples without an answer, the start and end position are -1.
+ """
+
+ def __init__(self,
+ qas_id,
+ question_text,
+ doc_tokens,
+ orig_answer_text=None,
+ start_position=None,
+ end_position=None,
+ is_impossible=None):
+ self.qas_id = qas_id
+ self.question_text = question_text
+ self.doc_tokens = doc_tokens
+ self.orig_answer_text = orig_answer_text
+ self.start_position = start_position
+ self.end_position = end_position
+ self.is_impossible = is_impossible
+
+ def __str__(self):
+ return self.__repr__()
+
+ def __repr__(self):
+ s = ""
+ s += "qas_id: %s" % (self.qas_id)
+ s += ", question_text: %s" % (
+ self.question_text)
+ s += ", doc_tokens: [%s]" % (" ".join(self.doc_tokens))
+ if self.start_position:
+ s += ", start_position: %d" % (self.start_position)
+ if self.end_position:
+ s += ", end_position: %d" % (self.end_position)
+ if self.is_impossible:
+ s += ", is_impossible: %r" % (self.is_impossible)
+ return s
+
+
+class InputFeatures(object):
+ """A single set of features of data."""
+
+ def __init__(self,
+ unique_id,
+ example_index,
+ doc_span_index,
+ tokens,
+ token_to_orig_map,
+ token_is_max_context,
+ input_ids,
+ input_mask,
+ segment_ids,
+ start_position=None,
+ end_position=None,
+ is_impossible=None):
+ self.unique_id = unique_id
+ self.example_index = example_index
+ self.doc_span_index = doc_span_index
+ self.tokens = tokens
+ self.token_to_orig_map = token_to_orig_map
+ self.token_is_max_context = token_is_max_context
+ self.input_ids = input_ids
+ self.input_mask = input_mask
+ self.segment_ids = segment_ids
+ self.start_position = start_position
+ self.end_position = end_position
+ self.is_impossible = is_impossible
+
+
+def read_squad_examples(input_file, is_training, version_2_with_negative):
+ """Read a SQuAD json file into a list of SquadExample."""
+ with open(input_file, "r", encoding='utf-8') as reader:
+ input_data = json.load(reader)["data"]
+
+ def is_whitespace(c):
+ if c == " " or c == "\t" or c == "\r" or c == "\n" or ord(c) == 0x202F:
+ return True
+ return False
+
+ examples = []
+ for entry in input_data:
+ for paragraph in entry["paragraphs"]:
+ paragraph_text = paragraph["context"]
+ doc_tokens = []
+ char_to_word_offset = []
+ prev_is_whitespace = True
+ for c in paragraph_text:
+ if is_whitespace(c):
+ prev_is_whitespace = True
+ else:
+ if prev_is_whitespace:
+ doc_tokens.append(c)
+ else:
+ doc_tokens[-1] += c
+ prev_is_whitespace = False
+ char_to_word_offset.append(len(doc_tokens) - 1)
+
+ for qa in paragraph["qas"]:
+ qas_id = qa["id"]
+ question_text = qa["question"]
+ start_position = None
+ end_position = None
+ orig_answer_text = None
+ is_impossible = False
+ if is_training:
+ if version_2_with_negative:
+ is_impossible = qa["is_impossible"]
+ if (len(qa["answers"]) != 1) and (not is_impossible):
+ raise ValueError(
+ "For training, each question should have exactly 1 answer.")
+ if not is_impossible:
+ answer = qa["answers"][0]
+ orig_answer_text = answer["text"]
+ answer_offset = answer["answer_start"]
+ answer_length = len(orig_answer_text)
+ start_position = char_to_word_offset[answer_offset]
+ end_position = char_to_word_offset[answer_offset + answer_length - 1]
+ # Only add answers where the text can be exactly recovered from the
+ # document. If this CAN'T happen it's likely due to weird Unicode
+ # stuff so we will just skip the example.
+ #
+ # Note that this means for training mode, every example is NOT
+ # guaranteed to be preserved.
+ actual_text = " ".join(doc_tokens[start_position:(end_position + 1)])
+ cleaned_answer_text = " ".join(
+ whitespace_tokenize(orig_answer_text))
+ if actual_text.find(cleaned_answer_text) == -1:
+ logger.warning("Could not find answer: '%s' vs. '%s'",
+ actual_text, cleaned_answer_text)
+ continue
+ else:
+ start_position = -1
+ end_position = -1
+ orig_answer_text = ""
+
+ example = SquadExample(
+ qas_id=qas_id,
+ question_text=question_text,
+ doc_tokens=doc_tokens,
+ orig_answer_text=orig_answer_text,
+ start_position=start_position,
+ end_position=end_position,
+ is_impossible=is_impossible)
+ examples.append(example)
+ return examples
+
+
+def convert_examples_to_features(examples, tokenizer, max_seq_length,
+ doc_stride, max_query_length, is_training):
+ """Loads a data file into a list of `InputBatch`s."""
+
+ unique_id = 1000000000
+
+ features = []
+ for (example_index, example) in enumerate(examples):
+ query_tokens = tokenizer.tokenize(example.question_text)
+
+ if len(query_tokens) > max_query_length:
+ query_tokens = query_tokens[0:max_query_length]
+
+ tok_to_orig_index = []
+ orig_to_tok_index = []
+ all_doc_tokens = []
+ for (i, token) in enumerate(example.doc_tokens):
+ orig_to_tok_index.append(len(all_doc_tokens))
+ sub_tokens = tokenizer.tokenize(token)
+ for sub_token in sub_tokens:
+ tok_to_orig_index.append(i)
+ all_doc_tokens.append(sub_token)
+
+ tok_start_position = None
+ tok_end_position = None
+ if is_training and example.is_impossible:
+ tok_start_position = -1
+ tok_end_position = -1
+ if is_training and not example.is_impossible:
+ tok_start_position = orig_to_tok_index[example.start_position]
+ if example.end_position < len(example.doc_tokens) - 1:
+ tok_end_position = orig_to_tok_index[example.end_position + 1] - 1
+ else:
+ tok_end_position = len(all_doc_tokens) - 1
+ (tok_start_position, tok_end_position) = _improve_answer_span(
+ all_doc_tokens, tok_start_position, tok_end_position, tokenizer,
+ example.orig_answer_text)
+
+ # The -3 accounts for [CLS], [SEP] and [SEP]
+ max_tokens_for_doc = max_seq_length - len(query_tokens) - 3
+
+ # We can have documents that are longer than the maximum sequence length.
+ # To deal with this we do a sliding window approach, where we take chunks
+ # of the up to our max length with a stride of `doc_stride`.
+ _DocSpan = collections.namedtuple( # pylint: disable=invalid-name
+ "DocSpan", ["start", "length"])
+ doc_spans = []
+ start_offset = 0
+ while start_offset < len(all_doc_tokens):
+ length = len(all_doc_tokens) - start_offset
+ if length > max_tokens_for_doc:
+ length = max_tokens_for_doc
+ doc_spans.append(_DocSpan(start=start_offset, length=length))
+ if start_offset + length == len(all_doc_tokens):
+ break
+ start_offset += min(length, doc_stride)
+
+ for (doc_span_index, doc_span) in enumerate(doc_spans):
+ tokens = []
+ token_to_orig_map = {}
+ token_is_max_context = {}
+ segment_ids = []
+ tokens.append("[CLS]")
+ segment_ids.append(0)
+ for token in query_tokens:
+ tokens.append(token)
+ segment_ids.append(0)
+ tokens.append("[SEP]")
+ segment_ids.append(0)
+
+ for i in range(doc_span.length):
+ split_token_index = doc_span.start + i
+ token_to_orig_map[len(tokens)] = tok_to_orig_index[split_token_index]
+
+ is_max_context = _check_is_max_context(doc_spans, doc_span_index,
+ split_token_index)
+ token_is_max_context[len(tokens)] = is_max_context
+ tokens.append(all_doc_tokens[split_token_index])
+ segment_ids.append(1)
+ tokens.append("[SEP]")
+ segment_ids.append(1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+
+ # The mask has 1 for real tokens and 0 for padding tokens. Only real
+ # tokens are attended to.
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ while len(input_ids) < max_seq_length:
+ input_ids.append(0)
+ input_mask.append(0)
+ segment_ids.append(0)
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ start_position = None
+ end_position = None
+ if is_training and not example.is_impossible:
+ # For training, if our document chunk does not contain an annotation
+ # we throw it out, since there is nothing to predict.
+ doc_start = doc_span.start
+ doc_end = doc_span.start + doc_span.length - 1
+ out_of_span = False
+ if not (tok_start_position >= doc_start and
+ tok_end_position <= doc_end):
+ out_of_span = True
+ if out_of_span:
+ start_position = 0
+ end_position = 0
+ else:
+ doc_offset = len(query_tokens) + 2
+ start_position = tok_start_position - doc_start + doc_offset
+ end_position = tok_end_position - doc_start + doc_offset
+ if is_training and example.is_impossible:
+ start_position = 0
+ end_position = 0
+ if example_index < 20:
+ logger.info("*** Example ***")
+ logger.info("unique_id: %s" % (unique_id))
+ logger.info("example_index: %s" % (example_index))
+ logger.info("doc_span_index: %s" % (doc_span_index))
+ logger.info("tokens: %s" % " ".join(tokens))
+ logger.info("token_to_orig_map: %s" % " ".join([
+ "%d:%d" % (x, y) for (x, y) in token_to_orig_map.items()]))
+ logger.info("token_is_max_context: %s" % " ".join([
+ "%d:%s" % (x, y) for (x, y) in token_is_max_context.items()
+ ]))
+ logger.info("input_ids: %s" % " ".join([str(x) for x in input_ids]))
+ logger.info(
+ "input_mask: %s" % " ".join([str(x) for x in input_mask]))
+ logger.info(
+ "segment_ids: %s" % " ".join([str(x) for x in segment_ids]))
+ if is_training and example.is_impossible:
+ logger.info("impossible example")
+ if is_training and not example.is_impossible:
+ answer_text = " ".join(tokens[start_position:(end_position + 1)])
+ logger.info("start_position: %d" % (start_position))
+ logger.info("end_position: %d" % (end_position))
+ logger.info(
+ "answer: %s" % (answer_text))
+
+ features.append(
+ InputFeatures(
+ unique_id=unique_id,
+ example_index=example_index,
+ doc_span_index=doc_span_index,
+ tokens=tokens,
+ token_to_orig_map=token_to_orig_map,
+ token_is_max_context=token_is_max_context,
+ input_ids=input_ids,
+ input_mask=input_mask,
+ segment_ids=segment_ids,
+ start_position=start_position,
+ end_position=end_position,
+ is_impossible=example.is_impossible))
+ unique_id += 1
+
+ return features
+
+
+def _improve_answer_span(doc_tokens, input_start, input_end, tokenizer,
+ orig_answer_text):
+ """Returns tokenized answer spans that better match the annotated answer."""
+
+ # The SQuAD annotations are character based. We first project them to
+ # whitespace-tokenized words. But then after WordPiece tokenization, we can
+ # often find a "better match". For example:
+ #
+ # Question: What year was John Smith born?
+ # Context: The leader was John Smith (1895-1943).
+ # Answer: 1895
+ #
+ # The original whitespace-tokenized answer will be "(1895-1943).". However
+ # after tokenization, our tokens will be "( 1895 - 1943 ) .". So we can match
+ # the exact answer, 1895.
+ #
+ # However, this is not always possible. Consider the following:
+ #
+ # Question: What country is the top exporter of electornics?
+ # Context: The Japanese electronics industry is the lagest in the world.
+ # Answer: Japan
+ #
+ # In this case, the annotator chose "Japan" as a character sub-span of
+ # the word "Japanese". Since our WordPiece tokenizer does not split
+ # "Japanese", we just use "Japanese" as the annotation. This is fairly rare
+ # in SQuAD, but does happen.
+ tok_answer_text = " ".join(tokenizer.tokenize(orig_answer_text))
+
+ for new_start in range(input_start, input_end + 1):
+ for new_end in range(input_end, new_start - 1, -1):
+ text_span = " ".join(doc_tokens[new_start:(new_end + 1)])
+ if text_span == tok_answer_text:
+ return (new_start, new_end)
+
+ return (input_start, input_end)
+
+
+def _check_is_max_context(doc_spans, cur_span_index, position):
+ """Check if this is the 'max context' doc span for the token."""
+
+ # Because of the sliding window approach taken to scoring documents, a single
+ # token can appear in multiple documents. E.g.
+ # Doc: the man went to the store and bought a gallon of milk
+ # Span A: the man went to the
+ # Span B: to the store and bought
+ # Span C: and bought a gallon of
+ # ...
+ #
+ # Now the word 'bought' will have two scores from spans B and C. We only
+ # want to consider the score with "maximum context", which we define as
+ # the *minimum* of its left and right context (the *sum* of left and
+ # right context will always be the same, of course).
+ #
+ # In the example the maximum context for 'bought' would be span C since
+ # it has 1 left context and 3 right context, while span B has 4 left context
+ # and 0 right context.
+ best_score = None
+ best_span_index = None
+ for (span_index, doc_span) in enumerate(doc_spans):
+ end = doc_span.start + doc_span.length - 1
+ if position < doc_span.start:
+ continue
+ if position > end:
+ continue
+ num_left_context = position - doc_span.start
+ num_right_context = end - position
+ score = min(num_left_context, num_right_context) + 0.01 * doc_span.length
+ if best_score is None or score > best_score:
+ best_score = score
+ best_span_index = span_index
+
+ return cur_span_index == best_span_index
+
+
+RawResult = collections.namedtuple("RawResult",
+ ["unique_id", "start_logits", "end_logits"])
+
+
+def write_predictions(all_examples, all_features, all_results, n_best_size,
+ max_answer_length, do_lower_case, output_prediction_file,
+ output_nbest_file, output_null_log_odds_file, verbose_logging,
+ version_2_with_negative, null_score_diff_threshold):
+ """Write final predictions to the json file and log-odds of null if needed."""
+ logger.info("Writing predictions to: %s" % (output_prediction_file))
+ logger.info("Writing nbest to: %s" % (output_nbest_file))
+
+ example_index_to_features = collections.defaultdict(list)
+ for feature in all_features:
+ example_index_to_features[feature.example_index].append(feature)
+
+ unique_id_to_result = {}
+ for result in all_results:
+ unique_id_to_result[result.unique_id] = result
+
+ _PrelimPrediction = collections.namedtuple( # pylint: disable=invalid-name
+ "PrelimPrediction",
+ ["feature_index", "start_index", "end_index", "start_logit", "end_logit"])
+
+ all_predictions = collections.OrderedDict()
+ all_nbest_json = collections.OrderedDict()
+ scores_diff_json = collections.OrderedDict()
+
+ for (example_index, example) in enumerate(all_examples):
+ features = example_index_to_features[example_index]
+
+ prelim_predictions = []
+ # keep track of the minimum score of null start+end of position 0
+ score_null = 1000000 # large and positive
+ min_null_feature_index = 0 # the paragraph slice with min null score
+ null_start_logit = 0 # the start logit at the slice with min null score
+ null_end_logit = 0 # the end logit at the slice with min null score
+ for (feature_index, feature) in enumerate(features):
+ result = unique_id_to_result[feature.unique_id]
+ start_indexes = _get_best_indexes(result.start_logits, n_best_size)
+ end_indexes = _get_best_indexes(result.end_logits, n_best_size)
+ # if we could have irrelevant answers, get the min score of irrelevant
+ if version_2_with_negative:
+ feature_null_score = result.start_logits[0] + result.end_logits[0]
+ if feature_null_score < score_null:
+ score_null = feature_null_score
+ min_null_feature_index = feature_index
+ null_start_logit = result.start_logits[0]
+ null_end_logit = result.end_logits[0]
+ for start_index in start_indexes:
+ for end_index in end_indexes:
+ # We could hypothetically create invalid predictions, e.g., predict
+ # that the start of the span is in the question. We throw out all
+ # invalid predictions.
+ if start_index >= len(feature.tokens):
+ continue
+ if end_index >= len(feature.tokens):
+ continue
+ if start_index not in feature.token_to_orig_map:
+ continue
+ if end_index not in feature.token_to_orig_map:
+ continue
+ if not feature.token_is_max_context.get(start_index, False):
+ continue
+ if end_index < start_index:
+ continue
+ length = end_index - start_index + 1
+ if length > max_answer_length:
+ continue
+ prelim_predictions.append(
+ _PrelimPrediction(
+ feature_index=feature_index,
+ start_index=start_index,
+ end_index=end_index,
+ start_logit=result.start_logits[start_index],
+ end_logit=result.end_logits[end_index]))
+ if version_2_with_negative:
+ prelim_predictions.append(
+ _PrelimPrediction(
+ feature_index=min_null_feature_index,
+ start_index=0,
+ end_index=0,
+ start_logit=null_start_logit,
+ end_logit=null_end_logit))
+ prelim_predictions = sorted(
+ prelim_predictions,
+ key=lambda x: (x.start_logit + x.end_logit),
+ reverse=True)
+
+ _NbestPrediction = collections.namedtuple( # pylint: disable=invalid-name
+ "NbestPrediction", ["text", "start_logit", "end_logit"])
+
+ seen_predictions = {}
+ nbest = []
+ for pred in prelim_predictions:
+ if len(nbest) >= n_best_size:
+ break
+ feature = features[pred.feature_index]
+ if pred.start_index > 0: # this is a non-null prediction
+ tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]
+ orig_doc_start = feature.token_to_orig_map[pred.start_index]
+ orig_doc_end = feature.token_to_orig_map[pred.end_index]
+ orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]
+ tok_text = " ".join(tok_tokens)
+
+ # De-tokenize WordPieces that have been split off.
+ tok_text = tok_text.replace(" ##", "")
+ tok_text = tok_text.replace("##", "")
+
+ # Clean whitespace
+ tok_text = tok_text.strip()
+ tok_text = " ".join(tok_text.split())
+ orig_text = " ".join(orig_tokens)
+
+ final_text = get_final_text(tok_text, orig_text, do_lower_case, verbose_logging)
+ if final_text in seen_predictions:
+ continue
+
+ seen_predictions[final_text] = True
+ else:
+ final_text = ""
+ seen_predictions[final_text] = True
+
+ nbest.append(
+ _NbestPrediction(
+ text=final_text,
+ start_logit=pred.start_logit,
+ end_logit=pred.end_logit))
+ # if we didn't include the empty option in the n-best, include it
+ if version_2_with_negative:
+ if "" not in seen_predictions:
+ nbest.append(
+ _NbestPrediction(
+ text="",
+ start_logit=null_start_logit,
+ end_logit=null_end_logit))
+
+ # In very rare edge cases we could only have single null prediction.
+ # So we just create a nonce prediction in this case to avoid failure.
+ if len(nbest)==1:
+ nbest.insert(0,
+ _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
+
+ # In very rare edge cases we could have no valid predictions. So we
+ # just create a nonce prediction in this case to avoid failure.
+ if not nbest:
+ nbest.append(
+ _NbestPrediction(text="empty", start_logit=0.0, end_logit=0.0))
+
+ assert len(nbest) >= 1
+
+ total_scores = []
+ best_non_null_entry = None
+ for entry in nbest:
+ total_scores.append(entry.start_logit + entry.end_logit)
+ if not best_non_null_entry:
+ if entry.text:
+ best_non_null_entry = entry
+
+ probs = _compute_softmax(total_scores)
+
+ nbest_json = []
+ for (i, entry) in enumerate(nbest):
+ output = collections.OrderedDict()
+ output["text"] = entry.text
+ output["probability"] = probs[i]
+ output["start_logit"] = entry.start_logit
+ output["end_logit"] = entry.end_logit
+ nbest_json.append(output)
+
+ assert len(nbest_json) >= 1
+
+ if not version_2_with_negative:
+ all_predictions[example.qas_id] = nbest_json[0]["text"]
+ else:
+ # predict "" iff the null score - the score of best non-null > threshold
+ score_diff = score_null - best_non_null_entry.start_logit - (
+ best_non_null_entry.end_logit)
+ scores_diff_json[example.qas_id] = score_diff
+ if score_diff > null_score_diff_threshold:
+ all_predictions[example.qas_id] = ""
+ else:
+ all_predictions[example.qas_id] = best_non_null_entry.text
+ all_nbest_json[example.qas_id] = nbest_json
+
+ with open(output_prediction_file, "w") as writer:
+ writer.write(json.dumps(all_predictions, indent=4) + "\n")
+
+ with open(output_nbest_file, "w") as writer:
+ writer.write(json.dumps(all_nbest_json, indent=4) + "\n")
+
+ if version_2_with_negative:
+ with open(output_null_log_odds_file, "w") as writer:
+ writer.write(json.dumps(scores_diff_json, indent=4) + "\n")
+
+
+def get_final_text(pred_text, orig_text, do_lower_case, verbose_logging=False):
+ """Project the tokenized prediction back to the original text."""
+
+ # When we created the data, we kept track of the alignment between original
+ # (whitespace tokenized) tokens and our WordPiece tokenized tokens. So
+ # now `orig_text` contains the span of our original text corresponding to the
+ # span that we predicted.
+ #
+ # However, `orig_text` may contain extra characters that we don't want in
+ # our prediction.
+ #
+ # For example, let's say:
+ # pred_text = steve smith
+ # orig_text = Steve Smith's
+ #
+ # We don't want to return `orig_text` because it contains the extra "'s".
+ #
+ # We don't want to return `pred_text` because it's already been normalized
+ # (the SQuAD eval script also does punctuation stripping/lower casing but
+ # our tokenizer does additional normalization like stripping accent
+ # characters).
+ #
+ # What we really want to return is "Steve Smith".
+ #
+ # Therefore, we have to apply a semi-complicated alignment heuristic between
+ # `pred_text` and `orig_text` to get a character-to-character alignment. This
+ # can fail in certain cases in which case we just return `orig_text`.
+
+ def _strip_spaces(text):
+ ns_chars = []
+ ns_to_s_map = collections.OrderedDict()
+ for (i, c) in enumerate(text):
+ if c == " ":
+ continue
+ ns_to_s_map[len(ns_chars)] = i
+ ns_chars.append(c)
+ ns_text = "".join(ns_chars)
+ return (ns_text, ns_to_s_map)
+
+ # We first tokenize `orig_text`, strip whitespace from the result
+ # and `pred_text`, and check if they are the same length. If they are
+ # NOT the same length, the heuristic has failed. If they are the same
+ # length, we assume the characters are one-to-one aligned.
+ tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
+
+ tok_text = " ".join(tokenizer.tokenize(orig_text))
+
+ start_position = tok_text.find(pred_text)
+ if start_position == -1:
+ if verbose_logging:
+ logger.info(
+ "Unable to find text: '%s' in '%s'" % (pred_text, orig_text))
+ return orig_text
+ end_position = start_position + len(pred_text) - 1
+
+ (orig_ns_text, orig_ns_to_s_map) = _strip_spaces(orig_text)
+ (tok_ns_text, tok_ns_to_s_map) = _strip_spaces(tok_text)
+
+ if len(orig_ns_text) != len(tok_ns_text):
+ if verbose_logging:
+ logger.info("Length not equal after stripping spaces: '%s' vs '%s'",
+ orig_ns_text, tok_ns_text)
+ return orig_text
+
+ # We then project the characters in `pred_text` back to `orig_text` using
+ # the character-to-character alignment.
+ tok_s_to_ns_map = {}
+ for (i, tok_index) in tok_ns_to_s_map.items():
+ tok_s_to_ns_map[tok_index] = i
+
+ orig_start_position = None
+ if start_position in tok_s_to_ns_map:
+ ns_start_position = tok_s_to_ns_map[start_position]
+ if ns_start_position in orig_ns_to_s_map:
+ orig_start_position = orig_ns_to_s_map[ns_start_position]
+
+ if orig_start_position is None:
+ if verbose_logging:
+ logger.info("Couldn't map start position")
+ return orig_text
+
+ orig_end_position = None
+ if end_position in tok_s_to_ns_map:
+ ns_end_position = tok_s_to_ns_map[end_position]
+ if ns_end_position in orig_ns_to_s_map:
+ orig_end_position = orig_ns_to_s_map[ns_end_position]
+
+ if orig_end_position is None:
+ if verbose_logging:
+ logger.info("Couldn't map end position")
+ return orig_text
+
+ output_text = orig_text[orig_start_position:(orig_end_position + 1)]
+ return output_text
+
+
+def _get_best_indexes(logits, n_best_size):
+ """Get the n-best logits from a list."""
+ index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)
+
+ best_indexes = []
+ for i in range(len(index_and_score)):
+ if i >= n_best_size:
+ break
+ best_indexes.append(index_and_score[i][0])
+ return best_indexes
+
+
+def _compute_softmax(scores):
+ """Compute softmax probability over raw logits."""
+ if not scores:
+ return []
+
+ max_score = None
+ for score in scores:
+ if max_score is None or score > max_score:
+ max_score = score
+
+ exp_scores = []
+ total_sum = 0.0
+ for score in scores:
+ x = math.exp(score - max_score)
+ exp_scores.append(x)
+ total_sum += x
+
+ probs = []
+ for score in exp_scores:
+ probs.append(score / total_sum)
+ return probs
+
+def main():
+ parser = argparse.ArgumentParser()
+
+ ## Required parameters
+ parser.add_argument("--bert_model", default=None, type=str, required=True,
+ help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, "
+ "bert-base-multilingual-cased, bert-base-chinese.")
+ parser.add_argument("--output_dir", default=None, type=str, required=True,
+ help="The output directory where the model checkpoints and predictions will be written.")
+
+ ## Other parameters
+ parser.add_argument("--train_file", default=None, type=str, help="SQuAD json for training. E.g., train-v1.1.json")
+ parser.add_argument("--predict_file", default=None, type=str,
+ help="SQuAD json for predictions. E.g., dev-v1.1.json or test-v1.1.json")
+ parser.add_argument("--max_seq_length", default=384, type=int,
+ help="The maximum total input sequence length after WordPiece tokenization. Sequences "
+ "longer than this will be truncated, and sequences shorter than this will be padded.")
+ parser.add_argument("--doc_stride", default=128, type=int,
+ help="When splitting up a long document into chunks, how much stride to take between chunks.")
+ parser.add_argument("--max_query_length", default=64, type=int,
+ help="The maximum number of tokens for the question. Questions longer than this will "
+ "be truncated to this length.")
+ parser.add_argument("--do_train", action='store_true', help="Whether to run training.")
+ parser.add_argument("--do_predict", action='store_true', help="Whether to run eval on the dev set.")
+ parser.add_argument("--train_batch_size", default=32, type=int, help="Total batch size for training.")
+ parser.add_argument("--predict_batch_size", default=8, type=int, help="Total batch size for predictions.")
+ parser.add_argument("--learning_rate", default=5e-5, type=float, help="The initial learning rate for Adam.")
+ parser.add_argument("--num_train_epochs", default=3.0, type=float,
+ help="Total number of training epochs to perform.")
+ parser.add_argument("--warmup_proportion", default=0.1, type=float,
+ help="Proportion of training to perform linear learning rate warmup for. E.g., 0.1 = 10%% "
+ "of training.")
+ parser.add_argument("--n_best_size", default=20, type=int,
+ help="The total number of n-best predictions to generate in the nbest_predictions.json "
+ "output file.")
+ parser.add_argument("--max_answer_length", default=30, type=int,
+ help="The maximum length of an answer that can be generated. This is needed because the start "
+ "and end predictions are not conditioned on one another.")
+ parser.add_argument("--verbose_logging", action='store_true',
+ help="If true, all of the warnings related to data processing will be printed. "
+ "A number of warnings are expected for a normal SQuAD evaluation.")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+ parser.add_argument('--seed',
+ type=int,
+ default=42,
+ help="random seed for initialization")
+ parser.add_argument('--gradient_accumulation_steps',
+ type=int,
+ default=1,
+ help="Number of updates steps to accumulate before performing a backward/update pass.")
+ parser.add_argument("--do_lower_case",
+ action='store_true',
+ help="Whether to lower case the input text. True for uncased models, False for cased models.")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help="local_rank for distributed training on gpus")
+ parser.add_argument('--fp16',
+ action='store_true',
+ help="Whether to use 16-bit float precision instead of 32-bit")
+ parser.add_argument('--loss_scale',
+ type=float, default=0,
+ help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
+ "0 (default value): dynamic loss scaling.\n"
+ "Positive power of 2: static loss scaling value.\n")
+ parser.add_argument('--version_2_with_negative',
+ action='store_true',
+ help='If true, the SQuAD examples contain some that do not have an answer.')
+ parser.add_argument('--null_score_diff_threshold',
+ type=float, default=0.0,
+ help="If null_score - best_non_null is greater than the threshold predict null.")
+ parser.add_argument('--server_ip', type=str, default='', help="Can be used for distant debugging.")
+ parser.add_argument('--server_port', type=str, default='', help="Can be used for distant debugging.")
+ args = parser.parse_args()
+ print(args)
+
+ if args.server_ip and args.server_port:
+ # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
+ import ptvsd
+ print("Waiting for debugger attach")
+ ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
+ ptvsd.wait_for_attach()
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ torch.cuda.set_device(args.local_rank)
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+
+ logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO if args.local_rank in [-1, 0] else logging.WARN)
+
+ logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
+ device, n_gpu, bool(args.local_rank != -1), args.fp16))
+
+ if args.gradient_accumulation_steps < 1:
+ raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+ args.gradient_accumulation_steps))
+
+ args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ if n_gpu > 0:
+ torch.cuda.manual_seed_all(args.seed)
+
+ if not args.do_train and not args.do_predict:
+ raise ValueError("At least one of `do_train` or `do_predict` must be True.")
+
+ if args.do_train:
+ if not args.train_file:
+ raise ValueError(
+ "If `do_train` is True, then `train_file` must be specified.")
+ if args.do_predict:
+ if not args.predict_file:
+ raise ValueError(
+ "If `do_predict` is True, then `predict_file` must be specified.")
+
+ if os.path.exists(args.output_dir) and os.listdir(args.output_dir) and args.do_train:
+ raise ValueError("Output directory () already exists and is not empty.")
+ if not os.path.exists(args.output_dir):
+ os.makedirs(args.output_dir)
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ train_examples = None
+ num_train_optimization_steps = None
+ if args.do_train:
+ train_examples = read_squad_examples(
+ input_file=args.train_file, is_training=True, version_2_with_negative=args.version_2_with_negative)
+ num_train_optimization_steps = int(
+ len(train_examples) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs
+ if args.local_rank != -1:
+ num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
+
+ # Prepare model
+ model = BertForQuestionAnswering.from_pretrained(args.bert_model,
+ cache_dir=os.path.join(str(PYTORCH_PRETRAINED_BERT_CACHE), 'distributed_{}'.format(args.local_rank)))
+
+ if args.fp16:
+ model.half()
+ model.to(device)
+ if args.local_rank != -1:
+ try:
+ from apex.parallel import DistributedDataParallel as DDP
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ model = DDP(model)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+
+ # hack to remove pooler, which is not used
+ # thus it produce None grad that break apex
+ param_optimizer = [n for n in param_optimizer if 'pooler' not in n[0]]
+
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+
+ if args.fp16:
+ try:
+ from apex.optimizers import FP16_Optimizer
+ from apex.optimizers import FusedAdam
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ optimizer = FusedAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ bias_correction=False,
+ max_grad_norm=1.0)
+ if args.loss_scale == 0:
+ optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
+ else:
+ optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)
+ else:
+ optimizer = BertAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ t_total=num_train_optimization_steps)
+
+ global_step = 0
+ if args.do_train:
+ cached_train_features_file = args.train_file+'_{0}_{1}_{2}_{3}'.format(
+ list(filter(None, args.bert_model.split('/'))).pop(), str(args.max_seq_length), str(args.doc_stride), str(args.max_query_length))
+ train_features = None
+ try:
+ with open(cached_train_features_file, "rb") as reader:
+ train_features = pickle.load(reader)
+ except:
+ train_features = convert_examples_to_features(
+ examples=train_examples,
+ tokenizer=tokenizer,
+ max_seq_length=args.max_seq_length,
+ doc_stride=args.doc_stride,
+ max_query_length=args.max_query_length,
+ is_training=True)
+ if args.local_rank == -1 or torch.distributed.get_rank() == 0:
+ logger.info(" Saving train features into cached file %s", cached_train_features_file)
+ with open(cached_train_features_file, "wb") as writer:
+ pickle.dump(train_features, writer)
+ logger.info("***** Running training *****")
+ logger.info(" Num orig examples = %d", len(train_examples))
+ logger.info(" Num split examples = %d", len(train_features))
+ logger.info(" Batch size = %d", args.train_batch_size)
+ logger.info(" Num steps = %d", num_train_optimization_steps)
+ all_input_ids = torch.tensor([f.input_ids for f in train_features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)
+ all_segment_ids = torch.tensor([f.segment_ids for f in train_features], dtype=torch.long)
+ all_start_positions = torch.tensor([f.start_position for f in train_features], dtype=torch.long)
+ all_end_positions = torch.tensor([f.end_position for f in train_features], dtype=torch.long)
+ train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids,
+ all_start_positions, all_end_positions)
+ if args.local_rank == -1:
+ train_sampler = RandomSampler(train_data)
+ else:
+ train_sampler = DistributedSampler(train_data)
+ train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size)
+
+ model.train()
+ for _ in trange(int(args.num_train_epochs), desc="Epoch"):
+ for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])):
+ if n_gpu == 1:
+ batch = tuple(t.to(device) for t in batch) # multi-gpu does scattering it-self
+ input_ids, input_mask, segment_ids, start_positions, end_positions = batch
+ loss = model(input_ids, segment_ids, input_mask, start_positions, end_positions)
+ if n_gpu > 1:
+ loss = loss.mean() # mean() to average on multi-gpu.
+ if args.gradient_accumulation_steps > 1:
+ loss = loss / args.gradient_accumulation_steps
+
+ if args.fp16:
+ optimizer.backward(loss)
+ else:
+ loss.backward()
+ if (step + 1) % args.gradient_accumulation_steps == 0:
+ if args.fp16:
+ # modify learning rate with special warm up BERT uses
+ # if args.fp16 is False, BertAdam is used and handles this automatically
+ lr_this_step = args.learning_rate * warmup_linear(global_step/num_train_optimization_steps, args.warmup_proportion)
+ for param_group in optimizer.param_groups:
+ param_group['lr'] = lr_this_step
+ optimizer.step()
+ optimizer.zero_grad()
+ global_step += 1
+
+ if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+ # Save a trained model, configuration and tokenizer
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+
+ # If we save using the predefined names, we can load using `from_pretrained`
+ output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
+ output_config_file = os.path.join(args.output_dir, CONFIG_NAME)
+
+ torch.save(model_to_save.state_dict(), output_model_file)
+ model_to_save.config.to_json_file(output_config_file)
+ tokenizer.save_vocabulary(args.output_dir)
+
+ # Load a trained model and vocabulary that you have fine-tuned
+ model = BertForQuestionAnswering.from_pretrained(args.output_dir)
+ tokenizer = BertTokenizer.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+ else:
+ model = BertForQuestionAnswering.from_pretrained(args.bert_model)
+
+ model.to(device)
+
+ if args.do_predict and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+ eval_examples = read_squad_examples(
+ input_file=args.predict_file, is_training=False, version_2_with_negative=args.version_2_with_negative)
+ eval_features = convert_examples_to_features(
+ examples=eval_examples,
+ tokenizer=tokenizer,
+ max_seq_length=args.max_seq_length,
+ doc_stride=args.doc_stride,
+ max_query_length=args.max_query_length,
+ is_training=False)
+
+ logger.info("***** Running predictions *****")
+ logger.info(" Num orig examples = %d", len(eval_examples))
+ logger.info(" Num split examples = %d", len(eval_features))
+ logger.info(" Batch size = %d", args.predict_batch_size)
+
+ all_input_ids = torch.tensor([f.input_ids for f in eval_features], dtype=torch.long)
+ all_input_mask = torch.tensor([f.input_mask for f in eval_features], dtype=torch.long)
+ all_segment_ids = torch.tensor([f.segment_ids for f in eval_features], dtype=torch.long)
+ all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)
+ eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_example_index)
+ # Run prediction for full data
+ eval_sampler = SequentialSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.predict_batch_size)
+
+ model.eval()
+ all_results = []
+ logger.info("Start evaluating")
+ for input_ids, input_mask, segment_ids, example_indices in tqdm(eval_dataloader, desc="Evaluating", disable=args.local_rank not in [-1, 0]):
+ if len(all_results) % 1000 == 0:
+ logger.info("Processing example: %d" % (len(all_results)))
+ input_ids = input_ids.to(device)
+ input_mask = input_mask.to(device)
+ segment_ids = segment_ids.to(device)
+ with torch.no_grad():
+ batch_start_logits, batch_end_logits = model(input_ids, segment_ids, input_mask)
+ for i, example_index in enumerate(example_indices):
+ start_logits = batch_start_logits[i].detach().cpu().tolist()
+ end_logits = batch_end_logits[i].detach().cpu().tolist()
+ eval_feature = eval_features[example_index.item()]
+ unique_id = int(eval_feature.unique_id)
+ all_results.append(RawResult(unique_id=unique_id,
+ start_logits=start_logits,
+ end_logits=end_logits))
+ output_prediction_file = os.path.join(args.output_dir, "predictions.json")
+ output_nbest_file = os.path.join(args.output_dir, "nbest_predictions.json")
+ output_null_log_odds_file = os.path.join(args.output_dir, "null_odds.json")
+ write_predictions(eval_examples, eval_features, all_results,
+ args.n_best_size, args.max_answer_length,
+ args.do_lower_case, output_prediction_file,
+ output_nbest_file, output_null_log_odds_file, args.verbose_logging,
+ args.version_2_with_negative, args.null_score_diff_threshold)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_swag.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_swag.py
new file mode 100644
index 0000000000000000000000000000000000000000..a6cfdbe311d5784c79e507e5455baab423a1ae7c
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_swag.py
@@ -0,0 +1,551 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""BERT finetuning runner."""
+
+from __future__ import absolute_import
+
+import argparse
+import csv
+import logging
+import os
+import random
+import sys
+from io import open
+
+import numpy as np
+import torch
+from torch.utils.data import (DataLoader, RandomSampler, SequentialSampler,
+ TensorDataset)
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE, WEIGHTS_NAME, CONFIG_NAME
+from pytorch_pretrained_bert.modeling import BertForMultipleChoice, BertConfig
+from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+
+logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO)
+logger = logging.getLogger(__name__)
+
+
+class SwagExample(object):
+ """A single training/test example for the SWAG dataset."""
+ def __init__(self,
+ swag_id,
+ context_sentence,
+ start_ending,
+ ending_0,
+ ending_1,
+ ending_2,
+ ending_3,
+ label = None):
+ self.swag_id = swag_id
+ self.context_sentence = context_sentence
+ self.start_ending = start_ending
+ self.endings = [
+ ending_0,
+ ending_1,
+ ending_2,
+ ending_3,
+ ]
+ self.label = label
+
+ def __str__(self):
+ return self.__repr__()
+
+ def __repr__(self):
+ l = [
+ "swag_id: {}".format(self.swag_id),
+ "context_sentence: {}".format(self.context_sentence),
+ "start_ending: {}".format(self.start_ending),
+ "ending_0: {}".format(self.endings[0]),
+ "ending_1: {}".format(self.endings[1]),
+ "ending_2: {}".format(self.endings[2]),
+ "ending_3: {}".format(self.endings[3]),
+ ]
+
+ if self.label is not None:
+ l.append("label: {}".format(self.label))
+
+ return ", ".join(l)
+
+
+class InputFeatures(object):
+ def __init__(self,
+ example_id,
+ choices_features,
+ label
+
+ ):
+ self.example_id = example_id
+ self.choices_features = [
+ {
+ 'input_ids': input_ids,
+ 'input_mask': input_mask,
+ 'segment_ids': segment_ids
+ }
+ for _, input_ids, input_mask, segment_ids in choices_features
+ ]
+ self.label = label
+
+
+def read_swag_examples(input_file, is_training):
+ with open(input_file, 'r', encoding='utf-8') as f:
+ reader = csv.reader(f)
+ lines = []
+ for line in reader:
+ if sys.version_info[0] == 2:
+ line = list(unicode(cell, 'utf-8') for cell in line)
+ lines.append(line)
+
+ if is_training and lines[0][-1] != 'label':
+ raise ValueError(
+ "For training, the input file must contain a label column."
+ )
+
+ examples = [
+ SwagExample(
+ swag_id = line[2],
+ context_sentence = line[4],
+ start_ending = line[5], # in the swag dataset, the
+ # common beginning of each
+ # choice is stored in "sent2".
+ ending_0 = line[7],
+ ending_1 = line[8],
+ ending_2 = line[9],
+ ending_3 = line[10],
+ label = int(line[11]) if is_training else None
+ ) for line in lines[1:] # we skip the line with the column names
+ ]
+
+ return examples
+
+def convert_examples_to_features(examples, tokenizer, max_seq_length,
+ is_training):
+ """Loads a data file into a list of `InputBatch`s."""
+
+ # Swag is a multiple choice task. To perform this task using Bert,
+ # we will use the formatting proposed in "Improving Language
+ # Understanding by Generative Pre-Training" and suggested by
+ # @jacobdevlin-google in this issue
+ # https://github.com/google-research/bert/issues/38.
+ #
+ # Each choice will correspond to a sample on which we run the
+ # inference. For a given Swag example, we will create the 4
+ # following inputs:
+ # - [CLS] context [SEP] choice_1 [SEP]
+ # - [CLS] context [SEP] choice_2 [SEP]
+ # - [CLS] context [SEP] choice_3 [SEP]
+ # - [CLS] context [SEP] choice_4 [SEP]
+ # The model will output a single value for each input. To get the
+ # final decision of the model, we will run a softmax over these 4
+ # outputs.
+ features = []
+ for example_index, example in enumerate(examples):
+ context_tokens = tokenizer.tokenize(example.context_sentence)
+ start_ending_tokens = tokenizer.tokenize(example.start_ending)
+
+ choices_features = []
+ for ending_index, ending in enumerate(example.endings):
+ # We create a copy of the context tokens in order to be
+ # able to shrink it according to ending_tokens
+ context_tokens_choice = context_tokens[:]
+ ending_tokens = start_ending_tokens + tokenizer.tokenize(ending)
+ # Modifies `context_tokens_choice` and `ending_tokens` in
+ # place so that the total length is less than the
+ # specified length. Account for [CLS], [SEP], [SEP] with
+ # "- 3"
+ _truncate_seq_pair(context_tokens_choice, ending_tokens, max_seq_length - 3)
+
+ tokens = ["[CLS]"] + context_tokens_choice + ["[SEP]"] + ending_tokens + ["[SEP]"]
+ segment_ids = [0] * (len(context_tokens_choice) + 2) + [1] * (len(ending_tokens) + 1)
+
+ input_ids = tokenizer.convert_tokens_to_ids(tokens)
+ input_mask = [1] * len(input_ids)
+
+ # Zero-pad up to the sequence length.
+ padding = [0] * (max_seq_length - len(input_ids))
+ input_ids += padding
+ input_mask += padding
+ segment_ids += padding
+
+ assert len(input_ids) == max_seq_length
+ assert len(input_mask) == max_seq_length
+ assert len(segment_ids) == max_seq_length
+
+ choices_features.append((tokens, input_ids, input_mask, segment_ids))
+
+ label = example.label
+ if example_index < 5:
+ logger.info("*** Example ***")
+ logger.info("swag_id: {}".format(example.swag_id))
+ for choice_idx, (tokens, input_ids, input_mask, segment_ids) in enumerate(choices_features):
+ logger.info("choice: {}".format(choice_idx))
+ logger.info("tokens: {}".format(' '.join(tokens)))
+ logger.info("input_ids: {}".format(' '.join(map(str, input_ids))))
+ logger.info("input_mask: {}".format(' '.join(map(str, input_mask))))
+ logger.info("segment_ids: {}".format(' '.join(map(str, segment_ids))))
+ if is_training:
+ logger.info("label: {}".format(label))
+
+ features.append(
+ InputFeatures(
+ example_id = example.swag_id,
+ choices_features = choices_features,
+ label = label
+ )
+ )
+
+ return features
+
+def _truncate_seq_pair(tokens_a, tokens_b, max_length):
+ """Truncates a sequence pair in place to the maximum length."""
+
+ # This is a simple heuristic which will always truncate the longer sequence
+ # one token at a time. This makes more sense than truncating an equal percent
+ # of tokens from each, since if one sequence is very short then each token
+ # that's truncated likely contains more information than a longer sequence.
+ while True:
+ total_length = len(tokens_a) + len(tokens_b)
+ if total_length <= max_length:
+ break
+ if len(tokens_a) > len(tokens_b):
+ tokens_a.pop()
+ else:
+ tokens_b.pop()
+
+def accuracy(out, labels):
+ outputs = np.argmax(out, axis=1)
+ return np.sum(outputs == labels)
+
+def select_field(features, field):
+ return [
+ [
+ choice[field]
+ for choice in feature.choices_features
+ ]
+ for feature in features
+ ]
+
+def main():
+ parser = argparse.ArgumentParser()
+
+ ## Required parameters
+ parser.add_argument("--data_dir",
+ default=None,
+ type=str,
+ required=True,
+ help="The input data dir. Should contain the .csv files (or other data files) for the task.")
+ parser.add_argument("--bert_model", default=None, type=str, required=True,
+ help="Bert pre-trained model selected in the list: bert-base-uncased, "
+ "bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, "
+ "bert-base-multilingual-cased, bert-base-chinese.")
+ parser.add_argument("--output_dir",
+ default=None,
+ type=str,
+ required=True,
+ help="The output directory where the model checkpoints will be written.")
+
+ ## Other parameters
+ parser.add_argument("--max_seq_length",
+ default=128,
+ type=int,
+ help="The maximum total input sequence length after WordPiece tokenization. \n"
+ "Sequences longer than this will be truncated, and sequences shorter \n"
+ "than this will be padded.")
+ parser.add_argument("--do_train",
+ action='store_true',
+ help="Whether to run training.")
+ parser.add_argument("--do_eval",
+ action='store_true',
+ help="Whether to run eval on the dev set.")
+ parser.add_argument("--do_lower_case",
+ action='store_true',
+ help="Set this flag if you are using an uncased model.")
+ parser.add_argument("--train_batch_size",
+ default=32,
+ type=int,
+ help="Total batch size for training.")
+ parser.add_argument("--eval_batch_size",
+ default=8,
+ type=int,
+ help="Total batch size for eval.")
+ parser.add_argument("--learning_rate",
+ default=5e-5,
+ type=float,
+ help="The initial learning rate for Adam.")
+ parser.add_argument("--num_train_epochs",
+ default=3.0,
+ type=float,
+ help="Total number of training epochs to perform.")
+ parser.add_argument("--warmup_proportion",
+ default=0.1,
+ type=float,
+ help="Proportion of training to perform linear learning rate warmup for. "
+ "E.g., 0.1 = 10%% of training.")
+ parser.add_argument("--no_cuda",
+ action='store_true',
+ help="Whether not to use CUDA when available")
+ parser.add_argument("--local_rank",
+ type=int,
+ default=-1,
+ help="local_rank for distributed training on gpus")
+ parser.add_argument('--seed',
+ type=int,
+ default=42,
+ help="random seed for initialization")
+ parser.add_argument('--gradient_accumulation_steps',
+ type=int,
+ default=1,
+ help="Number of updates steps to accumulate before performing a backward/update pass.")
+ parser.add_argument('--fp16',
+ action='store_true',
+ help="Whether to use 16-bit float precision instead of 32-bit")
+ parser.add_argument('--loss_scale',
+ type=float, default=0,
+ help="Loss scaling to improve fp16 numeric stability. Only used when fp16 set to True.\n"
+ "0 (default value): dynamic loss scaling.\n"
+ "Positive power of 2: static loss scaling value.\n")
+
+ args = parser.parse_args()
+
+ if args.local_rank == -1 or args.no_cuda:
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ n_gpu = torch.cuda.device_count()
+ else:
+ torch.cuda.set_device(args.local_rank)
+ device = torch.device("cuda", args.local_rank)
+ n_gpu = 1
+ # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+ torch.distributed.init_process_group(backend='nccl')
+ logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
+ device, n_gpu, bool(args.local_rank != -1), args.fp16))
+
+ if args.gradient_accumulation_steps < 1:
+ raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+ args.gradient_accumulation_steps))
+
+ args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+ random.seed(args.seed)
+ np.random.seed(args.seed)
+ torch.manual_seed(args.seed)
+ if n_gpu > 0:
+ torch.cuda.manual_seed_all(args.seed)
+
+ if not args.do_train and not args.do_eval:
+ raise ValueError("At least one of `do_train` or `do_eval` must be True.")
+
+ if os.path.exists(args.output_dir) and os.listdir(args.output_dir):
+ raise ValueError("Output directory ({}) already exists and is not empty.".format(args.output_dir))
+ if not os.path.exists(args.output_dir):
+ os.makedirs(args.output_dir)
+
+ tokenizer = BertTokenizer.from_pretrained(args.bert_model, do_lower_case=args.do_lower_case)
+
+ train_examples = None
+ num_train_optimization_steps = None
+ if args.do_train:
+ train_examples = read_swag_examples(os.path.join(args.data_dir, 'train.csv'), is_training = True)
+ num_train_optimization_steps = int(
+ len(train_examples) / args.train_batch_size / args.gradient_accumulation_steps) * args.num_train_epochs
+ if args.local_rank != -1:
+ num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()
+
+ # Prepare model
+ model = BertForMultipleChoice.from_pretrained(args.bert_model,
+ cache_dir=os.path.join(str(PYTORCH_PRETRAINED_BERT_CACHE), 'distributed_{}'.format(args.local_rank)),
+ num_choices=4)
+ if args.fp16:
+ model.half()
+ model.to(device)
+ if args.local_rank != -1:
+ try:
+ from apex.parallel import DistributedDataParallel as DDP
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ model = DDP(model)
+ elif n_gpu > 1:
+ model = torch.nn.DataParallel(model)
+
+ # Prepare optimizer
+ param_optimizer = list(model.named_parameters())
+
+ # hack to remove pooler, which is not used
+ # thus it produce None grad that break apex
+ param_optimizer = [n for n in param_optimizer if 'pooler' not in n[0]]
+
+ no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
+ optimizer_grouped_parameters = [
+ {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
+ {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
+ ]
+ if args.fp16:
+ try:
+ from apex.optimizers import FP16_Optimizer
+ from apex.optimizers import FusedAdam
+ except ImportError:
+ raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")
+
+ optimizer = FusedAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ bias_correction=False,
+ max_grad_norm=1.0)
+ if args.loss_scale == 0:
+ optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
+ else:
+ optimizer = FP16_Optimizer(optimizer, static_loss_scale=args.loss_scale)
+ else:
+ optimizer = BertAdam(optimizer_grouped_parameters,
+ lr=args.learning_rate,
+ warmup=args.warmup_proportion,
+ t_total=num_train_optimization_steps)
+
+ global_step = 0
+ if args.do_train:
+ train_features = convert_examples_to_features(
+ train_examples, tokenizer, args.max_seq_length, True)
+ logger.info("***** Running training *****")
+ logger.info(" Num examples = %d", len(train_examples))
+ logger.info(" Batch size = %d", args.train_batch_size)
+ logger.info(" Num steps = %d", num_train_optimization_steps)
+ all_input_ids = torch.tensor(select_field(train_features, 'input_ids'), dtype=torch.long)
+ all_input_mask = torch.tensor(select_field(train_features, 'input_mask'), dtype=torch.long)
+ all_segment_ids = torch.tensor(select_field(train_features, 'segment_ids'), dtype=torch.long)
+ all_label = torch.tensor([f.label for f in train_features], dtype=torch.long)
+ train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label)
+ if args.local_rank == -1:
+ train_sampler = RandomSampler(train_data)
+ else:
+ train_sampler = DistributedSampler(train_data)
+ train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=args.train_batch_size)
+
+ model.train()
+ for _ in trange(int(args.num_train_epochs), desc="Epoch"):
+ tr_loss = 0
+ nb_tr_examples, nb_tr_steps = 0, 0
+ for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")):
+ batch = tuple(t.to(device) for t in batch)
+ input_ids, input_mask, segment_ids, label_ids = batch
+ loss = model(input_ids, segment_ids, input_mask, label_ids)
+ if n_gpu > 1:
+ loss = loss.mean() # mean() to average on multi-gpu.
+ if args.fp16 and args.loss_scale != 1.0:
+ # rescale loss for fp16 training
+ # see https://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html
+ loss = loss * args.loss_scale
+ if args.gradient_accumulation_steps > 1:
+ loss = loss / args.gradient_accumulation_steps
+ tr_loss += loss.item()
+ nb_tr_examples += input_ids.size(0)
+ nb_tr_steps += 1
+
+ if args.fp16:
+ optimizer.backward(loss)
+ else:
+ loss.backward()
+ if (step + 1) % args.gradient_accumulation_steps == 0:
+ if args.fp16:
+ # modify learning rate with special warm up BERT uses
+ # if args.fp16 is False, BertAdam is used that handles this automatically
+ lr_this_step = args.learning_rate * warmup_linear(global_step/num_train_optimization_steps, args.warmup_proportion)
+ for param_group in optimizer.param_groups:
+ param_group['lr'] = lr_this_step
+ optimizer.step()
+ optimizer.zero_grad()
+ global_step += 1
+
+
+ if args.do_train:
+ # Save a trained model, configuration and tokenizer
+ model_to_save = model.module if hasattr(model, 'module') else model # Only save the model it-self
+
+ # If we save using the predefined names, we can load using `from_pretrained`
+ output_model_file = os.path.join(args.output_dir, WEIGHTS_NAME)
+ output_config_file = os.path.join(args.output_dir, CONFIG_NAME)
+
+ torch.save(model_to_save.state_dict(), output_model_file)
+ model_to_save.config.to_json_file(output_config_file)
+ tokenizer.save_vocabulary(args.output_dir)
+
+ # Load a trained model and vocabulary that you have fine-tuned
+ model = BertForMultipleChoice.from_pretrained(args.output_dir, num_choices=4)
+ tokenizer = BertTokenizer.from_pretrained(args.output_dir, do_lower_case=args.do_lower_case)
+ else:
+ model = BertForMultipleChoice.from_pretrained(args.bert_model, num_choices=4)
+ model.to(device)
+
+
+ if args.do_eval and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+ eval_examples = read_swag_examples(os.path.join(args.data_dir, 'val.csv'), is_training = True)
+ eval_features = convert_examples_to_features(
+ eval_examples, tokenizer, args.max_seq_length, True)
+ logger.info("***** Running evaluation *****")
+ logger.info(" Num examples = %d", len(eval_examples))
+ logger.info(" Batch size = %d", args.eval_batch_size)
+ all_input_ids = torch.tensor(select_field(eval_features, 'input_ids'), dtype=torch.long)
+ all_input_mask = torch.tensor(select_field(eval_features, 'input_mask'), dtype=torch.long)
+ all_segment_ids = torch.tensor(select_field(eval_features, 'segment_ids'), dtype=torch.long)
+ all_label = torch.tensor([f.label for f in eval_features], dtype=torch.long)
+ eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label)
+ # Run prediction for full data
+ eval_sampler = SequentialSampler(eval_data)
+ eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size)
+
+ model.eval()
+ eval_loss, eval_accuracy = 0, 0
+ nb_eval_steps, nb_eval_examples = 0, 0
+ for input_ids, input_mask, segment_ids, label_ids in tqdm(eval_dataloader, desc="Evaluating"):
+ input_ids = input_ids.to(device)
+ input_mask = input_mask.to(device)
+ segment_ids = segment_ids.to(device)
+ label_ids = label_ids.to(device)
+
+ with torch.no_grad():
+ tmp_eval_loss = model(input_ids, segment_ids, input_mask, label_ids)
+ logits = model(input_ids, segment_ids, input_mask)
+
+ logits = logits.detach().cpu().numpy()
+ label_ids = label_ids.to('cpu').numpy()
+ tmp_eval_accuracy = accuracy(logits, label_ids)
+
+ eval_loss += tmp_eval_loss.mean().item()
+ eval_accuracy += tmp_eval_accuracy
+
+ nb_eval_examples += input_ids.size(0)
+ nb_eval_steps += 1
+
+ eval_loss = eval_loss / nb_eval_steps
+ eval_accuracy = eval_accuracy / nb_eval_examples
+
+ result = {'eval_loss': eval_loss,
+ 'eval_accuracy': eval_accuracy,
+ 'global_step': global_step,
+ 'loss': tr_loss/nb_tr_steps}
+
+ output_eval_file = os.path.join(args.output_dir, "eval_results.txt")
+ with open(output_eval_file, "w") as writer:
+ logger.info("***** Eval results *****")
+ for key in sorted(result.keys()):
+ logger.info(" %s = %s", key, str(result[key]))
+ writer.write("%s = %s\n" % (key, str(result[key])))
+
+
+if __name__ == "__main__":
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_transfo_xl.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_transfo_xl.py
new file mode 100644
index 0000000000000000000000000000000000000000..0ea7b320536f5320ff919d1a660d6b14e4f73bf7
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/examples/run_transfo_xl.py
@@ -0,0 +1,153 @@
+# coding=utf-8
+# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" PyTorch Transformer XL model evaluation script.
+ Adapted from https://github.com/kimiyoung/transformer-xl.
+ In particular https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/eval.py
+
+ This script with default values evaluates a pretrained Transformer-XL on WikiText 103
+"""
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+import logging
+import time
+import math
+
+import torch
+
+from pytorch_pretrained_bert import TransfoXLLMHeadModel, TransfoXLCorpus, TransfoXLTokenizer
+
+logging.basicConfig(format = '%(asctime)s - %(levelname)s - %(name)s - %(message)s',
+ datefmt = '%m/%d/%Y %H:%M:%S',
+ level = logging.INFO)
+logger = logging.getLogger(__name__)
+
+def main():
+ parser = argparse.ArgumentParser(description='PyTorch Transformer Language Model')
+ parser.add_argument('--model_name', type=str, default='transfo-xl-wt103',
+ help='pretrained model name')
+ parser.add_argument('--split', type=str, default='test',
+ choices=['all', 'valid', 'test'],
+ help='which split to evaluate')
+ parser.add_argument('--batch_size', type=int, default=10,
+ help='batch size')
+ parser.add_argument('--tgt_len', type=int, default=128,
+ help='number of tokens to predict')
+ parser.add_argument('--ext_len', type=int, default=0,
+ help='length of the extended context')
+ parser.add_argument('--mem_len', type=int, default=1600,
+ help='length of the retained previous heads')
+ parser.add_argument('--clamp_len', type=int, default=1000,
+ help='max positional embedding index')
+ parser.add_argument('--no_cuda', action='store_true',
+ help='Do not use CUDA even though CUA is available')
+ parser.add_argument('--work_dir', type=str, required=True,
+ help='path to the work_dir')
+ parser.add_argument('--no_log', action='store_true',
+ help='do not log the eval result')
+ parser.add_argument('--same_length', action='store_true',
+ help='set same length attention with masking')
+ parser.add_argument('--server_ip', type=str, default='', help="Can be used for distant debugging.")
+ parser.add_argument('--server_port', type=str, default='', help="Can be used for distant debugging.")
+ args = parser.parse_args()
+ assert args.ext_len >= 0, 'extended context length must be non-negative'
+
+ if args.server_ip and args.server_port:
+ # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
+ import ptvsd
+ print("Waiting for debugger attach")
+ ptvsd.enable_attach(address=(args.server_ip, args.server_port), redirect_output=True)
+ ptvsd.wait_for_attach()
+
+ device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")
+ logger.info("device: {}".format(device))
+
+ # Load a pre-processed dataset
+ # You can also build the corpus yourself using TransfoXLCorpus methods
+ # The pre-processing involve computing word frequencies to prepare the Adaptive input and SoftMax
+ # and tokenizing the dataset
+ # The pre-processed corpus is a convertion (using the conversion script )
+ tokenizer = TransfoXLTokenizer.from_pretrained(args.model_name)
+ corpus = TransfoXLCorpus.from_pretrained(args.model_name)
+ ntokens = len(corpus.vocab)
+
+ va_iter = corpus.get_iterator('valid', args.batch_size, args.tgt_len,
+ device=device, ext_len=args.ext_len)
+ te_iter = corpus.get_iterator('test', args.batch_size, args.tgt_len,
+ device=device, ext_len=args.ext_len)
+
+ # Load a pre-trained model
+ model = TransfoXLLMHeadModel.from_pretrained(args.model_name)
+ model = model.to(device)
+
+ logger.info('Evaluating with bsz {} tgt_len {} ext_len {} mem_len {} clamp_len {}'.format(
+ args.batch_size, args.tgt_len, args.ext_len, args.mem_len, args.clamp_len))
+
+ model.reset_length(args.tgt_len, args.ext_len, args.mem_len)
+ if args.clamp_len > 0:
+ model.clamp_len = args.clamp_len
+ if args.same_length:
+ model.same_length = True
+
+ ###############################################################################
+ # Evaluation code
+ ###############################################################################
+ def evaluate(eval_iter):
+ # Turn on evaluation mode which disables dropout.
+ model.eval()
+ total_len, total_loss = 0, 0.
+ start_time = time.time()
+ with torch.no_grad():
+ mems = None
+ for idx, (data, target, seq_len) in enumerate(eval_iter):
+ ret = model(data, target, mems)
+ loss, mems = ret
+ loss = loss.mean()
+ total_loss += seq_len * loss.item()
+ total_len += seq_len
+ total_time = time.time() - start_time
+ logger.info('Time : {:.2f}s, {:.2f}ms/segment'.format(
+ total_time, 1000 * total_time / (idx+1)))
+ return total_loss / total_len
+
+ # Run on test data.
+ if args.split == 'all':
+ test_loss = evaluate(te_iter)
+ valid_loss = evaluate(va_iter)
+ elif args.split == 'valid':
+ valid_loss = evaluate(va_iter)
+ test_loss = None
+ elif args.split == 'test':
+ test_loss = evaluate(te_iter)
+ valid_loss = None
+
+ def format_log(loss, split):
+ log_str = '| {0} loss {1:5.2f} | {0} ppl {2:9.3f} '.format(
+ split, loss, math.exp(loss))
+ return log_str
+
+ log_str = ''
+ if valid_loss is not None:
+ log_str += format_log(valid_loss, 'valid')
+ if test_loss is not None:
+ log_str += format_log(test_loss, 'test')
+
+ logger.info('=' * 100)
+ logger.info(log_str)
+ logger.info('=' * 100)
+
+if __name__ == '__main__':
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/hubconf.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/hubconf.py
new file mode 100644
index 0000000000000000000000000000000000000000..755e181d201f3f84621070874b088b1e4c485181
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/hubconf.py
@@ -0,0 +1,187 @@
+from pytorch_pretrained_bert.tokenization import BertTokenizer
+from pytorch_pretrained_bert.modeling import (
+ BertModel,
+ BertForNextSentencePrediction,
+ BertForMaskedLM,
+ BertForMultipleChoice,
+ BertForPreTraining,
+ BertForQuestionAnswering,
+ BertForSequenceClassification,
+ BertForTokenClassification,
+ )
+
+dependencies = ['torch', 'tqdm', 'boto3', 'requests', 'regex']
+
+# A lot of models share the same param doc. Use a decorator
+# to save typing
+bert_docstring = """
+ Params:
+ pretrained_model_name_or_path: either:
+ - a str with the name of a pre-trained model to load
+ . `bert-base-uncased`
+ . `bert-large-uncased`
+ . `bert-base-cased`
+ . `bert-large-cased`
+ . `bert-base-multilingual-uncased`
+ . `bert-base-multilingual-cased`
+ . `bert-base-chinese`
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . `pytorch_model.bin` a PyTorch dump of a BertForPreTraining
+ instance
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . `model.chkpt` a TensorFlow checkpoint
+ from_tf: should we load the weights from a locally saved TensorFlow
+ checkpoint
+ cache_dir: an optional path to a folder in which the pre-trained models
+ will be cached.
+ state_dict: an optional state dictionnary
+ (collections.OrderedDict object) to use instead of Google
+ pre-trained models
+ *inputs, **kwargs: additional input for the specific Bert class
+ (ex: num_labels for BertForSequenceClassification)
+"""
+
+
+def _append_from_pretrained_docstring(docstr):
+ def docstring_decorator(fn):
+ fn.__doc__ = fn.__doc__ + docstr
+ return fn
+ return docstring_decorator
+
+
+def bertTokenizer(*args, **kwargs):
+ """
+ Instantiate a BertTokenizer from a pre-trained/customized vocab file
+ Args:
+ pretrained_model_name_or_path: Path to pretrained model archive
+ or one of pre-trained vocab configs below.
+ * bert-base-uncased
+ * bert-large-uncased
+ * bert-base-cased
+ * bert-large-cased
+ * bert-base-multilingual-uncased
+ * bert-base-multilingual-cased
+ * bert-base-chinese
+ Keyword args:
+ cache_dir: an optional path to a specific directory to download and cache
+ the pre-trained model weights.
+ Default: None
+ do_lower_case: Whether to lower case the input.
+ Only has an effect when do_wordpiece_only=False
+ Default: True
+ do_basic_tokenize: Whether to do basic tokenization before wordpiece.
+ Default: True
+ max_len: An artificial maximum length to truncate tokenized sequences to;
+ Effective maximum length is always the minimum of this
+ value (if specified) and the underlying BERT model's
+ sequence length.
+ Default: None
+ never_split: List of tokens which will never be split during tokenization.
+ Only has an effect when do_wordpiece_only=False
+ Default: ["[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]"]
+
+ Example:
+ >>> sentence = 'Hello, World!'
+ >>> tokenizer = torch.hub.load('ailzhang/pytorch-pretrained-BERT:hubconf', 'bertTokenizer', 'bert-base-cased', do_basic_tokenize=False, force_reload=False)
+ >>> toks = tokenizer.tokenize(sentence)
+ ['Hello', '##,', 'World', '##!']
+ >>> ids = tokenizer.convert_tokens_to_ids(toks)
+ [8667, 28136, 1291, 28125]
+ """
+ tokenizer = BertTokenizer.from_pretrained(*args, **kwargs)
+ return tokenizer
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertModel(*args, **kwargs):
+ """
+ BertModel is the basic BERT Transformer model with a layer of summed token,
+ position and sequence embeddings followed by a series of identical
+ self-attention blocks (12 for BERT-base, 24 for BERT-large).
+ """
+ model = BertModel.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForNextSentencePrediction(*args, **kwargs):
+ """
+ BERT model with next sentence prediction head.
+ This module comprises the BERT model followed by the next sentence
+ classification head.
+ """
+ model = BertForNextSentencePrediction.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForPreTraining(*args, **kwargs):
+ """
+ BERT model with pre-training heads.
+ This module comprises the BERT model followed by the two pre-training heads
+ - the masked language modeling head, and
+ - the next sentence classification head.
+ """
+ model = BertForPreTraining.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForMaskedLM(*args, **kwargs):
+ """
+ BertForMaskedLM includes the BertModel Transformer followed by the
+ (possibly) pre-trained masked language modeling head.
+ """
+ model = BertForMaskedLM.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForSequenceClassification(*args, **kwargs):
+ """
+ BertForSequenceClassification is a fine-tuning model that includes
+ BertModel and a sequence-level (sequence or pair of sequences) classifier
+ on top of the BertModel.
+
+ The sequence-level classifier is a linear layer that takes as input the
+ last hidden state of the first character in the input sequence
+ (see Figures 3a and 3b in the BERT paper).
+ """
+ model = BertForSequenceClassification.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForMultipleChoice(*args, **kwargs):
+ """
+ BertForMultipleChoice is a fine-tuning model that includes BertModel and a
+ linear layer on top of the BertModel.
+ """
+ model = BertForMultipleChoice.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForQuestionAnswering(*args, **kwargs):
+ """
+ BertForQuestionAnswering is a fine-tuning model that includes BertModel
+ with a token-level classifiers on top of the full sequence of last hidden
+ states.
+ """
+ model = BertForQuestionAnswering.from_pretrained(*args, **kwargs)
+ return model
+
+
+@_append_from_pretrained_docstring(bert_docstring)
+def bertForTokenClassification(*args, **kwargs):
+ """
+ BertForTokenClassification is a fine-tuning model that includes BertModel
+ and a token-level classifier on top of the BertModel.
+
+ The token-level classifier is a linear layer that takes as input the last
+ hidden state of the sequence.
+ """
+ model = BertForTokenClassification.from_pretrained(*args, **kwargs)
+ return model
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..67c56ead389b640bfd4b2df505587034fcb15634
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-MLM-NSP.ipynb
@@ -0,0 +1,4815 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Comparing TensorFlow (original) and PyTorch models\n",
+ "\n",
+ "You can use this small notebook to check the conversion of the model's weights from the TensorFlow model to the PyTorch model. In the following, we compare the weights of the last layer on a simple example (in `input.txt`) but both models returns all the hidden layers so you can check every stage of the model.\n",
+ "\n",
+ "To run this notebook, follow these instructions:\n",
+ "- make sure that your Python environment has both TensorFlow and PyTorch installed,\n",
+ "- download the original TensorFlow implementation,\n",
+ "- download a pre-trained TensorFlow model as indicaded in the TensorFlow implementation readme,\n",
+ "- run the script `convert_tf_checkpoint_to_pytorch.py` as indicated in the `README` to convert the pre-trained TensorFlow model to PyTorch.\n",
+ "\n",
+ "If needed change the relative paths indicated in this notebook (at the beggining of Sections 1 and 2) to point to the relevent models and code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:26.999106Z",
+ "start_time": "2018-11-16T10:02:26.985709Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "os.chdir('../')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1/ TensorFlow code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:27.664528Z",
+ "start_time": "2018-11-16T10:02:27.651019Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "original_tf_inplem_dir = \"./tensorflow_code/\"\n",
+ "model_dir = \"../google_models/uncased_L-12_H-768_A-12/\"\n",
+ "\n",
+ "vocab_file = model_dir + \"vocab.txt\"\n",
+ "bert_config_file = model_dir + \"bert_config.json\"\n",
+ "init_checkpoint = model_dir + \"bert_model.ckpt\"\n",
+ "\n",
+ "input_file = \"./samples/input.txt\"\n",
+ "max_seq_length = 128\n",
+ "max_predictions_per_seq = 20\n",
+ "\n",
+ "masked_lm_positions = [6]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:30.202182Z",
+ "start_time": "2018-11-16T10:02:28.112570Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import importlib.util\n",
+ "import sys\n",
+ "import tensorflow as tf\n",
+ "import pytorch_pretrained_bert as ppb\n",
+ "\n",
+ "def del_all_flags(FLAGS):\n",
+ " flags_dict = FLAGS._flags() \n",
+ " keys_list = [keys for keys in flags_dict] \n",
+ " for keys in keys_list:\n",
+ " FLAGS.__delattr__(keys)\n",
+ "\n",
+ "del_all_flags(tf.flags.FLAGS)\n",
+ "import tensorflow_code.extract_features as ef\n",
+ "del_all_flags(tf.flags.FLAGS)\n",
+ "import tensorflow_code.modeling as tfm\n",
+ "del_all_flags(tf.flags.FLAGS)\n",
+ "import tensorflow_code.tokenization as tft\n",
+ "del_all_flags(tf.flags.FLAGS)\n",
+ "import tensorflow_code.run_pretraining as rp\n",
+ "del_all_flags(tf.flags.FLAGS)\n",
+ "import tensorflow_code.create_pretraining_data as cpp"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:30.238027Z",
+ "start_time": "2018-11-16T10:02:30.204943Z"
+ },
+ "code_folding": [
+ 15
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "import re\n",
+ "class InputExample(object):\n",
+ " \"\"\"A single instance example.\"\"\"\n",
+ "\n",
+ " def __init__(self, tokens, segment_ids, masked_lm_positions,\n",
+ " masked_lm_labels, is_random_next):\n",
+ " self.tokens = tokens\n",
+ " self.segment_ids = segment_ids\n",
+ " self.masked_lm_positions = masked_lm_positions\n",
+ " self.masked_lm_labels = masked_lm_labels\n",
+ " self.is_random_next = is_random_next\n",
+ " def __repr__(self):\n",
+ " return '\\n'.join(k + \":\" + str(v) for k, v in self.__dict__.items())\n",
+ "\n",
+ "\n",
+ "def read_examples(input_file, tokenizer, masked_lm_positions):\n",
+ " \"\"\"Read a list of `InputExample`s from an input file.\"\"\"\n",
+ " examples = []\n",
+ " unique_id = 0\n",
+ " with tf.gfile.GFile(input_file, \"r\") as reader:\n",
+ " while True:\n",
+ " line = reader.readline()\n",
+ " if not line:\n",
+ " break\n",
+ " line = line.strip()\n",
+ " text_a = None\n",
+ " text_b = None\n",
+ " m = re.match(r\"^(.*) \\|\\|\\| (.*)$\", line)\n",
+ " if m is None:\n",
+ " text_a = line\n",
+ " else:\n",
+ " text_a = m.group(1)\n",
+ " text_b = m.group(2)\n",
+ " tokens_a = tokenizer.tokenize(text_a)\n",
+ " tokens_b = None\n",
+ " if text_b:\n",
+ " tokens_b = tokenizer.tokenize(text_b)\n",
+ " tokens = tokens_a + tokens_b\n",
+ " masked_lm_labels = []\n",
+ " for m_pos in masked_lm_positions:\n",
+ " masked_lm_labels.append(tokens[m_pos])\n",
+ " tokens[m_pos] = '[MASK]'\n",
+ " examples.append(\n",
+ " InputExample(\n",
+ " tokens = tokens,\n",
+ " segment_ids = [0] * len(tokens_a) + [1] * len(tokens_b),\n",
+ " masked_lm_positions = masked_lm_positions,\n",
+ " masked_lm_labels = masked_lm_labels,\n",
+ " is_random_next = False))\n",
+ " unique_id += 1\n",
+ " return examples"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:30.304018Z",
+ "start_time": "2018-11-16T10:02:30.240189Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tokens:['who', 'was', 'jim', 'henson', '?', 'jim', '[MASK]', 'was', 'a', 'puppet', '##eer']\n",
+ "segment_ids:[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]\n",
+ "masked_lm_positions:[6]\n",
+ "masked_lm_labels:['henson']\n",
+ "is_random_next:False\n"
+ ]
+ }
+ ],
+ "source": [
+ "bert_config = tfm.BertConfig.from_json_file(bert_config_file)\n",
+ "tokenizer = ppb.BertTokenizer(\n",
+ " vocab_file=vocab_file, do_lower_case=True)\n",
+ "examples = read_examples(input_file, tokenizer, masked_lm_positions=masked_lm_positions)\n",
+ "\n",
+ "print(examples[0])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:33.324167Z",
+ "start_time": "2018-11-16T10:02:33.291909Z"
+ },
+ "code_folding": [
+ 16
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "class InputFeatures(object):\n",
+ " \"\"\"A single set of features of data.\"\"\"\n",
+ "\n",
+ " def __init__(self, input_ids, input_mask, segment_ids, masked_lm_positions,\n",
+ " masked_lm_ids, masked_lm_weights, next_sentence_label):\n",
+ " self.input_ids = input_ids\n",
+ " self.input_mask = input_mask\n",
+ " self.segment_ids = segment_ids\n",
+ " self.masked_lm_positions = masked_lm_positions\n",
+ " self.masked_lm_ids = masked_lm_ids\n",
+ " self.masked_lm_weights = masked_lm_weights\n",
+ " self.next_sentence_labels = next_sentence_label\n",
+ "\n",
+ " def __repr__(self):\n",
+ " return '\\n'.join(k + \":\" + str(v) for k, v in self.__dict__.items())\n",
+ "\n",
+ "def pretraining_convert_examples_to_features(instances, tokenizer, max_seq_length,\n",
+ " max_predictions_per_seq):\n",
+ " \"\"\"Create TF example files from `TrainingInstance`s.\"\"\"\n",
+ " features = []\n",
+ " for (inst_index, instance) in enumerate(instances):\n",
+ " input_ids = tokenizer.convert_tokens_to_ids(instance.tokens)\n",
+ " input_mask = [1] * len(input_ids)\n",
+ " segment_ids = list(instance.segment_ids)\n",
+ " assert len(input_ids) <= max_seq_length\n",
+ "\n",
+ " while len(input_ids) < max_seq_length:\n",
+ " input_ids.append(0)\n",
+ " input_mask.append(0)\n",
+ " segment_ids.append(0)\n",
+ "\n",
+ " assert len(input_ids) == max_seq_length\n",
+ " assert len(input_mask) == max_seq_length\n",
+ " assert len(segment_ids) == max_seq_length\n",
+ "\n",
+ " masked_lm_positions = list(instance.masked_lm_positions)\n",
+ " masked_lm_ids = tokenizer.convert_tokens_to_ids(instance.masked_lm_labels)\n",
+ " masked_lm_weights = [1.0] * len(masked_lm_ids)\n",
+ "\n",
+ " while len(masked_lm_positions) < max_predictions_per_seq:\n",
+ " masked_lm_positions.append(0)\n",
+ " masked_lm_ids.append(0)\n",
+ " masked_lm_weights.append(0.0)\n",
+ "\n",
+ " next_sentence_label = 1 if instance.is_random_next else 0\n",
+ "\n",
+ " features.append(\n",
+ " InputFeatures(input_ids, input_mask, segment_ids,\n",
+ " masked_lm_positions, masked_lm_ids,\n",
+ " masked_lm_weights, next_sentence_label))\n",
+ "\n",
+ " if inst_index < 5:\n",
+ " tf.logging.info(\"*** Example ***\")\n",
+ " tf.logging.info(\"tokens: %s\" % \" \".join(\n",
+ " [str(x) for x in instance.tokens]))\n",
+ " tf.logging.info(\"features: %s\" % str(features[-1]))\n",
+ " return features"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:34.185367Z",
+ "start_time": "2018-11-16T10:02:34.155046Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:*** Example ***\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:34 - INFO - tensorflow - *** Example ***\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:tokens: who was jim henson ? jim [MASK] was a puppet ##eer\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:34 - INFO - tensorflow - tokens: who was jim henson ? jim [MASK] was a puppet ##eer\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:features: input_ids:[2040, 2001, 3958, 27227, 1029, 3958, 103, 2001, 1037, 13997, 11510, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "input_mask:[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "segment_ids:[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_positions:[6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_ids:[27227, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_weights:[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n",
+ "next_sentence_labels:0\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:34 - INFO - tensorflow - features: input_ids:[2040, 2001, 3958, 27227, 1029, 3958, 103, 2001, 1037, 13997, 11510, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "input_mask:[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "segment_ids:[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_positions:[6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_ids:[27227, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]\n",
+ "masked_lm_weights:[1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]\n",
+ "next_sentence_labels:0\n"
+ ]
+ }
+ ],
+ "source": [
+ "features = pretraining_convert_examples_to_features(\n",
+ " instances=examples, max_seq_length=max_seq_length, \n",
+ " max_predictions_per_seq=max_predictions_per_seq, tokenizer=tokenizer)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:34.912005Z",
+ "start_time": "2018-11-16T10:02:34.882111Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "def input_fn_builder(features, seq_length, max_predictions_per_seq, tokenizer):\n",
+ " \"\"\"Creates an `input_fn` closure to be passed to TPUEstimator.\"\"\"\n",
+ "\n",
+ " all_input_ids = []\n",
+ " all_input_mask = []\n",
+ " all_segment_ids = []\n",
+ " all_masked_lm_positions = []\n",
+ " all_masked_lm_ids = []\n",
+ " all_masked_lm_weights = []\n",
+ " all_next_sentence_labels = []\n",
+ "\n",
+ " for feature in features:\n",
+ " all_input_ids.append(feature.input_ids)\n",
+ " all_input_mask.append(feature.input_mask)\n",
+ " all_segment_ids.append(feature.segment_ids)\n",
+ " all_masked_lm_positions.append(feature.masked_lm_positions)\n",
+ " all_masked_lm_ids.append(feature.masked_lm_ids)\n",
+ " all_masked_lm_weights.append(feature.masked_lm_weights)\n",
+ " all_next_sentence_labels.append(feature.next_sentence_labels)\n",
+ "\n",
+ " def input_fn(params):\n",
+ " \"\"\"The actual input function.\"\"\"\n",
+ " batch_size = params[\"batch_size\"]\n",
+ "\n",
+ " num_examples = len(features)\n",
+ "\n",
+ " # This is for demo purposes and does NOT scale to large data sets. We do\n",
+ " # not use Dataset.from_generator() because that uses tf.py_func which is\n",
+ " # not TPU compatible. The right way to load data is with TFRecordReader.\n",
+ " d = tf.data.Dataset.from_tensor_slices({\n",
+ " \"input_ids\":\n",
+ " tf.constant(\n",
+ " all_input_ids, shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"input_mask\":\n",
+ " tf.constant(\n",
+ " all_input_mask,\n",
+ " shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"segment_ids\":\n",
+ " tf.constant(\n",
+ " all_segment_ids,\n",
+ " shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"masked_lm_positions\":\n",
+ " tf.constant(\n",
+ " all_masked_lm_positions,\n",
+ " shape=[num_examples, max_predictions_per_seq],\n",
+ " dtype=tf.int32),\n",
+ " \"masked_lm_ids\":\n",
+ " tf.constant(\n",
+ " all_masked_lm_ids,\n",
+ " shape=[num_examples, max_predictions_per_seq],\n",
+ " dtype=tf.int32),\n",
+ " \"masked_lm_weights\":\n",
+ " tf.constant(\n",
+ " all_masked_lm_weights,\n",
+ " shape=[num_examples, max_predictions_per_seq],\n",
+ " dtype=tf.float32),\n",
+ " \"next_sentence_labels\":\n",
+ " tf.constant(\n",
+ " all_next_sentence_labels,\n",
+ " shape=[num_examples, 1],\n",
+ " dtype=tf.int32),\n",
+ " })\n",
+ "\n",
+ " d = d.batch(batch_size=batch_size, drop_remainder=False)\n",
+ " return d\n",
+ "\n",
+ " return input_fn\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:35.671603Z",
+ "start_time": "2018-11-16T10:02:35.626167Z"
+ },
+ "code_folding": [
+ 64,
+ 77
+ ]
+ },
+ "outputs": [],
+ "source": [
+ "def model_fn_builder(bert_config, init_checkpoint, learning_rate,\n",
+ " num_train_steps, num_warmup_steps, use_tpu,\n",
+ " use_one_hot_embeddings):\n",
+ " \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
+ "\n",
+ " def model_fn(features, labels, mode, params): # pylint: disable=unused-argument\n",
+ " \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
+ "\n",
+ " tf.logging.info(\"*** Features ***\")\n",
+ " for name in sorted(features.keys()):\n",
+ " tf.logging.info(\" name = %s, shape = %s\" % (name, features[name].shape))\n",
+ "\n",
+ " input_ids = features[\"input_ids\"]\n",
+ " input_mask = features[\"input_mask\"]\n",
+ " segment_ids = features[\"segment_ids\"]\n",
+ " masked_lm_positions = features[\"masked_lm_positions\"]\n",
+ " masked_lm_ids = features[\"masked_lm_ids\"]\n",
+ " masked_lm_weights = features[\"masked_lm_weights\"]\n",
+ " next_sentence_labels = features[\"next_sentence_labels\"]\n",
+ "\n",
+ " is_training = (mode == tf.estimator.ModeKeys.TRAIN)\n",
+ "\n",
+ " model = tfm.BertModel(\n",
+ " config=bert_config,\n",
+ " is_training=is_training,\n",
+ " input_ids=input_ids,\n",
+ " input_mask=input_mask,\n",
+ " token_type_ids=segment_ids,\n",
+ " use_one_hot_embeddings=use_one_hot_embeddings)\n",
+ "\n",
+ " (masked_lm_loss,\n",
+ " masked_lm_example_loss, masked_lm_log_probs) = rp.get_masked_lm_output(\n",
+ " bert_config, model.get_sequence_output(), model.get_embedding_table(),\n",
+ " masked_lm_positions, masked_lm_ids, masked_lm_weights)\n",
+ "\n",
+ " (next_sentence_loss, next_sentence_example_loss,\n",
+ " next_sentence_log_probs) = rp.get_next_sentence_output(\n",
+ " bert_config, model.get_pooled_output(), next_sentence_labels)\n",
+ "\n",
+ " total_loss = masked_lm_loss + next_sentence_loss\n",
+ "\n",
+ " tvars = tf.trainable_variables()\n",
+ "\n",
+ " initialized_variable_names = {}\n",
+ " scaffold_fn = None\n",
+ " if init_checkpoint:\n",
+ " (assignment_map,\n",
+ " initialized_variable_names) = tfm.get_assigment_map_from_checkpoint(\n",
+ " tvars, init_checkpoint)\n",
+ " if use_tpu:\n",
+ "\n",
+ " def tpu_scaffold():\n",
+ " tf.train.init_from_checkpoint(init_checkpoint, assignment_map)\n",
+ " return tf.train.Scaffold()\n",
+ "\n",
+ " scaffold_fn = tpu_scaffold\n",
+ " else:\n",
+ " tf.train.init_from_checkpoint(init_checkpoint, assignment_map)\n",
+ "\n",
+ " tf.logging.info(\"**** Trainable Variables ****\")\n",
+ " for var in tvars:\n",
+ " init_string = \"\"\n",
+ " if var.name in initialized_variable_names:\n",
+ " init_string = \", *INIT_FROM_CKPT*\"\n",
+ " tf.logging.info(\" name = %s, shape = %s%s\", var.name, var.shape,\n",
+ " init_string)\n",
+ "\n",
+ " output_spec = None\n",
+ " if mode == tf.estimator.ModeKeys.TRAIN:\n",
+ " masked_lm_positions = features[\"masked_lm_positions\"]\n",
+ " masked_lm_ids = features[\"masked_lm_ids\"]\n",
+ " masked_lm_weights = features[\"masked_lm_weights\"]\n",
+ " next_sentence_labels = features[\"next_sentence_labels\"]\n",
+ " train_op = optimization.create_optimizer(\n",
+ " total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)\n",
+ "\n",
+ " output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
+ " mode=mode,\n",
+ " loss=total_loss,\n",
+ " train_op=train_op,\n",
+ " scaffold_fn=scaffold_fn)\n",
+ " elif mode == tf.estimator.ModeKeys.EVAL:\n",
+ " masked_lm_positions = features[\"masked_lm_positions\"]\n",
+ " masked_lm_ids = features[\"masked_lm_ids\"]\n",
+ " masked_lm_weights = features[\"masked_lm_weights\"]\n",
+ " next_sentence_labels = features[\"next_sentence_labels\"]\n",
+ "\n",
+ " def metric_fn(masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids,\n",
+ " masked_lm_weights, next_sentence_example_loss,\n",
+ " next_sentence_log_probs, next_sentence_labels):\n",
+ " \"\"\"Computes the loss and accuracy of the model.\"\"\"\n",
+ " masked_lm_log_probs = tf.reshape(masked_lm_log_probs,\n",
+ " [-1, masked_lm_log_probs.shape[-1]])\n",
+ " masked_lm_predictions = tf.argmax(\n",
+ " masked_lm_log_probs, axis=-1, output_type=tf.int32)\n",
+ " masked_lm_example_loss = tf.reshape(masked_lm_example_loss, [-1])\n",
+ " masked_lm_ids = tf.reshape(masked_lm_ids, [-1])\n",
+ " masked_lm_weights = tf.reshape(masked_lm_weights, [-1])\n",
+ " masked_lm_accuracy = tf.metrics.accuracy(\n",
+ " labels=masked_lm_ids,\n",
+ " predictions=masked_lm_predictions,\n",
+ " weights=masked_lm_weights)\n",
+ " masked_lm_mean_loss = tf.metrics.mean(\n",
+ " values=masked_lm_example_loss, weights=masked_lm_weights)\n",
+ "\n",
+ " next_sentence_log_probs = tf.reshape(\n",
+ " next_sentence_log_probs, [-1, next_sentence_log_probs.shape[-1]])\n",
+ " next_sentence_predictions = tf.argmax(\n",
+ " next_sentence_log_probs, axis=-1, output_type=tf.int32)\n",
+ " next_sentence_labels = tf.reshape(next_sentence_labels, [-1])\n",
+ " next_sentence_accuracy = tf.metrics.accuracy(\n",
+ " labels=next_sentence_labels, predictions=next_sentence_predictions)\n",
+ " next_sentence_mean_loss = tf.metrics.mean(\n",
+ " values=next_sentence_example_loss)\n",
+ "\n",
+ " return {\n",
+ " \"masked_lm_accuracy\": masked_lm_accuracy,\n",
+ " \"masked_lm_loss\": masked_lm_mean_loss,\n",
+ " \"next_sentence_accuracy\": next_sentence_accuracy,\n",
+ " \"next_sentence_loss\": next_sentence_mean_loss,\n",
+ " }\n",
+ "\n",
+ " eval_metrics = (metric_fn, [\n",
+ " masked_lm_example_loss, masked_lm_log_probs, masked_lm_ids,\n",
+ " masked_lm_weights, next_sentence_example_loss,\n",
+ " next_sentence_log_probs, next_sentence_labels\n",
+ " ])\n",
+ " output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
+ " mode=mode,\n",
+ " loss=total_loss,\n",
+ " eval_metrics=eval_metrics,\n",
+ " scaffold_fn=scaffold_fn)\n",
+ " elif mode == tf.estimator.ModeKeys.PREDICT:\n",
+ " masked_lm_log_probs = tf.reshape(masked_lm_log_probs,\n",
+ " [-1, masked_lm_log_probs.shape[-1]])\n",
+ " masked_lm_predictions = tf.argmax(\n",
+ " masked_lm_log_probs, axis=-1, output_type=tf.int32)\n",
+ "\n",
+ " next_sentence_log_probs = tf.reshape(\n",
+ " next_sentence_log_probs, [-1, next_sentence_log_probs.shape[-1]])\n",
+ " next_sentence_predictions = tf.argmax(\n",
+ " next_sentence_log_probs, axis=-1, output_type=tf.int32)\n",
+ "\n",
+ " masked_lm_predictions = tf.reshape(masked_lm_predictions,\n",
+ " [1, masked_lm_positions.shape[-1]])\n",
+ " next_sentence_predictions = tf.reshape(next_sentence_predictions,\n",
+ " [1, 1])\n",
+ "\n",
+ " predictions = {\n",
+ " \"masked_lm_predictions\": masked_lm_predictions,\n",
+ " \"next_sentence_predictions\": next_sentence_predictions\n",
+ " }\n",
+ "\n",
+ " output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
+ " mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)\n",
+ " return output_spec\n",
+ " else:\n",
+ " raise ValueError(\"Only TRAIN, EVAL and PREDICT modes are supported: %s\" % (mode))\n",
+ "\n",
+ " return output_spec\n",
+ "\n",
+ " return model_fn"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:40.328700Z",
+ "start_time": "2018-11-16T10:02:36.289676Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x12a864ae8>) includes params argument, but params are not passed to Estimator.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - WARNING - tensorflow - Estimator's model_fn (.model_fn at 0x12a864ae8>) includes params argument, but params are not passed to Estimator.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - WARNING - tensorflow - Using temporary folder as model directory: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
+ "graph_options {\n",
+ " rewrite_options {\n",
+ " meta_optimizer_iterations: ONE\n",
+ " }\n",
+ "}\n",
+ ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2, num_shards=1, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - Using config: {'_model_dir': '/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
+ "graph_options {\n",
+ " rewrite_options {\n",
+ " meta_optimizer_iterations: ONE\n",
+ " }\n",
+ "}\n",
+ ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2, num_shards=1, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:Setting TPUConfig.num_shards==1 is an unsupported behavior. Please fix as soon as possible (leaving num_shards as None.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - WARNING - tensorflow - Setting TPUConfig.num_shards==1 is an unsupported behavior. Please fix as soon as possible (leaving num_shards as None.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:_TPUContext: eval_on_tpu True\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - _TPUContext: eval_on_tpu True\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - WARNING - tensorflow - eval_on_tpu ignored because use_tpu is False.\n"
+ ]
+ }
+ ],
+ "source": [
+ "is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
+ "run_config = tf.contrib.tpu.RunConfig(\n",
+ " master=None,\n",
+ " tpu_config=tf.contrib.tpu.TPUConfig(\n",
+ " num_shards=1,\n",
+ " per_host_input_for_training=is_per_host))\n",
+ "\n",
+ "model_fn = model_fn_builder(\n",
+ " bert_config=bert_config,\n",
+ " init_checkpoint=init_checkpoint,\n",
+ " learning_rate=0,\n",
+ " num_train_steps=1,\n",
+ " num_warmup_steps=1,\n",
+ " use_tpu=False,\n",
+ " use_one_hot_embeddings=False)\n",
+ "\n",
+ "# If TPU is not available, this will fall back to normal Estimator on CPU\n",
+ "# or GPU.\n",
+ "estimator = tf.contrib.tpu.TPUEstimator(\n",
+ " use_tpu=False,\n",
+ " model_fn=model_fn,\n",
+ " config=run_config,\n",
+ " predict_batch_size=1)\n",
+ "\n",
+ "input_fn = input_fn_builder(\n",
+ " features=features, seq_length=max_seq_length, max_predictions_per_seq=max_predictions_per_seq,\n",
+ "tokenizer=tokenizer)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:46.596956Z",
+ "start_time": "2018-11-16T10:02:40.331008Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Could not find trained model in model_dir: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d, running initialization to predict.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - Could not find trained model in model_dir: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmp4x8r3x3d, running initialization to predict.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Calling model_fn.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - Calling model_fn.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Running infer on CPU\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - Running infer on CPU\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:*** Features ***\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - *** Features ***\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = input_ids, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = input_ids, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = input_mask, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = input_mask, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = masked_lm_ids, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = masked_lm_ids, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = masked_lm_positions, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = masked_lm_positions, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = masked_lm_weights, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = masked_lm_weights, shape = (?, 20)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = next_sentence_labels, shape = (?, 1)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = next_sentence_labels, shape = (?, 1)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = segment_ids, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:40 - INFO - tensorflow - name = segment_ids, shape = (?, 128)\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:**** Trainable Variables ****\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - **** Trainable Variables ****\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/predictions/transform/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/predictions/transform/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/predictions/transform/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/predictions/transform/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/predictions/transform/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/predictions/transform/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/predictions/transform/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/predictions/transform/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/predictions/output_bias:0, shape = (30522,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/predictions/output_bias:0, shape = (30522,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/seq_relationship/output_weights:0, shape = (2, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/seq_relationship/output_weights:0, shape = (2, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = cls/seq_relationship/output_bias:0, shape = (2,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - name = cls/seq_relationship/output_bias:0, shape = (2,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Done calling model_fn.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:43 - INFO - tensorflow - Done calling model_fn.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Graph was finalized.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:44 - INFO - tensorflow - Graph was finalized.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Running local_init_op.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:45 - INFO - tensorflow - Running local_init_op.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Done running local_init_op.\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:45 - INFO - tensorflow - Done running local_init_op.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:prediction_loop marked as finished\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:46 - INFO - tensorflow - prediction_loop marked as finished\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:prediction_loop marked as finished\n"
+ ]
+ },
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:02:46 - INFO - tensorflow - prediction_loop marked as finished\n"
+ ]
+ }
+ ],
+ "source": [
+ "tensorflow_all_out = []\n",
+ "for result in estimator.predict(input_fn, yield_single_examples=True):\n",
+ " tensorflow_all_out.append(result)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:46.634304Z",
+ "start_time": "2018-11-16T10:02:46.598800Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\n",
+ "2\n",
+ "dict_keys(['masked_lm_predictions', 'next_sentence_predictions'])\n",
+ "masked_lm_predictions [27227 1010 1010 1010 1010 1010 1010 1010 1010 1010 1010 1010\n",
+ " 1010 1010 1010 1010 1010 1010 1010 1010]\n",
+ "predicted token ['henson', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',', ',']\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(len(tensorflow_all_out))\n",
+ "print(len(tensorflow_all_out[0]))\n",
+ "print(tensorflow_all_out[0].keys())\n",
+ "print(\"masked_lm_predictions\", tensorflow_all_out[0]['masked_lm_predictions'])\n",
+ "print(\"predicted token\", tokenizer.convert_ids_to_tokens(tensorflow_all_out[0]['masked_lm_predictions']))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:02:46.671229Z",
+ "start_time": "2018-11-16T10:02:46.637102Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensorflow_output: ['henson']\n"
+ ]
+ }
+ ],
+ "source": [
+ "tensorflow_outputs = tokenizer.convert_ids_to_tokens(tensorflow_all_out[0]['masked_lm_predictions'])[:len(masked_lm_positions)]\n",
+ "print(\"tensorflow_output:\", tensorflow_outputs)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2/ PyTorch code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:03.556557Z",
+ "start_time": "2018-11-16T10:03:03.519654Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "from examples import extract_features\n",
+ "from examples.extract_features import *"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:03.952710Z",
+ "start_time": "2018-11-16T10:03:03.921917Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "init_checkpoint_pt = \"../google_models/uncased_L-12_H-768_A-12/pytorch_model.bin\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:12.307673Z",
+ "start_time": "2018-11-16T10:03:04.439317Z"
+ },
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/16/2018 11:03:05 - INFO - pytorch_pretrained_bert.modeling - loading archive file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz from cache at /Users/thomaswolf/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba\n",
+ "11/16/2018 11:03:05 - INFO - pytorch_pretrained_bert.modeling - extracting archive file /Users/thomaswolf/.pytorch_pretrained_bert/9c41111e2de84547a463fd39217199738d1e3deb72d4fec4399e6e241983c6f0.ae3cef932725ca7a30cdcb93fc6e09150a55e2a130ec7af63975a16c153ae2ba to temp dir /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmpaqgsm566\n",
+ "11/16/2018 11:03:08 - INFO - pytorch_pretrained_bert.modeling - Model config {\n",
+ " \"attention_probs_dropout_prob\": 0.1,\n",
+ " \"hidden_act\": \"gelu\",\n",
+ " \"hidden_dropout_prob\": 0.1,\n",
+ " \"hidden_size\": 768,\n",
+ " \"initializer_range\": 0.02,\n",
+ " \"intermediate_size\": 3072,\n",
+ " \"max_position_embeddings\": 512,\n",
+ " \"num_attention_heads\": 12,\n",
+ " \"num_hidden_layers\": 12,\n",
+ " \"type_vocab_size\": 2,\n",
+ " \"vocab_size\": 30522\n",
+ "}\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "BertForPreTraining(\n",
+ " (bert): BertModel(\n",
+ " (embeddings): BertEmbeddings(\n",
+ " (word_embeddings): Embedding(30522, 768)\n",
+ " (position_embeddings): Embedding(512, 768)\n",
+ " (token_type_embeddings): Embedding(2, 768)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (encoder): BertEncoder(\n",
+ " (layer): ModuleList(\n",
+ " (0): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (1): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (2): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (3): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (4): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (5): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (6): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (7): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (8): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (9): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (10): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (11): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " (pooler): BertPooler(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (activation): Tanh()\n",
+ " )\n",
+ " )\n",
+ " (cls): BertPreTrainingHeads(\n",
+ " (predictions): BertLMPredictionHead(\n",
+ " (transform): BertPredictionHeadTransform(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " )\n",
+ " (decoder): Linear(in_features=768, out_features=30522, bias=False)\n",
+ " )\n",
+ " (seq_relationship): Linear(in_features=768, out_features=2, bias=True)\n",
+ " )\n",
+ ")"
+ ]
+ },
+ "execution_count": 16,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "device = torch.device(\"cpu\")\n",
+ "model = ppb.BertForPreTraining.from_pretrained('bert-base-uncased')\n",
+ "model.to(device)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:12.351625Z",
+ "start_time": "2018-11-16T10:03:12.310736Z"
+ },
+ "code_folding": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "BertForPreTraining(\n",
+ " (bert): BertModel(\n",
+ " (embeddings): BertEmbeddings(\n",
+ " (word_embeddings): Embedding(30522, 768)\n",
+ " (position_embeddings): Embedding(512, 768)\n",
+ " (token_type_embeddings): Embedding(2, 768)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (encoder): BertEncoder(\n",
+ " (layer): ModuleList(\n",
+ " (0): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (1): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (2): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (3): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (4): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (5): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (6): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (7): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (8): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (9): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (10): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (11): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " (pooler): BertPooler(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (activation): Tanh()\n",
+ " )\n",
+ " )\n",
+ " (cls): BertPreTrainingHeads(\n",
+ " (predictions): BertLMPredictionHead(\n",
+ " (transform): BertPredictionHeadTransform(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " )\n",
+ " (decoder): Linear(in_features=768, out_features=30522, bias=False)\n",
+ " )\n",
+ " (seq_relationship): Linear(in_features=768, out_features=2, bias=True)\n",
+ " )\n",
+ ")"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n",
+ "all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)\n",
+ "all_segment_ids = torch.tensor([f.segment_ids for f in features], dtype=torch.long)\n",
+ "all_masked_lm_positions = torch.tensor([f.masked_lm_positions for f in features], dtype=torch.long)\n",
+ "\n",
+ "eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_masked_lm_positions)\n",
+ "eval_sampler = SequentialSampler(eval_data)\n",
+ "eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=1)\n",
+ "\n",
+ "model.eval()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:12.792741Z",
+ "start_time": "2018-11-16T10:03:12.354253Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([[ 2040, 2001, 3958, 27227, 1029, 3958, 103, 2001, 1037, 13997,\n",
+ " 11510, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0]])\n",
+ "tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0]])\n",
+ "tensor([[0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0]])\n",
+ "(1, 20, 30522)\n",
+ "[27227, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010]\n"
+ ]
+ }
+ ],
+ "source": [
+ "import numpy as np\n",
+ "pytorch_all_out = []\n",
+ "for input_ids, input_mask, segment_ids, tensor_masked_lm_positions in eval_dataloader:\n",
+ " print(input_ids)\n",
+ " print(input_mask)\n",
+ " print(segment_ids)\n",
+ " input_ids = input_ids.to(device)\n",
+ " input_mask = input_mask.to(device)\n",
+ " segment_ids = segment_ids.to(device)\n",
+ "\n",
+ " prediction_scores, _ = model(input_ids, token_type_ids=segment_ids, attention_mask=input_mask)\n",
+ " prediction_scores = prediction_scores[0, tensor_masked_lm_positions].detach().cpu().numpy()\n",
+ " print(prediction_scores.shape)\n",
+ " masked_lm_predictions = np.argmax(prediction_scores, axis=-1).squeeze().tolist()\n",
+ " print(masked_lm_predictions)\n",
+ " pytorch_all_out.append(masked_lm_predictions)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-16T10:03:12.828439Z",
+ "start_time": "2018-11-16T10:03:12.795420Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "pytorch_output: ['henson']\n",
+ "tensorflow_output: ['henson']\n"
+ ]
+ }
+ ],
+ "source": [
+ "pytorch_outputs = tokenizer.convert_ids_to_tokens(pytorch_all_out[0])[:len(masked_lm_positions)]\n",
+ "print(\"pytorch_output:\", pytorch_outputs)\n",
+ "print(\"tensorflow_output:\", tensorflow_outputs)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python [default]",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.7"
+ },
+ "toc": {
+ "colors": {
+ "hover_highlight": "#DAA520",
+ "running_highlight": "#FF0000",
+ "selected_highlight": "#FFD700"
+ },
+ "moveMenuLeft": true,
+ "nav_menu": {
+ "height": "48px",
+ "width": "252px"
+ },
+ "navigate_menu": true,
+ "number_sections": true,
+ "sideBar": true,
+ "threshold": 4,
+ "toc_cell": false,
+ "toc_section_display": "block",
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-SQuAD.ipynb b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-SQuAD.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..c91822d8f9688f5d431a43d29e0c3a275876135d
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/notebooks/Comparing-TF-and-PT-models-SQuAD.ipynb
@@ -0,0 +1,1644 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Comparing TensorFlow (original) and PyTorch model on the SQuAD task\n",
+ "\n",
+ "You can use this small notebook to check the loss computation from the TensorFlow model to the PyTorch model. In the following, we compare the total loss computed by the models starting from identical initializations (position prediction linear layers with weights at 1 and bias at 0).\n",
+ "\n",
+ "To run this notebook, follow these instructions:\n",
+ "- make sure that your Python environment has both TensorFlow and PyTorch installed,\n",
+ "- download the original TensorFlow implementation,\n",
+ "- download a pre-trained TensorFlow model as indicaded in the TensorFlow implementation readme,\n",
+ "- run the script `convert_tf_checkpoint_to_pytorch.py` as indicated in the `README` to convert the pre-trained TensorFlow model to PyTorch.\n",
+ "\n",
+ "If needed change the relative paths indicated in this notebook (at the beggining of Sections 1 and 2) to point to the relevent models and code."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:33.636911Z",
+ "start_time": "2018-11-06T10:11:33.623091Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "os.chdir('../')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 1/ TensorFlow code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:33.651792Z",
+ "start_time": "2018-11-06T10:11:33.638984Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "original_tf_inplem_dir = \"./tensorflow_code/\"\n",
+ "model_dir = \"../google_models/uncased_L-12_H-768_A-12/\"\n",
+ "\n",
+ "vocab_file = model_dir + \"vocab.txt\"\n",
+ "bert_config_file = model_dir + \"bert_config.json\"\n",
+ "init_checkpoint = model_dir + \"bert_model.ckpt\"\n",
+ "\n",
+ "input_file = \"../data/squad_data/train-v1.1.json\"\n",
+ "max_seq_length = 384\n",
+ "outside_pos = max_seq_length + 10\n",
+ "doc_stride = 128\n",
+ "max_query_length = 64\n",
+ "max_answer_length = 30\n",
+ "output_dir = \"/tmp/squad_base/\"\n",
+ "learning_rate = 3e-5"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:35.165788Z",
+ "start_time": "2018-11-06T10:11:33.653401Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import importlib.util\n",
+ "import sys\n",
+ "\n",
+ "spec = importlib.util.spec_from_file_location('*', original_tf_inplem_dir + '/modeling.py')\n",
+ "module = importlib.util.module_from_spec(spec)\n",
+ "spec.loader.exec_module(module)\n",
+ "sys.modules['modeling_tensorflow'] = module\n",
+ "\n",
+ "spec = importlib.util.spec_from_file_location('*', original_tf_inplem_dir + '/run_squad.py')\n",
+ "module = importlib.util.module_from_spec(spec)\n",
+ "spec.loader.exec_module(module)\n",
+ "sys.modules['run_squad_tensorflow'] = module\n",
+ "import modeling_tensorflow\n",
+ "from run_squad_tensorflow import *"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:37.494391Z",
+ "start_time": "2018-11-06T10:11:35.168615Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000000\n",
+ "INFO:tensorflow:example_index: 0\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] to whom did the virgin mary allegedly appear in 1858 in lou ##rdes france ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\n",
+ "INFO:tensorflow:token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\n",
+ "INFO:tensorflow:input_ids: 101 2000 3183 2106 1996 6261 2984 9382 3711 1999 8517 1999 10223 26371 2605 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 130\n",
+ "INFO:tensorflow:end_position: 137\n",
+ "INFO:tensorflow:answer: saint bern ##ade ##tte so ##ub ##iro ##us\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000001\n",
+ "INFO:tensorflow:example_index: 1\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what is in front of the notre dame main building ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 13:0 14:0 15:0 16:1 17:2 18:3 19:4 20:5 21:6 22:6 23:7 24:8 25:9 26:10 27:10 28:10 29:11 30:12 31:13 32:14 33:15 34:16 35:17 36:18 37:19 38:20 39:20 40:21 41:22 42:23 43:24 44:25 45:26 46:27 47:28 48:29 49:30 50:30 51:31 52:32 53:33 54:34 55:35 56:36 57:37 58:38 59:39 60:39 61:39 62:40 63:41 64:42 65:43 66:43 67:43 68:43 69:44 70:45 71:46 72:46 73:46 74:46 75:47 76:48 77:49 78:50 79:51 80:52 81:53 82:54 83:55 84:56 85:57 86:58 87:58 88:59 89:60 90:61 91:62 92:63 93:64 94:65 95:65 96:65 97:66 98:67 99:68 100:69 101:70 102:71 103:72 104:72 105:73 106:74 107:75 108:76 109:77 110:78 111:79 112:79 113:80 114:81 115:81 116:81 117:82 118:83 119:84 120:85 121:86 122:87 123:87 124:88 125:89 126:90 127:91 128:91 129:91 130:92 131:92 132:92 133:92 134:93 135:94 136:94 137:95 138:96 139:97 140:98 141:99 142:100 143:101 144:102 145:102 146:103 147:104 148:105 149:106 150:107 151:108 152:109 153:110 154:111 155:112 156:113 157:114 158:115 159:115 160:115 161:116 162:117 163:118 164:118 165:119 166:120 167:121 168:122 169:123 170:123\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 2003 1999 2392 1997 1996 10289 8214 2364 2311 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 52\n",
+ "INFO:tensorflow:end_position: 56\n",
+ "INFO:tensorflow:answer: a copper statue of christ\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000002\n",
+ "INFO:tensorflow:example_index: 2\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] the basilica of the sacred heart at notre dame is beside to which structure ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 17:0 18:0 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:6 27:7 28:8 29:9 30:10 31:10 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:27 51:28 52:29 53:30 54:30 55:31 56:32 57:33 58:34 59:35 60:36 61:37 62:38 63:39 64:39 65:39 66:40 67:41 68:42 69:43 70:43 71:43 72:43 73:44 74:45 75:46 76:46 77:46 78:46 79:47 80:48 81:49 82:50 83:51 84:52 85:53 86:54 87:55 88:56 89:57 90:58 91:58 92:59 93:60 94:61 95:62 96:63 97:64 98:65 99:65 100:65 101:66 102:67 103:68 104:69 105:70 106:71 107:72 108:72 109:73 110:74 111:75 112:76 113:77 114:78 115:79 116:79 117:80 118:81 119:81 120:81 121:82 122:83 123:84 124:85 125:86 126:87 127:87 128:88 129:89 130:90 131:91 132:91 133:91 134:92 135:92 136:92 137:92 138:93 139:94 140:94 141:95 142:96 143:97 144:98 145:99 146:100 147:101 148:102 149:102 150:103 151:104 152:105 153:106 154:107 155:108 156:109 157:110 158:111 159:112 160:113 161:114 162:115 163:115 164:115 165:116 166:117 167:118 168:118 169:119 170:120 171:121 172:122 173:123 174:123\n",
+ "INFO:tensorflow:token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True\n",
+ "INFO:tensorflow:input_ids: 101 1996 13546 1997 1996 6730 2540 2012 10289 8214 2003 3875 2000 2029 3252 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 81\n",
+ "INFO:tensorflow:end_position: 83\n",
+ "INFO:tensorflow:answer: the main building\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000003\n",
+ "INFO:tensorflow:example_index: 3\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what is the gr ##otto at notre dame ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 11:0 12:0 13:0 14:1 15:2 16:3 17:4 18:5 19:6 20:6 21:7 22:8 23:9 24:10 25:10 26:10 27:11 28:12 29:13 30:14 31:15 32:16 33:17 34:18 35:19 36:20 37:20 38:21 39:22 40:23 41:24 42:25 43:26 44:27 45:28 46:29 47:30 48:30 49:31 50:32 51:33 52:34 53:35 54:36 55:37 56:38 57:39 58:39 59:39 60:40 61:41 62:42 63:43 64:43 65:43 66:43 67:44 68:45 69:46 70:46 71:46 72:46 73:47 74:48 75:49 76:50 77:51 78:52 79:53 80:54 81:55 82:56 83:57 84:58 85:58 86:59 87:60 88:61 89:62 90:63 91:64 92:65 93:65 94:65 95:66 96:67 97:68 98:69 99:70 100:71 101:72 102:72 103:73 104:74 105:75 106:76 107:77 108:78 109:79 110:79 111:80 112:81 113:81 114:81 115:82 116:83 117:84 118:85 119:86 120:87 121:87 122:88 123:89 124:90 125:91 126:91 127:91 128:92 129:92 130:92 131:92 132:93 133:94 134:94 135:95 136:96 137:97 138:98 139:99 140:100 141:101 142:102 143:102 144:103 145:104 146:105 147:106 148:107 149:108 150:109 151:110 152:111 153:112 154:113 155:114 156:115 157:115 158:115 159:116 160:117 161:118 162:118 163:119 164:120 165:121 166:122 167:123 168:123\n",
+ "INFO:tensorflow:token_is_max_context: 11:True 12:True 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 2003 1996 24665 23052 2012 10289 8214 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 95\n",
+ "INFO:tensorflow:end_position: 101\n",
+ "INFO:tensorflow:answer: a marian place of prayer and reflection\n",
+ "INFO:tensorflow:*** Example ***\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:unique_id: 1000000004\n",
+ "INFO:tensorflow:example_index: 4\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what sits on top of the main building at notre dame ? [SEP] architectural ##ly , the school has a catholic character . atop the main building ' s gold dome is a golden statue of the virgin mary . immediately in front of the main building and facing it , is a copper statue of christ with arms up ##rai ##sed with the legend \" ve ##ni ##te ad me om ##nes \" . next to the main building is the basilica of the sacred heart . immediately behind the basilica is the gr ##otto , a marian place of prayer and reflection . it is a replica of the gr ##otto at lou ##rdes , france where the virgin mary reputed ##ly appeared to saint bern ##ade ##tte so ##ub ##iro ##us in 1858 . at the end of the main drive ( and in a direct line that connects through 3 statues and the gold dome ) , is a simple , modern stone statue of mary . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 14:0 15:0 16:0 17:1 18:2 19:3 20:4 21:5 22:6 23:6 24:7 25:8 26:9 27:10 28:10 29:10 30:11 31:12 32:13 33:14 34:15 35:16 36:17 37:18 38:19 39:20 40:20 41:21 42:22 43:23 44:24 45:25 46:26 47:27 48:28 49:29 50:30 51:30 52:31 53:32 54:33 55:34 56:35 57:36 58:37 59:38 60:39 61:39 62:39 63:40 64:41 65:42 66:43 67:43 68:43 69:43 70:44 71:45 72:46 73:46 74:46 75:46 76:47 77:48 78:49 79:50 80:51 81:52 82:53 83:54 84:55 85:56 86:57 87:58 88:58 89:59 90:60 91:61 92:62 93:63 94:64 95:65 96:65 97:65 98:66 99:67 100:68 101:69 102:70 103:71 104:72 105:72 106:73 107:74 108:75 109:76 110:77 111:78 112:79 113:79 114:80 115:81 116:81 117:81 118:82 119:83 120:84 121:85 122:86 123:87 124:87 125:88 126:89 127:90 128:91 129:91 130:91 131:92 132:92 133:92 134:92 135:93 136:94 137:94 138:95 139:96 140:97 141:98 142:99 143:100 144:101 145:102 146:102 147:103 148:104 149:105 150:106 151:107 152:108 153:109 154:110 155:111 156:112 157:113 158:114 159:115 160:115 161:115 162:116 163:117 164:118 165:118 166:119 167:120 168:121 169:122 170:123 171:123\n",
+ "INFO:tensorflow:token_is_max_context: 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 7719 2006 2327 1997 1996 2364 2311 2012 10289 8214 1029 102 6549 2135 1010 1996 2082 2038 1037 3234 2839 1012 10234 1996 2364 2311 1005 1055 2751 8514 2003 1037 3585 6231 1997 1996 6261 2984 1012 3202 1999 2392 1997 1996 2364 2311 1998 5307 2009 1010 2003 1037 6967 6231 1997 4828 2007 2608 2039 14995 6924 2007 1996 5722 1000 2310 3490 2618 4748 2033 18168 5267 1000 1012 2279 2000 1996 2364 2311 2003 1996 13546 1997 1996 6730 2540 1012 3202 2369 1996 13546 2003 1996 24665 23052 1010 1037 14042 2173 1997 7083 1998 9185 1012 2009 2003 1037 15059 1997 1996 24665 23052 2012 10223 26371 1010 2605 2073 1996 6261 2984 22353 2135 2596 2000 3002 16595 9648 4674 2061 12083 9711 2271 1999 8517 1012 2012 1996 2203 1997 1996 2364 3298 1006 1998 1999 1037 3622 2240 2008 8539 2083 1017 11342 1998 1996 2751 8514 1007 1010 2003 1037 3722 1010 2715 2962 6231 1997 2984 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 33\n",
+ "INFO:tensorflow:end_position: 39\n",
+ "INFO:tensorflow:answer: a golden statue of the virgin mary\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000005\n",
+ "INFO:tensorflow:example_index: 5\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] when did the scholastic magazine of notre dame begin publishing ? [SEP] as at most other universities , notre dame ' s students run a number of news media outlets . the nine student - run outlets include three newspapers , both a radio and television station , and several magazines and journals . begun as a one - page journal in september 1876 , the scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the united states . the other magazine , the jug ##gler , is released twice a year and focuses on student literature and artwork . the dome yearbook is published annually . the newspapers have varying publication interests , with the observer published daily and mainly reporting university and other news , and staffed by students from both notre dame and saint mary ' s college . unlike scholastic and the dome , the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university . in 1987 , when some students believed that the observer began to show a conservative bias , a liberal newspaper , common sense was published . likewise , in 2003 , when other students believed that the paper showed a liberal bias , the conservative paper irish rover went into production . neither paper is published as often as the observer ; however , all three are distributed to all students . finally , in spring 2008 an undergraduate journal for political science research , beyond politics , made its debut . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 13:0 14:1 15:2 16:3 17:4 18:4 19:5 20:6 21:6 22:6 23:7 24:8 25:9 26:10 27:11 28:12 29:13 30:14 31:14 32:15 33:16 34:17 35:17 36:17 37:18 38:19 39:20 40:21 41:21 42:22 43:23 44:24 45:25 46:26 47:27 48:27 49:28 50:29 51:30 52:31 53:32 54:32 55:33 56:34 57:35 58:36 59:36 60:36 61:37 62:38 63:39 64:40 65:40 66:41 67:42 68:43 69:44 70:45 71:46 72:47 73:48 74:49 75:50 76:51 77:52 78:53 79:54 80:55 81:56 82:57 83:58 84:59 85:60 86:60 87:61 88:62 89:63 90:63 91:64 92:65 93:65 94:65 95:66 96:67 97:68 98:69 99:70 100:71 101:72 102:73 103:74 104:75 105:76 106:77 107:77 108:78 109:79 110:80 111:81 112:82 113:83 114:83 115:84 116:85 117:86 118:87 119:88 120:89 121:89 122:90 123:91 124:92 125:93 126:94 127:95 128:96 129:97 130:98 131:99 132:100 133:101 134:101 135:102 136:103 137:104 138:105 139:106 140:107 141:108 142:109 143:110 144:111 145:112 146:112 147:112 148:113 149:113 150:114 151:115 152:116 153:117 154:118 155:118 156:119 157:120 158:121 159:122 160:123 161:124 162:125 163:126 164:127 165:128 166:129 167:130 168:131 169:132 170:133 171:134 172:135 173:136 174:137 175:138 176:138 177:139 178:140 179:140 180:141 181:142 182:143 183:144 184:145 185:146 186:147 187:148 188:149 189:150 190:151 191:152 192:153 193:153 194:154 195:155 196:156 197:156 198:157 199:158 200:159 201:160 202:160 203:161 204:161 205:162 206:163 207:163 208:164 209:165 210:166 211:167 212:168 213:169 214:170 215:171 216:172 217:173 218:174 219:174 220:175 221:176 222:177 223:178 224:179 225:180 226:181 227:182 228:182 229:183 230:184 231:185 232:186 233:187 234:188 235:189 236:190 237:191 238:191 239:192 240:192 241:193 242:194 243:195 244:196 245:197 246:198 247:199 248:199 249:200 250:200 251:201 252:202 253:203 254:204 255:205 256:206 257:207 258:208 259:209 260:210 261:210 262:211 263:212 264:212 265:213 266:214 267:215 268:215\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True 175:True 176:True 177:True 178:True 179:True 180:True 181:True 182:True 183:True 184:True 185:True 186:True 187:True 188:True 189:True 190:True 191:True 192:True 193:True 194:True 195:True 196:True 197:True 198:True 199:True 200:True 201:True 202:True 203:True 204:True 205:True 206:True 207:True 208:True 209:True 210:True 211:True 212:True 213:True 214:True 215:True 216:True 217:True 218:True 219:True 220:True 221:True 222:True 223:True 224:True 225:True 226:True 227:True 228:True 229:True 230:True 231:True 232:True 233:True 234:True 235:True 236:True 237:True 238:True 239:True 240:True 241:True 242:True 243:True 244:True 245:True 246:True 247:True 248:True 249:True 250:True 251:True 252:True 253:True 254:True 255:True 256:True 257:True 258:True 259:True 260:True 261:True 262:True 263:True 264:True 265:True 266:True 267:True 268:True\n",
+ "INFO:tensorflow:input_ids: 101 2043 2106 1996 24105 2932 1997 10289 8214 4088 4640 1029 102 2004 2012 2087 2060 5534 1010 10289 8214 1005 1055 2493 2448 1037 2193 1997 2739 2865 11730 1012 1996 3157 3076 1011 2448 11730 2421 2093 6399 1010 2119 1037 2557 1998 2547 2276 1010 1998 2195 7298 1998 9263 1012 5625 2004 1037 2028 1011 3931 3485 1999 2244 7326 1010 1996 24105 2932 2003 3843 3807 7058 1998 4447 2000 2022 1996 4587 7142 9234 4772 1999 1996 2142 2163 1012 1996 2060 2932 1010 1996 26536 17420 1010 2003 2207 3807 1037 2095 1998 7679 2006 3076 3906 1998 8266 1012 1996 8514 24803 2003 2405 6604 1012 1996 6399 2031 9671 4772 5426 1010 2007 1996 9718 2405 3679 1998 3701 7316 2118 1998 2060 2739 1010 1998 21121 2011 2493 2013 2119 10289 8214 1998 3002 2984 1005 1055 2267 1012 4406 24105 1998 1996 8514 1010 1996 9718 2003 2019 2981 4772 1998 2515 2025 2031 1037 4513 8619 2030 2151 8368 15709 2013 1996 2118 1012 1999 3055 1010 2043 2070 2493 3373 2008 1996 9718 2211 2000 2265 1037 4603 13827 1010 1037 4314 3780 1010 2691 3168 2001 2405 1012 10655 1010 1999 2494 1010 2043 2060 2493 3373 2008 1996 3259 3662 1037 4314 13827 1010 1996 4603 3259 3493 13631 2253 2046 2537 1012 4445 3259 2003 2405 2004 2411 2004 1996 9718 1025 2174 1010 2035 2093 2024 5500 2000 2035 2493 1012 2633 1010 1999 3500 2263 2019 8324 3485 2005 2576 2671 2470 1010 3458 4331 1010 2081 2049 2834 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 63\n",
+ "INFO:tensorflow:end_position: 64\n",
+ "INFO:tensorflow:answer: september 1876\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000006\n",
+ "INFO:tensorflow:example_index: 6\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] how often is notre dame ' s the jug ##gler published ? [SEP] as at most other universities , notre dame ' s students run a number of news media outlets . the nine student - run outlets include three newspapers , both a radio and television station , and several magazines and journals . begun as a one - page journal in september 1876 , the scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the united states . the other magazine , the jug ##gler , is released twice a year and focuses on student literature and artwork . the dome yearbook is published annually . the newspapers have varying publication interests , with the observer published daily and mainly reporting university and other news , and staffed by students from both notre dame and saint mary ' s college . unlike scholastic and the dome , the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university . in 1987 , when some students believed that the observer began to show a conservative bias , a liberal newspaper , common sense was published . likewise , in 2003 , when other students believed that the paper showed a liberal bias , the conservative paper irish rover went into production . neither paper is published as often as the observer ; however , all three are distributed to all students . finally , in spring 2008 an undergraduate journal for political science research , beyond politics , made its debut . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 14:0 15:1 16:2 17:3 18:4 19:4 20:5 21:6 22:6 23:6 24:7 25:8 26:9 27:10 28:11 29:12 30:13 31:14 32:14 33:15 34:16 35:17 36:17 37:17 38:18 39:19 40:20 41:21 42:21 43:22 44:23 45:24 46:25 47:26 48:27 49:27 50:28 51:29 52:30 53:31 54:32 55:32 56:33 57:34 58:35 59:36 60:36 61:36 62:37 63:38 64:39 65:40 66:40 67:41 68:42 69:43 70:44 71:45 72:46 73:47 74:48 75:49 76:50 77:51 78:52 79:53 80:54 81:55 82:56 83:57 84:58 85:59 86:60 87:60 88:61 89:62 90:63 91:63 92:64 93:65 94:65 95:65 96:66 97:67 98:68 99:69 100:70 101:71 102:72 103:73 104:74 105:75 106:76 107:77 108:77 109:78 110:79 111:80 112:81 113:82 114:83 115:83 116:84 117:85 118:86 119:87 120:88 121:89 122:89 123:90 124:91 125:92 126:93 127:94 128:95 129:96 130:97 131:98 132:99 133:100 134:101 135:101 136:102 137:103 138:104 139:105 140:106 141:107 142:108 143:109 144:110 145:111 146:112 147:112 148:112 149:113 150:113 151:114 152:115 153:116 154:117 155:118 156:118 157:119 158:120 159:121 160:122 161:123 162:124 163:125 164:126 165:127 166:128 167:129 168:130 169:131 170:132 171:133 172:134 173:135 174:136 175:137 176:138 177:138 178:139 179:140 180:140 181:141 182:142 183:143 184:144 185:145 186:146 187:147 188:148 189:149 190:150 191:151 192:152 193:153 194:153 195:154 196:155 197:156 198:156 199:157 200:158 201:159 202:160 203:160 204:161 205:161 206:162 207:163 208:163 209:164 210:165 211:166 212:167 213:168 214:169 215:170 216:171 217:172 218:173 219:174 220:174 221:175 222:176 223:177 224:178 225:179 226:180 227:181 228:182 229:182 230:183 231:184 232:185 233:186 234:187 235:188 236:189 237:190 238:191 239:191 240:192 241:192 242:193 243:194 244:195 245:196 246:197 247:198 248:199 249:199 250:200 251:200 252:201 253:202 254:203 255:204 256:205 257:206 258:207 259:208 260:209 261:210 262:210 263:211 264:212 265:212 266:213 267:214 268:215 269:215\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True 175:True 176:True 177:True 178:True 179:True 180:True 181:True 182:True 183:True 184:True 185:True 186:True 187:True 188:True 189:True 190:True 191:True 192:True 193:True 194:True 195:True 196:True 197:True 198:True 199:True 200:True 201:True 202:True 203:True 204:True 205:True 206:True 207:True 208:True 209:True 210:True 211:True 212:True 213:True 214:True 215:True 216:True 217:True 218:True 219:True 220:True 221:True 222:True 223:True 224:True 225:True 226:True 227:True 228:True 229:True 230:True 231:True 232:True 233:True 234:True 235:True 236:True 237:True 238:True 239:True 240:True 241:True 242:True 243:True 244:True 245:True 246:True 247:True 248:True 249:True 250:True 251:True 252:True 253:True 254:True 255:True 256:True 257:True 258:True 259:True 260:True 261:True 262:True 263:True 264:True 265:True 266:True 267:True 268:True 269:True\n",
+ "INFO:tensorflow:input_ids: 101 2129 2411 2003 10289 8214 1005 1055 1996 26536 17420 2405 1029 102 2004 2012 2087 2060 5534 1010 10289 8214 1005 1055 2493 2448 1037 2193 1997 2739 2865 11730 1012 1996 3157 3076 1011 2448 11730 2421 2093 6399 1010 2119 1037 2557 1998 2547 2276 1010 1998 2195 7298 1998 9263 1012 5625 2004 1037 2028 1011 3931 3485 1999 2244 7326 1010 1996 24105 2932 2003 3843 3807 7058 1998 4447 2000 2022 1996 4587 7142 9234 4772 1999 1996 2142 2163 1012 1996 2060 2932 1010 1996 26536 17420 1010 2003 2207 3807 1037 2095 1998 7679 2006 3076 3906 1998 8266 1012 1996 8514 24803 2003 2405 6604 1012 1996 6399 2031 9671 4772 5426 1010 2007 1996 9718 2405 3679 1998 3701 7316 2118 1998 2060 2739 1010 1998 21121 2011 2493 2013 2119 10289 8214 1998 3002 2984 1005 1055 2267 1012 4406 24105 1998 1996 8514 1010 1996 9718 2003 2019 2981 4772 1998 2515 2025 2031 1037 4513 8619 2030 2151 8368 15709 2013 1996 2118 1012 1999 3055 1010 2043 2070 2493 3373 2008 1996 9718 2211 2000 2265 1037 4603 13827 1010 1037 4314 3780 1010 2691 3168 2001 2405 1012 10655 1010 1999 2494 1010 2043 2060 2493 3373 2008 1996 3259 3662 1037 4314 13827 1010 1996 4603 3259 3493 13631 2253 2046 2537 1012 4445 3259 2003 2405 2004 2411 2004 1996 9718 1025 2174 1010 2035 2093 2024 5500 2000 2035 2493 1012 2633 1010 1999 3500 2263 2019 8324 3485 2005 2576 2671 2470 1010 3458 4331 1010 2081 2049 2834 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 98\n",
+ "INFO:tensorflow:end_position: 98\n",
+ "INFO:tensorflow:answer: twice\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000007\n",
+ "INFO:tensorflow:example_index: 7\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what is the daily student paper at notre dame called ? [SEP] as at most other universities , notre dame ' s students run a number of news media outlets . the nine student - run outlets include three newspapers , both a radio and television station , and several magazines and journals . begun as a one - page journal in september 1876 , the scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the united states . the other magazine , the jug ##gler , is released twice a year and focuses on student literature and artwork . the dome yearbook is published annually . the newspapers have varying publication interests , with the observer published daily and mainly reporting university and other news , and staffed by students from both notre dame and saint mary ' s college . unlike scholastic and the dome , the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university . in 1987 , when some students believed that the observer began to show a conservative bias , a liberal newspaper , common sense was published . likewise , in 2003 , when other students believed that the paper showed a liberal bias , the conservative paper irish rover went into production . neither paper is published as often as the observer ; however , all three are distributed to all students . finally , in spring 2008 an undergraduate journal for political science research , beyond politics , made its debut . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 13:0 14:1 15:2 16:3 17:4 18:4 19:5 20:6 21:6 22:6 23:7 24:8 25:9 26:10 27:11 28:12 29:13 30:14 31:14 32:15 33:16 34:17 35:17 36:17 37:18 38:19 39:20 40:21 41:21 42:22 43:23 44:24 45:25 46:26 47:27 48:27 49:28 50:29 51:30 52:31 53:32 54:32 55:33 56:34 57:35 58:36 59:36 60:36 61:37 62:38 63:39 64:40 65:40 66:41 67:42 68:43 69:44 70:45 71:46 72:47 73:48 74:49 75:50 76:51 77:52 78:53 79:54 80:55 81:56 82:57 83:58 84:59 85:60 86:60 87:61 88:62 89:63 90:63 91:64 92:65 93:65 94:65 95:66 96:67 97:68 98:69 99:70 100:71 101:72 102:73 103:74 104:75 105:76 106:77 107:77 108:78 109:79 110:80 111:81 112:82 113:83 114:83 115:84 116:85 117:86 118:87 119:88 120:89 121:89 122:90 123:91 124:92 125:93 126:94 127:95 128:96 129:97 130:98 131:99 132:100 133:101 134:101 135:102 136:103 137:104 138:105 139:106 140:107 141:108 142:109 143:110 144:111 145:112 146:112 147:112 148:113 149:113 150:114 151:115 152:116 153:117 154:118 155:118 156:119 157:120 158:121 159:122 160:123 161:124 162:125 163:126 164:127 165:128 166:129 167:130 168:131 169:132 170:133 171:134 172:135 173:136 174:137 175:138 176:138 177:139 178:140 179:140 180:141 181:142 182:143 183:144 184:145 185:146 186:147 187:148 188:149 189:150 190:151 191:152 192:153 193:153 194:154 195:155 196:156 197:156 198:157 199:158 200:159 201:160 202:160 203:161 204:161 205:162 206:163 207:163 208:164 209:165 210:166 211:167 212:168 213:169 214:170 215:171 216:172 217:173 218:174 219:174 220:175 221:176 222:177 223:178 224:179 225:180 226:181 227:182 228:182 229:183 230:184 231:185 232:186 233:187 234:188 235:189 236:190 237:191 238:191 239:192 240:192 241:193 242:194 243:195 244:196 245:197 246:198 247:199 248:199 249:200 250:200 251:201 252:202 253:203 254:204 255:205 256:206 257:207 258:208 259:209 260:210 261:210 262:211 263:212 264:212 265:213 266:214 267:215 268:215\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True 175:True 176:True 177:True 178:True 179:True 180:True 181:True 182:True 183:True 184:True 185:True 186:True 187:True 188:True 189:True 190:True 191:True 192:True 193:True 194:True 195:True 196:True 197:True 198:True 199:True 200:True 201:True 202:True 203:True 204:True 205:True 206:True 207:True 208:True 209:True 210:True 211:True 212:True 213:True 214:True 215:True 216:True 217:True 218:True 219:True 220:True 221:True 222:True 223:True 224:True 225:True 226:True 227:True 228:True 229:True 230:True 231:True 232:True 233:True 234:True 235:True 236:True 237:True 238:True 239:True 240:True 241:True 242:True 243:True 244:True 245:True 246:True 247:True 248:True 249:True 250:True 251:True 252:True 253:True 254:True 255:True 256:True 257:True 258:True 259:True 260:True 261:True 262:True 263:True 264:True 265:True 266:True 267:True 268:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 2003 1996 3679 3076 3259 2012 10289 8214 2170 1029 102 2004 2012 2087 2060 5534 1010 10289 8214 1005 1055 2493 2448 1037 2193 1997 2739 2865 11730 1012 1996 3157 3076 1011 2448 11730 2421 2093 6399 1010 2119 1037 2557 1998 2547 2276 1010 1998 2195 7298 1998 9263 1012 5625 2004 1037 2028 1011 3931 3485 1999 2244 7326 1010 1996 24105 2932 2003 3843 3807 7058 1998 4447 2000 2022 1996 4587 7142 9234 4772 1999 1996 2142 2163 1012 1996 2060 2932 1010 1996 26536 17420 1010 2003 2207 3807 1037 2095 1998 7679 2006 3076 3906 1998 8266 1012 1996 8514 24803 2003 2405 6604 1012 1996 6399 2031 9671 4772 5426 1010 2007 1996 9718 2405 3679 1998 3701 7316 2118 1998 2060 2739 1010 1998 21121 2011 2493 2013 2119 10289 8214 1998 3002 2984 1005 1055 2267 1012 4406 24105 1998 1996 8514 1010 1996 9718 2003 2019 2981 4772 1998 2515 2025 2031 1037 4513 8619 2030 2151 8368 15709 2013 1996 2118 1012 1999 3055 1010 2043 2070 2493 3373 2008 1996 9718 2211 2000 2265 1037 4603 13827 1010 1037 4314 3780 1010 2691 3168 2001 2405 1012 10655 1010 1999 2494 1010 2043 2060 2493 3373 2008 1996 3259 3662 1037 4314 13827 1010 1996 4603 3259 3493 13631 2253 2046 2537 1012 4445 3259 2003 2405 2004 2411 2004 1996 9718 1025 2174 1010 2035 2093 2024 5500 2000 2035 2493 1012 2633 1010 1999 3500 2263 2019 8324 3485 2005 2576 2671 2470 1010 3458 4331 1010 2081 2049 2834 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 123\n",
+ "INFO:tensorflow:end_position: 124\n",
+ "INFO:tensorflow:answer: the observer\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000008\n",
+ "INFO:tensorflow:example_index: 8\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] how many student news papers are found at notre dame ? [SEP] as at most other universities , notre dame ' s students run a number of news media outlets . the nine student - run outlets include three newspapers , both a radio and television station , and several magazines and journals . begun as a one - page journal in september 1876 , the scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the united states . the other magazine , the jug ##gler , is released twice a year and focuses on student literature and artwork . the dome yearbook is published annually . the newspapers have varying publication interests , with the observer published daily and mainly reporting university and other news , and staffed by students from both notre dame and saint mary ' s college . unlike scholastic and the dome , the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university . in 1987 , when some students believed that the observer began to show a conservative bias , a liberal newspaper , common sense was published . likewise , in 2003 , when other students believed that the paper showed a liberal bias , the conservative paper irish rover went into production . neither paper is published as often as the observer ; however , all three are distributed to all students . finally , in spring 2008 an undergraduate journal for political science research , beyond politics , made its debut . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 13:0 14:1 15:2 16:3 17:4 18:4 19:5 20:6 21:6 22:6 23:7 24:8 25:9 26:10 27:11 28:12 29:13 30:14 31:14 32:15 33:16 34:17 35:17 36:17 37:18 38:19 39:20 40:21 41:21 42:22 43:23 44:24 45:25 46:26 47:27 48:27 49:28 50:29 51:30 52:31 53:32 54:32 55:33 56:34 57:35 58:36 59:36 60:36 61:37 62:38 63:39 64:40 65:40 66:41 67:42 68:43 69:44 70:45 71:46 72:47 73:48 74:49 75:50 76:51 77:52 78:53 79:54 80:55 81:56 82:57 83:58 84:59 85:60 86:60 87:61 88:62 89:63 90:63 91:64 92:65 93:65 94:65 95:66 96:67 97:68 98:69 99:70 100:71 101:72 102:73 103:74 104:75 105:76 106:77 107:77 108:78 109:79 110:80 111:81 112:82 113:83 114:83 115:84 116:85 117:86 118:87 119:88 120:89 121:89 122:90 123:91 124:92 125:93 126:94 127:95 128:96 129:97 130:98 131:99 132:100 133:101 134:101 135:102 136:103 137:104 138:105 139:106 140:107 141:108 142:109 143:110 144:111 145:112 146:112 147:112 148:113 149:113 150:114 151:115 152:116 153:117 154:118 155:118 156:119 157:120 158:121 159:122 160:123 161:124 162:125 163:126 164:127 165:128 166:129 167:130 168:131 169:132 170:133 171:134 172:135 173:136 174:137 175:138 176:138 177:139 178:140 179:140 180:141 181:142 182:143 183:144 184:145 185:146 186:147 187:148 188:149 189:150 190:151 191:152 192:153 193:153 194:154 195:155 196:156 197:156 198:157 199:158 200:159 201:160 202:160 203:161 204:161 205:162 206:163 207:163 208:164 209:165 210:166 211:167 212:168 213:169 214:170 215:171 216:172 217:173 218:174 219:174 220:175 221:176 222:177 223:178 224:179 225:180 226:181 227:182 228:182 229:183 230:184 231:185 232:186 233:187 234:188 235:189 236:190 237:191 238:191 239:192 240:192 241:193 242:194 243:195 244:196 245:197 246:198 247:199 248:199 249:200 250:200 251:201 252:202 253:203 254:204 255:205 256:206 257:207 258:208 259:209 260:210 261:210 262:211 263:212 264:212 265:213 266:214 267:215 268:215\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True 175:True 176:True 177:True 178:True 179:True 180:True 181:True 182:True 183:True 184:True 185:True 186:True 187:True 188:True 189:True 190:True 191:True 192:True 193:True 194:True 195:True 196:True 197:True 198:True 199:True 200:True 201:True 202:True 203:True 204:True 205:True 206:True 207:True 208:True 209:True 210:True 211:True 212:True 213:True 214:True 215:True 216:True 217:True 218:True 219:True 220:True 221:True 222:True 223:True 224:True 225:True 226:True 227:True 228:True 229:True 230:True 231:True 232:True 233:True 234:True 235:True 236:True 237:True 238:True 239:True 240:True 241:True 242:True 243:True 244:True 245:True 246:True 247:True 248:True 249:True 250:True 251:True 252:True 253:True 254:True 255:True 256:True 257:True 258:True 259:True 260:True 261:True 262:True 263:True 264:True 265:True 266:True 267:True 268:True\n",
+ "INFO:tensorflow:input_ids: 101 2129 2116 3076 2739 4981 2024 2179 2012 10289 8214 1029 102 2004 2012 2087 2060 5534 1010 10289 8214 1005 1055 2493 2448 1037 2193 1997 2739 2865 11730 1012 1996 3157 3076 1011 2448 11730 2421 2093 6399 1010 2119 1037 2557 1998 2547 2276 1010 1998 2195 7298 1998 9263 1012 5625 2004 1037 2028 1011 3931 3485 1999 2244 7326 1010 1996 24105 2932 2003 3843 3807 7058 1998 4447 2000 2022 1996 4587 7142 9234 4772 1999 1996 2142 2163 1012 1996 2060 2932 1010 1996 26536 17420 1010 2003 2207 3807 1037 2095 1998 7679 2006 3076 3906 1998 8266 1012 1996 8514 24803 2003 2405 6604 1012 1996 6399 2031 9671 4772 5426 1010 2007 1996 9718 2405 3679 1998 3701 7316 2118 1998 2060 2739 1010 1998 21121 2011 2493 2013 2119 10289 8214 1998 3002 2984 1005 1055 2267 1012 4406 24105 1998 1996 8514 1010 1996 9718 2003 2019 2981 4772 1998 2515 2025 2031 1037 4513 8619 2030 2151 8368 15709 2013 1996 2118 1012 1999 3055 1010 2043 2070 2493 3373 2008 1996 9718 2211 2000 2265 1037 4603 13827 1010 1037 4314 3780 1010 2691 3168 2001 2405 1012 10655 1010 1999 2494 1010 2043 2060 2493 3373 2008 1996 3259 3662 1037 4314 13827 1010 1996 4603 3259 3493 13631 2253 2046 2537 1012 4445 3259 2003 2405 2004 2411 2004 1996 9718 1025 2174 1010 2035 2093 2024 5500 2000 2035 2493 1012 2633 1010 1999 3500 2263 2019 8324 3485 2005 2576 2671 2470 1010 3458 4331 1010 2081 2049 2834 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 39\n",
+ "INFO:tensorflow:end_position: 39\n",
+ "INFO:tensorflow:answer: three\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000009\n",
+ "INFO:tensorflow:example_index: 9\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] in what year did the student paper common sense begin publication at notre dame ? [SEP] as at most other universities , notre dame ' s students run a number of news media outlets . the nine student - run outlets include three newspapers , both a radio and television station , and several magazines and journals . begun as a one - page journal in september 1876 , the scholastic magazine is issued twice monthly and claims to be the oldest continuous collegiate publication in the united states . the other magazine , the jug ##gler , is released twice a year and focuses on student literature and artwork . the dome yearbook is published annually . the newspapers have varying publication interests , with the observer published daily and mainly reporting university and other news , and staffed by students from both notre dame and saint mary ' s college . unlike scholastic and the dome , the observer is an independent publication and does not have a faculty advisor or any editorial oversight from the university . in 1987 , when some students believed that the observer began to show a conservative bias , a liberal newspaper , common sense was published . likewise , in 2003 , when other students believed that the paper showed a liberal bias , the conservative paper irish rover went into production . neither paper is published as often as the observer ; however , all three are distributed to all students . finally , in spring 2008 an undergraduate journal for political science research , beyond politics , made its debut . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 17:0 18:1 19:2 20:3 21:4 22:4 23:5 24:6 25:6 26:6 27:7 28:8 29:9 30:10 31:11 32:12 33:13 34:14 35:14 36:15 37:16 38:17 39:17 40:17 41:18 42:19 43:20 44:21 45:21 46:22 47:23 48:24 49:25 50:26 51:27 52:27 53:28 54:29 55:30 56:31 57:32 58:32 59:33 60:34 61:35 62:36 63:36 64:36 65:37 66:38 67:39 68:40 69:40 70:41 71:42 72:43 73:44 74:45 75:46 76:47 77:48 78:49 79:50 80:51 81:52 82:53 83:54 84:55 85:56 86:57 87:58 88:59 89:60 90:60 91:61 92:62 93:63 94:63 95:64 96:65 97:65 98:65 99:66 100:67 101:68 102:69 103:70 104:71 105:72 106:73 107:74 108:75 109:76 110:77 111:77 112:78 113:79 114:80 115:81 116:82 117:83 118:83 119:84 120:85 121:86 122:87 123:88 124:89 125:89 126:90 127:91 128:92 129:93 130:94 131:95 132:96 133:97 134:98 135:99 136:100 137:101 138:101 139:102 140:103 141:104 142:105 143:106 144:107 145:108 146:109 147:110 148:111 149:112 150:112 151:112 152:113 153:113 154:114 155:115 156:116 157:117 158:118 159:118 160:119 161:120 162:121 163:122 164:123 165:124 166:125 167:126 168:127 169:128 170:129 171:130 172:131 173:132 174:133 175:134 176:135 177:136 178:137 179:138 180:138 181:139 182:140 183:140 184:141 185:142 186:143 187:144 188:145 189:146 190:147 191:148 192:149 193:150 194:151 195:152 196:153 197:153 198:154 199:155 200:156 201:156 202:157 203:158 204:159 205:160 206:160 207:161 208:161 209:162 210:163 211:163 212:164 213:165 214:166 215:167 216:168 217:169 218:170 219:171 220:172 221:173 222:174 223:174 224:175 225:176 226:177 227:178 228:179 229:180 230:181 231:182 232:182 233:183 234:184 235:185 236:186 237:187 238:188 239:189 240:190 241:191 242:191 243:192 244:192 245:193 246:194 247:195 248:196 249:197 250:198 251:199 252:199 253:200 254:200 255:201 256:202 257:203 258:204 259:205 260:206 261:207 262:208 263:209 264:210 265:210 266:211 267:212 268:212 269:213 270:214 271:215 272:215\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True 160:True 161:True 162:True 163:True 164:True 165:True 166:True 167:True 168:True 169:True 170:True 171:True 172:True 173:True 174:True 175:True 176:True 177:True 178:True 179:True 180:True 181:True 182:True 183:True 184:True 185:True 186:True 187:True 188:True 189:True 190:True 191:True 192:True 193:True 194:True 195:True 196:True 197:True 198:True 199:True 200:True 201:True 202:True 203:True 204:True 205:True 206:True 207:True 208:True 209:True 210:True 211:True 212:True 213:True 214:True 215:True 216:True 217:True 218:True 219:True 220:True 221:True 222:True 223:True 224:True 225:True 226:True 227:True 228:True 229:True 230:True 231:True 232:True 233:True 234:True 235:True 236:True 237:True 238:True 239:True 240:True 241:True 242:True 243:True 244:True 245:True 246:True 247:True 248:True 249:True 250:True 251:True 252:True 253:True 254:True 255:True 256:True 257:True 258:True 259:True 260:True 261:True 262:True 263:True 264:True 265:True 266:True 267:True 268:True 269:True 270:True 271:True 272:True\n",
+ "INFO:tensorflow:input_ids: 101 1999 2054 2095 2106 1996 3076 3259 2691 3168 4088 4772 2012 10289 8214 1029 102 2004 2012 2087 2060 5534 1010 10289 8214 1005 1055 2493 2448 1037 2193 1997 2739 2865 11730 1012 1996 3157 3076 1011 2448 11730 2421 2093 6399 1010 2119 1037 2557 1998 2547 2276 1010 1998 2195 7298 1998 9263 1012 5625 2004 1037 2028 1011 3931 3485 1999 2244 7326 1010 1996 24105 2932 2003 3843 3807 7058 1998 4447 2000 2022 1996 4587 7142 9234 4772 1999 1996 2142 2163 1012 1996 2060 2932 1010 1996 26536 17420 1010 2003 2207 3807 1037 2095 1998 7679 2006 3076 3906 1998 8266 1012 1996 8514 24803 2003 2405 6604 1012 1996 6399 2031 9671 4772 5426 1010 2007 1996 9718 2405 3679 1998 3701 7316 2118 1998 2060 2739 1010 1998 21121 2011 2493 2013 2119 10289 8214 1998 3002 2984 1005 1055 2267 1012 4406 24105 1998 1996 8514 1010 1996 9718 2003 2019 2981 4772 1998 2515 2025 2031 1037 4513 8619 2030 2151 8368 15709 2013 1996 2118 1012 1999 3055 1010 2043 2070 2493 3373 2008 1996 9718 2211 2000 2265 1037 4603 13827 1010 1037 4314 3780 1010 2691 3168 2001 2405 1012 10655 1010 1999 2494 1010 2043 2060 2493 3373 2008 1996 3259 3662 1037 4314 13827 1010 1996 4603 3259 3493 13631 2253 2046 2537 1012 4445 3259 2003 2405 2004 2411 2004 1996 9718 1025 2174 1010 2035 2093 2024 5500 2000 2035 2493 1012 2633 1010 1999 3500 2263 2019 8324 3485 2005 2576 2671 2470 1010 3458 4331 1010 2081 2049 2834 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 182\n",
+ "INFO:tensorflow:end_position: 182\n",
+ "INFO:tensorflow:answer: 1987\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000010\n",
+ "INFO:tensorflow:example_index: 10\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] where is the headquarters of the congregation of the holy cross ? [SEP] the university is the major seat of the congregation of holy cross ( albeit not its official headquarters , which are in rome ) . its main seminary , more ##au seminary , is located on the campus across st . joseph lake from the main building . old college , the oldest building on campus and located near the shore of st . mary lake , houses undergraduate seminar ##ians . retired priests and brothers reside in fatima house ( a former retreat center ) , holy cross house , as well as col ##umb ##a hall near the gr ##otto . the university through the more ##au seminary has ties to theologian frederick bu ##ech ##ner . while not catholic , bu ##ech ##ner has praised writers from notre dame and more ##au seminary created a bu ##ech ##ner prize for preaching . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 14:0 15:1 16:2 17:3 18:4 19:5 20:6 21:7 22:8 23:9 24:10 25:11 26:12 27:12 28:13 29:14 30:15 31:16 32:16 33:17 34:18 35:19 36:20 37:20 38:20 39:21 40:22 41:23 42:23 43:24 44:24 45:25 46:25 47:26 48:27 49:28 50:29 51:30 52:31 53:32 54:32 55:33 56:34 57:35 58:36 59:37 60:38 61:38 62:39 63:40 64:40 65:41 66:42 67:43 68:44 69:45 70:46 71:47 72:48 73:49 74:50 75:51 76:52 77:52 78:53 79:54 80:54 81:55 82:56 83:57 84:57 85:57 86:58 87:59 88:60 89:61 90:62 91:63 92:64 93:65 94:66 95:66 96:67 97:68 98:69 99:69 100:69 101:70 102:71 103:72 104:72 105:73 106:74 107:75 108:76 109:76 110:76 111:77 112:78 113:79 114:80 115:80 116:80 117:81 118:82 119:83 120:84 121:85 122:85 123:86 124:87 125:88 126:89 127:90 128:91 129:92 130:92 131:92 132:92 133:93 134:94 135:95 136:95 137:96 138:96 139:96 140:97 141:98 142:99 143:100 144:101 145:102 146:103 147:104 148:104 149:105 150:106 151:107 152:108 153:108 154:108 155:109 156:110 157:111 158:111\n",
+ "INFO:tensorflow:token_is_max_context: 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:input_ids: 101 2073 2003 1996 4075 1997 1996 7769 1997 1996 4151 2892 1029 102 1996 2118 2003 1996 2350 2835 1997 1996 7769 1997 4151 2892 1006 12167 2025 2049 2880 4075 1010 2029 2024 1999 4199 1007 1012 2049 2364 8705 1010 2062 4887 8705 1010 2003 2284 2006 1996 3721 2408 2358 1012 3312 2697 2013 1996 2364 2311 1012 2214 2267 1010 1996 4587 2311 2006 3721 1998 2284 2379 1996 5370 1997 2358 1012 2984 2697 1010 3506 8324 18014 7066 1012 3394 8656 1998 3428 13960 1999 27596 2160 1006 1037 2280 7822 2415 1007 1010 4151 2892 2160 1010 2004 2092 2004 8902 25438 2050 2534 2379 1996 24665 23052 1012 1996 2118 2083 1996 2062 4887 8705 2038 7208 2000 17200 5406 20934 15937 3678 1012 2096 2025 3234 1010 20934 15937 3678 2038 5868 4898 2013 10289 8214 1998 2062 4887 8705 2580 1037 20934 15937 3678 3396 2005 17979 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 36\n",
+ "INFO:tensorflow:end_position: 36\n",
+ "INFO:tensorflow:answer: rome\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000011\n",
+ "INFO:tensorflow:example_index: 11\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what is the primary seminary of the congregation of the holy cross ? [SEP] the university is the major seat of the congregation of holy cross ( albeit not its official headquarters , which are in rome ) . its main seminary , more ##au seminary , is located on the campus across st . joseph lake from the main building . old college , the oldest building on campus and located near the shore of st . mary lake , houses undergraduate seminar ##ians . retired priests and brothers reside in fatima house ( a former retreat center ) , holy cross house , as well as col ##umb ##a hall near the gr ##otto . the university through the more ##au seminary has ties to theologian frederick bu ##ech ##ner . while not catholic , bu ##ech ##ner has praised writers from notre dame and more ##au seminary created a bu ##ech ##ner prize for preaching . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 15:0 16:1 17:2 18:3 19:4 20:5 21:6 22:7 23:8 24:9 25:10 26:11 27:12 28:12 29:13 30:14 31:15 32:16 33:16 34:17 35:18 36:19 37:20 38:20 39:20 40:21 41:22 42:23 43:23 44:24 45:24 46:25 47:25 48:26 49:27 50:28 51:29 52:30 53:31 54:32 55:32 56:33 57:34 58:35 59:36 60:37 61:38 62:38 63:39 64:40 65:40 66:41 67:42 68:43 69:44 70:45 71:46 72:47 73:48 74:49 75:50 76:51 77:52 78:52 79:53 80:54 81:54 82:55 83:56 84:57 85:57 86:57 87:58 88:59 89:60 90:61 91:62 92:63 93:64 94:65 95:66 96:66 97:67 98:68 99:69 100:69 101:69 102:70 103:71 104:72 105:72 106:73 107:74 108:75 109:76 110:76 111:76 112:77 113:78 114:79 115:80 116:80 117:80 118:81 119:82 120:83 121:84 122:85 123:85 124:86 125:87 126:88 127:89 128:90 129:91 130:92 131:92 132:92 133:92 134:93 135:94 136:95 137:95 138:96 139:96 140:96 141:97 142:98 143:99 144:100 145:101 146:102 147:103 148:104 149:104 150:105 151:106 152:107 153:108 154:108 155:108 156:109 157:110 158:111 159:111\n",
+ "INFO:tensorflow:token_is_max_context: 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True 157:True 158:True 159:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 2003 1996 3078 8705 1997 1996 7769 1997 1996 4151 2892 1029 102 1996 2118 2003 1996 2350 2835 1997 1996 7769 1997 4151 2892 1006 12167 2025 2049 2880 4075 1010 2029 2024 1999 4199 1007 1012 2049 2364 8705 1010 2062 4887 8705 1010 2003 2284 2006 1996 3721 2408 2358 1012 3312 2697 2013 1996 2364 2311 1012 2214 2267 1010 1996 4587 2311 2006 3721 1998 2284 2379 1996 5370 1997 2358 1012 2984 2697 1010 3506 8324 18014 7066 1012 3394 8656 1998 3428 13960 1999 27596 2160 1006 1037 2280 7822 2415 1007 1010 4151 2892 2160 1010 2004 2092 2004 8902 25438 2050 2534 2379 1996 24665 23052 1012 1996 2118 2083 1996 2062 4887 8705 2038 7208 2000 17200 5406 20934 15937 3678 1012 2096 2025 3234 1010 20934 15937 3678 2038 5868 4898 2013 10289 8214 1998 2062 4887 8705 2580 1037 20934 15937 3678 3396 2005 17979 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 44\n",
+ "INFO:tensorflow:end_position: 46\n",
+ "INFO:tensorflow:answer: more ##au seminary\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000012\n",
+ "INFO:tensorflow:example_index: 12\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what is the oldest structure at notre dame ? [SEP] the university is the major seat of the congregation of holy cross ( albeit not its official headquarters , which are in rome ) . its main seminary , more ##au seminary , is located on the campus across st . joseph lake from the main building . old college , the oldest building on campus and located near the shore of st . mary lake , houses undergraduate seminar ##ians . retired priests and brothers reside in fatima house ( a former retreat center ) , holy cross house , as well as col ##umb ##a hall near the gr ##otto . the university through the more ##au seminary has ties to theologian frederick bu ##ech ##ner . while not catholic , bu ##ech ##ner has praised writers from notre dame and more ##au seminary created a bu ##ech ##ner prize for preaching . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 11:0 12:1 13:2 14:3 15:4 16:5 17:6 18:7 19:8 20:9 21:10 22:11 23:12 24:12 25:13 26:14 27:15 28:16 29:16 30:17 31:18 32:19 33:20 34:20 35:20 36:21 37:22 38:23 39:23 40:24 41:24 42:25 43:25 44:26 45:27 46:28 47:29 48:30 49:31 50:32 51:32 52:33 53:34 54:35 55:36 56:37 57:38 58:38 59:39 60:40 61:40 62:41 63:42 64:43 65:44 66:45 67:46 68:47 69:48 70:49 71:50 72:51 73:52 74:52 75:53 76:54 77:54 78:55 79:56 80:57 81:57 82:57 83:58 84:59 85:60 86:61 87:62 88:63 89:64 90:65 91:66 92:66 93:67 94:68 95:69 96:69 97:69 98:70 99:71 100:72 101:72 102:73 103:74 104:75 105:76 106:76 107:76 108:77 109:78 110:79 111:80 112:80 113:80 114:81 115:82 116:83 117:84 118:85 119:85 120:86 121:87 122:88 123:89 124:90 125:91 126:92 127:92 128:92 129:92 130:93 131:94 132:95 133:95 134:96 135:96 136:96 137:97 138:98 139:99 140:100 141:101 142:102 143:103 144:104 145:104 146:105 147:106 148:107 149:108 150:108 151:108 152:109 153:110 154:111 155:111\n",
+ "INFO:tensorflow:token_is_max_context: 11:True 12:True 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 2003 1996 4587 3252 2012 10289 8214 1029 102 1996 2118 2003 1996 2350 2835 1997 1996 7769 1997 4151 2892 1006 12167 2025 2049 2880 4075 1010 2029 2024 1999 4199 1007 1012 2049 2364 8705 1010 2062 4887 8705 1010 2003 2284 2006 1996 3721 2408 2358 1012 3312 2697 2013 1996 2364 2311 1012 2214 2267 1010 1996 4587 2311 2006 3721 1998 2284 2379 1996 5370 1997 2358 1012 2984 2697 1010 3506 8324 18014 7066 1012 3394 8656 1998 3428 13960 1999 27596 2160 1006 1037 2280 7822 2415 1007 1010 4151 2892 2160 1010 2004 2092 2004 8902 25438 2050 2534 2379 1996 24665 23052 1012 1996 2118 2083 1996 2062 4887 8705 2038 7208 2000 17200 5406 20934 15937 3678 1012 2096 2025 3234 1010 20934 15937 3678 2038 5868 4898 2013 10289 8214 1998 2062 4887 8705 2580 1037 20934 15937 3678 3396 2005 17979 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 59\n",
+ "INFO:tensorflow:end_position: 60\n",
+ "INFO:tensorflow:answer: old college\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000013\n",
+ "INFO:tensorflow:example_index: 13\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] what individuals live at fatima house at notre dame ? [SEP] the university is the major seat of the congregation of holy cross ( albeit not its official headquarters , which are in rome ) . its main seminary , more ##au seminary , is located on the campus across st . joseph lake from the main building . old college , the oldest building on campus and located near the shore of st . mary lake , houses undergraduate seminar ##ians . retired priests and brothers reside in fatima house ( a former retreat center ) , holy cross house , as well as col ##umb ##a hall near the gr ##otto . the university through the more ##au seminary has ties to theologian frederick bu ##ech ##ner . while not catholic , bu ##ech ##ner has praised writers from notre dame and more ##au seminary created a bu ##ech ##ner prize for preaching . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 12:0 13:1 14:2 15:3 16:4 17:5 18:6 19:7 20:8 21:9 22:10 23:11 24:12 25:12 26:13 27:14 28:15 29:16 30:16 31:17 32:18 33:19 34:20 35:20 36:20 37:21 38:22 39:23 40:23 41:24 42:24 43:25 44:25 45:26 46:27 47:28 48:29 49:30 50:31 51:32 52:32 53:33 54:34 55:35 56:36 57:37 58:38 59:38 60:39 61:40 62:40 63:41 64:42 65:43 66:44 67:45 68:46 69:47 70:48 71:49 72:50 73:51 74:52 75:52 76:53 77:54 78:54 79:55 80:56 81:57 82:57 83:57 84:58 85:59 86:60 87:61 88:62 89:63 90:64 91:65 92:66 93:66 94:67 95:68 96:69 97:69 98:69 99:70 100:71 101:72 102:72 103:73 104:74 105:75 106:76 107:76 108:76 109:77 110:78 111:79 112:80 113:80 114:80 115:81 116:82 117:83 118:84 119:85 120:85 121:86 122:87 123:88 124:89 125:90 126:91 127:92 128:92 129:92 130:92 131:93 132:94 133:95 134:95 135:96 136:96 137:96 138:97 139:98 140:99 141:100 142:101 143:102 144:103 145:104 146:104 147:105 148:106 149:107 150:108 151:108 152:108 153:109 154:110 155:111 156:111\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_is_max_context: 12:True 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True\n",
+ "INFO:tensorflow:input_ids: 101 2054 3633 2444 2012 27596 2160 2012 10289 8214 1029 102 1996 2118 2003 1996 2350 2835 1997 1996 7769 1997 4151 2892 1006 12167 2025 2049 2880 4075 1010 2029 2024 1999 4199 1007 1012 2049 2364 8705 1010 2062 4887 8705 1010 2003 2284 2006 1996 3721 2408 2358 1012 3312 2697 2013 1996 2364 2311 1012 2214 2267 1010 1996 4587 2311 2006 3721 1998 2284 2379 1996 5370 1997 2358 1012 2984 2697 1010 3506 8324 18014 7066 1012 3394 8656 1998 3428 13960 1999 27596 2160 1006 1037 2280 7822 2415 1007 1010 4151 2892 2160 1010 2004 2092 2004 8902 25438 2050 2534 2379 1996 24665 23052 1012 1996 2118 2083 1996 2062 4887 8705 2038 7208 2000 17200 5406 20934 15937 3678 1012 2096 2025 3234 1010 20934 15937 3678 2038 5868 4898 2013 10289 8214 1998 2062 4887 8705 2580 1037 20934 15937 3678 3396 2005 17979 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 84\n",
+ "INFO:tensorflow:end_position: 87\n",
+ "INFO:tensorflow:answer: retired priests and brothers\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000014\n",
+ "INFO:tensorflow:example_index: 14\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] which prize did frederick bu ##ech ##ner create ? [SEP] the university is the major seat of the congregation of holy cross ( albeit not its official headquarters , which are in rome ) . its main seminary , more ##au seminary , is located on the campus across st . joseph lake from the main building . old college , the oldest building on campus and located near the shore of st . mary lake , houses undergraduate seminar ##ians . retired priests and brothers reside in fatima house ( a former retreat center ) , holy cross house , as well as col ##umb ##a hall near the gr ##otto . the university through the more ##au seminary has ties to theologian frederick bu ##ech ##ner . while not catholic , bu ##ech ##ner has praised writers from notre dame and more ##au seminary created a bu ##ech ##ner prize for preaching . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 11:0 12:1 13:2 14:3 15:4 16:5 17:6 18:7 19:8 20:9 21:10 22:11 23:12 24:12 25:13 26:14 27:15 28:16 29:16 30:17 31:18 32:19 33:20 34:20 35:20 36:21 37:22 38:23 39:23 40:24 41:24 42:25 43:25 44:26 45:27 46:28 47:29 48:30 49:31 50:32 51:32 52:33 53:34 54:35 55:36 56:37 57:38 58:38 59:39 60:40 61:40 62:41 63:42 64:43 65:44 66:45 67:46 68:47 69:48 70:49 71:50 72:51 73:52 74:52 75:53 76:54 77:54 78:55 79:56 80:57 81:57 82:57 83:58 84:59 85:60 86:61 87:62 88:63 89:64 90:65 91:66 92:66 93:67 94:68 95:69 96:69 97:69 98:70 99:71 100:72 101:72 102:73 103:74 104:75 105:76 106:76 107:76 108:77 109:78 110:79 111:80 112:80 113:80 114:81 115:82 116:83 117:84 118:85 119:85 120:86 121:87 122:88 123:89 124:90 125:91 126:92 127:92 128:92 129:92 130:93 131:94 132:95 133:95 134:96 135:96 136:96 137:97 138:98 139:99 140:100 141:101 142:102 143:103 144:104 145:104 146:105 147:106 148:107 149:108 150:108 151:108 152:109 153:110 154:111 155:111\n",
+ "INFO:tensorflow:token_is_max_context: 11:True 12:True 13:True 14:True 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True\n",
+ "INFO:tensorflow:input_ids: 101 2029 3396 2106 5406 20934 15937 3678 3443 1029 102 1996 2118 2003 1996 2350 2835 1997 1996 7769 1997 4151 2892 1006 12167 2025 2049 2880 4075 1010 2029 2024 1999 4199 1007 1012 2049 2364 8705 1010 2062 4887 8705 1010 2003 2284 2006 1996 3721 2408 2358 1012 3312 2697 2013 1996 2364 2311 1012 2214 2267 1010 1996 4587 2311 2006 3721 1998 2284 2379 1996 5370 1997 2358 1012 2984 2697 1010 3506 8324 18014 7066 1012 3394 8656 1998 3428 13960 1999 27596 2160 1006 1037 2280 7822 2415 1007 1010 4151 2892 2160 1010 2004 2092 2004 8902 25438 2050 2534 2379 1996 24665 23052 1012 1996 2118 2083 1996 2062 4887 8705 2038 7208 2000 17200 5406 20934 15937 3678 1012 2096 2025 3234 1010 20934 15937 3678 2038 5868 4898 2013 10289 8214 1998 2062 4887 8705 2580 1037 20934 15937 3678 3396 2005 17979 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 149\n",
+ "INFO:tensorflow:end_position: 154\n",
+ "INFO:tensorflow:answer: bu ##ech ##ner prize for preaching\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000015\n",
+ "INFO:tensorflow:example_index: 15\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] how many bs level degrees are offered in the college of engineering at notre dame ? [SEP] the college of engineering was established in 1920 , however , early courses in civil and mechanical engineering were a part of the college of science since the 1870s . today the college , housed in the fitzpatrick , cu ##shing , and st ##ins ##on - re ##mic ##k halls of engineering , includes five departments of study – aerospace and mechanical engineering , chemical and bio ##mo ##le ##cular engineering , civil engineering and geological sciences , computer science and engineering , and electrical engineering – with eight b . s . degrees offered . additionally , the college offers five - year dual degree programs with the colleges of arts and letters and of business awarding additional b . a . and master of business administration ( mba ) degrees , respectively . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 18:0 19:1 20:2 21:3 22:4 23:5 24:6 25:7 26:7 27:8 28:8 29:9 30:10 31:11 32:12 33:13 34:14 35:15 36:16 37:17 38:18 39:19 40:20 41:21 42:22 43:23 44:24 45:25 46:26 47:26 48:27 49:28 50:29 51:29 52:30 53:31 54:32 55:33 56:33 57:34 58:34 59:34 60:35 61:36 62:36 63:36 64:36 65:36 66:36 67:36 68:37 69:38 70:39 71:39 72:40 73:41 74:42 75:43 76:44 77:45 78:46 79:47 80:48 81:49 82:49 83:50 84:51 85:52 86:52 87:52 88:52 89:53 90:53 91:54 92:55 93:56 94:57 95:58 96:58 97:59 98:60 99:61 100:62 101:62 102:63 103:64 104:65 105:66 106:67 107:68 108:69 109:69 110:69 111:69 112:70 113:71 114:71 115:72 116:72 117:73 118:74 119:75 120:76 121:76 122:76 123:77 124:78 125:79 126:80 127:81 128:82 129:83 130:84 131:85 132:86 133:87 134:88 135:89 136:90 137:91 138:92 139:92 140:92 141:92 142:93 143:94 144:95 145:96 146:97 147:98 148:98 149:98 150:99 151:99 152:100 153:100\n",
+ "INFO:tensorflow:token_is_max_context: 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True\n",
+ "INFO:tensorflow:input_ids: 101 2129 2116 18667 2504 5445 2024 3253 1999 1996 2267 1997 3330 2012 10289 8214 1029 102 1996 2267 1997 3330 2001 2511 1999 4444 1010 2174 1010 2220 5352 1999 2942 1998 6228 3330 2020 1037 2112 1997 1996 2267 1997 2671 2144 1996 14896 1012 2651 1996 2267 1010 7431 1999 1996 26249 1010 12731 12227 1010 1998 2358 7076 2239 1011 2128 7712 2243 9873 1997 3330 1010 2950 2274 7640 1997 2817 1516 13395 1998 6228 3330 1010 5072 1998 16012 5302 2571 15431 3330 1010 2942 3330 1998 9843 4163 1010 3274 2671 1998 3330 1010 1998 5992 3330 1516 2007 2809 1038 1012 1055 1012 5445 3253 1012 5678 1010 1996 2267 4107 2274 1011 2095 7037 3014 3454 2007 1996 6667 1997 2840 1998 4144 1998 1997 2449 21467 3176 1038 1012 1037 1012 1998 3040 1997 2449 3447 1006 15038 1007 5445 1010 4414 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 107\n",
+ "INFO:tensorflow:end_position: 107\n",
+ "INFO:tensorflow:answer: eight\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000016\n",
+ "INFO:tensorflow:example_index: 16\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] in what year was the college of engineering at notre dame formed ? [SEP] the college of engineering was established in 1920 , however , early courses in civil and mechanical engineering were a part of the college of science since the 1870s . today the college , housed in the fitzpatrick , cu ##shing , and st ##ins ##on - re ##mic ##k halls of engineering , includes five departments of study – aerospace and mechanical engineering , chemical and bio ##mo ##le ##cular engineering , civil engineering and geological sciences , computer science and engineering , and electrical engineering – with eight b . s . degrees offered . additionally , the college offers five - year dual degree programs with the colleges of arts and letters and of business awarding additional b . a . and master of business administration ( mba ) degrees , respectively . [SEP]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_to_orig_map: 15:0 16:1 17:2 18:3 19:4 20:5 21:6 22:7 23:7 24:8 25:8 26:9 27:10 28:11 29:12 30:13 31:14 32:15 33:16 34:17 35:18 36:19 37:20 38:21 39:22 40:23 41:24 42:25 43:26 44:26 45:27 46:28 47:29 48:29 49:30 50:31 51:32 52:33 53:33 54:34 55:34 56:34 57:35 58:36 59:36 60:36 61:36 62:36 63:36 64:36 65:37 66:38 67:39 68:39 69:40 70:41 71:42 72:43 73:44 74:45 75:46 76:47 77:48 78:49 79:49 80:50 81:51 82:52 83:52 84:52 85:52 86:53 87:53 88:54 89:55 90:56 91:57 92:58 93:58 94:59 95:60 96:61 97:62 98:62 99:63 100:64 101:65 102:66 103:67 104:68 105:69 106:69 107:69 108:69 109:70 110:71 111:71 112:72 113:72 114:73 115:74 116:75 117:76 118:76 119:76 120:77 121:78 122:79 123:80 124:81 125:82 126:83 127:84 128:85 129:86 130:87 131:88 132:89 133:90 134:91 135:92 136:92 137:92 138:92 139:93 140:94 141:95 142:96 143:97 144:98 145:98 146:98 147:99 148:99 149:100 150:100\n",
+ "INFO:tensorflow:token_is_max_context: 15:True 16:True 17:True 18:True 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True\n",
+ "INFO:tensorflow:input_ids: 101 1999 2054 2095 2001 1996 2267 1997 3330 2012 10289 8214 2719 1029 102 1996 2267 1997 3330 2001 2511 1999 4444 1010 2174 1010 2220 5352 1999 2942 1998 6228 3330 2020 1037 2112 1997 1996 2267 1997 2671 2144 1996 14896 1012 2651 1996 2267 1010 7431 1999 1996 26249 1010 12731 12227 1010 1998 2358 7076 2239 1011 2128 7712 2243 9873 1997 3330 1010 2950 2274 7640 1997 2817 1516 13395 1998 6228 3330 1010 5072 1998 16012 5302 2571 15431 3330 1010 2942 3330 1998 9843 4163 1010 3274 2671 1998 3330 1010 1998 5992 3330 1516 2007 2809 1038 1012 1055 1012 5445 3253 1012 5678 1010 1996 2267 4107 2274 1011 2095 7037 3014 3454 2007 1996 6667 1997 2840 1998 4144 1998 1997 2449 21467 3176 1038 1012 1037 1012 1998 3040 1997 2449 3447 1006 15038 1007 5445 1010 4414 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 22\n",
+ "INFO:tensorflow:end_position: 22\n",
+ "INFO:tensorflow:answer: 1920\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000017\n",
+ "INFO:tensorflow:example_index: 17\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] before the creation of the college of engineering similar studies were carried out at which notre dame college ? [SEP] the college of engineering was established in 1920 , however , early courses in civil and mechanical engineering were a part of the college of science since the 1870s . today the college , housed in the fitzpatrick , cu ##shing , and st ##ins ##on - re ##mic ##k halls of engineering , includes five departments of study – aerospace and mechanical engineering , chemical and bio ##mo ##le ##cular engineering , civil engineering and geological sciences , computer science and engineering , and electrical engineering – with eight b . s . degrees offered . additionally , the college offers five - year dual degree programs with the colleges of arts and letters and of business awarding additional b . a . and master of business administration ( mba ) degrees , respectively . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 21:0 22:1 23:2 24:3 25:4 26:5 27:6 28:7 29:7 30:8 31:8 32:9 33:10 34:11 35:12 36:13 37:14 38:15 39:16 40:17 41:18 42:19 43:20 44:21 45:22 46:23 47:24 48:25 49:26 50:26 51:27 52:28 53:29 54:29 55:30 56:31 57:32 58:33 59:33 60:34 61:34 62:34 63:35 64:36 65:36 66:36 67:36 68:36 69:36 70:36 71:37 72:38 73:39 74:39 75:40 76:41 77:42 78:43 79:44 80:45 81:46 82:47 83:48 84:49 85:49 86:50 87:51 88:52 89:52 90:52 91:52 92:53 93:53 94:54 95:55 96:56 97:57 98:58 99:58 100:59 101:60 102:61 103:62 104:62 105:63 106:64 107:65 108:66 109:67 110:68 111:69 112:69 113:69 114:69 115:70 116:71 117:71 118:72 119:72 120:73 121:74 122:75 123:76 124:76 125:76 126:77 127:78 128:79 129:80 130:81 131:82 132:83 133:84 134:85 135:86 136:87 137:88 138:89 139:90 140:91 141:92 142:92 143:92 144:92 145:93 146:94 147:95 148:96 149:97 150:98 151:98 152:98 153:99 154:99 155:100 156:100\n",
+ "INFO:tensorflow:token_is_max_context: 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True 156:True\n",
+ "INFO:tensorflow:input_ids: 101 2077 1996 4325 1997 1996 2267 1997 3330 2714 2913 2020 3344 2041 2012 2029 10289 8214 2267 1029 102 1996 2267 1997 3330 2001 2511 1999 4444 1010 2174 1010 2220 5352 1999 2942 1998 6228 3330 2020 1037 2112 1997 1996 2267 1997 2671 2144 1996 14896 1012 2651 1996 2267 1010 7431 1999 1996 26249 1010 12731 12227 1010 1998 2358 7076 2239 1011 2128 7712 2243 9873 1997 3330 1010 2950 2274 7640 1997 2817 1516 13395 1998 6228 3330 1010 5072 1998 16012 5302 2571 15431 3330 1010 2942 3330 1998 9843 4163 1010 3274 2671 1998 3330 1010 1998 5992 3330 1516 2007 2809 1038 1012 1055 1012 5445 3253 1012 5678 1010 1996 2267 4107 2274 1011 2095 7037 3014 3454 2007 1996 6667 1997 2840 1998 4144 1998 1997 2449 21467 3176 1038 1012 1037 1012 1998 3040 1997 2449 3447 1006 15038 1007 5445 1010 4414 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 43\n",
+ "INFO:tensorflow:end_position: 46\n",
+ "INFO:tensorflow:answer: the college of science\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000018\n",
+ "INFO:tensorflow:example_index: 18\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] how many departments are within the st ##ins ##on - re ##mic ##k hall of engineering ? [SEP] the college of engineering was established in 1920 , however , early courses in civil and mechanical engineering were a part of the college of science since the 1870s . today the college , housed in the fitzpatrick , cu ##shing , and st ##ins ##on - re ##mic ##k halls of engineering , includes five departments of study – aerospace and mechanical engineering , chemical and bio ##mo ##le ##cular engineering , civil engineering and geological sciences , computer science and engineering , and electrical engineering – with eight b . s . degrees offered . additionally , the college offers five - year dual degree programs with the colleges of arts and letters and of business awarding additional b . a . and master of business administration ( mba ) degrees , respectively . [SEP]\n",
+ "INFO:tensorflow:token_to_orig_map: 19:0 20:1 21:2 22:3 23:4 24:5 25:6 26:7 27:7 28:8 29:8 30:9 31:10 32:11 33:12 34:13 35:14 36:15 37:16 38:17 39:18 40:19 41:20 42:21 43:22 44:23 45:24 46:25 47:26 48:26 49:27 50:28 51:29 52:29 53:30 54:31 55:32 56:33 57:33 58:34 59:34 60:34 61:35 62:36 63:36 64:36 65:36 66:36 67:36 68:36 69:37 70:38 71:39 72:39 73:40 74:41 75:42 76:43 77:44 78:45 79:46 80:47 81:48 82:49 83:49 84:50 85:51 86:52 87:52 88:52 89:52 90:53 91:53 92:54 93:55 94:56 95:57 96:58 97:58 98:59 99:60 100:61 101:62 102:62 103:63 104:64 105:65 106:66 107:67 108:68 109:69 110:69 111:69 112:69 113:70 114:71 115:71 116:72 117:72 118:73 119:74 120:75 121:76 122:76 123:76 124:77 125:78 126:79 127:80 128:81 129:82 130:83 131:84 132:85 133:86 134:87 135:88 136:89 137:90 138:91 139:92 140:92 141:92 142:92 143:93 144:94 145:95 146:96 147:97 148:98 149:98 150:98 151:99 152:99 153:100 154:100\n",
+ "INFO:tensorflow:token_is_max_context: 19:True 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True\n",
+ "INFO:tensorflow:input_ids: 101 2129 2116 7640 2024 2306 1996 2358 7076 2239 1011 2128 7712 2243 2534 1997 3330 1029 102 1996 2267 1997 3330 2001 2511 1999 4444 1010 2174 1010 2220 5352 1999 2942 1998 6228 3330 2020 1037 2112 1997 1996 2267 1997 2671 2144 1996 14896 1012 2651 1996 2267 1010 7431 1999 1996 26249 1010 12731 12227 1010 1998 2358 7076 2239 1011 2128 7712 2243 9873 1997 3330 1010 2950 2274 7640 1997 2817 1516 13395 1998 6228 3330 1010 5072 1998 16012 5302 2571 15431 3330 1010 2942 3330 1998 9843 4163 1010 3274 2671 1998 3330 1010 1998 5992 3330 1516 2007 2809 1038 1012 1055 1012 5445 3253 1012 5678 1010 1996 2267 4107 2274 1011 2095 7037 3014 3454 2007 1996 6667 1997 2840 1998 4144 1998 1997 2449 21467 3176 1038 1012 1037 1012 1998 3040 1997 2449 3447 1006 15038 1007 5445 1010 4414 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 74\n",
+ "INFO:tensorflow:end_position: 74\n",
+ "INFO:tensorflow:answer: five\n",
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 1000000019\n",
+ "INFO:tensorflow:example_index: 19\n",
+ "INFO:tensorflow:doc_span_index: 0\n",
+ "INFO:tensorflow:tokens: [CLS] the college of science began to offer civil engineering courses beginning at what time at notre dame ? [SEP] the college of engineering was established in 1920 , however , early courses in civil and mechanical engineering were a part of the college of science since the 1870s . today the college , housed in the fitzpatrick , cu ##shing , and st ##ins ##on - re ##mic ##k halls of engineering , includes five departments of study – aerospace and mechanical engineering , chemical and bio ##mo ##le ##cular engineering , civil engineering and geological sciences , computer science and engineering , and electrical engineering – with eight b . s . degrees offered . additionally , the college offers five - year dual degree programs with the colleges of arts and letters and of business awarding additional b . a . and master of business administration ( mba ) degrees , respectively . [SEP]\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:token_to_orig_map: 20:0 21:1 22:2 23:3 24:4 25:5 26:6 27:7 28:7 29:8 30:8 31:9 32:10 33:11 34:12 35:13 36:14 37:15 38:16 39:17 40:18 41:19 42:20 43:21 44:22 45:23 46:24 47:25 48:26 49:26 50:27 51:28 52:29 53:29 54:30 55:31 56:32 57:33 58:33 59:34 60:34 61:34 62:35 63:36 64:36 65:36 66:36 67:36 68:36 69:36 70:37 71:38 72:39 73:39 74:40 75:41 76:42 77:43 78:44 79:45 80:46 81:47 82:48 83:49 84:49 85:50 86:51 87:52 88:52 89:52 90:52 91:53 92:53 93:54 94:55 95:56 96:57 97:58 98:58 99:59 100:60 101:61 102:62 103:62 104:63 105:64 106:65 107:66 108:67 109:68 110:69 111:69 112:69 113:69 114:70 115:71 116:71 117:72 118:72 119:73 120:74 121:75 122:76 123:76 124:76 125:77 126:78 127:79 128:80 129:81 130:82 131:83 132:84 133:85 134:86 135:87 136:88 137:89 138:90 139:91 140:92 141:92 142:92 143:92 144:93 145:94 146:95 147:96 148:97 149:98 150:98 151:98 152:99 153:99 154:100 155:100\n",
+ "INFO:tensorflow:token_is_max_context: 20:True 21:True 22:True 23:True 24:True 25:True 26:True 27:True 28:True 29:True 30:True 31:True 32:True 33:True 34:True 35:True 36:True 37:True 38:True 39:True 40:True 41:True 42:True 43:True 44:True 45:True 46:True 47:True 48:True 49:True 50:True 51:True 52:True 53:True 54:True 55:True 56:True 57:True 58:True 59:True 60:True 61:True 62:True 63:True 64:True 65:True 66:True 67:True 68:True 69:True 70:True 71:True 72:True 73:True 74:True 75:True 76:True 77:True 78:True 79:True 80:True 81:True 82:True 83:True 84:True 85:True 86:True 87:True 88:True 89:True 90:True 91:True 92:True 93:True 94:True 95:True 96:True 97:True 98:True 99:True 100:True 101:True 102:True 103:True 104:True 105:True 106:True 107:True 108:True 109:True 110:True 111:True 112:True 113:True 114:True 115:True 116:True 117:True 118:True 119:True 120:True 121:True 122:True 123:True 124:True 125:True 126:True 127:True 128:True 129:True 130:True 131:True 132:True 133:True 134:True 135:True 136:True 137:True 138:True 139:True 140:True 141:True 142:True 143:True 144:True 145:True 146:True 147:True 148:True 149:True 150:True 151:True 152:True 153:True 154:True 155:True\n",
+ "INFO:tensorflow:input_ids: 101 1996 2267 1997 2671 2211 2000 3749 2942 3330 5352 2927 2012 2054 2051 2012 10289 8214 1029 102 1996 2267 1997 3330 2001 2511 1999 4444 1010 2174 1010 2220 5352 1999 2942 1998 6228 3330 2020 1037 2112 1997 1996 2267 1997 2671 2144 1996 14896 1012 2651 1996 2267 1010 7431 1999 1996 26249 1010 12731 12227 1010 1998 2358 7076 2239 1011 2128 7712 2243 9873 1997 3330 1010 2950 2274 7640 1997 2817 1516 13395 1998 6228 3330 1010 5072 1998 16012 5302 2571 15431 3330 1010 2942 3330 1998 9843 4163 1010 3274 2671 1998 3330 1010 1998 5992 3330 1516 2007 2809 1038 1012 1055 1012 5445 3253 1012 5678 1010 1996 2267 4107 2274 1011 2095 7037 3014 3454 2007 1996 6667 1997 2840 1998 4144 1998 1997 2449 21467 3176 1038 1012 1037 1012 1998 3040 1997 2449 3447 1006 15038 1007 5445 1010 4414 1012 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:start_position: 47\n",
+ "INFO:tensorflow:end_position: 48\n",
+ "INFO:tensorflow:answer: the 1870s\n"
+ ]
+ }
+ ],
+ "source": [
+ "bert_config = modeling_tensorflow.BertConfig.from_json_file(bert_config_file)\n",
+ "tokenizer = tokenization.BertTokenizer(\n",
+ " vocab_file=vocab_file, do_lower_case=True)\n",
+ "\n",
+ "eval_examples = read_squad_examples(\n",
+ " input_file=input_file, is_training=True, max_num=16)\n",
+ "\n",
+ "eval_features = convert_examples_to_features(\n",
+ " examples=eval_examples,\n",
+ " tokenizer=tokenizer,\n",
+ " max_seq_length=max_seq_length,\n",
+ " doc_stride=doc_stride,\n",
+ " max_query_length=max_query_length,\n",
+ " is_training=True)\n",
+ "\n",
+ "# You can use that to test the behavior of the models when target are outside of the model input sequence\n",
+ "# for feature in eval_features:\n",
+ "# feature.start_position = outside_pos\n",
+ "# feature.end_position = outside_pos"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:37.525632Z",
+ "start_time": "2018-11-06T10:11:37.498695Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "eval_unique_id_to_feature = {}\n",
+ "for eval_feature in eval_features:\n",
+ " eval_unique_id_to_feature[eval_feature.unique_id] = eval_feature"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:37.558325Z",
+ "start_time": "2018-11-06T10:11:37.527972Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "def input_fn_builder(features, seq_length, drop_remainder):\n",
+ " \"\"\"Creates an `input_fn` closure to be passed to TPUEstimator.\"\"\"\n",
+ "\n",
+ " all_unique_ids = []\n",
+ " all_input_ids = []\n",
+ " all_input_mask = []\n",
+ " all_segment_ids = []\n",
+ " all_start_positions = []\n",
+ " all_end_positions = []\n",
+ "\n",
+ " for feature in features:\n",
+ " all_unique_ids.append(feature.unique_id)\n",
+ " all_input_ids.append(feature.input_ids)\n",
+ " all_input_mask.append(feature.input_mask)\n",
+ " all_segment_ids.append(feature.segment_ids)\n",
+ " all_start_positions.append(feature.start_position)\n",
+ " all_end_positions.append(feature.end_position)\n",
+ "\n",
+ " def input_fn(params):\n",
+ " \"\"\"The actual input function.\"\"\"\n",
+ " batch_size = params[\"batch_size\"]\n",
+ "\n",
+ " num_examples = len(features)\n",
+ "\n",
+ " # This is for demo purposes and does NOT scale to large data sets. We do\n",
+ " # not use Dataset.from_generator() because that uses tf.py_func which is\n",
+ " # not TPU compatible. The right way to load data is with TFRecordReader.\n",
+ " feature_map = {\n",
+ " \"unique_ids\":\n",
+ " tf.constant(all_unique_ids, shape=[num_examples], dtype=tf.int32),\n",
+ " \"input_ids\":\n",
+ " tf.constant(\n",
+ " all_input_ids, shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"input_mask\":\n",
+ " tf.constant(\n",
+ " all_input_mask,\n",
+ " shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"segment_ids\":\n",
+ " tf.constant(\n",
+ " all_segment_ids,\n",
+ " shape=[num_examples, seq_length],\n",
+ " dtype=tf.int32),\n",
+ " \"start_positions\":\n",
+ " tf.constant(\n",
+ " all_start_positions,\n",
+ " shape=[num_examples],\n",
+ " dtype=tf.int32),\n",
+ " \"end_positions\":\n",
+ " tf.constant(\n",
+ " all_end_positions,\n",
+ " shape=[num_examples],\n",
+ " dtype=tf.int32),\n",
+ " }\n",
+ "\n",
+ " d = tf.data.Dataset.from_tensor_slices(feature_map)\n",
+ " d = d.repeat()\n",
+ " d = d.batch(batch_size=batch_size, drop_remainder=drop_remainder)\n",
+ " return d\n",
+ "\n",
+ " return input_fn"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:37.601666Z",
+ "start_time": "2018-11-06T10:11:37.560082Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "def model_fn_builder(bert_config, init_checkpoint, learning_rate,\n",
+ " num_train_steps, num_warmup_steps, use_tpu,\n",
+ " use_one_hot_embeddings):\n",
+ " \"\"\"Returns `model_fn` closure for TPUEstimator.\"\"\"\n",
+ "\n",
+ " def model_fn(features, labels, mode, params): # pylint: disable=unused-argument\n",
+ " \"\"\"The `model_fn` for TPUEstimator.\"\"\"\n",
+ "\n",
+ " tf.logging.info(\"*** Features ***\")\n",
+ " for name in sorted(features.keys()):\n",
+ " tf.logging.info(\" name = %s, shape = %s\" % (name, features[name].shape))\n",
+ "\n",
+ " unique_ids = features[\"unique_ids\"]\n",
+ " input_ids = features[\"input_ids\"]\n",
+ " input_mask = features[\"input_mask\"]\n",
+ " segment_ids = features[\"segment_ids\"]\n",
+ "\n",
+ " is_training = (mode == tf.estimator.ModeKeys.TRAIN)\n",
+ "\n",
+ " (start_logits, end_logits) = create_model(\n",
+ " bert_config=bert_config,\n",
+ " is_training=is_training,\n",
+ " input_ids=input_ids,\n",
+ " input_mask=input_mask,\n",
+ " segment_ids=segment_ids,\n",
+ " use_one_hot_embeddings=use_one_hot_embeddings)\n",
+ "\n",
+ " tvars = tf.trainable_variables()\n",
+ "\n",
+ " initialized_variable_names = {}\n",
+ " scaffold_fn = None\n",
+ " if init_checkpoint:\n",
+ " (assignment_map,\n",
+ " initialized_variable_names) = modeling_tensorflow.get_assigment_map_from_checkpoint(\n",
+ " tvars, init_checkpoint)\n",
+ " if use_tpu:\n",
+ "\n",
+ " def tpu_scaffold():\n",
+ " tf.train.init_from_checkpoint(init_checkpoint, assignment_map)\n",
+ " return tf.train.Scaffold()\n",
+ "\n",
+ " scaffold_fn = tpu_scaffold\n",
+ " else:\n",
+ " tf.train.init_from_checkpoint(init_checkpoint, assignment_map)\n",
+ "\n",
+ " tf.logging.info(\"**** Trainable Variables ****\")\n",
+ " for var in tvars:\n",
+ " init_string = \"\"\n",
+ " if var.name in initialized_variable_names:\n",
+ " init_string = \", *INIT_FROM_CKPT*\"\n",
+ " tf.logging.info(\" name = %s, shape = %s%s\", var.name, var.shape,\n",
+ " init_string)\n",
+ "\n",
+ " output_spec = None\n",
+ " if mode == tf.estimator.ModeKeys.TRAIN:\n",
+ " seq_length = modeling_tensorflow.get_shape_list(input_ids)[1]\n",
+ "\n",
+ " def compute_loss(logits, positions):\n",
+ " one_hot_positions = tf.one_hot(\n",
+ " positions, depth=seq_length, dtype=tf.float32)\n",
+ " log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
+ " loss = -tf.reduce_mean(\n",
+ " tf.reduce_sum(one_hot_positions * log_probs, axis=-1))\n",
+ " return loss\n",
+ "\n",
+ " start_positions = features[\"start_positions\"]\n",
+ " end_positions = features[\"end_positions\"]\n",
+ "\n",
+ " start_loss = compute_loss(start_logits, start_positions)\n",
+ " end_loss = compute_loss(end_logits, end_positions)\n",
+ "\n",
+ " total_loss = (start_loss + end_loss) / 2.0\n",
+ "\n",
+ " train_op = optimization.create_optimizer(\n",
+ " total_loss, learning_rate, num_train_steps, num_warmup_steps, use_tpu)\n",
+ "\n",
+ " output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
+ " mode=mode,\n",
+ " loss=total_loss,\n",
+ " train_op=train_op,\n",
+ " scaffold_fn=scaffold_fn)\n",
+ " elif mode == tf.estimator.ModeKeys.PREDICT:\n",
+ " batch_size = modeling_tensorflow.get_shape_list(start_logits)[0]\n",
+ " seq_length = modeling_tensorflow.get_shape_list(input_ids)[1]\n",
+ "\n",
+ " def compute_loss(logits, positions):\n",
+ " one_hot_positions = tf.one_hot(\n",
+ " positions, depth=seq_length, dtype=tf.float32)\n",
+ " log_probs = tf.nn.log_softmax(logits, axis=-1)\n",
+ " loss = -tf.reduce_mean(\n",
+ " tf.reduce_sum(one_hot_positions * log_probs, axis=-1))\n",
+ " return loss\n",
+ "\n",
+ " start_positions = features[\"start_positions\"]\n",
+ " end_positions = features[\"end_positions\"]\n",
+ "\n",
+ " start_loss = compute_loss(start_logits, start_positions)\n",
+ " end_loss = compute_loss(end_logits, end_positions)\n",
+ "\n",
+ " total_loss = (start_loss + end_loss) / 2.0\n",
+ "\n",
+ " predictions = {\n",
+ " \"unique_ids\": unique_ids,\n",
+ " \"start_logits\": start_logits,\n",
+ " \"end_logits\": end_logits,\n",
+ " \"total_loss\": tf.reshape(total_loss, [batch_size, 1]),\n",
+ " \"start_loss\": tf.reshape(start_loss, [batch_size, 1]),\n",
+ " \"end_loss\": tf.reshape(end_loss, [batch_size, 1]),\n",
+ " }\n",
+ " output_spec = tf.contrib.tpu.TPUEstimatorSpec(\n",
+ " mode=mode, predictions=predictions, scaffold_fn=scaffold_fn)\n",
+ " else:\n",
+ " raise ValueError(\n",
+ " \"Only TRAIN and PREDICT modes are supported: %s\" % (mode))\n",
+ "\n",
+ " return output_spec\n",
+ "\n",
+ " return model_fn"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:41.104542Z",
+ "start_time": "2018-11-06T10:11:37.603474Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x120df3f28>) includes params argument, but params are not passed to Estimator.\n",
+ "INFO:tensorflow:Using config: {'_model_dir': '/tmp/squad_base/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': 1000, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true\n",
+ "graph_options {\n",
+ " rewrite_options {\n",
+ " meta_optimizer_iterations: ONE\n",
+ " }\n",
+ "}\n",
+ ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=1000, num_shards=8, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n",
+ "INFO:tensorflow:_TPUContext: eval_on_tpu True\n",
+ "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n"
+ ]
+ }
+ ],
+ "source": [
+ "is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
+ "run_config = tf.contrib.tpu.RunConfig(\n",
+ " cluster=None,\n",
+ " master=None,\n",
+ " model_dir=output_dir,\n",
+ " save_checkpoints_steps=1000,\n",
+ " tpu_config=tf.contrib.tpu.TPUConfig(\n",
+ " iterations_per_loop=1000,\n",
+ " num_shards=8,\n",
+ " per_host_input_for_training=is_per_host))\n",
+ "\n",
+ "model_fn = model_fn_builder(\n",
+ " bert_config=bert_config,\n",
+ " init_checkpoint=init_checkpoint,\n",
+ " learning_rate=learning_rate,\n",
+ " num_train_steps=None,\n",
+ " num_warmup_steps=None,\n",
+ " use_tpu=False,\n",
+ " use_one_hot_embeddings=False)\n",
+ "\n",
+ "estimator = tf.contrib.tpu.TPUEstimator(\n",
+ " use_tpu=False,\n",
+ " model_fn=model_fn,\n",
+ " config=run_config,\n",
+ " train_batch_size=12,\n",
+ " predict_batch_size=1)\n",
+ "\n",
+ "predict_input_fn = input_fn_builder(\n",
+ " features=eval_features,\n",
+ " seq_length=max_seq_length,\n",
+ " drop_remainder=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:47.857601Z",
+ "start_time": "2018-11-06T10:11:41.106219Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Could not find trained model in model_dir: /tmp/squad_base/, running initialization to predict.\n",
+ "INFO:tensorflow:Calling model_fn.\n",
+ "INFO:tensorflow:Running infer on CPU\n",
+ "INFO:tensorflow:*** Features ***\n",
+ "INFO:tensorflow: name = end_positions, shape = (1,)\n",
+ "INFO:tensorflow: name = input_ids, shape = (1, 384)\n",
+ "INFO:tensorflow: name = input_mask, shape = (1, 384)\n",
+ "INFO:tensorflow: name = segment_ids, shape = (1, 384)\n",
+ "INFO:tensorflow: name = start_positions, shape = (1,)\n",
+ "INFO:tensorflow: name = unique_ids, shape = (1,)\n",
+ "INFO:tensorflow:**** Trainable Variables ****\n",
+ "INFO:tensorflow: name = bert/embeddings/word_embeddings:0, shape = (30522, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/embeddings/token_type_embeddings:0, shape = (2, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/embeddings/position_embeddings:0, shape = (512, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/embeddings/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/embeddings/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_0/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_1/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_2/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_3/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_4/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_5/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_6/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_7/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_8/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_9/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_10/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/query/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/key/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/self/value/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/attention/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/kernel:0, shape = (768, 3072), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/intermediate/dense/bias:0, shape = (3072,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/kernel:0, shape = (3072, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/beta:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/encoder/layer_11/output/LayerNorm/gamma:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/pooler/dense/kernel:0, shape = (768, 768), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = bert/pooler/dense/bias:0, shape = (768,), *INIT_FROM_CKPT*\n",
+ "INFO:tensorflow: name = cls/squad/output_weights:0, shape = (2, 768)\n",
+ "INFO:tensorflow: name = cls/squad/output_bias:0, shape = (2,)\n",
+ "INFO:tensorflow:Done calling model_fn.\n",
+ "INFO:tensorflow:Graph was finalized.\n",
+ "INFO:tensorflow:Running local_init_op.\n",
+ "INFO:tensorflow:Done running local_init_op.\n",
+ "INFO:tensorflow:prediction_loop marked as finished\n"
+ ]
+ }
+ ],
+ "source": [
+ "tensorflow_all_out = []\n",
+ "tensorflow_all_results = []\n",
+ "for result in estimator.predict(predict_input_fn, yield_single_examples=True):\n",
+ " unique_id = int(result[\"unique_ids\"])\n",
+ " eval_feature = eval_unique_id_to_feature[unique_id]\n",
+ " start_logits = result[\"start_logits\"]\n",
+ " end_logits = result[\"end_logits\"]\n",
+ " total_loss = result[\"total_loss\"]\n",
+ " start_loss = result[\"start_loss\"]\n",
+ " end_loss = result[\"end_loss\"]\n",
+ "\n",
+ " output_json = collections.OrderedDict()\n",
+ " output_json[\"linex_index\"] = unique_id\n",
+ " output_json[\"tokens\"] = [token for (i, token) in enumerate(eval_feature.tokens)]\n",
+ " output_json[\"start_logits\"] = [round(float(x), 6) for x in start_logits.flat]\n",
+ " output_json[\"end_logits\"] = [round(float(x), 6) for x in end_logits.flat]\n",
+ " output_json[\"total_loss\"] = [round(float(x), 6) for x in total_loss.flat]\n",
+ " output_json[\"start_loss\"] = [round(float(x), 6) for x in start_loss.flat]\n",
+ " output_json[\"end_loss\"] = [round(float(x), 6) for x in end_loss.flat]\n",
+ " tensorflow_all_out.append(output_json)\n",
+ " tensorflow_all_results.append(RawResult(\n",
+ " unique_id=unique_id,\n",
+ " start_logits=start_logits,\n",
+ " end_logits=end_logits))\n",
+ " break"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:47.912836Z",
+ "start_time": "2018-11-06T10:11:47.859679Z"
+ },
+ "code_folding": []
+ },
+ "outputs": [],
+ "source": [
+ "def _get_best_indexes(logits, n_best_size):\n",
+ " \"\"\"Get the n-best logits from a list.\"\"\"\n",
+ " index_and_score = sorted(enumerate(logits), key=lambda x: x[1], reverse=True)\n",
+ "\n",
+ " best_indexes = []\n",
+ " for i in range(len(index_and_score)):\n",
+ " if i >= n_best_size:\n",
+ " break\n",
+ " best_indexes.append(index_and_score[i][0])\n",
+ " return best_indexes\n",
+ "\n",
+ "def _compute_softmax(scores):\n",
+ " \"\"\"Compute softmax probability over raw logits.\"\"\"\n",
+ " if not scores:\n",
+ " return []\n",
+ "\n",
+ " max_score = None\n",
+ " for score in scores:\n",
+ " if max_score is None or score > max_score:\n",
+ " max_score = score\n",
+ "\n",
+ " exp_scores = []\n",
+ " total_sum = 0.0\n",
+ " for score in scores:\n",
+ " x = math.exp(score - max_score)\n",
+ " exp_scores.append(x)\n",
+ " total_sum += x\n",
+ "\n",
+ " probs = []\n",
+ " for score in exp_scores:\n",
+ " probs.append(score / total_sum)\n",
+ " return probs\n",
+ "\n",
+ "\n",
+ "def compute_predictions(all_examples, all_features, all_results, n_best_size,\n",
+ " max_answer_length, do_lower_case):\n",
+ " \"\"\"Compute final predictions.\"\"\"\n",
+ " example_index_to_features = collections.defaultdict(list)\n",
+ " for feature in all_features:\n",
+ " example_index_to_features[feature.example_index].append(feature)\n",
+ "\n",
+ " unique_id_to_result = {}\n",
+ " for result in all_results:\n",
+ " unique_id_to_result[result.unique_id] = result\n",
+ "\n",
+ " _PrelimPrediction = collections.namedtuple( # pylint: disable=invalid-name\n",
+ " \"PrelimPrediction\",\n",
+ " [\"feature_index\", \"start_index\", \"end_index\", \"start_logit\", \"end_logit\"])\n",
+ "\n",
+ " all_predictions = collections.OrderedDict()\n",
+ " all_nbest_json = collections.OrderedDict()\n",
+ " for (example_index, example) in enumerate(all_examples):\n",
+ " features = example_index_to_features[example_index]\n",
+ "\n",
+ " prelim_predictions = []\n",
+ " for (feature_index, feature) in enumerate(features):\n",
+ " result = unique_id_to_result[feature.unique_id]\n",
+ "\n",
+ " start_indexes = _get_best_indexes(result.start_logits, n_best_size)\n",
+ " end_indexes = _get_best_indexes(result.end_logits, n_best_size)\n",
+ " for start_index in start_indexes:\n",
+ " for end_index in end_indexes:\n",
+ " # We could hypothetically create invalid predictions, e.g., predict\n",
+ " # that the start of the span is in the question. We throw out all\n",
+ " # invalid predictions.\n",
+ " if start_index >= len(feature.tokens):\n",
+ " continue\n",
+ " if end_index >= len(feature.tokens):\n",
+ " continue\n",
+ " if start_index not in feature.token_to_orig_map:\n",
+ " continue\n",
+ " if end_index not in feature.token_to_orig_map:\n",
+ " continue\n",
+ " if not feature.token_is_max_context.get(start_index, False):\n",
+ " continue\n",
+ " if end_index < start_index:\n",
+ " continue\n",
+ " length = end_index - start_index + 1\n",
+ " if length > max_answer_length:\n",
+ " continue\n",
+ " prelim_predictions.append(\n",
+ " _PrelimPrediction(\n",
+ " feature_index=feature_index,\n",
+ " start_index=start_index,\n",
+ " end_index=end_index,\n",
+ " start_logit=result.start_logits[start_index],\n",
+ " end_logit=result.end_logits[end_index]))\n",
+ "\n",
+ " prelim_predictions = sorted(\n",
+ " prelim_predictions,\n",
+ " key=lambda x: (x.start_logit + x.end_logit),\n",
+ " reverse=True)\n",
+ "\n",
+ " _NbestPrediction = collections.namedtuple( # pylint: disable=invalid-name\n",
+ " \"NbestPrediction\", [\"text\", \"start_logit\", \"end_logit\"])\n",
+ "\n",
+ " seen_predictions = {}\n",
+ " nbest = []\n",
+ " for pred in prelim_predictions:\n",
+ " if len(nbest) >= n_best_size:\n",
+ " break\n",
+ " feature = features[pred.feature_index]\n",
+ "\n",
+ " tok_tokens = feature.tokens[pred.start_index:(pred.end_index + 1)]\n",
+ " orig_doc_start = feature.token_to_orig_map[pred.start_index]\n",
+ " orig_doc_end = feature.token_to_orig_map[pred.end_index]\n",
+ " orig_tokens = example.doc_tokens[orig_doc_start:(orig_doc_end + 1)]\n",
+ " tok_text = \" \".join(tok_tokens)\n",
+ "\n",
+ " # De-tokenize WordPieces that have been split off.\n",
+ " tok_text = tok_text.replace(\" ##\", \"\")\n",
+ " tok_text = tok_text.replace(\"##\", \"\")\n",
+ "\n",
+ " # Clean whitespace\n",
+ " tok_text = tok_text.strip()\n",
+ " tok_text = \" \".join(tok_text.split())\n",
+ " orig_text = \" \".join(orig_tokens)\n",
+ "\n",
+ " final_text = get_final_text(tok_text, orig_text, do_lower_case)\n",
+ " if final_text in seen_predictions:\n",
+ " continue\n",
+ "\n",
+ " seen_predictions[final_text] = True\n",
+ " nbest.append(\n",
+ " _NbestPrediction(\n",
+ " text=final_text,\n",
+ " start_logit=pred.start_logit,\n",
+ " end_logit=pred.end_logit))\n",
+ "\n",
+ " # In very rare edge cases we could have no valid predictions. So we\n",
+ " # just create a nonce prediction in this case to avoid failure.\n",
+ " if not nbest:\n",
+ " nbest.append(\n",
+ " _NbestPrediction(text=\"empty\", start_logit=0.0, end_logit=0.0))\n",
+ "\n",
+ " assert len(nbest) >= 1\n",
+ "\n",
+ " total_scores = []\n",
+ " for entry in nbest:\n",
+ " total_scores.append(entry.start_logit + entry.end_logit)\n",
+ "\n",
+ " probs = _compute_softmax(total_scores)\n",
+ "\n",
+ " nbest_json = []\n",
+ " for (i, entry) in enumerate(nbest):\n",
+ " output = collections.OrderedDict()\n",
+ " output[\"text\"] = entry.text\n",
+ " output[\"probability\"] = probs[i]\n",
+ " output[\"start_logit\"] = entry.start_logit\n",
+ " output[\"end_logit\"] = entry.end_logit\n",
+ " nbest_json.append(output)\n",
+ "\n",
+ " assert len(nbest_json) >= 1\n",
+ "\n",
+ " all_predictions[example.qas_id] = nbest_json[0][\"text\"]\n",
+ " all_nbest_json[example.qas_id] = nbest_json\n",
+ "\n",
+ " return all_predictions, all_nbest_json"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:47.953205Z",
+ "start_time": "2018-11-06T10:11:47.914751Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "all_predictions, all_nbest_json = compute_predictions(eval_examples[:1], eval_features[:1], tensorflow_all_results, 20, max_answer_length, True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:47.994647Z",
+ "start_time": "2018-11-06T10:11:47.955015Z"
+ }
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "OrderedDict([('5733be284776f41900661182',\n",
+ " [OrderedDict([('text', 'empty'),\n",
+ " ('probability', 1.0),\n",
+ " ('start_logit', 0.0),\n",
+ " ('end_logit', 0.0)])])])"
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "all_nbest_json"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:48.028473Z",
+ "start_time": "2018-11-06T10:11:47.996311Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\n",
+ "7\n",
+ "odict_keys(['linex_index', 'tokens', 'start_logits', 'end_logits', 'total_loss', 'start_loss', 'end_loss'])\n",
+ "number of tokens 176\n",
+ "number of start_logits 384\n",
+ "shape of end_logits 384\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(len(tensorflow_all_out))\n",
+ "print(len(tensorflow_all_out[0]))\n",
+ "print(tensorflow_all_out[0].keys())\n",
+ "print(\"number of tokens\", len(tensorflow_all_out[0]['tokens']))\n",
+ "print(\"number of start_logits\", len(tensorflow_all_out[0]['start_logits']))\n",
+ "print(\"shape of end_logits\", len(tensorflow_all_out[0]['end_logits']))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:48.060658Z",
+ "start_time": "2018-11-06T10:11:48.030289Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "tensorflow_outputs = [tensorflow_all_out[0]['start_logits'], tensorflow_all_out[0]['end_logits'],\n",
+ " tensorflow_all_out[0]['total_loss'], tensorflow_all_out[0]['start_loss'],\n",
+ " tensorflow_all_out[0]['end_loss']]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2/ PyTorch code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:48.478814Z",
+ "start_time": "2018-11-06T10:11:48.062585Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import modeling\n",
+ "from run_squad import *"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 16,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:48.512607Z",
+ "start_time": "2018-11-06T10:11:48.480729Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "init_checkpoint_pt = \"../google_models/uncased_L-12_H-768_A-12/pytorch_model.bin\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:51.023405Z",
+ "start_time": "2018-11-06T10:11:48.514306Z"
+ },
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "tensor([0., 0.])"
+ ]
+ },
+ "execution_count": 17,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "device = torch.device(\"cpu\")\n",
+ "model = modeling.BertForQuestionAnswering(bert_config)\n",
+ "model.bert.load_state_dict(torch.load(init_checkpoint_pt, map_location='cpu'))\n",
+ "model.to(device)\n",
+ "model.qa_outputs.weight.data.fill_(1.0)\n",
+ "model.qa_outputs.bias.data.zero_()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:51.079364Z",
+ "start_time": "2018-11-06T10:11:51.028228Z"
+ },
+ "code_folding": []
+ },
+ "outputs": [],
+ "source": [
+ "all_input_ids = torch.tensor([f.input_ids for f in eval_features], dtype=torch.long)\n",
+ "all_input_mask = torch.tensor([f.input_mask for f in eval_features], dtype=torch.long)\n",
+ "all_segment_ids = torch.tensor([f.segment_ids for f in eval_features], dtype=torch.long)\n",
+ "all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)\n",
+ "all_start_positions = torch.tensor([[f.start_position] for f in eval_features], dtype=torch.long)\n",
+ "all_end_positions = torch.tensor([[f.end_position] for f in eval_features], dtype=torch.long)\n",
+ "\n",
+ "eval_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids,\n",
+ " all_start_positions, all_end_positions, all_example_index)\n",
+ "eval_sampler = SequentialSampler(eval_data)\n",
+ "eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=1)\n",
+ "\n",
+ "model.eval()\n",
+ "None"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:51.114686Z",
+ "start_time": "2018-11-06T10:11:51.081474Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "[torch.Size([1, 384]), torch.Size([1, 384]), torch.Size([1, 384]), torch.Size([1, 1]), torch.Size([1, 1]), torch.Size([1])]\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "torch.Size([1, 1])"
+ ]
+ },
+ "execution_count": 19,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "batch = iter(eval_dataloader).next()\n",
+ "input_ids, input_mask, segment_ids, start_positions, end_positions, example_index = batch\n",
+ "print([t.shape for t in batch])\n",
+ "start_positions.size()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-06T10:11:52.298367Z",
+ "start_time": "2018-11-06T10:11:51.116219Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Evaluating: 0%| | 0/270 [00:00\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 4\u001b[0m \u001b[0mspec\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mimportlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutil\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mspec_from_file_location\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'*'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moriginal_tf_inplem_dir\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m'/extract_features_tensorflow.py'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 5\u001b[0m \u001b[0mmodule\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mimportlib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mutil\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodule_from_spec\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mspec\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mspec\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloader\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexec_module\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmodule\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 7\u001b[0m \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodules\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'extract_features_tensorflow'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodule\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 8\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/importlib/_bootstrap_external.py\u001b[0m in \u001b[0;36mexec_module\u001b[0;34m(self, module)\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/importlib/_bootstrap.py\u001b[0m in \u001b[0;36m_call_with_frames_removed\u001b[0;34m(f, *args, **kwds)\u001b[0m\n",
+ "\u001b[0;32m~/Documents/Thomas/Code/HF/BERT/pytorch-pretrained-BERT/tensorflow_code/extract_features_tensorflow.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[1;32m 32\u001b[0m \u001b[0mFLAGS\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mFLAGS\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 33\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 34\u001b[0;31m \u001b[0mflags\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDEFINE_string\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"input_file\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 35\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 36\u001b[0m \u001b[0mflags\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDEFINE_string\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"output_file\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/site-packages/tensorflow/python/platform/flags.py\u001b[0m in \u001b[0;36mwrapper\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m 56\u001b[0m \u001b[0;34m'Use of the keyword argument names (flag_name, default_value, '\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 57\u001b[0m 'docstring) is deprecated, please use (name, default, help) instead.')\n\u001b[0;32m---> 58\u001b[0;31m \u001b[0;32mreturn\u001b[0m \u001b[0moriginal_function\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 60\u001b[0m \u001b[0;32mreturn\u001b[0m \u001b[0mtf_decorator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmake_decorator\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moriginal_function\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwrapper\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/site-packages/absl/flags/_defines.py\u001b[0m in \u001b[0;36mDEFINE_string\u001b[0;34m(name, default, help, flag_values, **args)\u001b[0m\n\u001b[1;32m 239\u001b[0m \u001b[0mparser\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_argument_parser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mArgumentParser\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 240\u001b[0m \u001b[0mserializer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_argument_parser\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mArgumentSerializer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 241\u001b[0;31m \u001b[0mDEFINE\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparser\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdefault\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhelp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mflag_values\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mserializer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 242\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 243\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/site-packages/absl/flags/_defines.py\u001b[0m in \u001b[0;36mDEFINE\u001b[0;34m(parser, name, default, help, flag_values, serializer, module_name, **args)\u001b[0m\n\u001b[1;32m 80\u001b[0m \"\"\"\n\u001b[1;32m 81\u001b[0m DEFINE_flag(_flag.Flag(parser, serializer, name, default, help, **args),\n\u001b[0;32m---> 82\u001b[0;31m flag_values, module_name)\n\u001b[0m\u001b[1;32m 83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 84\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/site-packages/absl/flags/_defines.py\u001b[0m in \u001b[0;36mDEFINE_flag\u001b[0;34m(flag, flag_values, module_name)\u001b[0m\n\u001b[1;32m 102\u001b[0m \u001b[0;31m# Copying the reference to flag_values prevents pychecker warnings.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 103\u001b[0m \u001b[0mfv\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mflag_values\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 104\u001b[0;31m \u001b[0mfv\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mflag\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 105\u001b[0m \u001b[0;31m# Tell flag_values who's defining the flag.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 106\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mmodule_name\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;32m~/miniconda3/envs/bert/lib/python3.6/site-packages/absl/flags/_flagvalues.py\u001b[0m in \u001b[0;36m__setitem__\u001b[0;34m(self, name, flag)\u001b[0m\n\u001b[1;32m 427\u001b[0m \u001b[0;31m# module is simply being imported a subsequent time.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 428\u001b[0m \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 429\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0m_exceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDuplicateFlagError\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_flag\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 430\u001b[0m \u001b[0mshort_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mflag\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshort_name\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 431\u001b[0m \u001b[0;31m# If a new flag overrides an old one, we need to cleanup the old flag's\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
+ "\u001b[0;31mDuplicateFlagError\u001b[0m: The flag 'input_file' is defined twice. First from *, Second from *. Description from first occurrence: (no help available)"
+ ]
+ }
+ ],
+ "source": [
+ "import importlib.util\n",
+ "import sys\n",
+ "\n",
+ "spec = importlib.util.spec_from_file_location('*', original_tf_inplem_dir + '/extract_features_tensorflow.py')\n",
+ "module = importlib.util.module_from_spec(spec)\n",
+ "spec.loader.exec_module(module)\n",
+ "sys.modules['extract_features_tensorflow'] = module\n",
+ "\n",
+ "from extract_features_tensorflow import *"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T14:58:05.650987Z",
+ "start_time": "2018-11-15T14:58:05.541620Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:*** Example ***\n",
+ "INFO:tensorflow:unique_id: 0\n",
+ "INFO:tensorflow:tokens: [CLS] who was jim henson ? [SEP] jim henson was a puppet ##eer [SEP]\n",
+ "INFO:tensorflow:input_ids: 101 2040 2001 3958 27227 1029 102 3958 27227 2001 1037 13997 11510 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
+ "INFO:tensorflow:input_type_ids: 0 0 0 0 0 0 0 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n"
+ ]
+ }
+ ],
+ "source": [
+ "layer_indexes = list(range(12))\n",
+ "bert_config = modeling.BertConfig.from_json_file(bert_config_file)\n",
+ "tokenizer = tokenization.FullTokenizer(\n",
+ " vocab_file=vocab_file, do_lower_case=True)\n",
+ "examples = read_examples(input_file)\n",
+ "\n",
+ "features = convert_examples_to_features(\n",
+ " examples=examples, seq_length=max_seq_length, tokenizer=tokenizer)\n",
+ "unique_id_to_feature = {}\n",
+ "for feature in features:\n",
+ " unique_id_to_feature[feature.unique_id] = feature"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T14:58:11.562443Z",
+ "start_time": "2018-11-15T14:58:08.036485Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x11ea7f1e0>) includes params argument, but params are not passed to Estimator.\n",
+ "WARNING:tensorflow:Using temporary folder as model directory: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmphs4_nsq9\n",
+ "INFO:tensorflow:Using config: {'_model_dir': '/var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmphs4_nsq9', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true\n",
+ "graph_options {\n",
+ " rewrite_options {\n",
+ " meta_optimizer_iterations: ONE\n",
+ " }\n",
+ "}\n",
+ ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_tpu_config': TPUConfig(iterations_per_loop=2, num_shards=1, num_cores_per_replica=None, per_host_input_for_training=3, tpu_job_name=None, initial_infeed_sleep_secs=None, input_partition_dims=None), '_cluster': None}\n",
+ "WARNING:tensorflow:Setting TPUConfig.num_shards==1 is an unsupported behavior. Please fix as soon as possible (leaving num_shards as None.\n",
+ "INFO:tensorflow:_TPUContext: eval_on_tpu True\n",
+ "WARNING:tensorflow:eval_on_tpu ignored because use_tpu is False.\n"
+ ]
+ }
+ ],
+ "source": [
+ "is_per_host = tf.contrib.tpu.InputPipelineConfig.PER_HOST_V2\n",
+ "run_config = tf.contrib.tpu.RunConfig(\n",
+ " master=None,\n",
+ " tpu_config=tf.contrib.tpu.TPUConfig(\n",
+ " num_shards=1,\n",
+ " per_host_input_for_training=is_per_host))\n",
+ "\n",
+ "model_fn = model_fn_builder(\n",
+ " bert_config=bert_config,\n",
+ " init_checkpoint=init_checkpoint,\n",
+ " layer_indexes=layer_indexes,\n",
+ " use_tpu=False,\n",
+ " use_one_hot_embeddings=False)\n",
+ "\n",
+ "# If TPU is not available, this will fall back to normal Estimator on CPU\n",
+ "# or GPU.\n",
+ "estimator = tf.contrib.tpu.TPUEstimator(\n",
+ " use_tpu=False,\n",
+ " model_fn=model_fn,\n",
+ " config=run_config,\n",
+ " predict_batch_size=1)\n",
+ "\n",
+ "input_fn = input_fn_builder(\n",
+ " features=features, seq_length=max_seq_length)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T14:58:21.736543Z",
+ "start_time": "2018-11-15T14:58:16.723829Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "INFO:tensorflow:Could not find trained model in model_dir: /var/folders/yx/cw8n_njx3js5jksyw_qlp8p00000gn/T/tmphs4_nsq9, running initialization to predict.\n",
+ "INFO:tensorflow:Calling model_fn.\n",
+ "INFO:tensorflow:Running infer on CPU\n",
+ "INFO:tensorflow:Done calling model_fn.\n",
+ "INFO:tensorflow:Graph was finalized.\n",
+ "INFO:tensorflow:Running local_init_op.\n",
+ "INFO:tensorflow:Done running local_init_op.\n",
+ "extracting layer 0\n",
+ "extracting layer 1\n",
+ "extracting layer 2\n",
+ "extracting layer 3\n",
+ "extracting layer 4\n",
+ "extracting layer 5\n",
+ "extracting layer 6\n",
+ "extracting layer 7\n",
+ "extracting layer 8\n",
+ "extracting layer 9\n",
+ "extracting layer 10\n",
+ "extracting layer 11\n",
+ "INFO:tensorflow:prediction_loop marked as finished\n",
+ "INFO:tensorflow:prediction_loop marked as finished\n"
+ ]
+ }
+ ],
+ "source": [
+ "tensorflow_all_out = []\n",
+ "for result in estimator.predict(input_fn, yield_single_examples=True):\n",
+ " unique_id = int(result[\"unique_id\"])\n",
+ " feature = unique_id_to_feature[unique_id]\n",
+ " output_json = collections.OrderedDict()\n",
+ " output_json[\"linex_index\"] = unique_id\n",
+ " tensorflow_all_out_features = []\n",
+ " # for (i, token) in enumerate(feature.tokens):\n",
+ " all_layers = []\n",
+ " for (j, layer_index) in enumerate(layer_indexes):\n",
+ " print(\"extracting layer {}\".format(j))\n",
+ " layer_output = result[\"layer_output_%d\" % j]\n",
+ " layers = collections.OrderedDict()\n",
+ " layers[\"index\"] = layer_index\n",
+ " layers[\"values\"] = layer_output\n",
+ " all_layers.append(layers)\n",
+ " tensorflow_out_features = collections.OrderedDict()\n",
+ " tensorflow_out_features[\"layers\"] = all_layers\n",
+ " tensorflow_all_out_features.append(tensorflow_out_features)\n",
+ "\n",
+ " output_json[\"features\"] = tensorflow_all_out_features\n",
+ " tensorflow_all_out.append(output_json)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T14:58:23.970714Z",
+ "start_time": "2018-11-15T14:58:23.931930Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\n",
+ "2\n",
+ "odict_keys(['linex_index', 'features'])\n",
+ "number of tokens 1\n",
+ "number of layers 12\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "(128, 768)"
+ ]
+ },
+ "execution_count": 11,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "print(len(tensorflow_all_out))\n",
+ "print(len(tensorflow_all_out[0]))\n",
+ "print(tensorflow_all_out[0].keys())\n",
+ "print(\"number of tokens\", len(tensorflow_all_out[0]['features']))\n",
+ "print(\"number of layers\", len(tensorflow_all_out[0]['features'][0]['layers']))\n",
+ "tensorflow_all_out[0]['features'][0]['layers'][0]['values'].shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T14:58:25.547012Z",
+ "start_time": "2018-11-15T14:58:25.516076Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "tensorflow_outputs = list(tensorflow_all_out[0]['features'][0]['layers'][t]['values'] for t in layer_indexes)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 2/ PyTorch code"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "os.chdir('./examples')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:03:49.528679Z",
+ "start_time": "2018-11-15T15:03:49.497697Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import extract_features\n",
+ "import pytorch_pretrained_bert as ppb\n",
+ "from extract_features import *"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:18.001177Z",
+ "start_time": "2018-11-15T15:21:17.970369Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "init_checkpoint_pt = \"../../google_models/uncased_L-12_H-768_A-12/\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:20.893669Z",
+ "start_time": "2018-11-15T15:21:18.786623Z"
+ },
+ "scrolled": true
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "11/15/2018 16:21:18 - INFO - pytorch_pretrained_bert.modeling - loading archive file ../../google_models/uncased_L-12_H-768_A-12/\n",
+ "11/15/2018 16:21:18 - INFO - pytorch_pretrained_bert.modeling - Model config {\n",
+ " \"attention_probs_dropout_prob\": 0.1,\n",
+ " \"hidden_act\": \"gelu\",\n",
+ " \"hidden_dropout_prob\": 0.1,\n",
+ " \"hidden_size\": 768,\n",
+ " \"initializer_range\": 0.02,\n",
+ " \"intermediate_size\": 3072,\n",
+ " \"max_position_embeddings\": 512,\n",
+ " \"num_attention_heads\": 12,\n",
+ " \"num_hidden_layers\": 12,\n",
+ " \"type_vocab_size\": 2,\n",
+ " \"vocab_size\": 30522\n",
+ "}\n",
+ "\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "BertModel(\n",
+ " (embeddings): BertEmbeddings(\n",
+ " (word_embeddings): Embedding(30522, 768)\n",
+ " (position_embeddings): Embedding(512, 768)\n",
+ " (token_type_embeddings): Embedding(2, 768)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (encoder): BertEncoder(\n",
+ " (layer): ModuleList(\n",
+ " (0): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (1): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (2): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (3): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (4): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (5): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (6): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (7): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (8): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (9): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (10): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (11): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " (pooler): BertPooler(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (activation): Tanh()\n",
+ " )\n",
+ ")"
+ ]
+ },
+ "execution_count": 26,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "device = torch.device(\"cpu\")\n",
+ "model = ppb.BertModel.from_pretrained(init_checkpoint_pt)\n",
+ "model.to(device)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:26.963427Z",
+ "start_time": "2018-11-15T15:21:26.922494Z"
+ },
+ "code_folding": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "BertModel(\n",
+ " (embeddings): BertEmbeddings(\n",
+ " (word_embeddings): Embedding(30522, 768)\n",
+ " (position_embeddings): Embedding(512, 768)\n",
+ " (token_type_embeddings): Embedding(2, 768)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (encoder): BertEncoder(\n",
+ " (layer): ModuleList(\n",
+ " (0): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (1): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (2): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (3): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (4): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (5): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (6): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (7): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (8): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (9): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (10): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (11): BertLayer(\n",
+ " (attention): BertAttention(\n",
+ " (self): BertSelfAttention(\n",
+ " (query): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (key): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (value): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " (output): BertSelfOutput(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " (intermediate): BertIntermediate(\n",
+ " (dense): Linear(in_features=768, out_features=3072, bias=True)\n",
+ " )\n",
+ " (output): BertOutput(\n",
+ " (dense): Linear(in_features=3072, out_features=768, bias=True)\n",
+ " (LayerNorm): BertLayerNorm()\n",
+ " (dropout): Dropout(p=0.1)\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " )\n",
+ " (pooler): BertPooler(\n",
+ " (dense): Linear(in_features=768, out_features=768, bias=True)\n",
+ " (activation): Tanh()\n",
+ " )\n",
+ ")"
+ ]
+ },
+ "execution_count": 27,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "all_input_ids = torch.tensor([f.input_ids for f in features], dtype=torch.long)\n",
+ "all_input_mask = torch.tensor([f.input_mask for f in features], dtype=torch.long)\n",
+ "all_input_type_ids = torch.tensor([f.input_type_ids for f in features], dtype=torch.long)\n",
+ "all_example_index = torch.arange(all_input_ids.size(0), dtype=torch.long)\n",
+ "\n",
+ "eval_data = TensorDataset(all_input_ids, all_input_mask, all_input_type_ids, all_example_index)\n",
+ "eval_sampler = SequentialSampler(eval_data)\n",
+ "eval_dataloader = DataLoader(eval_data, sampler=eval_sampler, batch_size=1)\n",
+ "\n",
+ "model.eval()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 28,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:30.718724Z",
+ "start_time": "2018-11-15T15:21:30.329205Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "tensor([[ 101, 2040, 2001, 3958, 27227, 1029, 102, 3958, 27227, 2001,\n",
+ " 1037, 13997, 11510, 102, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0]])\n",
+ "tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
+ " 0, 0, 0, 0, 0, 0, 0, 0]])\n",
+ "tensor([0])\n",
+ "layer 0 0\n",
+ "layer 1 1\n",
+ "layer 2 2\n",
+ "layer 3 3\n",
+ "layer 4 4\n",
+ "layer 5 5\n",
+ "layer 6 6\n",
+ "layer 7 7\n",
+ "layer 8 8\n",
+ "layer 9 9\n",
+ "layer 10 10\n",
+ "layer 11 11\n"
+ ]
+ }
+ ],
+ "source": [
+ "layer_indexes = list(range(12))\n",
+ "\n",
+ "pytorch_all_out = []\n",
+ "for input_ids, input_mask, input_type_ids, example_indices in eval_dataloader:\n",
+ " print(input_ids)\n",
+ " print(input_mask)\n",
+ " print(example_indices)\n",
+ " input_ids = input_ids.to(device)\n",
+ " input_mask = input_mask.to(device)\n",
+ "\n",
+ " all_encoder_layers, _ = model(input_ids, token_type_ids=input_type_ids, attention_mask=input_mask)\n",
+ "\n",
+ " for b, example_index in enumerate(example_indices):\n",
+ " feature = features[example_index.item()]\n",
+ " unique_id = int(feature.unique_id)\n",
+ " # feature = unique_id_to_feature[unique_id]\n",
+ " output_json = collections.OrderedDict()\n",
+ " output_json[\"linex_index\"] = unique_id\n",
+ " all_out_features = []\n",
+ " # for (i, token) in enumerate(feature.tokens):\n",
+ " all_layers = []\n",
+ " for (j, layer_index) in enumerate(layer_indexes):\n",
+ " print(\"layer\", j, layer_index)\n",
+ " layer_output = all_encoder_layers[int(layer_index)].detach().cpu().numpy()\n",
+ " layer_output = layer_output[b]\n",
+ " layers = collections.OrderedDict()\n",
+ " layers[\"index\"] = layer_index\n",
+ " layer_output = layer_output\n",
+ " layers[\"values\"] = layer_output if not isinstance(layer_output, (int, float)) else [layer_output]\n",
+ " all_layers.append(layers)\n",
+ "\n",
+ " out_features = collections.OrderedDict()\n",
+ " out_features[\"layers\"] = all_layers\n",
+ " all_out_features.append(out_features)\n",
+ " output_json[\"features\"] = all_out_features\n",
+ " pytorch_all_out.append(output_json)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:35.703615Z",
+ "start_time": "2018-11-15T15:21:35.666150Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "1\n",
+ "2\n",
+ "odict_keys(['linex_index', 'features'])\n",
+ "number of tokens 1\n",
+ "number of layers 12\n",
+ "hidden_size 128\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "(128, 768)"
+ ]
+ },
+ "execution_count": 29,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "print(len(pytorch_all_out))\n",
+ "print(len(pytorch_all_out[0]))\n",
+ "print(pytorch_all_out[0].keys())\n",
+ "print(\"number of tokens\", len(pytorch_all_out))\n",
+ "print(\"number of layers\", len(pytorch_all_out[0]['features'][0]['layers']))\n",
+ "print(\"hidden_size\", len(pytorch_all_out[0]['features'][0]['layers'][0]['values']))\n",
+ "pytorch_all_out[0]['features'][0]['layers'][0]['values'].shape"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 30,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:36.999073Z",
+ "start_time": "2018-11-15T15:21:36.966762Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(128, 768)\n",
+ "(128, 768)\n"
+ ]
+ }
+ ],
+ "source": [
+ "pytorch_outputs = list(pytorch_all_out[0]['features'][0]['layers'][t]['values'] for t in layer_indexes)\n",
+ "print(pytorch_outputs[0].shape)\n",
+ "print(pytorch_outputs[1].shape)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:37.936522Z",
+ "start_time": "2018-11-15T15:21:37.905269Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "(128, 768)\n",
+ "(128, 768)\n"
+ ]
+ }
+ ],
+ "source": [
+ "print(tensorflow_outputs[0].shape)\n",
+ "print(tensorflow_outputs[1].shape)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 3/ Comparing the standard deviation on the last layer of both models"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:39.437137Z",
+ "start_time": "2018-11-15T15:21:39.406150Z"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2018-11-15T15:21:40.181870Z",
+ "start_time": "2018-11-15T15:21:40.137023Z"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "shape tensorflow layer, shape pytorch layer, standard deviation\n",
+ "((128, 768), (128, 768), 1.5258875e-07)\n",
+ "((128, 768), (128, 768), 2.342731e-07)\n",
+ "((128, 768), (128, 768), 2.801949e-07)\n",
+ "((128, 768), (128, 768), 3.5904986e-07)\n",
+ "((128, 768), (128, 768), 4.2842768e-07)\n",
+ "((128, 768), (128, 768), 5.127951e-07)\n",
+ "((128, 768), (128, 768), 6.14668e-07)\n",
+ "((128, 768), (128, 768), 7.063922e-07)\n",
+ "((128, 768), (128, 768), 7.906173e-07)\n",
+ "((128, 768), (128, 768), 8.475192e-07)\n",
+ "((128, 768), (128, 768), 8.975489e-07)\n",
+ "((128, 768), (128, 768), 4.1671223e-07)\n"
+ ]
+ }
+ ],
+ "source": [
+ "print('shape tensorflow layer, shape pytorch layer, standard deviation')\n",
+ "print('\\n'.join(list(str((np.array(tensorflow_outputs[i]).shape,\n",
+ " np.array(pytorch_outputs[i]).shape, \n",
+ " np.sqrt(np.mean((np.array(tensorflow_outputs[i]) - np.array(pytorch_outputs[i]))**2.0)))) for i in range(12))))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "hide_input": false,
+ "kernelspec": {
+ "display_name": "Python [default]",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.7"
+ },
+ "toc": {
+ "colors": {
+ "hover_highlight": "#DAA520",
+ "running_highlight": "#FF0000",
+ "selected_highlight": "#FFD700"
+ },
+ "moveMenuLeft": true,
+ "nav_menu": {
+ "height": "48px",
+ "width": "252px"
+ },
+ "navigate_menu": true,
+ "number_sections": true,
+ "sideBar": true,
+ "threshold": 4,
+ "toc_cell": false,
+ "toc_section_display": "block",
+ "toc_window_display": false
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__init__.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__init__.py
new file mode 100644
index 0000000000000000000000000000000000000000..28d215d8bdfc5b678ce4be7f1137e10809cd6865
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__init__.py
@@ -0,0 +1,24 @@
+__version__ = "0.6.1"
+from .tokenization import BertTokenizer, BasicTokenizer, WordpieceTokenizer
+from .tokenization_openai import OpenAIGPTTokenizer
+from .tokenization_transfo_xl import (TransfoXLTokenizer, TransfoXLCorpus)
+from .tokenization_gpt2 import GPT2Tokenizer
+
+from .modeling import (BertConfig, BertModel, BertForPreTraining,
+ BertForMaskedLM, BertForNextSentencePrediction,
+ BertForSequenceClassification, BertForMultipleChoice,
+ BertForTokenClassification, BertForQuestionAnswering,
+ load_tf_weights_in_bert)
+from .modeling_openai import (OpenAIGPTConfig, OpenAIGPTModel,
+ OpenAIGPTLMHeadModel, OpenAIGPTDoubleHeadsModel,
+ load_tf_weights_in_openai_gpt)
+from .modeling_transfo_xl import (TransfoXLConfig, TransfoXLModel, TransfoXLLMHeadModel,
+ load_tf_weights_in_transfo_xl)
+from .modeling_gpt2 import (GPT2Config, GPT2Model,
+ GPT2LMHeadModel, GPT2DoubleHeadsModel,
+ load_tf_weights_in_gpt2)
+
+from .optimization import BertAdam
+from .optimization_openai import OpenAIAdam
+
+from .file_utils import PYTORCH_PRETRAINED_BERT_CACHE, cached_path, WEIGHTS_NAME, CONFIG_NAME
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__main__.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__main__.py
new file mode 100644
index 0000000000000000000000000000000000000000..a2aae9e9ce9d9610bb804525b8ee3c009801bc0b
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/__main__.py
@@ -0,0 +1,83 @@
+# coding: utf8
+def main():
+ import sys
+ if (len(sys.argv) != 4 and len(sys.argv) != 5) or sys.argv[1] not in [
+ "convert_tf_checkpoint_to_pytorch",
+ "convert_openai_checkpoint",
+ "convert_transfo_xl_checkpoint",
+ "convert_gpt2_checkpoint",
+ ]:
+ print(
+ "Should be used as one of: \n"
+ ">> `pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch TF_CHECKPOINT TF_CONFIG PYTORCH_DUMP_OUTPUT`, \n"
+ ">> `pytorch_pretrained_bert convert_openai_checkpoint OPENAI_GPT_CHECKPOINT_FOLDER_PATH PYTORCH_DUMP_OUTPUT [OPENAI_GPT_CONFIG]`, \n"
+ ">> `pytorch_pretrained_bert convert_transfo_xl_checkpoint TF_CHECKPOINT_OR_DATASET PYTORCH_DUMP_OUTPUT [TF_CONFIG]` or \n"
+ ">> `pytorch_pretrained_bert convert_gpt2_checkpoint TF_CHECKPOINT PYTORCH_DUMP_OUTPUT [GPT2_CONFIG]`")
+ else:
+ if sys.argv[1] == "convert_tf_checkpoint_to_pytorch":
+ try:
+ from .convert_tf_checkpoint_to_pytorch import convert_tf_checkpoint_to_pytorch
+ except ImportError:
+ print("pytorch_pretrained_bert can only be used from the commandline to convert TensorFlow models in PyTorch, "
+ "In that case, it requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+
+ if len(sys.argv) != 5:
+ # pylint: disable=line-too-long
+ print("Should be used as `pytorch_pretrained_bert convert_tf_checkpoint_to_pytorch TF_CHECKPOINT TF_CONFIG PYTORCH_DUMP_OUTPUT`")
+ else:
+ PYTORCH_DUMP_OUTPUT = sys.argv.pop()
+ TF_CONFIG = sys.argv.pop()
+ TF_CHECKPOINT = sys.argv.pop()
+ convert_tf_checkpoint_to_pytorch(TF_CHECKPOINT, TF_CONFIG, PYTORCH_DUMP_OUTPUT)
+ elif sys.argv[1] == "convert_openai_checkpoint":
+ from .convert_openai_checkpoint_to_pytorch import convert_openai_checkpoint_to_pytorch
+ OPENAI_GPT_CHECKPOINT_FOLDER_PATH = sys.argv[2]
+ PYTORCH_DUMP_OUTPUT = sys.argv[3]
+ if len(sys.argv) == 5:
+ OPENAI_GPT_CONFIG = sys.argv[4]
+ else:
+ OPENAI_GPT_CONFIG = ""
+ convert_openai_checkpoint_to_pytorch(OPENAI_GPT_CHECKPOINT_FOLDER_PATH,
+ OPENAI_GPT_CONFIG,
+ PYTORCH_DUMP_OUTPUT)
+ elif sys.argv[1] == "convert_transfo_xl_checkpoint":
+ try:
+ from .convert_transfo_xl_checkpoint_to_pytorch import convert_transfo_xl_checkpoint_to_pytorch
+ except ImportError:
+ print("pytorch_pretrained_bert can only be used from the commandline to convert TensorFlow models in PyTorch, "
+ "In that case, it requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+
+ if 'ckpt' in sys.argv[2].lower():
+ TF_CHECKPOINT = sys.argv[2]
+ TF_DATASET_FILE = ""
+ else:
+ TF_DATASET_FILE = sys.argv[2]
+ TF_CHECKPOINT = ""
+ PYTORCH_DUMP_OUTPUT = sys.argv[3]
+ if len(sys.argv) == 5:
+ TF_CONFIG = sys.argv[4]
+ else:
+ TF_CONFIG = ""
+ convert_transfo_xl_checkpoint_to_pytorch(TF_CHECKPOINT, TF_CONFIG, PYTORCH_DUMP_OUTPUT, TF_DATASET_FILE)
+ else:
+ try:
+ from .convert_gpt2_checkpoint_to_pytorch import convert_gpt2_checkpoint_to_pytorch
+ except ImportError:
+ print("pytorch_pretrained_bert can only be used from the commandline to convert TensorFlow models in PyTorch, "
+ "In that case, it requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+
+ TF_CHECKPOINT = sys.argv[2]
+ PYTORCH_DUMP_OUTPUT = sys.argv[3]
+ if len(sys.argv) == 5:
+ TF_CONFIG = sys.argv[4]
+ else:
+ TF_CONFIG = ""
+ convert_gpt2_checkpoint_to_pytorch(TF_CHECKPOINT, TF_CONFIG, PYTORCH_DUMP_OUTPUT)
+if __name__ == '__main__':
+ main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_gpt2_checkpoint_to_pytorch.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_gpt2_checkpoint_to_pytorch.py
new file mode 100644
index 0000000000000000000000000000000000000000..51d52a6694f6e32a97c0af4d7b710fa96a10c525
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_gpt2_checkpoint_to_pytorch.py
@@ -0,0 +1,72 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Convert OpenAI GPT checkpoint."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+from io import open
+
+import torch
+
+from pytorch_pretrained_bert.modeling_gpt2 import (CONFIG_NAME, WEIGHTS_NAME,
+ GPT2Config,
+ GPT2Model,
+ load_tf_weights_in_gpt2)
+
+
+def convert_gpt2_checkpoint_to_pytorch(gpt2_checkpoint_path, gpt2_config_file, pytorch_dump_folder_path):
+ # Construct model
+ if gpt2_config_file == "":
+ config = GPT2Config()
+ else:
+ config = GPT2Config(gpt2_config_file)
+ model = GPT2Model(config)
+
+ # Load weights from numpy
+ load_tf_weights_in_gpt2(model, gpt2_checkpoint_path)
+
+ # Save pytorch-model
+ pytorch_weights_dump_path = pytorch_dump_folder_path + '/' + WEIGHTS_NAME
+ pytorch_config_dump_path = pytorch_dump_folder_path + '/' + CONFIG_NAME
+ print("Save PyTorch model to {}".format(pytorch_weights_dump_path))
+ torch.save(model.state_dict(), pytorch_weights_dump_path)
+ print("Save configuration file to {}".format(pytorch_config_dump_path))
+ with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
+ f.write(config.to_json_string())
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ ## Required parameters
+ parser.add_argument("--gpt2_checkpoint_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path the TensorFlow checkpoint path.")
+ parser.add_argument("--pytorch_dump_folder_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path to the output PyTorch model.")
+ parser.add_argument("--gpt2_config_file",
+ default = "",
+ type = str,
+ help = "An optional config json file corresponding to the pre-trained OpenAI model. \n"
+ "This specifies the model architecture.")
+ args = parser.parse_args()
+ convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path,
+ args.gpt2_config_file,
+ args.pytorch_dump_folder_path)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_openai_checkpoint_to_pytorch.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_openai_checkpoint_to_pytorch.py
new file mode 100644
index 0000000000000000000000000000000000000000..566008aaa0b3f45ceca515e98847bdfe4535724b
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_openai_checkpoint_to_pytorch.py
@@ -0,0 +1,72 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Convert OpenAI GPT checkpoint."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+from io import open
+
+import torch
+
+from pytorch_pretrained_bert.modeling_openai import (CONFIG_NAME, WEIGHTS_NAME,
+ OpenAIGPTConfig,
+ OpenAIGPTModel,
+ load_tf_weights_in_openai_gpt)
+
+
+def convert_openai_checkpoint_to_pytorch(openai_checkpoint_folder_path, openai_config_file, pytorch_dump_folder_path):
+ # Construct model
+ if openai_config_file == "":
+ config = OpenAIGPTConfig()
+ else:
+ config = OpenAIGPTConfig(openai_config_file)
+ model = OpenAIGPTModel(config)
+
+ # Load weights from numpy
+ load_tf_weights_in_openai_gpt(model, openai_checkpoint_folder_path)
+
+ # Save pytorch-model
+ pytorch_weights_dump_path = pytorch_dump_folder_path + '/' + WEIGHTS_NAME
+ pytorch_config_dump_path = pytorch_dump_folder_path + '/' + CONFIG_NAME
+ print("Save PyTorch model to {}".format(pytorch_weights_dump_path))
+ torch.save(model.state_dict(), pytorch_weights_dump_path)
+ print("Save configuration file to {}".format(pytorch_config_dump_path))
+ with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
+ f.write(config.to_json_string())
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ ## Required parameters
+ parser.add_argument("--openai_checkpoint_folder_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path the TensorFlow checkpoint path.")
+ parser.add_argument("--pytorch_dump_folder_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path to the output PyTorch model.")
+ parser.add_argument("--openai_config_file",
+ default = "",
+ type = str,
+ help = "An optional config json file corresponding to the pre-trained OpenAI model. \n"
+ "This specifies the model architecture.")
+ args = parser.parse_args()
+ convert_openai_checkpoint_to_pytorch(args.openai_checkpoint_folder_path,
+ args.openai_config_file,
+ args.pytorch_dump_folder_path)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py
new file mode 100644
index 0000000000000000000000000000000000000000..13d96384fdae76631279ded5e4307363ee2b2de5
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_tf_checkpoint_to_pytorch.py
@@ -0,0 +1,66 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Convert BERT checkpoint."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import re
+import argparse
+import tensorflow as tf
+import torch
+import numpy as np
+
+from pytorch_pretrained_bert.modeling import BertConfig, BertForPreTraining, load_tf_weights_in_bert
+
+def convert_tf_checkpoint_to_pytorch(tf_checkpoint_path, bert_config_file, pytorch_dump_path):
+ # Initialise PyTorch model
+ config = BertConfig.from_json_file(bert_config_file)
+ print("Building PyTorch model from configuration: {}".format(str(config)))
+ model = BertForPreTraining(config)
+
+ # Load weights from tf checkpoint
+ load_tf_weights_in_bert(model, tf_checkpoint_path)
+
+ # Save pytorch-model
+ print("Save PyTorch model to {}".format(pytorch_dump_path))
+ torch.save(model.state_dict(), pytorch_dump_path)
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ ## Required parameters
+ parser.add_argument("--tf_checkpoint_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path the TensorFlow checkpoint path.")
+ parser.add_argument("--bert_config_file",
+ default = None,
+ type = str,
+ required = True,
+ help = "The config json file corresponding to the pre-trained BERT model. \n"
+ "This specifies the model architecture.")
+ parser.add_argument("--pytorch_dump_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path to the output PyTorch model.")
+ args = parser.parse_args()
+ convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path,
+ args.bert_config_file,
+ args.pytorch_dump_path)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_transfo_xl_checkpoint_to_pytorch.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_transfo_xl_checkpoint_to_pytorch.py
new file mode 100644
index 0000000000000000000000000000000000000000..8d6b9651c700d11e00202dc09f961df15bceb7cd
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/convert_transfo_xl_checkpoint_to_pytorch.py
@@ -0,0 +1,116 @@
+# coding=utf-8
+# Copyright 2018 The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Convert Transformer XL checkpoint and datasets."""
+
+from __future__ import absolute_import, division, print_function
+
+import argparse
+import os
+import sys
+from io import open
+
+import torch
+
+import pytorch_pretrained_bert.tokenization_transfo_xl as data_utils
+from pytorch_pretrained_bert.modeling_transfo_xl import (CONFIG_NAME,
+ WEIGHTS_NAME,
+ TransfoXLConfig,
+ TransfoXLLMHeadModel,
+ load_tf_weights_in_transfo_xl)
+from pytorch_pretrained_bert.tokenization_transfo_xl import (CORPUS_NAME,
+ VOCAB_NAME)
+
+if sys.version_info[0] == 2:
+ import cPickle as pickle
+else:
+ import pickle
+
+# We do this to be able to load python 2 datasets pickles
+# See e.g. https://stackoverflow.com/questions/2121874/python-pickling-after-changing-a-modules-directory/2121918#2121918
+data_utils.Vocab = data_utils.TransfoXLTokenizer
+data_utils.Corpus = data_utils.TransfoXLCorpus
+sys.modules['data_utils'] = data_utils
+sys.modules['vocabulary'] = data_utils
+
+def convert_transfo_xl_checkpoint_to_pytorch(tf_checkpoint_path,
+ transfo_xl_config_file,
+ pytorch_dump_folder_path,
+ transfo_xl_dataset_file):
+ if transfo_xl_dataset_file:
+ # Convert a pre-processed corpus (see original TensorFlow repo)
+ with open(transfo_xl_dataset_file, "rb") as fp:
+ corpus = pickle.load(fp, encoding="latin1")
+ # Save vocabulary and dataset cache as Dictionaries (should be better than pickles for the long-term)
+ pytorch_vocab_dump_path = pytorch_dump_folder_path + '/' + VOCAB_NAME
+ print("Save vocabulary to {}".format(pytorch_vocab_dump_path))
+ corpus_vocab_dict = corpus.vocab.__dict__
+ torch.save(corpus_vocab_dict, pytorch_vocab_dump_path)
+
+ corpus_dict_no_vocab = corpus.__dict__
+ corpus_dict_no_vocab.pop('vocab', None)
+ pytorch_dataset_dump_path = pytorch_dump_folder_path + '/' + CORPUS_NAME
+ print("Save dataset to {}".format(pytorch_dataset_dump_path))
+ torch.save(corpus_dict_no_vocab, pytorch_dataset_dump_path)
+
+ if tf_checkpoint_path:
+ # Convert a pre-trained TensorFlow model
+ config_path = os.path.abspath(transfo_xl_config_file)
+ tf_path = os.path.abspath(tf_checkpoint_path)
+
+ print("Converting Transformer XL checkpoint from {} with config at {}".format(tf_path, config_path))
+ # Initialise PyTorch model
+ if transfo_xl_config_file == "":
+ config = TransfoXLConfig()
+ else:
+ config = TransfoXLConfig(transfo_xl_config_file)
+ print("Building PyTorch model from configuration: {}".format(str(config)))
+ model = TransfoXLLMHeadModel(config)
+
+ model = load_tf_weights_in_transfo_xl(model, config, tf_path)
+ # Save pytorch-model
+ pytorch_weights_dump_path = os.path.join(pytorch_dump_folder_path, WEIGHTS_NAME)
+ pytorch_config_dump_path = os.path.join(pytorch_dump_folder_path, CONFIG_NAME)
+ print("Save PyTorch model to {}".format(os.path.abspath(pytorch_weights_dump_path)))
+ torch.save(model.state_dict(), pytorch_weights_dump_path)
+ print("Save configuration file to {}".format(os.path.abspath(pytorch_config_dump_path)))
+ with open(pytorch_config_dump_path, "w", encoding="utf-8") as f:
+ f.write(config.to_json_string())
+
+
+if __name__ == "__main__":
+ parser = argparse.ArgumentParser()
+ parser.add_argument("--pytorch_dump_folder_path",
+ default = None,
+ type = str,
+ required = True,
+ help = "Path to the folder to store the PyTorch model or dataset/vocab.")
+ parser.add_argument("--tf_checkpoint_path",
+ default = "",
+ type = str,
+ help = "An optional path to a TensorFlow checkpoint path to be converted.")
+ parser.add_argument("--transfo_xl_config_file",
+ default = "",
+ type = str,
+ help = "An optional config json file corresponding to the pre-trained BERT model. \n"
+ "This specifies the model architecture.")
+ parser.add_argument("--transfo_xl_dataset_file",
+ default = "",
+ type = str,
+ help = "An optional dataset file to be converted in a vocabulary.")
+ args = parser.parse_args()
+ convert_transfo_xl_checkpoint_to_pytorch(args.tf_checkpoint_path,
+ args.transfo_xl_config_file,
+ args.pytorch_dump_folder_path,
+ args.transfo_xl_dataset_file)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/file_utils.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/file_utils.py
new file mode 100644
index 0000000000000000000000000000000000000000..17bdd258eaecd8823940cf50b23efde939bbfb04
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/file_utils.py
@@ -0,0 +1,270 @@
+"""
+Utilities for working with the local dataset cache.
+This file is adapted from the AllenNLP library at https://github.com/allenai/allennlp
+Copyright by the AllenNLP authors.
+"""
+from __future__ import (absolute_import, division, print_function, unicode_literals)
+
+import sys
+import json
+import logging
+import os
+import shutil
+import tempfile
+import fnmatch
+from functools import wraps
+from hashlib import sha256
+import sys
+from io import open
+
+import boto3
+import requests
+from botocore.exceptions import ClientError
+from tqdm import tqdm
+
+try:
+ from urllib.parse import urlparse
+except ImportError:
+ from urlparse import urlparse
+
+try:
+ from pathlib import Path
+ PYTORCH_PRETRAINED_BERT_CACHE = Path(os.getenv('PYTORCH_PRETRAINED_BERT_CACHE',
+ Path.home() / '.pytorch_pretrained_bert'))
+except (AttributeError, ImportError):
+ PYTORCH_PRETRAINED_BERT_CACHE = os.getenv('PYTORCH_PRETRAINED_BERT_CACHE',
+ os.path.join(os.path.expanduser("~"), '.pytorch_pretrained_bert'))
+
+CONFIG_NAME = "config.json"
+WEIGHTS_NAME = "pytorch_model.bin"
+
+logger = logging.getLogger(__name__) # pylint: disable=invalid-name
+
+
+def url_to_filename(url, etag=None):
+ """
+ Convert `url` into a hashed filename in a repeatable way.
+ If `etag` is specified, append its hash to the url's, delimited
+ by a period.
+ """
+ url_bytes = url.encode('utf-8')
+ url_hash = sha256(url_bytes)
+ filename = url_hash.hexdigest()
+
+ if etag:
+ etag_bytes = etag.encode('utf-8')
+ etag_hash = sha256(etag_bytes)
+ filename += '.' + etag_hash.hexdigest()
+
+ return filename
+
+
+def filename_to_url(filename, cache_dir=None):
+ """
+ Return the url and etag (which may be ``None``) stored for `filename`.
+ Raise ``EnvironmentError`` if `filename` or its stored metadata do not exist.
+ """
+ if cache_dir is None:
+ cache_dir = PYTORCH_PRETRAINED_BERT_CACHE
+ if sys.version_info[0] == 3 and isinstance(cache_dir, Path):
+ cache_dir = str(cache_dir)
+
+ cache_path = os.path.join(cache_dir, filename)
+ if not os.path.exists(cache_path):
+ raise EnvironmentError("file {} not found".format(cache_path))
+
+ meta_path = cache_path + '.json'
+ if not os.path.exists(meta_path):
+ raise EnvironmentError("file {} not found".format(meta_path))
+
+ with open(meta_path, encoding="utf-8") as meta_file:
+ metadata = json.load(meta_file)
+ url = metadata['url']
+ etag = metadata['etag']
+
+ return url, etag
+
+
+def cached_path(url_or_filename, cache_dir=None):
+ """
+ Given something that might be a URL (or might be a local path),
+ determine which. If it's a URL, download the file and cache it, and
+ return the path to the cached file. If it's already a local path,
+ make sure the file exists and then return the path.
+ """
+ if cache_dir is None:
+ cache_dir = PYTORCH_PRETRAINED_BERT_CACHE
+ if sys.version_info[0] == 3 and isinstance(url_or_filename, Path):
+ url_or_filename = str(url_or_filename)
+ if sys.version_info[0] == 3 and isinstance(cache_dir, Path):
+ cache_dir = str(cache_dir)
+
+ parsed = urlparse(url_or_filename)
+
+ if parsed.scheme in ('http', 'https', 's3'):
+ # URL, so get it from the cache (downloading if necessary)
+ return get_from_cache(url_or_filename, cache_dir)
+ elif os.path.exists(url_or_filename):
+ # File, and it exists.
+ return url_or_filename
+ elif parsed.scheme == '':
+ # File, but it doesn't exist.
+ raise EnvironmentError("file {} not found".format(url_or_filename))
+ else:
+ # Something unknown
+ raise ValueError("unable to parse {} as a URL or as a local path".format(url_or_filename))
+
+
+def split_s3_path(url):
+ """Split a full s3 path into the bucket name and path."""
+ parsed = urlparse(url)
+ if not parsed.netloc or not parsed.path:
+ raise ValueError("bad s3 path {}".format(url))
+ bucket_name = parsed.netloc
+ s3_path = parsed.path
+ # Remove '/' at beginning of path.
+ if s3_path.startswith("/"):
+ s3_path = s3_path[1:]
+ return bucket_name, s3_path
+
+
+def s3_request(func):
+ """
+ Wrapper function for s3 requests in order to create more helpful error
+ messages.
+ """
+
+ @wraps(func)
+ def wrapper(url, *args, **kwargs):
+ try:
+ return func(url, *args, **kwargs)
+ except ClientError as exc:
+ if int(exc.response["Error"]["Code"]) == 404:
+ raise EnvironmentError("file {} not found".format(url))
+ else:
+ raise
+
+ return wrapper
+
+
+@s3_request
+def s3_etag(url):
+ """Check ETag on S3 object."""
+ s3_resource = boto3.resource("s3")
+ bucket_name, s3_path = split_s3_path(url)
+ s3_object = s3_resource.Object(bucket_name, s3_path)
+ return s3_object.e_tag
+
+
+@s3_request
+def s3_get(url, temp_file):
+ """Pull a file directly from S3."""
+ s3_resource = boto3.resource("s3")
+ bucket_name, s3_path = split_s3_path(url)
+ s3_resource.Bucket(bucket_name).download_fileobj(s3_path, temp_file)
+
+
+def http_get(url, temp_file):
+ req = requests.get(url, stream=True)
+ content_length = req.headers.get('Content-Length')
+ total = int(content_length) if content_length is not None else None
+ progress = tqdm(unit="B", total=total)
+ for chunk in req.iter_content(chunk_size=1024):
+ if chunk: # filter out keep-alive new chunks
+ progress.update(len(chunk))
+ temp_file.write(chunk)
+ progress.close()
+
+
+def get_from_cache(url, cache_dir=None):
+ """
+ Given a URL, look for the corresponding dataset in the local cache.
+ If it's not there, download it. Then return the path to the cached file.
+ """
+ if cache_dir is None:
+ cache_dir = PYTORCH_PRETRAINED_BERT_CACHE
+ if sys.version_info[0] == 3 and isinstance(cache_dir, Path):
+ cache_dir = str(cache_dir)
+
+ if not os.path.exists(cache_dir):
+ os.makedirs(cache_dir)
+
+ # Get eTag to add to filename, if it exists.
+ if url.startswith("s3://"):
+ etag = s3_etag(url)
+ else:
+ try:
+ response = requests.head(url, allow_redirects=True)
+ if response.status_code != 200:
+ etag = None
+ else:
+ etag = response.headers.get("ETag")
+ except EnvironmentError:
+ etag = None
+
+ if sys.version_info[0] == 2 and etag is not None:
+ etag = etag.decode('utf-8')
+ filename = url_to_filename(url, etag)
+
+ # get cache path to put the file
+ cache_path = os.path.join(cache_dir, filename)
+
+ # If we don't have a connection (etag is None) and can't identify the file
+ # try to get the last downloaded one
+ if not os.path.exists(cache_path) and etag is None:
+ matching_files = fnmatch.filter(os.listdir(cache_dir), filename + '.*')
+ matching_files = list(filter(lambda s: not s.endswith('.json'), matching_files))
+ if matching_files:
+ cache_path = os.path.join(cache_dir, matching_files[-1])
+
+ if not os.path.exists(cache_path):
+ # Download to temporary file, then copy to cache dir once finished.
+ # Otherwise you get corrupt cache entries if the download gets interrupted.
+ with tempfile.NamedTemporaryFile() as temp_file:
+ logger.info("%s not found in cache, downloading to %s", url, temp_file.name)
+
+ # GET file object
+ if url.startswith("s3://"):
+ s3_get(url, temp_file)
+ else:
+ http_get(url, temp_file)
+
+ # we are copying the file before closing it, so flush to avoid truncation
+ temp_file.flush()
+ # shutil.copyfileobj() starts at the current position, so go to the start
+ temp_file.seek(0)
+
+ logger.info("copying %s to cache at %s", temp_file.name, cache_path)
+ with open(cache_path, 'wb') as cache_file:
+ shutil.copyfileobj(temp_file, cache_file)
+
+ logger.info("creating metadata file for %s", cache_path)
+ meta = {'url': url, 'etag': etag}
+ meta_path = cache_path + '.json'
+ with open(meta_path, 'w') as meta_file:
+ output_string = json.dumps(meta)
+ if sys.version_info[0] == 2 and isinstance(output_string, str):
+ output_string = unicode(output_string, 'utf-8') # The beauty of python 2
+ meta_file.write(output_string)
+
+ logger.info("removing temp file %s", temp_file.name)
+
+ return cache_path
+
+
+def read_set_from_file(filename):
+ '''
+ Extract a de-duped collection (set) of text from a file.
+ Expected file format is one item per line.
+ '''
+ collection = set()
+ with open(filename, 'r', encoding='utf-8') as file_:
+ for line in file_:
+ collection.add(line.rstrip())
+ return collection
+
+
+def get_file_extension(path, dot=True, lower=True):
+ ext = os.path.splitext(path)[1]
+ ext = ext if dot else ext[1:]
+ return ext.lower() if lower else ext
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling.py
new file mode 100644
index 0000000000000000000000000000000000000000..2b42d9df8d829a353354e095908e51ab36971864
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling.py
@@ -0,0 +1,1229 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch BERT model."""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import copy
+import json
+import logging
+import math
+import os
+import shutil
+import tarfile
+import tempfile
+import sys
+from io import open
+
+import torch
+from torch import nn
+from torch.nn import CrossEntropyLoss
+
+from .file_utils import cached_path, WEIGHTS_NAME, CONFIG_NAME
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_MODEL_ARCHIVE_MAP = {
+ 'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz",
+ 'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased.tar.gz",
+ 'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased.tar.gz",
+ 'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased.tar.gz",
+ 'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased.tar.gz",
+ 'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased.tar.gz",
+ 'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese.tar.gz",
+}
+BERT_CONFIG_NAME = 'bert_config.json'
+TF_WEIGHTS_NAME = 'model.ckpt'
+
+def load_tf_weights_in_bert(model, tf_checkpoint_path):
+ """ Load tf checkpoints in a pytorch model
+ """
+ try:
+ import re
+ import numpy as np
+ import tensorflow as tf
+ except ImportError:
+ print("Loading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+ tf_path = os.path.abspath(tf_checkpoint_path)
+ print("Converting TensorFlow checkpoint from {}".format(tf_path))
+ # Load weights from TF model
+ init_vars = tf.train.list_variables(tf_path)
+ names = []
+ arrays = []
+ for name, shape in init_vars:
+ print("Loading TF weight {} with shape {}".format(name, shape))
+ array = tf.train.load_variable(tf_path, name)
+ names.append(name)
+ arrays.append(array)
+
+ for name, array in zip(names, arrays):
+ name = name.split('/')
+ # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
+ # which are not required for using pretrained model
+ if any(n in ["adam_v", "adam_m", "global_step"] for n in name):
+ print("Skipping {}".format("/".join(name)))
+ continue
+ pointer = model
+ for m_name in name:
+ if re.fullmatch(r'[A-Za-z]+_\d+', m_name):
+ l = re.split(r'_(\d+)', m_name)
+ else:
+ l = [m_name]
+ if l[0] == 'kernel' or l[0] == 'gamma':
+ pointer = getattr(pointer, 'weight')
+ elif l[0] == 'output_bias' or l[0] == 'beta':
+ pointer = getattr(pointer, 'bias')
+ elif l[0] == 'output_weights':
+ pointer = getattr(pointer, 'weight')
+ elif l[0] == 'squad':
+ pointer = getattr(pointer, 'classifier')
+ else:
+ try:
+ pointer = getattr(pointer, l[0])
+ except AttributeError:
+ print("Skipping {}".format("/".join(name)))
+ continue
+ if len(l) >= 2:
+ num = int(l[1])
+ pointer = pointer[num]
+ if m_name[-11:] == '_embeddings':
+ pointer = getattr(pointer, 'weight')
+ elif m_name == 'kernel':
+ array = np.transpose(array)
+ try:
+ assert pointer.shape == array.shape
+ except AssertionError as e:
+ e.args += (pointer.shape, array.shape)
+ raise
+ print("Initialize PyTorch weight {}".format(name))
+ pointer.data = torch.from_numpy(array)
+ return model
+
+
+def gelu(x):
+ """Implementation of the gelu activation function.
+ For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
+ 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+ Also see https://arxiv.org/abs/1606.08415
+ """
+ return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
+
+
+def swish(x):
+ return x * torch.sigmoid(x)
+
+
+ACT2FN = {"gelu": gelu, "relu": torch.nn.functional.relu, "swish": swish}
+
+
+class BertConfig(object):
+ """Configuration class to store the configuration of a `BertModel`.
+ """
+ def __init__(self,
+ vocab_size_or_config_json_file,
+ hidden_size=768,
+ num_hidden_layers=12,
+ num_attention_heads=12,
+ intermediate_size=3072,
+ hidden_act="gelu",
+ hidden_dropout_prob=0.1,
+ attention_probs_dropout_prob=0.1,
+ max_position_embeddings=512,
+ type_vocab_size=2,
+ initializer_range=0.02):
+ """Constructs BertConfig.
+
+ Args:
+ vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `BertModel`.
+ hidden_size: Size of the encoder layers and the pooler layer.
+ num_hidden_layers: Number of hidden layers in the Transformer encoder.
+ num_attention_heads: Number of attention heads for each attention layer in
+ the Transformer encoder.
+ intermediate_size: The size of the "intermediate" (i.e., feed-forward)
+ layer in the Transformer encoder.
+ hidden_act: The non-linear activation function (function or string) in the
+ encoder and pooler. If string, "gelu", "relu" and "swish" are supported.
+ hidden_dropout_prob: The dropout probabilitiy for all fully connected
+ layers in the embeddings, encoder, and pooler.
+ attention_probs_dropout_prob: The dropout ratio for the attention
+ probabilities.
+ max_position_embeddings: The maximum sequence length that this model might
+ ever be used with. Typically set this to something large just in case
+ (e.g., 512 or 1024 or 2048).
+ type_vocab_size: The vocabulary size of the `token_type_ids` passed into
+ `BertModel`.
+ initializer_range: The sttdev of the truncated_normal_initializer for
+ initializing all weight matrices.
+ """
+ if isinstance(vocab_size_or_config_json_file, str) or (sys.version_info[0] == 2
+ and isinstance(vocab_size_or_config_json_file, unicode)):
+ with open(vocab_size_or_config_json_file, "r", encoding='utf-8') as reader:
+ json_config = json.loads(reader.read())
+ for key, value in json_config.items():
+ self.__dict__[key] = value
+ elif isinstance(vocab_size_or_config_json_file, int):
+ self.vocab_size = vocab_size_or_config_json_file
+ self.hidden_size = hidden_size
+ self.num_hidden_layers = num_hidden_layers
+ self.num_attention_heads = num_attention_heads
+ self.hidden_act = hidden_act
+ self.intermediate_size = intermediate_size
+ self.hidden_dropout_prob = hidden_dropout_prob
+ self.attention_probs_dropout_prob = attention_probs_dropout_prob
+ self.max_position_embeddings = max_position_embeddings
+ self.type_vocab_size = type_vocab_size
+ self.initializer_range = initializer_range
+ else:
+ raise ValueError("First argument must be either a vocabulary size (int)"
+ "or the path to a pretrained model config file (str)")
+
+ @classmethod
+ def from_dict(cls, json_object):
+ """Constructs a `BertConfig` from a Python dictionary of parameters."""
+ config = BertConfig(vocab_size_or_config_json_file=-1)
+ for key, value in json_object.items():
+ config.__dict__[key] = value
+ return config
+
+ @classmethod
+ def from_json_file(cls, json_file):
+ """Constructs a `BertConfig` from a json file of parameters."""
+ with open(json_file, "r", encoding='utf-8') as reader:
+ text = reader.read()
+ return cls.from_dict(json.loads(text))
+
+ def __repr__(self):
+ return str(self.to_json_string())
+
+ def to_dict(self):
+ """Serializes this instance to a Python dictionary."""
+ output = copy.deepcopy(self.__dict__)
+ return output
+
+ def to_json_string(self):
+ """Serializes this instance to a JSON string."""
+ return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+ def to_json_file(self, json_file_path):
+ """ Save this instance to a json file."""
+ with open(json_file_path, "w", encoding='utf-8') as writer:
+ writer.write(self.to_json_string())
+
+try:
+ from apex.normalization.fused_layer_norm import FusedLayerNorm as BertLayerNorm
+except ImportError:
+ logger.info("Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex .")
+ class BertLayerNorm(nn.Module):
+ def __init__(self, hidden_size, eps=1e-12):
+ """Construct a layernorm module in the TF style (epsilon inside the square root).
+ """
+ super(BertLayerNorm, self).__init__()
+ self.weight = nn.Parameter(torch.ones(hidden_size))
+ self.bias = nn.Parameter(torch.zeros(hidden_size))
+ self.variance_epsilon = eps
+
+ def forward(self, x):
+ u = x.mean(-1, keepdim=True)
+ s = (x - u).pow(2).mean(-1, keepdim=True)
+ x = (x - u) / torch.sqrt(s + self.variance_epsilon)
+ return self.weight * x + self.bias
+
+class BertEmbeddings(nn.Module):
+ """Construct the embeddings from word, position and token_type embeddings.
+ """
+ def __init__(self, config):
+ super(BertEmbeddings, self).__init__()
+ self.word_embeddings = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=0)
+ self.position_embeddings = nn.Embedding(config.max_position_embeddings, config.hidden_size)
+ self.token_type_embeddings = nn.Embedding(config.type_vocab_size, config.hidden_size)
+
+ # self.LayerNorm is not snake-cased to stick with TensorFlow model variable name and be able to load
+ # any TensorFlow checkpoint file
+ self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+ def forward(self, input_ids, token_type_ids=None):
+ seq_length = input_ids.size(1)
+ position_ids = torch.arange(seq_length, dtype=torch.long, device=input_ids.device)
+ position_ids = position_ids.unsqueeze(0).expand_as(input_ids)
+ if token_type_ids is None:
+ token_type_ids = torch.zeros_like(input_ids)
+
+ words_embeddings = self.word_embeddings(input_ids)
+ position_embeddings = self.position_embeddings(position_ids)
+ token_type_embeddings = self.token_type_embeddings(token_type_ids)
+
+ embeddings = words_embeddings + position_embeddings + token_type_embeddings
+ embeddings = self.LayerNorm(embeddings)
+ embeddings = self.dropout(embeddings)
+ return embeddings
+
+
+class BertSelfAttention(nn.Module):
+ def __init__(self, config):
+ super(BertSelfAttention, self).__init__()
+ if config.hidden_size % config.num_attention_heads != 0:
+ raise ValueError(
+ "The hidden size (%d) is not a multiple of the number of attention "
+ "heads (%d)" % (config.hidden_size, config.num_attention_heads))
+ self.num_attention_heads = config.num_attention_heads
+ self.attention_head_size = int(config.hidden_size / config.num_attention_heads)
+ self.all_head_size = self.num_attention_heads * self.attention_head_size
+
+ self.query = nn.Linear(config.hidden_size, self.all_head_size)
+ self.key = nn.Linear(config.hidden_size, self.all_head_size)
+ self.value = nn.Linear(config.hidden_size, self.all_head_size)
+
+ self.dropout = nn.Dropout(config.attention_probs_dropout_prob)
+
+ def transpose_for_scores(self, x):
+ new_x_shape = x.size()[:-1] + (self.num_attention_heads, self.attention_head_size)
+ x = x.view(*new_x_shape)
+ return x.permute(0, 2, 1, 3)
+
+ def forward(self, hidden_states, attention_mask, uniform=False):
+ mixed_query_layer = self.query(hidden_states)
+ mixed_key_layer = self.key(hidden_states)
+ mixed_value_layer = self.value(hidden_states)
+
+ query_layer = self.transpose_for_scores(mixed_query_layer)
+ key_layer = self.transpose_for_scores(mixed_key_layer)
+ value_layer = self.transpose_for_scores(mixed_value_layer)
+
+ # Take the dot product between "query" and "key" to get the raw attention scores.
+ attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))
+ attention_scores = attention_scores / math.sqrt(self.attention_head_size)
+ # Apply the attention mask is (precomputed for all layers in BertModel forward() function)
+ attention_scores = attention_scores + attention_mask
+
+ # Normalize the attention scores to probabilities.
+
+ attention_probs = nn.Softmax(dim=-1)(attention_scores)
+
+ # if uniform:
+ # bin_mask = torch.ge(attention_mask.squeeze(), -10).long()
+ # eff_lengths = torch.sum(bin_mask, dim=-1)
+ # uniform = (1.0 / eff_lengths.float()).unsqueeze(1).unsqueeze(1).unsqueeze(1).expand_as(attention_probs)
+ # uniform_masked = bin_mask.unsqueeze(1).unsqueeze(1).expand_as(attention_probs).float()
+ # attention_probs = uniform * uniform_masked
+
+ # This is actually dropping out entire tokens to attend to, which might
+ # seem a bit unusual, but is taken from the original Transformer paper.
+ attention_probs = self.dropout(attention_probs)
+
+ context_layer = torch.matmul(attention_probs, value_layer)
+ context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
+ new_context_layer_shape = context_layer.size()[:-2] + (self.all_head_size,)
+ context_layer = context_layer.view(*new_context_layer_shape)
+ return context_layer, attention_probs
+
+
+class BertSelfOutput(nn.Module):
+ def __init__(self, config):
+ super(BertSelfOutput, self).__init__()
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+ self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+ def forward(self, hidden_states, input_tensor):
+ hidden_states = self.dense(hidden_states)
+ hidden_states = self.dropout(hidden_states)
+ hidden_states = self.LayerNorm(hidden_states + input_tensor)
+ return hidden_states
+
+
+class BertAttention(nn.Module):
+ def __init__(self, config):
+ super(BertAttention, self).__init__()
+ self.self = BertSelfAttention(config)
+ self.output = BertSelfOutput(config)
+
+ def forward(self, input_tensor, attention_mask, uniform=False):
+ self_output, attn_probs = self.self(input_tensor, attention_mask, uniform)
+ attention_output = self.output(self_output, input_tensor)
+ return attention_output, attn_probs
+
+
+class BertIntermediate(nn.Module):
+ def __init__(self, config):
+ super(BertIntermediate, self).__init__()
+ self.dense = nn.Linear(config.hidden_size, config.intermediate_size)
+ if isinstance(config.hidden_act, str) or (sys.version_info[0] == 2 and isinstance(config.hidden_act, unicode)):
+ self.intermediate_act_fn = ACT2FN[config.hidden_act]
+ else:
+ self.intermediate_act_fn = config.hidden_act
+
+ def forward(self, hidden_states):
+ hidden_states = self.dense(hidden_states)
+ hidden_states = self.intermediate_act_fn(hidden_states)
+ return hidden_states
+
+
+class BertOutput(nn.Module):
+ def __init__(self, config):
+ super(BertOutput, self).__init__()
+ self.dense = nn.Linear(config.intermediate_size, config.hidden_size)
+ self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+
+ def forward(self, hidden_states, input_tensor):
+ hidden_states = self.dense(hidden_states)
+ hidden_states = self.dropout(hidden_states)
+ hidden_states = self.LayerNorm(hidden_states + input_tensor)
+ return hidden_states
+
+
+class BertLayer(nn.Module):
+ def __init__(self, config):
+ super(BertLayer, self).__init__()
+ self.attention = BertAttention(config)
+ self.intermediate = BertIntermediate(config)
+ self.output = BertOutput(config)
+
+ def forward(self, hidden_states, attention_mask, uniform=False):
+ attention_output, self_output = self.attention(hidden_states, attention_mask, uniform)
+ intermediate_output = self.intermediate(attention_output)
+ layer_output = self.output(intermediate_output, attention_output)
+ return layer_output, self_output
+
+
+class BertEncoder(nn.Module):
+ def __init__(self, config):
+ super(BertEncoder, self).__init__()
+ layer = BertLayer(config)
+ self.layer = nn.ModuleList([copy.deepcopy(layer) for _ in range(config.num_hidden_layers)])
+
+ def forward(self, hidden_states, attention_mask, output_all_encoded_layers=True, uniforms=[False]*12):
+ all_encoder_layers = []
+ all_attentions = []
+ for i, layer_module in enumerate(self.layer):
+ hidden_states, self_output = layer_module(hidden_states, attention_mask, uniforms[i])
+ if output_all_encoded_layers:
+ all_encoder_layers.append(hidden_states)
+ all_attentions.append(self_output)
+ if not output_all_encoded_layers:
+ all_encoder_layers.append(hidden_states)
+ return all_encoder_layers, all_attentions
+
+
+class BertPooler(nn.Module):
+ def __init__(self, config):
+ super(BertPooler, self).__init__()
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+ self.activation = nn.Tanh()
+
+ def forward(self, hidden_states):
+ # We "pool" the model by simply taking the hidden state corresponding
+ # to the first token.
+ first_token_tensor = hidden_states[:, 0]
+ pooled_output = self.dense(first_token_tensor)
+ pooled_output = self.activation(pooled_output)
+ return pooled_output
+
+
+class BertPredictionHeadTransform(nn.Module):
+ def __init__(self, config):
+ super(BertPredictionHeadTransform, self).__init__()
+ self.dense = nn.Linear(config.hidden_size, config.hidden_size)
+ if isinstance(config.hidden_act, str) or (sys.version_info[0] == 2 and isinstance(config.hidden_act, unicode)):
+ self.transform_act_fn = ACT2FN[config.hidden_act]
+ else:
+ self.transform_act_fn = config.hidden_act
+ self.LayerNorm = BertLayerNorm(config.hidden_size, eps=1e-12)
+
+ def forward(self, hidden_states):
+ hidden_states = self.dense(hidden_states)
+ hidden_states = self.transform_act_fn(hidden_states)
+ hidden_states = self.LayerNorm(hidden_states)
+ return hidden_states
+
+
+class BertLMPredictionHead(nn.Module):
+ def __init__(self, config, bert_model_embedding_weights):
+ super(BertLMPredictionHead, self).__init__()
+ self.transform = BertPredictionHeadTransform(config)
+
+ # The output weights are the same as the input embeddings, but there is
+ # an output-only bias for each token.
+ self.decoder = nn.Linear(bert_model_embedding_weights.size(1),
+ bert_model_embedding_weights.size(0),
+ bias=False)
+ self.decoder.weight = bert_model_embedding_weights
+ self.bias = nn.Parameter(torch.zeros(bert_model_embedding_weights.size(0)))
+
+ def forward(self, hidden_states):
+ hidden_states = self.transform(hidden_states)
+ hidden_states = self.decoder(hidden_states) + self.bias
+ return hidden_states
+
+
+class BertOnlyMLMHead(nn.Module):
+ def __init__(self, config, bert_model_embedding_weights):
+ super(BertOnlyMLMHead, self).__init__()
+ self.predictions = BertLMPredictionHead(config, bert_model_embedding_weights)
+
+ def forward(self, sequence_output):
+ prediction_scores = self.predictions(sequence_output)
+ return prediction_scores
+
+
+class BertOnlyNSPHead(nn.Module):
+ def __init__(self, config):
+ super(BertOnlyNSPHead, self).__init__()
+ self.seq_relationship = nn.Linear(config.hidden_size, 2)
+
+ def forward(self, pooled_output):
+ seq_relationship_score = self.seq_relationship(pooled_output)
+ return seq_relationship_score
+
+
+class BertPreTrainingHeads(nn.Module):
+ def __init__(self, config, bert_model_embedding_weights):
+ super(BertPreTrainingHeads, self).__init__()
+ self.predictions = BertLMPredictionHead(config, bert_model_embedding_weights)
+ self.seq_relationship = nn.Linear(config.hidden_size, 2)
+
+ def forward(self, sequence_output, pooled_output):
+ prediction_scores = self.predictions(sequence_output)
+ seq_relationship_score = self.seq_relationship(pooled_output)
+ return prediction_scores, seq_relationship_score
+
+
+class BertPreTrainedModel(nn.Module):
+ """ An abstract class to handle weights initialization and
+ a simple interface for dowloading and loading pretrained models.
+ """
+ def __init__(self, config, *inputs, **kwargs):
+ super(BertPreTrainedModel, self).__init__()
+ if not isinstance(config, BertConfig):
+ raise ValueError(
+ "Parameter config in `{}(config)` should be an instance of class `BertConfig`. "
+ "To create a model from a Google pretrained model use "
+ "`model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`".format(
+ self.__class__.__name__, self.__class__.__name__
+ ))
+ self.config = config
+
+ def init_bert_weights(self, module):
+ """ Initialize the weights.
+ """
+ if isinstance(module, (nn.Linear, nn.Embedding)):
+ # Slightly different from the TF version which uses truncated_normal for initialization
+ # cf https://github.com/pytorch/pytorch/pull/5617
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+ elif isinstance(module, BertLayerNorm):
+ module.bias.data.zero_()
+ module.weight.data.fill_(1.0)
+ if isinstance(module, nn.Linear) and module.bias is not None:
+ module.bias.data.zero_()
+
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, state_dict=None, cache_dir=None,
+ from_tf=False, *inputs, **kwargs):
+ """
+ Instantiate a BertPreTrainedModel from a pre-trained model file or a pytorch state dict.
+ Download and cache the pre-trained model file if needed.
+
+ Params:
+ pretrained_model_name_or_path: either:
+ - a str with the name of a pre-trained model to load selected in the list of:
+ . `bert-base-uncased`
+ . `bert-large-uncased`
+ . `bert-base-cased`
+ . `bert-large-cased`
+ . `bert-base-multilingual-uncased`
+ . `bert-base-multilingual-cased`
+ . `bert-base-chinese`
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . `pytorch_model.bin` a PyTorch dump of a BertForPreTraining instance
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . `model.chkpt` a TensorFlow checkpoint
+ from_tf: should we load the weights from a locally saved TensorFlow checkpoint
+ cache_dir: an optional path to a folder in which the pre-trained models will be cached.
+ state_dict: an optional state dictionnary (collections.OrderedDict object) to use instead of Google pre-trained models
+ *inputs, **kwargs: additional input for the specific Bert class
+ (ex: num_labels for BertForSequenceClassification)
+ """
+ if pretrained_model_name_or_path in PRETRAINED_MODEL_ARCHIVE_MAP:
+ archive_file = PRETRAINED_MODEL_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ archive_file = pretrained_model_name_or_path
+ # redirect to the cache, if necessary
+ try:
+ resolved_archive_file = cached_path(archive_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find any file "
+ "associated to this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_MODEL_ARCHIVE_MAP.keys()),
+ archive_file))
+ return None
+ if resolved_archive_file == archive_file:
+ logger.info("loading archive file {}".format(archive_file))
+ else:
+ logger.info("loading archive file {} from cache at {}".format(
+ archive_file, resolved_archive_file))
+ tempdir = None
+ if os.path.isdir(resolved_archive_file) or from_tf:
+ serialization_dir = resolved_archive_file
+ else:
+ # Extract archive to temp dir
+ tempdir = tempfile.mkdtemp()
+ logger.info("extracting archive file {} to temp dir {}".format(
+ resolved_archive_file, tempdir))
+ with tarfile.open(resolved_archive_file, 'r:gz') as archive:
+ archive.extractall(tempdir)
+ serialization_dir = tempdir
+ # Load config
+ config_file = os.path.join(serialization_dir, CONFIG_NAME)
+ if not os.path.exists(config_file):
+ # Backward compatibility with old naming format
+ config_file = os.path.join(serialization_dir, BERT_CONFIG_NAME)
+ config = BertConfig.from_json_file(config_file)
+ logger.info("Model config {}".format(config))
+ # Instantiate model.
+ model = cls(config, *inputs, **kwargs)
+ if state_dict is None and not from_tf:
+ weights_path = os.path.join(serialization_dir, WEIGHTS_NAME)
+ state_dict = torch.load(weights_path, map_location='cpu')
+ if tempdir:
+ # Clean up temp dir
+ shutil.rmtree(tempdir)
+ if from_tf:
+ # Directly load from a TensorFlow checkpoint
+ weights_path = os.path.join(serialization_dir, TF_WEIGHTS_NAME)
+ return load_tf_weights_in_bert(model, weights_path)
+ # Load from a PyTorch state_dict
+ old_keys = []
+ new_keys = []
+ for key in state_dict.keys():
+ new_key = None
+ if 'gamma' in key:
+ new_key = key.replace('gamma', 'weight')
+ if 'beta' in key:
+ new_key = key.replace('beta', 'bias')
+ if new_key:
+ old_keys.append(key)
+ new_keys.append(new_key)
+ for old_key, new_key in zip(old_keys, new_keys):
+ state_dict[new_key] = state_dict.pop(old_key)
+
+ missing_keys = []
+ unexpected_keys = []
+ error_msgs = []
+ # copy state_dict so _load_from_state_dict can modify it
+ metadata = getattr(state_dict, '_metadata', None)
+ state_dict = state_dict.copy()
+ if metadata is not None:
+ state_dict._metadata = metadata
+
+ def load(module, prefix=''):
+ local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+ module._load_from_state_dict(
+ state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
+ for name, child in module._modules.items():
+ if child is not None:
+ load(child, prefix + name + '.')
+ start_prefix = ''
+ if not hasattr(model, 'bert') and any(s.startswith('bert.') for s in state_dict.keys()):
+ start_prefix = 'bert.'
+ load(model, prefix=start_prefix)
+ if len(missing_keys) > 0:
+ logger.info("Weights of {} not initialized from pretrained model: {}".format(
+ model.__class__.__name__, missing_keys))
+ if len(unexpected_keys) > 0:
+ logger.info("Weights from pretrained model not used in {}: {}".format(
+ model.__class__.__name__, unexpected_keys))
+ if len(error_msgs) > 0:
+ raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
+ model.__class__.__name__, "\n\t".join(error_msgs)))
+ return model
+
+
+class BertModel(BertPreTrainedModel):
+ """BERT model ("Bidirectional Embedding Representations from a Transformer").
+
+ Params:
+ config: a BertConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `output_all_encoded_layers`: boolean which controls the content of the `encoded_layers` output as described below. Default: `True`.
+
+ Outputs: Tuple of (encoded_layers, pooled_output)
+ `encoded_layers`: controled by `output_all_encoded_layers` argument:
+ - `output_all_encoded_layers=True`: outputs a list of the full sequences of encoded-hidden-states at the end
+ of each attention block (i.e. 12 full sequences for BERT-base, 24 for BERT-large), each
+ encoded-hidden-state is a torch.FloatTensor of size [batch_size, sequence_length, hidden_size],
+ - `output_all_encoded_layers=False`: outputs only the full sequence of hidden-states corresponding
+ to the last attention block of shape [batch_size, sequence_length, hidden_size],
+ `pooled_output`: a torch.FloatTensor of size [batch_size, hidden_size] which is the output of a
+ classifier pretrained on top of the hidden state associated to the first character of the
+ input (`CLS`) to train on the Next-Sentence task (see BERT's paper).
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = modeling.BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ model = modeling.BertModel(config=config)
+ all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config):
+ super(BertModel, self).__init__(config)
+ self.embeddings = BertEmbeddings(config)
+ self.encoder = BertEncoder(config)
+ self.pooler = BertPooler(config)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, output_all_encoded_layers=True,
+ uniforms=[False]*12):
+ if attention_mask is None:
+ attention_mask = torch.ones_like(input_ids)
+ if token_type_ids is None:
+ token_type_ids = torch.zeros_like(input_ids)
+
+ # We create a 3D attention mask from a 2D tensor mask.
+ # Sizes are [batch_size, 1, 1, to_seq_length]
+ # So we can broadcast to [batch_size, num_heads, from_seq_length, to_seq_length]
+ # this attention mask is more simple than the triangular masking of causal attention
+ # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
+ extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)
+
+ # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
+ # masked positions, this operation will create a tensor which is 0.0 for
+ # positions we want to attend and -10000.0 for masked positions.
+ # Since we are adding it to the raw scores before the softmax, this is
+ # effectively the same as removing these entirely.
+ extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype) # fp16 compatibility
+ extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0
+
+ embedding_output = self.embeddings(input_ids, token_type_ids)
+ encoded_layers, all_attentions = self.encoder(embedding_output,
+ extended_attention_mask,
+ output_all_encoded_layers=output_all_encoded_layers, uniforms=uniforms)
+ sequence_output = encoded_layers[-1]
+ pooled_output = self.pooler(sequence_output)
+ if not output_all_encoded_layers:
+ encoded_layers = encoded_layers[-1]
+ return encoded_layers, pooled_output, all_attentions
+
+
+class BertForPreTraining(BertPreTrainedModel):
+ """BERT model with pre-training heads.
+ This module comprises the BERT model followed by the two pre-training heads:
+ - the masked language modeling head, and
+ - the next sentence classification head.
+
+ Params:
+ config: a BertConfig class instance with the configuration to build a new model.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `masked_lm_labels`: optional masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length]
+ with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., vocab_size]
+ `next_sentence_label`: optional next sentence classification loss: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, 1].
+ 0 => next sentence is the continuation, 1 => next sentence is a random sentence.
+
+ Outputs:
+ if `masked_lm_labels` and `next_sentence_label` are not `None`:
+ Outputs the total_loss which is the sum of the masked language modeling loss and the next
+ sentence classification loss.
+ if `masked_lm_labels` or `next_sentence_label` is `None`:
+ Outputs a tuple comprising
+ - the masked language modeling logits of shape [batch_size, sequence_length, vocab_size], and
+ - the next sentence classification logits of shape [batch_size, 2].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ model = BertForPreTraining(config)
+ masked_lm_logits_scores, seq_relationship_logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config):
+ super(BertForPreTraining, self).__init__(config)
+ self.bert = BertModel(config)
+ self.cls = BertPreTrainingHeads(config, self.bert.embeddings.word_embeddings.weight)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, masked_lm_labels=None, next_sentence_label=None):
+ sequence_output, pooled_output = self.bert(input_ids, token_type_ids, attention_mask,
+ output_all_encoded_layers=False)
+ prediction_scores, seq_relationship_score = self.cls(sequence_output, pooled_output)
+
+ if masked_lm_labels is not None and next_sentence_label is not None:
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
+ next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
+ total_loss = masked_lm_loss + next_sentence_loss
+ return total_loss
+ else:
+ return prediction_scores, seq_relationship_score
+
+
+class BertForMaskedLM(BertPreTrainedModel):
+ """BERT model with the masked language modeling head.
+ This module comprises the BERT model followed by the masked language modeling head.
+
+ Params:
+ config: a BertConfig class instance with the configuration to build a new model.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `masked_lm_labels`: masked language modeling labels: torch.LongTensor of shape [batch_size, sequence_length]
+ with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., vocab_size]
+
+ Outputs:
+ if `masked_lm_labels` is not `None`:
+ Outputs the masked language modeling loss.
+ if `masked_lm_labels` is `None`:
+ Outputs the masked language modeling logits of shape [batch_size, sequence_length, vocab_size].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ model = BertForMaskedLM(config)
+ masked_lm_logits_scores = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config):
+ super(BertForMaskedLM, self).__init__(config)
+ self.bert = BertModel(config)
+ self.cls = BertOnlyMLMHead(config, self.bert.embeddings.word_embeddings.weight)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, masked_lm_labels=None):
+ sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask,
+ output_all_encoded_layers=False)
+ prediction_scores = self.cls(sequence_output)
+
+ if masked_lm_labels is not None:
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), masked_lm_labels.view(-1))
+ return masked_lm_loss
+ else:
+ return prediction_scores
+
+
+class BertForNextSentencePrediction(BertPreTrainedModel):
+ """BERT model with next sentence prediction head.
+ This module comprises the BERT model followed by the next sentence classification head.
+
+ Params:
+ config: a BertConfig class instance with the configuration to build a new model.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `next_sentence_label`: next sentence classification loss: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, 1].
+ 0 => next sentence is the continuation, 1 => next sentence is a random sentence.
+
+ Outputs:
+ if `next_sentence_label` is not `None`:
+ Outputs the total_loss which is the sum of the masked language modeling loss and the next
+ sentence classification loss.
+ if `next_sentence_label` is `None`:
+ Outputs the next sentence classification logits of shape [batch_size, 2].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ model = BertForNextSentencePrediction(config)
+ seq_relationship_logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config):
+ super(BertForNextSentencePrediction, self).__init__(config)
+ self.bert = BertModel(config)
+ self.cls = BertOnlyNSPHead(config)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, next_sentence_label=None):
+ _, pooled_output = self.bert(input_ids, token_type_ids, attention_mask,
+ output_all_encoded_layers=False)
+ seq_relationship_score = self.cls( pooled_output)
+
+ if next_sentence_label is not None:
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ next_sentence_loss = loss_fct(seq_relationship_score.view(-1, 2), next_sentence_label.view(-1))
+ return next_sentence_loss
+ else:
+ return seq_relationship_score
+
+
+class BertForSequenceClassification(BertPreTrainedModel):
+ """BERT model for classification.
+ This module is composed of the BERT model with a linear layer on top of
+ the pooled output.
+
+ Params:
+ `config`: a BertConfig class instance with the configuration to build a new model.
+ `num_labels`: the number of classes for the classifier. Default = 2.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary. Items in the batch should begin with the special "CLS" token. (see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `labels`: labels for the classification output: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, ..., num_labels].
+
+ Outputs:
+ if `labels` is not `None`:
+ Outputs the CrossEntropy classification loss of the output with the labels.
+ if `labels` is `None`:
+ Outputs the classification logits of shape [batch_size, num_labels].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ num_labels = 2
+
+ model = BertForSequenceClassification(config, num_labels)
+ logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config, num_labels):
+ super(BertForSequenceClassification, self).__init__(config)
+ self.num_labels = num_labels
+ self.bert = BertModel(config)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+ self.classifier = nn.Linear(config.hidden_size, num_labels)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None, uniforms=[False]*12):
+ _, pooled_output, attentions = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False,
+ uniforms=uniforms)
+ attentions = torch.stack(attentions, dim=1)
+ pooled_output = self.dropout(pooled_output)
+ logits = self.classifier(pooled_output)
+
+ if labels is not None:
+ loss_fct = CrossEntropyLoss()
+ loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+ return loss
+ else:
+ return logits, attentions
+
+
+class BertForMultipleChoice(BertPreTrainedModel):
+ """BERT model for multiple choice tasks.
+ This module is composed of the BERT model with a linear layer on top of
+ the pooled output.
+
+ Params:
+ `config`: a BertConfig class instance with the configuration to build a new model.
+ `num_choices`: the number of classes for the classifier. Default = 2.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length]
+ with the token types indices selected in [0, 1]. Type 0 corresponds to a `sentence A`
+ and type 1 corresponds to a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, num_choices, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `labels`: labels for the classification output: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, ..., num_choices].
+
+ Outputs:
+ if `labels` is not `None`:
+ Outputs the CrossEntropy classification loss of the output with the labels.
+ if `labels` is `None`:
+ Outputs the classification logits of shape [batch_size, num_labels].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]], [[12, 16, 42], [14, 28, 57]]])
+ input_mask = torch.LongTensor([[[1, 1, 1], [1, 1, 0]],[[1,1,0], [1, 0, 0]]])
+ token_type_ids = torch.LongTensor([[[0, 0, 1], [0, 1, 0]],[[0, 1, 1], [0, 0, 1]]])
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ num_choices = 2
+
+ model = BertForMultipleChoice(config, num_choices)
+ logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config, num_choices):
+ super(BertForMultipleChoice, self).__init__(config)
+ self.num_choices = num_choices
+ self.bert = BertModel(config)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+ self.classifier = nn.Linear(config.hidden_size, 1)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
+ flat_input_ids = input_ids.view(-1, input_ids.size(-1))
+ flat_token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1))
+ flat_attention_mask = attention_mask.view(-1, attention_mask.size(-1))
+ _, pooled_output = self.bert(flat_input_ids, flat_token_type_ids, flat_attention_mask, output_all_encoded_layers=False)
+ pooled_output = self.dropout(pooled_output)
+ logits = self.classifier(pooled_output)
+ reshaped_logits = logits.view(-1, self.num_choices)
+
+ if labels is not None:
+ loss_fct = CrossEntropyLoss()
+ loss = loss_fct(reshaped_logits, labels)
+ return loss
+ else:
+ return reshaped_logits
+
+
+class BertForTokenClassification(BertPreTrainedModel):
+ """BERT model for token-level classification.
+ This module is composed of the BERT model with a linear layer on top of
+ the full hidden state of the last layer.
+
+ Params:
+ `config`: a BertConfig class instance with the configuration to build a new model.
+ `num_labels`: the number of classes for the classifier. Default = 2.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `labels`: labels for the classification output: torch.LongTensor of shape [batch_size, sequence_length]
+ with indices selected in [0, ..., num_labels].
+
+ Outputs:
+ if `labels` is not `None`:
+ Outputs the CrossEntropy classification loss of the output with the labels.
+ if `labels` is `None`:
+ Outputs the classification logits of shape [batch_size, sequence_length, num_labels].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ num_labels = 2
+
+ model = BertForTokenClassification(config, num_labels)
+ logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config, num_labels):
+ super(BertForTokenClassification, self).__init__(config)
+ self.num_labels = num_labels
+ self.bert = BertModel(config)
+ self.dropout = nn.Dropout(config.hidden_dropout_prob)
+ self.classifier = nn.Linear(config.hidden_size, num_labels)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, labels=None):
+ sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
+ sequence_output = self.dropout(sequence_output)
+ logits = self.classifier(sequence_output)
+
+ if labels is not None:
+ loss_fct = CrossEntropyLoss()
+ # Only keep active parts of the loss
+ if attention_mask is not None:
+ active_loss = attention_mask.view(-1) == 1
+ active_logits = logits.view(-1, self.num_labels)[active_loss]
+ active_labels = labels.view(-1)[active_loss]
+ loss = loss_fct(active_logits, active_labels)
+ else:
+ loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
+ return loss
+ else:
+ return logits
+
+
+class BertForQuestionAnswering(BertPreTrainedModel):
+ """BERT model for Question Answering (span extraction).
+ This module is composed of the BERT model with a linear layer on top of
+ the sequence output that computes start_logits and end_logits
+
+ Params:
+ `config`: a BertConfig class instance with the configuration to build a new model.
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the word token indices in the vocabulary(see the tokens preprocessing logic in the scripts
+ `extract_features.py`, `run_classifier.py` and `run_squad.py`)
+ `token_type_ids`: an optional torch.LongTensor of shape [batch_size, sequence_length] with the token
+ types indices selected in [0, 1]. Type 0 corresponds to a `sentence A` and type 1 corresponds to
+ a `sentence B` token (see BERT paper for more details).
+ `attention_mask`: an optional torch.LongTensor of shape [batch_size, sequence_length] with indices
+ selected in [0, 1]. It's a mask to be used if the input sequence length is smaller than the max
+ input sequence length in the current batch. It's the mask that we typically use for attention when
+ a batch has varying length sentences.
+ `start_positions`: position of the first token for the labeled span: torch.LongTensor of shape [batch_size].
+ Positions are clamped to the length of the sequence and position outside of the sequence are not taken
+ into account for computing the loss.
+ `end_positions`: position of the last token for the labeled span: torch.LongTensor of shape [batch_size].
+ Positions are clamped to the length of the sequence and position outside of the sequence are not taken
+ into account for computing the loss.
+
+ Outputs:
+ if `start_positions` and `end_positions` are not `None`:
+ Outputs the total_loss which is the sum of the CrossEntropy loss for the start and end token positions.
+ if `start_positions` or `end_positions` is `None`:
+ Outputs a tuple of start_logits, end_logits which are the logits respectively for the start and end
+ position tokens of shape [batch_size, sequence_length].
+
+ Example usage:
+ ```python
+ # Already been converted into WordPiece token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
+ token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
+
+ config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
+ num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
+
+ model = BertForQuestionAnswering(config)
+ start_logits, end_logits = model(input_ids, token_type_ids, input_mask)
+ ```
+ """
+ def __init__(self, config):
+ super(BertForQuestionAnswering, self).__init__(config)
+ self.bert = BertModel(config)
+ # TODO check with Google if it's normal there is no dropout on the token classifier of SQuAD in the TF version
+ # self.dropout = nn.Dropout(config.hidden_dropout_prob)
+ self.qa_outputs = nn.Linear(config.hidden_size, 2)
+ self.apply(self.init_bert_weights)
+
+ def forward(self, input_ids, token_type_ids=None, attention_mask=None, start_positions=None, end_positions=None):
+ sequence_output, _ = self.bert(input_ids, token_type_ids, attention_mask, output_all_encoded_layers=False)
+ logits = self.qa_outputs(sequence_output)
+ start_logits, end_logits = logits.split(1, dim=-1)
+ start_logits = start_logits.squeeze(-1)
+ end_logits = end_logits.squeeze(-1)
+
+ if start_positions is not None and end_positions is not None:
+ # If we are on multi-GPU, split add a dimension
+ if len(start_positions.size()) > 1:
+ start_positions = start_positions.squeeze(-1)
+ if len(end_positions.size()) > 1:
+ end_positions = end_positions.squeeze(-1)
+ # sometimes the start/end positions are outside our model inputs, we ignore these terms
+ ignored_index = start_logits.size(1)
+ start_positions.clamp_(0, ignored_index)
+ end_positions.clamp_(0, ignored_index)
+
+ loss_fct = CrossEntropyLoss(ignore_index=ignored_index)
+ start_loss = loss_fct(start_logits, start_positions)
+ end_loss = loss_fct(end_logits, end_positions)
+ total_loss = (start_loss + end_loss) / 2
+ return total_loss
+ else:
+ return start_logits, end_logits
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_gpt2.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_gpt2.py
new file mode 100644
index 0000000000000000000000000000000000000000..063c525d98cd91bbd5364dba809436f8aeed8233
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_gpt2.py
@@ -0,0 +1,711 @@
+# coding=utf-8
+# Copyright 2018 The OpenAI Team Authors and HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch OpenAI GPT-2 model."""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import collections
+import copy
+import json
+import logging
+import math
+import os
+import shutil
+import tarfile
+import tempfile
+import sys
+from io import open
+
+import torch
+import torch.nn as nn
+from torch.nn import CrossEntropyLoss
+from torch.nn.parameter import Parameter
+
+from .file_utils import cached_path, CONFIG_NAME, WEIGHTS_NAME
+from .modeling import BertLayerNorm as LayerNorm
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_MODEL_ARCHIVE_MAP = {"gpt2": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin"}
+PRETRAINED_CONFIG_ARCHIVE_MAP = {"gpt2": "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json"}
+
+def load_tf_weights_in_gpt2(model, gpt2_checkpoint_path):
+ """ Load tf checkpoints in a pytorch model
+ """
+ try:
+ import re
+ import numpy as np
+ import tensorflow as tf
+ except ImportError:
+ print("Loading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+ tf_path = os.path.abspath(gpt2_checkpoint_path)
+ print("Converting TensorFlow checkpoint from {}".format(tf_path))
+ # Load weights from TF model
+ init_vars = tf.train.list_variables(tf_path)
+ names = []
+ arrays = []
+ for name, shape in init_vars:
+ print("Loading TF weight {} with shape {}".format(name, shape))
+ array = tf.train.load_variable(tf_path, name)
+ names.append(name)
+ arrays.append(array.squeeze())
+
+ for name, array in zip(names, arrays):
+ name = name[6:] # skip "model/"
+ name = name.split('/')
+ pointer = model
+ for m_name in name:
+ if re.fullmatch(r'[A-Za-z]+\d+', m_name):
+ l = re.split(r'(\d+)', m_name)
+ else:
+ l = [m_name]
+ if l[0] == 'w' or l[0] == 'g':
+ pointer = getattr(pointer, 'weight')
+ elif l[0] == 'b':
+ pointer = getattr(pointer, 'bias')
+ elif l[0] == 'wpe' or l[0] == 'wte':
+ pointer = getattr(pointer, l[0])
+ pointer = getattr(pointer, 'weight')
+ else:
+ pointer = getattr(pointer, l[0])
+ if len(l) >= 2:
+ num = int(l[1])
+ pointer = pointer[num]
+ try:
+ assert pointer.shape == array.shape
+ except AssertionError as e:
+ e.args += (pointer.shape, array.shape)
+ raise
+ print("Initialize PyTorch weight {}".format(name))
+ pointer.data = torch.from_numpy(array)
+ return model
+
+
+def gelu(x):
+ return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+
+class GPT2Config(object):
+ """Configuration class to store the configuration of a `GPT2Model`.
+ """
+
+ def __init__(
+ self,
+ vocab_size_or_config_json_file=50257,
+ n_positions=1024,
+ n_ctx=1024,
+ n_embd=768,
+ n_layer=12,
+ n_head=12,
+ layer_norm_epsilon=1e-5,
+ initializer_range=0.02,
+ ):
+ """Constructs GPT2Config.
+
+ Args:
+ vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `GPT2Model` or a configuration json file.
+ n_positions: Number of positional embeddings.
+ n_ctx: Size of the causal mask (usually same as n_positions).
+ n_embd: Dimensionality of the embeddings and hidden states.
+ n_layer: Number of hidden layers in the Transformer encoder.
+ n_head: Number of attention heads for each attention layer in
+ the Transformer encoder.
+ layer_norm_epsilon: epsilon to use in the layer norm layers
+ initializer_range: The sttdev of the truncated_normal_initializer for
+ initializing all weight matrices.
+ """
+ if isinstance(vocab_size_or_config_json_file, str) or (sys.version_info[0] == 2
+ and isinstance(vocab_size_or_config_json_file, unicode)):
+ with open(vocab_size_or_config_json_file, "r", encoding="utf-8") as reader:
+ json_config = json.loads(reader.read())
+ for key, value in json_config.items():
+ self.__dict__[key] = value
+ elif isinstance(vocab_size_or_config_json_file, int):
+ self.vocab_size = vocab_size_or_config_json_file
+ self.n_ctx = n_ctx
+ self.n_positions = n_positions
+ self.n_embd = n_embd
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.layer_norm_epsilon = layer_norm_epsilon
+ self.initializer_range = initializer_range
+ else:
+ raise ValueError(
+ "First argument must be either a vocabulary size (int)"
+ "or the path to a pretrained model config file (str)"
+ )
+
+ @classmethod
+ def from_dict(cls, json_object):
+ """Constructs a `GPT2Config` from a Python dictionary of parameters."""
+ config = GPT2Config(vocab_size_or_config_json_file=-1)
+ for key, value in json_object.items():
+ config.__dict__[key] = value
+ return config
+
+ @classmethod
+ def from_json_file(cls, json_file):
+ """Constructs a `GPT2Config` from a json file of parameters."""
+ with open(json_file, "r", encoding="utf-8") as reader:
+ text = reader.read()
+ return cls.from_dict(json.loads(text))
+
+ def __repr__(self):
+ return str(self.to_json_string())
+
+ def to_dict(self):
+ """Serializes this instance to a Python dictionary."""
+ output = copy.deepcopy(self.__dict__)
+ return output
+
+ def to_json_string(self):
+ """Serializes this instance to a JSON string."""
+ return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+ def to_json_file(self, json_file_path):
+ """ Save this instance to a json file."""
+ with open(json_file_path, "w", encoding='utf-8') as writer:
+ writer.write(self.to_json_string())
+
+
+class Conv1D(nn.Module):
+ def __init__(self, nf, nx):
+ super(Conv1D, self).__init__()
+ self.nf = nf
+ w = torch.empty(nx, nf)
+ nn.init.normal_(w, std=0.02)
+ self.weight = Parameter(w)
+ self.bias = Parameter(torch.zeros(nf))
+
+ def forward(self, x):
+ size_out = x.size()[:-1] + (self.nf,)
+ x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
+ x = x.view(*size_out)
+ return x
+
+
+class Attention(nn.Module):
+ def __init__(self, nx, n_ctx, config, scale=False):
+ super(Attention, self).__init__()
+ n_state = nx # in Attention: n_state=768 (nx=n_embd)
+ # [switch nx => n_state from Block to Attention to keep identical to TF implem]
+ assert n_state % config.n_head == 0
+ self.register_buffer("bias", torch.tril(torch.ones(n_ctx, n_ctx)).view(1, 1, n_ctx, n_ctx))
+ self.n_head = config.n_head
+ self.split_size = n_state
+ self.scale = scale
+ self.c_attn = Conv1D(n_state * 3, nx)
+ self.c_proj = Conv1D(n_state, nx)
+
+ def _attn(self, q, k, v):
+ w = torch.matmul(q, k)
+ if self.scale:
+ w = w / math.sqrt(v.size(-1))
+ nd, ns = w.size(-2), w.size(-1)
+ b = self.bias[:, :, ns-nd:ns, :ns]
+ w = w * b - 1e4 * (1 - b)
+
+ w = nn.Softmax(dim=-1)(w)
+ return torch.matmul(w, v)
+
+ def merge_heads(self, x):
+ x = x.permute(0, 2, 1, 3).contiguous()
+ new_x_shape = x.size()[:-2] + (x.size(-2) * x.size(-1),)
+ return x.view(*new_x_shape) # in Tensorflow implem: fct merge_states
+
+ def split_heads(self, x, k=False):
+ new_x_shape = x.size()[:-1] + (self.n_head, x.size(-1) // self.n_head)
+ x = x.view(*new_x_shape) # in Tensorflow implem: fct split_states
+ if k:
+ return x.permute(0, 2, 3, 1) # (batch, head, head_features, seq_length)
+ else:
+ return x.permute(0, 2, 1, 3) # (batch, head, seq_length, head_features)
+
+ def forward(self, x, layer_past=None):
+ x = self.c_attn(x)
+ query, key, value = x.split(self.split_size, dim=2)
+ query = self.split_heads(query)
+ key = self.split_heads(key, k=True)
+ value = self.split_heads(value)
+ if layer_past is not None:
+ past_key, past_value = layer_past[0].transpose(-2, -1), layer_past[1] # transpose back cf below
+ key = torch.cat((past_key, key), dim=-1)
+ value = torch.cat((past_value, value), dim=-2)
+ present = torch.stack((key.transpose(-2, -1), value)) # transpose to have same shapes for stacking
+ a = self._attn(query, key, value)
+ a = self.merge_heads(a)
+ a = self.c_proj(a)
+ return a, present
+
+
+class MLP(nn.Module):
+ def __init__(self, n_state, config): # in MLP: n_state=3072 (4 * n_embd)
+ super(MLP, self).__init__()
+ nx = config.n_embd
+ self.c_fc = Conv1D(n_state, nx)
+ self.c_proj = Conv1D(nx, n_state)
+ self.act = gelu
+
+ def forward(self, x):
+ h = self.act(self.c_fc(x))
+ h2 = self.c_proj(h)
+ return h2
+
+
+class Block(nn.Module):
+ def __init__(self, n_ctx, config, scale=False):
+ super(Block, self).__init__()
+ nx = config.n_embd
+ self.ln_1 = LayerNorm(nx, eps=config.layer_norm_epsilon)
+ self.attn = Attention(nx, n_ctx, config, scale)
+ self.ln_2 = LayerNorm(nx, eps=config.layer_norm_epsilon)
+ self.mlp = MLP(4 * nx, config)
+
+ def forward(self, x, layer_past=None):
+ a, present = self.attn(self.ln_1(x), layer_past=layer_past)
+ x = x + a
+ m = self.mlp(self.ln_2(x))
+ x = x + m
+ return x, present
+
+
+class GPT2LMHead(nn.Module):
+ """ Language Model Head for the transformer """
+
+ def __init__(self, model_embeddings_weights, config):
+ super(GPT2LMHead, self).__init__()
+ self.n_embd = config.n_embd
+ self.set_embeddings_weights(model_embeddings_weights)
+
+ def set_embeddings_weights(self, model_embeddings_weights):
+ embed_shape = model_embeddings_weights.shape
+ self.decoder = nn.Linear(embed_shape[1], embed_shape[0], bias=False)
+ self.decoder.weight = model_embeddings_weights # Tied weights
+
+ def forward(self, hidden_state):
+ # Truncated Language modeling logits (we remove the last token)
+ # h_trunc = h[:, :-1].contiguous().view(-1, self.n_embd)
+ lm_logits = self.decoder(hidden_state)
+ return lm_logits
+
+
+class GPT2MultipleChoiceHead(nn.Module):
+ """ Classifier Head for the transformer """
+
+ def __init__(self, config):
+ super(GPT2MultipleChoiceHead, self).__init__()
+ self.n_embd = config.n_embd
+ self.linear = nn.Linear(config.n_embd, 1)
+
+ nn.init.normal_(self.linear.weight, std=0.02)
+ nn.init.normal_(self.linear.bias, 0)
+
+ def forward(self, hidden_states, mc_token_ids):
+ # Classification logits
+ # hidden_state (bsz, num_choices, seq_length, hidden_size)
+ # mc_token_ids (bsz, num_choices)
+ mc_token_ids = mc_token_ids.unsqueeze(-1).unsqueeze(-1).expand(-1, -1, -1, hidden_states.size(-1))
+ # (bsz, num_choices, 1, hidden_size)
+ multiple_choice_h = hidden_states.gather(2, mc_token_ids).squeeze(2)
+ # (bsz, num_choices, hidden_size)
+ multiple_choice_logits = self.linear(multiple_choice_h).squeeze(-1)
+ # (bsz, num_choices)
+ return multiple_choice_logits
+
+
+class GPT2PreTrainedModel(nn.Module):
+ """ An abstract class to handle weights initialization and
+ a simple interface for dowloading and loading pretrained models.
+ """
+
+ def __init__(self, config, *inputs, **kwargs):
+ super(GPT2PreTrainedModel, self).__init__()
+ if not isinstance(config, GPT2Config):
+ raise ValueError(
+ "Parameter config in `{}(config)` should be an instance of class `GPT2Config`. "
+ "To create a model from a pretrained model use "
+ "`model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`".format(
+ self.__class__.__name__, self.__class__.__name__
+ )
+ )
+ self.config = config
+
+ def set_tied(self):
+ pass
+
+ def init_weights(self, module):
+ """ Initialize the weights.
+ """
+ if isinstance(module, (nn.Linear, nn.Embedding)):
+ # Slightly different from the TF version which uses truncated_normal for initialization
+ # cf https://github.com/pytorch/pytorch/pull/5617
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+ elif isinstance(module, LayerNorm):
+ module.bias.data.zero_()
+ module.weight.data.fill_(1.0)
+ if isinstance(module, nn.Linear) and module.bias is not None:
+ module.bias.data.zero_()
+
+ @classmethod
+ def from_pretrained(
+ cls, pretrained_model_name_or_path, state_dict=None, cache_dir=None, from_tf=False, *inputs, **kwargs
+ ):
+ """
+ Instantiate a GPT2PreTrainedModel from a pre-trained model file or a pytorch state dict.
+ Download and cache the pre-trained model file if needed.
+
+ Params:
+ pretrained_model_name_or_path: either:
+ - a str with the name of a pre-trained model to load selected in the list of:
+ . `gpt2`
+ - a path or url to a pretrained model archive containing:
+ . `gpt2_config.json` a configuration file for the model
+ . `pytorch_model.bin` a PyTorch dump of a GPT2Model instance
+ - a path or url to a pretrained model archive containing:
+ . `gpt2_config.json` a configuration file for the model
+ . a TensorFlow checkpoint with trained weights
+ from_tf: should we load the weights from a locally saved TensorFlow checkpoint
+ cache_dir: an optional path to a folder in which the pre-trained models will be cached.
+ state_dict: an optional state dictionary (collections.OrderedDict object) to use instead of pre-trained models
+ *inputs, **kwargs: additional input for the specific GPT class
+ """
+ if pretrained_model_name_or_path in PRETRAINED_MODEL_ARCHIVE_MAP:
+ archive_file = PRETRAINED_MODEL_ARCHIVE_MAP[pretrained_model_name_or_path]
+ config_file = PRETRAINED_CONFIG_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)
+ config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
+ # redirect to the cache, if necessary
+ try:
+ resolved_archive_file = cached_path(archive_file, cache_dir=cache_dir)
+ resolved_config_file = cached_path(config_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} and {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path, ", ".join(PRETRAINED_MODEL_ARCHIVE_MAP.keys()), pretrained_model_name_or_path,
+ archive_file, config_file
+ )
+ )
+ return None
+ if resolved_archive_file == archive_file and resolved_config_file == config_file:
+ logger.info("loading weights file {}".format(archive_file))
+ logger.info("loading configuration file {}".format(config_file))
+ else:
+ logger.info("loading weights file {} from cache at {}".format(
+ archive_file, resolved_archive_file))
+ logger.info("loading configuration file {} from cache at {}".format(
+ config_file, resolved_config_file))
+ # Load config
+ config = GPT2Config.from_json_file(resolved_config_file)
+ logger.info("Model config {}".format(config))
+ # Instantiate model.
+ model = cls(config, *inputs, **kwargs)
+ if state_dict is None and not from_tf:
+ state_dict = torch.load(resolved_archive_file, map_location='cpu')
+ if from_tf:
+ # Directly load from a TensorFlow checkpoint (stored as NumPy array)
+ return load_tf_weights_in_gpt2(model, resolved_archive_file)
+
+ old_keys = []
+ new_keys = []
+ for key in state_dict.keys():
+ new_key = None
+ if key.endswith(".g"):
+ new_key = key[:-2] + ".weight"
+ elif key.endswith(".b"):
+ new_key = key[:-2] + ".bias"
+ elif key.endswith(".w"):
+ new_key = key[:-2] + ".weight"
+ if new_key:
+ old_keys.append(key)
+ new_keys.append(new_key)
+ for old_key, new_key in zip(old_keys, new_keys):
+ state_dict[new_key] = state_dict.pop(old_key)
+
+ missing_keys = []
+ unexpected_keys = []
+ error_msgs = []
+ # copy state_dict so _load_from_state_dict can modify it
+ metadata = getattr(state_dict, "_metadata", None)
+ state_dict = state_dict.copy()
+ if metadata is not None:
+ state_dict._metadata = metadata
+
+ def load(module, prefix=""):
+ local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+ module._load_from_state_dict(
+ state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs
+ )
+ for name, child in module._modules.items():
+ if child is not None:
+ load(child, prefix + name + ".")
+
+ start_model = model
+ if hasattr(model, "transformer") and all(not s.startswith('transformer.') for s in state_dict.keys()):
+ start_model = model.transformer
+ load(start_model, prefix="")
+
+ if len(missing_keys) > 0:
+ logger.info(
+ "Weights of {} not initialized from pretrained model: {}".format(model.__class__.__name__, missing_keys)
+ )
+ if len(unexpected_keys) > 0:
+ logger.info(
+ "Weights from pretrained model not used in {}: {}".format(model.__class__.__name__, unexpected_keys)
+ )
+ if len(error_msgs) > 0:
+ raise RuntimeError(
+ "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
+ )
+
+ # Make sure we are still sharing the output and input embeddings after loading weights
+ model.set_tied()
+ return model
+
+
+class GPT2Model(GPT2PreTrainedModel):
+ """OpenAI GPT-2 model ("Language Models are Unsupervised Multitask Learners").
+
+ Params:
+ config: a GPT2Config class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length]
+ were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, config.vocab_size[
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+ `past`: an optional list of torch.LongTensor that contains pre-computed hidden-states
+ (key and values in the attention blocks) to speed up sequential decoding
+ (this is the presents output of the model, cf. below).
+
+ Outputs a tuple consisting of:
+ `hidden_states`: the encoded-hidden-states at the top of the model
+ as a torch.FloatTensor of size [batch_size, sequence_length, hidden_size]
+ (or more generally [d_1, ..., d_n, hidden_size] were d_1 ... d_n are the dimension of input_ids)
+ `presents`: a list of pre-computed hidden-states (key and values in each attention blocks) as
+ torch.FloatTensors. They can be reused to speed up sequential decoding.
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+
+ config = modeling_gpt2.GPT2Config()
+
+ model = modeling_gpt2.GPT2Model(config)
+ hidden_states, presents = model(input_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(GPT2Model, self).__init__(config)
+ self.wte = nn.Embedding(config.vocab_size, config.n_embd)
+ self.wpe = nn.Embedding(config.n_positions, config.n_embd)
+ block = Block(config.n_ctx, config, scale=True)
+ self.h = nn.ModuleList([copy.deepcopy(block) for _ in range(config.n_layer)])
+ self.ln_f = LayerNorm(config.n_embd, eps=config.layer_norm_epsilon)
+
+ self.apply(self.init_weights)
+
+ def forward(self, input_ids, position_ids=None, token_type_ids=None, past=None):
+ if past is None:
+ past_length = 0
+ past = [None] * len(self.h)
+ else:
+ past_length = past[0][0].size(-2)
+ if position_ids is None:
+ position_ids = torch.arange(past_length, input_ids.size(-1) + past_length, dtype=torch.long, device=input_ids.device)
+ position_ids = position_ids.unsqueeze(0).expand_as(input_ids)
+
+ input_shape = input_ids.size()
+ input_ids = input_ids.view(-1, input_ids.size(-1))
+ position_ids = position_ids.view(-1, position_ids.size(-1))
+
+ inputs_embeds = self.wte(input_ids)
+ position_embeds = self.wpe(position_ids)
+ if token_type_ids is not None:
+ token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1))
+ token_type_embeds = self.wte(token_type_ids)
+ else:
+ token_type_embeds = 0
+ hidden_states = inputs_embeds + position_embeds + token_type_embeds
+ presents = []
+ for block, layer_past in zip(self.h, past):
+ hidden_states, present = block(hidden_states, layer_past)
+ presents.append(present)
+ hidden_states = self.ln_f(hidden_states)
+ output_shape = input_shape + (hidden_states.size(-1),)
+ return hidden_states.view(*output_shape), presents
+
+
+class GPT2LMHeadModel(GPT2PreTrainedModel):
+ """OpenAI GPT-2 model with a Language Modeling head ("Language Models are Unsupervised Multitask Learners").
+
+ Params:
+ config: a GPT2Config class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length]
+ were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, config.vocab_size[
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+ `lm_labels`: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length]
+ with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., vocab_size]
+ `past`: an optional list of torch.LongTensor that contains pre-computed hidden-states
+ (key and values in the attention blocks) to speed up sequential decoding
+ (this is the presents output of the model, cf. below).
+
+ Outputs:
+ if `lm_labels` is not `None`:
+ Outputs the language modeling loss.
+ else a tuple:
+ `lm_logits`: the language modeling logits as a torch.FloatTensor of size [batch_size, sequence_length, config.vocab_size]
+ (or more generally [d_1, ..., d_n, config.vocab_size] were d_1 ... d_n are the dimension of input_ids)
+ `presents`: a list of pre-computed hidden-states (key and values in each attention blocks) as
+ torch.FloatTensors. They can be reused to speed up sequential decoding.
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+
+ config = modeling_gpt2.GPT2Config()
+
+ model = modeling_gpt2.GPT2LMHeadModel(config)
+ lm_logits, presents = model(input_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(GPT2LMHeadModel, self).__init__(config)
+ self.transformer = GPT2Model(config)
+ self.lm_head = GPT2LMHead(self.transformer.wte.weight, config)
+ self.apply(self.init_weights)
+
+ def set_tied(self):
+ """ Make sure we are sharing the embeddings
+ """
+ self.lm_head.set_embeddings_weights(self.transformer.wte.weight)
+
+ def forward(self, input_ids, position_ids=None, token_type_ids=None, lm_labels=None, past=None):
+ hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)
+ lm_logits = self.lm_head(hidden_states)
+ if lm_labels is not None:
+ # Shift so that tokens < n predict n
+ shift_logits = lm_logits[:, :-1].contiguous()
+ shift_labels = lm_labels[:, 1:].contiguous()
+
+ # Flatten the tokens
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
+ shift_labels.view(-1))
+ return loss
+ return lm_logits, presents
+
+
+class GPT2DoubleHeadsModel(GPT2PreTrainedModel):
+ """OpenAI GPT-2 model with a Language Modeling and a Multiple Choice head ("Language Models are Unsupervised Multitask Learners").
+
+ Params:
+ config: a GPT2Config class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the BPE token
+ indices selected in the range [0, config.vocab_size[
+ `mc_token_ids`: a torch.LongTensor of shape [batch_size, num_choices] with the index of the token from
+ which we should take the hidden state to feed the multiple choice classifier (usually last token of the sequence)
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+ `lm_labels`: optional language modeling labels: torch.LongTensor of shape [batch_size, num_choices, sequence_length]
+ with indices selected in [-1, 0, ..., config.vocab_size]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., config.vocab_size]
+ `multiple_choice_labels`: optional multiple choice labels: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, ..., num_choices].
+ `past`: an optional list of torch.LongTensor that contains pre-computed hidden-states
+ (key and values in the attention blocks) to speed up sequential decoding
+ (this is the presents output of the model, cf. below).
+
+ Outputs:
+ if `lm_labels` and `multiple_choice_labels` are not `None`:
+ Outputs a tuple of losses with the language modeling loss and the multiple choice loss.
+ else: a tuple with
+ `lm_logits`: the language modeling logits as a torch.FloatTensor of size [batch_size, num_choices, sequence_length, config.vocab_size]
+ `multiple_choice_logits`: the multiple choice logits as a torch.FloatTensor of size [batch_size, num_choices]
+ `presents`: a list of pre-computed hidden-states (key and values in each attention blocks) as
+ torch.FloatTensors. They can be reused to speed up sequential decoding.
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]]]) # (bsz, number of choice, seq length)
+ mc_token_ids = torch.LongTensor([[2], [1]]) # (bsz, number of choice)
+
+ config = modeling_gpt2.GPT2Config()
+
+ model = modeling_gpt2.GPT2LMHeadModel(config)
+ lm_logits, multiple_choice_logits, presents = model(input_ids, mc_token_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(GPT2DoubleHeadsModel, self).__init__(config)
+ self.transformer = GPT2Model(config)
+ self.lm_head = GPT2LMHead(self.transformer.wte.weight, config)
+ self.multiple_choice_head = GPT2MultipleChoiceHead(config)
+ self.apply(self.init_weights)
+
+ def set_tied(self):
+ """ Make sure we are sharing the embeddings
+ """
+ self.lm_head.set_embeddings_weights(self.transformer.wte.weight)
+
+ def forward(self, input_ids, mc_token_ids, lm_labels=None, mc_labels=None, token_type_ids=None, position_ids=None, past=None):
+ hidden_states, presents = self.transformer(input_ids, position_ids, token_type_ids, past)
+ lm_logits = self.lm_head(hidden_states)
+ mc_logits = self.multiple_choice_head(hidden_states, mc_token_ids)
+ losses = []
+ if lm_labels is not None:
+ shift_logits = lm_logits[:, :-1].contiguous()
+ shift_labels = lm_labels[:, 1:].contiguous()
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ losses.append(loss_fct(shift_logits.view(-1,
+ shift_logits.size(-1)), shift_labels.view(-1)))
+ if mc_labels is not None:
+ loss_fct = CrossEntropyLoss()
+ losses.append(loss_fct(mc_logits.view(-1, mc_logits.size(-1)), mc_labels.view(-1)))
+ if losses:
+ return losses
+ return lm_logits, mc_logits, presents
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_openai.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_openai.py
new file mode 100644
index 0000000000000000000000000000000000000000..f956462ddbfff09dde4c8595cada600ade404ed2
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_openai.py
@@ -0,0 +1,822 @@
+# coding=utf-8
+# Copyright 2018 The OpenAI Team Authors and HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch OpenAI GPT model."""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import collections
+import copy
+import json
+import logging
+import math
+import os
+import shutil
+import tarfile
+import tempfile
+import sys
+from io import open
+
+import torch
+import torch.nn as nn
+from torch.nn import CrossEntropyLoss
+from torch.nn.parameter import Parameter
+
+from .file_utils import cached_path, CONFIG_NAME, WEIGHTS_NAME
+from .modeling import BertLayerNorm as LayerNorm
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_MODEL_ARCHIVE_MAP = {"openai-gpt": "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-pytorch_model.bin"}
+PRETRAINED_CONFIG_ARCHIVE_MAP = {"openai-gpt": "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-config.json"}
+
+
+def load_tf_weights_in_openai_gpt(model, openai_checkpoint_folder_path):
+ """ Load tf pre-trained weights in a pytorch model (from NumPy arrays here)
+ """
+ import re
+ import numpy as np
+ print("Loading weights...")
+ names = json.load(open(openai_checkpoint_folder_path + '/parameters_names.json', "r", encoding='utf-8'))
+ shapes = json.load(open(openai_checkpoint_folder_path + '/params_shapes.json', "r", encoding='utf-8'))
+ offsets = np.cumsum([np.prod(shape) for shape in shapes])
+ init_params = [np.load(openai_checkpoint_folder_path + '/params_{}.npy'.format(n)) for n in range(10)]
+ init_params = np.split(np.concatenate(init_params, 0), offsets)[:-1]
+ init_params = [param.reshape(shape) for param, shape in zip(init_params, shapes)]
+
+ # This was used when we had a single embedding matrix for positions and tokens
+ # init_params[0] = np.concatenate([init_params[1], init_params[0]], 0)
+ # del init_params[1]
+ init_params = [arr.squeeze() for arr in init_params]
+
+ try:
+ assert model.tokens_embed.weight.shape == init_params[1].shape
+ assert model.positions_embed.weight.shape == init_params[0].shape
+ except AssertionError as e:
+ e.args += (model.tokens_embed.weight.shape, init_params[1].shape)
+ e.args += (model.positions_embed.weight.shape, init_params[0].shape)
+ raise
+
+ model.tokens_embed.weight.data = torch.from_numpy(init_params[1])
+ model.positions_embed.weight.data = torch.from_numpy(init_params[0])
+ names.pop(0)
+ # Pop position and token embedding arrays
+ init_params.pop(0)
+ init_params.pop(0)
+
+ for name, array in zip(names, init_params): # names[1:n_transfer], init_params[1:n_transfer]):
+ name = name[6:] # skip "model/"
+ assert name[-2:] == ":0"
+ name = name[:-2]
+ name = name.split('/')
+ pointer = model
+ for m_name in name:
+ if re.fullmatch(r'[A-Za-z]+\d+', m_name):
+ l = re.split(r'(\d+)', m_name)
+ else:
+ l = [m_name]
+ if l[0] == 'g':
+ pointer = getattr(pointer, 'weight')
+ elif l[0] == 'b':
+ pointer = getattr(pointer, 'bias')
+ elif l[0] == 'w':
+ pointer = getattr(pointer, 'weight')
+ else:
+ pointer = getattr(pointer, l[0])
+ if len(l) >= 2:
+ num = int(l[1])
+ pointer = pointer[num]
+ try:
+ assert pointer.shape == array.shape
+ except AssertionError as e:
+ e.args += (pointer.shape, array.shape)
+ raise
+ try:
+ assert pointer.shape == array.shape
+ except AssertionError as e:
+ e.args += (pointer.shape, array.shape)
+ raise
+ print("Initialize PyTorch weight {}".format(name))
+ pointer.data = torch.from_numpy(array)
+ return model
+
+
+def gelu(x):
+ return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
+
+
+def swish(x):
+ return x * torch.sigmoid(x)
+
+
+ACT_FNS = {"relu": nn.ReLU, "swish": swish, "gelu": gelu}
+
+
+class OpenAIGPTConfig(object):
+ """Configuration class to store the configuration of a `OpenAIGPTModel`.
+ """
+
+ def __init__(
+ self,
+ vocab_size_or_config_json_file=40478,
+ n_special=0,
+ n_positions=512,
+ n_ctx=512,
+ n_embd=768,
+ n_layer=12,
+ n_head=12,
+ afn="gelu",
+ resid_pdrop=0.1,
+ embd_pdrop=0.1,
+ attn_pdrop=0.1,
+ layer_norm_epsilon=1e-5,
+ initializer_range=0.02,
+ ):
+ """Constructs OpenAIGPTConfig.
+
+ Args:
+ vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `OpenAIGPTModel` or a configuration json file.
+ n_special: The number of special tokens to learn during fine-tuning ('[SEP]', '[CLF]', ...)
+ n_positions: Number of positional embeddings.
+ n_ctx: Size of the causal mask (usually same as n_positions).
+ n_embd: Dimensionality of the embeddings and hidden states.
+ n_layer: Number of hidden layers in the Transformer encoder.
+ n_head: Number of attention heads for each attention layer in
+ the Transformer encoder.
+ afn: The non-linear activation function (function or string) in the
+ encoder and pooler. If string, "gelu", "relu" and "swish" are supported.
+ resid_pdrop: The dropout probabilitiy for all fully connected
+ layers in the embeddings, encoder, and pooler.
+ attn_pdrop: The dropout ratio for the attention
+ probabilities.
+ embd_pdrop: The dropout ratio for the embeddings.
+ layer_norm_epsilon: epsilon to use in the layer norm layers
+ initializer_range: The sttdev of the truncated_normal_initializer for
+ initializing all weight matrices.
+ """
+ if isinstance(vocab_size_or_config_json_file, str) or (sys.version_info[0] == 2
+ and isinstance(vocab_size_or_config_json_file, unicode)):
+ with open(vocab_size_or_config_json_file, "r", encoding="utf-8") as reader:
+ json_config = json.loads(reader.read())
+ for key, value in json_config.items():
+ self.__dict__[key] = value
+ elif isinstance(vocab_size_or_config_json_file, int):
+ self.vocab_size = vocab_size_or_config_json_file
+ self.n_special = n_special
+ self.n_ctx = n_ctx
+ self.n_positions = n_positions
+ self.n_embd = n_embd
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.afn = afn
+ self.resid_pdrop = resid_pdrop
+ self.embd_pdrop = embd_pdrop
+ self.attn_pdrop = attn_pdrop
+ self.layer_norm_epsilon = layer_norm_epsilon
+ self.initializer_range = initializer_range
+ else:
+ raise ValueError(
+ "First argument must be either a vocabulary size (int)"
+ "or the path to a pretrained model config file (str)"
+ )
+
+ @property
+ def total_tokens_embeddings(self):
+ return self.vocab_size + self.n_special
+
+ @classmethod
+ def from_dict(cls, json_object):
+ """Constructs a `OpenAIGPTConfig` from a Python dictionary of parameters."""
+ config = OpenAIGPTConfig(vocab_size_or_config_json_file=-1)
+ for key, value in json_object.items():
+ config.__dict__[key] = value
+ return config
+
+ @classmethod
+ def from_json_file(cls, json_file):
+ """Constructs a `OpenAIGPTConfig` from a json file of parameters."""
+ with open(json_file, "r", encoding="utf-8") as reader:
+ text = reader.read()
+ return cls.from_dict(json.loads(text))
+
+ def __repr__(self):
+ return str(self.to_json_string())
+
+ def to_dict(self):
+ """Serializes this instance to a Python dictionary."""
+ output = copy.deepcopy(self.__dict__)
+ return output
+
+ def to_json_string(self):
+ """Serializes this instance to a JSON string."""
+ return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+ def to_json_file(self, json_file_path):
+ """ Save this instance to a json file."""
+ with open(json_file_path, "w", encoding='utf-8') as writer:
+ writer.write(self.to_json_string())
+
+
+class Conv1D(nn.Module):
+ def __init__(self, nf, rf, nx):
+ super(Conv1D, self).__init__()
+ self.rf = rf
+ self.nf = nf
+ if rf == 1: # faster 1x1 conv
+ w = torch.empty(nx, nf)
+ nn.init.normal_(w, std=0.02)
+ self.weight = Parameter(w)
+ self.bias = Parameter(torch.zeros(nf))
+ else: # was used to train LM
+ raise NotImplementedError
+
+ def forward(self, x):
+ if self.rf == 1:
+ size_out = x.size()[:-1] + (self.nf,)
+ x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
+ x = x.view(*size_out)
+ else:
+ raise NotImplementedError
+ return x
+
+
+class Attention(nn.Module):
+ def __init__(self, nx, n_ctx, config, scale=False):
+ super(Attention, self).__init__()
+ n_state = nx # in Attention: n_state=768 (nx=n_embd)
+ # [switch nx => n_state from Block to Attention to keep identical to TF implem]
+ assert n_state % config.n_head == 0
+ self.register_buffer("bias", torch.tril(torch.ones(n_ctx, n_ctx)).view(1, 1, n_ctx, n_ctx))
+ self.n_head = config.n_head
+ self.split_size = n_state
+ self.scale = scale
+ self.c_attn = Conv1D(n_state * 3, 1, nx)
+ self.c_proj = Conv1D(n_state, 1, nx)
+ self.attn_dropout = nn.Dropout(config.attn_pdrop)
+ self.resid_dropout = nn.Dropout(config.resid_pdrop)
+
+ def _attn(self, q, k, v):
+ w = torch.matmul(q, k)
+ if self.scale:
+ w = w / math.sqrt(v.size(-1))
+ # w = w * self.bias + -1e9 * (1 - self.bias) # TF implem method: mask_attn_weights
+ # XD: self.b may be larger than w, so we need to crop it
+ b = self.bias[:, :, : w.size(-2), : w.size(-1)]
+ w = w * b + -1e9 * (1 - b)
+
+ w = nn.Softmax(dim=-1)(w)
+ w = self.attn_dropout(w)
+ return torch.matmul(w, v)
+
+ def merge_heads(self, x):
+ x = x.permute(0, 2, 1, 3).contiguous()
+ new_x_shape = x.size()[:-2] + (x.size(-2) * x.size(-1),)
+ return x.view(*new_x_shape) # in Tensorflow implem: fct merge_states
+
+ def split_heads(self, x, k=False):
+ new_x_shape = x.size()[:-1] + (self.n_head, x.size(-1) // self.n_head)
+ x = x.view(*new_x_shape) # in Tensorflow implem: fct split_states
+ if k:
+ return x.permute(0, 2, 3, 1)
+ else:
+ return x.permute(0, 2, 1, 3)
+
+ def forward(self, x):
+ x = self.c_attn(x)
+ query, key, value = x.split(self.split_size, dim=2)
+ query = self.split_heads(query)
+ key = self.split_heads(key, k=True)
+ value = self.split_heads(value)
+ a = self._attn(query, key, value)
+ a = self.merge_heads(a)
+ a = self.c_proj(a)
+ a = self.resid_dropout(a)
+ return a
+
+
+class MLP(nn.Module):
+ def __init__(self, n_state, config): # in MLP: n_state=3072 (4 * n_embd)
+ super(MLP, self).__init__()
+ nx = config.n_embd
+ self.c_fc = Conv1D(n_state, 1, nx)
+ self.c_proj = Conv1D(nx, 1, n_state)
+ self.act = ACT_FNS[config.afn]
+ self.dropout = nn.Dropout(config.resid_pdrop)
+
+ def forward(self, x):
+ h = self.act(self.c_fc(x))
+ h2 = self.c_proj(h)
+ return self.dropout(h2)
+
+
+class Block(nn.Module):
+ def __init__(self, n_ctx, config, scale=False):
+ super(Block, self).__init__()
+ nx = config.n_embd
+ self.attn = Attention(nx, n_ctx, config, scale)
+ self.ln_1 = LayerNorm(nx, eps=config.layer_norm_epsilon)
+ self.mlp = MLP(4 * nx, config)
+ self.ln_2 = LayerNorm(nx, eps=config.layer_norm_epsilon)
+
+ def forward(self, x):
+ a = self.attn(x)
+ n = self.ln_1(x + a)
+ m = self.mlp(n)
+ h = self.ln_2(n + m)
+ return h
+
+
+class OpenAIGPTLMHead(nn.Module):
+ """ Language Model Head for the transformer """
+
+ def __init__(self, model_embeddings_weights, config):
+ super(OpenAIGPTLMHead, self).__init__()
+ self.n_embd = config.n_embd
+ self.set_embeddings_weights(model_embeddings_weights)
+
+ def set_embeddings_weights(self, model_embeddings_weights):
+ embed_shape = model_embeddings_weights.shape
+ self.decoder = nn.Linear(embed_shape[1], embed_shape[0], bias=False)
+ self.decoder.weight = model_embeddings_weights # Tied weights
+
+ def forward(self, hidden_state):
+ # Truncated Language modeling logits (we remove the last token)
+ # h_trunc = h[:, :-1].contiguous().view(-1, self.n_embd)
+ lm_logits = self.decoder(hidden_state)
+ return lm_logits
+
+
+class OpenAIGPTMultipleChoiceHead(nn.Module):
+ """ Classifier Head for the transformer """
+
+ def __init__(self, config):
+ super(OpenAIGPTMultipleChoiceHead, self).__init__()
+ self.n_embd = config.n_embd
+ # self.multiple_choice_token = multiple_choice_token
+ self.dropout = nn.Dropout2d(config.resid_pdrop) # To reproduce the noise_shape parameter of TF implementation
+ self.linear = nn.Linear(config.n_embd, 1)
+
+ nn.init.normal_(self.linear.weight, std=0.02)
+ nn.init.normal_(self.linear.bias, 0)
+
+ def forward(self, hidden_states, mc_token_ids):
+ # Classification logits
+ # hidden_state (bsz, num_choices, seq_length, hidden_size)
+ # mc_token_ids (bsz, num_choices)
+ mc_token_ids = mc_token_ids.unsqueeze(-1).unsqueeze(-1).expand(-1, -1, -1, hidden_states.size(-1))
+ # (bsz, num_choices, 1, hidden_size)
+ multiple_choice_h = hidden_states.gather(2, mc_token_ids).squeeze(2)
+ # (bsz, num_choices, hidden_size)
+ multiple_choice_h = self.dropout(multiple_choice_h.transpose(1, 2)).transpose(1, 2)
+ multiple_choice_logits = self.linear(multiple_choice_h).squeeze(-1)
+ # (bsz, num_choices)
+ return multiple_choice_logits
+
+
+class OpenAIGPTPreTrainedModel(nn.Module):
+ """ An abstract class to handle weights initialization and
+ a simple interface for dowloading and loading pretrained models.
+ """
+
+ def __init__(self, config, *inputs, **kwargs):
+ super(OpenAIGPTPreTrainedModel, self).__init__()
+ if not isinstance(config, OpenAIGPTConfig):
+ raise ValueError(
+ "Parameter config in `{}(config)` should be an instance of class `OpenAIGPTConfig`. "
+ "To create a model from a pretrained model use "
+ "`model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`".format(
+ self.__class__.__name__, self.__class__.__name__
+ )
+ )
+ self.config = config
+
+ def init_weights(self, module):
+ """ Initialize the weights.
+ """
+ if isinstance(module, (nn.Linear, nn.Embedding)):
+ # Slightly different from the TF version which uses truncated_normal for initialization
+ # cf https://github.com/pytorch/pytorch/pull/5617
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+ elif isinstance(module, LayerNorm):
+ module.bias.data.zero_()
+ module.weight.data.fill_(1.0)
+ if isinstance(module, nn.Linear) and module.bias is not None:
+ module.bias.data.zero_()
+
+ def set_num_special_tokens(self, num_special_tokens):
+ pass
+
+ @classmethod
+ def from_pretrained(
+ cls, pretrained_model_name_or_path, num_special_tokens=None, state_dict=None, cache_dir=None, from_tf=False, *inputs, **kwargs
+ ):
+ """
+ Instantiate a OpenAIGPTPreTrainedModel from a pre-trained model file or a pytorch state dict.
+ Download and cache the pre-trained model file if needed.
+
+ Params:
+ pretrained_model_name_or_path: either:
+ - a str with the name of a pre-trained model to load selected in the list of:
+ . `openai-gpt`
+ - a path or url to a pretrained model archive containing:
+ . `openai_gpt_config.json` a configuration file for the model
+ . `pytorch_model.bin` a PyTorch dump of a OpenAIGPTModel instance
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . a series of NumPy files containing OpenAI TensorFlow trained weights
+ from_tf: should we load the weights from a locally saved TensorFlow checkpoint
+ cache_dir: an optional path to a folder in which the pre-trained models will be cached.
+ state_dict: an optional state dictionnary (collections.OrderedDict object) to use instead of pre-trained models
+ *inputs, **kwargs: additional input for the specific Bert class
+ (ex: num_labels for BertForSequenceClassification)
+ """
+ if pretrained_model_name_or_path in PRETRAINED_MODEL_ARCHIVE_MAP:
+ archive_file = PRETRAINED_MODEL_ARCHIVE_MAP[pretrained_model_name_or_path]
+ config_file = PRETRAINED_CONFIG_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)
+ config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
+ # redirect to the cache, if necessary
+ try:
+ resolved_archive_file = cached_path(archive_file, cache_dir=cache_dir)
+ resolved_config_file = cached_path(config_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} and {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path, ", ".join(PRETRAINED_MODEL_ARCHIVE_MAP.keys()), pretrained_model_name_or_path,
+ archive_file, config_file
+ )
+ )
+ return None
+ if resolved_archive_file == archive_file and resolved_config_file == config_file:
+ logger.info("loading weights file {}".format(archive_file))
+ logger.info("loading configuration file {}".format(config_file))
+ else:
+ logger.info("loading weights file {} from cache at {}".format(
+ archive_file, resolved_archive_file))
+ logger.info("loading configuration file {} from cache at {}".format(
+ config_file, resolved_config_file))
+ # Load config
+ config = OpenAIGPTConfig.from_json_file(resolved_config_file)
+ logger.info("Model config {}".format(config))
+ # Instantiate model.
+ model = cls(config, *inputs, **kwargs)
+ if state_dict is None and not from_tf:
+ state_dict = torch.load(resolved_archive_file, map_location='cpu')
+ if from_tf:
+ # Directly load from a TensorFlow checkpoint (stored as NumPy array)
+ return load_tf_weights_in_openai_gpt(model, resolved_archive_file)
+
+ old_keys = []
+ new_keys = []
+ for key in state_dict.keys():
+ new_key = None
+ if key.endswith(".g"):
+ new_key = key[:-2] + ".weight"
+ elif key.endswith(".b"):
+ new_key = key[:-2] + ".bias"
+ elif key.endswith(".w"):
+ new_key = key[:-2] + ".weight"
+ if new_key:
+ old_keys.append(key)
+ new_keys.append(new_key)
+ for old_key, new_key in zip(old_keys, new_keys):
+ state_dict[new_key] = state_dict.pop(old_key)
+
+ missing_keys = []
+ unexpected_keys = []
+ error_msgs = []
+ # copy state_dict so _load_from_state_dict can modify it
+ metadata = getattr(state_dict, "_metadata", None)
+ state_dict = state_dict.copy()
+ if metadata is not None:
+ state_dict._metadata = metadata
+
+ def load(module, prefix=""):
+ local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+ module._load_from_state_dict(
+ state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs
+ )
+ for name, child in module._modules.items():
+ if child is not None:
+ load(child, prefix + name + ".")
+
+ start_model = model
+ if hasattr(model, "transformer") and all(not s.startswith('transformer.') for s in state_dict.keys()):
+ start_model = model.transformer
+ load(start_model, prefix="")
+
+ if len(missing_keys) > 0:
+ logger.info(
+ "Weights of {} not initialized from pretrained model: {}".format(model.__class__.__name__, missing_keys)
+ )
+ if len(unexpected_keys) > 0:
+ logger.info(
+ "Weights from pretrained model not used in {}: {}".format(model.__class__.__name__, unexpected_keys)
+ )
+ if len(error_msgs) > 0:
+ raise RuntimeError(
+ "Error(s) in loading state_dict for {}:\n\t{}".format(model.__class__.__name__, "\n\t".join(error_msgs))
+ )
+
+ # Add additional embeddings for special tokens if needed
+ # This step also make sure we are still sharing the output and input embeddings after loading weights
+ model.set_num_special_tokens(num_special_tokens if num_special_tokens is not None else config.n_special)
+ return model
+
+
+class OpenAIGPTModel(OpenAIGPTPreTrainedModel):
+ """OpenAI GPT model ("Improving Language Understanding by Generative Pre-Training").
+
+ OpenAI GPT use a single embedding matrix to store the word and special embeddings.
+ Special tokens embeddings are additional tokens that are not pre-trained: [SEP], [CLS]...
+ Special tokens need to be trained during the fine-tuning if you use them.
+ The number of special embeddings can be controled using the `set_num_special_tokens(num_special_tokens)` function.
+
+ The embeddings are ordered as follow in the token embeddings matrice:
+ [0, ----------------------
+ ... -> word embeddings
+ config.vocab_size - 1, ______________________
+ config.vocab_size,
+ ... -> special embeddings
+ config.vocab_size + config.n_special - 1] ______________________
+
+ where total_tokens_embeddings can be obtained as config.total_tokens_embeddings and is:
+ total_tokens_embeddings = config.vocab_size + config.n_special
+ You should use the associate indices to index the embeddings.
+
+ Params:
+ config: a OpenAIGPTConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length]
+ were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, total_tokens_embeddings[
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+
+ Outputs:
+ `hidden_states`: the encoded-hidden-states at the top of the model
+ as a torch.FloatTensor of size [batch_size, sequence_length, hidden_size]
+ (or more generally [d_1, ..., d_n, hidden_size] were d_1 ... d_n are the dimension of input_ids)
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+
+ config = modeling_openai.OpenAIGPTConfig()
+
+ model = modeling_openai.OpenAIGPTModel(config)
+ hidden_states = model(input_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(OpenAIGPTModel, self).__init__(config)
+ num_tokens = config.vocab_size + config.n_special
+ self.tokens_embed = nn.Embedding(num_tokens, config.n_embd)
+ self.positions_embed = nn.Embedding(config.n_positions, config.n_embd)
+ self.drop = nn.Dropout(config.embd_pdrop)
+ block = Block(config.n_ctx, config, scale=True)
+ self.h = nn.ModuleList([copy.deepcopy(block) for _ in range(config.n_layer)])
+
+ self.apply(self.init_weights)
+ # nn.init.normal_(self.embed.weight, std=0.02)
+
+ def set_num_special_tokens(self, num_special_tokens):
+ " Update input embeddings with new embedding matrice if needed "
+ if self.config.n_special == num_special_tokens:
+ return
+ # Update config
+ self.config.n_special = num_special_tokens
+ # Build new embeddings and initialize all new embeddings (in particular the special tokens)
+ old_embed = self.tokens_embed
+ self.tokens_embed = nn.Embedding(self.config.total_tokens_embeddings, self.config.n_embd)
+ self.tokens_embed.to(old_embed.weight.device)
+ self.init_weights(self.tokens_embed)
+ # Copy word embeddings from the previous weights
+ self.tokens_embed.weight.data[:self.config.vocab_size, :] = old_embed.weight.data[:self.config.vocab_size, :]
+
+ def forward(self, input_ids, position_ids=None, token_type_ids=None):
+ if position_ids is None:
+ # This was used when we had a single embedding matrice from position and token embeddings
+ # start = self.config.vocab_size + self.config.n_special
+ # end = start + input_ids.size(-1)
+ # position_ids = torch.arange(start, end, dtype=torch.long, device=input_ids.device)
+ position_ids = torch.arange(input_ids.size(-1), dtype=torch.long, device=input_ids.device)
+ position_ids = position_ids.unsqueeze(0).expand_as(input_ids)
+
+ input_shape = input_ids.size()
+ input_ids = input_ids.view(-1, input_ids.size(-1))
+ position_ids = position_ids.view(-1, position_ids.size(-1))
+
+ inputs_embeds = self.tokens_embed(input_ids)
+ position_embeds = self.positions_embed(position_ids)
+ if token_type_ids is not None:
+ token_type_ids = token_type_ids.view(-1, token_type_ids.size(-1))
+ token_type_embeds = self.tokens_embed(token_type_ids)
+ else:
+ token_type_embeds = 0
+ # Add the position information to the input embeddings
+ # h = e.sum(dim=2)
+ hidden_states = inputs_embeds + position_embeds + token_type_embeds
+ for block in self.h:
+ hidden_states = block(hidden_states)
+ output_shape = input_shape + (hidden_states.size(-1),)
+ return hidden_states.view(*output_shape)
+
+
+class OpenAIGPTLMHeadModel(OpenAIGPTPreTrainedModel):
+ """OpenAI GPT model with a Language Modeling head ("Improving Language Understanding by Generative Pre-Training").
+
+ OpenAI GPT use a single embedding matrix to store the word and special embeddings.
+ Special tokens embeddings are additional tokens that are not pre-trained: [SEP], [CLS]...
+ Special tokens need to be trained during the fine-tuning if you use them.
+ The number of special embeddings can be controled using the `set_num_special_tokens(num_special_tokens)` function.
+
+ The embeddings are ordered as follow in the token embeddings matrice:
+ [0, ----------------------
+ ... -> word embeddings
+ config.vocab_size - 1, ______________________
+ config.vocab_size,
+ ... -> special embeddings
+ config.vocab_size + config.n_special - 1] ______________________
+
+ where total_tokens_embeddings can be obtained as config.total_tokens_embeddings and is:
+ total_tokens_embeddings = config.vocab_size + config.n_special
+ You should use the associate indices to index the embeddings.
+
+ Params:
+ config: a OpenAIGPTConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length] (or more generally [d_1, ..., d_n, sequence_length]
+ were d_1 ... d_n are arbitrary dimensions) with the word BPE token indices selected in the range [0, total_tokens_embeddings[
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+ `lm_labels`: optional language modeling labels: torch.LongTensor of shape [batch_size, sequence_length]
+ with indices selected in [-1, 0, ..., vocab_size]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., vocab_size]
+
+ Outputs:
+ if `lm_labels` is not `None`:
+ Outputs the language modeling loss.
+ else:
+ `lm_logits`: the language modeling logits as a torch.FloatTensor of size [batch_size, sequence_length, total_tokens_embeddings]
+ (or more generally [d_1, ..., d_n, total_tokens_embeddings] were d_1 ... d_n are the dimension of input_ids)
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+
+ config = modeling_openai.OpenAIGPTConfig()
+
+ model = modeling_openai.OpenAIGPTLMHeadModel(config)
+ lm_logits = model(input_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(OpenAIGPTLMHeadModel, self).__init__(config)
+ self.transformer = OpenAIGPTModel(config)
+ self.lm_head = OpenAIGPTLMHead(self.transformer.tokens_embed.weight, config)
+ self.apply(self.init_weights)
+
+ def set_num_special_tokens(self, num_special_tokens):
+ """ Update input and output embeddings with new embedding matrice
+ Make sure we are sharing the embeddings
+ """
+ self.transformer.set_num_special_tokens(num_special_tokens)
+ self.lm_head.set_embeddings_weights(self.transformer.tokens_embed.weight)
+
+ def forward(self, input_ids, position_ids=None, token_type_ids=None, lm_labels=None):
+ hidden_states = self.transformer(input_ids, position_ids, token_type_ids)
+ lm_logits = self.lm_head(hidden_states)
+ if lm_labels is not None:
+ # Shift so that tokens < n predict n
+ shift_logits = lm_logits[..., :-1, :].contiguous()
+ shift_labels = lm_labels[..., 1:].contiguous()
+ # Flatten the tokens
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)),
+ shift_labels.view(-1))
+ return loss
+ return lm_logits
+
+
+class OpenAIGPTDoubleHeadsModel(OpenAIGPTPreTrainedModel):
+ """OpenAI GPT model with a Language Modeling and a Multiple Choice head ("Improving Language Understanding by Generative Pre-Training").
+
+ OpenAI GPT use a single embedding matrix to store the word and special embeddings.
+ Special tokens embeddings are additional tokens that are not pre-trained: [SEP], [CLS]...
+ Special tokens need to be trained during the fine-tuning if you use them.
+ The number of special embeddings can be controled using the `set_num_special_tokens(num_special_tokens)` function.
+
+ The embeddings are ordered as follow in the token embeddings matrice:
+ [0, ----------------------
+ ... -> word embeddings
+ config.vocab_size - 1, ______________________
+ config.vocab_size,
+ ... -> special embeddings
+ config.vocab_size + config.n_special - 1] ______________________
+
+ where total_tokens_embeddings can be obtained as config.total_tokens_embeddings and is:
+ total_tokens_embeddings = config.vocab_size + config.n_special
+ You should use the associate indices to index the embeddings.
+
+ Params:
+ config: a OpenAIGPTConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, num_choices, sequence_length] with the BPE token
+ indices selected in the range [0, total_tokens_embeddings[
+ `mc_token_ids`: a torch.LongTensor of shape [batch_size, num_choices] with the index of the token from
+ which we should take the hidden state to feed the multiple choice classifier (usually last token of the sequence)
+ `position_ids`: an optional torch.LongTensor with the same shape as input_ids
+ with the position indices (selected in the range [0, config.n_positions - 1[.
+ `token_type_ids`: an optional torch.LongTensor with the same shape as input_ids
+ You can use it to add a third type of embedding to each input token in the sequence
+ (the previous two being the word and position embeddings).
+ The input, position and token_type embeddings are summed inside the Transformer before the first
+ self-attention block.
+ `lm_labels`: optional language modeling labels: torch.LongTensor of shape [batch_size, num_choices, sequence_length]
+ with indices selected in [-1, 0, ..., total_tokens_embeddings]. All labels set to -1 are ignored (masked), the loss
+ is only computed for the labels set in [0, ..., total_tokens_embeddings]
+ `multiple_choice_labels`: optional multiple choice labels: torch.LongTensor of shape [batch_size]
+ with indices selected in [0, ..., num_choices].
+
+ Outputs:
+ if `lm_labels` and `multiple_choice_labels` are not `None`:
+ Outputs a tuple of losses with the language modeling loss and the multiple choice loss.
+ else: a tuple with
+ `lm_logits`: the language modeling logits as a torch.FloatTensor of size [batch_size, num_choices, sequence_length, total_tokens_embeddings]
+ `multiple_choice_logits`: the multiple choice logits as a torch.FloatTensor of size [batch_size, num_choices]
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[[31, 51, 99], [15, 5, 0]]]) # (bsz, number of choice, seq length)
+ mc_token_ids = torch.LongTensor([[2], [1]]) # (bsz, number of choice)
+
+ config = modeling_openai.OpenAIGPTConfig()
+
+ model = modeling_openai.OpenAIGPTLMHeadModel(config)
+ lm_logits, multiple_choice_logits = model(input_ids, mc_token_ids)
+ ```
+ """
+
+ def __init__(self, config):
+ super(OpenAIGPTDoubleHeadsModel, self).__init__(config)
+ self.transformer = OpenAIGPTModel(config)
+ self.lm_head = OpenAIGPTLMHead(self.transformer.tokens_embed.weight, config)
+ self.multiple_choice_head = OpenAIGPTMultipleChoiceHead(config)
+ self.apply(self.init_weights)
+
+ def set_num_special_tokens(self, num_special_tokens):
+ """ Update input and output embeddings with new embedding matrice
+ Make sure we are sharing the embeddings
+ """
+ self.transformer.set_num_special_tokens(num_special_tokens)
+ self.lm_head.set_embeddings_weights(self.transformer.tokens_embed.weight)
+
+ def forward(self, input_ids, mc_token_ids, lm_labels=None, mc_labels=None, token_type_ids=None, position_ids=None):
+ hidden_states = self.transformer(input_ids, position_ids, token_type_ids)
+ lm_logits = self.lm_head(hidden_states)
+ mc_logits = self.multiple_choice_head(hidden_states, mc_token_ids)
+ losses = []
+ if lm_labels is not None:
+ shift_logits = lm_logits[..., :-1, :].contiguous()
+ shift_labels = lm_labels[..., 1:].contiguous()
+ loss_fct = CrossEntropyLoss(ignore_index=-1)
+ losses.append(loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1)))
+ if mc_labels is not None:
+ loss_fct = CrossEntropyLoss()
+ losses.append(loss_fct(mc_logits.view(-1, mc_logits.size(-1)), mc_labels.view(-1)))
+ if losses:
+ return losses
+ return lm_logits, mc_logits
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl.py
new file mode 100644
index 0000000000000000000000000000000000000000..e8fffc5b60894b5a41f757a7c3a3ced3f21be162
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl.py
@@ -0,0 +1,1387 @@
+# coding=utf-8
+# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" PyTorch Transformer XL model.
+ Adapted from https://github.com/kimiyoung/transformer-xl.
+ In particular https://github.com/kimiyoung/transformer-xl/blob/master/pytorch/mem_transformer.py
+"""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import copy
+import json
+import math
+import logging
+import tarfile
+import tempfile
+import shutil
+import collections
+import sys
+from io import open
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn import CrossEntropyLoss
+from torch.nn.parameter import Parameter
+
+from .modeling import BertLayerNorm as LayerNorm
+from .modeling_transfo_xl_utilities import ProjectedAdaptiveLogSoftmax, sample_logits
+from .file_utils import cached_path, CONFIG_NAME, WEIGHTS_NAME
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_MODEL_ARCHIVE_MAP = {
+ 'transfo-xl-wt103': "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-pytorch_model.bin",
+}
+PRETRAINED_CONFIG_ARCHIVE_MAP = {
+ 'transfo-xl-wt103': "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-config.json",
+}
+
+TF_WEIGHTS_NAME = 'model.ckpt'
+
+def build_tf_to_pytorch_map(model, config):
+ """ A map of modules from TF to PyTorch.
+ This time I use a map to keep the PyTorch model as identical to the original PyTorch model as possible.
+ """
+ tf_to_pt_map = {}
+
+ if hasattr(model, 'transformer'):
+ # We are loading in a TransfoXLLMHeadModel => we will load also the Adaptive Softmax
+ tf_to_pt_map.update({
+ "transformer/adaptive_softmax/cutoff_0/cluster_W": model.crit.cluster_weight,
+ "transformer/adaptive_softmax/cutoff_0/cluster_b": model.crit.cluster_bias})
+ for i, (out_l, proj_l, tie_proj) in enumerate(zip(
+ model.crit.out_layers,
+ model.crit.out_projs,
+ config.tie_projs)):
+ layer_str = "transformer/adaptive_softmax/cutoff_%d/" % i
+ if config.tie_weight:
+ tf_to_pt_map.update({
+ layer_str + 'b': out_l.bias})
+ else:
+ raise NotImplementedError
+ # I don't think this is implemented in the TF code
+ tf_to_pt_map.update({
+ layer_str + 'lookup_table': out_l.weight,
+ layer_str + 'b': out_l.bias})
+ if not tie_proj:
+ tf_to_pt_map.update({
+ layer_str + 'proj': proj_l
+ })
+ # Now load the rest of the transformer
+ model = model.transformer
+
+ # Embeddings
+ for i, (embed_l, proj_l) in enumerate(zip(model.word_emb.emb_layers, model.word_emb.emb_projs)):
+ layer_str = "transformer/adaptive_embed/cutoff_%d/" % i
+ tf_to_pt_map.update({
+ layer_str + 'lookup_table': embed_l.weight,
+ layer_str + 'proj_W': proj_l
+ })
+
+ # Transformer blocks
+ for i, b in enumerate(model.layers):
+ layer_str = "transformer/layer_%d/" % i
+ tf_to_pt_map.update({
+ layer_str + "rel_attn/LayerNorm/gamma": b.dec_attn.layer_norm.weight,
+ layer_str + "rel_attn/LayerNorm/beta": b.dec_attn.layer_norm.bias,
+ layer_str + "rel_attn/o/kernel": b.dec_attn.o_net.weight,
+ layer_str + "rel_attn/qkv/kernel": b.dec_attn.qkv_net.weight,
+ layer_str + "rel_attn/r/kernel": b.dec_attn.r_net.weight,
+ layer_str + "ff/LayerNorm/gamma": b.pos_ff.layer_norm.weight,
+ layer_str + "ff/LayerNorm/beta": b.pos_ff.layer_norm.bias,
+ layer_str + "ff/layer_1/kernel": b.pos_ff.CoreNet[0].weight,
+ layer_str + "ff/layer_1/bias": b.pos_ff.CoreNet[0].bias,
+ layer_str + "ff/layer_2/kernel": b.pos_ff.CoreNet[3].weight,
+ layer_str + "ff/layer_2/bias": b.pos_ff.CoreNet[3].bias,
+ })
+
+ # Relative positioning biases
+ if config.untie_r:
+ r_r_list = []
+ r_w_list = []
+ for b in model.layers:
+ r_r_list.append(b.dec_attn.r_r_bias)
+ r_w_list.append(b.dec_attn.r_w_bias)
+ else:
+ r_r_list = [model.r_r_bias]
+ r_w_list = [model.r_w_bias]
+ tf_to_pt_map.update({
+ 'transformer/r_r_bias': r_r_list,
+ 'transformer/r_w_bias': r_w_list})
+ return tf_to_pt_map
+
+def load_tf_weights_in_transfo_xl(model, config, tf_path):
+ """ Load tf checkpoints in a pytorch model
+ """
+ try:
+ import numpy as np
+ import tensorflow as tf
+ except ImportError:
+ print("Loading a TensorFlow models in PyTorch, requires TensorFlow to be installed. Please see "
+ "https://www.tensorflow.org/install/ for installation instructions.")
+ raise
+ # Build TF to PyTorch weights loading map
+ tf_to_pt_map = build_tf_to_pytorch_map(model, config)
+
+ # Load weights from TF model
+ init_vars = tf.train.list_variables(tf_path)
+ tf_weights = {}
+ for name, shape in init_vars:
+ print("Loading TF weight {} with shape {}".format(name, shape))
+ array = tf.train.load_variable(tf_path, name)
+ tf_weights[name] = array
+
+ for name, pointer in tf_to_pt_map.items():
+ assert name in tf_weights
+ array = tf_weights[name]
+ # adam_v and adam_m are variables used in AdamWeightDecayOptimizer to calculated m and v
+ # which are not required for using pretrained model
+ if 'kernel' in name or 'proj' in name:
+ array = np.transpose(array)
+ if ('r_r_bias' in name or 'r_w_bias' in name) and len(pointer) > 1:
+ # Here we will split the TF weigths
+ assert len(pointer) == array.shape[0]
+ for i, p_i in enumerate(pointer):
+ arr_i = array[i, ...]
+ try:
+ assert p_i.shape == arr_i.shape
+ except AssertionError as e:
+ e.args += (p_i.shape, arr_i.shape)
+ raise
+ print("Initialize PyTorch weight {} for layer {}".format(name, i))
+ p_i.data = torch.from_numpy(arr_i)
+ else:
+ try:
+ assert pointer.shape == array.shape
+ except AssertionError as e:
+ e.args += (pointer.shape, array.shape)
+ raise
+ print("Initialize PyTorch weight {}".format(name))
+ pointer.data = torch.from_numpy(array)
+ tf_weights.pop(name, None)
+ tf_weights.pop(name + '/Adam', None)
+ tf_weights.pop(name + '/Adam_1', None)
+
+ print("Weights not copied to PyTorch model: {}".format(', '.join(tf_weights.keys())))
+ return model
+
+
+class TransfoXLConfig(object):
+ """Configuration class to store the configuration of a `TransfoXLModel`.
+ """
+ def __init__(self,
+ vocab_size_or_config_json_file=267735,
+ cutoffs=[20000, 40000, 200000],
+ d_model=1024,
+ d_embed=1024,
+ n_head=16,
+ d_head=64,
+ d_inner=4096,
+ div_val=4,
+ pre_lnorm=False,
+ n_layer=18,
+ tgt_len=128,
+ ext_len=0,
+ mem_len=1600,
+ clamp_len=1000,
+ same_length=True,
+ proj_share_all_but_first=True,
+ attn_type=0,
+ sample_softmax=-1,
+ adaptive=True,
+ tie_weight=True,
+ dropout=0.1,
+ dropatt=0.0,
+ untie_r=True,
+ init="normal",
+ init_range=0.01,
+ proj_init_std=0.01,
+ init_std=0.02):
+ """Constructs TransfoXLConfig.
+
+ Args:
+ vocab_size_or_config_json_file: Vocabulary size of `inputs_ids` in `TransfoXLModel` or a configuration json file.
+ cutoffs: cutoffs for the adaptive softmax
+ d_model: Dimensionality of the model's hidden states.
+ d_embed: Dimensionality of the embeddings
+ d_head: Dimensionality of the model's heads.
+ div_val: divident value for adapative input and softmax
+ pre_lnorm: apply LayerNorm to the input instead of the output
+ d_inner: Inner dimension in FF
+ n_layer: Number of hidden layers in the Transformer encoder.
+ n_head: Number of attention heads for each attention layer in
+ the Transformer encoder.
+ tgt_len: number of tokens to predict
+ ext_len: length of the extended context
+ mem_len: length of the retained previous heads
+ same_length: use the same attn length for all tokens
+ proj_share_all_but_first: True to share all but first projs, False not to share.
+ attn_type: attention type. 0 for Transformer-XL, 1 for Shaw et al, 2 for Vaswani et al, 3 for Al Rfou et al.
+ clamp_len: use the same pos embeddings after clamp_len
+ sample_softmax: number of samples in sampled softmax
+ adaptive: use adaptive softmax
+ tie_weight: tie the word embedding and softmax weights
+ dropout: The dropout probabilitiy for all fully connected
+ layers in the embeddings, encoder, and pooler.
+ dropatt: The dropout ratio for the attention probabilities.
+ untie_r: untie relative position biases
+ embd_pdrop: The dropout ratio for the embeddings.
+ init: parameter initializer to use
+ init_range: parameters initialized by U(-init_range, init_range).
+ proj_init_std: parameters initialized by N(0, init_std)
+ init_std: parameters initialized by N(0, init_std)
+ """
+ if isinstance(vocab_size_or_config_json_file, str) or (sys.version_info[0] == 2
+ and isinstance(vocab_size_or_config_json_file, unicode)):
+ with open(vocab_size_or_config_json_file, "r", encoding='utf-8') as reader:
+ json_config = json.loads(reader.read())
+ for key, value in json_config.items():
+ self.__dict__[key] = value
+ elif isinstance(vocab_size_or_config_json_file, int):
+ self.n_token = vocab_size_or_config_json_file
+ self.cutoffs = []
+ self.cutoffs.extend(cutoffs)
+ self.tie_weight = tie_weight
+ if proj_share_all_but_first:
+ self.tie_projs = [False] + [True] * len(self.cutoffs)
+ else:
+ self.tie_projs = [False] + [False] * len(self.cutoffs)
+ self.d_model = d_model
+ self.d_embed = d_embed
+ self.d_head = d_head
+ self.d_inner = d_inner
+ self.div_val = div_val
+ self.pre_lnorm = pre_lnorm
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.tgt_len = tgt_len
+ self.ext_len = ext_len
+ self.mem_len = mem_len
+ self.same_length = same_length
+ self.attn_type = attn_type
+ self.clamp_len = clamp_len
+ self.sample_softmax = sample_softmax
+ self.adaptive = adaptive
+ self.dropout = dropout
+ self.dropatt = dropatt
+ self.untie_r = untie_r
+ self.init = init
+ self.init_range = init_range
+ self.proj_init_std = proj_init_std
+ self.init_std = init_std
+ else:
+ raise ValueError("First argument must be either a vocabulary size (int)"
+ "or the path to a pretrained model config file (str)")
+
+ @classmethod
+ def from_dict(cls, json_object):
+ """Constructs a `TransfoXLConfig` from a Python dictionary of parameters."""
+ config = TransfoXLConfig(vocab_size_or_config_json_file=-1)
+ for key, value in json_object.items():
+ config.__dict__[key] = value
+ return config
+
+ @classmethod
+ def from_json_file(cls, json_file):
+ """Constructs a `TransfoXLConfig` from a json file of parameters."""
+ with open(json_file, "r", encoding='utf-8') as reader:
+ text = reader.read()
+ return cls.from_dict(json.loads(text))
+
+ def __repr__(self):
+ return str(self.to_json_string())
+
+ def to_dict(self):
+ """Serializes this instance to a Python dictionary."""
+ output = copy.deepcopy(self.__dict__)
+ return output
+
+ def to_json_string(self):
+ """Serializes this instance to a JSON string."""
+ return json.dumps(self.to_dict(), indent=2, sort_keys=True) + "\n"
+
+ def to_json_file(self, json_file_path):
+ """ Save this instance to a json file."""
+ with open(json_file_path, "w", encoding='utf-8') as writer:
+ writer.write(self.to_json_string())
+
+
+class PositionalEmbedding(nn.Module):
+ def __init__(self, demb):
+ super(PositionalEmbedding, self).__init__()
+
+ self.demb = demb
+
+ inv_freq = 1 / (10000 ** (torch.arange(0.0, demb, 2.0) / demb))
+ self.register_buffer('inv_freq', inv_freq)
+
+ def forward(self, pos_seq, bsz=None):
+ sinusoid_inp = torch.ger(pos_seq, self.inv_freq)
+ pos_emb = torch.cat([sinusoid_inp.sin(), sinusoid_inp.cos()], dim=-1)
+
+ if bsz is not None:
+ return pos_emb[:,None,:].expand(-1, bsz, -1)
+ else:
+ return pos_emb[:,None,:]
+
+
+class PositionwiseFF(nn.Module):
+ def __init__(self, d_model, d_inner, dropout, pre_lnorm=False):
+ super(PositionwiseFF, self).__init__()
+
+ self.d_model = d_model
+ self.d_inner = d_inner
+ self.dropout = dropout
+
+ self.CoreNet = nn.Sequential(
+ nn.Linear(d_model, d_inner), nn.ReLU(inplace=True),
+ nn.Dropout(dropout),
+ nn.Linear(d_inner, d_model),
+ nn.Dropout(dropout),
+ )
+
+ self.layer_norm = LayerNorm(d_model)
+
+ self.pre_lnorm = pre_lnorm
+
+ def forward(self, inp):
+ if self.pre_lnorm:
+ ##### layer normalization + positionwise feed-forward
+ core_out = self.CoreNet(self.layer_norm(inp))
+
+ ##### residual connection
+ output = core_out + inp
+ else:
+ ##### positionwise feed-forward
+ core_out = self.CoreNet(inp)
+
+ ##### residual connection + layer normalization
+ output = self.layer_norm(inp + core_out)
+
+ return output
+
+class MultiHeadAttn(nn.Module):
+ def __init__(self, n_head, d_model, d_head, dropout, dropatt=0,
+ pre_lnorm=False, r_r_bias=None, r_w_bias=None):
+ super(MultiHeadAttn, self).__init__()
+
+ self.n_head = n_head
+ self.d_model = d_model
+ self.d_head = d_head
+ self.dropout = dropout
+
+ self.q_net = nn.Linear(d_model, n_head * d_head, bias=False)
+ self.kv_net = nn.Linear(d_model, 2 * n_head * d_head, bias=False)
+
+ self.drop = nn.Dropout(dropout)
+ self.dropatt = nn.Dropout(dropatt)
+ self.o_net = nn.Linear(n_head * d_head, d_model, bias=False)
+
+ self.layer_norm = LayerNorm(d_model)
+
+ self.scale = 1 / (d_head ** 0.5)
+
+ self.pre_lnorm = pre_lnorm
+
+ if r_r_bias is None or r_w_bias is None: # Biases are not shared
+ self.r_r_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+ self.r_w_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+ else:
+ self.r_r_bias = r_r_bias
+ self.r_w_bias = r_w_bias
+
+ def forward(self, h, attn_mask=None, mems=None):
+ ##### multihead attention
+ # [hlen x bsz x n_head x d_head]
+
+ if mems is not None:
+ c = torch.cat([mems, h], 0)
+ else:
+ c = h
+
+ if self.pre_lnorm:
+ ##### layer normalization
+ c = self.layer_norm(c)
+
+ head_q = self.q_net(h)
+ head_k, head_v = torch.chunk(self.kv_net(c), 2, -1)
+
+ head_q = head_q.view(h.size(0), h.size(1), self.n_head, self.d_head)
+ head_k = head_k.view(c.size(0), c.size(1), self.n_head, self.d_head)
+ head_v = head_v.view(c.size(0), c.size(1), self.n_head, self.d_head)
+
+ # [qlen x klen x bsz x n_head]
+ attn_score = torch.einsum('ibnd,jbnd->ijbn', (head_q, head_k))
+ attn_score.mul_(self.scale)
+ if attn_mask is not None and attn_mask.any().item():
+ if attn_mask.dim() == 2:
+ attn_score.masked_fill_(attn_mask[None,:,:,None], -float('inf'))
+ elif attn_mask.dim() == 3:
+ attn_score.masked_fill_(attn_mask[:,:,:,None], -float('inf'))
+
+ # [qlen x klen x bsz x n_head]
+ attn_prob = F.softmax(attn_score, dim=1)
+ attn_prob = self.dropatt(attn_prob)
+
+ # [qlen x klen x bsz x n_head] + [klen x bsz x n_head x d_head] -> [qlen x bsz x n_head x d_head]
+ attn_vec = torch.einsum('ijbn,jbnd->ibnd', (attn_prob, head_v))
+ attn_vec = attn_vec.contiguous().view(
+ attn_vec.size(0), attn_vec.size(1), self.n_head * self.d_head)
+
+ ##### linear projection
+ attn_out = self.o_net(attn_vec)
+ attn_out = self.drop(attn_out)
+
+ if self.pre_lnorm:
+ ##### residual connection
+ output = h + attn_out
+ else:
+ ##### residual connection + layer normalization
+ output = self.layer_norm(h + attn_out)
+
+ return output
+
+class RelMultiHeadAttn(nn.Module):
+ def __init__(self, n_head, d_model, d_head, dropout, dropatt=0,
+ tgt_len=None, ext_len=None, mem_len=None, pre_lnorm=False,
+ r_r_bias=None, r_w_bias=None):
+ super(RelMultiHeadAttn, self).__init__()
+
+ self.n_head = n_head
+ self.d_model = d_model
+ self.d_head = d_head
+ self.dropout = dropout
+
+ self.qkv_net = nn.Linear(d_model, 3 * n_head * d_head, bias=False)
+
+ self.drop = nn.Dropout(dropout)
+ self.dropatt = nn.Dropout(dropatt)
+ self.o_net = nn.Linear(n_head * d_head, d_model, bias=False)
+
+ self.layer_norm = LayerNorm(d_model)
+
+ self.scale = 1 / (d_head ** 0.5)
+
+ self.pre_lnorm = pre_lnorm
+
+ if r_r_bias is None or r_w_bias is None: # Biases are not shared
+ self.r_r_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+ self.r_w_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+ else:
+ self.r_r_bias = r_r_bias
+ self.r_w_bias = r_w_bias
+
+ def _parallelogram_mask(self, h, w, left=False):
+ mask = torch.ones((h, w)).byte()
+ m = min(h, w)
+ mask[:m,:m] = torch.triu(mask[:m,:m])
+ mask[-m:,-m:] = torch.tril(mask[-m:,-m:])
+
+ if left:
+ return mask
+ else:
+ return mask.flip(0)
+
+ def _shift(self, x, qlen, klen, mask, left=False):
+ if qlen > 1:
+ zero_pad = torch.zeros((x.size(0), qlen-1, x.size(2), x.size(3)),
+ device=x.device, dtype=x.dtype)
+ else:
+ zero_pad = torch.zeros(0, device=x.device, dtype=x.dtype)
+
+ if left:
+ mask = mask.flip(1)
+ x_padded = torch.cat([zero_pad, x], dim=1).expand(qlen, -1, -1, -1)
+ else:
+ x_padded = torch.cat([x, zero_pad], dim=1).expand(qlen, -1, -1, -1)
+
+ x = x_padded.masked_select(mask[:,:,None,None]) \
+ .view(qlen, klen, x.size(2), x.size(3))
+
+ return x
+
+ def _rel_shift(self, x, zero_triu=False):
+ zero_pad_shape = (x.size(0), 1) + x.size()[2:]
+ zero_pad = torch.zeros(zero_pad_shape, device=x.device, dtype=x.dtype)
+ x_padded = torch.cat([zero_pad, x], dim=1)
+
+ x_padded_shape = (x.size(1) + 1, x.size(0)) + x.size()[2:]
+ x_padded = x_padded.view(*x_padded_shape)
+
+ x = x_padded[1:].view_as(x)
+
+ if zero_triu:
+ ones = torch.ones((x.size(0), x.size(1)))
+ x = x * torch.tril(ones, x.size(1) - x.size(0))[:,:,None,None]
+
+ return x
+
+ def forward(self, w, r, attn_mask=None, mems=None):
+ raise NotImplementedError
+
+class RelPartialLearnableMultiHeadAttn(RelMultiHeadAttn):
+ def __init__(self, *args, **kwargs):
+ super(RelPartialLearnableMultiHeadAttn, self).__init__(*args, **kwargs)
+
+ self.r_net = nn.Linear(self.d_model, self.n_head * self.d_head, bias=False)
+
+ def forward(self, w, r, attn_mask=None, mems=None):
+ qlen, rlen, bsz = w.size(0), r.size(0), w.size(1)
+
+ if mems is not None:
+ cat = torch.cat([mems, w], 0)
+ if self.pre_lnorm:
+ w_heads = self.qkv_net(self.layer_norm(cat))
+ else:
+ w_heads = self.qkv_net(cat)
+ r_head_k = self.r_net(r)
+
+ w_head_q, w_head_k, w_head_v = torch.chunk(w_heads, 3, dim=-1)
+ w_head_q = w_head_q[-qlen:]
+ else:
+ if self.pre_lnorm:
+ w_heads = self.qkv_net(self.layer_norm(w))
+ else:
+ w_heads = self.qkv_net(w)
+ r_head_k = self.r_net(r)
+
+ w_head_q, w_head_k, w_head_v = torch.chunk(w_heads, 3, dim=-1)
+
+ klen = w_head_k.size(0)
+
+ w_head_q = w_head_q.view(qlen, bsz, self.n_head, self.d_head) # qlen x bsz x n_head x d_head
+ w_head_k = w_head_k.view(klen, bsz, self.n_head, self.d_head) # qlen x bsz x n_head x d_head
+ w_head_v = w_head_v.view(klen, bsz, self.n_head, self.d_head) # qlen x bsz x n_head x d_head
+
+ r_head_k = r_head_k.view(rlen, self.n_head, self.d_head) # qlen x n_head x d_head
+
+ #### compute attention score
+ rw_head_q = w_head_q + self.r_w_bias # qlen x bsz x n_head x d_head
+ AC = torch.einsum('ibnd,jbnd->ijbn', (rw_head_q, w_head_k)) # qlen x klen x bsz x n_head
+
+ rr_head_q = w_head_q + self.r_r_bias
+ BD = torch.einsum('ibnd,jnd->ijbn', (rr_head_q, r_head_k)) # qlen x klen x bsz x n_head
+ BD = self._rel_shift(BD)
+
+ # [qlen x klen x bsz x n_head]
+ attn_score = AC + BD
+ attn_score.mul_(self.scale)
+
+ #### compute attention probability
+ if attn_mask is not None and attn_mask.any().item():
+ if attn_mask.dim() == 2:
+ attn_score = attn_score.float().masked_fill(
+ attn_mask[None,:,:,None], -1e30).type_as(attn_score)
+ elif attn_mask.dim() == 3:
+ attn_score = attn_score.float().masked_fill(
+ attn_mask[:,:,:,None], -1e30).type_as(attn_score)
+
+ # [qlen x klen x bsz x n_head]
+ attn_prob = F.softmax(attn_score, dim=1)
+ attn_prob = self.dropatt(attn_prob)
+
+ #### compute attention vector
+ attn_vec = torch.einsum('ijbn,jbnd->ibnd', (attn_prob, w_head_v))
+
+ # [qlen x bsz x n_head x d_head]
+ attn_vec = attn_vec.contiguous().view(
+ attn_vec.size(0), attn_vec.size(1), self.n_head * self.d_head)
+
+ ##### linear projection
+ attn_out = self.o_net(attn_vec)
+ attn_out = self.drop(attn_out)
+
+ if self.pre_lnorm:
+ ##### residual connection
+ output = w + attn_out
+ else:
+ ##### residual connection + layer normalization
+ output = self.layer_norm(w + attn_out)
+
+ return output
+
+class RelLearnableMultiHeadAttn(RelMultiHeadAttn):
+ def __init__(self, *args, **kwargs):
+ super(RelLearnableMultiHeadAttn, self).__init__(*args, **kwargs)
+
+ def forward(self, w, r_emb, r_w_bias, r_bias, attn_mask=None, mems=None):
+ # r_emb: [klen, n_head, d_head], used for term B
+ # r_w_bias: [n_head, d_head], used for term C
+ # r_bias: [klen, n_head], used for term D
+
+ qlen, bsz = w.size(0), w.size(1)
+
+ if mems is not None:
+ cat = torch.cat([mems, w], 0)
+ if self.pre_lnorm:
+ w_heads = self.qkv_net(self.layer_norm(cat))
+ else:
+ w_heads = self.qkv_net(cat)
+ w_head_q, w_head_k, w_head_v = torch.chunk(w_heads, 3, dim=-1)
+
+ w_head_q = w_head_q[-qlen:]
+ else:
+ if self.pre_lnorm:
+ w_heads = self.qkv_net(self.layer_norm(w))
+ else:
+ w_heads = self.qkv_net(w)
+ w_head_q, w_head_k, w_head_v = torch.chunk(w_heads, 3, dim=-1)
+
+ klen = w_head_k.size(0)
+
+ w_head_q = w_head_q.view(qlen, bsz, self.n_head, self.d_head)
+ w_head_k = w_head_k.view(klen, bsz, self.n_head, self.d_head)
+ w_head_v = w_head_v.view(klen, bsz, self.n_head, self.d_head)
+
+ if klen > r_emb.size(0):
+ r_emb_pad = r_emb[0:1].expand(klen-r_emb.size(0), -1, -1)
+ r_emb = torch.cat([r_emb_pad, r_emb], 0)
+ r_bias_pad = r_bias[0:1].expand(klen-r_bias.size(0), -1)
+ r_bias = torch.cat([r_bias_pad, r_bias], 0)
+ else:
+ r_emb = r_emb[-klen:]
+ r_bias = r_bias[-klen:]
+
+ #### compute attention score
+ rw_head_q = w_head_q + r_w_bias[None] # qlen x bsz x n_head x d_head
+
+ AC = torch.einsum('ibnd,jbnd->ijbn', (rw_head_q, w_head_k)) # qlen x klen x bsz x n_head
+ B_ = torch.einsum('ibnd,jnd->ijbn', (w_head_q, r_emb)) # qlen x klen x bsz x n_head
+ D_ = r_bias[None, :, None] # 1 x klen x 1 x n_head
+ BD = self._rel_shift(B_ + D_)
+
+ # [qlen x klen x bsz x n_head]
+ attn_score = AC + BD
+ attn_score.mul_(self.scale)
+
+ #### compute attention probability
+ if attn_mask is not None and attn_mask.any().item():
+ if attn_mask.dim() == 2:
+ attn_score.masked_fill_(attn_mask[None,:,:,None], -float('inf'))
+ elif attn_mask.dim() == 3:
+ attn_score.masked_fill_(attn_mask[:,:,:,None], -float('inf'))
+
+ # [qlen x klen x bsz x n_head]
+ attn_prob = F.softmax(attn_score, dim=1)
+ attn_prob = self.dropatt(attn_prob)
+
+ #### compute attention vector
+ attn_vec = torch.einsum('ijbn,jbnd->ibnd', (attn_prob, w_head_v))
+
+ # [qlen x bsz x n_head x d_head]
+ attn_vec = attn_vec.contiguous().view(
+ attn_vec.size(0), attn_vec.size(1), self.n_head * self.d_head)
+
+ ##### linear projection
+ attn_out = self.o_net(attn_vec)
+ attn_out = self.drop(attn_out)
+
+ if self.pre_lnorm:
+ ##### residual connection
+ output = w + attn_out
+ else:
+ ##### residual connection + layer normalization
+ output = self.layer_norm(w + attn_out)
+
+ return output
+
+class DecoderLayer(nn.Module):
+ def __init__(self, n_head, d_model, d_head, d_inner, dropout, **kwargs):
+ super(DecoderLayer, self).__init__()
+
+ self.dec_attn = MultiHeadAttn(n_head, d_model, d_head, dropout, **kwargs)
+ self.pos_ff = PositionwiseFF(d_model, d_inner, dropout,
+ pre_lnorm=kwargs.get('pre_lnorm'))
+
+ def forward(self, dec_inp, dec_attn_mask=None, mems=None):
+
+ output = self.dec_attn(dec_inp, attn_mask=dec_attn_mask,
+ mems=mems)
+ output = self.pos_ff(output)
+
+ return output
+
+class RelLearnableDecoderLayer(nn.Module):
+ def __init__(self, n_head, d_model, d_head, d_inner, dropout,
+ **kwargs):
+ super(RelLearnableDecoderLayer, self).__init__()
+
+ self.dec_attn = RelLearnableMultiHeadAttn(n_head, d_model, d_head, dropout,
+ **kwargs)
+ self.pos_ff = PositionwiseFF(d_model, d_inner, dropout,
+ pre_lnorm=kwargs.get('pre_lnorm'))
+
+ def forward(self, dec_inp, r_emb, r_w_bias, r_bias, dec_attn_mask=None, mems=None):
+
+ output = self.dec_attn(dec_inp, r_emb, r_w_bias, r_bias,
+ attn_mask=dec_attn_mask,
+ mems=mems)
+ output = self.pos_ff(output)
+
+ return output
+
+class RelPartialLearnableDecoderLayer(nn.Module):
+ def __init__(self, n_head, d_model, d_head, d_inner, dropout,
+ **kwargs):
+ super(RelPartialLearnableDecoderLayer, self).__init__()
+
+ self.dec_attn = RelPartialLearnableMultiHeadAttn(n_head, d_model,
+ d_head, dropout, **kwargs)
+ self.pos_ff = PositionwiseFF(d_model, d_inner, dropout,
+ pre_lnorm=kwargs.get('pre_lnorm'))
+
+ def forward(self, dec_inp, r, dec_attn_mask=None, mems=None):
+
+ output = self.dec_attn(dec_inp, r,
+ attn_mask=dec_attn_mask,
+ mems=mems)
+ output = self.pos_ff(output)
+
+ return output
+
+
+class AdaptiveEmbedding(nn.Module):
+ def __init__(self, n_token, d_embed, d_proj, cutoffs, div_val=1,
+ sample_softmax=False):
+ super(AdaptiveEmbedding, self).__init__()
+
+ self.n_token = n_token
+ self.d_embed = d_embed
+
+ self.cutoffs = cutoffs + [n_token]
+ self.div_val = div_val
+ self.d_proj = d_proj
+
+ self.emb_scale = d_proj ** 0.5
+
+ self.cutoff_ends = [0] + self.cutoffs
+
+ self.emb_layers = nn.ModuleList()
+ self.emb_projs = nn.ParameterList()
+ if div_val == 1:
+ self.emb_layers.append(
+ nn.Embedding(n_token, d_embed, sparse=sample_softmax>0)
+ )
+ if d_proj != d_embed:
+ self.emb_projs.append(nn.Parameter(torch.Tensor(d_proj, d_embed)))
+ else:
+ for i in range(len(self.cutoffs)):
+ l_idx, r_idx = self.cutoff_ends[i], self.cutoff_ends[i+1]
+ d_emb_i = d_embed // (div_val ** i)
+ self.emb_layers.append(nn.Embedding(r_idx-l_idx, d_emb_i))
+ self.emb_projs.append(nn.Parameter(torch.Tensor(d_proj, d_emb_i)))
+
+ def forward(self, inp):
+ if self.div_val == 1:
+ embed = self.emb_layers[0](inp)
+ if self.d_proj != self.d_embed:
+ embed = F.linear(embed, self.emb_projs[0])
+ else:
+ param = next(self.parameters())
+ inp_flat = inp.view(-1)
+ emb_flat = torch.zeros([inp_flat.size(0), self.d_proj],
+ dtype=param.dtype, device=param.device)
+ for i in range(len(self.cutoffs)):
+ l_idx, r_idx = self.cutoff_ends[i], self.cutoff_ends[i + 1]
+
+ mask_i = (inp_flat >= l_idx) & (inp_flat < r_idx)
+ indices_i = mask_i.nonzero().squeeze()
+
+ if indices_i.numel() == 0:
+ continue
+
+ inp_i = inp_flat.index_select(0, indices_i) - l_idx
+ emb_i = self.emb_layers[i](inp_i)
+ emb_i = F.linear(emb_i, self.emb_projs[i])
+
+ emb_flat.index_copy_(0, indices_i, emb_i)
+
+ embed_shape = inp.size() + (self.d_proj,)
+ embed = emb_flat.view(embed_shape)
+
+ embed.mul_(self.emb_scale)
+
+ return embed
+
+
+class TransfoXLPreTrainedModel(nn.Module):
+ """ An abstract class to handle weights initialization and
+ a simple interface for dowloading and loading pretrained models.
+ """
+ def __init__(self, config, *inputs, **kwargs):
+ super(TransfoXLPreTrainedModel, self).__init__()
+ if not isinstance(config, TransfoXLConfig):
+ raise ValueError(
+ "Parameter config in `{}(config)` should be an instance of class `TransfoXLConfig`. "
+ "To create a model from a pretrained model use "
+ "`model = {}.from_pretrained(PRETRAINED_MODEL_NAME)`".format(
+ self.__class__.__name__, self.__class__.__name__
+ ))
+ self.config = config
+
+ def init_weight(self, weight):
+ if self.config.init == 'uniform':
+ nn.init.uniform_(weight, -self.config.init_range, self.config.init_range)
+ elif self.config.init == 'normal':
+ nn.init.normal_(weight, 0.0, self.config.init_std)
+
+ def init_bias(self, bias):
+ nn.init.constant_(bias, 0.0)
+
+ def init_weights(self, m):
+ """ Initialize the weights.
+ """
+ classname = m.__class__.__name__
+ if classname.find('Linear') != -1:
+ if hasattr(m, 'weight') and m.weight is not None:
+ self.init_weight(m.weight)
+ if hasattr(m, 'bias') and m.bias is not None:
+ self.init_bias(m.bias)
+ elif classname.find('AdaptiveEmbedding') != -1:
+ if hasattr(m, 'emb_projs'):
+ for i in range(len(m.emb_projs)):
+ if m.emb_projs[i] is not None:
+ nn.init.normal_(m.emb_projs[i], 0.0, self.config.proj_init_std)
+ elif classname.find('Embedding') != -1:
+ if hasattr(m, 'weight'):
+ self.init_weight(m.weight)
+ elif classname.find('ProjectedAdaptiveLogSoftmax') != -1:
+ if hasattr(m, 'cluster_weight') and m.cluster_weight is not None:
+ self.init_weight(m.cluster_weight)
+ if hasattr(m, 'cluster_bias') and m.cluster_bias is not None:
+ self.init_bias(m.cluster_bias)
+ if hasattr(m, 'out_projs'):
+ for i in range(len(m.out_projs)):
+ if m.out_projs[i] is not None:
+ nn.init.normal_(m.out_projs[i], 0.0, self.config.proj_init_std)
+ elif classname.find('LayerNorm') != -1:
+ if hasattr(m, 'weight'):
+ nn.init.normal_(m.weight, 1.0, self.config.init_std)
+ if hasattr(m, 'bias') and m.bias is not None:
+ self.init_bias(m.bias)
+ elif classname.find('TransformerLM') != -1:
+ if hasattr(m, 'r_emb'):
+ self.init_weight(m.r_emb)
+ if hasattr(m, 'r_w_bias'):
+ self.init_weight(m.r_w_bias)
+ if hasattr(m, 'r_r_bias'):
+ self.init_weight(m.r_r_bias)
+ if hasattr(m, 'r_bias'):
+ self.init_bias(m.r_bias)
+
+ def set_num_special_tokens(self, num_special_tokens):
+ pass
+
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, state_dict=None, cache_dir=None,
+ from_tf=False, *inputs, **kwargs):
+ """
+ Instantiate a TransfoXLPreTrainedModel from a pre-trained model file or a pytorch state dict.
+ Download and cache the pre-trained model file if needed.
+
+ Params:
+ pretrained_model_name_or_path: either:
+ - a str with the name of a pre-trained model to load selected in the list of:
+ . `transfo-xl`
+ - a path or url to a pretrained model archive containing:
+ . `transfo_xl_config.json` a configuration file for the model
+ . `pytorch_model.bin` a PyTorch dump of a TransfoXLModel instance
+ - a path or url to a pretrained model archive containing:
+ . `bert_config.json` a configuration file for the model
+ . `model.chkpt` a TensorFlow checkpoint
+ from_tf: should we load the weights from a locally saved TensorFlow checkpoint
+ cache_dir: an optional path to a folder in which the pre-trained models will be cached.
+ state_dict: an optional state dictionnary (collections.OrderedDict object) to use instead of pre-trained models
+ *inputs, **kwargs: additional input for the specific Bert class
+ (ex: num_labels for BertForSequenceClassification)
+ """
+ if pretrained_model_name_or_path in PRETRAINED_MODEL_ARCHIVE_MAP:
+ archive_file = PRETRAINED_MODEL_ARCHIVE_MAP[pretrained_model_name_or_path]
+ config_file = PRETRAINED_CONFIG_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ archive_file = os.path.join(pretrained_model_name_or_path, WEIGHTS_NAME)
+ config_file = os.path.join(pretrained_model_name_or_path, CONFIG_NAME)
+ # redirect to the cache, if necessary
+ try:
+ resolved_archive_file = cached_path(archive_file, cache_dir=cache_dir)
+ resolved_config_file = cached_path(config_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} and {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_MODEL_ARCHIVE_MAP.keys()),
+ pretrained_model_name_or_path,
+ archive_file, config_file))
+ return None
+ if resolved_archive_file == archive_file and resolved_config_file == config_file:
+ logger.info("loading weights file {}".format(archive_file))
+ logger.info("loading configuration file {}".format(config_file))
+ else:
+ logger.info("loading weights file {} from cache at {}".format(
+ archive_file, resolved_archive_file))
+ logger.info("loading configuration file {} from cache at {}".format(
+ config_file, resolved_config_file))
+ # Load config
+ config = TransfoXLConfig.from_json_file(resolved_config_file)
+ logger.info("Model config {}".format(config))
+ # Instantiate model.
+ model = cls(config, *inputs, **kwargs)
+ if state_dict is None and not from_tf:
+ state_dict = torch.load(resolved_archive_file, map_location='cpu')
+ if from_tf:
+ # Directly load from a TensorFlow checkpoint
+ return load_tf_weights_in_transfo_xl(model, config, pretrained_model_name_or_path)
+
+ missing_keys = []
+ unexpected_keys = []
+ error_msgs = []
+ # copy state_dict so _load_from_state_dict can modify it
+ metadata = getattr(state_dict, '_metadata', None)
+ state_dict = state_dict.copy()
+ if metadata is not None:
+ state_dict._metadata = metadata
+
+ def load(module, prefix=''):
+ local_metadata = {} if metadata is None else metadata.get(prefix[:-1], {})
+ module._load_from_state_dict(
+ state_dict, prefix, local_metadata, True, missing_keys, unexpected_keys, error_msgs)
+ for name, child in module._modules.items():
+ if child is not None:
+ load(child, prefix + name + '.')
+
+ start_prefix = ''
+ if not hasattr(model, 'transformer') and any(s.startswith('transformer.') for s in state_dict.keys()):
+ start_prefix = 'transformer.'
+ load(model, prefix=start_prefix)
+
+ if len(missing_keys) > 0:
+ logger.info("Weights of {} not initialized from pretrained model: {}".format(
+ model.__class__.__name__, missing_keys))
+ if len(unexpected_keys) > 0:
+ logger.info("Weights from pretrained model not used in {}: {}".format(
+ model.__class__.__name__, unexpected_keys))
+ if len(error_msgs) > 0:
+ raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
+ model.__class__.__name__, "\n\t".join(error_msgs)))
+ # Make sure we are still sharing the input and output embeddings
+ if hasattr(model, 'tie_weights'):
+ model.tie_weights()
+ return model
+
+
+class TransfoXLModel(TransfoXLPreTrainedModel):
+ """Transformer XL model ("Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context").
+
+ Transformer XL use a relative positioning (with sinusiodal patterns) and adaptive softmax inputs which means that:
+ - you don't need to specify positioning embeddings indices
+ - the tokens in the vocabulary have to be sorted to decreasing frequency.
+
+ Params:
+ config: a TransfoXLConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the token indices selected in the range [0, self.config.n_token[
+ `mems`: optional memomry of hidden states from previous forward passes
+ as a list (num layers) of hidden states at the entry of each layer
+ each hidden states has shape [self.config.mem_len, bsz, self.config.d_model]
+ Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
+ Outputs:
+ A tuple of (last_hidden_state, new_mems)
+ `last_hidden_state`: the encoded-hidden-states at the top of the model
+ as a torch.FloatTensor of size [batch_size, sequence_length, self.config.d_model]
+ `new_mems`: list (num layers) of updated mem states at the entry of each layer
+ each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]
+ Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_ids_next = torch.LongTensor([[53, 21, 1], [64, 23, 100]])
+
+ config = TransfoXLConfig()
+
+ model = TransfoXLModel(config)
+ last_hidden_state, new_mems = model(input_ids)
+
+ # Another time on input_ids_next using the memory:
+ last_hidden_state, new_mems = model(input_ids_next, new_mems)
+ ```
+ """
+ def __init__(self, config):
+ super(TransfoXLModel, self).__init__(config)
+ self.n_token = config.n_token
+
+ self.d_embed = config.d_embed
+ self.d_model = config.d_model
+ self.n_head = config.n_head
+ self.d_head = config.d_head
+
+ self.word_emb = AdaptiveEmbedding(config.n_token, config.d_embed, config.d_model, config.cutoffs,
+ div_val=config.div_val)
+
+ self.drop = nn.Dropout(config.dropout)
+
+ self.n_layer = config.n_layer
+
+ self.tgt_len = config.tgt_len
+ self.mem_len = config.mem_len
+ self.ext_len = config.ext_len
+ self.max_klen = config.tgt_len + config.ext_len + config.mem_len
+
+ self.attn_type = config.attn_type
+
+ if not config.untie_r:
+ self.r_w_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+ self.r_r_bias = nn.Parameter(torch.Tensor(self.n_head, self.d_head))
+
+ self.layers = nn.ModuleList()
+ if config.attn_type == 0: # the default attention
+ for i in range(config.n_layer):
+ self.layers.append(
+ RelPartialLearnableDecoderLayer(
+ config.n_head, config.d_model, config.d_head, config.d_inner, config.dropout,
+ tgt_len=config.tgt_len, ext_len=config.ext_len, mem_len=config.mem_len,
+ dropatt=config.dropatt, pre_lnorm=config.pre_lnorm,
+ r_w_bias=None if config.untie_r else self.r_w_bias,
+ r_r_bias=None if config.untie_r else self.r_r_bias)
+ )
+ elif config.attn_type == 1: # learnable embeddings
+ for i in range(config.n_layer):
+ self.layers.append(
+ RelLearnableDecoderLayer(
+ config.n_head, config.d_model, config.d_head, config.d_inner, config.dropout,
+ tgt_len=config.tgt_len, ext_len=config.ext_len, mem_len=config.mem_len,
+ dropatt=config.dropatt, pre_lnorm=config.pre_lnorm,
+ r_w_bias=None if config.untie_r else self.r_w_bias,
+ r_r_bias=None if config.untie_r else self.r_r_bias)
+ )
+ elif config.attn_type in [2, 3]: # absolute embeddings
+ for i in range(config.n_layer):
+ self.layers.append(
+ DecoderLayer(
+ config.n_head, config.d_model, config.d_head, config.d_inner, config.dropout,
+ dropatt=config.dropatt, pre_lnorm=config.pre_lnorm,
+ r_w_bias=None if config.untie_r else self.r_w_bias,
+ r_r_bias=None if config.untie_r else self.r_r_bias)
+ )
+
+ self.same_length = config.same_length
+ self.clamp_len = config.clamp_len
+
+ if self.attn_type == 0: # default attention
+ self.pos_emb = PositionalEmbedding(self.d_model)
+ elif self.attn_type == 1: # learnable
+ self.r_emb = nn.Parameter(torch.Tensor(
+ self.n_layer, self.max_klen, self.n_head, self.d_head))
+ self.r_bias = nn.Parameter(torch.Tensor(
+ self.n_layer, self.max_klen, self.n_head))
+ elif self.attn_type == 2: # absolute standard
+ self.pos_emb = PositionalEmbedding(self.d_model)
+ elif self.attn_type == 3: # absolute deeper SA
+ self.r_emb = nn.Parameter(torch.Tensor(
+ self.n_layer, self.max_klen, self.n_head, self.d_head))
+ self.apply(self.init_weights)
+
+ def backward_compatible(self):
+ self.sample_softmax = -1
+
+
+ def reset_length(self, tgt_len, ext_len, mem_len):
+ self.tgt_len = tgt_len
+ self.mem_len = mem_len
+ self.ext_len = ext_len
+
+ def init_mems(self, data):
+ if self.mem_len > 0:
+ mems = []
+ param = next(self.parameters())
+ for i in range(self.n_layer):
+ empty = torch.zeros(self.mem_len, data.size(1), self.config.d_model,
+ dtype=param.dtype, device=param.device)
+ mems.append(empty)
+
+ return mems
+ else:
+ return None
+
+ def _update_mems(self, hids, mems, qlen, mlen):
+ # does not deal with None
+ if mems is None: return None
+
+ # mems is not None
+ assert len(hids) == len(mems), 'len(hids) != len(mems)'
+
+ # There are `mlen + qlen` steps that can be cached into mems
+ # For the next step, the last `ext_len` of the `qlen` tokens
+ # will be used as the extended context. Hence, we only cache
+ # the tokens from `mlen + qlen - self.ext_len - self.mem_len`
+ # to `mlen + qlen - self.ext_len`.
+ with torch.no_grad():
+ new_mems = []
+ end_idx = mlen + max(0, qlen - 0 - self.ext_len)
+ beg_idx = max(0, end_idx - self.mem_len)
+ for i in range(len(hids)):
+
+ cat = torch.cat([mems[i], hids[i]], dim=0)
+ new_mems.append(cat[beg_idx:end_idx].detach())
+
+ return new_mems
+
+ def _forward(self, dec_inp, mems=None):
+ qlen, bsz = dec_inp.size()
+
+ word_emb = self.word_emb(dec_inp)
+
+ mlen = mems[0].size(0) if mems is not None else 0
+ klen = mlen + qlen
+ if self.same_length:
+ all_ones = word_emb.new_ones(qlen, klen)
+ mask_len = klen - self.mem_len
+ if mask_len > 0:
+ mask_shift_len = qlen - mask_len
+ else:
+ mask_shift_len = qlen
+ dec_attn_mask = (torch.triu(all_ones, 1+mlen)
+ + torch.tril(all_ones, -mask_shift_len)).byte()[:, :, None] # -1
+ else:
+ dec_attn_mask = torch.triu(
+ word_emb.new_ones(qlen, klen), diagonal=1+mlen).byte()[:,:,None]
+
+ hids = []
+ if self.attn_type == 0: # default
+ pos_seq = torch.arange(klen-1, -1, -1.0, device=word_emb.device,
+ dtype=word_emb.dtype)
+ if self.clamp_len > 0:
+ pos_seq.clamp_(max=self.clamp_len)
+ pos_emb = self.pos_emb(pos_seq)
+
+ core_out = self.drop(word_emb)
+ pos_emb = self.drop(pos_emb)
+
+ for i, layer in enumerate(self.layers):
+ hids.append(core_out)
+ mems_i = None if mems is None else mems[i]
+ core_out = layer(core_out, pos_emb, dec_attn_mask=dec_attn_mask, mems=mems_i)
+ elif self.attn_type == 1: # learnable
+ core_out = self.drop(word_emb)
+ for i, layer in enumerate(self.layers):
+ hids.append(core_out)
+ if self.clamp_len > 0:
+ r_emb = self.r_emb[i][-self.clamp_len :]
+ r_bias = self.r_bias[i][-self.clamp_len :]
+ else:
+ r_emb, r_bias = self.r_emb[i], self.r_bias[i]
+
+ mems_i = None if mems is None else mems[i]
+ core_out = layer(core_out, r_emb, self.r_w_bias[i],
+ r_bias, dec_attn_mask=dec_attn_mask, mems=mems_i)
+ elif self.attn_type == 2: # absolute
+ pos_seq = torch.arange(klen - 1, -1, -1.0, device=word_emb.device,
+ dtype=word_emb.dtype)
+ if self.clamp_len > 0:
+ pos_seq.clamp_(max=self.clamp_len)
+ pos_emb = self.pos_emb(pos_seq)
+
+ core_out = self.drop(word_emb + pos_emb[-qlen:])
+
+ for i, layer in enumerate(self.layers):
+ hids.append(core_out)
+ mems_i = None if mems is None else mems[i]
+ if mems_i is not None and i == 0:
+ mems_i += pos_emb[:mlen]
+ core_out = layer(core_out, dec_attn_mask=dec_attn_mask,
+ mems=mems_i)
+ elif self.attn_type == 3:
+ core_out = self.drop(word_emb)
+
+ for i, layer in enumerate(self.layers):
+ hids.append(core_out)
+ mems_i = None if mems is None else mems[i]
+ if mems_i is not None and mlen > 0:
+ cur_emb = self.r_emb[i][:-qlen]
+ cur_size = cur_emb.size(0)
+ if cur_size < mlen:
+ cur_emb_pad = cur_emb[0:1].expand(mlen-cur_size, -1, -1)
+ cur_emb = torch.cat([cur_emb_pad, cur_emb], 0)
+ else:
+ cur_emb = cur_emb[-mlen:]
+ mems_i += cur_emb.view(mlen, 1, -1)
+ core_out += self.r_emb[i][-qlen:].view(qlen, 1, -1)
+
+ core_out = layer(core_out, dec_attn_mask=dec_attn_mask,
+ mems=mems_i)
+
+ core_out = self.drop(core_out)
+
+ new_mems = self._update_mems(hids, mems, mlen, qlen)
+
+ return core_out, new_mems
+
+ def forward(self, input_ids, mems=None):
+ """ Params:
+ input_ids :: [bsz, len]
+ mems :: optional mems from previous forwar passes (or init_mems)
+ list (num layers) of mem states at the entry of each layer
+ shape :: [self.config.mem_len, bsz, self.config.d_model]
+ Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
+ Returns:
+ tuple (last_hidden, new_mems) where:
+ new_mems: list (num layers) of mem states at the entry of each layer
+ shape :: [self.config.mem_len, bsz, self.config.d_model]
+ last_hidden: output of the last layer:
+ shape :: [bsz, len, self.config.d_model]
+ """
+ # the original code for Transformer-XL used shapes [len, bsz] but we want a unified interface in the library
+ # so we transpose here from shape [bsz, len] to shape [len, bsz]
+ input_ids = input_ids.transpose(0, 1).contiguous()
+
+ if mems is None:
+ mems = self.init_mems(input_ids)
+ last_hidden, new_mems = self._forward(input_ids, mems=mems)
+
+ # We transpose back here to shape [bsz, len, hidden_dim]
+ last_hidden = last_hidden.transpose(0, 1).contiguous()
+ return (last_hidden, new_mems)
+
+
+class TransfoXLLMHeadModel(TransfoXLPreTrainedModel):
+ """Transformer XL model ("Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context").
+
+ This model add an (adaptive) softmax head on top of the TransfoXLModel
+
+ Transformer XL use a relative positioning (with sinusiodal patterns) and adaptive softmax inputs which means that:
+ - you don't need to specify positioning embeddings indices
+ - the tokens in the vocabulary have to be sorted to decreasing frequency.
+
+ Call self.tie_weights() if you update/load the weights of the transformer to keep the weights tied.
+
+ Params:
+ config: a TransfoXLConfig class instance with the configuration to build a new model
+
+ Inputs:
+ `input_ids`: a torch.LongTensor of shape [batch_size, sequence_length]
+ with the token indices selected in the range [0, self.config.n_token[
+ `target`: an optional torch.LongTensor of shape [batch_size, sequence_length]
+ with the target token indices selected in the range [0, self.config.n_token[
+ `mems`: an optional memory of hidden states from previous forward passes
+ as a list (num layers) of hidden states at the entry of each layer
+ each hidden states has shape [self.config.mem_len, bsz, self.config.d_model]
+ Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
+
+ Outputs:
+ A tuple of (last_hidden_state, new_mems)
+ `softmax_output`: output of the (adaptive) softmax:
+ if target is None:
+ Negative log likelihood of shape [batch_size, sequence_length]
+ else:
+ log probabilities of tokens, shape [batch_size, sequence_length, n_tokens]
+ `new_mems`: list (num layers) of updated mem states at the entry of each layer
+ each mem state is a torch.FloatTensor of size [self.config.mem_len, batch_size, self.config.d_model]
+ Note that the first two dimensions are transposed in `mems` with regards to `input_ids` and `target`
+
+ Example usage:
+ ```python
+ # Already been converted into BPE token ids
+ input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
+ input_ids_next = torch.LongTensor([[53, 21, 1], [64, 23, 100]])
+
+ config = TransfoXLConfig()
+
+ model = TransfoXLModel(config)
+ last_hidden_state, new_mems = model(input_ids)
+
+ # Another time on input_ids_next using the memory:
+ last_hidden_state, new_mems = model(input_ids_next, mems=new_mems)
+ ```
+ """
+ def __init__(self, config):
+ super(TransfoXLLMHeadModel, self).__init__(config)
+ self.transformer = TransfoXLModel(config)
+ self.sample_softmax = config.sample_softmax
+ # use sampled softmax
+ if config.sample_softmax > 0:
+ self.out_layer = nn.Linear(config.d_model, config.n_token)
+ self.sampler = LogUniformSampler(config.n_token, config.sample_softmax)
+ # use adaptive softmax (including standard softmax)
+ else:
+ self.crit = ProjectedAdaptiveLogSoftmax(config.n_token, config.d_embed, config.d_model,
+ config.cutoffs, div_val=config.div_val)
+ self.apply(self.init_weights)
+ self.tie_weights()
+
+ def tie_weights(self):
+ """ Run this to be sure output and input (adaptive) softmax weights are tied """
+ # sampled softmax
+ if self.sample_softmax > 0:
+ if self.config.tie_weight:
+ self.out_layer.weight = self.transformer.word_emb.weight
+ # adaptive softmax (including standard softmax)
+ else:
+ if self.config.tie_weight:
+ for i in range(len(self.crit.out_layers)):
+ self.crit.out_layers[i].weight = self.transformer.word_emb.emb_layers[i].weight
+ if self.config.tie_projs:
+ for i, tie_proj in enumerate(self.config.tie_projs):
+ if tie_proj and self.config.div_val == 1 and self.config.d_model != self.config.d_embed:
+ self.crit.out_projs[i] = self.transformer.word_emb.emb_projs[0]
+ elif tie_proj and self.config.div_val != 1:
+ self.crit.out_projs[i] = self.transformer.word_emb.emb_projs[i]
+
+ def reset_length(self, tgt_len, ext_len, mem_len):
+ self.transformer.reset_length(tgt_len, ext_len, mem_len)
+
+ def init_mems(self, data):
+ return self.transformer.init_mems(data)
+
+ def forward(self, input_ids, target=None, mems=None):
+ """ Params:
+ input_ids :: [bsz, len]
+ target :: [bsz, len]
+ Returns:
+ tuple(softmax_output, new_mems) where:
+ new_mems: list (num layers) of hidden states at the entry of each layer
+ shape :: [mem_len, bsz, self.config.d_model] :: Warning: shapes are transposed here w. regards to input_ids
+ softmax_output: output of the (adaptive) softmax:
+ if target is None:
+ Negative log likelihood of shape :: [bsz, len]
+ else:
+ log probabilities of tokens, shape :: [bsz, len, n_tokens]
+ """
+ bsz = input_ids.size(0)
+ tgt_len = input_ids.size(1)
+
+ last_hidden, new_mems = self.transformer(input_ids, mems)
+
+ pred_hid = last_hidden[:, -tgt_len:]
+ if self.sample_softmax > 0 and self.training:
+ assert self.config.tie_weight
+ logit = sample_logits(self.transformer.word_emb, self.out_layer.bias, target, pred_hid, self.sampler)
+ softmax_output = -F.log_softmax(logit, -1)[:, :, 0]
+ else:
+ softmax_output = self.crit(pred_hid.view(-1, pred_hid.size(-1)), target)
+ if target is None:
+ softmax_output = softmax_output.view(bsz, tgt_len, -1)
+ else:
+ softmax_output = softmax_output.view(bsz, tgt_len)
+
+ # We transpose back
+ return (softmax_output, new_mems)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl_utilities.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl_utilities.py
new file mode 100644
index 0000000000000000000000000000000000000000..647ba7774c1522ba90c9c8804215fb8068f10a01
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/modeling_transfo_xl_utilities.py
@@ -0,0 +1,402 @@
+# coding=utf-8
+# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Utilities for PyTorch Transformer XL model.
+ Directly adapted from https://github.com/kimiyoung/transformer-xl.
+"""
+
+from collections import defaultdict
+
+import numpy as np
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+# CUDA_MAJOR = int(torch.version.cuda.split('.')[0])
+# CUDA_MINOR = int(torch.version.cuda.split('.')[1])
+
+class ProjectedAdaptiveLogSoftmax(nn.Module):
+ def __init__(self, n_token, d_embed, d_proj, cutoffs, div_val=1,
+ keep_order=False):
+ super(ProjectedAdaptiveLogSoftmax, self).__init__()
+
+ self.n_token = n_token
+ self.d_embed = d_embed
+ self.d_proj = d_proj
+
+ self.cutoffs = cutoffs + [n_token]
+ self.cutoff_ends = [0] + self.cutoffs
+ self.div_val = div_val
+
+ self.shortlist_size = self.cutoffs[0]
+ self.n_clusters = len(self.cutoffs) - 1
+ self.head_size = self.shortlist_size + self.n_clusters
+
+ if self.n_clusters > 0:
+ self.cluster_weight = nn.Parameter(torch.zeros(self.n_clusters, self.d_embed))
+ self.cluster_bias = nn.Parameter(torch.zeros(self.n_clusters))
+
+ self.out_layers = nn.ModuleList()
+ self.out_projs = nn.ParameterList()
+
+ if div_val == 1:
+ for i in range(len(self.cutoffs)):
+ if d_proj != d_embed:
+ self.out_projs.append(
+ nn.Parameter(torch.Tensor(d_proj, d_embed))
+ )
+ else:
+ self.out_projs.append(None)
+
+ self.out_layers.append(nn.Linear(d_embed, n_token))
+ else:
+ for i in range(len(self.cutoffs)):
+ l_idx, r_idx = self.cutoff_ends[i], self.cutoff_ends[i+1]
+ d_emb_i = d_embed // (div_val ** i)
+
+ self.out_projs.append(
+ nn.Parameter(torch.Tensor(d_proj, d_emb_i))
+ )
+
+ self.out_layers.append(nn.Linear(d_emb_i, r_idx-l_idx))
+
+ self.keep_order = keep_order
+
+ def _compute_logit(self, hidden, weight, bias, proj):
+ if proj is None:
+ logit = F.linear(hidden, weight, bias=bias)
+ else:
+ # if CUDA_MAJOR <= 9 and CUDA_MINOR <= 1:
+ proj_hid = F.linear(hidden, proj.t().contiguous())
+ logit = F.linear(proj_hid, weight, bias=bias)
+ # else:
+ # logit = torch.einsum('bd,de,ev->bv', (hidden, proj, weight.t()))
+ # if bias is not None:
+ # logit = logit + bias
+
+ return logit
+
+ def forward(self, hidden, target=None, keep_order=False):
+ '''
+ Params:
+ hidden :: [len*bsz x d_proj]
+ target :: [len*bsz]
+ Return:
+ if target is None:
+ out :: [len*bsz] Negative log likelihood
+ else:
+ out :: [len*bsz x n_tokens] log probabilities of tokens over the vocabulary
+ We could replace this implementation by the native PyTorch one
+ if their's had an option to set bias on all clusters in the native one.
+ here: https://github.com/pytorch/pytorch/blob/dbe6a7a9ff1a364a8706bf5df58a1ca96d2fd9da/torch/nn/modules/adaptive.py#L138
+ '''
+
+ if target is not None:
+ target = target.view(-1)
+ if hidden.size(0) != target.size(0):
+ raise RuntimeError('Input and target should have the same size '
+ 'in the batch dimension.')
+
+ if self.n_clusters == 0:
+ logit = self._compute_logit(hidden, self.out_layers[0].weight,
+ self.out_layers[0].bias, self.out_projs[0])
+ if target is not None:
+ output = -F.log_softmax(logit, dim=-1) \
+ .gather(1, target.unsqueeze(1)).squeeze(1)
+ else:
+ output = F.log_softmax(logit, dim=-1)
+ else:
+ # construct weights and biases
+ weights, biases = [], []
+ for i in range(len(self.cutoffs)):
+ if self.div_val == 1:
+ l_idx, r_idx = self.cutoff_ends[i], self.cutoff_ends[i + 1]
+ weight_i = self.out_layers[0].weight[l_idx:r_idx]
+ bias_i = self.out_layers[0].bias[l_idx:r_idx]
+ else:
+ weight_i = self.out_layers[i].weight
+ bias_i = self.out_layers[i].bias
+
+ if i == 0:
+ weight_i = torch.cat(
+ [weight_i, self.cluster_weight], dim=0)
+ bias_i = torch.cat(
+ [bias_i, self.cluster_bias], dim=0)
+
+ weights.append(weight_i)
+ biases.append(bias_i)
+
+ head_weight, head_bias, head_proj = weights[0], biases[0], self.out_projs[0]
+
+ head_logit = self._compute_logit(hidden, head_weight, head_bias, head_proj)
+ head_logprob = F.log_softmax(head_logit, dim=1)
+
+ if target is None:
+ out = hidden.new_empty((head_logit.size(0), self.n_token))
+ else:
+ out = torch.zeros_like(target, dtype=hidden.dtype, device=hidden.device)
+
+ offset = 0
+ cutoff_values = [0] + self.cutoffs
+ for i in range(len(cutoff_values) - 1):
+ l_idx, r_idx = cutoff_values[i], cutoff_values[i + 1]
+
+ if target is not None:
+ mask_i = (target >= l_idx) & (target < r_idx)
+ indices_i = mask_i.nonzero().squeeze()
+
+ if indices_i.numel() == 0:
+ continue
+
+ target_i = target.index_select(0, indices_i) - l_idx
+ head_logprob_i = head_logprob.index_select(0, indices_i)
+ hidden_i = hidden.index_select(0, indices_i)
+ else:
+ hidden_i = hidden
+
+ if i == 0:
+ if target is not None:
+ logprob_i = head_logprob_i.gather(1, target_i[:, None]).squeeze(1)
+ else:
+ out[:, :self.cutoffs[0]] = head_logprob[:, :self.cutoffs[0]]
+ else:
+ weight_i, bias_i, proj_i = weights[i], biases[i], self.out_projs[i]
+
+ tail_logit_i = self._compute_logit(hidden_i, weight_i, bias_i, proj_i)
+ tail_logprob_i = F.log_softmax(tail_logit_i, dim=1)
+ cluster_prob_idx = self.cutoffs[0] + i - 1 # No probability for the head cluster
+ if target is not None:
+ logprob_i = head_logprob_i[:, cluster_prob_idx] \
+ + tail_logprob_i.gather(1, target_i[:, None]).squeeze(1)
+ else:
+ logprob_i = head_logprob[:, cluster_prob_idx, None] + tail_logprob_i
+ out[:, l_idx:r_idx] = logprob_i
+
+ if target is not None:
+ if (hasattr(self, 'keep_order') and self.keep_order) or keep_order:
+ out.index_copy_(0, indices_i, -logprob_i)
+ else:
+ out[offset:offset+logprob_i.size(0)].copy_(-logprob_i)
+ offset += logprob_i.size(0)
+
+ return out
+
+
+ def log_prob(self, hidden):
+ r""" Computes log probabilities for all :math:`n\_classes`
+ From: https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/adaptive.py
+ Args:
+ hidden (Tensor): a minibatch of examples
+ Returns:
+ log-probabilities of for each class :math:`c`
+ in range :math:`0 <= c <= n\_classes`, where :math:`n\_classes` is a
+ parameter passed to ``AdaptiveLogSoftmaxWithLoss`` constructor.
+ Shape:
+ - Input: :math:`(N, in\_features)`
+ - Output: :math:`(N, n\_classes)`
+ """
+ if self.n_clusters == 0:
+ logit = self._compute_logit(hidden, self.out_layers[0].weight,
+ self.out_layers[0].bias, self.out_projs[0])
+ return F.log_softmax(logit, dim=-1)
+ else:
+ # construct weights and biases
+ weights, biases = [], []
+ for i in range(len(self.cutoffs)):
+ if self.div_val == 1:
+ l_idx, r_idx = self.cutoff_ends[i], self.cutoff_ends[i + 1]
+ weight_i = self.out_layers[0].weight[l_idx:r_idx]
+ bias_i = self.out_layers[0].bias[l_idx:r_idx]
+ else:
+ weight_i = self.out_layers[i].weight
+ bias_i = self.out_layers[i].bias
+
+ if i == 0:
+ weight_i = torch.cat(
+ [weight_i, self.cluster_weight], dim=0)
+ bias_i = torch.cat(
+ [bias_i, self.cluster_bias], dim=0)
+
+ weights.append(weight_i)
+ biases.append(bias_i)
+
+ head_weight, head_bias, head_proj = weights[0], biases[0], self.out_projs[0]
+ head_logit = self._compute_logit(hidden, head_weight, head_bias, head_proj)
+
+ out = hidden.new_empty((head_logit.size(0), self.n_token))
+ head_logprob = F.log_softmax(head_logit, dim=1)
+
+ cutoff_values = [0] + self.cutoffs
+ for i in range(len(cutoff_values) - 1):
+ start_idx, stop_idx = cutoff_values[i], cutoff_values[i + 1]
+
+ if i == 0:
+ out[:, :self.cutoffs[0]] = head_logprob[:, :self.cutoffs[0]]
+ else:
+ weight_i, bias_i, proj_i = weights[i], biases[i], self.out_projs[i]
+
+ tail_logit_i = self._compute_logit(hidden, weight_i, bias_i, proj_i)
+ tail_logprob_i = F.log_softmax(tail_logit_i, dim=1)
+
+ logprob_i = head_logprob[:, -i] + tail_logprob_i
+ out[:, start_idx, stop_idx] = logprob_i
+
+ return out
+
+
+class LogUniformSampler(object):
+ def __init__(self, range_max, n_sample):
+ """
+ Reference : https://github.com/tensorflow/tensorflow/blob/r1.10/tensorflow/python/ops/candidate_sampling_ops.py
+ `P(class) = (log(class + 2) - log(class + 1)) / log(range_max + 1)`
+
+ expected count can be approximated by 1 - (1 - p)^n
+ and we use a numerically stable version -expm1(num_tries * log1p(-p))
+
+ Our implementation fixes num_tries at 2 * n_sample, and the actual #samples will vary from run to run
+ """
+ with torch.no_grad():
+ self.range_max = range_max
+ log_indices = torch.arange(1., range_max+2., 1.).log_()
+ self.dist = (log_indices[1:] - log_indices[:-1]) / log_indices[-1]
+ # print('P', self.dist.numpy().tolist()[-30:])
+
+ self.log_q = (- (-self.dist.double().log1p_() * 2 * n_sample).expm1_()).log_().float()
+
+ self.n_sample = n_sample
+
+ def sample(self, labels):
+ """
+ labels: [b1, b2]
+ Return
+ true_log_probs: [b1, b2]
+ samp_log_probs: [n_sample]
+ neg_samples: [n_sample]
+ """
+
+ # neg_samples = torch.empty(0).long()
+ n_sample = self.n_sample
+ n_tries = 2 * n_sample
+
+ with torch.no_grad():
+ neg_samples = torch.multinomial(self.dist, n_tries, replacement=True).unique()
+ device = labels.device
+ neg_samples = neg_samples.to(device)
+ true_log_probs = self.log_q[labels].to(device)
+ samp_log_probs = self.log_q[neg_samples].to(device)
+ return true_log_probs, samp_log_probs, neg_samples
+
+def sample_logits(embedding, bias, labels, inputs, sampler):
+ """
+ embedding: an nn.Embedding layer
+ bias: [n_vocab]
+ labels: [b1, b2]
+ inputs: [b1, b2, n_emb]
+ sampler: you may use a LogUniformSampler
+ Return
+ logits: [b1, b2, 1 + n_sample]
+ """
+ true_log_probs, samp_log_probs, neg_samples = sampler.sample(labels)
+ n_sample = neg_samples.size(0)
+ b1, b2 = labels.size(0), labels.size(1)
+ all_ids = torch.cat([labels.view(-1), neg_samples])
+ all_w = embedding(all_ids)
+ true_w = all_w[: -n_sample].view(b1, b2, -1)
+ sample_w = all_w[- n_sample:].view(n_sample, -1)
+
+ all_b = bias[all_ids]
+ true_b = all_b[: -n_sample].view(b1, b2)
+ sample_b = all_b[- n_sample:]
+
+ hit = (labels[:, :, None] == neg_samples).detach()
+
+ true_logits = torch.einsum('ijk,ijk->ij',
+ [true_w, inputs]) + true_b - true_log_probs
+ sample_logits = torch.einsum('lk,ijk->ijl',
+ [sample_w, inputs]) + sample_b - samp_log_probs
+ sample_logits.masked_fill_(hit, -1e30)
+ logits = torch.cat([true_logits[:, :, None], sample_logits], -1)
+
+ return logits
+
+
+# class LogUniformSampler(object):
+# def __init__(self, range_max, unique=False):
+# """
+# Reference : https://github.com/tensorflow/tensorflow/blob/r1.10/tensorflow/python/ops/candidate_sampling_ops.py
+# `P(class) = (log(class + 2) - log(class + 1)) / log(range_max + 1)`
+# """
+# self.range_max = range_max
+# log_indices = torch.arange(1., range_max+2., 1.).log_()
+# self.dist = (log_indices[1:] - log_indices[:-1]) / log_indices[-1]
+
+# self.unique = unique
+
+# if self.unique:
+# self.exclude_mask = torch.ByteTensor(range_max).fill_(0)
+
+# def sample(self, n_sample, labels):
+# pos_sample, new_labels = labels.unique(return_inverse=True)
+# n_pos_sample = pos_sample.size(0)
+# n_neg_sample = n_sample - n_pos_sample
+
+# if self.unique:
+# self.exclude_mask.index_fill_(0, pos_sample, 1)
+# sample_dist = self.dist.clone().masked_fill_(self.exclude_mask, 0)
+# self.exclude_mask.index_fill_(0, pos_sample, 0)
+# else:
+# sample_dist = self.dist
+
+# neg_sample = torch.multinomial(sample_dist, n_neg_sample)
+
+# sample = torch.cat([pos_sample, neg_sample])
+# sample_prob = self.dist[sample]
+
+# return new_labels, sample, sample_prob
+
+
+if __name__ == '__main__':
+ S, B = 3, 4
+ n_vocab = 10000
+ n_sample = 5
+ H = 32
+
+ labels = torch.LongTensor(S, B).random_(0, n_vocab)
+
+ # sampler = LogUniformSampler(n_vocab, unique=False)
+ # new_labels, sample, sample_prob = sampler.sample(n_sample, labels)
+
+ sampler = LogUniformSampler(n_vocab, n_sample)#, unique=True)
+ # true_probs, samp_probs, neg_samples = sampler.sample(n_sample, labels)
+
+ # print('true_probs', true_probs.numpy().tolist())
+ # print('samp_probs', samp_probs.numpy().tolist())
+ # print('neg_samples', neg_samples.numpy().tolist())
+
+ # print('sum', torch.sum(sampler.dist).item())
+
+ # assert torch.all(torch.sort(sample.unique())[0].eq(torch.sort(sample)[0])).item()
+
+ embedding = nn.Embedding(n_vocab, H)
+ bias = torch.zeros(n_vocab)
+ inputs = torch.Tensor(S, B, H).normal_()
+
+ logits, out_labels = sample_logits(embedding, bias, labels, inputs, sampler, n_sample)
+ print('logits', logits.detach().numpy().tolist())
+ print('logits shape', logits.size())
+ print('out_labels', out_labels.detach().numpy().tolist())
+ print('out_labels shape', out_labels.size())
+
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization.py
new file mode 100644
index 0000000000000000000000000000000000000000..aa59c7d7ecf717b11bc2ad84f7e9aac07bae5697
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization.py
@@ -0,0 +1,180 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch optimization for BERT model."""
+
+import math
+import torch
+from torch.optim import Optimizer
+from torch.optim.optimizer import required
+from torch.nn.utils import clip_grad_norm_
+import logging
+
+logger = logging.getLogger(__name__)
+
+def warmup_cosine(x, warmup=0.002):
+ if x < warmup:
+ return x/warmup
+ x_ = (x - warmup) / (1 - warmup) # progress after warmup -
+ return 0.5 * (1. + math.cos(math.pi * x_))
+
+def warmup_constant(x, warmup=0.002):
+ """ Linearly increases learning rate over `warmup`*`t_total` (as provided to BertAdam) training steps.
+ Learning rate is 1. afterwards. """
+ if x < warmup:
+ return x/warmup
+ return 1.0
+
+def warmup_linear(x, warmup=0.002):
+ """ Specifies a triangular learning rate schedule where peak is reached at `warmup`*`t_total`-th (as provided to BertAdam) training step.
+ After `t_total`-th training step, learning rate is zero. """
+ if x < warmup:
+ return x/warmup
+ return max((x-1.)/(warmup-1.), 0)
+
+SCHEDULES = {
+ 'warmup_cosine': warmup_cosine,
+ 'warmup_constant': warmup_constant,
+ 'warmup_linear': warmup_linear,
+}
+
+
+class BertAdam(Optimizer):
+ """Implements BERT version of Adam algorithm with weight decay fix.
+ Params:
+ lr: learning rate
+ warmup: portion of t_total for the warmup, -1 means no warmup. Default: -1
+ t_total: total number of training steps for the learning
+ rate schedule, -1 means constant learning rate. Default: -1
+ schedule: schedule to use for the warmup (see above). Default: 'warmup_linear'
+ b1: Adams b1. Default: 0.9
+ b2: Adams b2. Default: 0.999
+ e: Adams epsilon. Default: 1e-6
+ weight_decay: Weight decay. Default: 0.01
+ max_grad_norm: Maximum norm for the gradients (-1 means no clipping). Default: 1.0
+ """
+ def __init__(self, params, lr=required, warmup=-1, t_total=-1, schedule='warmup_linear',
+ b1=0.9, b2=0.999, e=1e-6, weight_decay=0.01,
+ max_grad_norm=1.0):
+ if lr is not required and lr < 0.0:
+ raise ValueError("Invalid learning rate: {} - should be >= 0.0".format(lr))
+ if schedule not in SCHEDULES:
+ raise ValueError("Invalid schedule parameter: {}".format(schedule))
+ if not 0.0 <= warmup < 1.0 and not warmup == -1:
+ raise ValueError("Invalid warmup: {} - should be in [0.0, 1.0[ or -1".format(warmup))
+ if not 0.0 <= b1 < 1.0:
+ raise ValueError("Invalid b1 parameter: {} - should be in [0.0, 1.0[".format(b1))
+ if not 0.0 <= b2 < 1.0:
+ raise ValueError("Invalid b2 parameter: {} - should be in [0.0, 1.0[".format(b2))
+ if not e >= 0.0:
+ raise ValueError("Invalid epsilon value: {} - should be >= 0.0".format(e))
+ defaults = dict(lr=lr, schedule=schedule, warmup=warmup, t_total=t_total,
+ b1=b1, b2=b2, e=e, weight_decay=weight_decay,
+ max_grad_norm=max_grad_norm)
+ super(BertAdam, self).__init__(params, defaults)
+
+ def get_lr(self):
+ lr = []
+ for group in self.param_groups:
+ for p in group['params']:
+ state = self.state[p]
+ if len(state) == 0:
+ return [0]
+ if group['t_total'] != -1:
+ schedule_fct = SCHEDULES[group['schedule']]
+ lr_scheduled = group['lr'] * schedule_fct(state['step']/group['t_total'], group['warmup'])
+ else:
+ lr_scheduled = group['lr']
+ lr.append(lr_scheduled)
+ return lr
+
+ def step(self, closure=None):
+ """Performs a single optimization step.
+
+ Arguments:
+ closure (callable, optional): A closure that reevaluates the model
+ and returns the loss.
+ """
+ loss = None
+ if closure is not None:
+ loss = closure()
+
+ warned_for_t_total = False
+
+ for group in self.param_groups:
+ for p in group['params']:
+ if p.grad is None:
+ continue
+ grad = p.grad.data
+ if grad.is_sparse:
+ raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')
+
+ state = self.state[p]
+
+ # State initialization
+ if len(state) == 0:
+ state['step'] = 0
+ # Exponential moving average of gradient values
+ state['next_m'] = torch.zeros_like(p.data)
+ # Exponential moving average of squared gradient values
+ state['next_v'] = torch.zeros_like(p.data)
+
+ next_m, next_v = state['next_m'], state['next_v']
+ beta1, beta2 = group['b1'], group['b2']
+
+ # Add grad clipping
+ if group['max_grad_norm'] > 0:
+ clip_grad_norm_(p, group['max_grad_norm'])
+
+ # Decay the first and second moment running average coefficient
+ # In-place operations to update the averages at the same time
+ next_m.mul_(beta1).add_(1 - beta1, grad)
+ next_v.mul_(beta2).addcmul_(1 - beta2, grad, grad)
+ update = next_m / (next_v.sqrt() + group['e'])
+
+ # Just adding the square of the weights to the loss function is *not*
+ # the correct way of using L2 regularization/weight decay with Adam,
+ # since that will interact with the m and v parameters in strange ways.
+ #
+ # Instead we want to decay the weights in a manner that doesn't interact
+ # with the m/v parameters. This is equivalent to adding the square
+ # of the weights to the loss with plain (non-momentum) SGD.
+ if group['weight_decay'] > 0.0:
+ update += group['weight_decay'] * p.data
+
+ if group['t_total'] != -1:
+ schedule_fct = SCHEDULES[group['schedule']]
+ progress = state['step']/group['t_total']
+ lr_scheduled = group['lr'] * schedule_fct(progress, group['warmup'])
+ # warning for exceeding t_total (only active with warmup_linear
+ if group['schedule'] == "warmup_linear" and progress > 1. and not warned_for_t_total:
+ logger.warning(
+ "Training beyond specified 't_total' steps with schedule '{}'. Learning rate set to {}. "
+ "Please set 't_total' of {} correctly.".format(group['schedule'], lr_scheduled, self.__class__.__name__))
+ warned_for_t_total = True
+ # end warning
+ else:
+ lr_scheduled = group['lr']
+
+ update_with_lr = lr_scheduled * update
+ p.data.add_(-update_with_lr)
+
+ state['step'] += 1
+
+ # step_size = lr_scheduled * math.sqrt(bias_correction2) / bias_correction1
+ # No bias correction
+ # bias_correction1 = 1 - beta1 ** state['step']
+ # bias_correction2 = 1 - beta2 ** state['step']
+
+ return loss
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization_openai.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization_openai.py
new file mode 100644
index 0000000000000000000000000000000000000000..99ac15e1089a512e0616348536debc6b52e0a812
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/optimization_openai.py
@@ -0,0 +1,161 @@
+# coding=utf-8
+# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch optimization for OpenAI GPT model."""
+
+import math
+import torch
+from torch.optim import Optimizer
+from torch.optim.optimizer import required
+from torch.nn.utils import clip_grad_norm_
+import logging
+
+logger = logging.getLogger(__name__)
+
+def warmup_cosine(x, warmup=0.002):
+ if x < warmup:
+ return x/warmup
+ x_ = (x - warmup) / (1 - warmup) # progress after warmup
+ return 0.5 * (1. + math.cos(math.pi * x_))
+
+def warmup_constant(x, warmup=0.002):
+ """ Linearly increases learning rate over `warmup`*`t_total` (as provided to OpenAIAdam) training steps.
+ Learning rate is 1. afterwards. """
+ if x < warmup:
+ return x/warmup
+ return 1.0
+
+def warmup_linear(x, warmup=0.002):
+ """ Specifies a triangular learning rate schedule where peak is reached at `warmup`*`t_total`-th (as provided to OpenAIAdam) training step.
+ After `t_total`-th training step, learning rate is zero. """
+ if x < warmup:
+ return x/warmup
+ return max((x-1.)/(warmup-1.), 0)
+
+SCHEDULES = {
+ 'warmup_cosine':warmup_cosine,
+ 'warmup_constant':warmup_constant,
+ 'warmup_linear':warmup_linear,
+}
+
+
+class OpenAIAdam(Optimizer):
+ """Implements Open AI version of Adam algorithm with weight decay fix.
+ """
+ def __init__(self, params, lr=required, schedule='warmup_linear', warmup=-1, t_total=-1,
+ b1=0.9, b2=0.999, e=1e-8, weight_decay=0,
+ vector_l2=False, max_grad_norm=-1, **kwargs):
+ if lr is not required and lr < 0.0:
+ raise ValueError("Invalid learning rate: {} - should be >= 0.0".format(lr))
+ if schedule not in SCHEDULES:
+ raise ValueError("Invalid schedule parameter: {}".format(schedule))
+ if not 0.0 <= warmup < 1.0 and not warmup == -1:
+ raise ValueError("Invalid warmup: {} - should be in [0.0, 1.0[ or -1".format(warmup))
+ if not 0.0 <= b1 < 1.0:
+ raise ValueError("Invalid b1 parameter: {}".format(b1))
+ if not 0.0 <= b2 < 1.0:
+ raise ValueError("Invalid b2 parameter: {}".format(b2))
+ if not e >= 0.0:
+ raise ValueError("Invalid epsilon value: {}".format(e))
+ defaults = dict(lr=lr, schedule=schedule, warmup=warmup, t_total=t_total,
+ b1=b1, b2=b2, e=e, weight_decay=weight_decay, vector_l2=vector_l2,
+ max_grad_norm=max_grad_norm)
+ super(OpenAIAdam, self).__init__(params, defaults)
+
+ def get_lr(self):
+ lr = []
+ for group in self.param_groups:
+ for p in group['params']:
+ state = self.state[p]
+ if len(state) == 0:
+ return [0]
+ if group['t_total'] != -1:
+ schedule_fct = SCHEDULES[group['schedule']]
+ lr_scheduled = group['lr'] * schedule_fct(state['step']/group['t_total'], group['warmup'])
+ else:
+ lr_scheduled = group['lr']
+ lr.append(lr_scheduled)
+ return lr
+
+ def step(self, closure=None):
+ """Performs a single optimization step.
+
+ Arguments:
+ closure (callable, optional): A closure that reevaluates the model
+ and returns the loss.
+ """
+ loss = None
+ if closure is not None:
+ loss = closure()
+
+ warned_for_t_total = False
+
+ for group in self.param_groups:
+ for p in group['params']:
+ if p.grad is None:
+ continue
+ grad = p.grad.data
+ if grad.is_sparse:
+ raise RuntimeError('Adam does not support sparse gradients, please consider SparseAdam instead')
+
+ state = self.state[p]
+
+ # State initialization
+ if len(state) == 0:
+ state['step'] = 0
+ # Exponential moving average of gradient values
+ state['exp_avg'] = torch.zeros_like(p.data)
+ # Exponential moving average of squared gradient values
+ state['exp_avg_sq'] = torch.zeros_like(p.data)
+
+ exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
+ beta1, beta2 = group['b1'], group['b2']
+
+ state['step'] += 1
+
+ # Add grad clipping
+ if group['max_grad_norm'] > 0:
+ clip_grad_norm_(p, group['max_grad_norm'])
+
+ # Decay the first and second moment running average coefficient
+ exp_avg.mul_(beta1).add_(1 - beta1, grad)
+ exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
+ denom = exp_avg_sq.sqrt().add_(group['e'])
+
+ bias_correction1 = 1 - beta1 ** state['step']
+ bias_correction2 = 1 - beta2 ** state['step']
+
+ if group['t_total'] != -1:
+ schedule_fct = SCHEDULES[group['schedule']]
+ progress = state['step']/group['t_total']
+ lr_scheduled = group['lr'] * schedule_fct(progress, group['warmup'])
+ # warning for exceeding t_total (only active with warmup_linear
+ if group['schedule'] == "warmup_linear" and progress > 1. and not warned_for_t_total:
+ logger.warning(
+ "Training beyond specified 't_total' steps with schedule '{}'. Learning rate set to {}. "
+ "Please set 't_total' of {} correctly.".format(group['schedule'], lr_scheduled, self.__class__.__name__))
+ warned_for_t_total = True
+ # end warning
+ else:
+ lr_scheduled = group['lr']
+
+ step_size = lr_scheduled * math.sqrt(bias_correction2) / bias_correction1
+
+ p.data.addcdiv_(-step_size, exp_avg, denom)
+
+ # Add weight decay at the end (fixed version)
+ if (len(p.size()) > 1 or group['vector_l2']) and group['weight_decay'] > 0:
+ p.data.add_(-lr_scheduled * group['weight_decay'], p.data)
+
+ return loss
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization.py
new file mode 100644
index 0000000000000000000000000000000000000000..3937d6e01189212cfa1f76e8cd694ab3d259f8f6
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization.py
@@ -0,0 +1,415 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes."""
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import collections
+import logging
+import os
+import unicodedata
+from io import open
+
+from .file_utils import cached_path
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_VOCAB_ARCHIVE_MAP = {
+ 'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt",
+ 'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt",
+ 'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt",
+ 'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt",
+ 'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-vocab.txt",
+ 'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt",
+ 'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt",
+}
+PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = {
+ 'bert-base-uncased': 512,
+ 'bert-large-uncased': 512,
+ 'bert-base-cased': 512,
+ 'bert-large-cased': 512,
+ 'bert-base-multilingual-uncased': 512,
+ 'bert-base-multilingual-cased': 512,
+ 'bert-base-chinese': 512,
+}
+VOCAB_NAME = 'vocab.txt'
+
+
+def load_vocab(vocab_file):
+ """Loads a vocabulary file into a dictionary."""
+ vocab = collections.OrderedDict()
+ index = 0
+ with open(vocab_file, "r", encoding="utf-8") as reader:
+ while True:
+ token = reader.readline()
+ if not token:
+ break
+ token = token.strip()
+ vocab[token] = index
+ index += 1
+ return vocab
+
+
+def whitespace_tokenize(text):
+ """Runs basic whitespace cleaning and splitting on a piece of text."""
+ text = text.strip()
+ if not text:
+ return []
+ tokens = text.split()
+ return tokens
+
+
+class BertTokenizer(object):
+ """Runs end-to-end tokenization: punctuation splitting + wordpiece"""
+
+ def __init__(self, vocab_file, do_lower_case=True, max_len=None, do_basic_tokenize=True,
+ never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")):
+ """Constructs a BertTokenizer.
+
+ Args:
+ vocab_file: Path to a one-wordpiece-per-line vocabulary file
+ do_lower_case: Whether to lower case the input
+ Only has an effect when do_wordpiece_only=False
+ do_basic_tokenize: Whether to do basic tokenization before wordpiece.
+ max_len: An artificial maximum length to truncate tokenized sequences to;
+ Effective maximum length is always the minimum of this
+ value (if specified) and the underlying BERT model's
+ sequence length.
+ never_split: List of tokens which will never be split during tokenization.
+ Only has an effect when do_wordpiece_only=False
+ """
+ if not os.path.isfile(vocab_file):
+ raise ValueError(
+ "Can't find a vocabulary file at path '{}'. To load the vocabulary from a Google pretrained "
+ "model use `tokenizer = BertTokenizer.from_pretrained(PRETRAINED_MODEL_NAME)`".format(vocab_file))
+ self.vocab = load_vocab(vocab_file)
+ self.ids_to_tokens = collections.OrderedDict(
+ [(ids, tok) for tok, ids in self.vocab.items()])
+ self.do_basic_tokenize = do_basic_tokenize
+ if do_basic_tokenize:
+ self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case,
+ never_split=never_split)
+ self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+ self.max_len = max_len if max_len is not None else int(1e12)
+
+ def tokenize(self, text):
+ split_tokens = []
+ if self.do_basic_tokenize:
+ for token in self.basic_tokenizer.tokenize(text):
+ for sub_token in self.wordpiece_tokenizer.tokenize(token):
+ split_tokens.append(sub_token)
+ else:
+ split_tokens = self.wordpiece_tokenizer.tokenize(text)
+ return split_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ """Converts a sequence of tokens into ids using the vocab."""
+ ids = []
+ for token in tokens:
+ ids.append(self.vocab[token])
+ if len(ids) > self.max_len:
+ logger.warning(
+ "Token indices sequence length is longer than the specified maximum "
+ " sequence length for this BERT model ({} > {}). Running this"
+ " sequence through BERT will result in indexing errors".format(len(ids), self.max_len)
+ )
+ return ids
+
+ def convert_ids_to_tokens(self, ids):
+ """Converts a sequence of ids in wordpiece tokens using the vocab."""
+ tokens = []
+ for i in ids:
+ tokens.append(self.ids_to_tokens[i])
+ return tokens
+
+ def save_vocabulary(self, vocab_path):
+ """Save the tokenizer vocabulary to a directory or file."""
+ index = 0
+ if os.path.isdir(vocab_path):
+ vocab_file = os.path.join(vocab_path, VOCAB_NAME)
+ with open(vocab_file, "w", encoding="utf-8") as writer:
+ for token, token_index in sorted(self.vocab.items(), key=lambda kv: kv[1]):
+ if index != token_index:
+ logger.warning("Saving vocabulary to {}: vocabulary indices are not consecutive."
+ " Please check that the vocabulary is not corrupted!".format(vocab_file))
+ index = token_index
+ writer.write(token + u'\n')
+ index += 1
+ return vocab_file
+
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
+ """
+ Instantiate a PreTrainedBertModel from a pre-trained model file.
+ Download and cache the pre-trained model file if needed.
+ """
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP:
+ vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path]
+ if '-cased' in pretrained_model_name_or_path and kwargs.get('do_lower_case', True):
+ logger.warning("The pre-trained model you are loading is a cased model but you have not set "
+ "`do_lower_case` to False. We are setting `do_lower_case=False` for you but "
+ "you may want to check this behavior.")
+ kwargs['do_lower_case'] = False
+ elif '-cased' not in pretrained_model_name_or_path and not kwargs.get('do_lower_case', True):
+ logger.warning("The pre-trained model you are loading is an uncased model but you have set "
+ "`do_lower_case` to False. We are setting `do_lower_case=True` for you "
+ "but you may want to check this behavior.")
+ kwargs['do_lower_case'] = True
+ else:
+ vocab_file = pretrained_model_name_or_path
+ if os.path.isdir(vocab_file):
+ vocab_file = os.path.join(vocab_file, VOCAB_NAME)
+ # redirect to the cache, if necessary
+ try:
+ resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find any file "
+ "associated to this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
+ vocab_file))
+ return None
+ if resolved_vocab_file == vocab_file:
+ logger.info("loading vocabulary file {}".format(vocab_file))
+ else:
+ logger.info("loading vocabulary file {} from cache at {}".format(
+ vocab_file, resolved_vocab_file))
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP:
+ # if we're using a pretrained model, ensure the tokenizer wont index sequences longer
+ # than the number of positional embeddings
+ max_len = PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP[pretrained_model_name_or_path]
+ kwargs['max_len'] = min(kwargs.get('max_len', int(1e12)), max_len)
+ # Instantiate tokenizer.
+ tokenizer = cls(resolved_vocab_file, *inputs, **kwargs)
+ return tokenizer
+
+
+class BasicTokenizer(object):
+ """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
+
+ def __init__(self,
+ do_lower_case=True,
+ never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")):
+ """Constructs a BasicTokenizer.
+
+ Args:
+ do_lower_case: Whether to lower case the input.
+ """
+ self.do_lower_case = do_lower_case
+ self.never_split = never_split
+
+ def tokenize(self, text):
+ """Tokenizes a piece of text."""
+ text = self._clean_text(text)
+ # This was added on November 1st, 2018 for the multilingual and Chinese
+ # models. This is also applied to the English models now, but it doesn't
+ # matter since the English models were not trained on any Chinese data
+ # and generally don't have any Chinese data in them (there are Chinese
+ # characters in the vocabulary because Wikipedia does have some Chinese
+ # words in the English Wikipedia.).
+ text = self._tokenize_chinese_chars(text)
+ orig_tokens = whitespace_tokenize(text)
+ split_tokens = []
+ for token in orig_tokens:
+ if self.do_lower_case and token not in self.never_split:
+ token = token.lower()
+ token = self._run_strip_accents(token)
+ split_tokens.extend(self._run_split_on_punc(token))
+
+ output_tokens = whitespace_tokenize(" ".join(split_tokens))
+ return output_tokens
+
+ def _run_strip_accents(self, text):
+ """Strips accents from a piece of text."""
+ text = unicodedata.normalize("NFD", text)
+ output = []
+ for char in text:
+ cat = unicodedata.category(char)
+ if cat == "Mn":
+ continue
+ output.append(char)
+ return "".join(output)
+
+ def _run_split_on_punc(self, text):
+ """Splits punctuation on a piece of text."""
+ if text in self.never_split:
+ return [text]
+ chars = list(text)
+ i = 0
+ start_new_word = True
+ output = []
+ while i < len(chars):
+ char = chars[i]
+ if _is_punctuation(char):
+ output.append([char])
+ start_new_word = True
+ else:
+ if start_new_word:
+ output.append([])
+ start_new_word = False
+ output[-1].append(char)
+ i += 1
+
+ return ["".join(x) for x in output]
+
+ def _tokenize_chinese_chars(self, text):
+ """Adds whitespace around any CJK character."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if self._is_chinese_char(cp):
+ output.append(" ")
+ output.append(char)
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+ def _is_chinese_char(self, cp):
+ """Checks whether CP is the codepoint of a CJK character."""
+ # This defines a "chinese character" as anything in the CJK Unicode block:
+ # https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+ #
+ # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
+ # despite its name. The modern Korean Hangul alphabet is a different block,
+ # as is Japanese Hiragana and Katakana. Those alphabets are used to write
+ # space-separated words, so they are not treated specially and handled
+ # like the all of the other languages.
+ if ((cp >= 0x4E00 and cp <= 0x9FFF) or #
+ (cp >= 0x3400 and cp <= 0x4DBF) or #
+ (cp >= 0x20000 and cp <= 0x2A6DF) or #
+ (cp >= 0x2A700 and cp <= 0x2B73F) or #
+ (cp >= 0x2B740 and cp <= 0x2B81F) or #
+ (cp >= 0x2B820 and cp <= 0x2CEAF) or
+ (cp >= 0xF900 and cp <= 0xFAFF) or #
+ (cp >= 0x2F800 and cp <= 0x2FA1F)): #
+ return True
+
+ return False
+
+ def _clean_text(self, text):
+ """Performs invalid character removal and whitespace cleanup on text."""
+ output = []
+ for char in text:
+ cp = ord(char)
+ if cp == 0 or cp == 0xfffd or _is_control(char):
+ continue
+ if _is_whitespace(char):
+ output.append(" ")
+ else:
+ output.append(char)
+ return "".join(output)
+
+
+class WordpieceTokenizer(object):
+ """Runs WordPiece tokenization."""
+
+ def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=100):
+ self.vocab = vocab
+ self.unk_token = unk_token
+ self.max_input_chars_per_word = max_input_chars_per_word
+
+ def tokenize(self, text):
+ """Tokenizes a piece of text into its word pieces.
+
+ This uses a greedy longest-match-first algorithm to perform tokenization
+ using the given vocabulary.
+
+ For example:
+ input = "unaffable"
+ output = ["un", "##aff", "##able"]
+
+ Args:
+ text: A single token or whitespace separated tokens. This should have
+ already been passed through `BasicTokenizer`.
+
+ Returns:
+ A list of wordpiece tokens.
+ """
+
+ output_tokens = []
+ for token in whitespace_tokenize(text):
+ chars = list(token)
+ if len(chars) > self.max_input_chars_per_word:
+ output_tokens.append(self.unk_token)
+ continue
+
+ is_bad = False
+ start = 0
+ sub_tokens = []
+ while start < len(chars):
+ end = len(chars)
+ cur_substr = None
+ while start < end:
+ substr = "".join(chars[start:end])
+ if start > 0:
+ substr = "##" + substr
+ if substr in self.vocab:
+ cur_substr = substr
+ break
+ end -= 1
+ if cur_substr is None:
+ is_bad = True
+ break
+ sub_tokens.append(cur_substr)
+ start = end
+
+ if is_bad:
+ output_tokens.append(self.unk_token)
+ else:
+ output_tokens.extend(sub_tokens)
+ return output_tokens
+
+
+def _is_whitespace(char):
+ """Checks whether `chars` is a whitespace character."""
+ # \t, \n, and \r are technically contorl characters but we treat them
+ # as whitespace since they are generally considered as such.
+ if char == " " or char == "\t" or char == "\n" or char == "\r":
+ return True
+ cat = unicodedata.category(char)
+ if cat == "Zs":
+ return True
+ return False
+
+
+def _is_control(char):
+ """Checks whether `chars` is a control character."""
+ # These are technically control characters but we count them as whitespace
+ # characters.
+ if char == "\t" or char == "\n" or char == "\r":
+ return False
+ cat = unicodedata.category(char)
+ if cat.startswith("C"):
+ return True
+ return False
+
+
+def _is_punctuation(char):
+ """Checks whether `chars` is a punctuation character."""
+ cp = ord(char)
+ # We treat all non-letter/number ASCII as punctuation.
+ # Characters such as "^", "$", and "`" are not in the Unicode
+ # Punctuation class but we treat them as punctuation anyways, for
+ # consistency.
+ if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
+ (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
+ return True
+ cat = unicodedata.category(char)
+ if cat.startswith("P"):
+ return True
+ return False
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_gpt2.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_gpt2.py
new file mode 100644
index 0000000000000000000000000000000000000000..07777292a3fa507346552278a89ee33f5af025aa
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_gpt2.py
@@ -0,0 +1,301 @@
+# coding=utf-8
+# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes for OpenAI GPT."""
+from __future__ import (absolute_import, division, print_function,
+ unicode_literals)
+
+import sys
+import json
+import logging
+import os
+import regex as re
+from io import open
+
+try:
+ from functools import lru_cache
+except ImportError:
+ # Just a dummy decorator to get the checks to run on python2
+ # because honestly I don't want to support a byte-level unicode BPE tokenizer on python 2 right now.
+ def lru_cache():
+ return lambda func: func
+
+from .file_utils import cached_path
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_VOCAB_ARCHIVE_MAP = {
+ 'gpt2': "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json",
+}
+PRETRAINED_MERGES_ARCHIVE_MAP = {
+ 'gpt2': "https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt",
+}
+PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = {
+ 'gpt2': 1024,
+}
+VOCAB_NAME = 'vocab.json'
+MERGES_NAME = 'merges.txt'
+SPECIAL_TOKENS_NAME = 'special_tokens.txt'
+
+@lru_cache()
+def bytes_to_unicode():
+ """
+ Returns list of utf-8 byte and a corresponding list of unicode strings.
+ The reversible bpe codes work on unicode strings.
+ This means you need a large # of unicode characters in your vocab if you want to avoid UNKs.
+ When you're at something like a 10B token dataset you end up needing around 5K for decent coverage.
+ This is a signficant percentage of your normal, say, 32K bpe vocab.
+ To avoid that, we want lookup tables between utf-8 bytes and unicode strings.
+ And avoids mapping to whitespace/control characters the bpe code barfs on.
+ """
+ _chr = unichr if sys.version_info[0] == 2 else chr
+ bs = list(range(ord("!"), ord("~")+1))+list(range(ord("¡"), ord("¬")+1))+list(range(ord("®"), ord("ÿ")+1))
+ cs = bs[:]
+ n = 0
+ for b in range(2**8):
+ if b not in bs:
+ bs.append(b)
+ cs.append(2**8+n)
+ n += 1
+ cs = [_chr(n) for n in cs]
+ return dict(zip(bs, cs))
+
+def get_pairs(word):
+ """Return set of symbol pairs in a word.
+
+ Word is represented as tuple of symbols (symbols being variable-length strings).
+ """
+ pairs = set()
+ prev_char = word[0]
+ for char in word[1:]:
+ pairs.add((prev_char, char))
+ prev_char = char
+ return pairs
+
+class GPT2Tokenizer(object):
+ """
+ GPT-2 BPE tokenizer. Peculiarities:
+ - Byte-level BPE
+ """
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
+ """
+ Instantiate a PreTrainedBertModel from a pre-trained model file.
+ Download and cache the pre-trained model file if needed.
+ """
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP:
+ vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path]
+ merges_file = PRETRAINED_MERGES_ARCHIVE_MAP[pretrained_model_name_or_path]
+ special_tokens_file = None
+ else:
+ vocab_file = os.path.join(pretrained_model_name_or_path, VOCAB_NAME)
+ merges_file = os.path.join(pretrained_model_name_or_path, MERGES_NAME)
+ special_tokens_file = os.path.join(pretrained_model_name_or_path, SPECIAL_TOKENS_NAME)
+ if not os.path.exists(special_tokens_file):
+ special_tokens_file = None
+ else:
+ logger.info("loading special tokens file {}".format(special_tokens_file))
+ # redirect to the cache, if necessary
+ try:
+ resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
+ resolved_merges_file = cached_path(merges_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} and {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
+ pretrained_model_name_or_path,
+ vocab_file, merges_file))
+ return None
+ if resolved_vocab_file == vocab_file and resolved_merges_file == merges_file:
+ logger.info("loading vocabulary file {}".format(vocab_file))
+ logger.info("loading merges file {}".format(merges_file))
+ else:
+ logger.info("loading vocabulary file {} from cache at {}".format(
+ vocab_file, resolved_vocab_file))
+ logger.info("loading merges file {} from cache at {}".format(
+ merges_file, resolved_merges_file))
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP:
+ # if we're using a pretrained model, ensure the tokenizer wont index sequences longer
+ # than the number of positional embeddings
+ max_len = PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP[pretrained_model_name_or_path]
+ kwargs['max_len'] = min(kwargs.get('max_len', int(1e12)), max_len)
+ # Instantiate tokenizer.
+ if special_tokens_file and 'special_tokens' not in kwargs:
+ special_tokens = open(special_tokens_file, encoding='utf-8').read().split('\n')[:-1]
+ else:
+ special_tokens = kwargs.pop('special_tokens', [])
+ tokenizer = cls(resolved_vocab_file, resolved_merges_file, special_tokens=special_tokens, *inputs, **kwargs)
+ return tokenizer
+
+ def __init__(self, vocab_file, merges_file, errors='replace', special_tokens=None, max_len=None):
+ self.max_len = max_len if max_len is not None else int(1e12)
+ self.encoder = json.load(open(vocab_file))
+ self.decoder = {v:k for k,v in self.encoder.items()}
+ self.errors = errors # how to handle errors in decoding
+ self.byte_encoder = bytes_to_unicode()
+ self.byte_decoder = {v:k for k, v in self.byte_encoder.items()}
+ bpe_data = open(merges_file, encoding='utf-8').read().split('\n')[1:-1]
+ bpe_merges = [tuple(merge.split()) for merge in bpe_data]
+ self.bpe_ranks = dict(zip(bpe_merges, range(len(bpe_merges))))
+ self.cache = {}
+
+ # Should haved added re.IGNORECASE so BPE merges can happen for capitalized versions of contractions
+ self.pat = re.compile(r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+""")
+
+ self.special_tokens = {}
+ self.special_tokens_decoder = {}
+ self.set_special_tokens(special_tokens)
+
+ def __len__(self):
+ return len(self.encoder) + len(self.special_tokens)
+
+ def set_special_tokens(self, special_tokens):
+ """ Add a list of additional tokens to the encoder.
+ The additional tokens are indexed starting from the last index of the
+ current vocabulary in the order of the `special_tokens` list.
+ """
+ if not special_tokens:
+ self.special_tokens = {}
+ self.special_tokens_decoder = {}
+ return
+ self.special_tokens = dict((tok, len(self.encoder) + i) for i, tok in enumerate(special_tokens))
+ self.special_tokens_decoder = {v:k for k, v in self.special_tokens.items()}
+ logger.info("Special tokens {}".format(self.special_tokens))
+
+ def bpe(self, token):
+ if token in self.cache:
+ return self.cache[token]
+ word = tuple(token)
+ pairs = get_pairs(word)
+
+ if not pairs:
+ return token
+
+ while True:
+ bigram = min(pairs, key = lambda pair: self.bpe_ranks.get(pair, float('inf')))
+ if bigram not in self.bpe_ranks:
+ break
+ first, second = bigram
+ new_word = []
+ i = 0
+ while i < len(word):
+ try:
+ j = word.index(first, i)
+ new_word.extend(word[i:j])
+ i = j
+ except:
+ new_word.extend(word[i:])
+ break
+
+ if word[i] == first and i < len(word)-1 and word[i+1] == second:
+ new_word.append(first+second)
+ i += 2
+ else:
+ new_word.append(word[i])
+ i += 1
+ new_word = tuple(new_word)
+ word = new_word
+ if len(word) == 1:
+ break
+ else:
+ pairs = get_pairs(word)
+ word = ' '.join(word)
+ self.cache[token] = word
+ return word
+
+ def tokenize(self, text):
+ """ Tokenize a string. """
+ bpe_tokens = []
+ for token in re.findall(self.pat, text):
+ token = ''.join(self.byte_encoder[ord(b)] for b in token)
+ bpe_tokens.extend(bpe_token for bpe_token in self.bpe(token).split(' '))
+ return bpe_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ """ Converts a sequence of tokens into ids using the vocab. """
+ ids = []
+ if isinstance(tokens, str) or (sys.version_info[0] == 2 and isinstance(tokens, unicode)):
+ if tokens in self.special_tokens:
+ return self.special_tokens[tokens]
+ else:
+ return self.encoder.get(tokens, 0)
+ for token in tokens:
+ if token in self.special_tokens:
+ ids.append(self.special_tokens[token])
+ else:
+ ids.append(self.encoder.get(token, 0))
+ if len(ids) > self.max_len:
+ logger.warning(
+ "Token indices sequence length is longer than the specified maximum "
+ " sequence length for this OpenAI GPT model ({} > {}). Running this"
+ " sequence through the model will result in indexing errors".format(len(ids), self.max_len)
+ )
+ return ids
+
+ def convert_ids_to_tokens(self, ids, skip_special_tokens=False):
+ """Converts a sequence of ids in BPE tokens using the vocab."""
+ tokens = []
+ for i in ids:
+ if i in self.special_tokens_decoder:
+ if not skip_special_tokens:
+ tokens.append(self.special_tokens_decoder[i])
+ else:
+ tokens.append(self.decoder[i])
+ return tokens
+
+ def encode(self, text):
+ return self.convert_tokens_to_ids(self.tokenize(text))
+
+ def decode(self, tokens):
+ text = ''.join([self.decoder[token] for token in tokens])
+ text = bytearray([self.byte_decoder[c] for c in text]).decode('utf-8', errors=self.errors)
+ return text
+
+ def save_vocabulary(self, vocab_path):
+ """Save the tokenizer vocabulary and merge files to a directory."""
+ if not os.path.isdir(vocab_path):
+ logger.error("Vocabulary path ({}) should be a directory".format(vocab_path))
+ return
+ vocab_file = os.path.join(vocab_path, VOCAB_NAME)
+ merge_file = os.path.join(vocab_path, MERGES_NAME)
+ special_tokens_file = os.path.join(vocab_path, SPECIAL_TOKENS_NAME)
+
+ with open(vocab_file, 'w', encoding='utf-8') as f:
+ f.write(json.dumps(self.encoder, ensure_ascii=False))
+
+ index = 0
+ with open(merge_file, "w", encoding="utf-8") as writer:
+ writer.write(u'#version: 0.2\n')
+ for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
+ if index != token_index:
+ logger.warning("Saving vocabulary to {}: BPE merge indices are not consecutive."
+ " Please check that the tokenizer is not corrupted!".format(merge_file))
+ index = token_index
+ writer.write(' '.join(bpe_tokens) + u'\n')
+ index += 1
+
+ index = len(self.encoder)
+ with open(special_tokens_file, 'w', encoding='utf-8') as writer:
+ for token, token_index in sorted(self.special_tokens.items(), key=lambda kv: kv[1]):
+ if index != token_index:
+ logger.warning("Saving special tokens vocabulary to {}: BPE indices are not consecutive."
+ " Please check that the tokenizer is not corrupted!".format(special_tokens_file))
+ index = token_index
+ writer.write(token + u'\n')
+ index += 1
+
+ return vocab_file, merge_file, special_tokens_file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_openai.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_openai.py
new file mode 100644
index 0000000000000000000000000000000000000000..214a476ce9604aeb787003d392598aae0ce3821d
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_openai.py
@@ -0,0 +1,313 @@
+# coding=utf-8
+# Copyright 2018 The Open AI Team Authors and The HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Tokenization classes for OpenAI GPT."""
+from __future__ import (absolute_import, division, print_function,
+ unicode_literals)
+
+import json
+import logging
+import os
+import re
+import sys
+from io import open
+
+from tqdm import tqdm
+
+from .file_utils import cached_path
+from .tokenization import BasicTokenizer
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_VOCAB_ARCHIVE_MAP = {
+ 'openai-gpt': "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-vocab.json",
+}
+PRETRAINED_MERGES_ARCHIVE_MAP = {
+ 'openai-gpt': "https://s3.amazonaws.com/models.huggingface.co/bert/openai-gpt-merges.txt",
+}
+PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = {
+ 'openai-gpt': 512,
+}
+VOCAB_NAME = 'vocab.json'
+MERGES_NAME = 'merges.txt'
+SPECIAL_TOKENS_NAME = 'special_tokens.txt'
+
+def get_pairs(word):
+ """
+ Return set of symbol pairs in a word.
+ word is represented as tuple of symbols (symbols being variable-length strings)
+ """
+ pairs = set()
+ prev_char = word[0]
+ for char in word[1:]:
+ pairs.add((prev_char, char))
+ prev_char = char
+ return pairs
+
+def text_standardize(text):
+ """
+ fixes some issues the spacy tokenizer had on books corpus
+ also does some whitespace standardization
+ """
+ text = text.replace('—', '-')
+ text = text.replace('–', '-')
+ text = text.replace('―', '-')
+ text = text.replace('…', '...')
+ text = text.replace('´', "'")
+ text = re.sub(r'''(-+|~+|!+|"+|;+|\?+|\++|,+|\)+|\(+|\\+|\/+|\*+|\[+|\]+|}+|{+|\|+|_+)''', r' \1 ', text)
+ text = re.sub(r'\s*\n\s*', ' \n ', text)
+ text = re.sub(r'[^\S\n]+', ' ', text)
+ return text.strip()
+
+class OpenAIGPTTokenizer(object):
+ """
+ BPE tokenizer. Peculiarities:
+ - lower case all inputs
+ - uses SpaCy tokenizer and ftfy for pre-BPE tokenization if they are installed, fallback to BERT's BasicTokenizer if not.
+ - argument special_tokens and function set_special_tokens:
+ can be used to add additional symbols (ex: "__classify__") to a vocabulary.
+ """
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
+ """
+ Instantiate a PreTrainedBertModel from a pre-trained model file.
+ Download and cache the pre-trained model file if needed.
+ """
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP:
+ vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path]
+ merges_file = PRETRAINED_MERGES_ARCHIVE_MAP[pretrained_model_name_or_path]
+ special_tokens_file = None
+ else:
+ vocab_file = os.path.join(pretrained_model_name_or_path, VOCAB_NAME)
+ merges_file = os.path.join(pretrained_model_name_or_path, MERGES_NAME)
+ special_tokens_file = os.path.join(pretrained_model_name_or_path, SPECIAL_TOKENS_NAME)
+ if not os.path.exists(special_tokens_file):
+ special_tokens_file = None
+ else:
+ logger.info("loading special tokens file {}".format(special_tokens_file))
+ # redirect to the cache, if necessary
+ try:
+ resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
+ resolved_merges_file = cached_path(merges_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} and {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
+ pretrained_model_name_or_path,
+ vocab_file, merges_file))
+ return None
+ if resolved_vocab_file == vocab_file and resolved_merges_file == merges_file:
+ logger.info("loading vocabulary file {}".format(vocab_file))
+ logger.info("loading merges file {}".format(merges_file))
+ else:
+ logger.info("loading vocabulary file {} from cache at {}".format(
+ vocab_file, resolved_vocab_file))
+ logger.info("loading merges file {} from cache at {}".format(
+ merges_file, resolved_merges_file))
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP:
+ # if we're using a pretrained model, ensure the tokenizer wont index sequences longer
+ # than the number of positional embeddings
+ max_len = PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP[pretrained_model_name_or_path]
+ kwargs['max_len'] = min(kwargs.get('max_len', int(1e12)), max_len)
+ # Instantiate tokenizer.
+ if special_tokens_file and 'special_tokens' not in kwargs:
+ special_tokens = open(special_tokens_file, encoding='utf-8').read().split('\n')[:-1]
+ else:
+ special_tokens = kwargs.pop('special_tokens', [])
+ tokenizer = cls(resolved_vocab_file, resolved_merges_file, special_tokens=special_tokens, *inputs, **kwargs)
+ return tokenizer
+
+ def __init__(self, vocab_file, merges_file, special_tokens=None, max_len=None):
+ try:
+ import ftfy
+ import spacy
+ self.nlp = spacy.load('en', disable=['parser', 'tagger', 'ner', 'textcat'])
+ self.fix_text = ftfy.fix_text
+ except ImportError:
+ logger.warning("ftfy or spacy is not installed using BERT BasicTokenizer instead of SpaCy & ftfy.")
+ self.nlp = BasicTokenizer(do_lower_case=True,
+ never_split=special_tokens if special_tokens is not None else [])
+ self.fix_text = None
+
+ self.max_len = max_len if max_len is not None else int(1e12)
+ self.encoder = json.load(open(vocab_file, encoding="utf-8"))
+ self.decoder = {v:k for k,v in self.encoder.items()}
+ merges = open(merges_file, encoding='utf-8').read().split('\n')[1:-1]
+ merges = [tuple(merge.split()) for merge in merges]
+ self.bpe_ranks = dict(zip(merges, range(len(merges))))
+ self.cache = {}
+ self.special_tokens = {}
+ self.special_tokens_decoder = {}
+ self.set_special_tokens(special_tokens)
+
+ def __len__(self):
+ return len(self.encoder) + len(self.special_tokens)
+
+ def set_special_tokens(self, special_tokens):
+ """ Add a list of additional tokens to the encoder.
+ The additional tokens are indexed starting from the last index of the
+ current vocabulary in the order of the `special_tokens` list.
+ """
+ if not special_tokens:
+ self.special_tokens = {}
+ self.special_tokens_decoder = {}
+ return
+ self.special_tokens = dict((tok, len(self.encoder) + i) for i, tok in enumerate(special_tokens))
+ self.special_tokens_decoder = {v:k for k, v in self.special_tokens.items()}
+ if self.fix_text is None:
+ # Using BERT's BasicTokenizer: we can update the tokenizer
+ self.nlp.never_split = special_tokens
+ logger.info("Special tokens {}".format(self.special_tokens))
+
+ def bpe(self, token):
+ word = tuple(token[:-1]) + (token[-1] + '',)
+ if token in self.cache:
+ return self.cache[token]
+ pairs = get_pairs(word)
+
+ if not pairs:
+ return token+''
+
+ while True:
+ bigram = min(pairs, key=lambda pair: self.bpe_ranks.get(pair, float('inf')))
+ if bigram not in self.bpe_ranks:
+ break
+ first, second = bigram
+ new_word = []
+ i = 0
+ while i < len(word):
+ try:
+ j = word.index(first, i)
+ new_word.extend(word[i:j])
+ i = j
+ except:
+ new_word.extend(word[i:])
+ break
+
+ if word[i] == first and i < len(word)-1 and word[i+1] == second:
+ new_word.append(first+second)
+ i += 2
+ else:
+ new_word.append(word[i])
+ i += 1
+ new_word = tuple(new_word)
+ word = new_word
+ if len(word) == 1:
+ break
+ else:
+ pairs = get_pairs(word)
+ word = ' '.join(word)
+ if word == '\n ':
+ word = '\n'
+ self.cache[token] = word
+ return word
+
+ def tokenize(self, text):
+ """ Tokenize a string. """
+ split_tokens = []
+ if self.fix_text is None:
+ # Using BERT's BasicTokenizer
+ text = self.nlp.tokenize(text)
+ for token in text:
+ split_tokens.extend([t for t in self.bpe(token).split(' ')])
+ else:
+ # Using SpaCy & ftfy (original tokenization process of OpenAI GPT)
+ text = self.nlp(text_standardize(self.fix_text(text)))
+ for token in text:
+ split_tokens.extend([t for t in self.bpe(token.text.lower()).split(' ')])
+ return split_tokens
+
+ def convert_tokens_to_ids(self, tokens):
+ """ Converts a sequence of tokens into ids using the vocab. """
+ ids = []
+ if isinstance(tokens, str) or (sys.version_info[0] == 2 and isinstance(tokens, unicode)):
+ if tokens in self.special_tokens:
+ return self.special_tokens[tokens]
+ else:
+ return self.encoder.get(tokens, 0)
+ for token in tokens:
+ if token in self.special_tokens:
+ ids.append(self.special_tokens[token])
+ else:
+ ids.append(self.encoder.get(token, 0))
+ if len(ids) > self.max_len:
+ logger.warning(
+ "Token indices sequence length is longer than the specified maximum "
+ " sequence length for this OpenAI GPT model ({} > {}). Running this"
+ " sequence through the model will result in indexing errors".format(len(ids), self.max_len)
+ )
+ return ids
+
+ def convert_ids_to_tokens(self, ids, skip_special_tokens=False):
+ """Converts a sequence of ids in BPE tokens using the vocab."""
+ tokens = []
+ for i in ids:
+ if i in self.special_tokens_decoder:
+ if not skip_special_tokens:
+ tokens.append(self.special_tokens_decoder[i])
+ else:
+ tokens.append(self.decoder[i])
+ return tokens
+
+ def encode(self, text):
+ return self.convert_tokens_to_ids(self.tokenize(text))
+
+ def decode(self, ids, skip_special_tokens=False, clean_up_tokenization_spaces=True):
+ """Converts a sequence of ids in a string."""
+ tokens = self.convert_ids_to_tokens(ids, skip_special_tokens=skip_special_tokens)
+ out_string = ''.join(tokens).replace('', ' ').strip()
+ if clean_up_tokenization_spaces:
+ out_string = out_string.replace('', '')
+ out_string = out_string.replace(' .', '.').replace(' ?', '?').replace(' !', '!').replace(' ,', ',').replace(' ,', ','
+ ).replace(" ' ", "'").replace(" n't", "n't").replace(" 'm", "'m").replace(" do not", " don't"
+ ).replace(" 's", "'s").replace(" 've", "'ve").replace(" 're", "'re")
+ return out_string
+
+ def save_vocabulary(self, vocab_path):
+ """Save the tokenizer vocabulary and merge files to a directory."""
+ if not os.path.isdir(vocab_path):
+ logger.error("Vocabulary path ({}) should be a directory".format(vocab_path))
+ return
+ vocab_file = os.path.join(vocab_path, VOCAB_NAME)
+ merge_file = os.path.join(vocab_path, MERGES_NAME)
+ special_tokens_file = os.path.join(vocab_path, SPECIAL_TOKENS_NAME)
+
+ with open(vocab_file, 'w', encoding='utf-8') as f:
+ f.write(json.dumps(self.encoder, ensure_ascii=False))
+
+ index = 0
+ with open(merge_file, "w", encoding="utf-8") as writer:
+ writer.write(u'#version: 0.2\n')
+ for bpe_tokens, token_index in sorted(self.bpe_ranks.items(), key=lambda kv: kv[1]):
+ if index != token_index:
+ logger.warning("Saving vocabulary to {}: BPE merge indices are not consecutive."
+ " Please check that the tokenizer is not corrupted!".format(merge_file))
+ index = token_index
+ writer.write(' '.join(bpe_tokens) + u'\n')
+ index += 1
+
+ index = len(self.encoder)
+ with open(special_tokens_file, 'w', encoding='utf-8') as writer:
+ for token, token_index in sorted(self.special_tokens.items(), key=lambda kv: kv[1]):
+ if index != token_index:
+ logger.warning("Saving special tokens vocabulary to {}: BPE indices are not consecutive."
+ " Please check that the tokenizer is not corrupted!".format(special_tokens_file))
+ index = token_index
+ writer.write(token + u'\n')
+ index += 1
+
+ return vocab_file, merge_file, special_tokens_file
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_transfo_xl.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_transfo_xl.py
new file mode 100644
index 0000000000000000000000000000000000000000..ddebc57c1068e6f6f2dce6bafc0c2f4f6d330865
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/pytorch_pretrained_bert/tokenization_transfo_xl.py
@@ -0,0 +1,586 @@
+# coding=utf-8
+# Copyright 2018 Google AI, Google Brain and Carnegie Mellon University Authors and the HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Tokenization classes for Transformer XL model.
+ Adapted from https://github.com/kimiyoung/transformer-xl.
+"""
+from __future__ import (absolute_import, division, print_function,
+ unicode_literals)
+
+import glob
+import logging
+import os
+import sys
+from collections import Counter, OrderedDict
+from io import open
+import unicodedata
+
+import torch
+import numpy as np
+
+from .file_utils import cached_path
+
+if sys.version_info[0] == 2:
+ import cPickle as pickle
+else:
+ import pickle
+
+
+logger = logging.getLogger(__name__)
+
+PRETRAINED_VOCAB_ARCHIVE_MAP = {
+ 'transfo-xl-wt103': "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-vocab.bin",
+}
+VOCAB_NAME = 'vocab.bin'
+
+PRETRAINED_CORPUS_ARCHIVE_MAP = {
+ 'transfo-xl-wt103': "https://s3.amazonaws.com/models.huggingface.co/bert/transfo-xl-wt103-corpus.bin",
+}
+CORPUS_NAME = 'corpus.bin'
+
+class TransfoXLTokenizer(object):
+ """
+ Transformer-XL tokenizer adapted from Vocab class in https://github.com/kimiyoung/transformer-xl
+ """
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
+ """
+ Instantiate a TransfoXLTokenizer.
+ The TransfoXLTokenizer.
+ """
+ if pretrained_model_name_or_path in PRETRAINED_VOCAB_ARCHIVE_MAP:
+ vocab_file = PRETRAINED_VOCAB_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ if os.path.isdir(pretrained_model_name_or_path):
+ vocab_file = os.path.join(pretrained_model_name_or_path, VOCAB_NAME)
+ else:
+ vocab_file = pretrained_model_name_or_path
+ # redirect to the cache, if necessary
+ try:
+ resolved_vocab_file = cached_path(vocab_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Model name '{}' was not found in model name list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
+ pretrained_model_name_or_path,
+ vocab_file))
+ return None
+ if resolved_vocab_file == vocab_file:
+ logger.info("loading vocabulary file {}".format(vocab_file))
+ else:
+ logger.info("loading vocabulary file {} from cache at {}".format(
+ vocab_file, resolved_vocab_file))
+
+ # Instantiate tokenizer.
+ tokenizer = cls(*inputs, **kwargs)
+ vocab_dict = torch.load(resolved_vocab_file)
+ for key, value in vocab_dict.items():
+ tokenizer.__dict__[key] = value
+ return tokenizer
+
+ def __init__(self, special=[], min_freq=0, max_size=None, lower_case=False,
+ delimiter=None, vocab_file=None, never_split=("", "", "")):
+ self.counter = Counter()
+ self.special = special
+ self.min_freq = min_freq
+ self.max_size = max_size
+ self.lower_case = lower_case
+ self.delimiter = delimiter
+ self.vocab_file = vocab_file
+ self.never_split = never_split
+
+ def count_file(self, path, verbose=False, add_eos=False):
+ if verbose: print('counting file {} ...'.format(path))
+ assert os.path.exists(path)
+
+ sents = []
+ with open(path, 'r', encoding='utf-8') as f:
+ for idx, line in enumerate(f):
+ if verbose and idx > 0 and idx % 500000 == 0:
+ print(' line {}'.format(idx))
+ symbols = self.tokenize(line, add_eos=add_eos)
+ self.counter.update(symbols)
+ sents.append(symbols)
+
+ return sents
+
+ def count_sents(self, sents, verbose=False):
+ """
+ sents : a list of sentences, each a list of tokenized symbols
+ """
+ if verbose: print('counting {} sents ...'.format(len(sents)))
+ for idx, symbols in enumerate(sents):
+ if verbose and idx > 0 and idx % 500000 == 0:
+ print(' line {}'.format(idx))
+ self.counter.update(symbols)
+
+ def _build_from_file(self, vocab_file):
+ self.idx2sym = []
+ self.sym2idx = OrderedDict()
+
+ with open(vocab_file, 'r', encoding='utf-8') as f:
+ for line in f:
+ symb = line.strip().split()[0]
+ self.add_symbol(symb)
+ if '' in self.sym2idx:
+ self.unk_idx = self.sym2idx['']
+ elif '' in self.sym2idx:
+ self.unk_idx = self.sym2idx['']
+ else:
+ raise ValueError('No token in vocabulary')
+
+ def save_vocabulary(self, vocab_path):
+ """Save the tokenizer vocabulary to a directory or file."""
+ index = 0
+ if os.path.isdir(vocab_path):
+ vocab_file = os.path.join(vocab_path, VOCAB_NAME)
+ torch.save(self.__dict__, vocab_file)
+ return vocab_file
+
+ def build_vocab(self):
+ if self.vocab_file:
+ print('building vocab from {}'.format(self.vocab_file))
+ self._build_from_file(self.vocab_file)
+ print('final vocab size {}'.format(len(self)))
+ else:
+ print('building vocab with min_freq={}, max_size={}'.format(
+ self.min_freq, self.max_size))
+ self.idx2sym = []
+ self.sym2idx = OrderedDict()
+
+ for sym in self.special:
+ self.add_special(sym)
+
+ for sym, cnt in self.counter.most_common(self.max_size):
+ if cnt < self.min_freq: break
+ self.add_symbol(sym)
+
+ print('final vocab size {} from {} unique tokens'.format(
+ len(self), len(self.counter)))
+
+ def encode_file(self, path, ordered=False, verbose=False, add_eos=True,
+ add_double_eos=False):
+ if verbose: print('encoding file {} ...'.format(path))
+ assert os.path.exists(path)
+ encoded = []
+ with open(path, 'r', encoding='utf-8') as f:
+ for idx, line in enumerate(f):
+ if verbose and idx > 0 and idx % 500000 == 0:
+ print(' line {}'.format(idx))
+ symbols = self.tokenize(line, add_eos=add_eos,
+ add_double_eos=add_double_eos)
+ encoded.append(self.convert_to_tensor(symbols))
+
+ if ordered:
+ encoded = torch.cat(encoded)
+
+ return encoded
+
+ def encode_sents(self, sents, ordered=False, verbose=False):
+ if verbose: print('encoding {} sents ...'.format(len(sents)))
+ encoded = []
+ for idx, symbols in enumerate(sents):
+ if verbose and idx > 0 and idx % 500000 == 0:
+ print(' line {}'.format(idx))
+ encoded.append(self.convert_to_tensor(symbols))
+
+ if ordered:
+ encoded = torch.cat(encoded)
+
+ return encoded
+
+ def add_special(self, sym):
+ if sym not in self.sym2idx:
+ self.idx2sym.append(sym)
+ self.sym2idx[sym] = len(self.idx2sym) - 1
+ setattr(self, '{}_idx'.format(sym.strip('<>')), self.sym2idx[sym])
+
+ def add_symbol(self, sym):
+ if sym not in self.sym2idx:
+ self.idx2sym.append(sym)
+ self.sym2idx[sym] = len(self.idx2sym) - 1
+
+ def get_sym(self, idx):
+ assert 0 <= idx < len(self), 'Index {} out of vocabulary range'.format(idx)
+ return self.idx2sym[idx]
+
+ def get_idx(self, sym):
+ if sym in self.sym2idx:
+ return self.sym2idx[sym]
+ else:
+ # print('encounter unk {}'.format(sym))
+ # assert '' not in sym
+ if hasattr(self, 'unk_idx'):
+ return self.sym2idx.get(sym, self.unk_idx)
+ # Backward compatibility with pre-trained models
+ elif '' in self.sym2idx:
+ return self.sym2idx['']
+ elif '' in self.sym2idx:
+ return self.sym2idx['']
+ else:
+ raise ValueError('Token not in vocabulary and no token in vocabulary for replacement')
+
+ def convert_ids_to_tokens(self, indices):
+ """Converts a sequence of indices in symbols using the vocab."""
+ return [self.get_sym(idx) for idx in indices]
+
+ def convert_tokens_to_ids(self, symbols):
+ """Converts a sequence of symbols into ids using the vocab."""
+ return [self.get_idx(sym) for sym in symbols]
+
+ def convert_to_tensor(self, symbols):
+ return torch.LongTensor(self.convert_tokens_to_ids(symbols))
+
+ def decode(self, indices, exclude=None):
+ """Converts a sequence of indices in a string."""
+ if exclude is None:
+ return ' '.join([self.get_sym(idx) for idx in indices])
+ else:
+ return ' '.join([self.get_sym(idx) for idx in indices if idx not in exclude])
+
+ def __len__(self):
+ return len(self.idx2sym)
+
+ def tokenize(self, line, add_eos=False, add_double_eos=False):
+ line = line.strip()
+ # convert to lower case
+ if self.lower_case:
+ line = line.lower()
+
+ # empty delimiter '' will evaluate False
+ if self.delimiter == '':
+ symbols = line
+ else:
+ symbols = line.split(self.delimiter)
+
+ if add_double_eos: # lm1b
+ return [''] + symbols + ['']
+ elif add_eos:
+ return symbols + ['']
+ else:
+ return symbols
+
+
+class LMOrderedIterator(object):
+ def __init__(self, data, bsz, bptt, device='cpu', ext_len=None):
+ """
+ data -- LongTensor -- the LongTensor is strictly ordered
+ """
+ self.bsz = bsz
+ self.bptt = bptt
+ self.ext_len = ext_len if ext_len is not None else 0
+
+ self.device = device
+
+ # Work out how cleanly we can divide the dataset into bsz parts.
+ self.n_step = data.size(0) // bsz
+
+ # Trim off any extra elements that wouldn't cleanly fit (remainders).
+ data = data.narrow(0, 0, self.n_step * bsz)
+
+ # Evenly divide the data across the bsz batches.
+ self.data = data.view(bsz, -1).t().contiguous().to(device)
+
+ # Number of mini-batches
+ self.n_batch = (self.n_step + self.bptt - 1) // self.bptt
+
+ def get_batch(self, i, bptt=None):
+ if bptt is None: bptt = self.bptt
+ seq_len = min(bptt, self.data.size(0) - 1 - i)
+
+ end_idx = i + seq_len
+ beg_idx = max(0, i - self.ext_len)
+
+ data = self.data[beg_idx:end_idx]
+ target = self.data[i+1:i+1+seq_len]
+
+ data_out = data.transpose(0, 1).contiguous().to(self.device)
+ target_out = target.transpose(0, 1).contiguous().to(self.device)
+
+ return data_out, target_out, seq_len
+
+ def get_fixlen_iter(self, start=0):
+ for i in range(start, self.data.size(0) - 1, self.bptt):
+ yield self.get_batch(i)
+
+ def get_varlen_iter(self, start=0, std=5, min_len=5, max_deviation=3):
+ max_len = self.bptt + max_deviation * std
+ i = start
+ while True:
+ bptt = self.bptt if np.random.random() < 0.95 else self.bptt / 2.
+ bptt = min(max_len, max(min_len, int(np.random.normal(bptt, std))))
+ data, target, seq_len = self.get_batch(i, bptt)
+ i += seq_len
+ yield data, target, seq_len
+ if i >= self.data.size(0) - 2:
+ break
+
+ def __iter__(self):
+ return self.get_fixlen_iter()
+
+
+class LMShuffledIterator(object):
+ def __init__(self, data, bsz, bptt, device='cpu', ext_len=None, shuffle=False):
+ """
+ data -- list[LongTensor] -- there is no order among the LongTensors
+ """
+ self.data = data
+
+ self.bsz = bsz
+ self.bptt = bptt
+ self.ext_len = ext_len if ext_len is not None else 0
+
+ self.device = device
+ self.shuffle = shuffle
+
+ def get_sent_stream(self):
+ # index iterator
+ epoch_indices = np.random.permutation(len(self.data)) if self.shuffle \
+ else np.array(range(len(self.data)))
+
+ # sentence iterator
+ for idx in epoch_indices:
+ yield self.data[idx]
+
+ def stream_iterator(self, sent_stream):
+ # streams for each data in the batch
+ streams = [None] * self.bsz
+
+ data = torch.LongTensor(self.bptt, self.bsz)
+ target = torch.LongTensor(self.bptt, self.bsz)
+
+ n_retain = 0
+
+ while True:
+ # data : [n_retain+bptt x bsz]
+ # target : [bptt x bsz]
+ data[n_retain:].fill_(-1)
+ target.fill_(-1)
+
+ valid_batch = True
+
+ for i in range(self.bsz):
+ n_filled = 0
+ try:
+ while n_filled < self.bptt:
+ if streams[i] is None or len(streams[i]) <= 1:
+ streams[i] = next(sent_stream)
+ # number of new tokens to fill in
+ n_new = min(len(streams[i]) - 1, self.bptt - n_filled)
+ # first n_retain tokens are retained from last batch
+ data[n_retain+n_filled:n_retain+n_filled+n_new, i] = \
+ streams[i][:n_new]
+ target[n_filled:n_filled+n_new, i] = \
+ streams[i][1:n_new+1]
+ streams[i] = streams[i][n_new:]
+ n_filled += n_new
+ except StopIteration:
+ valid_batch = False
+ break
+
+ if not valid_batch:
+ return
+
+ data_out = data.transpose(0, 1).contiguous().to(self.device)
+ target_out = target.transpose(0, 1).contiguous().to(self.device)
+
+ yield data_out, target_out, self.bptt
+
+ n_retain = min(data.size(0), self.ext_len)
+ if n_retain > 0:
+ data[:n_retain] = data[-n_retain:]
+ data.resize_(n_retain + self.bptt, data.size(1))
+
+ def __iter__(self):
+ # sent_stream is an iterator
+ sent_stream = self.get_sent_stream()
+
+ for batch in self.stream_iterator(sent_stream):
+ yield batch
+
+
+class LMMultiFileIterator(LMShuffledIterator):
+ def __init__(self, paths, vocab, bsz, bptt, device='cpu', ext_len=None,
+ shuffle=False):
+
+ self.paths = paths
+ self.vocab = vocab
+
+ self.bsz = bsz
+ self.bptt = bptt
+ self.ext_len = ext_len if ext_len is not None else 0
+
+ self.device = device
+ self.shuffle = shuffle
+
+ def get_sent_stream(self, path):
+ sents = self.vocab.encode_file(path, add_double_eos=True)
+ if self.shuffle:
+ np.random.shuffle(sents)
+ sent_stream = iter(sents)
+
+ return sent_stream
+
+ def __iter__(self):
+ if self.shuffle:
+ np.random.shuffle(self.paths)
+
+ for path in self.paths:
+ # sent_stream is an iterator
+ sent_stream = self.get_sent_stream(path)
+ for batch in self.stream_iterator(sent_stream):
+ yield batch
+
+
+class TransfoXLCorpus(object):
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, cache_dir=None, *inputs, **kwargs):
+ """
+ Instantiate a pre-processed corpus.
+ """
+ vocab = TransfoXLTokenizer.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
+ if pretrained_model_name_or_path in PRETRAINED_CORPUS_ARCHIVE_MAP:
+ corpus_file = PRETRAINED_CORPUS_ARCHIVE_MAP[pretrained_model_name_or_path]
+ else:
+ corpus_file = os.path.join(pretrained_model_name_or_path, CORPUS_NAME)
+ # redirect to the cache, if necessary
+ try:
+ resolved_corpus_file = cached_path(corpus_file, cache_dir=cache_dir)
+ except EnvironmentError:
+ logger.error(
+ "Corpus '{}' was not found in corpus list ({}). "
+ "We assumed '{}' was a path or url but couldn't find files {} "
+ "at this path or url.".format(
+ pretrained_model_name_or_path,
+ ', '.join(PRETRAINED_VOCAB_ARCHIVE_MAP.keys()),
+ pretrained_model_name_or_path,
+ corpus_file))
+ return None
+ if resolved_corpus_file == corpus_file:
+ logger.info("loading corpus file {}".format(corpus_file))
+ else:
+ logger.info("loading corpus file {} from cache at {}".format(
+ corpus_file, resolved_corpus_file))
+
+ # Instantiate tokenizer.
+ corpus = cls(*inputs, **kwargs)
+ corpus_dict = torch.load(resolved_corpus_file)
+ for key, value in corpus_dict.items():
+ corpus.__dict__[key] = value
+ corpus.vocab = vocab
+ if corpus.train is not None:
+ corpus.train = torch.tensor(corpus.train, dtype=torch.long)
+ if corpus.valid is not None:
+ corpus.valid = torch.tensor(corpus.valid, dtype=torch.long)
+ if corpus.test is not None:
+ corpus.test = torch.tensor(corpus.test, dtype=torch.long)
+ return corpus
+
+ def __init__(self, *args, **kwargs):
+ self.vocab = TransfoXLTokenizer(*args, **kwargs)
+ self.dataset = None
+ self.train = None
+ self.valid = None
+ self.test = None
+
+ def build_corpus(self, path, dataset):
+ self.dataset = dataset
+
+ if self.dataset in ['ptb', 'wt2', 'enwik8', 'text8']:
+ self.vocab.count_file(os.path.join(path, 'train.txt'))
+ self.vocab.count_file(os.path.join(path, 'valid.txt'))
+ self.vocab.count_file(os.path.join(path, 'test.txt'))
+ elif self.dataset == 'wt103':
+ self.vocab.count_file(os.path.join(path, 'train.txt'))
+ elif self.dataset == 'lm1b':
+ train_path_pattern = os.path.join(
+ path, '1-billion-word-language-modeling-benchmark-r13output',
+ 'training-monolingual.tokenized.shuffled', 'news.en-*')
+ train_paths = glob.glob(train_path_pattern)
+ # the vocab will load from file when build_vocab() is called
+
+ self.vocab.build_vocab()
+
+ if self.dataset in ['ptb', 'wt2', 'wt103']:
+ self.train = self.vocab.encode_file(
+ os.path.join(path, 'train.txt'), ordered=True)
+ self.valid = self.vocab.encode_file(
+ os.path.join(path, 'valid.txt'), ordered=True)
+ self.test = self.vocab.encode_file(
+ os.path.join(path, 'test.txt'), ordered=True)
+ elif self.dataset in ['enwik8', 'text8']:
+ self.train = self.vocab.encode_file(
+ os.path.join(path, 'train.txt'), ordered=True, add_eos=False)
+ self.valid = self.vocab.encode_file(
+ os.path.join(path, 'valid.txt'), ordered=True, add_eos=False)
+ self.test = self.vocab.encode_file(
+ os.path.join(path, 'test.txt'), ordered=True, add_eos=False)
+ elif self.dataset == 'lm1b':
+ self.train = train_paths
+ self.valid = self.vocab.encode_file(
+ os.path.join(path, 'valid.txt'), ordered=False, add_double_eos=True)
+ self.test = self.vocab.encode_file(
+ os.path.join(path, 'test.txt'), ordered=False, add_double_eos=True)
+
+ def get_iterator(self, split, *args, **kwargs):
+ if split == 'train':
+ if self.dataset in ['ptb', 'wt2', 'wt103', 'enwik8', 'text8']:
+ data_iter = LMOrderedIterator(self.train, *args, **kwargs)
+ elif self.dataset == 'lm1b':
+ kwargs['shuffle'] = True
+ data_iter = LMMultiFileIterator(self.train, self.vocab, *args, **kwargs)
+ elif split in ['valid', 'test']:
+ data = self.valid if split == 'valid' else self.test
+ if self.dataset in ['ptb', 'wt2', 'wt103', 'enwik8', 'text8']:
+ data_iter = LMOrderedIterator(data, *args, **kwargs)
+ elif self.dataset == 'lm1b':
+ data_iter = LMShuffledIterator(data, *args, **kwargs)
+
+ return data_iter
+
+
+def get_lm_corpus(datadir, dataset):
+ fn = os.path.join(datadir, 'cache.pt')
+ fn_pickle = os.path.join(datadir, 'cache.pkl')
+ if os.path.exists(fn):
+ print('Loading cached dataset...')
+ corpus = torch.load(fn_pickle)
+ elif os.path.exists(fn):
+ print('Loading cached dataset from pickle...')
+ with open(fn, "rb") as fp:
+ corpus = pickle.load(fp)
+ else:
+ print('Producing dataset {}...'.format(dataset))
+ kwargs = {}
+ if dataset in ['wt103', 'wt2']:
+ kwargs['special'] = ['']
+ kwargs['lower_case'] = False
+ elif dataset == 'ptb':
+ kwargs['special'] = ['']
+ kwargs['lower_case'] = True
+ elif dataset == 'lm1b':
+ kwargs['special'] = []
+ kwargs['lower_case'] = False
+ kwargs['vocab_file'] = os.path.join(datadir, '1b_word_vocab.txt')
+ elif dataset in ['enwik8', 'text8']:
+ pass
+
+ corpus = TransfoXLCorpus(datadir, dataset, **kwargs)
+ torch.save(corpus, fn)
+
+ return corpus
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/requirements.txt b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/requirements.txt
new file mode 100644
index 0000000000000000000000000000000000000000..63c4a4a5afd98883d77e7fbf03df5bfb9e1ca2d4
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/requirements.txt
@@ -0,0 +1,14 @@
+# PyTorch
+torch>=0.4.1
+# progress bars in model download and training scripts
+tqdm
+# Accessing files from S3 directly.
+boto3
+# Used for downloading models over HTTP
+requests
+# For OpenAI GPT
+regex
+nltk==3.4
+beautifulsoup4==4.7.1
+spacy==2.1.3
+jupyter==1.0.0
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/input.txt b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/input.txt
new file mode 100644
index 0000000000000000000000000000000000000000..d1e3f410d07833e4c5c233ffd54f8d2b54ebb7cf
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/input.txt
@@ -0,0 +1 @@
+Who was Jim Henson ? ||| Jim Henson was a puppeteer
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/sample_text.txt b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/sample_text.txt
new file mode 100644
index 0000000000000000000000000000000000000000..a42812060c576bae870eb29b1ac083fda0d239d3
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/samples/sample_text.txt
@@ -0,0 +1,33 @@
+This text is included to make sure Unicode is handled properly: 力加勝北区ᴵᴺᵀᵃছজটডণত
+Text should be one-sentence-per-line, with empty lines between documents.
+This sample text is public domain and was randomly selected from Project Guttenberg.
+
+The rain had only ceased with the gray streaks of morning at Blazing Star, and the settlement awoke to a moral sense of cleanliness, and the finding of forgotten knives, tin cups, and smaller camp utensils, where the heavy showers had washed away the debris and dust heaps before the cabin doors.
+Indeed, it was recorded in Blazing Star that a fortunate early riser had once picked up on the highway a solid chunk of gold quartz which the rain had freed from its incumbering soil, and washed into immediate and glittering popularity.
+Possibly this may have been the reason why early risers in that locality, during the rainy season, adopted a thoughtful habit of body, and seldom lifted their eyes to the rifted or india-ink washed skies above them.
+"Cass" Beard had risen early that morning, but not with a view to discovery.
+A leak in his cabin roof,--quite consistent with his careless, improvident habits,--had roused him at 4 A. M., with a flooded "bunk" and wet blankets.
+The chips from his wood pile refused to kindle a fire to dry his bed-clothes, and he had recourse to a more provident neighbor's to supply the deficiency.
+This was nearly opposite.
+Mr. Cassius crossed the highway, and stopped suddenly.
+Something glittered in the nearest red pool before him.
+Gold, surely!
+But, wonderful to relate, not an irregular, shapeless fragment of crude ore, fresh from Nature's crucible, but a bit of jeweler's handicraft in the form of a plain gold ring.
+Looking at it more attentively, he saw that it bore the inscription, "May to Cass."
+Like most of his fellow gold-seekers, Cass was superstitious.
+
+The fountain of classic wisdom, Hypatia herself.
+As the ancient sage--the name is unimportant to a monk--pumped water nightly that he might study by day, so I, the guardian of cloaks and parasols, at the sacred doors of her lecture-room, imbibe celestial knowledge.
+From my youth I felt in me a soul above the matter-entangled herd.
+She revealed to me the glorious fact, that I am a spark of Divinity itself.
+A fallen star, I am, sir!' continued he, pensively, stroking his lean stomach--'a fallen star!--fallen, if the dignity of philosophy will allow of the simile, among the hogs of the lower world--indeed, even into the hog-bucket itself. Well, after all, I will show you the way to the Archbishop's.
+There is a philosophic pleasure in opening one's treasures to the modest young.
+Perhaps you will assist me by carrying this basket of fruit?' And the little man jumped up, put his basket on Philammon's head, and trotted off up a neighbouring street.
+Philammon followed, half contemptuous, half wondering at what this philosophy might be, which could feed the self-conceit of anything so abject as his ragged little apish guide;
+but the novel roar and whirl of the street, the perpetual stream of busy faces, the line of curricles, palanquins, laden asses, camels, elephants, which met and passed him, and squeezed him up steps and into doorways, as they threaded their way through the great Moon-gate into the ample street beyond, drove everything from his mind but wondering curiosity, and a vague, helpless dread of that great living wilderness, more terrible than any dead wilderness of sand which he had left behind.
+Already he longed for the repose, the silence of the Laura--for faces which knew him and smiled upon him; but it was too late to turn back now.
+His guide held on for more than a mile up the great main street, crossed in the centre of the city, at right angles, by one equally magnificent, at each end of which, miles away, appeared, dim and distant over the heads of the living stream of passengers, the yellow sand-hills of the desert;
+while at the end of the vista in front of them gleamed the blue harbour, through a network of countless masts.
+At last they reached the quay at the opposite end of the street;
+and there burst on Philammon's astonished eyes a vast semicircle of blue sea, ringed with palaces and towers.
+He stopped involuntarily; and his little guide stopped also, and looked askance at the young monk, to watch the effect which that grand panorama should produce on him.
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/setup.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/setup.py
new file mode 100644
index 0000000000000000000000000000000000000000..fe7990447df4440bc44cfa3c24482f42e5f894f6
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/setup.py
@@ -0,0 +1,71 @@
+"""
+Simple check list from AllenNLP repo: https://github.com/allenai/allennlp/blob/master/setup.py
+
+To create the package for pypi.
+
+1. Change the version in __init__.py and setup.py.
+
+2. Commit these changes with the message: "Release: VERSION"
+
+3. Add a tag in git to mark the release: "git tag VERSION -m'Adds tag VERSION for pypi' "
+ Push the tag to git: git push --tags origin master
+
+4. Build both the sources and the wheel. Do not change anything in setup.py between
+ creating the wheel and the source distribution (obviously).
+
+ For the wheel, run: "python setup.py bdist_wheel" in the top level allennlp directory.
+ (this will build a wheel for the python version you use to build it - make sure you use python 3.x).
+
+ For the sources, run: "python setup.py sdist"
+ You should now have a /dist directory with both .whl and .tar.gz source versions of allennlp.
+
+5. Check that everything looks correct by uploading the package to the pypi test server:
+
+ twine upload dist/* -r pypitest
+ (pypi suggest using twine as other methods upload files via plaintext.)
+
+ Check that you can install it in a virtualenv by running:
+ pip install -i https://testpypi.python.org/pypi allennlp
+
+6. Upload the final version to actual pypi:
+ twine upload dist/* -r pypi
+
+7. Copy the release notes from RELEASE.md to the tag in github once everything is looking hunky-dory.
+
+"""
+from io import open
+from setuptools import find_packages, setup
+
+setup(
+ name="pytorch_pretrained_bert",
+ version="0.6.2",
+ author="Thomas Wolf, Victor Sanh, Tim Rault, Google AI Language Team Authors, Open AI team Authors",
+ author_email="thomas@huggingface.co",
+ description="PyTorch version of Google AI BERT model with script to load Google pre-trained models",
+ long_description=open("README.md", "r", encoding='utf-8').read(),
+ long_description_content_type="text/markdown",
+ keywords='BERT NLP deep learning google',
+ license='Apache',
+ url="https://github.com/huggingface/pytorch-pretrained-BERT",
+ packages=find_packages(exclude=["*.tests", "*.tests.*",
+ "tests.*", "tests"]),
+ install_requires=['torch>=0.4.1',
+ 'numpy',
+ 'boto3',
+ 'requests',
+ 'tqdm',
+ 'regex'],
+ entry_points={
+ 'console_scripts': [
+ "pytorch_pretrained_bert=pytorch_pretrained_bert.__main__:main",
+ ]
+ },
+ # python_requires='>=3.5.0',
+ tests_require=['pytest'],
+ classifiers=[
+ 'Intended Audience :: Science/Research',
+ 'License :: OSI Approved :: Apache Software License',
+ 'Programming Language :: Python :: 3',
+ 'Topic :: Scientific/Engineering :: Artificial Intelligence',
+ ],
+)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/conftest.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/conftest.py
new file mode 100644
index 0000000000000000000000000000000000000000..841ebc8df9ee11b40fa2b8dc7fbe5e9004fcee70
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/conftest.py
@@ -0,0 +1,19 @@
+# content of conftest.py
+
+import pytest
+
+
+def pytest_addoption(parser):
+ parser.addoption(
+ "--runslow", action="store_true", default=False, help="run slow tests"
+ )
+
+
+def pytest_collection_modifyitems(config, items):
+ if config.getoption("--runslow"):
+ # --runslow given in cli: do not skip slow tests
+ return
+ skip_slow = pytest.mark.skip(reason="need --runslow option to run")
+ for item in items:
+ if "slow" in item.keywords:
+ item.add_marker(skip_slow)
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_gpt2_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_gpt2_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..8f4581b37f847333175c53a65fc9d0dedc39ac50
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_gpt2_test.py
@@ -0,0 +1,229 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import unittest
+import json
+import random
+import shutil
+import pytest
+
+import torch
+
+from pytorch_pretrained_bert import (GPT2Config, GPT2Model,
+ GPT2LMHeadModel, GPT2DoubleHeadsModel)
+from pytorch_pretrained_bert.modeling_gpt2 import PRETRAINED_MODEL_ARCHIVE_MAP
+
+class GPT2ModelTest(unittest.TestCase):
+ class GPT2ModelTester(object):
+
+ def __init__(self,
+ parent,
+ batch_size=13,
+ seq_length=7,
+ is_training=True,
+ use_position_ids=True,
+ use_token_type_ids=True,
+ use_labels=True,
+ vocab_size=99,
+ n_positions=33,
+ n_embd=32,
+ n_layer=5,
+ n_head=4,
+ n_choices=3,
+ type_sequence_label_size=2,
+ initializer_range=0.02,
+ num_labels=3,
+ scope=None):
+ self.parent = parent
+ self.batch_size = batch_size
+ self.seq_length = seq_length
+ self.is_training = is_training
+ self.use_position_ids = use_position_ids
+ self.use_token_type_ids = use_token_type_ids
+ self.use_labels = use_labels
+ self.vocab_size = vocab_size
+ self.n_positions = n_positions
+ self.n_embd = n_embd
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.n_choices = n_choices
+ self.type_sequence_label_size = type_sequence_label_size
+ self.initializer_range = initializer_range
+ self.num_labels = num_labels
+ self.scope = scope
+
+ def prepare_config_and_inputs(self):
+ input_ids = GPT2ModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.vocab_size)
+
+ position_ids = None
+ if self.use_position_ids:
+ position_ids = GPT2ModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.n_positions)
+
+ token_type_ids = None
+ if self.use_token_type_ids:
+ total_voc = self.vocab_size
+ token_type_ids = GPT2ModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], total_voc)
+
+ mc_labels = None
+ lm_labels = None
+ mc_token_ids = None
+ if self.use_labels:
+ mc_labels = GPT2ModelTest.ids_tensor([self.batch_size], self.type_sequence_label_size)
+ lm_labels = GPT2ModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.num_labels)
+ mc_token_ids = GPT2ModelTest.ids_tensor([self.batch_size, self.n_choices], self.seq_length)
+
+ config = GPT2Config(
+ vocab_size_or_config_json_file=self.vocab_size,
+ n_positions=self.n_positions,
+ n_embd=self.n_embd,
+ n_layer=self.n_layer,
+ n_head=self.n_head,
+ initializer_range=self.initializer_range)
+
+ return (config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids)
+
+ def create_gpt2_model(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = GPT2Model(config)
+ model.eval()
+ hidden_states, presents = model(input_ids, position_ids, token_type_ids)
+ outputs = {
+ "hidden_states": hidden_states,
+ "presents": presents,
+ }
+ return outputs
+
+ def check_gpt2_model_output(self, result):
+ self.parent.assertListEqual(
+ list(result["hidden_states"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, self.n_embd])
+
+
+ def create_gpt2_lm_head(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = GPT2LMHeadModel(config)
+ model.eval()
+ loss = model(input_ids, position_ids, token_type_ids, lm_labels)
+ lm_logits, presents = model(input_ids, position_ids, token_type_ids)
+ outputs = {
+ "loss": loss,
+ "lm_logits": lm_logits,
+ "presents": presents,
+ }
+ return outputs
+
+ def check_gpt2_lm_head_output(self, result):
+ total_voc = self.vocab_size
+ self.parent.assertListEqual(
+ list(result["lm_logits"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, total_voc])
+
+ def check_gpt2_lm_head_loss_output(self, result):
+ self.parent.assertListEqual(
+ list(result["loss"].size()),
+ [])
+
+ def create_gpt2_double_heads(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = GPT2DoubleHeadsModel(config)
+ model.eval()
+ loss = model(input_ids, mc_token_ids,
+ lm_labels=lm_labels, mc_labels=mc_labels,
+ token_type_ids=token_type_ids, position_ids=position_ids)
+ lm_logits, mc_logits, presents = model(input_ids, mc_token_ids, position_ids=position_ids, token_type_ids=token_type_ids)
+ outputs = {
+ "loss": loss,
+ "lm_logits": lm_logits,
+ "mc_logits": mc_logits,
+ "presents": presents,
+ }
+ return outputs
+
+ def check_gpt2_double_heads_output(self, result):
+ total_voc = self.vocab_size
+ self.parent.assertListEqual(
+ list(result["lm_logits"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, total_voc])
+ self.parent.assertListEqual(
+ list(result["mc_logits"].size()),
+ [self.batch_size, self.n_choices])
+
+ def check_gpt2_double_heads_loss_output(self, result):
+ self.parent.assertListEqual(
+ [list(l.size()) for l in result["loss"]],
+ [[], []])
+
+ def test_default(self):
+ self.run_tester(GPT2ModelTest.GPT2ModelTester(self))
+
+ def test_config_to_json_string(self):
+ config = GPT2Config(vocab_size_or_config_json_file=99, n_embd=37)
+ obj = json.loads(config.to_json_string())
+ self.assertEqual(obj["vocab_size"], 99)
+ self.assertEqual(obj["n_embd"], 37)
+
+ def test_config_to_json_file(self):
+ config_first = GPT2Config(vocab_size_or_config_json_file=99, n_embd=37)
+ json_file_path = "/tmp/config.json"
+ config_first.to_json_file(json_file_path)
+ config_second = GPT2Config.from_json_file(json_file_path)
+ os.remove(json_file_path)
+ self.assertEqual(config_second.to_dict(), config_first.to_dict())
+
+ @pytest.mark.slow
+ def test_model_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+ model = GPT2Model.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(model)
+
+ def run_tester(self, tester):
+ config_and_inputs = tester.prepare_config_and_inputs()
+ output_result = tester.create_gpt2_model(*config_and_inputs)
+ tester.check_gpt2_model_output(output_result)
+
+ output_result = tester.create_gpt2_lm_head(*config_and_inputs)
+ tester.check_gpt2_lm_head_output(output_result)
+ tester.check_gpt2_lm_head_loss_output(output_result)
+
+ output_result = tester.create_gpt2_double_heads(*config_and_inputs)
+ tester.check_gpt2_double_heads_output(output_result)
+ tester.check_gpt2_double_heads_loss_output(output_result)
+
+ @classmethod
+ def ids_tensor(cls, shape, vocab_size, rng=None, name=None):
+ """Creates a random int32 tensor of the shape within the vocab size."""
+ if rng is None:
+ rng = random.Random()
+
+ total_dims = 1
+ for dim in shape:
+ total_dims *= dim
+
+ values = []
+ for _ in range(total_dims):
+ values.append(rng.randint(0, vocab_size - 1))
+
+ return torch.tensor(data=values, dtype=torch.long).view(shape).contiguous()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_openai_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_openai_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..4e7d9d542b0ba4a3ff41e5f1a5fcbb4af2da6444
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_openai_test.py
@@ -0,0 +1,241 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import unittest
+import json
+import random
+import shutil
+import pytest
+
+import torch
+
+from pytorch_pretrained_bert import (OpenAIGPTConfig, OpenAIGPTModel,
+ OpenAIGPTLMHeadModel, OpenAIGPTDoubleHeadsModel)
+from pytorch_pretrained_bert.modeling_openai import PRETRAINED_MODEL_ARCHIVE_MAP
+
+class OpenAIGPTModelTest(unittest.TestCase):
+ class OpenAIGPTModelTester(object):
+
+ def __init__(self,
+ parent,
+ batch_size=13,
+ seq_length=7,
+ is_training=True,
+ use_position_ids=True,
+ use_token_type_ids=True,
+ use_labels=True,
+ vocab_size=99,
+ n_special=1,
+ n_positions=33,
+ n_embd=32,
+ n_layer=5,
+ n_head=4,
+ n_choices=3,
+ afn="gelu",
+ resid_pdrop=0.1,
+ attn_pdrop=0.1,
+ embd_pdrop=0.1,
+ type_sequence_label_size=2,
+ initializer_range=0.02,
+ num_labels=3,
+ scope=None):
+ self.parent = parent
+ self.batch_size = batch_size
+ self.seq_length = seq_length
+ self.is_training = is_training
+ self.use_position_ids = use_position_ids
+ self.use_token_type_ids = use_token_type_ids
+ self.use_labels = use_labels
+ self.vocab_size = vocab_size
+ self.n_special = n_special
+ self.n_positions = n_positions
+ self.n_embd = n_embd
+ self.n_layer = n_layer
+ self.n_head = n_head
+ self.afn = afn
+ self.n_choices = n_choices
+ self.resid_pdrop = resid_pdrop
+ self.attn_pdrop = attn_pdrop
+ self.embd_pdrop = embd_pdrop
+ self.type_sequence_label_size = type_sequence_label_size
+ self.initializer_range = initializer_range
+ self.num_labels = num_labels
+ self.scope = scope
+
+ def prepare_config_and_inputs(self):
+ input_ids = OpenAIGPTModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.vocab_size)
+
+ position_ids = None
+ if self.use_position_ids:
+ position_ids = OpenAIGPTModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.n_positions)
+
+ token_type_ids = None
+ if self.use_token_type_ids:
+ total_voc = self.vocab_size + self.n_special
+ token_type_ids = OpenAIGPTModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], total_voc)
+
+ mc_labels = None
+ lm_labels = None
+ mc_token_ids = None
+ if self.use_labels:
+ mc_labels = OpenAIGPTModelTest.ids_tensor([self.batch_size], self.type_sequence_label_size)
+ lm_labels = OpenAIGPTModelTest.ids_tensor([self.batch_size, self.n_choices, self.seq_length], self.num_labels)
+ mc_token_ids = OpenAIGPTModelTest.ids_tensor([self.batch_size, self.n_choices], self.seq_length)
+
+ config = OpenAIGPTConfig(
+ vocab_size_or_config_json_file=self.vocab_size,
+ n_positions=self.n_positions,
+ n_special=self.n_special,
+ n_embd=self.n_embd,
+ n_layer=self.n_layer,
+ n_head=self.n_head,
+ afn=self.afn,
+ resid_pdrop=self.resid_pdrop,
+ attn_pdrop=self.attn_pdrop,
+ embd_pdrop=self.embd_pdrop,
+ initializer_range=self.initializer_range)
+
+ return (config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids)
+
+ def create_openai_model(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = OpenAIGPTModel(config)
+ model.eval()
+ hidden_states = model(input_ids, position_ids, token_type_ids)
+ outputs = {
+ "hidden_states": hidden_states,
+ }
+ return outputs
+
+ def check_openai_model_output(self, result):
+ self.parent.assertListEqual(
+ list(result["hidden_states"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, self.n_embd])
+
+
+ def create_openai_lm_head(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = OpenAIGPTLMHeadModel(config)
+ model.eval()
+ loss = model(input_ids, position_ids, token_type_ids, lm_labels)
+ lm_logits = model(input_ids, position_ids, token_type_ids)
+ outputs = {
+ "loss": loss,
+ "lm_logits": lm_logits,
+ }
+ return outputs
+
+ def check_openai_lm_head_output(self, result):
+ total_voc = self.n_special + self.vocab_size
+ self.parent.assertListEqual(
+ list(result["lm_logits"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, total_voc])
+
+ def check_openai_lm_head_loss_output(self, result):
+ self.parent.assertListEqual(
+ list(result["loss"].size()),
+ [])
+
+ def create_openai_double_heads(self, config, input_ids, token_type_ids, position_ids,
+ mc_labels, lm_labels, mc_token_ids):
+ model = OpenAIGPTDoubleHeadsModel(config)
+ model.eval()
+ loss = model(input_ids, mc_token_ids,
+ lm_labels=lm_labels, mc_labels=mc_labels,
+ token_type_ids=token_type_ids, position_ids=position_ids)
+ lm_logits, mc_logits = model(input_ids, mc_token_ids, position_ids=position_ids, token_type_ids=token_type_ids)
+ outputs = {
+ "loss": loss,
+ "lm_logits": lm_logits,
+ "mc_logits": mc_logits,
+ }
+ return outputs
+
+ def check_openai_double_heads_output(self, result):
+ total_voc = self.n_special + self.vocab_size
+ self.parent.assertListEqual(
+ list(result["lm_logits"].size()),
+ [self.batch_size, self.n_choices, self.seq_length, total_voc])
+ self.parent.assertListEqual(
+ list(result["mc_logits"].size()),
+ [self.batch_size, self.n_choices])
+
+ def check_openai_double_heads_loss_output(self, result):
+ self.parent.assertListEqual(
+ [list(l.size()) for l in result["loss"]],
+ [[], []])
+
+ def test_default(self):
+ self.run_tester(OpenAIGPTModelTest.OpenAIGPTModelTester(self))
+
+ def test_config_to_json_string(self):
+ config = OpenAIGPTConfig(vocab_size_or_config_json_file=99, n_embd=37)
+ obj = json.loads(config.to_json_string())
+ self.assertEqual(obj["vocab_size"], 99)
+ self.assertEqual(obj["n_embd"], 37)
+
+ def test_config_to_json_file(self):
+ config_first = OpenAIGPTConfig(vocab_size_or_config_json_file=99, n_embd=37)
+ json_file_path = "/tmp/config.json"
+ config_first.to_json_file(json_file_path)
+ config_second = OpenAIGPTConfig.from_json_file(json_file_path)
+ os.remove(json_file_path)
+ self.assertEqual(config_second.to_dict(), config_first.to_dict())
+
+ @pytest.mark.slow
+ def test_model_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+ model = OpenAIGPTModel.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(model)
+
+ def run_tester(self, tester):
+ config_and_inputs = tester.prepare_config_and_inputs()
+ output_result = tester.create_openai_model(*config_and_inputs)
+ tester.check_openai_model_output(output_result)
+
+ output_result = tester.create_openai_lm_head(*config_and_inputs)
+ tester.check_openai_lm_head_output(output_result)
+ tester.check_openai_lm_head_loss_output(output_result)
+
+ output_result = tester.create_openai_double_heads(*config_and_inputs)
+ tester.check_openai_double_heads_output(output_result)
+ tester.check_openai_double_heads_loss_output(output_result)
+
+ @classmethod
+ def ids_tensor(cls, shape, vocab_size, rng=None, name=None):
+ """Creates a random int32 tensor of the shape within the vocab size."""
+ if rng is None:
+ rng = random.Random()
+
+ total_dims = 1
+ for dim in shape:
+ total_dims *= dim
+
+ values = []
+ for _ in range(total_dims):
+ values.append(rng.randint(0, vocab_size - 1))
+
+ return torch.tensor(data=values, dtype=torch.long).view(shape).contiguous()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..5cde383fdfe76d04f3692a2e1461df320304c947
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_test.py
@@ -0,0 +1,321 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import unittest
+import json
+import random
+import shutil
+import pytest
+
+import torch
+
+from pytorch_pretrained_bert import (BertConfig, BertModel, BertForMaskedLM,
+ BertForNextSentencePrediction, BertForPreTraining,
+ BertForQuestionAnswering, BertForSequenceClassification,
+ BertForTokenClassification)
+from pytorch_pretrained_bert.modeling import PRETRAINED_MODEL_ARCHIVE_MAP
+
+
+class BertModelTest(unittest.TestCase):
+ class BertModelTester(object):
+
+ def __init__(self,
+ parent,
+ batch_size=13,
+ seq_length=7,
+ is_training=True,
+ use_input_mask=True,
+ use_token_type_ids=True,
+ use_labels=True,
+ vocab_size=99,
+ hidden_size=32,
+ num_hidden_layers=5,
+ num_attention_heads=4,
+ intermediate_size=37,
+ hidden_act="gelu",
+ hidden_dropout_prob=0.1,
+ attention_probs_dropout_prob=0.1,
+ max_position_embeddings=512,
+ type_vocab_size=16,
+ type_sequence_label_size=2,
+ initializer_range=0.02,
+ num_labels=3,
+ scope=None):
+ self.parent = parent
+ self.batch_size = batch_size
+ self.seq_length = seq_length
+ self.is_training = is_training
+ self.use_input_mask = use_input_mask
+ self.use_token_type_ids = use_token_type_ids
+ self.use_labels = use_labels
+ self.vocab_size = vocab_size
+ self.hidden_size = hidden_size
+ self.num_hidden_layers = num_hidden_layers
+ self.num_attention_heads = num_attention_heads
+ self.intermediate_size = intermediate_size
+ self.hidden_act = hidden_act
+ self.hidden_dropout_prob = hidden_dropout_prob
+ self.attention_probs_dropout_prob = attention_probs_dropout_prob
+ self.max_position_embeddings = max_position_embeddings
+ self.type_vocab_size = type_vocab_size
+ self.type_sequence_label_size = type_sequence_label_size
+ self.initializer_range = initializer_range
+ self.num_labels = num_labels
+ self.scope = scope
+
+ def prepare_config_and_inputs(self):
+ input_ids = BertModelTest.ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
+
+ input_mask = None
+ if self.use_input_mask:
+ input_mask = BertModelTest.ids_tensor([self.batch_size, self.seq_length], vocab_size=2)
+
+ token_type_ids = None
+ if self.use_token_type_ids:
+ token_type_ids = BertModelTest.ids_tensor([self.batch_size, self.seq_length], self.type_vocab_size)
+
+ sequence_labels = None
+ token_labels = None
+ if self.use_labels:
+ sequence_labels = BertModelTest.ids_tensor([self.batch_size], self.type_sequence_label_size)
+ token_labels = BertModelTest.ids_tensor([self.batch_size, self.seq_length], self.num_labels)
+
+ config = BertConfig(
+ vocab_size_or_config_json_file=self.vocab_size,
+ hidden_size=self.hidden_size,
+ num_hidden_layers=self.num_hidden_layers,
+ num_attention_heads=self.num_attention_heads,
+ intermediate_size=self.intermediate_size,
+ hidden_act=self.hidden_act,
+ hidden_dropout_prob=self.hidden_dropout_prob,
+ attention_probs_dropout_prob=self.attention_probs_dropout_prob,
+ max_position_embeddings=self.max_position_embeddings,
+ type_vocab_size=self.type_vocab_size,
+ initializer_range=self.initializer_range)
+
+ return config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels
+
+ def check_loss_output(self, result):
+ self.parent.assertListEqual(
+ list(result["loss"].size()),
+ [])
+
+ def create_bert_model(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertModel(config=config)
+ model.eval()
+ all_encoder_layers, pooled_output = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "sequence_output": all_encoder_layers[-1],
+ "pooled_output": pooled_output,
+ "all_encoder_layers": all_encoder_layers,
+ }
+ return outputs
+
+ def check_bert_model_output(self, result):
+ self.parent.assertListEqual(
+ [size for layer in result["all_encoder_layers"] for size in layer.size()],
+ [self.batch_size, self.seq_length, self.hidden_size] * self.num_hidden_layers)
+ self.parent.assertListEqual(
+ list(result["sequence_output"].size()),
+ [self.batch_size, self.seq_length, self.hidden_size])
+ self.parent.assertListEqual(list(result["pooled_output"].size()), [self.batch_size, self.hidden_size])
+
+
+ def create_bert_for_masked_lm(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForMaskedLM(config=config)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, token_labels)
+ prediction_scores = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "prediction_scores": prediction_scores,
+ }
+ return outputs
+
+ def check_bert_for_masked_lm_output(self, result):
+ self.parent.assertListEqual(
+ list(result["prediction_scores"].size()),
+ [self.batch_size, self.seq_length, self.vocab_size])
+
+ def create_bert_for_next_sequence_prediction(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForNextSentencePrediction(config=config)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, sequence_labels)
+ seq_relationship_score = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "seq_relationship_score": seq_relationship_score,
+ }
+ return outputs
+
+ def check_bert_for_next_sequence_prediction_output(self, result):
+ self.parent.assertListEqual(
+ list(result["seq_relationship_score"].size()),
+ [self.batch_size, 2])
+
+
+ def create_bert_for_pretraining(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForPreTraining(config=config)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, token_labels, sequence_labels)
+ prediction_scores, seq_relationship_score = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "prediction_scores": prediction_scores,
+ "seq_relationship_score": seq_relationship_score,
+ }
+ return outputs
+
+ def check_bert_for_pretraining_output(self, result):
+ self.parent.assertListEqual(
+ list(result["prediction_scores"].size()),
+ [self.batch_size, self.seq_length, self.vocab_size])
+ self.parent.assertListEqual(
+ list(result["seq_relationship_score"].size()),
+ [self.batch_size, 2])
+
+
+ def create_bert_for_question_answering(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForQuestionAnswering(config=config)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, sequence_labels, sequence_labels)
+ start_logits, end_logits = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "start_logits": start_logits,
+ "end_logits": end_logits,
+ }
+ return outputs
+
+ def check_bert_for_question_answering_output(self, result):
+ self.parent.assertListEqual(
+ list(result["start_logits"].size()),
+ [self.batch_size, self.seq_length])
+ self.parent.assertListEqual(
+ list(result["end_logits"].size()),
+ [self.batch_size, self.seq_length])
+
+
+ def create_bert_for_sequence_classification(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForSequenceClassification(config=config, num_labels=self.num_labels)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, sequence_labels)
+ logits = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "logits": logits,
+ }
+ return outputs
+
+ def check_bert_for_sequence_classification_output(self, result):
+ self.parent.assertListEqual(
+ list(result["logits"].size()),
+ [self.batch_size, self.num_labels])
+
+
+ def create_bert_for_token_classification(self, config, input_ids, token_type_ids, input_mask, sequence_labels, token_labels):
+ model = BertForTokenClassification(config=config, num_labels=self.num_labels)
+ model.eval()
+ loss = model(input_ids, token_type_ids, input_mask, token_labels)
+ logits = model(input_ids, token_type_ids, input_mask)
+ outputs = {
+ "loss": loss,
+ "logits": logits,
+ }
+ return outputs
+
+ def check_bert_for_token_classification_output(self, result):
+ self.parent.assertListEqual(
+ list(result["logits"].size()),
+ [self.batch_size, self.seq_length, self.num_labels])
+
+
+ def test_default(self):
+ self.run_tester(BertModelTest.BertModelTester(self))
+
+ def test_config_to_json_string(self):
+ config = BertConfig(vocab_size_or_config_json_file=99, hidden_size=37)
+ obj = json.loads(config.to_json_string())
+ self.assertEqual(obj["vocab_size"], 99)
+ self.assertEqual(obj["hidden_size"], 37)
+
+ def test_config_to_json_file(self):
+ config_first = BertConfig(vocab_size_or_config_json_file=99, hidden_size=37)
+ json_file_path = "/tmp/config.json"
+ config_first.to_json_file(json_file_path)
+ config_second = BertConfig.from_json_file(json_file_path)
+ os.remove(json_file_path)
+ self.assertEqual(config_second.to_dict(), config_first.to_dict())
+
+ @pytest.mark.slow
+ def test_model_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+ model = BertModel.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(model)
+
+ def run_tester(self, tester):
+ config_and_inputs = tester.prepare_config_and_inputs()
+ output_result = tester.create_bert_model(*config_and_inputs)
+ tester.check_bert_model_output(output_result)
+
+ output_result = tester.create_bert_for_masked_lm(*config_and_inputs)
+ tester.check_bert_for_masked_lm_output(output_result)
+ tester.check_loss_output(output_result)
+
+ output_result = tester.create_bert_for_next_sequence_prediction(*config_and_inputs)
+ tester.check_bert_for_next_sequence_prediction_output(output_result)
+ tester.check_loss_output(output_result)
+
+ output_result = tester.create_bert_for_pretraining(*config_and_inputs)
+ tester.check_bert_for_pretraining_output(output_result)
+ tester.check_loss_output(output_result)
+
+ output_result = tester.create_bert_for_question_answering(*config_and_inputs)
+ tester.check_bert_for_question_answering_output(output_result)
+ tester.check_loss_output(output_result)
+
+ output_result = tester.create_bert_for_sequence_classification(*config_and_inputs)
+ tester.check_bert_for_sequence_classification_output(output_result)
+ tester.check_loss_output(output_result)
+
+ output_result = tester.create_bert_for_token_classification(*config_and_inputs)
+ tester.check_bert_for_token_classification_output(output_result)
+ tester.check_loss_output(output_result)
+
+ @classmethod
+ def ids_tensor(cls, shape, vocab_size, rng=None, name=None):
+ """Creates a random int32 tensor of the shape within the vocab size."""
+ if rng is None:
+ rng = random.Random()
+
+ total_dims = 1
+ for dim in shape:
+ total_dims *= dim
+
+ values = []
+ for _ in range(total_dims):
+ values.append(rng.randint(0, vocab_size - 1))
+
+ return torch.tensor(data=values, dtype=torch.long).view(shape).contiguous()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_transfo_xl_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_transfo_xl_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..e5c5f3d163adc76762106c8489d03e7082f7c425
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/modeling_transfo_xl_test.py
@@ -0,0 +1,237 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import os
+import unittest
+import json
+import random
+import shutil
+import pytest
+
+import torch
+
+from pytorch_pretrained_bert import (TransfoXLConfig, TransfoXLModel, TransfoXLLMHeadModel)
+from pytorch_pretrained_bert.modeling_transfo_xl import PRETRAINED_MODEL_ARCHIVE_MAP
+
+class TransfoXLModelTest(unittest.TestCase):
+ class TransfoXLModelTester(object):
+
+ def __init__(self,
+ parent,
+ batch_size=13,
+ seq_length=7,
+ mem_len=30,
+ clamp_len=15,
+ is_training=True,
+ use_labels=True,
+ vocab_size=99,
+ cutoffs=[10, 50, 80],
+ d_model=32,
+ d_embed=32,
+ n_head=4,
+ d_head=8,
+ d_inner=128,
+ div_val=2,
+ n_layer=5,
+ scope=None,
+ seed=1):
+ self.parent = parent
+ self.batch_size = batch_size
+ self.seq_length = seq_length
+ self.mem_len = mem_len
+ self.clamp_len = clamp_len
+ self.is_training = is_training
+ self.use_labels = use_labels
+ self.vocab_size = vocab_size
+ self.cutoffs = cutoffs
+ self.d_model = d_model
+ self.d_embed = d_embed
+ self.n_head = n_head
+ self.d_head = d_head
+ self.d_inner = d_inner
+ self.div_val = div_val
+ self.n_layer = n_layer
+ self.scope = scope
+ self.seed = seed
+
+ def prepare_config_and_inputs(self):
+ input_ids_1 = TransfoXLModelTest.ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
+ input_ids_2 = TransfoXLModelTest.ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
+
+ lm_labels = None
+ if self.use_labels:
+ lm_labels = TransfoXLModelTest.ids_tensor([self.batch_size, self.seq_length], self.vocab_size)
+
+ config = TransfoXLConfig(
+ vocab_size_or_config_json_file=self.vocab_size,
+ mem_len=self.mem_len,
+ clamp_len=self.clamp_len,
+ cutoffs=self.cutoffs,
+ d_model=self.d_model,
+ d_embed=self.d_embed,
+ n_head=self.n_head,
+ d_head=self.d_head,
+ d_inner=self.d_inner,
+ div_val=self.div_val,
+ n_layer=self.n_layer)
+
+ return (config, input_ids_1, input_ids_2, lm_labels)
+
+ def set_seed(self):
+ random.seed(self.seed)
+ torch.manual_seed(self.seed)
+
+ def create_transfo_xl_model(self, config, input_ids_1, input_ids_2, lm_labels):
+ model = TransfoXLModel(config)
+ model.eval()
+
+ hidden_states_1, mems_1 = model(input_ids_1)
+ hidden_states_2, mems_2 = model(input_ids_2, mems_1)
+ outputs = {
+ "hidden_states_1": hidden_states_1,
+ "mems_1": mems_1,
+ "hidden_states_2": hidden_states_2,
+ "mems_2": mems_2,
+ }
+ return outputs
+
+ def check_transfo_xl_model_output(self, result):
+ self.parent.assertListEqual(
+ list(result["hidden_states_1"].size()),
+ [self.batch_size, self.seq_length, self.d_model])
+ self.parent.assertListEqual(
+ list(result["hidden_states_2"].size()),
+ [self.batch_size, self.seq_length, self.d_model])
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_1"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_2"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+
+
+ def create_transfo_xl_lm_head(self, config, input_ids_1, input_ids_2, lm_labels):
+ model = TransfoXLLMHeadModel(config)
+ model.eval()
+
+ loss_1, mems_1a = model(input_ids_1, target=lm_labels)
+ lm_logits_1, mems_1b = model(input_ids_1)
+
+ loss_2, mems_2a = model(input_ids_2, target=lm_labels, mems=mems_1a)
+ lm_logits_2, mems_2b = model(input_ids_2, mems=mems_1b)
+
+ outputs = {
+ "loss_1": loss_1,
+ "mems_1a": mems_1a,
+ "lm_logits_1": lm_logits_1,
+ "mems_1b": mems_1b,
+ "loss_2": loss_2,
+ "mems_2a": mems_2a,
+ "lm_logits_2": lm_logits_2,
+ "mems_2b": mems_2b,
+ }
+ return outputs
+
+ def check_transfo_xl_lm_head_output(self, result):
+ self.parent.assertListEqual(
+ list(result["loss_1"].size()),
+ [self.batch_size, self.seq_length])
+ self.parent.assertListEqual(
+ list(result["lm_logits_1"].size()),
+ [self.batch_size, self.seq_length, self.vocab_size])
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_1a"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_1b"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+ self.parent.assertListEqual(
+ list(mem[~torch.isnan(mem)].sum() for mem in result["mems_1a"]),
+ list(mem[~torch.isnan(mem)].sum() for mem in result["mems_1b"]))
+
+ self.parent.assertListEqual(
+ list(result["loss_2"].size()),
+ [self.batch_size, self.seq_length])
+ self.parent.assertListEqual(
+ list(result["lm_logits_2"].size()),
+ [self.batch_size, self.seq_length, self.vocab_size])
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_2a"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+ self.parent.assertListEqual(
+ list(list(mem.size()) for mem in result["mems_2b"]),
+ [[self.mem_len, self.batch_size, self.d_model]] * self.n_layer)
+ self.parent.assertListEqual(
+ list(mem[~torch.isnan(mem)].sum() for mem in result["mems_2a"]),
+ list(mem[~torch.isnan(mem)].sum() for mem in result["mems_2b"]))
+
+ def test_default(self):
+ self.run_tester(TransfoXLModelTest.TransfoXLModelTester(self))
+
+ def test_config_to_json_string(self):
+ config = TransfoXLConfig(vocab_size_or_config_json_file=96, d_embed=37)
+ obj = json.loads(config.to_json_string())
+ self.assertEqual(obj["n_token"], 96)
+ self.assertEqual(obj["d_embed"], 37)
+
+ def test_config_to_json_file(self):
+ config_first = TransfoXLConfig(vocab_size_or_config_json_file=96, d_embed=37)
+ json_file_path = "/tmp/config.json"
+ config_first.to_json_file(json_file_path)
+ config_second = TransfoXLConfig.from_json_file(json_file_path)
+ os.remove(json_file_path)
+ self.assertEqual(config_second.to_dict(), config_first.to_dict())
+
+ @pytest.mark.slow
+ def test_model_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_MODEL_ARCHIVE_MAP.keys())[:1]:
+ model = TransfoXLModel.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(model)
+
+ def run_tester(self, tester):
+ config_and_inputs = tester.prepare_config_and_inputs()
+
+ tester.set_seed()
+ output_result = tester.create_transfo_xl_model(*config_and_inputs)
+ tester.check_transfo_xl_model_output(output_result)
+
+ tester.set_seed()
+ output_result = tester.create_transfo_xl_lm_head(*config_and_inputs)
+ tester.check_transfo_xl_lm_head_output(output_result)
+
+ @classmethod
+ def ids_tensor(cls, shape, vocab_size, rng=None, name=None):
+ """Creates a random int32 tensor of the shape within the vocab size."""
+ if rng is None:
+ rng = random.Random()
+
+ total_dims = 1
+ for dim in shape:
+ total_dims *= dim
+
+ values = []
+ for _ in range(total_dims):
+ values.append(rng.randint(0, vocab_size - 1))
+
+ return torch.tensor(data=values, dtype=torch.long).view(shape).contiguous()
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/optimization_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/optimization_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..c6924bd4bcbe31e5d717455e53e3af4acf3a4a1b
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/optimization_test.py
@@ -0,0 +1,91 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import unittest
+
+import torch
+
+from pytorch_pretrained_bert import BertAdam
+from pytorch_pretrained_bert import OpenAIAdam
+from pytorch_pretrained_bert.optimization import ConstantLR, WarmupLinearSchedule, WarmupConstantSchedule, \
+ WarmupCosineWithWarmupRestartsSchedule, WarmupCosineWithHardRestartsSchedule, WarmupCosineSchedule
+import numpy as np
+
+
+class OptimizationTest(unittest.TestCase):
+
+ def assertListAlmostEqual(self, list1, list2, tol):
+ self.assertEqual(len(list1), len(list2))
+ for a, b in zip(list1, list2):
+ self.assertAlmostEqual(a, b, delta=tol)
+
+ def test_adam(self):
+ w = torch.tensor([0.1, -0.2, -0.1], requires_grad=True)
+ target = torch.tensor([0.4, 0.2, -0.5])
+ criterion = torch.nn.MSELoss()
+ # No warmup, constant schedule, no gradient clipping
+ optimizer = BertAdam(params=[w], lr=2e-1,
+ weight_decay=0.0,
+ max_grad_norm=-1)
+ for _ in range(100):
+ loss = criterion(w, target)
+ loss.backward()
+ optimizer.step()
+ w.grad.detach_() # No zero_grad() function on simple tensors. we do it ourselves.
+ w.grad.zero_()
+ self.assertListAlmostEqual(w.tolist(), [0.4, 0.2, -0.5], tol=1e-2)
+
+
+class ScheduleInitTest(unittest.TestCase):
+ def test_bert_sched_init(self):
+ m = torch.nn.Linear(50, 50)
+ optim = BertAdam(m.parameters(), lr=0.001, warmup=.1, t_total=1000, schedule=None)
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], ConstantLR))
+ optim = BertAdam(m.parameters(), lr=0.001, warmup=.1, t_total=1000, schedule="none")
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], ConstantLR))
+ optim = BertAdam(m.parameters(), lr=0.001, warmup=.01, t_total=1000)
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], WarmupLinearSchedule))
+ # shouldn't fail
+
+ def test_openai_sched_init(self):
+ m = torch.nn.Linear(50, 50)
+ optim = OpenAIAdam(m.parameters(), lr=0.001, warmup=.1, t_total=1000, schedule=None)
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], ConstantLR))
+ optim = OpenAIAdam(m.parameters(), lr=0.001, warmup=.1, t_total=1000, schedule="none")
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], ConstantLR))
+ optim = OpenAIAdam(m.parameters(), lr=0.001, warmup=.01, t_total=1000)
+ self.assertTrue(isinstance(optim.param_groups[0]["schedule"], WarmupLinearSchedule))
+ # shouldn't fail
+
+
+class WarmupCosineWithRestartsTest(unittest.TestCase):
+ def test_it(self):
+ m = WarmupCosineWithWarmupRestartsSchedule(warmup=0.05, t_total=1000., cycles=5)
+ x = np.arange(0, 1000)
+ y = [m.get_lr(xe) for xe in x]
+ y = np.asarray(y)
+ expected_zeros = y[[0, 200, 400, 600, 800]]
+ print(expected_zeros)
+ expected_ones = y[[50, 250, 450, 650, 850]]
+ print(expected_ones)
+ self.assertTrue(np.allclose(expected_ones, 1))
+ self.assertTrue(np.allclose(expected_zeros, 0))
+
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_gpt2_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_gpt2_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..4ae804a060bb20745d6c9217ad2b9f7313f4a3c1
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_gpt2_test.py
@@ -0,0 +1,77 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import unittest
+import json
+import shutil
+import pytest
+
+from pytorch_pretrained_bert.tokenization_gpt2 import GPT2Tokenizer, PRETRAINED_VOCAB_ARCHIVE_MAP
+
+
+class GPT2TokenizationTest(unittest.TestCase):
+
+ def test_full_tokenizer(self):
+ """ Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt """
+ vocab = ["l", "o", "w", "e", "r", "s", "t", "i", "d", "n",
+ "lo", "low", "er",
+ "low", "lowest", "newer", "wider"]
+ vocab_tokens = dict(zip(vocab, range(len(vocab))))
+ merges = ["#version: 0.2", "l o", "lo w", "e r", ""]
+ with open("/tmp/openai_tokenizer_vocab_test.json", "w") as fp:
+ fp.write(json.dumps(vocab_tokens))
+ vocab_file = fp.name
+ with open("/tmp/openai_tokenizer_merges_test.txt", "w") as fp:
+ fp.write("\n".join(merges))
+ merges_file = fp.name
+
+ tokenizer = GPT2Tokenizer(vocab_file, merges_file, special_tokens=["", ""])
+ os.remove(vocab_file)
+ os.remove(merges_file)
+
+ text = "lower"
+ bpe_tokens = ["low", "er"]
+ tokens = tokenizer.tokenize(text)
+ self.assertListEqual(tokens, bpe_tokens)
+
+ input_tokens = tokens + [""]
+ input_bpe_tokens = [13, 12, 16]
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(input_tokens), input_bpe_tokens)
+
+ vocab_file, merges_file, special_tokens_file = tokenizer.save_vocabulary(vocab_path="/tmp/")
+ tokenizer_2 = GPT2Tokenizer.from_pretrained("/tmp/")
+ os.remove(vocab_file)
+ os.remove(merges_file)
+ os.remove(special_tokens_file)
+
+ self.assertListEqual(
+ [tokenizer.encoder, tokenizer.decoder, tokenizer.bpe_ranks,
+ tokenizer.special_tokens, tokenizer.special_tokens_decoder],
+ [tokenizer_2.encoder, tokenizer_2.decoder, tokenizer_2.bpe_ranks,
+ tokenizer_2.special_tokens, tokenizer_2.special_tokens_decoder])
+
+ # @pytest.mark.slow
+ def test_tokenizer_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_VOCAB_ARCHIVE_MAP.keys())[:1]:
+ tokenizer = GPT2Tokenizer.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(tokenizer)
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_openai_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_openai_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..a57f86be57b39c99bdac6967c5ffbb212d116889
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_openai_test.py
@@ -0,0 +1,79 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import unittest
+import json
+import shutil
+import pytest
+
+from pytorch_pretrained_bert.tokenization_openai import OpenAIGPTTokenizer, PRETRAINED_VOCAB_ARCHIVE_MAP
+
+
+class OpenAIGPTTokenizationTest(unittest.TestCase):
+
+ def test_full_tokenizer(self):
+ """ Adapted from Sennrich et al. 2015 and https://github.com/rsennrich/subword-nmt """
+ vocab = ["l", "o", "w", "e", "r", "s", "t", "i", "d", "n",
+ "w", "r", "t",
+ "lo", "low", "er",
+ "low", "lowest", "newer", "wider"]
+ vocab_tokens = dict(zip(vocab, range(len(vocab))))
+ merges = ["#version: 0.2", "l o", "lo w", "e r", ""]
+ with open("/tmp/openai_tokenizer_vocab_test.json", "w") as fp:
+ fp.write(json.dumps(vocab_tokens))
+ vocab_file = fp.name
+ with open("/tmp/openai_tokenizer_merges_test.txt", "w") as fp:
+ fp.write("\n".join(merges))
+ merges_file = fp.name
+
+ tokenizer = OpenAIGPTTokenizer(vocab_file, merges_file, special_tokens=["", ""])
+ os.remove(vocab_file)
+ os.remove(merges_file)
+
+ text = "lower"
+ bpe_tokens = ["low", "er"]
+ tokens = tokenizer.tokenize(text)
+ self.assertListEqual(tokens, bpe_tokens)
+
+ input_tokens = tokens + [""]
+ input_bpe_tokens = [14, 15, 20]
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(input_tokens), input_bpe_tokens)
+
+ vocab_file, merges_file, special_tokens_file = tokenizer.save_vocabulary(vocab_path="/tmp/")
+ tokenizer_2 = OpenAIGPTTokenizer.from_pretrained("/tmp/")
+ os.remove(vocab_file)
+ os.remove(merges_file)
+ os.remove(special_tokens_file)
+
+ self.assertListEqual(
+ [tokenizer.encoder, tokenizer.decoder, tokenizer.bpe_ranks,
+ tokenizer.special_tokens, tokenizer.special_tokens_decoder],
+ [tokenizer_2.encoder, tokenizer_2.decoder, tokenizer_2.bpe_ranks,
+ tokenizer_2.special_tokens, tokenizer_2.special_tokens_decoder])
+
+ @pytest.mark.slow
+ def test_tokenizer_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_VOCAB_ARCHIVE_MAP.keys())[:1]:
+ tokenizer = OpenAIGPTTokenizer.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(tokenizer)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..fe120a522c32a6416417479b17828f892c9b57dc
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_test.py
@@ -0,0 +1,140 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import unittest
+from io import open
+import shutil
+import pytest
+
+from pytorch_pretrained_bert.tokenization import (BasicTokenizer,
+ BertTokenizer,
+ WordpieceTokenizer,
+ _is_control, _is_punctuation,
+ _is_whitespace, PRETRAINED_VOCAB_ARCHIVE_MAP)
+
+
+class TokenizationTest(unittest.TestCase):
+
+ def test_full_tokenizer(self):
+ vocab_tokens = [
+ "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn",
+ "##ing", ","
+ ]
+ with open("/tmp/bert_tokenizer_test.txt", "w", encoding='utf-8') as vocab_writer:
+ vocab_writer.write("".join([x + "\n" for x in vocab_tokens]))
+
+ vocab_file = vocab_writer.name
+
+ tokenizer = BertTokenizer(vocab_file)
+ os.remove(vocab_file)
+
+ tokens = tokenizer.tokenize(u"UNwant\u00E9d,running")
+ self.assertListEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"])
+
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9])
+
+ vocab_file = tokenizer.save_vocabulary(vocab_path="/tmp/")
+ tokenizer.from_pretrained(vocab_file)
+ os.remove(vocab_file)
+
+ tokens = tokenizer.tokenize(u"UNwant\u00E9d,running")
+ self.assertListEqual(tokens, ["un", "##want", "##ed", ",", "runn", "##ing"])
+
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(tokens), [7, 4, 5, 10, 8, 9])
+
+ @pytest.mark.slow
+ def test_tokenizer_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_VOCAB_ARCHIVE_MAP.keys())[:1]:
+ tokenizer = BertTokenizer.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(tokenizer)
+
+ def test_chinese(self):
+ tokenizer = BasicTokenizer()
+
+ self.assertListEqual(
+ tokenizer.tokenize(u"ah\u535A\u63A8zz"),
+ [u"ah", u"\u535A", u"\u63A8", u"zz"])
+
+ def test_basic_tokenizer_lower(self):
+ tokenizer = BasicTokenizer(do_lower_case=True)
+
+ self.assertListEqual(
+ tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "),
+ ["hello", "!", "how", "are", "you", "?"])
+ self.assertListEqual(tokenizer.tokenize(u"H\u00E9llo"), ["hello"])
+
+ def test_basic_tokenizer_no_lower(self):
+ tokenizer = BasicTokenizer(do_lower_case=False)
+
+ self.assertListEqual(
+ tokenizer.tokenize(u" \tHeLLo!how \n Are yoU? "),
+ ["HeLLo", "!", "how", "Are", "yoU", "?"])
+
+ def test_wordpiece_tokenizer(self):
+ vocab_tokens = [
+ "[UNK]", "[CLS]", "[SEP]", "want", "##want", "##ed", "wa", "un", "runn",
+ "##ing"
+ ]
+
+ vocab = {}
+ for (i, token) in enumerate(vocab_tokens):
+ vocab[token] = i
+ tokenizer = WordpieceTokenizer(vocab=vocab)
+
+ self.assertListEqual(tokenizer.tokenize(""), [])
+
+ self.assertListEqual(
+ tokenizer.tokenize("unwanted running"),
+ ["un", "##want", "##ed", "runn", "##ing"])
+
+ self.assertListEqual(
+ tokenizer.tokenize("unwantedX running"), ["[UNK]", "runn", "##ing"])
+
+ def test_is_whitespace(self):
+ self.assertTrue(_is_whitespace(u" "))
+ self.assertTrue(_is_whitespace(u"\t"))
+ self.assertTrue(_is_whitespace(u"\r"))
+ self.assertTrue(_is_whitespace(u"\n"))
+ self.assertTrue(_is_whitespace(u"\u00A0"))
+
+ self.assertFalse(_is_whitespace(u"A"))
+ self.assertFalse(_is_whitespace(u"-"))
+
+ def test_is_control(self):
+ self.assertTrue(_is_control(u"\u0005"))
+
+ self.assertFalse(_is_control(u"A"))
+ self.assertFalse(_is_control(u" "))
+ self.assertFalse(_is_control(u"\t"))
+ self.assertFalse(_is_control(u"\r"))
+
+ def test_is_punctuation(self):
+ self.assertTrue(_is_punctuation(u"-"))
+ self.assertTrue(_is_punctuation(u"$"))
+ self.assertTrue(_is_punctuation(u"`"))
+ self.assertTrue(_is_punctuation(u"."))
+
+ self.assertFalse(_is_punctuation(u"A"))
+ self.assertFalse(_is_punctuation(u" "))
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_transfo_xl_test.py b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_transfo_xl_test.py
new file mode 100644
index 0000000000000000000000000000000000000000..bf0ac5db2fedde843f251688ca1563eb97299ee7
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/tests/tokenization_transfo_xl_test.py
@@ -0,0 +1,80 @@
+# coding=utf-8
+# Copyright 2018 The Google AI Language Team Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import os
+import unittest
+from io import open
+import shutil
+import pytest
+
+from pytorch_pretrained_bert.tokenization_transfo_xl import TransfoXLTokenizer, PRETRAINED_VOCAB_ARCHIVE_MAP
+
+
+class TransfoXLTokenizationTest(unittest.TestCase):
+
+ def test_full_tokenizer(self):
+ vocab_tokens = [
+ "", "[CLS]", "[SEP]", "want", "unwanted", "wa", "un", "running", ","
+ ]
+ with open("/tmp/transfo_xl_tokenizer_test.txt", "w", encoding='utf-8') as vocab_writer:
+ vocab_writer.write("".join([x + "\n" for x in vocab_tokens]))
+ vocab_file = vocab_writer.name
+
+ tokenizer = TransfoXLTokenizer(vocab_file=vocab_file, lower_case=True)
+ tokenizer.build_vocab()
+ os.remove(vocab_file)
+
+ tokens = tokenizer.tokenize(u" UNwanted , running")
+ self.assertListEqual(tokens, ["", "unwanted", ",", "running"])
+
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(tokens), [0, 4, 8, 7])
+
+ vocab_file = tokenizer.save_vocabulary(vocab_path="/tmp/")
+ tokenizer.from_pretrained(vocab_file)
+ os.remove(vocab_file)
+
+ tokens = tokenizer.tokenize(u" UNwanted , running")
+ self.assertListEqual(tokens, ["", "unwanted", ",", "running"])
+
+ self.assertListEqual(
+ tokenizer.convert_tokens_to_ids(tokens), [0, 4, 8, 7])
+
+
+ def test_full_tokenizer_lower(self):
+ tokenizer = TransfoXLTokenizer(lower_case=True)
+
+ self.assertListEqual(
+ tokenizer.tokenize(u" \tHeLLo ! how \n Are yoU ? "),
+ ["hello", "!", "how", "are", "you", "?"])
+
+ def test_full_tokenizer_no_lower(self):
+ tokenizer = TransfoXLTokenizer(lower_case=False)
+
+ self.assertListEqual(
+ tokenizer.tokenize(u" \tHeLLo ! how \n Are yoU ? "),
+ ["HeLLo", "!", "how", "Are", "yoU", "?"])
+
+ @pytest.mark.slow
+ def test_tokenizer_from_pretrained(self):
+ cache_dir = "/tmp/pytorch_pretrained_bert_test/"
+ for model_name in list(PRETRAINED_VOCAB_ARCHIVE_MAP.keys())[:1]:
+ tokenizer = TransfoXLTokenizer.from_pretrained(model_name, cache_dir=cache_dir)
+ shutil.rmtree(cache_dir)
+ self.assertIsNotNone(tokenizer)
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/visualize_attention.ipynb b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/visualize_attention.ipynb
new file mode 100644
index 0000000000000000000000000000000000000000..677089ec9d6595ff14bafc9c01ff0e6bfa1750d5
--- /dev/null
+++ b/dark-secrets-of-BERT-master/dark-secrets-of-BERT-master/visualize_attention.ipynb
@@ -0,0 +1,2964 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 71,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import csv\n",
+ "import glob\n",
+ "import json\n",
+ "import itertools\n",
+ "import random\n",
+ "import requests\n",
+ "from collections import defaultdict\n",
+ "\n",
+ "import matplotlib.pyplot as plt\n",
+ "import nltk\n",
+ "import numpy as np\n",
+ "import spacy\n",
+ "import textacy\n",
+ "import torch\n",
+ "from matplotlib.gridspec import GridSpec\n",
+ "from bs4 import BeautifulSoup\n",
+ "from nltk import word_tokenize\n",
+ "from nltk.corpus import framenet as fn\n",
+ "from spacy.symbols import nsubj, VERB\n",
+ "from tqdm import tqdm\n",
+ "\n",
+ "from pytorch_pretrained_bert import BertForSequenceClassification, BertConfig, BertTokenizer\n",
+ "nlp = spacy.load(\"en_core_web_md\")\n",
+ "%matplotlib inline\n",
+ "plt.style.use('seaborn')\n",
+ "tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Functions used"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def convert_to_bert_tokens(sentence_1, sentence_2, max_len=60):\n",
+ " \"\"\"\n",
+ " Converting two sentences to BERT tokens\n",
+ " \"\"\"\n",
+ " tokens_1 = tokenizer.tokenize(sentence_1)\n",
+ " tokens_2 = tokenizer.tokenize(sentence_2)\n",
+ " \n",
+ " # taken from the original repo\n",
+ " while True:\n",
+ " total_length = len(tokens_1) + len(tokens_2)\n",
+ " if total_length <= max_len - 3:\n",
+ " break\n",
+ " if len(tokens_1) > len(tokens_2):\n",
+ " tokens_1.pop()\n",
+ " else:\n",
+ " tokens_2.pop()\n",
+ " \n",
+ " if tokens_2:\n",
+ " tokenized = ['[CLS]'] + tokens_1 + ['[SEP]'] + tokens_2 + ['[SEP]']\n",
+ " else:\n",
+ " tokenized = ['[CLS]'] + tokens_1 + ['[SEP]']\n",
+ "\n",
+ " to_pad = ['[PAD]']*(max_len - len(tokenized))\n",
+ " return tokenized + to_pad, tokens_1, tokens_2"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def convert_to_bert_inputs(sentence_1, sentence_2, max_len):\n",
+ " \"\"\"\n",
+ " Converting two sentences to BERT inputs\n",
+ " \"\"\"\n",
+ " tokens_all, tokens_1, tokens_2 = convert_to_bert_tokens(sentence_1, sentence_2, max_len)\n",
+ " input_ids = torch.LongTensor(tokenizer.convert_tokens_to_ids(tokens_all))\n",
+ " input_mask = (input_ids != 0).long()\n",
+ " \n",
+ " length_1 = len(tokens_1)\n",
+ " length_2 = len(tokens_2)\n",
+ " segment_ids = torch.zeros(max_len).long()\n",
+ " if length_2 > 0:\n",
+ " segment_ids[length_1 + 2:length_1 + length_2 + 3] = 1\n",
+ " return input_ids, input_mask, segment_ids,"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def restore_model(path=None, sts=False, mnli=False):\n",
+ " \"\"\"\n",
+ " Model loading. STS and MNLI require different initialization due to a different number of classes.\n",
+ " \"\"\"\n",
+ " if path is None:\n",
+ " model = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)\n",
+ " else:\n",
+ " model_path = path + '/pytorch_model.bin'\n",
+ " config_path = path + '/config.json'\n",
+ " # Reading config\n",
+ " config = BertConfig.from_json_file(config_path)\n",
+ "\n",
+ " # Initializing the model\n",
+ " if sts:\n",
+ " num_labels = 1\n",
+ " elif mnli:\n",
+ " num_labels = 3\n",
+ " else:\n",
+ " num_labels = 2\n",
+ " model = BertForSequenceClassification(config, num_labels=num_labels)\n",
+ "\n",
+ " # Loading weights\n",
+ " model.load_state_dict(torch.load(model_path))\n",
+ "\n",
+ " model.eval()\n",
+ " return model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 29,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def visualize_single(att_map, sentence_1, sentence_2, max_len, n_layer, n_head):\n",
+ " \"\"\"\n",
+ " Attention map for a given layer and head\n",
+ " \"\"\"\n",
+ " bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(sentence_1, sentence_2, max_len)\n",
+ " if tokens_2:\n",
+ " crop_len = len(tokens_1) + len(tokens_2) + 3\n",
+ " else:\n",
+ " crop_len = len(tokens_1) + 2\n",
+ " \n",
+ " plt.figure(figsize=(16, 12))\n",
+ " plt.imshow(att_map[n_layer, n_head, :crop_len, :crop_len], cmap='Reds')\n",
+ " plt.xticks(range(crop_len), bert_tokens, rotation=60, fontsize=12)\n",
+ " plt.yticks(range(crop_len), bert_tokens, fontsize=12)\n",
+ "\n",
+ " plt.grid(False)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 31,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def visualize_all(attn, crop_len, n_layers=12, n_heads=12, title=\"\"):\n",
+ " \"\"\"\n",
+ " Full grid of attention maps [12x12]\n",
+ " \"\"\"\n",
+ " fig, axes = plt.subplots(n_layers, n_heads, figsize=(15, 12), sharex=True, sharey=True)\n",
+ " \n",
+ " for i in range(n_layers):\n",
+ " for j in range(n_heads):\n",
+ " im = axes[i, j].imshow(attn[i, j, :crop_len, :crop_len], cmap='Oranges')\n",
+ " axes[i, j].axis('off')\n",
+ " fig.colorbar(im, ax=axes.ravel().tolist())\n",
+ " fig.suptitle(title, fontsize=20)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 32,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def visualize_before_and_after(before, after, title='', cmap=\"Greens\"):\n",
+ " \"\"\"\n",
+ " Visualize the difference between base BERT and fine-tuned BERT\n",
+ " \"\"\"\n",
+ " fig, axes = plt.subplots(1, 2, figsize=(20, 10))\n",
+ "\n",
+ " ax1, ax2 = axes[0], axes[1]\n",
+ "\n",
+ " vmax = max(np.max(before), np.max(after))\n",
+ "\n",
+ " im = ax1.imshow(before, cmap=cmap, vmax=vmax)\n",
+ " ax1.set_title('Base model')\n",
+ " ax1.grid(False)\n",
+ "\n",
+ " im = ax2.imshow(after, cmap=cmap, vmax=vmax)\n",
+ " ax2.set_title('Fine-tuned model')\n",
+ " ax2.grid(False)\n",
+ "\n",
+ " fig.colorbar(im, ax=axes.ravel().tolist())\n",
+ " fig.suptitle(title, fontsize=20)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 33,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# See spacy docs for tag-pos relation\n",
+ "def detect_all_pos(sentence, pos='PRON'):\n",
+ " \"\"\"\n",
+ " Detect all tokens with a given POS tag\n",
+ " \"\"\"\n",
+ " if pos not in ['PRON', 'VERB', 'NOUN']:\n",
+ " raise ValueError(\"POS not recognized\")\n",
+ " \n",
+ " pos2tag = {'PRON': ['PRP', 'PRP$'],\n",
+ " 'NOUN': ['NN', 'NNP', 'NNPS', 'NNS'],\n",
+ " 'VERB': ['VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ']}\n",
+ " \n",
+ " doc = nlp(sentence, disable=['ner', 'parser'])\n",
+ " targets = []\n",
+ " for token in doc:\n",
+ " if token.tag_ in pos2tag[pos]:\n",
+ " targets.append(token.text)\n",
+ " return set(targets)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def detect_all_negations(sentence):\n",
+ " \"\"\"\n",
+ " Check if there is a negation phrase in a sentence out of a list of manually curated negations\n",
+ " \"\"\"\n",
+ " negation_words = ['neither', 'nor', 'not', 'never', 'none', \"don't\", \"won't\", \"didn't\", \n",
+ " \"hadn't\", \"haven't\", \"can't\", \"isn't\", \"wasn't\", \"shouldn't\", \"couldn't\", \"nothing\", \"nowhere\"]\n",
+ " \n",
+ " targets = [word for word in negation_words if word in sentence]\n",
+ " return set(targets)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 35,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def detect_all_dep(sentence, label):\n",
+ " \"\"\"\n",
+ " Get subject-object dependencies\n",
+ " \"\"\"\n",
+ " doc = nlp(sentence, disable=['ner', 'pos'])\n",
+ " \n",
+ " label2dep = {'SUBJ': ['nsubj', \"nsubjpass\", \"csubj\", \"csubjpass\", \"agent\", \"expl\"], \n",
+ " \"OBJ\": ['dobj', 'iobj', \"dative\", \"attr\", \"oprd\"]}\n",
+ " \n",
+ " targets = []\n",
+ " for token in doc:\n",
+ " if token.dep_ in label2dep[label]:\n",
+ " targets.append(token.text)\n",
+ " return set(targets)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 36,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def get_max_target_weight(attn, target_indices):\n",
+ " \"\"\"\n",
+ " Get the maximum attn weight out of target tokens (given by their indices)\n",
+ " \"\"\"\n",
+ " if not target_indices:\n",
+ " return 0\n",
+ " avg_attn = np.mean(attn, axis=0)\n",
+ " target_weights = avg_attn[target_indices]\n",
+ " max_target_weight = np.max(target_weights)\n",
+ " return max_target_weight"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 37,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def analyze_target_attention(sentence_1, sentence_2, max_len, model, feature='NOUN', n_layers=12, n_heads=12):\n",
+ " \"\"\"\n",
+ " Analyze the attention weights for the target sentence and for a given syntactic feature\n",
+ " \"\"\"\n",
+ " weights = np.zeros((n_layers, n_heads))\n",
+ " bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(sentence_1, sentence_2, max_len)\n",
+ " if tokens_2:\n",
+ " crop_len = len(tokens_1) + len(tokens_2) + 3\n",
+ " else:\n",
+ " crop_len = len(tokens_1) + 2\n",
+ "\n",
+ " if feature in [\"NOUN\", \"PRON\", \"VERB\"]:\n",
+ " target_feat_1 = detect_all_pos(sentence_1, feature)\n",
+ " target_feat_2 = detect_all_pos(sentence_2, feature)\n",
+ " elif feature == \"NEG\":\n",
+ " target_feat_1 = detect_all_negations(sentence_1)\n",
+ " target_feat_2 = detect_all_negations(sentence_2)\n",
+ " elif feature in [\"SUBJ\", \"OBJ\"]:\n",
+ " target_feat_1 = detect_all_dep(sentence_1, feature)\n",
+ " target_feat_2 = detect_all_dep(sentence_2, feature)\n",
+ " \n",
+ " bert_tokens_feat = list(itertools.chain.from_iterable([tokenizer.tokenize(feat) for feat in target_feat_1.union(target_feat_2)]))\n",
+ " feat_indices = [i for i, token in enumerate(bert_tokens) if token in bert_tokens_feat]\n",
+ "\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(sentence_1, sentence_2, max_len)\n",
+ " _, output = model(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "\n",
+ " output = output.squeeze(0).detach().numpy()\n",
+ "\n",
+ " for l in range(n_layers):\n",
+ " for h in range(n_heads):\n",
+ " weights[l, h] = get_max_target_weight(output[l, h, :crop_len, :crop_len], feat_indices)\n",
+ " \n",
+ " return weights\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def extract_subj_verb(sentence):\n",
+ " \"\"\"\n",
+ " Get subject-verb dependencies\n",
+ " \"\"\"\n",
+ " doc = nlp(sentence)\n",
+ " \n",
+ " subj_verb = [] \n",
+ " for possible_subject in doc:\n",
+ " if possible_subject.dep == nsubj and possible_subject.head.pos == VERB:\n",
+ " subj_verb.append((possible_subject.text, possible_subject.head.text))\n",
+ " return subj_verb"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def get_max_source_target_weight(attn, source_indices, target_indices):\n",
+ " \"\"\"\n",
+ " Calculate the maximum source-target attention weight out of all possible source-target pairs (given by their indices) \n",
+ " \"\"\"\n",
+ " if not source_indices or not target_indices:\n",
+ " return 0\n",
+ " \n",
+ " weights = np.array([attn[source, target] for source, target in zip(source_indices, target_indices)])\n",
+ " \n",
+ " return np.max(weights)\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def analyze_cross_attention(sentence_1, sentence_2, max_len, model, relation, n_layers=12, n_heads=12):\n",
+ " \"\"\"\n",
+ " Get maximum source-target attention weights for every layer and head\n",
+ " \"\"\"\n",
+ " bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(sentence_1, sentence_2, max_len)\n",
+ " if tokens_2:\n",
+ " crop_len = len(tokens_1) + len(tokens_2) + 3\n",
+ " else: \n",
+ " crop_len = len(tokens_1) + 2\n",
+ " \n",
+ " if relation == \"SUBJ-VERB\" or \"VERB-SUBJ\":\n",
+ " feats_1 = extract_subj_verb(sentence_1)\n",
+ " feats_2 = extract_subj_verb(sentence_2)\n",
+ " \n",
+ " \n",
+ " all_feats = feats_1 + feats_2\n",
+ " source_tokens = [pair[0] for pair in all_feats]\n",
+ " target_tokens = [pair[1] for pair in all_feats]\n",
+ " \n",
+ " if relation == \"VERB-SUBJ\":\n",
+ " source_tokens, target_tokens = target_tokens, source_tokens\n",
+ " \n",
+ " bert_tokens_source = list(itertools.chain.from_iterable([tokenizer.tokenize(token) for token in source_tokens]))\n",
+ " bert_tokens_target = list(itertools.chain.from_iterable([tokenizer.tokenize(token) for token in target_tokens]))\n",
+ " \n",
+ " source_indices = [i for i, token in enumerate(bert_tokens) if token in bert_tokens_source]\n",
+ " target_indices = [i for i, token in enumerate(bert_tokens) if token in bert_tokens_target]\n",
+ " \n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(sentence_1, sentence_2, max_len)\n",
+ " _, output = model(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "\n",
+ " output = output.squeeze(0).detach().numpy()\n",
+ "\n",
+ " weights = np.zeros((n_layers, n_heads))\n",
+ " for l in range(n_layers):\n",
+ " for h in range(n_heads):\n",
+ " weights[l, h] = get_max_source_target_weight(output[l, h, :crop_len, :crop_len], source_indices, target_indices)\n",
+ " \n",
+ " return weights"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def read_dataset(task, path):\n",
+ " \"\"\"\n",
+ " Reading formatted GLUE data from path\n",
+ " \"\"\"\n",
+ " if task == 'MRPC':\n",
+ " dev_data_path = path + '/MRPC/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = line[0]\n",
+ " string1 = line[-2]\n",
+ " string2 = line[-1]\n",
+ " data.append((label, string1, string2))\n",
+ " \n",
+ " elif task == \"QQP\":\n",
+ " dev_data_path = path + '/QQP/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " try:\n",
+ " label = line[-1]\n",
+ " string1 = line[3]\n",
+ " string2 = line[4]\n",
+ " except Exception:\n",
+ " continue\n",
+ " data.append((label, string1, string2))\n",
+ " \n",
+ " elif task == \"SST-2\":\n",
+ " dev_data_path = path + '/SST-2/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = line[1]\n",
+ " string1 = line[0]\n",
+ " data.append((label, string1, \"\"))\n",
+ " \n",
+ " elif task == \"RTE\":\n",
+ " dev_data_path = path + '/RTE/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = int(line[-1] == 'entailment')\n",
+ " string1 = line[1]\n",
+ " string2 = line[2]\n",
+ " data.append((label, string1, string2))\n",
+ " \n",
+ " elif task == \"QNLI\":\n",
+ " dev_data_path = path + '/QNLI/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = int(line[-1] == 'entailment')\n",
+ " string1 = line[1]\n",
+ " string2 = line[2]\n",
+ " data.append((label, string1, string2))\n",
+ " \n",
+ " elif task == \"STS-B\":\n",
+ " dev_data_path = path + '/STS-B/dev.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = line[-1]\n",
+ " string1 = line[-3]\n",
+ " string2 = line[-2]\n",
+ " data.append((label, string1, string2))\n",
+ " \n",
+ " elif task == \"MNLI\":\n",
+ " dev_data_path = path + '/MNLI/dev_matched.tsv'\n",
+ " data = []\n",
+ " with open(dev_data_path) as csvfile:\n",
+ " reader = csv.reader(csvfile, delimiter='\\t')\n",
+ " reader.__next__()\n",
+ " for line in reader:\n",
+ " label = line[-1]\n",
+ " string1 = line[-8]\n",
+ " string2 = line[-7]\n",
+ " data.append((label, string1, string2))\n",
+ " return data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "-------------------------------------"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setting parameters"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 75,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "max_len = 200\n",
+ "path_to_data ='../glue/data'\n",
+ "model_base = restore_model()\n",
+ "n_layers = 12\n",
+ "n_heads = 12\n",
+ "task = 'QQP'\n",
+ "path_to_model = '../pretrained_models/v2/{}/fine-tuned/'.format(task.lower())\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "--------------------------"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Workflow:\n",
+ "0. Pick the dataset\n",
+ "1. Load the dataset and the fine-tuned model\n",
+ "2. Plot 12x12 attention map for a random example (before and after fine-tuning)\n",
+ "3. Compute and plot **averaged over data examples** cosine similarity map \\[between `baseBERT` and `fine-tunedBERT` \\]\n",
+ "4. Analyze attention to target-sentence tokens:\n",
+ " * NOUNS/VERBS/PRONOUNS\n",
+ " * OBJECTS/SUBJECTS\n",
+ "5. Analyze cross-attention (source-to-target tokens)\n",
+ "6. Repeat for other datasets as needed\n",
+ "7. Analyze base BERT"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 1. Load the dataset and fine_tuned model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 42,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "data = read_dataset(task, path_to_data)\n",
+ "model_finetuned = restore_model(path_to_model, sts=False, mnli=False)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 43,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# for big datasets\n",
+ "if len(data) > 1000: \n",
+ " data = random.sample(data, 1000)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 2. 12x12 attention map for a random example"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 44,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxwAAAMCCAYAAADj28DjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsnXm8TtX+xz/7Gc58zKQQkefgHAcZu5QIJTQbirokw63URQndHyl1lZJriJQG0U2FoklJ45U03EZJ5ZYhmR1nPs+wfn/s/V1r7b2fc87znHpo+L5frzqPPay91l7TXt9pGUIIAYZhGIZhGIZhmATgOdEZYBiGYRiGYRjmjwsvOBiGYRiGYRiGSRi84GAYhmEYhmEYJmHwgoNhGIZhGIZhmITBCw6GYRiGYRiGYRIGLzgYhmEYhmEYhkkYvOBgmN8IWVlZuOqqq050Npg/CD179kTPnj1PdDZ+k1x11VXIyso60dn4Q/DFF1+gf//+yM7OxpgxYwBw2/sl/N7e3QcffICsrCzMnz+/SvdPnjwZWVlZ2L1796+cM+a3hu9EZ4BhEsXq1asxZcqUqOf8fj/q16+Pbt26YcyYMTj55JOPc+5+f+zfvx/Lly/He++9h127dqG4uBg1atRATk4O+vfvjwsuuAAeT/kyjA8++ADPPvssPvnkExw6dAgejwcnnXQSOnfujKFDhyIQCLjumTx5MtasWeM67vf7Ua9ePXTr1g2jR49Gw4YNf9WyMr8Oy5cvR/v27dGyZUt5bPv27di0aROGDx9+XPLw5ptvIhwOo1evXvLYuHHjcPjw4ePy/D86M2bMwI4dO/D3v/8d2dnZAIDp06ef4FyZRGt/DMOcGHjBwfzhGTBggO1jAwCOHj2KTz75BM888wxeeeUVrFy5Ek2aNDkxGfwd8Nprr+HWW29FKBRC3759MWjQIKSnp+Pnn3/Ghg0bMHHiRDz55JNYuHAh6tSpY7u3rKwM06ZNw5o1a9CgQQP0798fTZs2RTgcxrfffou1a9di5cqVuP766zFu3Lioz58wYQIaN24s/52Xl4fPP/8cq1evxiuvvIJ///vfOP300xP6Dpj4KCsrw6xZs3DHHXfYPvheffVVPP/888dtwbF06VI0bNjQNgZ06tTpuDz7z8A333yDli1bYvTo0fJY9+7dT2COTMprfwzDnBh4wcH84QkEAjj//PNdx4cMGYIzzzwTkydPxty5czF37twTkLvfPp9++inGjx+Pk08+GY888ohrYTZq1Cj8+9//xowZM/C3v/0NTz31FPx+vzw/a9YsrFmzBpdffjmmT5+OpKQk2/033HADbrrpJixYsADVqlXDX//6V1ceOnfujLZt29qODR48GGeddRZuuukm3H///Vi0aNGvV2jmF7Nt2zYEg0HX8S+++OK45SESieCrr75iDVgCKSsrQ1pa2onOhovy2h/DMCcG9uFg/tRceOGFSE5OxpYtW2zHy8rKsGTJEvTv3x+5ubk444wzcPHFF2PFihUIh8O2a3v27InevXsjPz8f//jHP9C1a1fk5OSgb9++WLduneuZL730EgYMGIDWrVuja9eumDZtGgoKCqLmLy8vD7NmzUKvXr2Qk5OD9u3b46qrrsKGDRts161evRpZWVl44YUX8Pzzz+O8885D69atcf755+OVV14BAKxduxb9+vVDbm4uzjvvPKxYsSKmd3T33XcjEolg3rx55WqBrrjiCgwdOhSff/45nn/+eXn8u+++w1NPPYWcnBzccccdrsUGAGRkZOBf//oX6tWrh7lz5+LYsWMx5QsAzj//fGRkZLjqrzw2bNiAYcOGoV27dsjNzcUFF1yAhQsXorS0VF4zadIkZGVl4b333rPdu3v3brRt2xYDBgxAWVkZAEAIgaeffhqXXXYZ2rZti7Zt26Jfv35YtGgRSkpKbPdfddVVaNWqFYqKijB16lR07twZ7dq1w8iRI/Hzzz+jqKgIt99+O/7yl7+gffv2GDZsGL755htbGllZWbj88suxa9cujBkzBh06dEDbtm0xbNgwfP7555WWPxwOY+nSpbL9nXHGGRgyZAjWrl0b0/sDgP/85z8YOXIkOnXqhJycHPTq1Qu33XYb9u3bJ6+ZPHkyBg4cCACYMmUKsrKyZBt95513sGfPHpfPUnFxMR544AGcd955yMnJQceOHTF8+HC88847tufrbf2NN97ApZdeijZt2qBjx474+9//Lk2lVq9ejZYtW6KoqAhr1qyx2ZlH8+GIRCJYtmwZLr74YrRp0wZt2rTBhRdeiKVLlyIUCsnrdu/ejaysLNx222349ttvce2116JDhw7Izc3F0KFD8eWXX1b6DufPn4+srCx89NFHeOSRR3DOOecgNzcXl1xyCT744AMAwOOPP45evXqhTZs2GDBgAF599dUq1YX+vLfeeguPP/44+vTpg5ycHJx99tl44IEHZHsmevbsidatW8dUBgDYsmWLrT6dfgix1pnOc889h8svvxxt2rRBu3btcMkll+DJJ59EJBKp9P1Ga38ffPCBrLvJkye77lmyZIlsp/p7iGds3717N6ZMmYJu3bohJycHXbt2xcSJE/H999+7rn3//fcxcOBA5ObmonPnzhg/fjz2799fadmceTt48CDGjRuHDh06yPeZn5+PQ4cOYcKECejUqRM6d+6MUaNG4eeff3als3btWgwZMgTt2rWTc8YDDzyAoqIi23XhcBjz5s1Djx49kJOTg969e2Pp0qUQQkTNXzzvgvlzwBoO5k+N1+uFz+dz+R5MnjxZLgyuvfZaBINBrF+/HnfccQd27tzp8g2JRCIYPXo06tati/Hjx+Po0aNYunQpJk2ahKZNm0rb5g0bNmDChAlo2LAhbrzxRlSvXh2bNm3CDTfc4MpbcXExhg0bhu+++05OTAUFBVi9ejWuv/563HnnnRg0aJDtno0bN2LHjh24+uqrUVpaisWLF2PixIn46aefsGrVKlx55ZUAgIcffhh33HEHWrdujdzc3HLfz44dO/DZZ5+hW7duaNWqVYXvcuTIkVi+fDlWr14tJ/sXXngBQgiMGDECXq+33HszMjIwZMgQzJs3D+vXr5f3x0JycjIKCwsrve7JJ5/EzJkz0alTJ0yaNAk+nw+bN2/G/Pnz8fHHH2Pp0qUwDAO33XYbNm3ahDvuuAPr1q1DcnIyAGDmzJkIBoOYNWuWXDjNmTMHS5YsQffu3XHllVfCMAz85z//wdy5c/HVV19hwYIFrnxMmjQJaWlpuPXWW/Hhhx9KX6OkpCRkZGTg5ptvxjfffINly5ZhwoQJeOmll2z3Hzt2DNdeey3OOuss9O3bF3v37sXDDz+M4cOH44UXXkCjRo2ill8IgfHjx+P111/HhRdeiBEjRqCoqAgvvvgibrnlFuzevRvXXXddhe/w7bffxtixY9GsWTPceOONqFatGr755hs8+eST2LRpE1588UWkp6dj6NChSEtLw4oVKzB06FB06tQJrVu3xr/+9S/MmDEDgGnnX6tWLQDmAn/EiBHYunUrLr/8cuTm5uLo0aN47rnnMHr0aMyaNQsXX3yxLS/vvvsuNm/ejGHDhqFevXp466238MorryAYDGLhwoXo3Lkzpk+fjhkzZqBTp04YOnRohWZ3//jHP7Bq1SqcddZZGDhwILxeL95++23ce++9+Prrr3HffffZrt+/fz9GjBiBfv36oV+/fti+fTuWLVuGsWPHYuPGjVEX106efPJJ7N+/H9dddx3279+PJUuW4IYbbsDVV1+N9957D9deey0KCgqwZMkS3HzzzWjbti3q168fV13oLFu2DIcOHcKwYcOQmZmJtWvXYvHixcjPz8e0adMqza9O37590bx5c9x00004/fTTMW7cOFmf5VFZnRGzZs3CY489hnPPPReDBw9GKBTCm2++iZkzZ2Lbtm246667KnxOtPbXvHlz10d0LMQ6tu/atQsDBw6Ez+fDkCFD0LBhQ+zcuRMrVqzAW2+9haeffhrNmzcHYGr5Ro0ahYyMDIwZMwYnnXQSPvvsM4wcORJlZWUxtR3AXARcd911yM3NxdSpU7Fhwwa88sorSElJwTfffIM2bdpgypQp2LJlC1avXo1p06ZhyZIl8v6FCxdi3rx5aNu2LW666Sakp6fj448/xuLFi/Hxxx9j2bJlcm6899578fjjj6NTp04YNWoUysrK8PLLL+Pdd9915Sued8H8iRAM8wdl1apVIhAIiIceeqjcazZt2iQCgYAYO3asPFZaWirGjh0rbrnlFtu1wWBQdO/eXeTm5orS0lJ5vEePHiIQCIjbb7/ddv2aNWtEIBAQc+fOlccuvvhi0bJlS7Fjxw7btVOmTBGBQEAMGzZMHnvooYdEIBAQixcvtl2bn58vunbtKjp06CBKSkpsZe3QoYPIy8uT1z788MMiEAiItm3bisOHD8vjL7/8sitv0Vi9enXUPJRHr169RHZ2tggGg0IIIYYNGyYCgYDYt29fpfd++OGHIhAIiMmTJ8tjt956qwgEAuK///1v1Hu+/vpr13uLxoEDB0Tr1q3FmDFjRCQSsZ2bNWuWCAQC4rXXXpPHNm7caHs/r7/+etT3NWnSJHHNNdeIcDhsOz5o0CARCATE3r175TF6F3r5hBDiggsuEIFAQEyYMMF2fNSoUSIQCIidO3fKY4FAIGqbpnq666675LEePXqIHj16yH9v2LBBBAIB8fDDD9vuDYVCYuDAgSI7O1scPHhQVMTSpUvFFVdc4Wq/9913nwgEAuL555+Xx6hNrlq1ynatM19CCPHEE0+IQCAgXn75ZdvxgoIC0aNHD9G5c2dRVlZmSzc3N1fs3r1bXhuJRETv3r1Fq1atZP/ctWuXCAQC4tZbb7WlS3VBfPrppyIQCIhrrrnG1T5Gjx4tAoGA+PTTT21pRssv9eNNmzaV8wZN5s2bJwKBgOjTp49tLLn99ttFIBAQPXr0sB2nfvzss8/KY/HUBT3vL3/5izh27Jg8XlZWJvr06SNatWplGx/y8/Nt11VEtP7nrON46oz6tHM8FUKIcePGiUAgIL766qtK8xWt/ZXXHoRQ461+fTxj+/XXXy/atWsnfvzxR9u1X3/9tWjZsqVtjrn++uujtpMFCxbI+q8Mytv8+fPlsdLSUtGxY0cRCATE/fffb7v+ggsusL3nn3/+WbRq1Ur079/f1taEUO3wpZdeEkIIcfjwYdGqVSvRq1cv27WlpaWib9++IhAIiHnz5lXpXdAYv2vXrkrLzPy+YZMq5g9PaWkpjh07Zvtv586dWL16NSZOnIiUlBSbZDcpKQmLFi3CvffeC8CUvh47dgxFRUVo1KgRSkpKopoAjBgxwvZvMkk4cOAAANNRfevWrcjOzsZpp51mu3bIkCGu9DZs2ADDMDB48GDb8YyMDJx33nk4duwYPvnkE9u5Xr16oVq1avLfLVq0AAD06NEDNWvWdB2nvJUHna9Xr16F1xH169dHMBhEXl4eAODgwYPweDyoW7duTPfSPU6Kiops9bdnzx68+uqruOGGG+D1enH99ddXmPbGjRtRWlqKvn37Ij8/35ZW7969AUCasgDm+7rkkkvwyCOPYOvWrbj77rvRokULlwbgnnvuwdKlS+HxeBAOh2XaZHoWLdTjJZdcYvs31YVTgk/Ho5lZODVbVIaPPvqo3Hfw8ssvAzDN0PTyFxYWonfv3ggGg6725OSaa67BU089hdNOOw1CCBQUFODYsWNSq7Jnz54K768obxkZGejatastb+FwGOeccw6OHDmCb7/91nZPnz590KBBA/lvwzCQnZ2NUCiEI0eOxPX8119/HYDZDw3DsJ279NJLAZjRrnTq16+Pvn372o45+3xlDBgwwCbNpjrv169f1ON6ulWpi759+yIzM1P+2+/3o2fPngiFQvjvf/8rj2dkZNiu+7WIpc7IBPSCCy5wjdvnnXceAMRsQvlrUdnYXlxcjLfeegvt27dHjRo1bHk+5ZRT0Lx5c1ue33//fdSpUwdnnnmmLd0rrrgi7rzp40lSUhKaNm0KIPp4or/njRs3IhQK4bLLLnNpVC677DIAqs1/9NFHCIVC6NOnj+3apKQk2T+IeN8F8+eBTaqYPzwLFiyIatoCANnZ2fi///s/l73yjz/+iHnz5mHz5s04dOiQy05Vt+kGTNMsfSIFIE1x6Fr6+NSjLRHNmjVzHduxYwfq1q2LGjVquM7RguV///ufbdJy5oGct8s77ixHecRiNw0gqj2vEAJCCNeHXCz3Es4Jn2jatCkWLVqELl26VJj2d999B8A0ZyqPvXv32v49depUbNq0CUOHDkUwGMSDDz5oc4YHzMXRvHnz8Pbbb2P//v2u9+T09wF+eR1FaxMZGRlIT0+v8IOfbKfPPffccq/56aefyj0HAMFgEA8//DDWrVuHXbt2uZxyo5U3Fr7//nsUFBSgY8eOFeZNN+s79dRTXddQn4vXWXjHjh0AENXMg/raDz/8YDte0fNj7Ve/pC1UpS6ilY+ECZXV/a9BLHVGfXXYsGHlpkN5LSwstPlfAUBmZqarn/4SYhnbf/zxRwSDQbzzzjsVtuH8/HyEQiEUFBREDQNeq1atqON9RXlzhnSnsjsDJdBxes/xtPldu3YBQFQfPufcFc+7SMSilvntwgsO5g/PoEGD0L9/f/nv0tJSjB8/HtWrV8fy5ctdEVYOHDiAwYMH4+jRoxg8eDC6du2KatWqwTAMzJ49O2qUHa/XW6GPAgDpRJyamuo6l5KS4jpWVFSE2rVrR02LJrzi4mLb8fJsf2O1CXZCmgnnx3h57N27F0lJSahevToAoE6dOtixYwf27t3rmrSdkENjNG3ItGnTbPb3Pp8P9erVK9dfwQn5eEyZMqXcEJmUZ6JatWro06cPnnzySeTk5EgpM1FSUoKhQ4fihx9+QL9+/dCrVy/UrFkTHo8Hjz32mEsiTvzSOnLa5RMZGRnIz88v977CwkIYhoHHHnus3P1SKovmNHXqVKxduxZt2rTB9OnTccopp0hfmAcffDCm/JeXtzp16mDOnDnlXuP8sKE+8GtAtv0V9c1Y+1o8/JK2UJW6iBZNKiMjAwBcjuOJIJY6o746Z84cV4htgsaIO++807VPz7Jly9C5c+dfmFNFLGM7Bf2gfYHKIzk5WV4bra0B0eeCivJWXl+urA3F0+bpb7S8OY/F8y6YPxe84GD+8DRq1Mg1AY0fPx533nkn5syZg3/84x+2c2vWrMGRI0dw3XXX4aabbrKdq2ziqQgaYJ0SOQBRnRnT0tLKdYamCaC8j89fC3IoL8+xXWfPnj3YvXs32rdvD5/PHFratGmDLVu2YNOmTZU6gpNJU7t27VznsrOzXWFx44HeU7169WL+GNm2bZt0bvzyyy/x7LPP2sqwceNG/PDDD7jwwgsxe/Zs270rV66scl4rw/nhS+Tn59vM5pykp6dDCIFAIFDuQrYi9u/fj3Xr1qFJkyZYtmyZ7UPjf//7X9zpOfNWUFDwq34oxgN9iEfrh3Qs0X0tHqpaF87IaQDkIrWitnM8offcqFGjCgNaAGZIbqeJYlX33Ij2bmKFFm0ej6fSNlzRPACY7e14SP4ravPO+YXaV7Q8O+eoeN4F8+eCfTiYPyVXXnklcnNzsWLFCpfdOpk+Oe1r8/LysH379io/kyT8pJ7WiZbu6aefjgMHDkT1FyGzg2imWL8mzZo1Q3Z2Nj7++GN8+OGHFV772GOPAbDbFA8YMAAA8Oijj1YoQS0sLMQzzzyDtLS0qHum/FLIbCCaj0JZWZkrLHEoFMKUKVNQo0YNPPXUU+jUqRNmzZpl0/RQO/nLX/7iuvfTTz/9tYsgOXDggEuTcfjwYRQVFVXoa0Maomjv4NixY5WaAf30008QQuCMM85wSTUraxuVcfrpp6OkpARbt251nTty5EiF5na/BvRuovVDMkUj2/jfAlWti2ghSakdx+qnlWgqaqdOE6pmzZqhc+fOtv90HzYnZFYUbSxymszFQ5MmTeD3+/HFF19ENefTx/CaNWsiLS0t6jywb9++uMKC/xIqavM0v1CbP+WUUwBEn7ucvlXxvAvmzwUvOJg/JR6PB3fccQc8Hg9uu+022yRGanzdHj4SiWDWrFlywipPOlURtWrVQrNmzfDFF1+47KWfeeYZ1/X04e2Ulh85cgTr169H3bp1o2oDfm0mTZoEwzAwfvx4bNu2Leo1zzzzDFasWIGcnBybE2FWVhYuvfRS7NixA7fccktUKWJBQQHGjx+Pffv2Yfz48QmR7vXs2RNJSUlYt24dDh06ZDv3xBNP4Mwzz7R9qC1evBhbt27FbbfdhmrVqmHGjBkoKyvDbbfdJq8hLYHTb2LRokVyAfNLpKblEYlEbHsFAOZO8ADQoUOHcu8jB+cnnnjC5msihMAtt9yC7t27l7sfDKDK63SEf//99+WeJXp5ydTD2Vc8Ho/rGOXt0UcftR0vKyvDNddcgwEDBsTsR6RDGsnK+is5I69cudK2uBFCyL7Zp0+fuJ+fKOKtC+KVV16xSbTLysrw5ptvIikpyaZNKCgoqNA8L5FQW/j3v//tKsPs2bPRpUsX7Ny5s9J0orW/mjVrwufz4auvvrLV808//YQ33nijynlOSUmRwQ30fYgA8yO9Z8+emD59OgDTUb5Dhw7Yt2+fzVEfSKxm1EnPnj3h9/uxatUq1wKM8kH9okOHDvB4PHjjjTdsgonS0lJXeeN5F8yfCzapYv60tGzZEldffTUeffRRzJ8/HzfffDMAc5BdtGgR7r//fhQWFiI5ORnr1q1DamoqhgwZgoceeghLlizBwIEDK/zAi8aYMWMwadIkjBgxAoMHD0ZGRgY2bdqEvLw8l83tlVdeibVr12L+/PnYv38/cnNzcfjwYTz33HPIz8/Hv/71L2m6lEi6dOmCO++8E7fffjsuv/xynHfeeejYsSPS09Oxf/9+vPHGG/j444+RnZ2Nhx56yGV2Nn36dBw8eBCvvvoqPvvsMwwYMADNmjVDJBLBt99+i3Xr1uHgwYMYO3Ysrr766oSUoXbt2rj55ptx9913Y8iQIbjqqquQmZmJLVu24Pnnn0eHDh3k4m3btm1YvHgxevToIT9+mjZtijFjxmD+/PlYuXIlBg8ejO7duyMtLQ2PPvookpKSULduXWzcuBG7d+/GjTfeiDvvvBPLly8HAHTv3v1XK0uDBg3w9NNPY9euXcjJycHu3buxdOlSVK9evUJH23PPPRe9e/fG66+/juHDh+Oiiy5CKBTCSy+9hA8++AB/+9vfpDlENBo2bIjc3Fxs2bIFM2fOROvWrfH111/jhRdewD333IMxY8bgtddeQyAQQN++faU/yIoVK1BSUoIzzjgDbdq0QcOGDfH+++/jn//8J04++WQMHz4cQ4YMwbp167Bu3TqUlpbi3HPPRUFBAVatWoWtW7di5syZ5dqqV0SdOnWQkpKCd999Fw899BAaN24cVYOWnZ2NK6+8Ek899RTGjh0rIzdt3LgRmzdvxogRI6I6+Z4o4q0LokmTJhg6dCguuugiVK9eHatXr8ZPP/2EkSNH2hb6F154IQ4cOHBcd4UnWrRogb/+9a944okncMUVV2Dw4MHw+Xx4++238dprr+HCCy+M6nzupLz2d9ZZZ+HNN9/EhAkT0L17d+zfvx/Lly9Hr1698OKLL1Y535MmTcJHH32EGTNmYMeOHWjZsiX27NmDFStWuKINjho1Cu+99x7GjRuHYcOGoU6dOvj000/x2WefoWHDhgnX6AGmH8zf//53zJ49G1dffTUGDBgAv9+P999/Hy+//DL69OmDc845R1576aWXyn1x+vTpg5KSErz00kto0qSJSzsUz7tg/jzwgoP5UzNu3DisX78ejz76KM4//3zk5OQgKysLc+fOxYIFCzB79mzUrl0b/fr1w7hx43DgwAG8+eabMoxnvAuOiy66CGVlZXjssccwZ84cVKtWDT169MCdd94pQ5sSSUlJWLZsGRYsWIDXXnsNzz77LFJTU9GmTRvccccdFUYA+bUZOHAg2rRpg8ceewybN2/G+vXrEQwGkZycjDZt2mDmzJm45JJLoi6AUlJSsGTJErzwwgtYs2YNnnnmGeTl5UEIgbp16+LMM8/EVVddVam99i/lr3/9Kxo0aIDHH38cc+fORVlZGRo0aIDRo0djzJgx8Pl80pTK7/e7pHCjR4/Gyy+/jHvuuQfdunVDgwYN8NBDD2H27NlYtGgRMjMz0aNHD/lx/NJLL2Hz5s3w+Xy/6oIjKSkJDz/8MGbNmoUXXngBwWAQubm5mDJligwtXB5z587F448/jhdeeAEzZsyAYRg4/fTTMXPmzEp9bAzDwNy5c3HnnXdi7dq1WLduHdq1a4cnnngCgUAAgwcPxtq1azFnzhz07t0bHTp0wKWXXopXXnkFDz74IGbMmIE2bdrgpptukh8fWVlZGD58OJKSkvD4449jyZIlePXVV/Hmm2/C7/cjOzsbCxYscPWNWPH7/Zg8eTLmzp2LBx98EIMGDSrXZG/atGlo1qwZnnnmGdx1113weDwxv5vjTbx1QQwaNAhHjx7F8uXLsWvXLtSuXRvjxo3D2LFjT2Bp3EydOhXNmzfHypUr8c9//hORSARNmjTBLbfcguHDh8eURnnt7+6778bs2bPx3nvvYePGjWjWrBmmTZuGwsLCX7TgOPXUU/Hss89i4cKFWLduHZYtW4bMzEx06tQJ1113nS3oRKdOnbBgwQIsXLgQCxcuRFpaGrp06YJHHnkE1113XdxhnavKtddei5NPPhlPPPEE7rvvPoTDYTRu3Djqe54+fTqqV6+OF198EVu2bEH9+vUxaNAgdO/eHW+99Zbt2njeBfPnwRDHYynNMMwfij179uDcc89F8+bNsW7durjvX7BgAebPn4/JkyeXG/KWcZOVlYXTTjsNr7766onOCvM7Yv78+ViwYAHmzJmDfv36nejsMAzzJ4R9OBiGiZsGDRqgU6dO2L59u0u6FQsXXXQRvF4vli9fXm4kLoZhGIZh/hjwgoNhmCoxdepU+Hw+TJgwAQsWLMC6devw1VdfxXRvo0aNMHz4cOzevRvDhg3DypVZ89EWAAAgAElEQVQrsWrVqoQ4WTMMwzAMc2LhBQfDMFWiRYsWWLZsGZo0aYLFixdj2rRp5UaxisbEiRMxfvx4HDhwADNmzMADDzxwXDYfYxiGYRjm+MI+HAzDMAzDMAzDJAzWcDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC4AUHwzAMwzAMwzAJgxccDMMwDMMwDMMkDF5wMAzDMAzDMAyTMHjBwTAMwzAMwzBMwuAFB8MwDMMwDMMwCYMXHAzDMAzDMAzDJAxecDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC4AUHwzAMwzAMwzAJgxccDMMwDMMwDMMkDF5wMAzDMAzDMAyTMHjBwTAMwzAMwzBMwuAFB8MwDMMwDMMwCYMXHAzDMAzDMAzDJAxecDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC4AUHwzAMwzAMwzAJgxccDMMwDMMwDMMkDF5wMAzDMAzDMAyTMHjBwTAMwzAMwzBMwuAFB8MwDMMwDMMwCYMXHAzDMAzDMAzDJAxecDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC4AUHwzAMwzAMwzAJgxccDMMwDMMwDMMkDF5wMAzDMAzDMAyTMHjBwTAMwzAMwzBMwuAFB8MwDMMwDMMwCYMXHAzDMAzDMAzDJAxecDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC4AUHwzAMwzAMwzAJgxccDMMwDMMwDMMkDF5wMAzDMAzDMAyTMHjBwTAMwzAMwzBMwuAFB8MwDMMwDMMwCYMXHAzDMAzDMAzDJAxecDAMwzAMwzAMkzB4wcEwDMMwDMMwTMLgBQfDMAzDMAzDMAmDFxwMwzAMwzAMwyQMXnAwDMMwDMMwDJMweMHBMAzDMAzDMEzC8B2Ph0S2vQAAMGo2BZIyzYNev/lXRNS/S45ax4T5NykDKD1m5TTVSk1A7P/STK9OC/OQP8386/EBsO4NB637UoBwqZUR61mRIOC30gsWW89PkvkVRQfN5E5pH3MZdw1MAQA0fGS7SisStpcVhrpBWOcMD8TzN5s/L77Pui+kXWfl2eNX93msaguXaXk37PcahjwmPnrMPNRxpLrGMM8ZGfVjLuO28831adaqn7WyCFvZDMOAoPqrCBEBgkXm76SMCq6j9+R1nbI/y54PM32zbo2aTSrPD4CD11QHANR+4AtbewCg6tIwAMO5Tje052uU5Jl/U2u6r6d6JURYlVHWuVZmKie9DxhAsND8Vev08orkIrK4v3nP0CXqOdRXZBsKyvaBMquOkqsBHke59XqmvNJ7EkKlQe01VKye6Uux0i9U/cP5zgHZZ2Otw8iaceb1Z92g3juVwerzhscLQfmkejM8MKx6lefK8iEOfG2ert/GyjeNG0UyvyL/J7OYNRqre6mNiLD8rdK3+mjxEYh9n5v35gyOqXwAENmx0UzvpNYy/zLtWPoezL5DCKoTZ1/S+tyM9qcAAKZ/vEdPpdL0hYjIdI2Mk2LKGwBENi8w72k9WOWH2qkcT21PVD+dfQsC4sNHzas6Xeu4xlBjoXyHEZWeNsbKMjnbjhAQe7YAADwtLoqtfHPOMp85coV679QnaM7zp2nzWLS+YeUjWCTboFG3pf2c12/2MQAif695Ta1map6QY4zhPqaPD1YaRr1WMZUPACLvzzfvadINwiqHUaOxPe1gkTm20G/AzAfNz5SncJk2r1J7tcqYlK7So34XLAF8yebvsgLzry9ZlgOptdzpW+OUUb1RTOUL3mV+H/huXAfXfOip5NPK1dc87vYt+2VEzf8R6xqPX6ufMnWfHHfoW0ebt4Il5s9aTWMqHwDc3sJ87u2f7JVji7MfGB733GwW0T4WGYah2kGUPiyvt+rLSM6MKY96vmSe4hhrTiT0fk/Y87cFK7/oV4Q1HAzDMAzDMAzDJIzjouGQEliPF6Jwn3ks05SYif1fWf8+GYIkO9bq3PB45QpdfPOSeazZuTBqB8zrHJJjw/Ao6SFpPSAAryXp8NKKO1XdlGJJTaR0IQlG9VPjLmPDpd/JvIujP5r5qdHEPGRpMND6fBiNTcmWOPSteU1aHRgXzTbPFx1SeafVvkPSY/iSIUgTlGxK5E2JmJD5Ny/0ACFT8uGxpHoiVKpdE5skVOf1veb6tIXhgQiZ0hKSvBia9FuXnpaL4QUcEgynBMX8R8VNVF3reKbhhUhKc11fEcFSSxoUKnVLqEiKlJQun6WkuNq71KWr1O6iZ9z6Q1JVjyaVstqi4VFaAim5I42cT0m94sC49B76pUnQSIJs1qkIFikJEfW/w9/CqNXcnpjHq2nUSCqnadhCVl5JqxkOQlhSTIPq1TBU36Py0Lv2pwEiTglMu8vNvynVVd78puS4aHpHAEDabW+p6zVtlbCuk8eSq8Fo0InO2v/60yB2f2Be3rCzeaasUOt/Wnu0yieoTZHmIK0OjNN6xlc+AEYda/wLFkmpuLDqTvYDm9TROlZyVI6LgrS+IqLqp9TSyJHEOVQq63/6hztVWSzNGklMkV5HtUtqzyRVBqJo+GJg52fm31aXaBo/6/2R5tCfpo6RtFtE1G9ZF14YLfqq8/pfj0+2E1k/Qqg+KPuzgLDGU/lM0lx5PED9tnEVzxj9rFWWo+Z7BiAKzLkRP34gn210GG7+tLQTKMlT9U/9y5ei5kSrbRk+NX4JS4Psse4TkbB8N7YxzJI6uzRx/lRtPo2DUqudZNSXacp+UUhzXap6x6SJKToAo06WLJt1o+rPsl6TVZmp/VE+DUPNsdUamMfCZa45R7WfSBTNWMX4/vaM+aPoMMTmR8xn9TDnevnsGo1hyDJoRJkihdTwWvOgNgfRnG+k1LCyG1LnPeY3gk0zR+/boz2I5uw4mP7eFzJtcfAb82A1SwNkfa8JjxdIr2tlVJsLqb6s/ih8qe6+Rnny+u2aPQCiNF/VCY1JIgKXxpPG1+RMpSVjfpMclwWHXFSccgaMdOtDxvqo0FVfhjXxicL95t+jPypzmzrWx44/TQ1a1LhtKj6HalNEVOfTVbEyDadpUBXLSB8fdVu5zZSyLPX5SblA8WFbns0B1+qQpPLOPFkzG7CraIUQ6iNcN0+hzk1lNIRcrMgPYt3cymUWVDljB5uTamTHGzBO7WbLX9TFQhSEbsrgUJ3qqtry1LR0HrCrct2mVVAmeun1KswTUf/+/wAAvru6BU5/epd5kAZGv2Z64MBm2iW0NkYfsEJTmwOWyZLXlpxhGPKYgLZolKYC1sMozZI8a/ETH7SoN3QzNitNkb/HOpepVPFkhlSnhTJNpHvLClTZgg4zB386RKlVv14rz6ESpSaPlMnrXOp/Sj9UEvdHAAkyUJqvFjDWwjP1hiesqyJyMSiO/mDeV/M0VRbKWySsPngKD5h/aeLzp7oFE5EwAOteYdVv8VFVZ/QxRAsqEdE+qKoIvW/ddAIAQiE1hlAbTqkuP27lR5tu+kZpyIlcLTikSU9pvioHmZcJNdag2OpzunlgjGZeOkb3G61ylKrFEy2I5MerRxvHaJz0A+EoZp4knCGTGnonZQWqDvTFi3C8C68fKMs3f1MbgNaHYxGy2Apo3etJAlLN91nywKUAgJTh95vn0uvKshtkvleap/JptW9RdAhGtVNkWXWEiKiPfXp0lLE1qimsrMOw7eM3bgwPxGFTIGfUamYekwKNkHrXyWa/N1KqaWOCtsigfkPzxHNmGzEun6eZ5SrBm3wWEQm5zZ3l4lSZnsVcrBSzTUUK98M4d7KZ3I43zL+vPWhe87fnXfcJIaLOk2r+c4z5gMvsOFp96PWsqpLS0hYFcSB++sT6EYHRrLf5mxbGNCZ4krR5Su/3ZC5vLQwjIajFgvX9R98okZASoMoC6SbFSggtZDuwykvzoAgrMzrmN8lxWXAwDMMwDMMwDGMSp5jid8/xWXAUmBoLm1pWrn5rm39L85X0IWRJQUvylFnS9pfNYyfrqmvltCfTlE5T5IBUpCSMRLBQnScV5cFtAEwNRdzSKgBGozOttIu17FllrG2ZlBkeiAOmtgeHTLMrtG6MqM3OKYnRVdLSbEVzEHM4potje0xNifkv8w9JqA0P4I1fw+HtP8G8vWEX5E89AwCQOetz+zNgVKjtsElhSHXquN7weCtOoxwJHT1fph+n9Fgc2w0AaPZ//4L4wjR5MFqYTtZScqIHF5AKNkNJp6TzNDTpXLItj8LwVtzGpGRHuK+jNpWUAXHM0khUaxhjCbW0DUMVgDQBJPUPFcPwWlohUlGXFQFptex58CarNkjmHWGrzCmauZclCRMFP8OobjmNklQqWGhKyADTTAzQNHGe6I7kFfDjqBwAQOPHdygTRavujHrZ1lXK3Muo3Vw9i55L96XXVSp/Mgui9+FLBqh/WeY3Rko1t6Yto57qw9Q2yBQpKS1uDY7MP2A3+7MFkIC9fqX2U3PGJU1HuFS1bal9ishrSJpIphw2h2297qkNURspPmKlFUZVEKWmNsFIranGVMpXipVP3bxC3ijsWkYz01o9WuUgsyzAblZDadB71N+xNOugZ1LdedX7jxXKT6gY8JrlSRn3b3UMAI78T2oCxIGtVv5rq7xb45uRnCmPCXKGJoSAsPJpD4oQJciGVX7hcug1VDuIp4xFR2TaRk3TUVnkmWOsdMz2+NS7oDZkM9XUgi+QmZ41dhiXzjFP/XcFjPZXy/LKdJ0mqnq5nE7m4SBEnmk26NKMlANp640aTaQ2nbQA32y4CgDQYlSJCjJgaVqdGicnNL/pc2BFGqZoc6X8rZkkxxpQQmfPndcAABo+/I3SAGVaFhy2vu00aTY07YVlkeD1Q+hjK7R36PFF0bDp70mdMxxzgs1pnCxImN8krOFgGIZhGIZhmONIFWTbv2uOz4Ijw7IdLCtUkguScJZYjkfBQhgkQTryPyt3yTLcHygspebISDafdkmtI0xkUppmO09OcOmqpmmFTU5qVfBtsKWdrNmfksSetDR7PpTOfbQqFyvHwfNX07ZcSsc1KVU0aYdwhpENFilNjSWlN1Kqq7CpPjM9QWWrqt34oR/Mv816IWOc6SQnPlxqPqOjKQkRkTCcDtciEpKSdbtDuMd2TFAYW3+qPaymusGVhltqo0nufKmIB5JsiR8+wJGV8wAAte6jMJe69Kj8NiLPeT0Qgt4D+blY11Tgn2KmobQ1rvJp7zbWULG2tEmaHSwCHA7G5E8lCn5WbdiqE1GaB4NsdsnJuKzAbscPKGl5JAyDJGHUdms0UZJ1j2Wva3gBpxMgaUKDxeWEQC2fxo+Zzpoib6cpHQfsYXnNh0pNhcjbpfJLYwD5/BgewG/5Tn2/wTzU6C/WKZ+S+pJ9fVmhKj9JvIPFqsxkt05aBsNTNZ26DGMsVF8maT5Ji5PSo4xlhl0DC5jvmjQu0v/Gev8iBMPpsO1N1pxBNU2XdPy1pNDUhtNqm9rrODGqW1q7sArhjRRrftB9KWRYU+udRELqnVhO3sLwqvnG2ca8fvcxj6GeIUO0p7vDXJMDrPDF71RttWsRLIZh9SEjzdL2O53IAWDXp+bf6vUBK/StDE6SeYoWJMA+/8HwRpEcGzAM+7xialN9tnul07jhrdK8aLS7Qj6P3jFpNmzzlJ++BSz/H2+S2/8oEnZrmGjubn81xBbLaZtCv4eKVb1KLUapGq8ofTlee2HUOC3uMso0HPXfYtXPAIBV556My17fTRda2Y/uw+FKNsavUd3xv7x7bA7lcdDgtiXmj2ARxI/vmc8LXGAeI7/SQ9uVVojaTOkxiLQ61sM1TaD0I3FYpxget/9RJASbFQfdpWlMzGeqgD/0HcT8etx999347LPPYBgGpk6ditzcXHmuZ8+eqF+/Prxeczy57777cNJJ5YckZg0HwzAMwzAMwzCSLVu24Mcff8TKlSvx/fffY+rUqVi5cqXtmocffhjp6bEFsDk+Cw4Ki2uLEmGtei3JgJFeR/l6WJsDGdUayqhO+Xf1AQBk3vURXH4atpWxPVSpzTZQ2sZH3LqsQnOzP8SxEZ4NylNpviY9IX8KS5pzSntl721tWhg8sBt+S6JlSIlAWEnPKGKRLBc0G3CrklOqA9ZmhdLW2pcsJWBy1a9rNqpgO172xpMAgOROo2A07GIepJCgO80ITx4r7C8AFTrXmyz9R4TNn8Eu+ZchPm1RqvR6sq6nDYx0TYgMrepzXR8r4qXbzXQvm4taHSybYCsfxTPNcqVO3+zQusRgkxtlUzZnVKtoEbd03xD3OU+VbHLVJlvFymaf+kWpaXNthpHUoqgBwIGvARmVyXpuak3VFsm3QYZAFSpiEbVTfxoE2fZbNulmdCiSlFPkJC1CVpy28cLqx0ZSphpbqI2QpD2tjtJEvDXXvKb3VJUIaQGS0pV25pQzrDzpfcgeoQ2+FNdmlqLwoAppTXVp679VUHHo0c8oTYrwoketIj8tfaNQ6Sdn5cHwqPohDZC2uZi0v7bqQeTvhUF+GhEZYs1lhy+O/GCeqpNVtbC4eh2UUshTK58RzdeCyk/l8iZpknIKm6r5s5CWjuzRtag2rihBgD2MKvUduldKxzUNT6yQxje1pnu+sDQpRv1clS7lc9s7QOuB5m+KINSyoTafxdKe3H52Qhiw+byYB81rPFVoowDETx+b9zc9V+WftNoUFv/gNtNvEjDnMcC+8S3hTVIaJql10nwiOowwj1l1L37+DEZjK5Ki1JKkKP8sqlfd5ykSpx+Otmmw4dBwCKuNXL5xHwqndQAApI5ZbD7q5Lbm+4bdgqEijUY0Pw0VfZI0OV7bd48d91wSE9R3k6vBaHEhPdn8P0UfPSkHsPwJ5TiUWkuNhaRNTqutxgm/3fpACKHGetqA0eODsCxgVD8UKj3YtwwQWojr3wu/9Y3w3n//ffTq1QsA0KxZM+Tl5aGgoAAZGRVs1lwBx2cfDjJR8HgB3UkNUI0sVCrVbRRuTuz/0vxIBzDnDXMgmX63RzqXOk1TzM7mMKnSjkXbwZmui1jqayOjfpU6ZvHMswEAqVPf1OKjWx1CTpRBIMXqwCHzef4bVqmJhkKWptbU9iZwVJHhdanPEdJUxTShiohyJqaBSS4AkuKfIAHs32aaup0aZYEHawEiQiVa3HGtrvWd0gHA8JX/oW5EdxqXAyzF//b65cBK6lXphGZ44o7GueUJM656l4ELlImalV7KqIXmv4/8oNozOcXBHXrSmXfnv5VTX/SFRnnoz6lKOxU/m/sbGA07qWfTxEmTi+HR9viwXmL1RmpPDtrVuFYzNXE4TRRCJWqS0MysZHoUcx0eIEQfcFY/2POh+c8mZ0PG2o+R7641J/fmT36jVP7SBLOaupA+pK1wlrYQxOS8bnggjuwwf5KDuD6u0McKfeCmVFfhK60FtlGtgftjkEINGx5pRhgXZKLlS1Zp0t4Yctd2v/rga9BRu9nx/vVJWnfaBYBgsRRgSFO86g2lqZJt92uP3XzQtpdMMP74/+QkbTToZJplAVqoZrXXgmzDcsGjmU/RewK03Zut/FHb1BYtSpCghR6Plneqd30vk3gXjiXkUC1Ufulj3PqQFt9tgNHqYvPcSVZwg+Y9lWDuu83m35zLtY9fd39xjxNC5lcJ47Q9f9SN8ZXJyW5zDwc0OQuAmpfk8wAY9bIhPjclpkbOZeZff5rqF9Jc2qPqMKQFXYBVb7rJH2AuNsLqQ9xKzAziAKg+K02htVDgMSJ+fNdM/vTzXOcMq06FEEibZpoiBef2AwD4xz5lCj0A287bFe3c7dzzyTaXkEmniEhTufLujxua67xJtoUNAGX2WFogv2vEwa/Nc7WaKaHLgR/MY2m11bgbRQin9pzS+iP1U+prMLTFh3U9vcOqCuGYcjl48CCys7Plv2vVqoUDBw7YFhzTp0/Hnj170L59e0ycOLHCtsYmVQzDMAzDMAxzHPm9OY07F3Q33ngjzjrrLFSvXh3XX3891q9fj/PPP7/c+4/Pxn+0oV1abcDhqCrVYyVHYVjqOKFtjCUOmxLGaUv+KY9JpzqXJECpzikUICKhKLvRKic22vFSD4UnpQhxlDH1//6jyiXV9JbalCTHJXkQJT+Yx0iL8+L/wXP5fPO3vikVSY6dkmYYcJmUAaajLwDDMkeDxwfxnpmup/tE8xqpbtTSiIOyEuudaZtJCadTaURIZ0d9U0dnQ9VDM8Yc8o/SsKRDhscryyQ1Hc58xUGn5R+qvEmHdus9WaZ2Yv9XyJ81AACQebcpQYY3OeaND1W57KEqbecqkHT9UqS0uyRPjXZ+h/1lOAjZdklbpZtZkale9VOVhpLaLmkHPNpu9uRIHA4qrRtJqZMzIcOLknT2mLWrcqhUpREjp89fbz3fr6SXtOuydF5W2gky8TL8qeo90HsXEaXNcgae8PiVyQCZLJbmq3cqJcdBLT3rGGnjIuEqaRppx1+jfi7k+yHpr3Sejqhd0uUOvqmqHOS8GSpTdSa1E5pZAm1KStLfskI1jvq1TRHJHIb6IQX5KCusUpAKcs5HJKTtIm1pFHeZY63R+CzAcAQrKCtUmiy9b8lw1WRypuqRwkt7rBDJIhxUElW5eWS6kqw7zINgGI6d3WOATE+KDrrMQGgeMBp3kyZARv125rniQxDk0H7ICqhSVqSc3J0BReCWits3iNOCkTi7WlUDqBCNLTNE24ZujnC8wWIYAfMDRRwwQ9OjekN3XwRUX6HQxjLctHZOalOD6j6r3sTO/ygzKxkgQJt74jXHsb4ZKhzzw2Uyn/4b1wIADo7PRZ15X7sujWW8r0z77Zw74p2XXOjho6XVBVlMWGN/crIak05uR7mRdUJBcwC4N+aT4by18VrXSFE7ob6XrMzH6TuA0hAizW0Rwvwi6tWrh4MHD8p/79+/H3Xrqg0kL774Yvn77LPPxvbt2ytccPzWTcgYhmEYhmEYhjmOdO3aFevXm0K8r776CvXq1ZPmVPn5+Rg5ciTKyswF34cffojmzZtXmN7xWQ7qkhI9pKF5Ul4jyO6ZVrh1W0KQvWqRtaGL7ihISLtQfUM17a9r5QztOvLvqNoGVW7Uyl5Kx0jCmHESQGF+SVrQtr+6M12tHF3iJinN8SmpA9k2+1MRXnETAMA39ll1T+5l7ryZiVdJstrsaWujOZsfgd0W0/AmyfCfS7qZWoFR7+51hbnVN/eLRiwb/olQqZIWVyDFixUlzY7iYEcOyA07I2PwJABAZMlgAIBnzLOatEdJGJ1hU6Mh30G4TNrJC7Kv1jQtJMUxNAmRfJcxlxAQh7ab99Roou6UjtnWvzXnSaFLk+h5tPlmqFiTRDo27QuXadJJ8uvQQlpT6OfC/S7/D6O5GSACZQVxhzk0yLHd43Vvhkmbc2maN2nLXb2R0ljotsSy7rQNxQjaZEr3KaB3SfUWLNW0PnbnYPh8VZPI6T4K0TaoAyxtkcNfIVTqllpHgtrYRxqvFFUGoWmKAVOCTu+MxrDUmuresCZZBsywr1Vx5Kxg01AZFjxcZg/RDAClx5T2mIIWZJykxn5DmytgjSnONqb7gVCdRUJyHJf2+WEtHGc5tvPlQ+3Jq9WdNXbp4USpzBR4pOZp8voZi0wfgukj0zQtmvKRK/fJ+rhL4Ub1odjpFxNvyF/igGmdgNOF9gDyIbK+A1KqyzZs1LFCxn+/QYVe1e6T74DmPTl2lrq0h/ClqrZrtQ2jaU/l1+P0OYNQVgIntY6tfGtnmH9HLHedkhp2b5Jyarbm/jrzvlZWH6lmWYTHr/y+YtQIqnDytBlpiqa5ixKGvgqYWlSY44QjVLH49Gnzmq43mI7jAMR3r5nHTuuhtBlk4aEFrZDWFvoYTcFz9A2BydeJvhdDxe5w99R/dWuW3wm/dYuqM844A9nZ2RgyZAgMw8D06dOxevVqZGZmonfv3jj77LMxePBgJCcno1WrVhVqNwDAEMfByyby1XPmw+rnqsGMOhVN2iKiBnD6MEivB0GqUVIt122hFh/SkVN9ZErVP3U03VFVd1iTH9xkAmHd50tVUWXkTt2VIw5/Z/499C2MzAbmQVJr6jtCS9MH68MkHFTmDbRbbLBILT7oOjnoC/XRJgdZH6KZWal9R1LtaYmIcq7LKD9mspOS/zOjiaRMeUcNqFFivDs/ZEXeTpupl3kuqPJDbUGaRUXZfwJq0nKqi+V5KhtgRSezftNCohJILYzkau6Bi9phSZ6cLPD9O7LMRufR1m81gal6svKhReZwRYkJB23lp2sqchaU0AdKLGXc96V1T23laEwfVdbELL59TTqryrJGQmqfCnoX+j4cVLYya/LzajsIU/SxozthkNMmLVRKj6n6l/s/OMxegJh3U7+9hVmW6R/scJk2yg9lfa8c6WydrHbGtfJt+FLUZG69I0PbF8QdGU3/kHMPq7LdamYP8ncc/ZAWjQiHlHlJSHOqBKyyO6LzibD2wZWk7qM2RYIMyku4LMr4A3e6OhSFjOo0rBy3DWv/iFiIkEOuPmaS2Z5uDqObNQFmm5ELAfqo9ELs/a/5kz4m9eudAS08qo5tUcd0p3IdwwsUW/OTFX2wMmS/KjmqlcdhZrznI6DmaeYx62MYkTCMU7ua5z99ynxm9kWq31l735RnNgU4BBnqKsgFj1M4pF8bx1gTeW6seTAIdgIAACAASURBVH+PCTJt+XFI71UINXbokdOc+9kYHi3ClPXx6aU+maT6KbXJSEgtQmmsMQzVV+XePFogFTIFJoFKZRQdrPQSIYR735pISNa5eNzckRz1T4Nx/gwrm1UzPBGRkMs8WY+GKAV28dTh2glmns6d7A4uQe8ccAfQEEIKRClQieeUM1SUQhIayIWRx77vi5VnV6AfEVHCAmqntFDxJSsT66pGGz3O3N0qvn2mfm2mbg1WftGvyO9rOcgwDMMwDMMwv3N+b07jv5TjExaXJAZ6vHKn5CxYBJC5B8W896bIVewbQ0wHyHNf/FGq6KKGj3Oa3dhiiOsOa47wuaRV8afFvbsxAKUVqdZISR1JslSX4rcXQKyfaf4++zrz77dvwMi93F6GtNoqtC5JhHRHKvpJzpSGR0msPJoJjpRuOMxBDE9UCWxlzHrW3MX59imaFKaiDkNSKtqHBZqE3pcMQar0qOYTzmOG61n2a5Rpnjofn6SodK4p1U+Z+q77JElw0usq6RS1tT2faHVh5SMpQ1oSGZ7KQ1UaUdpctDC6v9gJkBBhJakiszQKd5tzGUDxz0kClVxNvVty7k72av05aLseHp8yG6htmkoYNRpLjab4zyLzWMerVRpktkYqe8MTd6jKW/pZWsmyAk2jab0rue+IR0k2ZahVL0SppeHJt2LK126utKmWxtGmQXVISQ1/qtL86RpWqZFzjH0ezewjHvQ9N6hOSLJImlGPV6sT0uKqYBFS61rjVPUOSLNBeS/Nd+9XAE1iq5tFOMotPnrc/GeXMXHvpQLANB0CzHFPOgtbZng0Nur7ilgmOqLkmNQOi29eMpNoMUCZhug7VwOW6Q3VKWk6Iqodyx3KSzVNiBWYhNp3tQYq3HmMiKM/mPdm1FdjsbY7OACgTpayALC0tEZydZXfzLoyv2LrOvN851GVP1wzGVVaYjUnQjdH/iW0sRxKvclKi1Fq7TEiw46L6Fp8+vX09eb1Qx/RTHrIPM7qV+GgMsPRdo2Xpm9aYBFX8JLnbzGTHPQgRJz7xYhj1nhZ7ZRyrzEMwxzHALd2BYBx9WMAgILbzkB6rrnPhT2Mdezo5VOmxVY9w1Da1HgSzbS0IaFSpQGWlhXW/O31qz2OKHhE4QHZTmn8F2WFEJ+sMI91GWPlT5ujHRpTUwNs/cNmueD4dpFhgatmKs4cP9hpnGEYhmEYhmGYhHF8fDge6G4+7OpHlKTDFZ5Q24BK2n6nQhz6xp7hejnKTpjswXXpf7Tdx6W/gea34ZTMEklpaiOZag1iLqMoNJ3bxSNDYFxpbRInbRw1aQP5E+g+I2TnHLacyzbeA6P3P8xj5HBO0j0RlulJSU+4TPlk2Oxvo234BEtyaWpdbI7qlRC829xUzXfTy3FrD0jKJP0kfCnK+c+y55SS7Si2qPEiRCTuepQSywo0XDabXM2/iDRkeP5282+vv8GwdmEne/aK0o2VX2qTKw6ZWiokZyp7fuqTtEvvxvthnDfNPKaXlRx0pa2toez+SbJFdtMpNVQ/JUnjh4/B6PBX8xjZ+AeLVX8nbZ7Nrtpy5JThFivm+wvNdtN0xS4t5LYlVSUJvtevxh8aJ3QpPLW9YInqY5Rf6rfBYmWHbLMzjiI/FA6fK+lvpYVfjsPmmDRRiIRUvuTO2ZRPbTM0WUYtRK+t/NZ5Z7jJ4iPK4Z20WqES9QzpXO9xO4vn7TJPpddVmt56rWIvozWeWg+go45/a1dE2SDUdsxZ31qaNJbN7mTWwc0f7I1boxi3v5hl1w6P37Xxn2y3xYdhSK0ObfaYBLHjLfO3tfGaUScAseNN83erSyvNb6y7WhO2a+Ox/19nhmM3uo3T2hFtAKjN0645W5ufKU/7PpeScnmdbaNK0uJpfjbk86HXietbwNJWrZkIo/9dZvK1msZUPpG307y+qk71eloigge6mJqS8ZuteehXtreRbTmOOX97X7NNNn/uZ8Dl41PJpyN918iAN16Iw9+baZA/oMMfw5luRf3Q6WNkajjIJ+734cNxT/aJ9eG49as/og9HPzOqjyjcrxoCfVySGUd6XaUilU5AqcB/V5vXdzVVcCjJ01SqtFeCGqjk/hsSQy1I5Eeypj6Wg5cWKaSCqELlQRMssroo1W+xZWpBDq+2Dx366IhAbLzXPHSOOUCj7WXKIcv10RDWYs6TQ2OaLI/qq9pgRWp5mrx0h9I42PuJ6XB8alUiQFE5LMdRsedD+YHpXFxEUw3H/TzDI/dYiZVtA82FScvV+8u9xqYiJ4fpag3VombIXACACBZDWO9dRk7SqGr5lIrcU6UoVXJxAcO2Gy4AVR8XzAS1Dxk5Swi5wNIjZUUKD5g/rL9ysg4H1Qc59eeO18h0lRlVqmvBoxY2QczoYZpS3r4ttoFxueVPPS0SdC1kpdmjLxmCohVJ05oU98eIL9m+hwUAw2Op7/WIeBGa+PzRJ2EZEckxURvuD6uYIPMFjx/wOMybbAjbH3i8bhM1EVELQ+dHm2GohYZs8ynuDz4A8GnCD2hjnmGoj4x4Fhx0T/VT1XhKdUH9zvDIDwwyz0QkrKIC6fVO74zMXbXgFWQaOXHF47L8FCmOTCWFX42xrjrTnhlz+fSIaXIHZ/tYbyRlqndNC3qvH6jZRJULgDjyA1BSYE8/WruyfdhXNGo4zI70D8AK7nIyd+q/AADj3xgJ+CgSmGPxWJ4Du7DPk8ZJuVEW7lqwGKcpo4A9shFgj9ymXwfAuPSB6LvKV4BYNd68d8hDVR7PpaN0sAR/v9kyrbaEB8Ian3+thUdV0tlZaN4T0Jy6BRnGaAsJ52IEgGZKZj3fl6IWf3ogC0rTKRQwvJW8V/diRLDRzm8adhpnGIZhGIZhmOPIn81p/PiYVL15t/mwjiOVNIekY9KB0gfXXhh62EPdHEqGwLPO6dIrMukg6UawCIIcYEkinJypTADIKVILvyf2fW4+vuUlMZdRmqqkVNd2xrUc2Q6Yu4oaabXdpifhoAy/K3ca9yVrKnRtB10rv/Qs2h1dHN6h1JeWQ6mRlC7NEmSoRjKZSKmhHF3JOTMGjt1gmqRUu/cr6ZxXVTMhISIQn/0bAOBpO1QeI6Sa1LGTePnpRVG5Uh5jDG8s47xHiYNu35HXIQk3vPbwtjA1XmsGngkAuPQVKx59NNU7qYDLKV95Eh6hSVXjMf2L7Nho3lMnS+Vfhi4lyVVQc/LMp4yo9mzFRjeSM5W0OepOzlae9VDQ0lyKQisW22PnA3YzLQoVHKtZ3EFrt+KUmjYJKABE1lthJ8++MboWU9P+qAQdjt7kKOnxIvK8qZH0XHy/eWnBPlXH0nQ0pNJwagvoPGJvo+ZzrBCpZUXuXcx1ja3cE8WsX3H4ezVmWPVqJKUrjZIrLKxQedVDVdIYro9RtA8N1SW1A69PhRSOI/Rv5CnT9M648B51kPIiw+NqO6fr+zzQ2K6HRS5xhOPU90mg8Vr2e82ROUKauBTNCV8LvgGzLUiNhRYgoyKUyVh5QU3o3w6tkwjLfX2EU1sOpWmXDr4/vANYIckpJLAo+FlpoiwTKXH0B8DaE0L2NTJJIpMvQIU3j4HIQjMmv3H1Y1o7ssZ1K4y8Ua2B3bwSMDVuzv4e1vaQ0dunmYoMS6yc9wVg7eFFO7PbQms7x+Kg6kuxhuCWWriU6u5zWojXaNJ5V6jiUKnqr2RSt9LcW8u4aqkKMkBmZXrbiGHvKT0UcjwmVTK4hC/Zvt8HoOYGX4qmCde+02ispzHc69e+caxQ6Xs+NM/Vy4Y48j/zt2UqiGCxelY0h35HvzU8XgjrmFG9UcxlPJHcm3NiTaomfXl8TapY/8QwDMMwDMMwTMI4PiZVR61NjgyPkvJJCZO+G7BD2RIq1aROmoMiSa5IwkZaitJ8JbnUQjca1upYbrYULlMrYXJmIwmZL1VKeuJBbF1rptduqCYptbJMzu2+VCWdk7v6pmq7TVvl8Wg7EMuwnUoiI53USDqaXkfan9KmZaLkKIxMK1wf7SZLO+V6k9Tz48CbrIV3JR+caJtDVYDQwqfSjtIRknKc0oGucm8CVAnRNgFy2wZXghZezxW21hai1i6ts0HSnMyTccmz7wMAZnQxpZ7TP/5JpiUlXPI9RnfkLM/u1vB4IaqijyWJeKhYtS3y5bAkgkZSuqals6SbvmS1ISdp5I78oKTWYadvRrqyufdqtvTUdqXGTgu3S1JlksCV5qnnxwppSSJBlS5pJc4eZ/7b8GrvwaFJpHupLNQmaFMwCrnqTwM6XmF7tAgWq13lKd1gkfqdTiEmyVbccI0VMaFrD+Rmno6xM1zmCl9r1Gwq7zU8Dq0SANumaYBd3+88B9jHKHp/Ukvi0IzEyzk3mn/13dTlRpWUpu6LoDnkO0KjG4YBoW80Cah2VbBfaT2ISMi1aafp62L5wlFwA10T6olXUhml7zq0aIAevtY6FNJCAevaf5Jek7Sd6uPn7TBO/Yt1szXmJKUDkVT7szw+4HvT8RzthtrTKCtQc2w8NLJ8djxeiGdvMJ99+XzzL2mCRNjtE5NRT+bVpomSGwNavp57tpj/bNhZC6pCviEhOY5IiX6oRNM8kmRXzcNSsxojckf7aEFO5Phezvitb8IIWH1J82sBYFg7mIsPHwWOWKG6z51sJeCR3y5Sa5BUwVhpeFT6cSB++sj8cUp71d6du4SHyyCclg5RglYIbeyQmrhT2psHio+o8MLUH4Mlqt3p/iLOMP+Upr5R4O+EP5lFFWs4GIZhGIZhGIZJHMdHw+HYhAaAigIi1zwCMkqHXMEKGfpT2vIbBoTcsIw0A5pEUq6qrfvKimD4LWkDSaZEWEmrLHtBQbbtp3azS65iLWLbK+mX+6RHSaRkNCsKS2sLM6hJ66iM0g6SJIxKi2EYmv2237JjJGn1R08AHYar9AAl3UupUSXJ6r7vTA3RaXpkEWk7XvG9UqJPm8wJISUyJOWQdr01m2ohcslOOUZNhyapjrYxZIVovjdCl5Rp+Y6GPQSxdV/JUcCyu5220AwxK94zo7ag8yhXhCjdN6RCW1zKV6ikStHUpGQ3ElJ9RW7gRCErI25NhAgryRP5xvhTlRSU7Hn1Ta6cGphQiXqmHpmK0pX2/0H1nKQ4wy9vfd7MRvalQMiyhaa2Q/0fUFI6XYJGEmY9XDCV37IDF5aU3ah1Ooz6baxjZDes2X5LjWmK8jmzwl7D2mAQabVUm4sHGcXFr8aHkKYdBUzNBfUhyzYa/lTpwyHLVZLn3vBPak0iSlJLmh4tMpSs+5I8JV0ljQmlES6rmhbHso9H5in2/ACARwtRbDj6na7p0ccmqm9q/+RD5/UpDZz+HF3jRWmlOTTfMqS7N+4oVWojM22MqmjeoX6lvUs5n2rRfGwhpQEYXa5V74HCrpfkAR+b/nNoN9AqSxg4yfL1c/qqpFRX7ysO9j61GABwSs9bzShQAMTuzWa+GnaReZfvXQ/pLP3jNO0n+WJaFgNSI15WqGmpaFO6ZLiiWiWlR40SKdN3+pDGiBlV0T526/NFRZGW5PeNiEjfQUEaGdJ0dByJdX3M6J4DetxsnvOnKX8J8q0TkXLnDqPSyGTRES/NNu8f85xqF7pWDLDeoSNkLiLqe07ztREUzZB8MmTo4rpaaHIrn+l13JGroOYVWwRF69nCGYXvN86fzWn8+Cw46pqOySIS1gYBh+pLaA6KcgER1kKbaipI10c4DVjaxCZ3SM5U4QMppn96XRWezXJ2NBqdSRlRYfTigcyV0mq7HcKsxQAK9iunOwoH9+VqeKyFgZwq9Q/JKLuvSlMpSv+nj2E0Psv8/e2r5rlW/aUJgpFSzcqHlVYUJ7dYKCp2OynG+lEfzURJkLMXpWe9G3H0B/VhFIVYzbji3llVNwtxLJAqQs+H/N5JrQWjcTfzd13TtGDzFeZux52f6AfI8rlNtSqcoKjOo33Qx4Jt/xnrfucusaES1basQANGRj3tOmuX4LydMMhJ0+U8HrQ7cAKm2QN9fNMHaijJ7lQOqI9CPWhEjBitrQ+o0mNqUU+TOo0r+p4b5KBZdAhGjSZW3qz8GB7Ab5Vno2UKcok1AfuSlRCEHHTz92rv0FoAGF5lKkmTrFfbDTxuUxzYd3UnYYYotj9XD/OrO0zTefpATq3hDuRBfyMhdxvTFxxSMKSVwRme1J/qNveKAbkzOITqlxGtbQHWR6VjP6VISNvrgcJ4QjPLst4TLTRFRAuNrC0yKNwz3edPVeGDyXzHCsEuIuEqLKoqfyd2M0vh+GsfJ5xBK+R1KdW14AHW3yP/A5pYH+sUgKD4KEB73ZAJLpkR1jjVbXYWAydfYgnhDEP2cZpnxYePmv8+4yrVVqxdyGH4VF1Q9/f4pVBNQmNDUoYKqU39T4RVeFkaw6o11Nqzw+TP8EYP6lEB0kQ74yT1zqQplZoXXeZThifKfOiB0APoAECQBB8eDHjNrCdhBapApyukwEMP6S+k2V9N27OritHNqsNICCLf+n6q4QjzHiqV7V/owl9pPk/vxANkWtsiUHAVCl4gdLNMq+0WHVR9jQLepNeV9a6GFS14QFWEG8xxg02qGIZhGIZhGIZJGMcnLO7XlpnDqV01zYYjBKDh0dTSpAr3qt9Sna6He7RW09JkQdtsjI6lVJcaEbkRmzcJtjCPAISl1TDSakvJhadJ95jLGL6vq3nPtU8pia1TbSuEJmHRJFfOMI7hMhWOkRwlpdRX2yBHhqVLjuJIKACfdYzUlhQquKxQqfRPyom5jJElF5n3DH1ESTkdO4VWtostYTY7ktBRCFyl/RIUwrTnJCvvmS4zhEp3zCUNR4zhOEmiBwhXuWJFqrlLjioJvwzjTBqpT7B9vLkjcNazVijeOM2jbJtxxRPm8LAVotejSd6lqtxPiUtNBYV0hi9VOXqSBtDjVVIsZ7AGfVdrj9bX6FlU50WH1G7KUpJNpiqa+py0epXwUDtT2jd647emdhPau3JKEM2z1l9DtSs93CJpB6hfJWvaQelcrcttHKY9Nkmnx5G+2lgw1nCqgNZOS44ocwoKX0nvSdc8ylDFYYhCS2pN2hYhtJ3DaXM/LSgH/XaFn4VmgqJtYFhGY1IUbXOM4UYBIPLyreY9Z09Qz6Z2p6ftcLS1jadyHPbaQ+nq5QmrwCTikGXSWft0uxbJlr72TFk+nwzaYdCmfJUgQ5bbtKOxTMX6NcqMRehSYUDVaahESe7JxC5vJ/DWAvNYdi8zjUZdNMm6NV5Rn06uJt99PO10c0+zbJ2f3SZDLyuTWs2UiTRkukmT3HXdqvOkdKVhorLp4VD1sMjmA1Ra1Cb1ABRUr9KMNlXOk4a1OW1l6HXosmqw1aXdPBeGt0LLgKhp0buhjShXTwSamloq44xhVpl8msamglC5cewW/0wXM71BG/dF39kbAETENifreXcekxqNaKGdaT6SdV+itFramO0skz39+ELhn2jub31iw+JO/ILD4jIMwzAMwzAM8wfhuDqNm1Iwh0TKoyTJwul45/Wo1a628Z0K9+hYnekhEUmCp2tECI9PSUDSLOnq12ZYW7QYEF/ZKMnRz6i05QrcegZJUbxJbgnvhrthnH+HeSys21+T06kKkSjTJI0FSTNDJRDbXzF/n5RtXp15MsSX68zfrQeZ50jSk1bLIbGLkWTNYVEL56r/LU+y4cQwDIgyy37e5aciYFjh/8THT5jXn/k3eV6Xkii7V7/rWXE7jdP7Sa6mNF+ONXllGg8p/UnWbKdJqkrOxfVaIdD3/9n7zgBLqjLt59RNnWZ6cibDIBnJEgQTICKKKEmMqBjRdVdFXWF03RXXXXXNYXV1/VQMqMiaRcFAEERQQSUzQ5hhGCZ2uqHO9+O8qapud9/bzDTgnOdH3+oKp86pk9/wvMeF+y59S7j2ok+2SWv8d2Wkmt1ApJcVo5XI+VXVZqrD4izyv2qOQupHnDvbSBHFbrcJpIaqFAh1RA7TfjNJ5GcuM1LknMag1TC02Z3hNVfcRs+OwbdIAspSVSvpZKn2A78P9yzcLzhE2+8ALxoNvy6ky46drtwjgbCEHrM+pMHAFoR+6B/+m34Tln7b4Ifd0v5aVAe0ntgnjfPeND467P+2ZbXaX7OPQnVA/WlkXLVSx5x/XHNEx3P+XmnLUCwzRfkmfW4q5AYHvEiPhS6ciTR4PDU+HFZznPfxS8rFdmSl+NQvk3khqJpPmwAH5rPaUfZ/4fdz3ZWq5v2doZ3keUJtLZczbbaRDpvAo1yXlryhaaT4CFowf+TLw/FAsKn3P7gIWEZzxxHnhXP3/ib8v/TQrv0bAOCwV5L9f7knpwWEfEt/8zdUQm81kVx3PHYarWu+PXnvjcM/fZORDeJcnglsaYlTbPpA12ONJSUo1J0J6OrZWiFRUgz1iS/OUW39HSUAJdXhmZ+F//JLw6tYu3bEeWC/pSn1uTZ40WU3h3cYn1uZd7gPVfrUd0OcxkvIO+F7lCBjkQQ5NEQt7DPEGphyDf7nHwjnjn9POJe24EdzFhs85yalrvthxPRiWjYc4ozpnK4pcxFyfdoyzn48uLaAEt3HjQvQBU+eVcY53ZjIPZt0spS4AFX4O4iVat/Twu/ykyitkSk5yGXMUcDZoY7A10oV7ZDr7wnndj+q6FRWMmZDjRw3u0sMJ7rXa3N2DccP3BSuzNkIbKIOzGnxd6oOTG1AWrgrlctsDmUiKW40JMJxUmm/+LesJIBRBydAhdJdGpxH/dq/qqp7OJjG+b65JhZAjkHD5KNzp3HjlMisLNwW2CwvM/GqSVjbzRWbw7nc5FmqACeEKLL43SUhpXW3wc3dQ9LLIxNfRE5OgXM8v7gEtIyc9/oWdUama66iC0d/41fCub1PGX8hUu7V9Ljd1bcI04/EC0ibWndsHvTAjeGeubsrg1CHEKazdmYR4oC4DmpSuB/lcQTSn4SrvyULT42wbOqXTbbYAXJorTBV+fXBdM3N3lXTtUITSt+v+VO4rxsTABnjajqOUnsVxqwZi3QTwBujvjkaEZzzUKoqa5YsVuk9rTHDSEZ1U+lV0yNm5XJJEA4BWudcv+0EPp1AIn03pP/IxpFNa3oSXcRZIgCuZ2HzS5R8RJ6l+aRUVYdjLn9jxJj5GHMYHsf529E3z6TfIcT0o+NFLn1X46Dfdkxlx3BzzY9Q/ByZE8fkG4mTca0fWP4seoDGFY723RzVGBUdmjYCwI1f+hoA4ODn/2fxIuUvOeTlOpcIa15FvruyVLYgBhltBUlZYYibuQTpg2EuFLNh6xjOG/UyCc5adTE37dRsbCKBlo3LUbhvCnNv3uTWe4/k5WEcXnFQGDtW3Hh+1+lOBhbSJG5Roc16mS8NkQF/w765cHlWN0CFOgyzrhHTYE53eJ1sNPyGleHc+rvglhwUjoWRlJjB2gmtH+cYJ0zL3y2iSVVERERERERERERExDbD9JhUGWcjD5bKspkVOzQ2dMfKEtGxjXAkZREKyqSsEnRWlVrJOEuma0Ey6++7BW5BkHCw2YF/6M+Ga5920yz5qvbDTcFU5b0Hh3yGaNI5h3irueHdOGskZi7V8pio656ljqJSNqYDLKXhMvhUvhNEsusBkt76PH+298Dm4KyMLhyO3dFv1uMJJHoi5Tcag3ZUtnnJj2oJEonTkhDdLwB8+PCg/n/rdasL7/Rt6EW7dvgmKYqbu3uRjtg6RecijXsTl0Tq0jmjZaPvTyZkqA3Azdsz3H9ciMCLkfXwt3wnHO/1PE3fZduQOMInpaKZQidlvIQi/p79GSMB50i1JiZFgySarP2o9KkmYsfDwq9xehbzBXEkbBmJpSGBkMjBxlE0J71zs3fR/HQpsbrvglMAADv899/USVak1FTe3tlGKkb11hxR0ydjKiBjC/c1pgTOmHFQ2zZaCjdLnWRF6lZIv3dqzo0NE1mYxzvKl/Sv5qhqj1mKnbZ03Bki5+IZizVf+Tgc5Z6iNDZtKQkDm3b2zdXvxHXOZo8umZo2lc2o4EzfypqVIKkU8wxnTFONuQxfZ82GjF9e2ri0k0pfth0DFKeF6i0fadme6xRMi+5LEzrB6kk18wumKXou4zTN1LJsYgcv2qkMnTnXF7U/P2epfjeu13W3h9/dngnkHeU7wEGvf104MPTKns0s2WQ1bWnbsc7VQjNryFO4/3Jd2nm/mTW7874FxyZy3ycCgud9yMSOYPNcpXFlDXOnyJBLpCz9z2kiUhMrpg1lbtZSoBivLDyW6txRUasAjjB+0e+DeerI+49EzwW/DNfzZAelivbJbgrJMXzm7qHxfOayBsisczi+Bl0bl7KeNCYcEy1lDe/CfbXPcZ30DIr2VTTMMxa3GU/YrBtFc/PHObYzBUfUcEREREREREREREREbDtMi4bDM31r2gLyTuOWxi4vOar0qQSAdufe0udaulkAaJnoyyQRcLN21sB/FOXbLTscvp+kdGz7TBoOV55aoKqLbiB603YRPa0dsI1iDMCvugbJboGaUJ1QZ6nzH8NKPfIai/pm+L+FgH/skwJ4jH7oBABA7z9fHV7P9u1LDs46Q3aITxwVtChvuqaoYZgMkwXpC/e4tseMf7g20BVLpHWjQSlKhYwvTaeZzGgxyG6cbYhZcmKj7rItt0sy0kZ+q2hdOF2WKtcGlCaSnRcbI/DsE3AFOcod+1a1K0dWOma1Kt3AnfGxcFDfYmhiczTOxl9KorlaSSQHC5uxWDUgrOngr91qhKBy4WG65KQc4rTcv0CdWlnquJptr/fPRgfvAMs+fAWVyUR/zmtOW6Mq1eZxqDkGlNgZmbUvqfGXIrAUuFTTfshR49ffI+Vj+2WfVg0tLdNfG83XVAL/2XrntpX7hnBloELXrg6aM3fEq7QOmVRjdL2J9M7B1kz5C+92+j1Zqzr0sNY1gyXNzTG5vxtpXsYHJzcvcF90M4yk0/pfVDn4m5GSs1ZE2oBSkI0vfgAAIABJREFUJDMhQEKB7/zYZniKas3Uyqj2q7ZD/ABoXrH+HZ3CBmsraDRyGnJA67wddaxF3qcqqRS15AD82lvo/vDd3P4vFCIHnoekjbQa+t26wXoKjGcl0kJvb4JLUtv19/w65GXZ4fpuOybx+JnTUoTrRjvL7+Ex/Dn/Ek5tWKnauTx1PUrdl5Hb4eCycbV4VvvkEtaIN8VCwFoKjEcC4lwiFhv2Xqa25ud63nklbn1h8Kva+9IH+E5Jw5dq6BrGn0U0zwzxSZtpgglbX87c+swlqkXnu1mr5H1WGwOQDy/1+Rv+J9x/yMvhN9wbjmeSFsNqpLug/I2YfkyP07g127EMKIBOwi5Rh0aG5ZUmR2HMXILC1GUnclG3mki3fD9P9GNbhJ1DmEZu+1n4PfSVU2LkEMdDE2m8gFI1sGcAUq7mdz+A6j+FDYdE3/RemVY4z/nvBmgZa4PATkeEY2apaIyi5x8uo/So4/Oi0KfGpKJz1eMbfnTdpPd4n5p4AzQQlHtNJNwsD/dE6QDG5MXG3KBvcs85S7HTV+7LPWjMDLqcQJx1qC7nFhA2gnSezcipeZOo6DORwElVTmZH3qdaFzxAtupCruD3eXb4/cUHlZ2D2T1g2N2m4CAni7W+uSZmzZDkIVxU8gVf59gMPVqee24Iv0sOVqfupFefBcLCz8bO4Wsc2ZudmxvDhQUcZu4QfmszuzbHcYa5jRmmNDo787u3tH8xa9amVXBLg6mYOHzP2d2Y7GQd251z8LQZk8103zxd+BNzjH0GCbWvzWbD1mPIMDoFO0V7Y64hggx2/HZ67kkk0NiyWp31uW96bxZwuQ1GW5YwVzRjKlV0fGbTOts2p7AxhjVDczlzTH5HqwFU2bx0VPIk9U15svOPz0ccL1eU1ESQQBb9tPn2m+5X8zfu62QOkiw9RDeuHUOZG/MxDCSvPtX7bLwYMU9ic5yynEtJuCcbJbsZkrIPazwZ+VYPi+mxnPv+ivD/2Z/JkrZ0ioE5mgdrwkblkHdRvtyOR4Zzlm2I54tKL1KOBcLpS7s1DuXCoKbjr38wbK7cwn3hVwbhm2NTXTZRLVVUMNohZIN0wFnFa+2iwNMawW9+EJj3JL6T7kvaEp9M8HYU2sbwOuz9bZoPKZaYtKmZS7PECh2C48p4M07IPMzsfN7L3Jk+RBvZwR01DhLMhtpxW6c+bQSrPpe+d7oxw96BPfSBc3fGki/cw7kLP0wUUKrq/PsEwfZGqhVNqiIiIiIiIiIiIiIithmmJ9L4rwMtnjvoJcWLVqpmIleGB4z6XuDUpCWfRrlWiO+B+hZxrhJpRm2GSv9YVSlxL2rwvw90c8nT391J8cLjTHPrXFEqyPm1Kk3exW9+kKgzkY1IzVIOloayihlBUgkATqI8m/dZaSLv9lmNzKYwriQqaKX7nBx/PTGkveelq42aNCfFd+1MBIrwaVOj89ayTpuB3i5rcjeek7poMfJSId9S6djgjpMXDhBKP//Hb8Id/FIqVjZya1D9shYjr5Y3aI5mSADCbUaSyTz/1mSI2yxrutbfrRJ1luqKCY1qb7qpw5QlfLN2MmZQWfgN94pzsTjSz95ZnXDZ4XXNn+DmB+5+NVkhSbOte/5OjSE95nKMbVRNi5gXsQPsJjX5YPOdSbDhtcGUafDiGzPaCABKp1quGTrlpuSbI557iZNR074rRBWUn6SkxA4spWuOZmPphDulPGwCqOlXpxZpnJ03bZTmVk6bV5uh/V/GFevAbWJzsORe4lbk6iF/Lk+JbUwQxeTHOtuzJqgLB/n0th+EZxY/Wd9TV2f7cNGYr3J7Mw7H2j8dwNHZbYR1IHzDPD13UtZj/na1GSiY8EqsixL8xiBZTpiycxJwlOqCtj5cNcc5KbaNw9Eu9hGbKpZzGkdAyznyCPyaP4bjOWTSsnGlOJCzltA/GOip0TtHTD+FRroD3Pbs8H32+MZKo+XkdkEapjQtjh3eq+ZKvnlapJVmKXmpqvOA1aTkzecqvSiQG9iYO5S+m7NbZwXkqO4o1sVEc6BvjupYYOJXTETE0g6FdzZGpK9xu1n9mmAGtfD9l0tsIInZ0wFaH3oKACB5/WVFE1XWEiYlXduIuXtLz/E8XK6ZuFm57+9K2iZs2AP+PmWTPj/Lv6Q5cr1zZK3XzZz4WOJjBzy2kcbPvzlGGo+IiIiIiIiIiIiI+DvB9NDiLiOpj/e6o02NrTsQtA8sgbQBY9iRdO2tAAA3f2+9lg8UWB/S3bFQctbgKsZpGyCKT5LEkSTI3xwCsLkDz4Lb55Tuy8g+HAMLik5tbP/aasDfHWjr3I5Hh3O/+ChwKgVG4jxVelU6w/bHRiIg0t4xE92ZJSl839hmiFTMm3SBIFGeghPgJfcE6dAKl7QXzBE6kfK4pFxwhJtKlHBxas6/y5XbaMcmhv+/oNFyL/iI0Vrkor4miQYvsjbs+fsqvW2pgCWPpHVSBaNXfxXWiNRmwG8MGga/JdjkOrb9rc1UiWBXhWzpL+eLJaarQ1RZt+Qg+C0kgd1ENsFzdjNSSrK5nrscKLPEnMrB9eZKANOxskStOkN9Qx68NpybsTQQNQAiEdOoyQMqkewQM/cnyl4TEVuCsrH/gvWR4muVfpGS+k0UBX3WzqpNZN8QGrc8oFJYq9XIUSZbiNZOJIOVqQXFY41Q2lBt2FoihFgYAmWiVDVO7azFTZUk45bvh/sPeZkGiyvnnIubo2ofbTSZPq8dsv2MIyKzT8eMRUXfkA4gknRb/9YvgX8z4x2CFqSPNFVkT+7m75Whic2UsWyol52RuuYJFEY36DFr+GRO2ksdWDuEEJgY7Wt+3MzkV25qLyOUZ0T6y+2wpD5aQkldU80Gn6sOqMSeNVH0zVy5N/gdoDsNx+7v+Yi+Q7RUuYCqgPZFG/WbUP/P48Pt//RzQ0OdLauvD4kvQCY4XZWuN8k3oFwrzBPpfdeH/5cd3v18IRrOviIpSj6YrUGG7KRbsgGbTn6eNXMOY9Fn7wQA3HzqYhzwlT+Ek11oONweh4aDxoh+Y3HMV82tWJEIkc0Q0MskFExPniptLbcte80G3uV0+fsJVXcNMtfmwiP4xjD8hnvCs08QDcf2hunZcEREREREREREREREAAASt809Gh5XmB4fjq8Ge/jk1A/Lufxrre2/opg15xLdTecD5pkAQ7LTtnaALAXrmaX2gkIxS2lU++G/8OKQ37f+urMCAvBkI+zX3hokv4AEuUFCeUlKRkpsJKGi9SGaxYf+DLfowHC88rfhtmWHU5pbCpLpjB2ytX/U3FH6JJm78+dwe5wY0h3coeMy3ntakFjs+L+r9NszMkGl2tit5u+fgLmk24B9mfQ58FXaUulsh9IOv5moBBujajOa16K5pEjBmzYl7xlfD2tXDkzIzBXybxhLQDa5JD3n9oXvXRh+X/BvcOzfsHDfjsoHAP6hIJVFz8wsrSEg7dXf9A31YbFSbPExYb+HET3H7Ejcr6oz1P6dqQ3X/lXpTrm/1YeymkfA2HKn+u0sTeoEELvxbz2Q8c+QPAGhXvJ2+K16UcLZGCmOJ6whagxnpbSSliueS8exk2VNB7qwG4dpp7bdsT9OPk+AlN+PaiBV0ciWqoY2Naelao4ZhiO2oVatSgZpTovKUvWkImm4Lhjx2Bch45/QgW/YpOmKBNj6RoTj9x4c8hcCqeU0lpP4psm1DgOppvf9Ltw/a8ei1iLjH5f34TDMZBnpOV1nraJoDSuG/dGMszw3cZ9omUCR9wepP5i1qr5FxtVkz+d2VD4ASP/nReFtp31Ey5TzMWg3x2fA1Ma/+wLcQeeEc1aLKv+7zP0hAzy3G9/JDEU3dE781YfhjqKgqJ3OF6R9RrlH/Um61JJIWt7L2PJo0ijMTRLYsIXLTgzlev7VnTOqffuI8I1Pu+JBOafMkfm+NF7GdO4s+D9Kmu3Wf5lEiu/KrZssu2K3QRwfK3ziwMdW5v/Gm6agYX8UmJ7SLg1mUL45WjQ34kV2qYqCutslmejMAILTUS42R8Yxjk2FeDFlIyTLYOd1EcQTG3NMN0eAky/ouoj+gRsonV11QORB3w6ytmycF1ZlE0Wqm7eX3scUmMZBkQdQ/9Cfw6WF+2fNDBjspPwXMp/Y8znh/JKDpmTK8Yt7w0D1cjjhFO8UhUm6VMlEFn+0UKo+igTdJcUhAIz8SzBz673o2mL03wk2Q20nCKcD8mQUwJou010yH3lVHW55YXD2J8M9G1bCr6JFSxcbDtmQulKWXhUAeoJDrTvitXrOGRrYHOWmr/RKDANRr/PCuVUvbCDcwv20XduozjyBcZwYNjMZfhhXnhHMMZ92VWeT5M8fpA1H2tRNQoHGuAydwMzGw+UWLWV16i4svJOKPiuOjRWz4DGTZ554QMw5a7op6wY8tiXl4sa/Hbgd9s3VeuV20BzT9PrmZO6HifKdmfBlLGLTqoZGQBY6cqY6brXfoEwGMmvzMxabTSFtvq3QJj+e2gWnPZePTp0YUw06d+H7z9U8M9jMzpj7+DS3aG011Kyuww2HE8d33TSIiQzabWzYPNAV7/fGHJOFW7Zd8KJ9dTCpcUsO0fcKrXW/fi8aFziP/o5fAP1KWtIp3vfB7wEALnreB425VG6jYc2tLGQRSbE0DjvXOCRzn6Ufl+hiNbOhYMpoppivaftHtk+6p7616znR/yTE93AnXqRjGNN9tzEPmzix1LS1hLLW/RxbWLSbNdIp73l7d3kCsLHB814i85LnbyjU1dWJN+E2snpuHpU0YYRAhiimnVmyz2zIoXS9SQm+9ASjxX2sMzDNiE7jERERERERERERERHbDNNiUhUREREREREREREREfDJx9ik6g1/lyZVEREREREREREREREAoklVRERERERERERERETEVkPUcERERERERERERERMIx4F4d4TElHDERERERERERERERGxzRA3HBERERERERERERER2wzRpCoiIiIiIiIiIiJiGrGdWVRFDUdERERERERERERExLZD1HBERERERERERERETCOS7UzFETUcERERERERERERERHbDNOi4Ug/eSIAwL3kC0BCr/Rp+HW050mbgG+F46QSfptj8OtuC7ctfjKdGwGao+G43Bt+S1VKs6XpNUay1yg9AEClF0gb4ZHVN4f0lx1B94yGdwBwc5d3XMaVL+wBAOzw5VVwxHXm0xYVsRT+96kpd0kflnO03U1b4XsAQLmHLjlNgy3/zDfMvxMu0e9p3wWE843wDd3snTsu4+cOCnX36qtW6Xdu0TeVOvBAYygccz0CWu9cB6UK0BgOx9UBuofy2RgFypReq6Hpc3nqlH65R9OozaD0qW2UanLs5u7RUfnGLto3ZOftvzRnfUiDyuu9L5wDAD+2OZyjfPhWHdiyJlycuZTu0nrjNpFuWBmuDC7T6yZ9z3U8tiWc65mp6VPZ3ZzdOyofAKSffk545qX/Y07moo0m5VCPeXBdc7/zLa0frl/+hdc0pE6qUjYdB8x7xjaG355Z9L46+Js4+YYTQ/vhSnM2+13hvZaZv29SApr1cFztC7+NEWBkfTiesSiblnMm7/TrrPyG7uNvxu8FpE+jVQfGNoW75+/VUfkA4M/Hh/fs+72HqD1amP/zYyycuW6+CY9TJdNfQW29kIY3fSE1d2fHJBnz0ibVI+AGd+ysgABWPCnk5aLfPzDBXSZ/ko0kO2YAoQ7qof+gd64+a/MNaJtIyrnxFuE9dq4CdHxLmzLfuDm7Tl44APeeVgMA7PjZP+q4x+M0j3OAjnVUFj/8MFzfvGz5mmPwa28NSSzan/Jm5j1Jj9tpGUi4LC29h+YEyY98v7rMk8l+Z3ZUPgBI/+vp4XXnfAaoUJ/i7y1zcY/2O5p34b3Ohfz9K33wW1aHFGphDAT/OqdjDNd9z6AeSxmb+o2pTQoqfcDIIyE5XmtMguG3h7ruvfAaPcn55TZi5y0ue1IqtqHmqI4VXC7bp7yZ14HcnD7BuMZjTX1I5mW3YJ+OygcA6dX/FZ7Z70W58c2WtRXmcz7mMvI3LoW2Hvohtefe2W1elmun3mu60s+dto18+mlD+2GHc37E9CKaVEVERERERERERERMI7Yzi6pp0nBsDlLCUqmqO3CWTLPko1QVrYPspCu9Kvnj54bXwf/qkwCA5LkXAzBS/cYoUO2n9MLO2JVrkg/PUlWXyDvc0kOzmfUpPEudu9Bw3PRgyPsO8PDD68LzLIkiOPPe7IWcBqKU6M4+f6t9Pv8cVLIIAB4qlcw868o56WRneNV5zw4HYxu13nxOUuig2qU+umdsE/y9vw2XdzkunGsMAaUePQaA6gx9mdFyhf9LkOZao3IkZfihteH4kTtC+osOpHylqjnpEBlpca59em5/aQv+kTvD8VzSLDTr0u68SM5cUVLF7TupyLvc4A7FjFB79olTiTFrNlKjBbTfq0O8779+CgBY8TLTjqwWkNFuJEz6cveUjUajTQKcBvfJcW4T9M7JXmPpXBfY4ePXhgOWbgIydviVvwmv3PFoI1UmSfamB4B7fh1OHfjicG1kfZDAAnC5clpNl4wlLiloHFy5R8+RdC8jPZc20jkuXUkaDptWmpWAOufajzWFinVhvGl3p3Ntxhhnrk8+lrlkojYyPi786ZXhoL5F30lSXpZ0A4CbtXM4R/0f5V64Sq6d1maIFF2/PV1LGzrWmHxKnWXGN5e5j9NCqaJzS4fY4aIvhoPe2QDNF+il8Yrq8vPH7YRXfeP/wrlbfhh+DzxN2q5/6M+ax76guXHUh7yRCBfaSH0zkIQ+yd/Kt+pAhfppvl5bY3A7PKWr8gHA+z4d+tNF586EfLvRDVpuhqO+ylpqOJ3vWcOIYWDjqnC487F0G6U5tgmoDUpeAYQ6beXXEzMgfbbSq88SZEzusHw9TztD85GzqPA3fT1c2u8FMo75B/8Qzi1+crafAvDlHjjOUx6uBO9z93uv7Q9cvw0dy2mukfsrPXC17uZDAEhoLPTNMbjSOG3cdnnbh2gskHwmffCkHX/vwUsAACtufLDts51lLnd/UprSnBExfZiWDUdy/GvpyKsajBaS/soPAwDcCSuK6s5yTTsQDxqlGtyzLgjP8mTA949tkonHLTpA7pEGz4NdbYaq7/INNG1MSR138tdpseISeBr07zwzmGHs+mXK0xQ6/KMCq5l5QjULC17AdoP6/XcBAGrlXq2rOg3YovruLS6EK/26wOPFZ30IaHH+cou5oYfgkkWaHiNv+uO9bOr8EJkvWTOKvNp8ElT3PVLfw/m1JhUA0KqraYg1qTEbEikTm3EUNj4eugmh51yiEyg/1zMo79AFhH2nWVR3iIuuC3X4wUMX4R3Xr57k7icgeAGfNo0wg4QPbDZZqsq3lo3E4A6464NvAwDsdslLAADeJbr5y5lH0j/hx9lFeHG5wufSoYfC/zVjQth2UzAxfJslUcZsM5zpOt3HFXisHlio/YfrgOrYj202AiwypehfAM8bu5oZh6gNyKKSF2IuQVszM4YVzIgJB6WV+cZtTBAnwgO3hKfm7KHmgjyekADm3LMPB1ZeH87tc1L4vf9GYDmZKC/YLzy2aRVwV9hMexaSiRDBm80TlaU5JpsLNRkz40nepApOzZ26wIyKfldpn1yXdmPO7+XNQqVfBD4s6AmCQ97gU99l0z47vtImw5V74FloxyZbrbqOsWnOtNk5FMzzJkHrpp8DAMpHn69jDed37+eGvK35kwg1xVSrVDXjio7/eevItmbU8vIxeDGpM/O8CG2NSeOjgPSXck371UQCJIOJxkLeaPzbIWGef+f1D7a9v5CP7S00998ZoklVRERERERERERExDRie9s/Tc+Gg52UfAr/648BANxx/xTOHfD84v3WeTovaS736Dk2lVh1dXhs52PhRCOiNSmqZJZCtBrwm+4Pt7FZjDVVmYLUkdMJGpXw/G6XBAky7+LfdUN7iXJb6elWgKjL2zkATwHVl34kHJRraupEqvyMeIYdvtlBLCkDq/8WbnskqMXdoa8AmjknWnZqm7lUtR5cFz7NOiQDwZGaTF6wNpALeJY6L9wv67DbCfZ7jh632OmPTfLU2dmvv5vyuYQuecCxhMtIQdn5mZ9l6VupqpLFjHNk1gQufGN+r3GoB4JjcxuTusngNwcn3Ld95F1If/uJkPRRbwzXSKrsemZp/lhi187syqbbpg2zedlEzwYihbxpUrHvdjwuc503WppuNWcq0hwTzZn0Deewy+veRvkmKWGlR7/3aDDt8KxBSSoq9bVmd3mTIuf0HdS+M+Y6LJ3sAv/wNCPhZ5NR1oq50Ed8uabmeKas3UoIuR20M5+aSOooZWyOar1282LrVGu1gIB8b1fSa+KsXarB5aW69S3aFnjOkIymgMtJtr03c4DRDnDbbuaIMtIGCuLpybCatN6HL9C6Y40MjT3uxDfDLTmE3k/5dolxFqf2l7aAQSY1ILCGqNKrc5sdS/PajMYwPBMY9JC5E7VNP7QWuIfm2GP27LiI5xxI2unGSGiPNg+MVj1LLgKEOuI+eNM3wu9hrwT6F+h1IGsinTNv9KkhkOH2abXICHXO5rFu/l5ZjVgHKA3Q+O5bwAhp+nPSf7dgX0NyQ3OVdYp3dn1D/amwDlCyB+lXpSqkR/Ec5Zx+N8/zF33bxgh8SvNrztR7ImS0t2w2PMrtpHsrifyYwWuiiw9dhAsm0LhHzcbfB6KGIyIiIiIiIiIiImIasb1to6Znw/Gdd4bfcy+BO+zlAKC+FmRz6pxTJ2cruagT9SfZPafDD4vENFCJAlgYqAD9Q7fCzSMbVt5JJ4naM3pNV/w0mDqVpYCN4anZVf8qSP/dMW+G99lm9M7rg1Q5veQ8uNM/SVkgSWTaUoczKqP3aXuHzEmudZxXS1XZzYM3XBJ+T9yz4HchNrr1IZW6Go2SO5js4q8nOlZX0rfnJUBJydBqknSuVdd0GZVe+Fu/G5LbM9g4+9U3URpliDNih6h/7V0AgNo7fln0j2BJVOLhZiwOxyLhVOmU+Jz4VMuVd2KFsT8WCs8K5HuwA2S4Mftg1dJLTkFzdfO3w9NHnw//ULAjT9fdHq6xje7iAzV/JH30zTF1lm2jiWt7jh0oJ5CEByKFcMzfQrRUph10DJY+z1yiWlEmADDtS0Bl9hvuBtaRU6pQ/Fbg64GUwM2g8YLbY3NEnUGZntjUM2sGvKGqdDOWUJnJNjspI0OB2iE+dmWgD36PlcjmHe6hfefR2D9PNNZMlJ5cG88RdjJwPY5tNs7EeXrRqvpTkJYb/Qvg5uwWjmVcnQGMkMRfpN6G4jZh51ZDfc31Irb+hvAj35+TMrrui4tIs17pAdgZl/vXQJDk+7WJzJPfeHEg7DjjQx8EWJsjvi33AwsDpbeMSUI64uDK1N54rKnOAEZDG3IzQlnSsU1qm89aY+r7zpWBfV/QXfkAzD/7jZROSfsiUc9iYJGWmbQxflUgfHC7PTP4rwFwTyH/z7FNht4255NSH1ZSDSoXZvQppT77+Y1u1DKS9tktDH4waI4qtXmH5XNnBmsNf8t34ZafoOUBVOtQKqs/CpdvwT6F/prRDOe0ikFLCjkO96i2UtZNrbHQX0z5pDRT7YdtoN+6e03HeGPGBdevxr8cHNrEe37/d+hbGAEgBv6LiIiIiIiIiIiIiNiGmBYNx5a/hqBBM9KmsMi4/vnhoje2iSyJFNvo/iB5BIB5TwrPzVxSYLBgCZjrnQ3Z0TOjSGJ29jZgDNPhsVSVmUKq/V2zGwFGEuNKGSlEeC299/RP4c4zg5Rz928GViWXlODzrA/ejytmyQSbm8C+GgDStX8BACT5oGIuKVJHdoJ9WYqjgYv8XVeEc7s9M/xW+gDyEwDbivqm1tGhrwjn6puVzSrHBOUfuRPgYHZDLGFeFKTsgGoOqgPA7STZ3Dv4Armlh9E9Y201CxOheuiJlI+WSqqEfYo1ETW105XgTmVTBiPpZKkV+xXQr2+WtS1mJPg5yV3aVKlqzm9p3OB8k8Ad9/bw6xzc0kMy11YcFDQ3lqpQ2m5eu9TNOzuUrOcDz00JGSpiZqcjqSf7FvXOUf8i/l37N+Cprw/HohGpwQ2EbyLSb2f8a1iaPBGdo0/VTpwk7xn/nSn44ezSxtS8vRaD6XiDj4AvVeCeKLSRs3YKvz2D8MzuRVJxpsJ2AwtVU7v08HB/Y1h8IMQnKW0BOYYjkQRXenUss0HVhL2IA0OWzZySY5izEvxOsdsx4Z2//BDc0W8K54hNyVM7dTsfC09UsGd86nPhnoEFRnNDGoiBRfAP/D48e1tgTnKHnUtpjilbE3+3mUvhiSFQWKCqAzrW8LdhDUrLBBacvUvHRWxce3lI5sln6ZjJdLjDD4f0+ucLVarbmb7JxpXKBGipcqncEgBwYGG4VK7BbyapOGtGnANIoyj3sw9IOBt+LCVyl3XouQz7nwF/f2AT4/nHX/PpkOzT3iG+JqJ5q/arNpm1BW0CymZ84NKsT0bGN8wGGczNW85ouqZChT8RrFbJ9QxOcvfkyGs23nvQIlx049+3tiPpXun8hMa0bDgu/V0Y+F9RqqgKXBxodZHhbUwOAEibhjJQ6Ub9w0FVmuxwhJwDAN8cVVVmu0WAjTTuc/R8NqIuqQrRRR+66yVhMNntG4YjPrfQckkJu31tJb0u58iXvbEzCjo70LQBO/8VVbSJLN67au9ChTukZgji1Ggoinlgz0d/hTE/SStq4sL1QotaXuQBgF8bzH7QN6dwH5qjcCe9j/JEiyqOgVKb2f0ied+TuaDm2RxlZtrUzSpPhiPrjfM8tSNrIsV1LHTBhhbXOg1ye5EI7P2FxaqYFZSqgO+eFtdf86mQ3pFvKFy76Jdh0mx98rlIXh3M5zqlQHy8wH8vOH67Mz6t35MWJmKmN7pJTU54Al/+bK0nduILDFECAAAgAElEQVRt6KJV6prv6RnUhcGmsMH2AwtRMK1xCRwvnjMOokAmMnAXWNxT3Gi23dTl3zeyHp765tYmqNjquPLj4fd5/y6xJXgOEGFJfYssMP3qP4Zz/fPlOm9QgmluqFOpM6Ym9qnZ9CktrpoU8gKuVXRutuNul2au3zgnjDVnXvEA/J0/C0nsdEz2puGHZf6TGt/8gI5NPOZWeoF5wZnb7UljGJGioHe2zoVkZuObozquDFLbrPTpuMZjmQgbenRM7wLXXRU2Kce8baGmxWAzJ5+KubTMSYM7qPBpJcVvWnqYjItu9s6UiEmT88zmWZVenaNoQ5MZk3lTeveV4Z5dn951+fC3UG+YuxxuycF0kupk10CxHtY5LFyitc/6e5TamU2d2qxXMmQbbfsrr13Y1LBkiAdIkPooabJTikmWzNpR35pfT/QM4scnhDH2xJ+smTC9duu+wj00x9nNxmTCVX2WNloT3hXxWCE6jUdERERERERERERMI7a3jdG0bDhe9u1fAECGAtP/eEW4+Ix3hF/rAClmJGWVjvcYqfbsXSm9rHbC//q/VD3N77GSAxs1Nh8UzzoRToHubZd/eG9H93Hkc96JP3L+HpjzsduLN3ZgDjQZValIdts5607FROam74dnTz1OHXzvIBX+PqfRTV6l8KQS9mv+BEfaKJH2NkdM3ZKDG5s2PPD7EA0agJtPlMpJWSV7pMpG31z49SGQHUvqHJtitcay0aY7wdhGPbYaL6C9CRRp0/zYZqUaNFSZnqLkFrUEJshUnh6XrwOBWMBoewAY6WaqmpAu4PZtQ0PNoPaSvOrr8P/z0vCaI0M03WS/06dM39zO3KdTiVXXeOp54bc5aiTSNBYMkcNq72zVlrGJR3MM/m8/CHnahSIZW6m2L2okxQm3fx5dMo6fRkPpWZPHDrM2uvgUzOKOeea+nd1I/YRNP1CqwrGTPEmVt/r330pwz74oHDTryGsKmdrZzdlNtEviFHvDV4Gn05xCJkTon6+akML31r4oRB7eSIZbhj6WCUaYHjTTH7pbOpzxgfdJHh1HzmaShevIfGrHQ4GVREf7JAoklzbh1/wpnGPz0bGNavLJ5shMbOEcsJlMJEXTOiBkLUwckex0NNJ1gbpcpM9s9twcy5kjdYYFbL2XNlWCz9+VNYd9c42JIeVveB3QS07diyhYXn1z0WxVKPNbWtemHoS+3JJ8iFkcSedpnkGzPo4WYXy4A16k+c1ruNksrVXXcZrb6JpVwOKD6L1sHtwvee98bMz2C+cSNTOX+d0GJZ0C3f+MxZPfBOCEHwdtRHrfdQCAZNnhhXvC+q+9WddklN2c97vOWoxdv/5g23u2Fv1/xLbD43O2iYiIiIiIiIiIiIj4u8D0mFQ9GJzGwVS0ANyx54cDoRFtQXbjZIePnkF4chRz4sNRMnSrLK0gp6zDXqkSZqLTRdo0UgeyUXVOpUQsEWebXmecTbsBBbSbCJkdOOVlzkdvhf9ykCbjpV+iPEx9H5iVJuf8D8w9Uwqkc9hZ4XlLJbn82ZSoSi6+c3ygK37Bd2/Ua+LjwY7fY0arpUH1AMAtO1zPVQwNLEugxKbbw5HtskgiOf201bXTuFt4AKXbMpoK4zNB//vbfhzu3ydQRbqeWSp9JSdPN7iD+h9x+iJBTeWYEag42VmctEdpC54CzklQKvofPbNVOt8NJpJYcZ3WBoDXfAsA8Nmjg+Pla397etEOu0OMS4ebg/T1R0PheGPItzv5Ayo5ZSmi9cPg789jQnVAy8fta3hdG+m3sZuWQHCUvtU45gMLhlTCX5Zks5alSyTVzhy//YN/CAfUTtzC/cf193o08Glrq/uEiG39Hidqv+A643bvEp0DyK/PHf1G9YupGj8Ncsh2tSxVcyZoXDsw4YBxii0E7TTpddxDNq+VQ/9/7w7PnkRa8iWBqtXNXKpz5hYiGRncAf7uX4dzrOEAgFU3hF/S/osWuDZTfcfYN6ncAy4zE0f4Vh1uEY1/Qw/nMuuEbrabEWD582huSJvwRGAiTufi8NxSEhiu8z1PVo1Nmeq8NaaacF4LjJry1HIOlz7V8ZcDc675ExwTqIgVhRLJuFKXWn/OT6mibaHJaxImHXDi0C/9cNdnqEamne9Gu/lfgh2q03ge3qequU3oGxmSiK7bKKDBStOaak3YcoG+l3NO8uNIszG84jD0XnQd3c95Mv66ea132lRrlIzWx5QNwC5fuz/46tr0hFZfg50+UbC9xTOclg3HyKX/CQDo2+dUdepmExSeHLwXpglR49U3qzkKs8mUKtlozoAOHr1zdDCyTkniNEeDUqVXB1dhrjLmN1NotO6p509+jzNMETxAugTJy78CALjypKC2PvYHqzHesDC56pHUss3RAiON7eRdR3AGgJ8F3nGc/hGgRGkT24mNYnvql4PpFdexW3ZE0fzHJWaTQPXJ+W01gEpJ7wNyzrUcbbUG/2DY1MjGMTXPdemQ2/pc2FCV3nhZYTNriNDhlhObFS/eenSxpdHHWwVOdGYdgUvgJQYJmUCUe7QtVnXz62Yuy5ZZJlavk0s3sMQJnFIbUyme9M676l4AwP8euwgvvWp8xpBOYz20MxeQc1uDK37/4DTrLTORROfl4c7p4osX/7/4d7invjmcs22V6qfA8GKdjdup8q2ZCLfD3iyJQ9j4dC9cSE7+x47uY0dWjkeAtKkmnWxyOrIenkxukgV7d5UPu5lKHwzxb5LFB3aVxrjgBblPdVPoielOxokxdQxnk6nN98PN3CGbVmNESSV4PrHmOUwmYBxsxWmcTWUAnVMymxU2FenOnCP9yzUhyWP/Ae45/0InyeF1GTEdbVylYwjj1svhDjw7XOdF632/07bNYxLPr81R3WiX1dxTBCnyLY3pD5dZYqFsBMjcCjlmuwlR5u+VatwrMSsc1DLzGMvtb2wjCoQH8DpWksO3CJRG1uvcnqlXej87KlvzVC6bOFx3P1+IWVjPLPNeGsvYibzV1LULs2qNbiiuf9ImxKSKBW/CdmmJDbh4lg3RmI5yecr9xTSmIkh1SpogfaKDjVnfit8hJQbLZNdnmDwQ2HyZN22WJazdJsluwuh68xNhrC+/8f/0Xm7XEY9LRKfxiIiIiIiIiIiIiGnEdqbgmJ4NR+8JLw8HVn1tTaSA4JgmUgrjoEnSGYkovffz20cMZvBu3+x0hQ7TSKv9lR8O104kNbbQX86ckkmTvyxEqXYv/u+Jb2RHYis5Ixz7g6A293+8BFgcNDAuJ3W0/NsTajra8O1n7p9ChGN3ZoiSjkqvqIQ9qb5dL5WnNQZcEahXcU74Fn7zA3DkWOvvvipcm7lEnR6FHpdNU8pSZynR4rqlhxkJSK+Wh53vWFJE3xflGlxpoKvylV5/qf6Tp8BkKVW1D/47FMuCvodfdS3cLk+jG40ZG3PCSzs1klSR6pvo1Kzx4TIMLBDTAq5PD6OlmkJMBX/FxeF5bvcmf22dwqnPvOTnd6P+H0FSVflHovE0/aTjWBs2mjNYHb/1XMk2ffSVAIDB/7hV20teE+rTAlmEe9rbVMPEEbpH1mvMA4ZIt0tQU0CuV6N9FImkR5F2lf91XUvGAeAjZwUJ9z9eNzFHvYwxNZK2mjYt+eydnTFd6Aa2rG5raTYY83fXYx63WVrLFLd9c1XLsC4Qb7i5y1UjwiYg1X6gydHEOToz10lR8puJcZAxt6L7ck7OziXw6E4r/uPLgwnUc15dh/9NGEdEw8bmX72zxTRYombvdbJo+N1g0H76+uZi7Kg6aUbKvVoGax7KZssS8duYKrN5DI2zvjVD4mB1hV6iu92yWjVRLNlnmBgTYgKVpqrNYPPG5qi2A5knqD3XZhoTJo6rUYJfF7SzrFVxiw6AJ02c0IyzhH0KsbckTka5p/i8MV10Ox4V7v/rD8Nv7xxx5GfnfTTHdE4oEBw4uMRo00BtLrcO8B5qPid1nRTu77KQ4deQ+uTHem/iRcn4nrZEs5F+9VXh2tmfl/ScREI3adI46cdUGzluTDOoZiO96avh2gFnTakeI6YP0Wk8IiIiIiIiIiIiImKbYVo0HF98x/sBAOdedR6Qo3KTX0vTytKBUlV26uygi7QJv/6ecG4uScEK9p5QaUhSyjp6AoCrwB1/IaVH54SatQVPQZM6pYQDABz8vI5ua6fZkGu8e9/3NNx9TqB5bUcBNyWH73waU3AeHf7X4wAAfSuuFcpEyQtLpFwCR5oNkZTN3kXtyIly2M3dQ+04JWKz2qKKAzEHFjSO6gxvHQ4pmixL7tz8vZWaNC9VGwcfO3pnAMCbr11dlKiYiLDunC/QNZLq7Pp0ox1Qx3CMKPUjPyvPsbCUqTbLPUpHOWORZoo1GwVplrHX7QZDG8a91M7xV84lvai8JdDG+h++J/we9RrxMenWaXhrtOF26FtK9vvNUW0vBZvjtEhYAKi9PF1zzsE/EmiXsYDomUVbUtQCwQR01PI50UoJrSo7lKepOlR2gX1mdSepFCd174vtqDkGTw7JGGTfhzZ209OM1vf/HQBQft23VQNK383NYD8poykkv4fgC0jSY+53W1YDj9wRzrEdPSFDZZzx4eB5qjhFcrT5jH18l7bjN6wL3/Y5zsE99S3hJPtYMM37xpU6B/H86BJ9F0fXTlvGz4WuibbC2r4ztfyY+mlw+Ue3ADnnefkeaUPT7wKbfvMjAMDg8e8qXrQ+cqK5YH/Olo7/VUMCkvf/EA1mWZ21GdU+1ZhwoEDv4eaTxQDPV+ycPhl5QBusf0+Q4M/9+B0azT2/1kjKOuc9+Wx60mn98HPNMeNnxJp51XCIzx/7XkFJRjxYu2Dum8jnowuww7tbdji8BKalb8dlSMqSV2/GM8/ajLM/H05YP5U8xXFzNONXCABo1eG5fTINv20vrOnc/8zw/9imQrkf74hO49sA9w3br5p1uNPTvngubQlLAowKzg0saJ+GXfSk3KCtGVebTieMCMZRd4JNwXi4493nAgCWX/pQ188WkJSx879/BwA6izg+TfjQD8JEftEKaE8RZ3sz0HGd8QaiVJHYKeDItvC62OJIsO0c+NlUKinpcctMErzoZ+fq/oXF+zvEIXOLZhHtzdfajBIygfZqvseJA5IxvZHo6ZOYR0lUeXXEnEqUaux1XPfPEMRM76Tg5LrmvF2x8GMU4XkKsWu2BYZWBTOKWeVa8fu3a0syHtlYCkq8AB5ruOwZ5/E2MR34SsYkh2O45IdbN3m9b0VYsgiZmNMmXLVPrgOA5yjMidtmG8PJ8N3vB4atF53fU2znbfuAxsmRsZKdb3sG4XuzY3r7cnVY1sKzriAMmQxvfTotmhujyJj6AZnFqsRQ4cjcm1ZpO+KNdGsM2HBfOBZ/eWMemOTGVUuokhoHbCFwsXNnyAcq3c8/M1/YhkhF5mxmLazr+wxZg7D4WbIYzkNJTaPpbk2Pv12roenyWqA5Yszzkuw7y71dE6l8/LchbytgNueyLzCbgLxwz5XMesZpmSg+jzMmmvKIbIBz6eQwnpBgyv141s7F9+XHrOaY9gnZQIxovKGWIfnJj4GcLTteS1kNsQzXb32zpiuMpEQAUh2ITuOPc0Sn8YiIiIiIiIiIiIhpxPbm0zAtG44LP/eBcGB3t0IHSmryUkXVZ4xSAjhWL7I6bxj+Zx8MZ04JvyLJqG9WNR87opVrKingHbHVdJQMpRwAjG6A/1GQ4LqXfLXjMq4lRcnyjp8YH84lwLIjKFuhbOnDFAV2zm6FGA7dYqpxOC782DvDwcgGdTSVCNzGWe/+4BDpdnhKOLdljZoOscSkMaKaDeuwHzKoUhRWo1utk0RT7YHfuDJzLuOE12UcjoOP2Yte72HV2XoOoT6oLJ7T96kxfWKH0pJ8I5/XzJSqmh5p1nxjRNunjUxrngFU44VSdWq0uGvItK07BtQMWIq28NO3ofHRk0JWX0/xL0xskLwafyptrlvu+Fn/QjEKXKJ928ZmAYIWKj/WpE34DfeEy4v2D+f6F2i0ZZGc1uWXaUkda9XKNaU+lojwavom5gk7HaNpTcFsac+5UzcbyJu+uWq/aadURm6vQw8pRWfOFKkbiKNrF8+86PI/UwZLcBSLQfj3WdKalDPmYoK8CWVtEI7qVKhkub8+cifwQIgT5fZ/kb6HpelCQlFV86oM1SiyEaw7jMg9sE8wFXUzFmXosjO/t34fbvkJ4ZhNf2btrGMumwAmJZVE8zjM40TF0G1TvY79xwmonvel8CoyO/Jrb4WbvVu2fGxqN7ZpatrUP/48/B5xnpzya0K9uoX70glL5Tpbz/EYyPW76X7ts7WsNtWv+bOmx88lZa0TznttRlECTt/LD63Vb9whLvzplZoHY44HQOm0bdtgR/0ZS4yWjDU9zWLsHvMN0kteF1I/89Mm7zla7vqQxhPLmXYGx+6SzWFH4Lgp/3vsYrzsV0xS0UG8ktqAzHvt1isaAoHMh01sJO+rmvdSjrzFkqywdYM1x3+Ua6OIbYvtbYMVERERERERERERETGNmJ7t4C5H6TFLAihyqR8KznxYuJ/eIzZ/NZFAyE69VIN79gpKi+n+yKFozZ+AtXeG+5/8YrrF0B6yxKY2Q9+R99fwHu6kFV0X8cjLH+j6mYmQp9fkiNofOmwR/ulacmpvYzc8kVShkHa3qLDzZp9K4Vg7YRw7JeiRleJbv4uQCU1XykH3j6xXCTFLNAxtoEjEmqNwRJvIjv5qS9pCt07V1b0PN//lv5HRcIgNqy9eh7G1zQR1yqUl9sUa+FIc75g62UpuChSvbkoaDnfsP3X9zLhplaqovCU4hl58eDAef+cNStW6VcgNuk2jZmghxdadbdFzdWTTL1Xh7/xVOGYNh41kzdLkqlItCxkBv9OnOQdyhLbNdffTj4ffVx+j16ag4djp4m91/cx48EZjJ1L/+0KEYDdvzzCmAvDkNOzm7J4LJhY0Xm0plQlTcj5vN37lqaqR0zyGl9k3F495HCIbejd7Z3V0tRrGfP9sAv6+oD1zOz81m5+0heJ4MTGGbw++T/3NUfXFyAeP2+NZSr0q7bqZpYMFMto2duiVAKRpC2iStoPqobrLXtp2+Z0jj8D3UCRsds5nStpSRdpBN/jSV0Nww1ecl2gbmJezAUhKwJjRStD78j4BrjoAv5FobnlO5La2cN9iP0rK2i+5D1vHclfU9AnxSKcwlNkFum+Obm7HIdImZShqZfxvqTaNy875H3kEOPbVeh8A/9CtcExkwWlU+6WseVr8R2sV8bJfrcYHDw3aiHdcvzpT1pDlYvufcA1Cmg2mwHU1M652SbH+RMb25jT+xK+xiIiIiIiIiIiIiIjHLaZFw+FYKuhT+AfIvp8DtrWj2xMmlYbaWFoaNQqCw7t5SXPZ4fCWpUjSI6kW76IbIyItEUYq4xfQLeMI8OglCJOBpQn/dN2DeO/BQXq14sY2lLnbMh/Ljw+/pYoyQ7CkzAZNY3o7tiP2KbA5aCD8fb8Lt+91impAyrn7++ebQE7MptMAkpq+Hwhth6VApGnxf708PLb78TlGoQ6w2+GaX2GKMYw4/MssXJZulbUR3I5ckqV6ttfKPdkgXEA28BNLbqwGQxg5NAChDcbUMeSbbB36QKbyfcdPgvT3ihND/3vGj9e0f32+jFsbYqNd137P7ctz3YwTbPBJJ1AeWTtR1jbK45TYg8/UYGwcwLI5WmSzcoke7/eMbPpwU7KNd/P36vqZcdNyiQaQZAnrhlXhd+mhcEsPBQD4my4J5/abZfxamOELU9LUTAgj+WXaT+mT4otj2rAwPcmfXP+n8Yn7pAQIc9o++Jw3GkjW6roS3C7HZfPIfog+7XrO6Ft+gOZbGO54rOM27IuMiYYKWGz8W03VbDBl8BCxW/UMmrqhb3DMq/UcjyszlxmNMGmw1hMldGMIuI60agec03EZX/oi0nTXh8TeXvKecBlGilrv5phqbNhHb2CxzjVs0ZDLb0i3ovfwXCjtpWw0V5SGaJN7lOmwU5jxV4K2SjswTEs5CthMkEiZ5xL4VUEj5HZ7ZrjvzuAD43Z9Otwiai9cz3N20/riOjSsje1p1Lv3pbKae9ZsfO7ooJ14zW9UI9RJMGJv+qNQytOazA+vmxI7aDF98l15VClFbCtMy4bD3/C/AIDkae8AuONw9E+K+BkaKnUOqw6kQZWpDtOhNbIQc/ND9FNH5lh+3e1wTLsqFJiGio2d/vrmqBMkmUo4ovVMNz8oizo3uGPHZUw5gumiA5A3r+nWLMSnrfY0qACQtnDR9WGzlN789ZD+fqe3NWVIf/LecJ1ijjxqExdWaffOKvK8C6XtiG44xBxqFG5BcOrzQpVbk+/sR2nQZ+pKl6AQW6W+JWPOwmn4az4THjmITOjIMRflmkTp7RSb/js4xQ9++AVSnjwfv3dJZpIIJ41quWQ2yEzHWGaH+jZxYticJSnpuX52enXmHTS5CD2yn1p0XHKMdnP36PpZSaPNpiGh9J5+eUj/llMXYO9LaUMsUWh1oc8TtMs7BT5K+E2BHjRZsLeJI2DGAmT7QVaQkTPfTMrqcM6LbCYxGFmvBAW8OOqfp21B6DZ71Bxr7k7Z9F0ypQ2HRLp/8X93/Ww7SJ2wKc2ux4UL5RrQoro78Kxwz5qbtV+bcXqr0+ca+lLJX37TYJcVstFXoUuWtIAWe5uD6avjyNmtNpv6UkMXpGT6C59qO+aFKW9oKwPd1yNHE28XD4nz3zdP0vW3/zjc/6Tn6pggJlUNNUkp5YQySUXbG7fJ2gxNj+JbffW0Y/Hi/yGSFBIG8hjhh9bCPXdFd+UDkDz3beHAzucsqEoov5WSCouGaS6euVRjaMwKfcY/cqemIfVKfbfVVOIDXrTv8jQ9pnWCH1qrpmnUhjnOiX/4b4Z4pLOx0VnndSZXyPUD55xutiwtdz4yd3UAbvmzw/EdPw33rQxrCux+go71TKc7ttHEDWKyFbMZFcGjyc+UhAK6YeGy8UbjYjKxuuD61R31fztf5Dcorm8uPK/FpiJIy6X/RMH2tjF64tVQRERERERERERERMQTBs5PNQRlF1h1epCcLfviPQVJvLy+OapScpZut+qi5neDQarkW41igCJLf5p3LK4PqZrVRnIlLYlfE5z33I5Hh2tpQxwL3SwOUjc5hCJzGr2A+Nt98LDFeMd1U3Qk7zAKNwAMvS1Q5PX981VQlTGrwY3jKR8zrLOn/T4soWmnLSFnMtGWuESfZRW5ddLNU0uWKhLpmykxJ8PIu4LEuvefr5ZzeWl+Rm1rJeWtrMTS28i47HxpHSE5vYeIJnLenplI6/SAZk6ocg3tc50kQnM7J2OWKMxbWbNQeE9zDF84LvSfc69aNe47vU87k0x12E5TlmoO7ljURGWQo6VMW2rCxlqkVl2djNl8hqVvSUWlz2tvDSnO29O0YTOGkTkWO6WKs2d9SLWpiw/sqHwAhCDB5U32Onm2E9MHdoYtVbNtFgB8Cv/wXwEAq991MgBg8efvymixxkUXY016LzlosyaC3k0voRMuKzXmc9xnOYBa2iqaIYkGxYmZigRZ7RlULQp/i1ZdnepJIi+O1yOPwJPkPmHK40mw+fwgWR/4199rW2GzPW5jzTEtl5gHNbWN8f0ugV93ezjkAKss8d/ykFJVc1nSJmDoqwVsrkkmy/6uK8P/fXOA20N9JKd+vKPyAUDKWpRZO+kYyO+Q/jRTCGSk3NbMzM4rXHdc5wNEqVrfovMva8F7Z2vkbramGFik3040WESG0DOokasXP7mz8v3wHQCA5Li3ybl22l+fo/iFM0QSjPqQjiNMoEPP+cawmsrZoHg8D/I3qvQC1pQLRpuattShfFCiQ04Kv/5uTU/CCuSCh7bqxizQBpVsZO9vRx5jSTbERNksScVs05A+5LX+TUP6wPT7j0KDP534xhGPLY3vGddOge76USBqOCIiIiIiIiIiIiIithmmZXs1d5clcuzztq5McVfpVccr2S2X9X5LIyeh7Puz98MXHMUygchEWlnX3XFe6m/T7wYsuTH0biIxY2nTONJcloC4CQLVpbf9MNyz6zNUskn26u+4+m742wM9KXZ7VrivXBOHOJaKi/Slnd1wB+h93RfoyMn3ZWknS21duQcp0Q+LBLA5aqR2xu+ApV4sCWmp3bQjm3mWsCQL9hYKPaHi7Z2jPhxHvymbftoCBjoLwsWoHf/K8M42zuYSFM0lEHtwqxx0pex9gGojOL8suakOaPucszsnoFIvG6guLwkaWadpTMUCVBzeZ3X/bBuMKzEvVXHuL4Ld9Q+fHbSTz/npmjbPTUyp2i0cST0zAT+5b1K/caWK1hOPA94bybnXNPi758eaLauVWpXbvulX0lZbY9i04jgAwMz3X8O5DD89g6rB66qQU9eidmRrzQHI0pb4Qfmhh8K1mUvg5oeokYvOeX249rP3IznhoinnqW0eyE8io0XMtXfnnAYmNM6xotmw0lGXG+eNVN3ff324Tehuy9rfhklL2TMI/+CN4T4iCRB/qOoMpZDtEP1v/ko4aDWACkmFJZgqzYnVfg0KyFLccg9kHOF3+lQkun86K2iE9vt2cLZ2PYMaqJSJUpYeBpAvC9PC+xv/H9y+zw/HVGa3QyDR8M1RoJHTWneCX38+/J76H9rP2AmfaZYrvcXxtr4l9A3OP4ImQvy+8uNEtR+49zfheKejNV0e66ie0L9A65V9ONgnYnSDUj93WDx35Ovl2I5nhfvazes5Wl5fG4BbclDm2pb3hO8/cPEf1edmjxPlPZ5fZcc3/kbi68fvMxqKLtDenyKrqb7lhUuxz3cf6izBHF3vOG8FAKS//zKSg1+We95qddlyxaQ5lfE0YtowLSZVoj7+wM3qyGtNZUCTh6jtVVUm5gNzaWE2ulHVcr00WPAis2IaHi9i7MJDVHxaZOYXl85uWAbqun0AACAASURBVLN0MTg5ZEKeguOS30SOjDOXTHJn2LCpU2RRfduOJUiifJto2PJMN2YO134ipLPH8arO5QmRVdnlWtbkCQgDQj6ORHMMMnHmzaeaoxphljcvMxarM+VvKR9PeV3R9KFKg3tjVEwDOnX+v/QpYaA/7Yr2DEuMdgvktqwg3Jb424gjuuFhN2xG8qxVT+eZWKSuvXw3N2fXjsoHAOk1wSQiOeAsU54JIsJ2YIIzGfh7Pfym5Zj3iTvGTW9rmP49/MqwQJn3iTuL0X85/VKlEN3ZJWWNZG3jIuTblWWf4rGsVM4+B8Df9oNw2+4naOYsSxkQ6pbS6KYO/QZaOOYXFfn7tgIjmESb5u81sFAXSySYqX/8VFTfeGl4VztTHUY3Y81P/jmkd/T5Ol5bwRJAhCCl3DVXjMAMb5gPc+1uZL2a3FCsIzRGdfHPpj/NseKCzbJl0dzS6Zwx+s+Baax2wZXaxtjkNzO28qaJylSqAKM8rvAGJSkI4aSNzN5FTEsZfnS9lNWv/QulW9VN2Bj9WmEIsUUlez2/o/IBQP39YU6tnH+5jlvS79hEpqn1JQQpZuykPuXX3yn9RsZzy+ol5CLM4tevAh8xEayo+VFmA0ebR/p2yTIbj2l8rDs3rD/mfvyOCe/LCqvkbDhlTXXzsX4yYz1f66dLOkZKDKqBRbpeorVE5j4WbHZBhiPto3dOZv4qlC8nwG07xzVG4O/8WbhO7SgzX3Jb5/qq9MJf/8Vw36Hn2je2z4dP4TcSaQiv5x7n+OZjbFJ1ejSpioiIiIiIiIiIiIj4e8G0bK/6X/dp/YelMqzSJRMID8NXLSZNiTiLW2m5f4SkpH1zMmn61TcLBR47m/uBRYB1DAWC1IpU7MJp71VqJpF2u9Bw3Pi8UI6Dv9+hatGgE82G3JuR/halxE//UXBMTR+6RRwuC1FHXaJaj24yunAfPSZVtzptmnyVyKSB4qNgeJ2oglX922Ocz5yeA7LOqlx3paqqi598Zrhy76+AWTuHc6wBy0iCupPKP+eZe8pxXvGXia5MJk/e2ZgLdD3zTpZOZp3cMvcYR1jPNLo2lgc5eaOfzcOMdHcKcPufWTy3FaPSt6O7ZU3QvE/cjn8/LGg7mdO903x0ik9cHaR4KwDp047j9ghlsZF4lwxvPUmwta4B5B34pb5KRfW9V0mrW/6ccC41To55CblLVKrdBW45PZA37Pu9icearUETyfTh3jrsMs0zmVtUz79MvrVnyvIpOLRnMIvHRKdjAX8/2z8YVjvlcuQLztBLW5NHIJjFsbTb9lNLlwsEOl12uhWShjbxWjrET38ZYric8q4SfP5b2XGQtC/+3l+GcyOb4PZ/EeWXHcpLGkskrwWAUwdx1vD1zTHxHMK46b99PnDMq8I5MpmTMlUHctL5Dsv4o5sBAM95S1nN3LiP2/l8A8XaGCJt9tw9lP6VLQfm76UOzPx9pF8BXBdiaTCwUMvLsR42PaD01mxmzCQac3cX6XinuPLWMNecNtmN+THUp+3NR3NkB3Lao60pkmhOiNoXaUs1jLn2OFVDFv9XMuU+6CWSv3xKLinBt4oxUZRSnuq+VIYjkzdNq42FC5XfuQTusNAm08svCJeee3Ghq2UinvO65AmCZDvjxY0ajoiIiIiIiIiIiIiIbYbpiTTOtrFm9+tv+X44c8jLw2+lV3fCEsjGK80Z2++3GlnpTTgZfq/4OPCCD4bjShv7ZpbS9c5WR17PNtz0buc6plG1OOibE9txdgMbZGcitLtHzs3fC589OkgJX/vbojQ5EyW7U7AksGe2SvnZafuYt3AOpD7c/KAR8Q//RX1qSCrqTWRtcX7kyPH3XCXRVsUZs6T+DEKBuNMx8KuIwpa1RCwZawx3Hf23+oyXhvTbBQyz0aGdSmCkLCxZzGhCyGeIfYcsxIbYRCgXekzS0vQvMNI8buuQ+8VptQv7fwxTBGKW9HWBTnwCJPBhfciQJagt79t/FySKN5wSNDYHf/e+LJUwihK+bnDhN78c0mo15JsJ8URdA5+xtiNj40w27L46Q/Ir9ZOngoTPBvcDgqSbxzCiKU3mLRd/Ef+Hr4Uk9j5ZM8xtpIsox3u/rPNoz1sL7PiaIf3g/ju2Sb6Tv5qCER795vDcFOvS7X2qHmc0TtC+4BKtF66ztGkCBZr2yueME7iU4a9EuHH4a7hgqgnhYGT9C+Bvp4BsHLyWfO9Q7sk5s06Ok9/+Jnp9ywQLbUNyQON0Qt/D+9QEC7XaAnqWg1By+ZIy/HffGo4PeWEoy6yd1TfiW+Gae9mX4W/9bjjH4xvTp45t7NqhGgAOWKKapoJ/pumLouHn9zWGtTwssR96OBN9PgPfEk2FpcwX/wNO12g9RJvA32l4XftxegKc+u1gCdFuvlDtagnI0ah7r8/IfJ221G9xmIlBeA3jsvTA8k6amzYEzYwbXAZ/+0/CMQcR5OfSxpTmfPfkcyif40cxByAa9+ueGzSiR/xfG+1rqQZU22tx5B0AfBvn9uS5F4diPHQrkgV7555Tmmw/xaCBEdOD6fFYkYVXUwfJPcmZkgfbVl2dv7kBJWWNRm3ja7AzWJ3NZ6iRnfRu3WgI40/dqMwp3dGNkKGTOfLTuuY1H9G6ozKOTX5Ph9gasTycS3DeL8Im6NJnhEFgMmfoSdNcuK/9L/w96o3hX+OUyQ6L+OXHwu/T3iyLf2/NIfJxGdjhbLdn6aRi407wwH57MC9ofuvdKL/pcr0O6HOlapGBbLLyUTTlEFWYnRpz/OK+pfzu1uyLVcQtjhBf1c1PbrII/PoaSRlAMHHgRRtNOJm4FcwKRsxkbvauZiPfRRmnsNGQZ7swq3DV/uLEZJiCDr4sbII/edQSvOHKO+WZRw0yEbHfzotwwbKKZSdP+FQdLsWkJC3cnxk3uK016L7aTPAigCMle59q2/nLVeH3yS8Ov82x9oKRybB078nvGQeP1pHcJWVl++MI0SOPqPnmoa8I16gNp+vvglt6aLjWzYt4bO+Zpe/LMfvYmEnSn5qjhQVLxiGVxyGKXO3m7Aoso7gL3IeH1wnbmbSB1EtUbiUtMQ7yXS5WsUhj56Ts4M1O9WJm54WsgB3bUR0w7dgwN/L7ae7yXw714M74KPCMfwzHdlPEfe2UwC7m//wtuP1OD8cP3qTpgsYcFoJ0gaXnvlX/Yed72vApY2JZF8K80SqVocJENuvqB0bXZ1/A7Ftb1gBb2DRquV7LCwvqWzQqtzAjGnPeVnd1yOaGGSa1HEGNN7GifGYTQvdbh3I20RRTTUq3PqRzCNfb2GYxaZQN2+hGuL3JqZ+FHL/9VPh/96d2HF8kA/Pe/IbYWyINOnf4t4OpoK8PGZM3M49xvtgE0DiKuzwTYLmnaNo8fy+kl5wXjs8gYad1PO+yDh9rbG8mRttbeSMiIiIiIiIiIiIiphHTQoubfi1wKbvnfqBgPpFRQUpkZo6MOgqw5oG1GM1R3QGzZKSm6nSlRyQkZSDntAuXKFUg30985GiOiKYlK9GfGBJFeCs4vm4NZKqVpEkffkqQur712gemRot7XZCWuN2fpRJiMkNx/P0AeKJQdOTQHcybmDrUOH7yOeZLZ6lHY1jrNE9tCqi0rNKvNHs7PIXSMI7c7ARM0ubJIJHUV1xn3peP+u3bO8rn5beWUpWdllnS7Zzeb81yWFLGnPdWImkpEhkcGbcLcgMxqXqcwHuP9x0cnB4vurGN6R+j00jjf/gSAMAtP0lP2sjZQKi3nFmKX3dboBC19/lU61qk5saMM+eInKHgZolwqaJ0o+VcGq2GaF+SHY/sqHwAxFTk0WirtgY4zpBfdQ1AkmU3gzQDbJ625k8A9b9kx6M6Tjtd+duQ3tw9VMqddxZP2kRsTltFzalPNQ0hJNF+7alPyPdMKpoG3++9zhUsJec2ZPIgJCSTYMtbglS6/19v0pN5TaiJDq1Uv6nOe/w9miOGvpmjdbMmbrNq7Fmqb81ZWau66X51sqaYHp6JXe6/Ef7qbwAASv/4m47KBwDp1f8V0tv7+Zp/jg0l8W00JpE4fPfN1ToWDfCwxrViOnzRjGzJRpsGwvzB35PNHLc8qJHY+XsxGkNi+pgsObij8jF5TWgPeVIJNfPJUGmjvQVDNhZJ7npz1LR5M+fY6NsAMPww/I1ktnnUG8I57qPr74KbGUzLujEZF8rdUqVrreidZ4SxYLdvhHHdWysF1gTzerDNMnQyS4909R8BAAmVx3u1NGDt0+Md3z3ysV0vnnp1pMWNiIiIiIiIiIiIiPg7wfRsr3Y5hA7aBPezTpgsibFSAraDZ2e/ar+ROpb1WX4uE1gOgEvUKa+Zyn3+J/8aLp/6n+Ec23lb6VYXuPW0ILGaLOLm1gjG1Qky0gH6xm+9Jkgr/Pp7gJ4gpXNdaDiE2rZcFQ2Ro3TE7rTVQOtrwXa3/AZyQnz4r3BzgtRM7HR9Sx1l89LE2kx1oFsdqBXdov213ntmSxpul6eFYyvt4/x26UDW984r9Nm8o2qqtuL+nl+FezgycWNYItZ6K82q5OxUbVtnMgR2ok9bou3wvw7B+dyxb5HvJZHXrc1vtXsHufSzgcQxOe/SwrWtEeSve3hceHWw+00ve1t4/ykf5AypVLDD1B64OET/XfrFe/RkIfqs9eUgX6T5e5vz9m25N1vJK9e1+Ce1VOpo7Zu5j7TTgnHk+C7wlZP2AwC89KoJNELTALG53vlY8y2oL5PG0+1yXFGa3EnaPO7DCa2xz9NMGwIHDSxaRaEeXQlwuSBkokVoFPMOFJyL/fAjKj1vkY8V0a2i1AtP9dhpO/3oT8I88e5/RTGAJCMp6zzJ/d60ZfbJ8PAQ/yget1YGTQT7z4R8c4DcPvjVfwjX2V+pNaYaafbPo/99YxjuqCKd9qRYFYLqYu/nq2SeSUCslorHwkFDnCBzfE3LxXM0f2XRkpggfybArF9N2qObLgtPPfPt8OvvCsdMM87PtcaALV36ODIFuA1yZzUbCGOpbyPXbTvW5mm8+bSlxG2jXZd24BLgwDPC8c3fDNeefHa4qb5Fyz4FUpyJWnYY94pl3PWS4Ffzy2cHbcNxP3hA5uf83Jzxg8kTqowD0WwwxXOl11jORDwe8fiw/4mIiIiIiIiIiIjYTrC9mRhNy4bj4U+/GwAw/1Onq+Q4b6Npj1nSlFRUYtFPkvjmmDB2uISkFCIB8CqSlt2x112vSE0qcM8jKSpLfYROdQT+gd+H9LtgAdrrlOM7uq9Tzca2kDYLO8bgDkg/F4JHdWWT+4XArpO89lKA7WgtyxIAlCrKHMXvXXqYUrgyK0/PrKx9NGDs5L1Is9zCIM1Fq24kkCaQI9uKb6CgUEzpOG85/Bj5OHSoxXnvkaG+V9z4oOY9H3QMEMpegb1m6tc3lW3DwrlEA08ZFiyR+jzjnfJuZvPIM5yE4HVTsL/c7+njXpoOzUZew+dcAk+ScnfSewEArU+eAgAovfJLKhHtEPOW7xwO0lYmgFQ+D66g9QA8+c7wNe+h9tHMQkf2+84lRnKp0nPH9uIiXU+k3fqH/xpOCZPOCPDIPV2VDwCeusPUaYO3BVypolLJvB/C2CZl/5q1U8dp+jt+Hp454CxDh9tZsDQNPkla0rEtej/fx22i0g83gxjFSMLtZi5VSTW1P1cb1GeEtpS0k2v+pL5RHdr/7z7T+C6KNiynmbF+bjyHWTpUbpveq68j+xMtOzz8X6pqusxgNbQWbgH5J/KYs+pGoaNnKnL/0J/DtXINWHdvR+WyWH3Z1wEAi0+80JSNwO2l0lv0SfEtY6FA9ZWm6gvFczWPuy5RBiv+XuUe9W175tvpWkXZx1qGlZLu75ZG3bM/St+8LFNSDhPS1+sZ0WZ5purP3JPT0LVD/wI4aod+kMppfSdXUiDevZ4/fho5iL/Ywn3h6zTH59dMhqVRxslSVerzuB/SfDq8TllJua65XY9uVCpnZrKy47ZQQLe0TbSy86tvjEz8fSIec0zLhuMnfwkq43NgOmR+8nDOTAbGAStveuKcLiClERajUWciArd15Mo5d7KK1SVwC7tXOX7ty2GCPOcVem6qm4at5cc/rvmWS5C87L+7Tu973/kdAOAFr4Uu/sUxz7yDNxI8kZQqcO3iDOTjGPAE0hjRurJ0k8IpbsyKyBHXLTowe39SKRIITIJ5PVP/7vlvbR3YGG3bAW9KqpW298lEJkQJxqG5EPN1cqz81L8BAHY+8g1dP7s10G7DLYvFJCy8Sm8IMXr87T8OFMld4Os/CuZZL7+gOLRZczRfmJgcCmYB3uuYVDac+HLdtEMAcKVi3/VeLXuoD8imsVQDlhzYSbEyuORv4b0XdP3ktoOYdUg/oHZbHYCbt3zc58bD8DeDQKj/gLOL75pgcdcWJTMWMDGFjeVBeZcNv4k5kAVT62aFZW5wB4lT0SkOXWBjSFF6QvHLc6NrHwU9Tx8PKNGEmANzmk1DykHl7J2l4zdTcO91kpqxEgGKjONTiDMCAIte+mbNQ9LG/BkIi1KeB2RsMzTG/E2aI8X+loHZpPEvf0cR/CBrNgfoRq1l8tEhfnjWMQCAk3+mpliZ8X8CtF0bjONUHqJw0z9MhNJuLnGJbpB3PDqTH8zfC35GZ+QpGbDAJ22pOV9mDkJunjNUvlLnVIc9g0UhmWzsZ0PqnNdkSUkFmfxuVwIcv4/bCAvtjAniEwTTar38OMD2ptGJiIiIiIiIiIiIiJhGTAstbkREREREREREREREwPePemzdqE/57fTS4kan8YiIiIiIiIiIiIhpROK2L3l/NKmKiIiIiIiIiIiIiNhmiBqOiIiIiIiIiIiIiGnE9ibx397KGxERERERERERERExjYgbjoiIiIiIiIiIiIiIbYZoUhURERERERERERExjYhxOCIiIiIiIiIiIiIiIrYS4oYjIiIiIiIiIiIiImKbYVpMqh56+UwAwIJP3SXnON6gI52SjT+YPcf3JXQuzRzTE5xq7piucbj7tMkvB/jZSm82s606kIb73aydOi5j+qlnh2de8kUgKY1zlwOao+GI3ut9CoxtCZdrA5SHRsiHzZ9LNA0uW6sRfksVLY8z726NUeao/JW+8NscBRrD4fZ5e3Zexs89Lzxz5qeAJNd0+P1pI5dXBL0hf3s+l7aAlMpYzpUxKen9nHeXFN/Zqptvnav3tKX1OGfXjsp3zdPD+4/4zt1AuUfLAwDNunnvWPZBl2g+HOfRA2ObwmHvnPBbqmqazVz9pk3Vr3K9NkdD3QJAUsm+03tgbGN45aIDOiofAKx71SwAwJyP/EW/N39r0e+aPsPtybe0jku18Fsfgh/dEG6bsUifzd/PfTsp6zmppxRIcu2Fv2V9C/zY5nBqycEdlS/9zMkhpbM/o+2F3y/5SYGR9eG4b274Hd0I1GZQIlT2Uln6q9Sdbdvy3RJNl4+5Dp0D6kPhuHdWNn2TJzd7l47KBwD+ERpHS5XxdfLOypK4zzVM2ag8rbq2cW7zaJemqcN8GW298rhVruk9dM7N3WOyognuOyOMVUs/f7v2gXw9ZrLHddvSNsvtqNUAmiOUL+pvXH5Ayy9jpy/2jVZT76v2596Z6rg+d3lH5fvlseFdx112Fxyl55s0rnCdugSFukgb2p56qD2Nroff8lB4hPshX6sP59KDfk/AtEUTD6BB34r7w9DD8DSWJTse2VH5ACD9/lvDa5/6Zp178n2mMSLjGPoXUFZamgdpp2NApT9zznVojyJri7QpbcK5rKzVpy0Z89yMJR2lO/KuUNc9F/zSjDW5tpk2dH7vmUnnWu37Lc3JqNJ3lzx6bfvS9+paj41RveZz/ZvrN0mk3bj5e3VUPgC48rjwvmN/sKbtWk3y1w7ybvo2aUPLWBts84BZs9H/hbVe2tTr0q65n9cB6kOdzvmPNbY3if/2Vt6IiIiIiIiIiIiIiGnEtGg4rGZjxUGLw++ND2buaSetCOdc7lzS9pjOtD9mqXNeQt4OIuXrDs2HHwAAVBrD8OtDed2iAymfJi85jYpzCXy1T44BwCclYIwkZixlyj5Fea2ZU232jlQWlg6IhMJKKbvAbd+5HADwpHO+0OZqSdPmLFmJiGgs6Le+RSR6btbszP0A4LmMLKnxHo6kUyLtSPr09RkJCEiL1UQ3OPz1rwkH5R74tbeGPLGksG+e5BtV0kSNkmSub1A1FiSt9xtXASOPhDRYYjayLvy6ElAjaRdL0AHVHNh2mhiNCSDfw29YCaz5U0iuCw1H2mJJdUmlZvzNqD35h26FW7hf9r2tluaF89wYguNzooHhNE3eSQLlH7kDbs7u4RS3jbTVRppM39J7kf52jCNOp2dTYDh8f5UEDks+kp2PDccspe+bo/XJGqmhtfBr/hjyu8vTsuXzXjVdPGaMbQIGgoRZNJhpSyXi9C6RaDeGgbT7SLOOJKXpF89G8sqvdf5gqVI8V66JxHLs4vBNet75q87Sazee2jEJoPbaZR0CWPqZ0P/gW8DQxsw1vzmMta5/gUisPfeFZUcATdYohXEFzVEgqWbzZzWuLE3nfp02pT07GUNHzPdz2WtpM6sx6QDHviB8a9u+Hfc/qxnkMZH7XKsB/8dvh/sPp/EqKct9jtsuozagbdxKzEULRKdGN8Bf+dFwfMQrw+089vXOhhtPij0Bkme+S47Hm/dRG1DNvqDSZh7uvg0xZF5p1/75nqQEnsM6Rc9bvhsOfAtwVT0GgGEa6wcW6QM8vrumsUhgjbAXLZBfd1u4xJqIZh0j/xbGn94Lr6G0qqpxY036wELI+CSaEBqjkn7VenWBY3+wJhxsWQ3Pmhd67/AHngUA6LvgJ1pf/Du8TuuVv0lSEe1Nu/Ve3urFruFkrVeqhjHVlFHmklK16374WGN7cxqfdpYqHnDGHYCeoCjvsi8AwG9cKRsNXnB6ngRadZlULFy7iTunSmxnUsbwacssxs3EwOYuPPHqExhXDToB+mcW8y4p8gDaGNEFlRlAPE9wFUqjOQq4hlwfP2FKtzkGLyZhvNCz+TEmH0BYXDbMYr4TzF4aUtpwD9yCff4/e98dZklVpv+euvd27p6e1BOZIQ1hhhwEJStJQAED65rRn1lRxMgKA6gsuqCuaQV31zWvrotpRVFUFBEkCUrOMEyOPR1vqvP745wvVN073VU9Mz0DnPd5+rnVFU6dUyd/4f3cORrc/KBp194PM/8Id67Tb0K0mRstvIttkvchZ+7AmxYr6ntaXNvlt8PscZLkndMgc0CaoMjsrKgWTdkx/VCf91pZbdKSeTHT92x8sNAiCzPaNNfLsH5iNWTmYdWGj76JV5+brtn8TktmBsVWZYZAG+k2Th/1nGWkNmEtUKTJx09IA26sMbMPVIs62ry2JBdkAOzwOjErYSjTPdpkkkCjrVee5e6lzCGoDdH/IxsbF+g5EL3le4g3Pe2OexdMOB3qf7TR+I9j3CLprTetmnCaWwvb/xQAwEzdHXbILXpM5yz36zetKBRlE7fLC925qAAgbSarTDlo0Umby1KHtDeqf1OQdlpT5pOpRSIvfKoj0hf1AnMMmHM+BwCIV/8dkd/cc5/wfci09co7CIUSzEH/SBkCADzy+r2w59dvc2eoP3pTQVselHz7BaAxBbdJcv+5n7apMC/9ZKKc1C7ix38L4zfoE8Vzcd63f78WAGCOfIeM0zQOknmYHqNp8T+8Xhb/tFlo7WGzPzNzsX/WtW07uBrtF/3JneMNqDKpIrPQ6rAIayh9mgOjkpgVds/JUUg/Zj16A6KD3+COfRnb33SF+3/9wzBzDk6Wt6WzYdNoTMRjn/VziZiWW5nX6dzoJjG9ou+rTdZoTURlTpsdB+x0CCZVAQEBAQEBAQEBAQHbDTssDsdzTeJhzvyMP4jwreNcmd70x5SEMKM000RFllqwRGDUSULInCJ5v6iCE9qCBs2Gv6fQAtvW/NpYmHfqaXxsU1JbUdGrJuWlHVY7gZPKs1CCoetp52Lt+KkkGqw6hUiI7aD7xmaqdxIjqUqhRaQ9WTHfS2miEjC01h13zvT58OrbmYuV+RQ547cr8yH//SuDrEK3ZA5BqvIZe0u5yAGStCaAOLuW+8UUJO2ou/b+pCN7VpzxCfcb1+Rblbx27C9fd//PWYxo4THuHEnq46rczxLTHhgyEyDpHTt2lmAf+KnL+r5nucdW3aMk0f57FVrR4GhJbaqlW6ReWcH1EPG3Ysd2IoGoV+U7ajOulNmf6ZihJMFI3l8bbdR+2Fg+mO6HDY6kPo3WnqRJXUZozSFpNqxvr4baa670klpU0mx87ajZeOfNW6/l4PRzPMNmiIUW6dtkQuEloPHy21kabJc5UxOz4CgeZ0lzbFu70WBqQqZMUVGZMvmxdt1DMLP285n3mo57fwyz9xk+T97cirTKUSFn6SBS6Rv/FfGhr3LpkRlj1Y+LAytlDKfx0Bjl6O3eueiHK7n/2W+91f2e/Sl3R/dc0RLyXLKx0bzGxo1aV+O+s1l0av6xdAt4Ls375qB/cAf1Klizueoed23uYe5aS4e0OerrHTPknHaUT99HY+p9P4Pdxc1NZsHR7lxiLeG1EAMrFGmBb+9tXkMwQTNqfmb67rwG4XHP90MzYx/Yx3/njnd/seRda6oBZ+Wgx2ckxx7L2mHSjPdIOZhQpMTnZD1ApBhCCvBswfNN4v98K29AQEBAQEBAQEBAwCRih0ca1xKPZ7O0g6TZq9+xG97w6Y8BEBtfsxW7btZYNNFsbA2yUgpqrL/pBgDAzNeodFL+J1Y7bbGPAJSzvJdY1MpsxxmxtMN/L5WGrZFDs9iDGnKus1boRFni49OvDIkUOyse+LVL/8QLYTcvd8dkk0sSUWPEuZociuce5jQaUBLmtqn8LcyUXdw50prURpmek22uWzpFikiO4aP9MD3z3Tlv62u8HbbtXSh5yoHKf7wTwMI/CwAAIABJREFUAND6oesh38pTJB/8Opf2xse5bvi7moL4eJCN7ciGhLYjgbgGsxvZffv3dM4UiRn542jpf1oLNroxSSGbBdpR3L+DNWnLbnX/7/7iRofnqCgSXu2on6bxZCf5orSRLudbgGKr3GeVY2NaW7clCs2M0GQMdEztLr7n++4VB/5j84ebp9j07DtvXoVHXu3Ktuh/Vk8orwAafM6yPST0n3aT9+fwtueWNAxTlN8KaUGafVM9DmiaVQ9rve23p2A2PXPFtp7a0N5nwHrSB1Oan0pfUVpnhL31apfW6Z9qpP0lf61CK7hu/NhhV/8NmLGPz4cfB9feL9qfc7/t7k9Q/BLdthpfSIrOvnfDqk9aeRYAhtbJfT2psk8Qz4l53ztA24EViEh7SmQUK+4EAETzDoP19O/i29johwkUYCMv4V9xh3/2cHfp6PNw2aFO47f0LqdxTPht+jEymrmv0hg0WhzYLKQ5DfBtp385QNTkpAH2Whx7w+Uwx57nb/dz+fB6GNJ2U3tuaZP0UuEOAGnPnN/KsHwzpdVoYOVV1PgTWdfsSETPruxuNYKGIyAgICAgICAgICBgu2GHazgIl9y18llt19n/QRdAb/Y1T/I5kjbEy28HAMd81IS5IQ1rYwnS05QW19+3FRoUCqhmiDkpA75yi5OEX4Ita0gS50nqVig1sm0V2xu0I83KkbavThZCqATTEhNbbMuvAdgo7c70OMYqZmYiCV/bVNiVd9Fd7ieuolHsov4nKT2xldXKLAEnbQHap4okln6H1gBky57W1hRaJUBUDlzxP48AAJZ+RNHiFlw9EM0nZuwlea4plpN0/ZQ6RfqcDlZlIqboZPrjaU3Yr3SwvFRAL9s6BRhel6+A/d7nYGidaNWonNP2cL+VIeWPQuwnKjCWt2+3rd0qgF05kUcUSjCk1WHfj6Iwa+qAiXRM/bXoy1fqQCJQZ04064Ok2Ygf+Cmifc+ccDqEtGbjjpf34bCfrcmRy4mBJMSmbz8JTkp1MOjz1D1XApi2qUBi5IPAfkIlsSMnrRRpHWtl2I1PunO+z5viFGG+oX5nazDt3u+CmHuI0jgq5PZxIPt/+/OPw5x1ZbJc1B+6+sTvgrSLu7xQ8kQB80qdMF5LYh/6hbtvn5e7a5Vh6bc0BjcZF60xQtvKfcKXr6VzQox4WfBsnvft95222Lz2mgY2MWqzVlEss6/i4GpYaq9MqT0gc8KsA9z97N9lcNGvnI+EJba12qgbHwFmn7LVYfZ5sx1+rqG2pANA5vDzuucsp+E88CerGrW9nkXLnHwRGjQXUxbI3KEp4GkMJE19gqEwNYeWOpoHiCa/KhAtrp/z45q40GUuYcBkYqfZcADPboeyVU85x1TtikcdwZBqFGrAoEHIyOaCeeATEdO3DL1AJ+fWphS7zTABOs437aPiZPDgk14wSXRQo3nPKxRhlLjuq00oHN0myG0iVLwDIKkapnOldhmIyIGaTYCMUGFmRY0m8gFl1rPJp+cHtYEVrOrmuhztZ7MIogx0Jkgp/nsdjR1+oqGJp1ZmsyxeFM05mM08aJIgTn3TPpWdofNgQacfwLXZWtpsqV5N0ssCbuPlFyrx2gdcGlN347pr3BiqyUNPUOz8qs4Vkm2RHKDRNiV3O/3KBy8FALzvlndJen7Tns4rgEQsF5taiNlnbuNJmmmS6VpUlLrzMKX2ho2x7o82HVtAR5LPAWp3aRMEjWjfMzPR21oby0J0jLGDynXoT1fJBjJD3VhFj5xrEUAbAtjGBTDRHK9/WDaxvu+gMiQLbJ0/WuiQ+Ruh0CK04UQyYOvcJk2LN2+sldnUhYkPCKX23PV46XGHAACW3rlCxjCi2db0nrQIpf41slHMNsk5HoAd8cQI+7zM/VKaxVZlwif93KZNz2qjSgDh369jRNA4tB3wbJ33zcvcWIN6WczbaCzlb1gQU9qiigVDjvwFNb+VvXCL5hAyX4urTLEPMt1q6wX3KL+gt/ddC7Pnie4cU3YT1XhLfgIOAJtZzmUScwYAWNMkZhmTcdTBc0Bd5YU2yUzRrAWuGdY8xqiNRipGW1RsnMt2cjzLLMC2GsGkKiAgICAgICAgICBgu8HYprYq2xh5zSI8drhDWQ5zo/ghF4U72uWFjdJ/iggKA3vHN9zRIT6IzsBKGA7a5be7g6vEWXEL1LYAEurGxiidW4a1sQTMItOhDIiJenLq7mhwjiXH3NX3wnjnMpay1asigSQpSayC35HKlSVqsYoY7tWna+7ndBPY/Iz7JZpfHZyLzBDIJGMckHmQvfVqmKNTTnDe7AKVQY4Ay2YMo5th7/yWO3ewd9atlRsd2ckkA1YCj+nI6CnJomnrZY2GXXOvO0eS9sogty9DpkIZEP/lq+6ZJa+UOiRNCUdQ38RO7Rw5/bb/hDn8XHeO8hnXRYtEdagDL5JEn9q/Kcg3UxJW+8uL3WUKPKa1W14tz5S245XvYW9SMusAaWsUeJHypqMxk/nN4GoJvEUOyJVBFzEegJm+yJ0jM62RDRJUi+q5rVfeESvtFkkWOSgWReCNpY1mLB+A3ONpWpO4rfGZw2fjo7dnoM/NMZ5yfyuURGrK2kvfbsv9oh2jehlaq+iKlZkHSY1JK5UgKPD3jW50v61TREtCEtOoiPirzkwpepePMM2S24KQRmQca75xmGsnb/7DShkn0yYrgJSd+pkxsA//yh2SyZwxzoQQkKBuWqtK4yppwQst0v+ofMos1Pr2ZSiIoYlYMm28w3omPMfnfaZkb6Z99zDGsMbWPvEHd27fM7l96YC+rIVLj5E6iKpJmhHpd/7upbNxwpVfcdeXvIJyoDLsNZlpLd8YiB/7jXumY4a0LZ5jfR+pDIhGxZPb2I1P8HtY+9YzT+Z8ateKbCS9drFxHQ0kEFFR+kLKnFqbZRmiG97J8YfjdyyN73E3Tq5GKGg4AgICAgICAgICAgK2G3YqH440lt7yCD5/pJOynH/r1geg2q7omsOHDVSxSkJn9jvbHVedxC2atjsqVzm7y5YLHO2s7Z6bzQmRdvotnWNqNtjun+gkTTQxijySEtq6Mj7UAXngA2Z5SRlTq7bDeilzRM501RGgQpLfaiINmEhJir1Ubua+SmPipdIDK4E197nr81/gfr10ytYruYOq2Uc9Le7h54o9M2lmBrxfRd9+LDHnb9gxQ4JAkfS6Ngp7gwsGaV52RfJFcV1perxUs3OGOKh77YcttHA+TJ8PRKYC5rHWI4eGA+3ey0gHZkr5q9j1D3MgQvYnOeR14iTI9I5lNNjpRoqOkSTGpFEYXA1L0jWSXNWrwKKjE+eY7hFxo4RrPNzzM/d7yoHSrkgyzoGoKkLtS+y17dNgn3G0uUQ7aiuDov1J+7S0dEuQQ8pjrQzT6tswURtrf5hKkuTB1qoTcsblQIbp4G36HkWZS99/8J8OQudlt7lnKX/lQanXlDbVSVi9tk07babwkVuXNQbv5CBmI+KMn0PDAeu/S6UsbauUDCQGANb7E+FRLz0+4q3Kd0pJYNM0of43ISn2Gkg9llq6XN6M6NxvJNJN2O2n/HnGw2tOFi0aa2QqKZ+zlk75jlozk/axKA/Aek0vERlQgFhbbfTNcG3DS9arfnxr7W7QVnNfrpVhiTo7Vyknhot/9F3cfaaTUB/00+1PUDBheA2nLbaIf0PCTw9AsQ2WAr4uOsWdG1rjtGjwPjaAJxLxVgLlFMmIrQNeS8njbfccWNKSe/+aE6651vn9qXOijVaarhxgjVZcgyYEceck6K+98fPueIkro5l9oPS1xDxMfifer26MPJmoAGuT/hpWEcWY9BoEys0rYKfETr3hMK1dvNHY6R3KNj0JALA9cxucq8TMZFQGH2+WEy+7FaXzr3fPaqcuWvz5STrhIK6cprPATMAxtRl++SY3mJz+a8Vcw36FWs2aGgjiGkdA5c1XbSQZbRqpiZ4nSTKjqIrJAW1u2nqBPU70r21J5iOuKSf8jFjzqPudf4Qswsm8iZhDHroOZvcT3DGZ2xRbRTVMk3apHTjqbT4v9URagJVFKF0b7Zf6pHzHdRms01G+EwxdOUBOttURMZui9+po5rT4pw3cM7fyJIk+F90ZtVG3YAVgCikHwkKLpK83DalYI4gKEj2XI9Kq5/LGcHjB6/2z2mldeNopD3x1xG2K7NAayRPHW+iApVgPWyIAANTEWhanRR37gGN4EG9+TZ4rTmAJVxqffU33JTru+vTdiG/6gjt3zAfcxZaOLcaracag13SBoMaXoaWOm7/zMudgbYvt+esQEAfuypAwUJHZJjGj9cyHme7bszf/sctuddHGAWnXoxt9DACIeRuNF7AiuPF9zFooMyTqzx1q4ZSKfqyjRGdE65Ij/TtbZaxjZ3fV14kRa5nbKOKenwNHv9O/3r+ztUfY7FokRhGXk9IlgUahJNdJeLD+Ed5wmg5iMSKH41Zh05sERLu/mDcaO/O8bx/4KQDALD5LhE8cy0dt7mgT4MchW6/KfE4CjVKHxKug9qXNilkY4IV5cV3aN/3WK0oI6WOEPP0n9+oFR4mj+kQQFdVGlPq7bGTN8ee7Y99X7YbHJIbUXT9wv8e8V8bNgp9XrAhhGpgsVbyShKCjRhu91JxoovwCqh2M4DQeEBAQEBAQEBAQEBCwjbBTazg0dvrIpN7RtEG7AQh9ZLEVmO8kWyThMLsciSdf7xy3d/2OM1+xtVFxDmsWm6KJKnEsZKHRzIKXfuf3Lr2htSJx0TzfcFopm1Lhm6iI+Jm/AAAiMtUpdYij2YCrT9vtnRRtLFSCXtqv45GQmY9z7nRNOF5zv0vfS99tVMptUoWpTiJjRzcBAyv8O7zq1zv1mr1PB0t2KBpxeUA0Ahx7YhRY7UyeOP4DUwbWRDtBMTBKPaLh8s7LaOkSLYEHm+MUig10slkw+kUXJr794j+L2p1MVEjCufBo0QqQOd6ux7M0jtpRPLRWSYd9u6c0o6KiMm7C/66/habfhFKf1yqKcCEjbv2m+z35E5KngnKGB5xzN7VbnyfTNQt2uZciUx3e+yPJ5/7n+HIpbQmZGIKcwTsb+qsttYPLTw7w/rvYylBjxPMM2BqNZeQ1G0TfbFo6J0SRvSWQZiMhmU7TAWcB1c/6hwDSgFHsoN5d3f+FFnZmtp421PQtEU2Vh+1/WkwzPaUt9yNrGzTFCTMro6TNHCcmJaczUW7TuNG/3QwAaD9TSWXJfJPSKrbxGGlmH+TOTd8L1mvT0eO1GsPrWCuHKfOTadSU2WOzuEQkYe5STrY2VZZ6dWJ1uA2wM8/7Zr9XyXEqlpBI642bi6D6bedMMQWkeX7G3kkzUyT7OWtFiWRCmRtxzJq5h/J4SebBGPCO+wswMZDZVms3YMgMmGLTkDltFXbj4/7YzyVds4V84MQL/f110eKk1jBuzZBezzRZ38S1hjEzYf3xLJOhP7tyu/V4vpU3ICAgICAgICAgIGAS8azRcBB22sikNzm6UXvG5UDK0TIBskNWNHcLrvylO0cS7qG1EtSNJPtbkq5lwNZqNgi1b70PAFB6//8pm2VfRnbCFft/sJN3lZ2e2YejPCAaCHaOU34gpEEhaXC9Iraj5HhcK4tmwdtysw19vQId6CoLNn/zEgDAlE/fKlJfL8G3g955sTLAElbyS0Fbr/gfMH1lkR3ZxYeD7P+taDHIrpel5ZA6r1fFPp7sVun/em1CdtWDq5zUqR0QiadN+3BURBShHK2ZopjK0/+0CiqW8isykZSbUK8oR3XyayhCbJa9vbpV/+f1AtzneH9gRWJMPg+Ut9po0u4XgF37gASR2+y0W2aPl6QkxRDpXkuX5JvKVBttDO5XHRGJcZH6CPmxlJBVS6kRb3DSRNM1q9GZXfuHsJOnokWl/kcOz8PrVT/0fS6hITDJNGqjDWkgKkhd+z639E6vrR3ZKNqFPLS4P7rAHRx0qvL38WMBERl0z+ZvSj5xCZ8ZKsH0vaRfkm+IDq5Hdcs+HGI7zpoQE215PJmAp2rbQu/TtvpeRHOc9sISlSdJiU3EGlD+BvWyaBE5iGZB6vMJp4U2u73YXWspiiaWECvSj0ilT/XK/kqKpjznWLqtsTPO+w+d44JT7v1DRZqQ6ge6nfE9UQGG/OC4XlUAVOXr4Q5i8Vuiea48yP463PbLm2WdsOcp/Cy9s2E8zgB787+59E6+SNqDSY35hZIKbqz6AgeLpOCxw7DXuWCJ5pVfbPa2RLLN/NCsni/ST1ubP9hvwKTiWbfhAICldzhn3csPcyY4F96x4xms7DIffTnjAoLNKqwVDn49UXJEcmJrogimW6+UstZOyLmqONVPiIkJnWImkDq4ImYnNCEm3qfMrdJsKJo3X6lrAbjyl1IOuzofPErxaJW7jF+7yamPP1pslYmAzHxo0Vpsk00jmaq19arJQjF5pBzPuXyFYuNC3Rh1TOZMQ/IMmQfp9tUQ5X183Pm429icaq367kjlRU1Ouv2lTLjMrANgV91ND/uTRSmPUW0CcBvG9OYrrilmq1R+im0SmTYrNnlTuIWqbfDCCfJOcqD1bD4JFhZiQyq2qbxT29ML8FRfV1G7ZXI2EKojm/iZkDM1AMPxTdQCgskEFEtOMzMYNlFTbFK08dQscYDre4l4Ff4eKwsIAG4jt6W+VmwFavkX5Fd/zcWaeOfvvijtghhrulV8CO5HTcYEQqkDhuoxZT5mjJFNvII4qaqYMKmNBd9jotymf9d915m8nP6aL0s6HNdDkQykN43FdjHRZDbAIWDlPe62PU9KvkizWvHiU0/7ipwjzVonjCDKfDA7EmZ72wBL/+zm2O8c7+r/db93G48ssae2B36/ytXN3iYCfysew9RYxpsQPXar8S8NnjdkLpF4WWo+SEc1r400zjn19BybDw9c7TYci09Z2iDstHruN6kxDhABD40rpQ7giNcm0kgSxSTfrTf++q1jYiLsmzsQUXAaDwgICAgICAgICAgI2DZ4dm0HPUg7QJqNnUHVSmYm40lbTDMJm5cEcITRYhuwzlO0zvNmOcqRL09U8S2CzE1yYMWfnKPjvLe0NEYFpa26kjAId7aS/Cq1vcQHSZqLWR11m8QexVa5znFN2iASD/99OHJunJsG8COf+5g7KLSwChvkjEp85DYWtTVFbjZGpL0FT0e44RHAm75ES17pzpGUsDos5itaMsxUueKEnDDDAoACceOPwFJskNkHZC7jIh+6wWhqTDKpIYnU4FqJKstaqAgg6lsyr6iVYWZ4dT5pP+haeUAkpr5cdt1DjZGKTROpH8e1GEVuLcCwj/2hpdb1pJOjHVyNaIY3afHmNGbuoeIU7x3Jba3MDroc64TMiOKalNXH1EG9KiQH1Pb1COtNW5hYoDoMu+Iud27fs7KX0dfNl140B+fd4rW73O/GcAAvtDQ4VEPF3qCIyFR+V29N0svjtF7qmBB98zt/651qC0WJok35JNri7jmwo95cqN+bnMw7nKmOWXNTL6uIxkbOAbAwUn+eShQ2btTS1kdVbBWlseP78zny7zfLk0Xo+B3UT3hMMDBRasyLCg3kA+ieCyx0fY2obYVqt9iofdI0vjReVgbZfAZLTnNp0bhijItJg3wGgNtKs8Hp+bgzr79x55j333GdJ5mwdfBSikxf17v528zaT2mOdRwLkzyHWPoVtV/qq+V+1K5xdN/F91/n7zdiCkhzefvURo1JpKwMmhDajIfFP/IELTaWdknvpTmvbYp6rx9j1z0sYyblz1oxW6W7qS9VRzhWl77Hpk1aS+2inSYzblZG7lizv4ngeabgCBqOgICAgICAgICAgIDth2elhiONnYE6Lzr85RN+tjFQYEFs0Ynu0EfBtUNrAS9Z1861zagy04F0ku/IT625aaOT3sxvqlkRO2TDzpekiYlYi6Mla+zZkHbyNEbo7UhqUa+6yNv+Ha4MhUafAPqtDuf3U/FSFLvxCZGyUEAtemes7Nq1liKdj9YelijFG590l3oXUAFF6uM1A3bjE+L8R1SEHdOVnX6DgeuEHOR2P/EYn2XlOMpRz/27umaJhJ7KU2xnqTDXb6lDoolTO6XsVYaBIS/l85JSM2WB1AlJqQAg9t+WtAMkSZ5IIKdpzpETpgAUyFlaSYcBmN6FDcHWTFR05dHQwSPTNtf1ihzrb0S0lFqCzD4R3veHNC0Dq4BHb3LXcmg4qG7Ou2UV27OT1JeQiDSeNV2v2Yg3PQ0AiHoXNGhTm9tVjw1OI88z6x5yz8w5SBynyZG7x9dxqQ2GfIZIK2MKQrpA0t26ilbO/hJeYppwrlcaNr7Pa//q1UZtI2kdqsPSl0gzOA4WfMYFjYs3PY3IjwusASXfh66+xqCA1sJ4CnZuRzdeBXPkW90xOdR7m/9401MsMTd9S9xvoSQEJZTv1h5Enr6UpMpUz/EvLoI55cJM5ZpM7PB5/zHnoI9D3iDty2vc6FtrYhOijrdDq2GonZAWvlCSNJgUxWtLRjejeN4v3DntK0lzPs1DtbL4vBEBAVHPG6P8jLK1UX4OcO2OLDDIl5JJSxp7tpm1v4wZiXHVt2eaN9q8Rm5kBXDH99y5k/7JnVt5N8ys/d056hv1sqKKV8FVAfc9zDg+HgG5cfnll+Oee+6BMQYXXnghDjig0aLiqquuwt13341vf/vbY6b1nNhwBAQEBAQEBAQEBDxbsLM7jd9222146qmn8IMf/ACPPfYYLrzwQvzgBz9I3PPoo4/i9ttvR6k0vhD7ObXh2JHUeeYFb9t2abVPgzn+wwDEdvfSQ52kvWm5tsBctSVJpDEGaO3Kna/Fpx7Fxyx5Iy0K00cqNgwvebCFktxH0muVbiMjTBPWCxMpajz/jpENsBufcJc97S6KvtFHRcBU8hVwl4NdWu3TRGqS8kNAsVVsbCkfxTYlhfXnqsMAsel4+3KWvLZPk/vou/TuCpbYEMXl+kcc9akGfQNmh8qJV33G/apAZkw3usYFKjRTFji7XCAp9eWgVhRIrEPyM+zt6r32zbROwcCnTgYAdH/2Pp/+X2HmHpYoB0rtQJ3YkVQwNmBiwcaGNsoxa6Vq8i7A1Q2zbymtA0mMNd1jlJSuJ+hnSftB2hpbA/s8MGVpDGnDioUNzn7ftpNvQXZorQNpNta820m9+776OF/Lnl5SE0oS9/imL3CgQMJEaLnHZZZpAjNjb/dbbBPNJpWbxpryoPg9rH9YniMfK7qvpRsNLFYkHY4KMiZR/W9eDjN110TeN334MPR+9m5+BgBQ9JJbG+dmqSIJuH3014hJc0jaVM9IZwfXJHxsJP/J72lO/Cd+f/Vfz3BJnOX80cys/RuCh9rNK2RsUmOvTfUX0tKZs64UG/qdEDtq3jdLvFayXhW65aedn6PZ3dMSq4C1HBy3e660P6obHQA17SPx8G9gd3fBL6lfoNiq/BX9+FYZEA1bmqYcFognwFTFPkwjsD5/zLDJfkVF0UCw344O5KeY3pi9z/dDCg7YMxd2xPtjkV9d32Ll/+H7b7G9Cbui8g99lrFU7ey45ZZbcOKJJwIA9thjD/T392NwcBBdXbJ+vOKKK3D++efjy1/+8rjpPedqR6tZ9f/bHRSvISpwJ92aiMC0CBi66FAAwMVfv8Kdj+tNo49PBgYfdY6cPVALmjQ1q+7wWpXMDl6KUpNV92Su4ZNUaVhSCSsHRKP58qerAdiddT+10dzRfzGooptzDAeKCO7rsjzAC012+F3wInYyj7xZlI2KEhWWIiWTc7amqhxc7d/TrhzPieZQqY/JaZMW4TMXw1aI5zwHbvqa+z3jcjlH+SQnP2vFWU9PGjQR0IZo80peHJFDJ6M6hK7Tz02cMt3zZKJh5+tqQz0RsYItD6qNZ0a00UBo5VmaIH1Ee9PSpUwFfBtt5lBpVRpMe1nk/NuBVT6/alNKfVMvQOmZdOT1qAj0LcpXPjTfTNBGY/U7dgMAzLr6iRzpEVlDauNxzAcQ/+Eqd3zcBbnzmU4/H/zif7TfRQoHYKY4+nBL7WNkPW/ITavfINcr6juLYILPcYwbMilqa3CGNa3dEkPA9/XeK+9NmjpCj1v13Caq9atf7XL4rh+r/JKpFm1WVcwBPw7a4XXijE3kA5ueZEKC0vu96U2kBBqpKNVo6eDFr5CRRDLusJmPf/fgKlhyVJ+yS65yThZ2yLzv68GuewgRmVAtfoU799D/AQDM3mfIPOcpuJvO4YUSAPf94zX3AwAiitVxxNtw1RFuM/qh29yYY20MU/TjWuwJCHrmiTCimeh8Ak7j3HbKA9Le0vTKQ2sbSVCqI7BEL04bqEILxAzKj4+KzMHQnESmWz86D+bsq/x1Mi0syXFaiBAV5FrANsG6deuwZMkS/n/atGlYu3YtbziuvfZavOAFL8C8efO2lEQCwWk8ICAgICAgICAgYBJhdvBfXog1CrBp0yZce+21OPfcc8d4IonnnIaDMNkOZSvf7aQRc//9qdzPNjpmWpYAdH76Hn8TOSUbcUJmSdfUhFaA0kikp9IHlGNgjnwOb3ZSrh4oKUfaqbe8GZZMTUiiooU5VI72XpHaMT2rM02xpqBMbryUpDIE20ImDIryMa1C1arXlCnBeFj+BWeGMP9rp4sklB2zleSETA0oknh5EMY76THNamefOA6TyQ2ZXVUG5duQs+voJskvBRScsXdDcCdWY7d0Cp1nDgze7GgVu0//JLhiSALFNKIqWjN96/IAS6O4vrr6AK81QHt7Ip8Y3QRz7PnufoqS2zFNJP+6Dsm5O0qZVJU6YAd81O+pu2UrYJ+KOM1B4bwpAJmnDa0TUwNy5O5/hJ1xE1oPoj4mkwHWpBmYrr7kNRtLXWtTAEqPaFdZe9YuTpE5QPS16JjRoO3o+8qD7p56tTmRBGnZNI2xpwPm+lXSenPM+wEA9a+eCQCI3nktxETMa3L1uJKmjI1r4rQtI7wFAAAgAElEQVScJ9L49Z92z7z8s2JGkgqqaHrmywNUF1FJtBeEdtXu2NFftf0RKr+vn1KH1DOVZ3gdS3jh26Tt8mZJMLm1qdF+x7mD0U0AfRdNBw0ALd1iFuZHatO7QOibSUreMw920I8Pz9zqrs0/wue/ytoP67U2ZupuShMnFNesCSR6ZNYqG+CZ290hpbuTgub6Sw+ZjYvv3L6BAe13nRm1ee01ThsL8PzGARjLm0GzrCVtdluvmAWzVkk0vWamow7nNI3BBbcs88lX+H6b0ljY4Y1ShzpQJ5DQ+Ofph2TyBG7raCTL6JyJBtPqkpp7S8r0kOZ/GldpPhvZqMwc3ThkTv6YvIPMmIstsOsfcdeJ2IX6ankIHHg2YJugr68P69YJLfmaNWswc6Zbs9x6663YsGEDXve616FSqeDpp5/G5Zdfjgsv3DLBRNBwBAQEBAQEBAQEBEwiIrNj/8bDUUcdheuvvx4AcN9996Gvr4/NqU499VRcd911+OEPf4gvf/nLWLJkyZibDeA5rOEgTJZD2ehQTgdlhaYSGLLZ5+Byqqq8NI9s0iOSkm8hzabpx/mD5HT0iBOvSVNJEkgKCCWhaekUqT3ZvQ5vaLRt9RJ+Y4yi/JUgeKyxAfmGjIj0hKgkvZTSVgaUQ2Q2zHu5C9CX0IyQBIZ0Qe1TYfufcWdIgt45AxhO1lc8uErR96a+lTGNUti2qSLZpHIOrHAO3ICk730lbGWwUbuUAd+/2Tl3vz0qStm8hEgc/1slbfptmyJSLMpftSzBGzUdMQC09bJ0jSkgo2IyEBWQDO7nv6fUsxX/iKxY55yHMW0PaeOUJ3Y27ofp8Q661mkkzNTdxJeF2qimHqZ8UJ3XK1JfpH0aWiuaGGqX1WE59lSQps3bQtcrsJt9W8qh6TBN+jtf83UZ//IimJd+ssn1tvQJkbA3TxAAUHi3o3Fd887d0fe1x7d8e9pOvFAS7UMe7OEl6cPrJH/kmDvsAoRFvQtg2WFf2XPrAKKA027ws95JtXdXd602CkvBPTtn+CSK4sNEY2f7NJHAEuWotiXPqeEwR3ga2199CubMz7qTfgyzjzu6VbPvWY3joDFCpU003v3LYLz2zK5wxA/Y5UX+OeUrR/4XUVHmBE84Yo0RjRVpAvmeEsyik3OVb0dj6V2rtv+837er+zVGaTG9VoiD97WK5ozGlfUqKB6TS9TAc0ycCijaNgUY9LTX1BdGNgp9Lr271Cnv7fLv1PP0BHw47B3fAgBEJ3xU+W4kx3Cj2g7BRAXlE+bH81qZNco851MfaumUNk59unu2nCMSk1pZKIfpexHpw8YnG30JA7YKhxxyCJYsWYLXvOY1MMZg6dKluPbaa9Hd3Y2TTjopd3rG2pyrsolgeN3490wCcg9AeUwAfFRp6khbvC9jlPAxY2iMgS+90Kk+3/vnlWO+g9PvzL4YiP/rHPfMq7/SqELVrFXp9+qYC16tb8v9MJ6DG520oPALvmKbTPgqvgXzgKtIvLJwN4k03ELWmyGko1tvARvf7vLT+7kHZaCmfJAqOCpKechMo9ShFqZkFjUE6+MZkIqcF61xDRim+Cp+g1avwnjmMDJpsCMb2UGb1cb0f73MTsvRrsdmKh8ADF3gNjAdl93ZcI0nBh1jgkAmOO4OlaA370kvwkwkPP7kXD9zH/k+VG86inTaRKmlkyP2Zi3j/x3l2uPpv17VYFIo+TaJuBIMYpGheqiNqgl8Fj8rIE55MsmZIu2k2X3DfmHrGYLs0FqONxHliMNhafGhvjFf0+UhExo/Wdt6RTk3y7eg4yxjjbUWPz/Z5f/lv1nd9LpLS397fy7HWLPmjW4cnfnVR5qMMTQmFGSxRe01KsoxbRCsVXE4aHGmhArU3tgEtCQbYt/X7cbHYaZ7B3/aqGsBAjlVk/nXOIiv+6i7/7gLwG3FR1rmvjSyHohdfi1vVnt5HBFmuBnyTVqS7IN2w6OSJ2qnxXYZo2nTXCtL+WmcogVvSRF25HEaf47P+5XLDgQAlC64oclY46GjX5P5aalDsY8ZuY9MLZUjtb8oaahxmdnViA3yhn8GDn+Du9a70N9E9Vzm+sw6HwJA/FsntIhe+C4pUrqMOiaIjrOkTamARNwsmcvVhjotxNz8DODjycDHl4EpMFkKl5FZBVtkc92TzYl5R+OOE3cMARDhsBsmNzr7c17DERAQEBAQEBAQELAzIXqeBSp8Xm04tqcjuV3zdwCAWXDUODeSdE5RvzbTZqQp33QSxBf/P+9xt7z6K7D3XwsAeN8tqxrvJ0dmopO0Nj9lLICRvzuHxM5zRIWadhI1UUHMHDgCs5hx2A2PuUsz9hLpMTlEkuROO/ySoaGJhJqR7quNilSOnZs9l3drd27u+CnHU7R4kcqCqGe9qtaYCPETN7rjPU6Ucsbi1A/AqY5JihornnA4CbVh0zOJacHSaTKjW/cQMPcQd0waEW0moqiCs6I+opzgddRaAJbrySSdirmMvs2QVLTYphyzk2mhXoH1kjqWDGtJM6VfVCYH1K7J7KXYlruMd6x3be4MbZany+DLLkI6JUUspIbDqMTSRqaOpXq2sZgWkNYgQY+quOeV1ieZfrFRG5gFLM0viYCUJHuaRIHMEPT7GGS+UeN+QnSzTZ3NVfqk2bj3LCeJ3+8na5JaR6Q0LROI4TDjgq/65KJGsyJdDtIykobRWhlvtNSV6o36pNbIMlmCMp+0KS3J5mdgyxRF2tOVMhtGlLuMy//riwCA+cddoLSnZKpCxBrTRZNMZnTKfIvM91wsIont4sri+1llMElqALjvQ9+LiAxapygNp/8OFAV65V0srd5ZaXHHwvaa94tv9m20MqiiafsxjOa5nvny3YnkoDYq7YXol9ungl1qSRNF2ioTyVxJ2g8TiWbj9m+4Uyd9QjTHpC1hKuSu3NTNAIDljqJXazUayGiKrTIuMu250iCmaZ+hxget/aB2R32ue46cI6f14XVA/7LkOXa8LzddLwXsPHhebTgCAgICAgICAgICdjS2D3/azovn5YZjuziS3/cr97vgqLH9L5qdS2kz7Gi/SPO8tJu0BkZJDszLJXib2fNknxTZ5FmxjydHauu1AZVBYAKuO+3v+w6/gykZUxGb44EViMi2kotXE78HloCrSM1eomdJYqhts+n+0X6WaIhkpY0dPtmu2dvEmzkH5y6fOfUil8b9P4FZ4gI4kWbDrvbOmD1zOYos1YUdXgf71J/c9dkHuN+oxI6cKPuAYT6SsJmyi1D7dXtb03pVnI8piODCo5PRXAGRxsb1CQ1WXae8XhV4C9IgHZSKNHHD65K0oYCTKNG9TbRZDZL9UjsSNssA0D4Va97lNCB9X/V1Rw69cVVpgrLh4s9/yCev/BVIwkvvbumUkZ7b12Zph2RDXxtliaGlaOiUZnVEypeISq+c4el+T4dpy8nAcbAxB7PLg2aO9GlabI34t//ssvKSjzdeLJREQjjWO5ukv99PnM/BircuxNz/SNOBq9ZZykdPDQCGfHZqyv8rStmJ0zcGuJ3YilBFc2DOvsUi+S2kNHJa6ks+C1GRNbik6Yj2Og3xnd905zwdNo0Jpm8xB8DMim/f6/r1x2EkcjP5tDEldT3hw8bl8u2JfIFMz1wZ61OBLM3sg+RZrfEhnzCShBdbYSl4rU/LTJnvy7efBD18FmNbz/tEo601gjQ3RcqXh+Yo+6uL3f2nXNJ8vPS079wetJaQ/Uq075U/OuLtAIDRy49G6xw3nphzv+t+tTZiIoGIFzg/FZQ3C212qk3ahN+m8qUk6wPqw8W2Bt8hDiJrY2mzzfzraP3QPk00tzRfEolL78IJWW4ETB6elxsOYNtHJjXHny/HY6j1ml6LlUMynFrcplikTLNJm5iZ2npdZGsgEcGUuLD7P+oW372ff9if70hGQs4IUevXYcs0OblFBzERmZ75TZ102SGcFuQmkoUglXXUD8zVYXH6IpVz1yxOL2HeQhFzyRmNTJCiYnKxkgF25d0ujcVny8l0FG4TSRwEGhg3PgGz6zHuePkd7tKcQ2Af/5073v8c/wJZKBlSB7MTa1vDIsiObga8AzMPsmRa1dYrDEfTds9cRnPEW+S9aVYQeu/D18Hs8/JE+REVZJJk9ppWtXElEzTPtDO8TswvaGK47iKY0zxzktoM933FRbBPxxxBodWx7OTBMz4to6LOUt/h2BBVwNAGQiKHM7SZE5eZ+pVPU0cVp81IdUg20co5ns0OaQPqv7sdXAVsdBHBzZyDMheR0mtghNL3qDg85sXOQdlWhpIR3gH3TVIEEGO+W6VLmPP1x/HLk51T/anXk0lno0lZrg2yjhdC7aKuSBoA104olo9mIqM4EuToX1cOqTa1YKtXAEOEDH4zb2Pl1Frn9M3BbrNu13smNKrrWjm3AOdjP/25OxhaC0ukGVxmVa825dRZr8qG06jNPgk/iLyEhD71iixqtUMvCS4Uu5wlx9w/OxMdnE4sZ1bMyXKVcufD0jtdDJVLD3Hj79K7Gk2Qs8ObTyXqPtm+bFwX89G13hSoOiLMgNTO4pqY3Jp0P1QkFzTflTrkXUR2ERVgXvkZ/w4ymetM/p8T5pA3+GLV5X167CCQGRixRJmCbJy0qaJJmqGKuaRKn9p/XFPvcH3Dbn4GxgvuOK17HYMeXvTuxrhcATsVQu0EBAQEBAQEBAQETCKyxMJ4LuF5RYs7FppqOnLQ4sZfORUAYF7/9ebaiC3AxrWm5gosFfXQ0kyWGLApR6tIE0gaOLga6CBzlCb0el5FLpzW4+PLB7l3vPfPTRzTlRMY56UqqtQGh8R6hSXJolZV+Us7zWv1Kp2rjYhWiCgCSWpfGWTJUlZHx/j3zkTN7P9KJfHWkloAIxthh70ElZwATSTOfPT+elnRVnoTGpLWGCNSWypXebNQ5JK0CEak8w3pV2D7He1utOtxmcoHAHbdgz4vRUW/6EHSs3pZaBtZ+7RJJP8kxbJxI0Wols6y86OXsg2tY4kxm0qZCPb+n7jjeU47xU7mtbJomOjcOIgfdNIuM/ewRpU+/V8dFZMSjmDbKhKzgtJ+KLMdd5+XFmvCAk25mnairw6LUzY5+5JpWlSEXeHoiaP9zslUPgASCXoCDufxUze59y08JvezWXDz6c4c7ahfrOFzbAI5RvyQNOKnb3bP9C5EmuOfv3epXb4z/VaGpM+QZLk6LNooMg1ih1zD9NIcm8VE3N9Jm2GmLBTTSGrPpLEa2cDaETN112zl+5s3ednjJNHYNovSTv2KzPIGVkjMFmWCwmMttUkdFycVqyhh/kXvfuJGjHzLmdyR6SzHlAFYs5onXsxzfd6Pv38uAMCcdpl8dzJRU+2LiUxqNKdbGeuZHn2qjFOkaaP+XRkWc9aq0nBoUyXAte10PBDSxrdNZfNAjtCdAWy+N472s4EOW5ld05okYX2RWna62FtJU3S9NhItiYV95i/uvtlOK7zsbfsCAHa55n4xS9OR0Xdi3HPyjnVyP/DX+WN5bQ2ChiMgICAgICAgICBgEvE8U3CEDQdha6nzzBs9NV1OxywTFZsGAxzLPpvf0eRdLEUgaR2ATx3qdvufuNNJ8qyNxXE1B979f07qaIfWJiKKA+BIqKZnXjIIILz0wlPZkbbBAmwLyj4RlKe4BrvxSXe/l4qa9qmSrpe6oNTB0p3Y2y5H3p/BljoUzWVG9KjIrV56ZCl4GjkSd86Ub0y0hCrCNAdZqw4DN37J3XfWlb5cXltQGxEJGGksWuezvbYl7VP37IbosExtTPnMCfvQL106B7xGNBAkeSOtRiKqu89zxwz+JgkNlooED0DRk9aTUj4A6J4NQ1JW7Rsyx/v1kCSQJNrlgUYb9vHwZxcZF688GKydIokxpd/SLZqIdq+tiYriL1TwZSn3N34bKkuxTWyZSdLY1tvc14MkfFqaCcA++uuGQG1ZMBHNBoE0G5UrXwIAaPnQbyecVjOQZqP+by6QYeFdP5lQftkJfOMTMFNcsErWNszcl+4SelH2hVK27fQ7skEk/tQWqF/pqPMUabtQEmkr+ZKZSALypYkSCiUVgDQjHnMU49j9RFhqRzyueQ1NsV1FN/fjYMe0Bl8PWxlKaLtduXyA1YHlMK3e/8s7maM6ImUgrVvffmh/r+87bO/v6210U4oM4rmDrZn3zWmX+QMj0nzSGitpvSXK+C5PQDKwkrXTNCaZ9l5pw6TVpcjkrV1AiozCtHTCkp/YgPNLQfdcrjt7hyc42MVrjVu7c/sZAQB8wEmrxzb2l1LLR6/ZobKiXoYlv8O0v4b7z/+qKOTpfmWtspzwc+PyO1yAVYC/xS7//pC/Z3Kl9QH5EUiLAwICAgICAgICAgK2G4IPRxOwXeeD2SXk8c8vAACYEz7CO/Sskr1mNLrNbBzzwNqYgw8xEw7Z+o9sZGmCSAvHx4a3Ocns1M/dr6hTUzSnGiR9LLahIfhaVShHxZeAJNJF0QYwtWpBpMvM9lRWgf9UACHA2bOSj8j0vTKVb+B9TgLY9cm/qGBCKSl2oVUk25rBiOkoyf663sjcoSmEyZac7Np1sEPSGoz2N/pSKMpM6/0PonmHZSofANSuOMJl830/F4kXvU8zqDVj+6hTAElPQTzaL/VKUidKy9Ylffom2neC31NQmiJPx+mlyvaRXwL+ONrjpEzli3/3aZfGYW9KBuRzZ/3/ShLG2qpqkh0FcN+AJMfMJKc0GE39i1JSwEIL7EbPHEQUuF6qaesVYNBJVaN9zsxUPgCwq+5xB+1TlQYsNT7YOhr6ZlxvtPEv9yt2MF83iWCLKR+m2qj4BOj+SH1da4AAN9aQVHb2gZnLOHi+84/qXHqzvE+XA/BlSbFOKYpO8Y2oSp8d9jTaXV5bUa+qvjhDztE7md64qPp9Sos1spHbuJmxT6byxT/7oLv/Re9gHy8e86ntaBpRHaixKuyErnwVYesiLRCxAQKwax9I5q06JH2D+mOt7FjTAOBvLoisOdbl0fYvY427TndcPMfn/d8f59rB8T9/WsZLqjueG6LEmA3A3+vrkzVNdRUUNWW5UB2VNEj7pWlnKV2t0efAuTSnVPl6wjdnHNC6RlNq25QmwajgnHqd0uCTYa1aB6T6kI2V36b4erDmiMfhGvdJO+LmV56PKoOcXjReAOadBPeesmNl/vtd/xz04Yi/81YAQPT6/2g0t8lAxbhd8rTmPpenJk7TF1/1gdzpjd70YwBA+4s/xuead8LGczyZ05zZpPM1fW4MGBNx3Ac2VaLBxximkcyzjel9naPXdAuy1ESvaXYpsjbn1aoFM20qIpC1jixSSX2qBibetLQqh1y1CKJ30Dup/xRagTjpeD8ervqNM3lZemlNFkzG54PU13FVLda8CU7rFMgEQgN83Gi+wZuRgix46LlCKbWQQpJ6lUALhOpoLnICwqaHndP4dE05yN+1SVTxtBM01MKo2MaLFHL8FxpDFTmWvlNLl6pPtVGkDbp2RgdgFhwFu+befAWcsZuUgRfEFK2eO5jkidT+1aFGJ/BCqXETaFXbpu/AZgQV6QfaeZRMuXwMCOLlN1N3F2fQPOAFuFWbxoqU2x1IeXnBU1NOp8qx2qSiajcVHqhvQt/HqDZP7ZTvow14a4NZYBZc+UtnXrT0U+3KvI+c+tWCRC9mdJ4S59SkTht8vSBjkgBquwYNZll6fKN31P13a5si1NIZcf1n/xUAcOovzlfUoGlBhkHDCG0MKDYOP1evwvrFvSwmyeF4SFFw+3pt6ZLr1K4LLWI+dtibkq8stkrk7Bwbjt+/1JmonvDL1dtk3s87BzZNg/peyiTYWoulN9+XK60/rHb1dHykhGGaXtul3Gjaq0gJZFy1yesaLR1qjaD6XFqIF1fVvNVEuDKBKNyXffiLAIBL7pINRzNq/+Z1suV6MumNdJNNS2Jjw4QrEZeR41yR8LRekT4ZsFMi+HAEBAQEBAQEBAQETCKebz4Nk2NSFRAQEBAQEBAQEBAAALh/B5tULZ5kk6rn2wYrICAgICAgICAgIGASEUyqAgICAgICAgICAiYRW+GO9KxE0HAEBAQEBAQEBAQEBGw3hA1HQEBAQEBAQEBAQMB2QzCpCggICAgICAgICJhERMGkKiAgICAgICAgICAgYNsgaDgCAgICAgICAgICJhHPMwVH0HAEBAQEBAQEBAQEBGw/TIqGI37qJgCAmbE3ZE9H8QZljych7RtjEfK1ehX2t//szp34T3TRJ2mb8owZk9xXjZl+rQz7wE8BANER7x67YDrNtQ+4g7YpQFTyJ+vpt8g5U6An5ZiuxXXYjY+723oXunOFVvdbrwCRrzYqV1zjZ+3aB92lqbsCxXZ3PfLpD63zr6kAm59xl/Y9K3sZNzzmD2KgOuKOWzqlHADsE39A5cdXukun++/XPQtm9gHuer97r5m+CKgOu+vtU30aqu7imi93ScpN36lecb/VYaClK3kfwwCVAXc0bc9M5Ysf+ZW7f+4h3GZsveqTo7wZLmvi+1Pe6VvXq7C3Xu1uO+q9PgmVJqWXaJtGpeevxf79Ubp8FvYPV7lLp/9LpvIBgB1c7ZMuNF6jfhHXXB1rGANTaJH8AzCFEmxt1OfPt0l6zlr5FlTG0U1Arewuf899E7zkHUBrt7tt9kHu2oM/d/8veQVQ9nU4fVGm8sVfOsnd/5bvqYJRnuqS17iePFevSX59OVEZkrz7dm7U+GJ1Wd1FPqbva20s1wmURq3MeeN+ngHcD+OaG28orwojnzkZhQ6X54GnngIATPvY92DmHOzS2OTOmam78TfmevB16fLuy0h9rzIElPy4Uvd1ueoemLmHunM89sX8HSzV4dTdMpcxvsm1bcxeAjPd91/j2xjVU1RI9hXKH40PHdPd79A62M3LEvkznX3u/5ZOgNow9b+WTjmmvm5joNjmDjc96e6esou7NryBz2UdT+OffdClccKH5Nv6vsljqikABV9mapP1inxjLruB/ePn3aEfa6g/2uW3w0zfy52jtjLaD3RMc8dVX/ZSGzCwyh13+W/DY18Ee/1lLtlzvp6pfAAQ/+LD7vHjP9JkblfflZAYT/kk38ff3Zen2XpBzsVoqMN6Vd5Bv/T+uAa77iFXxj1OzFa+Z/7iktLt2qbKFdekv1MdxjU1/6tHf/Q+l96rvpi6YuSb0DhbL6t1Qyy/dOzbqh7zqp8/FQDQctHdmcoHAPGf/9Xl4IB/QMP8lJgHU/NfXJe8ULnLm2Gf/KN7ZO/TpWyUFrU3SiOxnlNrAz/ucPlp7q+OwD78S5fEC96ZuYwBk4dgUhUQEBAQEBAQEBAwiTDPs0AcxjYT929rDHvJurXb9ANfdshsAMDFd63aZmkm0DEj86129b3uoKtPpP8kRdEaBpIeaekESbFph18osRTTaYXAkmEY06hBqQyKpJ8lDUWRRNRcfuzoZndH+zSW+EULj8lcxhuOdXk+8Verpdws0an7Isse1pKkUUnnqf7j5Xeg+p0PAQBaPvxbKRuVgZ8lKXmhUcpTGVQaFs6Q+3ngJ8Bfr3O3v/2nmcpnB1dxGeKH3bMDV58PAOj+0A/ctam7AqWOZLlszHXdTAKe1rC5a7bh/rzgNDpnZn4mfuAnAIBo4dENdcfttToiUmxqy1HBaQEAoNhKOYB94vfu0T1O9qe8VKu8WdokpVsvi+St4rVbcQV2eL07Ht3kbmdtRgT79x+5o+M+kql8V+3v2sYFf9k2Y4KNqV03SiS3VdoAYLpmZX4u/sH/cwcveB3MzMU+gZR02ERN8xyv/Ku7TNqk1X8H7v5fd/Hp+9y1t3zX/V8ouXoEgJpoDcbSjtn/Pc+l8QoncbfrHgKeuBkAEJ1wYfYy3u/yZKbuIZoKPz7a9U4SbaYvAkpe8+SlnLZelfGBJLDD62H/7LWNL061o1oZdtmf3bXdnWTbLr+dNTZ25V3u2tzDGrXT/tvYobXAnd93ZTz7S5nKZzc+If/48cTe9AWX/AvOdedbpzSMK7YyJOOP0g7Yx37j7vMaLNIa33rmfBz5335uonGiXklqTACYYivPDzym8nhbgH38d+7wgNdmKh8AnvcB4JJD5gAAlt65IlGePBhrPN0W4PRJ+zXe/RufdAeldpmvKoPu2qM3uLS0ZoCe+/lHYV52hfuHnjMFfhZtvckXVQZdG4PShCptKmlmzMx9pb/y+O3H2WI7a/LMtD0ylQ8A1ryxBwDQ97XHMz+zJdi4DvutNwMAojd/e6vTa5r+M7e69Pc5c5unvz3wyGnbfl7Jg0XXpa1wti8mVcOxrXdztNG41G88lm6vjUcWkLq6vFk2DrTg8gO3rQzAlP1A4M0XUCvLQptUhaWOxkGBBhBATTR+4C11yiRLavHuObAr7gAgiwv+/i0dMFN3z13E6a1+gNNqcDJHKLY3PsD3WfkGlM81D6DUN99fpkavTMuMSaRhoiI/y5MvTZqAmMjQYqh9KtDSijzggbtvCcyilwIAei473F270S0GcNx5YobgF4l25V95kWxbeyRB2iSWGr/NRPsCL1JNpExBsoPala2VwQsyMqmhNkztEOB32OF1vCi2/WtcWlPmw8w5xD9DE6evm/5lsllmM4aCUv/X5Bwtqsj0g9Jqnwqz7xm5ykcbjU8eOhsX3bn148HWbjTGErKYqNDUvHNcrPLmlj3zYJff7s5R3yDzI9UOZWNjxQyIMLQG6HYLUfMaZ8ph+592/7dOUSY9ZMbZWBZbr/JmESec50/6cm16CpiZzaQxgV5npvLUe47Awq//PVmmjpk+n8tgetwYYnmRZmWRxZuwOnDg2f4Zb9I5xY89pQ6YXY5KvJrSBCALc1gl6PDmazQ2tU+FPfwNuYpnb3ALTnPqJfLew97oDnijbqSP0xhCG3ZdvtoI7v+we3bJj7zpmN8UHfn9e2Su4TKVpE0W1Rjp2zqbA6pNbP+X376AFAoAACAASURBVAMAmHpNjg2HwtI/u7H16qPdxuMdv/cCtbTAaAxsr43GhNOPlFkWtRP/++hl7wcALPqv4wHawNA499JLQJsQo+rT6rqFmudK7cBd/+1OvuTj7re8WUwgaZyNq5IPEoa1TuG0bIOJd4YiFn0e4lpCmKhh43qmcdJEBdj9XpI7D0ByzbHFejIGeNKZueFZsuF4nik4gtN4QEBAQEBAQEBAQMD2w6RoOJqZj1ivbjUZzZaSzmYmkR5pNr57/Gy87sYdpOWgHX61BuslbIZV38pBniRV2uGJJJAtXhJl62A1rDaloufSjlT1qjhOscNhJOYw3qSKTRNqZclvDhz41je7g/IgS2vs3c451xzmrlmAnYtJ02OKbWiQ487YE9ir3z0zsNLd59uC3bxcJLFeAmdrSupOUjljRDqf/tadM4FFL8xVPjNzX5+EBUv/KV2SNt3wWeDot7tT3kTBzDlYzOJYumrEyTTV/q21fL/RWposeWRnZJuUdmYFm4MVRBNHTvtsgldhUxXSepi4LuYrPURGUBIpK30vMtvrmq1MBH0ZRzbyu8ikxMzYG3jKmbRg0Sk+KXJUjpLalhy46M5V+OShs/lYY1ubdo6F8d4zkXyYcz4HALD/fR5w9uXuHLWLH3/E3/NlJaH248vwOhkfvJTe9O0H9C1xz978NXfu2PMoc0IyQON0XAcKvu9r51GvfWBJJ0lT+5bIOJinjF6auuD4o8HO4t6shPta/zKRHlP7q1VkjGUNoGWtgema7V9A4/WIvLTsx6N6maWopnuuXKexhbUCvj8/8itg/uH5CnjUO9xvVIBd/6g7fsCZcZqj3y/5NiJhBuC0f16bzOOJKWDxBd5cjZzFvZkYHvwdzMkX++RIwm0gvthEVFIT4hMiACDYGFPO+UC+8gGJedC0ufbxjj85k6qbTnf1cMwvVjf0gS31TzYJbutpvLYtTFTJbCmrKbWe5/i9Lh8rfLNa1DFd2ivdP9oPdHoCDi3zZadybzrI5AB1oOTbHI2NbEIN0ZxEBViTcuiGmnMmMF9MO/40d1AdgSWTr9bU968Mwfq1xViaDmtjoL17y9cVGUkaTLhi7Zbr2sbA3CVbTD9gxyM4jQcEBAQEBAQEBARMJp5nNlWTs+FgZzt5XTMpwlg7XEa9ItL1lPP0a3+/Anef6SReB/3U2Zlrx8xk5Sa1JJwHazm9XE2BJPCtPTD0JNuu+3sSPgdeElEeENt5SqNQEukcSTK0w6KmAfbvsWvud4fdc3xaIzDzvNSNJHLknNvWC9Q09WA2lP/2J/f4y0RKYQ5/C+fBH6gyekmNjUWiSN+9dzc8+dFXAgB2/bZI+wBvh84UrYrKNN05R/vFsT/l+Gzap2Ht1UsBAH1HnZ+xgE6CY9p6YGMnSbKr/+bOneKlhNVh2F+4dHGWo/+16x6EmeKd+Vq8NLc6wvVp0j4ccTWzZmNMR0nydcgBu/4Rl97cQ6RdMH2h/7+1G4Y0S3StvVekwa1d/Jy90zn/MR0n2dJXh+RbUN20dnMfMH37wRcMZvcT3G1rnK2+2eVFvnxV2Os9Bfabf5itfJuXu4PuufjEHU6a+sRrnZR61295x8dCC7L2bqL9NRPRJmVJnyhj8xBU3OBokM0b/h32KdcnMe8wd+7ln/Y3CS2xKXqpd/dc0Qj6erWblwF/dFSn5uzPunNPeyfqhcfAkNZVawsILFktigSe6XO9NDSuw/7eaWRMDkpV8iPBWZ90lK0AO8Qa0pK1dLLUlYkTIqUV05qOx5zTM/Z/dfJFmnCDSBFW3wvQOErQ8wi1Z58vs8sRsE84us8G7cAWwP25PMBaFEvzHjn+6v6SpunWKJRgDnmdOyYpvddafek1r8Z793F282aB91Wpaw23lxYXW4EZ+zTPbKEF5sBzMpVLo7lmy/W7Y69z8/O3jpuNN9zoNdxMFz0imliNMfw9smo2qJ029UdIS+7Hg/dbsqP9MPSs1z4d9xOnwbVP/hFm12P9y70P5NBaGLI2IN8n/e40tW5UhDnW0SiztqQyJP1gZKP7LXXAbnDaMvbV2+T9sXoXqjk6Bw51czSqw7CeSp+0ikx0UShmTNuMWYfYgo9IIgVjxvZ727g8Qz4CdhSChiMgICAgICAgICBgEvE8U3BMzoZjS+wGDfeNodkQVo0mkkalOSHNBuGyw+blYq8yxiTSywxtu0jSihSNq5kyH3bFne4cSS+658GQhI19E6JGZiOtzWHNhn9nSyfT5dmnHQWlmbV/MoAUALD9cjQh6XjrHk4q7aQMPv+klVF2zSyBoO9Yrzb6jGiJSNrW3MbKJ8b7RLR0sd05s3m0dIp/Spolq30aKuV8Why71tOC7vJCsUUlLZH/hqatF7VljnGlSN+wvJml+Rw0rdQhGqVUXeb122gGYwxsbWT8G9OItYQ05QtEkttim2gGSatRHVK+HkraO2uvVMZ8XRZaVeAwxYjDefbnRjaKRH3RqT59FfRxjyPylI79oNx44d6x2/ecpuNrR7n2/86bc4wHE9RsZLYp17bYGfGVKx1F83vP/hzM7i92JzkQp9dIVIdZOm6Z2rgoWg8/NtioiJW/cQEv5/7DV921PU+Rl3EQRNFYEJkcS90rQ9L/vWSax3JrRTOaR0juNVVmygLRaN/n6a13d1TeZtoiGWu1308zCejAWs6Pu8//u3kZTO+u7h8K7rjgRTLeMvtapMasFGNXWy8wuD5H4QDb7wMv7iJ+ZuZIp+l95NVO47Ho+0Kda+m7ltq5b/I4Wx1x9MYAzK7HJfL2vpuXw/7QB2D1Wg9ERVjSZpLU3cYJuusEqiMiOfd+bpnKSH5CSqqd7g9v/MMqxP/jA6O++su+jM19fpiieF5OfxmFsdYi9g9eE/eyqzKlZb3WLNr/nMaAhqQZ3e14cGMjdrOHfgMQs1T7dL5GgWKj4y5wKel2nKYub+uV66RVro1K/dD4QxTjsQpEe9IlmcoHAOj3GoNdj0NELIU0rtM43dabScNkjIG91Y1doLE+dT0LtnifKaD+J5d+Vhr1gMnF5DiNk+PhOFzzWTjvrY0xfLEzH+j85F3jvjvrZoMXWPUK7PLbAADR4ldmehYA7EPe4W/fM2WSogGBJu2hNTxY2lX3uPtbu0SFTotmW+fYBBLFtElUT+0YRhR5u3vaOWOEO79ZdNWRDe64Z17mMqJa8cWysuikdMikodAC4xfntq4c3kfVxgEAim2Yc8AiLi+XA3AL1XTMDf+MO6ccPsncgqL0EgVkVMDco5N0l+OCBn+F9ARl61UU3++imWLIbW7jn/0zCm/+D3edvkvXLDH3QDZe92YYm6pxAg7Hs1zEd2jHePrWtGgb2ehMqADmNTcz95XrKoIs8/43A9VrQXHCk5mdN+1CXAfm+jRKqcW9KQC7ZY8T49JoQs/s8Y4/OdONbx47G2+83jnIGo5yv22x1ZPnGHjZHjI+8qKfxhje8KlNLQsHRsUMhMkQChgZ9P2U2m6CCta/i/pVs/ZoYxEGePMOULyAuIba8GCu8gEA7vZ9bPHZnH8iphAa8aq0GU3VXSg1nKvc/GNXjBf6CMQUf2L63mIuxhHMi26DDQBtilAhTppSQS28hn5+DQCg++TLspWPhD86Irb/hot+4KltCy280Oc66ZihTOVcPuJNT8pilsrCMXWGYF7uiAXsL5xZKMVIASDjZqmNKVSbkj1MxHyT5rBxqG/NKzwJgsp7M6GM/emn3MG7s8VVyo3u7PGMAACz9+dDduCnNQzFVtF9w8e+Mke/rzGtQhGjNzqz0fZjz/dpiqN0FqGtjWswxo1/FkmBh42KsPe4WC3Is+Ho25fT4Q2OJ1cgE9xnzt0L8//zSXeOYy5VuJ/qujSv+Gz2d+eEMQaF11653dIP2HoEk6qAgICAgICAgICAScTzLdL45JhUZYyimyl4jIkyaTbGwiWHzMEld61s/u6oHayWzgGzz8v88wUgIilr0jHdFlvZVADt09y5ZbfC7OEi3LJGoqULprU78WwzabZoXK2oxr/n1PLmH/9NNCctqci0JlJ0kjkwzTtSlgdE/U7OlYq2sUHaWi+zWYeW2Lfs4SVE1ZT2o6WzUcNRr4Al8nUV5I+oMumdZLITlYDufNJrpsxsAqu0AOx47R3zold+is0Hat9wlLmlD/wSxmuPxnRUbPau7WiOw1CRahskwm1TYNc+6I59MD678QmhCGVJZEGIHtKS70KL0lzRuZJIp6ltrnkU5oBXuWM2YynL/zmDG2Yxy3zTH1dx5OP0OPBsQM900k5EYtpI35M0NsaA+wv1r1KHtF3SbEZFzN5zFh8DkHagI8OzI6tt1ETXRuU+ygdrFwza56hAelkxywcmjWvKnJSoqomWuShO8GzGWhfzMmoL1WFYr53ltLTWt1XTkSPRr+yfXDBEc9R7gWLq+yjnXlPI55DLpnqmoAK8EWW1ipROWgIKLPrNc2H+nwsCZxVRCR71NLh7pwJlFlvB89AZn/JFLoqZXaenIt/0NFBx7STqXZBMPyoCA6tzlQ8AMOLNzHx6W0SU7LP2mlfBvON/G24zp384fx5ywOx9yvg36ftbmlC8muQaxpgI8QM/c8cHvtafMw1jvC20oDbQz88k08ioLW3t3uLcYUyEoSedtiyXa/wyFzwYcw+RvPq1C2m95r3sFWzpQOQNjgChyZqFwiF05tQmZcVTPvDfbidsn/QDtgpBwxEQEBAQEBAQEBAwiQgaju0AkpQYdkrLhgdf0Yd9rl3TcN72O3o2M2UCkjM0l2rqPE6EqhJ+546O6Sw9swOr5BwARC2ww85ZkOkOF50qNrOaqrDs7T5bRJoJwFG3egkfSXNtHIud8yv/xd8njsySvpLQVSfgcLzYa2IKJZEGVlw6dtDVEzmvA8pPZeZiYNTbdnd7LUJ5ADj0Fb5sFCxMSQ7JTrRF2Y7T9yF75eoI7CM3uOM9nERDaIFHgSNfl698dgwncw7UWFDtzkuTZu3H3/ORPzka3cXvGRaH+pzSHB0gcMwBiSTXeUB9sF5VGq/GdxjSgJH0t20/qXPSQBTbRHvppaMo+DZWq8h91P7jmqMyBoRacfE+Qpt6kg9eRvXbOQNm6q75y5gBF7/9SJelq52flnn7j541g//Dj7lveIQKgsr+QuRAqn0DSJpebGUHU7YPL7ahf5Xz6+hs0KYqLRj7kBVkfGOn1SlCWes1JxyIsNSB6maX31yhRp+42/2eANZy2cd/69L2NKO2PCB9UWvryMfBj7FxvYrWA70vEPt/+LY5slG0I/Sd4qoQfRz1Hp9+LFpX6kNFP/6ufQAdBx+bp3RcD7b/aaDHl4G1uv5LjW4SHynS1pxxofgEUMDUzhmwuzjfLHuf0wyYxWdLWchBnIgCtIaaacoXwt79fXf9AEcdzNLo9mmsCcmFX3/B/b7pm2Pe1kBNf/ZlHOSP572WztzjaBYkNQI5+3+z/NDY1SpaMrPE1cXARxYDALqv+Jua64XivvvyO/1hkgrd1sqKprxJEGU/pg4uPQpdlzqCBuv9cWQuidH96dvzlQ8AFrpxMhFegOYNGsuPfD3sjb6uX3qpL1bUfDwtbHkNmKbWbhr8UTnSN02/d27juYCdBpOj4ZggK88+165BvMFx50fTdpcLPduuUf3qFLdgOvV6pTIex8mtKbQalNk5kpsFO7gCptubMhT9N6mOyMBO8Ths3MQcSk2oOmI14BaCtJmgxZ2NxdEvPZHBCOtMHtD7aiNi/uUnItPjJ8TKADsfmgU+nkJlELjJc/B7B0YU25QZVMrRsdQhGw0qV1wDL1mU4zkvcFf+1f3y5q7YyFw1XvGIw55UxgqJKOFps4xamZ2h9736Rnff768E9nAMNNFep+XKRyJPY5lXTaAO2aSma3bjRkObNNHGjRY3w+thmNnGTzhxDfapm1x684/0iShCgzTDU1QC/AaFNzT1MnCUM0PjRVWrmCowg9X+45hl0P0ZzdGid/448f9lh8zGxTnY7HYkpvGcLWU0U9z3kejTasLX3zO9WRhYgYdXusX33NSCS8ePsSN+cdE2tfHbFlrYHIgWwcJoVsHoGvdd8/B91fqdOU4pKsq4M/tA9+vblWnp1Hal7v2r/86CIktmY2vvBxalCCSI6WnDozCz/aKexstCKWEu5V4mGy1ZVPqNSlQEDmhk3RkLdp1jujMz9+Hx2T7lYnkwscjAKhjPZsRmNr0LuZ+YLt+HohKzcJnD3kRv4DRYGOUJSOzGJxDN9hsUMokstgJ+o8GbLWo3pXZg/ZO5ygcA5o3/lfsZAIj6JFr0FYc7AdXHbl8FMIHKtkOiLWc0/Wb49uJMiH178fM1m6MBLHjq/qxjQaz962konvd/AIDKVScBAErv+VFjBHtFOsMO9X4MtnFdTBp9PXVd8XcWfPI4QG211N5g7pUFLMCDTTK2ARIrpXMWsK9ny2Nh4DBsekMW14E17htYX5djjdPNBG7GGNn8NCvPRNZuAZOGYFIVEBAQEBAQEBAQMJmYQCzGZzMmx2l8K0wVEpoNTm/b1RJpNrQT6UTSZ6forlks/bf9ngavz6lSTe+uIjHe+Jg7N3Ox0AYqCbL15jKGzY38rr42ypIQ/q5Kg2RJClGvKufDFId3XOP78tTM2s++DQDQ98W/AR1e+jXkTclIotk2FSzt8eZWaJ8G8zIfAbnk1fobnxCKTjINmOkj3ZYHRNpNtI3ds4WWkikfR2D28w7HFLGVTDk2PTUxk6NxYIzhvLEEqtjK2gJ2FH/h2zD4qZNd1v/lVP+stKsskvjx+g1F1Taz9h/zvkSaXhKOYqtIrHSEe8Br2JxEm80L23tF2suUqkYivbOplopNQhIuTadrveRt3cMuibZe0eyRAzBpsOI6mtKwjoE7z3RSysN+1miKqZE2W7jo5vtFKkmao/ZpYLO5bTjmJPPh08/xzMOb3d2LbNxAtDFmFF4AuNPz4J/utQVP3YbjzvX0302oqFmaSFqFLbVJGoOWOUpxTHXjtl33EHrOfufYeWqC4l4+Yrc25aB3U7/Wmkiqz74lYDMh0uaYCLjn5+6+Rd4xmOIk9O4mGmYyzzIREiZkLhUxQ21NOQsv/6tEOB6DJVqDTAXtrdcgOuYD7iRRmlNZpu/Jx9ZK/IV0tHgUSsA+rlxDn3SEJ52f+quUiaTORF4xfU/OB9NTT99TKKsHViTvb+kC5h+SrWAKAx91cZt6vGR/IvjobW5usJuXw671VNZEsrKtkdfMmNpQazcaejCbQMXAiDcJvOkbAIDCgSdwuy6++hJ+hGOpcJwRIZ2huCmsuVArVW7no/0yHhM4ponlSOEmDxU+mWBGxcZI4DQmtHbDkmUBzY3rH2Yttr3l39x9L3gr8LDTiHMcDkXykTle2xbGPGMMMLIpUxoBOwZBwxEQEBAQEBAQEBAwiXi2+A1uK0zKhiP2dntm5uIt3pP1w1trYa/7hHvmtE81PJuZUjR1/9I7nVTH1iuwf/oSACB66T9nSgNI+V8QpW06KqtRAfpmH0Q5kOBS7HNRB37syoi3fs8/S/aaPWJf3KyIZH9baGki7aRAb3XUv+noc6OPZ3cku/9pJ1nsa+kSjQtJsXVQQXovaW6sFak1SSce/T1wr4vUat76fbkPcN+IpHfsrGgaHe00zSpJdigfg6uA+37tjvc9K2MJmzmpJSXhibuVky1TAZMkqtiG0UFXn2nyRBvXlOR+KwacFc5BvVnU1i2BpVxTd5c2VUz6ZthNT3L0ZUN+HVr6V/d5bulUUmZfdySlap2i7OspsNkgtwO2DTYFiX7ODr2+HxTbAIoCnRGH/PuNme5roJ5U0vLlb3cUpHOu+mOCBGFnwWkfdD4vdtmfAfKdYQ2o167eey2w7+nuHJFGtHbDnPRPyft3PQIYdJSWLEH2NvTW2gatkx3ZIH2ZqLj7n2bKZLPkFYn0zfS9YNc+mr+QR77ZJbN5ubQV0mzQWBtXk75tgAtO6skfLLx0t3chcKort336Zndumg86+qd/A17wBp+e1/DUyiIZpsCmbb1Cx0vOsvT/wiNh9pqSq3j23p8AAMyRbxc7e9J+UxTy3l3ZT4Ql2/WKjKtM1W2BOx1VbufFXoJM32p0E+zjN7o0PC0rKoOw3r+NfERgIhm7vHZKgv0Z4H4fiDGHdqHrwz/KfO+4aOlG/5feBQDo/bzXjm7jxZod8BrjJlYVzR/w7b9eUcESfTuMlUUC+dy8yBEQ2Kdu4vt5TdTaDWzwkeX7nGaItfWmINYANM/VRlS0eKXp8r6fNqWFMyYSzVUO2PX+Wyu/mqagOYQcvqfvzVpyc/A/umvD64C1y3LnYTwk/Cs3T4C+OWDSYOy4OvitR/yoW/hFcxvVsmNtEJo5Ddm4Dvsbt9GITlm6rbMKG9dgf+eiYUYv/1zm5+KfX+CeecnHG9PMuwmqV2G/da5L79zvZM5DVti4hvqXzwQAFD/2l8zPxX9zmx+zx0nKmVLMtPj/dDltzPex+rdexedftAsA4PxbvaOpVc5gNmU+Za16BzF7SLo80NEgXa/g0sPdYvGSB6uZymfX3O/S6Jp4ZHCNyuecSVXLB3+9TdJLY2ipU713/ssTmZ+5ZB83+WWNP6HbbnrzFS+/g3nPIz+Z5sVYTFx2ZAPsBmd6GC16aab0Nr/HbYB7/uWBcd+rkRBaUDsbWCmLSh+FPq9ww20uk6wrCbIHIpjwMV2yIL7axfwx//g1xV6TigJfGRTiBXbyryrzS5/nuIavHOv6yXtuetpd8fdYzczEG89hSUPHVKEFF23c6FpcxxePdum//29jsMClsOzVzlxjl28+03Ct2ZTVjNmN76sO48qj3ff98G2NxADjzUF0rYE9iNKvV/C7M5yp4kv+WG9Ioxm+eKDbmJ93i+SnuYAoFcupCUuPjWsY/LgTYHVd8ffx79ffKG2m2vRZi/Xvcxu0Gf/Zn6l8APDbY127f8mvtn4RaG3MMZeyxvXKl77lRT3FPBkP8bJbAABRxvuTbSkVh8PGeORVzkF+r/8d2xx0rPTHGosefpWb1/a6Lt7iPWmMXuw2Gm0f+8M4724UzDWUsTwA+wcX5d6cdFHi2njINNbWq7B/uQYAEJ247deG2wNPviIfc+u2xq7Xlse/aRsimFQFBAQEBAQEBAQETCKeZxZVk6PhIKq2Sw+ZjaXPEupJAI0OWGPArr7XHXT1iTo1TsW/iGsSf8Ofc1JEpX7199uNTmrNkk9SkderwvHNKu9IaRr8ubZeMTdgznnvUNXaw+rViKhrM+DBU907dGwUkdZ6CYdy6NK82lqSA3jTBu/IaU7/NJebQc8y931bo1NrdVipchVFLYD4mnNg9jrC3X7mF7IVkGKpbCM0SHi8KcalL9oHS290nOvo8BLhQuvETQRytFPSxJkXf1QktWQuxZGjK8qZT8zwWLJN7S4qwP7yEpfeGY7umNOsDIkpAUnMa6ONacSK0lmRAQAAiq2wv/6ke9WrvpatfLe5+yIiE9gK2LiOTR90RAZTv/DIVqfX9B2+TeTScPzsg+6gWII51h9zXByibO1WY4ySrBIJA9ESD6yE/d1V7niqN4siJ2Zbl1hCZG4y7wVNnDbFjJJpkilWxrqH2NwnesVXspfxem82+6L3KBO7lAlJdVjGBK+Bsg/+DGavpDbMxQPymgJv5moHXblMx3Rpb9x2B2EoNsb/Z+87Aywpqraf6hsnb87LsuSwwJJzFhAkiSRFJRhQUUwgosACJlREX15ERV9QMYARswISJIkEUUEyLGEDu2zeSXfu7fp+nHOqqsO90z27O+hHPT/m9nSo0F3xhOfU2KSsOtYSU5iI4KxNfeoPxvQz2OqYTPWT8R1Qlkr19yz1PYDff3WMNZF1xxATCd1O3XoxmVcah2B24n3ubTMw+2oyIzOmUm5keJ6jVKFsSChM25A2FZSg2Sw62OSgTPUDsP7HU0MgQPW+bn/6Rqf/6TGjWVsXMyvTTzLG+9Ar2fTNoWI1xBP9HHeqc3LCGkD//gKoI2hcg/SvjgkmqryN78HffNVLwAv8Dbc/yV6TuXQxxaxR03YhynrAUsK7eXMsLDV+i0z1A4Dw5osAAME+Z7e8L4uWUP/rRuh7eCw4gyw3lBOvZH0gXETvIthQxALrGS+85bXVcMz6+ehqOF5npFweHh4eHh4eHh4eHqOJUTWpmvfwYvzPHmSn+OG/RjUdulEzktBWSPXraOHYmxdahyNLp+hIbo3EVihtra2vsZ02Ub81jKRK8i1WrJ2qSJkkyJTRasAJ1FYzAfD08ywJ2fJNxuZVJHK6bql7VXsyuN1wGGhIkZ1v4PpdNIFr/2zqX+oAFjaRGuuGTddFXNpXKCERpVwog+ceBv3PW4etUyRboTHumZnrudS0dAh9K0v9D2FpLTu7zrvjIWhxqDPUsQ1A8TcUp1T55kiXHIV3XwkACA69NHvBprB0q9YHLZI50ayJ5LRQtO9f+mTfciN5E+2Y6tnI6oRFsucEi0u0iaCEBAplm4ZLQQqQhmTM1OQzLbA+NBsCFRQw5isk2b1idxq3Pnb/etbQjiRQVTs7KE/a3JAA4MEf0m8PlVPt/E57P39LrZUNjCnvuFQF5jBVrGhZmBaYaDpZcmkIMHTEth8AaV2FQlUoPQ2xRQGYM4LAl5sdQMmseN44jRuNjfiLDK6xtM3SV6bONdJ4QwG9eiHUTI6Y/PJf6dpkCnyHcpeR9uulT9C5tnG2TRZYE6CUdVZfyxpeIbQYv3lmqbhA/461GcdebjTcal/2g3J9ZOLanXJnxN8CADDUh1c/S+1+4v9yHfj9z/7m/VBxaXehZMdQt09y21CG9pv7Y1hH7boPAgCqn/l3rnquVwj1OI8Tp/+FxqFb3jgVh/yUA7+mBAWWqOWq2p24FrszX3kcenQzPovWi8kGVNvYCPUrAKjDL7FrDKe8WvyfpD2Yb96BS844BwAw6C9ybgAAIABJREFU70HScOjVL1PwVjhtWTes439iTAV0Kf9Yo3Y6Jdt9GTRLaquj0Lj9e3R8/bvo5Ltu4MI1Er5uI0JjdCX264rXG0uV13B4eHh4eHh4eHh4eGwwjIqGw9VAiGbjq6zp+Cj/n0W7YdOL2gta1pARaieiiUOLVC/PcwGXX4eWscXQfApLkooE6aNTQdRPAaD/pR5CS6l6bZqGdpYl0kP9RoshttNKBaYeWtg3mLoS9QFb3hzY/ji2ja73GxpYzUwdItnUhbL9BmJz6/hmGMmaUsA2+9LxKmbHYYmNXvYU1FiWtrJ9s3lHgJUYqcBKNEwe/M6rXVBzD8tXQbHbXi9QUAeeC8DxZXHpC58mSmBdoveHlQuBN3yKjh0qzFYuVmqvD+Qvlth419ZYrZxIwcRGXWtrp14UP49eKJAU11DIBkVgqwPomL+DdqV5RnPCkrVar/2e0idKJfsNhebR9DwFTN8+V/XEzjvuZzBicLv66L3URv9vX2qjZ9zxotHYrJOUagTjldqVGOxe/ch2mPBVpkbei2iuL9mHKDXnPfhO23Zc/yYZf6Rvljth/C0eoMBk4PTVpG2g/0IU4Zi2i83fMBw57dmkG2WSU93TgTEb5a6jYbVrG2uDqEmZpR8FJUAC100QzV2vpRqVAI7t42x60zigoJS3PmDTe+YO+t36CNsnXEY8gWg6pA0/9GNg/w/nq96h55t66kXsz3ULaSzVqd+zNxa4/0ndGzUoV4sIQBcrmPDRq016AKAf+wUAILz9eyh8iIMexmmq6ST9bdQtLa+RktvAieUTLshVP8D6hsXLO1IIY52wSEn5DvnjIix+L/mnTP32/MRzw2s2OP1lRN8sAeuGhWiHwjq08dPk7yTtcuLWMFpCp9+kjuvSJjkQrmYrB92/AtPaXSsJnivdQK0gDaDxl0jxqTAakzwIszPLNYNrkVA48FQAwDOXkk/I5qcO2BsN6731Pcs7tqqe/zwa85Z4fSk4RsmkShyjuqeZjvCRe8h8Jfwj0ZepfT9o1NKaHa6URK92wTzTAOzAzxhusxFxcI5tVgwGVtjFax41uUxurtO4MVvgATctkmlYt/m5zrjiNC5RpE2sgkLUCR0AimXoV9hpkOMW6LBhHemkHH3MKd82xsZjyMo5DuDV+8ghdOI7q/a9bbQPVzXFrK1VFO3OqXj+G18GAMy+XkwJeAE3ZW6UYx5gVXUsvbAOJeYQstCUSXL2vlh87oEAgGkHX5itgsbhttO2FaEtdZzbEs7gjhmdrb82TpMm+jjXSb/6JNQeFH3ZOGr2L3M4/8dyukNmwym8+WbBDlgnQ96oZcJLvLiZsZuzoJDFlXC362S/GLepbddG9T+E+ReeBgCY/SM2s5J3UgqSaVS67ATi8MRrJQsSHbum0fdtmpg6r3hzpur1XkChnjs//89M9w8L+SbslPyuu2gs02sWATI+OW3Q0DJn3fjw4gJZFzkA9PN3AADGX3STdRhl7v6LPn0i3TS4mmKhOGXQxYodg2RzWaxCP3MLHW9KUaqNw2u5CxjgcU1IK3QVuhStk1JBgjwCNV5IqAD6OY63M/edyIw/0+IbJ309Zn4K+3+xYjfQ7gZFxg6eH9SYjaH/8RM63u1dUmpOs27JEmbvSb+NQYe0gsfdsJgUJImAaPfToB9iJ9jDPpuperqXzLJUsc1skPTGvLmWOa7S45jGyCsoJBeTqgCMnU3HIlzi2Dzf/8DZOO04irUUzNid03BpUXkeLJSgq01iiagCxTLJi/VtLtJUKKkw5RqaL0N2nm/ceB5KH/5D4k5x6k5bfKtyTgdmcYpv1Gz/lXYosZGG+mw7kdcRNuw/hsRhgmnfxvRZHPo7p+A9tz5J98mmcMVzUNN3o2MeDzD7QDuvy2ZFTF3LHXaOzlFFmZ/WCxVxvQY8R21xs49FaWt175L1Y8q8ms2iJ89Z57Q81j+8SZWHh4eHh4eHh4eHxwbD6JhUiWlFioN08MZLEveLZsM1kTIq5VI79IK/0X0b7d00z/BmcqQNDr3IppviSGXKKNqFSo9DSbvZMDVznhdH2o4JVmJmqD+FzrUAo8J2zR0M/WDBnquOoeMYZaleNR+qM+ZIG5Rt5HLRtLhB8UzdWHIcFBOObFkwYU8ygYpKrqUeyaBS8p51fdCq+gORDlYwddvZkbpZlb+VwEkgH9IIMA2g7JPrNWiR8kubAdMOlzswdvb0fBUMUoJfsTmQ6/QeV/OmmQMqFRg1u5FIimR22k5WEtnFZmQ3fhQ49Vq6LpKzSpfV4rmR3ClTa7KQp45j2KyuMZSkGXXMR2z75HOO2Z5tr0X0TIxFPXbU/MaAwy27imoAAG2lgqavO3TKhXxD1HrTbAia0G2qrqm4bFf6dp98gM1CHW1GZpOurnxO8QCw5HIypZt89ZOJwH/qOKaArg+kalu0a+oE0ibfez5pkfa+iWk+WSOrdQh19BcpPdF6OGOYSX+oz7bPXtbq9ZB5ol69ELedTSZab/hLDg3HpI3pt1CybUsk/6Ltqw/YSNmmPSlLDSqa5WIV2OYoTo/bPGts9NJ/G62wSVcFxhxQr+ZxvXu61YAacg/HBHarnOabMm6N38LWbyMKjKuZvlwNDUCLlt30TW36q5ae3xgCOGgpRLLO95z27W8b8hCZ4/SaxYBIk1mKrrW2WiyeJ7R2iEEk/RZzbgKF9Uv5aSh/4+fdvjmRaKyv/9nfcfqHhJ6eCQ2KldZmRTkd/w2pR2T8F01fLCAunDGv3m/ndzfILJuWGtNns+apwozy0uYeuBH6pYfp/l1Oo3MDK4HlZBZmNH9Fp90Y0+7syEOhOyxKVWAakU+ouRx9XN7di/dBL2HiiwOIuh1B0c71Up7htGbSR/9L4J3GPTw8PDw8PDw8PDw81hNGRcMRRPwEojapgs/sPAUXPhSlnHQl6VZiWGip2TB5smYjqyO5K6VWE/Lv6sWRjSTgIs2P0eEFBceB2KF9jdn/a62tNMcJvgSA/FwMVW6XJGyoGuvXkjSxcNxFli7P2ByLrbwakb1kbdF8AEC1xT2pO/ZCOUll3KhhaCXZx1fjdKiho52ZtZskDNRYAmcklxXj+Gx8OUQyHzaw9Cly9M3ssirSkZL1k7DlHoEkQiRr8XdSbINy8gAAdebPjQRM/+qTdHL/9xtHehMgEFYLgWVP0fF069A7LKRtu0EW4xqmiAZQR+9xywCgY2qShtI+Gg14RadE4ufQ6YrE1nUuB4ByJ1YvJOlsVkJH0XgpQ/+6jkgE47L4xE2/pTxZGt64+i0onv07OudqFVoExcIgayRz+HC8vJj6wWSXotalywag174CNXZj+kek9cVq5FvwndhhDucd08hBa6tVEGnrwGpLrSvVGeqz0t6OWBDKQgm7bDMmc90Maqyl6F9u0jaa5wlC0atsIUy70k7QM6FXLgMcpM1Kfh3tntwvjr5Ttrf+ZBJUs1CydTO+HKyZq46BXvp4ruoZmuFC2QaDnU0+Z5fsTGWcd/+zjvbPaobt93G0hJPYUb4YHZ3Vlm9C3zwaQ9svomBohvQBsH09CJw+HmunjSGjOcgD/eLdlNqsfXM/m5reArL/VzP3bHqPzONn3LU40Te0ChLU6i7Cbx4HACice1+28jBhSjD7AHvO+ILz2FdqTwa97X0VijUcJijqUB/0MzfTuW2O43Kz/0VjyGqihLr98IutNlnaow6hpu/O2cdozYf6oOffSc/m8aVaX+QblDMwxNYuYvUidZixK/DSv/g+mcMd/yp5d0GptVZgPTi5jyZeZwqO0dlwhK9QQwrEAdqBLEAuuP+FBL972kZB1weh77mK0tv/48PmrVQALc67GSIy675l0EtJfRxsnc1RFQBHswXUlLl24QRn4gag605ncONXFNg0QRZaQRGQWAyyaJMFZ6ndDiZibqVDY5pTPOsmOje42mFp4Xpz7AcUKnZhloOZKRzo5+rYTZzZJMmgnjYg6JBYUAA7AAZFdG7LC2Upp7y3oGicdY1ZW9hwYhaIKYFlmBE+frM4WbMAY6fmXOis18EV0I/St1B7nZW8Fo9LohvW5Oq4K+h3qJfilQCJhaJSAXq/ex4AoPOKt2Uuk/u9TX+TdmQ2QdrZ3PECRod24nRYpMo7v0EKFE2z1mtY1BIOxQD0gvvpoFBJqu2dOAQTd8jp/NeWP75MK7SKrxDENnpf+N5DuJAD8kbMq1qRJ2Rk0HGx00lH04ETnV4b8xUeO914BLJBDxuO6RGjWMWD/6RN3QHx9q+UbRMlJ+K8WcBxnpUe6CWP0iM81mhpZ4Uy/vdOGncyUjcQZBHVNt6+P2HKknakQyCUOBUsfOmcjMQmRAXo+/oZAICOz7GTvSy0p+9mFj9KohM3ao7ZFi/6tLaLJRH0OAvCu88iZ/397nAdsptD//3HlOce77WbBO6H8x5ayHkPGAY/S5AS2HFQ3tFQrzV5kzHeMTtrv5j7mmyeq93WpIwXzeiaak15pH5sHqeHeu0YnQcd68HR2IFhLsx6P7ebwa+ROV35Qz9vKYhQJ1+Rr0DGnNGaEVtzYru0io+zatwmph9q7TBYPsvfaetjOQkrRDPpCoOYw/wlZpJ6YCXwAm/ytmITQsMAWAYW8IJ+bvYq6qf/SOltY9dCxjQsxbQ6IjRNG/e2PIR+pfx1Zt2qdENvf1TknF74AIJNDub0nLnDMbmlPJ25pyezeNHjNYA3qfLw8PDw8PDw8PDw2GAYHVrcuFTNgZHquiYercxXGjXgSZbK7J8teyN1Ew7rVjE/Su2xsmTEU3fQ78w9HUkw7+cMHZ62UjeR+FccjnLDL1+E0Y4YqbNITgKgGHP4rg9Yp1GWuoQrV1izKSmPUNyN32JEzkqqJLzjDcsPL46GLFGLpOuaI8VVncWqldAFIqlw0jRO832Re9w8dLFinemXseOu5NMxGStfIWmfJWAdBjqbdNLc3sKRHACwUbqpE90bMzFzaNlFehU+9gtgJplDGMdWQSFA+4775SovACuprHQ77zFmJKdD6MUUuVexVFn3LrXSI/luhRIw9y1c6CQFrtQxYlLAdKCmj61ZZM1chHpYzIMKJQRHnZereq+lE94FDy7CpTuRpvGih9dzRHIHf/vxrwEAe5zyHXvSvDM2z3FNRI2JJwCOHm1IMho1LB2M3eeYJRqTBxNJvj35jgsBYOi7ZXyrmeemRa0Hs6GLJdGudlTIFISiMxyKOm4DTAcs5Bhc9kIJ7Zuxpky0zQ3WVgyusfGMJK3aWjvusGZBr5hv4zOImZk8V27HjtvnlOZPFGm9a4Ir5iPWHCZh+rpmke1rounQIXDbN+jwGCZLGTOGL9VtXUw/t5orQ7tedajSpX+zCZka6kB4w8fo3DmHZ65i3ujrwyLn+Cwof+J2evzG9wEnUFyZ1DXA375Pvy1MtiLg9pB1zFFlaxhq5w6hfA7ReI4IL4oJbbYTj8mZB8Onfk/XWTOnemZaJ3Tpf05ajX/QewjelKm4hGk7JeuRagkQI9VIeScqKFgK92KMCr3cYU39uMzXvOMUnHn7s3y/aNobtk7GSsQxk0sLpfAfDO807uHh4eHh4eHh4eHhsZ4wOk7jU3MYDQK4ei/aBX/g3qSUUFW6oN77sxGVwzho9a9IDyoIki63ckprmvZBLIl1I3vGJBU6KFknPbG5rg9YGkrXR8FIh2K2oW56IiUpVo2E7+JdKUDTvHufcGgkWRIzjqVqQdFKlXOg8hZ2Zh7qgwZLKESKKDbBKrD2peIQq1Qy6jQAzDmIfll6aqK1Fio24JZJa42R2oRcL9U+AfpltnuVwFfs86CKDUw/9sScFWwS+Ar2XUeIDJwI92laOUNVnAWOo7ZxON7hrdAr2YabbbTvP5kktbv/cj7UkZ/Lnr5ANEdudFxpW6HjDCvaDCeKrdXcyWMBtGhMjNbDkRya6OMVmw8HNcRa7ttP3AGws2z0fYLeCVOE/jdAKWU0GxtS0zFnB7K1jwRwM0EgJ/E1ZSWBxjes4AT8JN8oveolHLn/xnRsbLOFCrmXgv8Blmq2UI748Jj05VuveI6uiGbs1cdx+mkH5q/krF25DH22TZUdkgwuiykrjwl6qN/2UelTtbXA3m+LPcttsmoDqRo/lHGbmmf1v39J52bt6/QTifrNdV70CDoOPD5X9ZQ4GrvaxThduwpsYEier245YScc8kc+Z7Q6a4AD3kOPxG3Y3fQHOKBusRtaAlqKBqt/haHPxVgmeRFfnb5lUCdclqt+9ODINBJNkxMLhZzPmbnzmMtQu/xQAEDlvDsoTXdcn3tcvnSrGX0E63EfuZS1AQIMraLvU0rTDsjY6xDRqM1F22R95MSpPDAkNnxJazx0K9Ho7vGpbMUGkD8YopNfqvQ+ToLgfE3LxE7nFvc7Y5isZapjYGegFHn5COj+X1O8zjQco+M0vmI+ACAQ1pRhkLbREOhGDfpuUotmcRpPg2obi/AR4nwO5p4STT9sQIuTuzgRZoDmiUFVuqBlIOLJ3ZgABEW78DaL67JdOJhfhx1GNiOsgtSLH3EYTpyFHDfci770IT4XWGfxMOawHdbtIJiDHQcLaELGzD0sK41ELJayuxGkxVm/2mMXIhOE7URbBioZROW3UbPpySTYOdksas1AP7gKWBWdkCGOj7UBYPe3Z6+b5Ava+CQGS3HyC9LNp9I2JNaMbHg+eldtbuIbDKyGGkMLCL30CQDAnLm8UexbllygZIEsQNYsctTvMX72oGBJCzhOhG4MWQf9+79N1/Y6C2ritnQs93c4bU42TkXHqU/M52RCfOM8a/In6T/2K7pn51MhzD3/bbiQHX8v3WnKet90VCcLg522bUX6o2N6GTE9BICw7sQy4Em7fzn6X6XNSls8loobR4EdOVF10pUVQqkKaGZkk/4n6f/xa0lzuwx49jzi6d/sxgW27/3hYsrjmC/RTcWq3SSzM7AaWOmQT3A9amuBgTWxHNi06tGfQW1Hggk1idsylLluIka7C3d5n+K0OmkO9LL5+Soo73rNImiZH+Q9GROoiiUL4WuH/PrZZGym+gDwGDn3anGKFiIGFdCGBDDvQzeGTHqax1dV7rLmVbIZEdOZoT7gCUofORin9A00F6kzfpT5mZYQh/iRojoG5XNuAUBCRwB2s1nuyO8Yn0pqwwQExuE7NG1HD1pylDRilbZzfmufgSOAqQ/YvihsVXAEqBwr5sZjdsJJv300tRwYXIXdf/T3fPUDzCYpbQPhCjziG6impkJNNgRaa7O51jyHnH/SlnbO+Tu1ITXnzXbeEqFAxB55dLwEPEYGb1Ll4eHh4eHh4eHh4bHBMCrbQTVm1oif/fMbSfpz8B9foRNBCWq/j65zmUSzEb5KsQwCjk+gggIwZYeRJ1woWy2GmOiIBLkx5NDCOrEPUihSjZNm3IG9bZyVNsq1vuVAG/N6S5ROHVqJTTxKdVC0UrI82NnRBsWcKk169Zox1bFxAAatyYdI5UtV64Qp2haJ9F1ss5JLMS1r1KzEUpzFqmOBR0jy9upDlwAAJl5OUehR6QRqOSlHuRyuc59BC/52pDiNa62deBLZokgknXFLJg3Fmob291xJ6f/jp0AHS3WnbJ8pfQBGeqmfvhVq+xPonFAnGi1VN9TkqJOtG6FXzT2JDsI69O1fpnP7xKl/VVICGBStw6Fo7sK64zHP5oM7WFM4Y9Ky97r3+dGESPwuenjxejevGnyVxsIOl0ih0tz0QRWs87SWCPciueycgtpAVANq2qFDvRkKVTjTb0fuU0Vj5KCXPEanJIL6Tm/C2p9cDgDI0xs3PYrNRVQADLI0er/306/rEBtzGtfP3ga1GWum2XxPr34JmDInUmY9yH1z8rZ2vBYUq0A8/kyj5tCR87M8ful//xqYmpO+WfIs2bz0U3+gMk5nc7Jar6E7F2gd2nGfNReqZwb0ZnvRcYz6XS9+xLyjgDXj4Uv3Qc3Yg+7na6pYtbTsIpHnOD9q0hzorofy1Q+AOv2HuZ9pmd6knO84/rxSVsLOGvFLuG/Oe3gxwO8kM1wthszdrIlw4w2ZMY/7qG7UTF8z5A1B0VoNxI3GChVEaPQBIByykn1eZ5x02yL88EDSMJ5yK8WdCW//Ij2255nWnDZXHS3ddtzkKUt8swSaPKOUspYh3L5Lb/6EnXfnEFVw+IP3Qx36ATrHlLlwHer/y0yU/suKu87wGg4PDw8PDw8PDw8Pjw2G0dFwrMM2TjQbF+9EErOLH16E/G5jzRGkRBUfEWWsSDGKFZjyiRagzBSEQTHpr5Fmh68CaLZXVWKv6Po0CLWcPOvSDzL1LaBtELRYoBzU1lr/ihx1XH0JSQ57vvCAlSyK7b44xLkaGZEIFqvWFl80PIOroV+4h445gJoyfiErbHrGh2OK4/8iUschqNOvBwBMOJqjBLPTuH7hLmAtPztj92wVFM1RCloGb2vi5Beu5MjInTn8ZFwUyo7NK0uVpjJN4cRt8NzpZHO+6b45fJn4Havt3pKkFDXUg9q+d9FglbusJEnuCwrAOKbt5Ijvpp3WB0zZLc3qkJWEL3+GytE+3kr2RNon/aZQBhY9mb1uAHovorbUcemDiWtu9O+syPOMa+ds+nnYwIV3ku209DnTDx2/gDz98KVHiUhgy8ZQUgPqRHdP9TVi/zRstA/9DqzE5IMPizwrdN5KKSOpVWlBW90I6lKX+fzeN+UAXysXonPvI3LUjtHNvghhA3rlfMpn/JaSMf269uCszVWbHGiJI+RbdEw2DtOmLRZYcjpxK5ue+IgFBfv9HL83zZTOogGV9F+66mLMuOCaXNXTosFevcAGkNzi8Eg5EBTt+3cJNaR/uE7Z4mv1NEer3uFtXL9trEO9pDXJfkt9P1Mr7/l+x/F/Pp0TTY5uGG1qHtz5JrJOOOD3S3I/m4oU5+t1xUUPsd/AwCr7XkU7NwyM/0vPzGRfY18CrcPIWACArQC47Spr6aAfprks2O3d0aSUgkYsyB0cDbpoIvuX420/v4uOZSzd9kj+fwj6+vdSGh++LVP9OHepbY5nWoCDScINTCo5xbS0ags7bmj+JsEpXzfjgWhL1p5PGv7Ozz4AvZDGn3WxqhlNeFpcDw8PDw8PDw8PDw+P9YTRYala+jgAQI3bzNgTR6RjTeBKDOc9SHRvulGDvvUL9OxB59KNLoOII+Hjg1y7SF0fhH7uzwCAYO47Mz9n7CO1hmLpmU6h3rWUjSkMF67E6m5mAjrq8/TLtqFqyg6tDf+E1aTYBiuViGlcAAx8k6Qo7V98tnlaMbz4PNl/b1fptmlXm1PJ2iBF2koKhXmj71Xg4d/RqW3fEn2uOsZKHcUmWSknDaf+IrUTyl+RGC18FHicNSh7fThL9ayUfn3hFWKWyqxhiYEkrTHJkrT1YgXL+qi9bIocEAaacifMNxRJsbTJgVVRbQcA3b8MqsCaNUO3XAXGb8zPxoaSQinZxof6gTJLz0UrF5Ss5M/Y17O0utIDbJGdFQcA2i+4s+m1PJqNkTzjjjOmnxeChMT02n3pPZ5+23PGlyIPtvgwUXDr52+H2oSppY30lLSAum+Zpb4WDVZQhJK2aNidJgJTmCJ3wQNUdvYh0A0naJzYpg+stH2M8wqH+kw91L5n0zWRvm60I7A0+xhjsPs7KL8X/mIpdh/5Mb8A0p6orqm27cr4W+sz9u5WO1W0WmHRMgk70T9/CmxJGh7F9+ihfjvWiKavbQyUsElx3STo6YyLv0eahDxYy6yGk7az2g7WFutFRF+qZu0b8aOhiw1HmyxtUwGslVBbHRkt4/JngCeJmUkJq2O932rdd3sXJxEYHznjvyU+gFoDrz6fr34A9rvqhtzPtILxDxIfl5GkEV93GCrwAurXngEAKJ3/QKa0VNVqZrWZ18UPU7QadRsUV9Y+7jjhjC968pb8qPX/4CtW+2iY1/qsfwVr71BsA568kY7ZD04JDXl1DDAtSpWbCeInVM3pD9kEmn3BVIqGI3Gv1khoVjomWLYytubo+CBr6QZWAo9QUFRsE1tTePxHQOnEimb9I3yB1HzitObCzT7V8TZ2TYd16D9eTOcO/0ziubTqpKXbbBOiwzr0Xf9D5T38C+kVSkH4e1oEBAecm+n+Vhsu3ahB30COUYEbTTgHWtaxUUP9yqMAAKVPZ3cGDJ+9FQDISVkWOK5TJUCLkfhCUzfMIsXQeIYN/PkIGnQO/v3CaBrFSsSsgDIfsiZPTiRexDdwjgnEFXuRuc/H/xWLct4EetEjlFTPjEz3D4fwhjMBAMHJ31ov6cWx9Czaaky8Ljtd5JU70Ps8+75sDsxuO4232fDRnwEDtCgJdjktcxni6Tdtp2sWGfrKYONsUdVrlxLhQ+njtzYfT3TobFptnBs7dvBvfdAu1pmPPkoJG9vQuwIDaaM6dOhlnQjeAG3cxClyXPZtY/hHItJXe30gJW0u78Aqa6om5WoMGnMjdyJ/7hSiXt7kB/PphKSpw0QEcwyscGisedG+ZqE1aZIx3hH83HYkmTcc/JdG5jr2n0+Lo7YL70tci47xsW/gnpP7ar24ZC9azM1juuKsc0ZECGSusdBE3nVYx28OpzoefU+2Ol62Lb27Tz5g+2GyHO7/rmlLnLK7jtrltAmTqNoJ6uJEHfj6kI2vYi/K+G37y5pPUr/qvuqVpnWK47HDKJ1tf5nPpCp93m8YCnbV3ty8q9V6In6dz9BPWDcbPpWyTklDuJDmziCj+U7LOV9rLDiD0plx3YuZ0ktLo+lYqjVueSNtqA+9K3s/rH2WTHjLH7t52LyB9H5lvmGjBv0QRXOPm40NBxsjKCmotUQQa6Dv/Cqlf+yVudJ/rbD47SOLc7K+MOUHOamg1xHepMrDw8PDw8PDw8PDY4NhVDQcHh4eHh4eHh4eHh6Exe/oGv6mDYgp18cDom5YeA2Hh4eHh4eHh4eHh8cGg99weHhtDzpjAAAgAElEQVR4eHh4eHh4eHhsMIwKS5WHh4eHh4eHh4eHB+F1FobDazg8PDw8PDw8PDw8PDYc/IbDw8PDw8PDw8PDw2ODwZtUeXh4eHh4eHh4eIwi8gSl/v8BXsPh4eHh4eHh4eHh4bHB4DUcHh4eHh4eHh4eHqOI15uGY1Q2HHrNQgCAKpST1zjuoFIKOqzTcUDF0mEDKigknln18S0BAN2XP2GeNWnpEJwIZ9CAfu42Oq5QkBU1bRegPkDnyhxa3jynsOrjWwMAxnxzWfY6rl0cKXszhC/fDwAIZuyeTMN5F03z0SEU183EbNQh9PJnAADfffP+AIDT71xgnpF3qPtepRPVMbac7RNaljcCfj41VqTz/i7ZeToAYN5d/6RLQ/0IxmxEx4Nr6b5yhy1fzk6nJa/6AFSpvcV9/D47JmZLt3cJl8cq/uJ1VUpBr+X7OifRPY1aom1rHUIveIDum76beZbuHwLAZePntNbmG2I1fTu18f7J9mwyCKGXPAYACDY5KFP9qI5LI2Vpel+sLWqto/2Mr4WLHqHjKTtE76/1AsVqtOz1AegnfgMAGPzV1wAA1fPvhF45n24bO5vuawzRb7EKvfBBquPmh69z/dLabVr9XKz4yOYAgLFfe7ppeq3SMG0V0XYlWHvBjgCAziteSqlNOlrXMWyaV7Px9NFjqR1v+8tXIulqrYEa9VfFY2daO4BuOO1TyqTN/8+9dSoAYNNf1zPW0PZFm55brtC5UyXKbI7DBl0LCghvOJOOT/pmJB+lVOKdpdYRdm4x89NQH10qVqF/chYAIDj9p9nqt/IFOii1p9SL6zzUB+WMk60QLqB+oiZsxWXi8ahvOSDjX2MQALDmUzuj64uPclZOG5I5MSjx/TX6LZSgX6H7g80OzVQewNZR33AW1Duu4zz66dz936H/uyZCbXeiPMFlKgBrqS0OfO3NAIDqp++CXvUyXea5JDIPcllVsZIsh7vGaDHHahl3eSwbtn7cD+G2RzPWUT1RqCTG7mZj7+CXaRyvnHtbMi+nLQPNxpr0c4LwG8dSkc65p1mVkvnyugaqkGynvF5DUHTmKTu+pM4Xf7mCjvf9aPQeZ06U9qeUgu5fQedkflXKOea83HZ679WUxCEXZ66jx+jBm1R5eHh4eHh4eHh4eGwwjI5JFe9Ydf9y2vG7GFhJ1zomAgOr6Jil7vr6M4BTrqH7eCesCmV0X3o3nWMJkyuFMtqUsZvQqXAIavL2kTQQFMzOWYmUqm0s/R+U0fOZu3NX0Ui96gPJnbdIXYb6zf1GMjxuMyAkia6+/Sv0u+spVtobf19ISmk0AgQTSOtz+u3z6TYdkuQBzvsptvE1DV0f5NSyQ9d66SAoAJymfuEvlM7G+5vyXvzwotiDGnpgNR2LxC6sm2+FmAZCv/IvqClzbV5ARMJipJqOdiNc+DCdm7iVzZM1Mlk1HJKuq0VKk0aJZsP8n6K5UyqAYi3WF3edAgA47wGWFgXFRLpKKajxJE2H/FLikfvMtw8KUFO2z1KpGOj5cOHfEUzbKZKmfpEkX2ryHKDSLQXgxxqAKtpjAFBFqEnb0qnHfk6n5hxP1wpl6BfuonOz9qV71iyC2voYAEB1iyPovqCQ7C+O5s60g8zVC7mWympK+FtesitJRufd8xhQHRupOxo1aPnmhZJJbswVTzTPiyXCmjU5qW0lTdPgSLI7Ln0wQ6XSka5taKIRa3YOwAt9lMacVMlrsp0mjlVyGpHiKKVQKCa1KsOD0w6HEv0r7Z2mJ2Hvqy98HgBQzqBNTauj1inaa3dsTtEctYRoE0pt9tut4XGT5wv99xuAvT4gJeFfDQzyWFrpsem1jTPp0e1c90IR0udFIt91wS2OZsN9HypaNtFQBgWgfXy++gHGskC9/TvQ/7qRjrc6ii5ucTD93z3DjCfud9Y8xrZdeB/9X+s1Ghrp1+GD36Xndno79Mt/pWsyXhTKQEz7raFMHWX+A/d1vfJFqyXJXEHua42akbqb9iJtRalhLBasBkIVxLJDNIE8zwUFW/eg3eatkm2umQaH5pdp2eoVeVDG5kFoWcf9/Qd0abvj+ZZy07ElUZZxGyXPAfQOpb25z/K6zEVUwwnzDRE2gPGzmpbjPxGvM4uqUfbhqPTYTioLA1adqq6pxrzHNMaTrzKNyZ1kBr7Ki5ZP0YLGfLWwDtxJKjX1lv+h9IMi0NkWTRcAembygVX9C+rfeScAoHT+A5mrFjHXkLLKJGQGtzLQMZlu6ZlB5/qWQz9zCx3vcQb99i+HXvwPSqKF6ZXJzq2XGfiSZkGintc6hH7lcTrXPT1jDZ16OROD2Wg4C1ONQqxcGoCYI4hKWAF3s3nDEZ+VitDP1B2T5gs6tHnIBKph29MQb4ZkEaBD4D5W279l52z14w2VqnS2NE3JAj2wykxgn7ifVPVLPkCb4In/+yR0IO8oX/qRBSZv1lUeszhpC+M2cc7xO5SNWeRd23tMuxMhQdt46Pl3UnqbHJDMa42o4/m7dk2xbYgXVUoFQCf1CdOuZOHVPR36JTJBVFsfm61+YpZZrEDLRMRmfOMrjsmG2TRxHw0K1mRj7MYmuTQTJIPYIiMrIt88ZdEwPGRRESTPDWPSmYZX+luUXxaw64ARTaoiCALMQidu0qFXvQTVE13ApG6IABR3PDhz1rp3aTYhhfv+p22VOX0AzmZF2f4n/YAFb2qPM52dG98+uBrfOIjyev891E/0iuepbwERkxsAtBGRPt89I3KPe59SymycTR+SxXjYgJKy5cESNkPc8k1Q25BplAjBcN936fcNn7DnHPNS9C+nczK21dYCq9jscPwWVL7d38MZKeBZ2phgGo/1xap9F2KO5M6J3L5UkRe0pXbotTFBWUboJY9Ccb7mvats43ukjc4iU26IGVF1DJdf02I68mDK/J4q8HDOTd08cX1YyDjpjiuvzqdLy5+l/yfPAZDsf6nlGttkQ1BqsyZaw9TNmtJFTRBVoQSdZz3jMerwTuMeHh4eHh4eHh4eo4nXmYpjlDYcsut3JXtsvsCmQGgMGQdFLerhRg1KibOilTiXx8bUu66KbbeT+X7OMxwCWJVs9AKdkx3HVJY0ye66UALCmMouC8QcJGwkJDYIrMmXqpIaXC9/jupT7YHalqQ/+uW/0W3P3Qe1zwfpHEu7jHO7K0GQcqY4/ulC2ZGyuI5ZVFdjcpYHRjul7beUc0Y61oBexmYo4zaz5ePrpgz9y4HN9oqU30guACvtEsmOUvZZlrCj2gMj3Zi+K5eH7+9bBmy+T67qqUqnPY5JUXJrIqo9CenMpKvpm3925ym44KHFudJLpK8UNEvA8oHbYqXbSoik/YhkrWdmUlrmOg22jTdlAJtliQmW+V46xOPzzgYAbPOLk+jc4BorsRSzjULJ9h2RvLKUSqkAmLh1vuoZR96K/Yb8Xc+69lt8rer0fzEZGLIkAEbSVoD+w4VUNNHCuRBJu25ufpcGV0Op//wFSv/oKzI9ywXLca/zVJPytRVTSCAAQDcyaUyaOdwLhgazO4sbiGZlcI1xPnY1TwCgOqcgTUOdCtHuiraZ25+q9lgNtOSZYsaR5lwecU7vSD7TEoGMJ9qRxIsTLv//7C0I2PTQ5F0dg/ff/rR5FiAttX6FCTpEg95N5jN65YtQUq8yzaVh7xIE46Ljf0Rrzv1RG3NWBb3saU53RvY6PkXaSezdSDhz691Ood+nb4ba/uTINaUUNJuLmXdSKAPP8/y48X50n2tqtztbBxiz4dC+szQtJb8TLfPl4CrgJTLLxZZHZauftPmX/w7Ie+H53dXWZNaA7ngMl43NpsQEu1BEwmSpUbMaXDGJ1To557vfddN9s5XDhTzvmgEf/Em6tIjfV5O+l1rvf/2WfmW+NvcG0CpuGdF6TI3PyVpr4J+/on9ykBt4jB6807iHh4eHh4eHh4eHxwbDqGg4IvR1smM10vaG/Y3ZC//1xK2xx2/Ix8O1ZVXHzpOE+c7ApKm6Y74Z/SutQ52Rwg8ZClJxenUlCIWTvpC/kiJtqPcDWpxOxalSfEzCqD8HQBIR0WI88BP63fc9VgIsUmynfFZ6IXadjhRAW21SQsrh+mAMrchfRyPxCqLaDjjfOACU2NiKpG6o30qe5Vu0TwAevZmOtzyS7nPbglAzOg7KRnMkEqCgBPOd1zCNZhc7xnVMBB67lY63PyVT9dK0GaZe3F5dh+LWaWnr6BdzhrvgocV46i0kTd/i51TucNnTCMZnt7HVWhuygXzgtlLrtVozkZSOYfvaMIQWB0ppp40aNPjYUEp3QN/xVbqPpV7GTwjANtf/K5I+CuUIhWFKpeiXJXu61AZ9L2klVFYNQLkzccpIpNlhPep0aB1qf3sCSd2O+tW/6VLbOKjDP9M0q8xtQTRHMuZ1iK9aABx8fqY0IhhcQ+k2Bs14otcwDfa0XSjtFKmurg84/lficxXipIs/QYeiOWSpsl76ODCFaHtd7Z9JzyXrMIehOUd/A2zyqTzaG4EyZZJ2aaTdkkdQyK55XMxaAWkLVcfhOk760CTNpETVcdBf8my2ctiHU9MEYHyO1GaH2jxE61AfxIrz6BuPvYLbaVAAStTvxDfQpNs9DWjweysxuYHro5RGNSx9lMdZpRR0KRs9rwt1wlckE4fUgzSWxndw4lbQC1hzMZXaWsQhOnDoUGXuVELzzrT1lW5DA6xfoTFHTdnBaiAdmuEEpaq0h0oPdP/qfBWURr/xnlajIdS6MpbqMOGDlJ6UTq5TZNysD1rndvk2A6sMzb+Zl7UGQh4PjMO8QybRgkK+KXjOTfUT5bHmyj2n4uz7ov4vLvGKC3XIBU2zSvjc5CRiUEoBLdL/T8TrzKLKazg8PDw8PDw8PDw8PDYcRifwn7NjNX4HIq3WzC5UrCZ8J/b42VPJtHSIgW8TVWD7xX/jc0L7WknSE5Y7kpJmVbBS2hjjk9Yhhn54DgCgMu+fmetoJIaFsj0W6YLUSzegX+VghWLDOLjGSmp2YKrQUoctny2ZU0adOOdK/QC2iUyhz5X/db2Wo26chdAcbn649XlJlEkZrYQOUyTbUuaBlcCcuJ1liq+PsIk4Ekntsl9JvkLbKO9hYCWw9YE5atfaT8MwtjSR3CTTUqk0fwLRbAguPWS/JJ3wcOmn0PEO/yCXvdwRofwEAHQw3W9Yh5Ewa9tfjcTJDUa2httunM1IKaudkwyqY2w7EC2GKkD/4RJ65M1fTZZ3ZU7mmNSgjdIPHRYU0QwwdS+qPTjySqbgdmz49WKmr2bpq5U4DyHuv9Q02KBIIlNslM14kJm6GVY7VG63ElpmKVr4bqLTnvad55L5qQBwpb0AST03537CdvOG8WnWvghvId+VlpJJFSTs5TV/X1Vqg562Y/a6CYQxrnOKfc8ilRZK74xB8SiAIdvDx4KmKqWMmDGvZDWiuR9Ym+kZA5Y2ax0mtMW6tobTnw4wdbj4GuDlBzDmuPfRsWik2sYanw2B7bcl65tVFo1F88CmAEy/jdjS52mfAqHWHlxjfD/0CqInFgpwVShhxeXECjn2cxyQrjFofCLMGFVqB7Y/JpY+96vBVcY/xWhQCiWErO0w17qnReZHF7p9PDDnyHz1k3e85HFA2NJEc1N31hxG05K0UoiA3w0mbUO/ouUvtVtNv/h1tI115iRHi1GIM7o5c6qwfOWBBDAstTv+lIFUAgBw9r0vm+Cem/yYx+uwARRS5klpsyntKc3npiUjZxok/a6pre/7D4GPNL4hIJ2k0unQ9nHjYjWiHlxjFy2yKKl0O5OlVbetXUzxFdrikVl1CHROiWStV73oxOSom/TVRO7UZuCx6stVz88HAESjLbSGKWepvamjsW4Aih3e7OKn0zqSv8SRqbfYHINfoPsM9W9qpin0mk5nTePiNmX51afp3Jm/aV0x93nXEcs4xEcnZ1oIWzpFzs3WUQaQSjew/EW+HDMNC+vOoopNdBpD1oRIopUP9RmneqHHtBXUwLIXMtctK/I4j/d/lpzi2y64d9h73c1GloWP1iH0TbQxDk75fuYyGfOpNBM92RgXynbDK+20tjZ1ksD4mYn0DFyHT8oAegWTJUha4RCwJTszxqlqoYHxOZxU3TSQjNvibtB0ges6blP6rfVCbRKlTlVKQd9MEdFx6vf4XJTWN5F9bJNPC9oWw+wDP6TfWTkcOt0xMfbOpl/7IpcjNIt2rdlE6vHfQM19e7T8OnQoiqMLb6gAAW80LBlAw4lvwvf3r4T+OZll4bTv23QB66SdF1KmNLM1GRPCBvSLFDNJzeJxFTrRFpVS0D2TuErJPmXuj1imyiJO+mLzb6iUgm7rbno9HU4/jG+ChEil3m/mMyWxN+Ycb6nCZRxUBYTXngoACN73CykUZxM6ggQWBNX7HAEB17PWB4RsRilxsByzSv0TIoDIM1+AN07onGw3v9LveXwJ1yzE2K89ZcsKUJwRibElC95KN1Z+kcgnxn7tSUpLzK3aJzgO9/xuepfa9ygmyzq0pCrGpErek8bar5DzevdVr2Srn2z255xgz0kdCo6QI6OAauj3VwEAykIhHLmB259ssmLPArShSc75Nu/aTV+iJHZ4+7DlMQhs/0uaPNk+scmPeWP8PAslJ2xlyD9c6H9R+1R7nJlSjxQTrFh9dKMWiUSeSP/RX9K1A3OSjXiMCrxJlYeHh4eHh4eHh4fHBsPoaDjKSWelhBSq0gUtgcI6HdOOeJTZoICnXyLJzqT4jjhFkqgmbm0irFqpeZeJ1mrV8qKhaMOqpSQZzKPhiEjExFzDlD1F2iESw8aQjXDK9LgY6kN54lRON6puJ4ftmGoTyWimpKqPmSi5NKhd43LUjpFmytBCepsuTbSmOqt+QtLjsXudFb3JSV8c/lTgaE6MCUvQNOidrnSj77fkcNx58IVNy7guCFfMBwAEMbpOQfXT94woXXlvPzloMk68LV3aplQAHPuV/Gm3krY5DoKm7caDWMbw/DUUYHPTmPOzUkEyArcqmKBdrvZDbXoIZxVvSwqLfkiBPKe96cutqmWfyGhmJvepYRz1X7yZgnJufGqmZHOryPvv/xMAoOP4HA+5AQdj/c9IesNGwoRG90xLaLNUoYxVlx0HABhzhUiOUwLDGemmTpoyldoBdhC25BHclooV1K4jE9jKJY9mr6NIx9uS45SrgVHjt+R8Y9qsOMT0LyNaRVFPRW9OEo44FS/csZFPFNvsmC0mpY0ha44s82pjEC/dQ9rxjT8Q66eFIPEN0wgbdKUrYcbr9qX+p8k8KZ/ruIzd9h1a53QuU/sEJ3I6U2svfAhq9gGchNXUlNrsuB8pu9bmPUmZdandmg1KeylUrJZc0mAqfgQlFKo5g1zG6JG5AJS/rGXax0eCkbZMrtR87MpCzz7c2FPsSGpHhkXaHN5CSyha4vDG9wMnXp0s3zqaEKlCOd0MUJBb0/ja4nVmUeU1HB4eHh4eHh4eHh4eGw6jo+GI23I3gV5LduxGSvbMLVBzk5Sme153R/rzzYLsiHTukRvp/5k7Aw/SsTrmy+ZZylth06vT028FVyKmS0nnUADQQcmRMPI1l26S6S5VuQMv33MfFfU9SfviLFI3sg2On7SSid4H/gwA6HzbsElZpFFtxmzWm8ENpiZplXvGpKYR/Y4pgb0cOkoj7ehfTr/i8BvWUR6fR0eVvS4C0Ww0cyQPrz4aAFA4K4fds4MT/rwYWpxnxW7dCWipf0f29erka0eUfgKuc6NIO1kqp1951KGttDa8s7/51+bpxQOZhQ3o52+nZ8W+utIN/d3T6Pg93D9FMlooYcqX78xZhbopW8v7YnSj5OTIkl9+5yh3YtLmOX1IEvm0DvzVNn1W/kTF4b5YjdigAzBBThEUrXZAhqaN9zf0zkYC2z0N3Z/6DScR84kDrB9Ig23flz8LTOVgjy5BhbwzoW99iHxesPM7UXr3t/PXMYPmVGtttOFapNmFSrrv00Bf/jI4GJYson9NvgRbSbtFg10ftFSmpj9Wk/TAhTKm7bQNlzNF+iskHpq/a98yE+TSoFEzY6guRP0gVVBA26zslN0GEb+92DguWgfnPehV5H+kZu1r27Oh6O3A8gVUvricnoIySvA7IXKoAIrfnbSl2lporqMJYOjQ/fYuoWuZtTjOeBl30tYvknZbzT4AWrQoQtmugtQ2WtivxWQcC1Ssw0aKb0jSf8md09Tux2WqVhQjDDJ6wlWJOQRQaNxHPhzB7u8dUbrAMPPzg7+m3z0+OOL0RxWvMxXH6Gw4VLo5BgDLUQ9NjkaAYX159ZsXYuI3uBOuJdMS1TUFGEhXXyulLE+1dLyBVcZER+1wIp2r9kDvKSwhPLAxfzZ6ZlimgxFAN2pGRayrY6NlgYausTMcLwxU1xToZzlehCyWJ83B9A9/fsRlAIZf6LSfOoL0ZbHiOhVLfjIwFasp5l3aqs3lndQH0fYG3kzKwN2wcSVMesZEwzEbS5v4ZcEhcRjq/SjufnSe2jkbmbz830FC5a21RnDmz/Lln0hXmbZ7w4EUQfik24QFpA71ppTo18PAXZAk2kfDYS4zjG6sKu+Zaa8578dENE7ZNNr2wgv5oT4Y0oS1zNJVrOAz19Dm+qIzU0wiY+w7w0IWyC7hBCO6GBNHcp4MCzD17//SGwEAbZ+6A9XjP9E0q7wbVPOcRLkuVoGDRjDxGlNJZVl/eOzQ99HiXh1yQdJsNSjadmq+pwaeIrMxtReZu1myB0D/m50wd3gr/U7fLbW+8cjV2OU0vlACFjPb38w9s9exbMfnxHc0DIOO2ahsQh6+HorzjrznvZvH4omTNKRuLrQ2cY/MfW5snj3ySG7gMEyNS7KpORteA7No07TpA6Cqlk2tOJk2rqkRpuPjWkeKGWqhbAlXZBwwggIA07fIWjOb7aK/U7Yz94BeTU7FSszAZCOlArP5UJJ/fcDOL+LwvnYxZu63d7RuUk4d2jmhnhL7qCTHitoC4LxjFuSsXoAJe++fr4LCfhkUDXmJwXJihNJTV0K1s6O8shs+LXO9s2mPR1yPQL7FgJifOW3DiaGiB1jwJgyBElNHFaDm5myjgDOvtt6GJQQ9DkujfpkizquZe6Cw+c75yzAMInnP3Ga9p++x/uBNqjw8PDw8PDw8PDw8NhiUbumBs36ge0ma2UolrXVoJR3iHFwfMBIJoxHomIgnjiOpx1a/WJKSUizdsAEjOXIcy+KRlMNFxLcfTJ1r0/9jNC5IS/S9avMUSb0x63DyZ02NcINjqA+osqOTPFfpQu1LB9HheXdkL4ODNEm2K5ENf/YhKuI7f5w5zfAZigweTNvJ5pNiwpIm+Y3fp2u90N8lT1z1np8mytkyKms8CraTvpHiDa6G/gFJj4Oz/5ytgs43XB8YqQS8FS7ZiaSAFz20yKbbxHE+tUxstoCi4yBpiAmE+tQZEtwotuZY+NKLCO+n2BVqt3dHMwrr1qTKSBX7gEYs/o0KKKI1ADV5Ozon37VQgv7b/1ExDvp0tgqu52/4/NtIwzL7RwvXa7qCvot3AwC0f+m5zM+YsXA9tavr9qM2dfpfFifzWg9t+KnjaTzd4vfZx9MsdXTHieHGu9rl5MxaPifjWJAT4f+RBij40C2Z7tdr2HzYceDO8q4jdXbMLZd8gDRMk67O3o6AqJbGmKXJ2MAxQFT3dPTOo7hRHV9+PnPa4dN/pOen7eSMKeLAzvNAfdBx3LbzpWIJvdG29S/HorMpsvXU7yTLEImJIvnECVfCITM/iAO3mav7lmHpuURjPun72SKOayYNUZXOpJZKNPpaO7Tgsg6oW+2VcTwPEP6WiDeCI7+QzKul2TFanLOa9/DmSyn9Y6/MVD8A6zSexpeW+tsnAHsQO4ba7iQu1/qbG3VjyJjsBtuPQJvzGmDlmeNf0/zHfCsfmca6wms4PDw8PDw8PDw8PDw2GEbHh0McCQuVFImUSB8cu+E4nStgHCB12MCLvZTGlml24wknY23TEV8BrROO7Goi+Y/oxhD6bVFyQ2sdoRd1y6K1xiPvIMfbub9gn5Fyu30/f+KovkddhvJHfzey/B1NQkJ55djyqqMvy522iRbbqFnbTrGZl3fcPt4EWtJiH+/4IhgpmtZQ7yTptVAVaicwox5iSXjnZCk8zDcVx7w1i+x1yV/SVwWot+dzVm0VcC8iCZRgg+IUWetNUIVqraEfoPolpP9oohmSAJnynUptjk0yvcuLHuR2M7AS4TMkTQ12/0D2Soq2Iaw7JAAieXPai6ElJQmMXnA/1HSSMFob7yIwZevoOfPNAyc98ZcoWcds1nqiczIw1Bu9T6LLl8ZDT5uTvW4bAMVSK63sukv/R6ZgTiFSkCuuX0H8Wthw/A+slnCLrhaaB+mbZkzLJqNyI423dWSjKo6lwL/N320kEnaa74L40pU7UexZ/5LEiKa1mo8w1gaodAKrOZThAIDGUIJKNVJnh0hiwtydzHEsIzu3yvy34AGojfay18HjkWiHhTqWI8/rRg1tG2+Zr36A8UnUK+ZDjWFyhCEb8A8AVM+MFOpjbXw7xckb/Ssweaft6Vi0EkZz0LB+PY7mNkEcUShZp25Jw/FXmbBnjuCbgInCrcvtpL0FHF8H7ieFgjPmcp4DfYAaMGUCQH5uXVFNdavxJX3M0Y57lxNEF4AOSsD4/AQV7jotrkUabtxLaBrf8xM0vs5EKltyVHf2RdFDfeY95QmuG0m/UIJ+jDWM/yUajtcbRmfDkViAJxfjKKQ0Msdkxo1yee9SuvfQ1EjasU7gsp24LFml9thtbLrVqOGJ1ZTGjk0rlITlhrdliEerhdaYe5OYLXA5G4O2ox3JjtwqwOrzaQLp+cqTLfJKRos2POBam3w1olFVtdbo+yw5yHVcPj97JXlwUIWyeZeaJ6WI+RRPNEZtHtbtYFW0EW6Xf5gcEcdf+QydkvQd8z0AACAASURBVO9Tagd4jE7jHzeLn66pSVWz810HPkdOhm2ffypb/VpxnDsLONlomP+bONQ17iFTsSBlw5HGoqRKKTzw8XOy56qOAR75A/2TY8Nh3qPb30ysDadMcRadGbvbb+zc96szSEX+5j/H4oWoIlCO1dFN03FCV5sd1rS8L3ySovjOvqnW9J4NiWK8Dg7WhzlAozcnuxFaT8ipkbnNRWe8cMbC53vp/N5pz7SI6tuyjE67XbaSvt3MZjenPZ9z0WGfc+YYp5/WltMGt3UkhJx5ufNJb06ikUIrlirLPpVkDnMWfo7Q4qW7iS1uo/cmqAmhlV2Q0U172/HSRJLWZkww6Yu5T1DEqw8SscPEHFUUAZUqVuxcKLFhxEHcmacsGYUyTs9iWoWxwKX/9yAA4OKzou/OzG8uGrWo4zhiJkey0JexrNSO73ybYuK89/0Z69fhvA3XiRuAlrnSbcci4OmMlkvQe8sPAAAd+32Mn81mWmfKkxap22mjg3/6DgCguu/Hm6bbDNG0hxcGuOWLrPVCboux90VkM+vW57XWwNIXR5TGa4XXGUmVN6ny8PDw8PDw8PDw8NhwGCVaXJaY9C516NpYvcixJ9A+AXrpvwEAATuPLjhjI0y7hqXTYu5S7sC82/9G6YkJikhuwkYyQm3YiHLtA0Cly8ZtMA6qbFpVrODkmx7KX8e1EiV9it34G3Utl085jmZdU2y+rlMbAAQFdF/+BNUxpg5Hba2JyGqo/QpFc59eSFIgNXZTqqfzrKipVbGK9kseyF9HUUeHDUMJqB+6nq7t/A76LbYhQSurAqvydiQuY8//KafHplSsbteLHoKaviunx5onV6KTIgnRfez8JO9GKVTOuCpf/dh58a9v2RR7/HZ4QoI0uBqZ4od/T+disTTWBdppy+qU7+R/XlTsz9wMtfnhcpb+vkKRoNW4Tc39psxDfVZ7VBdzmXYc+6cXqDj30LtWEjVeh9BLHqNz4gzeu8RSPxte+aqJK2H6sZiRVTqx8fdzOsG2MovLGKPDxZQrH6ZnpQ8tY23csqeANexQuRE5fqu2cY4zKLfVko0WbTR+jlau6zP3ZS5LK8RNVPNIC8eWJYZBiokqf+u4RjhreZRS6Civm8nZSDVJblsIOIq0OKOjfTyXL8hsIpJWNvPclE2GuTsGY9LrOI27pooAUB9I0Xo6GhynHc/4n7sjZTJS6FqfpRhmB+CX3z8XM657IZZsYEwZQ6aIVz1sQqtDjP9sfmd7/QTHH9rqSEDoXMUMbA3n0TUV0Dz3Bo6WPG4aFg7hoqso7pBQw5qYPrP3t/TQJr5Gu2NhkBK3yZzjPAdW4d1X5KMZbzXW2Bgdrc2h3DYUFC19OIAoFW2M2jhrW3XTL4/LTi6SVtb18VzhAz+ncsXCF3x9nxk4655F65S3Ugo47fs5S/raYn06zf83wGs4PDw8PDw8PDw8PDw2GEZHwyGojrHSaZGWS+TPjkk28B9j6uW3W+2AY/Na+yZJ08vncsA8I1GvQ//tOgBAcMC5lH6gUn1ItHFUS9ojhj8kKW3hnHsyV02/RDa0aqujAOVoXABbZ1WAEidn8WXQYcRmFwBJfBMRZismDUs3aoNBIZSIrPxc1xSrYRkkyb0J0vbMLVDitJUHrvO9+J3scmr0Hu04/8OhUY1rOLQGHmMfhIkcrEfse2fuaetonBZCKy0yjnBFK0mSwJBit61D4CmWym12aKbqadZS7fHbJQi/S21MnRqVmLjBJSM23OY6a/Pqg8YZWnFbC1eSVDEQB8oRQPLUesBSLxrH+iwJ8DecuI2VZDdi7UmppBapPmD8SYTSE2M3gV5CWklsdRg/KtJxWC2ioDrGSm8dJ2TtBOYCAAyyFLrSZYOcieRyWFjJaEKaGZcgZ4HYkrukCAB0/wSs+PpHAADjvvg3ydA+Z/JwfLrSJKIxf6AsaO1MmtGp20ljZa2JlE2HZvzNrwWw42q5mn+acYMPNgvEqRu11D5oHnMikhdOJP84M05v+Sa+ZunYRdsQ3vFlyPzRCpF3sefpw94fgRCFBEUk6KhFc53i00UBNXn8MVGn61CVWPxtM94WbXrcNmZ8/WH7biQAYaXHSPsHv3kaAKD6aZr/9IrnodrG5KsfYP0agpINNih1vOtb9HvEpQ7hhNPW4nUcXA1MZ6dxPqe2OILrpdB3Fb3/jvNv5jyLdixK6Xcy5pn0C2VghqV7zwK3fSX8KfKMMYy2T1LZL9ljMwDAvAcpeGCaP0rmMro+TadfP6I0UlId+aPiE+a8OwBYU4PTDkZCMiHpj+6S1iMf/Nfx8PDw8PDw8PDwGE28viyqRmvDIew2jqRKpBRd0+l/HUJ/+0Q6PpPs/PDM7cCu76LjhtCDVlE6/gJONibNC0rAnKOT2TMFn5GQd01JUuvJ7rpYgTrsQ3kqFyuDI1lhbUNEsxKTWtFOPybBc5hxdFrwogJLjkRyVaxAr2Wp83IOijRhK7vbF+mI2MaX20dmOyj23NUxsD0lxlwU1qFfJHtitdE+XPYQhi1MyqBDYKO5dCz+M/LeVAEoxpqmW94+lpy3WYm5Gr8Z5y+SUQ3M2CFX9VzfheA0kgZ9eTfytTn3bzYoWiupqrmnWCGqQwei2VgXu3SDQmVkkiDxJ+iaCs2aL4iGYQH5K2DMrAhNNUA+MqI9Uj0b0a9SwNiN6Qb2XTC+OkEJgz+iYH1tIinVDQBcZqOlct6DtPWuqTb9rmkjqp95HjAScj0CjiJ93dspiTN+RCdYW6Vm7IZxX2JfL5FQhw2gWo3kqZSCrnG/EVt6R1ugb3gfHb/rF5nLtD7sqt3j9kJzat6WrFet4NjND/YN5X7cSkCbU/Zm6YcA11UovX9Ndvp6GWkb1V5Jhje130dzlRWA1VxnhaEAVWZs1KLJkXlyYGUiqKdSKsEgRxToseCdokEPh6yfCD934xtm4uTbYwFonTJVzhS6ctYCdE83VOe5MJ/8CTH7AMMCaRTWu1O/0n+4GOqYLyWflbnAGU/w9F10PJU0Ea5PXPu5v6ID0fSEdWjxDXEDxMYo9w09bjgEzGd/qqwacXnnYcNqZyJslVThLP1V6xB6LfkNvms7/v4yV9b7HbrxTEVLSV9baxIZszM+B6w/XwP9+E10sM1xkXTP+/Tbkv63I6isfpqsJtSOp61LMT02EEZXwxHh+efFjiyagwJwxDmR+1b84DKM2+W0yDmtNS2IALsoksVuY9DGhTAxHYZsA3ZjEMh1edbtUGNzOgCCzYAAYHANtBlco/SCAKyqW5y8K532nAx+5U7HgZzLJ4NbudO+R9mEldqg2Flam2jQDSComDLRszRAq+m7m4E8V5fmQY8cLbk8xsHXRodWsw809wE8aUi5jGpYA3cTDaCoxs1gXetNOLxHNpft/D0DZzEkjsfitAsN3MuLxKyc3Ma0wuYlG41lZ9OGZtxXH8+0CNNhw7bxOI2uUutuXlXvB8QUqmNS9ufkfQ6udgZ2nlRm78//atv+pMxdU237FMKAoAf6jq/S9QO574rTPjSqH7mJk3O/IbfdwFmQa1mFcH0k9oNqg36ATCTVGz+frX5B82+T6twppdUhpDe4k2sgGw1Jw6V4jH1XXetNNaV45EQaT3Z465vpyZO+YdII3npN0zI1Lat8h/qgXZgtfoRS3MiS2yacwBtDCfNSrTWO+RxFIBbiC6EK1T8/F3g7ERO0MhFJjfjt0HNvdsnV+evomhfxWJB38RPZ2N3LJjyHnU3nZu1H/4f5N0OpeT1KbV3t/4lsD/D4q0uO8EfavThAO/064gwuY7fjyK9XELmCat81mo8K7FzH7ebE636SEqPJETA8z6bEMg+qAHoVb9AmbZOtfgCw6zvM8+GrRO9uhBWyQTvkPIAd+YVOnRzYeU50TcX62PyLN07hcq5zzwyoKsf8WPY0nXPGVVPX+oDjpC/jEP9farckEHnhzk2OYJQKWbdzMtI3/FScEFhOhBTTL2Nne4nz8eJ9UFscznWRTY6Nm9JqXHPz0ytZGDktu+lYs3gffJWLXs9sQqZ4M5cwQTv6i+Y9/v4wEjgd8fsXE8JIYBiT0o0PyFSO/xSo4PXlRv36qq2Hh4eHh4eHh4eHx6hidDUcSlmpt0iiRUIeNhxJAe1gx554duRZ+lE2+iibwBjpWrFKkljA7P7Rt8hS8a2cT5fGbAwtEpcpbHbjSin6l+Wumh5gytnOSYhHbI04q4nkw6FPNeccqYgJRGiob1llu3ohqbgBR21et5qQPlZ9Dw1YB3IdVfPqpY8jkKjRecCaEl2sWhM0kTJEHHJZG2XMrUoAvx9jDjDUD2y6M91nzCZYclJqS2p44HxnI/0q2Wd01AQI9QFgs5i0b1g0Ny0xwQmzQgVW6jg1GUJSNBurPk4RfNMCPLZMvtSOcBW34XE5NHLSzitdVvIpkb5FmhgUoH89j46PvRwAoJc8CiXO/W7Qsl3JNMK2NUcCZ0wKnQi1NhQu3163Y4AJSmilWnrOMdnrti7QOqrlHAnSAjcCmD6Vxrq+f90PAOg8eV3lPE4/keBs04maV0ccjmP1cYNnOoQWtdt/CACoiDZZtCBv+xb0Y2zqNecEfrAxjDRTpMn8TQtl6Du/R8e7vS9L5Uw+ACLO0MptM1IfkXa2cOAFAOzCpnExjaIqlNF3Mb279ovZ+b9/BTSTA8QpjZtBbXFIllpZcFtJLavk5WjQXM2aTimL9M04aYBGwc5HTLuqpu5k36Urc+Qx/NL3XQgAmPfgaTb9GKFLFijJtzEENX5zOhYLhEWkkcPErawmx9DHK2vmaVMDxjCVPI//5lvWeg3hh7w7vexpqCnsZO6aqpm5WUhJGDpMRPoeFmb87Lffyszhzpw7FKOW1qEth0sYIOajj7BJ+WE0BqtZ+wBMaW9M7LQGlENTD0T6d7xdaa3zacJbIEFMkSdgX5MxUjlR4I/4E33LS3aehnkPLUzJv8U4nWg3Hv9J8E7jHh4eHh4eHh4eHqOJ11kcjlHacFibv4TTFjujqXHjrU257P63eRPgSsnBO/U/s03wO66VZG36QqcnebaNM3aVRkoT1q2k3Uj/HTvIuzldExgtA1Yxhd3Ere25tJ2/SCRdu8u4lM61QzdBDdmBz5FqQigNXcmFBFjqmAijaXj+Tnp2zvH06zhH50JgbUY1HGkUEJVKG+2H1FFbe1P57kHR6Wxxe9aGkaLrpY9T1pO2cWhOHemUke7EbNNVIeozlAFZnFBvesNkHHvrK8OnpVSqZiMO0WyESx9H4LadDAgmbJnrfiqY1SKYbyFtzGiaFNSRHARLpN2T5jj+MXyXUui/iuy02y66j5PnNqyU1WK4fV6+uZOX8XXpmGjPMQav5vQ//1S26o3UobqJHbQbyHFYhA3j/2OkjmEda1ZSIK8JuyQ1bib9PIV1KIUNhNL1+jPoltOSFJgRTStLtpVSqJxNElUVc0ZGUMAlp34YAHDxw0zooVKk6xFndK6/4xQdnP7dTNWKJmqdxuPvPqJtiCtxmjjG43efod/jv0K/okEuVNB2wV/oWMat9vG2HnFJeBPou79JeZ7wrUz3m+fChiPl5j5xz7fp98jPJ30tGjUnaKAzXvE8qmN91DwD2EBybkA548OiIPPFRR85jMvjEJXIXJMHQoXbqNu0ZK7b/I30WyghvJl8iNQBH6d76v1QMWd5HRShTHBZmbv5fQ31QYl2QDRj1R7UryTq46DMc9MZP7C0rDGaWV2oQM09KV/9pB0679y0VccPUAf1SNki6wL5llobv5bG00TeUZDP0LvUWjUIFXqpPXV90Wz8U0o5PnvZkSVYatrYmNZvAVifXdYIRfwxdCNyz7y//AP1K8m/s3DSF+m+yds5WseUMVvS9/iPxOhsOKTRRtgi6DeYsAX/rwE2edLinNw5JdGotA6BNxCLVFqDizvh6mIVmqMg1+77LQCg8sEboHixFu9IWmtgn3flqx9ArFCCMDrA2Gij2kYbNgxT1dRdromibpyz5X/txOZw2Kdkgt/7g3Tb4n9Y8zJRLfeyU1y126Sfa6FjnDcdlXDB2VRIihJZm50Adb1mn+13TKue+zsd7yuPumnyQDRxK87TcUytOM668u6EAUjYVErtwIJ/56ldJqfUY299JdN9WmvU/4c2rKWP/HHYvIOJW7eOXJuC8AfUToP3/irT/UD65GAWV47TdN8XDgZgzUzCV580Znja2SxUdzk4tcxKBUkWrWIl8iw/GIn6HEdl37dkrptbtjyRtgFa+KVGJ//uOym9038wfCJKWYdusxjqxewrb6PLPOZEnJmvPYWOP3RL9sJyHjooJticZKOh3XFCvu9L9yGYfQCXj8cmrYzpn3H4l/fQGDKxAMx3619uTDn1g9+l5Lc/3pJQSLwUiTOhFPSypyL1z4bhI6aTiV508ZHmwK6UArYkZ3pVTcaTMPONPLd6gTGp0i8Q4x42OTjdTEWem7JFjro59VKAsBSa9N7I5oy9S80i1FwrVoAUtjV9A20MA2F4dK/FNiF65QuWqciYPdXMWKreygK9QR5Lq2Ohf8zO9h+7K2sVjSmRzAOUVnekProxBOx0MpeF30nRmt2YyPBtY7HiU3sBAMZ+jduTfI+uKUmChN6lKHyI5nvDiLfkMWPaJQySmsk7VPd0rL2UyE66roxGvG6G6DdsgVj7iqThbHz6PkekHbIBNmuT8Zu3bOdZUfvGWwEAlXn/zP5QTqHdsMk9SOaVah9uT05fNWsjMTes9qB49u/o3D9vpAQmbtWSGMSQjByWL2r8a4bXmYbDO417eHh4eHh4eHh4eGwwjI6Gw6G0VTGTCuO0FtaBGpkeqB6WkvUts3ErXFpAoeVsIc10eclvvewKAMAhv3yMzlXH2PzddCm1EVElKtdZzEg+rGYjASeqs3WqdaS/cl0k9iL1CQrWjEic0FyncRG3rHwBWuImiJTsGZKiqs0PNRGUc0FoWAuV9DqZ+1gCK6p8ra2UdRFpNdTMPYA5LB1PkUja4+ZmaQiHHLUyp9FHDv+quwJsunuu6mVFVslSOFTLl25MWj2slL60DhFZHSQicqsCqm/+KJ+ib/7iJ47GrB8sSD68xb7Jc5JuqsaGpaiBY3rUJJI0AGDWCMgNmpVnBJzyjZUkYc0imUkzO9AqgGrjdiDmEE7fq3P6ub6kvE+3faTRR8uxmCosfwF645R+K5qNuGld3zITE8U1/TBS4Z1J+6MXPmQpVOVZU8bCiEw5smioKIZFXLOWblIlZU2Fjplldk/HxTtRvS9+uLm0O5L+jqcMW95Ilq36uMwltbWJ+Ym4DVLGS6FZT2njifsrXTGqatD3qkukZ6GZF3MyjUY/zc25JJSiRUijnJY5VgUmvo+hgV2z2JomO6QBnbNmc3FiawfAatSkDTeGbL2lX9b7oV/mSPNMn6o6rTly24wRUpQPg6zjTXUG109FNV4ArLZSCHCWPAoIiUcLZ3EXpWn56f5HLoFv8txguslTpL1GNOPcbrYjk0790r1Q09k0NS36+uqluUvqMXrwTuMeHh4eHh4eHh4eo4i8pr//7RidDQdLHSK72Li9dKEEbLRX9JwKUu0kF11KgdymfeeF6O0pzosolHHILx6lY9ESNIagX7oXABBscrA8YZ5d9dXTAABjvpmDHtelahTpjQkkx1KXWi/gBjdixB2ztBvlWRz8xOdDVRzJpXWaM+mK7fNmh1pHwkduoHNz2Va21gdUokHLMkG5UqoWkg92/jXfw5FYqE0OojLVerH6R18AAIxhOs5U59NWQX5SnLyFIlYP9WHgN18HALTv+/GW1bJ55pPm/PoQ0hwdfUvSiVwphfI5f86Vnn2W6h4uexqB0Emm3Xdi/oBq6fkJRabz/rc+mg+on876v8cj1JyCRZcR1en0a19MSTlFmi721PJ/UKD2C0SdWRmrvvZuAMDYa7L5crQawFt932Z+M6ueJTrknISZNt1CCeiaknpNa421L9IYlksHYEgm3PGUxw6XnjoWNEvP3idBh6pUgNq17wcAVM67QxKj354Z1ifFjKclBBz8TcapYPYBCJ+9lW4bsxFfs+Oh/ul5dPzxu7NXkfNNC7KZrglt3XfDb1DQxcJZv0lcS/rxhUazkVXbaHxxzs7Y5+U7RaS0sf7i+DK4WnDLLG1pgvvmUzvtatXGjQ9cd5I+NygBgcxXPK4KicPgKtSWk+Q4V9x58etxyAjipDE0vnCZOXgf+pZFyT8AQIco7nQIPxr369ROIFF+T+3jET5G/iyvfI3G/ynXPG/HGnlPRlteR7D9QXlq1xIj8SUbfOVlAEBbwSHIAbfzuI9cz8yIP4fc17JMK4cnO4kja0C/ZGYNpBFMYNbO+fI334m/76x90f9ZWie2XXBv8oGt19839Fj/eM01HK4ZUfj99wKAYTXRL9wNtQ1NFK55xpRLftYivZjzmNY2qjeb86BYgRofdWB0F/3d5/w4fz14IFO6YZ2ajHkBv+ZSmx1wJfp3pdtw1uuCOIjXkkwcshgL63ZCEHafSrflNxezjbBhTdOms9N4wUnTvPcckDgcpXZETNDQ3JQB4G8gZXWi43Zusrm9Tkf8/DAqYsf0x5wyTvgyuSlUZ+Zj49L8HRJsPfH7uLyy0dB9rwJt46M3DfVB/+JjlN7J+ZhrBMH4zVuWSd92GV076isjSj+ZoMRP0bhkD4qsPu8+ityr77kaOOiTkduVUpj2neebp5dmKhVf2Nf6gGJzg6IxVzyRoeAWhqUuZTPaahHbDN2zZ+fKP5Gna2IZ73N9y9A2aQTc+DK21QecPs1j3K20iVcHnRtZsAKAGruJ7WtM3qDbxqL8UXbMdIks6Az0v35Cz257HJ1Z8KCNVFwWc5cCFAtuZBzVCx6k56buiOBdP8xfxxaIOJqa49YLPHXYB7Nn4LCNmc3/NccjeO/PTL7xcmD/0/PWovm5iFmaNaWKP+cu3lUpuUiV/xNR4FPhxKGJt1NVRFjLZx4KwJo5RUypeVMjsZSKFegazSuqjU0Nn70TeuzsSBoIh4AFT0bqETHPZnNsXek0zwkT1uSLmOHJjTRu4gU5zFFLowLM4dByoW/MyOum/6ebfts0Ku/7buScuceJVSQbsL4vHIL2T/4pmlfb2NYCl7f9b5ZqNS1fHjTbqNR/TXGdyh87NHdZBLLR0LF2qlQA3MJCuDk5GcdeK3incQ8PDw8PDw8PDw8Pj/WDUXIaFycvIC7ZkTgLqn0igjOIMk3Mffpv+BzaLz6Kzq2h6JO6ezriTlWR3bc4lEs+Q73WxEfMjgplk6+RdAhda8fEiLlTZhgn8IKjpo1J3Rz1uVAVKqUoZgGsVFbX1kIV6LoecnnSAb38WYqJIHkBlF8sqjOKVatZeYipEt+8D6Wx4H7HlCwHmNIQumEcw0195D06EY4jUq14JHAVIJjLUg6RYhlpXmhFeiKRCoqO9MZKgEy+ojGSOod1YNsD8tUv43ePS3tU+wRcsw+ZzbznLnYyDQpQJ30jX/5peXG7SuNDVwedlzs9Iz2r9RoqYYM1XPaOCbjoN7+mY7nnmQeBAy0RA8BmGCLRTjMR4ejyWpykB9cCIoEUJ8igAP3MzXS8xRGR4igVJCMCDwcnwnUCjjS8lbTR1bQVj/lEvvxT8jQSZpHCPsF0nWuWovLmc/KnKW2hWLVllvc0mxwqVak98Vik3Qrtq9bQ/4+97wy3pKjWfqt3OPukOZMTMzDMkDNDkDjkHAThmpAgKIIgigEVQUCSehW9XDOKGMCLIuIFRZAoIskBiYLCkCYxOZx89u76fqxaVdXdtXt37zlzhvtR7/PM7D4dVlV1V+he4V2KjlxwoK7uI7349flkpXvfXcfROZse4HQ/M64Pqn9OM4QNcg25isSzfKciQqAQnU9c1gw7b0ZeNxBNEc4EJUN9et5hq494/7UIlxDNNt8nJqhA6xiLtCMj2BJdnGD28RxpWb+d/ZQtUZalrm2nfeoUJBMEHHH3HD7G9MDoU+1i+uPlL6F9twMaNMhRsqK0Fa1jIVWwsOxbQfuU9V0ERYBzTLDbzOwP6XVZcJD0mgXAVOVeyvPPAFnN5bKXICYpK75+JmP1eaJrYyXfZKbnuYkJDeTCvwMT81kzm3HRTJMRd5/Va1t1QM9n8haiky2N6oq4iCtpqRbxgKmQ80DP6+lW/zTY72nF3Y9eJxlRqPngyZ/Tn7NPATYbPpIRj+HHBnep8vDw8PDw8PDw8HhH4R3mUiVkumPnsED2LKHCUpM4hSZREGs1BrohlEY0XEzJaoLJO+CNU0g7svHP5zcuO9I8o+mJ+3OHb1Km5GD6nlj00RkAgCk3djeUz9BBk5N3NDt7VaI9pp2UodEUsFYiKFoaS6XNHlhrNMWaRlBpv3qWau2k1qrVBo02ia8LCoaeVgWx62ylgz06JkR0uANanW18jPwjA0VRB7itTPF760zQVe2H/OVHSZ4jK3IatNbRCjKOaz1lbQjyRiX/rDuyyeV7uA6TwNW70v38whOLgAzJy9YFus3t2eMA5GoV3F2sWI7hMS25CIymkGXXBi2rAfuOFyD/eBFtHnk5V0r9hkZjy5rTah+0hVMnpSsYba9lgQRA1o+7SW5wfDb/4+F4hjaWn0exLOOufXlY5NmQUmLlpyhh3NjrVma/bpjb+Mv9qc9+6IHFybKaiHuJ418nUh/a4o/Z48aGu43V75BmtXhu47nAlVSNYnHY0hKlMRciQHgdzYnBJ+/LVB+5gvqTnYgwy72OxGQoDT/K7ej5Eq07HVdFk7o1svhEA/BjSTl5/Sq3o+/K/QEAbV+fl94wC6GyXIrJOxiZTKhixxNyIDfXMygYaz+vk4PdmH/W9gCAaTe8ybU3MlzxYpwsV1tTy4bARluOa7qtSz6zBwBg0i/WZmqf7KYYvqzWjEYI5xHhAFP2wrboay8Auo9D3z4SpU+rGA79LlGIej1Y7YAI5AAAIABJREFUEEIgVEksg62Py1yn4RyHMqzq5MOiDpFGLnmxV9f7jpiMA3/zNMmftN06yx8JdJ8/bYOW3/Gtxu/Qwwlv4fDw8PDw8PDw8PAYQQyXQuX/Ckbmg4N9l8ttjiR3CmEt6Qdr+Q1qv9naIF5bRl/702OJxeiLN8qeRNoPZokxLDwmpkBpPJRlQlYHsHYl1XdKjiaK8aSpjHSg9gmRc+wv8oiPJdffotHV1oG49aN1jPY1thlnxKipAIDLdiM/6UvmLjQ+rCzX0j43Q3cnZilK2+qA0aoo9inJTEOldsiVSgs2egb9VvsBldyJ2bwAQJz0I9rHPss67saco++hTCaIk2sXASqpod6naQ8DiA/+IF8D2TcYBQfTlqXNVJq7YLNDTf2VteULj6i296/UGjBZomOBpgytaa0+xxWRFchiTEFjy4hcTZq+PBYO7ffbt8IkoIuzfokA8smbaHMO+fDXvn8iCufcRseZ7rKlE3JvYpbjMa7Z1EQAyJjvfbEC/Yy5/wlhWKq4/NDS9O5xRva2Yfgn8EJh/fFqCCFQHag1PjGB2BxnH3EmW3RIsDTbu4wNE/v038w0xBcGRTcLXbxuuj+0odzSzDJTv42ZJVjtCXY+PPN1zj4khGE7VHTE8kVFsTvzQGDz3fJVLnBYMZjaWM/TtcQ8HakbW6mFQHn8ZHUJ0yIbKnZpJc8DoKzkUaLnSBI9vvccByQKCMpN+PBzuX0rTXnKi0HHqbSOhYjVT0qpYxE0O9PqN7HRu09UMnqj9asOmLWNY86k1HFMUseVCT3Xyf5VtIvZHZ//X0w4OiezEa8XIcycyN4ZzbA7sScEx8Vx7NOYTc2aoCw/xdOvM/NlQZUxuNZ4Naj2sQeDLLUa+U1AhrVEYloDkamdIigi/CtR1YsjLo/KrxNXZ47b1vjYuqL60gE/vx+S5Z+w7vGTHoSrrroKTz/9NIQQuPDCC7HDDjvoY7/+9a9xyy23IAgCbLXVVrjkkktS+8LIfHCol2VXkJ+GY2G3TcvmBbyG+xfTuXPi3NSu/BD2S6q9HctSqSenaj+eXE4ytqjTHCfYBBwZjDEzMpBcVKoDFgWucUHpuZioJ9u+Mjdyvii3Wxz3qj3Wh80lcxeasnTgtXqh47pVRqP/6jm0efkL2duoXlZFsUXXWVqLnsbYzVT11DMLCuZD07rvaz5LH5Fd33wpWg4HFsN+gTJdVXJ/Kk9JugFY8ge+RoGOla88n619HLToyvlh9Vf+0NDH7AVTL3KtqP2Q8sWUPvWn6PlBwcqbwvIFnLzldSClBG6/lP74WDK3QD3otrXZNL6OINL9orlLCp+4w1xrjcvfn7A7AOD4ex0c7yL+smSNcdttJP7xax177WySv+ltTdByDgP6e9ZvuS3t+V2VUqkvUz40bHccu48/uYLkbR0nQxBCUWCnvzhFj6ltawyv6s4Z+I/hcUO06zX0t1upWnvnoMeN1ye2Romt36235b8eo40o70F9pAT12/N7Yn6ToXnGVob0t56meXx6nCyh0pX4CJXtExMB+IDU64U+j+fZoIi1C2hdceR2rgsO1hbFFrPecVZxDnyv9utyIuszt4NzyUzaHpdfQfmkvvyeb0fKkaU24z7FqA0CAdNC2/lqVNs4D5X6FXuchevnkIrxjPf9OFv77HudyCGU/4O59/ukvGm//EkAQDDW0LrL2AeqGDMjkaMClnueiCk7AWDg+nPotCv+mblOWl4dooF6cLklAoCc9496JaSXb+WASkQBMAHPhK0hFw6/6+s7GY8//jhef/113HzzzXjllVdw4YUX4uabbwYA9PX14Q9/+ANuvPFGlEolnHLKKXjqqacwe/bsuvK8S5WHh4eHh4eHh4fHSOJtnmn8kUcewcEHHwwAmDVrFlavXo3u7m50dHSgtbUVP/sZMcv29fWhu7sbEyYkP3RtjEjQuIeHh4eHh4eHh4cHoeczOajC1wPav5me7PLiiy/Gfvvtpz86PvjBD+LKK6/EplZC3B/96Ef4+c9/jlNOOQVnnnlmqry39+eVh4eHh4eHh4eHh8cGhcs+ceaZZ+Kee+7BQw89hLlz56Ze7z84PDw8PDw8PDw8PEYQIhAb9F8jTJw4EcuWLdN/L1myRLtNrVq1Ck888QQAoFKpYM6cOXjyySdT5fkPDg8PDw8PDw8PDw8Pjb333ht33UX5Xp5//nlMnDgRHR0UpF+tVvGFL3wBPT1EAPHss89GXK1c8EHjHh4eHh4eHh4eHiOJt3kejtmzZ2PbbbfF+9//fgghcMkll+DWW29FZ2cnDjnkEJxzzjk45ZRTUCwWseWWW+Kggw5KleeDxj08PDw8PDw8PDxGEL0XpFsE1jfavv7qiJbnXao8PDw8PDw8PDw8PNYbvEuVh4eHh4eHh4eHx0jibZ6HY7jxzmqth4eHh4eHh4eHh8eIYkQsHLJnKQC4U91zCEltEKLYQvuqA3S++ts+TwgBOdBNO4tl2lcoq3NCYKhPy6ODAcBy+tcAAC6bsyMuuZ/ovDBqI/qtquuKrcBQL106OntSFtmzRG0J3U5ZG6JdQSFxLNF+AHLFy3T6uM0RvvkIbU/fM3m+DFXTCkiDkU2/8p6rSOYhFyGc/xhtb3FU48axlCXPU7kdk1LKDCHUV7sMa3XrKWUI8P3hX1VPVPuBymjaHqRnLe+4GOLEa2lfWKV9f/sexI7vpe0XbqdjWx6mzhmC6NqYyudn3Ai9RP8mB9YC5fZofec/TtW58fNoueBe2vfWc3Twjcchdj6JtgfWUpmdk0kOANHSmWy/6p+67/atgGgdGzknXPICxKipdJ66H1K1XQRFPQ7EmBnZ2gdAdr+lt/m5SNWfeOzIGz8GcdrPaZ8au7JnCcTYWbT91/+m6/f9FPQzE+oZB9aUUu2nXx7XT/4CYvYpXBP6GeyBXPwPErHR7nRkwePq790A0HjJ+gy5fdnHht6j+619jnzrWZI3aXv65bEdud4awy/9gc7bZG/aUaxAvv5X2jeLkidBjQvUBiFXzgMABJsekN4wuzQ118Tr2/A6a2yafRLyhpOpDh/+ZUY5qr3cb0QAcw8cc/yNHyH5Z/4+e2V7l9U9pPtrtR+i1OaoG68Vpq2S1wWuZ7FV/VkD1HrDa0b/N49C5Ut/VfusOVyNBd3GdqKHFEEBUvV1MWpapubJtYvo/EKp/jlhzYxRx1zK90GIQB9njalzrVXtk/Pug5ipgjv7V9G+X52LoZXUvtIhZ9CxyarPT9oO4PU3Ze5PlLdmAV1jrePO82Jtk1I6x5mcS1mNxS6n0q/VRmcoat9yOrZIzS+bHpDUKFtrafjM/wAAgj3ObdAyhZQ+6kLkHUb1l55LaX3vuOKpXLLqy+exSb/h948DABTOud2c2DY+u1BeEyPPxPS7ZB1Cva3fA+xnw/eMxxzP6wNrAV4nZc2co9YT7kPhYz8ysjbnPryazhm3BVBT43D8VtnbuAHhGqf/P8NbODw8PDw8PDw8PDw81htGhKVKrnyNfle/oTVA8rlbqQJKIy2HeiHGcMS+0moKkdDcAIC882IAQHDkFdFyGjQl1cLCX9VhDfKBb5L8Y69p0DILObUdjTB4zaEAgPKn7x5WuYzwOrIMBJ+8L/M1cu1CAEYrD0Q17nrfIPEyo1ihv1fO09pxjcEezDt1cwDAzF8pbV9EgxJ9VqQVUloRZYFCpctoSliLxn8DWHQ2aTmm3tSTrYEObU4a7L7D7ef7AVHA1btNAQB88Ylo+7JC1oaMhiem8RNC4FcHkLbxA4/VssvsXqzrB0sbRfvUGKsNAPyMpWV1UvdYt7U2hJ8fMB0AcMqDC9W1SiNcKCOuaZa1IWP14D5S7jDPju8PW0sg8cfD6B4e9deMbeRn2L0EomNitO3WvXMhbhEEhNYAv3HmdgCATX5BWlsZ1oCQ29qSkBvRZsbnGLZ8tY6BVPLF2M2ytc9q43Bh8JtkeSl/5p5hlcsIf/w+AEBw3r3ZL7LaGLeoyL/+FwBA7H66tvw1rMO9V1MdDvpi4hhbP0SpVZeXd6yGfyXra3DoVzKdn2aJi1gclYUXtpWUxwfPNcUKHj2WxsmedyxBPTjXR/YECIpGHo99PlYo45X3TQYAbHZH9rlGj8U1CwC+t7dfRE044duqKeUGlnAzjsIVyho4dmb0nNDUKWoBSo53XpuEsmDbZYerXif5U3fJ1Dzt1SBlQ4tqXRnW/R/8+oEAgJbPP9CcLMsipvexN0ipFZLX/PMfzC6Un2HVrAly6QsAANHSpepegOik/ue0cLD1r1hBuIgsOWLCNupatZbc+imIwy6kfZYVLWGRdTxrPYdWRiNcQnULZszJ3sYNiL4v5Jj31wNav/ryiJY3MkHjarIRXdP1S6h2hXnxDgBA32++jrYvPRA5XyLQLyjCcnGRM9wTAn2gRF+AZeSlyry08SSkX+RAnVcUi5DTtm+unQCeOW4itv+dcluxJmy7jllQ+vhvqF59K+m61jF1z21mgRQn/zjX+QCi7jJaUNRsKoQA2E2AzfvKtUntpJ9SK+6dT5uznHVPukVo1zTuTyKAVP3JuEqw+V7gsVep7xzfqF0swTbfN3g5tY9x3wHsficxGEb7WKOyE+cFhcQ+++++6jqYY2UNGFQfbup+Rj4G2pXZne//UC8wQC6JsmMyVwbbjo59tFjuRfrjQ39QBNq9THI/KLeZj5rBtWpfpz5/i64m29c+IeGW2Og5mONmnpCLKHPqxj9+CQAQ/u8FdMYxX4XQfS9NlrUt1PixxjK75eX64FAI592HYOaBkX1Z+m0cxS13o2vV80pz82kKM3Zcp8vNOKO++JVP/ScA4JK/n5ddxgEX1D/Yt4J+SxtFystVx73OyXlBunsxAKB/FWo3nAkAKHxYzdeiYNw9WfESFPCuU06sW1S8T0Rcb3h8C6HnV32eNffOPOkUNI3OKXoe/8m1pGg84zh6hrZiwoXIOEpzHXXcT+dzjLl32S/oomt6ffkuqHeLZ98z1az5/PKrxnijtdm4g0uUPvqTfOUnhImku5PtGvy+HApUBd0nCyXoNZndlbhdvG7QBXQoKBjlm5pXZKEF3d/6EACg82tq3mMZJ/yX1QyHK5YpIKLcBGBcVAEM/IjcN1uv+lem9m1weJcqDw8PDw8PDw8PDw+P4cHIWDhqys2l1GY0Kuz6MoG+lltnbWs04/zFXCjrfdLe95b6et0mWoyU0nzt6q/gZKC2E0q+LJSB5a9nbVkC29/4PORfSJMg9olq4PJozuQbD9M1mx9Of8fM/s3Itc3HciEFzYuYeToVcZM7SXDUgbdZiyYsrYXSiIc1TG2tp9mq98zUPg5uDozWRfcrbUKuYGIlp7cgu7xIaUy93E9VmyMuMlx2WDXmZk1WUMBmo1SwfkzrJMOa1my4A+9Ybg0yiFnirGDzTTua8Ibk8mQItHTwTvrhMVZqNfv4mZfbY8+druuu6krTb8HSIA6R+4LksVjth2SNoiJ8QLUfslsFQasAeVvr0zfUnMenEAJCWTYkk0DEAowbQwLzn6bN6XuRjCMuoyP/uBFiZwq2TrMquLR0kfPmU0Artj4uZ90AMX3PRN9qKghxgZpPXRbM4cCCl5q+lMaK1WcBfPnrnzDH2TWPn+1Qr+k/bE0XAaDcX9g6Z2v75VMULCz2+0z9ethWAYe7pw4ozxpUzetUwYx/Y1lQY6RQRuE0CpLVxCtjzHwtXyYXOLHt8Xo8OfuimtfYQoTaoHnWPOfVarpOsqTKZ4KJUVPNeG0KRvN+xkPkevnDfWisn/XwYsTvZ16LvQgKph/EyD6S5w5jH1f3a8pGnRZBhnIjtt0oRTGyL+p6abkgKSKDpi2Ng73GglOwnjUVClFID953QhNDCOjnFLdsiIJpL1vBK6PN+dZaUh6tiFHUO6GMrBeK2CH1GTr6hRr7UkqURtX3BHlbwls4PDw8PDw8PDw8PDw8hgcjY+EI6Es9EtDE2tVxFDgsu8YbulPlN42lr0BsfwJtl9X51X5g5t7OYoQQRpvKCIeSmtlqn9H2czAe+2PLGrDJbjkaF0NlNAKlKTPax/pfsfW0OfJe0mwFWxxJOxyWjbyI3P8HlL/odu/LLiDIqHGJB9DVhpQ1AtAaiqCI+b3u+1L3fvH+inLslzJpdSmTtkMERbxZR35dKM2inPdnCPaN15pAFbBaq1o0j+xbbmmqdJB3gHdNcNGHKo2cDq5my8wa3S6t6QwKWkMluX3WM9hyav0YgrrQNJBFy5Ki4p44viKsmoBo1rZW+61xbGifH1tK9d8/HqwoZYLmV5bajFz2Py+1Q3QpKlGtpTfWsKdW0PYOWZvnoGwUlgaMjmXtFwJY/LKSER1/YueT9RwiVbCpGDszYdEQQmjtNNNRRiiRVy/OWBcHrBiSZmI39LU1ev7BOmjbUsvXtNf5EZmzVF8UB1Nw6f1HTMIBd74VOV+KwMwT/CwEDOGBIyYqbtlwEQ7Ylk0nNXID6tcEUoOMlWV0qA9CzWccZyKLFT1exDbvVrJKkIteoc3UmDM1rvpXG2pUnld6lwMto6IXdE7W18m3Xs3SqjrlWtpxhY/9lYg0wpfvhpj2LtrJ8zpk4nwAkPd9jeQ5Av9FA8tGFsi/fItkHXF1tguU9XvclfcZi/jjFGsjdj1VCZWQxbjFv+iwfArIJUTBHWx2WFP1R7kdCSIQy0qig70nbZddZoolHg6SBVmxgu7icSRhFW/MJWvn5gmiDZGwbLjmEldwvj03L3mGYkOmpjTJY8PBZxr38PDw8PDw8PDwGEHkzaX0fx0j88Gh/PrQNk7vYr9q7XNfq2otjhi3JZ1T7rCsE5avfs1Qn9qQUhr/fv5atv0LGcVKfa2bCKKsC3lRG7SsLOqrn9tYbkt0sHoaSTEmJblezMfTldCrIUZPbHxOvE6sxWCrAmD8fm22E5WIRyfvG1hjtI2ssZM1HL+d0ZBEYcfiWBoN9vHksgploI9YQdCqygrMPTlidjbKzEi5ANA5RZevEw4tUDEvnRtBKgpAjbCW8H+XMsTUbRQ7l3BoM5kZh32+rfgCbS2pDVq+3iU+qGWO32bzHG2DkYkoAw1rsSTXqcWmhlLlldrNtoWNY3EkEaavGAUlAGPZ4NiYcqflo5+kwJ7WNnys3br/DvVmi+eQIbD/uWrTQd3J2r/R9JzfPG06pt/wZlKOmveM5crqK7vnZ/9JS7wVqtivYGO3FdgFMaM+K58dkwS4NYxUlxSq4U2aZ/2rKxPA/neYe60thkM9gLKs2fen+oMPAABK59/VuACV0C+OtLlb3vRx2veJPzeWD9SJh9OS6f+OiZBrF0fqJNrHW1bHPi1jYClZelpdlib2HOC/28YZWmp+nuWOpD+5pS0fWkqxF3nsOK5+alOYAgBmHYJV528BABjz7X8nzo9gx/fkKD19nDix7TG55CNUlsrOKYadasf/oLKf+y39vdVRZm5nK7kVw6ctOTIEOmLrioV4clYUytb6b82R3K+sZMNG/uR87YNtbU56YhgWqkHDFsn1K7UB/cSwyZYzUShh4iYcY8HxjVxOkmHLWaZNmR+JL6HrujbK/17jMXIYmQ+OuKkWxs2BB6OwaNE0FezkHU2n0i4oJTxzPi0eO96muMwtilsZN8u5uL3DsG72UyECvHYhyd/0tsGGTdMyebC4TLvKfUwO9pBbSaS8OibwnjV1y4oHkzX1lbx2Ze5LwudpEg22PUHvM1OdFdDNtJ/8slLpMsHEPMMERUw44cMR+fqFrDpoJsw1tNDJ7kUm0/WqN+j8iduayVwH7SlhhRZ0Hntmvgbyh9zEbRPUfCKNmz0oJp6BlCF+9md6Ifr4ZQ7TcCy41H6mWlaxYj4u9Qedkf/r28l8/KGLGzXMAn/8FlstulE1ftSLmrR45TloXq54WXOn6w/o2gDe94F9dX0i9QS0idwQPrRA9qqPGpVfQBQrkIrrXYzehM5XWZhl3woc8L5DcjQOcLliJM4otVn5fVJcBkQAvHw/bU7cJnlYPxP6nX7DmwiXPE/7VL4TsfHepk7arccq61WV0XrjvRrWO1GuA3k+NBi1f8+la11lOV+Is0MIATlv3bMox2UCiNKN8/ipk0V5YCn1qSxOoc24pQ2+RRzfmZ0cXUoIfYhrWQLY3dDq13ps8hwpCvjxXTQnnnepo+6uPDGlSrQetgui/jA2d+vGO14EAJx+Wf0mJdrh6Kf8oWG7p/GHRriQXKl7v3s6Oq78R/Jadqt2wOXS55qT08ZOmnwn1PmiUNYfhFyPQBFKcLlA/Y91JQT9130MANB68SPJwy6aW33MeuZx2l8rt9HA9WcDACpfeb5+PWJIJathxUlQNIoJ/b42kHSplSFkGHX5akQ7rxUJLL9YgQh43nW829WqiX1va/igcQ8PDw8PDw8PDw8Pj+HBCMVwZHOLSLjWVAeMqwp/CYY1tOldGZP9JE9KpZMMCs18dWbQrJbbk2bD6oC2tkS0NPHgLwvNBoiuS2ApAE3Ra8tKBPoWWyw3IKWBHOwFOHCONRayhhe+9x0AwHaHfjlajqWlkaNUMq7OKeB+JBSVMoIiUODOoLQdiooVIsCr36PkUrOyBgEqzWJUC8bJuIyFzaYapPZ1azcOqxWYUpf2t467QSIoNTQuSIV4BjyBGe31+0jdcv/2AwBAELvnACwK4JLWbGnt6KiNTL2UW5xc+CJWPEPa6/Euel+2ztgJ8pT7EbsgoNyhn7dOmsjUpaU2rH3+7wCApI103cDaxvBff6Qdmx2aoMwUQgA71E+o5pTLtKUTtgYA9F+xNypfvL/++dvkdONogNxuJABEobksyZnRZBbmLEiMmbAaSb7Jx4JyEwQL8bJSMmIH5byWoLS5wU6eqeYa20rI46l3Of22jMJeE1LmAu16oiwj9tqit60yRYwyVxSx3zBH4brWoGDqbADAd+9ejM9f6biojvWqnrzkOUEiMXBU/rjkvjRYCefi0BZUGUIuV7TTykIcsTRZltaWTbasLy8D9XajY+XNslJvRCQoOcY6lKCFl9KQmdSUZ0a1P0kR3L0EXVtzHZJUyOz9IKH6pAyNtY/dx/rXQLJVmgl/WEZtCG1b7tREGz1GCj5o3MPDw8PDw8PDw2MEwYyP7xSMzAeHM0V9EsaHX2laK13mq1dR5YqujTF1y8lKbH5tHgDSGsWS4Bitj8DErWbkkwcAA2tVnZO6WC1bSq2pCpV2SlRGa8rTiGZ7633rFuXWZMQsJ7UhR6yHRee4ZXafcQ0OELeTYA2SBl77BPevMpoi1s6VKiaYjQkEWsdgm2/eFKm7xpqFRoYKQJfdiyAmK+0FP+7agLE8DFJMirz/29TWY7+GTa++KVfzbDpW06csKwZAPrRxUoC+VUBljJJh4onSLBxxqlz7WqtGRsuYqKvA+CbyOGGnExK7dL8oWNTFrD1la0NldHK8jdscYTWljVrTbD1fjgnh/l9qh5z/GB3j+IPAWK3CMF/QeF7rHdNOy2q/M/BfPkwWISiLUEP5hSj9c+WLD0D+41e0a5dTIzKklJBzf0H7Dr8qc505JiZiOTKVpp9avz5uazld2tCgs75md12togCA9uG2Txkk6mX5ddvHan29615WiqUmHMpJ/ctzKZD0RddjXliWVTXYB3u0n74OwC62YOudyHKYmEshTJI1O+Tb0sCT/JIuSyf+47FaCjB1h1n52gekWxNScMHji3Qgedc1nDRSQqr4KKEsIc3AZf3S9V1GZQllVW8Iq11ai8+kHBzLtep1QMUe6uST7RPM2hgYq7o89iv1y4onyqv2O+Mq0iAO+Xyu8wFY6QKkMUrIMPoLK+ZPJS9EWE16rAQF1PppHBY5gJ4XcxkajwGWO9hjUSUrFErG2hGXLwQwsO7j3GP9YWQ+OPgDwhGo7crEq3nqa4NAVQ3MR39Jv0degrZ3EU91lg8NZ54L13UcCNvSgZadD2goN4GYeS9aCTWAqgMmSL5TvZgM9Sd43YUIIJdm5z2XtUEd3BnPB+GCECKXfA1uY23QTKx3XkoyD1cuOjI0Hxf83IOaca9idhIZQozfKtYQNQm9+iDEpnNoH7OzlDuAvuXcAvoplE0ZKghbHHWFaef4+iZqFzTrhpSQkhcEtUA9SB8yGDsdYlcV7K4mWVHuSObLCKvY/Vs/r1+Wyt0AZryq9ieC7CBrkMspoBJTd1WNMjkGtvjGr3K1DwDEKApCtV+w9QsBv9wEBcNown2zf5VhAONA/UIZ//scTfBnOPJP6Pwe9gLB7lNjN9NtFDPUsw5jAX+lVtz7NL1UJT+T3JDWYpjnRUcUKxGlA8vAzsqlKpKFnQ8nXWwSZRZbgC1pvpI/VCw7pyiu/rZxkNscnbmOGikZg+X9XyfZe59jZdrmZ7gaIr6AAxAnfqO+vLk/o3N2PY3+tuaayHkpyh9xzBWJfeuCNNem6HnmhbfQWp+VLKu8NAQtOV22dD+xx43aZndGWTNuUGsWUB1bxxgGPU1AItA6Ywt1jVpD+AVQSjNHcr/pX2nNw5whejUxxtmwmJRaZm6br33I/qGRePmXNf2h8afDaB464u63gDz5IxrVjcfEagr2F13TjKtuVvA9DmvRDzcAYswMOja4FoLz7zx7Kx2bfbIV/G3RrqR91MfHnMPtV9aqev+69OVIMVqBOWDWO253n2KjtPuNnh+Frpee18sd6Jv/GgCgE7FnLgqGuZQ/WlzvVMVKfbYsUcDgMiLrWHcHyhHCO4wW953VWg8PDw8PDw8PDw+PEcUIZRo3wWqZ8lAora8olIwJcTpxucuBtZALKAhLKncb1tq5rBlO+SJIWBWYUxtSAsvm52hctBxnHVjTU7ZMsBxk1WL2Rcy7qo2ZyrbpIbN+MS99I7N8DU09WzHajsO+RPtY29bSac6ztHNMeao1NeEQ5P8Sn6v40PVce/p/63cDivpOdpMZWnRMMsGn7N7EombSAAAgAElEQVQEGO0J88prrYiAvPurtHVKVkuAuv9BAUIorTxTth54gTrFur9skVm7wGh9Fj9Np82Yg+rt1wAAyp82wfZ0WRVi9Awlj601Sa21CIoAu5HxeVaOE/mna2k7T7Z4fjblDt0WrWG0g/VtqxwQyc1hAnRrKLMiLZZDQ0okrGxCCMheRcfMpnr7fuos7UZ+Xv4GrZHLd1n02n5y+xOVUYC2kmXLeuuUy1asM28BAFy2K1mZLn1yEcTYmU3Us/5NCRxZmPV1lS6nK0l4w+kAgMI5tyflKcuGllGHJjdt3pE3Eh2n+NQDdc/Jg7T7Hrk3lqtStWdtU/JspAbn5qXjVC4y0XwwTMrQocvRuWy4n4iCGR+cLyQo4Y17HwAAzDg92hYhBCRbM0JlwayMMRZWds8bO8thFTfuoUsfvAcAMPGM7E1kemsRz1sUL8XhFsf7jribqO/DZS8Bj/yUDh/z1eyVaADO2i5lCHk/WfrEsddkvFitgUIAiN53vudi432M1WEXlXOn2GIsIpxBvm0c5G0X0b5TbrDk6sIi+6SDRl0KmZjTDZ1sC+Sdl9P5p/82W/tsFJKE0jaNerwudp3DVx8AAASbHYqOrdR6pvN3WdbhWMoAGYauadciN1FeHa/9heTPPBDllMB7jw0PHzTu4eHh4eHh4eHhMZJ4h+XhENIVRDHMkN1vqdKEIxmPKr53maG9i/l+A9CaHhRKRjujM5NzxmUBufQF2lI+4nLlqxAbKf93Dop76xlKKghYwXgqEVlLJ0IlP0hL9haH0oSvC54/nrJkbvu7JcPiVxxH+AppqYJZBxv5HfUzmieu//ef6JqJ2xjtIWu2laZMPvUriN1JDcZZckXXdDOwtJ++AFSCRx0gbp9TjVLPyqUvQEyZHd33xt9MQraSSjLHCddax+lMvEJRlDaEeoayNmho/tiKoZIeXnnaebjo7wvV+RRTIhc8YRIDKguLKLU10IjGssU7sl8TNa3U8oCollVv18mM7IJc/aZpF2ur+Rn2UPvXXnkYOr9GSQXlGyoxXdfG+nnpPlNqh+xdSvu6FN0ta0llaMU9qHb1LQdaYwHKg93GysG/mmygFVIlfgx4DDdqHz9/V0Cn3hEaraNF1OBKFCZfJUrbYOZBybIc9JBy1Wskl5Odto6B5Cz1HIdjZZLnY8F2783UPgCQPUsjdc98nd2vzV7In1Ci0+AjN+eSF5GSEiAc/oJinoKPJS0odcFj0ZVt2EoG5kq0Go/FsS0F+rkrX3NZG4JcOY/2jdkUANB3+T5ou/Rx1S4rQSTHVnCGbyaKCArGEsGW3AbI8gwjsXmhZUHUQbUcdzhK73MmuuXzWav8xPUQ7/oo7VNtkq89CKxS89oiFajdpyx97/u+iQ3JsV7ItUqeKDS9jum614Yg/0AWAHHQ52gfz3uO8Ry5lp9NWpwlgPD+rwEAgqP+M1vlhmHNv/NQup9syRluhH8ha00w59NmZwq9cAI8DsOaseLH3k2ilLkOemabIIXfBbkO3C/CWjI7em1Qr1H8XMMlzwP/UBaamXvQr7J4iBn7aRp5MYzxPusTA5duv0HLb7n02REtb0Q/ONyTob1ox15Kg2KUuQlksvvLkfRiPuePS5LyEoOhzoIVd9+w5P/rBJK/xZ058hwMw+Rjo/odCiYtnnvHsMm0X1bD6+gFJ/jkfdmvd7zMZSmPy4wfu2wXIne/9MlFmesQl+GSy+V9Zy8KODz3H9ncHdJeAlwv+uZgzXlPrtiFyr9o7uJUeXXrI0Oz0DvGzk/nkPwP/z27O0e2sQiT/0OzlNSsl1Xzgm27B9WTFxlj/LLIZvFyu/ngcdzDa/ekPvLJZzK2cZjHYe9l7wIAtF3yWK7rsnLjD15zKACgfNGT2WU7+um65OYJf3A8AKBw9m25r21UppQS8qcnAQCCc+/OLnyYn2P3l8iVw5XBOg4n0QjS29v/1f0AZM/inPWjMf4CF83bY9asS2eT21LWuTTxsUySInWy5f9wb5prPvZU9rkm5I/1Sflequz7b7sGybuIxSk4ypWkI11eHPE2IqxB3kNMccHx/51NrmKdcuVSss6yK0LnW3Ov/XE8dM0hAIDyZ+5pXLb9AeB6h3Kg9l3K+VO84NGG8jVyjkOtDLDncsnvZEW8eSqtF9NuIMVXc0qTKEukraBe8nFyPZx4w5pccjcU3mkfHN6lysPDw8PDw8PDw2MEsU504/8HMaKZxtODxq0b7whQ0u5QhRL2+cYPAQA9XyYXhbbPkRVAvvkIxNbH0TbT3BbLJjCJza4u+Zb7x2ZX35C1YesNUvG6D6drVaRzNyPPYS518pnHOMYx2K1djeycJNuNDp0yGmkYba1RQqOknrsMCthq1PDlcHAfM1nIdR+ztC0t7HXmcDfJPNEkKABNfxhVasY4yWMxRetXG0oGdVv5QIz5vIrAGttRWXWCiLUcJmsYcrj5aMJ3FJrk0Vv7+e3Q8dW49sbQTqfB7o+t5/+O9rFlyHIp4eBF4aD7dvZfdutR1iPZNh6lj/6sQUuScMluduEi+uJapH4uN6Ws9XCdI4t5s3BHteuJY5puu5g6L9oy2nY90HkMYVVTTwuVCRp9K7Sbp90XElp3y42pvEP9vEl1aqd+sz63dCvSyVs0ngu09hkSGFIusOyCPGl7PdalCmjX17WMwjFb50/6k9eywYj0K15LCiWIOedSfVQwulQ5ncKbPo3ieX9IyMligdP9qxAAe5/dVH2j8uJlCeemBrvpWX1Z51dRgecyrCYs6BEqbtc7lII9BoKOMdka4UDE6mRlvY/XwfVuJa2A+mk/JFddPQeq38v23QGXPK6o+rVbrkzKC4raUqSbO6CIVNrGY8I1T+VtmscIwls4PDw8PDw8PDw8PEYS3sKxHqCzmg7pxDAuP3jtQ+rSWtkBRS89BABou/ihyDGx1THma5ez7L75CDBqOh1nLXvrWKO5ZV91aWnbX3yA9u10Ss6Guq04zaB0/l0AkPDNtX031wXBGfmTxiUy4tY/kf5njUixxXrenOhHYFFfutbOlhE9yEFodn2UfNbOigALU+SvC4ymU+0olM0+SxPVU7USyDVVjiOJmvXstfx8UtWvI+EmB+pX+3QyJ92ugTUm+NtKOLZ5V3wcK4sHpCMZYs1YTmwaY9a8cuyWJX+LzuZCzDq++ix6L9kNAND+lb9reVkQsdatpYRrwUYkK1xGdNXB+C2clo1Uudw3LQtCyNShTWqDhwPBx38PALhsF8qw3GxMVT2Ik69vfFLyqrpHdJD3uC0yyxBHXV732OILKdv81B8TWYgMq4qwAYZ6E4ZK3cw/lvyD69MRO2EH0tY9pZZIOCklkvWAwPTdd6gvJ2aRgayZdfLlB+l3wlbQ94THKN+DsIpJs3fK0KjGyGuxj1B1czJSNXcFim73ihvm4svnDUPlcpBvUDU4W7Zt9c0ZS8VzXW0AxTMpufHaiygYetQ3XiSZz9wMsdNJ9avhIhJhi4PVv8R7v5WtTrZsizTBXu/sMsiKktJeTZlehHz0Otrc95NqH1FAX/L4a5Zni5JV7U16uwxY3hIL59K+eYrYZJ/zIOf+giQccmnGFnqMJHziPw8PDw8PDw8PDw+P9YaRsXCwliZCS+fQjMZYFyJagiFOOtdiNKJDSkuqtLFy2T8hRm2kyiTtqpi4HdDKvousGTL+6Kw5wpoF6u+2VK1TFsRjDbJaPCKaCkUH+uXLTgUAhAstFpspO+vzmqmXEMJQ1uahyEuLY7A1HEy1yNr+sGbROxpa3Cmt9bVB6TETnMyoYl+hjql+EpSwUWtz2vGov2rST1+3gZ9ztd/0ba2578DMTse9AflSO+OIstSNE3UVK5jZ0YTlxJFMysRTcLsMzaRua7FV++za1snJ9iMAomNHx/KofjDUQwkHATMma4OaqpdpSY1GrIAJlSZJ9GSIti8/TJssv2t6fjmP3Ugyjt9F1XEG/d2/WiccTa2Gbcl1abUfI60mNjs0f92GCRzDcPp2cdab5mNDIvJXkm+2aJ+Y/1oHYxFTz4qg4GZCQ9wSUdAJPLWlXFvdqhi3+aYR+Wgda87TyU5bjdVDW+LU2iECSEWRnZUW12ZirAubuto6Xz8epsUtd6A454ORNkSfW9TCIbuXGAvDrH30Pqg5TnsCzCOWKbHZYRB7vz9bu+xSHXXRcQrqXopSayLGzY5Z0M+k2gep1mjRPknJoLn2orv/gvBFolwWWx6daLd2+RciPTZI99Oslo58sVSp1o+gqK0d33+QYjgu4LpudUx6LRrEPmrEaNczQc/FpYRlX68NIsm6Fbm/bPWWITBPxVjso4TYVg229qp3ODu5oe4HQcHESaqkrFIlIISUwBvP5G/jhsQweMP8X8LIfHDwRGab9+IZMYVA/IVSVgeM+0GZTY9DwOZ70SWxl2UxafvEy6Bcu9Jw4nOZbz4KqBdujCezvJiwFf0dVoFNs3H+uyESAzMN9iQRyf5748cBAMFZt6pjsfvVTM1sV5HfEC+3yENVqd1lLLcxbVa2Jjp+VjxZ1QYikwdfV06xqjsnZ247B6vJMOlexQt4UEAhyHevuEy5ch7AL7/xhU/WTFlC7StWTKB8qaLPP3xblTvj7z+lY7t/RF0XJF+Uqv0Qxfjbu5nA9eJmBertuH12TnwNVU+7LJ0TRI01ORRYH+VMPVizFh/17Kv9mLucjh/gyK+jX1z43pUthYMOPCybrOuWu52qGJ5bRXJ3ztg8fV+DgnlpVB8aeYOiAQAt7apKHFxa1r/80Q7+8CiUEM0+rF5yVs+n46OI4tcEwpaBUTndOOB2oWBExngWinAZAndS5ubpN7zOF1oCU5RAtpw0V5l7vkm/p/06tV312pHYp16cwsd/jIDHVLQyfIG5Vq0ViTFWakPL5x+I7Fp+/tYYd+3LkbIA6wUoPiYBiLZYfplGUEQJqXlG+lYCrSpLeL9ygWrpNPMPf7xDAL0qJ1XaCycrTXqXAipAXmeL/tXHIU78hhKnKOM3P9xc17MiX/tIqqlfDCLiWh0Lxq8OAOUobT1KbZD3kkuQeO/3aF+F1nXROUWv4xyEjEKL9fytDw+nskrh/v+i35NvzNa61x6g8jfeR99b/WHLL8F2TinOuVSwSBS0m1sIOY8o6i94TH1Y8ZqPMJqHRZ2vx7VF0lLvg0pKqTNyix0/lKl9dHJyPOtnImJro6NctVPXef4DVIfpHy4lz2dlMcMiE9HrZKnV3Bd2pedxKQSW//VeAMCEMxo1zGNDwAeNe3h4eHh4eHh4eIwgPC3u+kCWRHGiAK0RYQ2AKCTo4GShDLHlEfXlsCaOv3rbJ0Auo+ArsKVjqBePfJ4sCHv99l/qfJPRVWx2SOP6JopVmoqhPm2VcSWli+9zaSQhAogDSXMnH/k+7drrHDoWVo1VwTK3Z3HRiWgh980fEK/N+kKYuipqQjCF34pXjLaP7zcCYFBpnsIhfezo75LmX9Ps2TLVteyWJdcsgFAaLa35KJSN9oRd7nTweAfmXPvLptonOqcioW3ntg/1GEsLa8p7lgEdyl1Ea71HY/RMspKIrY6Itk8E+j5IbktYS1DLwkqUpzWCSksvixW0bTIrX/sArSmVbSWjjWYtNj8HV7nS0tSVDM30f2xpss0CUY2V7iP8zGtDxm1SZSEWYzczfYi1aSWjfTx0q9acDUzRquaxbPA1nNXYdayTgliv2pV+L/x7MsEjAIiuadEa2kGXe388d530/bTICtwF16MgtysjLR8JNT/wXCjDRJ+0g5YjVo9hdg1gTbVwuIGwm6Oce4exGjLCqq6/qXOYSgMep/Ees//RbosO90+22NuZl/MSeTis2hosq22ccQdkOuZC2Wh4rflcbndctE2R+sfWnAnbQP7uswCA6nxypyse+QnzDPW8YNGtbn1UvvahjrbbdV783tUZp+I99YOeZcxy8ZN9p+Ijf6XxaLuhOi0bLP+4jBnG+fxpKtO1/Y7Clg3bMsy00A5adOPhEUBssjdtc3Zvdu2qDkD+iUgPxLuVFapvBWS8LeWSGcuxbiWEAKa9K1f7qP5qTUwhyHBSgIfGbd1+h5v2gQ+ra1LIEqy1RK9J3KChXoSLn6btRZRkU+x5tj5/3PHetPF2xjvLgczDw8PDw8PDw8PDY0QxMhYOR7r7BFUfkNSSubRGsmbkuZBIziRNkJmiuMTiF7HL3lvq41QN9UXe14yvKqC/wF0auYxUebamR77yKO07/LLoOYUSZN9K+oM1w6W2BH2cM/DTvr/zHqPfPP6cXMZQn9kumsBJABTIpQIotV/rmvkQLcrPnX2ShQD+pqh5Z+xHv9z+Shf0/VR9Roze2NA0agtYYIgDWPvO8mUNePRm2t7hgxkbaGmbEn7VlhaUtT18P7lMAGhRftXFMgaWLQEAlGIaV4hAa7HAcUi1QaN90wGiBeP3G/fhFQKDyygZXS4bgIo3iGgVhaWNA6h9OtCP46qq2upkNLsBnl5Mdd3EoU3XCdq4jwQl8wx5DMvQjBnW+urg0Rr+PZ+ebxPRKgnotvSv1L7rDa95k8YhB3W7NHNffOwNAED4g+MRnPW7LFLVr4Bc8ARtjc1hrUqxGHM/FY2sHwwRABM3ccoVItBzDSdZo8D7pHY+taxxTQTrF1I0qmr8BR/7bfJgIokktSO87+u0feiXk8fjFpHd3h8LQgfFHHHcBVvnOEBcBJAPf5c2j7i6br0jUJpbWTSjN3EPZejop7bFxYrre+UB2ubYs6gg+uF5JawBnWSFLv6Hmv+X/NvMP/G4uLCq5WOj7PGNdl+sB2FZyxtR5cpVFGMkxifpkONlnPHQQoS/VtbDvUmrjknb6zaxR4BN8KKtSZ1TU+uRLLx+bCoFxXNaABOvaiwb3GZp7ns8aL1tPMTRVwIAwu+TJSv4yE2G+p9pojunWt4PKkbCtoI0Q8+e8uzsJL7xZyeCgo6njbz/dE6I7INFrBOPR5PVgUTMlaz26QSdtdvJIlXY9TQ6VmoDOnLGUm1oBN6lavjhWCDNS3DaJONwiwiKeOakbQEAO9y2xHkcsDp5ud24OCm3Fzl6E7Tsfka0bvrlcSxePYlcIGb+PuXDpl6NGzAuZebpXr1EVcsRtNUazRgqpTQBrG20QLkm+UiZq99KL98F9TFmB+tzsLiewKsD5qV7YC0da59g3Ks4+LEyGm89THlUppwWNaHLsGaeB7/o1wYNK4w9cZbUtfyCYgVAr55L8q3PgXTEXQpsqJdie4GEtWjofqfdyATufoKe4XuYwcqWG3t5R+sY66OG+6RIBHlLUdH1eOyRVwAA+2dtHwD5lsq+PXXXBFe75PsLAfQvj7axUNRjVXI297eextEfP0md5ujX/DLFLkBBCVBZjLWLyNpFkKteo30bExmEZLev/lXY6+MfzdE6Vfd6R7hurWPNvdYfWRX3mHyYAkh1AK1LLrtInfU7YvsBgHY1RsKqflFN3G8ZAo+rQOrtc7AApQS+pr3c1cWCf0XqFZGn5hp7ztEvGg4CAicW/it3lbLkaXDOjXXm1epzxNVfzkAGFnmh1S+ObdqdMYgxUQkhIP+pcgGkePtGUGAiFdd6YbJfy8RHoPsj7xefvRAAcMqDp6fIU65yQQnisEvUUTW+J1l5PFgBYLnu/f7znwcAHP/IZ9PbZYM/cBr0SZvUAABQG3S78LxwB/3O+XTDooUIdHC5nq8WPQUxWeUTKUTHJCAgX/wTbWZ0PeIs7Zyjx25D1BUvOu6dsqSEfIRyVAQHfTEiC5C6vwRn30a7agNGLucnKVaM8ogVBXzvpYT8+8/o/MOvytS+RtDtifC6mPbH3eEB4J/fvAIAsO0+nDjF1Z/Vr52/i+eccgdEheS6ssvP/z65nk0/8Es5W+MxEvBB4x4eHh4eHh4eHh4jiOFIEv1/CSNk4Vj3zNg2BjNYBm2TeOKhhjUKsKIzYxcGqA7VsL6QmZVgp+xBekIIhP9DGoPgoxmpJ3c8MrN8jZIjmC+unQoKSHSr0OLG11SOwEOvkHvNe+My7edlU2FqjYrqTzI0xx197JmXiM98Tp3mJMCm/WKDYaHrJ2N/R03qS/uZytah4XNpNhPBrjLhImL3n2X9GfuSff0odm8xdJQJyBAoKK11P91DuWY+xLTdaVsHfG8O6SBGMPXkPmHdT37+bBHrX23c4jSVtZLfPjEhf7igA6ODdItj/+uknc/MYK/dIZi8Is2dJMDQotcAALnsEinuRnlBrm+D+S7iXDPWWE5FE1TeafkSmkF1LVnNMt3nyuhIMK9G0VBeU91Mn6l2r8kuH9ksOPEy0jBvbfZxIoSwLCdMgGEHNKs5zApsn9/bxDgs5iN80Gt2vTHz+rP56wBAsMW9a5PUoHHMe7L+MZfc8Vsl96VZrNJkiQDhS8p986CYLOHIN1NoMXlY2O3YdslWJC62FUL+k/ISob6x1oH6JBzudqSfl3cq0O3NGG5cqzWfNsBj/eOd9Xnl4eHh4eHh4eHh4TGiGBkLh9IcyyDb135anIOUEpvOGJXYXx8OGQNrLRpZdZz97LuXYMqs/Blx09CoPU6twOtK27L1u5PXxCgjpQwhTiL6XJ34sDqQoAGNJD5j+Tuflr0hKVo5OxlQnKJQBwpTLfTvUNjY3146gkDtTNSJQDjOcl5uQ28tp1Yua/bvRMZol1ZFopVvFwfURwIKHZaBeGI9KU2CPAdVZGsTo1cufwkAEChKVypGtYfjGYIiEKg6dE6hKpXbzNhVAc5yzQIs/P1vAADTjvtmsjAd/G7fV2XFWEwZYcWk7QAVz6E1dCqDLAa6seyOmwAAE0/8QdYWqt9sz56zbGP0Jk6/8cqW5POdmfgh5ocs+1fphIcurXZx0+0y1TNW60gZkSNZY8RslKIZ5NNizqSUpi9aPv7xBJWR8puwcA+3q0Flevag/Hply1dV5u2ZByWOFae4grXrI0LVGj+mY2QMjXQ0bkfdW2vuO2lWfbN/4hnWBo2qmeecoV4ihgAgmeKb6YcBHLdF/slmuHMMiJOuW7frR03FpbNpPrv0yUXJ4x/8YS55cg0l9BQTtk4/r8H6TxshxH5pVPUxEg8VHwkAchnN6Zi8g5nDmXbdXgP3Oy21nk5w4LcoRGmgAWudqpkYRhUvg2JLgqpfhlVs88ETGxYZuSec3E/du/C1ByErFE8WTN4hel3fSmz83pNyNO5tAJ+HYz1AvzTVN8FHXrzVJChRSPJVyxCllowvhpZswBrwfcuBMTP4IP3qINIqCq2ZHSgM+lSQbSz7eaRcqy76Zazabz527HO2S3F5SpiqRTSzKQD58t2aWYcDdXWuDBkC2xyW3h4XdNZhs7jJVcTQo7OE9iwxL5gcaFrtN5Miv3xP2h4f+MVtCXkAgL6VxDgBmAHZv8YEo+ugssC0V7H9cFZpUW7HodffkbeBqj6Oj0AuJyiYZ6cg33oGmLoL/cEfV8UKTthHLdw6Q7v1YaJYf6SwWLviLzrVfshuFdzPgapWIOYBe0WDVzNB85obGBcKdmMRVsZe5Xr02kPAVsfScX4JKrZgwjab1S+LTf12/+fymaXLziujx6nJJDtmyy3ztC73i2qgMxT3uU/oXeven7U+ldEIl/6Ttl0vJgM9zUitf6SZBaxzbN1r9fzLblRDvUQCAZi+URtMD1bvzMYItj7R+woF+GZ0AnNCbLo//Tru0+Ab5HrXIHzeQLPAOVx+dcbtfqNA4Q+UUhvAeY4sNrtZZ31Ky6M6WjLjrFO1Ie0WJ9ktZ+GTwDQKftat0x+KAhudfn7WlpliU3KppMHFXCVlCHnTx2jfh36Suy4M/tBgYgr98SxDyN9foORny98kKkk6ksT6HhQtRZLjdctmJJyUonzQ5ClVXX+hlEEYVHNU30or9xXnrVDjtmWUUeTkgc4TVQWgXv7jbtSRqHHV1/pWQLbT+qfJMoIi5IRN1LWOvE3x9wDXXF4oA/+mjOxy4rbR8ypdwPgm1kSPEYMPGvfw8PDw8PDw8PAYSXgLx3pAhkzjroyuUU2S0n4GBXRssX1T1dBf5CteBfjrmDU9PYqGtmMSKhunaG3roTUb/7NuE5sbHW48QgjIjikNZWie67eeg5i8I23/njIji6Muh00lp67MJL9uuazZqA1Bu3U8eyv97n0u/baNM2ZdK3eFYPOnnU2dOeNZy6PqW73uQyic/B06jbU4xYqxHrDmrdpvtOiaytDKP9KVU9vBlK2uwEJL2wc7UzQATNk5SS07sBZtn/1dtH2sTSy1QT76Y9p+kzSv2O4giNmKE5+zspfbIPjZsZVK102ict7/5GsfAKEsMbI2aMZZzOVFhrWkpmzMproOkoOWyx146sHnAADv+gw/V4sCUluFuI9LbZ1jClzI0MwPcfnFCl55cC4AYKsvZmuf5PweIki2K8W1IcKXr3cGWqPo0lJmdrMap6woKgBf34NiC1AbXoKKrDkNIvU74b/qHqt99xgAQHA29WX5p8sgjqWcFoYoopiac0Ec89XMdcmCZgLKRWHdl7o45TqsHAK5XxwsulLtIRWnAJWhyZszpOpfGWPKsindl76WUljcHWeNtnazG6F89m6IzQ6m49otR/0WSg3ku5HXshG5ltca2w14HV2qqN+o+Und12/tQa6l5z+6GDj+mnwCO5LZgRLrO+C2bPAh+7xR0+qXpXNPqXG9doGxdN1J9ZZHXQAxQb3XlNW9L44z9WIvhBwwQdsGkX4EKBpltZZUeN0YlcirAQCrbqN1b8z+n0sWZrv0ArR+xN6PxNRdDW2x9jQwBA99d5L1q/3gS+Dx9oMPGvfw8PDw8PDw8PDwWG94m9LiunwDWUtTwwt/eAAAsF2GnGBO7eNWxxhNsfrV2hgZYv4DfwEAbHxuniorP/Sh/kSwdupl9YLG/0SZReV7VQZbO0u71mKqfRO3NTKO/xafBaM6q1n7QPvvU9qcjP6qVC5OR5IAACAASURBVFeleQuKur1ij48pkeoZByUgVBp61lIte4mCg2Fp1GSInqsOAQC0X/kPbhgAoHj2LUbLx1aSUjsgleab4wBs7YfSdoiSiW/p/28KUKtc/kK2BjqSUCb8Sq16aqq+oT4dcyKfIA2LmH0yBn9ESbhaPns3nccxLQNrIeZ8SrWPk+LZyQOtmB7WGLGVoNfECoW//AQd+sJj2dpHDaLfsAoU4kGAxrIoQ8uPHICYuA0SsQPhEFYOish5sJ5vMoYjGRtCDQiS+0gK1uTMvWkyCCefW0NLhOqv4Z8pOVVwyEXAycoSxT7fSqtYd9ymyGWt8sDX9gcAtHz+AYgTv1XnqhSk+ITnsWwA1A5589l07Qd+lDheOOf26I4jv5IsUwSQbOGzNdJcxm8/SftOzW+Rc6GZgPJwIBm7lBdJ65GVtKyQc41T1lpN2QpLm8zzeqUL4BguXq+EMOWy1bNQxsK77gQAbPS+WJ2lNPMa95v2icbvn+fx91wDlLiMWKA6BFY88EcAwPi0uOYYwpWvAQACjpdsAtyPZFiF/N1naF+KRS5VlqPfnP8oJcyVvcsgH/g2nZeVoMKKzYvPBbZ2PxGHau+z5n/57C10bJdTI+dHkj3ymjdmlklGeyZdJ1e8DEDNe7xO8LpUbof85//S+Xt+Ilv7bARJal62viUsw7E2hssovikYvwVGb6M8HTi5K1vwRZAMMnfWw7x78Joo59P6F0zfA62zts3VrA2N4SZWeLvDWzg8PDw8PDw8PDw8PNYbRihovP5XHH8dy6UvQExULA3ad9ROAMfUogJbHbhb5NoImwFrUNliMdRrtCSskVj8D1OW9YUNAChWMG2/ffM0Tl2utEI5rBtA9AtXKuYe0TYe4r3fi8p1lcV/W5qBOC0nbdJj/sm+5K96xkOLIVw0po3AlHelitGCaV9kpXUYtOj6lHZOTNpea7uNRqeIts8SS5WJ9WCfZAGwhp2Zsd54GGL6nqp8erYDXz8ILZ9UMSTqOUrW7LR0onz2L/K1T9fRsO7o2AxFEYne5ZoqVtOCBgVof+5dlAqw0IKWz/zJOg7Y/sOym2KG2Oox76TpmPkrSninNWe/ORfY5wy6hhmOLGaUwtm35Guf1UbYfufcNvUMw+7FOr5GrnqNjpU7td+zbVk67JKLYwUomWEVqDFFsRoT/astX2Sl4Vv9pn7+omvjqPzaIHa9+KpczWO2KdvS5TwvPneEVT2OxMEX0jm1Ich7v0b7jiSrh+2/bxSWli6O2Zw0k1er9nXm+1e+4D5dV3n/N+jYe76buY1O9pZ1QTl7gjY7vimiueVxx+PQjkfIIT8OF4uTfnbVpDVZytBo7S2rcPtHlZ97leO0LCpgbSlQ+169D4GivrXbIVe8QttsgVRsXSIoonTCRfkaltI/be03x7BpulE6QL99K+m3cwo2+tGLtO1aO2OQc38GseuH6Q+Om3r6Zk2PjOl7AACqPz2TqvrJOzDuW/mT7omulJiEvLKCIoZeJ7a3XEkyLaRZJUXbeMjF83LWKQNNfIN92tJVGQXMPqXu+QnrTPuESGJAAEDnRqZfKe3/ZXtQPOqlTy4Cdnh/3fo2QpRNKmbpsI+F0Vg+ABBjLUrqYy+O1M+sQf3aS0DLdcRwyPmPAstepj82VSl9+1bQsYFu4IgLcrdtg+IdlmlcSFdkzzBDB2Q7bm40AM/qaAAQFBPmSCEEHjiS6Nb2/+OSpLwEnZ7D3FkbSrh72IGIL76H5G/1pwwpzRlM8zlMqH7naABA8dy81K71Yd/D8DrK7x188r7s1/MHREa3DZcJ2T522S5TAbg50bPKr2fKFkLgv/ekD6xPPF1NXOuU17PUWde43MSQsXjIbVyxC5V/0dzFqfLq1keG+uPDdc+vVx+Qp8/N1j4g/RlG2sUf7vyCKWtWPg1+CZO4bFd6qXA9w2QOh9C4kPFHTrndvEA57uG396Q+cv4zGds4zOOw9zIKUGy7JIfbGtKfr31s8Bqiri5flD3LsaufNpV/Q10X/uB4AEDh7NtyX9uoTCkl5E+JGz849+7swof5OXZ/ifKpdLD7ZgrqvZimtbf/q/sBACpfeT5TfdLmGleZsAPUHfVJyy+RRS4hmkPFXhN/uDfNNR97KvtcE6q8JcGkfCQv9v3X9awNQt5F7nzBUVfmlhdHvI0Ia5D3Xk3yj7s2m1zHe02yLOtvx1xu568Z+iYF7Zc/e2/jssOag542md/JBhNAFC94tKF8jZzj0FYoWjsB0Efjm6fSejHthjdpX975qjZoyY6+L4pCCUvOmgkAmPjzNbnkbijU/nOvDVp+4XN/G9Hy3lmfVx4eHh4eHh4eHh4eI4oRsXB4eHh4eHh4eHh4eBBq39h7g5Zf+OzDI1qet3B4eHh4eHh4eHh4eKw3+A8ODw8PDw8PDw8PD4/1hhFiqfLw8PDw8PDw8PDwAHweDg8PDw8PDw8PDw8Pj2GDt3B4eHh4eHh4eHh4jCSCd5bO/53VWg8PDw8PDw8PDw+PEYX/4PDw8PDw8PDw8PDwWG/wLlUeHh4eHh4eHh4eI4l3WND4yHxw9C6re0jKEAAgRIDLZk8GAFzy5GI6VhuCKJRyFcV5DIcl+r9tfPZzVRulDCEEGY5kWKNjYZXqVGxB+MeLaPvwy+icpS8Arz9G+2Z/iM4vlCF//znad+zX6Dcwj0rWhmifdW/4PmJgDf22dAG1QdpW57O/oCi1Ibzji7TrvddlbqLsfktvi6DgPtY6FgNfO5Cq8KnfqWNLIEZtpOo1StVpEPKur5Csw75Mv8UWLS9c9i+q3/gtkm1U9xOiAKh77Xre4f1074Kj/jNbA1P6aTPouXg2AKD98ieHVS6j/6p9AQCVK/6Z/aKMbbxiFxqLF81dnLteWSB5TAQZp6CsY3GYn2FehD/5AADglz+/HwBw8gOLgDXz6aAaAzw/RNDEXNMI9tw6nAhv/wIA4J/XXw8A2PZ3S7JduD7auGYBAJj5ZZggh3oBAJe9axYA4NInF2W7MGMb5VolLygm5i6di1eGZq7j31Ir5MK5AAAxeSd9TfjDE0ncWb+lY/Z6wesQ9wNZA4b6VflqHh/sBgo0/8r7ab4UB3yG/l4xD/I2Wq/yJAozbSyYNZHbxu2RNaBQVvu4nsI6ruZ8UcDS87YGAEy49gXaVawYmbKmzwMAVPsAdRxrFtJvsQUotZn2AkC5Qx9b87ltAQBd31+erYGOPmqeXc3su+U8qtoxV9LfS56DmLhd5DrROgbhz04FAASn/swht/FYllIm+5K6pyIoIPzVmST/jFvrtymOzHNN9L1LVvv1c5U3qXJP+rHz3SUv9LtG+0T6+y/XkPz9PmPkd05pWr7H+oO3cHh4eHh4eHh4eHiMJIZZGfR2h5D6k3w9grX/Q71AsTVaAYdm2q4SHw8XPwMACCbvgPDOi+nYYZeokwp1ZWWFrA6QjGILwnuuorKOvSb79WuVFiUoJbUMgz20UWojzQuVxEeNZoe1LTLEbYdMBQAcf6+xKsTliXK72WdrxRi6M/P9pDKFEHj4KNIO7H2/dX4DyJ4l6vr0QRLXxji7mAxx2a7TALi1hy6NjktO2jO/dk/S0p/3dDW1vhrDrB3vvnAHAEDHVc8Mq1xG35V7AwBar3wp+0UZ2xjeRRpNcShZnzDYDdHSCSCqzQpf/ysAINhkn0xy2bKB3hX02z5B99m41QwAwjcfJflbHp1Jflr7wiVKMzpmU6PZDSxNW8jat7LeNfiNgwAA5c/em5Ane5bS+e0T6G8pLY2s6re1AasMdd8s7XP12qMAAMUvPNaoZaZc1vAJYcYYWzO7lbWhcwowRHONaKF5RYY1qg9gtL8QkHeTpTHg+dQuy2VN1dpnqY/F9+n7CwF579W0K8d8mrmfziVtcLDLqdllZ5G76nUAwBvn7AkAmHHjwmwXZrVwrH6Dfu+6HOIIuv/2PQOic5s998k3aMyJjWn8Y6gPcj71nwWXnQIAmPaTV5SoAOC1TfUDZ33Cmh5/4RtkxRDT3kUHqwOQC/9OVdzmhEzti7TxrWcRKFlpc7g5Jq21gy333fjzuzcHABzyxzfpOmucpnk2RKwqVbLsmLnMrH8vvIfWi23vyrgm8ntN3wqgMkbXneph1j5nnZQFTc8ThRY8dNQkAMC+t/6bDlW6stXDAftecn2eejet+Tv/Oceav0L1o0IZotSafnKk/DC5/lf7IP+k5ppjvppZFssjCP0MoTwi7L4i71NeDcd8M5f8DYXwvw7YoOUHn7x/RMsbWQtHdQD6RVstYBIpHwsyNB8Tozc2+8PYoh57oW4E5yRgv+w08+GiTbn9gBqY/BETMQ/zyweXIUPj8gQzGY8um7rS6Q3qJB2TiHY9Utfq+yW0/HwwC0Pio4ongbBmzNb2lfHzITCqlPKty/eO76UMkeX52verq0mrrawORNy7moUorbuMNATl5uXLsKr7rFz5KgBAdE2ng7VBiP0/zSfSb7GS/KgVBWD1fLd8KbVLH99Lul49Q3sxjX3ARj4212Z0ZcmA6i/ItaF0zm8BWYyVLcz9sPpQ8bCP1ZUnl71I56kPDnWRvhYAZFBMuh9a8oNDzszfEJcboZIp++hDTnRMMkqQ8uambqzcsF5q5fgZ9cviOcR2geC5zh6Odp9IyN8kQ6OaRNfkyJ/2i3Nkf15X23u+AQDY+PDDMsnQx7JJN3P+9kfpD119r/V4scrSc3kAMWW2EsJriMSa75wNANjorC9FzweAUgX1EPmQ4TZM3VWXRb8CPd8/BwDQ+d/ZPzjMWt9iXhi5PG5XzzLIVvWyzs8trAEFdv/i+acFEyv8MZuc2F2uRLof8P0NitF13pYPgdZKst9kQrHV+mhKr5fer9ZIu0+9sJravC+7Ha8DXPV5cQ3t2zmHHPmz02ljzikQW7+b9vHazB98A6shKqNj5QfWtqpLoQXi8KRSo27Z9nsGu4YVSpBKWeK6t2K/8zPL9xh5eJcqDw8PDw8PDw8Pj5GEDxpfj5ChMeV3q2DUjknqoLnx+ovYZu0tW+4cMW0JazJsM14anF/GdvAqB9TlgXJVkIufBsZuRtvPUdA0Nj+Yyii3Q1sxBtbSb0sXwK5R7BYhJfb7mTJ1KY2lZK1OtR/yyV+SvDmfTpSv729YTWqClKZHBkVsfd1D+dvIZuC+lZAcpKkDuZXmtFDW5ndM3cXUmV3p+PzaAD513tGquQ7NIZdVMtfJecqtpYusXWLsLBMQye1W91AWyjj5EyfmbyMAhFXItcrlp2NypG6y2q+DFRuhdSPS7LrM28OB0qSNG59UD6Jg7veYTWPHBNC/mrbZzU8E5nxhaQLXuAOGhRBaU2vvk2wlW/I87ZuyE+J64cj9qSO/HtK00MX/IDcxlNqMVWfMDC6VNKAkxex7QY3D7d+bLGzU9MSuhBtBbUi32aV5xwv30e8uZ9RrUiZoKxJrR2UIzH+CtsdtZk4sRN27qN0p/TFrX1VzTdI9JoeMZrBywfqRu1xZh3Y+wtoZtaSvE0GJmtfE1F3M3MVzHlsngqLbHTVufS1VcNOjNF7PvvzEiHxUB7Q82dD1WJWlXZX47xb8+m/kPpSrl3I5T90CHLFDdJ+2dI7SVgc9doS0xo9aE4MiJk7gtYADxFNeX+w22vMVWwJ5HWJtfbEFXROMi3IeyDXzgTEz6Q8mbVFWG1tL36i/bDearS35PDYa1E7L2r4rv/e8OJWIIeQbDxv3ylj/k2nPISIsgHzpD7S97XtoV8rcYN8ndumMEgTEyxWQL/+ZtmZ/OFudPEYU3sLh4eHh4eHh4eHhMZJ4hwWNj+gHh2gbZ/5QtGWsoZZIWicimgDLbzl8/TkAQBDTGA4b/eOCf+W+hH0yMXVXE2C5CwXw2RpxrVnhIDMZRqlsQZrQ77ybqGU/8UiclnRU1LIRL59RKJtA0iDqry5EgN++dw4A4MRHa8gMLsOloWGthwwhxm3OtVLll5La8aANP/3BHQCA009yyIv5sYqgCMw8WJVRM3ILsWAUS/ty+49+AwA49tT/SW2Wsy2lSc5DWa0bANC/kIIm2+JavWHC0Fskv7lIDhNPkdDABUWETL056xB1uomniiDMGJCvoMfo+C3136m8Fa7YpFT59eOMAhVkK4d6IbqIsCBCsdy/irZtf+SWZDySlqetI6aucrA3cR4H9GKj3ejXHqsp8ushTaMuRm+i6yK2PFJvq6MAE1i0WuPm6bvpd+eTk/LSYpmsuI1ULf9zyjK55yfqn9Ms4v1P1gC4fPFzao33Jw1p9Q/XAgDKu38kfX1CTu2xngeF6bPcLwaUdbFllNH6B1ZMTJyCFAF6+TYoS7Kua6kV4UKi5RZTyHvfHhtGhkDy3rBlLsBgjmVCX81z5UFfSASBmxinUqJc2xIoFVWvCAr40ZO0nl2agUo78qysdwe+x7p8jgcICrjpCaLKzd1LV7wC+exttL0ZratQcTBOqyZgSBY4lqfUir3+g+OFVN0sOuOmCXFYfrEF2/zHsfmvV+uwmHWwfq/pv5ra2PJ5ss6Klo4I/W4cdkyeVDT3ed/VIn0+Ztmwj8mxM3PJ9RhZvLM+rzw8PDw8PDw8PDw8RhQb3KVKx1/UhgwzReOLMsuvR0uXflE+rWoElvbFJCayKV7jmqUAPVeQNaP9K4+rKwt43875NZ9JqHZzvIhFo3vY7hObkKe0eBYrh2H2sVhq4v7cLq2clOhOU467NENc5hAzWAWa5cwVH7FmqDmtkCsZ3bBom5rA+kre5pIXaZfNvJSGuIUpIyTTnpba0u9nkwmi3FSUipK60GL6kp1MUjFn2dcuvuMWAMDU45I0i/HkhSIoQKqYF93PwxrEJvvq43GsvJuscONyJOB0IZFIURQQLiLNdjCTqH2llNq3PPL8m2VTy+h/jVJzfSQT4jE+gz1R642uT87xs4iSaRZ3PkjvSvPBzy2fWf2CYpJGnGMIixUd3+GypmjDYFjFBDa8Ktp1WWZNfgAxalqizOS4EohbOk05EhNb8/v/azr4oR5IXntUW2XvclMXNe7s9mgvAa5T/xp8ZEfLjx/Re5JIbmjL0DEshvXOxCAqD4uBPnxw1/q0wWkQMw+E2PzwSN1M7IkZ81GPDbXfmhJ+cT3FH5x6miNWrllY/fKOG24HABz70RyXc52teb7yxb8AANZ8jhIxdn79hbqWHJIRozhuANfzdVFEJ6x0I5DhYdjhg8Y3DKJZs9M7XDhAE2eW4djMy6HkF9p1hBkA7nYw9IeGNTl8/1FyzchOIucAv3ywe5LlfnDDg5RDII/5ODWvhp1rhBcVfsETQn9MGlZEgVLKo3G/EKt9ikM9djRSN3I9SGmMA/GFNlKm9YLKZAaN+lbQkt39yi4/2u9TXuoK+YdvJmpPISA6o5mb6y4obz6buw4AIDjwv9H4fOPppuQ7y7R45DWNs+Z0b3XeaxnW70TuLOmxl8EGH6hhbXgWSVddxKYHUBk/OJ7+PuMmK/+Gha32zVdWnIK0NpSeOXiLvXPJz1WXg74Y3RGj52T0XU75NFovfiST3GD/zyXLSnmOOl/LlzP2Vw6UdrmsdRhlEAfq2sHTxkWWTwoxiT8IVB6QSF3bxiJygRDmY0DnhQJC5Q4omKaWXzKDIqa3NdFPmRa7w7inagUR98OgoPPFyKdvpn2z9jdzPOc1KVUwZvrEZNsYyg1NtFJbpa340wHHoVakxF+kZVBE+9Sp+dsIRNx6tdwGtOpGYWfquWx4Xjui5VjjcvnA8L7cjvrPfyb2NfooD3+raJvPub2u3EZrQl2qYSEgb7+C/vjEfqkyPDYM3jYfHB4eHh4eHh4eHh7vCPig8bcB+lcCAGRljPNrtjBqbGJfHsiwWkc7SRCd4+oeayy8lghq0pS5EU2IlbRQZSzmQHoAmJbC0JcWoJWoC2DUrdb5m3Y0YSKPWABiWk7OQt45BfKPZJcRRyttQxgC4CRwRsP6gT3qZ1K1M7+b8qMUvC6tqt1fjt93SuJ4Gsy1Dg2KzoZd1JYbGct0CiCSmTko13claWTFq3+dcb8QheZN7lQ+J/KLu1AA8tYLqJyTyNVHBkXLdG25gOx3Vr5yWbPK/UAUnO52+u99m0iMV69se9xwm8sua5nBUEq0rNPdLZbYzBXkayOsroP7ZgPoep1ONNqyZymwXBFizDzQnLOurghBAZITNCpikASZw3qAlBJSZQQXOlg0OTcBQMvxn8knu5+IPOR9XwcABEdekdpPi0d/Kl/lmSzF4fIbsSQzzfcCojgWU2ZDVOK0pAXs+y6maHYEx1tkIVouU/AqF0D5wm1AUT2zbY6Pnl8bwvZ7bI7cUJb1SBs1pb2ygr/xGKBIRsTWR6vrOkyf5PODInqWEFV5m2s8xZPlWSQXtruhrJNkFAD6lxBBSz67dKzYuJW8NqitKI3c7trX89tYe3H9uRx9dTeij//CE3GSmygKKZaNNLjeB1wIzvhVU/I9RgbvrM8rDw8PDw8PDw8PD48RxdvTwsGUsXVQ6yb6yma/ltKsGwCA7hVNSkYk0MsETZvERia4TZ8FtMZ8bAEsayL3IBDTDuv4A6sshYV9Tfhz6sBwx52/h4Jq5ZyzII65mvbxfQ6rwCBRDjLNIQCUu9z+1gBM4jnlz0xaMlWuRS4gbcpPC0IIlLrS+1E9OLWOdp9h7a1OdhiYeljnhYODdctoFENRtx4ubXoOROWJyI+NVU8+DAAYfXLSimLXQbCPdVawlY01rC2djjZaVp8Gc0EeRCyCcU1ZnXvZ3ztUX55rHMQCGePbcQwO5KMVbgRnfBUnmRu1USLJGgBgrtI67npaU2UJEUCqBJlOwo2n7qTfvT6ZS3698rT2OKxBVJQWPR40jNjc8JRKOLZdMhmo02KsgpzFoRdbsuLP0bImPP1H2syacCxFUxshqOBga06iWihrba+eh4TA0lcoUeEmrjmarSlCmmNtMSv+1J2Ah39M24oKW3LMkxBY9RolWGxFDmirsHVf2Wqy6Cn6c8pOJoZj+UsAgGCTfR3xdAJt45OkDrqoFa/QMUW3HQUTOFQj87N9DAAqE9xU6HmQsJJHKPGl4zyDofVg7LTXlJ7q+gtQvuAsFS92/QepvA//0h0T12yyTI61CWupnh1afj7pGw4+aHz9QdYGM5naI4wEyrTNWbJF2zgUZu5Ix9Yl0yvL50lAZfQWbeOATbbLLSd89QG6ftruxnWKFw7FyBGuXQDRpbJP68ncMSj7VuBz37uKtnWeAGshXaCCzBWvOqQ0iw+bqqsDxqVKZ4s2zDkf+e63crdRv0wM9kByUJ/aV53/MgAguOVCFM6+VVVZmbSFMHWwFsTKu8+rX9RalUG4Xb3Q1oaci3R8UjP3q4jSUc3x/jcMWtMTnutl3Fxb6+1ep3Ib1aPaQ+xjzfE41Yf8f+x9d6AdRfn2M3varWk3jfSQEAKhhx6adAQp0kIN8ANBpYuCgISiiA1QEMEOKlIj0gSxIdKkCNIFgYQkhJCem9xy7tn5/nin7e7snt1zz73BL/P8cc/e3dl3ZnZnZ3bf8rx+BflS6Fr7ZTsjVSneLc4KuXDIPB/jqgQUN2STb76ohMeGv+xdqnPgWOUCxCRbTaUbnIls3TIL+vDNMOWMM4XcqPsUX03uA6yVXra5X4k8z9yvKLIGlQ1cfEyzhoEYfcIXMvXP7KONuUVnqdbBuCrrNLi6X9K1jRWbE3OBJH0EWz+qQqx1jHlAsTdOKhqRl7mch4Vn0Yv4qJ+RaxXdA8tH1XE/jZdre4FZTS/w/D/EHORtd4qlPcaH91E/TNUHBelayCuGwil0zCDHYGKbcw7+wq3UJvGByMtdGLm5cHnqIHfkAMucUPaYbkeRl8G2jYCDyX2MP0SBvewgWoPQ04Wh06Zm6R1BBHLzxrboXCafu8Yh4Pd/nuo7+BtU3vIcobwWXgLbGRs4Nvh/4J7qD63YOdXvAStkm0mVeyj3A/cHAPjKDwAAjx61HfZ/VMw1hosaFwoXJhm6/B588Y67hQzLXCOfa/nb0wlmIUjg0j1bfVDqNX/Wr+7J1L8sYMf9iDa6aKx1f3dvFC/4s2wVlWEe0P4R7RJzZmr5xvzCRR2sZGEVk/1Py7Lo0K9wLlUODg4ODg4ODg4ODn2G/nWp8rLrYqXJ3H/yRvp/xpnAVofSdh3MUVKG/zq5FbBtTwK2OCS7nAm7CXmGVkJuCO0Ha9laa6CUa0mHNt/L/jQOwXNXXgQA2P6BqGZNZWm2mmtlrgwj14CRaZU2cnj7KnJvmPJwFisAV/0x84gAQOGcB0W/CirvB5f3u2Jqx3Uw4A8/T4GcZz19XKQmNnJL0WZP9yfUV+v9N1za7voiBTTPfDYD8XiNsLvXBGmE652/o965OZRcL4fmXT8T3mkty98iVxW25THpZEs5Y3dKVZ6/Q1mw2Xanp5QfH/jPBk/UZRRBgxiPuaI6hw3bVJWbfzsFzY/d97KovJCWjnm5qEsTt2hrDc310jmkeR/2mWiej3gk0G2bRArFEPME8yggF4Ym1q+gaz65o9jcZbJa3XQ5PfbLC8myVGO2j0QM3WRyqOKYZyJrZuN3/pLxvIzPtnI3jbrAyTXPWgtj4C+RixrfXLiHFRpQ/IywFofvOaBobhMpSw1Xoydv/jkAYOfND6BjY3dGbv8vJvUmrrXi1wjkl9r7QRNEWzzwSTuIY9S+RadPwsjvPkH7pKte1yo0nC4ILEQOH+nOyTlXGdZN6JwYwi3Sy4N3C1dduZ7I87w8Cidls1JJdzcz8FxdW2Fx2f+RhdoqIYPHcwVwkFVRu8/l8NtZRwIAjv3rwmhd8v6IX9M1OVAupNnnhlvbQ6ceDgA46Mka0sZXg3wXEdaH4pceQ8/3afwUzn1Elyv0Lr8Yr3Qry4Yv3Oi8IZOi7fhfwXrGUrV+9dbBwcHBwcHBwcHBoV/BeD+kZ5R+hfZAWe2vePk2pHW8m7nyRgAAIABJREFU/EWRDbQKfW2fI0NArPKdhOHvLDUrMrlYsQVvH0GBaRvdIXyOf348cOR36LxB46UA5ePujd4uWpeQZ7VwyARRuYIOhJVf0Ya2w18oshBP3jd9H9uFz7otE3e7yPjbOBiLziB/4pE3UlI4vvIDnUxKJoHiHPz3lFzLO+zaqDzTxzx8LKXFwH/4UpJ/xM2J5ZTchHGaFsrXFgwdV+0MAGi87Gkht77f95mTjQGAzPBdBeFncZ0j7bOYsn99Bf+v3wIA3HUljemj/7II/M37AQBsKlmNrOMgS/B9yj5ymXitzhS13d/bGwBQXkFxb81XvZjuxD7oo/84WYa83bPR3laDnH9eP5qsYtN+tzipuEbKPvIPX6KN1pFqPo0jwACgYhjh5cHnUwwfG7UN7etuB/+I5lqV1d60rqq4OcP6pyjGxXzVuULFeKy9kuKqmr5C1kv+8JXA9kdT9dOiQfexfRT++rYYGR1rl6O6w+1TFiDR73wjHj+YrAa7P7hQyNXrUBJVvHwOSG7ouFxLvBxeO4wISqY9mjJ6W4xRzv3g9aZKVZ1XbEuZ3mc//i9ZKfgqilFk0tLaOkrFfXiDxkX7kGLNs5GMmDTtUj6TsZ9pUON8agZ3v3AwXdfp96d8hlLIBvS9Vu8y8nkAss016xD+zQeu0/q9Mx7q1/r65YOj3i8B5ev2AwAUznu0rnIlZGZe7/wn0p+U0MdA9uvQw0IBckE3EMZYzS98wUzb8QxO39+RTNXn/DsDS05SH02WmAhTCo9MyIwxfH06teHSF6Lc3dYcB+pYPNuHKf8HO5H8s19O2Ue1gEQn7rQw6+/4Bi3cjZc8WZOsJPkA0HUNZVNtuOr19ALSvqyGmHDirod/6ywAgDfr1nRyZfslS1WhKfFa+3dSQKl38t2p5Nd7rnnzs7RYTp1TfbG0T6XancTWz3kn0svIuHsy0NKl6GPgRVPCwlDGGEPlJnIhzX3h96mqV8+6KV+9/AXdvRhj8G8hVw7vvMdTyQeQfpyG51Pj5c9E17f2AACULvxbJrlKSWPJ+WM+6z0/oBeH/EXPppMv8haBc31fLMxMCOfK4b6eVw0X2Sun03px2fOCbCPAZqfddpT8UD4MgAXma/MYyxVxnVgvzsuwXtiUcOqYr9169L2Le35oHciyXgQ/AmRfeegZQOCD48adSf6ZL9W+Xtj6oI5ZmNTM/t26G9V/4uMfBs5LiyQmNcY83LsXKfsOfzqDS1Xq9xqtaKMfz+i3zoPi30Gusd7MW6pWHbiu8qPRzCAfur+MefDvIRdx78T/jXwc69sHh3OpcnBwcHBwcHBwcHDoM/SLv5L8sudL3gIbOoV2lqWbkQ6YNr+EAfFVK7fnkZbYG78r8qf8VMgNlU+pmbZpyP13icLN23AvsGNvzN5H9QVeiLZBmOd5sUm5V8ks1ah0K9pcDBij2nfkeJ2J1mwnAHAReK7clGCYqCUtrqlFkdpqScuJHA6ZnD1LdXKwtlFfNwWNa8pUptzAYAS7NeWl3Kg1I437UbV73VLj6O6NS1XgPpXL4ldQIPYyYC4sn+X77vHl919IdRwq8qv0dIMVLGHFu5xQWwUpM3FjuyNrk29BwI3DliNAljPaM+W4YyL7VDlJrSvdIngFfAll8mYDRtG+cocK5FaBnjJDfa6IMYenC7YPtq+69c+0pgbJOiR9rhibLAe2Lbl6+SvmUXHh0sErZU3pK+495xyQ2nlJdV1s0Vp0Rcsr5mTkgOkhAoI6QrmJDJ5AvzHzRlrLhpL7+u8AAO9cTrTFU+6JWrjMsZA/O6umkM59+6iR2OhuobEPk3vEtU26IJV0XorLnnwDALD2ih0BAE2XCatqQCNsWNLD9M2VbrWv40qS0XgJWaR4TxfO/esbWTpHCI0FE6ZVQ7keG5YeaVHivljP/DK++n8UXH6DsFyf9fQiUTxq1bKuJczyHKsqOb5w2t7Z+wgQSYoM9JdujHKtNWmajblGrgmqAYVGHDJd5JR58gY6ssvZSkZS7gkJ63zQJSmWW7DPNkMix6shyVUtME4jOXLKer6TlojOVcBIInmIzDXWnFPG/2IM2XKZBbwnxmRPabBOsZ7l4XAWDgcHBwcHBwcHBweHPkM/RWRLqtYGnYxKasTElz43jnEZKNzTDRRFEwfoxD5KSyEtJ/pT1/giz9Y1NmILo7k1hLVI/1heAZi00AjNjfL/ZUZ2WE//yv4avrbz1lCfNlXxEIafqyWQWn/uG5qIcNJA5a/KsHQVtW1Cli6m+hrnivpO+7X6gQB32sexRrnKJpMJ6P+r12+2sb2+SZwzg1eCdIj1l5+d3tCqqQ8F6HPOwQ78OkRB+jV82AOWxfefo30yWFVq9no6lWbdmoTLRumq2mjc+/kiuHazozL104q1FOSMpjZtCeSmJjRI3QmWAzbaydpGAOD/uImOHXAV7aiUNUGESdNZDCbJ1MQORWDyjtn7ocggbMfkHJLXVpxAgkCxYc4TKwUZxMAxQVnMU0njuJl0UyZjNGMCZJ9U2+QxH1gRpfmsF5hBMVxP9PzxxwCACTtv1Sfy5X2afPih+trJQ0IzHLhv5lwespRyzsFvJ0tM40ZiHZPrbK5gpQqNZm3vUfes8UIRG6kCt3vA7yDfePZFg+K0Ksw6YuZu7muLuLSYVcp6vpHXhnn41yNEvvHFe8LWJPv6odYOaU0IXIdozMW7f6Akj5OzMgAbCehYKDltwKphzL02a/ed/6Qkx5/79qmxMrLCTI53/3PkGXF8hvP5vH/Qxujtg1ZOGOu732OJcTL/F+XyJWALiufCkjdpn5xzuteq975g3Ia4t3IcFJsj1jmjtcDU/TP07hMAR4vr4ODg4ODg4ODg4OBQH/SPhUP63HUsA4SvLV9AmlGZZAvMU9pBqbXiXk77JQ4YTfs4B7pE8h7hJ20ylGTVgis0av9GLjQumbzrpPayvAZcaDT5a/eSnInEJoTm4Zrmz2QOkb6OMtai0Ij9vkQaK6VhlFaNzhXgi1+jUyeQXM65liu16X6P1lDJtonYD948FFufc16W3ul6BOKtHUzTKl5O9I1mckOTOesLO0vtS1TrHb76jHmaOUmOiVwhqm0xtOMnzRiUtmsRxPnJZ2GwyjUL7avQTvKyHN8tcadkQr4puxxr0q+QxYwxBl4Wvr/ymKERD1gPVxL1ZdiKE7ZoadlC6+jHm58C13xFlJEmCYnxIMLPmnk58LKkjC0EzhMF9Pa/SevJNz86KnfbY4Py8yUwmahMUWJ3KYunvO8cRl2vUGJDpExsGGlf5JihDVWxW/pe6OdEisqBL4+xQAS06Ua/pfZcWReYYdEJWocZ88BX9B21Mhfxb6ypra5yV71LScWGHFVLwrsUkPdwr3P0s9Mp1jX1LFnGMOeR+88Yw5sPUQzixt+9g3bKeKvyWrW+cBga9tA4gMlqFta+e3m88yCN0ylZLoeKHUrwGGCetmzYEgMrC2sRH3eK7Q9p/YOFMl6fZlwjMTbNeLFoeYZlHTUSdhrMYcnxerZ1TqMoH90aEiSnQWMue//Y+N3EVsLan1JLz/Il8Od+Rf9s9mn6ldZy25poxsMaFpS49ZcxD/yF2+mfUdNTtcmhf9E/HxySWnODrdVEwMZQABiXi1dPp3rpV5OCaZ70DPP9n75Pm8f+JFANYx44glSQqV8QzYfmcXKVwIm7Vj9Pnq5eXBq0CXQyBaExgxOai3y76iOpp1PzkIssoYx5eOxaasM++35N7QMA3tgGT3xoqLoZi5rNc0V9beXDKsow5uG1G64DAGyWJcOx/EiqlNWCptzGDIrMxm32EOWEGbR7DXj4xYVXUGgMfmhx2Qde0S/ppnuB7IdJqRgJNNP3MVfDCzl1JZoduhb2aPVSK/rFjQk/6jKWnYrXL3dlblNarL5iDwDAgO9QsCg33RxM7EJZ3DMH2uftmXIj2OnkTGKT2qHuA/c1gYJxX03OeoW9zoyV6w3bJCjDryhlBUQeHTZ+F/Dff4XK7UO/nlC6AAB2/3yabgX7kdRH81gxmFUcQICOWKGg54WILFseHHmupD1tGQH1XHeRW4hyuwKAks0FtE546R763ftialulXJXCNg2GnETzLt76R6rySn6q0roka91AzwEiD4ZcD3jDICPgWQQeW0gWOOd4axVtbyw/AuVLa2kA+Mf0DHvDp0VbIWXkGxJfHP+7mspNSd0/A5Wy8byFqJptSsKAO7Q8xvGOaMP+IsN64NlNur8ht+TQQVXPW6uo/9sndMUGvujfeu16kFxR2Sm3G20L5jzhXkG3STwvrHEwZn3nStonnitebFLnsRrzSkglHcuXcNg138wuoEe8Qxguov7b5FbHJlN6Asa8SPZ3a1v8CrANucbKuVMd61qtP5It98maayMsn3Ngq/R5Yj4RcEHjDg4ODg4ODg4ODg4O9UH/WDiUViOqkeBL3gIgTOLCbUppXXq6NLWamVjmwMuEECNAUkJov/2FIhvr0CkBd6lYSPm5Atg+F6bsmAaX5vBKN7iw6PDl79E+SQ/r5ZXWn3PFCQv+ImlD2I6kLeY93dj7OpEdWwbSS61i10pwaSI2tMSRxDuAtpwoekxhVcmXsOm3fp65j9IlC6b7gvpCN75ddz9L7BJ9zJcMLZMMAiuDeUbgPLVMdka5kqEkEySWteZDaKJNKkelMTcSZJnJpbKgKgVgWjk5QR5goxb0gwHlwQBRXT41FXEdwbmPlgspCZwKuORc07ua1hmRTZgL18hggHJUEyvdIXmHGJsDRkFpe0PuTYwxTcFap36pdttg0Yyz4fE0i+FM3hzAlfvuAQCY/TzRtaLcCUzclsoN1LTX1A4GZtE69wZmpmMbcQZX9JLGeBq3ZVV56h52rtTWEanNZDlNL/7nbwMAvIMMbWqC/N6CbT8ruCMmwDbz87uS3MCWPf1XAMDQ46q0I6t8aQnKNxr3TBAZSDexQrNhCRTrJQcg6Jh5q3bfO/iInemcoVMD5cG5smz7c8law0ZuGQ2293s0gYvQsLPBE+lY92rsd0R6a79ChyBpMNZf7VJpJMFTQd2GJU66lUkrud+DPUcG1xDbvIKGQWKfYTlRVPQNQEW6LYsxLNcSL4ddx2SbT9U8PXJLPbcLy4a/jFzy2OANdd/lPGEm+5XtrXTjnW9fCgDY6E4a04H13bBUJLYpbKU15uz/fPMiAMDU3b+SvpMWqmZvo/2FTON+NaZwaWRMu5yGD5Va4S+mBLbe8E0jx9f8gK5J6zWvJM7jtVqC1hlc0LiDg4ODg4ODg4ODg0N90D8WDstXstoeQRpE/vYjYPJrv9gSKae0CYVG8CXkk8pE0phAVeKLno0lukl/xVywNBYOQxvIlwqrywbpKRGZ6bMs94mgtkAyLhVHIfrl5fDRb34IABg5Q/iLFxrR8xjRMhY2PSwotGGQNZ7Aqs2U11MGmQvtF8uXwP9+GxXa+qR0HQSMJI0Glait3g6hoZNJfby8oZUSGq5cAXPfJO34ZmGtTa6o2q7k5wrgXcLqIbWDrSO1RUOOD6XR8fDRO6QJrE+IdnYkJU2yJcNSUAmzckBSoGGNFpxqYMxTVjoZ/GgGeQc1S/EWGFsiJy6tfX//Af3u87XkhE+1UFTHwYgVSquR5vddQG06/PuWg6G2MQ+X/fy7tC2tgV4ObOoBwWKmFehBij1gx/wiVXti2xlJaJa3J9S0aUjXLo+VG7k3DQPhf0D0pCoZ18ittOVkP4p9CIz99qUZe5MBEQ1hfXyiVz/8SwDAkD37KGmh1EBzH+oZl9dQWPqD0Fp9ZdmQ8HJ4+N6nAACfPlpaR4QGnzEtTyaKtdF0M6ZIAJT1Qz7zhSb86d4nAAD7npuuewDA33iQ5G1zgmp/dO1iyndfxzV2ReI4+bynsMmsE2Laz5X1W81Tfo8up5LOFlWiSJlkV65LfOm7GHdsxnixBLpaaR2i5yf0DBmWaxU0nSviyY9pLG9kuT/VLBuqXNhKKy0e+RJeWEbyp4ZPSpIXQ/4BIOgVkjCf6nvOwB+4hLaOuilSTlo2wpZjAGi54qkE+ZqIhj98OW2d8Jv4djusM/TLB0dikKOcZDc+CP86ZDgAYKu7hSuSyUxgZDlmI7dOXbc3aHzmNgZycmSEGfybKlCRc4y8SWZx1ZPxbff8CwBwytm2LNw23vGgWR7MU+4FzGKivOe3jwMAjjqnSodMpMwnwUaGXCjyDYHgaIkH5tO+abZgXVuAnwxglZmamWfM5cHyjDE8SslM8YVUra7ygVADStPicyxE2K8qZeUiyIXrA8bvohlmjEmVzmfIb1JDDoeUYKHnJi6vjXbhSAnhPvHBr24BAIzf97LkdgzdOJv8JFm2TOnVIEkRbPLCLwHMA5rJtYC/Spmq2ZYzgWbJjmW5hgnys8D+wZeSPWbHDAxZoA8MAHh/1iQAwPgvfwtsi5l00JJ7hW3/f5nkxyGSO6J7LSBcNAIvHXJbZZPPw79VuKmc+Ev6ZXEvwHTewO+9FZDLK906EDuUT4UVGuHfQdfQO+XedJ2RY8HLR+dG4yWPr5xPv/fRixqO+G7URZjlsN9BFEzLWkaIPht5m+S8nTNcdETuHVkXf+ePYON3oX2KNUqzHO51cNZQagBDyZ3ItnZxGEpI9bEg5l/zuZJtH70d+Av3BtulZHrwX7+PtrcSvm9+BWBiPhWuXaynSwc/S4WAZLpsHa0JUOqAas+ejX1pRXfoWEokvWeY17KzBv1UkiuXjZzBhoCCZcCw6uWl61m5Q38QSve8YrOeM833ACm/LZRL6JMOFzTu4ODg4ODg4ODg4OBQH/RTpvF0UF/gUjsWx0cdptarN+rkxpHGzQSMRQOeAXywRpZL9wWsNUeG6d3QooXx4doavqzTBjiFMyHzCpQFwrDANMlmWb/yLa5EIc1lsMow/z+Dl7GL9bJsKHkHfT19YS+HK6aTC9rsJyl4DrmSMV4sY2nfS3rZwnikdjmS1qmUcju/sy8AYOz1f6Pzq9ABZ5WfFlUDyCX2TG8CZIzhxhNPAgB88Z4HxE5PWTasmsjd0trfakdWStjqAunaTfg1mRCv2HYsZr9wfOBY30D2Q1gEAlpXM5BYupXpe9uzlNwri6nmZJOsQdQVyEMjNPJMWyL8JQsAZNDgheTbjoH7irIYnxGWwJzZZz3n3TGHLOLHnx10gSEa3XD5KI0623BPvd7KNinXzjzun0MkLId9uVrHDDQPjz1kWpjC1z+YT8sIfl/+UeDcAEZuLo4ZQel5SVAiTAcNgwDh7qvaJrXljCn56wq5Wh/PcA6uGGRdD0lmnXOCbLhD7KHIPFXpBhckFGz/y8UxT7/X2DCm7wgq+gQuaNzBwcHBwcHBwcHBwaE+6BcLR1oN2473vE3l/yK+amd8XiXDCwRef/Rv2g4Hz6VtT0xyKHV88ask3xL7EHtOQLMX8g8OU8IipNlaLbIpC/9bANhqSJhK2E6bGt6n6sp5WkMlg6wN39/pbTVoIo1M6GEE7rEM0jOShoXbzHs6cdbFJ8bXJemApSaOMfC5FLi46gbylx743dd1eenjKWjxuF/B5y45NWXHon2pVRscGKd/pQBiZolTCGvYGfMw++n/0LEV76t9XFI/N4iAT6mtyxXB/06BzCxD8sa0z6IMvoyL3ZBgMhA1JdS4X0rPOoZMimqTTdpYCxlDrQhe83T3d/U3DwYA5dMflBe9lmf+6RXaUEG7nqI5tdFzr72OEmG1XDc/VXtqQdWxLLW8KWNcwpnpZz/7Lu7di+biz94jAsrNoNm6xamEEhPmCto/30gaa+tv4bTbYmTGXBtB9/zGLIrn22TOYqsVVSJ30s+qd8CEEWMh5eiAf3nM0zEHH1LMHR81XWVlZkbsw3FnHUnlJO26Vdtt9DWkFedz/4GuO68AADR89XHRDOlLvxaHfPGYbP0DInFgJuzEJ9FroooVm4BPfzlwbiDuUsR6yXmLdywDE+vQ7UftBQA47pE39bUVNPaSqAb5ErD3WZn6l3ouTVmu6KUvHxg3KWMra9Eup40DSwtvyqcT6ooSVLADvwEAuHybDej3xQ8TPRG8yfvWoZUOfYV+ysNhyZdhK9ZOL95sr6/S/w9/DTjoanFUTkC+YtOoFSxXSA4QtmTXrQq5qHp5wxocDC6El1MzKJcfPD2dOkhPBihyhj23Fi8nKg9Hoy7TvTYow8vpc8VCidZR6lwVgC3rho+tt6khuErmGrG+mBgfRuGJslI27MWGa9XQ8YF2BaRJLnr5wVFqAbooi/OAc34qqmJAO+Vp4B/Txweb+CktpC3KYpYGvXE9CZzbFr/gWidyyQzVRvl8+e2ngx0r8rH4mm1Egg+d0Lv2JaEsPhpLVTi+0mYMFyh+gdhD/nsaMbhNvnNh5BkMtDGj/DQwM8lXW9ybxk5MkiR+NZMef/Nh2rOVeEHrXqPZgSxoHJ2O1CJQq4XFJVKm3KGC5HXOA+1uFAjkfINyrrCtT4iXJ59RqUwAlCKDr12Kwx/7ILBPz2UAf+shkr9T+hc6232x5XXhr8yhg9uJwHReUetMoHwhgW3HVr/I67DxQXsLWX6U6MH8gM06TqUs7iNMkMENViU11y4nly0MGgc0iIzk8gPWdAcSme7NfARyvKg6WU7Nq+q+FhpROv4a2paKpWJelceKGvLhJLnXKXamQiT7eOxL7rzn6bhgfwygLIPg6ZqwlhFKCXXEfmI+feYnYLucTeVknif5bsIBLHiJtkWeiWpIO5emYXBijGFUU+gZk8+vuT7KZz9fCj3XoOsdCrw35Q9vqCPjX41I7coawuy/PgcAuG/vETjkMfGeaCXPEfJ708j+hAsad3BwcHBwcHBwcHBwqA/6KQ+HNJVGtUSBYkKzq75cD/w6+M+EpnDaHnRsxzMULaPMy8CqaWFtdZm0gYDSeLBSa5TWNQX4YxQgzPa/wk5RC5DmKJxVG0xbVOSxjmVoOVvw8kurgsyW2t0OpaWU/fYKSlOj+Nc7V0TpDaX8Sjcaz7glex+Fqw9aRkS1C9KU7eW1G4BEoVFpXiTNHl85H5iyD22vXkjlBkqLBAf+TG5CfMoM2jVoHNiE3UU/pJbWA2+h4D+WF9pe6SrAPGCjvTL3sZ7oepisE43TZ1UpSYhQD57wC6hxYnFLKf/xJwCAUgbNcVqkfqZSmvOVXJGl/tEFdA83qhaon1G+SY8aGaNS+14aoJ8XWx4ZQRnKis3I7XmSkGuxhAhLo7ICVbqBsZRVHNKNyq9o9w2RadxfQe5TbNB4sF2yu6rEkmkYCNCIW65xwGVu9cfV5UmrQq6otcNyHmrdQM8xYs7hy96hpg7dGDyF/Lj6kvYxxsDFPQjTrUbKlwZka8Aico3jnTQWPOZFxkBgLcsqXz7PXavAhTVCu1RJucYYniGecV4Bv13QGB8nLL3c1xalTuk+22bITKCll/LH7qTnaLGWYX9ysYLfY7VCV0XHCvqVa5KJwHOXRBsvvQS6ksep4ZoKgAgFxHza+TFZf4oTd9T1SlfNsmGxyzhO/Y9ojHgjNrc0W7gz93Sp/qsxWinrNUwSSqxehINuFrkjQp4EjDH1rsPfJDIKPmQC2KjpVK5rje5Tx8e6/4B2Nexcif1+1H+5KbhfsXuPrCbyBlhzzVjkyPtfpGt4yL0voHzdfgCAwrl/oGNmnjfp4ZFAWPCJggsad3BwcHBwcHBwcHBwqA8Yt2Y+qi/4Gq05SOvPGN7336MpKHHyXR8FNJD1gr+IAtG9kVto+SmTBgKGlt4rBDUZACD8gVEaAMhrIYPVOpYDTcIX16AqbL+IghVbvin8SqWfr9+jg4qHTBKlmdYESf9PLxelwxUyGGNYM5v8YJu/8176PoprxESypMAxM1g6HCBuaDtkOb5yHt44mdqw6V1zqbzpsyr9juVvsVlviz6yQNb1cHA9x9wTxgIAJszpStfBtUvSlUuJtVcQBWDjZc9Qy9JSzRqPZFISya5v7QEAKF3xaoZGpesjF+WkL3hcMK7/1A8BAN7OX0wnVwa1CgIANnHPxOvi/5O0uN4eF6WSn9Q/03+Ym3SYgNA0RTXoS8+aDABou+EdizzbfBVM0IiuldoioawOOo5i1QWUOHHATRnGXp3HacfXdwYANF4an81XoiqNcfjZ51yN04YrX0vfqJR9lAn9vFm3JrYvKzWw/8craWMwWaW87U5JLK/kN1dPbAYYsXbda8AasllHnj+YtLfT7xLEC8UW8FXCaiatCUYMR6ZEtADu2pPIS46870Xa0TICfBWtb96obVK305fPeNsUYzxIanPRFi+nkxDK+X/FPGUNlOCrFur4FJncUM1N2sKjni0vB95JFhb+9h9p36Q9dcxPo0jGudkRogIffDmthd64Gan6x9fIuBaLNdUsZ4lbCMetgPt4eH/S+n/6fiKoUAHtFlngPLKmBuYhaWFR8UEcfz6AAq/3/nuGDIC9mGtssWZhkofUsuSa2LUSKJF1ynwn1PIpoSNr26i2Rvcz/F8evU7r9066s1/r+0Tl4XBwcHBwcHBwcHD4/x7rWdB4/3xwlIWfrcUiYWrEfvMp+gI//m/EQsC71yith/kVm2TZSMPgYoM3cotU8mOhkiQZ/tXK/7BVHeMLSWvENtwTAFC59XTkTiJffDQLrVSljObPXa+2AQB5oalgHljLBrKl9JcxrQEQ15p83yWFqrjNvvDzLTSicda3svcxyd9e9j9X0Ow1UkPT3a6TdOWkb2kbJm45QcgNDkPGGHyhsfOEFYdzH+gR2nHpH58rRWMbDAazUVtNTduzuiEQQ5DTFqUsCGqqwtYiqbnylPy+wJzPTAMAHP7nhGRbSG/ZkFBUm4MM2tQEeNvXRm1srdvQMCbRYpsY8oO3E+SFE5Zx8A+IFnbNTWcAAFqufgmrv0LXsvVbwkJozE2t33kjVTtqgYyXisQGhdDw1b+llskYg//3awEA795Ic0gO5zNuAAAgAElEQVRgbrYwjpW+/Fhq+Zmx28mR+mzI/AzuSjET9x5Ic8gRf0m2cGRmtgsxEQEhNiK9V/zoeWWbG+6lTWkR714DPPNz2t7rwsj53A/SsgfqUdTpq1WswRFfnEn7ZDzOI5frF6PDf1S9b7KlQ8g6aI4JFWMBQ8MftvwNHGs0X1gHBozGh6dtCADY4KfvB+thnp4ncwZDmdCEY5BmK2RbEg01X0FJKzXNeAnLrzgQAND2sxXpOqiuo48wA6dmkPLA3yELC8Saj0pZU++LeRCNg3HAF0TyzHAMJAAu36HEPeLgug5Zpmu1jtsU7I0qtqjYhD2/kC6WsF6Q85y0dHnjd9X07lllyfHXMEi948l55z+Hk8Vvyr2LM9O0O/Qv+sWlqt4uACpo6LxH6ypXwr/5MACAd/4T6U9KdOVIcDey5O9gjAV4p7MgUFfYvUP+xxi+vyOZI8/5d0964Ul9NCbYSLA8uDXHwlXTqQ1fe2GRpR82M7SFu93y0if3/2Ankn/2yyn7KPpXrzwcHd8g03zjJU/WJCtJPgB0XUNB9A1XvR5XPIq0LlUpP9zDLi1V5YY/jAtNyS5Vd36e5J98dyr59Z5r3vwsLWZT51SnBa02ldr6OfcEcqMYf29Ktz8gVR8554jQkgbq13NNz40HAQDyZz6Yqno1r5i0nIbrRlh+5UeHAgByX/pHKvkA0o9Ty3xqIyZR7ocX/i2TXPXCa/lANZ/1nh/Qy2r+omfTyZfuOJxHtZyBedvItC2PKXpbnbfjiunk5jr7eZHPxbwGlvlY7TPrChOayP7lCrh2B5pLz38l/XqR5EptvizbXIPCrrKMeZnWC3McBNy4Itdau3bduDPJP/Ol2tcLG3Wz2jbXSN0A1e5f7Eb1n/T4hwEZaRHM0B6Vf+9e5Ip2+NP1cakKvtfIa6ZdVHW/NUGFfwcRHngzq5PWBK6dSi2gP+zC95cxD/49pCjwTvxtVfmfBPi3zlyn9Xuz7ujX+pxLlYODg4ODg4ODg0N/Yj1jqeofC4eDg4ODg4ODg4ODAwDAv+3YdVq/d+Lt/Vqfs3A4ODg4ODg4ODg49CfWs6Dx9cue4+Dg4ODg4ODg4ODQr3AfHA4ODg4ODg4ODg4OfQbnUuXg4ODg4ODg4ODQn1jPgsbXr946ODg4ODg4ODg4OPQrnIXDwcHBwcHBwcHBoT/hgsYdHBwcHBwcHBwcHBzqA/fB4eDg4ODg4ODg4ODQZ+gfl6q1S2IPybyDjDG1DZGqHoxF0tcDDN/dYQMAwJf/uShRXq24doeRAIDzX+lJf1JCH2sB7+kEALB8Q/SYvBaiDPKN4LedBACoLKNrkjvgTHTP+TYAoPSlP5CsUouWUe6gfQPHpm9Uij5yzgFeEe3rol/mgRUaI2WXnj0ZAND2g3fscgAti+VUf/nj19GuGZ8HpNxcUZ4oqsxh0ecmAABG/rq9arsBgK9ZTOcmBHJR/3xVRxJunkHj6Iwn48epsQeAGLO+GHee5fE06v7u9iT/glfrO05pfFmeH79MdctrDcAX48478ZcJ8mRfeeK1tcH/5Qkk/wt/SHdCmv75lWiw3pqPwVqG03F5jY0y4XmF/pf9EteqvFaXn/ckHRmxBco/Ph4AkN91Jh0cvRX1afS2uv6moVXbrWSv/pBk5wrRY+Z86tOzI8cp5zzUfgDcR8eVO1ETLv9nVJ4Yi0yMxYCMkHxbOwBg7WXTAQDN35uXvo/L3iXZDQOiMuXzwTygWzzb8lnJN0TcFBZ9bkOM2PsAOuWom+hXXQdfzRnqPO7TfANo+YUmNWbCawvvXAn+wCXUjFl3pOvf6oWq3WqNE9dTzXnUQPrt6dDtkH2tlFW7+av30ua0w2iffEZ7uoDyGrGvJPZ1khzZVwB88WtAia41a6N5GWWxvhQawV+9h5q701mp+gcg9XoRGZN+Wc0x6powD9fuSOv+l56NzqdJcsP7gfBzTP9nXvdTrvm8m64/Kzank5sSS86cBAB46i0aowc/9pF6XvVYtcy3WeaaVfNJjvEeYt4TAED3GqDYFDgvMHeK8nz+M8DzdwMAvM9eH6xHvd/pc3n7R2AtI0LlovdVjZtKF/jDl5H8mT9P18F1DRc07uDg4ODg4ODg4ODgUB+s86Bx82tVbbOoxsz8Yl6boIDojWVDYnW51yIC4FLzmW/QWv+80DZxbtWc3bzbBADA55+yaHOkVqagtQps1q0Agl+QDRd/JrZNP9x9IgDgzJcyaMdl9ZUyILWawlKi4OW0dk1YH+I0xQ+/TJqZExLqYqaWX8rb++L4tkFrVv/wJmnoTk6QH6grjbahayVYw6BgnTHatKRxZNPSqH0WzbU+UT8bvRnq3K8ENN9mmxjzIlo5Xl6r76uJgcOq1qX7qhvMu+jes1JLwHoZLJ9Oflrw1fQs8fJasMETgseWvAE0DqZ/Kt30W2yGf8fpAABv5i2B8owx8I4VtC3O4/mSuj9s0j5SMgrnPSJOEuOra7WS4//2cyT//+ak7ofNsqEgrYD5hojlIW6u7VmzGrFgCTISLHxmuZ61a+Llx51vWDYiMs3+NwysKmuDn7yPy7ch7fjlR/8oJNOLGvPMeaDUmqKtA3HlN8kCMHtW1eJ0ju1ZkvUa64F6Nox5UM1Tcg0B8NAF5wIADnrsmKDMQiONS/M8S5/Y2B2j7THa+OhXSP7+T2SwcKSAdUwa9ZpjrD3Duhz3HhDeb/5fz3XftCAqa+fkfcWunuC6JtB+6dYAgJav/ytVHW3XvwYA+IxvWMTk/GNbjy7cDADQekN1C5ESZ/GwkHOMstiUWsDFnGkb1/IesnEzsOL6UwAAg0IWDtvaa1o3kqypqq/5BrQ/+TAAoHVmbJc+WVjPgsbX+QdHLejrW5SrcwV8wXMAADZ2J6BjOe2ULzeGKw0XDzdjDIdsWkIs1nxMv60jdR2G2wrJ8BLdyw6balnw0sKcLOXiv3YZ/TYM1G4AFfq44rmSdaJozIXdikyE3FVSwuxrsY72O3kt+ftPABsfFKgrbnHbbFBS/0IodwAZTe5TB2aQH8bapeDNZFrnKz+gfQPH0e/qD8Gf+CFt7/oF+m0epj6WeUEvauwz36ytfuOlMelDjx3y7drkG1D3Tn5ILHpZuxWIj9h5lxyD8T95hfYVtesh+/Rl8YLDLw3Ms4wFpkeydBUwXvjYgbNT98MG9UIq+/gMfRixHT9nzCfRD35znmjZ7ZD4Cjrpo0rPV9nRunuC/BrA1y6ljdKA5I8vA0Mb4p8VmwtdVgxOmK57BzGexFwKAJyVAscYY2gp9GIuSIHWfN/KNxGnwPmfgnG/pPsaly5w3e3gcq73aPwyxlAc3BYrzjpG5fxjzkPC7dWmHCoOHZ6hA6H6e7rAxIcrn/8s7Zz4KXkUfNHLtDlqW7XPpkAasPO+tTUg5Xho3mHv2uQ79AucS5WDg4ODg4ODg4ODQ5/hf9LC0deoxcJhuoVEguCk6xPzwEUQonTDUIF8IZSa4jV3vENo+FqkxoJBWQRE0Bj3kjV/SfKrwdRYyHr4P0gjzvabrYMe842R8ia2GJKgNZMB8YWm+DJVsOUgv3qhlFB9mLRXau3bS8uo3AFpCtfQz5eF/AMznwkVIA0AbND4wDHeOAiYsiv90zqKfv2y4SJn3P+nb6Z9M87M1gBbQLwFSv4+l2cSb2pJVXtF+/nwTbWFRWgEh4waqLWCHcJa17qBJiWwQbpKSouFXwH3jMBj2lBWTS4tJ/K8prZk+SmgNJ6i2ivO+RYAYPYLZ+udgfJybjL27ZPgoigD1Hth4WD7XlLzuWbwuWq7uD8sV0x2PzTQVkqYaxRJSe36t2EJFpQkBLX5QsYqCijnrSN1YLi0NDW1Qd1XRaiRx3bb0TPMhRVSaaO5D75UkHLIYHDOqxJehLHVtuOrF4oB575B/mIP2jb3mdpxE/X2PAijnhZxabkAr6D9xtMAAK3fEhbUQqN1/svvfHhq8Zxz8CVvAQDY0Kn6gMUdXaIwI738MJjhvudtuGf4KDCKiCFs7rMmlvzpIQDA8CNuyFZ/ymdzyZ/uJ/nHZxK/7rCeBY27Dw4HBwcHBwcHBwcHhwCuvvpqvPzyy2CM4eKLL8YWW2yhjj3zzDO49tpr4XkeJk6ciG984xvwvPiPqH794LBTmsVbBgKaVGNfvo8/Cgs1yLcGPcn2jyQaTPgVQNDMYcBoAAB/+w86YK9R+3AWivG3hrVtFKiTrk0w4D4QcGjRNhRK9bn1qo97f5V+2xcbsSXCr9z3rV/yz35M+6bYBOd7p/kFgH8uJfmbZTwvaZzyJW/p+ylR6VbacfO8ia0ZtJ4mFWdKbJhFfrg67tN4BMCXkQaUDSGaReSKYCO31O0CiCKz0KCOKwyo3S84FWqWzwNafAB6DN57MXDanbQt4jpajvwSeDsFUzJp1QHA1xD1JWscEq2iFAps7ukCKwnNJpMB+T4gSQbktTRiOJT8NutTkBmzX1goJVu1yBLmOF11yXYAgIHfeytabtgmvW7Tyq+S/EG3LMt8bnANEH7sYu5kGayC89ckqMczPneZ5aeFvE8D5PhjABNWjKIYM5WyVTs+75W5AICpEd99pp7rsDUsCxa8SvJrGqWVbnBZt6RXVhTAvt6nwot4gPwDoHEwzBK/LFErHb553pA6xuFoQg4PLcddSjtlDNyHL6oYMs98vha9HS8vtH4yxgDbs5mkMU+QXw0mVTp/8Taqaosj6bfQpNYS5JJfnoZut0OM/Oh8VUscz9DtZ2Q+Z53iEx6r9M9//hNz587FnXfeif/+97+4+OKLceedd6rjl112GW677TaMHDkSZ599Np544gnsvvvusfLWL3uOg4ODg4ODg4ODg0Minn76aey9NwXiT5o0CStXrkR7u85rNmfOHIwcSQrmIUOGYPny5Yny+tXCYf9ijfczjtvXUz/XfCvKNchPssoo39yOZdrfUiaUGryhYnhgE/dU5+VborSQCkYSLtrwtUYl8A0ZrzHIN9UnCZFqg2DE4Z0rwKTmVyYD6unUWnFDOzciwe+5HiwlQ5L8tpPAfXDFsCGup+znqw8ZFg6xL8YfesvB1esPJDNLSKBmux5p5MeDaYYfYTFTTGPFZqBZWBaUVpTZqTznC7/kzY/KVntaH/IP/k2/dsVYUg3GtrC0LXye/h2/maa+lde/aZD2kzcsHImxC12r6FfGYeT19bFZFbT20fBb70VshIlIosxKOUKHak1WCCBXSq/arYVBKN+YoJrOAmllXPZf+r9lJCAsT2GKZyD4zCQzxtXGiGdi8148i2bCOwDGHGm0RxzjaxaDSTY5I35l6AbSahbfl96wcbWNSliPqsEr6LFvUsQDlDROWACVhtvwdjCR5NmQdkxGKcD1ebV4NlSrBz2dQKdY6ztX0u9jN4Idd3P0pM0yMjiFn/lcEYljOav8ENS12uyzVO07j9FvUxu88buEmmZ/DjFuWqxsM4EfAHCWS81CpzA+qz+DQxKWLFmCadP0PRsyZAg+/vhjtLRQPKL8Xbx4MZ588kmcc845ifL65YPD+mIsg+FkcHD3GnDpssHEC3X7hzqg1aCPO37j+MklPKnGZhxN4HU+ZuPsM0/iR5J8UWvdIDgRQbhqDJsakfGjP1BW3i9dbqsr1L5AbgYLr7kFNz5CrhcXfi+2SCqoRaJCwere8E1VH3+wE3Hff3bjPMbc8iqVN3JY7HP3i7FywxmOrWWqvPwccu8LaboQgZXrW3KJ73NpNOMx52D5aDsfnE/3KckpRXOI21/6kvr3u3nV5VetF8Y1lpm2yx3RaxCXj2DBf1LXWRPd5cLa3ACC9YiXi8n70b+T94P/lCA52P5U+p20l/Xlv+v7tLg2XPxEtI5QFlyWK2o3xm56sWKllkQ3g87rDwUANH4j6s6UBTqvRpQqk8sXne41YANGIYx866DIvojcmP/TINdcPZdFGqhxOtx4aRH5YiQFMJhnfX7fa49vd2/ocCXeWkny9894XuB6SqWN+bEqPoz5n4ke+srLfonZz4os7MY9HvrtZ4S8qOtNRHliIODaJPN/WK7fkGueTt0nJVt8XJiub1x+nMvnJN+gsqhzuU5WympbvS/4FQwqJlEbWxR+fjDHFF0v+ws55xwDE+TXClZoBB8ugvWbxMfxyb+yKm9W3XQ2AGDQddE51dY/021L7UsYyytvopfBwT8+MUsXInKZWAvYJoeotoXfu+LmiaX3/QIAMGyfSy116HwatWLJnUQLPvzA79Qso1/xPxY0blOkLV26FGeccQZmz56NwYOTFWj/W711cHBwcHBwcHBwcOhTDB8+HEuWLFH/L168GMOG6WS87e3tOO2003Duuedil112sYkIoF8sHNpUaktaI7QpzUP1157Mcjx4YkQGAPzpA/rK+py9NmvdkVIJLh2Pf0Bf6xNiS9QRzLNqJfw+zrNUb/EskMiNrvlZDz9FO/5xs0qmZmpqPjiT/GTG3TY/Ki8FbWo1bet7Z1Cw6oa/r18KWV7p1hYl+dux1KAo1mju46drQC9yNyaBFRpx1XTyy7z0Obo3cffDO/VO636r3BgSiCRkkR+HiLsRAP7SH2ljU0riiH/dAex+gTxKbfPyKB52AeJgt5IGKXjN+m1BkaXDL8raneyQScZCGl+JntUr+rT6ntWr6itQalO9PPzVZKllknLYsCKbY2t8S99OqJN6QeAQRuA5kfduGwrQveyREyKWEMYYFp1Lc90GP3k/W11q7Ca7OC4+f3sAwIhfJWSlD8u2BPVrzwNh4fB7DNp46RFQiKzPzMslBuZbPQwsc1acBYAxVp/Af0t72AjB6qOo3u3uzO0r6LjN3pg0T6a1OrYvJ0tSb504bW5pl29DltPZzwp3R5azEqm0feMvdakzDkOv+WtN8tcZvPqNub7AjBkzcMMNN2DmzJl47bXXMHz4cOVGBQDXXHMNZs2ahd122y2VPEeL6+Dg4ODg4ODg4OCgsM0222DatGmYOXMmGGOYPXs25syZg9bWVuyyyy647777MHfuXNxzzz0AgIMOOghHH310rLz+/eDgFW3lkJScImAaTUPBBo4V5WTwUFlZPZQfqpfDQRsnBDnKYNAYn/jYphlf0PtNrVOQow0yMFB+qfMKbNqlpAC53tDGSdQ1yRGCbVLbQsu79I/3om2vL9O+Zm2OG77JhvVtRAgbTJtYvVBmMH3vpDa7eai15IruPqjewPKuvpO9tUzKKAKjeWmAVWPIZSxSL/xuk8DLpJVLO9JtwYrKsiDNhtwHaxBaxpUfUJkdTjWEGKwRC1+nX+GvnBVV41YWUGwTtqlJfEqEYskQel772Jwq47vqBjNmbcAYUYm+Z7brnRTDUQ/UUz43x58cz7JPXauM4HJtVW5sTVrvkmIeNbFJkv9/qTm7OTUpSJ2/+GvaeONxsON+Kk8QzbVbW04/dOP4ujppnmINFNzOe7oCyerS4IzDplYvZNaZch0u33wMACC/60wqv+3JVhmjTj43U/1KRkI8qonRp8Zba7PA1t/LHvs7bcj3r2KrtVz3zRQ/0nDR43VpS0T+j04g+Ve93ify10dccEFw3Eydqp+TV199NZOs/v3gYDkjkFpMQpLxx3hY5KQRh7tfozctWzx81klGnWc8HHe/Qi9RZ9UkKQpzAeEfUPAdk/3OFcDlBCsn6FwBOw6Np8pKmuDSToLbJ8ivDVzXLycdwVbVdt0ryrxutu+XD1Jw3BlfrXNTBH72MAVXnnl172XJdr9+xGhses8CANqNjPd06SBHY3E977BJ6eV3rwErZmMOu+CwvvtgO/ABEbgo7uFPdt0An/vHomhBG3NVCgSDOxMWzIzPs5UgQtw7vlQEoLcvAjs2yBKz+rKd0PrNl+QJ+sAWh8ZXJpnmGgaK04yMyjD7FyJAUJmtc8nya4Ak4wi4OMrg0mKLDhI2xinvqZ/LoQ1+T6V6oUww5hrltiOVVN3WZ7G73k0IobOO8s12c4g1QeZciGHuefpVcos7wCovjTtO8nqRJD9edvwHDNv6ONrY8mj1MRV0t4q258Y5RKxwgc0LMURqUct7wA/nvAkAOP+KdOXTKvzyx1IAs3KtgvFO0CFoRJvasOahnwMAWnc7P10DVEPir7OZeb799zTnDdjv69nkp2mCfA6F4umdo0dh8q2v0bEmrZArzrq+Nvkp3XGLJ32/JvnrDJ/wPBz1hgsad3BwcHBwcHBwcHDoM6z7PBymK4bQGHLDjGyjj6snX7YNdZdvaDTZGJFV3OiXvi5aw/vKCmrETjZxisvcko1bBMZy5BI1MK8J+dV5BdIhoM1SQY2i3zHuDn3tcrSiji5Hst2b3jU3mi/Dy0eCIRnzcPMDFEB33iUpKkipkTO1Oz96gCw459fBghOBHJ/CfePULx4M/wnSTnm7GqZ/Q0NXK8I5FALjVspvrj2juZInM6kPGq9zGAi3lJaLHtBEFoKmEwA6biQXgOYrnosKDruY+RVtuTWtGGENZJegqW0cgs6bTgIANH3rv5n6FIsEtwrGGAIZ0Gkv/Eq8tbMeGZzrDiNoXNcnNnL25yjJa6w3+SkkasndFG1HNLu2docT/eJ+lJqS+4oyNmwt5NzXZAGK7MKL3BdbpmezHUmUtNXA/R7DTSqcu4gDXRSILtc1vvxd8AYKbZZZ0pEr4pSdyIphau3D7U3rImaOf4D6eNKO9aFvDkNaNsx5Tl1rkUcGAOa+tRgAkDWTROIzZrxn/PdtyrO0dUb5YVgt0tKKIeqbfEeUCAYA2r9H7mUDvvtmzfUn9XfN9eRS1XqDxRrvsM7hgsYdHBwcHBwcHBwc+hP/Y3k4eotPwAeH1IR5OimQTArI8pHsk8iVEr1N66FZq7dOLqCJy+kEhkk49d7H4g/KYEHjmnR9k2jJyitJI9x8wd3g7R9RXeN2pl9D23HavX9M3wHZ9hTXlnM/6AMPBP3FDW3iHiOSNKtBDRTANdHAB0S3y8buFEgICQT95Xcclk0rl6Z/rNAY1TAyCy0qMvqNxwRKJqG9Btf71NpcaXkUFjN2xPfVfeSCthr5BvBHv0HHD7s2Rd3JAdTW+ItHrqJjx/+6qvyqMGMolGWGNHNX7j0Ds58XWjnjOWm6OIFmMZKA0yPaZABYI7jLW4arDMNcxHqYlsnGi/6cuRtJgfoB///E8Syel54uFAYMTKjMsNREDqVLzplvzhabVBWW+cRu9YhaWK3i6rDoN2ZcSVXiW2uiQhmjAqPtloBqZbnwsMO55wb2cXnP/YqyIKh1I1eIXBu++DWdmb5tCh2TFoeGQdjhnLOzddBAcHyExmKuqJ43LqiNWcMgbfFV7azgV89SP84OeT0kje/YNlnu+a//KeQnnpkdfJWgFhdJjOOS82523+JMcq/elqjLLzqHImu8E36RmIBv699nkw/Y5xBrrJ2KKxLljTgjcy3vKadfFGtJFFvu7EMmFYde4xPwwaHNuHJi4rbM2cbiujLhRaseJvxVfRRDyTnXE7yx3zb5XX/wPgCA85+1mAbFdWKsoOTKTMglc8IZFp+D+nsH7wsA+PKr6Rlkqn5o0AZU79RE4xlBtFrGayup3zsm1mpcKdnv8bsmNFKPnXdWU117J8qP1sW58QFjmOgBAOUOlWOB+3ox5HIxV/fXw8QEbv7IB1X3GqDUEipjOV+5zHnYaGB2N4esL1fqZSHw0qDdVvjiuRnqzp6Hg3/0fmr5VeuXC6XXGMlkPPuFBdZr8+zhFJi/44PRxZoVgi6NJEu4TbTSywBjXtTdzLiWSv5fMvjkJATqB8eM8eIaKSg+JKu58iW43gRf7kP3U36MWHIq1Aoe+nCg+TT4Ih7sv25nKaEJ1izVPDSHmZmWLflUchmXHVvgty2AW82bcq7xu/U9M2T87XsULPupvS8OCs15ym0nqX9szPaqXlXOILF4/LofAAD2OCR90G/SMx6ZVwF9jfMN+hzlqirD55Pl6XtjBp5Hs4vb7mGlj8jaZNA0Nz4QbXPN+8ePBgBM+PWCVHK/+hzloFHZ6HmyG9n7x1GujAm/S+/LnP59KsQIGPhYCI2rGupOM5ZMhsz/GbigcQcHBwcHBwcHBwcHh/pgnVs4rBlCq/Fa15vRNYQ1PX3z1ckYAxca0Gp9bEq6M4aWm2R5ys0CgvOeF5sT66h3Hg6lWclw6ZYJ62fYNYP7lSpa0XRuOR93pm8LnRu9KGEtDS80aS2OcnMwgjANC8t/Via1M1gXt1DiWu8f0wPj7QT5WWDVHnUuF+0SVpc1HwOtG4hyRtunkrse7yTaTNYwSMj0E7VtabVdbMoOqcqlgaKE9cvKHUlpxIyAb+kyxorN2P7/jg+UC2jdRPCqsliUOwytsBgjFUObKNxo+G9Op9NO/jW2P/mYzP1If10TLJIGQUVPu8i1kpQ53So/Yb9h/ehcQWOjt45V1n6ndEVMylmTuAYlyDfPW9bVV2uG6HMuecKWtLxyvEn3WVPTHxzDeu7S++JRy5qYlY436kYbLN8pDAQ2V75wXcGxkm69WN1XJCbKmpr8upVLSr5lg5x/hOsba4q6WAYsOAnkEFlgdcsVcyYvSppqu9tYviAskRZyg1rXC/NYoaG2tAgO/YN1/sHh4ODg4ODg4ODgsF7BBY1/8pHvC6bFgD9udj/ApORlgWMJQZgmZh0Qn9SNLyM6VJmZnecbwN/9S6AMm/gpcDmYZZuYtoicut/YxPr7AxWlZAsNQ+bVJfi/wrNSeaYIqOa+YVmy+ZNqLd3WbcKHXgaIyvvAudbsyMDMVQvA2jaq3kYjsdu0wTWMU0smbpW5WGpF/YoK0EcHaafRMgJaU6qvE5txJgCg85pPAQAaL6WAfnSvVTEp2orQo5MlWhLU2cB2z5gEKwnCMojuNeBh+tSeLnCbVW1yQoSRtF5Iy9zKeWBDZTZkSXZR1tq/lSLeZdcT0smPQaI/c0jDnSBF/DKUho0UmyGrG/fJsgZWWF4AACAASURBVAWI+5/G+hiV3zRiRIry6ZF1buCcK4tx1Nc/GvMCsMzzTslbt77jymJt1aIHYxjMWEKTJjlM7S3BmIdSP7wXMSNewwa1t49e0vy608UIpIxhGjw6nlrcGvMiiS9MEgWRcV1ZWuW9zBUwbOzgVO2oCulNoZJuIhKLEIy/0GtisW14sF3S0tq12kigGl2jErPWG890qW1Y1t449CP+Jz84HBwcHBwcHBwcHP5n4Swcn3z0hS4pQPvWB/Kzwu/WzBNASJun4jWEhjVfUv71SiMJbmi7oiwdflfGAIcqMDUQaTWQScwxit7R4qebVn4xo9YxDYNTde2uZuR4eRltHxjW4hvNZlJLJBNcVYNR/yvLqb2HpztTtSuyL6wNZ0zX0yC1YlHWEQDgc86jYhf/PSjCYNxS9828dlX8mZX8e4nuk518d6ryiZB1FluiGm4vB9UvmViz2Aw25dOiIZp9TEP6+Yt73mxo16QGz+8BXy6S+gl60lU3EfHmoGvfApv6mczdSGSLW/waldmgSnovacHKeSgeIqxI0gJkxAn5t55Kxb74gKw8XSMVexRQOrSOVqoa0aOmguBcuK6YbdKy7qSRAQDTN6XYKUXBXBpA//d0gq8limbJlgTuK6a9AGWwInWSDFYiVtArYPrUAVX7VCui158DllVYJeSVFmMj8V892CmzWqlSW9pC7Y1D0w77Vq3TtAKt/c5BdN7Ff9IF5H21rGUtOx9QVX4aMNOyoeptEu2zxd/otqyaSxTBw0NrImsYaJRPlhE9psuveJesyPW1qTrUC+v8g8NOaaZzc+hyhhtHH7epoQYWx8QXUdNULLOpywWh0qWDyoz+PvbUBwCAQ20P3/BNxYZBHzxc5CeVHyG5kvECEQ1Uf+hZovmcWa1jqWEPEgPiF4TOGEZexliA8tF6PAXqSS5g5oMJ5z8w3RHMtiUF5kcWK85TfemaY6QhV7tLVeB5U5nrRb+YB774ddoUOVxQ6VZuSIHrv9FO4pTgVMJNemT1cmdcEJkboKHKi8yUnat3yoLAS4x8qeoSwdHznwUmEe20dCfjy94BaxkZlbNcuC+O2Jx+zQ8V6YpmUowq8gRxjbw82DDxvApFwcDLdW4PJX/whOydtMCr9qEhYLqyrbj5AgDAkO8fEizDPP2hYexLJd+YD5f96CsAgKHbnZ7q3HqDMYYe6UGU+OJSu7axJ+OjaLqKRFyZxHgKfNwalKpqjKncIzkseI/cawYK0gYlv9AEJtzhzCzsYVcVymAepQCWZRfNow+ZWpxykui9qbrwa0iUDhUwFFSBgHgqHyEeSfkRYs6HiUQtFtT6kcPLHWp9M/u+8Hd3AADGHPztqnUxxtB0+TNCoLHQKbdd6cat5c//3V0AgHHH3Za9zcb19FfMI9kDRuk6DCVNEgZtZHcV5z2dOk8Mi64XaT/uhmyyceJxh3WLdf7B4eDg4ODg4ODg4LBewblU1R+83CFqawhqCAFttuXMuPi2gD6xr6cLx08TWgwVjCsjAivafCndRPweQ5sivo4rXfprWmopCzrI6siN65OoKgzGmEo4ppArWb/ad92kNV5QmD7QL4Mv+Y+shH5aNzA0WuLaGdqHPTeuT/ZfHRAsNMWe1pzqfnGdJM/QXmw/NCExngrIFfI6V4A1Dg7UyZjWbEFkVTepW6e31c9VQml3Vi9K1EabbTtiSvVxJAkF+GNfB9tvdqK88P9HTa5Bw6YC/hoDWlDRGPXLRm8nTpBjrAJ4oUBTvwJM3D3Qj4Clr4useSi1ijI9WgP5AmnZ2IwzdaCjrFFYPTj3gYm7Ze8jQpow2V5hVWRjdjCSAIrfYZtoi6GhMZSWDZt8HnKR4O/+FWyzI4L7lv4HfM5ldM6WZFVhW2sq3Dj5tSJ1JnkDxebG2GMRmtVKOTHQP6ylZ14ORQtdZzWk0Wjag0ujmm3uVzBpAA+cE3yegsfgVyJWMTQNjW0L5xzjW6yHEtpuZPo2yDwAKLrtQOCtSvZqav91/ZscSKQNKju4cpkyAsQ7ltFvqVWRVUDQXvMVc8GKrbpNgE5wWmrFxgfska2DMNf9UmQ88rKgdGVMky6Eky3K4wDQ04kzT5pB28JLQGnT/R7whS/QtrTwdSwDF+uFXut7tJVTUsm2jhJVlnHmrGwEDqmfNUX9qvtno8gdfdPLCXWFro1fBl8mXDWfu51+9zgXXd//LACgdP5DVFyOAy+Psbe8ktzOBARczgeOkVuiST3KpSrumZT/s1w+sg+AeF8LkpjwShlM3uPVH9LvgFFWa5aUpd4zHD6RcBYOBwcHBwcHBwcHh/7EepZpvF8+OFghqkHTiZUyNiFfwh1v0BfulyPBuPloMKqXi/qQmppJrv3WZbvueou0tV/O0KwkjVzgS1/4yQc1VdFz359LmhgrUZ6yIsiA1zzY0Cm0bSQ3U5BaOiOg7M15pCUaXq1jVRDUUEhNtwwYNQK+w1oeAOUEA0Q4kJkXW5RFiy9/j3a2TdYFpJ+yEayaNYYjVSBnk+2ORONXOOd49SMapxMSKxVasp1ONeoytMQyWDkUI8Q58PJiOhZPoGxpqdRamppgS6wPD8WkmAkPlSwvB/7Cb2h7r69G6xKWCn+ZjlNQ13H7UyLlVGI+E8/9in4PmZ6+k3GQz4TFN5j+kdnT5L4cIBMaNlq816U1Vd7/YRb/4WIL2GevpO0/fIfET6Bkhqx5mEqwiOZ60TnGa/pjkRR/Fp5PeQVAQdUBhJ6XiJY6By9fi8U4SnQRpmsFmK5HxbNZAo65j1bRjcg14ZVITERg25yn5fiMaLQ5hhRrI6jglniFamuI3DatifP/QXTUY06R64uMC6loa7+aG3N6DZF1No8AcvlgORlnwDwlf+w5GfpoWffVMaG5ttJ02yjjC034wS+eBACcc3rQ+s89BjZGJAiVz7BB463k50vgHj3HMoBer18FXP9Lion40peS+6XbW92KyDmPxrfFXJfKz4guO3/mg5a6gteGAyrmjG91KB0rtaJ03v203SOsS4K8gzGG8s0UsVm45IWq7U6G9BQRXgi8Yg8kD7Wd+xXkNt052B9VxtPWfmnNePvPYNuJdULSchsWwfB8wH0f+c1rs4g79A/6x6VKuLuo4DVbmbRBXmuX4EtfO4m25YA3XSHkJNwtTMvL3wUbNJ72SfNcpVsFrfK/XkciPn0V/V/uwJcuPTFVv0ykaXvqLL0AXhIMRLbXrAhbEPOiHOWc68Br+TJjvES+uoK26/142oPnmXUx3e+RD9PLzRW0CXvQOAAAv/tM/PamOQCAg3egBaT1W6+qcw59bGG2tqfJZlqKurrFvSC8KTKBH5QkT+alEKZ9+sf44LC4Uki8spz2HRbb6ihs98Fm3rcFKVrx4TvW3WZ5bwh9EvHOVfrjwgi8Vy8dkQWcAQvestdbA1Swf75BZwmXL2PMs75A8Hceo8ObHxUVWAoGvLORW4K/RwHh/sM0r3iHfx1siPgwnvVLKmd8TPO3/0j72qZk7o8N1VyQbMfzTfH+QOFrEiBMEO55vFLWShyV+0ePKa8U/+IZj3RMNUHWMHsZliugPY6gwhz78kXGr2hSAzmH9nSol/ToHMew1vKtnAaB5zDp3qmPW67cVs3yY25+zdq2wP9F3Vcu+7V2KQBgyUUz1LGh33yCzh2g56Qxt7xWrSvRNgvGRJb0Mc19+weGRdYpe9jlMMb0u4AMuF/0gs5rJJ5TDk/lxJEftLwk2JHWLsFpu9UpT0WobdF99g8V77CoS22sXC8HLpid3jrrQADAxhd+HWzr46iA+Kgx68odeWVq+Yl1yz5VYd2KnOfl0P4XYhts3fcy63EA6n2Nb3Gkdpk2P5ZVO0LXkQGrH/stAGDAIddnaptD/8C5VDk4ODg4ODg4ODj0J1zQeF/UUouGS5rMZJCx/KpuxIJ7fg0AGHPQ1eKYNLEZ2oSCMC0Pnqi1VPLm5ooqWA7TheZSaoIKDZh/DwVhjZv585raXQ+0l6uXCUB++as+ckPbGKXFXdndywamgNJO+BUwizaEP0fXl21/avRYkruG0PAt/tPDOOY0YT/YJ8r1z5/9McnY62s1tb+3qGTwsggEwCtq1SpazxraVA+++gCaEsgNwvDLVq1nkuWPD4jPvtsrRCxH9jYsvoEoXUf+OGrhiLabAYICN3fqb8SunJqLdFZnPbYXff9CAMCoHc+spRcZYc9vUKkxJ49yz6sYk5VlAa10rK1JfjpEXa9saE1OaB8E8wx3EWFtzJcSXw5qoVLPBDnW/Ird5XPBP6nYpL2riuKcK3dA3kV0t83Dh6JhL6Edt5AC8PnPkvy0+YKAdLl20s5HhSa0LyWrk3XGkeufJ0lTRmnPih5xL1tG6DblgxTfvDQA7ctonPZdxpFkzPsK0VJP+E1Ky7zo89TbBWGMV7C6qkm8dwHJn3R/jLmvH9A4alzqsqzUCv9lsliwLcgdrNr61ThqbO2Nc+hzOAuHg4ODg4ODg4ODQ3/CWTj6AAnBYxKpfR1zBTz2Ln2hnxJJUOQZ/uBCq19qDSQIAkBaK5kZU/p5ymNeHo+9R/L/r2qr+w4dGX2CdVCh2OHlE5MRdvl9z46g/KNjNF23nHspAODzT0UtHNbxEOrPiB++oYPkC1Hqze+cfQUA4MLX1o2Fw89gguA9XVYr0Ccd3mfT+8qypjb1fF6+DdEXz35hYaLWKov8aggkFg3TdBrxDSbF7/DZ98TLq4TMkN2rdWyI/H3zD8AwimFhI7ak88rkR86GTMLIK+Pl1xtxvuOdyylwvWaibCOuyXYvu9pJc5zBFpbaEpeW/ndFd/r5jjEG3iqSQArSAHQsB8w4qxBWZpBfE6QGP2ef0xddTavVqJ/NjRyzZvKWGv7BEwEADafeqNdCQY9rWuI+vJrm6NF3HJu+zSIJYRJS0zcXmvDsexQIfajteJg8gOV0XJSKa6xo642I5VBU2fkSnn6fLH2Hp2tR3dHdldHyIPssvTXM5I0Wi19HZ99Rxiqa4YTgcQDwpmybSS7b4mgAgH8zRSp6p9+DIEFEKE5l4+3g8MlF/3xw1NONw8ujMZ/0JqfIycV/TLPJmFmdedwbPau7eTzIEx1kOolbWAcXrbsD8kJ76Ue65YCByzfeUH4TxhgGZ2RV6QtMG5R+AqTMzqJv4mWOdyzT7EGSiatxiDpns8Hrto9ZXKpSuR+EwKu4kfQHeDtlrGctKfnOxHM3+x8U3D/vxDEYd9v8QJEAg5t0ixCsMr1B5KMcsC/Q5ktQ0gKqsvoKgflGsDy5b/CV9OK34OZvYMyNzwtZIpDTJB4o1CcfTm9QDze7urvq1RnLujKeIF9qZFAxW53YxyVZ5SfAytwU87/E/MX0Mh7/SQQEnDDlGJcf18On6bxNTK8TEu9/RC/jo6u0PU1bgSDDUJqPDsYYSknFukWQf9HI+RNqRyDXlwqq1m3sc7e4KmhsTlj0bZBtr4h1sWsVoFgUo66GLS1Z/AozNqXKh4ZEz2vENFbcv3pZcv0TeYBmCsVT12r1IWsbN+V/E+FBaV19NTokwrlUOTg4ODg4ODg4OPQnPuGKmnqjnywc9fRTS75BAf5xCamJlFrw0gBtjgxplhlj2TTTKZDEqx6H6W3x2n+tsTHoU0Xwn3LlYB4gtcJG8DYAoNCIrYdkN6/WksU4CU2ZRh8HX7UgsIcVmjTVMULXBAyDCvW3cPBKdyRHSBxO2DReZRbJ0P7z44FT7wyUCWj6LZaxEy1pH/oKnPv2+17MqKGXgdN/I9rYcbd9AP4zERB40q1UxqReLWZM4ZwAM7uzpkPV+Q2MkvpYOSHgWd4TGWDcuUJpGGU27tFnXAL+kcgg3EJuZIrEYfg0oLwG6xr5Ym+XAR2MrsapnCtyBeTy695PeU1GbxU1x8q1o2Fw0CUPwXlwVR1JOGpZLwY3pKD0NnN+9HQG9vHVixTduBzXJv1uW529PWtZQ94QNOMH2A4q9y1hnRkwSs2V6rlnnuF6Fa3/VUEzfmDmltUOc00dfdznsp0sM8dLC0dTm/bmWLsEAODfdwkAwDvxlxh37Mm9bm8cwiQvcS7C+S33TC0rYJFbLrKqj5sBCKu3L97nmMjHxZiHwjbVSRMc1h3W/Urg4ODg4ODg4ODg4PD/Lfon03hKbUZSplUti2HBmhRmKDNGQmo6hH9nINmTRQn9YZ1ZHAPa05QmtLdXUzlbiJXWdktZXFMPe0L7nisYwXIGtaPA+2uyJ/6r1bIRl3DsTx+SvHRhZAze4AlKnpBsaZOu5y+L6NgM1BFVYi3MMfzwf+k+fd5STmtQBT3rCT9N1mIqf11hwco34KH3qa6zUjY9CyLWLL8HfC1p1JgMqAXAF1Mshsr0W10yld//CvE/gycsO/4dp9Oembfo0h/9m/bJ5J29gqYWjdxHv0cHlEqLRb4B+OuPaPv4n1nkiXEoSTG6Vms/eGFdY9udrC0h8qznf0kbw6cBf72Ztjey6m0zI80cGkal3DuaTPMZlMQOJkW5NeSsF6ilj/mEoiZJQBwC1kZLh7w6eEak6VfcXNpRFjGLwpot17hANmZp6fYKOtO4XCcHjdPnSKueymrtZbYQpWmzeTy0xxg04pp4eRw8XvRRJu+Tz52RfVo9u4wh4g3hd+t3gQXP0b7RFGTMckUcNr7vgqrjYD47b918IwBgE0tSPCsag5ThgTEqCA7YCZra/183E0389CNurqmtSrYteWz4/uaKVo+IVQ+TFXvQ3hfHyjdj7VSi0Yl76mNy/WmMxnK0P0opE/5nEv+tZyxV61dvHRwcHBwcHBwcHBz6FZ+ooPF0WhCeKuGZ1Q9WaDDtLE/6vHpoq+LaklS3iaM/d0iSQPqV2pxcEfyF22j7w7fo91PnaIYHoR02NQHHn1ofjWoYZrJG3Vz7Bd22LZvqU2re+OLXSe7waeAinkJrVrVWZbNB9Y/hqGblMfvankorKMobsRB2DZLQzBnxI8trYMZJqx0O95PlikqzFJDx55uowKzqFg7SdIp+QMtQGuajbgrIBwD86Yf0u/FnqspPDeYZSRbFuHn7EWBjkUTSsH6wg69MEhT8t9isYqn4AmKmYiO2AH/7Udre8FNUbqK2uSXLtyMpliorWxTnHF6udstlXJ06lo6vs7hIcxwlzQRJlg1bjI/ZN4l6MBxZE/qZcXoIttW8/pM2H0M7ZfyFZNkyrWvSv5/72uot+9C1Gjwnxr3wBDAtfRtvpq2aWVF9rrEk0Azt4pzjrndp58XhuDHzOUigFuemJW7szpHjd71Hxy9NbG128A6inVaMijGYevf8xONhXDGdrBiXfedskr/XV8FlXIdgVzPHyza/yyY/DmnmGGZYmMxxWiln4PvnvpqfrXOMsH6Y82E5K7XwusZ6ZuFY5x8cNgpA2z4z8K23Qd3risKRc64merMLtheH66+/HwBw/jE/tkgS10Q9cBzejqeL7XSB3d+74Q8AgC+fnrr5iQgujDo4FxATPQuuyIwxzE1wjQu4AkgItzE2ahtTUqiQ/v+jzqwvXzrgPDoWxW+5wzDlSwrminZRUEGJDBNb4gdqhI613AEuFlLbc6CgAjqBKQOzPwhpx37yy6Sxr7O913UnvfDxrvoFVav6cwX9kSP3bXyg3mZ6Wnz22K0BADs+uDgqL5yR2aS73VC6AHhgW58QPG/gmKj8v9THnSM2+D1SULoR5ZFvic+tHJ5PTPeYpPla5ihhuQKKAwZm7kdSW/T/CDxvYiNyHmMMjQkrXSTgNcCbHB9kbMrvM0pVNceIubSnRwXjms/TP58nQo1PhV+4cwXdP2O88pBcmtOidLgSz7+0CACwR6amx88h1sBg497ZzmnOdA9tz1NUvnmvM2WjzwKhJNJKDs+6Ps+dNREAMOHXCyLHbJj9gignPgw552AGLXwY758wAQAw8b46MhwI2J4hey6t9Gty3Me1paR10+GTh/Xr88rBwcHBwcHBwcHBoV/BeFofHwcHBwcHBwcHBweHXsN/9GvrtH5vv6v6t75+rc3BwcHBwcHBwcHBYb3COo/hcHBwcHBwcHBwcFivsJ5lGncWDgcHBwcHBwcHBweHPoP74HBwcHBwcHBwcHBw6DM4lyoHBwcHBwcHBweH/sR6lodj/eqtg4ODg4ODg4ODg0O/wlk4HBwcHBwcHBwcHPoTzsLh4ODg4ODg4ODg4OBQH/SPhWPtklTFePca+l34PO1YuRBsi5m0XekGALB8Cf7dZwIAvCNvjMoI5TFkjIFzn/4pd4jftUChibblb3c7lS+1wr//KyR/5s9TtRtA6j6uvnAzAEDLVc+K+hvBLF+5/t+vpTbsdn4quf6y/9LG/ZcDANis29QxZqFe85+5heTveUkq+QDA2z8ieV4uuVznCirXMIj+r5TBcoVgme418H96LAAg94Xfp6tfjAF1rxqHJJQtg//6VACA9/mHUsm33UNZJ8sV1T5/xTzaN2AU7ejpAis2R879z+HDAQBT7l0clMl96z2X45S/80dq90b7JzZ34anjAQCjbl+TWC6AFOOUcw6+6GUAAGvbiPZ9/Aa80dtGyj5/MPVx2/sXR47VA/NOHAMAGHdPZ7oT0s415bViQ8wXPR1gTUMj5d4/fjQAYMKvF6SUS3MMf/fPAAA2YXelxbKNkaVnTQYAtP1sRSr5AOzj1O+hDUbPJmNM75ModwDFlsAuxhiu23EkAOC8ZxZF5YrrY5tDko6ZuGvPEQCAo56pJJYLINU49fX9A/3yp24Cm3FWYB9jHviaj2mXuMfV+sN7xHiTa0alDL70LTo+ZgfaJ++rl1fy2bBNUnUv9TitlEmumD8v32YDXP7ih+nqSCFXtoO1bmCd6yKwPCOxSNlHuXZ5QyZR29YuBWtqi5RLO95UebFePTNzcwDAjg98BFS6SEa+IV5+87BU8tP27x8H0hw540F6vviil+FtsHW6OhLAe6gvWENzLxs4Nt2JfXAPeccy2ijQHMdXzoMn1g4TTx9E12KnB9OtF+p9rms1/ZZaADGv2cbpe8fSmjzxvu5U8h36F86lysHBwcHBwcHBwaE/4a1fTkbr7INDaas90YSeTvBfnAgAYMfcAAD46OpZGHHDZ+h4sUmdy/b7apJkWcrYJ7YLjeLXkCW0JdzQ/LFPnZe6H1nRcvnfg22JwYvfvQYAsG1KC4fU2PBdZ9H/VbRAH/zwSgDA+AwWDpmkhpc7wOc+Qbuktk9eP8YClg3a52lNhS/2eTl4s35K5TpX0a+wbLGRWwKlVnWu+vWElSRXEvK79fiRViz1y8GO+G76vsVhrdDctI5Uu6QmSV3jov0xemgBtX1KyqqU1WPyfqnKvziXtLCjUsrPAqWpFVokNnILa7mH5lObo7aP+uDFhTRexmU8L2DplJp+Mfb4v24H21TMK+Ka8+41QGkA7RNjijEPwyePjq8jZFUAOJh8rqccqPYhZHWVlizGPLROmJCpXyb8Rf8GGz6N/pHaTlE/WRWLwXbmS2TdBdR9Ra6ArQYH22eDTbus5k6/x7gGQTDGMH1Y3yS3YswD76H+MDGn//ArV+PMp86WJXTZFFrrwJwp55qGoqoLwkJl06ym1opnRbfQ7Apr7qdH261E3Kf91azPCmL95R+/Qee1bhB7DwFjzKaTngn/PnknAMBWvyetN3/9AbBtT4oWlO8M+VIquayFLGs73vcB/c8YeC7hXLk21Rk7H38obXSRZR4vzQFX82tJt61zJW03DEwnWK59JV0+0SIp1lmbJbe3iHgbDBpvLffoQhpjO6WVK/vRMEDvTLDAvfIxPQcTU8p36F+sOwuHmtzEgMo34P+x990BlhTV+l/1nTt5Z2fTbA6wCwssLMsuEhYkyAJLTkp4iKDwUxQEQUwoQRT1+UTF51MxPX2IiiBPMYESVXIQkAwCC8uyCTbNTrqhfn+cOtXV3dV9u2+YWd7W98fcO911T1V1V1dXnfAdzF4EwN+MfO/hQVymHxzjAWpJeiCpnO3FmDRdBh7Qlq7YcllgLiw0eLPDC2MIa7Oe2ZBxIacW+bxQNBdctsnn5TdpcrVPC3bofuTbgNkH03dewLB7gXndlRuAlNJfnHi+a9Xg1SSj9WK1edn2XVSm0B9wVwhDqkVV4LrywtGYcAvfPo2ae/HDGXoZxJpPLQYA9HznJbMFgboAYb3G718cM07LJSAXr9lI6zJw0AFzUpWrBO0+wi/jchGQoYWNHq9BdCd4X9QDnVXOUOY1lPydNxLz3wO5/H461kGLkjWXH4ee777Iv9a/bd3/hKRaQnVZFuMSlmyy/v9Nex9XsS+xKA3690W5o5ouorJZbdr1hqtkKDr8+WFa0Msq2NI0Y1HkIuXM+adjdNR9pW4ILT7WDtRnSWxduIfcQocFIdfLv6/2sIe1WMqNBkOVFxN38Y/pZzwqy+YCWi/cv5ZkL+ADY2PUC0muXkkwfpc4nr3a7691wd/fy5UDAIpP34f8ofloOVaypQXfk+aoAtWKloQHPQPCbn6A70rKm0H5/J8gdjwm8tsDJmVwqzTrNN0dEzZVu056mzntuEzjDg4ODg4ODg4ODg4O9cEIWjiCGnEpAfRsS+eKtFv+xJFT/V08a2HzbX5ApmUHbYM03Gz8uv2AQsB3OxBeE+Sr99D3GLNgTSgqU/ZGCpQTY+f4GlhDY7p4cmVRVgsKa3OK/Vqe1AFyfp93mOZrRbIiYLFgjeqgcotq7vCtV3zdhdBtMV3emrdha0yoH02tvqm/ZydLC1TdpYLh/qLkq/sovSbkps+tuX8T/vPZ0LmyrzFWfUbLaEjWMLJrQy6PPz9GgcAnhlufy6cKxrWdN8fpY/e9ACC9eTogxwzkV22Gpz57V/pWPm5L/1uQncqtjLWjwsMRM+LdcZLcPCpZ4BjzZ0UDrasGj8t8G8Ss/em7cmOY8IXfW7VND1x1FQBg7wM/ZRGY0PeAG1XQzdPs7yv/+UUAr8DgrwAAIABJREFUwLZLLqvYfLtsFh16vjxDG8zPlSeNcv6cIcv2ftjqMTWMhoCIO47Zx3KpsstW1VCuZDJP9e85vmx9trIGHId/BwBQQciwBOdq+ZmkpwD3T8k/bKrd0mh9F6SRu0G5G3VM8F2KLJacevXPv57+M7HneEWWwdby0dP9d7b5zKh1gekSreXyHCa4vOnGq+brXD7Z9ayYkpgiXLfxvuA2SrbMixxw0Efo2Fp6lzTt+i7dzsC47F9Hv7EFzKtzom2MeVR9+rISx7cic0FHT9qu2WGzdnN/+VpL+3PYntEQZ1RKf8lkHFsq11R1BQ7DgLeZ/cnBwcHBwcHBwcHhbY6tLA/HCG44eMdqHNqkKNhGvwkA+Mu9r+EY3k0bQdY6UDlBbtCfWlkxAhrJ4C7ZjBUQdaCsowrZomIcU36UYjxr34VVK/Wbl+nz4xGRvhYj8LtwIF1Tm0Xb4f9/1wukiTm5QhdssAaOska80KfvFWuRpKERMdvc9zTFVnSE+i+EABJjUZTmxKKJM48NvfRPAEC6EMNQ/dzGy/eiNn7hUVW19LW57CPv5aCvraE5W7rQ1Eb5kIV+435FtTWJWipDk7zbvikpOG2wxMZo2R0TAA74Z22ilwvE5DBuf53aao0mSZhM02qan3+N/J9r0cnxs63bXRqCfPZ39H0zaQ6f+PIl2PUmRX1rXJs9/+vXCYJ5fEf9i/U8JMt+QLeFFnfWV/83Y2+Mazd6hh/8q7TDYpO6by2j/LEoVflCwdfiaqunQNc4u7VTCKGpys0g84jVrYJnbueYZIKMmsB0mWrOmdQWo+XlmCSR7pWnCS/0XNMMmRQ7GI55qhea/KBiAHhwrYcannpDrrr/ZtxAUkB1gjUvC2zxTvP2V9Eb/NytfhpQFLkBK5Wizw3EnbA0i8XCjxvMJZbT8tnak5ZeNlQPAE1eYGLwOiK5afkE0Z3LZ2+zW6IS2ha0bOij9MHPfqUYjZRjvyKs7w7VFqblnvoO63PIsam2OKRkK13UOmzD8jU0v01LLOUwUnAWDgcHBwcHBwcHB4fhhLNwDBeCO1YpJdCtAhfWLQMAvLRJGD7Hxs62OX4nn+TDataVuFO2aCmqgq0tzOZQgRa3OaYbse2WhkY1qZxCvopxnsY3WubbfV9Rq1aGf1dGrt1+H233R0rp++kyEroopYTXnNW2EUX7p/4UrNLL+TSjmv1EWjV3P72HrsNHEUJTa2Zfcl2/8bs77yCf4MOuqE2OtkiZTGLhGCvTAmj8dlV/yjqqxKo6sQ4BRl+aWoHVKlHmTocBAH77moddw/FAQGKSrHT9E7EMXyQ/6q+dGvl2fx7RtLjKmtu/zqfo1X7VRaApqqW3hYRo8BhP7Kv0jbm2JKMxMSJ1Qciy+8Q6Dwts8RpZX+w2bXM+gW2rUQuHUpCqdX298pnx820yMo7Q4uepux8DACy4QM0x3TOt73HROSlyLBVClNFxsCUbrAdkiemz1Xhc/bK9YFOVlsC07GlJ4zcLrBbE0BwXM5aSw7lSsogmwNu6SJ/edhixDUd4AAkhgB2Opn+UefqCe0+0us389ggytx5726q61B3G4+/ZBgCw4C8JC4UYyOKgMQHwBkfJKQ74k57aeIjmDsPVw8+COq1dBn5rTsAytLkApO/uot1HjCy8OvBe1Zlvx6yO7IsA23Wz3ke10YjmKTDLeehfRRlzw9NgbD0JPPG28oV15JpXE9mhohsOQC/g/JwiNsrGXMwwq8dCHAAGsw9PjUC2c17U8OJNCHBQsR5rMde+pcExeoUa+hgHIQSw70f5HwDAJbfcbr0vP3v3gQCA990dzcIdK5u++R8JdJcPnr4PAGCvO7J3VJhydVZftdiW0n/uc+omeVF3KAC45hGady631ZFiMVMpUPnXD1Fg/pkVJWUHu5rwPHn6X/37FKDI/vohAIDmj9+WTq5twRt6xgOudD86hb6fd3sq+abrnXb51fO6UqwMbdZueOwyd969r/vz6hC5G4rWbsjbKW8TDvqMbpOWqd3JPK7cd4tRi2wzK71kWlyVPVq0j4O859v0/ZD02o20uUF2vYbuiVxLmdzx6oOQk3YN9qNcglxFLrJMn16xfr6enu/6mDT3yjVPk/zx1ZGNxOHZh0m5sYDf5cdfZa+/l8ZuWrIafW3SbjiY9GBUfTM3SSm1OxqeI7cxrHsd4vAvRsq+76R4ov8070W6p0HCHxO7v/uwyg12GDE4lyoHBwcHBwcHBweH4cRWlodjy9pwDJImTK5/FQAgOnqA0RT+Y2qT9pjVwERSAGbvUH2IqrBmQVWDKt/uJ8jhjMCm1qXJ79dzGy2B4SwtFKBFcpSGlJPyNLVELZSGq9gzG0nuokodqgXcvnLRavatl6Y/vvo6qN850695X8PZzW1BdADidNYVXfpSohq3OEaQSjkUDF4uQq54hMpNfQcd618HqdyLzLbvPq46d5m0NJ57TGqQmwfTV6rnceA770f75Q9Gih2/T40UkhWw8IBoAGxayHLJ11qzhpqzMRcHdIC41jTnmlHetAIA4E1ZqOWcs3e8C2nmDNYWnLR/Co7vKlF+4RYAvta7vPwBeDPfSceMcdp0xPmZ5PpWaWV1zrdF3EcDz/CB/y+T/KRkkTq4t3U0ym8S9bXonkXtevIGePMV1YdpfZ21e7RN/H84WFhAkwFo+vEpi/QYEuyXYroYTV+IrEg7Zsq/IqtM7sM3Upte/iawR/B6Ci8HTF4Q+W1i/WwBsCSltZa3BKPXA799ja7DAk4sWLD7oYqubKHOvkXMQk5jQ9fUTPITKqYPfsPJMq3VAEBliJd3XWWd45ff/w8AwKxzbWIrU1cLISDLTLcfPb/pCUromjJXu8MwY+uKWHFwcHBwcHBwcHBwGFZsWRYO3qkzfaQR0Gb6pA72FcK/TCk+XfKnwsBgVfLNOkwE/GnZD9iI4fB/62sE2lIoh8z+aD9gJd9GJ2tqK1u9BgZy+i2kjxhNl6gxwquSpaBcLMaeS13HCqLDFXPIB1yWCr5PNFOM5pohOeBP8jVuwn49MTYOS6K0arDb9DpZ+ngceWoMCQ9iwjyog/TZOtp6rZ9cT8cOylhl2gRlT6+la7htRvkVoayJQmmwWz/wDWuxex8hv+dD6l2/wmsPPwUAmF3Fb00NMmtxdYyYLOtEY6J7G/83yjfexNo3yIo3wV5JFS0LYtMqoh6uQNpZFcRU5RPOc9t0Pw1mYL5/6CY6uMNR6eSG5lMtJw6P/54+d0lHNJ7mXSSl1Ikp9Vw6aZfIO0YIYY0hS6yLLTgqDiRQhw1VWGTT9tHbh1KjylVP0MH5S/32aerpMuSG5XSobWzGlqQji5GbKKZQjLWSfFeN1lyIKrsYs77I+KwFkian+0Um+STbYonm9x9bGjavgWSiGB6va1/Vz6Tkn0qJGSe/L6GyytTVZgyHDaMOPyNe/pYIx1I1vDAnT3n31fRl+/0AAG9dcQTGfoNMvppVw8th5td+X1VdaVmqui++qSr5Zh1cT/icPqZcdIJt8X+7KcWeyt6H+IAqc3D3lapf7AcCjhNgKzP0NVqaNl90O7r+g4L0qnXbsPX/zfPoZTHuWy+i/bL7M8mz1jE1FOTmNekZVLTSopWuR8gdAsBtb1C5+WGZNbinmHj4VdrwHF3FbwP3UL1U+B0vf/tJiIMuon94Adv3FmQnmc3N+3rGPvHG6/DLKpChPuXmf8mB26fsUTbIzavpC7vDPXoT5PTF6qRqt5fDO3bssvy6xrqNvk+am43zPyDHZG3jLNEquFj+6QqIJZ8Mln/qtyj+/VcA6PljzL3m7qRK6LOGDfLEL/+56t9WBM+nasH983dNxal3qeBbMz/CqT/MKFbTbhnH4t0AxbH2QOA4pHGpFEIAY2bRd9W/35zyLhx3e5QsRSg3stR18Ryk3GyEEJAJ91hM26tie1PVaylz1Yc/BQC46EG6b+VV/4zMkUJ4QM+8zG1QP07VnnoHizM+eZ4KZFZjatMV70LX156tWW40a3syqnHjtb7nw67EHRMoUz2gn0NpnPffjcBD1/wIALDHkV+OymW30BgXZZalSQ0sWPGz7wIAph2V7Xl0GB6M+IbDwcHBwcHBwcHBYauCs3DUH0nazMCxfT9Cn8rNaOwX7vI1MWyekx7WfP4EAMDEa6J81kl1yQBXdCjPAwdb5/LovYrM4l3fro52V9dg0yhsVtnUWSNgcNib2LE7m8sT0yZiYCN9jrJxlvtZ2HfoqoFvtNDv61Q09abhehDKMC6Ep9vXdAZpIGRpCPLhn9D5Re8LlAdguGFZgiq5N6bGXJmpx37lfiW/APnEL+l3+1xQRSe50uAjQhqWcGb0bJzjslyqi5WjrwaPsUCbw5qowy4DBtU4Yq1TU0uEvlMIgb89Qab0IyvVgeo0bK88TFSZO2T+ZTyklBCjZ9A/3M8ln/ILGO2++VFyNzq9lrpsgbwKDz5IBBkHViPcJGNg7XSe5k6x9HMAu54oNwex83H40YWXAQDOvtCwKj79Ryq3vyWHteWeJ8FqBXjxTvqcsThdv7KAXePU8zQ6b3/oMj9vbDHifggP6FVWMcvcWi+rZQSl4EM+qzNmUtHa4ZTt4PmaLX2aVjkGjcqkDuCkHZnKna3/Ma6iyuWJiWQqQctTAfJoGZU8jjcpSuWO+hJFDL76PACgVb2jOk+8yHjXGdYXRZpjpWK3IUQUk2QZAACp1gYiIbdQGojw/fGaos/95N2sv/3b6vhM4xG5cfUnzEGPLadr7DKNb5nYurZXDg4ODg4ODg4ODg7DimGxcKTWbCofcQwqKtK2Mb5WFUUtqzCUTduiNR0crFUa1EG+msaW6SRzeZQL1QWlp2rLc4rGcf576EC+zXp9Dt1zYjbBTHP41G/p/z0/qE/5NI7+/nKXvSpotCzQ2iEj0N1aTgWQCaWpkVJCKEpeMZ588mW5iPJDNwMAmoy2Jsplmk+lifKUfzNg3EcjNgaPK9/xWiwctr4qrbjUGe8lhEW7NCYujrOCGVUOqoReLclhtt3xcaKZEBl/LV2Qj19P53Yk24X8xUeBD94YKb/0qPhETpnqjMH2Rx9alfxKdUvWYDNVdGnQ2qbOGI15HNhSqi0n+XafltNCmT2muQbyhlyzERMzpI8BpMWUPKex9UMO4uy/LqPvirRCNnfg1iu/CgA4bP+PB/tiJA9N0uCbVhybFeyZf78EADDvXZ+tqpvx9Zb1/eN4lsOO28seO8dJ7FIHHKuxoO6nHFgP2U+JRLV12uiz7KfA+Fq1xxF0BOXd9oYHq+44bfI3Bl+jUQZVqta6W+51Be15LZj6H8EYHxGn4W/PGCyuEun+6zSK65v9yxXaciVz9CwGnvm2lJaFjLjljhcBAMd+Rr1Lxm9rt4q3ZCRz1bERKddXCQlIa0aYKrdcss6nTVWGjqa1sL7t0lo4l6oRBAdB8iIv4PbhT6hPr4g3myW5bemMnE2t0XJGjoo3ltGCOcv0Y2WFivhKSYidQs4nxUHIyEJE4LHHaVG9r7WuoDkWgDbDil3VRkaW9HntAqQDQD289g/id88Skmt3UwtOBCYTVyCoVYTcrLwmnQk89SDkl0SCST2QXfhNMsHXtC4foIUE1EJCSmlsQkKZ3BF0KYnNkyFLEVetADhQ2/ZTQ35LneYq22QuFiknIna3OvPn1oDv7/2S8nWce162ukw2r6T2/PZaWowc95G0vUkJHWzM9dvy5wBrBzK+wXhhzvNJBTeB5X0kP1uGAQtywfbTs6fa3qzakm/T7TM3Py9sonLhHL1CeJBqjNvY98z5JAxzLP1TMZlVGfIbCyE8yEHFjqNcybwP3Wifp26kzZQ47b/TCQ+Py7ZxEG3jVL3R+V3+6fN07r0/S93+LODn/qIHVljPD1xJb4q2z92bSp7OIG/MpeGs8uZzWLz6cABA/uKH0zc6JcRYxUHHSoCYDUfpGmKzajo3JWmMeiZm/+ylyDHrGLn2LDp3bn1JDo65/m/0hfOcGExqgfp/RZOcOOm76QRrRapiS6ww18jfkduoOPnH6eTHVmtZ/LNyA+F5NfjOmtiWUYFj5IYCQMqbiHLDlx/nUumwZWDL2nA4ODg4ODg4ODg4/F/H284kUxtGPGg8gM1r6HOUykw7uBFoG8NSlCxgyQVnq+9h2k3fBUBrUHPNKC9TGobHyIVHHPVlrfXXpk0OusvlMfe8T2TuY8WAeDriU8AqLXkcRS9rPu11RTWKHDTNgWGezXXAoD1cX0WqEW1ZGeyFfJKog8VccnnhLNQQnnaf0hYOs73soiDLaL3gN/Sd3TtWPkbFx87xtVxGVvUw77g9+FqNk1IR+Q/9PHsnw2gOujVZM/cGzvt9fXVzdVUmua+Y8lcNVCc/KtMy1sJBqEZAv1m+N6P3oY0+OKk9zQ2IxZVSBoOBAQCl4HnVjjT6suA8pNrOloQE8gohPBRrUciVhrQLIWdO18+elLoNcj0Fpou2bp98g12vWkZh6VRFi8xuQWrONZ8v6xyeQKNqlt9/TrzFrmZwkLzSzpe+cwxyH/ltpFhWWlz9LuLr29IFef/3SNbeH6ZP0yKYkRY3NZR7Md+TzZcsRMcXHo0Ua/3s3zOJlap/8q5vkvwjrozQkwcytZ//x4wNz9CWdUT+4k0g0gJZ6LO6iHnvsVCpJkFZTPq+QJTBHV94FJLnNdv8c9Ql2eSnxND3zwQAtFxIlhm54VWgZ+dAGSEExNKLq6tApiOAEQd8rDr5YTm2tU7IKq+vM4LvrGc3VJdrJMll0JS/JqtF2mFYsXU5kDk4ODg4ODg4ODg4DCu2rKBxBK0Tcu2zEX9HIQTknMXqe5SeVAeDmj7H/crPt1Np/TkoHYDkQCqzjTMWpWxvFVA79WDSnjoEV3H7tS9lMuIoWytUQn9buwD28WcqP60V8+9JWDsK+L7jslxE8UdnAADy5/+JTnJyqaHNAcsGyTU1qyHrlNlCtnYJCXmd0kRekJDYrAJkL1Eji+6Z/rGUFruJcYrd4iDQXCmbamX5kzL6w8bXF7QUCiEgtcaKNUz2tgzUkEAyDdYP1S7fFkultYJshRvqtcbOzOyIv8amFo+kCvhzWDlaRoSsRcLD9PYa7qE5/sOZpg3rg2CLsfC0JUAa2Y5vfZ3kbNcaClpNGdBot9L6FNz3vkTWlxNSScuIUKyFyNtjcTLT1vJ7Qf9OAk//lb4qC0dN8tMiZEX6x6PLrXF9mcFxaDsbkTsqyBpee7R8I7HhNfpUFo641Nm1ZBjX4OfdEjsWG6yeAba5u3kPdY353fzmi0YSQ/O5yTjX8dxSUGQqrWUtI2uC1VoRmWMHNxneKT6qfWdpAgwv78fCWKwe9SJSGTZsZUHjW1dvHRwcHBwcHBwcHByGFVtWDEdIwyjGGRxKmnZNAnf/hI7NsVBmsubC9Gtk+lTW6uXbfC0Ka5BMTeRDN9Dnzicmt7cKyL43VbVM49pqVWzEJniqhABDFFNaBhMrCSHQ01n9XjOgzeAYB6blzbdFNbmQ2mdWGprl3NhJUXkAseqoBImi05aESWlvysWIBcRnQSoDYzJSC9tq4gRxAXCcCCdcyhv1+9aCuPiGSrTCZrxSJNmh4We9ugZ/1YBWWj8LRh2c6Iv7WC751kAFIQSm1KKhT4FJrbXLt8ZSMYsUj8eWLmv5p9bbGZyoYFD7TL9jRjxWtVkSVxr+489upOewKptqcRCS5zsdc8PzXymqPZNl//kwzmnmGKVFZMrQIJuab7HQ4mI00eFy9biH8Qi1a/ej6iNW3z9/Pll1/30AgEkfGB6tMVUSvIerYp/5jG3he2eM+2RtawP72snvAfUseibDkRFPxQksu6YGfh4YhyZjk0o2234QJfKVpQKg4pRkOzGOaWu5l4PkeJla+sLeGWaM6GjVP0XNjK4p9jhMpm4OsTBKKfWzGaDWZhnakiASk4xquvqMXcqGoIU3jHK1U4E51yZYE6uWP1LYyiwcI+ZS5Qdv+SZr2aceSLUox9N/APamAHGdUVYIyB7fvSUqOEq3Kdh8OV7lKy4OBDIoAwgO4rHBCa1a2LLuirGzQ6Xsj/+flJtD6iwHnOeiawp9Wh5K8z7co5K2hluTBOsLlhckajEaWMhagkr1tWhqwbqHKdBx7GnRBSGsG41QOyyBf2aQWd9TDwEAKi3vwwhkMH/kp3RQZUOHLPuThEmLqTc6/sb48Kn2SVcObdbjztwM2vKlRGDcg/23q971IXAvTcpBxhDlAtELkqE+Kx1okutfPcz5Lbn6v0GkLPt5MpjEYP0rwOSFAILPznHbJEpSn5ZFOH+WBnVAt2ylhYF5PQ7eoQb3lWK/vzHk3DB8Lt8epRc1y+t7nccJJ6i8v7zINhYLcuPrgf7IrqnGODU32eGx4V+bvY87sLr+xSBAlTlEi0TZrOaf+38Nueu/BcoLL+cvVsNuY4gZp1ox4m/kJn751tDv/Pm9XlmcK2FzMeZZ0m56KV27+P0YoGRPek4bt5LTGwgjc7YMuwgCwG8upU+VD0j/3rxvptKS7/W+irPba9LvFesc+5ev0eeZN2VpfrAtFlet3l/9BwBg1JUnAQDkW/+y/1a9uyPHhQjdJwWuSylSK+aoGDUp8XxahMkF1EFViWpD2Z4rrS1hxWmVyzDcfZPGad57u+04ti5sXdsrBwcHBwcHBwcHB4dhxYjl4YhqAgQwSu3weTe/+CORhFYAgL3fnyBYBSorzaXI5SEf+AGdW0UZP7HzoTqRmxg9nY6xNkR4wMKTsnQlHppq19DWaJeGoJYwjK4YFjhZKliDpXz4msbhMPlHEvO8+QKgsolr7b2Vvhbo3nHnyDEtN6TtMN2nWJtSKVCzbZsdUvYiCPO6Dd1GicJadz9DnfR8079xX21Wne0OXmyVLx+/Ht47PkDfDY2w/mpJ5OS3yW/bhH0PSNUfaxvM8cHJ/VhjXS77dXOjjGsdoANNGGJJiSIjLkAx5QcbEJQuhOe3Q7kEilFTrONpys7xJo5wVm0ppSZRYI03ygXgTUqyKeYcosrx7wXGzN2u+o60j/fvGbuIaRpj6d8cvsb5joALCeOa68gS+JFzmCqYRUifopyfR9NFQ0YtPAxTg3zTz+8CALw7ZXLISjCtmNr9i8fwCVfa54WEZJrWcfrqPfSF3Wy2OxRY8yx9Hz+Xf2nIT066Vi8cMjemnqyuGWzhW64S+Y2f6wc127TpjXTE4bmG+5Bvhc2aiiWVs3/6JA3GOP3HdXRk15MhN6nEiZ3K3Zaffy8HLD6tuvZXQOfHr6cvfF3fes1eUCWWTAu5geiu8bRKhLjX2ZDrKMmhN84yr7RGg7irAs97OpC7ybBwqLmmFOXdl1Li9DPeZRUppfQtP2bbeT7VKROmQD74I6pqn3O5QfRXCOz7/vq7wTcUW1keDmfhcHBwcHBwcHBwcHBoGLawTOMh7WehD2izaFtu+zp9nnxN5JTWjBiUrGL+u+m7CppC/1t+IhkOejY1KX/9Ln2eGqTkzQph80Nl2l6b5caAF7PxjbVu8DXjwLCOCYnyRzVl93Vk7VHQOiWD7Zqwo26DDmazBciVi7FBp1LKqJZS5IABCvhDWJtrbWsp1o80C1o+daflaLJ1ivHlH94PAPhciEWTrRtBGYZGLsH6ZR67/seUjOuUDyU2w4qAlSJ0HUUu73vgM41qjAZ1WW82DY3/fAatKnFYXUWCylTg5577FeN7/9rjpHXb3nIu7PsvZVk/1/p5yDVDjom3kqx64hkAwLTYEgmQZYvPvtAfvgXGj9sIWAxBz3IHx5lHYs4Mym6m+Q1YTjlOpeAHr2vtp29h7mjQWyaQwJE1rP3rfAuk+Zz2rqZDo+OvtNl/MUVFzxVU9s5SAVBWKz/+0AgMViQX2kpfL4Sej9ueH8B7beWyxnBwfE/Pjv6xpiQrTQN94zmejceOjWgBCNDZx8F8N7GVvPfGbwAAuvY4C8jPVuUs10kloK07Vj9Nn8oyJnY60lpMvkoxjWLmO1OJFcoiIiepeyg8S4yoIX/5A1Rsrr3+tPAtjMb6Jhd8P0jLcyCEwFe/ewcA4FMfiJ4T4y2zLI9TTW4gIBZ/RP0m+k66+Roi/Dn29F9W6MUWAhc0PnIQIRacODP4rT+mDNWHWTYcWpZ5I5mRguWPmqIfFtvE8/hPfgYA2O3U/0nT7ACSXJmklAYPOC8C7IwcWTIQSyn9haHqayD3hYVBqa8KVxVbQFw0F4qAVO5pcpOKTG8fF8kTILwmiEnb2uuJWWhz0K18VN2XhafFBlgLLwc5pQZ3FV2vLddHumtXlFWaSytsRhkNzYERzm8Qs7lLl4vb8ruU17A/Lki2RkQ27jH9u/5luv+2HMRRRhhPz1ms8IDwNNOabSzd+iK5JZyZtuFmfWabIxu4+A0lHfPv7yu9RvuDpfy5g+dLIaKKglw++lvj+q7ob9A9FCLKbDjTnqWi8APKG9R80e0J8gxyjxbakErFKCeEgNgt6HJjXgf5i49SuRpy/oQhy0W9GOdNzql3LDfO+26nxf+kRaTOaWRpp+mmpMf/xF3UMYMMg3/HzEi5Zsj/pm2OOPfPtXcsBBuLohVrlMvNjvbTERlKTucnKMi8/OKfIWbsQ+VU/wNKwTX2YO5aIbYjV0o9x7CbYhh3fp8+z0i34UCTeiYn7GTWFl/+PvXerHHDYYPehPOnLFvzuZy7tPKGPEDGwO7F6lP2vQk88BM6tvSKwO+EEDj8sF2yN95h2LB1ba8cHBwcHBwcHBwcHIYVW5SFIy3Wx+Q3qAhPaXW8pkTq0c3F2FMVUVFzy4FjFTTnm9IlDI/8LpFS1dB+9NfQx8rtUaZxTTlsdwPre4TclTqPSytXuUioLOeyNBTNsGyg+CSZqJuPTye/3qhW+79FQAX9mTkZbBau9ipnEBtltA0tDUrgnBalaq1UrH3OJ6e+rUZ6PbMHSykDLixp66b60+lEAqvmAAAgAElEQVSrSlvAYyBL1blXpr7GjQj+lOWopTHmmpf6KeeEbaa1Z4EPEQpQwfi2FBrl22hWX+EaPnsvfe53YSYZgi0KHRP8/tpcz14glyMcVKmlGVEKpgCI7Wfma8yuTencU1HIsKhQSD/XBF0vbdnipZSQpRQLDzV3Ss+gwOX7JXJAhz34XUoJWWzgwqYR2Mpcqrau3jo4ODg4ODg4ODg4DCu2KAtHIEs04v3GxyQrDRMqULvv4gAkJxI0gssB2smPT+dCnxlCCKIchT0eIlg2m2wdKKkzkkYDAE0tRX44tpoc8GUEt5qa7fLgQG3yRS4xWVC5UK0pzIcZ/FovVKYsjqcbNdFcxT1MTK5kQiWT0vEkpUJE2yplGQvHxquvw77+QghIlf1X0xx2TTML8Bddfq/ak8UnwqQltmmPc6JK9bx6/mRxUPtu2575piruoRmTFafxllIikoyLTqi2+NrWrry9j9Z4DbMOpsUUXoAC25QPAGObG2fikCpztKY1H9wIEU54CCC/V7YM5FKNf7n+FQCAGDtH+6cLHeRsXNf5B2eSnwYi16yzXzPJQXnt8/A07bh/jZvP+kF6ucKDVEH0hWsoSWL+wr/oBJU6iaxpPT7xqqr6kAZha2fsuD70Y9lkqfEpb7iAjp38X/79HD2DfuAZMUAHnVNN8yu36Vki9xALKQZIFocgLPGp4uRvZZO7+kn68phKVLj0874smwfFMV/KJJ/kpFuI6LGo4jZkigD/MMKZ5oUQkDpzuZpXuqag/CTFSYl9zw/WjcoW5S0OW5mFY4vacPgv0uRySdmNExEw90UXQ7pYleIzNaWCqbKcsRF6sk5wMTJRsCfBTgVZHLAENvuLZN0nzhjf2g3JHeKbm/PQOXd+fB0J10cvgoY2Q7R2qWNRZprmGTZuoYxQC2TkohmKK2FCHOlLuRBxBQss7PhFaVxj23WY0FLFSE26nv6RyOYCpSE/INrIpj2lPZ5pTGe11segJ1hh5lQJ8/8LXwnQ1tmg3X8E9uewmk0dAH+uKRmbXsuGo7mGzLiBIOdIALswXKV8dwfbRjPNXGDPv6FQKmgWM6v8Rk6ozF7DbWqJeU5ffow+908pl4PkNduO1IHwVsXYsscyNDoDOGiWg9fbYnIpMINfWqiA46bRilBFVhgEzDzYQCTldQHgX4tEIYpNDcLv04LD6NiG5RB5uo56jmXFoxDA0Oaq2lsxw/fcQ+kLB1SXBiNkB1JK7RqVXKdxn3g+6VX3fnAj2GlFKoKcQD6TSkqmjEh0i40h/OlbTwwVnbaTg72B3xLxhrq2TLZT6IM4+mL6rsakbBur2iEwtHYlAOBttu3YarB1ba8cHBwcHBwcHBwcHIYVW5SFg91XoLni7Rmq91zQk00uu4dsJjOy3Pi65qvm3TFrfoXwMGtR7XSqsdAa3mStSJZAzmpQg4GDNEGtilZQ3x9Lezmbu9HXyxdSAN/lj74BcSZxZds0RUnXR59r7TKOKc3OBpXFdfR0iH9L72YQC6VZrAbr42L0rPTCZh6OdGOkKvIE1rJZLGEBbVg410KgvO+G1DkqQZfEVgzWTApPu2rJPnKpEp09UauLkTeifXRjMzhXusZdGT3pND2kdjcS/nymNePsdtRUk2uj6ZpnuoYBoPusA2P5WEG7KJjuXXO6EkwQrDHWc3NTNOBWeH45nmsNzfGMGCtYXRDKayTXvWTNRSBOujqb3P63SB5niZ+4C+TdlM9BLPksfZquHEdnd1dJBU1HTAOl98qD0PmlJyLFdN6QtOD7OZNoRIWXg5TxywExsfF0oxUtBdMr58UK5OFQ3684nVxvLn/0DT0+hSWbupj6jtRtBdK7G7EbqeDs5ptX++7GhiyZqzzXmdYEqZ7D0gqi821q7gQKA9a2CSG0NadeSEsaYbbhteVkxbCu4BQVdaCPPK+o51GM2dbPw6WsfWZfN6wkq4fVgrIlIi7h2v9ROAuHg4ODg4ODg4ODg0PDsEVZOLS/PGvLC5shWStg+OZvWLUBABBKExgPDvhVskRHj5GYKkTnJoD+1WQJsXsh1ggV5MgJ+gAzWM7X9E/vyKYV1JrVIeUHaQmcNOXPyCg/AOH5GkDWKLElQEpfe6T9aT19by/9AwXQydIQ5POUpErMPYJ/qmXoYCojkDistQloeNnHkxMflouQL1KSKrHgfdX3la9rhSB/60+rvMTWDPUWDJWrSN6YRraUhhbbklWWn0WvCTc8S9cnTFQZuFdG7IceGxNZs9oUG88ihMDtj5PG6oTKrU6NYLzMIB+0BnIOZTQF6uBpHaNSilo1jWSftcRSkezwuDSuu87C7VuM/PnHtz72FhKsiWydsVnd1H0zLdE2a2V/IxNUao01aW8Hv/8BtH7aknyvNSb2IQ6cMb5nHv2fbwfGTqdjNs22Zb6tCziuQAXFlwZjqFPZoualfGtxkH2n/x5KzFJeyk6pmhZyFQU/i0kU0ycL/dZnUS77G5Xbbmm8rEDQOFlxLv13Ixic30khi6AQHuTy++l798wqexKDe39In0sv023U9mPjeZGrHqfvKjlhJYixlDg3d7pKgCxyEYt8QP4aynhe9/5lwB0raR5ZZDlnDXTncdA9i/73coB6Jm3l719Gz0c93xcO9cOWteFwcHBwcHBwcHBw+L8Ox1I1gmDGEabna+6JaP8B4MVV5Kc4DemgZYQ1fgj5rSu89cZGAMDYTI1PB7lSaTFm7UcHjKR4pubspP0mZRPMmstbvkj/H//NiFxT/gH7V8/gFMuUoiA3LKdyo407xCwrk3ejMlICd/+Eju14TKp6mYaT5XtKwwMgQssHAPLBX9OXWiwcETYuvx1x944Rx1KVSGUKAL3EtKETVsVgTJ3oRiP+vl4OUsU78TMpX38ImLV/pPyBk9Kr6AM0q+qTNJLB8WnK331K/acoakeQ1SyOISZrjIWWy7EMZvI2i994dy10KgGNtK+pBQDkmv1rree9so6r8q0eTVibkG8smS2uHCnHfTU1zUnya4GUUtPFsoW3+eiP2emY33qRvo+Lj88L9FWxOKGonvXeVcBY0gyHYw2F8CDXvUTfOybU2KsQOoPe7j+/bwM+bCtnGVvJUPdzG1ObnjCfZJafoSXhmJsYa7KYsW9lWebcqsbsdV/8NgDgvQd9JsLGFWB6m7pnmuZmRt+9fwAAdBymaGv71wHKyhCIW5ywUzbBPM6LftyGbezr7wljv1akZuxqWAsc3g7YsjYcloUcwxzIA1Wb6H33kMiDYby8C8XqF3L2QE6jmskL6Au/tMolPyO30c7HHlsBALBNsZHFEqBN+uLwy1Shkj4vYSw4AEB4ePWR5wAAtRLHhicaKSXQPlZVZ2zsLC4X/auoj/4WoQLUuBDdM1K1p7hqGYAaKfKU65h21TLdvUxXFb7WxkKrOcZDQZaL1pwM+hp2xiefMOW31kk5Yr2HTKbAnOgzFkfGtRACf19Njdgta6UGSUNSex5dQQvJujsBFJRSg10rLXlGgDrE9OWTR3dWl614WILH+fnTrlzG4sroa5o+WvNxGBua6EKj8TTjQgif71/RnYq5R9oXPX9UQd2n/XeiPA1+BzEHe3OHDvq1zu93/Sd9vrcxi1Z+7s++Z4X1/MAXaePQdsl9qeTpvEI9/iI3PCcF5tKrDwcA5C9+OH2j04I3qdptyz4gi997D7Xh/D+lk9tE7jjvve0lJVb4OUZsG+hrz6Rz592eTn5KtF/0v6oCdT3Hz7WWk7/5FJ0/5fvpBDOhgPG+SFrwy1uvVPLjn4E0sG4ulGtqYP7Ryge//NSsruKheUeWS/58psv478SJrcOR1KCO2MosHFtXbx0cHBwcHBwcHBwchhVCJqWSdXBwcHBwcHBwcHCoK8qP/XRE6/cWnD6s9W1ZLlUODg4ODg4ODg4O/9fhXKocHBwcHBwcHBwcHBzqA2fhcHBwcHBwcHBwcBhOpMxW/38FzsLh4ODg4ODg4ODg4NAwuA2Hg4ODg4ODg4ODg0PD4FyqHBwcHBwcHBwcHIYVzqXKwcHBwcHBwcHBwcGhLnAWDgcHBwcHBwcHB4fhhKPFdXBwcHBwcHBwcHBwqA+Gx8LRtzZySJaL9EXk/IOlITr3z1/RqV3eA9HUqsqXVHmBZadNBwDM+tnrUbmyrIp5xjGVTH2oV9UzCLSNjZTjsus+tj0AYOwP1qXqHgBrH2vB5xdOAgBc9ujKusplXHcAyT/1wWL6H73N+vg/+5P89z2Uso+2carGjjDo66Qap9r/UggIrylS/t/fQfV/6qFs/dPjNVRvWP7fj+gBAOx7Zzm98JT3kJ8j08c03BYA+Irq46cz9jEt/ns/kv/+h2u4h3y/vDwA6oe+xjwPQULkmumbcY3j+idlGeC5xmsyjiu5hT76bGoBX0Ph5SLyv74nyb/wn/V9DmW5qOdT5FrUwZLuo9mPt86n+W7ct16sLFeWIdc8Q/88qubpQy8HwH2K6rDKPzwJAOCdd3tF+Rppx+nQZvqSb1f1232ibz54IgDg6L+sSieX30UrHgUAeNP3Siz/p0NI/mF/L6WSn7p/AxsBAKK1K7mcZZ5KQvmB7wMArvjwZQCAyx99I9Xv0D4+XTkgfR+LgwAA0dSSWO57+9CzcvY9GefTzWtIfseExHLPHE/z6Y63pJxPU7wvpJR+ubYxdGzj6/C6Z0Z++9rp0wAA03+6PCpXrX94DgGMMbrqSTo3eTd9LlC/+n/jJ+cBALq+ne4ZAJD6nVi+7iw69u5vUJnn/gBv/smR3xa/dQQAoOm8P6SqXg7Smk1uWEbyJ+xkzLu5SPnyDecCALzTf5lKvsPwwrlUOTg4ODg4ODg4OAwntrI8HCO34VCasIDmWGkKxS7voQOG5tDczd7wPO32P2EXHD3Cu/3mDjpQbrFq4rjsnU+SVumEip1oHLwGj8OyrFym0WhusENfIYPiPx58oXyNkdaElwrqlN8RczxXew/D2inbOQDoLTZykLBs1QZZBkR0umj0dJmrRwVsxWArVLkMefMnAQDiyCsBANcfPBMn3fYaHTOsAJ/8451WkUJ4kIV+kpfPqWNC1yWf/R0d2+VEQJeLauE/dsv9NXbODuE1QfK4HKT5DF5ez7Ha6iI8/OtlOj8ujVzhQfSQplQeepkvX2mnpbam+FrIF2+ha7j9edX3x0TAOl6ka4t8mzpXClicUslTz7HIkQWMnjt1j9rHGcdUtZZFwoyO+k+oslyCvP8a+mfxR+iz0GfV1MsXb6W2bbc0es6mdZ9L5S69cQ4dKxX04sd2/crL/g4A8HY8ttruxIPnU+3F4Fmv8er+KuUrq4yUMtEC1JfByFgJui+QgLLgoDBAn6/8HXKH0fS9hSxXQniYvGjXJIHmB62HlHeI6JpKB8t+B6TuJ1vhc+jYZrtquxOA9RqO5jGpzrV0Wi0h3klfyVaZ8kr56xkHAgD2//0KJL51llyUTb7DsGLkNhz6AVKmwlzef5oKxsyizJAmdu5OmNz1C6g9tk6US5Be1PWKMbMBLw/dhNAiMm4CbG/wnSkOw4bD76u0XufWBvexVI8+qpeFVK59gITuluE2oxdaCkIIjEvwEMjqAmHD+JYG3kS9qFPtG1hvdafobo4cqivGNtehj8osr903hQfMXBAoUigLgDeQxoZDdE23ipSyDPQrl0tebCPnX7dNyhVBliOL4oCbWgU3j7SwucCBFz280ClvApqmqGJ+uZlTOhLkRucrfYwXUtrFEP61M+RP2rbOfeR6m1qw+UsHAwA6LqUFMbwcpDA2gApHnrh/vGDtema43PEx6Wstkp7VHU84Km03KkJfX1mG2OMDqnI1f+ZarHOHmLhLgsBySK4E/qU20i/cS5+TFwCeWvjn1VxnXEcxdk4NPaqAcmilL0uBZ5AxtjVyKBF6vHC/KwTo5uupADMW/LL/LfqmNhfoHG+sT/x7OLiWXJ3sr0VLH/g7u6KZClqL0mpgJbmgxz/tllptLsW2uWbqTvRZUC6Oz/0V2P7wQDkpJbBJue5ZxqttYyz73wQA7Hfq0aqQ78oqLUpr9KZ0DdxisHWFUW9dvXVwcHBwcHBwcHBwGFaMnIVDaQeDGr6QNtNmpQDwWp9d0ySlBJrarOeoMtYSNce6VAFAUwP9RNJqtOtp3rWhNddIK45FA2JBte4y9bAOpK1Daz91QKNRZ1FpjvPmmPNdsMbGWCBIdtBVy16wHCRVQLDvXY20LoT6LQc2QFgsHF31sEAkoCtfB/la01ny/3/6bvo+nwKaewtGcYsrWxTC1ygamjaptOSYvpt/Lt/h/6bO8AkxlLXNHIthDagwzqlrIZFDYcge6EzW56BmNXBpuK+5Fv8797HsW4tKQ4YFpCYo2Ya1UTfI861XvobUt2IPribtrvXtEApWllL68lRdtrkmMA8N9GbvTliessSJlk7632vyragto+gzxt1I3vplOv2ebwePl4b88WkQW4h55BolJ5F7HFq74bvZWeTffTWdO/EH1XTNl2Obu9nlj/tYGLBaOKa2ZZ0LVB0ltoi1WkllGN11nFCFfl4k8Kev0sH30rX7ysmn4zMPrVDl/OvQ/rm74wUahBca6v0jNypZE0Zb2mHI/+xd2ToRi7C7rQSmLqLvbFWdu59+NqWxrive/DUAQPOFh0TFGnOSrqmLAunlAR/yy3EdOa7eGFN//paqv34WR4f6wQWNOzg4ODg4ODg4OAwnXND48IAtG1Lt0kWT4aDJ2of+dcCoSZHfzoqJsRBCQLJftSX2Q0OWIaXF/0+hkRYOP9CYg7m8oDZEtWlCRn9VLV/5w9oC/0z5dQnGjUGUatheLpUi2So/JfVjDcpxrSUNW9lk2R+frLkul4LxAQrrhuztDPjBJzYiqoUz+/7mQGURVYM15Rxj1Rl9DgFgzUD8vaiHJWp1gvy0kMqvWLQTFTbFcMyn7+p53FyE1qoG6B43Ke1hy/YhoWUgFz99iu5ZqpxMfKnI3lWqfJQmMw18Qgw1TntX02fLKN8SwGO4XPQDgw3LWecYu1VYeDl/PmX/c1n0LTsDG+izucOPf2FtpjH/NI8alb1jtvZwXw1P4PZzf6JORikyzXnoN7cR5e8pn7UJDj5nZKkiq0dSjI05Th74DcVE7F1DYLy2bOiYBqGJCQT7wa99Vvu/B56rxWfYZeaafXIDfY0k5JvP01e+h8UBQFEMSxUoH5jHF51UZa/CiJJw6Hc1a66b7Z4NqxLnmqjlQo8Xc22RYGVcvZ4scTNiS6SHHzQOYMk59KkCoD997TU+uQQT5QgPg1+jQP7Wi/8WFVi2xJfxHD1mm/h2GHPw0NWk9W+57InU/bDP3SEKbAHIl1Sbdz6OPtvHG3ONL6PppCvj67KRFbx6DwBg7TeJ7nbC1U8F5pZIG4+6JFa+w8jDxXA4ODg4ODg4ODg4ODQMI2bh0BoAnWzL1xxLpXURrd1+eWOnnqTp0H6g1krTJVTLNXIbpul9o7t/8/tbg1XKt2j6bPK3BFrcRsep1IV2V2ug2CJl+M17ni5jS0K0NsECkRRD5JdJ1u73NZQWV0H3y+7fnEQ9XI8Ym8116KNoJs1xQDP24kP0ueg0AMAR08oRzRkAQCV+w/iQhUN4QDPNNcLGHGN+2vyvGWueps9pe6TqSzyCNK7oXamTm2oLS7loTXS6ftUmAEA3LFDzqU5aWCr78ygzs3l5o4/Rcd2/jubzLOw4SQhcx86JfFB92p+rdYPx48jWZt+akg4r++v4LPIcvmmFHyfGlutxMWxRa56jz6m7R8+F4yH63oQYRxSpOomj8AyvgGgMDDZGk9FVA5s1X8+nFe5h8jur8vWvNB+tj7FIZ0OoLwDEaLJeSo5VyeWt5v3CxvUAAKtzg+W6aZhWj4Q+FpX85PSKlWF9d61UFrP5auw+9htg2p7Rcn1vZqtrIlmix++1Hx3INetYD2tfFSPY2wbOpWp4YM0S+btP07lDPkcHjGA+c3Atnpwk1+JKpKkqiTJNjI5SXZryZ86fFV9Bjbj3SHJNWfz7VaptJeu1+PQH980kN9xHWPpo4sRzo1lA6wXOHKsXq6VCKLCa0Gjq3/GtDdpVDdICTa5SpumOHojwghTAkdOqSwRiBt0nvUDmz+6sSn46hAIDYxYBgykTK1fdinrMx7wINxcli0+hTzXH7HDdc9ZrffOF5wMAjr0t+LyQ241tE6ZkqMWbECJRCXDrR88EABz29zOS+1ABuu2cS6Jrqh6n4PxDXt76u+6e+HEUmU+9JmCQNhDyL8SpL478MpKW5h2T7e549YBcRi4XYiflyiGlde3ZU+1ckEu3PJuekUo9yd1QH+uaCnkXBWvj4M/Qp5e3/2b7aP4NfS78fumYoAONSzdeDgBo+uhvo/Wb9M0zs72PKiHQB014oJba5RJsWr/JCUHj9qB+NY+y21iSmzVqGCPBlkTaU1ZKBV48Y9KukazqUkqseoncIW0q08TcMjZ6WD5lHCsNVqvFTIHFir65lQLXxdJL7e+ut17NJFY+eh0A4MWb/wgA2O40kbzzVOPaYcuECxp3cHBwcHBwcHBwGFZsXVENI+dSZdHwDD5ByZtaxv0PHeiZY82cus2Z52arzKDDBeKtCozWY8/PJj8D9r7uYfUtWXP8k2vpWnzgg+nkaspDdjGogPt+/AsAwD6nXZeugizQdKEi+H8IQ3XJBB6PhrlsKe2UGDub/s9FNccA8OhbdG93zCg+jbsVALz5Bmmw65NWLQRtZTEyjSP6zDSSfACoE30zW9yMxHsbv/sxAMDorzwCAJB3fR3isC9Efnr0WQmZldmqaN5/vm795L6AUZMStdmHnvO+dH3ICCE8TdErH6dnHT07AFN2j7SlebTVmSpGroDk7Misde9/U7uXhSlmAaBlQoJJukZc80F6F5z9t2PogCwbboA+jv9gtgzZ7PIrN6js82NmJZZf+MEzM8lP7W6YN9zWgHiignI2UyMnVMt1q9mjVIS2aNjm63ByvnpCWcy0hUPaXwybMrtXstUvHd1tPWhxrfe1dw19TlXrj4d/CbFvkF1ACIEp82bFyk2i841St9tRr0zjZkZ4jX/9lT5VolS57iVgUjBzuhACeE4lmtwzurCx9nEBkRXMuaibC0XM3oH59SXlKmvx5nIYeWxd2ysHBwcHBwcHBwcHh2HFyMVwWDQBLe+l5EXYRPENWPsyYNmUF5+4CwDQfPDn0lXGGhNj55xI2fnkHfS5uAGWjseup893koaVtHHRNizZJtut0TEcm5U2pWtqYvkd50STBNULOsBUB1fb+5IUcFwPDJXrEHBsGx9GEC4Aa3wKAJS2gMD8qmHx47ah0TEcdSE3sNzDrr0Ooi88N0zZSZ8LUEm+RP7XSYGcQahkV5xsrAL9cfHFfwCIC8lPjyjNsvTbN3Mf+nzuVmDKokCzhBBo3W5+Zbk81r0m/ztrVr0cJAe/K8pWjlsRQgA91VH+psEJ8xWFKgdXCw+2sNihV54FEHMfrVBjoILWWJd+7ZnUkjNh/tH0qbT/cs0zEHyNTaRsp4a6rxsefwAA0H16m38NbWHFuSp52tM0Zd3LAADBJAeybG3D/O4qXxicPLE5mbZg1NiEpMEpYV1XzFzMJ+lz3tHW8sufoOsQjQYEEufhlBbx1Y8+BgCou71RloFpC+h7meYEMTaG3GDbhXYRxvzlf5eAitEq3XcTAKBp4Xv9hIKWmDTMXJCx8SMMFzQ+PAi/IIUQENPVg8lBiQMbrWa2W275JwDg6I/b5Fr4uNndiINHY0y2jIev/x0AYI+U7kxZIPa7QH0xXsgK5uRz43P0Ur8wrVzu46gpsWVM+bc+QYwZp6RueXZUchtoifdqqws2FyqXSQs9XmXZd2/g8WSb+AC8mRCjV48cFcv76Lc7VC0hAeFNemnIunHstHedRNShj/l62GBVBueAawW3SS3U3vrB5zDuWyfSKcMlp/nIc6wipSzrvAVo7YoWYLaU0TM0qwpE9Po1HXp2yk4kI3qNhZ+Hg3NJ7HYqPr+IFBGXPeIHV4rjvlJZruk2xteRcykV+iEmkOOgvJFcRfyM1wLikM9k60wM9DPIC0ivCeMv+ZWqxuNC1nfLvx6gDcE8m1x1H4VakEppbNaUq5iUMnEcv/EgsZlNy9KhGGiXlXIBeqGp8pyI1m593hyn8pmb6dhup0XlsUsh38NySed9GX2xChYvDuhnXiuvjHeUfOUu+j7mjFq6ZoVgZi2eX2Lcw/66mu5J2iWlZhrTpBHJWPYyrTuSQ8vTQZprjPCmvdDnX2PN+JbHpG0T2qnycEiD8S78PEjPDzfXbGBGXq5REyzzVDUIB6mLHOTjNP6wRD3rQ706r0wAHfarS7Kic5gcTfPV4Boiw2lqak0Mkkd7Pe6eQ6PggsYdHBwcHBwcHBwchhPOwjFcCGWrNNGiqNVax1h5ux9/i27S0dFfpgu4FV5iHo7XlOa4VmZ8e9UJ2YmNNvVWqZ1Pq01u5DAPW5mkLFvvS6NdqupBuyu124jSDgqfqlbyVRSeoZVUGqWmFnTH+MnQWAtm3E3dHuPattUjoDoWgisMfoZLJVn66zCZ1sUtjTX9TE8ry1j+e9LITT/hmwCAh1/oxaGWnwp2RwofFx6kzY2Fx7mRv0XKhBwQMxZXbn+V0FpWPedIw7Lhj79bjiW9/2F/XpVNPo95Q9Mu3vNf9IVdyppa8fSp5K4279baHng9ngzXGNm7kr4oVzGznGnp8JLGacjVxtQey1eJvMOzkJeY74yW9gRTX0b4LqkloG8tN4o+OyYY54159g2Vh2M3i7wwBauQvgWTx4ZJQc/WcraMNLUAa16qpUuJEOr51PcrZzd9v5WQSyVRfkqXo4E6uocG1i1P/oYO7q2smb88D/jAz6mc54/VznO+Fy/Qtm5Qgf/yjqtI1hFX+u5GmpTGv5btH76mqr6EYZvXBx+noPHWQynTt1zzClS2vY8AACAASURBVMSoKB22mH1QtrrGk/3+bw/T3LRUCCS9O8XUd2SS7zC8cEHjDg4ODg4ODg4ODg4Nw8hZOHTUYuAg/X2DAinF5IXwqTjVOSmx+7h4tafNb1xrTjgoTkrIfHugnKkVWzR55D3NGmFpM69JoQ4B1fH1eIn/M4oNtnDUQzsuIpSKhu8snzMsZtLo67Je+zW2+6umbI8hPztNZKaKAvXx8xJGPeNkbOivRx/DQaPCw/SfvELflZb+0P99WlsnTM3xtUeTxux9d68MiJTloh+bYUDfH87kLMuRh9mco357PKmkj723vtH31D4mbWDLjhmH4GeT3pyB8VTKsj+PcgwLB/vCmGMMrfW4CfaxUw+IbQ8K1mueM4798fWsFNVq/CdoTE35f36KrsWpqeWnaEGuWWegZytAsH4jTnHp59PL9XIoq4Rq37jwiwCAjz+wMlrOtHrsk5GKPgPkgMrAbVqaLMkyd84YNK4tfJzwtwKRSs+o+gcVCiEg5x5M/7CV4pjLreO19zsfAQB0ffWpRHkaeXW9tn8nn4Xw4nXIxRsuAwA0f+6odI2PgS1OtmUXZQku9NPn5jXWtdjQNe+l8p+4I11lKj5u6Q0P+8csJED61C9VDNlH/5JO/ohj69L5j9zK2jZYbqFJUxx8MR0oF3Sgmzm4546Lv0lWNw7l5iJXq4yfUxbFunsIITB9bzuTQlbYHrjyesq0KUYprghZ1hO7WX7J5PjJ1bqpYvN3H5lZMcrnovA3U/5EccCs+tx6W4Bm+YVb6Ps2B9LBoU0Q7eMjbT9tp/gJ3tpHduFgJq7OiUhyzTt2oS1na/q+APAD/Vi+EP6kqheynb57DQdewsMZO0XdAen/srHhNsYh94/vpRD+pkYHVDJbUA6L5jQw07haVOps2uWSlQlnRpVNCAamxt/DjqY67Bo7xqu6OLC4DPmzs+jQieRShUI/oElq/Hty7B4xQY5eU2BzyZBqbHAWetuC1RzThx++a+R8WpiBzDISaCsAqDnEuNYBBhjQNd9xtP0aSymNgHejr5x9O8eBryX4Gxj/GtPPcigMNY7KTD70Y/qy94f1Mds4+rf5GVmWePw/9BOSub+FocTACYfauYXiIFUQOF1XDirma224MxbJpVPyvC4E0Bx66EoFyNdV/oFZ+4UqKkcXaFJCLKL8LxfeQXkO5GAvwItVdh+V/vhh+WLeuzP1Mwzb+0K735VbdH0wA/hVuXndSYpGSw4Hvp7sZmQEctvGSGt77Xk4rNi4nD45bxO/N0KQpaSdfwJbYo9PhZBI1FGBLMfaJs3+lvPrU27G0nBRxV5nBOoQ0/bU99V0N84vJbey4Pinc/54U+Vzef85VNdMDKxXOWMA2TLK/y1A93l3m6O9w5aCkVflOzg4ODg4ODg4OGxNcEHjwwtzRy4OJ/OuqcW14Zcv0G8utsqz0OIyHeCUhb78mOzQAPCrGx8EAJzyiTQ9iIdNy+B1z0hV/vY3qP17p5Srzd9d8bS45jW56xW6xlESxWyw9jEcYGma5o3yP32KNBmXppSraSADwWjxD+zv/kGZuE+PLVG5Tus4Ye0bWx1yed/1iF0xhMANz1P/PhGSK4Rnb3bOi6/T0p6HXyC618PiS1cNoVyotFYwZ7cqruqvUn4gCDL+Hm4o1GFC1lpV1vQKiHdToCW4ny/dqYMczbZ1nvzpGJESGFD03axx9nJaq/faZ44HAMz40fOQa1VA78SdSb7hptd0+Eer7pY5XoVlrtTWBkNz7lvZ/HLPbqRjYcpYIQRsVL6aT4A1jKVBhGm+peEq8OoauibTK/YoHQLvjHd8wN64EO5+kQbqyVZ5rPk13Gv5es47MlCnPh9qx/1/ozwfB6Tsg7A946HAYCkl5O3/QeWPUtTFshR188w1QyqK4ojmPsadtbxCuagsU5aLvT7o38NI21ogR9fn7lnndaZX5usZ8764/hVq3+VWudF+6n6MTkdW/KqazOLfoJVh7d+2S9QXRec6foeIJUIIAa+pMvFAgBKZx+Qd36DPY/4D2vVcV57T5Qsb1gPIlvPH5spnC2CXf1MB6Ud9if5//SGIWfsrGYaXQtdk1axQX2PWenKA2vzcB8ltbIcb39CLdNs9l5N2jumJw5aArcuBzMHBwcHBwcHBwcFhWDGCQeNKq8S0o2a2ZtNvXruz+hqBSUkJQYeUf6Qt6Yym+CxBSp+2MoyejElbG4G6JDxLQF0yONeIprfBdjecBAswxoz2QxW+tcPwP53e4RMdmL+TVl9aERmLslxEJHjS8I2f3t64mxiJZ5HSqjxufhvcQz3HMJpatGVDY9TkiG+yEAKYfaBdpixBrn6Syk1QWcqbO/XcNf1A0u5hcCPw5O/pu9ISSxXPJISAmFLfeDGGEMLwbVbjrTCoY4LMsTahJSsJR2juLpcSeWendtc5XkzHJOTo+gKApwLXDe2r2fZF4xMFK3lGHwbJOgqm3R07J9ZaAACzxtaPFtdoGMS7Lgy2zWZxAiDa4pPGheN2ICXw3J0AgMI/bgMANL/jA0Be0eHavARa6pQ0LgXikiy2eNXNd2kTkA40isQkPG5yzda2PPHkagCAnYjbIostQQcoK6kQ/vvCMqafeYziR3dP224kP/8BC8PoifSpEhTipfsAZeEIyNq8NltdKiZu7pHKSiTLvvnGZnXnWKe3C1JSNo8kvvSlL+Hxxx+HEAIXX3wx5s+fr88NDg7i0ksvxQsvvICbbrqpoqwtv7cODg4ODg4ODg4ODsOGBx98EMuWLcP111+PK6+8EldeeWXg/Fe/+lXsuGN67r+Rs3AMkf+5aB3tH2PNlUUDHEiKF0PmIGU5SK0XFUKfXj5R25GUJGq4MNg4YhcAW4Z1IVfldU6rsSrWwwCgLRYWH9OiCmBoaoZW/xttaom7xlJGgsViWUUivvF+ueb6szj6CLNzDW3WFIUmGm3hGJ2vw020sPRIpspk3/R7/hs4/uvBn0kJYbOUsixlqdDWklxe36/eZ4jae1TraGA7xRyk5qbAvU6ar7KANYuctE1KmKxcVLE/YMy4hdaYcWRa4oIGFH8epbpLFk2dn5wrl6/PQNXXzezHaqIQFWF2plAbWtoSLBDsW2/eF3Pcq/+jz6gvv7undsY4tqYKnTTSQ3n9K/S9c6Kuy/rb5xQz4O5nBI8HYk+YaQ+QuxM9aX5AWXLybfbrwHJeocRuYkL6xUVaSL7GHC9QLvjfDUzN+Kjo8avkS4Pdy9bHKY1ib+bkjYo+Wr50G8ScaJrRxb94LFaE9Z4sp1jTK445DgBw2cPL/feVF/XgWHTdE5mbbqebDrIvAgC2U9S8fM0nbGOVNfS7qwEALTsdZ6ktmtBPtJPlTh5wFgvx302wPNO3f5c+a2RTGz5sAYvNBNx3331YsoSsS7Nnz8aGDRvQ29uLzk56li644AKsX78eN998cyp5I7fhUCZaqQIvRetonx6PF3JDvYr6NIj9euy2TyE8lNe9Qt/HzIoWMEzLSYvWORMbRI8Hw0XHyFJtC0LsSXIbSyNf5KIuOob8fJXm6UxtsQRamkja9CTdn7QZrGOSY2eDkaXZaAF98EJTlv3zxmJofcEIQjV/7eWCwX+xdUc3xqYZe7ViLMxGyJkS3G+mRW2yPxNJbghpN4ZJKNRjv6HcYkT3LDogPOBFxQO/4BQAwKrbb8WkE8lFIUBjzQvame9EALIMMUqFl+ogV5/GuPMMRYCRa/apcZuiD7V88wX65dg51XTNB98vdjFqavXHoqaeLBn3xB9HO+1qz08ghAfZv47+YcWQLOtFjdz4OpXrnAio4E52FzNfpD3ztqu2V1YE5pXIAljqus0+/vFfNC+enbIO0abokLd9l5IVv/ACgL/+k/pfCymnzvBt0uOGAsTlxtchbMQjE+zjRwihKdOlKav/LfqcrtwjZAkYos2HbBmtf6uRMvC6EsIB+gCMzUV0DjWRxK5sdQPjhbHaaFSah9YPJp5OhYj7GuATfaj1jZi0m5Wo4OnTFwAA5v3v6qhcdQ/NgHpv+l4AgMsefFmJ8iKbRnMOfukskj/7dxkS71hgu9Zy+eP0ZZx6G8WMl/xxn4yVaw0Cf+0+AEDh1xSM3vzxW5LdkA6unoTDIYq1a9di3jyfTmTs2LFYs2aN3nB0dnZi/fr1qeVtAXpuBwcHBwcHBwcHB4ctFdZcZRkw4rS4aDGSs3FnWBMYkyBneR/t3hdYzgnLzlpfJA5yFJ4RcBTVeixfW18axwC4DdbMsX5b5ozKdmN14j9Oitc1NVGLPqm18RaOShqlpLFbi1acUY9M4xps9s/l/XuntVRNhimbr7XA0QtIC2ANGldmdt3EtrFRqsDNayC1G5Pw6wdpg3ZbFE+xXCt8Fy7Vnxha3KRs8Un30DZx2coXy7WPA62t5vmkpQtCWTbYMjDxxLPsv42zPMgy5PL7qcw0RV7d3O5bZ1WyQbn+VeA3n6Nyp/2QzhkB66KrPppj3+pm8TvhBFwbX4c3cZfI6d7VZMWwOgXx/Gy6pamxoDM3Fwd1n6LECAKFdTTW683FIUyyBkYMucH7j94hk+zyK3fTl3uupc+T/st3U7HMp4ccWEc7o34OBPCGyjo9mTLSixZ7MlMxcb71OAB/TjLp5jkY/pVH6XPuYf4cZpPPxAg1wqrF1nrPEH11CJsSFPM2uX4FUZdCG4bqEDQuzHvH1bew27jqn5e3trezM97tz08O7FsX2JtBvkhZtcXsJX6dOhGtbzVqG5X9CbRaqW2ZvnvmBI+teRGYYQl/v/cX9Ln94ZF6rFbESbTKyx9vpyeP4KHr6XPHY9OVH2ls4Xk4enp6sHatH+i/evVqTJgwoWp5zsLh4ODg4ODg4ODg4KCxzz774NZbbwUAPPXUU+jp6dHuVNVg5C0cClJKnxp3kALKOchKnwfttJ/fmLQrjPf596lLixENgym/kZSxIkzHGYP71lCbl1Qop+WyRsO08IQ0OqYG4bW+kd9r9hYbu7tvr8foZp9Ym2aRx6sRL2Nq7n/6AI3jC21xGJbYJIaW0TYmEvxnBvI++NBrAIAY4ta6wKaxq6/cZPQW6lBZTlmk8oavOFsicvzcTAL30ZwL5K/J51iccW1QpshBTCMfajSrZ1p4fl2K/lFM2hVy6UV0TFnGAvJvJd9k8b5f1NRF2xynLTrK6qFjWEJ49Q0qN8l2MkxaUC75mnIVf0fjNDzX+PPLpjUUV1J7WDXLNsaOQTGchG/fRIn5Pv6pCvL42EwVhD5pV/q/KVk7/NM/UCzOh76YWCwVdOxBcQDoGMNH6SPGwiFfUEHju5wYlRdOKDiwEaKH/LLlWuX/n++wBhrr37xOAcpiTDQYuFZE5pgYa0WpWgtE2rmmQe+jyLshpn93LSMLU3KiWiOgmu/rdof6ckXIc8Po0hOvUdBflsSG9gS8lhfrP2n8gWPd1vzL/kzuc2rqetQJAMDQr78MAGj55JJk14g97fK3WGzhtLgLFy7EvHnzcPLJJ0MIgcsuuww33XQTRo0ahYMPPhjnnXceVq5ciZdffhmnnXYaTjzxRBx11FGx8kZsw2E1PW5aRZ8cXJVvBZALlQfmj4kfcNaMu/zi7VfBLZ09Ce0BtklM9FEbOAhTu3nEYJeEPlrlqpwQUmWQ9abbcpT72Hvylm3KqwfqYa1MXMiwe1yuGcyPb5bfuTvjPeSJtEgvBrnyMW1S5s2NuZDbfvLIJ4zZXOVLOm1AeYKXQXpY2KEkux7xImvhada547fX/hkAcNwZwePCy0Eq4osgR71ajI/dlj7z7RCT5qty0ZfLr/7rVwCAk2vccETg5SA3raB6eaMxtNm6YJ0wKuk1EGSOEV7OZxViF0CvKfE+doypExOXBWk3rpmZB3ljX0wXSRzH9BUrPsX4F02tkN0z+RfqoL0isUP8Sz5StrULvVfsCwBon60CQvf8YHJbbO4xdUIgSJ4OWBe13Rm5XLRcxYgpW7oS+9hoN2OTJcyGNQNpZFh+zPmg8u2W8/7/rbn69M82dm/+Ds1fxxx+BR2Ys499jK+kjT+2W5quMsVS1XKOcpWC8IlMbFj5NH3OPTKdfIeKuOiiiwL/77CD7576rW99K5OsLXt75eDg4ODg4ODg4ODwtsYW41IFAPJx0vaJ3dmoaFcFLJiT1TivTJr/up3+W/DexNITds4WYJgF8u5vUhuWfj6x3CHvnJVNMGvbn6U+ooKFY9b+e2WTnwHRIOk6Z45VfRW55kRNYWuDqH99tynWyNnrWZ/ZHYjJDeiHonubQObkMPo21YHHsUYIVHeN02qm65FLJTbHCeCbtGPa8/g6Om9jjbfKZQpO5Xoicslq2eWb621pNMYmu4vx3JCPElUAQGEoiSbT0j7OTdIbpe+0yh9o3DhNaynbMJTxOvO4MNx6k9CXmWk0mnPAihVMN6qohU26Y5YkpaHlTmedb//IDwAAq684CQAw8QMVCB5KtVGpJkGuIa20GK/eu7JknfdO3i/qmZAo959qPTFLufm0jDIScEf7u9M7o4QKw4kYXo5YaPda5R4qKuT0STRk1gApJSa0cEC8GpvjtrM/kxvXJMoBQi5oKtO4Pz9LIw9HNJAem99emcbrQY7zdoKzcDg4ODg4ODg4ODg4NAwjZuGwaabELsfTuXUv0f+jplrjLTZvIGfH8ZEzMWAt40zlhzq4CVL5Mtt2mAMrKaFVPdzHwxCHXJqq3LV/XgYA+NBlKQWroEax7zmpit98470AgGMvTCk/A6LJ6uy7+EKVQYCm1jhJQ1BtfEHF+lkDYyZ8s2D1QBpaWF/TqYNF2c/ey2l/VSmDj6oQAm/202/rm1YtGxqtn6mHRkRqOmwVD2bGarC2bGiTnzncwJLJdn9hKaWOtQlQXKu65Fs0h6FnJ187bIklOWxqHbg4DQTiRFSCOE6ehY6JgKbo5PIC2+y7MEFeNJBeo62bzj36P8BOKuUdJwg0EpqN2fOAKnqSEhyfV0G7+9n37ZpJLMfCrbyMMhZP/sEr+pxtzvnQ6dliHBJpXE2UghnkUS5aAvQFZAp5gXu4inzpx8yeRSeLg5BMue1F5xrZQE2sptzVfbDHqXz3NrKofeJL6eR6uxL1tRwg0oJK1/yle/8JAKi3f4ONFt3WlqwxKnr2LZe1XPO5C6N+NPGhuC4hsPcHlVcK92vTCmDs7OhPlz+buhYhhI7nlSseoWPbLTUSrXI541quWZZavsPwY4tyqdIDiTOuFvqsCxovIQIwMUO15pQv+oFHInoJSoV6UOPEQLMMJUcZZmZYCgU5xrkxMTq3rDvfENRjKVdLtuw0eUasG2+TESiUm8MsV9c8I1Wi0U3IHOxrhYX9JuzqZGQBN+8J5/yJQsa4u6nyKtgRAODFM+K9sInKz0NtsG1geQByNm75s7OB/3dD5Ldei93VCrAE9JobY3UNxfwTITcuJ1mKDKPWBFGpEWL+oqZG79lTd1K+iQUfSyeWs8hP+thVdKA05N9vS+D2K3eRAmfbC9LJTw0OWlfzu9z0BgQTEgSQ8XpPJveh/Kwdffl9Kvu4RcnXWLArqeqr8KLPJ4BctXOBYpGr9E5s1Ii1MRjakDlonfNw9JGbkmgfC3+ui47RrkZoUBWk2khoF9tRMVxYXePsxxH3jlXHVqh8NHMOjSVOoHrj5W+Z2LqcjLau3jo4ODg4ODg4ODg4DCtGnBY3oOFVWkHBZvkYjcD/Pk87+/PS1qVM0JK1jjG0e4ybHyF++fellJ8FkWzSBsxr8VbGOEt2M5Id8VkgzczjqxLcfYYL+QZvd6t12TIR1U5Jw4Qbb74GgMEE9j4zEzP/HzGzN3dEqFrNMbKxMPL3sLnKe5jWclSPoHEdNM3uabkWRPSZg5sg2TXKuA8nHL3ALrNcgtz0BhVnjbDX5LtorXyCPqfvBbmB8qWwpQGGK96RR+9eTY8isOZLUXONLCjXr+OvhPzNx6nUsVf5v118UrxcC1WwX6eydAz1QnRSFo/yH1VW9cOu8Nuza0oKzJQIBImmtDw+q3I32e5m+FkUQlBuEQBihiLfECJSlzmGn3uTZNhsD1mh21MuAeO38esHINrGWJ8duZo0wGJ6lAxEW6lYIw749Mi7Hu6fU8fM94SW8SblGRGapreO4Octl0zznT0wX1dAfysSC1QrP2UrKuQ0SvO6Ctx7pvRmYgNZ1tcybPEUwsPaBnKMiO6J3ED6ZFfHSMGM7yyeT6V5dRIIF7bwvBYRuKBxBwcHBwcHBwcHBweH+mDEPfkDWgfeHVusD0Htf0KCoqQdo+bES46fqNpXtEaYbV+XlcaRZSRoJM3df72SANmQlha3evmGhjMBWZNxWevSFHwGLR/XyxpDL0faSD4PsjjN6rRfYymlXz7pfiGqqdK/y+Uxqa2B9zDsu18cgMi3R8pVn83d11IlWTtGNdWhj+wPbswr8i7S8IsDKZO4vPVKiBOujrQjd8Y1VpEilwdGTVZyOTmppwPU//jRswAAR/z5Dd+qqSwoAfmn2+XXA3rsqKz2QngQx32dzun7m8P1H6PAhlPuPCWbfJZhJDH1DqdU25yIFDkPD376IwCAve44O3MfbDCfe9a8e+PnRttnjKujD5iVSl4Yb312fwDAuP98MbFNBx/9jsTzWaAtR6V+wxdejRkjON6kB8cTv6ODFguHfiewpb84AKEs4ZKtCs2d0M8k128mqX3xLjo3e0kNPbNDhLTyce+KameCxHeigaH68jdoyI2UgJNjY+Rr9+tEigECie9eFy/D4u0hX/07AGDZZ8hCOevaV3yLQC5ohQeAxd/5dTXNj8D6vGxWSZUtc23gt0ckpwOIQM0xV3yc0glc9sgnkTQSxDvTkeY4jAxGfMPh4ODg4ODg4ODgsFVhK3OpGkFa3AQtde8q+uzoibjpSSkxoyObrkNrB5jGEoBU2tqwNkVKiV26G6TqyICcqL/22uxrX4MoY8P12P5nVMuylJZSsliPGI4Ea5v2oRVCa5RMTdRT66nf7w7LFCLCPmWVL7yI3680aJtW9jfwHoa1ghbrBgBsqpLQzbyHSdavusSpMCMdfEsE3vXpwDlxzFfhx9P4mtarDyON8QX3rwyKLJcMv2JjjCh/6iP+oGhxyyUgH6RsNeX/+EiK4TjzkfomVpOybFjdFMoF3xrD2m6Z7SEhOmCVjIvn05bRkRcnW3WklJgxMV0yumogxs6JP2e06Qd/IrrM81PTjJM1auwX70wl/4ZfPwQAOOWTKeWngGjuAFTMhGiyxDeY9OCHfyG93KZWlP9wCQDguq9fCwA47e6V0YKG5Urse35q+Vmhaas9f8zY5oQ5XdW995lm37PRtBqY3NGg+bRdxVioOU9M3d3avzvOORUAsOSWVbGiAmyG0/YEAMy86o90wMsnzqWvX0oWzOk3DMSWSQOrJWr8NPrkOaF3FcAxawYGvkkpVNsuuS9dZYp6+9Jb7+DaDU8VS9t+RlZU8bG70sl3GFaMoIUjOlrkLfQ2EEs+QwdKg4AX3RjsPzVjVez6whlNp9gfeK5n3sF7ZqzADmtw30bK8YEOph6UfsC3UX7fnvjJ1SqXJ21NbThRnzP5v+l/D++cVp/wHVtbyq/eQ8emqIDYQh+Z5UPlT9o2Yx/ZXNynsol29MAMigvjyHn2RXKlOoMH2fXEkM/ZT9XCS+Y7/EBgfslB4LS5dupbKcvGpGlsKLh/TIUphJ/rg90AOfAZHhZuk61/WSD5xcEbLlm2Zs2empz+IF4+99Vwb7Q9k231cKnyeHPnX395z7fpyO5n0KnSIITK0myOg6Nm2OsXXk7nJgi0W+fh+Bedm+hnL7b174R3jI4cSwtzYSbDmws6Sh98rb0mazDpLqMTXP9s45/dcBT/P2QJkKENMc81Xg7FQgJ7Qo2QT/ySvuzK7mDCep2P3S6jfyXTiz91M0ldnOyqccz+2YKptcuZ8BBxl+T2S+mPJ57zhNDZ7DVKBcjlD9D3bQ4IVVT2N8Z8D6WEWEr5oN67RLkUDvb69M38vOjg9aKWL3Y+MVM/wwjPsUIIYIBIWtDSpetj1zFz7lwwJul9YVFgqnEvcsGM1JFyCh1d8fTQaWEP6Ffrjh4iv5abVkKMmRX53dxJmRNxkFyDKCbJLW3MtNSZy3x5RYOqmK8Z58bQc7cE9jmTvqr3huiZ579DNGlBCS1HnUu/CIx/qHvF9MjsFpaPBp8PrNfndb6qsqGsWXRU5j6OLLauMOqtq7cODg4ODg4ODg4ODsMKIYctQ5ODg4ODg4ODg4ODg3z5rhGtX4Qtog2GCxp3cHBwcHBwcHBwGE5sZUHjzqXKwcHBwcHBwcHBwaFhcBYOBwcHBwcHBwcHh+GEs3A4ODg4ODg4ODg4ODjUB27D4eDg4ODg4ODg4ODQMDiXKgcHBwcHBwcHB4dhxdal89+6euvg4ODg4ODg4ODgMKxwFg4HBwcHBwcHBweH4cRWFjQ+LBsOuXkNAEAIAZ1ncGgzfeZb6VPkoFPbcxlZgsg1B2WVS/j1kikAgHffsSp4TpaB0hD9k2uJ1inLqi7fsCNCN1xKic8vIvmXP1tI38m+tenLpsCKs2YCAKb8cFld5TKu3msSAOD8J4rpf5Syj+V/XAsAEPNPpgMD6yE6JkTLvXALAMDbbmkqubKX7veai/YGAPR87yV9b8P3EQDKj11H8hefn0p+2v4l1Wni+Xf3AAC2v3F1JrmQJZLvJT+eX9uD7uFFT9b/HspyMVUb3vh/swAAk3/wSvo2ZMAP96U+nvVoyj6m7V+Jnm2RyyeW+9dJVP/s61emk6vzqPIYSTYif31Pkn/hP+t/D9ecMxsAMP7qp3WbRFNrpFz5+g8DALyTvpu+DUj/HJSvOYHkX3B3euEJfTTn81VnUx8nfv1BAMDQd09G88f/TOdFTrdPDqyn763dFnllVS7+XklZBsr0XIKfCeM5lf1v0fdx21fqGaHOc83Q1w4CiyCKYwAAIABJREFUADRfdHs6udwXNaeKrimJ5dd9bDsAwJjvr0slH8CWM5+W+VlvTigNfGERPYuXPJPuWTTXNWnakVROSllRThjlB74PALjzsksAAAfdsiq2rJRlf3y3j09fScp7+O3FdO3OuecN/TvbOz/tva72d/pZ7ujJJN9heOBcqhwcHBwcHBwcHBwcGobhcalS2lLk8lorhEIffbKFozjw/9t784BJqvre+3uql2dfZp55ZoPZGIYZhk0WkcUARgRUiIoLouIl6lWMuJu8GqKA+sbtqsTrvSrRJN5cFdHr1UQ0Ekx4ZRUQREWGZRgYhtm3Z569n+467x9nrarT1VXdXd39ML/PH0/1U3Xq1Dm1nKrzW8EKPQAArsp4DuljeQaXXHhsYFVAgyGlWuo4vDQD/pyQfiEvtR5HvFCX43LOpWfO5Wm898V99fSyqfQORiWRzeRAKTtVnnfyFcEVDkkHAGz5+JUAgLU3JZMes/4lAIDRr20262IkHp9820cAANdtSqjhSEhS6YznpTvHul6W7LGcKGd3DWtpNhT9IwOZtQEAtk1l08damg1TMGW9+t5ItuOhueyu4ej/2Fy7EIDdv/w5AGDpZenqT/ocfPobdwEAPvHBdPXXPC7LYfE1QpuK7iEAQOHi9zvvXf5//0rscvmNjvqicrewpkr8lBoNJu8d6zmd/fKrRDM+/WjifiQh6TkuvPcnqerlj/8MAPDVP/+vAID33RM/Bg9/9oFU9QeONfaskaorLYP1nKg+8vKsKVOaEL/VOztXxNI1I6Kc1E4qTZNtxWCfL/1bHpNX5vRzrzU8ShPp5XHpSj9lx5TFRC62WGppvq80Z9F6lQQfADAirCBe8paLAQD+s/cCPQvEvmFNW2VOn8ss+Iuv3yCOq65lV5X3wrjUgNTQqCn47LhY/uJTYsUln4u1AOAHt4pt80XDUUMD/nyjRT4c1qAtHyZ+cAsAgPVIFbfT9Ilp6yr90HIf/lzJXb9vqULVy8DLAQuPDtbhbJvZxn3uKNdaKuVK7UINkGfZ9TE8+FdTF1fKKQd4fYBkA31fvr3XsVzO9vhZXsOklGdmMq2/J9f+Pj7f8SsZ36dZvlP/IMwysewUsXzqPuDYV0XLnXBB1Sqc5hpqjPGtj9uyvNelWZq9X/Gc19fX/mahPtC7+hOWFybNp48kHINnD9XRKMngEU5zZiB0zu0JRFEJ/cz2XU8Ls7VBh7Ag0Ue9fWzHh94du8W642rXJEn33MRNJFz12vdX5NsIDOhfJn6u2CiWm+8ATnp9oA5z/rJ9xjd/Sgj1jr5ZSi0qZffXpRQMJKbQCwBg575PLBkDR/Xzx3pH0tVPtBRyGicIgiAIgiCIlkJO483HJVlQqlWllWCeNXu3TaREea3h9nLwClWcv/yKKaik38zTzmLasa93EZDzAofS1515KPbEO5e1gqy1LLkM7/OwtKmq9KneLiZUUWcsuG378bO8hknhz3MFRN1auIQUvfafwLlSCof1OuiuJdBNSUAjMSuDj6j3yOpTnBpWTOyLq1EurQdKvafsd5fLxFcxdTBh69OR2Fl2y+2inArUUYuVZwEAXnD2hmTt+IMw2WLLTk5WvwVjnvUej+lPwBwqqoGYLDX4rMQEiwGAsbDhRM3q0n0+xWk2AqZgjnpd5rZ8Qpgn/ej6zwMAXnvbdv0NFT5/vIbDfKP89FnRvg/oA1ax0JBm80lR54xbJlixz0Kx/ebwRHUOLwMygiAIgiAIgiBaSktNqoR0Q0opZEg5LFgjW9JtJBABe08xm9XSKuaBV2L8G3wppmDS6dr3jR2gllrZkqzQbJl5qMxlK/FLAkvpcJyWLKXvUQmj75RY1d3HKvbAYfJt1gAUMp7OZ+wikqwNc9n6GjVbOp6WtI7/aclSkZlUOp7LZ3uSmz3WBPqzbL1YKqnwgeecGlY+viemPsfYpN83RlOOkIQ6cJz9zyVsfTqSOhyz9a9MV/G+xwEAf/8vYvnej9Wo/9S3pKu/Wj0J+lNPiNhm0Aka41pw24djVvjtXPpXV4ltezcBT90ptp7xrtCe2XauFHkNVDteve2YBxenHg6zPByk4SAIgiAIgiAIIjNapOEwWgoupUls8fFiU0Ha3PGKkSppSZLLztPD1t88BgAwgd9kuXy3CL0LGB8Oz4eJYmX7j1SZWTIPv3hYRORIaA2bCblctnPBLDUcUQmjuy91S48ThpKbbL+i6nlPoZitdHw6WwVKTeazpjGr8M1pmcvSDWZSJqJTY8L+be5yvdGEf4moEdlI050wOlRWqChS1cKRhpkUCd3+dKm4OLW0CnxmDEDz5cxptBmN3qa1jpNWY1xvErvGsI61XYRgLv32dgBA17nvB9/wcvdudsqADIhEoqvqt5kswmSYxOeYZ+tzRzRGayYcc9MAIEPdyRunKMycdGZwLweuwg6qJZh2AlL5MlCawNGvf41YF46VXimZbLA568aTIQD52LOiVttBvWtQbFMOgeVZvOHN59Xb06ZRmp1fjpw24fB/1V4quUK6RujrfWi7WA4dGVv+mKFsvuZ0DPcaTohp385aXa7OX41cEYuyC6seyb5c7RpOT4jnd0FG7ViYUR+TZhqvN3hD0hCYAwnTgWQJz9jzP9P5zFMPiuWfihPJLvgbd7ktD4nl2fUdhjFmAp04KD/5WwBAs11zEzuNy7C4SU81W3MeAODRMbHHcbU+6KZTZBhPAWMs8bPSXefXSpJM8gAwU6kzb1ILsY/JZ8Q13/xrMfE4rmchWM9C9341Q/E2xvhcyPS9yvGybkeWk6pMOMzycBxevSUIgiAIgiAIoqW0RsMhs4mLpC2hOY7OfsmgNRVFo54Oh4jjhV5s/j8/BgCse+M3rH1l6DdPSon1ftyYWalsp11D+liR+nNF/OR7twMAXvO+pB1sPu2QnjQNpdFSIfAqs9px36a7N6V4V4dQtqQ8MdKrFb0ZSW6VWaAddtNBWn9qrU1IKBLO0twofD6r3Y/5rE2qslL0JQxpWW9SvKSSvCzNjbh8DlmNUJR+JVszhKEMtTjTzz4FAOhVCWV3PgS2+txowbWnNXysOG1D/tgzGq7ffdCEJij7ZVb5ReuT1SvH0pednNDUbOfvxXLt+cnKp0GeT3/fE+LfBUdFxlgA6B3orrN6Oa5WSjpEvouNw+meg+SJ/GqXr8tR/gRhPrVj+h8BABtLkyYze/i7xq9kql142TL5/Kn7tTIHOJ57nZqgiiamGonN16aEqSD6RlPV3zbm83deHZCGgyAIgiAIgiCIzGip03gAaUOtl7mi8b9Qsz6/osMSmsQ3Hrp6YkRmSgpuz/AjCZu4Ppar/qU97Y83mrVddaZ44fDD7ll8Oa14V1+jZJKatDa5iUkodaz3EiaVdHVEGMeMffQys/9XialkIi1emXP6c+Qi3pAJq08okctUQVTDP0WTsZRtJsN75Au3iORn114rH7b+pe6Cq8+sq/5AQra483TixXXVn6AByco5NMixSGn3wd3CD2CoVvnuwXT11wEbXi1+cN8pjZ8Ym4msS4J+l8YlbgSwdSKlD0dKjUHSxH+Jkdf8KdnulxZ6zPdP9ODp60/BvXtF/Wepd321saeYMKhBiMTnp7vmnUy0kRZNOHiV3zA3JvctpzwzQERuNO6ju7+aWpQHTakAMbFQDm/jMlb64BGRmOqmigr6WpqdxE3spKoJTGf1MQ6AhV9+VUw6ZiZn09Urry3vX2LWxQykd+8WfWy2EUD4xVFtMOwuZvshd6DU/hlHV1+292na7L9JCZsc1HIeT11/whfkeEb9A6J9rEZXd9bXMLv79BPf+br8JY/x0A+Alzkcxx+4SSwveUHdx4o1ifnp58SP9ZfUXb+LWo7OGpXPKiF8230AgPHxuWQ7LDqmdplqx/LL1gdvNEeKNnkqT4mVhR7w8Z3id/9iWc7D8KKUkyrrGIAMFCGf83DgD8YYTlyQTkKU1BndlDdBbuL2iRNW2O1mQysBAO/41jfFtidvBe79vtj+5m/qcnLHTB2qL10b6k+l7DSpQmlcLBOaVOlzpkyxekfid5DR1DCwPL5cx9D+d3grIZMqgiAIgiAIgiAyozWy/ED4UCnN0FJwZSZjz/RiZn1eHo9sEWFuFzuPo6QIxkQK3cIxTjtPMgY114pIEZiHX2wX206s3orMydqiioU1TW0g6xwH/mEmPWgHzMtWZpFlJu5OoBO65/vZ2sV5WfayJKXi6h2z4aXucvt31FV9UtO40t5dAIB49/wMmdwtlsOrkpUfFtLxwcGEgXzHt9fRKEFSTZsKgc8YAwaipnF7d4trXcVornY7LC2mS0v1m33iGl9QZ/01jx9jYpzUadxut//7HwEAKveKIDr5d30X/OVHRcoBAC+XAC+7u/PBXULzslo967yKaVc+XRu0dqoroUmfvIfmDRQWlyAIgiAIgiAIojm0RsOhnWw9Yc8JgOsQeNL2tNhnHDlVee6LULew7CTLszjx6KDjkbbzK88AMzLjqh0WTWk2crK7fkXPLLnOgm5sHS89qt6ONo/pceHfUGd+3JokFWzVA585KH50SQcuf84ZjnDp+hXp6pUJIfnvfwAAYCdfEVv+yo3ZzKeTSj3LaVPX6vqT2Qb359svH58+JK5JVi6lC1qU+K+ahNFPqWLRYxFX9v7xQ+xIfZE+k7Vlco/4ocKBw33PlmayTTJ64sLm3qfmHPuY+M6nAQAD118o1v3sv4H/15vFbzu06p9dH1Nf9efNhL72EdZHqXcZ8/Loeve36+pLLRKPNd+/BgBQ/NCtieplS4T+fsV/vydR+bGvXAUAWHDjZYnK10OtPtYb2yDpeJp+uG6eBj1xcAKbA8IndfMdIqnlhg+OVPdxyGeYJRbAY2PB7yjO3ONerRDd1UjqYxfxHyU6ipa6R3POzUtgyfFyrXy4uG8cigLmVvaADyBXrJ6hOt8N9IUczzkHPyBitav8EGz0WDOpsT44xEYPuXz7s1WWZhI689VJpo7xYfVnlY+u7X/cCgBYl7ReNfk87lIAwUherkH6jmeVmjcZOtO97VxXkV69VgQ1/swd4vdyEdufH9wCtvh4vV02Dsd8Szhmqo9bO/oaytJhXk2CwYLboRwEZR/VC6NicpF86N/uTtiz7KiUszXHKWSlg03o+F9vpvGkZGk6yRLGok87qUrLlpTRf5LCvBwGrvk38Y8UbrArv+00l5n5uzcAAHqvu89VU2SNHltcUQ/1buY4W64+BwCw9l+STd7sj2B7AiVXmnYpAZ21LmIuw33kl6x0t99pzsaA0mSgD9oZGQhO1OS2oY0nJepXPSTOsVA3yepN/RjMCuEmt/KG2cJScWgGTAhzO23uwzwTrUmVm5sE3/ek2Lz81NCBuBXNUwXZ4WAXXgsAWH+BCJLAK3Pm3lH3pqq/PAN/l8il4q1PHlHNnlRHtlnX7WP3b5fr5PG6+h3lffD7/0Hs88K3u44WXSXPsfp2Q9+oORd5k99NtLUC/svPAAC8V90Q37GO4fAy+yaTKoIgCIIgCIIgMqNFJlUqqzczkhc1Y7WzistQaW4VqJrFzmBw/fEIYoXAVdIfLaEqAXd/S/xed7ZYjh5rxYlWqkBZ/9wsRteYsKvtYnAkW9Xgzqns6o6qrt2zeC9lIgmtrtVmdvH7PzORqnqwvMO+xaHKZce8Ivh/fyR8AQDgyTcL27x1P9gV3ODl61dxW0/s9eeL+/m6Tdlqw+LoX5Ctm+ye+kLv1yRpKMtcShWLyeeTbGjdN9t+CVfWIbgnmnx7xj33fO8msCOiWcWLi9zPaK36AmFFlVaymI/sNzwalejGwQKBVGLyCyUwJWHMAy7/RmhdfA4k/46vAAC+ce1XAQDvvntn9fq9HPCOm2q2oxr+z/4GOPMdoi7lDK4tDLq0xoYrrUuhB3xsmygvTXBY3yiO/bCU5M9O6HKygbo+l3ZLv9vLs2By3FWSe71froiPXxtvoluN4DFlXyzpP1chWn35IHgFax+lYSoC939P7PPq8P3LwD11z6kcVwDn6t40AXIi45rSYAHAf3xNrEuh4YgzCQ2agdUeJxnzwE5/R1yJyBp/XAR7+OFl5wEAXv/LneqURZ5b5uXAXCGxOxnKNE4QBEEQBEEQBNEcWp/iTtmKKmmGK4yZ9qdwVzH93DMAAL2n7ajJQzsxD9goU78NHlm7fYxhbPu+YP1tIHUW7pRkmuE4IWnt47V9qB+yZ61Clo7xScja/j/H2u80PlfK1uG4u9336WEQntqvzFM/HJgAFUwlAx13h7/1lh9dvQ6XD4EKYKJ8uLwCMCfVwvKdZe83tG59Pc1vHipQR63EaIrxvQCAl69JeHGmROK1qpncY2Av+2sR0AWwsn2rd7x1zrXWlxlNiPURcOBmoZVZeNZ7gnUASBRe1MsFfDXD9d9zo3D8P/vyf6xdV61jah+OnGmn+vaxMp4bXxsGFGIiSGjtiNKg+FbbLT/YUJtsP0f0zLOQsYDujzGG4ObcEvOODsipTRAEQRAEQRCHE4eXSVXLE//p32FfC4ctmzNUJfNQmZkOrTORgSL1sRzQLcOzKmlVYIYcShQIhtJstlJbokFcts4OMlYS1STrpHUdMVQ9z/uYdQSnlG5MmTBXqtQu1AA9TdZSBTQSO/8gVi6QscxnDrk1FouCUZxqom3llZ284xzJ9wiHB2/NCenqbzbKvyQpx4v0dptvFH4Dq2sU52UZ4TFlswCIaEI5qb1Q714v6gcTsGzwrOh9koO7hO/GSCRKV7KkeQCL+LZwawDbdEhc67MT1BRuW2SL3UZ1D8nom64QuJz7QhPkgHMe8S9kzNN+Lts/eBYAYPm3no7c+7r+mTGwV38hUa/qIWmksdQRyeQ327BtzBAT2k/Xn6x2osW0SMNh3SBqkD4kYkjruMm5IrjHAmXAmDV/sMICRm5W9T83qlsdWtcHG5GBV8slU06rN0NvQy8HL+MM2EnI5bN1r8nyYzzs9M/9itOZLy26voTl822+jBkn4e6ILNyVjGd1WZrjJGE++/Qlfbnnq4UZbxKlJt8igf7s2SKWx8r/tz8KuCK4xgRpcJ0fPdYo8xfmAd3Bm9Ee03i7b5SkmZgVMrToo4dEu6vkZ9ew7gV1NEruyzzrCzDoVB34zcx73OWEPFPlRmKMJbvXY/KsAMCBlHM2FJMFdknkUB2TIdtuY+C87d0EAFiwaplYMbkH/IB4HtiKM4KV9C9pyju4Gv98nmjDFbdLk8aZg2A9jntGOesnzKvhLRET+T+95VkA8lzGBVzx2xdApS7aPW60GHIaJwiCIAiCIAgiM1rvwyFn2Wx4lfjfyjyp1X+qjFMywFEplRzrIZyxcuHZLwN/9l7xc5Fw7GM9w06nMQCAX8486V4Ssnc4zrByKYXR6mq/HNEkcc7RN5QupCpXTpsymzzvXwKl7wjfK5xzHDfcZhVA1uZGHSAc4THq7ST71ZK+F9uu4ajvJGefzCxRK+Qyvg1dvenC4nJlmiqTcjErk7mLpVlGTi5KUxNlgjO8zH3Opw+lqjaSjA+wpLOOaBQT+1PV33RyKV/lUqK+dybh/Zm2fgvuV6KDlX5HeND36Zy0Tsh3gYcCYjAvh8GhBrNl++WIZN1+Tlf2pxzLagS30cVUGF91bMbAQvcQ5z74t/8LAMC78p+r1hUwx1orguH0Xnu+WddfJaQ/8zD7ufMAAF3X/yG+wXXw1v8vFFbZpd0AEmcMV3CZNPGG84Xa8oP37owdW8PnlegsSMNBEARBEARBEERmtMhp3JZuy1mpnom6xAMxIgOvgIc2jQEAXqJKBzQjjlmvsmdU7fAKWsMRmSXnuvC9zWLdx6u3InNYxn4klQyl7xFbUYfNJWMM0xPpjGaVvw8PXDv3eWKM4YF9Ytv5zhLZk/U17AQfjrSJ8RRJJf9T2foz16ReDU4nkDS5YWk6nUZXSRG5TNRai13TtcvUCzv5cvlD3k9HnOgu+OQDYlnLWSFyAHUOOVCRGg4ppLUlrRO/+RUAYPBtKetvFsp30ZW81MUjtwIALlud8AGbazBTbCQcrUP7nSuEyljlrDqaoT00z7XRAqbV+vNDMjmhstaohvL10H6jjs+umTGwy79W/Vhhv8jSJPj+zWLjoz8X286+Wn9XKU2C6ic/8BSKH/jX+Ha2AHdS55jyMtzzicPq2lsJOAtR1SmX57gDlP+Eg9ZMOORgyPPd5vfO3wIA2OLjRJlcl5VnQdw0PFcwExM1uMxN4dzXnheoXu83OwFU5M2oIlPZkavyckCbmwby8qUpJyH6I3luGh+94gWN9LYpzExWMRtrEkNN1jwGMquqgVUNKpWSzvBqc+SpGxPVp9fJqBz8j3LgfNE7YTLFRlW1V704pSNlQuyMuGKF78zImvZ9qF+CVkZaU1e0sqOz6V6gLSaKijsSzMLzXhFZF4f/zB3ix6O3ifovvNZcQ8c5vPp11e+RRlBmQfrjuUr/BhYkcwzV9c4IYYj+QBtYFlu+7WZ/ALxcukmjztKsTKsK8eeoO8O3DD8ocjIpsy424s6HwSspIw+qd4rqY64QDTBi5Vrwig2a+1RrRsKP65kvvhIA0HPNXYnqZZd8DgCw4aJrE5Xf8b5TAQDLvzuZqHzgWC5nZS2EzBlTWT1ZcgWGAUoz4hq6nfwdUS7n5ExXvnvsd4QrE/uO6XQDtldroiH50hki0/iH7lU5YjjCn8SsZwH41F75XzRrffgDnRX7wJbKyfXSKpNsWNGqFq4FH6+eTb5VJJ1oaKYPAAAm1ScF88R3ZLX6M3SMz4ROsItuIWRSRRAEQRAEQRBEZrRGw6EkQdw30swhKR0oqeytnol1XlEqfitzpj1zXS7C3GrNhpIMezlgdjZ0TB6N6c1946AmpdScq7jrDLkVbc4aC9QdmjepRKwek6pYZ61APPXacTA55/C6q3uTOtuvQgdue0Qsz2ColpODc45cVzZSR53VV2lt/IpTTd47kNC8QaHuf33+7IsUPR+r+jIMSSul2Fz1i1cA5hgujj4jui6OMREOe+ZeYQbQc8EnYk0NCiedl67+JpPapEqZ/U3tA1Bbtd8JOXPrFrL5ycxxChkJ8TjnwPbfiX+WnyKWjhCijDGw3urqQJc2T9dj35Mx0tPC4HDa5qei1rjO5+oMdJLUiTdDRRzf9yQAgC05Xq7wneN631C68ZSpd7vWUsWr9Ue6sunk+Fy1MP4hvHQO1alJ6bDdEch37eKuoDkdMT+hTOMEQRAEQRAE0VIOL5Oq1kw4lA2llwNXdu+z42LZNyqWtgOwto31IhIdniti/Jc/AAAMveLTspzc5uUBFRbOck7jzwiHPizaIDYNLDPblQ+H5Xj+9A9vBgAcdcV36uxw4+TqTMaV1JGunuTCiZ30QllkXUmxGGPY87AIzxdv5W6hJFTnf0TVgmqiN8YY7npoNwDgoqT1J4QNHhFc4XICBLB/n7AhXp60XlVPlfrC3LNH3MMvTlh/GiI+MS7tBoA9X/sbAMCSb7w5Ub3eiW8EAPSccJmolrFYyfH/vO5/AgCufvVXEtWflHD4xGr39vgBcQ2XJq1Y3fsJ7bu3jGf3wsksNK/WHicbRLozMqtmjIGvPD2wju/fDCbfKYF+x2hjwufHTiSn+8gKRpOuyln3bWW6QafqhG2rRs9f/SxVvXzfEwCAz150DgDgrx+It+9f9nf3pao/cKzybFS6bjmRMxmunv+b8CdhL/2o8QOUDtfMy2PxSy6ofowEoVL93Y/Akz6jYUk5Ywyv+7N0vptcOi+7/BNtrv2V8FflYzJ5Xd9oxOGZ+xXw/xCZwJn6rknaDstnsqoGjPvg/yrGavbnP0hVf1sZFRk9/3BQvOvOsJ9NYt7RIpMqh2mIcjR0ZBCvdT8p57EI9oeLqpdbx1KqVdvkJ6K6ZZiYbnNonBbQCY9s2vj/+qNh5oBcsSowOQ0zkM+ml0k/5LI2l8k4CFYiss4z0ap3SzWn8bTd02NYB0iukl6bSjnlnaomxPlkCTYOZZnWSEXqWXSMWD5zH/iRLwIQ6v+ytVWrcD/PyqRKjVE8cjPYEXe6jz6hruY3jVxK880tdwIAVlRPcB0kafQrB64Pcg5zzvXvl/yl2Dg3BfSOqL31Pjtv+wUAYPnljmMkuNfZog2xUZJ+/jMxMfiza2pWJUgoGFLO62xAii1cgxpjwNRYwgOHdk3gKM2YB3//jprlOo6SyGHSY53q9uY2IhqBTKoIgiAIgiAIopUcZpOnFuXhsE9qNP61+Z8Fyrtnshy9C4eCa7T6mxsth50p9uA2Wa+UagytdErEVfkFwxk5G6egUMw2vFs9JlVJiZoouM/1gV3CrC5ZNH8AXQOivpFjzHGqmPoAwAP7xHHPTlp/QpJKWHqL2Q4mUWfE1pPvytb0byplNNN6qdaeeqVpScM/tqp/caQNi6v7VsOURDGWYYRvtkoaFCoTubV/4rxm5Ud/DQAovtxRR+w1tvNHVB/Xdt4uwjwvf2eiZjeftPfplNAS9+SSqhAbyHkxO2HapzQl0lSNewUwuY5Py2AcPQvAt/9GHHWpyDCNXBGjxwuTaGfI9CRa5/KMCeqgcjlIUy/GPJxz/FC1Pat0TGkGo2NgIGDB8EqxUgXD8ao8b+MH3Oth5ZdwaDNsE6M4k6rK1ESV1nYuTOb6sceQtLk8iM6BNBwEQRAEQRAE0VLaLzRsJS2acDhsY8OzU86tYjEXgXkY37UfAKDMT41zshcoJ1ZWgEXr1EHEwstX16IwD1v3COnHiuqtyJ6sbeMzrT0ZXjVJTxX0dU4oWT2Ybe5EAkC+mO0QMttmd6rMs8V3wAsnl1LDkZYyz9Ax/rn7AQBsnQwNUcU5PD+4IFW9+r1gaVB5jM3+wMhAqvqbTtqQp3IM3T2T8NokHHOdFI2jiHnfOuTsOoCMB7b8NLnSvKk23/EQAGDDB5Ml/ouUsdrh6s837zpkl722AAAgAElEQVQEAPhIZIsbvk1ozbSWrUp7dHZwpZ2w/Ea1dmJ8B9ibvi7XOST4oe8l7le0L6MKAMBGj9PfQKxLJA/kUqvCn/kVClf/OGHP0pPUrzF1pvFp8a23cdg3+ytfXIdfEWUa72xIJ0UQBEEQBEEQRGa03qRKznD55B4Alk1irssk8pOzVA5EQqyCc5TnglFVtJSgPANUpG1ml0z0xHLGDlQmM4Nf1vVyHrKNZAyFDpiGTY+LfmSVTmowPgdSQyipigqtyrnvlGiMHrUkXb2lSbF8+Pui/he+Lbb8m9a2V49TLtd3/Dh7XZv+jKJwpWFqTERgqZ5WrTEW1h8cJ5akkra0EZzMGCbGmnD43TCLurO7hirhWa02jB8QIV0Xp60/oVRzTX9z+hgJh8l9YFbYpesw6w/+EHyZDG9qhzw/9eKYetU1tqXSLLjNL2sbfB3a3brGfRe/o44eNQ/+rTeJtrzzh4nKszOEs8lV3z81Ufmp688EAPR9cWvqttWWesv7yDnemX3TJqvlth+nOEBsW0opg7WxlWelagdzaMj0fTawPLZtEb9ILwf/LqERefxrImT4hh/tju6nNF+rz3Uev1nsebeIAjf6NZHEEZN7wfodI4rynSkki3DHFqwBAJzz3fvVmvgEier7b75ATuMZoCYZ3DOmTEPSYElPJHxgRoaFK0jVJ/OgVKrcits9tGQouE6pXfPd0Wyi3NcqR3UzstGNgC8duGR5UxfDkUuSPQxZUprJMpYkMJDllZcvjlrxsnc9IcL09SetV6pQ2YlvsOpXHz3RD8cH9orlusgWN7wss8/bDqJKfasmq2Dg2+4Vv5aKDxs+9gzYomPlZjMxXv9PMv66qsPOM6OOZZtCqD6oCbdveRSr+1qtYx4+fNtvEvYsO1KHVE1JVlmqw1QLi8v9ej+W2/8iqTXRUOTy9UlYkjrUN8u0MZAJHBAfkKtkSAgpYGIXXINwbiUAmP2pyOfSc/IVrpoj5cPCL3gF/XwaExnz7G79+ucBAKsvTJZDwZ7wRnN+qOvBzHvKGt+YFpZZH9QDwdAbkXdjAAbMifGHDa+RVZRR3dzZR8+qpKNoBxIbhMYwl3Yok5NdbptqRXLUMPCJXeKnNHMC84Ci/K2u09wUfGUatTw8CeTmPaSeac7BXvpRAMB6GU448L5QZluqPeVZ+LvF+8I75pWJuxgWHga2WQIHPdFQ9C2Klvcr4L8TOc5wyltdR4uuCgtN/LJ+B3IWkkZxH/zO/y7Kv/ILru4QbYacxgmCIAiCIAiipbRfMNVKGKe0jQRBEARBEATRMvjex9p6fLZofUuP1wHeCgRBEARBEARBPF8hkyqCIAiCIAiCaCWHmdM4aTgIgiAIgiAIgsgM0nAQBEEQBEEQREshDQdBEARBEARBEERToAkHQRAEQRAEQRCZQSZVBEEQBEEQBNFKyGmcIAiCIAiCIAiiObRGwzG1t6nV/d0ZSwEA7793Z1PrVVx3yjKx3DSXfKcm9/FLLxJ9/NCvs+njJ08R9X9iUzn5TjF95L6sh+Xw+9csAQCc8J1HxLaHb4Z31ntStU/Vx7wm3KK9i5KVc/SPz06IHzMHxLLQi/FrXwwA6H/752X9C8FWnCl+V+Q9kyuAP3krAMBbd1G03pmDAADWPZysbQ78x34q6j/5yuQ7WX3ksq2THz8VAND74j8TGxatBkZWi9/Dq2RhH2xknfhdKQEAWL47vn3bHxTtW35K8vZVo5FrKHObMkua5G/6V7HumJfLFRWwfFdk33svXgwAOOOnu9O0NjG3XCCelVfeWUm+U5PHmq+eJcaCq+/OZqy5Xo411zZprKmHr58t2nDVXZ3xzuCTe8SPmQOAL6493/kwAIAtPk5s6x4CckXxuzwrlrkCMLFLlJ+bEuUXrgV/7BbxW93PKp/v9H7w2XGxTT2/c1PA7CHxWz3DXYMAk/JH7ovF/s1iv+FV4E/8GwDAO/XtifoHQF9DXp4V7XbAmFvmyWUblFMtY0yPxfyHHxCb1r5QLBcfDbb6PLFthxhz2NBKfQ64Gru9PFjfqFj38M1i3ZC4bt6GS3T9bMHqVP0DzFjKx7aKFer69o2CyWvI7XGzT4wraixFoQf8pqtEWy6/MXIodb+o9teD/7/FtfPe+ZPE+/BJMe7Z10ndT+benMH0Z18GAOh+1zfEOpYDW3KCPLB87v058H//W9GGV3w6eix1flS9deD/60dF/Zd9s+46WgtpOAiCIAiCIAiCIJoC+XA4YODtbkLmpn1N7yHLiQVjuGePmMeeoKRnS491t8EheQ7X1wi6/gbqYF394odaAhj8wqPVd7Ck5F9885UAgI/cF5WqNqLZUHzq8ncAAK7ddGXqfblf0dK13ncLiRobWim2lSbABo8QBZUELlc0v7m5eyY/LrQXfZ96MFI/G1oRXMf9qhLNYDlzX0z89YkAgP4btiftWgTX/eVtuCS0wj0U3rNXtPeMuo8ezwN7RdtemVH9hIBnPKSnfWeoe9If3wG24CixbpXQnMKT2gDm6XK8YMZX3rNQ/B5YJovlAalF5Xd9NXigJceArZNaj/KMXE4DhR7xOy+X4zuAXlEv3/V7sW6nGOfYacfo+uvCy8PWVCTBNU7wnQ8BAMq7hBZh921Cg7z8i7eDefJ9MSDHrWK/1qqwngVi3cyYbgeOlFrX7qFI/Yk1HHZ71bEWrhUr1NIu49rR0vx86YZ/AQB82KHhaESzofjkl34GALjuncn3cV0H1jUQXJHvQu9191WvRPUxV8CXrheahw87NByNaDYUn7z22wCA6+aNhuPwgiYcDngHqLmyfkFm2cdDyqpADVaFXme5pC+femlG/erjF1yavLAcUJoUv/NqgGT6hWMzGmNxpEwGknyAVyOu/powD/DlhbrrH0WbXnS5+H/HI8AJrxXrpOkFK/YB8kNHm4Bwjp6roi9HUX0O/oSYaKmXZdK+2tet9z3fStihdHBfXU/TJtf9UshYB5xr/1CTOZ0wnnYs0wfApx+Q/8iP8sUbxb/5HnAltJHPKgcD3yE/jEc3iHU9I2ISAYCd8maxbk78j9lDxhy0W354c9+8YNQ93ztiPtBHjhHFpICBc24mK/XglwE5IeDKqEIJFTwjWLIFDTz0AmSMAU+Lj9r88WJitrx/UNY/Z5n0uh5YHlrCmAPJ88Q5B566W6w79jVpetc0sn5Kcqw5HxXhawMAqEiTP0tw4zKH7o/54owVQCZkoNB+YTFRHTKpIgiCIAiCIAgiM0jD0aHMv2hpSrLAMKQE/8oEp9oeTZDyJ6q/KbVZtSS8OH7s1sZbFV9/PIwxcCYf/yHpwFiSDvL5LvBxacKkJKu8AqYct5l15Dhn7kqKoAvV6BlpvA4X8p6rJU2rNHKSE1Dm8+5BT00nmKhmTd1anL5RYFpqIKSmEa/4hFh6Re3AbZzGi8Ah8WxyabLIehYCypFXOiNrk81Cj95XaRN4vjuosQWAXM5oB4rCfJQtXKd7p53M64H7AFdjPDfrxIGjxTkHIvcMA2alZrlbmreuPjF4jGAlRoujtJl+xVHOCtZQakCLMw9oltWEa8zk6j6q8S4vxYynzbBImE0Re4NoPTThIAiCIAiCIIgWkrVZeadBEw6iOagQhODGLl3ZcCrJeYh4zYbRmNRLVpqTpKKi+KM33r+GUdK9ZcIWHCURZhP7ngFbL51QD20T63JdJryhqSBei6Xsehshcsz6CdgIy3q5ZUPuul+6Go9dEEt3rv3S/6ztajvBh6Nz3+vMhKY95VUAAP7U7WLLMRcBXujqVErAwBKzr0KNteoeVh3uGgT/44/FqpOkj1ZYym+Xt+uww0Q3Ep6c+0gdpiTsYwIA01KL0y+1nhWpGbKf25xsp5cz+6ql7Wen+jNnjVHTDWhxQsT5o6j1YfIxD2Iz/Bvi6m8YfQ3i25f1aFfOWCNNNAZNOAiCIAiCIAiipXSsJCQT5uWEo3OlVYcxzIQ9jEgZnm8XLJSUqj464JwoqdQBqcXolVFsdm7RScW0hqE0AXgy5KS2e2ZgKjGgC79xg9rY+tPW5ZLi1rgO5YxFciWyOT688cvmXlR+GstPFkvumxC56lnKFYCZQ2a7hE/vByD9OQAjSvYrwNLjxSotaedmX53sD2ac1n4jyqdhyNRfTx+ZF9W8OPRq9vNptGLWEV8gE5P+6h8AAKXnngIAFE96TfQcesXoM86YOf7smCxnaT1U/U3A7ktSrUQlZqxphulNXP0No+8nu53RNmf91vM64LVKVGdeTjiIzsGEjVUmVT6OHQq/sKrsK52KXSFljUq9gbapzKVJyzvV1rIdOh9FwWT1XSeyq2JuGpAOnPZL/eUb3eGAAZgP+SoZeJPw6vXp45YH+yj7ueo0sVQmVWe9EZBZwrWZwejRANbLWqwPE52x2BEnvko45FTMyA8DlRekWah7U5mz2OfDopixvVFvB4zAWVshdILTeKfl4dDP4YLVJsy0MmHSk4su8AekI7k0cQT3wY4QGba5Mln0K8CWu0R9Kku5Ok6uCDtHkqi3CK7uf3ksZptPyTGJq2zYnANP3Sl+rz43RR993WY9EQiXyXebj1Q1xjLPXDArnO7Uje8FAPT9PyKfRFGOoSxXMPXr/CJdZp0O2WqZTy48WvxQ55r7mPz61QCAga+8NnEfkxB+r3DOn3+2+xFzVHf//Iyfw0wnVUTDdMDrjiAIgiAIgiAOI55vE88azMsJR6dJq7Ig6z56Te+jqs8zUlslsZqp4owX64jYhPZ59WsPNEoCZjtHb/sDAICvOkv8b4dgtNo9uOrI6vVqs6T62zhyVHqpf9CsSGkqZJv3bBbL0jRw5AnBbcyD7ps6J3kGxGVM95sRFndB3bsa7VtIhs8Y+G5xDdmiY8W6XB5cZV3WxRhOXVhd/t8MR85TR57/Y00nkHUX094BOtP4oedMaNrnhFaRrTlH/J/Lg510GQCAT+0T23qGwZ++Q2xXCQK9HLD6TFFOhaLWJkYzQLEvcGzOffMMy3K8MmfGY/W8qISBPQuBNWel7CEQ0IQq7UL4GIzpYA06tKqXs0JqG9Oq3jd9XJR77n6x6qAMD7z2HDCZMZzL8MCM5c0xuTgmn9ytTc743sfENnluWP9i9L3l2jr6GCSg1QEC2hrznHGARd99fbFJ8RoPIT/YhNchAJM0VYc45oC8P3UyXK8AyIzk3DJBPnUkZjxVGqsGAhScvugwGMzmMfNywkEQBEEQBEEQ8xfScBz2dEIYx6xpuhzAkryYsLhSpDLrDotrpECO2KPNCGnrCv8Yg1NSrdqhpC5eHhgSYSlZTtg98+n9ln2uPjjmxoSEsNt5sMb7N3tAOHL21ChXFSWp2iWlfep8VeaAQamdye0Sy+7hgL2zJu4cNyOkbcpraKOvSeR5ZmDDa8RPy3bdtrFWPD0p1p1adyvi2TIh6n9JRvUTraHe8ZT1LdHaTr7yTFmZkpJz8Imdopz0h+K7/wj0SsdwewzZKqX+i4WDuL6XCz2O4A2W8zSCvhKBemUCQADAM78WyzV13KleDpEPK1fQBu3Azk17rDG5cts3AQC5114nVhS6TTvl+KzOE/LFSL9Zz0JzDDW+WVrYyr//vWju6Vcl7loU5aCuEot64DrJotL4uPeMD+na+DdJXNK9NOgEkrafpdKi5ZRfoemkrZXZPC5+v9hZceMxyDdPPP+/3eYzNOHoULI27WvWpCr8kcYYw0P7xaDyAqVeXbi62s5NaUNM4xqvQr34bLOEs98rq5eRuQaWWoc0JgA3/uceAMCHXfXm0jt8h7nhdmGqdl0d+3LOzflZKs2KVF8n9oANrxS/VSbxouUAbn0s8Nv+Vvy66ProQbrrN4fS7bz1U6L+13297jpcZgi8z+Hkrsub+2b7VPV7qBmOn9smG66iYXLzZKxphKxbUG8f+dY7gQVy8ju1VyyXSHNG5oGNyGzfc9NiVe8ilL70cgBA8S++J9YxDzjhdaKcDOLAlYCkPAPWK/JW6A9Ev2yEQXZ0JzUmyQhOfEIIG7zRY4ET35C6b/rdwHLWBEJ9VEbNEZk16eGhSICMMeQWSRPSzdKB/bF7xLbLvmwJhJQZFTcBIVS/5ibN8WcOiqU22WTIjcaYwCbEHv/1Oi/0IV3lVpmJiVjXjLFmstzcpyAQVUxlp6/Rzn0xqZma0ce9M+0fa4jq0ISDIAiCIAiCIFoJOY0TRP1wztFfUKFkpbq6SkjE7OWOzas/4CAsHci1FNFyfLTJOkt1TwNZqhmzQiPc932x3CCdVfc+A75EhtcsCRE8G1gGKKdqywyBnfPe6geZbTxzLzvvAw3XEVt/jQE/6zCOfgdI/w8HOva9Xug1dpgqBLQOmd3lNCksnCMdyQ8+I1b0Gw1rxFST+8GAF2KlMTeyNQPqJCnNwIEtovSiDWl65IBHfzvCntcMwnDGG8XyQZE5vTIlTHVzpQkwnTtEvmtY3vRH9bU0ZTTVc5PB5dBK4IzL0nSq6WQf2CDTRBzir3UI13Xs2OeQaAk04SAIgiAIgiCIlnJ4zcBowkE0By2xYlg3YNkKA8C+p4G10V1ipctNSPzXDJtQriRmyjY63wN+03vEupe8UywLvcDSF8gdTLjJv/jIG6vXqzQHoZCVafjLv7y07n0BGIfpcRn+clIuF60Cnhb20Vgikv3xQ9vAlCTVtrtWPh5x9TdAbP2NIDMncyvsrktLNdi4q00sg4Ws5Zq1ORwS/3UqbNF6naAUEzK0qPTXgB2mWUnwvTywXPp4qOeLV8APPi3qGw1pI0ocXPpk6GeJc5MMT12bXBf0YKt8HpT/iF829S88KnUfhZN66NlySr/jx2u2TGRg55UfiGZueJHYMPYsMHKM2Cb9TtjQCqAkNLG8pIKWMDA1Pu8TWcrRN2KOvbz+0BDaP0YlVLQSxUbiTjNmtEhKq+PlcNZoTMhY+R5iDYyp5yxpznOoQy8r3xS/Ar7tXgAAk0ELwHJA96AobwVBePnyuDDjjYf+veTIGEcYou3MywlH1tkkc+z5/4Isek0afKyJhuIFP9om18kMt6ddGYj8o5j78oWiLR+6NVqx4+XjqsMUD5bnnKP0314KAOi69nfVO1ADPcDbmbTf+k/Vj2/FV7/h8zcBAD546Q3R8g1MNBSf+/yPAAAfu7K+/bXj+jukSZX9Yeh6SeoPEvPS+8xpYhLysQd2RutfelJ9DdNN4Lj+1OUAgOs2NSGnh42Mx1/rI+dQ2BolIUlzdDQrckwjZG02lrVTehLmMj7P+XrfGf1LjUzllLdGt89NiaXKoM052EqVE8M4VH/24gsAOJ7D/sVA3yJdDoDIPq4/FqVQKJCfR47bC9fKTR6uf4UYq9M8h3oi5eXNeKLzD8m2eLAyjFs2qNr525e7lfH//okw87zmvq2BOsB9XY4ph3uYepk6d1ZeJnbS5cH6yzO4/sXHp+6jrk+dP3WsQsLYgdZpv3uP+NA+01WsCcKbO3Y1JyIey4USeuQ8YKWIO+U2o5ITCAbc8pz47TLSa2Siobhlm6gjq6iCRGPMywkHQRAEQRAEQcxbDjOnlnk54VCqR5WZUqslrZk3nxQhSdG7yEhVVPlyCcgph18ZLztvsiWcOdp+Dcel62Ssa2l6o9TcgT7KrKoo9Gg1PJ/cLcrZjoTKaU5m/mSM4fVrmtPHcP4JxhhuOn8FAOCyW6XT4d7H4S17QWTfwnt/HFdzgqPbfQiWZ4yh+IGfJqgjPUlNteLjqjfOrF//YMU5j/QjIGFKWPVFRygJoTTR0FmKDwpHcwBchfsEgJ4R1QKxqJSsgwUzg7NCLy5cno2KPOk1PElmGtfPoTpH+W5z76u+25I/+czx8qwZaxTqWWYeNgy1f6xZqCxzQlqtQNhLJS22wjnrjMO8An0N1XhqhQIdblw42zDFJqT1iaORqxh7LybUhCrNhn/zXwAA9vzy5wCA0c/cBk9qKvjYNlOn0iZIyTnf9QfzvD58s2jXSTIUbt8orntwR7LOWLhDfzsiaeSdmYpkJXLp5XDNXY8DAJ64TISvfeSA2Pjqn/werH8xAMA/KLQfrHvYPI/qmZ05qJ3k+d5NYtMCqcXpX4xr734sSbcyo5DgHo0zO3JZANj3lpfht23S8VR/uynNnWXGp/N7qO+afHdEm8LtnF4hrRJjHk7vgG83ojrzcsJBEARBEARBEPMX0nB0PHtnQ8l1XLNrKTVhjIHzUPlcPmDPGWZXBySPmZqUdqTa/tEh/tDJlDxEMmIzD8YhMNrXfdNi27omtddm05hqs2xfodddMJdO9BmVotS4TinrJ9LTExFYyvCIs+NaYqrVX+EEWNWwJHV9bR6hDpbkPebFNCSQNTkkYfNysc/wWKn9Y01FKqXipZSObTpzspW52lFH1v4TnUCMe1kszQhsEWChkP7/5+NCgvwGLUlG8PkLZfJmPQvNPa60Kkmf1xaTy4nnaEGX48ayfVJ06F/PbGuCn0BWKC2cK+mhRodMdmmPHD54ll9MTwd87S0ZMI7mAILBEJT2SzneW34rJmllyfRJ+wVJDRkDVvWRhqOT6dynjyAIgiAIgiCIeU8HzHnT8/SknPnHSWCK/fqntg3UkoJirGTp2cmGm9gwWiqYUiLD7EgfMRqgcoaCgKU9ocR/tpTNRklrspKkqfqJKLxiNH8lYTPL7TCcoSgxAItcJ8Y89PRJ7ZnSonFjG861pEpF2KlEfH7ArIRjOuqMuTkHets7RG2fCoYKdY4bUjIstKlynfb1MOFGXftur/JotJLpJG4ycRoea5urj7MdEKky60hc9cL9Svx7LGk96jkdEBGp3vD2i8X/938XuPgz4reSildK1tgotM98ai9YUWqiu8W7kx96DoDQfmjfgbraFvUXqwsZ3rd/WIwnp66WmhjXOM998wz6yg9rQvh2AJaPlfB9ZFb97UI9J7HRmuKeQ/vqOM53KcPnMGlUvj/uFddiTVwUL/XtxryoX0q+O1Zj/Fvp13NsgjZ3BOQ03vlUEqjoY8Oz1WCm0v6b4LYdoq0nxoTDY7apklKDJ3Qy/KWs/8X1NS+KCi8IDwu7lKpTvgjGngUcTuPxof70l1vdTWpGKEGjyrUmR1MyXr51rlnIbIxzjnWD1b9ylMNtIx8bawca+IpiOf2C5Q/8L7GuSw70h3YBJ8usuxPSWbR3FGzwCPFbnhNe7MPKr/9WVBd6CbDhVfC33A4A8NacFz28ej4d5n7wzMto9d8/nLJjUVzO0NrRu0a2+Bwz+1TD3mbCIye7rvkO0DEnET40Mp52woQja+p9Evmu34Gre1BlGle5JNa+1JiFalMhD3zHg+Ln8GqxbvAIYEIGCzn1v4h1p10plpU5+A/9MwDAO/kKccxKyTJzlJPlJSeYZ3fjq8W2aZGXh3Nf14++xek7OTcFrh24g4KMQBAU66NVByRQJl/MA3/83wEAo697u9i240mx38xBE/pXBkaBlzcTLBk0JuBIrt4NMr8IH1oJ/tgvRDmZ76MezFij+uL4aIb7eYpz6jbnpvozV+tjP0uncRWCWF9nzp3vtq1aWFz905PFTUaqmWdLnplo/7cbUZ15OeEgCIIgCIIgiPnL4TVBmpcTjredK6QZPGyqk+/WEgB/1+8BAGx0o8ncWhEmHrw0ZdTHSuIiVa2MMXzg3IEMW5+MD3zpbwAA/r4nAEinPgDoWWD6uPsRsW1oJbiSuj/3gFi38kwTNliFtFywRiy9PD5+w181t8HMmI3MVoJmKBha4dxFmdywuLCIdvmQ2taWHLmkO7r+RLW7cUri+0bdhUP7PXkoRireBDOKLeON7M2NecHZ7xVtcknIamQWnvnKawEAXR/6maxW3HN8YjfY6nMBAP4OoQVB1wDY8CpZTl672UNRkyp1L3UPY+bLrwIA9Pzt4wn7FcUpnU+o/bpwtXjWdFhcRaFHP4c6VGOhOxh6GwDK08akTIXHVeZZXh6vWtF+j+pzFqsw40oVYRxO9bMmM7Oje4HZrkzmyjOwwwUDwWf63KXt76OyzIsz/XBpHXkkYR2MJsCSNg9Uj0ESi1eHNJ2t+pPoygERBt2/66vi/013ieVrP6c1G+o5ZINHmOdPXie+9U5AaUz+eIsod/qfiyXzdP11Ueh1nO/o+BfQFDrGR7bxEgDA1GdFEsL9z4rQ90e+5rM6BK8/IcIDs4HlllbIhJjXoXpnD4lNo8eZYx/3qrQ9i+mDQ+tZg4J2Go9qZPVz6AhPreDcj0QvsM9jkrC79RIZT6t0+aq3iXeCDpVekdeoZ1iPGf5Wce+yJScB6jtNadV3P6JNe1n3ArFNvo9ZroC/fNsZjXaFyJAOUOgTBEEQBEEQBPF8ZV5qOL57p5gdvyvGno+NbhRLLweubO1l+DTWvSDicGRLIb51t5B+fLhpLU7PbZ/8FADgZT97l1ihZvVWu9nCo8WPXNFIPNYICQK6hy0hg5TqWXaTP7rucwCA1130t01qsTna7pnwtioWzs7QftH69JpIorpaYXHj6s+erP1U/UZ0N34ZXGn3fOHcr23JOXc7jYekd4wxFF94kVylHL5lkR2/BaRGDTIZJUoTRopqNz3s72BJk7te9Iq0PWsq9+0QbVkV0YhYHSiosIw5YyhdsJzGtZQxGsThV7tEuWOa2Oa03L9P+ovFad26hgCoZ05KW5UWI9cV1VJZ/EbWf14zGlsn09olIJ3W0fjkxL8qJ+bqaxfnfmJfmPh65IM3dRAAUJkQy9z0fmB4JQAp9QfADz4NNiQ1jarPQyvApKSY94nknFz6cLDeRUYz1HBLG0Deb1sfEQkMx6QS8Uj7/BWldQLzIg7WrNBrQuSqe9e+5gk17VkRDmzgvFdr3Ssx93elAwIn/McPbwcAvPTPB4MbrHcAWyr9PXNFo0VW3zALjjJ9tNMCSP7lh/cAAF7Tzo+3NBxeFlXzc8IxIp2STbxqQWDglqpHeD3mA0ZFTU0iIsAAAA3xSURBVGLM3ODa2dXc8As7IH3DEb0hZ2XdTqsQt0wgZDll+sG6Bq2PRVnKM+YEuv46sKOOGJMDFcw/h/e8Wn4+yegg9kAeMGmQjoi8f4lZp8poJ3QrclFFOvoyz6xTzodyUhnIxNqIo2MTSOIQHGviUcNkLM8acxo30duUiYijDTVMAyZv+x4AoP+c98s1stNHnm4cUkdlzBCvYEWUUfe3ZaqiHGStZ3Hi1u8AAAZfdUOibjUbbR4YyLUROhfW+ePKREA9m15B7+P6WCi139oIGwblsxZyeAWMKYeOZNY1ZLYrx3u/bJ5/9axbE7ROyKbejPd63LNav0Nuk784NpwPAMgdeYL4X5niAsZUcWglZv7uNQCArtdfI7YNLAcfe1b8XiuEVqwr9FHYbmQG6vUXnSP+H5Kmrba564yYJKFnWJiJAdDnuG9UmgTCjDH2vrMN2ag2zEyCSXFcMIpaE9eZDgjesFcOGZF+2H1WEwmXYzmzBF9eKJM8gP0dkNeIqA6ZVBEEQRAEQRAEkRnzUsORKIdEQEoQnvUyxKkeOyFm+4YLzhI/lNRFhizlrMdIQFQYxW4GzAgVOnb/Qa4bNNJj7fgsTYxYDqe97NQGWscRlvTavmrFP32L+BF27grXMr5dbFeqfNf819bSKCdVWBIOKUk2mhap/QI39S85PlGvmk13ksyxVjhha6Xa0VoTNWdqXooKpTGErtsl7XbRt1HeR2HNhdJGAeAyaAPryptyKg7+9D4wK+66WKmuaR/6jzutrh41i958eDAw50pfR+ueM/1XYZTzZp3jXHZC9t8RrYwI3Xf2fVqxbIZ4SDvlz1n3anTwHO1q/4Dak8Rp3NKORjS34h/5V5m3mjr66nQabxqqnf3SuVtqNliXyUelJcbc19rn8k+F5jD/5i/qePMuh+Swtrwd8ANPiR/HymDuPTLohd02pSXlPlCSQWUOOTTdWgNntOqqfrb0xCa3PBm5BoXztTTinSD7H05i5azGlZwjaIM/B90Tx306VGj/WJOOTrgqrYM0HARBEARBEARBZEYHyNfSc7yyCQ5lGOVewbI5FqEqea5ozY7lks9aSYFkXVYSt+OG2i/NYetOFz+UL0ogZKWUykitBusaNGE7x2W4uXIJXEqZdfZxyycut/70RlrnWGckC9v+xycBACtuFMnj+MQOMEdYRbZwrVg6nDW1ZEPbenLtuGoKVRDWGihNAWOsZkjXrDGOqjHzeqf/hOpLfIbeqYYSqXNnQiog7KMTXB9u68Tv7gUADFxuEj4BACtaoaWlNJFXSla4VFmud5FlRy3Pk5W5e+zhXwMAhtN1rmlcdLYM6SwdaLV9cbHfOBKXVFjcXhMOVz57/ODTYIPL5b4q6Zjxc3jdmYsybH0ynhwX13O99jWRC/vek+MjY8xo49R18zyzk8PuetMhUb6REadR0jrMhscTIUVX5yOqqUqSjDZT1Bjzu/8jlvI9wM+40oRUV2HkC73o/sD/1b8BgP/nF4BTLxe/N/1cVGmHiW2CY3ujsEXrAQD+LV8AAIw99kcAwIJPX2QKqfclALZKakLUtdy/WQdO4TK5IgZkMtMugI20M3QD0K+H0KBGHLCeRaU5djm4W8E29OvTeh6H2htDBQBwwVtkEBAVxl89S93D5j2hUhosPt58lynn8T2PmnFUaqz0t0WuiEuveFmm7W86lGm88ymqVpunKloo+KWkfoiFXzY3rTKHsD6muhtPkdA4a4VJlco1wlwZxNUkg/vGWW5yn9xv0mT/lB/tzDZ3OOacuptWK3rG1n3iuCvUOvVBFt7XlWU6VJ8d15yHzTWc2Y/t+js/SlXSGO311p/k2NUmHnaZauW6l8oXtgoQoMyhSuNgWCbWqQ+dwHNqm5UptXkpUq57ybIavciWqX3iuerVDu2OaEx2AAo1IfHl9q5B0z+d1dz0b3yvMJlsZ+afur+Vw5HJqhBze7WMROYq+n1SbbsJjBEm0yzOaThCOourfA2W07jKy8TAwWXkONYrIlKhf0Tn30C3dBb3G5JoNB0+vgMA4M+JvnUNSQGULbAKZDQPeUkXBxD4wAUC7xBtgrt4YzObnZhZ3VyH6a0iJkM3mOc2h4zU3z723fsrAMCiy0LfM9Y1ZINHih95x/t74AhtWs265fW3nsedd90BAFj+niY1mGgq7RdbEARBEARBEATxvGVeajhyudA8SasSre6UjdOqCSkbDG0pdxZrLIlAlhk5k/Lc9SI77JE3bhIrVHZiOw/HyDrxI98NtkiEHuXKWbB/aVSzY0kCnr5GOHYf9ZPLmt727dOhMKB2aEabhkWfNcSKbRatNhp8oJb2o8IbEKvacc8Tmku42vP4nUL9fdz7VX1SM9U7Ysr3SrMh5llScXlyCr3RddZxHrn7UQBAIyEOGmHvDqGBWBSWbrvCOALGGdXOm5MLZ+E1+27bLRzqlzevyak5WKoeAELjGW2kLqelrVYQDse9dLAUWdVy5tTliw05Gm27Ma2yQqk7xp12hzfW7dws8hD4T9wPAPDetFEH5WD6GjJtzqol5kedLd4ZAMr/9E4AQP6ESyP1txM2ILSdBx57DADwxDNCc36WXSjfY35rqwB5f/csBHKyv5MiSzlGRC4rxhgweEQWzU5MWQ9/6e5Ra2OsiU4nhMXV+CoIhRXiXqH64Fd0AAP9ash3GcsFVYc2Qcs1rvZvOe1/rlpJB3xaEwRBEARBEATxfGVeajhu2SqW74tk/7WQ0grGGCA1Ikoi5/SHsPjJs6L8cQ22sxH+4ykxe39rQUhsnFJoO5yocurMSSmN7VTmcPa95Wmx7r3NarBl/7xQmV4qG2DLhyPgeBwnrXHYU4clP3G+B2KH9s6nOyG8clX8sk74p32B8pafgpLmasm+Q/rr5dBVCIcolJ2eGQNXYYzV81YpRzLHIt9l6lbHsrSTfW0eoX65U7RtQ5fwsnCGmyz0RNbpLO65IsI22fZ9e9uO9jtUh7NkO/vo8LfSY1JY4xziUJ1ZuJtJEvv12CzkNcaSUr3SY78c78uWEH1PSbt3b+ka8f/kbhM8Qz3fpUnAk8+YeocU+/UznztDJAXk20TABrb+4gYzjZtAJw0hr09FqqvWLJXPnRWyWd+nvAJw7ewpFn7Z+AXI55lL52XW2/7gDeVEWrK4ezT+HHfC++imB8W75moZAMbZ5oGovlePnVJbJ9dGyn37YXFff6zBdraMDtActhLScBAEQRAEQRAEkRnzUsORJMShU0qXcDY51+4QhwDGpNAmTrIWCCcrpVMcXdH9HN0ea5JddThBHOcc60Zl6EGVjHDLPcCRL5LtskL9OSTDpmLLLrMKta5nbP0JUZFdIJPXIVc0CaJUmN5CN6CSGyp/IpbDGzdUb7tOhtdAG69YV7/IiuUKOgKa/w/CX4htlGEkF64A+qUkSfUxVwQb3SB+S4kiL/Ri3Q92uQ8wsAz+ltsBAN6a88Q6S5Crrx2zhiB1zxbNug0/2p2qX0nRYaRlWFCAO581pcBJa8NuR1ervq3xZF/NIG3I2LR0QpSqrE9zvV3k4ztMVCiV5FX5vOW7jKZRjUNeDhjfabar8ipE+mnCN48rqf8zd4GPyqhLKqpPZU5EMQTACkITzgZX6PqY9N3gD35XLNeVTdsCEuaknfStxKWCWs+TM0FjTrR10fHCXzF3pAhjy8eeAaR/h9ZYFHp1dDw1zsEvg3E53u56TLdNVLo+6P+REv0eVH4FvpX4U40rOhGvF/T7gnhfr+qvfhep6+nSNCZl7WD6fVzXQWuurUSS/DnhM6RDC3t5EwlM+9gyXLzGFQZeVSOjqMVE4qqlaXzt6g74eCOqMi8nHFm/wDohxGG9HwGunBYumt5HayBY+U9b5DrZlrOvdpo/zX35QgBA8UO3OhoYk5vDdXiHudXcF88X9X/i4dimx6EHvy4rcOmiDdWPb31A/+gJMdB+0FVvEyZDN28Wy3rVx0x+bHvv/rFc4wolrQoz6M82y4H4sy8UjqYfvX9ntP7V9YdeFk3guP5UoV6/blOT7XJk3+MmBkD9jpZxGa0D5eqrvqlk3YZOGE/LGXey3okjG1ph/gnnGQJMSGk1ufArgM5pZEz1PvMn4iP8Yw/stLYAWHhUIIM6APD+JWA6N5X8MLZM/9AjQuWys6/W+113tpi0pHkOucqJwXLmw16FzdZjSc7kzrBzLql8Gb7JofWZs4WD98d+vTXQf5E5XX6sLjnBNECFBlYmnXbgg9PfbvaF+Ij+5Jmi/ms3pQ8HrJ9zZVrqCskeF9IWwNMTMeZSTTC723wo/T5Owa0rB8iKs6qWD7wTN4vz/WHXsWqcnyTc9JS4xz/RcE2togMGxxZCJlUEQRAEQRAEQWQG4zU9bwmCIAiCIAiCIOqDNBwEQRAEQRAEQWQGTTgIgiAIgiAIgsgMmnAQBEEQBEEQBJEZNOEgCIIgCIIgCCIzaMJBEARBEARBEERm0ISDIAiCIAiCIIjMoAkHQRAEQRAEQRCZQRMOgiAIgiAIgiAygyYcBEEQBEEQBEFkBk04CIIgCIIgCILIDJpwEARBEARBEASRGTThIAiCIAiCIAgiM2jCQRAEQRAEQRBEZtCEgyAIgiAIgiCIzKAJB0EQBEEQBEEQmUETDoIgCIIgCIIgMoMmHARBEARBEARBZAZNOAiCIAiCIAiCyAyacBAEQRAEQRAEkRk04SAIgiAIgiAIIjNowkEQBEEQBEEQRGbQhIMgCIIgCIIgiMygCQdBEARBEARBEJnx/wO9WMEQtQ6e4AAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "rand_example = random.choice(data)\n",
+ "input_ids, input_mask, segment_ids = convert_to_bert_inputs(rand_example[1], rand_example[2], max_len)\n",
+ "_, rand_output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "_, rand_output_base = model_base(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "rand_output_finetuned = rand_output_finetuned.squeeze(0).detach().numpy()\n",
+ "rand_output_base = rand_output_base.squeeze(0).detach().numpy()\n",
+ "crop_len = torch.sum(input_mask != 0)\n",
+ "visualize_all(rand_output_finetuned, crop_len, title=\"Random {} example attention map: fine-tuned model\".format(task))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 45,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAxwAAAMCCAYAAADj28DjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzsnXd4VFX6x793ZtILNUhTqRMgIaA0FVwEUSmi4kqRjroUFf0hioAuCKKiroiABRDpKrCAECsqqMsi9o6AiIUepKQnM5M5vz9OvTOTZCYQcPH9PE+eubn1nHvOPfeet1qMMQaCIAiCIAiCIIhKwHG2C0AQBEEQBEEQxLkLTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARxlkhNTcWQIUPOdjGIc4SuXbuia9euZ7sYf0qGDBmC1NTUs10M4n+ITz75BKmpqZg7d+5ZLcf/Wt/dv38/UlNTMXHixAodP3fuXKSmpuKTTz45zSUjzjaus10AgjhdrFu3DpMmTQq5LSoqCrVr10anTp0watQo1KlT5wyX7n+PrKwsrFixAlu3bsW+fftQWFiIqlWrIj09Hddeey169uwJh6N0mcUnn3yCNWvW4Msvv8SxY8fgcDhw3nnnoUOHDhg0aBDcbnfQMRMnTsT69euD1kdFRaFWrVro1KkTRo4cifr165/WuhKnhxUrVqBNmzZo3ry5Wrd7925s27YNw4cPPyNl2LJlC0pKStCtWze1buzYsTh+/PgZuT5x5sjPz8dLL72EYcOGITk5+bSeu2nTpnjmmWfQpEmT03pegvirQhMO4pyjd+/eto8NADh58iS+/PJLrF69Gm+99RZWrVqFBg0anJ0C/g+wadMm3H///fD5fOjRowf69euHhIQEHD58GO+99x7Gjx+P5cuX49lnn0XNmjVtx3o8HkyZMgXr169HvXr1cO2116JRo0YoKSnBTz/9hI0bN2LVqlW44447MHbs2JDXv+eee3DhhReq/7Ozs/Htt99i3bp1eOutt/DKK6/Qh8CfDI/Hg5kzZ2L69Om2Ccfbb7+N11577YxNOBYtWoT69evbxoD27dufkWsTZ5bvvvsO8+bNQ58+fU77hKN69ero3r37aT0nQfyVoQkHcc7hdrtDvigGDBiASy+9FBMnTsTs2bMxe/bss1C6Pz9ff/01xo0bhzp16uDFF18Mmpj94x//wCuvvIJp06ZhzJgxePnllxEVFaW2z5w5E+vXr8dNN92EqVOnIjo62nb8nXfeibvvvhvz5s1DcnIyhg0bFlSGDh06oHXr1rZ1/fv3x+WXX467774bTz31FJ5//vnTV2nilNm5cye8Xm/Q+u++++6MlcHv9+OHH34gDdhfhDPZtwiCODXIh4P4S3HdddchJiYGn376qW29x+PBggULcO211yIjIwMXX3wxbrjhBqxcuRIlJSW2fbt27YqrrroKubm5ePDBB9GxY0ekp6ejR48eyMzMDLrmG2+8gd69e6Nly5bo2LEjpkyZgry8vJDly87OxsyZM9GtWzekp6ejTZs2GDJkCN577z3bfuvWrUNqaio2bNiA1157Dddccw1atmyJ7t2746233gIAbNy4Eb169UJGRgauueYarFy5Mqx79Oijj8Lv92POnDmlaoFuvvlmDBo0CN9++y1ee+01tX7Pnj14+eWXkZ6ejunTpwdNNgAgMTERzzzzDGrVqoXZs2cjJycnrHIBQPfu3ZGYmBjUfqXx3nvvYfDgwbjooouQkZGBnj174tlnn0VxcbHaZ8KECUhNTcXWrVttx+7fvx+tW7dG79694fF4AACMMbz66qv4+9//jtatW6N169bo1asXnn/+eRQVFdmOHzJkCFq0aIGCggJMnjwZHTp0wEUXXYRbb70Vhw8fRkFBAR566CFcdtllaNOmDQYPHoxdu3bZzpGamoqbbroJ+/btw6hRo9C2bVu0bt0agwcPxrfffltu/UtKSrBo0SLV/y6++GIMGDAAGzduDOv+AcB///tf3HrrrWjfvj3S09PRrVs3PPDAAzhy5IjaZ+LEiejbty8AYNKkSUhNTVV99KOPPsKBAweCfJYKCwvx9NNP45prrkF6ejratWuH4cOH46OPPrJd3+zr77//Pm688Ua0atUK7dq1w//93/8pU6l169ahefPmKCgowPr1623296Hs4P1+P5YtW4YbbrgBrVq1QqtWrXDddddh0aJF8Pl8aj9pk/7AAw/gp59+wm233Ya2bdsiIyMDgwYNwvfff1/uPZR26Z9//jlefPFFXHHFFcjIyECfPn2UrfqSJUvQrVs3tGrVCr1798bbb79dobYwr/fBBx9gyZIluPrqq5Geno6//e1vePrpp1V/lnTt2hUtW7Ystx6yLVauXImNGzeqfnXppZdi6tSptnFN+kA899xzePHFF3HZZZfhrrvuUtvDbf/S6Nq1K/71r38BAK688krVvuVdd+/evZg4cSI6deqEtLQ0ZaYZ+DyF8uGQz7TH48ETTzyBzp07Iz09HV27dsWSJUuCynj8+HHMmDEDXbt2RXp6Ojp06IAxY8bgm2++Cdp3x44dGDZsGFq3bo22bdti5MiR2Lt3b1j3wizbqYw3APDRRx9h2LBhaNu2rarbjBkzQpokLlu2TLVf586d8dRTT4UUOkR6L4hzE9JwEH8pnE4nXC5XkO/BxIkT1cTgtttug9frxTvvvIPp06fj999/D/IN8fv9GDlyJFJSUjBu3DicPHkSixYtwoQJE9CoUSOkpaUB4B+899xzD+rXr4+77roLVapUwbZt23DnnXcGla2wsBCDBw/Gnj170LdvX2RkZCAvLw/r1q3DHXfcgYcffhj9+vWzHbN582bs3bsXQ4cORXFxMV544QWMHz8eBw8exNq1azFw4EAAwMKFCzF9+nS0bNkSGRkZpd6fvXv34ptvvkGnTp3QokWLMu/lrbfeihUrVmDdunXqY3PDhg1gjGHEiBFwOp2lHpuYmIgBAwZgzpw5eOedd9Tx4RATE4P8/Pxy91u+fDlmzJiB9u3bY8KECXC5XNi+fTvmzp2LL774AosWLYJlWXjggQewbds2TJ8+HZmZmYiJiQEAzJgxA16vFzNnzlQTp1mzZmHBggXo3LkzBg4cCMuy8N///hezZ8/GDz/8gHnz5gWVY8KECYiPj8f999+Pzz77TPkaRUdHIzExEffeey927dqFZcuW4Z577sEbb7xhOz4nJwe33XYbLr/8cvTo0QOHDh3CwoULMXz4cGzYsAHnn39+yPozxjBu3Di8++67uO666zBixAgUFBTg9ddfx3333Yf9+/fj9ttvL/Mefvjhhxg9ejQaN26Mu+66C8nJydi1axeWL1+Obdu24fXXX0dCQgIGDRqE+Ph4rFy5EoMGDUL79u3RsmVLPPPMM5g2bRoAYOrUqahevToAPsEfMWIEduzYgZtuugkZGRk4efIk/v3vf2PkyJGYOXMmbrjhBltZ/vOf/2D79u0YPHgwatWqhQ8++ABvvfUWvF4vnn32WXTo0AFTp07FtGnT0L59ewwaNKhMs7sHH3wQa9euxeWXX46+ffvC6XTiww8/xBNPPIEff/xRfcxKsrKyMGLECPTq1Qu9evXC7t27sWzZMowePRqbN28OObkOZPny5cjKysLtt9+OrKwsLFiwAHfeeSeGDh2KrVu34rbbbkNeXh4WLFiAe++9F61bt0bt2rUjaguTZcuW4dixYxg8eDCSkpKwceNGvPDCC8jNzcWUKVPKLW9pvP/++/jtt99w8803IyUlBVu2bMGrr76Kw4cPY/78+bZ9v/32W+zfvx/33nuv8p2rSPsHMnXqVLz44ov49NNPbX2rrOseOXIEAwcOhN/vxy233IJ69erhyJEjWL58OQYOHIhXXnklrInXpEmTcOzYMYwZMwZerxdLlizBY489ZjPly87OxoABA3D8+HH0798fTZs2RVZWFl555RUMGjQICxcuxKWXXgoAOHjwIIYOHYqSkhIMGTIEDRs2xK5du3DbbbcFtWl5nMp4s379ekyaNAmNGjXC6NGjUb16dfzwww945ZVXsHXrVqxbtw7x8fEAeN965JFH0KxZM9x3331wuVzYvHlzSK1TJPeCOIdhBHGOsHbtWuZ2u9n8+fNL3Wfbtm3M7Xaz0aNHq3XFxcVs9OjR7L777rPt6/V6WefOnVlGRgYrLi5W67t06cLcbjd76KGHbPuvX7+eud1uNnv2bLXuhhtuYM2bN2d79+617Ttp0iTmdrvZ4MGD1br58+czt9vNXnjhBdu+ubm5rGPHjqxt27asqKjIVte2bduy7Oxste/ChQuZ2+1mrVu3ZsePH1fr33zzzaCyhWLdunUhy1Aa3bp1Y2lpaczr9TLGGBs8eDBzu93syJEj5R772WefMbfbzSZOnKjW3X///cztdrOvvvoq5DE//vhj0H0LxdGjR1nLli3ZqFGjmN/vt22bOXMmc7vdbNOmTWrd5s2bbffn3XffDXm/JkyYwG655RZWUlJiW9+vXz/mdrvZoUOH1Dp5L8z6McZYz549mdvtZvfcc49t/T/+8Q/mdrvZ77//rta53e6QfVq20yOPPKLWdenShXXp0kX9/9577zG3280WLlxoO9bn87G+ffuytLQ09scff7CyWLRoEbv55puD+u+//vUv5na72WuvvabWyT65du1a276B5WKMsaVLlzK3283efPNN2/q8vDzWpUsX1qFDB+bxeGznzcjIYPv371f7+v1+dtVVV7EWLVqo53Pfvn3M7Xaz+++/33Ze2RaSr7/+mrndbnbLLbcE9Y+RI0cyt9vNvv76a9s5Q5VXPsfbtm0r5Q5y5syZw9xuN7v66qttY8lDDz3E3G4369Kli229fI7XrFmj1kXSFvJ6l112GcvJyVHrPR4Pu/rqq1mLFi1s40Nubq5tv9KQbdGyZUtbWzDG2JAhQ5jb7WY//vgjY4yx7du3M7fbzVq0aBG0byTtXxZyvNi3b59aV9Z1//Of/7ChQ4eyzMxM2/qPPvqIud1uNnny5KDzzJkzR62T/WjkyJG2fiPHMvMd8uijj7JmzZqpfiQ5fPgwa9OmDevdu7dt38D2Zky/U8y+WxqnOt4UFhaydu3asUsvvdT2TmFM90c5Dvl8PnbJJZewNm3asBMnTqj9/H4/GzZsWNAzGMm9kH13+/bt5daZ+N+CTKqIc47i4mLk5OTY/n7//XesW7cO48ePR2xsrE2yGx0djeeffx5PPPEEAC59y8nJQUFBAc4//3wUFRWFVCePGDHC9r+UjB09ehQAd1TfsWMH0tLS0LBhQ9u+AwYMCDrfe++9B8uy0L9/f9v6xMREXHPNNcjJycGXX35p29atWzebs2SzZs0AAF26dEG1atWC1suylYbcXqtWrTL3k9SuXRterxfZ2dkAgD/++AMOhwMpKSlhHSuPCaSgoMDWfgcOHMDbb7+NO++8E06nE3fccUeZ5968eTOKi4vRo0cP5Obm2s511VVXAYAt7GKXLl3Qp08fvPjii9ixYwceffRRNGvWLEgD8Pjjj2PRokVwOBwoKSlR55amZ/v37w8qS58+fWz/y7YIlODK9VlZWUHnCNRsyTp8/vnnpd6DN998EwA3QzPrn5+fj6uuugperzeoPwVyyy234OWXX0bDhg3BGENeXh5ycnKUVuXAgQNlHl9W2RITE9GxY0db2UpKSnDFFVfgxIkT+Omnn2zHXH311ahXr57637IspKWlwefz4cSJExFd/9133wXAn0PLsmzbbrzxRgA82pVJ7dq10aNHD9u6wGe+PHr37m3ThMg279WrV8j15nkr0hY9evRAUlKS+j8qKgpdu3aFz+fDV199pdYnJiba9iuPjh072toC4O0DAJ999pltfcuWLYP2rUj7R0qo63bq1AlLly7FtddeC0CPM3Xr1gUQfn8ePny4rd+E6gdvvvkmGjdujIYNG9rqGBcXh7Zt22LXrl1q3Pz444/hcDjQs2dP23WuvfZaJCYmRlTvio43n376KbKzs9GzZ88gB3z5THzwwQcAgF27duH48ePo1KkTqlatqvYL9f6K9F4Q5y5kUkWcc8ybNy+kaQsApKWl4Z///GeQ2vy3337DnDlzsH37dhw7dgyMMdt206Yb4KZZgS8zaYoj95Ufn2a0JUnjxo2D1u3duxcpKSm2AVwiJyy//PKLTfUcWAbpvF3a+sB6lIbf7w9rv8D7JNcxxoI+5MI5VhI4mZM0atQIzz//PC655JIyz71nzx4A3LygNA4dOmT7f/Lkydi2bRsGDRoEr9eL5557zuYMD/DJ0Zw5c/Dhhx8iKysr6D4F+vsAp95GofpEYmIiEhISyvxA+vnnnwFw+/bSOHjwYKnbAMDr9WLhwoXIzMzEvn37guyzQ9U3HH7++Wfk5eWhXbt2ZZbNNOu74IILgvaRz1xpduOlIW3jmzZtGrRNPmu//vqrbX1Z1w/3uTqVvlCRtghVPylMKK/tyyKS8wbWDYis/bOzs4PqFmg+FYpQ1wX4x+/SpUuxe/duFBQU2LaF258DzRgD+0Fubi6ysrKQlZVVZh0PHTqEKlWqYN++fUhJSVHmShKXy4ULL7wQP/zwQ1jlAirex+QzESpcefXq1VG1alX1TOzbtw9AeO+2SO8Fce5CEw7inKNfv35KggVwjce4ceNQpUoVrFixImhQP3r0KPr374+TJ0+if//+6NixI5KTk2FZFp588smQNqlOp7NMHwUAyok4Li4uaFtsbGzQuoKCAtSoUSPkueQLrbCw0La+NLvxcOzJQyE1E4Ef46Vx6NAhREdHqxdFzZo1sXfvXhw6dKjUF77k8OHDtmuaTJkyxWZ/73K5UKtWrVL9FQKRPh6TJk2yhWg1CXy5JScn4+qrr8by5cuRnp6uJICSoqIiDBo0CL/++it69eqFbt26oVq1anA4HFi8eHGQRFxyqm1Umg13YmIicnNzSz0uPz8flmVh8eLFpeZLKS+a0+TJk7Fx40a0atUKU6dORd26dZUvzHPPPRdW+UsrW82aNTFr1qxS9wn8cJHPwOlAfmiW9WyG+6xFwqn0hYq0ReBYB0BJzAMdxyMhkvOG6r+RtH+fPn2CJtahnJ0DCXXdNWvW4MEHH0SdOnVw5513okmTJoiNjUV2dnapIbpDUV5flONPs2bNMHny5FL3k2NkUVFRqVrlUO+KsqhoHyvrmZDlkAE+5LMRzrst0ntBnLvQhIM45zj//PPRoUMH27px48bh4YcfxqxZs/Dggw/atq1fvx4nTpzA7bffjrvvvtu2rbxJRVnIl5IZEUkSKFkD+Eu8NGdoOcBH6kAYKdKhvDTHdpMDBw5g//79aNOmDVwuPpS0atUKn376KbZt21auI7g0abrooouCtqWlpQWFxY0EeZ9q1aoV1BdKY+fOnXj11VfRtGlTfP/991izZo2tDps3b8avv/6K6667Dk8++aTt2FWrVlW4rOUR+OEryc3NtZnNBZKQkADGGNxud6kT2bLIyspCZmYmGjRogGXLltk+JH755ZeIzxdYtry8vLDb5nQjP5hDPYdyXWU/a5FQ0bYIjJwGQE1Sy+o75RGqT0Zy3kjaf9asWSHH0Irw0ksvwel0YvHixTYz10iiQYWD7DterzesOsbGxpZax1B9tDIo65kAeJvLesn+F867LdJ7QZy7kA8H8Zdg4MCByMjIwMqVK4Ps1qXpU2CUjOzsbOzevbvC15QSG6l+Ngl13iZNmuDo0aMh/UWkiVAoU6zTSePGjZGWloYvvvgiyBY7kMWLFwOw2wz37t0bAH+xlyVBzc/Px+rVqxEfH18pybWkyUcoHwWPxxMUltjn82HSpEmoWrUqXn75ZbRv3x4zZ860aXpkP7nsssuCjv36669PdxUUR48eDdJkHD9+HAUFBWX62kgNUah7kJOTU64Z0MGDB8EYw8UXXxwktSyvb5RHkyZNUFRUhB07dgRtO3HiRJnmdqcDeW9CPYfSFK1Ro0aVWoZIqGhbyLqYyH4crp9WKEJ9oEdy3kjav3Xr1ujQoYPtr6Ls378ftWvXDvKpK8sXqiIkJSXhvPPOw2+//YZjx44FbQ8c4+vWrYujR48GfcB7PB789ttvp7VspVHWM3H06FFkZ2erZ0L6vITzbov0XhDnLjThIP4SOBwOTJ8+HQ6HAw888IBtYJeZsk21vd/vx8yZM5Wda0UkbNWrV0fjxo3x3XffBdk1r169Omh/+eEdKC0/ceIE3nnnHaSkpITUBpxuJkyYAMuyMG7cOOzcuTPkPqtXr8bKlSuRnp6uHAoBnjfixhtvxN69e3HfffeFlLDm5eVh3LhxOHLkCMaNGxeRs2q4dO3aFdHR0cjMzAx6yS1duhSXXnqp7UPthRdewI4dO/DAAw8gOTkZ06ZNg8fjwQMPPKD2kVqCQPOO559/Xk1gQtX3VPH7/Vi3bp1t3aZNmwAAbdu2LfU46eC8dOlSm68JYwz33XcfOnfuXGo+GEDXN9AR/uOPP1Y5S8z6SrOtwGfF4XAErZNle+mll2zrPR4PbrnlFvTu3TtsPyITqZEs73m95pprAPBnzfy4ZYypZ1M6Qf8ZiLQtJG+99ZZN4uzxeLBlyxZER0fbwmPn5eWVaZ4XyNatW4Nyf0hH/DZt2pR7/Olq/9L6XGnUrFkTJ06csGloDh06hOXLlwM4vc9vjx494PP5sGzZMtv67Oxs3HDDDbjtttvUunbt2sHn86l7KMnMzDxjGo727dujevXqePPNN4McuOU7ST43zZs3R1JSErZu3WobQ/x+P9asWRN07kjuBXHuQiZVxF+G5s2bY+jQoXjppZcwd+5c3HvvvQD4IPr888/jqaeeQn5+PmJiYpCZmYm4uDgMGDAA8+fPx4IFC9C3b98yP/BCMWrUKEyYMAEjRoxA//79kZiYiG3btiE7OzvIpnbgwIHYuHEj5s6di6ysLGRkZOD48eP497//jdzcXDzzzDPKdKkyueSSS/Dwww/joYcewk033YRrrrkG7dq1Q0JCArKysvD+++/jiy++QFpaGubPnx9kdjZ16lT88ccfePvtt/HNN9+gd+/eaNy4Mfx+P3766SdkZmbijz/+wOjRozF06NBKqUONGjVw77334tFHH8WAAQMwZMgQJCUl4dNPP8Vrr72Gtm3bqsnbzp078cILL6BLly7qQ6hRo0YYNWoU5s6di1WrVqF///7o3Lkz4uPj8dJLLyE6OhopKSnYvHkz9u/fj7vuugsPP/wwVqxYAQDo3LnzaatLvXr18Oqrr2Lfvn1IT0/H/v37sWjRIlSpUgWDBw8u9bgrr7wSV111Fd59910MHz4c119/PXw+H9544w188sknGDNmTJkRcOrXr4+MjAx8+umnmDFjBlq2bIkff/wRGzZswOOPP45Ro0Zh06ZNcLvd6NGjh/IHWblyJYqKinDxxRejVatWqF+/Pj7++GM89thjqFOnDoYPH44BAwYgMzMTmZmZKC4uxpVXXom8vDysXbsWO3bswIwZM0r1OymLmjVrIjY2Fv/5z38wf/58XHjhhSE1aGlpaRg4cCBefvlljB49WkVu2rx5M7Zv344RI0aEdJ49W0TaFpIGDRpg0KBBuP7661GlShWsW7cOBw8exK233mqb6F933XU4evRo2Jm7MzIyMHjwYPTt2xe1atXC+++/j6+++grdu3cPSwt7utpf9rknn3wS7dq1w/XXX1/m/j169MCiRYswduxYXHvttcjKysKyZcswefJkPPbYY/jxxx/xyiuv4IorrgjrPpTFmDFj8P7772P+/Pk4duwY2rVrh2PHjuHVV1/FsWPHbGPfsGHDsH79ekydOhV79uzBBRdcgJ07d+Ldd99Fenp6WMklT5Xo6Gj885//xD333INBgwbhpptuQlJSEr755husWbMGrVu3ViamUVFRuOWWW/DMM8+oscXhcOD9998P6d8Tyb0gzl1owkH8pRg7dizeeecdvPTSS+jevTvS09ORmpqK2bNnY968eXjyySdRo0YN9OrVC2PHjsXRo0exZcsWFcYx0gnH9ddfD4/Hg8WLF2PWrFlITk5Gly5d8PDDD6vQppLo6GgsW7YM8+bNw6ZNm7BmzRrExcWhVatWmD59epkRPk43ffv2RatWrbB48WJs374d77zzDrxeL2JiYtCqVSvMmDEDffr0CTkBio2NxYIFC7BhwwasX78eq1evRnZ2NhhjSElJwaWXXoohQ4aUmYDwdDBs2DDUq1cPS5YswezZs+HxeFCvXj2MHDkSo0aNgsvlUqZUUVFRmDp1qu34kSNH4s0338Tjjz+OTp06oV69epg/fz6efPJJPP/880hKSkKXLl3Ux9Ebb7yB7du3w+VyndYJR3R0NBYuXIiZM2diw4YN8Hq9yMjIwKRJk1Ro4dKYPXs2lixZgg0bNmDatGmwLAtNmjTBjBkzyvWxsSwLs2fPxsMPP4yNGzciMzMTF110EZYuXQq3243+/ftj48aNmDVrFq666iq0bdsWN954I9566y0899xzmDZtGlq1aoW7774bBw4cwMqVK5Gamorhw4cjOjoaS5YswYIFC/D2229jy5YtiIqKQlpaGubNmxf0bIRLVFQUJk6ciNmzZ+O5555Dv379SjXZmzJlCho3bozVq1fjkUcegcPhCPvenGkibQtJv379cPLkSaxYsQL79u1DjRo1MHbsWIwePfqUytOxY0fUr18fCxYswM8//4ykpCQMGTIE48ePD+v409X+AwYMwNatW7F161Z8//335Wqlxo4dC4/Hg02bNmHatGlo2rQppk2bhiuvvBIFBQV44oknMGvWrDITRoZL1apVsXr1ajz77LPYsmULXnvtNTWey4SkkoYNG+Kll17CU089hcWLF8PpdOKiiy7CwoULMWfOnDMy4QCAnj17okqVKpg/fz7mzJkTNGaaQrIxY8bA5XJh1apVePzxx1GjRg307NkTo0aNCjJ7i+ReEOcuFqtsY1mCIP7nOXDgAK688ko0bdoUmZmZER8/b948zJ07FxMnTiw15C0RTGpqKho2bIi33377bBeF+B9i7ty5mDdvHmbnGElFAAAgAElEQVTNmoVevXqdtvPKrNXjx4/HyJEjT9t5CYI49yEfDoIgyqVevXpo3749du/erZI/RcL1118Pp9OJFStWlBqJiyAIgiCIcxOacBAEERaTJ0+Gy+XCPffcg3nz5iEzMzPshFTnn38+hg8fjv3792Pw4MFYtWoV1q5dWylO1gRBEARB/LmgCQdBEGHRrFkzLFu2DA0aNMALL7yAKVOmlBrFKhTjx4/HuHHjcPToUUybNg1PP/30KSUfIwiCIAjifwPy4SAIgiAIgiAIotIgDQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0aMJBEARBEARBEESlQRMOgiAIgiAIgiAqDZpwEARBEARBEARRadCEgyAIgiAIgiCISoMmHARBEARBEARBVBo04SAIgiAIgiAIotKgCQdBEARBEARBEJUGTTgIgiAIgiAIgqg0XGfiIuzYbr7gKwYc4pLOGP7rcIqdGFjhcb7s9wEArNgqYMW5fDmuOt9WnAO2fRFfd/mdfJ1l6XNafA7F8o/yTQkpgK9Q7Ceu7SsEohP1MgBEJfDfkmKww9/worX4e9h1/LVPNADgwuX7VBnA/OK65ryO2Q9kDP/tXQcA0PH1I8Y+ok6sxH4O5gcscc/EfVL3FAD8XrG/U63aeRM/f7O1RxCIlZBSXtUUx26tCgCo/swuozwB5RN14ussY13AvfD7gBJR1qg4WRp9TrWfOL9D1yckoc4vjrWqnF9OzTh59/D9Eqb+V98/VYcQdTHLxALaVZYB0O1j3g+1T4neFtRvnFD9JeQ95euspDqlVyrwcpnj+TFX3KvPrepm/C/LINdZpdx/WUddqIBzQtef+XWbO2Rdma5bqDaWY0G1hqGvH7j779vUudQxuYf4b2JtUTQHUOLRy4C9z4m6z7u8Ae7cIscuMU7I5yX3MJB4nthWxH+j4oP7oXw++ErbupPj01D1yW/5ljD7KAD4nrgEAOC8Y6M+s2WXHTHGQowdTC1boh8xxmzLfMEvDtPt8dDFvI9N/eIgZBvLa/Lj7OvMcsjzI75m2HV8qFkUv97n+3T/8Bbw3+gEWevQ44/sY3Kd5QB7dQxfvHm+KJg4zlsERMeL3Xg/ZX6jzeQ1o+L0+XzFwef/aBYAwNHrycjq98UBwCv6lku8Ez3GNWU/dUaLI413g7yvliNgzIB+Dzhcqrzsj518l9qt7OMOXwge19SY44Bq34RaYdUPAPz7P+XHuGLAjv3El2ulibrxe872bobl7snXifvAsr6DdV6GOIkYX5xR+puh4A++nzxn1Qv1+1y2YdYPsOpezNflZfHf2KpgXyzlx7S7xX5+h1PdY0uOE+VwcCDvh3Xm79H3UY1l5qdVwBgeCssKMXbI40oAR5ReBuzvBtlHHFGh3zGqGKIN5bgVBrKfPvTloaDxIdS70TKuH2o8YaIvWrKvB+4LALkH+T7J9cIqozzWsiz17EZSx7OJvL9n7fo7vWf0eqThIAiCIAiCIAii0rAYK2vafXrwf/cqv1jNVLCv+DIyhPbgwOf897w0rWUozuG/sVVgxSQBANhhLgnEsV+BC9oDABy1WvBtcgZtOWyzXb6t7OqF2q8iEjmWJ7QHJcVgX4v6turPt731EN/W5BJYjbvxdR/O5uvcnWGdfylfd3wPX+eMhhXN662kqKakS0hULaEZYL5iLdVzGpIVKU2IrcL3U9JoS0knIpFY5dzBpbtJj38HduIXfnwil3xaMYniGiU2yaiEKYm5pesj9lOSxaKTfFtMcpCkNFKY36eluWHW0f/uQ3yhaVetUVMbxb2Lr66lTabkXl24xNhfat6i7PsZ0i+znuoeKYmQK7g/y3IYmq5IpDns+M9BZVCSKtnHYGlpa4m4XnE2EJMsCyp+9T2WEkub9FVJD0X9/b7QWrkgTaApGRP9NEwNAMv6QZQnQbeFqMu09lzjMWXTh4Anj583ub4om1fXWY5D3gJdNvEMqfo5XGA71vE1LfrwdZ4C3dZOo86B2qkQ/SZcqSoAsLzDonyFdo0e3yquEaXLIMtcnKO1iQXH+N7MryWJnnx7+UqK9bGy7b0FWnoupeSxVdT9lFJoJs4P4/yRaOKUxnjRDl0uqVERY62V0kzfQ7mPK05ro6TUGwB771F+zNVTRNmlZM/S9YiOV/ur8dQcdwOfX3OcE9p5q2azsOonNfAozgW8+aJeol2zuCYCtZorLR0rPKHKozR3Yh3b/wmsum34uoB7bFlWyHeiVZYkPLCsFdRS+ZcO4Nft8y9jzOD3kGV9r8tY5yK+IO4JO/wNrIZd7GXYmQnUac33r3qh7VxwRgPy3SHbvCgb7KR4R1UV98sVq9tMPgei7dmJX9Q4alVvHFb9WM4BvuAtAHvvcb58+e38N49rVa2UFnaNnFwKuP/M1Pi7YoP2CdQMRNqG6hqIcKwR73lExYFte44f33YoXyc0ccyTC6t6E3kR/utwak2dS2rnoJ81UUcY73w+3sB4X2jpuyU1YoY2VY15pjWC1HBEMNacTf5qGo4zYlIF8+Mt7Vr+Kz+uY+WLrFCrjaUa+/jPYLVb8eWkumLdbyFML8qeXAQ93CH2C3eCUir5Qm3rjILV/Fq1DAA4Kbad1xLs14/4snwIXbH6IT0qXjQpzYJVo+JclmWB2UwZwF+AoUyc1EB3ekjsy81x2Oo7YPWdx1eKDw1mqk3lQx/S5Mgoc8DHpxXLTbaYJ1+VPfBlGf46q2wVdgisS0YCAKa1uxBTP/3VVjabqZCcrCmTBqduC9kODhfMFwzf5FJlDfWysAwTDb2utHPoj/EK4SvSg768npq0RhumAWJbfE0tCJAfnwV5+kPcJz/M5ADqMMw2THPAgImkZUGZOsqPO/nh7y3Uy+Ei+9DxPXrSKPrSlCkDxU5MtaEUDiDtOliB44+vWNevKJv/yomVwwWcly4roc8rPh7BhDAgP0uXI3CiwZhxvyLANAdTJpTi3H6xzZOnP75kH4qK02UV98mSEwWzHrL+DmfwRKYoGxDCBUSJtvEVa3MX+WyIcZ0Vnoy8fgAueOptfvyBz/VH7ueirS4Q74TqTQxzTHHfzftpTrjaDeaLxaIdjcmLFkiI8dLvM9rIMC00BQvyvBJXHCJBvW/A9PMkhW/JQoiTVEd/TMl+lb0PqN6IL8sPtKIc9ZyEetcFjSERfqhWmHxxr5kfbMu/+LU7j+O/cpLhLdSTVfGhaV1wWdDHp9XwCrA97/J1VS8AAPieuxEA4Lp9rerPqp/GJmuzLGXG6dRjXoBZkFWlPlBsPgvloz/+/bCufYwvf/AEAKDow7UAgLgp2/T+ZXxjWA4XmBQghWofZ7R9/wq0oeWI/HOP5eznC1+uhnXlRL7uyHf8fEJYY8VUMcxKjYmcfP/LPlziCfpOseKq8X+ZP/gZcjgR+G3HJ9Byu/4mUueoQB2JMwe1DkEQBEEQBEGcQc7Q1P9Pw5mZcAgnL1S9UC/Lma5Uj3oLYIkZLhNmVMg5oJya2Rv/5Ovqp8OKrXYaCmV3bg0y+YkUKSXzFWlpqJQONr+c/5YUAzs/AADsWrYcAJD6Yj89KxeSQyuhVgiJmemEa5eEMBhSYtmk3gJD+hdwDstRMSnXt1zCZA18FuyrlXy5VT/bLtxZVajNmXZQDTSzMlWjpkkcAFjlaGZCawcCNQFOw4QsXPg5pry6CN/dxE14Wr4qnIaliVsp9y1YRe7XtztAqn86JIwVNjlzGNoEKcmOr8F/5TNQ4tF9R0oHffmG2ZThzBhoLiQ1bM5oQ/ouy+rT0q5Ax1TA0GaEcNQPE/bHLlUH9hOXMlqtuWmj1X2qKFsU2MHPxbYBep00x/ri33xbjynK9MNK4eabSsvjcBpmBNIkJzFYcppUF3bHcWin4+hEw4wtAqS0s8Sj7yeTpnbifrritIRfaa6itLmCaHuWf1QHjpDmmLK8viKYYwYA4TzttJfDk6eX5bgl63XwCyApfBMOxcnf+GUbXA4mnf5F+Sz3Nfx/h1OX1dRESpMqqZ1zxSL3sesAAMlPijaWjtpR8YB830gsh66PModkgCvAcddwlPcLSbAy9ykHJeX2e7WG93weDEAGSmG+Yi2V/kOMQ7HJuv/I5zE2GezEXr4stGlWoBY2gjKdNg1Ii87yxLA638MX1/Ffq+9cvi0myd4WgF37LUw6ma8IqNWcrxN90TV6DQDgzV4N0fMtIYmX7xJXrB7/TcfzwKAtkuI8ZdIbce1dMep5sq64DwDw4WM8iEB3T542M5RO6Q5nyHtd1n0/Y1qpQKRJVduBYNt5wAXlcM8ME22JabYtx3PZ151RygwQUrNhOrwHEd671rZ/RcZT4oxBGg6CIAiCIAiCOIOcrXnk2eKMTDhUKLwSD7DrPb4sfTekdNWcEedwCQ8Y0zPmK+7iv9+sASvis2QrkWsVIpUM8HWBdq0BfhGRIm1OXTGG7bSQsDURDnA73gAa8FB9qXdzn5TDE7qizgIhncoT2h9DEhPS7jNQuu0vgSUkcUzaUjqjDSc0KTk+NQmW1Y9LbRBXHWh8BQDg+HhuJ1t9tpQsF+vwgsrRWYccNSVvzBkTsJ9wagsRUpCxyMsdsc2qlDTWuQhfHeflTRfOr8qJH1ZY5QjlOP9nQDqmWjHJuh/JPqP8MYr1AdIPgzEjLKno365Ye5ACfmaxf4m265c4XHZ7akBIJKW013ACBbiEOkJNjlUzlZ92zyYgKtZWL5bDwy2i6CSQw6Xm7Khwoq9SG9b5l/HlHkKbUeKDVUv4aUjJnHyWnDHBwRuKc43QpuKewq8lfFKSHi2leSxyHxVAS4Qt4/5IzYskytL3WmpdfMWAUyxLP6tqjXQbS0xfFmbXBMNy6P5i+hwprYLTtr/V5JrI6wcAwveD5R/V12s/hK879DU/d902QIkYOw59yfdJPl+9F+Q5EFsFyVPEe0f2baklTqilg2vE8HozxrTDK6TGyqXHeOkvIB2OHQg7bHMQJV59j6V0XgZKMYMWSIrzVNmVViMmCVaNphW7vkFp41qFpet7P+O/Fw1U/cPqOwcAwD54Smzrq3xoVCj7qDg9xggpuhUVp3w3AkOW93z7ANgro/mqAdyxmXny1beDdLi3ohO0xkqO50wHPpCBT8KFmT5qASG1u7/Dx5clneti+Jbfg477s74fghC+s1ZcNbCTYswUVioyGAQPWiBCEMv74C00fEiN0Orm9x5g69/BVgL6PpnfP6UFBrIsq2LjKXHGoLC4BEEQBEEQBEFUGmcm8Z8pVUoQM9x9IuFVc5H0x7TlFDbFVrVGYFJy9916/uuMUrNoPevViXXClRwEz5L9xjb+G5FcR9ogFp7QWhlpw/7fhfw39W+AlJjm89CONVu8oyVndXnYPxTnKekNC/BnYIwZNrzSR4NxiQ5gTzLnNbQuRo3MukZSR3b0R35MfA0l3aj+8Ba+7WUe4ckaOF+3jyynMzo4AlFgnQAluWIlHh0xKNR+suzlRK6KGHlfouIx5JXX+bKQjh2ZeCUA4LwX9oCpeXrkdtJnG0tKmHwe3RZKKiUTjgX70DBPHiwZLUgl74zS98xv+G4AQjsZYJ9raiukBNm0q1Z9N0QSyXCR5fntS1gdR9vPJ7U7NZoCQiJ89P9aAgBSnvpCSztVskmHCrcqbY1tiTKlhFte0xWjQnvK8KTsyPfBCazMSHJBkY/CQI4J3kLdZjISlrBDZ8V5sOKlNkb6fHh1mGOlkY3V26WWQv7GVVPhc2WZWX6WDiUsMUPGymde9aV4wBFZBCcAQHw1fe5jQgtVW0QdOihCqtbvoBND1m0n6hPNQ83KZVEW6Z9jNfibKKfUBGtNFVNhdI3n2XwW5PlUOE4jJLGsb5io9050UvBGqcGPr6nPmyWiOh76CZCJ8qRGoM7F0D58HHtUH7uPXOB2/r9f30tndMh9Iqam8Gcp8WmtkIzE1JEnYtw/JgP1F3FNjUoGJ9sP0H4dLqY0OpZsEyPKnvV3oTER3wtFs65D7P3v881SM+Iv0eHcq4o+afroFEUYpUpK3/0lWsspnwPhxzn8w0M4OoaH2a05RfiGndcSzOez1TmUFcCfAauaiIjGSmBdNZkvy3ov4eFxreHLAJkWQI4lcdV01C+pBTd9c2TbGVYItu8FIGTES5uPaIC83JbaoAJ1PRv81ST+ZyQPh8o07ozR8dmzhZpRDPJW1Qu0w6EMtfr9elhtRwAAprXjg8aU1/4NJHOHXoccSOR1yolNXdYAyoyYzhWKOb5WZLLt/H/6w1w4lbLvxWSpWQ9YwoGSHfqKr4tOVHGx2b7t/Lg6rfUHTtUGooCGc6SKDR+jtwW+BM3MsWZOAFFHSSQ5HLKGczO4lHk/GSpkI2MvuLpVOU4acbT1B6n++Kyo43No06syCLMd/T+9xc9bt43+cBL3mn3LHRSx+2NYfWbaznvWXxCR9NNvhbP/hX8zJgvSqVK8LEynWdmGxblqEqydAKNDOJIb4USDHMT9WqggiYqzm3CZmNl/w82lsoiHysSlg3RYTOXkK8rvK9Y5F2Qs+bzDRux2sV9cNfjn38SLMmIxXydDmAL6GZBjWmwVXWefMdkP7B9mOGvpqBqBSQzL3qePl3WTJl9ROpcEOyKEOjJHQ4lHjbEyh4AVV1XXwzgWgHAGF2aPMkx1cZ7uE+qDz9ITGflRLrM7xySr+kdidpQ3jk9qEh7crMc56Ux9jJtvWrXSdWZpmXW6emNtSiM+Pi1XnJ5cy/Yzc89IUz7ZJ8ycCKZQS00sZWhVo19Jc8ww4/+rcKNFRm4UOTbKfBQlXv2+EFm74YyCVZ33FfY+z2pudRmnjw0xJpWVmyqc/WzjWwRjzcGbeZ+p/dRH3IQTMEJDy/dZFNjrD/DrdH+Qr4tJ1pMOw+Gb5QtTHuFwrExAGTPaU5sRKlMq2SYlHn19NUGUY5lhIinMMstD5d6yvccCTBCN3EOHb+dO7+c98obO9aGCPMSqY081B1W5RJJf7MSvfCE6Psghm4kw/lZ8DR3G/nceBtiq10592ylTuSr11VgZ+N3BJxJmFnUAYNrUvUQHIAkdgCZgch1BfrGzyYzmZzcPx4M/not5OAiCIAiCIAiCABAsizrXOTMmVXKGG1cdOM4lUcgRWVXPE9nCcw/BiuPmHkwmZ3JGK2nPlA08qy9+eBvoMMJ+/jKUNKVpPYKkN8asuiIKOeuah8Thflg13HxZSsIadeK/X6/SQflaiuzErz8MDF3Clw+LxH8XdtJhIZUq0QgVGuSUbTimKideS6vchROlTBjIw/JFbspRc+oGXUdZLm+AeVfOPrBt3ITM6jlDH1wSECLVUwAmJIqBEh2zzUJlkTc1NKVprSqSidWq25Yv+Aq1hFpK5kR2exzfB/Y6z1Zs3SQcIB0uJf00Vb9nXfMRAquBCFVZnKulfcq0ypASqqzY0skyQYeozBZ9M7m+krYrUyMpfTK1TyWGFEVK0WXiJ4cLzCElW+KayjwwXi+HmcPS6sMTjCmTGUCZG7FfPuD/H/4RaHApX87i2kjUbMKddGFIx/w+WN3vFuUWBTDDbors0iokdn6WrquUwhYeD5aqy3saKhFiGDCRNNVKaR7saG9omqz6HfiicJa3ElK0JF46fOYc0CZfKgGkkGT6SwCHCNogNZSm1lK2cd5hQIYql8+mDGd+fI8OHxwBCdOFw7E3X5vSiIzRJRu4htF59WjlrKoychfnqKz00vkavmJdLhVuVdQjKg7shNCOpHAJNMs9rKXdsh1jku2JMQFDmxfgdB8OZqJEf4Bzv8CKrQKWLcLtCnMylvWD6qeox4OxsKKT2hxKSK/L0uab2cfL2+9UqD2MPztWUl1932UdZVjmY98Df+PWAeyNh/i2dn1h1RFaOTlelHj1fZLmgyrstlMnRpT7e/JVIBWZ7JHteQ9W8+v5OqWpNDSzvgDta3kIs0TL1CqqF7xMlHlSBYuo/Tx/vx8c6UbdZ7/h222mw6f+vjjtoY2lSXtUnN3UDcaY74wG2yZC5l5xr9ioQ9SqMPeeAruGGKYZVVQ5Y6GoV2mJhSHfuX81I6X/Lah1CIIgCIIgCIKoNM6MSZWUjpV4gO/e5svpMlyimIlHJ4J5+AzaihG2mWnXaQnndm57jlY9Q1xA202GmuGH56YSYHsZKYa0RYWNk3bZrz/K/+8/G+zkr3ydnJ2nddFS8SoiQZbl0Al36ghHcsOhj0lnLCnBs5xgJ7ljpbINtZzBDtrSydVyGgl6wkdqbpTkCIZtt7jHjgsvB7uAa3SWdea2s0M/PKQ1ILIvRMcbfh2lJ8YzJRbBYfN8pfpxnJKExxkd7Ogs7YXbDQa+4v4c7AMuTbe63KecBJmUiCK0s+ZZx5T8S6mv7E9SIwdwp3IA8Mu+lqyl+1W47xTLPaSl46pPGFqSwLaxrIAkXELCFbif7NfeohDJK8tBlsNXaGhTxLN9RIZuNqSlwlHUqt4ELFeEzZVjTmwVFRZS2+9L51CmbcSldN/h0n3ab9j3y3uubP/9ev8QwRHKRfkXGSFV5X1SEsMYHfpWSiJLDF8ZOU46o+3+JsY2uOKgk3sJnNHKx07548RV18eY/QtiLC/NR6cs1D2K0vdNaESd1cU4efIg0Lgb3136Ce77AhCJyZhIlmdVb6y1S8qHTPiVOKOCkuUhOl4/71I6W+JR/cNRk4+DzFd6IIywq+kt5Jp//p+4pgwJ61f+fSqxXN02akza+egEAECzxV9qH8QAO/hTcdE8ZWn5AaGx9/uM8NrC4Vf4pFgNr9Bj7HVcc8UyJwE3drTtD4cLvhXjAABR48Q3hKmRUxorUdaYJP38i7HZSr/JCChgjHXiXGwPD51s1WwWXv3kuQwNhy0UPMD9JeRzJywO6i78GcVP8CAkMWP/LcoTozWLgb5U5WB/z5zid0wg0teC+fWzILRzJc9xbZFzzDpYnXnbsJ9FcuBGV+oyKG3vCRWqnAX4l1qWQwW+0Yl/raDEgNy5PjClgWkN8eezKiiL/63SnjpnxGlcOuMioRZwUMRLlxFHpFlCXHU9uEgnzIQUbqYAaIfynz8CGnBTAaueMHMJ8ZCFzF8RFtqkyhaRpryjhAMZO/QVsF/U8TzhfCZfzLFVYFXhDtUslzttImsnrJZ9+XIBvxds7xZYacLkSjnZyXjwhhrRjP0d6qUgByIVdURHFLICnAzDwb+DZ252NOisz2VmMRbn1mZTIsLPl8thZYg6ysEnP0u/WAPylpRmAlfRgADh1lGa/qGkWH9AKSdD0YZ+rzJpURnlj/0KK120l4rR7zXyUEh1cCXN7yNxAhTOtTxCkpzwiXLKj0V/iWGaIF6qDpc9EzDA1e2BEzJl9hYiApOZc0NO6Pxe3adVXgfD4VJgyQ//cvB/wuPwo6ZbOynLyc1uERRAZHQGtHkSGOOOjoBy5EZUPNiOjfyYFr35OrOvBmTNhd+nJysqU7tPCxcCJ92WQ39IRuBQrdqQ+YP6mG1yI9vJb5isqXHEiEwmJ3iy7PJjzO/Tk4X46nofeazKWl+s21N+IJjmceKZCDcLNwD4V93GFzqN1hGAfv0PP88FIl9KXDXDrMlw7pb3wOivvrk807jrztfs5SvO1fU3o6Qp0x+ZBduY+AaOxZYDEJMRq3arsOqnxpqiE/rdFuj47CtS9VJmVDn7YNUXGck/fZGvq1JbBe2wWvbTZYLdfOpMO42/9zd+f65c/6O+j+IDVuXEyT8Cq7YQqolvAeYr1oIM2ZbeAj3BVyaAolyuOEAI+ZTJTv5R6Amc+IC1XLrvygABsr+W+JTpVbhtqK5ZDupdKScozK/63J7+vC5NlnyvnxNnxRyJwzbjjeSd/5t45qo3Do5iqCLi5XKHcHAhFCAmP9KUVUYHTKqr74EcM5VJu8OIDijHLWPslDAW9J1gdxoXyxF8u51NHm1xdp3GJ+8gp3GCIAiCIAiCOGf5E7p5VipnJtO4lE5GJ4HJrOO7ReZXN1eJW84YsKMiVrp03rUcyiHz8J08Xv55DywFcsWMWbWWnukGzvDDVQeHdE6OBKkOTKwNSDPYQzYAACAASURBVGm3RKm7GdhmES+8LZdEFb/+HGJTe/DNJ0WM8EZXgB0TjqHyfpnS8cDymSY7KnSpdiRX9VFSe8PxKoIq+tY+DACIHt9ZmwkFmhMY5lFMqE+t9D5gUlLS/Aa+MaGWdioLI3dKee14OpzkpFmGwwiLqO6dkYXWqtlcbBRO1LFVUDzn73y3u0UI5NiqQZKYPwVm+GRlhiCkfbKccdW0ZkOa/DijtbRJhcwtAeKS7OdVqvLoIPMpOKK0xNwmkbRL3VUoRL8vOFt5eRwSZhxVz9caHI/oZ02767JJqVuKyCruK1IaC/bmdL7t70/Du3kpACA6LeCZZn6dNVc+S65Ybq5o1AXOmGApvJQUxiSjQkp1KZFnfnU/1fN1vnCGd2mTKpVxvcQDma+B/SFCWtZM1eWT0mEZCvrYTzpEqKxXlBkeU0qQjRxKQjuiQoOHGWI0EKvreHG+OLA8ITUVDr8ylwJ8hSqvjMyzgewDsC7k5jjsnUf4cd0fhOsfy/j2QI2lmetCamm8hbrPSs2GpyB4rMsXgU8SUiKSGgPQzrjMr68vNUtSqs8YWKHQ9svnwBg3UVfkdIqtChwV5oJgtt/yXmWV6TR+eRc5TjL1rLDj0vRXBBJIqqPvhQzpW1KsxqLj47m2ofozu4BfP+L7SQ2Haq8Cff+lqWRSbd1npRYxOkGFV5VjjH8516RZw5frMS9MWJEIuxybXM6O8h0i9ivOUeuavMz7csGMzogb9jjf3rCLKGOEQU8q4T1jKY29Ty0rczGhOXIkpOjAQDJr/B87lampCkvM/GArRvF1/1jFf6WpFGNB2gzLGRUU3IYHvCk9H1eF8hoRZ4w/kXE5QRAEQRAEQRDnGmfGh+P3//IFMxmYyLStZqRVLtS+BkL6iLjqwJ7NfLmu6fMhpDKNrxJXkDNc0zY1MBt5IAFSf6UlMM4RQfIYJnxN2BfLgDpC8iSlaUKqg7iqgAj9Kx0vrdqtdObOrB1828dLYQkHOnUOm/1/QJlNvwnTr0PWQ4Vq1YneVCbvSOw535vGz3fJ6FKl96G0TIwxbZ/721a+MqW5PeQkYGRE9uissxWEt7u05wyvHbUPh8fmzMbPZ0qpA/qO3wf20Wy+fEhopnpP03bCMrllRRyEwyESHw6ZNM5ISKUk5sJ3innyVGhRHUbUoyWwZojjwISO8t64YnU2cSlhL87VUmIztKg8n9wWwucgKLt1Kfh38QzxVs1U7Uh8UoTuPb+Dqi87LkKtCu0rO7ZLSb2lTTnz5AIyQZ7UHMjQmX4fl+wDxrPJgu2Qmd8IeyrvjVlPFlH9AD3WwJOv77uU2Mvr+oq0P5HUKjFfkP01HC5dnsDkfZ48wCGWZV+W/nWAESq4RNtmixDcaj/LqbQEkfip+JcIX4Qr7tD3+cc3+HnShZY0tooe72TZna7gMd1ygC3jodStm5+zbyvO1s+PzV8uhBY5sG1NDbvUmJ3XMqz6ycSL8OYHB6iQTvnOKCjtvby/hceMwASizom1gZ3CP6nDP+zngmXUxXxPCj/FMoKrnIo/HAD4fxAO0VUbwJIaYllXeQ+dhtZTapOMbNKyjodub4Haj77JyyUd6c3QwvJY+Rw4o7TmRL5zAe3AbQuDD+Ten4Gkh7gGxarVIqz6Sb/NcDT0QccGfpf4ijDncv583LVNhEL+E/j8FU/j313R97yjVwb6lQJam2z2O9mPZT+NiuffRwCs9rfq/cRxZfkYlRUMyLZOapsjSGh8Nnk87ez6cNz/w7nowyEdw2Or6k4oO5p8yTtcuiMLVZ3ligWrw02KVNz47zKBmg3t5whB+fMo2dHtA2+F51/yhXBwB9BAODUeFpl+G3fl/ztjgnXcThfyHuAmZInTP+ZlaH6FfphlfHHprOrzGA6phsN2oCrRsqBeVvJDWg40zK8f1kjqmHo1P6aMAbZUta6YXFhuHmXs55vro/ESMcEKyCtyKlFfNBXIpyIzxyaeF8I0T0w8oDOiSuc+5nDpKB3iHDtHXIxmK4WZgxUw4QsR3CBSKprnwyor6pMYpC0Yg7WcGJiRU4wILzKyiDYhMgbQwGtFxQdHZbIc+iNEOZTrbdPa8Of+oZ3hDYzf3smdgzMWb1eZxtUTJ51Ho5O0A7O4tlW7tRZ0yCN2vAFcIMw75UtTPodF2cbHrjDZiooDCxR+WA79YRT4jPoqEIUL0BM5VgIwYSIls2rLF63DZZRBRmiK1R938l54C8D+EJm7qzYQ5xXmOIUn9MRTEpNsnFdmkHdo8zI5JskgAt4CbUYTwYSj8LvtAID47g+qCTGrGyDIgQUUiomTbBd/id1sEOD36frp4hi//RzMrzOTy3wf1ZtAPaNmngA57vkD+iJjevIVJuzQV/xaKc2NIAXCOV5c03y61XMb10R9SOtnzwKa9ZB72sttjtXyYzw6Ue1W1uuuNOfycJn+95sBAFM2fQBWJMZMFWBBvJuOfAurzsX2Az15Rq4aflyded/psUX2Mb8RwShwwl1w3JhUindzdLwRBCDedq6kR78wJpDhofpLjabhTdbkcWakJfEss7xDGDvrn3x5Zybf5hZt6oxGqAnimeCXj78HAKQ6XLrfq8hRxsRbmGraBL1m8AmA/3+hCPSjxk5xTocTzB9wD51RYZn82fb5M5kvE0GQ0zhBEARBEARBnEH+avOjM2NStVuowuu1R2CccLvquvxZrCm5Ztm/81V7tvB93VdpNafIgcGy9+kZuTQxKClWs29LmPMwkdkb2fuABK4JcKT2DruO8rqwHNrsQoY7NLU60rxEhDGE0wVLOlrKcppmO1JiYEgJ2e/b+LIID8kOfAb8JjLzNhLxyy0n8C3PDK6yoEtNBwAmHBMd9dqGXUf/z9zR31GndanZPksjSLrv92L/rTxnSP3F4t4bpjpKeyDzj5SjXg6tXhXHShV8eWWUjm8VGAVU/xRSILYzE19O4hl026wT/VRJ90/dQdMmfYwkfLMMAxsVr8uhwtAasfLNMMeAPa+FmTk8UOqo+oNlaOCkmY9HP/9me0rJWWB4XkCb44jcH+XWT5pRRSdp7aW4lv97buJhNbkKNud2WY/AUMh+n1EWI3wjAETFYs5l3Azqrq28/7LjP+vs1jI3R9EJLelT2iJDQymk22GH4oShsSzOhZLYy5wQpmN/YHjNwpNKs6HGzuhEbc4l62rLOO9Q+/F9CnV2c9mWUXFBjtcqF5EzRmUdj8TMYWd3ft3U1b8Hi+Flv4o2tAqmVkP2J1MTd1Q4yUuNjemYLdvADNsrx3ApHY9JCg4Bbmg12G5u7uNocyvCIdRYE+pVbM8xINfJoAryeTUynUszIplzad929T6TAUhY3hFYsuzy/Vd0Uo+xSWK8lJoXb5HqVyqISRj4F/NQ6NbfnzZML4XZ9OGv+bbEOjrXgxw78/+AlWwPg80OfKY0ltq0U2qfosGkNYF0Rvd7wA7xa8jn2apzkf4WkLlPZF6Qr19Woa9lvqnykKaNpyPPEvMV6e8E0TZH7+b3OuXpb3R/lZrECLXkzF+i39WRmOBKk1JXjJEVXJrDyvxqXh262gxRK8cEU6sdEEqcbXmS73/5XUHBE9iJX1S4XTWemt+LgeUB09rmMN8XZ5sn0s+uSdWE78+sSRU5jRMEQRAEQRAEUWmcGZMqU3ITUrMRdIA6jgXOiRjTk3sprWh0Od+Ud8QIe1ig9pGSNRkSD8W5wRJOKS0rytFJaSKhSMzcY6sFh/I0M4hK+2tVf5eWbCkppQOqknLGniRCAcIC8oTTqKxPUl2gvkiedIzblSIxBdZlI23XUpmUY5LKNt4tDSkNYkw7okeo6VCSGocL9Z/jEqiiR3n7xU7moT3hyQdT9qHhOuSJOpbIjOxRERwrLnUK+k0liZQ+AU2uwcXPcVvcFVc2AAAM/uDgabummeQowiP5T4lXS8MtQ6IvCTx3dKJhiyv6d1G2Xg7Mpu2MMhw0pcF4iX7ezOAGytZXhmcVWoI8IzlkuChJfLERjldIOBv8zfY/AD1OOKsYmh4jxK+on5SWWtW5Vg5RcRh7W2d7XRwuLbmWUrodrwNNruDrpPOqCisLhNJ2lYuS5luANyBErfz1FgAsVpULgLg3QgIptX4lnhC+JdLJvEQ7javzFurxTQV58Ab5qUjNMXzFwecPA/czPOEiCo5rKXyOcP6XGj0WF+zAbRuHDCmw0GywI0ISXpX7k7AtTwPtB4qdZDLEQvuzIFCJ5+S9MyXbdS6KqH6m5l6PAVI6HEoOaIQ7D9Vl1PMk2klqMI7+DFRvJPYRz0FcdcMB3qnKw7K+48syjKmElQCuyLJfm+cGY/i6L7/frdcJh2h5v7yFQb4BNo2t7E81mmrrBenP+aNIypn2d+3orRLSFqsEiapveAu0b1GJXSNopfXR/StMDUdZmo1ws7SrMdwZA8TI/sfH0pR5PMEne+8RsJ+4BYNjjExcGeAvUc61zHCykYw4Mtw2zr802CfDCMEfaMHA/MZYbyQcVgFypEbqb//Ht3nydH8RY64VX9MemIJXRNegggkS/0z8xSyqSMNBEARBEARBEETlcWY0HCLSUmipbHBIW20vbUiCzFCZMmJDDJdWMKld8BZoO1w5M/Z7lW+EJaQbzJuvIxJJW7/ti/jv+a0qpuGQNoZGKE+FirjhV1JfJS2r1kDXUUnpXHYJMKClCn6nlupJ6XLhCR4aEVBhGXOnXo6kf74rChAQkSsmuUKRoEpWTeKX/b+3gs5ZLmakCgAAU+ti7+Uh99j+T/kpTWmhlES5YmzJGQNR65xa6lNRv4jToukAUz4Pg16YDwDIf4BroRKmbVf+PRW9ZkWjVNmk+yUBvhNm6F+XDLksw+KWICjClPnMSulUiRHiUkn+xTZfSfD+Ph+U3MNlPz9jJRFLgNiJXwGAh7sVNuhMPs/xwm7bDNkoJfGwtM261NYk1ALL4RJZZXv+5Uq+d5vBsIaIMUNE4bOqXmCE9BVho1sPUM86KxC+F7kikV29dlriXjsj/Eoqqb5DaEOhIhZZUlMblaDvp9ScuqINDbA47thulTwvKLS2twCIMsOrgveVwASQ3kLAISPMCf8U6UNw4DNYjbuFXzfJgW/4b52LdD+VEYbEuG8faw0NQGCUKssBJkKzWxdybaqKnBgTD6taY/u1nS5dR+m35PdrzYaUyJuJLd95jC+PWBNW9UI992X7AgREHIPWKjNm2f2qjP2siwcrfz317sk7rH2HUnhyPlacY9Q1YFyI8mtbfukjEQ5ROmxt65d5Ykbv0zzyUtS4t/m26AQ9xkvtJAv+FmB5R4ztQqvehPcr9vliWG2G8W1SOxdXXT/jUoMbW8WI3uWy7c8Of6X71Wkg3LHZ5qMqvxOiZcJCEcGz2wN4/Unu09L7H1KTHKODHVvhvfMq9L6Q99wVE9wvJFFxUN9k6v1nJiM2nkMRll1pqXZwjY3V4gYj+pgYQ2OrBGsanQ4gMEqesoiBSsD8v8JfzWn8zEw4RAdlzK87k8RQu9qyZMtfmbnati1gv5O/6vPJTihexIipopzhVIZaVywQEAISbYRa/ecPKjbhEIO5FRVvhMYURBnO4+KFbCWKl3XBce2kp+pgqA3lxMD4eJOZdNUD+vNHsNrfwquznJtRJQ6bDraLf8hb7YQjo+mUVQEzB+eQOWq5tI//0gY9lWOFabMBJh335MeZCJuZO6k1kp74wX7eEo92SA1HhWxZEWdTLysOeMREJ6rABYjnJgJrt3JTuKFF2UBCQLtGSIXD6UqzhIRa+toqTnpAuGV+hF4MdJp16DwB6tdhmNkocyiZfyLOcPSUk3AdZliNDVKl7oyOeES2ql6gry8nt/JFLl+Y3iJdXpld2BGtg0pImF+b4uwUH0iJ4uM8Jll/0AgTELZnkw7Fe+hLUZdYHaBCfNypTMl+L6y64QdtUJhhWWXdREhfliBCG8cY961EfKT4zVwqQghTs5neT4XPNUwVAp41Zhyr24YZfSZAuNGwS0AOoTCRfdJXqM6pHL6Ls/lvdJLuw+rj0qknWvIcDiesCzrxksp8GXJi1HagElhZMtdIiQ8sWwSyiDJCtJ8QgTtEnhbTtMS67tHI6xgRwc+BHqcs+/ME6DZyRumcRnLMzzmg+roKduItAETfUaazcqyITY4oT4zkj03rAQAp1z2uJvZRIp8DWzKUn/vmZ3UfVua2UfbvAgBWtUbakV8JSMS2dreC7X2fL0uzSU+ecvhn+z/h21TeLgQJQ6w6FweHOy4HlbX8NJj2WJalTb4Dn5cSD3q/w81x2eeL+bov34D1j9V8nUP3DRVS+bSElofKQ4R6bQ2Tb2E2LCe83sIgARrAjOAG4lvB4QKqNRB1kubgIm+Pae4m+98vH8FqKFIK+ALeGyZm3p2KfLsRZwwyqSIIgiAIgiAIotI4I2FxZQZtK7FWyORnqjAhErkEF89I6Caze4sMtKjlBqQkRqqM42torYeccednccc5AEyq1qW0bt8XwHlc6udIvTb8OorkWYhO0DNuMwOxqqSU3BimF/KeWIZZitTKHPxSrwO4ZkapDeV1ongGWkBLqQBYcUKKVSyymotsr+zwt4DQJjjqtgm7jv4F1/PzDHrRqFPFEhIxU4oeeKwnH2wfT4KozKsSUpREUWlLzFB/oa4RYVhciFCeFTZXgiHh8Xt1ewb0MVaUg923cYfG1FVCkmqE8Iz42hXNNB4o+VQ7+bUE0GdIswOT+xmZsnWWavOcMhiCDAEdY/R1h94WqKKXYXQNbagtZGkZ+OdyKabVb5YOt6kcSWVdnIbGNNhURav2vWAyY/Y+Ye7XrLfapqTqUpNjWTrcNYwxQJkgiXqqxG15gEge5sgYFFb9AOiwuEUntHS4SEj9ZUZnyxkcvpIxI3ytTBiYq52ElQO9OKcnX/dL+esr1u1jhiqX7a+k1GKsLTyhQlRKM4pweO0yfs+uf2e/XilN3qQU03IiyOTUcuj6Ghm8VZK2hIAQrPs+htVAOP/L9oyvoYN7OA3TG2VCJdpR9nWHC+zXD/hiev/wKijGmtOFeq/KsUZqGj25WnMpAxlk7wMOCZM18Y7g2jfxDMj7q0IH+5VZoEOEYg+HL7rxPnDxKz9oUzsZFl48YyzviO4Xsg6+Qv1MyTC/UQlKw8FEX1chqM1xRdbBCEetQjQn1QGKpVaSj5ksh2sOrPgaKvS8o8nV4VXwNLehJFCDzxgL0iruHXgBGs7iQUmskO/wMt7LEbwv8sfzZzfh4S+DAsTod51Ph7EvJyM4M8dgwLC6iNFjmDDtY0d3aA2sNPdjfnvodb5SnN+hNTwRjDVnk6danl3H9/HfUVhcgiAIgiAIgiDOEc6MD4eQDDHGDKmYXaofSpth+9+UwkrhpJTmyG3fvwWr8918WTpCshItsZT7xVYFkz4XIsEQ28OT2iG5VsR24wC0VMj0RZFIZ0fL0j4pEk+BDgP6/+x9d7glRZn+W93nnJsnJ2aGmSEMYUDSECUpgoKosIjKAiqCAWTBRdRVV4L+TLu6urorumbFNa1izmIWRIKAZCTMwORh8o3nnK7fH/V99X0d7r19DjOXQep9nvucvt3V1VXdFb/wfiyZi2tinzqT6P54V580YNdTEKuFzl7VrrwF+OU17vrxr3G/M/aGvZ6C6pzybneOHdVnLmnZXhUAzIuvlmPvqDa6xmrMvOKq10DknL+iinfuXHWRq/8un3nYp7NKSu4lJmxXmnKqbK156/bWrjN3ih6X68e+KiQxNN0V7HXq81y6LzvfG/Oaa0UDZyWv7eHInoJ2ata2r4D0k0oH/PeMOLjTiJLkcxtuihSZJVWJsr22rPXokvt8OrGvF4rqDBVyUqD9GAfmvC+5g+EtPl/vw6Wl4HxuHWlfZy5J9z/A2b/3kf8Z2xpznToni6SVJciNYelzp74XALDm4n0x6yoKOMjSOvbpiGspzVbLiKoiAeYxxL/fpoxJ2qGf68aSwFq30sBUU9dQ7Zb8WMI4XEAbbiIZ8zxNMuXfNbUtH46XvPsKKvOIBN/j9zxA4373tLzGzERKGsw+SnVPU8z+LPYJRzmKaXvCrr3LJWd/mqQuGpEZRJEaVSHaPHY8ZuKDwfb8VFrAeH5lfjz2js88lsZpDSPg/DFqfXIdgP3T57Dt544Qoe/fHD2u93fpmZWnyi2Bgy8lytNaj7R5Lie1jxvOPRxH/4Dowv08EMlYxKQOynfT9MxKlR1RRXwNqJ0mX3kjotd93aVnR/fmiGg7OBgla06TuuT7FCP7fY0xYiNB72X3r6/Eg2c6zf3iz5MVRN+cdN/dDuh+p1sXWZuIht6m+4GpdKYCU7o0yGnOLYzMOTazJoOBffDn7mjvF7rf2Qd4zaHZnYgnkoZo8djnjteXbfj8BUwsJmbD4dlntGlRlDrnzFiyi1g1QEGnzziNsyPn7H1kkGQWhG2r5ZjUpxjcADxGnfRwcqief7j73fKYd/JtCUWmGX7RRq9ZOzyx+tBEiolLLd692pDPMWtKFzBtMT/UpZ5zEOxCx06Fx0lV/sSjGL79twCAzlPp+Z7xqbMt5ypLE3+U2hyO3sH1ZqRwwZwzL1PsN/Td53zgJy7Jd94Mc8ZH3XU2KemaqhbLNNA+mQWcVs2Osvlta+GvF9eAK/NBJ7tjYnKxP3gHzIs/QJelW2YjDT/pqLZMpuAo4KQ8gCwWRwZkcafjZnD52dykOSRscN5sUbd53oyQCWB9SMycGEldNuQxL2rJ2XN4q1uwtgJvWlQQa4Y3LyP9QEQOnxxXo6mioPMGyTZh1zvyAjNrfzonhBaGyR64vOvvB/ZzJl32QedkPusDv4BhB12/iN3b3dcYRv0z5wEAalfeUb6O3iG45gUrdvkNLm92TO+epvoXlTmqiOOtNn1jMxP+5jUyj1h7t5gmcBvpni6MRfwto6oazwkdinFIR44viz05xokBOmnRyd+lQf2/MaxMvdQzsvEC4prf9HuTQjIrMVMWwv7OOTeDNhx203KAIpODSAjs1tU+e8NxLXwU8kl5pqvtjNLjjh9z2ZxRRYb38S6GJb6MYvXpfcPH0nnQO7VbV/q6MrFHGQzf6Ex+Oo95o7RZv0h0fe3oH6xUQhCqY7XL90UvLBjaIuMEj49N1a49yYr79tGF1+F7L3Dm1af96CFKFwkjG5Mm8HsY3AC7jdjjcFDpOk4UfDmtmEnu9W1nUn71IW4zeNWtK/1GY7sJqLzAx+S/CZs2N+vyTYbFLMrkWA0h8bUYkQjAzD5kwk5Me/be78Hse7o7x+bzf/oczLGX0L0ZQQlMW4LUpxLRM2x/FEyqAgICAgICAgICAgJ2GCZEw2E3O8e/qHuGi0+Qvko/idDJeifPukjdCiKTm6kkaWInKGuFN59VsY/d6KWq7GRmH/ot8ARJuniXTNINVDpFItsC3n2ok6ZcdevKvMTeS2SUJKBTOL+z6kiYCGBnVZYIselEXIPheAIswR3aDBz8Mncrp48q6NzvdHkuIOYvcQfsLc70xDzvitJ1jDiaK8pJUMaTxvs8TKYZxlWvUfDPfOnHvCTn6ttW+aRe81CppP9vR8KjNUCZ/PX/5epuYL0pCzt0UryYnpkw+zppjiXJDQa3wP72P9zx8W/hTODNCDLav3Y1HfYvX3X3H3qeqN+9VoIlRUoq7c2Laio6MVNKdktf5f7G79Am0tZZvW8qyDn0wijnZo666/730sgW8PjFhwAA5n/mASDOOjdyP4hVHBqq+/BWgGlRveNzxZuRefMppt1NGtJHOf/OyTBMt83jVsckidNB45XQjU5G5fBTW66j3bRMykLmRWbWfpn6bBPNgHKCR5w2LULcIWVljQDz/6uYC17T1hj08YzysUygxjqlhSmQdI4L3Sa8qRfT+7LGsCLtj8uiz2lqUBo/fZviPl4fBPYi6k1qH6ZvF9iZzvyNCTcQVYDbnWkcjnqd++V2HcWwRK9cdtSxyrSxzHgyrjMut0F+R97Zv0vM4oqIWqiN2N7pQD85QfM4tcJZAZj9Xwq7dVXu3vHQ8YbPShno/XiNipeEd6qI9TQeKbICjiFiB9Z782cPXy/A/tZRtpuTr6K8RvCSax0F79/OddqnPb/6iDJfJssK/r6Ap7QuC2+e1YITdrvwJCteoz+ChLSoV93qNI72f18L8zKnpbKVWjq9EdO6VmZGu9VZhUTTF8MOclyjzPojioXIhsaCovnJWuvbAa9TEo5DNHepMj0nM/vFJwP0fCYDMsdeKuO0Xy+Kmb594m/uSl+mreykeIYpOIKGIyAgICAgICAgICBgx2FifDjIDjstLc74AOgdsbflL/CH0JoOkmp5u9z+tQDbWjOF4tTd/M6ZfRCw+HnAHLJ15nOMNfcBcw8sVS2Nq25xmhXn4JWlamUnq7qKuE3St/qQ0NX6axWRGvJ70VLIJO3catfdA9z5Y3fupH9xv1GM37zM+XU858cUFIwiHZupu7t30CKSPzuJlTns/MKAf758HExJpZGgRllKu7w2wg5t9vVn50I0R3DVb25xj+BATvMOl+i5Gem4Hd7qHSNLSxFUYMFcmZTDsXbq9tfH0qx4u1UlkaFzTBlqdz8WWE3Omje7CNbm4HP8d7fNRiovmzR922hFSmIOOdcdNIYz/hYKlY58cCcd8VU76nNbZIklay4ipc3gb6ihJbGcHz+DnQKr3flAoeNg/iedxAxx1Uu+DFO0ekl/Q+quA0pl+6a14kPiI3STVLVziqfM9T4ZD/8adq3z08AR57nfe38IsycRBDQovZcQNtNS+JJIRQbnfrKV/CpY2hrFvn14SWzvbKWxona0+g6YOQem68jS8cGNXhPr+5epAJVqJt0GIRTg/LndrL1HaV9aqCP71thEBZxkm3Gyvx/aLBJrfqfNER9t3ZfZNoVExEd9p2/ROQX42+/ctV2cdsz+/mPAIM0LR7gI1qZ3DuwBpDHmNsFtc2QAoBN9awAAIABJREFUpq9FGk7uuy1GpB7vnGiG2L8hVhpbHqObLrI4IFTjzzoD9o5vUjJKN4kcxYe3Ahuc5BjzjyhVXgDA7d92vye9S0Xxprx5futfBx7B7B8/6c4tPUusF1jaPWmuRKlmR+9YaInNif/i6wYAdmij7wt7ft75R9lffQjm+Mtcumy/iys5/89x0TlxQeaywQWt6YTZK60dNWd/Gp85zpX9db8hunX2u6hUYNvxb4zUczsVMU4WHKxVz/l6TQC4Nk9zsp8vmUijWReNrA/m2Q+QI799wIU+MItPUT5kdI01xh2TWvIxCph4TMiGw/iIw0COVWTMG1UaraLPghvg3EMKonVKxGl/zhiAFnqsFt/yUcfuNOkd3xGzpFagzTWSTBm441U65BqXaWiTOFgybCIc32QOxtz7pmu6V0vzxGrmHAS77NZ0FkNbcPynviX5AbJYagwB33mXO37Tr0pX0Rz0j+634Nv5yU8NjDoSq/B0J+n0Rc9R5mbeuTGueWeyiCaE7zxvNk7/Jan6M4Ob6ZySXiQ/SZjUQj/9SGPEKT698eDNNGXSO0tu5MV4VbjkOUquXUMT5M/+nzCMMbS5XjsOctwGan3Sp7wqO5LykTOf1WpzzwqjmKn8t6JK8jtXG8+UEy/HqdAMP/RdfZ/wrF6DPk5AaXCfawznmVr0JsOnI5ONLY/DTCdGInYQ7p0lH5lNL7zTu/FMNz6vmXsBUygaNi/Qqh2yuGTBy/I/uv8XHgMsaCPSOI9P9QFVPlog99IiMYplw8XsO4kSeDBqffLeM4xzNqnn+2mkYphwXrUeMUP1TvX07O7p+WeWgY5lRLFg/KKLI7uvu0diEPAGdqRf2hMvqmu90rZ5oUsO/Jg0F9jXETj4zfXDtwN7uYW1jzS/4SGYLjbxo/dD0eTN7APKzWfbGdq8Ukze6D3wO2jWFVkFR4YegGGyB2b4GdwAs9RF/+YxvP/TjmWq94pfSXyrVrAbEbE0xEk9Za4IuG/Di9UT3ubO1QdkHmHzvo4+2BU0x3HEeU1ywWsM3ox2TvaEFvaRX7tzx1+G1Zc4s6k5n3pAysbl+DU5zZ/z5XL1Y5PK3nLsVuMxjbVkLmutCFL4G66+E6+73o079gfvcHk9z5nn2p4ZAJlEcf8pA27/7p/iTbK1VuZ3Nq2qdMicqQSQ3nyc5zGaP4yJ8qyVVZtj1Ru4+gh0X31Tuow8zndMkrnsaYJnGqlWMKkKCAgICAgICAgICNhhmJhI46uJ8lE58mQlwm3lS/EoQA5tZsbeKhom7a4bQ2JexdLYwY0i/cw62W15zEtwo/1fXr4s7PxbJM3TEXlZOqx2+swR7zUxzWFPAenryDCxOEh6k4YnJD/vwGbEWZUpf9mc4NHfO4csAGbqotJ1TG5wEiBz4Nm6cu7caE5iGfjvXh/wdI0mq+GBSHu8M1qt1z/Lv8+RfqEyZHUvm1iMbBPTLpZcjwMfwVlFTi1MVxQJNlu/pKE0BlmtngFIDewlkXFNOVyTduH+n3gHObPwaEpP9WuOiHN1C1FVLUveK53KCZyd21k7ocgaPGWtcv7letQH8uZYmoI12xf0O6koiloV9yKV/8g2Me0qG2n8xv9y6fd9ifQPraIHFE03xFzHGDHR0NSiHBWcnca9yVJFSbjJJCkZgb3Xqf6x0EWSx5YVXiJrWErM1LRJE/bu77nsnvP2UvUDIMQY6v3kNKfVHolrwt8VSrPG0u7BjUJvy1oKfg/NEfmGbJKkJYi63TCRALdh1gIMbfIalpS0dBwkP7/S3XP4BfIcH39junp+xtTWJjmTUxgD+zhFimdTLZYK3/8zmH1PTde7UvNaFW8q2DVN3jG3D0+GEMH+1Gkio5f9T6n6eY31dhJxitM4U8cqAgjfJzkq/UZlqkjfafAJiZ1CJnD2vu+7JFMWAH/8gsvivG+WLtPdL3BtYsk3H5Xn8XhH/cluXSkmidy2GkN5s7XGkGq71K55HuyZ6cc1b6K6bTWytNCm1ivtlLWErOGoD8DSOiHa9ahyFdxBkcbLIqsRsfXBnDbrppc4jecRX7tDtAls3VECCZsyzTtMqfQz85pNcpHGkQpfoMYcPs6aW8VVGZ957mGNDCDzhZr3eL7wpnaTF8iaZ07rZvFPBT5+4FMbafzSO0Kk8YCAgICAgICAgICAvxNMjNM4S2IVfV9xhOoxpOXaUZV39iyZY+rKdfeKgyLv5gc3+qjiXlo6dREs00mS9Hvowy8AAHRc9GXgzutc+hY0HF46UxGbbSm82vWzDb2iZbTe0U9RhnIUT7ZnLwjK5iXyvXMkMBNJ+u2Wx4C1ZKc6e386R9FY5x4C+/if/Lsoi/WfchS6sz51jjrbnoTOVLvH9LHwbYA1FvpZLDipdMBMd0EQvZ3uoue4i1oiWRakpTLZ4HS5suUjwebSRBXRhBRFIaagcV6rBQvEvXIMwOxxAuxKZ7dcv+YVAIDqxf/nktT62rMb9w64yp/IS4KpLB2ThKLSOwFXxIaXbd1tosgPqP3rKMzeCbtT8iKJopdKdU8vkEizf8cw0J3Xfo2F4R84DUfn3qemNRVAmk6xyCeMy6tss+0GFzTMayc8ZfeI+A+wZK5rGsw+L6R8lb+BugcA7DLy4djtOcCy21uqn8ub3nlHJO/zJiIaOOpCd62qNFgsMWwOAwlJtNc6ggIza38ZdzrSNLp2/X1eUuilqJXOfL+qdSvne/K52viIu2/KwrYMlc0hpEXVzvwcaNFLwgdFy8Djan1Q6sGUmj2zROLJJBOk7TUHnSVtsZf65Lr7RGvIAQPX3ZsOdAjA3uH6onnWGTDHX9paBQcz2prtBQ7yZ3m+jKXPe01rDIC0qKwRsQnA5CuscSBKctM1DXaOUCSXxb4nH0tlaqYJBgD/Xk1HnwQSZe1HpVM0cF90c4254Guwj93ojpkYhvy77Ia/CYUztQMzaZ5YDrAmonNqXgP3pdf4/EEUrSip4WBK3SIN/URAqNJpvqh2iQ8rfesjvuu+6cC7j0LXaa6NmuPeUv4hKtJ7dh7zCg9FY5+66GnvFYkOW5sQiYDhtUzSlHGK0/fMlLFVlyejJfFjc30A9qv/5M69+ffl6xgwYZiYDUdAQEBAQEBAQEBAAAAgMjvco2GnwgRpOIroRoukXnmmH3+sJJGWLMHsepLgs29I3y5CAcnPbNYBZhdhu90nHhQ6ycectLHj/P92aUb6ge426O60pK2SsQXXLDCejk+CheXeRb1fyse2qSR1ss0RCWLFbC2rbhemDJYSdE4FFpGkhvxBvATppi/APLcFKQdhxtn/lDs3Fh1sEW1sysaTpVg5ZrHi/LLPMpUOOTf3MJfmhk+4xPuf7qUnpcEUfCWD+40FVy6qM0vwqI3ovFPaAmbY4rbbNQ1mgfPdqJxC1M7fJVv/F7xd6jeORiYFZnOJYsBmfCb42/Sv9e9CpEkSnNLqAIlZ2lrOK6oASVo6ZTcv91Jkb0ec1JXtLklBva9Fl7BalUTHme+gvLoUHW+aMjmlffFBx5S/Qg/5aYwMCD0nS+6JOhPDWyS4lA+E15/PN2mKJpJpqXd1/h127d3A4pL24griBzXkx1az9Fy+SuXbKtpBHTCP2oyvV31A+T2x9pU00pMXCs0of//hzdIWeFxJBoUli4KSes3s8JZ0YMCSsH/5OtXrVaItYg0tt/dKh9SXfZv0s6rCrGSZlYp9/Kgv2l9+AOZYGtdYgzd1kRq7qU/O3FfmFtaeH/Yayt3A/oECz53ygXIV5LF4e8NLlZW/GEP3VbY6YOsAGKDP2ft7HyFuNzYBHv5L62V56fvlfpZs1wq0Adlx2kRixfCaawEAG/95b0x9/w3uOrOPkQbD9O2ivhfNJc0Ilv1weP63TUXRSnPIeS4A7s2n7YJDr72ltfrtqG/YIlLzSZY+l8b07qtuwHuPWAQAuOLe8nP/0DXnAQC6rvxTzmfERKK5yM2XJs74cwCAydH96wCYeU2oFa10yiImzTAozIs9wNIXl65bQDm8//3vxx133AFjDN75znfigAMkQOYJJ5yAOXPmIKYgux/+8Icxe/bo65GJ2XC0yKWf6kAFFGwenlpTNTxvXuWcrc3U3Z15EQAzxXE0267posrjCJbdtHhf+1dRQbcC76wXI2fqUhndMcgY45y9AFE7V3vgJ4qVd7nfqeTs+MQD3pTDrrvH5bHrkbIQYrOx3tkSs+ELr3TnXvlp9//0hUq13AL2Psndrxbk7fLI68842iA5Xh4pMEXwka8HAKy7dAlmfuC3pcrmkVnwPxmkaHFLTEzGRECcjiaOqOIXg2avU9y1Gc4E8M+vOhJHfLV1c5zUJixLMc2mASkTgWxMmfS5HOW05nrPUOaa3jlqccExQFScAE0fTfjAEc7R+J33lKNW/fxrXRToC36zTN57horTRDFshRdm/KvigfhovsNivpQQ3eZkN17YratFKMAxHSbNVZs3ei+VLnGW5kU4p3ng18CzXlKqXhp+w2cbADKOk2w6ZFScAU9nXBFzJN5kDG2WyOV+40rxLrauhOneP/3wWi9yJAjWipMvL/JUNHLvIF3S8d/dpxz82dk5G5vIRDK3ZDfNgKy1rU3R27p60EJm6jx/j/22i9FgzvgPeMdkHtdhhS7dnxN6YHNQC+a3gLxzNl96kvCb0Gy8mLgmZmQr3ILazF2qCBqob0YVgM1r2aSO24q1wGGnt1ymRy4+HgCw+xfuVmaFmYjZQ5vErIz74kh/Ll7T1A/dnqa8hmrzUxYoWmb6lmvv8vO9UJB3e3IXvyGm8hz29bvTY1cJ2Edpfll0/Ci06C3kpQRUfrxsQ+hls2st6kd2y2N4x6uXtpzflhWO3KTbGN+1ytL3FgmXbTbumh9z1XzJm8ZKp7bbkox1jBlA2kXcAUwr7xC/M2BnZ8X985//jGXLluEb3/gGHnroIbzzne/EN77xjVSaz3zmM+jpKbf5Dk7jAQEBAQEBAQEBAQEeN954I0488UQAwB577IHNmzdj27Ztbec3IbS4AQEBAQEBAQEBAQEOnzjoqXWjvvj2sS0HrrjiChx//PF+03H22Wfjfe97H3bbzWkPTzjhBBxyyCFYsWIFli5dissvv3xMzVxwGg8ICAgICAgICAiYQOzsJlVZZPUTl156KY499lhMnjwZF198MX72s5/h5JNPHvX+YFIVEBAQEBAQEBAQEOAxa9YsrF8vAS7Xrl2LmTNn+v9PP/10TJ8+HZVKBccddxweeOCBMfMLG46AgICAgICAgICACYQxT+3feDj66KPxs5/9DABw9913Y9asWejtdaQyW7duxQUXXICREef4f/PNN2Px4sVj5hdMqgICAgICAgICAgICPA455BDst99+OOuss2CMwVVXXYXrrrsOfX19OOmkk3DcccfhFa94BTo6OrBkyZIxzamA4DQeEBAQEBAQEBAQMKH45MFPrcz/or+Uo5vfXggajoCAgICAgICAgIAJxNPNafzJIvhwBAQEBAQEBAQEBATsMAQNR0BAQEBAQEBAQMAEInqGqTiChiMgICAgICAgICAgYIdhQjQcyc+vBACYw84Hql3pizZxv80RIGlSqWru0oaHgVu+6u495T3uWn0QaA5Tus70r/Z/rw+437gq+fKzKh1Aw1F52ft/6PJ/1iuoHMNAs+7OTd2tfB2/+mp3z4s/kK9bXKNETXUHl9VIuog+h20CI/3uuKNP0gFA0pD8+D1EFcDQ3nFkQOrIvGf8Xnz+CdAYcrlOWVi6jlfvUwUAXHXL4/K8hJyOoliexfXh5ydNeXZSp/TV/L2cZ3MkXVZdf77O91E9UHNUbWhw2+jw95q+uaXq98jp7r0u+t8VPlqmpTIaKk8hx4JNgK0r3fGk+VLPOpWNvqHO0+c3tEWlobz5GSaS+g8QF3bPLMq/4d+lmbygVP0AIPnUi9w953wm3z40T56JuXJyjt973CFlqA+6Y+qz/j6dF7fJWjf8d+RvDSvnuD1z+7aJf2bZOn7rSPf8l16/SqpCz5Jvp+vE7VENhbodjmx1x51T0w8yWlaj20RGZNUcSY87Oo1t+vbaSj/8+IGurJfeuDp3Ld0+3bExY8uVLPeTcdI9aXTPKJ2Ux5qrb1tV3OdyyKfx371Z92ONycw/1tpcZFx9ruh98vfLjhEAYHrnlCirmi9OvlJOFs0Tfm6gfpU0pBw878HAbnjIHXE74jpFFTWGqjy5n9qmz8On4zpzn4aB3er6UzT3kFL1A4Dkjx91d+99iozP/FyeB0wsZeW+aBNVFvqtdksd+3Zx52o9quxUj4ZaG2R5P/V78GsCrmtHy3XccrGLR9D3wTsz45lGgQjbmPQYD7jy83xRpe/qv1FSwGFqVBtV9crOl3oOprHMTN9r3Loxkt/+u7vn4HPy87Rv96ag/kbmZh73bAK78jZ3ld+xni84P79+iDPjLOdPdfL58/qvCbv+fgBAtNtzS9cxYOIQTKoCAgICAgICAgICJhDPMIuqidlw/OH97wUAHPujS0QSMbjR/bKUotKpdswOZvqewPP+xf1DEgG74mZgxZ0AgOiYS1PpbXNYJJVK+mEqWUmAAapuZ232P9P9srSgMQI7tMmda0HDseW3PwAATD7tw7C/+4i7/7jLKG/apcdlJYgxbGclfS+DpQyAknApdPSOn72JYb20tTyu/L9r3UF9QCQP/B17KPpkXENWEojGMOy2R90ZksDZLStgfFkpHUsfB55IS/IBoKIkk14abUXCvm2t++2l+4Y2F7+fMdDRXfX5Wpa28Rl+ThSL9slL4i3QO9sds5TeJkBC0vmstNTEcuw1WIBIpfxTpb/w+8hKPFvEe/7TBfG5+lXVcVJmygTk32dcdX/joXNSe/mbKK15KIGXfv+vki+1HQt6h/wNtTaJy580RNtA3+T2l+6KA792DxUl/775G46pGah05CT0vj30bwC6p5etmseG4dGnqbS0vtx0tsM1G23gyq9/DgBpJ1hbndVmAtJmuE+ObPN9xlKftKvvFM3D5Pmp52S1G9lz+nhURYtpoy+yhL3WI5JtrwWm8vevhaG+Y7e48c1Mmi+aRtbSD23078jQvCdt00j781qDppIss1alDr8cyLYH24Rpo52+54K3AQCuuvVcVW8aV/V4zn3Ra7VtWsvoKiJja+cUXzdXrQYQ0XxGY7IxUUrzlMofAGLWqkj+psX5ojppCuUby9jCZdrqtI9m0lynzQdc2wSAarfUWVsmZDThHuO0r1T/Ha0vxxHQNW3MfArzPvgcKt+2XBuw9CwzyhjNa4zU9XmHAgA+epTrh2++SbS0lscr0qDrtlvUTy19LxmbY2DWfmWqFfAUYUI2HMd+7VY6kgWiXXu3O7X6XgCAOfy10vn5t3OK2jgQ/vJdmJOvcHn4gZQG76HNMiixaYtWwfGgEFd9x8yaythtq9yg3iImf+Q+l58xwHFvBgC8e6kz5bnq1hV0rfzEvqMXAaMNEmNi+mK+WQbWP7mFgTnhre5aSl2vFsc8mPOEUOuWgdgvtN19dsUtMItPTqUfpRay+ODJhZ8dVVterM654ivuoDkCNCm/hmozAGCrSoWbSJo4vViAMbCP/9kdLz6FnqAnfpM+p7+3NxmrqnQFYPOBFnDlJ5wJx+rXL8KcTz/a8v07PXhiGt4qYwcvWKNMu9TpAdjVt7tTC48FABzwoufADm12F8mkiic3m/qG4xWJ+gpvYrkt1XpQZAo0bn4F94w1MT8dYWY/yx0kdT8G8LdILQx57K/ThsMYwKiFK+DMEacuaqscqffKpmdtbvZT4DYwvFUELdzvGfV+GS95TmoMylhBAhuTNGDXP0DlTZvHpTa73hSm7sde3zaLTK/0mGRbH2vO21fGuFx5vNlMpMZMep5RJlLc1uuDyI2Z3jTRAkhS16wSOKTqwead/M45TWM4//7HQcesuXJvtZvKSRs/3uw++DOZyziNLhPXs6aulUCRKeAOAZuXd09HQiZt0bQ93LXxNtqZ69YmPr/L/uRMkG98kROkHfn9VWlBossgb7ZoYn8u2w9t0hShRMBOiWBSFRAQEBAQEBAQEDCB+DuRD5XGhGw47OblAADTNxf2D//tjo/5J3eNd/hRRSTHfvevpOUsfTjiHNgnHnTJ5h/uzrE5Tc8M2C1u5xz5L6lNNth5yYra2mhnPMB0TRPVZyvgnbWSYlx9m3NC+8ChTp3/jlvyTp5PK7B0Kq4B29YAAMxz3yLnAMBESqJLUom4Ctz1PXd8xOvcb1SBJUdrb7rGJi17nCjSjlHMG1z+dd9W7CO/dWmWnO7LgWFyyMa8cvVjswxlDoXOye6XtWNJE3bIOXAbdp5WGjOMkES81ivt0xeYpKVRJW3ex9e8hC8j/dJI1DuttmYCAADY60QAwOyPnYW7/8FJl5ZctyaVJG1G8vSSnNsVNwMAzIKj8yY4rOmwSYG2w8BMWUSn6L2fdjVw30/dMbVbm6hv6CWzrPUQaXKh03H2HcY1aVctoOhLPF2+T1lYGoNN9wyRsrL5YYqYgOrdQWZ71qbNGgHgvt8Ac5e2VY7Ue81I6f012xzdlGU0PIu0niaGXePMAM30vd05JoOYsbdyICfJ/dAwDI9JrJUffAJY6+ZE7M1jL2tJKzJuc3njDp9fqjXlyCOUBUEbc2KlRmN4YwjWzw8Fjtw5E65EytK/zv12TlFaCZq7+TalcU/ZvUViIksJ8xYTZD7tTBtLmCMrbHvIWTX0AULqweMzmS+ZxScXmMHm21TKUb6EiV47/d2321ZuimWJyJoNO7jB5TOOiVaujKnx0dX7yB+4uee/n70LLvEkGAVmjt4SZSzNsn3mreCfZggajoCAgICAgICAgIAJxDNtezQxG451f3O/8w6HOeJ8AID9288BQOwbG0OK+o8p64bhpRMsmfjD54GTyF+AP5d3FN4EM3lXd6uyRzVaigA4CYKXcDbplJOGJPVB5cxcHivfsA8AYO7nlnlpkCWJzNtvdpqOZP39iGbsnbrPaupP0vDY5gjMKL4L28t209O9tnLTPY5CGEtfKZL/bFmSOixLlpQ9vTnw5e7Y27HXRELCmoUsZTIgfgpRxdkZIyOBf/iX7tweJ9DzFbVy1iZ5PDz4B/e76HhF0WjTZasPwhRJzFnboaWwhZR+jKwPx2iUtJl0sfZ7aaMd8LuetieWfOkvAIC1F+4OAJh1zf30DGl7WlL/tJCiD4qGyTuosh+O16CqdzfwBADANoYA1sSSvbyZvAB21p4uXUZaahORamepk1NIUVZmnMfjqmh4dwCebtqpFLQTOGmN7YAjqGD7eDRGAO4q3BebI3kp9oEvblvyWfQOi5x6W33H/Z9+EwCg9/23w7Cjq3ekpnbSGPbltiucH6SZtZ9of7k9N4aB+Qemyma1j4YX/itaaO1DwXkwlOM1QG292pqPAQBEHNUs0hpgpntnIgkrWgF2nq/1ABGlZ0rzka3wFclaLzSHYLiNWOXLwZoubcXAz+LnM5lAY0jG/JLoffPX3JNu/jyw8Aj3KPY94m+kabFprEFcBTomZ3KzpTQbTwZtjQOsvW0Mw7ATOM3blghjTNfU4ntzeUXij8W0/NSGL7lxNd5ziPsWV942uiWI9m3N9k0TVcTxPGCnxM5HTxIQEBAQEBAQEBAQ8HeDCdFwfOutjh7vZb88V2wb9z09nchEIqXyNHkRLPlnsAQDL3gHsPwGd8y2/ywZ6FK0bUWMIp52sAE0SbJIUg3LlHTd0x3bVYuYsutsyTvFngEYlvRM3wuWfB+YRtaYSHwd+GvUB0RqmpEspaT7GSmBe+QOlGoucJR2qHYJ5SjZ2LLU0VQ6hY1HUxt2pKlR7dZVzj6b83OFd7/9a4VmloOXxVUvvbKJkhp71pGe9DOHNqXpM8tgcKvkwZoKloixtLzWm2dTq3SKrSt/ywhAxJqCtL20TRpS1yK7XpZ6Jw2pD0vYbYbRpUVw0CxjDDDJ+bbM+oRjivv2SU47eOav1uTve7pIyStKM5iV3noqUvUNWWvWMQmWpakskUxGRpfsFgbbUqeKgpDmgs4lhfeOh0rBpx+TPlL7qTxdvuOmZe531n7+uxkeQwbJ7r5rqrw/L+0fgpejcd/tm4fRtIHuveX9GUzGj8ERIdFcEWWpoG3L9vE9x7v5z255XAKTJhkJe6UTdgNZB5DWzQ5ugCFaWM+INHU37w9n2Q+OpNGjBYLk+cXqcYh93kiLarW/WIvSfwCIK6r/pZjZoLT6HdI+lRbZHxulJeHyeC25+KQIjbnyseL5nvtfrRdIMgFNff6xpz4v3UOYnvfwC5D8z8vc8UXfcde2MS3uPPH14nKogK7CelfUb8vN5UUUyNutn3vf1ALrA6p/8ruPICJmzqLnp8o34Pw/iujAr7h1FaV372bofceg6103pNKMq2l/mrFURU+T4Xh7YUI2HHO7eBCIpWPpiNCMWHOCAxjpVwtuyqNzEuzD5Bh6wFmp59ikKR24aNjgZ8ZVALzI5+eLQ7TliJyla6jUenEVNmsaptJYVj/yRieuphdJAFDtgX3QOauafV48+kMLTDiKFxwFg1AbCx3/feoD4End6GivgBu0OR13/qQJkKMZLwxM3CELQlal83vomKyuuUHa6oi0avHtTfJ4MPf0u9oxsiSq1BaiGDD0DB9/g/5vDMtmwTuUD8hGwzsI1mC3kVN8LRP9N+VkauS+rOlfpVMWAbUM1WcUtTW4fuR5zvTicsV/zhtW3mjY+oA373jaLFAJN7ztIgDA0T84TcwSM+aJtjki59Svb8s+CnIF4AjnWQGGiqljG5nFm4aJ/KYlT6dqxqF9LkZZp3Exh+OCJrDcb3f276pIKGAyUac13TGP3zxObHwYZs5BlAnVtWtKzpyNYbR5XXEKdTjauzPF334MvOdfPwsAuOrWdwPD7JAtUbXdv3WYaWTSx+ZBI/1yXUcTX08+sIOtAAAgAElEQVS07LmNhqakhVzLkKUgqqg4Den6GWNg22gvq1Y786lZ1U4Z2/Tz4BaX2fgX7llUWBb+dUwCNj7sjtksWeeZiXRtrZUKe1PKOL/CYxOvWo83qy6NO7/lfo+6ENGF17ljv8Zg06Fhabd8buWtMHMPpdLyeK42VHGasnh85GlkC1O14TRub/2yu+ew8/NrKnrn0XFvRvK/r3XH53x21PJZa2Wz6Nc/BeMflVNvNnT/yhKu+M1mFCuhdcDOiOA0HhAQEBAQEBAQEDCB2MnFPtsdE7LhePYXfg2AdqKslqZARezkjY4+ZXpAEhy7TaTfBLvxEWDOXu44K6XvXwtbI1MYogy1mlqXg0NVpuUkEV7lWx9oS/q/7iEX3G8hxLHJmnw+LHX1gQaX/wFml0PSZYkqMHudkrs3l1fJwHaF0s82JKtY7jRL2OUgMal64jGX30xSkSoneJY+2uHNMOwkx1I0LZ2vdcm9AOy2ZTDT90o/WwemUs7/dj1Fgp66Oz2TpFntBOeidgVrAZuWxEk5YpG6VV2d7Lr7HIWlfn59yKvVQdHVPUwE5JzGtZ0MactMBMvO6D4KK0fGbrYV+O+yDznzRq2azkln6wNIPns2ACB63dfd0yudOU3ZzuhI/uyLX+8OhjbDsgmODngGuH7mpYlC12kpsJXhwFbNYXH0JOdSm/pO6W+XMiPQgSi5XdeVCR7gNF1tfMMysvQiil4X4T5Dx7yzgh2pi0x5+Ds2G2kpPwBsXQPMYw0oB40rcCQfA6O261G0GO30g8ufT+akgxvF9JO18+uciSM6JnmtG5uf2saQkCCwGWl90AemzAb+A0zexDWlYVVmVvVMkFN2GE7aoP0FsE0F+hYtC5srsoVBflxxtNVsgtUndSQzZMmTx6Ek359tAumfsU9vs26rKVO81mCOutA9attaF1Fc58MS/A0P+XnQr3Wm7SHz0zCZ8Va7pSytl6RcqjbGanOAMxVzZcuQXmgihbM/DQBI/vBx9//Rl6jr6rlsmpVbfyhNnDeZljmc82p8/FRULv1R+k5tllwpMP0K2GkQnMYDAgICAgICAgICAnYYJsakaqtzBsK0PcWGnyXSTEGbNEWqQY7VtlkXClIO/jRlIeyd33fnvFM2SWK6pgo9aaKkOlkKwPog7Nq73CmypUzRn7YRjGvG7nNz58poIMyCY2C/c5k7PuM/3W+RJGZnwILD3K9NxNmWHfd9kKgEw//h/Co63uQc6MykeUqSS+k0fWUm4JFhu3lAOX5rKkeSmCRN7wQt4PzrrUuMFh1F+ap7vVO6cswnR04z+wD3O20PsUn+6/+5c/ufKRSJSooIkATHmPQ5a5FyFgdgk4qidKV2zYGqOqfmHPHLwBz6anquUYHr0hJ62zUd0etdPf74EudYfvQP1yjnyx7JIydRfWrxH1c7G+K3/vm9/pyXqsbKX8sTSLD9b1XaEo9DXdNguH7a1wws1WZNZl7yLYGqrEgz/TdnjUMFto0RuIyjYfr7am3fTq7ZyMGmxm0AihpageeHxS/IX0saonVFWtNRqAlSvi7edyIei2bbSsy8MWqi0b2A5r+oKkHjmFCFyByQNL2EnwlC0DEZIAdxdtpF0gBYK0KaVpsou3auV1Moxn1JtebBU7nSu6xSG4ZRQVTL48gXHObrYT1xg9IyuoJ6Kwbvz1DrRaHml/svjZNWE7/k2oSVOYfGKzvSr5yf0/XncrYEHje19oznDSY6mDQX9ndO6o9jLnbnJi8AtjiLCF+elAZh5+mj9qHrAQBm35fkyF0sZOznsdAccykAILnrW8CSf+Bc3LWogmTTo+54zoH+XpelEC9IG5FvytfiS34o4zn3aU+sUpG5eeqiJ1HricNOZiCwwzEhG461/+YcimZ/8gE5yYsrdpirdDj1MuDVw6beLwstXgwNbwX2eV7qnIeJ0hsNdyDpKqLOM3MOdsdZ05tmvWAROz663/rDltKnzFJe/H4AwF2nu3gi+/3v3TKZxDtmIddW1NGVd7rfaXukTdBchu43rqLj8p+4Y73Q8YMEm9TdLwtyVod7Z8mG4lBX9c9GH4/Udb8JocX6wPqWv+Njlz8fALDgi49IeZkP3soEaGbtn35mXIUfVPc/UzLk6LyeIz4fc0Mc3iryvnS8iCxfO7cLqA1KC7AbH3FZd09XDCkFqnL6vkf/0JmF3XPGbCz59qp0XmNGfX1qcPknXF+yzRF5nxlmr1SUcJ48198H00PxfNjsc+tq4H6KNH7Y+ZSH5GmRyR+jmPHZgsUT6P21wf5z+dkHlkvIY4x+Bk/w7HD50C8AirCejRH0lEKNy54Jj/u/jkORMf9xfY7Gedo42o0Pw8zeP5V94UaD/9dzgmYAzJjDygal9QjHvv1VOvLEJVrI5k3zKP+B9TIG0ILLrr0L2LKWC5+qg7VWNr+6/RW1yaxzszdJGxT2qxbqGHVzPUZyC3H//KSpzESZhWpYysXnrIrXwXl4E+QY/t15p/RYmS5mGQGRE+4ApngjOxYatOCNq2Jmy3kwy+HwFuCwV7lHkoO6SRrC0sQmVSP90l+ZZKJkm9qhZq1zD1L/mNSPb39xWhgDANH+ZyJZe7c7JvNIa63M8Uwiw0yVMDLnQ9ZGOWGdEvQkD//K5b/XCyV9O6biAROG4DQeEBAQEBAQEBAQMIHYucR1Ox4TsuGYtXSp/OMlxxlplYmEZrSqnCpZ2rLWOQejo080IeyMWUBdmAZLosVBFE2mYqXdOUnRHB/3gL6rFH58hjOvOfXn+RgGY8EY43f9+3/XSanstrXAMElMvARge6NFylgAZglR9FY6lNlZxpk5qgjVIH/PoU2wmx93x6yW7V+XcpIHoMweINcUbaFXU/tI5hFs1unPmwfFaDVy64JP3T76RdZW1Hphb/yUe8azHQWr3fw4zDQykeD3oqKqMlJO2hxnhTnvrRXzKX4Ptd58ZF4vTR+PzrMYPz7vVADAi34h7XQsCRlLk5ZctwaWInGDnR93wuHyl+96BwDgpB+eBcRkopclatAOpQQzY28Zm1iC3dEHuzeZ6HipKuWpzCWtMp8qfJej0qmidakqgI9//Q4AwOWXjZ3Om7mNpSXd4/lKsrgTgdtazyxFV0zfQMcEyNJLV7skdtNkF7sC3dM9/z/HsHhyUuGM6YuJ0Op4+o1v3QIAOPtC5EyqfF6VTjGhZEl8p6IM51JM29NrqXwJleY0r8GMxOTK6LG3WBOHSidMxmG7FPYmE9Xhrcp0KCMNj6vKVE6ZWfnYFewg3y3jo9cO6blHOZxzHYY3qutwTvY8T/gYHZRnrVc0ISUx9CE3NnS+64/y3Kzzejwdhk27NGEDa0S6FdkKa5F2JjubDFWvA5mGVpiYR2mfeCxMGl6zkfzpf9ylI98gbZyjlWttIeeXcuCnOVOZ//I9rNlIfvMh9/9z3gr0zXkSlQ3Y0dg5DK8DAgICAgICAgICAv4uMSEajvomRy1ZUxS19pYvAQDMEa9ziZKGSAC0lJi1DUyTOLwFdtaSwufYjY9IoCQ5K4faAdBLjOmZvWS/3RxpS257yiWvaeOuUdA1Bf1XHu6K9b4xpO5PCm3U0ju/xmILzhFVvd9BBNPFFI0cIG8KsMJJ9MyuTuplTSyUoxwMMVakARz8iGxhbdIoDszEUpHHb3LpFxxNZa20THX4izOcnfdJP1mpztJ7YptciGOcty+dultOwgMTeYlcocRKBb7i53CQS6+1gUk7zevytCFVBYDDdh8lcvY40JHJ7XIXkMnMfpZILNumdNy+OPFaZ9fr2xYAy748rPHatlb6u1G23xzBmjVzsKLZ4l8OYtU5NS8JHt4KdLpxRQde8+VoFlAtt0HBPfNJvOpCClK2I99JviEAT0OMxpDSTGcjTavxgMemkUEJFsvfe8VtMHucMO4jx4vSnAuk6G8UEoiy+Me3Ob/GFC2ybwv0gEbD93+7/n4qQyR15XpGFdjbib76qIuosKPLEp2DuFDFuieqYIC2gGygDafxTd/6BABg6uGvzfuMsBY3iqUeTaUJyAbtq3QCXaQNiDNBUGHymgsrATe9diXl15WJUB9VJN+S6HijC4qH4S35fsx9Pa4ozZz4sdiVNB/ucZK6x9U/R2Nc8C0nTAvCPoRGBdVjP0lNWuLHNHEQ57ktOvIN7rYHfgyz4BhKx+MPEQAM98PwuBtJv5b2GeXP0XuNnvNWl//33gpzytVtVfOpws6kzJoITIzT+D2OOWB+YwSo0UKSOKxTcTascqQDgP71sFsdm4OPVWGMX9D6iadCg+f0xcVOmDZ7zkg6nmx5AZI0ZaBqAZ/+4BcBABe++IMt35uFiWvoeZeKXQK0HgV1B2DrFW6z0PdvdwnDVMHkb3NsHxZmt+emzpi+OXJvxnHfrrvXq/B1Xn5xnjF9AQCz6Hj/LACpDUJZbOFmkmJGyTilYhQH0W1ugWtIpWsbQzln8RRvOQ+aPsp6p1qkCpuRNxdkh0O1AWvVBAAArv2L2wS9peU7xYzILHSThm3Wd7oR86bzXTs78nsrfDsx2c1dz0z17tRCqIuccf3k2Q1DBBYpZ30AJlIxLTivWk9+o5E0kN4kZkzr2viGa1on0RsdxsiGnzDewnsicOf5TnBw4HdWpk1RslBkFQDcd2qOpM6ZvU4uFe+kbSfddjb/+5GpXr1fOVTzhorKb61fBPs4D9bCkklxqhT7nkr3ZklTVBvTi2LvhKvZHJn1ic0H1ZzTxpw49b2/p3rVgYSFVVQuNhUFZPxnJ+xkBMiaQTcGZUGaaLNSytOTjDAjV1MIZBqRui/zLrQ5b6vR4k86DgBw9W2rxk6Y3cjHAMgcKBWfhFnK2okhtaOwntjPembId+LfZsFmmQXKw1uEHYzHkz1OUm07s0GrdAqTGbfDAnNAJA1YHYsKgK2TqfWL3hcije/kCE7jAQEBAQEBAQEBAROIZ5pPw4RsODq6nJTCdIoKznrHWKaAE2cgv/udNM9LjL2UMu4Qc6xnX5xOP7hBqZl5N14VqQpLPGwi57z5BElGhrfAXv9hd+bsL5Wu4+svGF9l3xJYQk8Sg8bHnQQrvuSH20Xy2E4evf98rTtoDIm0ls1VNPVghoYUjWGRaHk6wC0iXWISAKbX65oqKmmWenG8Fp0vrHe4ZEczM4kcRYefkEi8JXHqc53jt4svkY5Sqx6OQmlm72xXDk0tyaaBTG7AdbKJcM6zBK85oqRzda6dSGxYs8E00lEk77QF+t8Lnt265mdURBXYG69xx0e4CN9FmriJlJgf+fW/jloORlqCqKmB07FRktV3+FgrqSjNIO0Om3GwyYaJ07ScfB+f4/ZA4+C2tx+A3v93U4s1BBqtW2F55ChgTTSq+Y1NGqViCe0IHPAZp+FNOeeXiPnizIXStKK2a6qYgXB/4j625XGJJdRQUnKOcO3NfKx8b+7jrPXatlrMQJhaeTz84r/do177jVy9UvEIsmZOURUm61xuE9F28PtqZEhRAG/iaQeegOFxmE1St6yCYQ0fL4P8uxqBJRpTM22PcvUD8LlTHKXqBb9fJWVlDUxPARmK0izmaHEBWHKgN32ZmFfNYU+ln4rV5E1zeG6qKDMlHmtVLIcSWjCNK79yjZQt+w29pskoMzJlDUBjt6XvY1ffATOdYqjw3D9E82KtF3aLM/P1mi4dS0VrU6nO2X6uo8W3MgqbeS6WyorX74P5X1iezrMuMaKyfdLGM8QcXsUp4XnPrvqLq8a8Q319rKdAlrgavhy+PtKeLdy3M4pEwrZIFBMwsXimbbACAgICAgICAgICAiYQEyK+mvHBX/tjv/Pn3TFJFWzPTJFOkCbCVDrhafsY9UGYQ86hzCg9ObTZR34LrHnQ3fvct/IDvTTDblrmrnVNlSCA3jaVdt5r7waOu7jlOprzrm35nrEzTDv1xZe4wILvO3QX/OvN5Ncy0baeWlrGkhyKIu8dNeNq2t8BcNLBAvpKrxXxNtf0rVf+VWgeWdpTaF9rvVOl8ZIqlX+rToD/+G56lAQcKrQf1xoWTlOkCWH/oGxgq0T5FOl68fdkyWJcSzuhA1I/E8u7n7lvmeoBAKZ88NbSaceDMcZrNv7nOEeVe+EfVxenm2DYpFkQ2HCccvhIzCQhn7JIJMUZTROGNistBmnfUt9V+XdwX1nnqL3NrkcCAHrOvTJNBV0SrXt9tAZx1IyLg0JOAAxpDMeCpiEe8xtb7RBN34ilotUe6YMpvwfSKHi67QrsakdHbOaSVFYFVjSt+oz1TKaiSfDMvL16UyiNvTO09sPgMcfAEuGBYS1ximqcA6XSOLR1pVBbs9S7e7rU2dI4xU7BzRFg2R/dccYXbywcMFXK6iX/uTnLKK2EniMy3yQVLZ7zpPdmVOA/n1cl7ydgjGiFfdRsXnM0nd8BWtAATBJNS0475Wl3B+RZXutqcvTAZuYSCXCZUD1Zk1UfFL9V1tStvUcC52onf9bAZoKQmigu8K0sAbISmP+F5bj6EKdJv+pWipJOdSjSOBpjxM+TYNVYyJoNr7XqnKJ87cppxE1hkL8dPTpuX+xkLpA7HEHDERAQEBAQEBAQEBCwwzBBBrpqG7eNAo5523+2fTReYuPt/IokWB19sJSHYcrAX/6b+//Ef4F9iGyiWfqjpB/e9jNpKqYLzSYDmBn7wG55vPUq+mB3k0rZGo+H3O6e8vzXW1bhE0e7d3fxH1cVp91RYLaNuOZpQs3Mfdw5r5GROnufhbgK+wRpnrxvSgQMEnUp20IzE9OzzlB0hcwSZfJSpPqAlxrZm77gkrFmC0Z8QyYvKFe/IaZ+VMwuWRpFVzH65TTq/WtpndfOZIJCVXuUFoPq1xjO266maH1JWqQCIRqiqW0Jtg0p11igb/KGXzkmuu+e6CTTp/1i9VOj2eCxoWdmTlKWkyArGGNg40xQsu7pQq/L34npQbumpah3ATh/I9Zy8LdP6tJOiOqVbZXNgqNgNzzkjqcvLl3H+a25Jo0LCaqVCZ4GZX9O7dVWu59C9qqs/0me9a0QigbWg4OsVTpVfXkuaMqzeByyFmbu0lQWfg557CZg9/KSfwDAMedzyXOsWrlnpyujxpv03OVupba+9m7379RF8g3Zvn/e4YrhUT0jwzTE5bKDG4BlzuYeR5SqHQBg8R7MKlVXcmeT+imkhU4aUkcK2Ihaj/jksZTfZ6m/vdJ0RAVjN/mMcp+0W4nWvXd268EN66PTxfn2WCvurOxr4LWwJgbu/r67ePgF7lePL6xt4nY7c1+kfFN8xqOUA2hz7JcMr7rV+ZF8ktYfb7xBtNml1jzNYfkWzHhIc78dWA9DQY7bHV/aZW4MmDhMzIaDqG3t5F1lcUnmIN7ZNmnIQMcDqPGutd4JNNm8XPGsk5PnCZe7PO75PswLrqA8lIM4LyQ4UmmlS1G78WDPEc3vytFEloG9njY9p75Pzj0JZ1mvXvWDoAykvNGwX3qVO/fqLxc+wzthZcyzUmVrpVAcHbUxrKL9Zp3TlKOpokD0CypvGhTLYMz58qAaqUHLmzSoBaHiJ7dfcXFczJkfpns5gn2cnojLYNV97nc/KDW1Sf8q5zuw6hsGosE2mV91zG2t2iUqdb244zbpqSCHhbudnef5ncW1tgZXNitE55Tts3DMTKynfd8tdFacvxDzPv9oJukEKFQ5LsycA1Rk8dH7odWChyKHb25P3omYvsPIViGcYCFKzwwA6UWi7V8vpo/seDt1kbvWM6stp+xl28ZP0wqypplMxZx6X6MsnHYYaDGXit6epd5MCaTyJmxG913uKxxPhb6drQ+Is3SinHs5P97gVzphiaDC9KajGZvdjm+dhvvnH3P3XvA1WOrvOUdf3b91DKnM97GwIsihDYQ3s6x0+PR+/msMKuEHUXevv8+Pv554gzcj6+8XGt8WEMVqrPexRWjBHBNlKhIpywgJYWp9Msayc3l9UJmXZsbYxpB3TE6NiD7WB903sk3u5bpOWeDzZyfosqMixwbT36noG0rcCJ6PlBmZnzgszNJz3dG9tPHgWCn7niZ16KBN6Ug/YFhAwnNTPb35yJRDx7coDSY8qQ/69sAbjWuPd/3g3N+sLBzbeWzl8cVUOj25iyd54fbaPaNgzZMpP7JrmPQmx2jH86cJnmEWVcGkKiAgICAgICAgICBgx2FCNBzvOe0MAM7ZyBC9qa1lAgklTcBmghZpJ0wO4jR1N3GWZSkBq+IOPEvSa4pdzqci9Glepcw7YpKem0XH+aiurcAc+0Y53g6SXB/4jnf2yqHK7/LP/SwA4PpT5uB5P3rMpVOOVP6eRDmV+TK2vrd+5M0vAgDs/tXl8HvzbDA+HayHn1HpSJsaAfT+09Imr9loDKt7VdRthqeercKc82m6RzQh7v+RUaU9o+GL73FUleef9C7RMmQRq28bqcB+Ni3NSQV1Y1MJaqewTSkbm+h0TJI68jXtWM8mWGxWYJvF5gjjwTv6bS/ZSkYaTnWcd83tWPZK55i68EuPuGsF79TaZPtqPjYsp3xHlzpqpL5XNNY7yXzLSmfetMVa1Q9Iqtc9XcgDWArL/XFoszff9A6gJXDZyXPGT/RkwFJImyhNJiFpwm52Yw3XMZqycNSsUuNVC2D62ihFUZ03jcs5ixdocVMgM1zWoJhKp8wFWiKeNYlJmt7kIxtkzg5vhsnOZ+PVb4Q1OMpp3I/Tav5jMyimr1VtzEdfNpHMiVxeP75sFWIUP5YMAd3kkMzj1pyD8oEHmbxl8gLYG2mc3e/M0nXsO+r57qAxlDMP80HeotjX32sRtdO+HuNYO+U1Id2+nDYb8C0yki9/X62l43ftLSEsQA7MZWGX/cE9ask/jJpGU6ynKZ7zZAd246PuaBFF42bt07bVQj+fWHWNtUaKAnlMk6Y2CCq4XXVO9s+z1CbP/TU5jzfrsJ42XK0HaNzzNL9RDLvOBRI0RArje2hU8XOHWGboughBg9dAUztgamEYA7vhYXfYu4PHyO2EMaecv0MEDUdAQEBAQEBAQEBAwA7DhGg4zt6dpCg6uF9WMhpHsrNliURUzQfjadaB9WRrz06zKakiS95JghF3irMxX1OSDG+3yfbYw1uBTekAN2Vgf/2fLp/TPpQLYDgefW2xfWL6XEKBcsycA+F3+8sdVeGJP10DSw5mlqVZBc6A1ur9Zes+HLt94negCsk7Z8c2srU1USz21IzGYM7hD/V+8a3h90NSJ7vsDzC7PYfu5cB/fcg5a8PCPvhT99w9n5/KA4MbhFawJM57z2WUvZaMKum1r1+BsyZLyrUDrtbYuIvyy8feKVVL8shuvDEEkf71SL34vnacAG/4vPs97UO579SWpmGY/G/Y+Y/fTaUTCz95izt321fcuaWvyvksGBPlbH01WvY16ptJ+eZtfXNjTvZaThJvVBslzQa/85GtojGtKCpSzo+liY1B2Pt+7HJb8mLKlt5zRy/QNb1szTw+c72zob7sfeMkbBMcSEtrn7y9dFyFmeYCZPJ4rSmIc3m1qUljZ1KdR1ZhUeiTU6BRSVGE6sBwAGCbfuz0AfV0MESWnFe6hPhi1hJ3juYp0zmlZft4c9jp7iBpev9E1ljInFgTm3evwVFlUz56hrRMrH0yU3encneoulLde2dJOppD7ZYVfl418w936di3rtIJHHhGS/UDgMZyZylQ1f503Fc8jXkNSDLjY7Mu2gvt18naKU8zTn2x2pP314gq0lfZuTuKlda5JnUDADtY0P/Hhll4jD8eLXijMSZHWmEUAYq/H5H3CfFzyZ3fcOkPeAXsGgpoSr45mubWbnbrFdM9Q+qv+g8/AWMEQx21jrzGijtG7cvfOWku/uH6Nbk65rSjADBtT3d9LJ8nCoWQrLwN0dxD0td0HXQwYMYY2taApx4T4zTu12zaMbOIczmjFlcRrVOLo02r0vknaoPiB1dxLMtFMK90+UKZbIwErWJvAeaF75FjFSuh1L0FHTl7LtrlYAC0oOXsFxwt6Sly7OeOdarEC36vGCTYvEPxxrdqbgQA+N5V7vdVn1dmQjRJ8+BnmkCWKqPaI4tuXjhXlQmCjsAKwCx4doGJ1JA8kzeMlS4Z9FndzhF0OziCfHn8+9s+CgB4+83/os5mWFU0Ct6hidSEw4sQvxgrYAxRMUuEq54njcnq3WQmyFGePy4OPl3Kuj1M/zITm2+31W75xoeeBwC44/Q5OOh7a/N5jLEhb3XB+p5zLgQAXH3baSqPseqpNyF5FiRbSZuBGTWh+3HFT4Iq3gwv1jonwyx9Fd+czqva7UkzWsGTiTTeCsZrH2Kyuf0dNe39JEhY+mp1Mm2WmRIMqBhO7IStBRO5hWYkC1lmO0yZ4HhmJ0rfHJZFXDNT30xMhTKwN3/XPXHpq3Mbbi+0aCqzUL3J8IKGjAkmkI+aXu1SYynlsWUFDDP3sUlfz0wxi6lnxt5KJwybvbaAB353OwBgv39W87Ifz5g0o5liInTX1KJSm9cwqyHHEOFv2BjKk3AkDTFD8++nYCzR0eVbJOEY+shLAABdV9yY6ytjjVtF5p7GGDEzo4W6OfBsuqEJM2t/vlvy4Gdym671iLBK6HYofxHstAIf4XzqIkXKmK7b6b+UtcZoQgB/vOJmd0DxXArT80Z+l4Ox6bK9AABTPvpAuQLzuNsOg+NTgGeYRVUwqQoICAgICAgICAgI2HGYEA3Hnp/8NR3ZAjpaUidbK5IY1likNASsubBAf4b/niUkgxuEqlI72Wb5uOv9ImnxDk0qkuufr3PHh19Yuo4b33IAAGDaxx4sfc9YGI3XWkuDTYFk7fzfOYlEcssXlXQwI7mNq7BeEtICDjiJMtCS+iwXuJK8sHNfvV++C2si4hpE9ZUR2UZViRLODrcmylE5YnADLLUj71THsTeqXTmJ8ng4Z0m+OxTHcshIwmzi34PV74E1Mexs6tMrnn9FxeolxV5Cq/bl3GQAACAASURBVAgPmhlpFqw4nKO8NMfselTptO2gqN3y8YHfXYN/O8xp4N725x0TQ2ZW5/hSyrKO6tZaZd7J0vxsVGRIH6gP+POmj2JujMHVb1W7aQXD2zmUShloU5HcNzNxrp88aczY3R/mSCjkCmRs0++RpLqsbVRaYTbX8FqCqKpMc8gEpN4PjPAzWeo/AmygsX3y/NRzUG+dwOHxG5ykd4GaYsS5mONRVaV98HzZHJYxwMeNUWMuz5l+zmtKvTiPjsky13Jsqq0rhdCFxlBxvLWjk2i0Cj0+MryGTJmvZk2vKp3KbCqjMU7FzzE+jY93w323WUAkor99iyZHHfP3KJUuZ26VNNT4oeaVZsYKIBu/CRDzsGq3mPSxVqtZl7mRtCVe22dMW07j+K0jUsFpHwKaZN6XIQDQpvJ6ntTaG3cuEXp8T3ygNDAZc0djjNdsJN91oQ+i0/9j7PKOY76+syE4jQcEBAQEBAQEBAQEBGwnTIiGwzvjwYhkqZGR/MVVZQvLjlwFojxjgEWHp8+RZMA+9idgzoEuGQcMSsQe1y77vbu26Hiho4woyJ+29z34lBZq5zB5r31avmdMZGlki5IUakFIOrD01bDfcOIzc+bH3SWtESly6BoP3aSliGtSPrKZ9BJdRRHrHf+aQ2Iry7bAzRF552xDTdJ8u/ovMAuPdcccCbZrWi5II3rnAPd+zx3vTmXjtpYlGyiBef/6aX+cpdTUEpyclLs+mA+MZi0saThMV6aNJQ2lUWNHeGVDXOdo4r0qInBXOn1jBPZhpzn0Qb5KQPeBHQHtZJyLDG8Tr9l4/Dxnhz3/8w+nqJxTZW2DUvXCi04Z9V75hlr7MnogKfdPxkFXS0j5HPcF1Qas9zPqBLY5vxVL2j128MXgBmkjFAywDOKnUCo2nr/ZWAQALT2HnEvHig4PrW0s+KZpzQiVkbWurEFtjsi91Nfs4EaYHvKtsezbV1OU1NyelS9Fi/5UU2ax34hoJ3IB4nQQQ3bkr3TI8/W38JpQpptl349YNK3sj9IYkvpzAN5J88WRPEu20WzIfMkO8yXwt22ufPuZSH0TVTcuZ8z1p/GiOSLHPGcl9bTfH9T7ajbg24EndxB/Ue/NENdyBBLSDoZb1jaasz9Fj7L5NpcqZ2YOV5YbqfQ8N1IAPO8f2RiWtsbanMGNMt6QnwV6ZkrwPPYNY0uPqNKe0/gpzm/TRDGs4TVDfhxlPPZqp/3b9UuP+3N63LXsn6LpyMHvsMBfl8CajeTh6xHt/rzRC1wrcCQP2GkwMU7jejD2znjU+LX5iO98akGXdfi2FlhGjkfzj0jnNecA6WhaDcmD8XTngIShTTC8gGOVKquYRwaA5Xe6Y/HJHhemozUe9hI5jp+ioMOnFgQvdcxZ9suvcdfO/+qTK9Hik/Nlo0HRetaRWBb7zKjUPUPer444y5ueTORYM+9w4cnXfPGshiW2GNz/c5iDzpL8NOr9BUwd49RvwbNd/oobP8t+YhEpcz0uv/oObALW0ScRU7mN6dgbbMqnF9v0LDvkTKVMs65ix9DAvOYud23OgSmWlPJ1bP2edlC4qFcT7fwvLAMAfPKYebjoj7To8bFMKv4+vwko+dzGOscNX4Xe6KTjG6TXBMq50m/uxnAC9hGoB9wiFFAsOCpCsh9fGkDvLHe8hhzmuR10TRt1szUWtrMVWtsoNKPKOXTbwnFqXNQUi02WWUouQJMu0IOlKCl2q8w8wos6E8tCjDdJm5YBbKLZYCFBE2b2Aeln+XatWPtKou8kFTOK+z2XIxWHIxNLCVCxo4StyZtBcbm5fl3TZJHKY7A26eRF7kg/DDtj8yaEx7eRbcDAupbqBwAveesb/fPsMJnSZmNi2CYkcjybLyomQO5b1so4yotUNs9tDKpI7/K9cuaQzRHf33z+niDGjM2cVISCcUKbD7mDxLOkpRzEs+saE0nZuRzeTC6Sb0ebRjv4hMyNPL4Mb5XI6cxutuFvLou+uS2zNgKA7acxq2MSUu8WgDWqH1CbnP859zyrBYreRDCGXXu3O0cCRY+k4eN7+P5g4tz7NIueg+SGT7jjI8keUW3Y7YaH3PHTxGn8mWZi9Eyrb0BAQEBAQEBAQEDABGJCNByb/tVJjqd89AFRm3npjHIQz3JpJyqqpI7qu/i5dCwOYgBcLA021fImKInPz0uco1jiR3D+ni6xAuzTurmJOedTLd8zZn7bwfmJ1fB45ecAAN890Wl/TvvJo21RqtrH/+TynXeESNf4fbMULY5E8sMOiTZJa7IAJzVjdTY7ejOv9tBmoX/1pl/GnzMzSFM1fTGwjSj5OD2bRWxaDjMzY+Y0DlZfchAAYM6nHxHJildDK+mUN51hClhlnpaStLGEKuPY2jFJ3p92AuT6MbWvjuvAUhwvZUVbKnK73nHjt2IaMWZ+o5AbpNKwc6MqL7fvi37/GB4807XLxd9ckclbOwWXQ2XB3lQeZeaDdF9Km1HRQdIoeJ9WtFlMesD1rHUj6/iZIrmIlOSdzQ49Rzw9dGQb7K3XumsveG+J2jm87ZMfLJ12h4I1jppGlsFS0Bs/CTvf9ato/5eXz3skHZ3dZZbVLMQZLQZc7IqsWUtSl2/D90aZ+QfwWlIz91AZH6usbWgqbXjaedlu+JvEvSgJ+7CjjI1MJI694CLy2KPO+/db4MBtmzDZeDE9JPXWY693JBdaeFTVuDlE43AXa4apHBsfBW4hIpUDzilfyQ3Un5OGmKPFGUph3W48Ba7q896kqinHPrK2coj22g+OyVVR6wimGa4LqUc2mrW1Plp56RGHx4YCymAd+2tMM8SUeZUqOyDrIZuIqRCPt327SEk9Gc4ArDdxdpour7UCxLSOtWBloLTE3vmePZ1TJoXVdL0AJD/6V1edU98n9esk82IirWGtko2qShMtGqHc/GIA8+yLXf7rnUN5ROsBa23rWqqnGDuLtnqiEDQcAQEBAQEBAQEBAQE7DBOi4Zj87JP8cU76ZJQU2NvSKskEX2cbUC111A63LvM0bSjnmw2QYxNxbmRJxBMqsAwHPmoB173IOY2/lCJuPllsT5pJ1nSc/kuKBrp1dc4Br1Q+c5ykEnFF6G1Zs6CdFL0Eiq6N9MMOrHeX2cdG2xFzWch3AXFVJDregbrPS6VYmqcl4HaLk6aZGXRfZ+uB/+Z87NbRL1Jbs4iBflcXH2SwOZx36oZqd8LJST+R9/Vgh3LbrMNufMRdZ0nb1N3EWXMGkRIwnXEb2g0AeOTSEwAAe3x99TgpS6IEuYEZo6wmrmLxt1xZ7PIb3EkV0LLVaOpf+a//AwC88qUfg9ZKZZEdh4rLaETCzlLSmKRvUSUf+M8moonStvejSd0qnTAHvqxErdL42D+9HQBw2Z/Oa/ne7QmvhS3SxrJC4bg3t0XBbdfd4+7vnZMTA47loOsCuGWCappIOVPTOf4mzWGRvns/NOU7qOnYybfQS5FpLDO9uyiK6nK47xc3AgD2eyOKCQmQlvB6ybZK4+sfVQEmpuBr5Idhembm+1DcIdJufg+DG1Qkb5onebydvhj2+Itaqh8AYA8id2kMiU9OVtMdxTKmebpVq6T2SgPMZfZjDn3LSizntJaAx2nW4lS7VRBEutdrMFvz9wOA5HPnuipd/INR07Q2f7Mje0aDFddkfOX30Zn+3u7cFBjOg/0neb5tDEtdWymR0uzm1iSGgzcnou303cbAvNBpbR8jgpBdv/gYMImsHjLO3cYYlV+RI3neN4s1G8mGh93/03aHLXovATsNJsZpPCAgICAgICAgICAAwDPPxGhCNhzNlQ/Tw5RNNlFF+h2vttduCluFl854idQIzJRFmSeQxOv6f4c5+arUuRR7CO/wTSQ7d5YcsQS5PgDLTEMt4B/OGYOqrS1kJGzbE11T0bzmDABA5e03lb/PSzQrIo1S75R/TYZy13b0ia8FS4M7+kR7wdoBtufc+DAMUzh20HdqjhQGOgT75dz8Zfd73GIpY7M1atxbz3L0sod+f23umpdYWyvPZFS6dEI+KDhHsInUtSGSVzONAkklijmFbcM9LaJi1GlDy7HbaS9p+Z4xYVrzNSrS3Hn73HmHujQ/ucKdP+7SljVV537pG/SgRPwISGLoJeONYd+WtOSskNI1y2LHkryuqbm6203LPL2tp062TfC0Yjctd/mzL0djEHajGxsN0XmXwYt2bT1Y4I6GsP5Qe1bMSKw9Fh+W8WGm763/46ekn6VTpLQemYBjDWF7EkYx1shGotnw7HNK++vTGcCSL2C1LvcCsGvuAKiflh2t933DRVTePLtVEcMXswDaUcYVT4dKYxMH8YOeczXzE2vuyPbeDm+BYV8E1tZUla/azcRwuMeJJWsILP+4m4sXfulV+Yu+3rHY8RcFpmMa1bgK20Ua5awVA1RQO++nYsW3guejxlA+8KB/D5X0OF4C0as/R4/KS9/bg/h2AUj7r/C4w+PFtD3lPXhmvA6vOber/uIu7fkCdy2uAspdqXSJ2K8JkLxZi+vbmFFaSPVNqMy7ftHRLSeffQXMOZ9MpXOUxnDfj/udbteFgXf5mss/mub6XrLmry37UgVMLIwdk+h8++CXx7lB4Hk/erwg0qfqBUlmUVUfEjUq05w1hxWVHaGm+NH9hFIQCdibQFQkEier7ueR+nekH9wZ/AKwBD64n6vX22/ePqYq2z1ybzZ/NtuZvKD0Pf1vWQQA6L5aNine5IcdxDXnfVFEcv+/oi3MLJztX/8PZl9aGFcKosgaNYFmKTN1rAt+hyVjHFy9jyvH1betKpU+VWYemP1CVtE8ezpEiXzsech5IVtIjyqDNi8CvAlWY1jMd1pwAvzvg1w7/acbtpNJ1Q6ErQ+IM2LJOn79CPdeX/Erqd9o8TjoP06VS2+tVaQItVQ6o6k7/eLJIre51IscP/5QXs0RgE0NW9hwfHh/l99b/rzzfUNP36qdnLkfsnlHCTxyuntHu3115ejPGmWhl4vsbJM8IYmOx8CLVC3o8hlHmfQQCm5eyA5v84tEM0eROoyB5I6vuPS7n4DC8RJw82GuvJA5kx3EjfHEBNqh3pdfx5gC3LvwtM3smCzmyN4ci01GjYFlB929X1SqfgBg1/zVHdR6ZYzn96kd77Pv2CZSRx1FnPqK30j4PGPpp5xH3KHWFpq8g96PJ5fplHKRGZLZ5eBS9fvIs9y3efNNO8+cX0y3CwBWSBG0I/k4SO5yAhyz4GhpU1nn/qI2NrhJNhBMDhNVZTPFAuTsfYCYtFc7c3Oojq+SpdN3myp3bytxjZ5KfP/op9bI6CV/bIyfaDvimabRCQgICAgICAgICAiYQEyIhiMgICAgICAgICAgwOGZpuEITuMBAQEBAQEBAQEBE4jIPLPk/cGkKiAgICAgICAgICBghyFoOAICAgICAgICAgImEM80if8zrb4BAQEBAQEBAQEBAROIsOEICAgICAgICAgICNhhCCZVAQEBAQEBAQEBAROIHRRmbadF0HAEBAQEBAQEBAQEBOwwhA1HQEBAQEBAQEBAQMAOw4SYVH3sAPeYN/1p9XbJj2MVmiehj8rGO+S8rLWATdy53tml87vr+W7vtt91qwDTyj7OAokLvmLiGpUhgd38uDs3eT6XsODWJl2KXT4AjHq2bdbT6eIO+j+B3eLyj+YuLV3Slf/YAwDY5bOPqPLwe1T/F8aSpHP+mgXqA+6wY1JBeuPL6v4teKc2KX4+ozHsrkxZWJB/Ht8+KgYAnPHzx4GIukZSTz/fxHKO62Kbct2XNwaGt7jjrqlyjvPkY18vVW7OI2n6toEozpTWAPV+dzR9r1L1A4BvHunyedn1ui9m32GmPL5MdJ37XdIAtq1xx5Pm5dPn3kmE3Hcd9flw72nIvUMza8moddLYfNF0V5wP3ztKO6Rn83uNq+63PghUOt1xc0SlpTxq3en/YVX9uB0k0m50e+d2ns3fGJ/e9M0tVT8AsOvudQcdfek6AcXtlMvE44FODwOMbHOHXVPkHJBu13wuUlNG0/UvxDXXVlP5+gfJN5y9f4naOfz6eNdOn/Pj1emxeTzYpvQtqO8ywmNNL10rmjskfx5HLddHzQu+L+rvTt+07Fjzk2NcHif/dCUM5cfPkmfbfN1tE7l+2BhyfwDQMdld4jyTZuqYLha/0+y30/knVLapi0rVDwDufoGrx5Jvr5K8+HlqPM+Xxfr2ZKh/2qTpxxozqXxfaQWW5+HeOaXSrz/fvevpH38gf1GP681MP4TN99dmHbZ/nTvlx1L1bbhN62/E9+o5ir9xUZph18/LjqUAcM1BrsxvvKG1tZtuV6nv69c61XHvHW19Z/XYlc2f10E9s1oq71OFZ5rE/5lW34CAgICAgICAgICACcSEaDi0ZuPqQ3Zxv7etaju/J6PZGC8PY4ySkJVHXx/v2K2SLKZ38cXPNE5CqDG8DXjCSU3MlAWjP9Toz5fPW6QI6XLYpAmM9I+e7yj4wyNOivbylLYh+1yTO6W1RiLpSkSTNJZGaKxvoa6lpGPuxOj3jYLTzzicsq2KFEVLjIG0hLHBkusuScdapPqASOlIssUSNCRNoFJR+cFJSDPtxW4VyaDpmUn5U1sZ3Ag7uNFda0HDcebH/svdY4xIPEnCbTonUzmVVNRLJOOcJPKJN+2L6f95D+WX/oYuD353eYktv8/UuQySh6+H2f2E0nUDgL6TznIHW1YC3dPcMUsYayTdNgam2kXPT+Ta4AZ3udtpSZJNy4EVN7tz+55GVankym2brl8Y1mC4/+SHNBFeglpzmkLbGM5LJFuAve7NiF5+TboeLB2MYtHeMOJaTnpoR/phqd5Y575lVPadR93quHgqsTYB7KZy+Skcf9nF7qAxJO00VmMsADRGgEpH+hwADLt+AfqOGOkXLY7XcBRo1dQYku//RtLmpO+t1Q0Alu7Zk85flcWmypEpZ3PEa8xM9wypCY3npmta6jlGa0YL+5loxnk8YY2ziUjjHncAzW1lq+ax33fW+uMy876MAwaI0+OJiWJgB2k25BmtLYemnXSGO0jqMu6z9lL1des13dOQA3//xhBM55R0Hty2G0NyzlsDGNFqeUuBPqW5pXmC32l9ULX98rjo1w8CAGx9UDSa1P4tjRdm9oFAtVOew3XgMaZGmtihTQDXsQDWz5f5OT81bvk5ia9q64anlxf2M81pfMJZqnjA2R4bj50J6za7Rc2C+qBf2NiNjwAAzNTd3P+NocyiZBTUeoDJY2w0niRMXMubwJTAvZva6x3GGFhSpsmiNZJBejsgNVkBsLD5Bdd4eUyiCbwxpMqWMdVSGyVUadGQNGSg9xPIIOzWle64ZwZdo4G0oiZTXkxFVVnA0ALZ9M0BBmmxll14GAMMrGupfgDwk8svAQC88KcvlTrRr1/cNIf9YlJvMrITwrSrfwI03ARjs5vmrEkWQGZW6clEbzayi2Ez/4jWF+TTdnV5bXoUhjccvKlTm1Gb+V6ujaZhat2wjREUIb1JKtsvMumiGGhjscrvJHr5NUjW/NXlPGt/LhiAYnMcN1krkz/AmXnRu+CNRvKFc12S865tWbhTaA5h29hUzSWzD5vIQofaml+41dT0VafFVxQpMxJ6bmMIdsj1IzFdk7ZvskIFVQ85oUwEt8MiYebH71HlKDINKRbE2EqXE3Ao/Oexu+Ofb3h83GcW5afP8QYm98y1d8PMfta4+Y+Fv8d5v/G3vwAAqnEtb27nzaNtfqORNKQt8X2NYVl9kjBEzGkrYo7pYeVcpMyteNPkzeG47wxlNrclwWWoD8o8tpUEyJOd+aDd+DDM1N2pDKqv8/zI5auKgKJwnMgKrcYQRrl7o9z/pcwuA54yBJOqgICAgICAgICAgIAdhqcsDsffm8Rjn2eR5KzSCfs/TtVqXvdNukq77pISfRPFQAtOpG0hJ5EeH/+4u5PGpB0NM2ZDRY6OKp03A2uOeDMAqyQf7kSSN+lBXgIIIOdU7aUeSUOkO2Vx4pvcb9yBlEYDgCFzJxtVlFaCHb+Vmpe1BcPbRAORMscCkDRhSF1ufR4Fe38Twa6/zx3ueqQ7R6p1u2Ul8MTDrdUPwHGHkDNdfSDv/KpMvgolvHzIGpiemc7sC4CpZZxxTVyQRxNA2oHVRHGBsyxrkLpEw1NWI8dlm7XE18tuWu7OsZTW6Ppwm2p6EwV+vt22Bpi1r69Pqio2UZLxseU2o0rdbNKeTp1NsqxFRHVKbvo0ACA64vW55EVSxFS/yrzb6DVfAQA0rzkN8Ru/N2oxikwg8hJJM6q51ZjoJqKFuKqkwEOp59nGkHzjwSfcta7pMrZxuvqgO58tl6tETmORHsPo2c1hGb+zdWxHgzO0mfKtI/nb9e54AZl0knbbAkpirdoYjWuWzFj++caV/5+97w63oyrXf9fscnp6TyAJoQcpoXeQqoAgUgUE9apXr72DXsCG4lUs6PXafl4LKhZs4BVFwEIRBUVA6YSeRpKT0/feM+v3x/q+tb4pe5+Zk5OTxKz3ec6z50xZZWbVr7yfJajQLPVmM5tKlyMmEM7pKqMdpLWSpp7BnD2N+d844F9p3i+/+EJzIEkoQvplc6ignHbaD8px7QUAVNqhaaxTHfQdpHkSE0TE2gOP22yWW3fkFqn0O1ybKAI2R23rcSZc3WYOUaxV7JrptI+sfeN7AddHf3EJ1Gn/ZZ7NGPeS5+T/rTQhNpuokRqnt3RsaxL/ba2+Hh4eHh4eHh4eHh4TiM0eaVxKPLZmaUfXB4yUSj/3F9zxi9sAAIecT069JBHQtQErnRwV1c7R79kYlIv7T6wmIcbOMWlDugk5Ckd2ZA2tpMU5IStrg5qSdkgpRUKDIu/XLe37dVNJSDPUvmH8G9refyvilMMCUegkTyzV6ZxuiQIUa2u6ZhqpuSko/bKmS/iWWKmX0xzY6wNroObsGbuPHY4xdQfoFg54zfCVW4zW5Z0fa3PlYakovy8pgWMpmio5bQ/7mpTboSZt5643gf1e8hsmqEDNcdx2XasS0J5FmdwCj/3F/B7wWlsmNZnKKL+pdUR2tK9WYs829Z0zHAVtK2R4DWf7MvB9/K6UI5goAtYohM5pmjUb0Q/eZP4nZ/JmsN8EypIVWKd2+k6lN/0M0f9eYNK76NvpNDKcO7PyGUs7BWsk6kPQ/cZmXE3b0ZSTNVDrnrCECUx4IJ27rQaqfbKT1MoxCbBU5DJdNEaguQ9aMohqWktrKxkUHmtGvnAWAKDtvTdDLbuAE4qVDWHdSailgzDnZecSDT1IGh6ypZe+gtwWNaeRpd2WFMgpmnY9Nul4C/xLzPsLDwEA6Js/ieDYS8xxidsXa20VNNFCWy1FWLPaMtsPK11QdE7f8EEAQPCyT5r/Be2ypKl2pBVMitHlxpNk+iqI+VDkBlNl968E2MeH5wZao+g1D7l5gLXET98BtejIWFLqhA80HWuyoCUVe+xCwg/NZlDK1JxsyQi2ruJuNLyGw8PDw8PDw8PDw8Njk2GzazgYl9/z/FZt16k3PAsACLY7GIdcTywOJAm/82Sj4TjwFyudhJ9sbq1kTqalIydFz9CIODYhCjY1Bm2FDYSVU+ECAHevNfvTQ1vcE2Md4mBRpaqw0xfS7lKO5tfSJlOnA60xgrLwq8iH1Q8ZVrEFUcNpGVg6KKT72gZayqK2FEVne1YZIA0wPhIqEQBLlZy0ncsdlJxUKmlDrcPCNI4A0FdnH4sA0Gx3nKBhVIGTorNUtFxOf4tSxdXNFczWIemHk01/rJpKpVRQcmxSORG+YJjBSr1PW3a4lEQ3aohAimkNhG2jHdMctWpKut0iSCKQaX+ss9roWMD9qi0tsWTNRvT1cxG89nsmX6FtyfSxoO+e1Z5Ys2Gl5F85A8EbflysvGOp51++b35f/C6oKYvi1ziA2dQdYN+91NJRm2GNlR7pg2JKUMsm5LSMMiCeSStjPJVBK5MIa82vNcHPf218s858T+T8w6zGljQ0lQ431luNS1d6vNFRig7XXpL+KNQPYsxUtm264LFJem4AGSxJ44Oted4fuNJQZXd//G9Oy5DwudFaiwCd1Ic2POsCRPKY2hhxWreTPmrusxqpNpduLPAoM2LxXBWJIJ7V+O1j9MGxAYinLnJzAY+BtEZRM3Z18wDn3zPP3c8a8dX/gOo4hJJop/KlNcFZ80Us2F+zYLg6cmyYxavqMQHYYjYcwNbtUPZ/rzoOAHDSr1c6kxBaLB50veEjj6kIOQ6BUJEra2IUIOolR9eZu6Xysp1TTIxJx9tREWXTfbbCGbun83NlyqBc5IEmawPVGAaY953ek6QqtellmDHYuoqFjGZnPV60h7W441oOzD/9XJu/HanJnAMUdV5DWbMpu0Doe97yi2taDCEIoGt9JjmOWM/fbXito0qU5hmcJ1F4otoNUBqWapnTL7cDI32F6gcAx811Jmq62WYuKLtNCC+Gep+B6jH9UtepDNVON+m0x6NU6yZ8r2kHap2aYFw7KMWjY+fA6vsM3ejcKYtcepaTnkysStKhXcRtSbbhgVWpWCqZGGVBLWOZxC+MMQZHDrOB4LXfyx37QBcwB1Kv/9GokYAltNa5ypvCwr05AXeOxxAbn2DIRtYGqM2M9MF+UxoTVNcMgEyObCRuSUCRpChtRnnMG3SVMA8tdxQea7bv5MW9aDtJam9Zd0n1W05sCMKai6XTMTV2aVxMTHRUuH5FsLXO+13v/qE5aIy42DqSRhYw8x23PZo3VFuPjZqumbq70m6FlorHCZ4rdejGeibnCMQYxuNbY9g5bXPsD75WaQcG1hSv5O+/aH5f9km36eR8JX0v58Obq/7n3SaH1zXzlqXMT/M4j9NZe003obvPY+K5pWErswDbaHiTKg8PDw8PDw8PDw+PTYYtSsPB2Bodyl5y/kvsMTvw8c7eUp/W+l10aBkoh6S4VmJRGlCSSAAAIABJREFUH0wFd8pC6+iboyAsZqoCAE+uNFKbudoFy7KmFoJaVWVJIJLUgOV2GzXV1kM4ljupcGCfU0nqWwhNS5Iyt1QtLB3HsleY36G1huoPcPTEI0RjWemyJlVW01FuTzv/NrSj0eRIzmye0zHNnePIsVqn1PEoVYVJ1bC7DzDS3YKBDQFgp4WkKWmMpPNjqVx90F0jKZtqnwzUB+JlDmuCDCBOMwuo1LkYlSxEdTh4IEvCZJoFRUA900nCpiMTXNL8Q8XIcOCWwdxqiWjKlQ6nbWqFUUJNp0kOhDmXLt4PrfZJ2ENmaR147Ix+SU6oL/1otnaC02vRnlpJykeuPApt77u16XO6UdwxXi063ByUq05SytGMI0HGwP2ONXHD66G658QTa4wI7RWNV1KLljR1ixrOrEi2E6aytWZ2wkG4YIDKR/pMeQ5EttbPHSYCflY6haaHLpU7mpKRjBY8jaGUEtY65JTP5ilBydHtbkJsdfP+IGkMpu8sTD/pXUuTTWuyJ75hNW7VoOuDjlqcNQdWy1yOm7syuP1x3iMboFfca25bfDSl32bTHwtRDI40RCp6xb2GalzmK6Or17iPkhane44wXxTU7zy2M8W1MK1KtlOttQioK0z/qP1rek/x9QCNb8VrulmwrUn8t7X6enh4eHh4eHh4eHhMILZIDQdjq3Io2884kOmw7uws2a6RKQVL7VDdZAe/gZyxJm8PfdPHAQDBCZeZa+X2fPbdlgKv2lISnOnfMQZpx4FvfzslGEJr9kWJ2x1DOXpRK50oVYEBQ8fKFKU6CqF7nzbnZu5K5cwqfEISDSERbAy5h7iOZEOqtXZ2rHnBEqtZS0UQvLTkylKlTt/J/HZME87Vbe65J/9ojnc8ngvurlmHcvqGlU4bCNHa+tcGnHSINR0seY0a0OuXy5Llwozz3inyjQeOsum1TbI+JjawX+cM6xhotVtBOeVc79qDtsQE7LCrw4b1sYl9wyhBG2ybUuC0gznRdcAxlFYIcBtNNCyjOUn4CAEABY+z9w2tt9+zpd/CKOQEjiY6IxBiQck4gEyfkXRQTKeFVC/5CADgsbPnYIdrnjRXeWxqUYbsAHFp7Wb1vTdDW9pO0kLa4HwjY6NUtWPnsCOXiBL1rnab64Brm1MWCttx+nb3Xgv1ItJeJjRsUrKvAq5XEKcNBkyfTfgp2ftrw4XH0xP36LLlTjrQ2uCV5TZLNBBzqmVNI2s1oobtp9ZHRfQbq2nM0IJnEQrYwJ62XFFa+7cJsdXM+zLIHWslbHsRdN8UAFZ1GVpZvfZx69enJaFFiqhDECEkg7MG5bRfTcc0qO2J0oUDxEpn8zEE++XgtWrm7mkfNF5/aA1N/iE8/6J9SpzKGTAaEe6bGbT/mWOs1UCLecXWQ2U8V4woxmNisUVvOICtx6EsuvnrAIDSrqcAAbOBpOMPcIdTZLKj77sW6rgPmGPJEc8mN1jSNE/JId/KWTzz2hhU5Fe87/MAgA/efUnTezQcF7aWTD280bDmJxpqxi7Ny2cfbLG4EnXIjJxbNFr7aorcvegotwjjiUM4u6tJlK5kaOJvzM8N9wJTF1HhMkx7eAPBaUR1YTLiIn5bZ8GkGUd9KLV4yoUlR7gyWJMqWiSSU7jh/6dzvIF77i9u0ltEaTRGLGMQqszEwlDGHAbymyNGDACANstsIhM3OdJaj7qYT+GAc9Ln7PvkyOqRm/gkv/0GM7awY76qdkEHOSKcj8Yln9iMxZzIS2NQ/vNE3taTYY7DWaWZiHb4/vPQHKfj7C+ZsqDkFpOJmCdKBan0s9IFlF0ERLcbJ1N1yH+Y33KbNSMtBObzH1rnSBfkJgQAdN21I178CQIHe22Hw5HcXKcYbpBl2gTETGSatcVyW+F2Ov2Qo20Zbf9IlM30FWozHF18YI2LxSNNVnicaLGhcuZn6cjwRkiU4cgOageFarfx2Crm/Qd+aX6P2TNt3lRxJtE2DhBv5DqmivlXmA6zqVwHO2fLKPdVdx9gxhI+x99zpB9gso7k/KKCMW0arem3bOOCbdHkVYGaROMkM8StX+7WOOQgj7Bu34VSJIQSm4X0WKNSmxxpZuVIaRJm3VsRvNO4h4eHh4eHh4eHh4fHOGGL13AwtnSHsnqvkeKVs8wdpLo9oXpXe56D6K+G6z7Yx0Sc1UEFIP5rzN9/o8tmIyhLqR47QE7KIcElXPLZ95v0GsPCtIcdQsnMoNyWchZWSkGTSZWV1KsAeu1j5njy9uaXzXNCIblksxBpIsAS0/bJNg/Nzr3s0AzlKG3ZAXw0zCWnuEqHcNhjExH6rkHZRIwHxDtwkdRlFG489WdzfdYeVHBnupI02TCxStj0ht7pSB80x2sh+kTrUB2UADJJw4ID89UPwIoPnGKq+tXHnJSfI6bzu5MxTNiRe+HhTnonJPZs3mGl3VabUHdOhVZKXBamP6KNWKfCpNRZCwfpfFj5YUNtPOfLj7p3xSaOnKNS0KUME5j5+5rrbAI2vN7QOwIASylFGq6YyQji8r7AVl+HiTgfUXGn+HT68ffZWtOpoEizET34C3Nul5PH7CyehYA0GwOXmXGr8/K7YtLe3OB+P3mB1V7oJ24153Y4xt3HfYqlrTGHcSp7WLNSVo5WbseNkb64GSaafdsW0tOwXrid/vpb5v2feN7XnAlqytxEaIhZYt0zJzU2ojEktMWJ/hXWrUQ6K16TnBvSxA98T0PEkphYbNHz/n7nmd+g7DTsiTFMKYXohUfNbfP3M/cAhnIbcO+12mPnMNbo2b5cqoi5wfVzDWo3bM7UPcvNs7XVNl2TT7czjy1QRb3GxItR8w9w43SSjCWDgMLRpDstiR5YbceCFKFIWHca4Ixx1BUotO8gOebpKNzqVAbbmsR/W6uvh4eHh4eHh4eHh8cEYqvRcDC2VIeycreRDkfrlgOP3QoACPa7qOn9MWe9vc6NX2wMA3P2MvfZCKF0/xh8LzID4iQCROXBXVdeAQA48Bdvc1LAHE69MtqqlUY0RpwUpBT3eRnVAdNGZxeOjtVE+lHDOe/nBUspBcWtlRhxoL5S2dnHSod2tjmXzuOLD4vfx1Kq2gC09Xtjabez17Zagmq3oxhmJ0C+Vm5zmqECqNeEDwV/O6G9sfmz9sbSM2sh7aUyr7wPypaBPb0DlxZLqVMOzUJ6l+HAahHWClP/zn7N+8xBrd9JG4WkEEhIwrjOjREbNE2TPbweXO0k5pYqNCMonPAfyqR2TGjLnM1+PTuq82jgskShsNlP59tKOxHsajRd0TX/BnX6f7n0RH02Jmhc14eMdi/69Yehjnh74ef1HV8zZTjybU7LyEFQ2S5dBjqT45DVxJH2sH1Kyo/GtslKZ0p6rKGcw30e0oJyW2HJ6gn/Z4K8bXj3ruj5+F9NXqxZk758DPZlCyqiH8Kdq5rr0U1mjFbHvJ8ea0v1K0kGoGL+HBl+cIDRPAabV3K8Jc774beMNq/81hugQyYv4LGbKNO1hpq3Lx0znWwFYH85SxDSDzVpAaVBNPk899cHY5YBAM0DPHbRfKjrg2nCGsZIXyooZC7849fmd8FBcGNMwpcDcFpMG5RzxGnOWRNR6YBe8XcAQLDdQbFsdFAS7a+FtlaVoSvZYx5UkKan99iioPQEeNpEnzcq8ODfrh2X9Hhi1Mwvf/LHm9/bvwqqe1bz663YZzpn5C7TNQeYzn/erStyP+PyT0RaboxAP30HACBYfFSh9HLnSXElnInB6Ii+ZTZG6uVXuQk+qYaXkyG/U62Fw2LZ3TdEZlA8mMbStHYolFbgBmdOSwXuvqQzMhxLlZqxa676fW5PU7a33v6MyItZQeSCJsHCJZ3bJDe/NVWaHC+bnCwsNFIDelgTjqGV+LUotOmr6Tvnqh8ALD/dTAgLv/WUeFcZ0PHFZ8zMjaEUNJvm7HYqn0ynxQtEacaU9V5t3mJxyNF35+7TvKwC0TWvMve/7EpRh2Rb1c4sgPMf7nWbSmY3q3aLjQmzcYk0kzzzUZjuF1o7Z1CelO3/7bZs1jE6Tx2fus08M3Wxe1cyYjqXKVlvORlz+wvrdlNtHT/le7Lp8YK35Nq/3FynnLFdv9RkPhLQwisPLt/VvO/L7n6u6can2QYvi7EryZg2FjSbK2KOrLyQHAWPnmze05LvPyu+U7y9SkfamHN3st2Fdcucp+bsSTmI8TgRSb3ZJqrpUkBHwDDFj8o5lgLAx3c3bebivxSbE5tBJzZaLclGNgY55/0v72Pq9/o/Pofk2B3fvMU3tPExUoznlrGQxglJupKxGUy188ZwOraS7PP8/QsIqj66m0nvg3cX/4ZZ/SWP6ee4oMDabXPi9hdv3ujoh9w8BpbEjYA3qfLw8PDw8PDw8PDw2GSYEJOq8dJsMGzEadJstFK1ttJuABtnNiBx5OIxcFzb/BNl0KF16k5TJpYTz44BYQ163XKTRgENxy+//AMAwMlnXO1OpiQVTegmreRX0AASbWoyMrmWplI6w7wjg09cJ99FFFp64rx4yyUkHQ/KIip0QjolTZ+YD7/c4bbuLMUf6bOSbMUxLWSEZJZmaqHNsHSDRC3IdIIQ7ZhN6kb6oInmsEgr2P7Sr1IdhRM+S2dFpHeV1MBkmLlFv/s01GFvNc9Kxz3AxDTpMmW2fP5Rw6VrKQ2dNss50pt3MvKJI9H2vpsL1A5AGzvcpvnapUQwZZpY6XQOtEznGNahVz1grtM3VGzOJ6P/MoV1xzRhKibMrIhuVlOf5jR0/ypnblVAw8GmiM+/YSfM+9qTsWvZUnj61hnmaRqAIg2jXkvOrWy61Cy9UtxETWaRmX57cVOOS7/wQZtC0tRLxt7RyXEFSMeRiEJwRHedpEOWdJ/cJ8OGMDcUFWOTK06DTUIbQ4WdxidN66IqiX7WggI8cZJ+OB5OyWi7zNnYczomU2T6UKEpsGahJehn7zKHs19kfsl0VNeHMzSyo2O8NBuMpFR8c5tYve5NJ5uDxogwJSLTWjatausBBim+D5s0yfdvY2/0p7Xq3L/qQ46ygEk8gpIxoQIcdXul3VFGc3lYMzvS60yBC+ADtz2cOmc1KiImSKZW0d7PB8Lhmy+ytrcxDH37l03djn4PpVtKvxMdxrW4lL9Jw32HrcV1fGsp53jBazg8PDw8PDw8PDw8PDYZtjqn8SxsCdR50xbls93NhXI70EvSIZaSU7CnaO3jVpqVZc8p0dQ/pVSxUs0i+MsLZn96csHnYloc69ahhUNy1v3p41Fyid2vA1U8mvpUE9BPN4aFvwG/YxHkjKUtVsI54uxvWaoejtjI3ZaWl6VNMamp01xZSl2mFuyeDTAFcDIadxQ62t8CeOwDrwYA7PiDFfHAbTHojHPp9qQOeZNtn1oG0AOgB19wmh3hLwCrRZH2zCR5jeI25m3/8X1HFZoXHZK+M7vtNG1TyfOBDIoXJziISSmTNJHNkNQGBiXowQ0tSpoNvfI+AMC8rz2JiKhi2ddL0qLmc8IsWerLgCi4I/YHEPb6WZHG7bUodJHFk3mpAHq10RKpaTvkr+TiQzhjd85KVCs2bXGRK2SlojFncNuO2LeEvlmtX9Bms8S0km4LtUGrkXV02FSesG59RPJi5hcfLHQ/I8v/Qv/x81CHs2N+fLyK+YFkaSlkIMcm9Np/OG0xDr9hZea1zYnNPe/XnzOBYqulqrNAYIl8m9Cgsm+YpSoW2mweO8I6gIQPh/Qz4/nIUsKK+VOSC7BWRCXaeWNkbAQVIrheatzMCgFgx58MXyeUYPsfa4WtdvUxYP5SupHe03C/G3flGBswMUnCr7EgwYjHxONfYsPh4eHh4eHh4eHhsbVgM5O/TTj+pTYcm5M6L6iMzYcjC0oF0LucYP6hXXt0wwfMtRMvE3a3cVvedDrNWrMakz3na/ZISA6BFPNJNksMkGTvQNSw0mOd1EREoZP2C0l4jKnFJOYklywxYglTWDN0ggWg9j5bpBVnbLH1U8pJBZWwIU1Q3+qhtU46nmShiUIbhEvoqERmjnLYBl5i6SPT4/Y+CTBlbwEsOZapekMXYCn5vbRuKXK33yQoO/tklnaRdE51zxbvhOpd6xfSO9KIlNoE2xe9Y75W7YSm9NWUhfkqaBleSkCSuSkVWHAU6AhqyqJ4usQMpofWizI5NjaNtFYxk+YUAOpDNuhcEah5y+wxazai/zVBQ4OLvk35C8l1K0YYpaCmLaFymrKzD0d044cQnHBZ8gF75CTnaWrVWPrTdxqlRhlgia4KkOqLseB2VB6pYUzSMQNAxWiIrZai7p6zvkvWTjxy/l/EsKZX3eeY0vjb8m9daD/ygpjQdLnD9ZOEf5PR1mR9u7gWIzjindaeXxOrm378D+aWhQdCTTJBKzVpyRGOZEjHVbp9UvqH37DCMe5tgdhc837l+NeZAx258YzfEwfcK1Xd3BGG9lz6m1dtQFtLF88scLUBoYUTPpBJuvWoEQ+eC7j/y+2F58NYGYTPW3J+z1pnZK49GsOiPNT+uM4zdnHtk/MUwQNjfc5qVhJ0z0/cArX46ByV8thc+JfacABxNav8f1Oj7Z03AKBJy054Y3eRUZPIvOeWTwEA+m7+EQBg0hFvgeqZ4/ICYk7WudJWCtrGssiPp14wg+r2Mq+MyKoWcgKnDUFsE0IDTJLyL+ZEbjc0wlneDuBCzZpV/4IxS/RvzbtWp3wiPSEwwpqI0E0DY8dUYIQcuHli6Jkn3eboJ4O/3MZoqEA3EiYjQ2utAyerxu1kNGM3gHnbC+B3P/0jAOCo17dum2nnP5XeXKLhHCHtJowX3xX3LLcRnlAA59BbGxCTSRS/1hiB6ipIb7jiEVset+AUZghosRG3GxNbSmimj+bvajf5pbgjIz+RFUmdESUWdNUu6A3PJO8aHUHazIE3Gg+ebhz1d71uVeajKbM4Fbio69TGNKUfnHAZInKaD2YtTaWV24yzaDwcwJrjIaqLCNwcp4K+03Cvi52i0s7RolDuHC/M7UZ2kdgQkrCg9ykojixP/V/N3itu3gU3JulKJwp+QejrLzVpnPnFbJpdIG62J6mNeX4RY6olAZlBFNn7LeFSuvZJkabRNQPJTQvCETfWsSNv2yTKs2EIDoBClKoTic0x76tFhwMA9LdfY/ufZsGENS1U0MNEs83ChfqgFfi5OFJdUByF+1eXAwCCkz5m/m/rySaN4XG4QfNguU2QIbTH02+bFG9PecF9V25mOH8hAJQCR3MQNiEeoWcGKTo6j6taOwEO96uH/w9qybHxNOqDbj7h+Z3KpRYcODazMY8Jw7/chsPDw8PDw8PDw8NjS8Y2ZlH1r7vhmGiHsm8dbyRLF/5+nKgAB0gCcMzFAIBJL36fvaRrRMdqTU+6CzlMaa0LOzkCwMMbjGTtsGRayJZ2qAyHs5iGo4mJSzwSbjl1zj1esbSYmVFyi1I5rn7a/IY1ERiNTJggzBxY6sO/Yc1KrLSMVs5qY5bQWgniILSVzrS5NMriGDDOhnzMzoKs1WkMOfrPAlXcc0dnwpM2UXMmVkmHcq112rwjKEOvuNfcNWdvup3uqfU5MxbSEqBUTUmTY1qopMS83A7da75JbsnqwRea36jhqJItnSoFDA1FgD5GfdhFBxYmXpZuNPEdoEruHEuCh9ZCs8ZHOINqcpC0TvQs8Sy3QU0t4EhN0Kv/YdKbvz+SX3+XH5JJTVhPB+EDXBBG1iJpDf2C0QpZOlRJH8zmVb8yplXqhMuBhHlkzKE0SekJBb3GUGuqAho5fctV5pmTPy40YwmpvDC5UMIsMxkgjk7SjTReMGlGrd+Zl9rAaNsJJ102g6m4dsRjAudTH8yk6m5dwQwzGHuN25jU0jEVsTBBdA/AaXASQSalJo7PlarCqZ+LU3b/JAOQ6tBRgG/hmMh5/7ZXGtPGQ69f4cblFC125Nqv1WaXXbtSYp7h8fylHzHPSiKOFFmAdgo3SWKQGOts+qyNLwoK2mmDgiJDSxozqeT2WkprO6PQvgNnjspBC0dcn+B+uORYNyfyeiWoQg+bgMHKtmsat1feDzWbAl9aEzSPLQn/shsODw8PDw8PDw8Pjy0R3mn8XwwT5VD2qpsea3qtJS1lM5AEzkoH2KG2MQz9vTcBAIYeM/bVnR/6czrPKBT25sl8NVAq/unLWayaCal0LKCcpfUrp96BDkpAnYNltSXSSjvcZr07HdZtHZMOrDooFw7GFY0YKUopKAmJUiKwWKnqnO+YcrDSCdRJOkXSLB1UXH0STv4Iyu5Zrle5HSmfh8E1jlKRJIyq1GPrp4pKVQF88Tbja/KfUIg5iQNQgZL/0nFrvwc1Y7d43Sz1Yx1oozZmbfiV+/5MYxo1hCYqkYdS1l8pL565+DQAwHb/uzymHTPpcQC8Uro+bd2urQnNmNVGsM8Ta2TCESfVtoG9prn2z3mX251jOGs6mAqyMQIMry1UPwCWrjarT7BW8eEzZmHnH2X4cST8mpRSAGs2ElJUSWetTvwQACD6588Q7HZqxn3x/GPXZ+3eqjrZ2Pcs8xvV0zS4kRxX6H4r0Q2cL4YM2sl1I2dtq9Uot7tz/I2jhkvP+ulopJyqSTOo60PW/j4v7rnmx6aaZ37BnrP90Y7boo3K+mVQ41pyA75miSoi17+spiijL0ch7PiTJPFQwRbru9EMEzHvH3L6MeYgarh2wv0rpslnGleam57/K9TcZXSb9AtKECCEgiY3w6dOBYlgqypwRCIUdDWe/hjA2ltZyhZO463os6GU07IxpG8Ka9EUvcNyu2v3fC0oAwNE0cxzj6QbrhYnw/GYOPzLbziAiXEo46jQWVzzY3Iet2rz+OCvyu1QF34TANCFNPLUUalARNLOj9MP5UEsIzp0bECka2xS1RhxGwOp/iXVqE4MQkoF6WimYnFtF4alinAcZ8dgoVJmrm92AB0FjQ3G9Keko1jUbQMRJdxOyOxsGzpzHWuOEwj1NkdGrbg05H0AENWFEyjVqX2KY7piggBWjYuorUUwhcl/YpGjs50B5XEmDzuUmwh4oOd7Oqa5Z+vMJFaB3cBJs5CEGYh756ows8r/u998h8ulc2Xe/pdyGodjH0o4YcqFu6YNRWyzncHQliIxCMqOzaYA9Iq/mYPtD0ttEjjfnX64wsVzse1KsDrZByKAxi7dY8aOVs7gwW6nov+Dhq2p+6N/zVHYCBjpM8fd+TePT77nZQCAxd99Lp0k9aPYQgcley7FFFbpgOYyWFMLYW7HG2JhDuOYq3icjNIbY94YTFkIDK/LXTcAuP4ZU979MiKZy5gmycj1FODAXjfFCOwYmoqRUh92QhB7zY3RdrySEdeTQqRS1Y3bWxE29bx/5ZduAQBc/Npq2qRO7hXteyUz4bn7pDaVWmvLcKXazSJfU5pZjHex7yzjUdBGI5W+ebhwHR89f0cAwE4/TMdhacVOlZWVCsrpTYo0u5asVDBt0/ZD7ifVbsuql4yLE+zw4o3fYHlsUmwTGw4PDw8PDw8PDw+PLQWB2rY2SNvUhmNTOpSpMdDMtk5wdKlstP5Jc+vk7fH5g1mKk9NpPUMtPxp+cpsx0bhAlC2tVlVCyk+UkuW2NM1qUBHUgAmpnHQyz5CsxZxVU/E64NJPSExGQ/VsijkgnZtZS9LhTNyYIhKd09zDycixOoIm9bZiqRM7fj/7Z3L4jZfXviNOSzq0srMvazy0hmYzqwJ1fPURM+hx6cg/usmfUuK7stS3VLXR1FUn09dqW17NqnFrZlQx2i7AxSGJQmerxyZwgpbT0X26qNetECQpUYsgRU8aWefhFHWpIEfIEufFpZLJA0JjeExaKvTMo0xEP+Q2SRJ8pYImjswJx2sVQK8z5qCKNBz2zsawpSiWz7FmI7rORLcOTv9sU9M7FZQQMdd+zuoBwML/+pnJNSPmD0T/t+ZVfEtYd1JTQYKQMqUS7dRpHp1Jo07khfoAdC9RGLOjP5ulqgB6cG2hOp62XYaG2baFOBGGPNY6tLnY/hvWTKwGwNJUS3NJpy0OqXpurImZHaYcjknTMtzr3vNW6Iy7qeb99739JADU5pK04MNElNE+JfU+jYY7oRGvdNix1JrREnRQcumRNjU2kvC3r3YDEZNbkFkhj9WVDmf6WQBLLriQkktruLPMp+Q5pzATWhzW1CfNp3Xk3kXgHN1tjK42Wl/VhwzBB2DXFyjTvCrT89gisU1tODw8PDw8PDw8PDw2N7a17dE2ueHYFA5levU/AQBq4WGj3JkjLa2FRI7s2qXPBUtTWCLSNglv+Z9Pm/tY2lGqOGl74DQNAKDrQ07aUcAZcP/p7CeRpqiVPhdJx1EtHRKlk1jMtjwu0UsGv9JapwIJxaTu7kG+WNhmlSlAMbjWSfJY2jRk3pdun+I0MywdHOnL9DWxQfFYS8LS5/n7u4BvLFXWoZOAkW8LOqY6OsNkIDmFlM1rHjRGSFMi7LitPbekrE1Ga44a7n1yHXXo7GmTUjxJG8sIa+6bi/as1z5uHumebU6wTX19CGr6joXqd+HStOZu1IB/jOQ31NoFiksE6jISPCElB9IOka4E5oe/axvVrzHk+mgRZNh2q+5ZTW8fvPwAAEDn5Xdl3zBrj1R6AIR2A8iaGoPTPwsAiB74MYKlr2iav/2uBaCoTBAS/RQxAeDGNpawhnXXBoVzufWVSdKDKuXaqXW+Vc5Pi9tE+xTgcWOzD46czmNtud0GIMyL+3tNefeWJxPjWyaduApsX7N+K6UqtBqK38fIaJPxQG3iXfL7sv2cxuO2Sa6uWzHGfd4/4d3mV0b95rYqHe/Zf4g1aZKUhDVuUd31E+vXx4QFSmjmGMqNuZzW8Dro33zCXD3po+aX1w+NkULU+RaH/Jv5bQw5+tkkFb70j2/lw6GUIXMBrN8YeNyKwjRZgdZO2819tNIJTf5SjlBH+NVtRLBlj02PbXLDAWwCh7KCEw6glVsYAAAgAElEQVRDC/YTZSdPZRdkSfYlc44GN14YVTqBaYvMfewgqLVTid/7XXNtnwvM/eV260RZZIe986WfMQdRCB0OcWEoTXbgKsfiGFiEIhoqPWejjRLHN5vbyMjkdhIUZk52cRGFcRaZWPoRNHOIT12Uq36aFpdq8vZuArGL0HZXJxrwrLpXKRdFlhc0pSpAmwrM3YfScowbKjmBBOWYqZK5b0jw6dNAOixiWnAU7Fy1M5j85i/YsmjrQ5qIjVEfcqp7dtpWJThFftmdC6rxZ5XYKHK6vNFd87Db1NECRvc9C8VMKDKuDABU2gvHi2nvdNGXR91gCBh1P79/PinqQHV3E6mLVaIzIn/H0rWPZDCtsYlZ7pIi3XZi+cRjZABAx2V3miuDa4SZoYzNkc3mJuvT6l2q3U/HurebRfiUzzwUT19HY1us8uKoPiz6NCfN/P+O/c6ZjnRa51uLsObMrJjco0046/P7tItzJ9zQljmoZjdVesOzdBv3l8HC5puvfK9ZyJl4KdSf7EaCN/tBegMBiHbqojvrPuNcb4U9WWxq1lRRRH63fT9y4/WQGZetsEUF0GtIoJZzLN1SMZ7zfmwjbRf/9K55LitVXftikpQV91rBpCV0CMoublEyTVUBNPdRaitRQzCvkZltWAOWvcJd5/wB2nRn0cyMUkc235Qmv2wWZdttBB0lTBDFsZYbeU6jg/qLFdqIGCKWWKbu5lXu54MvuPdO79rO850z05sWjy0K2+yGw8PDw8PDw8PDw2NzwMfh2MYwXhKP+jffAgCovvumYjS4goI1dpopLZMRQwEXrXPevnQtQLD9oZQc3V8bMBJiAGrPsylN0hZEDWANSSIXHZG7qB99zbsAAJfec15r9ayMsgrSzgQdsXNQykT0hZDGZkUGb5mPstIQZyIgeMo5SnRePE0mJ5O3Q4pzn6Wf4YiTurD2pdwhuO5ZMjvozDisZIhNPCpOis+SqPqQMMdiqeqIo4VlDRpLHUc2FKaMBQBMIxMlFbh8rCbKORha6b2MhI6kKj0C0xE7CLpR1vZwNPiumdC9T9HxLPtrKUvXkWnVdoeY/xsjzlwpJ2a+64t0JEzGWDrmwipnm+ck2i1U4CTFSa2ddGgU31erhElXbcCly/zxLN0L6850sggy+klWDBt3jfLvmA59jZGsB+d/3V7XTaSCWfz6zagwp37WRCuXjuTmYgl6DKYcegNJ7LtmOvO+hBlQjOaXixXVnaM/398Ycd+5Rm2Npb0dU50TakD3NEagq87UBQAQNqBDfrY7nkbfing7yoM1pP0MysI0pIVJFY/rQrId+xbsVMv9lV9IqU20ddYGR64vcHutD6XNzSQpQts4k6JsZozHvK+/8+8AAPXqbzupPGs2WGurAiCk90rtRi04yN6vWesUlJyZM0vw+Vs2RkR8D2pnQdmNSWz+G9WdaZ8SbZ8xlmjxrE2UfTjZn1UJTi1MxdSRoOSVWg9OI0FoUe2MU6UDJi4Ot0/qa6pUhX7kV+Z44eHmGllK6LC21cWL2dawzW84PDw8PDw8PDw8PCYS25iCw284GBtLnVc5+W0AUEy7gbhvRuw8SbKztB8qGUAsIz0tbJQHLzWakK6P3GOuBSWgvzhF3gc/YSKcxxzQrP03SUJiFLgOmu1MbSRQBZDdsabIy1ZKFNacFDdmr03pDiZsjAFD3QjEg3ixY1peKsfJxpcEpaqTVCX8ZZTwf3ESqGHnSG4pOQPgod+Y60eYSM7OKVPYS5PPg6q2i/dGeTdGrFbKtgei4tSVDueMXgCDV50BgNpCyk+F6ij1vCxdrnRkfFcFNNjvIu7DogfXCJpVSq9zOhSIPpffRbULir87S/aEM6QmDUDegfn+914EAHjRT1ygqlQfk4lJogI+ZW2P63GJorxW6YwH1YKzx46hfZLTDpA9NPtgRH3PW/+lIrD2+gX9xpRSUKTZiL5p6C7Vq/7X2fYnA2/l9OGQsI7kj95onlty/Jg0cYr9DEpVpyEiG/gssgTHaFvNaKciqBpT2nKaN10Bddx/0m30raudgNSGAUYCnYgY7dqOLlzHZ397EwBgu1epdJDTrDGfx3UR/d3WLmpYDajT6pBUfWidG//ap9p62rbL41yl043pVktH+Yysh5q8oFD9thZs1Lx/KtGoB2XxfeI00kopRL2Gvj6YtdRcqQ86AglG53TofkNprybROMFtVKXne6WUo5aleUB1TnfzoCU7IM1UuR16PdG0T985fx0FaUaSRt3O+Uplz/kp5/K01tlq30Y2CKf6Svwe87D5rQ0As3Y3p7hPz9jF/L/hOZeexxYJ79Lv4eHh4eHh4eHh4bHJ4DUcCcSo8x7MT1kZ3vglAIBafJSVFGWxyeQGSTeiFx6hDExZAtrdjw4nTei8/M74pfoQsOjgwkX69aeNffwJx/2nk3hmaFsyqfHYn8RKjxuOvaMUtyFXgspRSk5suqwRkZI+tvFkSVNYz6ASHAWCdcMxgDBLEzMSlZ19rtV+VKymR0eCjWv3l9J9bH9Nv0NrnU00Mzg1RlyezLJTqjrbWdYcSDauruZUqM3Qt9IwbHWJOjGkJKqVZNvRwdYcKwjfx/bl5XahxWGax3qclcVcFJqtuL+GXv+kY7DKiT3e/h56WNipZyIRPE0EPotRMnM5qf9pQe+sLQMMPdcYTtsm68hpELjuLH1evxyafGoKsVTRO8mip86L4MJvAgCi778B6tSPU7niQRkLs3yJZ4IdTzDp3/5FqL3PKVw+fdc3THqHv919owoz8KWZleJS1ASjmCq5tphk+DvgIqeltT4M2mpR7DeW1Kd8P/fNandheuMF31gOAIie/xvUnL2omAk68cyAkhnfW5WAkqn/bSeZMeGQnzvabSeRdv2WpeaWKl0yAyb9Uard+Fc3/hjLvD/0xYsAAJ2X3eH6NP9WOUBfBDWTJPKShbCDx00OLDnoNA/MAsVtamSD0/4iPV7xfKRDwerEYyrfU+uH6o4H9swDTVTQwYvOSl2L0XKnxopsrYd9P+WEr5EqC3854etok3PsVszEabVEbAXQNdMFit1KfDm2NadxpTNbxfjiy/uYSfsNt+WMgj0BaBVdWXNkXOZbz4HvHWgmtHNvWTnKnfnQynwgdxpyYoYzz9L1Iejn/gIACHZ7ee70ops+ZNI58HUxUxQDwYnNA2vMwY0GERvNNnJOclkmT7xA4EWodH5kMwCl4mZYgHB0jCyFrCJV9mi4fFeT1mV/etwNiDZdMfjZ98r11CJyO5Uxqmc7EPLzLReKiZglgKu75GgnZ1c1bYfRK0d4+iyzmVvwjSebLlblos1FkM12itXP3W3um7ssXj7ptM0bqHIHUnULSm5lmHTU1xr62T+b23Y7bdS6AUD0t2+ZYuz0EpdXUo0vj7m84QiVD47GVcYwSC42VeDS54myMZw2B9AhdJ8x1WATM/3Ereb/7Q9xZmwFJsjoqdvNM5Ki1Dpc8sZPyJL4vTZqmeZAuo9MOYjEIW7uwO2Z0i+3uWO5gOJ+aiMui7ZesB8CwEd3M+X/4N3jM2dYyuscQojm9MaJPiHpPmlTqaYszFWex0817WTxd5/NiNLMyTbb7CfaddRItbHYOGjHDBdpPJ6e/c89A7jnGsN2rFZM8Z0DH97VfMNL79ly5v1WsJvmnCa4dr64+7nUtSzChVi/Si67dOTmC2l2nEgvi7whZpIXG38h8tH2uIgZJ9dxPCO0jxdavYu833Bz4/4TNq+R0R43FiS72Eh4DYeHh4eHh4eHh4fHBGJb82mYEA2Hh4eHh4eHh4eHh4fBPzazhmP3CdZwbGsbLA8PDw8PDw8PDw+PCYQ3qfLw8PDw8PDw8PCYQGxrLL5ew+Hh4eHh4eHh4eHhscngNxweHh4eHh4eHh4eHpsM3qTKw8PDw8PDw8PDYwKxrcXh8BoODw8PDw8PDw8PD49NBq/h8PDw8PDw8PDw8JhAbGMKDq/h8PDw8PDw8PDw8PCI44orrsDZZ5+Nc845B3//+98z7/n0pz+NCy64YNS0JkTDEf3zpwCAYOFhG52W1hH0DR8AAKiTrjC/GdxiWWHvs9OL36ejEPq2q015T/ho7nJF3zrXpHP6ZwFF+zgdml8l9nURn1NcAHc9qpvfsI7a508FAFTf9gtzLqBPVesD2ibF09UaaAwn8iy5Y95Xjmwwt6x+ALUfmXfX/uEHctdRr33Uls+C8tX9K0y2PfOgb/sfc23PU+meIajpO5v7Vt5n7pu1B3Q4Yo4nb5/IKAKg43WEEvWBOxfWzGGpQs/Sc0EJ6F9p7pr9olz1633jdADApE/9E1b2EDUoK5Uoj0Bj2H0fVaJyhOh7/14AgJ5P3Bu/FtaAUpXSp3dZqro8ZbupD4nrohxRiNpnXgIAaLssexDIguZ3EpRyPwOYfpLZz0b6zQGnV6P/wzpQ7TbH1S7zO7zevj9NbRGNEeCFh02ZtjvYXPvl5eb/U66w70JN2zFXOZ871+Q192tPIC0/4hinitqYOKcC1zf5Ww6uAdqnmMvcvjKgKS2lgvR4khFX1V5rDNs2oXrm5qgd5ffCI1TOClBuM8fD680vjwOlNugn/2COe03fxMIDoaYuMmn84Ysm32PeB73+SZPcjF3y5W/rJN5n7NhBKQW94VlzPGevXOkDwOW7mvd92V3LXdpBYrrK6othzY1P3O6iEPrGD5lHjnoHnaN+Xe4ASpQu16tUAWoD8TyDssuP227HVPNbH4a+5b/Mbad/MVf96lfsZ7J/2y/h+n3UvF4WGql2HdWh/2jmLHXYWykNZcuGSkf8nNa2/3PbhY7ce+OxhlEbgP7rNQCA4Jj/zFE7gxXnm/4/5yvLXelz9A/oMD7um5PAwBpz2DUzlUbL9FzCzd+xjoChdebSzN1aV4wQ/dS8a3XcB6F4XNPJQGpZ6w/t2poYU9e8eScAwIzP0ZzM8wV0+n00W9fwGMbl4HFZa2x47x4AgMlfeqFFreLY8ObZAICeK+9L10XW1ZZHfK/keBqOQD93t7lrwUEZ9RDPps7JfBPrAH5PjWHoVebdBTud2KRGHkVw11134cknn8S1116Lxx57DJdccgmuvfba2D2PPvoo/vznP6NSaT5HMryGw8PDw8PDw8PDw2MCoZTarH+j4Y477sCxxx4LAFiyZAl6e3vR398fu+cTn/gE3vGOd+Sq74RoONR2ZjerR/qg2no2Li0VAC/9CADgh8fMAQCc+VuSrosXmOdlZt6nFNShby5crvoTZmddbQwD9UEAgB40Ehk1dQfz/9pHoabvxBmZn6jhpPQsxah0oDKJpGcsWWXpU9hwO3qW0jWGnDRZSnBYwsD3dU4zl5Ycj+ouNxau451nGC3FwdevsudYohNM3s6eUy/5SNM0VLeRmOjaAO49c08AwD4/WxW7RyNwEiuqtypVAJXRXIOOzHx0WMfaDxwOAJj+tfXNKyXQ827euSug73kAwI1n7gMAOPYkI5EMzvyY+4b8bQZfALpmUXlYW9VAz4f+SGWPSwy1anf14xylxE3Wk6W0SQRlVN/6s1z1iuX9+G9NFjse785FaU1cpjaD22nAbXIE+peXmvtP+7Q5x+9kuBeodvKD5rdjKrjdK9bSQQPd9O5Im6NO/pj5vzYA/SujZVQXfj9X/b57n9GavQtZA6r4X2VoeEoJ+UvXTNv+WkG1eG+txiFVbs+VfhK1z78cAFB9+/VO80JSXynhDZa+Ivac1hoYMH1NHX8pJdaP2ldfBwBoe9f/mWuizdn0aExT1S5Rp7R0Ug+T9J/GeR3WoK81k5F628256/iWw6gMOoRe9Q/z/Lxl5twgSWjLHUCZ+lZME8Hflso3uAb9t/4EANBzwmXmHGuE2nqclpHT0JHTInPbDSpubO00mlAr1a4Pon6f0Sa1nZ6vfqULjRZYr3nIzg8YWG1+SauGUhkotVFe9H7rQ67/cXkbI8B2e5vrvU+b+yfNp/QfhJq11NzH2jAhQbZtVwVI6QbsfNQObL9fvooJzP7y4/b4iv3MXH3xn5/jxGP1iuebtSxR0Dx/tehjred91Vx7pALojmktns3Ac4/YQ9uPaa7VT99hkp2/v5ubSWumV/wVau4yLjD9ljD9IrLcKLeb+6QmkefwwGnjnJZKcyGcRryc0FKpAD1v/Hyx+gH48Z2mr12UNZ5mjaHuYno8FeNdaw37KGu3zPYBoNIBXels/axHIaxZswZLly61/0+bNg2rV69Gd7dp09dddx0OOOAAzJ8/P1d6E+M0Tp1ctfVAU6dTzRZSudIzjfWsm415yOcPNoPZW+9YsRGFpKRVAD0GV57KVJrwn7gFas7e8Yu0UFPtU9Oq0aAEKJoIeOEHBexlFoR63RPmDC3UzWRBA0yDBhcEGapi5a5b853QpbHzoYXrOGl0jVlu6Of/ijmzcgwOBU1/LIbXo2tOfjMVAMDPaaP0umsBet/H/7cxJbBmCf0rADY94QVHWIOizSVPLnrNg0AfbYR3OzWWTWyCHGv9AOi1j5k0pi7O/Yyat2/6JE9mSVOK1MOl9P+DG7g09GN+9ep/Qi04IHbOmA9S3dn0p9yezpdNXMIS0J9vs8h41x1PAQA+deBcvOeujR8PUmY8440xpF855iIAgB5a6xbLPNHa/q+heWHKi+awDthFFX2n5+9B9fjXmmNuw2wyVO1xG2gqp47C7IUOo607nv7qB4GjX1u4jlOPOwMA0P/BA9B1IfVL3nBwHQMhVOG2qQIAbCbkxtPuU99oTv3DbDzUzi81v0pBJzcoKnCbiUiYAKm4GZKtf7mKyosvLFbBWz5r0uXNNeBMtHisyVrkldvjwgkAOqoDD91qnjj2YD5r/p+y0PQxAdWszbF5XyJPHYXAU38x/+z0kuZ1Sidojy6+y5jV6W+a9xT2G/Ol8puvL5DeFoZTqV0O97rN3EgfAODDZ5wHALj0VzdBzdzdXKuY76Dm7WfnNRZGaa2hdjuJjtlUSZhd95OgYPICcyqqAUh8L1UG2ifZ9GLXAGBe8U3jk/30/MAqaGvKNnbDGJXTbHPM6ffM2aTpjze2tkjjUqC1fv16XHfddfjGN76BlStX5nrem1R5eHh4eHh4eHh4eFjMmjULa9assf+vWrUKM2eajeedd96JtWvX4rzzzsOb3/xmPPDAA7jiiitapjchGg4rpZ+2xGo2NKmPFe2ac6fVGIHuNVJMNm1hzUb0vdcjOPcr8fubOLs2TT9qWGdjFCnbUSTFG3gBuo+cJCct4ETNz8AqKKvWJgQlpwZlyVYtBHY+zqTBJhMNMmeRz7MEM6o7KXJUc3UhsyDF5j6s2o0aTvpcALudeDhVp+HyuO0LJo/D3566n011VFBKO/Mt/xNmn2oc7ZMSHb3yPqg5e3IidCnLITeyz6S+cWME1cPjJiWj4rg3md+oYaXuatGRJq/7f2SuPfsAdOcMc40cfVX7FGvKYJ2ml/8Jas/R85d1ykt0wM+xtKsQuP9FodMyCKc+U4AAmutRJRPI+oCTlNq21oA683P2GICTxE1bIpxQRTtlqV3N2YHqP33VPLPsvHhaUQh13peL1Y/yfPefnsfVpPl8yzhoPrcorHvG/Nb6AJbotXQqFe+8fbI5tuZBM61UFn1k7jKb+l65zT7LjuVSm+b6oXL5JUkx2icD/c8Xr2OPMVvq+o8vu7bCaVvTvrJ1oLdmjjpy2p6aqZd+9s9Au2nHavFRdF9IZRcaY0EQYUcrylOrIO3oy+Yzg2uBzinF6nf4G8xvqWLbrH6KTDAXH003KTfWshYwrEFzf+Jv3qgB+56TuI/TvB1q11NSdbFaKqn1pnlIW+sD4cy+0zHF6gc4x/u2bpufuujbAIDfv9TMSUcVsV4mp+5C83IRsGbPWhOMAhov9dBaqJ555hy9/0svN4w9qnuONUfEcK+5f9V9RssBQNsxNXLrGtJC8tyvoYAuM+fYeTQoZ8+HCfNYDaHxpP5dRMtw6XtM29F/+AJw4EXm5JRF8ZuUs7BopbHXOrLzyiYT7NcGN1XK2yQOPfRQXH311TjnnHPwwAMPYNasWdac6sQTT8SJJxrn/GeeeQYXX3wxLrnkkpbp+TgcHh4eHh4eHh4eHhOJLdymatmyZVi6dCnOOeccKKVw2WWX4brrrkNPTw+OO+64wukp3ZpHblygN5BErtSWtg8dg6ZDC+pHIC4RvvNkIzk56PpVqefs86NoPXSD6Fon5XOEAYDoEXK4nLbE0UAy3StLKp6+01J/YmitOTe8PqUJQaXDOUayTwA5OarOGU5iTFoPveFpJ5G39vDKSa5Zc8KSwbYeYIjSm71H7jr+9TgjLZFO3q2k8jom7YxrInRjGA+dZd7PrtclnMYlzSE7q2Y4g+mo0dQeWYd1rH+XsZ2d+pV1o1cOQPSQsSeOaX/ID8M65RKtn3nASBHXferVmPrvV5pniY5PD6yGIntalXBGbEoxm/EuWcKZVU920FXTdshRO3pmhaHQVZPmiXwdrWuyfDHHRXbqtlqSBvTvP2POHfkucxtrNRpDQBtJ061PQJeztee2HjUsVS+e+J1J60Vnmf9r/dB/NDSjwWn5HB6jr55m0njlV139biCpy2HGjj+Ysn3quWZIvpvxhv3mBca/6JpXmWdOkerrBG1zULISeSvNbowASdrOqAH9C/N+1KmmDVuJbLnD2ppbOuXJ26UoQJUKhNNsnSvm6vgbU87gzPzaquiG95i09z3faXKpP7FGRt/5NahDSFNQH3YPk608tz/95B+AlQ9ResQVL53MuayWWly7MdY66QqqVq4329/3Pg39iCFjCI54d676sf+VXnW/JZywmmgr9Q7NN5CoDQhHeUfewA7nrIliXx3VM8dpfIRTvPQdsGBNF88XfH/UMBTRQG6KcQDGjw1mzsrWTgO979wVk696kB6gsvSvcE7vUgPMtLXs6zLO0DQnM4X7aIj+afyB0DYZauau5pi1MKQFx0gv2MmZ00dYc/1kNb2jxUfZ9YxdhPJ3i+ruezEpQ7XTzglaUjhzG+U+zBqytkk2zxQNfQvolfeb3w1P44UrjBZtxqf/astABc7tixitNUQCQYE5qwisVUeBdro58ejJm1fmv+P1jdFvGkd4DYeHh4eHh4eHh4fHBGILV3CMOyZmw0GSmCypLkv2dG0gP3MVS/Mz0ktqNv77kDl40+1xG+5RbeRbBPlqigpJfZ//G9RcQ6XKEgUb7G7GLs6fg+nheuZCkzbDSp1FQC+9zkgE1NQl5lpYE9JGoxlRHdOdbTZL6epD0OuN7wy6ZlP68139krR5OdDTVrB3jEKz2tmdowykiUEW3d0oAbJqQwV376SZilmYJiTPav7+LgDi8yag3/rV/ZjGmiuWIN/xNeDIt2Vm06z9ZZ4nbRuqGV11DKOV8y+aJ86OTjOptbb9zkpFVQCseLRJmQJrJ2/7UywYpWAEYknqDmS7boM5VoEVjlozF066NF2Hkz8OAPjRi00/OOPmfIwasXJuQRj++20AgI5TK2nGKMtgNwJFkupYMDIeO63WYxhD990JAOh8BbHoUF+T/g1WwxiFlu4ypumwGoNS/FrUQOOfJv0iI84/Pmdolpd+97WCyUuw9gBQh77RSfkrTtpq+wy1ATVnTzR+9gkAQHnvcxM5KVtHl48W7Hjc1kspDYdF+1TgXqIZz6vh4DF8zt5OCo2EpkUFThLODFJt3U5zzHPJ+qcAtv/fntgH5T02AJujWLcMZpZdLkpTcHM9B1ZBryENURENB/tzdc7IoLI173LKZx5GxGx7rLlgX0MktL33/dicO+DfcpehCPRyQ22cV8OBXhpLX3SYe1fsY8najM4ZsH2I/BfuufAALPuuWRNgd+JRrg9CUwDUgNcPXC4VuD5MfTqm8bZtXwQ25Pt4zI5CR5lcRMOxmoIQTtsRMz59ty0rAOiHbqD6n5U7PZDWCy00HEV8GVPPkjXN1qLh2NYwMU7jz94FAM6cKAOq2oVovRk0RzN5sFFjT/zQqHknNxujQWttHYTVgW/K/dxXXv5iAMAbbnnMTUxMBzx/f/N/w0V9tSYUw70ALQKts5oKbMRMtd0hXDL6VS6GATubt3U7FSrn3TEV7Pqo+T7BtO6cLXMOrgCGawWt70Q0Zx3ypEcTeakNw4O1zMdiA01HPmdM61zJ77xURVtnsY3jdW8x3/uMm52zt6aBXcYF4YjM7Dy++MyznTMcLa7VCZdB//4qc3z0+wqVI4YWvOL6mT+Z9IvQ4mZG7KZ2Eglq5WQkbiAep4CxZyKiqzRBYTMXNnEJR4Tj74jL/anbTbZsjsaLp0kLgJ2bjxlZWPXeIwDEoxszXnGTcZqM/vkzqJ1OMHkkKEMnHsUtWsNBMqHI2hjy+5XkEnyuPuDOsZmf1ih3JpyERZo2PVoMZpowai0ougli8VqenX+Bw7AU/h3TRAyCSjztoJyOUySvczutDWLNA2ahM6fNEWeYNEruWTlujxgHXxuPA4hvhCE2eVEdAw+bBWTeKFOKnIBjlLVJc7Ryu1vc9ZGp1KQFsG2Gn334N8CuL4mVDSUum6AsJtNG1TE17iwOGMd8fpbNHrme/SuBzHFjFAyuznWbIidk/csPmv9PvDz7xlk7ZZ8fL+SMMM4YvNZsYrv3PMe+M03tRZqn2u9Jc/6yHzzs2rLYIPR/2myGJ33qwVg+sT6XjG0BKRQC4vTQMo0S6t/8DwBA9T//lr+St37dPP/qa5ywiNvdQlqbDK2Fptg0bgzR9jg2n4/Eg8ZlYSwbDYuhfObTHpsH3qTKw8PDw8PDw8PDYwKxUZurrRATsuEIWmg2YveRZkOT81qzqORBDs1GK3zuoDl4253Zmg+lVDEVIeFVxwkpHkuKWDXO0tyw7tT9JLnXYQ1gSj0RdVRNEzSPgKAuraWlz7UB+6xezVF594N+jCL7zl8WL6zWzlG9AEY4npaOmjrRxpwQZTmT0avVNKkAACAASURBVIkbQ+ie0hF7Rprv2I4YtNZS6ChhosM0nlEDpWoxs7H95qW7Q0pyLNXbLCXe70xguZHS6x6izF14BDA8ujRnNLQckIhmsRBSUWwB6eBLJ0T05YTphSkUH0AtMlTJzlnctHk93OtMJG309ciZGfY+48pjaUYTUbeDEtQeL89dNSBbs2GLzeY+u52Ky5cZSuPL72lN2bqpJ4SxOKN37X906pyWkmzAmM1Ycgn6Jn0rnAmriFxc7s4eZyXttKORleZE4rslKGPtfSoYU/DOHyw37+XSxogbTzlN/l9Sitvgpg1hhkTvtlTB5AWz3XUg3q4tlSyNzYGL8G01BKVK2syKoQJ0bLeoUP2sGW270ODa8tI4xE7ccGYw+u5vQu1PFOzsvL7vq6zGGmpRvIg6ctouTWa6tYG4BgwASm2Ovp5pU+mbq5m7OTPSInianIuJWrwZrMPxCcYcUv/jJ8AeZ8avURm3JKx7xjjSd8uTth/Qj1LO0ZvNntsm27Ym6YlLnbGURJL5qP2VUs6aIUNzW548PXVuVBxxkfmNGq79sxaFTMX12qehllJwW6KKh46ElkUGH4ybi407NnFgQY+Ng9dweHh4eHh4eHh4eEwgvIZjEyBaQ85QM/L5C7BmQw9vsNSisfS+YyQ8wflfH1N5mmk3bPpkex+c2DpqokT76//HHAytcxSqTG3L9v0dU6Efv9Uck2RYTV5gbfGtxFxKPa3TMEkVRvqcwzcHZdORox5lGsXGENRSIx3W9P4dtJNSFMDMKY5y10rI2XExpmFIBNKSUiprJ11CtT3uHyHLZ4XuZHeug8np+6S2p5TWZqhyseY9b9lSk1cWLayU+FsKUKKsnbfM0vdihbG/1cN9jg45Q4NjyzjKgNPSgW4sjNZJh1dzkgojvhNr1LKcwCW1LbdP9jVhO/ewBlTY4Z61VSVHzWj9tBT03d8zyc3d25xhX6bhXpdn3uqRNH8034xL/9+nAAArXr8IQGvNyJaGwb+ZAHFdp2toZLRPgPocvXd6F2r6jpY+1X7DqIaobp5N6Vp05O6z0v9Kmtq81CY0sPR9pUP5UB+K4nyeKsptwgGaf6nOIxucj0XD+QRB07jDRB71IQSVBJUs9wMdpnwWEDVcO7bagchpFFnzbrUkyvmo5QQHitO9z7gAnrY/0/utDwKd02LX1F5nuW8t/aGGyXadvy/7CKrAfQvWENQH0+N/oN14ze+G34HWcR+ZnHjoM2b+3C0jKGwWmOQAS0+3c6YmB3IVlKCmbhoqVZs/+9XkxPwzzgdAbVy2SQBaaK6Yzhc/Nb586mUfc3TTVnsXouNVH6P0EgQQYc1ppmUwXRsglX7rg04LZDWMglTixHcWqh8ARxGvQwA8FiR8L6MQeh05/vM40D4ZQCNRxyhFEZ+FjXEaV+2bhjLZY3wwIRuOsfJmq/ZJlmM6xmB1Qj4mkDwY/rhxMm2/+Pfu5B6njSEhGlSEU5+NiUELKL3mQajFJj87kQ33usGf+bRVyTJMqVnEtiBjWthBhc5FDbvgU5O3M+eCinU4VMwJztHKK4Fgc8gfh2PBy8nUrDEkOMNpmcILktgihfJrRNA/J67/V3zGlmUaxa7QNnKwm8BTpg86NBGN+R3QfXrAqLXB72v7w8z/UQOdF30id90AoL7eMItIIy476PGAH9ZdnWlg1Cvvcwuu3U8yvzdeCez1kkL5J6G1dk6j/H3lpmX6krEkygdi0RWP9G4OE/EydOg2jpIMgNsRT7A2unjVtdkKRxpvuMUaX2sMAXPM6lKx4yH3g0oH9CoyEZybUxUfUrsZZWQL9jZRzed8xfy2MrPc0lCd4gQTyfbJiJlh2I1vlPHNAd3IXkxKZ1U7DmeQGCilXN/kGBlioR4OmEVYEeOxxa9+Y6qcSLAzxcwtI4pJ1PuM22Dxpvbad6B6EhOAJBZzUQiotIlUiqwhrIs2S+MVMcfpZ+5E6bjXF6idQO9TAG847HjJZCA9aZPGxlD6PUQRQCyGNpo0bzgawy5uVRP2JwDQQQmKBYJyo0F52xghBbDr958o/AwAU+c2U+97TzNz514/XVFY+FAYSbPE0bBgafocj2+irJYd8nQjyNTP/glqrjFz1l83juLqrKuA9iRBCvfftvQmBBAbZfoNKm7DyfMVb3baJsW+f25oIYxKCKvUYmPaqTv/bpkwNRMF9D0LNXP3eBph3TFldTY379oYIZyNV9YyBY/NBW9S5eHh4eHh4eHh4TGR2DTxZLdYTMyGI4carRlsVGOOclzpcE5t4wDWbFx98BwAwFvuWAE1dVHxhPppZ79dt+A9J8kDS6dnLbUSYBsluns28KiJsIxD3+zSW2eoQTGfVJpW6hQJKkgROVzr+LnISeKtI/nio0T6ywtX8eorvwsAeNvLr3ISF5Z8tjnpScxhFACCMtRpnzTH1qG+hvA3Jhp0ZY8zYvnImCxsMqKCcto0KSg5jY4wIbC481rzu9f5uer3uV8aaf0lH24uH1EiRoulzJ2/P5xZEuV/5tV48iJjArDoO3HH57yqYqVUSrMRu/brz5p/3nhcrvQAuCjCUEKLkSg7AJCkyJqsiMjVTsqmgG5yEuTvyo6Rq/8BsGTPmtE50wy9wlAzqmk7ApOJNIHakotLM99RReeE/vn7zbNnfqH1fYm29Nbbn0H4JaPZDN7wI1HuDGrHcYQtR4FnvvETQ5n9+ncKbWIyvczyim9uNZM1lHfIwVmfjHidTJlNRKjfSpOq0m7FqI0B4J4vmgjz+51wmZPQi8jLAEj6T22Rooqr6R1WEmvN6l79bdx2uumLh91gJMrWubbS6UwjZd+2NNssPdaObpe0N9bhf9GRWPFGMyfNveZ1+SrIdRhe795Von6q3B4jqzDl7XJajA6SEndMdTS+K0xkaMwx5omodgnzGmrLgoxFsylaqWK1tGyCa0292ia5KNz5agcA+PGJ5p0Xi3sTb7t7/8zE8tC1ATF2bSKM0saTqN/0DQBAde/zhdad+5+gxQVrXSkGzoKD7PdU53zeXOucgYGPGarunivvj+UT01YquWSL5wVVgu5mTRTdT+OxUgH0T4xTPt7xu/yV5DYj+0aFSUCo38zdx2mWmRRk/XLo200fVryuiRrAQLG2kF2kFq1w6IWNTt9j08FrODw8PDw8PDw8PDwmEN5pfFOAHWqb0NzmgY2aO7jG2bWyNHUc8JY7jP32I2fOxo7fJorBAnaraskx5iAKnT21pXal11xud9I3tmGMGlBHkDMXS5UBfPjV5txld5xiTkjpDks02FZZBU7jwIHnBH2rWnKsuSYcK7/9JiN1uPAv/567jjtPSjs760qWc2683kop5/xHz0V/+DxKcxZmZyTr2soxWkp37bug9FfeD0ybl/VUU7zzDAroJ53G2SchSDu4xyhzrZbAOQ/e9oyRTi5MSinDWorqU42BOjRca9psIa1sjRx42ybHfDFkWaAz2nBtIOW7AqWcb5WVspFEbPLCuI8Rg50eKdgX2nrsd+Toy/a5sBanDc2BDbf/GgAw5czW96UGelVC8NprTPaPGzpptd3BY7N73sR43Ve/BADQz98DkC24bT9MgaujtEagPgQkHGN13/PA4n3N8RO3mJMLieo4KMeoWQHqD0miBhHEMeYvAgB9K4AFxaP+Lvviz6l8K6D/ajSrat8LzEVum6XIzQVZ9tw03ulHb8Qh3zLR2SMKqIqbSbJ83Luhuo122440Ud2ly/2l2unyZYpgast61QOY/V8FpMaAbVdqh2PwxCvNOLXoGopcLfxxUuNCqQI1JT5u6qjh/ASI0n3d240/xtTPPpwdrJHTl/S4LHWfs2c8fR2NyYejazxXF5VOgDQ7aV+HcUKUHYi2Gapv+wUAQN9wCaLH/w4ACC4wUn2rmW0MCXIXYfnA8wrXpT6E7o8Y8piIorrbsXLSAqs5tO1ca6dVlHOkta7g+cppWtRrvl2ofgCcT1bbJKdJ4fwkVb9tR6T9m7w9wBHhJWX6nT8wSSw5Lla+ccPyO83vkgJaf48Jg9K61YpufBA9dD0AINjuoNS1sTASRL8wZhPBKcWcgrOQlX/0249T+p/Onc53DjCd7/xbneNpVtqtGIvcwk/jh8eaSeisguroLKQdzhRuP9kMiIfeksValI3okV+Zp+ctSw1sjsEoSJvehHVnXiYmv6sOMnV85500kUhH8SQjTVBy1+XAlzLf4gla4c5TTB0PvjlfHQffa0wAOi+/K9f9o+Hrh5uFzGv/sGmckR8509RvpxvyOztevqtZSI0WfyILdgMrzGX0TYaJJjjugy2ey9fH030jgr7HTJLB4e/KVcbbjjZlO+T6FQmzhlRu9KuanwtrKe75lv22qcFJ/LrbqLqNGC968yC6xbxztd+rramDY40T/YbLLlnVbDRpR7yw5i1mcTrjalroyI18itUso89FDWCIhCVJZ1AVYOSTLwYAtH/4gdx1dN9xZZN3bsoSY8Oi+iRNiBDWcS3FSTr75ufdfbJO8UKnC6TD7PN07faXGfOjvOPp9w407/jcW/KN7/ExPNGOtIb+s2FsDHiRJ55r1e+y31t6Eagf+DGln19A9a39zVj/qt8VG/+yWPwkgUaMQGaMyJyHOf3Ehq4Zop++FQAQHH/puOR/44lmDDjxxvGb8+W3f+xsk/6SXzQyn8nCV5aZb/j6P278HKZ1BL2CNmbESDie0DqCps1awEKTLRzLT9+8sWUWXTcy+k3jCG9S5eHh4eHh4eHh4TGB2MYsqiaIFpd2sw+/YhZ2/vGqxNXRpIMZ6Z1k+KojinQZzC6usrdpZXxxdfR7C6dz7lkHAiCTA4odwVFwNdPSDq+3KlTN0seoIfjP6V2U23HGm8m5kVWarE7uXwG13SHmmB3ERzY41SxrHspVJ7FiGkfQ/9VuHPzufBJjiY+ccjIA4LJ7VgClhBRMOrMlX2mGSl+HdbzuSCMNtWpVkr5qHVkzAEvHWerKjLXRDHqkH3su2370GwU63nND8/QKauK01njNbx6MnYvWPwkA+N25B+DI806mk6Q2P/vqtBmTCtz1pCQ7qmPH7xSP/vuu42e4dCytZiJeSkxLZWvknLrZ5G1wLdARj5OjZZpW+0X0wZnGX9rlxaYqLGGvD6XSHw3raiz9dXk1daAe7Vy5zcXSIfNNrkPcFGC0NpF9XQVl6JHi0eiHrjcxfzqXXSBMKFWsfFBKUOY6cwdtY0nQe60PYtoeNH7WmT5VSJA5Daaf7pqZNtEJykAPmSWxczERhejaAKqz5heu4557kAlPVIe2dN4d9hwAQ/nK406VzMdq/Y5Ug7/R0FqcdhxRvkbOXA+AeTdJjWwUundWI3PgapejPmfnYs5nYDX23i+fVJxxzs9MFG7d+7Qz3eW21jHFll8lTEV1Y8Q5d9u4CCPAAmNaFz1v0lXTdjL3PPgLYLeXmfvs+2s4ynahLUnGVoqZgM7avVD9AOCCW4trUbksWed0wmn88mWGsOKyu58tbJqTScJR1CldOlIn2xD3l44p9l1bjebQOtv/tNDGHf+Fb5pzrJHk2FpBGXqVcSRnOl3U+gFeV0iTSTZfJzMuzVHrVYDFn/pJsfoBWDlU+JHWIDPPJBnCePgyKBVskSawHg7bGCmXh4eHh4eHh4eHh8dEYmJMqkhiufOPV+FTBxhJ2LvvMjaBMaemvLtceoY1G0nb8o0vb/F01BKSPAy+4CSE7LTJUoxKZ8zHwOaViNILAFhoAp3pe4nadRFpNTpnOEkw04eWKi4PKaVm6keWurC0rtwBzM4X9V1i0XgKD4bXozK5ifOf1k4oLKRIRaBX3Yfq1ObBhTKfIftPGzRLoLAERofQzxvqVyw8LJbukRedDrzoeHPtQeNsqofWQZGUGIMkSe2cbh39tdUW0Hce7nVBlKYuzl2srnPe79LhtKxjLEdQloEXZb70DiLhDL7SRJhN2dfXhwFLlZzhQyHTT1BKWigFrHs2b9UAACfdWOz+UUGS+isPMNLU9//ZjFuFxqtWGIM9escFHzUHI31OY8HBVeU4kJSsQgniDtI6vfAI1CkXm2PWopKGVk3eDghIEi4DmpFWwUpnZdsgelq+pu/7EfDS9xeuY9fxJiCj/sm7oE65goovxkzA1E/ShQOmPVnKbtIO14dQPf615vj3JviaOuytlFYJNjK5oIe285L8xlQ3dsy3GuyogY7jLyxUP/13Q72sDnit87vhd2z9bDLmoVJbOmhfYxh4wWg7LXlJSGWcvTQVFFIGTIz5MKgs7R2gG0PA+uX03xE5akdgyXsBzXRL8HxGfeayu0yZ9HdfD91mtBPBGVdvfPo5oQ41wSl1FLq2yUQCTF1c6RTfk+bmjmn2/tj6h4PWMsWxDQjcj+suOh0A8IrfmDFfP/wrqL3Pj99X7YZiTSxT1rI2TGuo2XEygDw4b+fxtPlRLsL6P4y2Re1iAuXqcvv4jKfB2NYLmwvbGkuV13B4eHh4eHh4eHh4eGwyTIiGw9kf7mM1G9G3LgIABK/6X3OtyU4vxeYQ1oEBCrI3aR5dIynB4AuObnasZQ3rwKCzV86NvUxwN13rszSRbMdspW9hzUmHreReO7pafgdDg1A7GkkVvzsrYZq1FNbulqWa9UEnnWuQFkO3Q/cTswTZHNv7RzYYys+COONI8761kGhyUMFg1tLU/fLbpb7jhmfQdtKbzHGCfUrffCXUsZeYc3ytVIUmaWbMJtZK4xLtZ+1ylI55dbEKtrD/LMymFjWcnTxrrpjp6PSroB+4zpwboe81sNJJqVlLNbjG0EBDaDFIgqPXPT6mgJpqj1dQ+eouwBdT1HI7DBrQvzUMcOro95j8XnjIBOkDoPuo/4U1qJcnmNzChL09gBjrkfVXEtJxlhhbSaxjJlMHv7FYBVOsShsLU5b33WXs0T93kNFCve3O8WIeGwNJIGvAdOTarNWcxlm16B/z0xh2vglSI8WarUEeVyngW6XTtbeW2jTlxi7b1qlf1odF2QpgIWmM932l0wBEiV+toZ8wGkKm/tb9K6FoXtBrycdp7RPADFNutcsJ9KzUziQYviD8GWxeop1y3djGfu2jwPxifoRqL0NfK7U0Nhgmj806SgdRjRpO2ynGPEVaVNseauQbdNPngFd+JV4XwNEXS/9BLof1JXNsgGruPoXqZ+pzr0lm/v6Fn81Mj3w2FbNdsv/SK78K/dPiPomp9HmdkpOlympEVFmMdzRP8NqhVHHvmN//8Hrnw2Ezj6D7yedlwPi5Kg7eGDVw+n9/zRzTGK12OMppkGxbbbi8WLMRuLx5PZC7fgAWfXQMVLpNoaGo3hu+8QEAQM9bDdOi2u5g4eOXpp7PlbrWQCZN/xaMbUvBMTG0uFZVn6HK/c5RZgI/75bnY3SY5p5sBUxSXRi7xtR2bM40lvJy+kWoKp8xVKqqZ45xpgUc5z3XY2CNo43kySuqAyXmd+fFdZvh7wZcZObHDUe+2vlEwX9v7tfP3xNzAgUAdE53MQwkXzZAMRjIGXnaktx1vIaof8+7Nd9iq9V31FED/ZeYSaz74/fG7ovFweCFDkcUl2nUh1x8luSGpj6IoSuOBgB0XvlYrvJGy83iRW6e8mw0tI6y68gbYxtzhdTtvU8ZnnLAqs/1Q/8HrCc1PC2y9eAa4Eaifj7UmGzISchGJJ6xa676AUD0cxPfRR1zMVJxMqTpQ9KBNGo4kzPm6R/uhb7BUEKyKYN0PLU88dTWZXyBmJOlJTegyZSjzId16NtMusGJV+Sr3+8/Ze7f76Jc948G3uQmI3T3v/9Ftt1K05eiKnJ2EFU9c3M/E33HmFKoUz/pHL3LCTOcoAyk6FPFotkWQNvNr1pqzDb0CyYOkZq8wAor9AaKETF9xxQVrVKBW+jwQpeduaMa9CMmNkpw0Jvz1/EeE8VZLTrCmaYm4sXo+38MtSPFGKK2pftXunGb3oleeT+w6p8mPd5wS9ML7gdM0NG/Amr6zrG8oCMXb4TbAjvAvvAw8PgfTbIv/kCu+unep8zvc/dAzdwtVgeLoGT7TIyMgb8rbxoEUYm+9TPmvv2pjbRNcv2aTbbCemrcBDTABAa8KbXzTM22myLzRfTcPaYaU4qRdzQDt0H0xOMrxRzeabGuf3MlgpdfFX9ea9u2gxlpk2KOA6QyhGeZ5VlDpCBtk923YJIPHvui0LUhu2l0sWw0OZer7lmwQjwSLlpBUH3Akc0s/7051zULATnyR3d+2dy/5yvM2gGAHiKhxFrqywsOchu0xPtrWcc+swlSYzRtTqXHZoj0HT55ijEtft/tj4v2P7YNBwAbe8fGpdnC8dSZm3eDtP0Ph0e/aRzhTao8PDw8PDw8PDw8PDYZJsak6uk7AADBoiNT187PkJaP5vytb/yQue8lH0k/y1InklapMUQ317f/t3n2+A/nf4g0HNjpeGcaY1Xw4jUHzsHL3sNaCdZEBCXo3mfMMTv0zjMmBrr3aSsd1xyZvGumkIYwLV63lchpCsplqRBLZeh1pDkoILE66+VptXo+DYCGVTUL04CuA5tEG9WRVa8+9GoTBXmXH7tgSEo4GTvH1QT1alhH+94FHBwBq+5NFIaK5OgLM7UZMoAWl421bSGly06Z1W7nGMhOufURYO+zzZO3UbTafV8JHPduSi5OFIDG8Ni0eFWSqEQNob1gikKh8RARbc1vZCTefB0wkt45RL9pzTU42GMNmqLOWrMcIZ115isC/A2tGUsITCkWLV7tfW6h+0dFImIvo+fK+3HVgUaS/s4/bYR51RjMjXp/b+ibp7z8Kqj2JC2x1DBlBMUjrSg/p8M6cK/RQGCvV5oiEY15LAAb9w2toZEIABmJoHikcWECBN0YBv5E5oMFNBxYazQA2LHNtVOmG+c+M2/vlEmZ6p7jtMP8bjumAnP3onMJ8oVan9NUEO0zSlXncE5Sf9Uz12gyAGvKaDXsUQgsPDB/3QBDjQrSWAbOOThWtnKbk/pGztwL0tEbMPTATCBy5Ntj9dR9z7lxh+fCWn/c5BFIBLnkts7pj1iJeRF587hLmVuYS1stHn2b/8/edwfqUZTrP7NfPyU56YUQkhACoZPQpIbelaIIF0RFVLzSFMFOF70WuCq2iwUpYuMHXkFRelGQKi10CATS2zk57Ws7vz/eeWdmy/d9u4eTg97M88f59uzOzs7sTn3L895/9fXY92hl7smmR162udWCcrhODS9rvoXuE5YZlW4n6lxt0Jg5c0RyL6tpkY3Jk8oz36m1b2Kaot+/9zvwt1falPmn0Ll1i4DHf03n9lGkCFNp/kQmBzDBRAoNh+4HxTHv2MFZSh9QtNn8Ls7/PZEnyKVPAuuVyRdHCc8WINWaJcmzpZSmjv8mGg7nNO7g4ODg4ODg4ODg4DBMGJnAf0z31gTf2GWyppxsBS9GsxF5ppLmSL+u7SmTwmPKxBQQM5Q03Q6oxtIpJZ3Qmg9AB1GDlzHpLIm1GKckx8//gU6wbeKEuca5UDnjBdxwlFZFrlsEvHQ3ndvmKPplaYrwUlGpMu77M9nkHni2Odd8h27bkIfSeTlUX6F6FEJ3ES0lpZ9z+c8aZ+/lTL4iZK/uZTH4T7J3bUsq9I5xGteS7SbVjNfICUBJ00Q2WEMdCBLQbUXs/FFyJgQgNyVpbO2np2LhA+QoucOvlb0w2+i+eBsw931NqxOLF/5Ov3t/xrTTkC+UyJUsLYNFeKAdkdXLyOSAZS+rUyGpaCZvtCRW8ELjBJsz+WsfktBwJDKpaXHlGw/QrcqJ+B1DB4iL2tp+5m76NgOXkpNv8fN3av+TxPBjND0t8MbrJMXrshplJPCYX4tqT2Q9SsMrPPPtYtux0joxYYP0gxpbgL4zPz9MtOFlgVykh7fEQ18jLfYet3zUevft5nkAxJhZ5v2xT0bOM/4sLB1tn4TyTw4HABTO+2sofbuxbbfHZzUWCCvIn5i4LR0zuQIH5Rs9DYPfJnrP0tdfTlZBHmsyOdP+ec5aSX1djNvCfFfW4AjP0lgp2H2TtdhKyyM6p0K+rbTvPOZLP0pjLTJA1iIyAYzGcd0i4yOYAuzjkNQnomV+r98LAPC2OLRxIjWWLfjTClNH9qnMlSB7yHJAFGNITl7+C12b/7Fk5Vn8MD1yyyPNuZC/hsiavqJ76Gv3QMw5XF1X47lfg1SBJbXDuQ7s2639PLUf3f6fN+Or0oyJ0ljIObQOYW2cmLozpan2QSqSg1QydSY6GabAfD73V9YmqfFItI2HvP/ndG66CgEQng9a5i8sCvB/D2xkCo4RMql69Q4AgGgyUCTdbEgpIV8ikwK7ozeC8DLwn/ktpd/u+GT5L6ZFmbdV8gXdoyeRGcIut7xlJjMefIpWvAlWm/NEVu0zk0/VRLWVt32Fyn+4MuvKGlMYVqvqTUhprNms8ORZHAOZU52ZWWfYJKHYBfm7M+naqTclruOr66l3pF3KSWtRqc26hEBudDzLUsAJsNQgVgdDq66Vcz2/12wB+THjG9wUjzdOo0loxg1LUt3XCL1focG+8xvPBM4LLxNRwUP6hmFNLZZzn94dO5wR6qJqghLbfgBrzyHHx7FXr01cJnHiVeooukllMwvp140pX9ZEJ44suoUHvOcUlV3QAV32LjfOiWwOVqsbx3l20My1QXLcB2bdylibndlRM8xm+PGHyFn2P/8+TCxSmcaLZV4YFL+iNnEsREAyU0MAkc1eEmyzYKdI3sYxXJ2wCAC06VO9as1wxnym/BaxOYW3VDSBqw0xfxsvY0w1MxbjFY9d/CwtPBHof5naf5owPrscsGOk/Nz2AwtuNtFgp1Y7DgezCMk6nnuIFr/z2ESON2OeMS3S7E+5kmVmGLMJCzPhZAp47lEiptg5SeUA/O5w6rsfuGuZ9a2o3GzSRkUPkhZIvxrdSBZGGWZF7qMW45s364BgXu0TTH9l0oFMNoY8QrXh8XOMPuwaFwAAIABJREFUw3YKPHkakXbM/98Vqe+NQxKmrNg+wWaH2VIsm6JOn5a58ak/0a+1DmlqDs7CpbnvM+Mgn/OyWP9NIm0Y9c3nVHlVe8/kTL6q/AHhDX+vSj+ENqEy8ywAiGIXqlcRYUL+gqcSV/H/HUHt9Li7lrdImRDdbwAAvM32BhAy2zzp6kjy1E7jzOo41PI5bFA4kyoHBwcHBwcHBwcHhw2GkYk0/jo7VDdRhabB+lWpkrNmQw4qlWKxsXpYCKGp6tJgm3nK0Sug7rbMG/h/ph5k6Vuu3Wg9bKfFOcoMjR3ebPkjSwXYPKLcYySMLBXpW641J9q5nGlU8x3ATkelruP6avRcMkmuMJJP6/2Ul5PULOS+GKSZVdFRA1oPhpcBwk7Idv6rSbKWtJFPe8+8hCmTof2g/2h4TYQdP22NA0sz172B2o2fBwBkTyIHSE0OkC2g68h00Y0BoHwV9YXCF+4z0u6wVE4I0061M/CAcTT1rbgi7BjPTqqsgu+YZEnvrS9QV+p/psqslTVFp27PWvsj4p3Lm+BTD7yZKn0rJJGw6W+Zb8d/7UJmdOeruB3DkX8Ya14gc4lJ1rlw37DzDVxjOk4dL0MgPzZoBmUczy1ncJ2vF6PpypjYDWxuZMUIKG2SnPef8ehdZG6552fbTHlYQm9rM1g7zPXtW2Hi07Apo5/FjBk05tuRl6m8A0aLFxejhJ/l1y3tNNPGMilCBrM3b6GJDeH9v/i9yrdmUaqqb8Oa6Fwp6uRerwBSaWl4rK+Vgbceo+vK4VjT6dYret7TyLdBf9d8m85X07GOUhTk3B9rvcBS+h6Yltw5fqejFiROmwRy7WsAYBytE0ITyfS8DdnJsbui/U6uVuZKim62Jfp6Wqexy2FRy0rPkIswOvZ/f8N74/rAwMUUj6R0wUMqg4lW2zUO/wAgcx6y7zkmVXkB4Jj/uSH1PU2xnjQleh0QjtsDGGuFv/0A2OezwfuFsMwomdDD+pZ9w6NNGyk4p3EHBwcHBwcHBwcHB4dhwohoOLyDL0iV/qJ5FATroieiUkIhBMQQg3ppKkgrsFwcvCE445a234OfgrDTnVUC4zjOEsGBNeaYnYm9jImmKkNaEr9mAlBxsKdKP0RGRYn9B9lBit0+rqNFY0ey3dS0ptI3tp4p8L7NlBNhk/cXuGZLpyPBj+paKhkXyDEoZYVFuwqL+rYSdbq1ghwyLWhSeJtEg0G9I2TzLZPYtvdhbYeYvCOy+ystxlM305U2kqSKHT8IFNNYxRPyH1cRa/0qUFPSMn5PTG6QLRppn7KblvUKOQwDAFMEZ3M6QJp21mMJtxBG68bI5E2bGLD8TjIhnyf+ppkcRFdKcgOuU6b1u98QOP8R8v/5yV40hp3+t+GKSG4wdmsK9Bjsh8HI2FJ4xk+Bv021z/Q/ji4ufYiQc6Zuk/AsybqSlMoYp3jpG40tmIhApfOrEG3pHY5nTFIS+lrZSPLj7OMrljYAUGMpB4FbxRXCmP3fp4oaCsxmBwBkSs1il9HQcJ8IED2wMzLRmMrBtRj1npSebRz408tY1N4hshG/ZiTAqj3Ltx6GmLmfKoalTd9sT5WfFQwQoP7IPif8rmrlILU1gzXhY2cHyyMlMJRo4Vvt0TpNGgzB3ymA9omQj/+SjhWVbMDnotSYdjcWHcm0WvGBioM08VJKYPHz8Rn4daMF43YgJUpfpWCTNoW5/OMX6NQRl9GvDqJaARb9M1F5baQJKpsov02Vhiy8HrDJQ1R9Vv36R5iw56fpHPfbQiekCM6TRiEuDLHDvws2Mg3HiGw4/KeJG9rb/oRE6eM2GoH87v0W5bfgvCGVRwhheMVtZhLO/8nrKP89P5M8z4O/SAf9q0m1CZhFFS88pdSRO9nJW7SNN6p6nvhrGaO2Z4deHjh6l0G0K5WymkiEXzcdt1dNgj1vQRz4JTpWqmjtzF4rR3nYE6CnbEX65YmKNwRW/Atd9rrhP9fOiRZLTmEbtakKu3j5de0YKd8mUwExcz8zSPFi0uY41zFPSvqZxZ3SORyj3Nc6TQyk9K1RT7GTCAGURqvroSjocRs24ZnFOH/LSq+J/rz8m3SO83zhz0Mqq23KZJzFlVmFfpdl4FFl8nHUDqrsnm6nOhL1hLnGfIVhxZJBiEEMUhozFDZjsVXk4ejmUgaZ3ZIgzODTKFkobsqwqLat6OqfvJf63E0HTMKxd6povS3iCyWFXynzAxHpOyL0znU60GYsbE4q60DJbD4oNS8CbDO/Ju9H1s2ilvu5jsdSjRG8tIbuTjbDG4+nzOIjfUMqwWNNtd9ioGJzobLOMMxYKHveNnEPbHIP/lbMpiN9E3V5lCJDKCgmK78aNfdsAW0W1LvCsAdy2ew2zJtxNl3cZDezueNxUNY1K5VmYeINTa4N+lvwmF+vmXfDC95F95mI53o+sh2P03/DP1/6dQDAEX89J/W9cRC8ERrq/Zmc3mhwe5FMsiI8812TYlZ0E8aRrmFH5uZ+xQQHNuOXPSYc+cX451R6IbV5oInHIlT785nJ8pH/B3HiD/U9AMx9AHBAevbNZrFP7NhLIulmkNsgm94q0zJZG4RcSRsu7o/jd909MifISp/V7nVJzN9sekY8h5GDM6lycHBwcHBwcHBwcNhgGBmncTYPGgIGLiO1bImpJwGIeY2dcZOCNRtxmg4xOz2Hv5YsjZtjSeKVlIN35P1rIEYrhzxhSRHDPPhCQK4mqkodL8Pml2ZpI0tpqgNGwrCrUhUXuyD7VgbSacmVlzW0pCmkRjv956dUmeuQUkny2QmTJTvSj0adlnXIF/5I6W1q4l0osraOHZIz2gk+J5TTOPyqkVz6xntdvvk3OnicohmLY69UaWrA/A8krhsAoGd1uvQaIt58a1MyW0smPRdRh1YvC/n6PXS8BdEIsrnDsnP3xuSLfp2+qMxJn+8wkdB1hGGmCs1CHEUqeU1zWxuAUBJgbeZUr0KuIydtLR3V9LiwpHfWuwnRNaJeMSaCYQdg6Ws6TtE5JVn9NPVzczOesLYhQFRgnw+Z4DTN00rDjqrH3bUcNx9AEuyj7wyaVxERgqLFbJm7Qd9biwEAeesuER5D7OfwO++crGkjjeO3AAqlwL26vVrmNr4yT/LCGi0AyOS12YjspzGHpa9U0GgMk1aY+hnS6NH7UeUJkxb4dSvuhDL5KvcYrQiTZjz9W2D+cSq/kNliqSswntAlW4OrxvLaILBiIZ2ytdMA5K/PBg45G0OBHFhjtB3aLJHNArNGI1hg7fSg6R885npZ3Tflqyr20g4q+FC132gyeY4b7I7EYxEz9jXp+H1oEpOsiUeSAoff8lzqe5qCHabfAbQJk2o3i06iiNQzbnjLMgtMiKk7RPPX1M2WVpvNoIrq/VtzmWSTvVyb7pMRbVlxtKUlZDPYrImNM+cwOjdzX8hbyOpDHEl0+rX/pmvZ034Z1LokRRONcdr4ZgAgOZp62Hwtk4eYrN4nzyEHnWX6t6qrfOnPQKeKb7WJIqHmtU+l36znUpfs3cFGZlHlNBwODg4ODg4ODg4ODhsOQqY1Ph0CZC9J9hLb+cXge++hXe1ZDy0zUsFhsokGyIYQAES2aKQT4ai5TeA/fwvdYzvXhaMUy7qlbVDSo9IYyJUkORMTFB1frQz5FEmvBduc+pazrZaIWZRyfJ21BYPrIJRjqHzwB5Rs//PVs3shn1fBE/c9P3EdL9mKvt8FTxhJbZwEWIbsLoWX0dJ0O5CcvP40KsOHfxm8r9pvnN0sKuNohN2qkdooyYm2Oa30Qd78Oco/YXDDi7aivFr5ECVF9b+JBjp3zu1Dul9KaaSOntE+AABqg+i7mBxFO65YnDxPDuBl+05wO9XB2iReOYE0cbOvedY8n/sbU+YWRhlplLZ/t5yX4/o7p7dtnUPRlA2E7idi7KxE9fN/+3HK6sivJ0rfCmnGmkZkCiz9939EtJTe6TfxBSsa9sTEZeJ2euHjb8doaprTVEvlGC2Ub4Ks9EHeqyiXD/pqw3tZW9poTLT7LgDTf2uDkA98FwDgHfGtVlXTeP1oei8zf7Uk2u8VhPACZBIAqL1wVG4uQ3UAq84hZ9IJP6AAfcGIxHH00KFp0a9DLiOnW01NzY7k5W6sPIcotSdem4wqlaNwy94V8BQNq64nj6l2cDcYrZ8Odqijled1ZGm8ci+l3v10lVfN9DlNpGB9Xx6XcyVjJ6+0xno+KnZpylwvRXC8a3eh/n/KfcNDnKA1snqcGj7Icq8WN2ufnhbwf0b+dd6J/2Pyiel/0SWW5XvFVPWZPAYupXerA4labTDgXM5QkcM5Qj2RN1iBHEHtCyC/o+oVhwBIF/hPa5iHyTfC52jxMxc0fqZNAWz7KQG0dulW890K5fOhorbLNS/r+cKbe/SwlHdDY+lJ6YlfhhNTbuhtnWgY4TQcDg4ODg4ODg4ODg4bDCPjw8F2imkZZyyc9RBJSV47cQpmXqOkObyzHwboQEm9KyL2rYnuZxt2wApMFbSNFyIHmQnaF0P6EJN3pGOWNsk6vn82UQmfdTfZHmtmi3y7la9lt51RUgFVD1EaYyRlB32FrinaQ9m/Gjd/gRgxjnsouYbDj7OMjLMZj7EBD0ul/PVLgMlBqbVmcbIDi8Xmz3a4lk2qZtNQeTx+LTBu09h6NML575seOZcssGE8VjxNmqtNwnn6dUPL2SR/IYSRcbFE1qLkXLWIpFdpZCSyW/lcdM2IBFDSz6hXsfn3/kLHHDRr+TPa/0j2KrYeSNOO9QMsG3ktneVzwmhT7KCDvjpXs6XOnCadArb/IWLv6hgmDUcaa+BGbYQ1u97HKIiWfPtROt81A+hMyYwD4IKrFd11/2pI1e61poM1nF5G+25oab70jV8Df6eBNcAMsoWWi5VklSmzMwVg0KIvBshunO/l/mef43bNPh0LbwGmRW3dW2Gzy66n+/tXGYkm+04oBiYpPNN2+d17GX0slbRTrn4Z4y69Q+cHAHIlaRhQ6YXYTPlH2fbi7C/C1LptYy0NtJIic173fw/jL78zXQWVPb8odsG/7ctU9CO+psphxryIZk1kDEuXgpS+8fXbmaTzF8+ndnXh40sgRBPbfXuuYx+eGUF2P+nXjO9hChQzw2w8EdHEDiPybUZzlBDiOPIXlD1vG01/u/Lr4Xle+obSVlrjIPsQaZpbH8XzaMyVT1xLyZYTGyB2OQngYIzMZDnYDbBWlK0lhIAcUD4Sam0kLMbM3Bk3p6of5R0KdvlOsVD1w+lM42yWoFHmRut/7ge5dsPSuTmPnRwIuQN4UfXDfxMNx8aGETGp8l9S5jsxUUqbLbjsotmUovJPtIDWA3SDexrl2+z5UkrI+75N+R/+X03vtfHb3WmiPf7u5Q3zbvQ8qzT66Df7k5PsCfcsxztFNJq1wANHUKfd595kNKIA4L9Kg4WYtL1ZnOpIvDave6hJ1cpmYLE2CT/fhwaMU+9TZj7aBK1g0f8xP2bGeoZnrulFc8jkCMDCY8kMb5u/JKtjzxn0TkZ9c3icHa9fQM8/+d7hj8UAAK9+kPLf/I+1FikN3onZWCSGgfQh77wcAODxpjbuvoSbtti++NSvKP89kjnlPn0wtY3tbl4Wb4ZgnqZ+Y+hjtblDLbgxiqRvllf4mg3LnIIXeR2TY9LFw39AmUDteKK1YAnGrQk4PrNTvu9HY8OIDPq+ShuM9kseoXM6NoW9abRMetiU0zMbDu0Mqhw6zQYkC/9HNPlnznsocR2fVd9xm9jvaL9Tq1wAYuml61XccghtUo6+Y0nwmn2vzjLmO/q1qPDDeudPHEv5z78z2Vhzw640Tp2UcGwI1j1mrlTxJbxQjCrbzC92PvKtd6XfW9QZWC4kUg5vl08mKi8AXK/qmHb8azTvh831hoJG6wNpxZJJbFJ1C9HMpo0zZpfD/ha3H0Lzz6F/Gb45387/leMp/9m31mPvicNPdqJv+MlhiCckpYRc/gwAwJu8fdN0QHohn5Q+5CqKH+WxEOFfHMtOfndNqiZf70yqHBwcHBwcHBwcHBz+j2BENBwODg4ODg4ODg4ODoRlHxo+t4ChYPJ160f0eU7D4eDg4ODg4ODg4OCwweA2HA4ODg4ODg4ODg4OGwwjw1Ll4ODg4ODg4ODg4AAgnp/i/zKchsPBwcHBwcHBwcHBYYPBbTgcHBwcHBwcHBwcHDYYnEmVg4ODg4ODg4ODwwhiKAGF/53hNBwODg4ODg4ODg4ODhsMTsPh4ODg4ODg4ODgMILY2DQcI7LhkOveAACIfHvjNNLXx0JEFS92uPvuc7cEAIz+zouBa0BMDEO/DvhVOi6rMO5tYwE0+NDSR/8lewAA2r/1esPyRm7rXU7l8zLN06l6ch2llLrRSb8eyYPPQfKvNNQGXo5+B9aY54+eTsn6lkO0jaPreQouc/HO0wAAFz6yCCJboGtt45PXcc1r6rkZyCWP0XGOvqkYuzmlKa8Hfvc5OvcfP6BzvcsgRm9Kx92LdZlEoYPSqTLoutJLoB9+N9Knb0kJ6TeTB/wapcvkVDqrLZR7VNlmJ6tf3wp+eGQgiGtj9jc051R5K32QN3ySzn34mlCdvEB71vXTbVJG0nGduW3I9Ush774CAOCd+ItE9QMAf/FDlM/Y2eY96sJzv5OI9g9zzi6zvO87dG6fz1Cy/jUqeR1on6COpTlXK9PhWupbYvwcc52RyZs0C2+lOh56eaL6yfVLKN9MPuYdB/+P3BvzTRYeOxkAsPVv1FhQ7affbAlQbU6/Ny+DZu/NtB31v6xj6cep30y9sS9R/YDkY03kPmussXHzAZMAAEffuTRYPrqLzgTauqpHXY2r/B4i9wKQPv5+1BQAwJ73hNpbs7Im6ovNJ2w7nfzN6ZT++B+oGzP6/qRjcjRGrumb8n8/DwDw/uOXCWoH+C/fTuk32blp+ZPW3b/6eFU/6o/ItdFvvt2Mm9xeyuvNdZ4HqgNAvawyU9+pNEb9X4W87jTK4oy/JqofQGMUAMg1r0CM2lSVq0jneqifYnAdxKRtTRkAIFcCKtTP5Nv/oHqN3wq1X1AZsmf9KVJ//cwWfbwZ/N+fCQDwTrkxUXruh4hZr+j5WngIj5tUTu4LZmy4eu+pAIBPPLgsmJXV51qtjRqdk1LiqaOpn+94R4p+qMZTQMTMsVHYzwuPu4CEfPb3dDhIazGxwwfp/9ogUFBB8LiOwgN63qbjzqn8AGu+4P6nfv0a5Gt3AQC8nT6SuI4OIwdnUuXg4ODg4ODg4ODgsMEwMiZVSpoi/RrQv5rOKemnfEtJMCZtb9IFpLpqp+5RUWVtEJ2Hf8TkBwB9K1WeEyHXLaL0o6bpPATnmy1ZhQpLjlX+5QGUDj01fR1ZOlZer6VzooOko1oqKjzICu3s5ZpX6dTEbeCvfJ6uj5lp0ilpjzd+DqWXRuIfljCItnFAaWzwXH6WkWCrsl34+BKTh9I0iBQaDpbaiPwoYLN91LngM7zSGODjvw2WpX0CZG2Qrk/aTtXHx2U7k9Tiq48vC+QBRCV7QnhAJmZ/zFIXJTlnzY30fVx94FYAgE88UUtWv/J69bAMoLQv5vlcnqjkLFaalmsDDvksABgp9s/eaHhPUHIloulESJrdMQnY87QGFWkMre2pVyzJtHpGuZt+8x3mW3BbK3drbZbkd/7KX4GaknJzO2i32lO9Eny4lwMK9H3E5O3Ns1XbMJoCNSwtvDXyHVpB3vAJOvjQzyFXPEfHY2YBAKpXHQMAyH3yBiO9VVowZEtaW6YlbfUK5v7iUSqa0s7KHI0hQnhm/NESzDj5jWhwDEBkMeXKh1LVjwpRVz++eVch7UlSCa+slbGyHGzbQSllsE50TaVn6Xiz/CUwsW0IZgPcnmplyCxJxcManVZ1DEhY1ymNScw3iq8jJ8/EnldndJ4yX0IqsKbSr5m6cl/j8vQugxy1SfC+8nrz/Quj6fevlwAFpbEIp5cSyKjvxJr+0tjIexCFTkgZtEDQGp/qADB1i3T1AyAX/53ymb4XoLQdENTvtIZ91FTo98haF0jSzADwZh1AZ+oVyGpwPNEahnxHZB6SsPqsPV+G0hlNVg3oX5+6jgBo3FD9UOeHTOD/MMJtTkpgSX+8JjbQ52Lzaj0nCSFQTKcQVTda1hZqnaXXWJN3VEly0dtEzBwGgdrdpI3Pnf3nQHqZyUfeIQBAWUYEM29Q1kwO8rFb6PjfRMOxkVlUjcyGwwz8AAqjgtfGk3kUMjkz+efVImNwLVAaF8xM+sDLj9Dx/qrT8mJbCIg2ZcahzDIAaS0M+JQ1AHmhVyAE8NrjdHxA67qZ28zAJfJB1aBUGw6RyRuzsglb07VyjzE94QVatgRU+1RRw6Y8wlLXqklL+tZkZbVgzs96F/Q3ozcAqdq7Z6tUw+VSeft+dDCXEhHzHQBt6tVH6iglZIx5XND8SC369GRdDydGe9rWzeUudJj3oxY76SGBV+8HAEz56SIAgH+DMks46adDzDOEFQvpd/qeKYoVp05X77rG7aViFin2hof7CufRNg5YrszstJmQ1Q7rqt/xhGRNzDoPL2u1zxA23wd45o9Ja0aYf5R6vAd0zTTPBbB+EZV1rF8zfYPHI+EBMlwOAbxyDx3u/BGTr74cXVSkNemQy5+m9HoDlgD8nSDNIo3HuEbvEgCq/ZD2og4A6hVs18V9yFdXPPN/eKPbAA3r7dcwdnJnojwCUAIEDKwx5pghk0op68asNDTm2BBCQE7fhguqkjdW7jcyPWuKMZu0TmOXiU1E/Lo1B6nvwBuDYpddKPoZWGuERNyHttofvXeRyVhnaEFN/Zf7ZEZfC5uMQWRMG/JCC8hcCfK1f6aqHwCgawb9Dq4FOiYGLsnnb6PHbnesEWSx0E9C90/J7bleRXa0WguwAM8SlslVZF7Npr1BWCaq3LZVXSW/e78GdE2MubcZ1DdZeDPElkcEy8uIWYzHQQiBwzcJmk/zpoWvv1NM22py+pv0psczQuI3aSMJteFIg9p6EmpF3oqXGR5/hsrgO8/DYYPBOY07ODg4ODg4ODg4jCQ2MhXHyG44/LqRLGpJp3FAlMrcSkuV80YypiXZ2SJw1JdMfoCRhglhHN9YPVwbhHyBpKRi1v4q3/aIKYIWsmeLwP6fHnodcyVL6qakA2xa5dcge96i46VP0W8mBzFzAZXhyV/RuR2Oh2DpUAhSyohkNVZl7NeNkyBLu2pkpiVybUD7pNRVk8upzGLTPaAl2uxUmWXpVD1WQmO0MUaLsdv4Bs5rfi0qqZW++UbK3EzmSvHaHpV+l0b5NwJLFmHaoFTtVTSTHMdACA9y1CR1rNqB0mxUvnMg8ufema5scfk3IWFoCH4/1X7It5R0fVMiSZD9pDIXnmfeP7/02qA5p8whxKTtIY/4qrqunMGXPm7y1OZK6r5MMWoC5FejWpcM578tmY6lgNh0d/OPMn/ifjj2m4+Z/HXfUL9xmh8vi7u/Qs7AB97+keizEpgytMLKb38KADDpupOT38TPyBbNeBpxLo6SEES0uQBJu63jQPoWiCVLCDuUCoFqOaFJow0ua74D/qqX6BSbl9YtKX4TzUagnKOD0l1bgxl2Go9zJNf5NHpW54TEVQNgJN+ZHCJ6Zp4/YjTDYtQmxoQ4pyTytUHUKkHzPqNxNBpnM1YK0kSrY3qUPYdwOqONrPeTKXAqqxye6yZuY+ZqfsdbHkr/+zXjIG6ZMpr1gZpL+lej3kfjSZbHIdtsZ9xskx8AZPKxZAAa4XGoNgj0rUtTO9P3XrgfspvMgsUuyhw7o+ZDP2kb9bHz8UcET3JdLM3MULUAUvpo2+/49Dcy0U5xlGn/Wx5J59gUNqEWBwDWL6H3FDZAjDdHTa8xriwnU/Gh2iU4bFg4p3EHBwcHBwcHBwcHhw2GkTepYv8MW7IAkAREHyv7yod+DLH32XRO2xJLiInKHpcl0nlllyxlREonVz4PsRXZdWsnx8F1Eeo/TfkGATF+q9TVsiXrYPtg1rxou2kYidVYsi8X4+ZADqylc11TTRq2U9U25qxRqBmHw5qlLZJhbY9HElDAvCdlUy+zVv5pELDnZqlpSLMA2xbXoqtTToOiazN9/6hGghFbEmtJeTQlLH9vWzJYUQ7PmTH6VL6YXPICAFBOmDZYsyF7mQgguZ2vmHNI7Pn8uXfil/uQxPXD95PER/r1VDSnUvoQ03ZLnF5DS47bIabMVwVVGhimVB5YAxHytZLVAaBCfkWCHVMzOeBN5fS87QfoGpdJeOTjAWj6VNmz2Phs+ZaDbDnkrFlSZcwUIJeS7biYslOy+lmaqPD7ZCdbVPuBmuoLfSSFLf/oQ3pMKJ59E6UrjsYBNz/f8pFx0nBzTVr+RSJSronfHYJtvOrP67+wKzo+eD7luf0H1APNs9hPQdaN5i5I1wmgvB673byILr16B12aqGhK28drsgpGUNJo0c76QU2jRq2CST9o/Q4jYD+Nnrcg2N9NS1TZCbpmSbSjfT0g0WZpNNNoswYzxjEeiP+WTaWs9jtOgrwhQzCOxir/PkWsUhxl3ieP4b3LId/4G923NZEgiBn7onOm8l0If1/pG00ja/yksKTulvSfx3DurzmVRyaP7OQZ6eoH6O+GXFtEw6G/l5c1c8i936Qku39Cz3s8lyCTQ+awM4J5aJ+TLKDIWMDEMH4d8Pi9qrXAS7dBbH6grlOg/vUqsNsH01WQfdR2PcH4kVaMRkCXkZ/hGW2qCGsbpUTlLSKSKWgygjCddnCeNXTHVl4h+vQAlrycqFqNoNtpyDdvzdlbYMwVC4NlkfXIOkAIgfGf/lZs3lL65lsEwgOEqMRbIH/8BUmr8y+Bjcyiymk4HBwcHBwcHBwcHBw2HEYm8B9LmjK5aLAoyzeD2Ui0FGrXU2PsigXkyhfoaIpiSbAlXyz1YNv7UdO074LWcNTK2pdCKA2H7VtbNJrEAAAgAElEQVQg17xC1+Io2RrAlhTJXmXPyZI4mwmDJUvjtjDnmKVjxj6cGeRzf1Dv4OOqXNFAQjpfWxoXZ9cv1LsuMO2cB5/9MTjoUpI6KsmnsG2A+ZpmRbHPKYmIl4XonBI8JwVufpPu2d5mMVN56HT8vgISG0MXqMuhpNd2/g8spnc2I2kFtcQsKh1izYYs9+qAhS2hNAJhymIAOOXetwL/337oVBz21+VJS0o+HLX0jBzmvWZ0EC7dJ/ld59sDvh4AgJ63IKYqjYiWqAmIWfsFH6DbetXy/1C+Q51TzbP4tzYAeR1R2YpTrw+UA0BqLQ7TSQtY0kDW+mlpad6IlnrI5rfwsR9h3dePAwAUOyfrMsrXiaVK2y3zcywmo0b2xwZRSaXOZ/kzlCKWXacBlI1/51fuMBS+6lzlWyTBzX38Z4YliMvXv8qwBbEfQNtY/S3EjH1VXmZcka/fTYfT96LyyrqRIts0rhzwkRn3WCKfKxm/ujRgKefEbY1mQVPJqmuZXKIgZAAgX3yYzu3z2cZpeN6J83VphTeepN/3JExvsbXJkN+FHCSNtw7cChhNR74DmK4eUlmvz2mpM/crzt/LmPlskmJCk9Jov+z3F5aY24xz2SG8E86nd4Vh3OL3zeUsjgYGlXZ6NrU/2b0YYgJZGUjWhHpZoF9ZArAWIWdZNmg2QV5r5IkdC9DtVczaPzhXwhojsiWTf1JI9c7bJxjmMJ5/+1aZ+oV9KQPPN32obynR1hdifD7CAWAhMrF0tI1Y5YTwUFtL80saKbNcRu1ajw2AoZ5XZRpz5fOQ696ka6OUlYbFdBbQDP79N/S7zXGR8ulje/7lNhlHiR+H50hLi21TaqveJbhI4xsCtmNvmPtfLUTh18wkyOpmEdPwvAxw34/p+ISf0K9N4cmDNN/bljUDGg9yuTbNIR1QPdMB8NA1dDgztJhKAuFB8ODKiwFrM6NNinixmCtB2IMkAGSLWHb11wAAU3dTcQVEzKey308i+kqrcT9wHf1uf1KC+/j2KD89D9jNHUhldLBt1tFsWtyY6L8mmYnTYr7fUMjG+dbWJlii0KH5yEV7c0fRgStpUG276JGYjILltDcbSR3l5O/OpXSfvr15oe17eEGeKxm66IzlpAoEI2Yr4gYxZScryraZ3Cs/OhEAUDiPFqY6T5ExpoVsniXrZpGkIyF3mI0G00dr/nwf8rfn0LmEEY7tSVhPYsqUJFA2XlxOJlMt2bMYY772YPA9AFh0EW34Z924FDaSRisXQjR1+Kzc8EUAQPGyoxPUjh+u6tUxCWE66PxniG4UtQG94JJLyJG/+6oz0HUl0YfqTbBfhVxLdMHalMp+lB4D2Qm5bjYTXI5qv25Duo5siih9SBUtmE32EkGbqcU45tsLEk0l3iImSKb1VGdvNDSVuW0O2wy51jFJWj1Tg016/aplbmX6kp5fLPO5y2+gmDMXnRsUOEkpIaZSNPMwrbh9DkDDeCcA8MC1NwMA9kkToorn3c4ppq1w29Eb05qmt9XvutwDDCgHbh5rSmPw1pXUV6Zf+6FIHZENuyHLgNkagFjTI12ufDv6f3UJAKBj19OT1Y/JLTh+Fv1HP2odInuXW/OEmdPixvgXXiXTtz2bkVHErQMSou8N6uddLdIFnstmcc3SeFkdQ+zi+bThuPCxxfFpP/jfKZ6OCNGFTYYROzbsf266/B1GFM6kysHBwcHBwcHBwcFhg2FkAv8Fdqkh6bd2iitDPkiaC3Hwl6OZWFEo5V6n6mM7j1jK2HoVcnCdSqYcw3uXabUsO6MGpFu7BSUoaSHZudY24QAgB1Zbkb2tqMqsvckabcvkI45TdWJHPqa/zBjtgE2Bq6kCQxSINmwtw+yd01eMJdz5dktKFo1oGj4OBgqELrts4AhmS9jY4TU+mqkHqVW3UTrO1NrKJMHTYDQb/lJSN3sNHJpLX7y7cR5NCsfXbj5gEo65q7GZlTjpJ03LGQvLWTYi5eM2ZlNpsoapfxVEjGlYbsf9gydsTSRLM22TJh2ozgRxM8HvdjRlY8zdO139rGBgYQTGIR4LeFwZM9M4gQojzb97MZVlVsLHp6XKXfosOYrObJiiCfya0U7xO1Z9VK5fqs02xYwFAIDedYNR6aaXs8YkEyANUKZL/A25Twx2G+0IjzG5EuSKZ+l4k10jxRTFNDJVlfVzJFEX8z8cvaYdkAXk26Q9FJs1bidSSmDBaekKkGGzkcaEAAHscEy6/Fnb3kwz42WjJmP2N9eUznUUPKZ3jylvOKBjIwfcBlpiKWX6IKqAMdm0y8KOxGsX0aWOSZo+l02Y/WtPhTg5OrZN++IPdHmAUL9iMzTVNuVgtzHrYZPOtvF6LaDHIe432SLaPnZFuvoF+kboOykiGJFvh7zjcjo+7BJ10QRjNFZrAjvtmm4UCJtZtSIdGTUvRYBYRhMNXxyBxIWP0bfsv3AXtF2s+qblPM5BA8W2709YgKDTuBAe/NXk/K7N0u3UbKI6YW7C/N9dbGQWVU7D4eDg4ODg4ODg4OCw4TCitLhxdvi2lGz5LTcAACYfooKJDawF2tk/g28QFrVq3ENC0pxswTjfKRpZ0TlFS1jCGhchBDm3DhW2D0fIgVsUx1jS84p5vnbwM86ab95EzlUzjlFSl2yTUDZxgo0Wtp7lB4j6s3jIZU3TBRCwk22s2WCYd+oZak6ticloqVw4j3h/jQY+HKyV8E074vL5IaFTS3itfTgCyZVmw1/yBLyp8yLX5S8/QiX5xO9TFoRw9J1LIcvGMRQI2VzfRv1EnHTtkPI3dvky9Fs3kleWnmVLFiGDkgpmCsCcBepWP5Ae9bpFFSrMfcqHxAT1lEbKyHXltu5lgU2j73U4EPAv4jKyEyiXJ5MfFglUM5+cTHYIMh/uQ34V+t2yI+fj1BauO/9CnHL3okD6aVe/aGlFWcJeMn5v4T7tZSDZQZellKte0IEiAxojpZ0K+28J4UEOQcOBSY2pyY3/mNSkAjJMu2qVRQgB+ZzyAZoV0sg1BJc/oU/YIkUPPfvgZOlZw2bRgZpH2xTr2eh9mtyCNR0eunhaUVpKyX6NsGiZbc24CNvGm3RSspWA8T0qDsU1jmmo6xVLY6rKMKBIBoqjta+jf4vySfvodWY85/5Z6QVevJ+OmebeJg8Jz4/5zqiWttxr5jB+d/zrV4E3ydcJ252QqHqyh5y8xbjZVrDGYLnl3d8BepQD+XpKj/YJ1rdTqSVQPOnSRM8FVHvXFh4Jx5D9/zNx/hqtfKMUwqQxbRc/CqhAztLyYVnzYwqkOv6qZBqOOM2ipzQbceNq9SZazxS2HUKQw3cDG5mKY2RZquLAk7tfw+TvK0563jT0r9LxAGS3YkGYMFezQEFF8A5MLH7IlKjSZ9TjyhFNFKaYSVvH/mDWjC7IfhoghtwUNHuQqjerJb0MxQBR9dXP6yYHK+1clu/E9FM+Gcgy1oRJm8QIc95eUOiFSWjwhkD+wKipQkuwE2q+3eQVo663vweVXUKuW0SX2cEuU8AHtwiZ11nOjIaxymb4EaZuAEQmr1msZJ9qE7x4yhZw4OyUjpwWA0gaeFPnxTuSH59SRR+CEJ6ejO88lCJuH3i7MbESB543hDzVO+f2BxgWFV5wDKwzDGscE2fpE4bDnu+VvmaTCbZFAF7eImlQZlzP3gQx9710jheymZy5JxzjRPqxavNm0CY205ubD5hJSv3mSqYt3folunLEpXj/rtHYLGkRT3ZA33faPkMwc2DH+1rFODePnkbnZpOT9yl/OtpaBBkmQHOsFl61QVS//z4AQO6cP6nkKo2XCzLsARCb7BpvNhZamJh2Vof8HcVPEKfelLiKYtJ2Da/p8aI2aJkXqW+35hWzIGETQS+X2qk7zpm7KYtVwoWZBsfGiIn9Yy+UYyO46zGe+7A0JD5xJqYWqxz9WCardr7aHFctlq088tn0s6GOvdQxKeLcr7+v8My1BRRzS65+2ZBVsInUmteAtuC7CpBAqPZssz5p2EI+Zqzkd8hj32A3kJR9UEELMqVvxboKxWOpV4EjKDaEXKni0Tz4YwgWqprcIGKEVmFIe5zlBbfNOKbWHiJXCt0nIcbOTlCrEHh9lm++VIws/q04TDq+yoLzMHbb7dOXoQHsMYYemUFuzvxhy99h+OFMqhwcHBwcHBwcHBwcNhhGyKQq6IxIx2p3qrm0rYiozJs9draRHD1K5lY4/DLglfvoepyzrhekZ5SFTm3GoXf92SLE+DnBEi5XjkhTdgQWP0onOS5GGmRypsxhB25bMshSp0qvkeJqcyOB/nvJDKfj4HDkTCPpD1DThhzHAufiovAufiZ11eRLxHHt7XKqLkMSda4QAlDRqZn6UPo1PLGcyjdLRCVv+plx5igBzm71HjtIA2A4wn28tYK+e2IDuSZxOFqBNRuaMCDfDqiIwOhKQQfaAAf8mTQ4g1+nNln84v3GQa6JNDiMYLRmdtxWUkGWPnZMIikjYByPN9sbWB/SIgGQz95C13f5GJ2wmwP3bTbN2OZYyJUUkVbHuBFZbaKDXqW9UZpLAJCvqTgY44L9tRGGFH0dqp1x9OGjvkEnvSx+eB9Re35xSLlaecPWxJqX9OT/Uv3mn5UiQx4nCqNNBAfOW41rcTTVADQtsjY9zBZw4y3ktP/hz8ZMB3FxfZog7LQsvAxe/j05gM9JQ6naRMuox/ZsIRD/CKA5Q2qTODMXyDefS/HwBs9tFp9j2UspMzMkKIxmpndGmutHNVdeFmvKXMaww3eMyZYNZR4pskXTZryouWxvJa19KoCqikOUyVvmwkybbCKoa4sCNr0rjTHts6ok9l3TseZWMhcce5DSDsStJwKaIEMbTPUyUc0jJlD5DpTv+RU9/sAL09VTeKYsbNrFVL8HfcFoEzfbS/3urZ3KNcmFX8PSs0g6P+XqRYE6BR5lf0sRviYMBXhMW6p8j4gNChc/m7xuiU2qom1GK5v2/Rwd1Aax6p8U/yukyx4SdB0tE8vyMzTnlk4chgeMADa2OBxOw+Hg4ODg4ODg4ODgsMEwQoH/2M4flu2qCgZmU3GGHWNzJb2L9Q4nZyA52AP/qTsp3V5KLKick1Aaq20pme4S9aqROqjdPwXjCkqCPBW1XPauAJ4jqSP2HkIQmXrVCurHgYyUzXWhE/LVu6gMMxfQ81a/qG0r5VvK/nyro9B21vWBbJtJqYTwItIOeNmATwQQtHXEvp9OUyvCpC0pn0of5Lo3KC8O5mVLQvib5qyIxNrG1lD5HnmQyo+pPNkPom28keKys6pfN1Kkikpv+5KEAhHCr2PeIVGKzqbgPPpXkzMjWkg1Y6CD1vWvgpipIuey5F5J8ML27onyVX2i+EVynFz32S0x+tIHU+fDNsZy8cNGM6IdKDkiroCYuI0+BkBawjZFh8rlH1gDTKF0WnOhNE1kVx2KZJ7NQ0zegY4D9LnqmOlW+Znrlxmfn6RgnxPhIdwpWlERGwmv8WmZ26Xs2XtX0Dnu05mc9b6CVNxxeQNRqaOUEtOmpNMgADD0vVmj7QzYszeAlL5pn/zO62V05OKl19KvG+mhPXY0QVjjAOkjX0xHxgCguVN16Fl0bL1j7bum2gLqxi8hYT1SI6njLoPnxHpVtyP97iw/gLB2WngZy45fjU3Vfpx1BI3Dpp0abb4sK2m66ueyfyXEmFnBc37daCE0wYe61rME2x2UciwFgH8qn539PmfGGPZh5LLn26IBIWuDWrOBrKGUH7PHAXQurIku9xince03VoqM3VJ6lv9LKHhlbRCFHRekqp7WpN1yng4sKQ76giqbmrfWLyWncrpqyp9T/Z7nSgBTrvi7ylj50umAoenlwnFjUf6M36XOR48vA2tiadGb38rlNtHix5/1HQDA6rPonYy98nmTJmSdAkQ1poHsY+pYOPnr6croMKIQsqlH9/BArnmFHjYUtpJwXn4Njx9NRjI7/++Kd5xfJP96BSs/TY6wE6/tSX6fcvwOqFdNptY1nujUa68OmIWZ5RC28kwqw4SrXtbnTH4hB3LhBZ/B13hQDTsNQqL8jQUAgOKlC5PXcZmKlzDKGCnFqW4jDu7St8pvynDxfDKzuuiJYBRnm5GKB3WRLUSdy4UHINx8Tf5/OoSi2B/xYExMkjgo1f5woftc2lCN/s6Lw5ovo3rlIQCA3JcfT37TMNfxonn0jsPfcLjw/LGkfJ97u98iJUFvWodJVX3ZfDLv+srjy4YlvzDuOYw2aPvdl7CNAv923/BvR9A33POeZN8QQKI6xrIeNkD1u4cBAHJn/zl5GVLAv4HifHgfvyVRetnHc5eI1EEGBEU2aUbonGVG+1+7UDs9/xHlqB0bAymGgCPA0sZzSHCOEsLDXw+hdnrwA8nbqY4wHyNgia9jFLZj8MU7EzFCXDuVMty24t6rbLz59+u47RCa1478W7I68jeMWww3M49rdP3bu9I3/Nwj6caa+Dk4eu7+w6kf7nNv8n4YV8fwkrEZQ2UYq8+kjcb4q15tmb7Vu4tD/4W7AADav72oYZp/JXSfPq51og2I0T9ePaLPcyZVDg4ODg4ODg4ODg4bDCMTaVxpNmS9QvRzgKFM+8vFlObAL0UjPMu65RjHfNl1zDuTTKm0GYEysUGhU0cthRVLI059HpaIGB5pD+M/+oVU9QOANz6yOQBgxg1LImpA5jVXD1D1UGXPdxjJkqecVoXAxB++Rvdyun7FW14aY9Swgz3WOaXN4LzqFWNGsF5JmpgWT3jaNCcNdD6jpkbiLgRpcYP7WAkv4JxIt9UxtS0oqbA1GFqdbEnHwpHl+XkAIhHJpV9PvZuW3dR2fn7kzjj1/qCkMC2klBj19SfoOExaMEzInnHzkMoFgLRf4bgjHEk+kzeaMq5/z9vEHw8Ymulnf4cLPn80HcfEQYhEuxde1MlXCGOawI7G6hv6r/wFW55+Rrr6vaZMFjc/MNoPm9GaxuUlJb78wHPqWLVzPV59ETVFJ5vZ7xQ6t+1xUVpSv6bfV4TYoTaIBf+PTQpS1NGK3G47TNpoJHWMSH3rFWwxSupjzlfdENHWJnVytvOf2DYESlWL3ryR+VPSvimlj8wmymyVNdGqLYts8R2bWUkpgWnpIhvLN8m51YuJkG5HemfKUz3O2WQAuh2Vcd4PL+ec6S+b3dnfr6zMmeo1oF05K9ctLTh/fx4HimMAAEs/MQMHfvv7qeoHAPLl2+lgq6NM3A2Ov8OmVV7WmFLGaXEYfhUXfO00VTdlFrrkMUo5fitD38rIlSDZbNuyHNDvJTwGSB+HX5CSGkLHPMmnMnuK7Yd+HWN4mFDEI9rywdLkJzUtitPuTGwSyqtxYY01QbyFQTwa9c2xuxHpiSYvUesaueZliE2U2Z4VqytJ3nafKO24V8My/SvCOY07ODg4ODg4ODg4ODgME0Y00jgA7YyrsTvTaWZ1YDl2rpXr3oBQTqPG9rKO+rMPAACyB36Z0heMs7lkOjpLWhVr8xeiVDMQwMv/oMP9klfLDtQXlUII65qqh2doEcMO7AHYTvUM1mxkjNNgILgXANQGIVmbpOwwmYJYFLtS013aZWn2PgFf1yewe4/cIzChIEP5Wb8RzYm0pONGcm2kLiwRZQmzwJh8SvckpU059f4luOkAsqd9/93Lm93RGH6Ngk4CECqgXRKJcCqwJDIVWPJeNxIkLo/d1rR0TUlR6xWjGeC21rfWCqgWF7nb0HbyfTIcoFJkTPA2zoG/addMoJCCwhEIjS+h95zSXU0IAf8NcswXW1HAQrHPmXTRDjA6oPpjnC+Vl2ugmaP/5RLSgiWl/QVgpNJCkCMsgGYSUA2/FnDG5rzmjVXHHIyRpdD1inEyjws8aiGsJTCatDrGbzIqWb1ssATUy0RoqhtR/jaGADYjCnUOvuYpWmaiD+dvyW29Gksl3jB3ISAnzEqcHgBE10z1/Bhpd1zwutDzAvCyqP+dHIKz8z7Eicx17WTNRBxW3TQ96zgzr6hgo4xJH/si8IjSpu6WPFq12ITs6THYred0HaDx+dvo3+2PR1zbldrhn526y8AY8uHQgXKnzNN5yieINl/spPhQs4UglTAA4eUgmfJXO2YzPW8NGLtp4rrZdUG9oiO7m28T9puxbotrs0LgoJnK0fyxa+jUvJNUXUoWrXC6ItrP3HTbodCzx6xdtDN7+sJU3ibLjSLPL8ofVK5cqNtsWk1jQOu66q3UZXIYOYz8hsPBwcHBwcHBwcFhY8bGZVE1whuOetVIURUFrpaq1Svahl4oCZton2T8EFgKlSkgc6DSivA5loZk8loyJrPGblquIpYgMZb8LGz7zYjUWQhg1+NSV03s1Vjy0ywokUTGuh4jRRRBG15IP6olqg0YiavSasinfw/MJWYWMWl7OnfHf1GaI7+WsFYhPKSoejlQG2BRniqJhV8DfLZt5eBKxhbc9p3Ze25HMH8tFbXs3hl+zUjHWdOSLSLMrGLyl9hm20mpqse2s177BK3ZeP0/SAIz81dLUuUlMjn4ZfoWrOEwErdaarrd2PKWqe+kGrOkkWbLl/9K92/3AbpkB+DS307lvupFE5CPNWXzToacoxhV2K563SK6Nn5LY1edsWx+Q1J+6dcimgctXR4zE3KH96epHcTozRpfHMI77/4JBa3quvIoOsHSfy+LzEf/h45Zk1irWFTQrN3xor4RMGNN+VcUxKz09ROSF0oHL8siaPduEMvgFJCYm2BoK5UyYauwzXvgfZkxLCzFlRJW/w6VRwh0ryJN3JjmtQrC8t0Ka09ix9NW6CCfQTGBfC38V6jte7MPNjS0moo7gzg6zkYaSilldExuBUvD3Rg2mxR0+kg5pI9MR1CDAOua7ss1FfTRrwM1pqVVY7CsAzk1hrN/RbsKz9a3Bii0paicynKNCh46eaeghh6A2Fr1J1nT44S05xBu4+rby95lwNuKenvHD0aeJXb+CKUbINYdke8wfkBMUWszdumgvJZ2b0lKf6qa6teZnOXPYc15ACQSMqn5dUzaTmk5WZPD4/Ocw0J1CCJJ/lL6KM1IoUVlsD9J0Wgp34mGvryS5gt2J9FteeZ+WmNva4viqMSbPb+yankgf4d/LYzshsPLADk1MGu1sbWIZBMR9Vu96ljkziHVq1mASogtj1CHfugaoo7nlT5tlqXT8SYHsKIh80JIQMzYN23NIO/6Ft393m8mvEOpD23+/2ZLRzsiO5st8AZt1FRAUp1kRU0qm++t+c3l6xxXRJl9VfuHZlJVHoiU35jDWGYOOto3mzvVzEZTc+RXUa+G6PmEWQTp/Pg+kTELINscjRfETJ/LJiP1Kvyq4ThPAtE5JXKONxo37keblxPueEM7vjeDlD5ERzQ/ABBe1lBGqgjsQwFHN093k2rnpTEQWx8dOCdG0URHEYJ5waK+QyYPueyflG7a7nQuVwLu+wEdH0386hyZHBCmjfHEH+eoDphvzMIC5UQqMjnIx8hUAu+dn6x+ObMwiot+2wh2nBc7Xde3nua76a9N8cnxA4pkiiLv/i+IQ8mpHBXTV+SNJIyod9OGLnv6b+lCvg3FL9ydoFKR0tLfV++EmEzxg7QTcFxqy0QnsmCpDWLvP1JbfOoY2lxvfw7FHxJ7nRVLN9qoPLHX61Vs9stFDcvWpNDqfmPWFUarhU+AmKCbFjqyjcoiZlFMB1kbNNGZtZmc1AIBO49G9LVCeJBM6pEUxSZmZrwxsMgbAg67WvhmYhX5fdQGzXbeoiHnDXFMzCvkVB6ybhb3q9VGgfvh7qdB3piOvAFQGw2ATJ61c7oqIUffFsIsph/7BZ3a8UQzxvP80jEFco+PqDKr9lC2SFPUmkFYJsVmjlP95WcnQnzoZ3Qqr4QmPJdmi8D270tdR6pT1trMR83hoovm+NgytZ51qnrKLJyFJ+X1sdT5WsijxyRjxhpnKl7voTaaynE3m4+cipLitIhmb6Fz+93VPaF3Uq9pIYN8+teUeKujjADHinXUzDQ5v83uicrxrwJDCrFxYOOqrYODg4ODg4ODg4PDiGKENRzW43hHzBSbvcsi0aWzx301SjMJQZGgAYjR0wLZCyGM+YKSdMr+VRBsCsJO6dmScZaLk56xShlNzDPCmNU6EmtgR55ULRmWnPh1IKMksVz2es28J67/8oUAO6Kyul9pdmSufWimg3sph0S/aqRL7IjIZYlTg3pZoL4+eM6vQWRCDsSs4fKywcjl6pyRhliRcDUda7RGXi4qnWkK1SZkx6SI9OTEe0hVm9SMQwgPvqKFFNtGzYJYs/FOApIxtaboStFOGQNrzTdjykqmLH7qd8AW+1PeLMXvnGIizXJAyeoAxKFfoWNtNpQx/+tI3EoS1rvCKitLbKVxag3Rx8p6rampYiz81lqteHMjAcQ5esZoPTgPTi+4f+13nknAkrl6BddfcycA4AMHkWNxlp1yq/0Rh/lEYG3ehK2NpJxNOAZUwLxiFziac6wJkm/oQdkMbvvTSQMq5v0H1xLyrm/Quf3PN88Oa668rJG6a82YcZ7X2rwYDWJDsBQ1Zny2KZhbRY/XmEiaN8HmoNb8I+8lrbRYoL5fZQCSJY9amxknWbXyH51SU9nMvM+iaNenLOIBWVE2cFqbL+G99/zAPcbB19IgcBC3tnFGC21HqFd1feTM4wEAu/2Bvptc8ypwyDnp6geQZhsAyr2a2MW0uxh61+3eb9XLMtcDIAfXGbOlta9T+jEz6GKlz2ipbqXxSBx+kbFkYBPQE75vzMYskycAkANrTdmSQlG9Q/oRR/uACVmEWjomiJ4QKJ32XQBA7/fIZLzzAtJ+ysF1RmPBmqnaAMAmkLofZpquKzIfuDRJrYJQlO6wtPphanMymU6WnTj6ayqPkPY5WzC05Wq+/Obum+L8hxdz6fUzm/b5/c9NVhCHdwXOadzBwcHBwcHBwcFhJLGRxeEY2X8bpMYAACAASURBVA2HXweYlo7f8z20qxdHfi3gQE7pa0aKYO+wQ45ZgY/G97L0sXOqkTaoQEYByWIoiI0QAjLOzrwVXlVUutu+P9bhkIrbQDreLJAOS3t7yQbZro9cQZShYtruAFP/FUjaKqe/x0jRlL2sljAm8EGIxYPXUT6zDoDkdxmS9sVLH4SRzDCyBaxbQd97ItvR6sB4JnigdhjNWvnGacoij/SwfiVpYTrjU0ShpMWNgqYBwMXzp+KiJ5Ymyk7MbW0TzJoNObAWopTKrRZiZgreZr6HJZ+lMab/sHSO3/U27zXvVfXJ7suPRtflD9M5fv/5diMBC0vqMx4ijqIdExHRItTLJoDX9D3VJZY+56z+nBDZ1sEV4wNleTH+CrBIEdgvKZpGn8uVTLut03uRa17FyReT1O2Sz14BALjwS5YNftr6AZb9fUlrhbhOvqI2FcWYtiTrxjeBpbnZovH12u2jdI59g7wcvv+V7wEAzrr/bDpXG9D02gE6WaV11vf61jsZZYKwJkYTDcCQAvS9ptruzAXR/Pb4NABALnuK/p+8o+knTM+bLTb3CVqmHI63PCphgZo4wbLEOqaaUkrjw2f7/v3zf+lwOgU+swOx6qCr7ARuj5ncDqRPvgIAdj1S9UPVNsX4OZD3fTdl/WD8qerVqKTcah/aWbiPtMiiNjpKJV7ogPSVnxLPz+ybUhwNDJBVgjiQSB5QGAW5nPyvRCdpn+SalyEyW5t7LIhCp+kTScHvsdAR0XrJgF9TcMxrROgg/0R+cKzZQJvyoSmNARShh3yJ5gsx+2DzfHsM0ZqwmACr9/6Erm1xWOIq6j4R128a0X03y+/+q+iWo74Rvahp2qnM5z/0BuRCRcfMfru5kkVzHBPQ+SnyjxMHb5u4TA4jByFTUX0MDbJvJT2sxW7OX3QfpWN+7VzJmiANU4lcr6JAJ1TR+8r05KVzjgEAbPmrl3T0c11Gu2P2KtXzxK0T5Q8A/uKH6J6xsxFmTjIPsWIf2M5fId51IYQxG1DmLnIxTZhi+h5m0OaBemAtwOxAatEm1y8hli/A4tdX5gnZgh60RUdyJif/VTINEZO2i5pQ8Lf169ZiwR5oQ83Mr+HxY2mhM/+WJeZezovz0FGVM1GHY1k3zzcn1Y+Ptz5GrGSb/nYASaBNDlJEjW2Gvgt2BgC0X/JYsuerCV+E1fMNUP+hinT9ub8lLpNdRz3BcvvT8TLqxqxQLxpqxpnVinXTdyFx7bdf/KjKiiPQSysmg7VI52+sIwObBZQ+p/qmECJ1HZN+w6QxUX65D5ljfvj+ZS3zgvRNP+Bz5R5j+hBTpmeOof63/V9jNjuNntejyCIswUFcpHFtemhvHiPRlz3I/yETBu+TN0XrYzKkX79uxqSFf6RL2x0LNt/SppCaACIHef2pKv8/Jq9jr1p8NtlcSCnNojhsghmC/zAttrzdPxm5FjEB9WuQL6i6zVGLM4uYJK7N+M/+nvLf9fSm5dDPZAFSk41VK2dcu9zf2pXa6ef+sTRSxsgUH9h41tEQmq0sgzsOpbn24AeapA/fvkKxSqnNQ6AsPH/0rtALa/O8nBmLFOsUSuPQ/dktAQBdV74UfI6UkflIdi825puc78Aas9HguUQxCaIwGpVvkgCncHGy2D9sKohMvmEslVZOzjY0McmdKpaE/e2tb6H/j1lLNGsvzx5N32HbNGMNr7WGYvoZg9VnzgYAjPv+K42fGfPueDNMpnmNhZsDl5HJZOnylyLX/hWx/qzJ7+rzO7/XeF7bEHBO4w4ODg4ODg4ODg4OGwwjYlKVlCeaJafaZKDcA6mlg1YeudZmE3a+lxxNErwL731S5d8ZdJYLl3EoJkeP/YryOeRCQ9UaKYxvJEs2bWEMl76WDrOEmaVAYdpfdc44Q1pqTo6HoZzlhUVhK4pDqCNznHsZE+E4TMPnZWMlOvp9W86qL6+n6zvzO7EkNlpS0+RbU5TYsObERLB+cRnVP3H82IpyWi90NE/XBHbds53pIiyzZkMqCbJ20m6UvjgEamPtkF01kjHWXLA5UqUXkmkmmdRh7WsQbeOCedSqKJ18qTpVD1xDvRKkawTI2bzOpjzqWm0wSBZAmegyiiM+l65+bLrRgvY5juI0Tjp4wlHJtZwkkQzTzpYhX1I0uJ0qHstme+tr2x51QPL8GUxf2sAJHODvIQPX5NrXISbvEMqrBhyjviFrmDivco9leqLe12A3kFHO6NtRvCK5ciGwXMU6mneySq7epV8FDvtC6iomMZsSQkTJJRogTrMRyCfw7CwuPpnICpKaT3oxxBBNoWlim5iOpdC0TmtvLEWPkAbY9Mj2d9JzIkf3ZgKWIoqZIRhCMHFLvRqtJz/fpvZmKbbwTLvj+c+vYdRONMdxvCTY40tIm0dEDsEyyzWv6nFBx+TiOadeQW7OvHT1U8+M064NJVbMEfOVGaTW1qh7q4MIUyHL7jcNoQdbMBS7TByQGC3AnH12TFwWhux+k/Lh99UqvY69FNX6AMCYs3/QMo9AXDL+1mpurP33och+QsUDU23HnnMLJ389UTkd3h04p3EHBwcHBwcHBweHEcRwmW//u+Bd23BEdv5+HXjwajo+nIJnyb4VhhbR8oeQy5+hUywpbIELH1Q2mWyH27vM7Nyn7aZSWdL4NWRfKMbOSpQ/AGAzSzrSiEoz4NNhBf5DfHoAxml+iarDFPMcloSjVgZKHGmWNBcikzPSIWWHCQ6AyNKgtHhNOcbPOqCpw1hsgCx+J0y96tdQbNTXbNpAy9kwzmnTNCMl5eDAf34NE4sppXKytdOg7fD3nd3I/vKzDy/V5+wy5s+5Nd3zFVizIcvrm/pz6CBWafK2tUlMb6vpFQ09rWBth5JmY+3rwBjVH7Svhwc8rZwYtziUTvnqnedKRtthS04tulg+Z2iuSRelHa9FBniJ/Low95hkFRxiBPdGA/+v/kh26B89L/ayutdul0FSDNkxCWKbY+mfkBYCWYEn/3AXAGD+mWkKa6iy9buy6YjVNaMNVYWp9Fp0qSYidSSAJH8jLwMo/zso7Zb8248gDlJUyOwvs8kukHcqyeXOHzXPV2XVmrEUkN1Eh8ltomE6ljK3sDH377gMAOBx2ePy0oOJxIV/J42NtOrfbHHgP3YN5b9PQo1cHEFBktus8cdud6sGG1sRRH04pA44Jm3/LR3kU71L9nPwqxisD4FNh6XyFqW5vsRazXyH7hfcDv0Vz0HoCOhW+ZiWXbVPO/CcprnvVDbxA4OQa9X8uJCCB4s9z4R85S90Trcri6Bi3rHp6sdzM6Ia8cA7l0br3gyDvdR3O60gxwCovkxewN/e8kHVfhYtAsH65cGm1+PQqv9F0tt0yyJmLG5LHqxWCE9beHD7yZ59G/5nbwqH8IkHo/5KafJ3GHmM6IYjnpGDF3kStUU0uec4yqjt2K0XQzmgvbGjswyrhesV4vAGNAuQrFchxm+l8lXmCXr+9YAWpiyxUKo/e6Bp6ixmD0JqYJZMSxLnXK7iIqBesaKjq0W85TzPUWKRKUDwgoOvs3rSr2tTtTTTiFAsNnYMgrDqtqGjo2IUkTyReFmsLqdQO9tOcnHgKNVsDicyWDZA+W/fOndCobUJlP0tz/2Hil48uE47Otuo/+gDAIDsWbclLUHwWYVO+OtoY+x1TY9clzcRN7449aYh5C7N++SNhzKhkN1vmE0qx3IpdJp0vEDI5AAVhyNioiF9q2+rd1Yrm+tsMrjkSYgJc1W+ylTLIojAnp9KWa3GC7lWzptx7HLbjE7hYBnXjutls8GorA9eEwLjOocwBMeZVbKz7ErlqDtmVoTBCtP3jDiECy8DGWm7Qj+n/wpahLV9njaWa269HuMO+jJdV+OKEB7w0esD+ZrFZg4oBBmBkiC8CW2IGDaeWBRbm0kGzGA4XgObV65fBii2rdhnDSO9pd2O4hnzwpGeJaa2ycA1M7JLs2hvVkYpzbeLMErVMSo3BJMqy0TS7OXqwd96BdqEUm88JlomZ6ovlnuAhURaghkLVK6WQCPCkpeDaJ9B6TZRM0ClD5ptiedCjtVU7gFeU8QUc49OVr+mDHOqwrVB00b5khU/xv7W4y9RAqowa6UQ5n0Nqlhat10E7K+Y4577E/12TTdrAzaFtfLPf/KaZPWykYKBKnxfXD/p/uYHAQBj/vvlZNmETStFBp94QDnrq3Wd1EI5gcpPyXSyeNnzQyv3SGMjo8XduPQ5Dg4ODg4ODg4ODg4jihHRcMRJ/vQOv9rHiZD71G/omKWfy5+B2HR3Omaq2jEzDC1jnKQpLB2o13RUZ3ZKE51TIJeSA7mYshNdqyj1aKHDRNFOg0nbRMsSQmC3HpCOhaPDZsxxnnbvgiVWuVLEPAnZon5nQqkU5aL7AaYFXEy0rHKGcixfsRBeghgRYWhTsw6Lyo0diLXCo0H9tUTJcPgfvmVIdWxJp7VpTpyEz3a4ZMlPv1Kfs/mGl8G8LVI6f7PamsuYEKLYhev2pXfyofsMzVzmtGvTPT8GrNmQioddMLECAPG+GC7zpLD7JEvMfeOYKKbtqh6i+uIfvgp8aF++mf769YCGUp207rO0kgDkmlchxhEtIjs6iglzIVe9QMfavNGS1jaj7Yyr1rJ/Ul6bvidyrRV5hYmhYcx0ttoyubYzNn+bulZp0OSD36f0e/wnJm87O3H+GtyX8m1G2s3SZNbcZnLN40bwd/XrkDd8nE4dr2ItcJ8qjELp1Cvp1NpFAICxF9+uNac8/tJv0JQMgs1xapC3folOnfiLxFUU47dMli4kAW2oYV23LFDmpm1BSquOqj13TILsVnTEo8mkQ7KE28sCvasSlVeDHaRjyA1aRU8PT6dCCPTXgnEgjAQd0TE08L+lmYy8N3UtW8DAUEyqeB61zStZYs5aL5togd9ntmTKyGZdlT5g7DRVLCa74PRFgGMY8dxQ7DLmR2yGXOmFXPOyuQ6YGDRrXgM6k1PE63o1gB5LskVE7AhitfUSYtS0xs/ivvbkjfT/5rtpCxC5zeH027cSeJlieIhdT+OCmDK1MLmKBZuNeSnnRJswxhpPR81JQcLRKF8euxSxgHz053Rtl1OR3yzZuOHw7sA5jTs4ODg4ODg4ODiMJDYyk6p3f8Nh282zEzTbHk+db6SvfyKnP5z0U+BZ5QSmtB82RNi+udBhJFHKf0BkC8AmuwTTcaTcQgfw2gN0PGv/5PWwHLEjgaTibHJZYgFLOhhOAxj/DpYCT9g6GOkZAAbXGUknS8DXLNYOn2K2qgf7pgxF0gHgmpPJJ+HUB5aZMifoL0J4kCr4k5Yc+jWMmbmJvk4HUcmklMqfx34nrM0RAuyYFsm/XsWoWcmo/DTCWpgUYM3GP99H5djxDysM3SvSO82Gwd9V6kjSo6z8k8NIZXOWvX3QWVSMng659Ak6VtJB8YErIDm4ZNcMk9/jZLsv9lXOsvY3DAVqFOO2MFJPlkwLz0gg2W+ENZJSQj5Pds1Jg3CKqfMTpWsKy/H8tsfJGfXEd5CdlpKzb9puH6P/M3k8dD85J++XJsMstfG4oHGiBWV4eGwSXgbX/vR2AMApHzK20JxGbLZXsA4xHT5Ifx0Keupl8cTPfgkA2DmFhmO4nf+x4g11vfWAFdSaWL4Q7MOhKVh5HPIgVy9OVU7uh2mXG418BI2Cufn3iUAHTiwaH4OIA75ARqT34ZCqr4uOyZGKatp3vxaMdg6Q/wx/R/YnyeTR94cfAQA63qP8ujzT1gVLve0HMc05f8/iaIiJKgI12/0rGlcxelOUryGfiGJSx38dvLQZRPT9x/g3SCnR/SVak4z+Bo29Rjsq9Jwk9jyDTg2sM5HI1XiAbAnYVY0tYXpwv4byFRStu3jpwmT1Axr4i6WDTegwuIz8L4ZA6B6FskjQVNz9q1F9i6wwkpFlO4w03v0Nh4ODg4ODg4ODg8NGhCQCkP9LGJkNh22bXwntvFniKqW269S7fkv74Z30UzpXXo/+B/4AAGg/lOhz9W4+1xYNXiR9Iy2zJK4R9gOWXpV7UXuY8s8feGHyOq4kSSWmzkvWiJiRI1uwbKFZ4mwCBHKgNDFZBe3JlUwdqyp923iTnrVEs/eFtoXn99NN9u1Yswhix5OS103hIxecofLrtwLycGBGi1GDYdupsgSKJXR+HYWtSBote5bQ7/KnKenU+aZ9MI2uHTxK1UdmixGfHfsdZmenDHTENqGD3UYbliAAmY0d/0C+RjfuNwkn3E7SFllWgZk46NQ7GGREUfkB1KuxzFgtwUxUz94IsfVRKlOmaVXSLOlDsAaQ+0x5PUTnVHVOvZOBtcAUYpiSa1+jrNonqjRZIynVUm+LAU5TxGaMhkVr3pjGtQcYNyNd/Vj6/A4kc/b3qfpWECpYttlxjHsN8wtRkCq/LBPgKyVs2/WUiKOsnt7O/jdsz6/+rQ6aZ2i/keZ+UZp2WdvZlzGuYwjTTMp+1xLa1yzKRJbkPgp2alG02nnWq6l9jSKaePuRFj1vuJzkwxGSjvt17DKOg3EqPzTut8IzzI0Mv26C1dlsRlwm3S5V/usWYXJ6pS/wxsP0O24LnRcz3ek6ZvIQ2WD7kNK3tB1Ki1HuQfsCRY1t+84AQN9KzU7INMaia7rlH2NpZ7j9sw+NDqS3GPldDxtCJZW2irWXCdtVZOyQEqO+Qkxw8jrlfzGD2LXELqeYsV5YGjd+p0xh7mUtNk9uo+pb1gZR+M8bklYpUC4AkLVybIDDtMiPUz6mTGPM9apXjIbdpsXndZIeV6xxl9ur1oLlkBuX0g/HYUQxMhsORQcpil3RKM5NJs3YCT1XwsJnyHxlV26YtuNdRB2ccuLKFrDuJVooTkxxm9jqSABqwA87hEmz4NKxOHgRMthjBlVrY9b71Z0BAO2XPBpIT2nUe9ExSjwz0KhFG1ELKnUxTz7ssDttd9SuovJmz384eR13J3WtyLWZKLJxUcXD1KTSj9oqellccvnvAQAXvV850Vq0k3oyZZW3tTHTbUZ40Xw1pWcOf76KTDmOOOHnyerHA9gwqJFPvGc5us8lB7bR33nxHecXhsjkUP0efcPclx9Pfp/qf2L+h9M9sCOmN7SNw8UnksNx0ojMGnY/5U8YXmQWu/DC504AAMy9/fhk+Q7BHK4Z3lR7xbiFX1pEY9dk4Ke3VNGbzuGAEB7uW0512y/8/m3zrJh4MM2dmz2dx5JuGptmpixXK6TZ9NV7iZY7lzLQli1wiDONIWQgB1OaN8b1J42o2ZR9TYSEK8LL4JY36dwXQlTIUko9d9jxbexNjaqMtbkKbhq9sZvjtfWU35zkNYTY4YTAcwGYBb6eIyzKXHOnZa5myFMu+TIJHS867NJg8s4p5l1whPKAKZN5FzoOVJgyevIO+OOHacPx3vdekayCPNfGmgInICewILwMvn0okbqc98iyFqkRGOfCsSpikcnj3mO3AAAsuDdFDBhtvmzG6yR1a0R1v+6llwAAE9jJn9Nbkcnt/MNmo/aGW/c/Swgy8AaRAgyLyZbDsMOZVDk4ODg4ODg4ODiMJDaySONCJoq65uDg4ODg4ODg4OAwHOg7d7N39fnt33ljRJ+3cW2vHBwcHBwcHBwcHBxGFM6kysHBwcHBwcHBwWEEIbyNi6XKaTgcHBwcHBwcHBwcHDYYnIbDwcHBwcHBwcHBYSSxkcXhcBoOBwcHBwcHBwcHB4cNBrfhcHBwcHBwcHBwcHDYYHAmVQ4ODg4ODg4ODg4jiY0sDsfGVVsHBwcHBwcHBwcHhxHFiGg4ZO9yOrB3c9IP/IpMzlyyYhGKkFONlBLy9bv5H0ozYWv698kbgS32o3MTt7aeKTgz+u1bBRRH07FfD17L5CDf+gcAwJt7TPI69q3kI7uwVhka10fDr9Kvl4X/I3q296lbgvlKHxCZ2Lwalo2fUa/QbyYHed2plP/ptybKAwBk3wquRcM0Qgj9PC4f/c/n6F3IegXynm/Tud1Po3Plbvq/c6p5d7ki/VZ6gXxH4Ply1YsQ47ekU+pe3daeuxVQbco75LJkFexfBQDwn/09MGFLUxYA8FRXER6QawveJ+tArazKW6JTPUuAVS/QLbP2N/da5QeSf8MwpPRNO93yqOT39S6j3zf/BnTNpDIUR9FF7gteBsi1m2MAKPdAyyd8akfykV8Am+9FefB38FQ/zuRNH7fbf7ZA9+p+50XGAt0P+lfr7+ltcWiy+vW8ZcrB+XEdUr53We2HvP+7lJ7baHWA/s+VgNKY0A2+qb9dd6nqym1alUc+cR0wQO3WO/iSJNWj+7gfSqnfbdy4ElfHSN+sVzH4jQUAgOIX71dl5vui76tVvpF+7tdR/+H7AADZ8x9OWEO7jtHxzm47ib9jN7WL+nVnAgAyx3+Nspi4NWT3YjoePd08U/VxO3+p26mqI3/HgTWQA2sBAN603ZJVUI01gXJy/kMYH6Qa2+XSJ+nE20/R7yuPAAecRXmN3pTSrF8KMWEuXa/202+9DNRr/FD1q/ptpgCofis6pyQqD2CNxTDvKi3s+VHeR/MF5hxAeU7ega9E20hgXjX1El5wycPvTWTy8B+7BgDg7fO5ZGULtFGPC2zOATTnZvJ0XBukXy8LPZ+r8silT6J8zWcAAMUv/y1QF7s+cf1Q2s+MtBehyyF/egI98rMPJKofYI2nmULMGKPWbg2k9JEyVwdQ/vYhAID8J35G1zom6/JBja3omES/XkaXP9D3uRyVPvrlOeXp3wCP0HrJ+8x9iev4bmKo8/+/K5yGw8HBwcHBwcHBwcFhg2FkfDgsCQOqtMu/eA+SiF7w57/SteIYoFPtdnlXLuuQSmKod4KyDjFOSZ9HTws+Z68zjYSRc4qRuJvnNMDYLRJWLPgcgCUPXqAaSe4DYCQhAPoWvQIAGBWWNg7B5k8/Q0kCAKC+jjQyqXKrKGlYrhiV6CiJjURGHxvpSllnIZUkSq59HYt+9n0AwMz9z6fU7eNVIt/cy22nMDoi0fEmbg3J1/8/e98db0dR9v+dPe323Nz0XgghQKih1yBFkN6kSRQpIsUGKoISULH38qLgK4IgguhrQ0WxIB3pTXoJgfR++zln5/fHM8/MbDl7dk9uLvBjvp9PcvZumbLTdp7yfZT2Q3S10987vR/rPkfS986UGg7dT7Y+BhHJSrgPBZALtB0AoGMinj9jJwDA7FuWxzyzcRDCQ/cPzgQAtH9/afoHWSo1aWej0VBaGS2VqwxojaOWuhbbo1LKXU/HH44gSenht70eTMOLmVpszaWVllS9UJ/jvNsKeOH0eQCA2X/ykQqch6URs6Xt4byDj4bauFoGplH+z5y6NQBgzvVP07XmUVoTI1TbB6T/wspD1JhmdzgF8rX0Un8rQfqp9EEUW+PvqCE5C58XuQIGV5O0vTmFFLp+uqH0vRz8gf666UbAmqRiK/xlT9C5UbMBAJUfHgUAyI2bCnHijwOPxWrH882kkQJQ+Pht0by6NlNlzes04upZa14XzV1Gw5kSet6yNXx8TmlL5Iqngen70jn1PuTTv9VpiC2PoIPBbsjXHwQAeFuSNglK0yJ3ORN6Pq6q/jp6C1M/1X+kbAEqSsKcK3GFOSfIl/5OR9ucmLqOWgPkV+Hf9GE6uWE1XTvt5/QbnjfDadjtsM/HVbqmnVQG0X4dWFeD3wSB+6z8xY6nJpYlAm4vW7Op8hVa02vlnVAOTJyHv/zzRQDA0Z+N63u16xc754SRL2HxfdRHptS+K4qBDer5MvzX7qFslEWJ1uYXWvX4Mm1S1e9H8vz4s/ejuPVulEbXLHUfry+tQHNXpG5hBObuUnvw4nYnQ64f+rXWYegwPBsONimAB+Sp8116mzKL4g/QO38I8e7P0TmeBETOPMuLtl+B/BOZH4iTrgpkU2vyyqq2knf+gJ477kfpn1EfM/LBayB2UeYX9qJCB+lMAKRE2x4H62Ng6FVv3pa7Z3+orFSY/IEKwJh6sfkYzESjJ+GcUScXVDt6BUzYanowff4ArgzofqIndS8HySp/XnDs7VJFmYuxerXci7bNZmerH09+/Wv0x6rkxVddkxA1Vcjhuoya0JEt/4wodHZlf4g/ZpY+BjF1TzrHH0tcr3IvZMh8DZBRFbrIYb8dxwTSNaaKFfOs3nxYZg4BhDaoDOFh2q5z09VLJ6XK2DQCks1FPGUeY29yEkwUNHIF4D+3AADm3PIGAARNHWOEG5HiWJvn6HUBPK5MGrc8KmUFDS7fbRYWPvRG3TLUQ8flZGJhm5cMJQof/mX2h6wNq1AbDblhCZ3gtjvqaxFTrtg5VvqQj95MV/e9IHhJSkQ2SQ28Q/nfW+nZcdtke9A2EWGTEs+sdehfq66p+bPYAnQq0y81buWSR4BRs2qVDAibFAsvaHLEUPnqjYJ9T+e0DJUKQng5iBOuBAB8f3cS9p13egOfHn5w/tdjd6gMNTj9tFBzzeW7zMDCOx+nU4vooxyzD6FfL59uvYDAUf/3MKWRZm5qAJN/8GD2h7RpcwvE9Pmha2q95nUbAPpoQ4mmEdb8qMq+6zF4/ZtkrjbliK+pS9lM7RLfhfDQ+1fayLYd8uVM6b5pcCZVDg4ODg4ODg4ODg4OQ4NhosW1zIGUFEGwBHntK/T3PucbzYZWVUojsbbUpwMvPgkAsOXsjCHRCLz+XPZn2Hlz7tHovZzU2c2X3ssXM5VJCAG57XsCz0hWhyepZbNgzv7Zn7HNZGTIxMWSDrEkUnQok7fBbiCvnL8H1tNv7wqUTlAaLUuKDgDI5Y1khdX8gHFw5HdQbDOSdXbkZi1M70p47/lE+roBRmJT7jPpsYSxRNqKdNIqAF4OzaNHUbIRiZWfPp0EFEaMrH9TGJzv6DmQT5D0Xsyhvqad8kvthmCA7x9YH+sk3Xrip+mYzaCW0dgUXZtpcw1tUmdpBLT2IVcEekkqJltJW6LHiV9B6dW+3wAAIABJREFU4T3nZqyfZWqg2pCdVwU7IyLlWMw3A7udrB4gSZz3oV8BAK7ZdxJO+zf186Q5Rwgv1hkYIMmv3P7I+uUIQ6W38N+PxpjBZHfOHfj+cQAsp/EhhrzhHACA+Og/sjxF//ethuxdRc8rk4vcB5QZleV8q83mpG+1h+q7g93A4v+q67W0bBuJpRnXDN1OFuEEr3/cX9omGOkxm8OO35bqA0Dy3HTHzyBOJHKDiMbHIuwIahpDc65VpoBZDAD4VYhiyHwlA6SUgHKwPv9eMv986SRyPp9545LgfUgYm9q8pylwmsymkx2YU4HXjrRQ89lRU6pY9vGdAQDjvquc9rOWQwhtUqT7frgfbyxY+5wF3O/8su53vBbKl2m+ENP21t8npm2E+Sbg9VpKjN1qljoMjkMhRF2TV74vCS0ztqhXo7cWnIbDwcHBwcHBwcHBwcFhaDA8Gg5lvy+aR2opjuwg/YRQUk0Mdhupttotr7toHkZ8WdkdlpRG5PnbUDrlSxtVHCmlplQTpbbQNR845IK4x+okqmz8B7vRfImiZLOpP0G2pml9OPDPn9DxDKJUHTLNBuP+X9BvBkpVI7EetOyMlWTDNxSkmkqWkStq2mChHL3EyJkYuI7sOUufPlA/S2lZEjimZ80VjfREO6z7JIUGLPteRTIwdmv4PzmJzs19b7r6qWcv328XLLxXSSxDUqH02gmBpo+SNFxrZtT7GyqJlfe+K7M/pPqRKLYBWx8Vew1+1bSFph5sivrmQOKN710CAJj001MoXbZht+hM4UUJD7T2odynx3acL0fPdQsBAO07fyhd/ZSfj8ybdmLNRoDSNg2kj4HrPwMAaPqkcjZWEr/T7lgMufoFlT7ZpctCS0Q6J6U0EutQu0vpA39SVJ9ZxiG/p2IbBr5ONJOlC/+izrETcDq7byklikd/MnjOkrgPie/Y/inbLgAe/y0QXaxZU2QRLUpzuGFpxGleLrobYvQcOmaJsfCAipqnWBJut5Mwzs2cT5g+FYiXyurz1Yz2/6xB9HKmX2hpsrICaJ9gtIrcJiufARY9RMdbqz5z2MWG2rd1rCqs6nO5gpVX3qSlx4Cl6WA/O25zyxFadiut9biMPlVQ0mte5xVYs/HHA8fh0L8y6UUNXy5GHBGFzmMI5tRa5A61oN7rtl+6EuiYBADY8DnyjWz/4gN0j5eD5Hk1TIBjw69Arnqerqs51KaCbZRW2IZcu4jS7Zhc5077mVd1mURLky4PAIjNDjQ38vvnNpJVoxFp6qTfKbvihfs+CgDYih+ztf4h36F6c09EIyKrWP4AUcXXoQVyeJPgIo07ODg4ODg4ODg4DCOGzFzubYJh0nDEUAaGgiihWjGSFWU7uvjltRihg2YpydT0fSCZJjEGaSRyQoiafDlCeDWvJSeqpKntE6B9NtiuXdvDC4RNiOPtvgUGV5IEqLSpbPzWLKt/Txg6qFnVaDZCAe8CUlzbnr7YErzPr6J/Jdn16jqyHWh10Ej7dHomyJlhPQPgqXPVECOYl0dlPdk4p5cNUeNc+ue/Gqmgr7QqFt2rRFACExuESQiUf7wAAFD81D9TlyAL5M2KJvIjf0//kG0nrgMYqjKzP0zzSNOutlbL9q0CAK+I5jaVXtUErdRg/wLdJgXDNMbvM18yxyoNyRIrISDyGaeoBC2Spm6s9EPE2IFHH/BQmjNPlwWADlwIQNOp8rsa+Nq70HQRaTeD0vBa1MoCmL5tqmrZkCtJ+yZGb4Gmi+8MpK3t+ksjUtm1CyEg/30t/TGHJOaBOnKeKeyra+IBYojC3BPSP8OauEDZQ2Ospcsc96/hgmqtpJaYFlrQ9xLRGbcm2YfXqVsSJbD/xvOJz9bOMw+pgzcqTTj7UgkR8ZUT47cHWIvIY3nEFDx8NEmtd/wdz+vG/p+Z9gwVbt6sSTbTWliDbNVv8WeIgnfKzX1oBLWoqQ/98yL0X0FseU2fpnlM5kuxfZaD0ImwL1kNZPV7lD3K16trZqr7dR+d/R4dBLD9y+TDIVUQQbHV4RHfNymMVitAafu3b9PhqdcE7g+w62nmS3MutRbyHrKawNQ90t0PkC8e2E8mHMiY/SwHzHqtKZ6rROsMaDZE0ToWPTpCQpgZrnZw5DBq+WEJL4/+nsF6VXJ4EzE8G47mzsgpvRgINck1C4RVqVvfssioIfn+5i789WxauA6+Ld1Hc2zk6xr89QDw+iXHAgCm/CoDf7z6CI6jlNQ8/TYFYtihKgS/nG3gZHWWl4PZFw7Dqy7NQsU9iCfAAB2qulYdMOf4N1+CXw050+qJtWCpZnlS6w/SRao0MKDUtryRsSKCSz/k2F4P/IE2ZktrQmcHSkMtmcgTrvsYsPLZlwAAk9K2ScY27H2JHGHb6twXyKObFkbROS1q6tOi4qD4FWP6uF7Rrg5uMNHENY2uQKlDxboIkTsIL2ccyQNRxTmauWqnvtXGLK4QMo8DUBw5KkPt0n0Qi3yTKZNVnvBHjhACr/72NwCA6cf/IHDNjrcj1Yda00V3GAdxTn9gvdkExWxyNvztRgBAx9HfT1E79dyYOVYeofqWRuh6heMV1OpX3S9QP0oicQ5SCmdz0K0ueQVAVofB+mPMjn0jm+ijTkzbJ/ox4+Xw30fINGSnTGVIj2WPEC3qxDr3adjxH8LvMWaTZRxvfSCn1i5rXP13PT0zr0Y8lNqwPt5kcDNgf9gtX01jPlMMhxiEx6fIl3Rk7e/vQY7k59/1GpCLERhw/Ie0eYXWVulXYk3l9P0sQEidAZvCNZko9XxJUeOjb7UhzVDzm4ihJRZeDs/+H1Erz0kIB2KXvxYZRS2s/uuvAQCjT/xpqvsBBL5PYmm9AaDQbOrDgidIiKl7BZ6T+SL+vYze2S4ps0+qY9xc/9QK6sPTU6b/puNt4DT+pS99CY899hiEELj44oux7bZGSHbzzTfjlltuged5mDNnDhYuXJj4/fLO0uc4ODg4ODg4ODg4OCTigQcewKuvvoqbbroJV1xxBa644gp9ra+vD7feeituuOEG/PKXv8RLL72ERx55JDG94dFwJDp7qd2vlzfSRpYuVyvGCZRNcqSP9nw2o6f4HVdtBzXhNbAPCzvmIWZ3LqvGCVdH5ow6IUspURqfUZakHP5Y2lpPSi7GZg/kZFSqOUAqKTTXhx00bRMAbsd8U1BTAQD5khWHKuS0We61TH9YAp0zGhAdeM4zEhgraB2nn1l6YFEvww9RVVqR1SXC7Wo5SJvE0DqiCRuLJOl0rjU7VWXAYTpsosaBx3JFyGUksRUTdqBbXv4XMGpzus593MuhOGpsMH3LCTCiQbFN4fqU6U/LKNMnGNYYyneOTl+5DNB9rspliu8rhVK8liBgYqBgaz1YQutveCNRcppryaKfUmANk5eL194CZNKgtahG+xTXj0qjxkTOJYKDnHrp6L5zbQ3QcaZAXFRx+5x9nH/rCxJjYFPW8vzWB91X9XuXaPLSr4lSWuapdr/X7RpNq32IOUtscNudfw9pU6/eeyLOumtp9EYrCGxD+Xh5TYgQZzaYOfBfgKjEWjsAE5Rx1XN49uNkMTHnZnLa1sFkEeyjs7+QkQSEzSiRTjPe9Ymrs6UPGI1FQjDQgEmxNddHymMHCEyJWNO6BK2Hk6APLe69914ccMABAIDNNtsM69atQ3d3N9ra2tDc3IxrryVz3L6+PnR3d2PMmOS1xLWPg4ODg4ODg4ODwzBCeOJN/VcPK1euxMiRxgepq6sLK1asCNxz1VVX4cADD8TBBx+MKVOSBeVvHZYqKyigkYzmDLUaOwUWmrHlrI2UmPkVLeHR0mrLIWr05tml/3GBfuxgVABIcqs0A/JekmaIPc+LONIJISC3yRiYT0kgtKSh3JdM/zlnn2zpA1a7CEQkOqzhqQ4Ym3xb28EOxOyoLCVGnKOCVXG7qzTkhqUQHgeBVI7EK56GmLRzsDyVfkPV2r+OisaSq1wRxfdemq1+LJHpXx+lo+Ty55uNtIclbeVeSH7XVj/qmDuvZlbxmosYrZsflDDZErHS5tulqVUQNm2hrSkCjHbIr0CwNmOQzomZ++sgiOy0Kcu98IpcrrDUSVgBzZR41Kbj5Haq9EOuIV8XMZKcNZnqVPoVoJBNSxQn9Q5fs8+zf5W/dhFEZ9AOGwDGzJ3DT5t6mRTpxzd+IBFNyOgtIB/4X7q+29mR/Js32zpVvQLgAGito6NSRNWG8tEbIXb6QPCa9CERdFaVUqKwyyE1s4o4aEpf9yE770Q/kaxzWUoE8+d53IcM+bUIIYZEw5Hku5LPZ5W6B4O7BdLVdakCZaX9s6g/9VzL9xVbsd/cdnW5zImpX8/SXHAa0pLmW1Jnvf5WzX0AkCtg0twGNOKqnHF+jXHg+p955xsof5f6ZP4jf1LFHIB8g+iA2Qm5EWhfM6Yeti0SOGhpmNa9FmyH+7C0XX233HLacTjueqKslovuoTun7wuEqWmkhJi5X7p8ofqipeGKLUMIYmLt9agmYrRJscElWcPPc30VUad9L49a37g0tsKO5LVIGvi7agiCPDtkQpzD/llnnYUFCxbgzDPPxLx58zBvXu1+NjwbDj/kKBoDYtMoBu8Xnv5Aef3D5GY0ceHP0fnej25UceJYK2TVmNAUDz2nkUSTLtLPYDegIraKPc+jc5V+8zGpWIJE80jgv8Q+g13PSpd92BnPcuSKHZAv3KfSz1BXHXV6AzBIHz06CrBmVskHOeYBMgNgZ1brI00wW4Yyy5LdSo2+8lmgXTFp88ajucssiBYjlY6toHjQdd6QEBMzuohq9XHBfCzzB9xzKs7BhhUQe3D0azb9s1g6GNIHjv1q7bw4botvzAizMgCJgy/JdD8ASPWxKkrt0U0y86VXBgypAkc1XvOSdnrmcSp7V8EfoA+i2JKHWbCKefBYEIpJSJb79EZDl5HbQVaBgYzRf3UsAS/aJgnwOqdC9lHEcx1V3ssjP3EGFWXFM3RurOW4qtm6TATniON5rqjHsHz4Ojo59xi6VmoHRqfnxNdQsWyC8RRUCf5wMR3MPy/iICsHesyz5iwwOWHjGopbkTaWUADj59S/Jwyej/PRDWfsxlELkCxmp96VdK5lFDpHbrx5Y5zpK5ejeURL3CO14VumorU+FIUX3XBbBByyW7EqdUzCQG85+iwdRE1LhYg3kbE3KQDgKwGQzEOWY5gm6yDtRiPmSb3RuG5fciR//7+XAsq8cyigCRWsTZEYs2W2RAIMYqGPfvWNcdyXPg8xcjrd8ehN+k4xbW86ZmFLx2TIDSrWiY44nlD+GMfzeBNDs0nW/SWLc7yOPm/FNeK0WRhlEfDoceiZ7wBtZlXpRyvLuyJCVi8Q/6cRSCkxstgQx+ibh7c4Le7YsWOxcuVK/ffy5cu12dTatWvx/PPPY+edd0ZTUxP22WcfPPzww4kbjrd2bR0cHBwcHBwcHBwchhV77rknbruNgt4+9dRTGDt2LNraSLhcqVRw0UUXoaeHhIJPPPEEZsyYkZje8Gg4Uu7iwjSTEJ6Wkk/6n0fpXLkf/zn/eADAzr8/J/DcRsHSeiz/IUUXHr/7+emfT+EYH5UuAjJvJJQ2v3jv8xRrpAGX0mi+MaguIirMLG/O0NuViO4TMGYo2rlaGDM42wRLaUTQxHUUkH+g9yyOUJoANtWYsa+RjrMkpG2cdn61g5noSPEcf4CpXaUPecd3VPrfSllDlW6h1TI3UhL5uceavMPma02diDhIl/sw8H2ib27+3L2RnIw2ow73f0jaY5vCyF+Spk987F91a2bS435u940Q9WauYGJiKO57b+zWFne/aqeWUVh0P/VTlpnF9znLoZdN4Cypl47mHuZ5lwIrHiIzinFpKxgjEdelqCeZZw0PEyGU2tH90L8BAO0hWlwqZpg+NGriI4Rn2nqb4+jaUprLxNQ9UX6SKEFLxycXLYA8x1WIjl7v2O/WfEw0dWiNoE3R2f2/FGm8/SsHR58Ja04TYlEA8dqH8q8uBwCUFh5bs2yR9BpsRyGE0Ta3mV7z0jLSfDWgT0pVjt611GdS0zhYkt1IujEUxLpvNZk1QseLEDnc+ARJhy+Io1kXUeKDcLoiTuthtcGdD5J0vLbxXRSssUdTZzatmEUuseAOciT3X/o7sITWLCgN81CY0gjLVFU+TvFixN4XZCpnnCZHqrlEzFtg1oud3k8Xq4OQz/yezk3bSz0gUfk5zefFC25PV/aE+sfNSfjdQvo9/2+p0gfMehFnSiMSQgtAeIG4R3xuSZ/qbzHa/FhH/qSyxRBFLO515lVDiR133BFbb701TjzxRAghsHDhQvzmN79Be3s7DjzwQJx77rlYsGAB8vk8tthiC+y/f7L57FvHh8PBwcHBwcHBwcHhnYC3gf/JhRdeGPh7zhxjInvMMcfgmGOOSZ3WMPlwKKmm5XBopJghyXD4Gt+v7KpR7sdOP6GIpP7/HEm370y/mPUuCJaMt6iAYYGdtEVPG6EWhL429ku3ZaxgPOIoKxOlElVjh5trzmgTnBG5UnabZslOxZUBoJ1sa43zIUsZPED7c6h3XxphOUIq34U3HkLlFYpEWlASEDFSqeOkNPezl1mhxbw7diq2o2X7VpRyAPKRG4DF/81WQeUEKBffh8otJJXNn/yNYNm8vNWfLd+ksCPdwAY0XfQPdV/oGpKDB6aDhPjg9dkfYymSHXCP62FL6tR75WB/0qauZTrGR2/EzBNPonPsh2NrhrTWT9XVpjbmc6UO8+5CUZXlusUYc9T7stWPtRPF1szRsbW0WWkiZe9KtF/6j0DZZI9i6GiJC0gY9SkArHGt3qmnnF4v23ECFt6RzFseCy6L1Y9iJZBxfSwkRZTST8VWkhZxeTZCM67r07fKSPXD0ZnrPGsTaOx9M2nibnkXaT2OuYHmeNG1WSTCdr3+Ejevj/3WA6nqZWC3W0gaHSDniK5PAR9HAKj24RN3kS8A90/5CmnmMGrzSOBJ369AdChdjyJjkbmiJiOQ61+ja23jdbYH3/pKxvrBaAyrg5n8qYLB7dS7nrEfZButOX2X7woAaL5UaY4topJG51UhBOSMvbI9xPOgTQurA+ApbbBfBXKhc4O9mmKcyU6w/CnkZw2hj0qoPFJKYOb2mdOR/WTJgFIbIlOMDhQZ/xmptbzc75Y/jQvfMz6YLreXl9dWE2g1VOtx7am1c/w+WUv1/F9w5BFD9w4dhh5C1ooTP4TQzkoBVXH0I8x6Qh9FHfV8/PJdNPGc9M9loWtm8k4ys4qb5O0B+uyxtCjN+UuGSNW9K+vfkwHdn6WB0/bFBj5IUmDwWwcBAIqffTj1M3KlcpxtGZrYCJftSO142cNLhiQ9G1JK3LAfpf++B9Lxq+uPSWy8ul5KiRdPoMl1s5uWDkmaYay7gDYDI65clb5cMWMxck8MYwgQLb/0K/j8TvThculDS2LvSZNuzbHoV/HTfYkM4PSHsrXhUL3rf72HFr/5f1o+JOnZkFJi0QJ6f9N+nd4pV/ZwWewNR7ro33Efy/oDbuH9oZv9iNmYHX8k/mMgmn75WwcCAIqfezRdBYEhn08fOJzacZc/DH07AsAbpxOL08Qb05EccBtujDmwvXR/fh7NdQsfjolfkTG9OPO4Ww+iNfHQu9LHUvBffxAA4Cmn6dTlsCKCSyt2g/wLmQR57/li7Wdj+mbgE0fHzrDY+gBAeJB3fFOln0D2YecVMw5NniH2SyqUvj9uk/nQUdSG8363LLkOdAbRudSQVkRZAyVeez+ZP0+9JRT3KAFJ60XcnIAw+5ZdBunjW7vTfH7B/cF+WotdMNonE77JpMRN+xPD2In3Z4/58WZg4LJt3tT8S5c9Maz5OZMqBwcHBwcHBwcHh2HEO43S903ccCRpIpIaQeCE35AkTrL5hDL1+dXhc3HcT8nxSyp1sBgxJRotGjGqal0sH1tc81DKOmxCMHd6SsllVjTU0ZvJjERWywg7VbMkR3i5SJk13R1gTGq6l+GYqVWTnn3NlgBpWs5kta2WXHGe61/HvtOHLjxuPc5vGY450bMSI8ezCymrtZGYRnL+oXcqfZTGjE96pEZC6p1XKuadFUJxU8r92hk/IKkLUz8P9uLdEzmyrop0z3SmRYvuwJJKhR0sbYlVNKZEH/afljG+wQCZKPj96yFGTFbpqncWw72fBOlXsedBO1J6S5Sj97i5Oo1wnAGbOjI2vbB0bmADpuy3b6qyBBHUBNFxWmKOKNe9ZJpp1iooDaYt1QzmVcdpO/S3yGcfh2xyIZo6olJOnvfzTanM5aSU2Omnd+hjAMZxf8xW6FlIGp7Wy4kqXK54Bt6E2uYncWNx/Nf/kb5yKcrLiJecR7Xyn7vlF4Gy6THnV7Wpinzqd3Ru/JYQU3anYyYRACAXq/pzNO4pu+k0D7n615nrkVWzwbDHZ6B9D76MysNr428+BgDw33gJufN+H72f07DfYdipXlhzwT4fz1ZQ1Q9FDBmMHo81h0pQYi9Xv4CRLUykQW2iyXOkjKlXbXIOO3+7L3WObYCCRtHeo21srNaE/xYB7U08ZLWMj1+oiDOYbv7Vu+ipcdsaWnyOAZUrRkIYBOiAmZyGTawf+wXee+EZWWrnMMxwGg4HBwcHBwcHBweH4YTTcGwC+GEJdvwOPJ3kV2LwylMBAKVP/pVOKWfM4265B1j7CqXF0UL9KpALSuFhR8o2uev0/RsoKF/uwrtTlEc9pSO8eqmdVGum5VfRsvMBdPwcBUASWxxm8glJaLNKzKX0kdti5/o3RgtmHXM0Wl0KlbawIttajvnamVdJznuWY8accea6nb6URoPCaYkcarcZTNA+lgr1rMCY2dMzVc/UT+poxebd2g6CMdLkEFWu378GbVMUZTAHlFNBHylwXrzfQi1E8pQ+iqMb0HAoJ0XZt9o4jlZDQcP8wRhaRRnV8giBefuTr5Fc+hid4qBStlaLHUat/hOQ9rMTvo5greqay2Pq/IyOnEpi63VONQ6ZinY3rWZDQ3joeZWccTuVxFsy/XJTZwxlseUorh3hTUT3iD+al0f/ohcAAI1QRMhyryHJaBBSSrR8+s8AgK/NJ3viTz2wRF8bCpV//ozrGnjIvNtajvGZ5tkHSQMgDvkC/T1uW/rtXYWWKYoQgsfw2Gj0d/tdxI7/x26h38m7pi/TkEECndPVYWjcVgeNhn/Js/Q7fktDe82BW5s6jTP6aMVCw39X+oERU4ampBmJHALgyNVM6Xzc9wEAV+85HmedPwQfbn46P7Hw/bI6GKDXBbKsyapPFVox4wekhXvq+OkAgK1/SXODfPXfwKx3030WTW3a7yX68dF28Z9SlskuXm0tambvX6+A3keIzKD1OFqnddR4Lw/B3xS8XvgVAFxfpQla8YwJ0BgmImgfi/LT1wAAshHsOgwXXOA/BwcHBwcHBwcHB4dNhjfNpMpIANlGP63EQ2D94tcBAGP4GV9JFZo6IVnSw9KKYpvlu6FzN9JWEQ7eBfQvpfQTwtpEYQVyCtt2Z4XwcpA7vZeOx5LN+JqPkV1j57eeaeCdhdIXHuT26bmTrQfp1/eNVJxtLFmLJfOQK0mSpiUR3ctN0EPFeoH/3IjW07+u0g21T6wGCkG7ZIAkd2HfCW53v4ziiZdnq5/W2viajtf4lzCdcy22jpCfRmkEChNmBMvL5e9ZAbR00X11pO6JJHJdExKfjYV6T6JtHOQi0uBpe26WGBVaYAffAgD0rbGoYFnD4SF3wpfpUElA5QM/pb93Pd1QWSu6TRRbLbpTpfUpdRjtyMR5wTylDxx2Sbb62e9T5e+vfpEusfYlJYQQaJk8NXiyRNoS/9W74E2Lal8i7SU8q1OEbi60oDS2gTbksZEQHC8Lyj86GQDwqX8pxhJlQ45cKTI/NuR/dMOH6dkMASq1L5ZftaSs6jdjgDBA6meYdUdy4NKW0cB+p9Nt7M9gzdvxwfJUaTQtbTbaV/V0tJQxTD9Rv68Y5iK/agIlchl5XpS+npvFfAosJ/tW63eq0/UHIZTWR/Yqal2mwr7zxxAHfSZj/eKhWc9YWxEXqLAWlDZGKkp2+dC1AIAP3b0U/q+pbjjm25RuIz6PHKiwI2V4SA58J7yGfS219q6lC0KQ71ReJbHuMzsBAEZ8+YGgdQHSj0NtSQJAdEzKVDYAer2ItURhPwybsj4JsgqvqPopr6u+0WrI5U9S+kzHbDGU8bouRk6H0dqE5trWcRC5t5mXwBD75r7VMTytEzZ3gTXQYEyrIg5ylf5oxFm/gtHnfI2O+eOCf3NFCBVLAcp5Va56DmLMVpwi/VQGIK8/k8pxvIpC3UqDXa5/A82n1qbdqwcp/YY3GiYNCfn7K6h8p5GKsPMbT6qLfsSkqiHc/j36nXVQhoesiYYXWG06o95trmhMErg9m0ZAriNud9FJ9JFiv0+g56sUt7b1chWdUlMVWpOrzb0vQu0NY4ajOblZ3T5hRwx+m9IvzX1vuupxHI51r+kPaO5/SR9c9DGCwHXZ3AlxBJlv/ODA7QAA5/37ZbqpdYwpv/WRERt1uFYEZ5EHNuYjoNhmotzyJsomVbA/pgDaMIY/nP0yFn3qcADAtOtepUt7nW+lodJlsx8vb70ftXkp9wbjrwTS97H+axSCu/NHKal/NY2rca7ljYZUG584J89aJgobXn0FANAVWty8aXtBditaTI6UnivGpCEgZS1TDYnBVZRGc0KVYgpLKXu5jRZuAED+GLWpUx9Q6z5NZnIjvvKwieGwMaZVB2d0xgVMX/C8aN46cnbUSV+We2NiMghg+St02EwxPTwVhVxKCY/NVTeQ06pc+QzEjPmhNGA2JBxzhsex8IBVr2WsnxF4RZzifWPaqelhbRM9hnZM9iB5UwE9EdFvoU3H2tBxMe66UjtgM0EApA+5VplcPUdxrsQOFGNH7nN2dnMj1Jkzc8ZUJu3LO5pQAAAgAElEQVRHulzxFADAU2QQ2Ok0nY937HcBAP6d36FzWx8BT0Vit8kikky65GoyYRLjUlKVWt81wt4g10g/EX4V8jVyoJ59iRLEqbgg8hfnQCy4JlAX2uSG4igJ22Q99M6lD/n6f+halo3HADlmC+47FhIjjcdBeFj85CsAgM1Z8MWCAOlDzDwgeL+9GeXvOi8PvUBo81Y1/1b6sPY5akMTycPhrYS32XbQwcHBwcHBwcHB4e0NR4u7SXKpr26OffFxkg8vDzHrQPVMOMgNjLMh7757V2rzDck0an4F1//sdgDAqSc1BdJC2zgjCcoCm5Z1I9VkQgiI7clJTL6qzF5mH2LSV9SfkqlHB3shmjqyZbLV/IbKBZA6XP6LNENiTzKXkEqKJtonBCTadFGa9mAn8BVPo/Wz5CSnKVUVLZ4YMQWoKMl6sUXlGXXMC5hdhR3PBzeg+JHfZasgR05t7oo4MpvozlGJa2yk2V5jtvDhU0g1rgkANj/YPMxSHJGDDJmRCSFqSszsgFiZwM91LzORhNl0gk0KmjsBqTRLrOGpDpp2YhrdgQ2Y+hnSlMnlFDVejJ5N1wa7DR3nqufo2rhtIJVJktFItZh+osan1H1lEB1nf7uxekoZjXCt+qX0q6klkB0z2AwrhsZbmcXJhyniu9jhFGOCYLebHe3XSgvSR2nSjLQ1MhhUJmoxjuuZUR0A/vFjOl5ADs8jvvY4/e3ljES1rPpIobluELDI+XuU0/icIzOUS2luYtYOTW276gXT3xj55lhqXhxDUmObwpiueaaOqt99/tiT4oOR8rymyqTHvPAgDv5c+rrRwwnXuI/JaJ8JkGcosyjPM9JuPZZ6TJlzbCqprAW2P1prazQJwrLHIZTDu3yRgsHKOcpRuXVsgPAlLYb6Y0rM3L9u+mJPIny5au/J+NDdimaV5zwvnzjutWlpWhSiekltMpZV01Fs1Vo1+Qa9f6HaVB6xEF/ejSwDPnO/0qT5fqyZVU0TXJGDmLxLurJY0KaHmZ+Epq/WmhCRw6wPfYQOQ99IQnhm/Qtr5MJQ1iu/OZJMAI/562JKY+w2GL3gggZK6jBceGcZkDk4ODg4ODg4ODg4DCuGR8MR44yXCjVs2n0VoEgwVZxNsWtRmwIg6k+W+nAQG7+C4w9Skkt2fFKBzuBXIJc+Qs8qG9BUUDaGQyXVGfgn0Tg2XXJ3JF3Jzrhc16y2lAD8+4jG0dvnwuyFK7YCOxM1MWuNBDvr54pGeqfbzzcSbaWBEuO3RfUn5KyaO4c0EcJ21lOajYCDeATC0mywBFDlWeqAvOEsOnX2H1NWTOj6RSS1sHxJwk/FtXlLF+RP3wcA8E4mLYBge+lcwQQt0s7olEsgT4nIGNCaDSEgf0e29+LM39aplwWWyjV1Wnbe6pxlE641OpL9dvpMG3AZWkah57qFAIC2Lz0czKfUYRz9FPEBckVDLsC+P17eouU1TrAAgHwTBm4mf6rm7Rekq5/S+sVKFrl+vSsDfjRA7XE7sJKcjAtxWktlMy12OIVK//I/zJxk5x+hsTbvubyWgu1lmhlzQ0f6KPJN6FtMvkUtMfOIlv6zk3ENiW2t9yeEQGXZIgAZpVsx0uNwXhHtRgKWnEe0xhP/V/kaWe2p/SSUBmPhfS9EnpeVAdOO3HeLhkii5/P7AADavvNGugJpH46oxjQ+iFqtY8IrZ1H9ZvyC8uc1QghhNJLsDzd+WyKuACB4HEzb29ACn6j8+7RGpwkrziHSkrHXrU9Xv5TI4mQtXySrhICGuAbOunMx/NVEaY2n1fy/xznmHcTkK19T3xVbHpWyQMaXKoLMVg4C8kZFrnCy0jhyAMa2cbjobyoo4+O/pHNbHWV8N/pISyVL7XqcylxQCwcA8vcX0blT0tNUi+aujPWwEKENF+i+ncrffuBnYx5Q87+ij4dfidDNS9/4sB559M6csMqvgIF/3AAAaD5gYePlHk54Q6sFfKtjeDYciUwUSj3u+2aQWuZJcar6td8ntWnXd58PJiU8sxhbE7rutOw8uvwpNH3iD3Q9H1rY8k3ov/ZTAICWrx6bpnaZkJbtJd/WkXAfL/7GoZAj8+rNh/QTGUC85qjjbGrkimYyUY52zNMuV78AUQylXWqDUAugXPEMlb1rFrxjPq/qodTQinMb5X7Tjjxvl3vNwsmbF+HpuBJ6cbSvHfSJhqqXelOR8KzMlbDsPw8CAMafOV1dswgSYiZyjr7K5ApxKnJ7cet+nBahTC2pNv9x8RukzXTD5lPr6INbtI2zzL9Uo8gqNqwi8572uA0hm6/krUjmvEhyJPOlj0GMUh+OBbXpZ+dHv4LyOlKvp3aq9mr3ecMIM1o7W+uPN78ceSdSSvSuonEVF6NXWAsdQAIQyexbbA7gV8yHQ8hZXUofg6vpwy/ThmMoSCMs+OXBwN+2uZFmUSvmY6/T38mxlGQo/VSIYYNL9ViNcdqzvr/+szwP2SxVtsCDHbNDGzMhBHpXkwAhdSznFPWLNdW0yCXsteThZVRONtDTdZHSbIwtEzAop/mAaSPPP+yEa7HRLV1K9XsznXG9FBuNQJRyJTC87CPEpLfwofMT5/E41rlEqLlMSi9yjt+d9H1tWlovHtCGR8hpvOOUENlGdRCCHeW3UQQofsXMw7bZoe670ThKA0/fDyAjQUVzNvNyIxDzIt8fUkpcfycJWD4c86yJv2TFM+LvGq6jJQzLnxcUJEoA99xFjIRB4zuHtwqc07iDg4ODg4ODg4PDMKIh+ua3MYZnw5EgkTPmBeaciTicTd1k05PqPG06P5Y+9K40dIBa+mocOUWuAekaS4fqmDelrVP+3N/UTcOWcF22xxYAYDk7JtdBLPjfVOWIRWXAOF+GpLaiY7J555o6tx9yDZltaMrcgfXA649SGuNV1F/LXCg25gZ3Et2fhHnf7CSpaC8x2AsobQpmvitdvdjcJ84pNolOMY5S1S9j5AwVnVdJEbUD+sAGkjIC2gQIwrNoo+3BoKRoocjnUvpomT4rXb1sWOY4tjQqVCOjzdDUgx1G4mZJ8cbv/+7gk2weZZMnsBTVy1njUqWx+BGtFRDT56v7TZTytj3fk75uSDe+hBCQrAnhfhbWdKr7Xl+SUbKrNRsWVWWNeBlCeNiwjDQ4qSXjmwDhWCABTZw2ZWCtc85oKcdupe5PcFYFkB89caiLnBnrB9KHRbYlswFNQcI6Vqk0SOKQVI5AlOeomZV9fVJzmJjA3MPjnM1CA87GmkZ+0DJ95phG6r6cj47WTfepEI7JBYsOOHDfeoqPFUfrGu1/UpNgsIlc+ZsHoPDRoGVDwORoA62d2vS1LkKxTAAYymbLxLeOZoPRPme7QHp6zHm5oAkqAAx2Qz5Aa7iYd6rJKwGlmVsnVycOGTWNgfESviYE+rIOEzatttdEXk99Nmk0ZtUbsrM3Owwj3lnbKwcHBwcHBwcHBweHYcUwaTgyRBNFfSnl7U+QXV9cOLcIfagtXWSHsal7mJ27tvlne9UcXnqIJCKZ5AEpI6am9eGQv1AOZO+/tuY9tqbj0tvJ/lP7AXiFmj4cUkrIP1xMabzv+lTlDqDYapy/ddRj9p0pGG2RbU87gYKJae1Hyyg8fQU5sc397QfpUWEkcMavw6aD1KmpH0uKzpJlacqz7HvkizPhhnPS1StJE5dEpxjXll4ePUtJYlYKS55KHfHSKIu+EahnGy/Qv5QkfpkoA5hStdQWlUbZ1M7KN0ZMIGdUueRRy+/EaJr8FVQGHTLQpoVlTQi/O78KVBUd52v30v07nqoli9ru2J4v1i7PUrvUMFLEWloewkBGiVzYDlmWexOdvBvimNAapmzzat1ktYQ5KrnV/mLVQRNVnu2rS+2Wz1zMOEnQfgwXchvpm2lrChDju9JIekOJ5zZQmXZLoJfXfd7LW9oEdX++ZEV2Dvomwcth+XqaG6YPaakRKJeOaq9JJEII+wbGpWHOAC2jAmcKF9yOm95FVOAn/nNZpvRjEaKlrVUeE8ixNoQQ6HmF6MNZ22lrRGQhND/lmyC2OZquMe34hB0S15BqI+QNDc41NUk4ss6noTVZVgaMj2P4W08OoqfyNnPCdnE4NgGSFiP7trBqNfARYAbOYftMC9wfGNzhRcB2rmIH9TUv64jXOvYAO3sWWjB7/rzk+sSBJ8l6TFzh+A6QiFXNto+smUTYaROw1cwqjd6VkOwYyBGWWbUrBGTTRhhx+GXIdcR9jef+RmnucDIXDrqt2Nyp3GfU4Sr2g6z0Y+ufPx6oj47g27sK6FBmGNokTliqfjazKpjjAWX6wyZe1UGM//b9GeulTA9ELsZZ09Q9sjGpDpp257pUBzHygGOC93F7VQbMRzibgomc2bTZZoAcy4IdOa3yNO9xWPq6mULoo0gsDzapKLYaUy82JWufYNT6vImv9MGbqKL56nqzSZUwDv3aCRWmDXvVeCv3QPCHwYCJL0Hp9wOjMkTFRboNfcAEjqPLr3k5lpVu7vb187ejfUf6S64E+foDdDx5t0jZRm25Rd30o6i/SMnKQGwMizjkZrIpRwxzT2gzKu+/GmL3c9T90b7EDrSB9z9lq1TlCOWsfodmQW7JsNLVdYKvYS6SBUl5RE2MLFQHLeGWYQ476ShqQxk2vREiuuZUBowJMOchff2s7gc8f5fasdOhe2aqH4BoHIaUELlCLDGBfJLMjMVOH8hcFsYJfycWL/9RYjMS2xxvyvv8X+jcLmenLGjtT3e9abIiw9vzROR+6aNlwRfj05LSxEri+Cn3XqVjYGGNIvhY9iQxkNFf/LD+O3fEp+pUKAacb4xZaFwf1ucq/YasRwu2fEyt8dkRYBjlXy8fHVeeiUMjeb3QcT48bN5ef3Pn8ObBOY07ODg4ODg4ODg4DCechmNTIJ3qOaxaDaTAEgOvgKZDzgjeH5cGw5KWaInQiKlGAqSirz5xMknh5l59B3L7vT9VeQNIKU2MmjlFtTNCCBNpOy6NsGRlYB3AEZw5vkPbOCMx5gjLygRCNHVY6vMGMNgD9CiVdJdyjOYyecKkze+93AcxeovAfaIyYJyDWYrPTt7CUodbFKxGw1Ex6bOkLqQ+D2ha0sILOmbbxzIgdQnFxsiVzH3aHKEI8e5LVXlDZjteDnItxQMo//RDAIDCfgvgKcmd6eu5gEOcKhCXDGK/7DFUJL9zKaMqaSsWjVAUxFpi6lcs7YWqR7kXWKk0Xdb70eXkNtFqfmMCJ5jesXcVMKAijHOkdx3dvQqsjon4nASm880Vour4GO2HPh45gzRrQMDJc2AdjaFWjikSY3ZnSywj84/Iac0Gesg8TCpiA5ErYnAtaXqGyjhKO5pmiXm0PUUA1xJuay6TbxC1s46zs3pxvETemp8j17Y8KH1ZGLa2rUHYUvK2kbXJQMP9omZMEcusDIBFdzoSrSMzlpPLFmOqaeaSaIwOKWxJs5nfFt33GABg5sdC2g8hjKkqw8sbraNNlhKSysteojAV+SYsf5hiU41PVzt6biPaTms27HE3L2UsnpppCvNtsR1p5L+5KxEmXPjAUmDrY2o9WiPB2hoOs86b8cD9LJ7uWABM2x9j2qhJLkrqd8p2xlSTx/zy/0Iq7TBbPMhX76S/p+wOsFVHFhQS2pDbxiZZsGI7RTWBAp7gMAgxpqxsZRJey+0UbDMzHUZBpVXue6eFtXjbwTmNOzg4ODg4ODg4ODhsMgyThmPjt52ahlVK3PN5ilK55x/PpGuWtCBe6xGSSOebzM5aBaTb5pfkeIVqGYu+fgEAYNqvz0tfviEIxmWXve8JCurWGucZH36uyQrOY9PUKimWlvQ1dehL1ecpMnRDO85cEegmaa2YvKs6ZweFC2lnhADKyp5XOR5L6UPeeyWd25Pfs5I0jpmrndK15DRfMtIp+12zbanSdLCfihQC8kEVlXX+Renqpelb49oyWfppB+YCAAz2wL+GNGXeGTeG8inrqOrFj/wumqdFCy3tAHJW8oAP+auP0n1ZIo1bttsy0vrGGVxLoJSTuRgxJah5AYB8M964/yEAwKQzQxoOWFJKO5uwjTmgg5AJGXR4RHUQ6x8l5/LU4adCdsOB2iWor+ldq76znuy8RedUi94yGz2kLTW3Aw4CALqVdrBjImS1AR7HBO1FI/NQ31XnAgBav/Bw5Jo3aefgiUO/FM1TGHpqqbRVHNhRCIHB60kTV7q8fuA2neZGSMdNuUzHe3kJaUziCHoz06/z+7foU7tV4L+OuAdiE6k/89bX4Jvj35PCFB+PSVeEHIhFrmDGt8f356ORofVaInD3i/T+soTClYpSO7A+ZYQuu1+BvPdHdHKPc4P3WLTMdclYQnTrF9xLjtT+oruBZc9SGvum9HXQ/hnp5gb7OyWsxQAA3PYN+j3lJ4H7AQTWBADA5gcbLdlUFbCw3KvnKXnHt+jazspaozoI3PFDlf6uqcoLxFlkpLtml9VeG59dpzTccSQsHPU+bdl4zmE/k0IznlpL6e+UKaU3D0NNHvFWh9NwODg4ODg4ODg4ODhsMgyPhoPtqgvNCEuK00ombMyZoSQvzGrEUhqRMzSqmorStiNke3iLySAswax0Y/T0canLsqngFeN9QmppcWrdS1Dv2pJQi9xGNH2lH2IW22WrPDjtQrORlLG0rXWssQtW714uugsDd5Nkvmk3pamaqOQSduA/JUUJMHsoG3MppcUOVTXnQIwd8kEV5CmthoNrZPWPNJFAgyxp6rd7GbwFV9MxsxS9TForscsHjZaA+2muEGUE8iu16ZYHNkAc/dU01QmWVUtlpZHQRVjhhGEWaSZ7WlktB64DgOxZgdGzp9Mpbn/2GxICku1/bd+ssK9TqcOi4w3Zqy9/Cm2bzc5WQeUngfYJmeYWKX0tTRWdU+ncYA+aRqi5hvstayRyBYQ1t/GB2kJ26IBmYPPXLkKuuQWZkZL1L3VyCVqWtP4Nuu8WmiLnG6GP1RJ4vxyhFW6EEWqHXWZQckvIF0GMsZizWCuWdl4Ntacs92L0lptnKpMeeyImwF1AWxoqk/TNeLH8pT76udPpMmuYLE2rZpzzLDpfHoeDTKNugsvJda/pdAEApQ4c/emgViEVikMXzlJ4eciXqe2wq2J74nXArybSljNo7WStq6qb8sMRk3eDfPR3WQuV7X5+TAiE21VKiTUP3AEAGPW+lP2bxz8HKpZVc27eKfTL47FaRv9jRJ3fckr6srL2QNQIXpqqmNa33sSWcIBKi02LNTZ1NCdmnQwxg/asxKSWN5+COxNcpPFNkYviTY59udwB4x6UMc9I/PkJ2micWgpOaNJyFI6LzKrvs80+eOLlyavYhleeJEfYBuJyDhmqA/2x57MstnEO9Ay/UoPrPAHsVCsSnLrohlCb5TyE3WLFzP3x1V+/DwBw2SWNqdzjIrxqjN8ON9xEtLinpvWttqJNh99d2veu7xu1OV54L7lYzrp5KV2bvm+6cnA7JU28LaOw7gJyxB9x5ap06QKAMmlLWqCl9KMfiXEUhSMm4yu/eh4AsPAi+nBOMoWhjVy4TxZhxmyo30zYEdfeRE7Lp6dldGwzbq3ZxkqMKUqxFQ8+QuZV8/l9pfiwiaRTw8zJ65yK5c+8AgDI5M4ZY0Ka3qQkeJ+UEqJQClwzN1vxcKznkpzvZYx5iZfPvsyIjXjfkbSEwJMPvgwA2GXhDhufXqg9RaEFK5+mcZA6pnqSw3HC2gWR0+/Ebq/Pf+6nAIDLDr0ieH+pTmwJ67pU85+nSA3sdv7j18kc57CjvpecngVNhZ7RWdneQASouzvHUnohoUW9zYb9nozQTbVh0RKKtNamoo+FRdkbJVJgilcZEaRQu4Zon6WP51+izU/S6pq4ec83m3KMmBxMHxLLX6D2mJ6QfgQJMdSSwxLEbZYl1umwZ8E+LiFjx0TcfKXfnR6HKq3W0VgxUK9CDm8m3lnbKwcHBwcHBwcHBweHYYWQGxMu1cHBwcHBwcHBwcEhE6rfyB5QcyiRu/DuYc3PaTgcHBwcHBwcHBwcHDYZ3IbDwcHBwcHBwcHBwWGTYZjicDg4ODg4ODg4ODg4AC4Oh4ODg4ODg4ODg4ODw5DBaTgcHBwcHBwcHBwchhPeO0vm/86qrYODg4ODg4ODg4PDsMJtOBwcHBwcHBwcHBwcNhmcSZWDg4ODg4ODg4PDcOId5jQ+LBsO2bMcAPDsseOxxS9eoJM5lfVgj/q7CBTb6JhD3EsfkNXAueePn4jNf/kqncs30aUhbDQpJVAdoHQ7Jqd/bvWL6qAKuVaVL1ei31fuofR2Oo3qBACeqr9fNvX1K5TE8qeBJ/9IzxxwibkPAETOHOv6p1NU6RiP5V7I+35MxTj4S2mrCNm9zPyh37n65XpBWuesthO5QD0u33kaLr3xairDFoelLgNg6iGECBwH7ln3GuSq5yn9ue9Nl66qn/BymcoTm5ZfxXXzJwEAFtyxJFBGKeVG91npV3DT/pT+ifdX0z/Xs4IT0P1c91P+O98E9K9T51Rfax5p2tMr6PQq36e2y3/kVnWC0ygBlX513AyA20v1E+6LlX6g0KLOBcc6ICCvOo6y/Pgd6erHfVQIU5acKu9gL/0WWyC84NQnpdR9mNtfVsuQK5+hc23j6dwLt9MDE7cHVtFcJiZsT+faxgXejUrZlIPrzvUd7IFct4jqN32fVPUDAPSupOQqA7puPAfo94v4eUGqOUaPR0jIZ1XbvXgfXdrnfPq72EblB0yb5IrmnH6HwtRNt7nqU34F8oW/0e07npa6irJ7aTDfOPgVs340daoHrbGwgcYdSu2Qr9xJyc0+RN0nTRq6HupctQy5fjEdr32Fnpuxn6mjelY+fjP9PWMvYOWzAABv+wXp6rfWWsO4jtVB+uW5QeSCayFAf9vHANC/Bv7PPkj5n/UrOsd1qvSZY25z4Zm66gL55rpOn8rx4gkTMGP+bgCyBQozbZiLzs9WvOG4uTAaj1hi0YIpAICp1y5S1cjVTMueY+PWi3AdIX2sOGc2AGDsdevTVZDHoV8Br3mmTGouEV7sGhUpByQGvrY/AKDponRzXVZUf3g4ACD3yXtTP8PfboA9x5jvCAAQxdbY+ujvgL7VdKZvDfDa/QAAb4dTo3lZ7yyxTKH3qee0/nWQr92r0v9A/co5DDuchsPBwcHBwcHBwcFhOJFSWPz/C4Zlw8E71i1+vRRaEsC7fZb2WdC7ZSEgvCKdU7vY2b9ejq/uTNLGT/9n6SYoq8A175oOADjtwUr6B4utnALE2K3pkCVLXTM5cUDkdT4AgFwRkiVbvHPvmonrr/gfAMCpB11Kl9Q7FEJAehvZbIUW3PHlrwIA9sug4YgbHMKWxkUQcy5H7bnwoTfw470mAADOvjtbO9qSolqaAjFiCv50/E4AgEPvSqfhMJLtQUvqyflkmxiEl8M6VkqFyjgUGjnh5dHUgCJGS4WqlmS3dxX9Ko0ZchKieSTd51tjwFNSa0uKtfa55wAAY/j9FEibIaWvNSfB9lL38SlLOhbWOgBAzyLSHLalrqClLeSyD5DEUj58A92yw0mQrWNDj+UgB/voj5LKzcsB/6JxKE64kn63O8nUr3Ma3WdpcGIKpN9JBKU2yHv+l44zaDikJZ1d87EtAACdn/09nVvyGF3a8sjYfCOaHd+HmLgjXZtzuDpnaUFCWp+a4PGv50EFLw9M2qlunWqlF5BK675oSVN5/eA5tNwLNI2gY6WVgpfH7849AwBw9N8tLS0A6eWjkvBcEeiaRTd0bWaVhyWrqo/taLQZd7xvdwDA/H+l03Cgovqamg8Dx1wXT8TOr7JK70EoLZK/6B78+/ck2Z1/BredKmO+yWhzFCjNUF+15jcJz7oP2Oy6Z/H0SST93/rCdNUDYDRMLaMgeW7xWUsaWgdDiGhEKgMolNQzob4Yl0atNaLWeiUBtIzqqF2XJAgvsj7YfyeVz37X1Z4NjeWfEpVuSj/TssHtJYTuI7o+1lhP/A5oHUO/+Wb49/2ajmM0HGnX2Mh6qvqSbO4CHlbzoNNwvCUxrBoOITxSqwGQrAIPT+Kw1aLWs9yppERhE28KW/NhdW4KqI8aVAbMhxyr3vvXAgBE12bg+krbX58X0rJahPwqjtlnIt3nB81lKMnaH2hxMCpnU6893rV1qmcD4IWwbzXQMopSZBMSbfLlpStXdRDbdPqB8sWpnBv9OJfSx5yubM/yh5x86DqI7enDEry48YJp3VdvgjxselDlm7a90mL3zaKb9Xowqu8qoDbzSz62MwBgwg+eoGv2WLTGXXhRkVJi1GEnx2dUGQi8syQktXHLQdGFKU1a0q+Yj1H1YXbLZbTJPu43x1mfW2o8SqkpCvWYEx6w5wdq5ONBsklBjNAkNXZ9X/ZntEDGw8hvPUWnlj8NAKj8lcwUC1sdZd1um5SE+q70IV/+N51Tmym7zfXctBFmhnKRMikdt036h9R8IoUX+WA25luDwGA3HbeN44umzBU1X+V87DgxxoQucmz1Bb2p4XcNbV4o9SbB3DN3Znv6ugHmw8zLW6ZawY/xyKaAYW9SAIjx2+HRNXTvfDZlC5iMsdlduvkwcl++CQ+vpv6SZdWQvJ6tfhFixBR1UtVVrf/2eqYFblZf05tfv4rxO+2YIfeUZdRtL9Cy5bzG0ljxXyNkzAj7XZfGTQyUaaiDwhXGTMz+kOqL9xw2Dnv8gTaQcv3rAGDaFOnKKkpt8M68idJQ3xJC9eWhMDMWQgDHf3+j0nDYtHAmVQ4ODg4ODg4ODg7DCec0PvQI7NhZs6E0AoLV3zGopYqspeEYil0yAPRUsqfBTuOiY4qWfLI6UtjSN4YtOWRpI0sWe1eiefoW0WcYcc6gSY5pYfhVFCbPSq5QHJQjsVx0D8SMfen4KVJhinnKlEDkYjUqEYezgfXY/UwlvVaSPcnK3uqAcWSOQSoJULWMGQvOSMaA++AAACAASURBVFUtA5XetF1x+W70fhY+uFjlqSRz1UGIFJJ7KSWmX/xdlayq+wBJY0UptYFQIsZ8Irs0R2sA+tdpCeSEHzxOF1myONgNlEhiW+9d995N5AZth3wheH+hOdL/Ghmf5QfIobl0yJdT3a/zjDGLO+70w1WhV2oNna5zpRuaJZwlyNIHVpDJmJy0C4DQeygG21FKP95RO0kjtkqRTWx2YJ2a2QlaDq9lctIWY7cCAFQH6O8C4jSbxpRGWloSdK+KzUYIATkUC2LPmuzPhJ2lAdOeVSbeyEEuf5Jubx1Nty99XGtS5BsP0bWJO2LKZ35A5+z06Cqi2gwBKE28zr91jNGm8JrFJASFZnR97H+y1U+n1WnyYAfdotKWFFvAJrhB2O1J0uLj5oTICnhOXfU8xKjN6ZhNYrJqWgd7cMg2rfXvC6NP9avH/wi86wI6Zm0gaz8sp3nhmTXCjDN2PF4D7E6msWHig1jNXVZUB4F5RzT0qNDWGkarI2qZUSbBb8CyIgtqfQ8kgNth9vR2yCeIJEFseyJdZIsHwJBE1IMmMAiboA3Rh3cuTNrh8FaC03A4ODg4ODg4ODg4DCec0/jQI+yUB0BLOtinA6UOBBw+1a/QTqjm2c6gCWsgnzTS73qS1iktDUgCJuygEvchlNTRptxTGUOEduDSpiNUElPROR2/u/EfAICjTrfsqekoPv86jnPqDKXgCTz+y98CALY/vV7NLCipN8bN1WmJ7U5QSau2q/RpJ1pjgxxT5qZOvP5/ZM85+chvBEtZR4OQShqSK2DVrTcCAMYc96P699vpj9kKp29r0RYDNSUzSWWs3nEdACC/jZLM5Wt03EZxx8/otwEHOTmwAaKDaHXlYqIqZOdhFNssPwbjTxXn/F4aNVZdD1EVSh9hG/RGpFiFEV3ZHmA7+GKr1Vaqr57wQ7pl2eNGi6El6X1AVY1b7ueQGPzzVXRq+xhfC8v+WB3UMLu330lQClv9m6KG3uXsdPUDgjSuLElV9W76FFHQyjcegpi0s8re0i6q/sxjTMKD/9CfKdk9z4uWXNdxIyTIj/2VfudflP4ZpmO2aXhtDQQAVPogpu0VuF+Mm6vfiZixn05u0RXnAgCmXW98WyLQlLo5ooG285Q+rVGA6TOsqZQSK75O7Tfu56ekqx9rx2TV1Ec518olj1A2Y+cCxfpLtGgZDc9TaWhqXaU16JgCyfTNPN6Lbdmov0uNOVOLieQTIYUH+cRv6NxWSsuofU2K1hxjrxPsL6bedds4YBnRnGPbIH0v+UM2JiHX81WuCCx/IdOztn+lphvPN6DZUPBGTQimX4c6ODNGjqt/Tw2M/tajenx0X0QaxLZL/0llY6fwNFAEJaItSNoh/UpAY9UwmAAlRAri8NbAO2t75eDg4ODg4ODg4OAwrBhekyrpG8mKkvRrtqpyL4SiWdO0r5atqb3rXT1QO4u4AENZaUmX9DWyw2ZthjR0cazZ4GBkMZIiITzI7jfoj3ZF4yiraM+H7zM2rUb6nK5kxm7ZPNBabKCOLG3asMQwL3VOp3PdZH8sK/0QTBdq2UtLhGynhYdJe+yqr4bLZ+hYWdIlUklXbanQqN3n170/UCSLqnDyNYuC53SAykJE2ltLY3btTQ8CAE5XgmORG1oNx+2/pmBmB30s/TNaE9ExSUvoxbS96aIOUCeBQUXRyFLffLOhtrTaYWAFMZcUImOsXvAmu09GmerC6aeWGwZoGYN0qnKxCmw3dmutaeSxJAfWa38yWyOba24JnAtWgiljQ4HVahUtpn651ozsRqH0ZFXZwnMZmIp48m4mkCMzycmq9h2QFmXx4GqSziYtBnbZs7LolFfT3JDSypugtZwSRi4mAz9oHgUMqACV3I6vP2j1Z6WxKrZixJjW2DJTXdQ5TdXqaX8raK2kMJoI87C65KGlM6OPA/uB+FXTF7k/jZ1Lf/eu1GtioLxasm+o5f+ziNr6SJYSc5oiBzFa+QNuUOuMyBntTCDt4Dqh35WXxwuv0HwwOlMlVTqjNgdUnThQI9PsbvmDPxmGJ8sPIOJ/KDyIPT4ULFeAKngw+FyukEkbJ4QA5mVjxNOamZ4V2odK+2HWCcDJCIylcvDDZqhZqsC03xmg32exVdej7Svk8yd/pvw2j/uGZomrV2YOdBrWcNh+RVIFChTNGbXbAKSiYhZjtsz87JsC5zS+6RCnxrU5nWtOeCGsGUzfSI0M2oH0gZtNPlw3L8ZpOmEwSukb9T0j34wn19K9+8dnlr2AofzvW07Hm2d5nheEaXtHy8Afr14udkGIJgZccz1Frf3gh2LiezRIIWtvzG77OTk0H3xWumfjNmYamke+Yj7MC0nxF4BixmaS6gMpjVM6gIbooeM41HX+ms7ZNyYfOm6G2fDZi+SzT1IMlbSEkpJNXwKmhbXH6NKnXgYAzEiZfrAt1HFOmZdM39eUg9tafQCJjkmRPieEh8f+SeYtO308Jq+YPhrXh5LmoJdup03jrAybxkAZYtpRlwXKHEpFwRaj59BHeqhMle6UkZU5z/BHu19NNNGp9NI7zrThsIQQhuqYncVV3+lbZcglmmjjJmZaM6Z2wm7FKy/TxiQ009IzIQdliiETMqEUObOB4zx9Y/a16nUyDU69fWRCkbhxoOZZ0T4RUhF1GBrdXNRB18tropOwyW4gqni7oUXVAp2BDboOsD4ugeBHf3/Y1z4NuE+UOkz51fvc6lf04SkfuwkYvy1dY0dyL2cRN5hNnbzhHDr80K9j8oo6zcuY74ikzbL8v0+r9P+Qqnp6IzF+O+tctu8N+/7e5yiGTtsm+ggtv0rzQJZxGBB0hs0qP/Bz+h3sSV1v+fsv0sGHf1s7T2ujwQKh1CaAtyuilrPele5+h2GFcxp3cHBwcHBwcHBwGE44p/G3AJTUURaaY3e2bY0E5suApqF+K5aJRtS8yzOBpCyNwPpyUnJZpSjRTl3ZmFc42G0k5FxmFdwQAGTLaJUv0zaWLbMBk/GOXY2IzdJjq3HZKPLSqOBFvgT/ESXZYXpAK1qxDQ5smDp/duRN6aC72ZjsFIDBIHChTsDB1lY8o80vxEyWFHmxk+MOh6WPkA0Aks06RkzV5UjCtPcckil9nU/MWJO2NJcjPRdUPx7sgVR92tZcTBxT25grVvoW0m4KS+MZV9exU+Nk7vUQNUGMg66HipotX/o7WMsjLfO+4ujGnUkpn1yidq7Ylc0QJ5C2/c5sOmCAHHT1OSat6DfmZZb2Z0PN+dSYVMU6LSttpii2Gg0gw2r3ymAFmZA0tjWRQcWUQ6r0q34s5erSBGsZrZmE3TeV1k9p12V1UPfduH76Wk8DH0bs+O9XrPqq4IpP3EJ/vvwgMPvddMxU1YMbgpolTuPwSxIyC40J4UXIaoQQVn+JkZgf9Il09eIcub9U+rSzvuT5hPvGwIaothjx77hl5qY1AypMnb1Rz0ejqav3W2zFt3cjc/CP3Uka6Vq0wN7Zv8mWp3qPksdhHfpdccrVmdJ3GF68s7ZXDg4ODg4ODg4ODg7DijddwxEr/eOd89pXIbpmRp6Z2rZxGo4APWSMI+/kluyS94B0rBryarfsutnxNkAfylSGLBWREtMTYsOFfURqScLDEmz7HU9vzf4OdXpeXtsgQ9Exyrso8NXPvnglPnjn0uCDlX4TQMySLG37swdq58XSMduWOyaoT5KN55Tv35tQm6yQOj+x3cl0qt9QOkt21rTK8cIGqvP2GXOKBElEvERs+VrqN1MzpV1b28aBwSrXfxz5LiX1VtSWKLYZcmNbOnjYxanzllJCtI3nvzixZCfkA7M5N+h35lcsibRqO9au9K2GmKwICzS1pQAUtaVUZRRCYPxeewfSDUhNLadkQEmJdZ52XWQoDTNe23Y9IFP96kHaFNwhgg4xbS/Iv3+Fjg/8rH6mOG7KxmesbOjjfHRy47L0UAV29PcKZv7keVIqqWfvSgimMOa2yBWt+VSVodyH7ecGqTtNP6lChqdPvwJNlsCSa7/J0l6F/HRkFRO3mpatfjYFrzkZrEu512jgeD6s9EH6rInL6bqcvueIYL10XcpmLuV1qdwP0TFR3c++TN2Qa16hY+VTIbQDtMQx+xj/j9TgtTVXtOpL7+wLp38SAPC5L5xufBi5vzaNjJkLcsCEhJk0QXNhp5XkC6Cd19OC65QrQTDNP88nXB6/Aps8BQD1qRDdvxACKKTz3dPZx6x98T5yClmokDNACIGP3f0aAODynWmsL3zojRpU/Q36nyrNRq3gqhub/psG5zQ+9OBBJZc9oaPAcmcMfgCpzsKq8M5pJmo3O4AVmjG2iRfwxrjhhRD6Y8Tm0gZo8IYZolJBRU6Hlw+YFwHQ0dXlnd+F2OU0Om5RjlHStxhZFKqDOOUMMiUxHP/WpKV56GlhkiIXGJDqJtTakEgpse/p781YQRiTm/WLTVFGKnfePqr/kTuYnVIgmjHHJtGTqtARt+M/OMMsJTUGZszHuUaCQ21W2IuvLqcyHZMD3WbxaTIL/0aZraGG2ZOFdRnIEyKw4r8EonMD+O7NT+GCv1D8Ce0gO9it6xsoYxa+c79i9dN0HPNCOTlnhmVSoT8MlRkXLN54w+YjtDOufOwXdHG7k1BeTn09LJYQwggPNLxCjbqE5jrLPEAufzVbvez6xF3RpijRczLfBPEu+tBjs1WU2rDiPiJvGHdaA0XhPPTHTLBPCSGw/oF/AQA6P5ghQXu+47HFa4Fi+xNNI4CK2lz4ilVKCICdTtnhutiK0qgasQJkFXoZ5Hy8gslLm/SUodsxF9xcAUChMyOjjpoPyUE9ZCbCa1KxzZio6Q/UdqCbBDq8MUa5FxtWU3lHcARzjugtPBPXiufgFlNW3V+au4Am9eGvIoRLHnuDPehbQ+8y04zKZot2bAo1BvafoNbuvT5sNlP86+Ug+YNZr88Scq1iOBodZxoUdhDPNjdKKfWGK9OcBgRNpXgTqOpy+V5zsfChNwJlk368aZVYcE368g72xBNGJJCtiPf+MHX6WcFjYuH9LwEAfrHfBJz8T2KMsoW7/D3HG7TM+QgvUcioI703lLrDpsbbbDvo4ODg4ODg4ODg4PB2wrBGGhdMf5dwT+CclwM8papcp6TqIybj2XW0T5q/EeqzsCTQ1nS83E0n01J9AjBSNyEg2bGJ6VOZ332v86NSd+lrCYmWgOSb8JMfU/Tfs04NmiIFJN7KZMBWnwa1GfH7fCEEbv3xzQCAwxbcmLKCpgxi1OyIs6Y47AoAQJf6DSBfMlI2rfUA/nQUqbAP/euymKzSxayoJR2XUuKFBeSEt/mtDfAcp8jP5iiXq1QU3JIhxmwZgtFlpGIq7omlaekqbYQKJcaBUar+dOE9ryLi1FpoNjSgmuIyhxfPpFEy6+ZoG0ayzBVitVlJJlXLLtwDADD++u666QfqEiNLYUdGmS/BxDLQhQNyqn9vd5JKy8P9dzwNANjn0zXmp5i865bRcnx8+Q4yK5z1yVSPEthkqI4DZSRf4UGytJmJNyoDqFaGnrzB1lxndqgGDM22ECaiewtT+iqNTa4I+ciP6NzOSj1jmVz4zOff1InC2Mmx5Qtol4UZsP5/fkandlKxBrwYiSzPafkm5MZMjl5PrJ+J3WLelbqmnIzjTVKEobRlE8G2cXjwNeoTk5Xk2Da34vgycaQI4bQBmPgeWlvS3FgcDuvdhvPc8w+q7FIa7TWTxfSsxj9O3BkAsP+fXqdrftW877iypzAXonlGvWRFIGHH3pEy4zqhzbesWC76kpL4P/BK9LF8SZMsaLMzCAx+g8wrS5/8R/2sa2jvk+Yg/8qjKcsL766bfsNQ9Tn5769B3kUUtWKPcwGQNk8+eC2d2/3DDWcR1iLZ3z/yKRXRfo+PNpz+sOLtZgK2kXhn1dbBwcHBwcHBwcHBYVgxPD4cyhnztmPm4qAb7wEAinQMaDtbOdhj/AH0g76RpKso3P7L/8KZt5JUUNvQ6gix8TbvaSSPRiLk49j/eyhlzayirnqOfl+5F5ionO5GkDOmXELBw8SM/Yx0Uks2pLZ1ZaddufolnPEtCpAjw1IdkTOSnoy7Y9sf5D3f/G6mZwFAvnQ7ZTttb6DK5WLnSw6CVTLaInZSzDdb0kD1np+9FYd87jPBcgVzo7xi6hgJrEh3Bm+qDmKzT305Ra2sdENahHoI2N8qu2K7bIt6hs6SNBqUTuBF5ZS+Q4Z0dPn6VkEqrZyh4bQdHZWfVI/SXHh54B6iHBTvvpSuVSqYefG3VLpRf6o4zUVSxG59v1XXsR/9ZobaWeNFeJBssx7O0ytE3qcdwdmWoO116RfoWNkey8WK6GDZsxBzjw6m3zwyvu8oyamUIpi+X8HMz3wjU/0AoyGR5b4ELYeIvGuqoxqb3E65IsYfQf5c/nN/oktTSauEcr+x9w/QlAYpSIPzbpSyt+vdx2eonYKyxZa5YlQCzu/Yy0HMWxAsX98a04/Y4bpvFVAMUk5rVAeNc7mer5ogdjlDVYfnOWF85nLs86TmvMFuIJ+NoprXRNE+sTY9tZePEEgIIQBeO5nOd83LOHi/zUx9AEih+oX0Iavsm1J7Xgv4pimfQ36nA1+dj+33aoCyVZUlOZCpNJolthIoNGP/P7ys0jB+HUsvJr/GiT9pxO+J5x6uY9DJXgiB7q8cAQDo+OGKdAnyu773SsiZe1IxJ5FmRq8ltcYn++2oPifXv47CKd8IPKvH1xA4e0vpQxzzhY1Opx7Md5SA2Iu0DPKms+naCT8CmjuGLi/2pWK/jUIz0DxiyNIfFrzDnMaFTPJKHSr0rhzS5C7bcQL9PrxkSNNlXLUXbW7OejiDKUBMHdOwScU5z0opcfk8YgUJ1zGts20SpJT48V70Ds9+JH0dJUfujYmibJukhSdIezGzNz2/PZDqeNTtQVYr+oBJF7E5lj0IAGQV9xxO6e/5z3QmI5rFyLNN1KIfbWnL8bsDqR8d+belkfsj5Y3JK3w98Fy1jFsPJjOOw+7OYAqQMBazmjsBwF2HkoPlXrcuj1xLjOqbsh8/fzyxZaU2i0tRv3p52vjzQZT/ITFmf3FtnnVMPnkUvb+5f01v1sQfq40huiFY/ZHNAQBd33s+03Np71t/4RwAwIgrV6UuZdp2ZCSNp6T5NAtqjkUpcfXeNJ+mXTP0XJq4/MqoUCmwYTLv+Dt70CbkY/dapkrJJQimYceoiKQvcd18mmve/+DGrRcalmAgbr2I9iOJJWeSQHLC1a/EZFYN3R80KzP3herI41X6WHfBVgCAzh+vjq1PBKqPxjL+WWt/HGlLGEII9H+JGPGaLr6zbtZp5xo7r8p3DgYAFC7JIFDNOA7tdouLg+JfdRwAwDvrlvRlqJFXbPrXvp/S//CtmdJ/s+D/6NA3NX/v7OF9T86kysHBwcHBwcHBwcFhk2FYaXFXf2RzdF3xbzrJ1JS8gy40AyWlDmOHKsuhWjuw5Ztx6f9cRpeZ2pEd5UQdTv+kMvrG1OCMH2Qz4wCMWi/gKKdpMJXq1XKEtmmBmSZTWtFJ3z/HSLJVAenXy+t3oZ37iq0RaU7AzCHMY9+3GkdtnY3zG4CmKIzl+E7kxo43o8l55ljdyE9oCZ3wkvfENdvZryKXVQHETvhJNK01rkXPS6waUOe4P3PfkFVtRlCvn9a87uWxMa6+Aa1TiBpawqaU5fgjfrB96CTmbqEiFXMd2Wyjd7Uel5KjzgqhpX12HBoz9mwnTMKELRrg/w8hMg4ymCK++5SD1aOqjGxW07MCeO0+OjdjPv02dRg6Uitvo5VSJibsBO1XsNWxjUu4hBA6Ta2Vk0wZWWtqj0piR15+G/2hTHkGvro/AKB49rXAsifoKa5jqSOxz2pOCEsT137Jn1PWKBlRKaeMXpN+gFiBy7Lr6OBoCWqfuT9b5mY8n+t4QzkTx0CZw0iLFGR2R8bRaGlwoxJ+Nk+MMaURXlSzNtCNI6ar6wNqnWSnYmFMwfT4yhXj25DNt0Imu/JX52Kfydllk3Lx/QAAb8ru0Wte7fnPNn3S7dSzEl0zFa112CzZimQeZ34UjPEVp00BUO5H+6wt6tYpDrXrAMRpd+I0yFJK+OXBjcozHkbjUB2gfpDF+C88Z9UvQ+36+q/dB2xGJCNxJCiNrLuBsnYvBzbfpe59bym8w0yqnIbDwcHBwcHBwcHBwWGTYVhpcbu+/iBuPZykCIf+QTlZK2drMWmXqATSDorHv32rgbGz6JglpxykqdiK9LbGoTLa0Tq7pmd6FoAJ/Cc8LQmTPktPVPlipATBCOVlff+0Q99D19cTLaAOlCOEkbq1cYAiEdDQ6HTDmg0YKeDYdx+ZuYramTHG7lbb5Ho5pHr3faux42TlONerbLubVYBEr2AkyQUTNCpNgEebRnbW9PY6d4fAzsu+H6/FyYhTDqW+PvD1gwAApQv/Qhcq/YCi+Iz3m0gX0PKQQ7O4iwcRkBixBFS966CTdzSQnC4nPHQcfAr94QcdU5dfsCvGfk9Jx+Nsqe30I/U0mbUccHKa6iRC94lBnifaEu4O4YDz6JdJEdQ8JNrG4ounURT0zz5AAclgBYKLL4ghptB/739++rJoWHbSrC3REv6sLnlSR1hHB9nplz71N7qy+iUs/zY5fo774X+jT8ZpU8PwKzqQXKZSxfhwAaE8/IqhHuc2lTLgx0VPedhtq45gulZEdt3//h973x0vSVG1/VRPuHnv3RzZwLJLWkCWFZaMZMkGgomgiCiCJF8FlWX1BVEEETOKqKiAAi9KElSi5CCgZFnysmwON89M1/fHqVNV3V0z0z1798Lvo54/7vTtrq7QXV3hhOfoKNGDVlRzla7QogPuoUNp3ThNuR/bbT0+YwNZY21RejrbnERC2pvL40FVtY3jdKmlPjN3xrSLVfOPEyps/UE88rPrAAAzXDdUz4jud0ivU0voOeJ8uQ/FzZVD9hoVANAO0NdUfayv6bulNYQ5iBnbpKtTPP+w0rBjd2SszStpv+0EvZ6w55Agn325Z7SxjbdRl7/RfEjlVH/fweTztPNfaH0jV7wAjN08UedUdeRn2D4OmH/ietVx2OFpcT08PDw8PDw8PDw8PIYGw+PDYflkHHj9EwCAi3cn6e9pN91J13IFs9tj6bIrOJkMgXZi/9ESKSXBkQPrkoH1stZVhkDLqOz3KV8U0TLK1Iuh/Tpsm2OYcyw5YKFbeQDoJB8XDkxkgggWgGZHG7WUWmlCZGjKY4kV0/i99QTQMTp94xhs1wwBCaWtYd8aHawqiNlHA5DSko4om9VVL2PS2ZfRMWu5xs+h///+HYgPLowUHclXa1PyOj9DH6keYqkXo077efY2Uma6z8YpHVMzEZUHUNzzU+omqtOrxxEb0LTfvGxROlsBurQEMoUcICwht/sn6qerAtKIqeep+wzTt8qU0iyJ7tv/AADo2O0UOtVDmr5xP3pG5x+RKouofb2NJNNLiL47rgEAtB3wnRT1qSPN1FormfQ1qMJuhKf+Qic/QLTHyKs8gzy+9mPVR5l+t9RrNHKcX2UwYROvqWzDCvA0UdFik31TtS+St22arjV7TE+dVkMngHWKuWnSXPpVPmdi9Cy0dHAQU/ZDatIF16I71s8z3wSsS7K0paoX5WSVEaWIBURMuw0aj1jarYK7oakDnZ9T9MPxAKS5QlKzDqmpzLXWb8I2kMsoCKRopfB3+v9xc9B2fHZ6Y4ZuD/sk8nwW5NONNSKHHca5/Vvk6lcguqbTKZZW18k3oXUd6Ma88Q3IJrXWyfisVS2jGvidrPwvsKuSXt9yPv0erqjd80WgfzXl2zyyar41fTz7VgLzj6vXorpIq512Idfs1mjIUl9qbUe19kkpEVTJPxXCEiTPtQ4fqdRQz2enw/agf2/5BgAgOPgCWr8BNbVVKQpYj3s9NjSGZcOhOael1I7hp9/7Ep16iLj95RZ5K24FbUrQPgGiixzFZKgmhXwz8AgtcrD/uQAsk41iu56ApLXIT6iILe715McigCf/jw6n7Zq6iWI0LSajXOcxvnbLoVGnkKpMQC8SxIhJuO2HvwIA7HcAxeNA0TKP0iZnZuGRiDwqcpaZlcqfzUFmfABPf/UzAIA5h12auo36Pdqbw6Yor7a0Iv2adYFj8B83B+u+sQMAoP3UK9XNarDe63+sDE1EaJ1v5Bk63h8AtIxC368pn7aLjqrfNiCq3ixHJ3+dJOXgKgotGLzzdwCA4hnklDt+i99wJo53mNF5LCigcvfv6XDn07Ldq8pjp9S403Za8gUhAjOJqUUqlJmDXPwoxERl8mUvEBmReBDVHPEDBE3ZommjlkmlHocqVeJlWKZOCmv/9kcAQOc+X1d5WJTJ846lAx1jxizGtfQg32yu8yaWJ+ywgt47Kf/2Qy+p1zCDWvTaKpK4/R260tmkFeXbaWOe34ziEGhT1bCMERfQWCxXLaL0hRazII7QjbI5ksqfOfLDEPLOy+na3CwLuuR7TAgySn1ACy0w9RibK5p7WFgDgdfO+wIAYNqVb6gsLCdvft+VPnNq6s7RdJUShDIH0eP0xLk6/dLvfg4AMOF3KYUADtpUyQt0JlJpGZlYqFfJDIUmFiopwRRHau+aDvkgvV+xw2fpWqEVcedpGwnzp80OQqHpf6qkrg4xcmN1EDgW4ulMn7XgY8p83Q7sSOO5fJmElWLazubb1kKOapvgWLn8PTd3AY9cocraIUXrrHETMroJbgBSSuRnkMBNb55Kag4KQ/NN1kGt8TqYNqehugFRwVvWeFX6PikBRZUsPvkzOqnmjfCXR0IcoWI6aeFlRuIfaZmHslD63Q7vNO7h4eHh4eHh4eHh4TE0GFancVkpJZ3A2QH8mZuBnclBU4ylQFHyX1cB25Mk3jjvtQLT53LGiXK0SUOPTbcbU9FZTsksOYg4QG60dfZGJoImwUjdWMJZzcwh3phk7wAAIABJREFU4nANIKxg7gylsdB0o0oavW4xRDs5XBkpVWDUnfYz4Xxj5jsQAlNnj03RqBjyyozKjgAfl3JIqYU8LnMVfTEI0H4mSXdxm4qwOqgkwId/z9SZNTdhyUTArQnjGN9yyu9TpHdDm7JxrpZTftqAS5U+Ze6h3kPzWXfT/+UByEfInAyb7kPZdk5198WqmgaJSi9JwBpyb7ffUxB1ZpZSarNAyZLrXJOz3U2j2XEzFhRt0jzLfIV/K5BlNosz0kH+LmQ+KcUrjhyTqVk1TRnY3MiVJiwn+rIQAiO+ci3/o6prnI3ls3+mS3MomJX897UQ2xwVTV8egPy3ymPafPotKvOsII/WU69O1a54vaqdM+Y5vUBT0jnedW9uB0UgEZc+55r0OxSjKJL12q9sgxEXPhNLb2ttooHhRJCDeP9h9RuVCjFzCSEiRBsAaH7gPmtF2J56mjLDiQeICyumHTyelvsg+0jKjOXP0W9zF8QYRZvK5APanLQV4754Qbam2P1JO63HSAcG1hkNjgVjjqn6q5QYPVWN5/G2ABDzKBiafJ6CfImNPwC0j1e32mON6tsRk1UAgz0YNa0BibGmvq3yvQFOIhV3XjmIeSqqPI8T+nkFWLgjfVMLHnmVTonAaIdYuxLkqnMq5IoQ234sXV0YVl8SQUybwhTyWRytWdOlNBxiDK2DUO5PmOBWG+eqmQIThXYdUgtXfo75J/6tp9V0CCEg2Zw0NpeLz1yFxw+lPjb3hsWqbKustKZq1SK7v1vhncY9PDw8PDw8PDw8PDyGBsPjw6FLswLf8c5/5t4AAPnGv6GlTkqCIeYdE6FlUzdi9WVfBQCM/H5ScqbtYVuV43f/mip1ie6E7R18zxVnAQA6tj22fpsYrkBGqgxZqe4MLCwHZa2JyRXx2CKSXu8fpzTsmp6UQMXLZbBGpW8l/bYaafFrL5CtYyarTl1/44OQCB5XpY3xY5lrgrz2bDq3P1FviglKs9Q0AnLJU3SuUwV7ikhrLQpQpxZF4abz6Pekv6ZrXi3NRUpKSTuvO+8nadsBVmBKAJCFnLGn1nTKImmjDuH0K6DkAZ64/3kAwPxUNYrVzyl5M3b9kiV20pZ6J9vev4RoDdtZEqt9UyoRjR0AkpjzM1j1MiVvG2flm7SD7n+bpF3rQwURt692SuSqONIuX0jS/3E/WRTLq0KSYljf9TZHmhs1AUYRYquPquOo35iUwLrvfAgA0PnT7NSxLug2OLQb1dB/+68BAK07fl7Vy3LUZu2TpGc24nvPGjtpljQXWkzQUpb6WxLMwb+TbXzzbmdmaIfL/4SvcRs7zNiZN34nJvii0G3ovYEcjNu2OyaWp9DptPS4uQvy7f/Q8QSiSpWlPkNhawWy5P8H/kz5t8z/YroGBkm9pJ7rmLRkoMrcFZ9r8k245QFy/D88dk0WWsxcN0X5oFRMgDl3AEAeG9S1trG45wGiov1gzUbFUEuzX4My1wmRg1RzuehUfoPcroF1OOeSL1N+t3+Lzm3zIUC9uwjFuvYJVG20iEjYaTm1VX2NQLFZKWSFEBh47lEAQNOh341edAQTrpqPg+SE7yu9SEQIWXQANX34MvpwAIBcTnNWwD5SFrb7C/l3DF5MBBqFL/4fUFRlMGFNDVIgIQTC5YqAZvTszHXz2PAY1g1H/GOwz4m9zjKLVpea1Vrw/fw+GhjOqvFR64+hdbQz0nAt1dyfH6QJ9ZNVU9Qoz3WtHmOMcrzWk0Qg8MJaOv5gPAqtCDRjhZ5sbUfJSGLVbhXzQW9QwhIeW0ntz7LhqNnGrANskMPlf6BItJ/9PC/crDYoxpwkM421GKk19uYC3PwHiidw0EmZquaur82S5mDpceG1bnc6IYTZjObUJjAswbXgrvVclw+knhpToVb04WptbZtPk0MiEq3Im8mO+eWlBNaS027vxbQIb1twn3FwdXyTLfP3z9iKJHTdre8wwRwVVpwLo9FbbunOS+SBtphZYlAAut+m/B/8JSXb7UtmgRpbZIogh47NsnP/Z2WHqWeO0HzSldF61VTzC00MsHA7ikex4P5ngaISljTFTBGlRPGEX6WqZ13oxaKq32C3WdSyw7UIjKnLIM0TsrkLubYOXR/AenaVUmK+kX0rIaZsb8oAIJpGGLOtQI1Jq1+hayM3RqEzK7NhlF3PhjZxbB4ZiWau28c5WGPiC2vcm2k7zg2bxUkZmgj1jLBinoMdBwsAwrKZj9I2z8rHHjMTsJnd7DrHEeSAJ5Vp4q6ncqXpp9hmBDi8uVh0J8ACLM5vsMeMNRyTgokzShXgmZvo3OR56Zo3xA6/5R7qr82NOGNXqU8kvhObGA4xqhFUOHH/b+lXbThcgsLCaSQglCv+CzGGNg4JUhxnPSTwxA30z6YHpavPOw3vNO7h4eHh4eHh4eHh4TE0GF6TqhqIUtYZSW88aixEHq2N1lpWd+SypQTFIDuXcy0pQ12JZMJspnb5cU1QVemCluKpdrPKHkAwxBvrWvEPbLW5/Sx2HccStJKuFwCSBCsVqo4rICWky2k8EZ3emHbNGT/03VtYFI/1lO/T2zP0o7AC9K2iY9VOmSuimiRUhhWMa94wnON2lPq6mqvtMzhalvux8AMkOV5w9+PqpEhIeSLfy/s+kj7/LIjREVfTQorDzkmfpwiwcI/tAAAL7nqMzlUGNQFBXJonpQQOyE436tKEpbqriiQyvPp0AEDwxZvSla20CuygK5c/b6I+K02D1GZNErj2K3R46l2Z6pssOtonZa5oSfyZgnzAmFcpqlQhAohcTFNsx/TRGg5h2lCKRaXvWQa59GlKtfFedI61OQ7NfT2kkwjLCNGHPufIoyxTEFmwNsjW5tkaFD2u8behys4V4KBEqY9aDs4cjyeoJL7FqtjqQypdYOoM0DvU1NPUnu4rF6LjfEWGwPSyFgW35Pbb8Vg2z6S/GXJktRJwmuDWyj+XLf8Ngl0/G/3fduiPWGwA6F+F8CYybxcHknm0HY/LZTkg3//xDVTxDQTvNO7h4eHh4eHh4eHh4TE0eMc0HEnNRWBsdFXgPzFmMwh2frQCIG3emU2ya3wjlANkecDSDiTt7Lfqyi45Xi97zsQuV2BdLQY7h129e9fPtusx+/ogj4H1MOd0O/o5qG8dsDUd/ZoZlaVM5p1Izj+wJFHVJHCRAoyWqFLeMDarWvsScwaPY6PW+v1IP498s4kE/Oq9dG2j+U7bbcbEUSkpJRuBpnysIxV7ieoKDopWC2EFXz6QKJ019WOhzUhbXdKeV8nPh8kl1geRb8RF8uDAy2ccTMVfnYyWzfb1Im8CCi54iAKaaslx04gkjamCEAJLziVflom/787QkmxjjYjY0icRDvSnyIPLFMnxZOwWkP+iIJfCdpynQhEquuv1lm7FbdtFYBzx+dsRObgou5/5JzmBb/vlmDat0GK+Qa5hWNa1lWuJtEB0TYNgDStLZfl/IfDGQ+SQOz1lU1L74Wj6aONzlPCXAtBZdI81Ea0aj40BzJirHa/XgvtVghJcSrQ1slKoykFraemDHIwW19BzJ3z3pDTkJyNncCZQN+o8uD3t+35Cj2FyzeuUbNyW5jivyEjsMYc1zKmbV/8dZvFvyLW2RfK16fKTGu4yEs79dRAUNwxlrBAB5AD7OtUmqxCT5kb/t7Q6CSrfKTvoIIzfn0+Uuafe/2ZNTZAY33hwQ48Nj3dsw2HYjWy1oBrwRlNsDvnKXRDK+cdWH86ZUl+FHY2CG1+oCu3ciY4JifvGTorF7UgB2zEzoeqUHP1U6kEiYmIUd4YMy5gz0rrHbkNY0hOpLKlFTb7JOOiFlomRjv7L8RWMM+CcruxKcpfzqa6fGnAQ5JJMEpUBSG3eYAbHbc7/qaorMyKpCaJcglyquP4fU7E6Nt/TmDLYAzxPQjxJWhuUGQt/mbGF6cAObE5mLAsPr8jomM8byem70f/lfiC2WORnL4Ic/vMWLWQ3Spt/Hej29K2IRKwGQGxLromzv9ZCOarqlytfRMunL1anVF7lPt1fnM9zIMtC3CrZtSnm/lUpO9hcQsu0w2zop2yrYgLFePVlWNFsP2yeIZc+A9E5hfJQRBAqd9UW5cRcVOZGIoex2zQwQVrmpVWTOMwY7XbYC61cS3WHzMRYVinpDZbuG6VevdHQC/QRk+laeQBBE5tXpYfTbFGZOWlzGCuugG5hxPlfjaGVErY5hMaORPu7l0K2dEXPlXrNpko5hqNtrHH+53I5vsxgL6buv1/mNsah28z5V0qmTH1tENIR32J6W3S+0H1fhglzIwAQzSMi6REUzDwRjyEhJWZmMQ9lsGO6TdbA5fUqVramdmusqWEaJISJHm3VS1UY8o6L6Hju4fQ7cycjVLA2KJr1UMcwCc0vmwWmRhpTJreJtcvsOCjEWK80Q5zj2Qt7o2Ztzjg2RuJemXrj0wh4o2G+/0nuhCnYplw49T5iSbv7oInY45al1ROW6wtP3lXwTuMeHh4eHh4eHh4eHh5Dg2HRcOhddqk3QW8WddZU0sOApNVik30hWRLC0pn28ehZq6jvlHRbtCQpCSPmRnCYILURHal2VGIJcqEVvWuNJCg1dORyhzQv4tBcn/pUigDzN45JtpiTvVIGVIwRuZZiIIiWkYb6NhJx15JewdaqCGw2OxnBthHoPLXU1rGHzRWNdNna44ppu9DBoHLqY/V+vlk7kg/8+z4AQNPUbSwTCUvrw9IxjtsxdnOdl1Dq2A2FeuYQxSxBZl2mcIVW4v8HjJRSxWrRktohhH6XzV2mDnGn3Di2IMluTdIE1SfXfvdIdH73STr35DV0cctDTMTgHKv8WVoLYJPd16st0ZPsBJt8MUQfqkovG83hmldJsjY27qAY5MgMjv6jv6M3gXB8/1qzqDqEdj4evxV6Xn4RANCZuKsGXKaE8SSO2DeAMG203tea50mb6CJ2jdwLROIXaYlpk6URVhGsmZIWxQ50v/BvAICt80kLYcenYa2DbVoV14QEeSsSMpNlFNC/hN5jC2uoOM/mTtPvuB82jYB8m+oM1tSUekkbBwAtaqzlmCO5PMpvU/7pjRxVHw9DPf5rSTXPB0HOmht4bszresi8oZPV+mqtdWOKW2FpR1S/tuZg/X6LbUbDouZcyXNKuR+VRvgpasRkYq2SyBUTEvzot2tZCSwmIgYxcrq6wtr8ArDtRyLXEJaNxkbFjRGFVityO2talIY8lweW0NiEmNnPeqHcb7RUNSCE0HSwrmv6Gal3+K3tp+Ocx8nMM2JdgWrPUiD3xT830IBsYM2G7F9jkWVY9VtNRBNi3BbZMlbf9O7XPmMcyff7Bl3jWGutYyDXUTwarcl6t8M7jXt4eHh4eHh4eHh4eAwNhkXDYaKetta0e487cMugoCUS8imSiIqtjkDPoJLOOTQbtfKNnGMpGUvD2GGs0IpKuQESQEc00ERT6wQctNPd9hxpTI5hiRzTM1qSbaG0NGjqNNIxTgfjL+LK/4H/kNPuUBEB1gwKKALIhC2rxG0focBM+/31LXXKaH+EckJuPvtefS3xXIUACipw5Pit6JwV3fq5T1HQts3/2hCpYya4/FsmNEhbG7FfZ3v5NSq6M0vQS32YnMIpvRG436Vbm/PaVz8MAJh25euRdGRPrCTMqs6dFz5lcpurwmpWjE+SljBrP5U8lp9PvgFjr6gSdTkDUkeJt6T4z7xCUuw9XNLauJ10PinJjNB9c7tUXxVCYPHLNO5k0nCk8OGoB9u/o54vUibwOMi/PcsQVrJ/f/Z35ArcSP8XtZZXa1bsfspaQTShac5OsfwNWYPJ3+gn1l1MdM/tX/mz1Z7Y/MS+Frki8lvsmL5xgNE6OAP/1RpLhSmXfSSaOo0GIj6HSmneBfs0VKOA5jmENXzs55hynk2gBl2w7fge9zWSMtS2/tpfDqF2FncGspy4bawEy2ehU3m5VcraUuKWQymvA259Q10bBFg7khr1xxPh0G7YGgu77YMXURDVpi/f4bwHgPaF/MYjr1vXrOfAxDjcNwLjz1T+IRFgFM56pG691xeiuVPTMEf68+uq7IwaDt03m7s0Ra686Wy6doCKLh/kgMVE3oANbN3g0RjeNXE4PDw8PDw8PDw8PN4TeI85jQ+PDwczzQgjsdXSd/Y5yDclJXYy1NI8sRWxT6w5cwts/RvaJYd3XUjptlb2m51TjD0fS4JzxYREx2Zz0FCsMrJ/Dab/9OHsbbz2FDo47LuWLbpCaCRhSWmiLWFUnW/tG/jUH0iyFj7xezo3dT6laBoRkeJTFqExZbbL5GccpxjsX4P9fn1rtgYCgGKgADPxVEEaLZbsW4V9L/wBnVynKEfZDwSBsQFn5hAZmrCQy5+ng1wBQgffsvw/AKB3OTY97xd1GhSvt0NylhKue15aR21lQte0VJguFjDNfqT9O1rwwlo63jpDPdPWwRlUSgfOzOs0U4/+HF0eWEvn+H1AGntyzjSsmL6raHFRsliqFt1J+W92CP0fljH68BMztC5mL+z41rgtLpYYhi113O1wkjpKHle4TatfhRg9S92g3le+yVAbW89Z9qm2WmMS5bkam37kkEzto0yZKatcUxruQvz9CyEwck8KkBaq70rbP8vQaJ9cfhO2T1WcCcnyWxux20GZ6khFMA2spaWVMYapsGJJ39X7DEtWgLuSyWuVsneP+z5VBiF57LDq3vH1v6ljS7vCLGPcP9aSdFwUWoGVb2VrH9uyM4MSLM2mreF0jUWsxeZr/auwz9YjIvdKi9UqHigvSjvLZUrTVmZu5DQ3nIntt3QEXa3byHhA29qI+AvFfT1FgDUXHgUA6Pr+C67CzL1xsJQ/yAMVqssB1yl/DdYECIG+Hx0DAGi76LVU9Y2y1CXp9Wshka5vBQrH/woAEN62kM5tRCN7sMWHksxhLuavSAHM5KXewWAPcsf8LFXdhgp6nlDPSeSKQEsjnlwx8Bh20Pn072+Po/8/9SugeQjy99hgEHJI9elV0Lt8SLM7dy5x+Z/7eLZBPi0u24UG3BMeL9dJacHRxtrOcMl09vVqbawV0TstpJT4+S6U/4n/St9G2cN0dEmHVEM3KZKc4S6KThnihn3Iweywv0djHJDKOUmLmSaKux3f5f6DKf+d70xn0iHXqQ1VYAbzWlHT69Xjhr2pH8Xb56pvvb6RuK9Swo370ybkkPsyOJDX+BZddalXv38eSBvCXW5OUhXWdCRP2Y9f+CjlP/uWlGY5KdpXr0wbN+9LpjoH3v521fzWJwbPfw6j9s25Pb3ZETvB2mW7hnH3teTCbNlJMwEAY3/8Uq1SVZ6ujVwynZ3/6tNmAwBGXpYhzkGD77HaxnHhdjQWrM+cUfVblBK/2JXG07RzhhlLYWxENZWttdm34grRj91PTNsvmk/O7Wc8tDiZp/N9xd5T1XSEK3an/D/92FDNF2V9LUGaYpsh6nsl3jqeNmcTf/mKo7DYZlRKQ2HtpMjXFdLnV59OhCOp+6nqo9Vobk0R1ccHewzpO29nAEDL1+6rW3S1eaha/gBQvmR/AEDha4/VvU8j5XdonVW/wjnHhZdR3KHghGvT16FKWc78f0ObxuDzN2fK/51C+Juj3tHyg2OuHtbyvEmVh4eHh4eHh4eHx3DiPcZSNTwaDg8PDw8PDw8PDw8PAED424+/o+UHR/9hWMvzGg4PDw8PDw8PDw+P4cR7zGn8vaXP8fDw8PDw8PDw8PAYVvgNh4eHh4eHh4eHh4fHBoM3qfLw8PDw8PDw8PAYTrzHnMbfW6318PDw8PDw8PDw8BhWeA2Hh4eHh4eHh4eHx3DCO417eHh4eHh4eHh4eHgMDfyGw8PDw8PDw8PDw8Njg2FYTKpkzzIAJhR95FpYgboIIHrdTm+Hrz9/3gQAwNmPLtkAtQUWzqX8FzxXTn2P7LbqInKxi6E6b9rIbbPjLupzYRnhzz5C5474Lv22U52QKwL8zAK1XwwKQJBPllUpqetcH/U8KwMo/fAwAEDx64+nb+PKRfp+NHeqLHnPqvIOy0CuELvTfq9SpytdcgAAIP/hr1Oqie9TyXNAvlmVNajakKe2RypUsYpQbeT2A+g+m/LruPStFK0D5LrFKqtinZQp8ir348UjpwIAZl3zGleSfiLPR52TIUQQ6zd2flb/B+ib+tPBcwAARzxYqXpfAr3L69c9rOi62OWavsrnAlR+fDAAIHfSjenrkAH95+8KAGj+32dTpZc9S3Xd5EA3ncw30W/k+Vb/Dm0MfHs3AEDTWXdTuiVP0X2dG1nfAPe9SrIfChHpk6ownb534U4AgLaLXkNayLVv0oHVj4T6DmVY1nVyjbeJvMr9ePEo1U+vft3UGaBvrtxPx/kWvkOX5cwv/hxlBQ8dMgkAMP+O0HFHlXys91i9DIn4nGGD2x8uugM9l30JAND+7aci17KA56r4dypLvVh5xtYAgNG/XJ0uLz1fCGsM5X5h2ifUuB6dJ2OolPDQYRsBAHa4Qb1D7ht23+N8hUjOUZVBU34QG7/7V+P+wzcHAOx8Z4Z32P22Ki9IPG/XvFczr7CCOw6gfrTXX99OXYe0kDLEn/aaCCDDeFpjLDXvK4B+rgpCBNZ3GuhzlZ8cCgAITryOU/INQKnPHAP0jvRz4znEGn84faFZl9O3cD4AoPW7i9K1D4Bc+V86aOqsOlYKIUx79fxeMX1QjSFy0R3AQ9fSPZ+6Qt/rytO+FqlPZCw1cyeVWYa8+gsAgOCEP6dq3zsO7zTu4eHh4eHh4eHh4eExNBgep3EliZYILIlLbPcqQy3N0aek1LtcI/2voLm6IHhI0FD+LLEY7DHHTSMAAHKZks4W2yBGzaRzvKEf7AFC0kTIllF0ToYQm2wHABCjZ8UKEkayrzUdeSMNsCVXLNmNI2hFftP3Z2mdyk9J/sv9QN8qAMDKs3cGAIy6+OloHW1EpPfqPUIgv9FsOjN9t2j6Uh8ES0cS2hK7GEf3tdrfNmtO9XtdUJI9GZYhFz9G2U3aLlJvhCUI1r7UQq4JPSzEUunjWopq9XYhfo9oG4uxVV5vLbC2Ea1jqkoWbQmunUbEJGrhLV+H2PP47JVICdm7HMUPntDYvVJCvnIXAEBsvBcAoOcb9C7bFtwPFNtUOjUeDawz0iZ1TYgAxU9eoI8BAGNJ0otcAVinpNQdJBmNaBWs9ynjmtuA/y+g5fgfZG6bUN+1rJT0WCNVH7to/hQAwBkPvlm3T1FezdjktHMi+drgYSr+7qvmF+9TIo/tTz6pbj0SYO1U8wj9joSWBqeri67C5PdjxRsrAQAd6+GkWU0DKQqtWPL6WgDA6LSZ9VJ9kC9C9q+hfNrH07mBdQAAufIlYMoOdI7H/Iqldedx5frTsP0Jx9I5relmbUY6TVfVuQKAbO7EjiccUz+POCoDuk4yUS9h/o/1UynDpGYrLGF8s1sLORQQIsCIGlNNLdC4ztop6quX7kTf4Sl3/xcotETTD3YDg6p/t6oekwsQHHs51SVIzmuS50E1R7nfqfUcm9oTV1vOvClFa+JZOrT9ZaU9UVpPWyOuYbeh0Eq/s/bHqp+cBgAYdXSy/mn6qVO7yv0nyOGtu/4OAJjU2LQx/HiPOY0Pz4ZDDaAIy0BThzpWAyh36FIf0KwW6A71mj5X7se2ozbcwAMA80Y3kD8vXPLN1sem8nngN/S7+0lITpLCTDi84aiUgLEz6JzLpIAVU7mcStNApx09Nfs9/K5aR+uFTvvkKbpWAGhCVOlqmqsEefQ/8yhlF2ujTLOgT4GV/3oYADAm9R2qnn2rIMZuqU6xmYEa1ML05kv9saQNvacaWDnYwE1qgrt53wk48K/K/EL1Pz35WahlPhMc8L+Q/bTQii8KhwKidQyw+SHZbmITvN6VQPcKOlbfY+tBn6P/y31mI8u/pV7I1a9SueNooyoLzUDbODrWJlLGBEKySVW9dsS+g8jGU+XfEIK8tXCjznb65/em/8v9euNUDwN3/B4A0LLzF5MX8y3Jcxmx6uYrAQCjP/TD9DepBZM9drhM+lKh0IJKOYPZYQNYNpDxhpyaI/LNEK3RxSSKarHYMQm2CSoAoNSbfCcz5qHvpp8CANoO2QDSOBGg79F/UP4fy3Afj+Ns3qPyIsTMyOziXO81V0R/emuuhjCYMf/IvCajG47P7q/m13yT9Y3yGqEIFGhWsjcX8tHf0rndTk8WVnOjkbK+//wx5fHhH6e/iesXliBVHRZ/bjMAwKTLabyECGoL0xgih66T3AIWueQJiInbpq9XFUw47eL1zsNjw8GbVHl4eHh4eHh4eHh4bDAMj4ajTOIfWe6HYOkNS8st50rpMLeKS7VQ6sXEFpYUDr1UVcoQ00Y0cCO3p3+1UfEpbU75FTI3yo+5EWApIksOcnmj2eB255uBqfN1fexrwnYGZ0dRIZxq2JqYuVv9NDFox1RIyCevBgAUz7wdALDw/dMAAAseflmn11KPIO+QgEg0f/zcSP4mfQ5SO4s3LtkZdfw3M96hyii2Q75J2hExdZdI+bKWiVcMI+M+7kPcX0c3NaLpo3YceOOLuHF/cjQ9+HblVM+SyLBiNJE1IKXUTom6bygHwVRmZ2lQyChh5++g2I5bz/ofAMABtx1NdZp3LF2rDCBuAoFiO8RI0iraam7BWh/WTqyi/i3GzNYSXKPBCBHXYEbNLaLmJFIqLU7DkNb4Sc//mz8kSfSCj5VS59J0yCnVL7LTuHoPTnOXOhh51GmZ0gMABnvpN8hBPvVHAICYS+9RPqscQjc/LJ3D8bJnMeMbJNXVzq0DpJkTLSMhlWmc6JiQvZ6gZ7LbtzOaxqk5EflmM3co09rXP0NmtFO+d5s11qs5Jd9iSdPV77hZaDn62+pUjFxCSmPaZJnHZH2HLUefnyk9Fa76ZqkXGIiaGWsTrsogELSmym7SOO6DSaflocDW47JphyLPmJ+/Q/auAAAgAElEQVSnOte8j+qrS56EGKGsAHjtIyt6nJJ5Na8HOWD0tNRlSxlqrZdNclJzjhm7cer8NVijG+R13pN+/KQqw5DcmPTV1yFCCKz9OY0FnRcdEL02fqvsdXNg2aVnAgDGX/m5Iclvg8M7jXt4eHh4eHh4eHh4eAwNhkfDwdRs13wJ8sCzAACiU+362R41IhG1JDi2cxkA5AqYfcAH6FhJq6TeNrkcXN2oanMoQ2y83571WuSAyqdllD7mvAun3gIAGLx4fxR3PZUrSL99qyFXK6npRjvqOuAtJUVIOI1bTqParrff6SQWpZCLST3eIP8JTNs1XfMQlSyxtJH9cxbcR1qcC+ZPx1cfWcw30E+pX/tASFsTs3pxrH5GYsRSG24jCZEcznS1bEfXLUvdNjuP8P6fmJOTt6dytAm4dDvSOfKatQ9RnhoJkIj8rC923mc9bF6b2jFvmvrmepWvA0vgCulogYUQkOz8ytJh/iYtSfh6aXZ0/pPSpbd8bj54ojI45+fPviqyYiStWusQQA720CFrHEUA+fqDdDhrf/ptHaVuk7pT6M9MVqr0DX7hnNA4P4ev3EPHYzZN175YvnEK5wUPKcpLhyZOSmlJRc31+88mythdbk4a6IuYhqmRd3jlV78FADjm0QXpbyq26vLEduSwzN+RmLWfupbSaXzMpnj51H0BABtfRXTjaBlprjeo2dD3iwCPf4Oe4XZ/PznVPXIVvScxfmvo/qHe5UaXv6gShUkJaJAjjQEAQL2bgXUYvP47AICmM2k8N5pYAeSiDuHVnltVzYGUGLh6IZW4zSdTtE7nQLdf/z/AtgfSmU32UXlWH2NsshhzMsSED3+KjlnrptookdIxvg6mfvSo9c6DIXY8kQ4G1hltsZorw18dDbHHsZRuc6Knl1ICrGF15RfzAxMiSIw1UkpjTRLTvgIAxm7SQEMcPqTPKefzrel5iVwhquWogY4tt3NfWLcEcsRkLoHybWCsGTt3buZ73lG8x5zGvYbDw8PDw8PDw8PDw2ODYXg0HLyv2XxX8O6VmZkQkLRGNHcaqaPNXMG2/PZunne+SuJdL3iXC1UlIsKm7s0ATd8baDthzSal6lX84nWm7iwyzxXwzcM+CgBY8BhL/CUwob5No5b4O7QbXBfA2NVHtEiT3lc3/wS0mF8YOk4VwEp0kj/AV//5gnl+6lc++xfghfso3Ycuomu5ArDi1fplijpMXKp/yBgzFgBg8fP187egpUe7nGyk4jqAVvW+Zdvu2/TNd/75AQDA3p/fMJ/ZVTeQFuzor2W5y0iPJv7ilciVuw4gSe8e1z9r+RXVyEmGsWB6gHAxI8X8DDIhqwRI16cIsZ96MEw7zX21bZx5v1YdRbNj/HmCpHliNtkcy5bRJg23i9mFrOCIkX5Yq/2P3EC/8z6bpnURuAKj2pqNhP+blPpZ2OlyG1jIlm9ErKU1T5bfBWvimrsoSWXQGaQz8Q6CPNo7159tqxa6WrI1UkzgIKeBRUkb1XRAhhGWIABA70rIO2gMFQeS5ghrl6I4cTod6zmxgYceCVAbHfKaJk1vPL/2kRAzdqdTHPBwhdLijN1cU8VbN1ruTqwVDoApiuZcPS+jQZVGm8/fdVDIrvWYuk2m5Pr7kpVIuQCM702umJgrlz35b4w/mujkIwFWU/jN1W2TfjaOdCnG9GR5dj9Sc5zSbMhX7qbT03aN0uDCraWSUkIO9MIF0TnFvEP1ncvW0Zn7cXkdBd5c/9C9HhsCw7PhUCpgMedQw8msF95qUbr2DTPwMNd4/2rzEbJ5ztrFQFeUSjK60ZCOczEefCGckT51OV3ZqSrtDyPBSc0TisV1rjcjLSO1GYT+QHNFvPg5Uo3Pvm5p5rroMnjidTjw3nkM0WfudU8DdJEi0AsWbfLliAC67qs0gLd/4usQh5BTYyQieovbO98d+8FAD0yDPbosVwwBdKRmxY+WlStWNZuyzdTsaLKJOBlBrjHa2gx4s6eBm6yFXHyjvvtVjwAAes7bC23fJJM7uy8nFnKVQeP8WhOqnMEe42hdI6q6XZ5U+addOujvMBcAbWMj9Q4mkbpdhhXtNKzfc77ZGpuM8ODNW2nDMeWIn6hzVgyIxHcltFlWhNo5ET/C4L9/uRUAMPvzKRsIx3uwjy2nTe6r8tFf06VtjkzGBJASm85MR++bqAf3/8EePU672viBmQ0EjLHbxosyvWByUeXybSL5LYoAfT1ZeWuzYVVfNiGVVIsqbVpsg+tf6td9R96mNhdBzmykeXOx2QGQY9XcGacYtxZ+ZoEcWmUQUYRctxiCzcyaR0brIQJgzwaccLl+h14ILehQm0WMVA7Mg+vMXKyEkLJ3OYS6LnX08zz6rqONVtvWUdMnal90sxbZhDCsDbcrNtK6P5BjfOeuZ2RqpgjykPGYNzyu2P1YrW/G/3yRcyG9auEHAQCjL/1vpvKjUdurL9DfOIPM2aZe258pf5M3P2O1qZpKmyb0rjAmikFLNG3s/r43ScjoEpHqDSQTachQv/+0m8e1r7wCIAsV/jsM7zTu4eHh4eHh4eHh4eExNBgmp3ElOawMGEk4S1aevRkAILY8BFpKYdNMdr9Np7qIMk60jIQcx0HxHBSU7KBajya21s5yfFzFOzSIqhmtujvUkfGgcY2gllQgFzQePFEI4VB5KymkaNKSio7zSEoub/9fyHYV6GgmOW+iMojSIyTdbdrjyxnLV5KQYrvWnrkck3vuIfrMjo/+LFP+NTHQrQNURtThDhQyaPSdjpJ10NzA1+vSLOhylVlc27kPYOE8krwueOQ1SpNLmiiIfDPkiPrO3LpMy9xKstazUJ0SUwgBuCTANVDLrFJL5Mv9eObj5KS9xVVKmti/DGhTcrGSUk01tWPSd2/LVLZc8gTVffL76WSQN7SkDqnqJj/+e+r8Gamdpflb2E4RPPQsS2g4hBBY9DJJlrMaXOj3akUDd+miHniFJKofyZZ78lQ8oCqEMfMUyW/RpisulxobUFMFNEMyyGc9COUUj7CizQB1fUtsAttktAS7UbR2+fRfDGWwMkGWqxYB9/6azn1iZy5B55mgVrfHAP4mO6daFLxx89CyyX/Lj2ZopaGyj2sD5RIyB73l2ANx4O3KzEoF0hQisLQrpq4tx1cPHJkwHyz3J8cWgUSYQfu9dpxyee3m1Lg3sRaxiCFkzzKuJP0W2xJBOYUQaJs0GVmQ0OCEFU3K4uqvozYaWrk/E0rIfDMWbkfzwIJHSIPhMnUEgNaTrqifr+Pds+YYhdaa3+Kor/y+bv7vKgTeadzDw8PDw8PDw8PDw2NIMDwaDqmcpla/Cqx9k86NVhRtq96g30Ir4vIHsim17EhB9p2YStRqSWlmFql9TIrDeYVlYOq8DPkgVpekPaW2Ua0Maudmjcqg8kWwbRcrmLxR/eiDLmme67rr2ibjGwnMZtHWppGysgPbHqei/AuSsuYVPSXCMmQpfXAyZ22EgGTfDe3kbQVBKjeWv1tKrnyDlj1t6IutergwtS1Lf5RI76lAmNHeuJaqeh0AlPqw4EGS/D9/OEndNv3ja/rZRuyE2ReCbbPTQtFhy8Eet6M5o5+cANGW0q/Kdny1pGIADJ1m/xpsftpXKfmyZyj5+K2AwW51ryUtZ/rosZubc1B9T2lMjMRYWtTW6llWBiN25ZxOFQS8RgEmsXEjVNzpwNpe2TraSLGZWltKrBlcfylbLdvxtaWhkeLpsZp9/GRovncd3K7J6gPqvZQH0NTSmBtphKq7xpg3JqObilxB35cYs6kJqhijr4UdSLJ9PKV/31GQix+jY0XRjCXPAO3q+1PvV9OPR96L1e/iyBWj/nUApD33ttZ3aE5CPbu+lSb4ncr7mwcfDAA4Z8HRCf8j2dwJ1/xcMzhcGAuQGH+WXKMavmPakT8j6vUNqehw8YwioNjpC5AcZNVygC9uvn3WgqP/VwYjdNdxtG63R7b8M2DB3Y9Tlf52Hp3Y+2y3X+W9l9HvET9JXqsF1pBVBiO+sAn8/Uf0u+lB2fL3GBYMz4ZDffzPn3wANr3wDwCgByB2dpNv/QtiKsUt0HEAwjJQZs9bNdF3TgU6VUTOGPuLEIEeUtOqwhMQQnGjZwRPeEEhOajxoG9zcmunssCoV3uUg3iQQ8f06QDStsMsVl3pI86CqszRM7OZqtD98QWWK42DO76pA/kTr6LjAeUY2L8WLz1CLFJbZq+JRtx0zm7/489QDIc9smbavQQoqvekTR9okfOdD30IZz26JFpmWNGLWV48Symx7Xk/StQpDq0WL/ebhXFKHHLBdzKlrwe9QW7u0nXd9Doyabxit4k47h8vUUJ7g5BvjP1HP4tiW+1Izxmfie57YcnEK2C+/kV30P8yhNjpC3TMpp1vPQ6hYq6g3KfrKMdtHs3fXoypd242vRUTI0c7uxZMBHdui+W4j4lzsrWvDpKbIAORK2p2QDtOQ0dh6Daurr4eNpB9zfHOMq3S6dhcrdRnxjnup0E+E4Nhvfq42pjZBJY30IVWayOv8rOiuuvyedPaMhJiooq/w339ib+itJLmjuLBUQGd7VBd6xkIIcxGO87gVhnEwLO08c72tavn1GKRd6g2nnP5dwEAb//kG5h46IXR+g32GOGAIn5AuR/yjYcoC9fmvJ4Jdb2ayhDyTZX/yOmZ7q252ZASomsq/TNfOd6LQJvRrT6NTDtHXvIiMCEZc6tmufFv3CG44c2OaOoAxjUQadwBZ1yldhq7xT5EaHDvwVOw6w0UX4zHPSEEsE00wnha6LYGOUjFTOk025p3SEP5v2PwcTg8PDw8PDw8PDw8PDyGBsOj4VBmEZt++jMAR5NkCRtL+ERgOa2pfVCpF1KZYAkVhVO+8RAwisyxtOOdU1XM9Lh1JBAxiaAUAZltARCj0ksEXNSz+hq3p328kyNeS3byhlLu33c/BQDY7ozqdXc5oNeklLWcsO77J0mr966auwPK5EQW2y2JUqwMKU17OIaKCIx0l+977X6MGcfagJhTY8pdvwzLkTbZ98qwgo3HZDOj0O+mY2LyopKm6CjqdplBDlK1z5YiXnXyKQCAT9x1RNUydd/IKskHcOHnySzoK0+fmfqerJo/Tvfpe5fgl7uQFOv4f5JGgihr1186zpqNmpqOrHnmipAsHVUaBrEpRTtGZdBEMFd0jmLSPNM3Larfvp9TxOA4TbCUYdIULMhD8him4+wUExJsu4/0/ozyb//+G6nbVjNyez2axaakqeaS/qGTsrn6Vb6B7GtpamqZBolCi6ZS1ghyGOxfP/PNeuUPZgzdxKYvTrMTk8p9mrXk3F/ftz+W/eRcAMCUKs6664VyP5Y8S87AM7Lcx+N6oTnRL8VWNCZO+OHBZvy3YmnIP55Gx0dfoeuAR6+lY4eGI228BhkjrYl8ww8oLfycI1PllQakOVKmYrqOQsfr6Pr+Czrt278kWt6JO5wwdOVbsT1e/hHlP3Pfb6a+3z1fxMcxaw5Q7dr15rd1ZHWdV1jB2p8T5XDXxY2bPGltn9LE2VYOSy4hAppJV32h4fw9NhyGKfCfh4eHh4eHh4eHhweA+gKi/88wPBsOJUUQu50M2RMNZKe1CLkm4/grC/qcDizHAcDGboHBn32Ssv3yP1T6mJMbohKPBH0crOBQ2m7VRNEuXX48AKBpwVOpmyj7Vpp/2K494SA+YBxI2ZotrEBLCLguvSux7bd+QJffVJJV5TSI1jGmk7I0FRIoqlA6ttMs28FqRzr1XMv92PPb30vdNgMl2Xj8dxDzjqVTbCOqqGKlbVftkNqzBFjM+iDGnnoJAGDFyfSOR//gOQBAuOQJBIpW1NZ+JIPrJbuvlv4GOUw5PaOPg7bJb0ZciumkM3ZAS7HL/do23uVDk9m3KAYpJcY24Pefplynr0lYxmfufYuO2Tci3wJ5GdFkis9dl70y8XJZ02FJt0uXkk1u8euPp8tEaRYkcpBvUiBDMWYzuhYaQ3vhoPNl22C2r5cyRK45FiiPKS6LbdHgfpyv9o2wnDdzUfII7cQtJYJCAxJp5agvix0xW32gpmOwA3LFC5g3Tfk/DCineabOFQJxB+x6/ScuEZWVEvaYVj/IYxw1nXvrTdIxx1m58iW0djRCkpGufCklRlT31XXD6mN2xGzK37zL5LUgSa+81UfxzOKzAACTeczniNeQCS2wOwq0NUcKQ4IAACj144m36dvJpOHgsb6Q9PzQ5ClB3jxP/m0ZCRzzG3Uqp9M/9FvScOyY1eHYgi4r4e8gcPtvbwQA7P/ZdHk5qaBl1JlHIme+IU4dBJYmVK1dckWMP/RjKgs1DnGawW7t86G1WkUrdJ6w1hKcTvcDlT4oYMbBDWgVNBmL6eDJMUAk/YMGe8wzZv9WESDIr/+SU/d7axzleuUL2ccaj+HD8Gw4mhS/dnNXYqKXltNpoiPbfYcX6sU2XHodLUy//BVX53Ko2VPsIm1TiW9f9SwA4NwFdW8zUNFZa7FFId+SHOgDAe1cx/VsG4vnF34JALDZ9TUijbvU8TZ3dT7JuQ8AKLTiH2eRGc7e95xWPf84lDmGmHecdY5UttJamDhVrXyOB8SwjFu/RKrjA26LOmGLSfMcm0SZIOUApOWAG90QyLCMK04m9e1nHvtSisZZdQMSE709oCYWVY5rCCvoijnj2nE7XHnU7DtxlPvQPoTOvvEyXZs7XWdrI1kp0WSSRk7jinrsjEhrvYdKb3emutubUDFlhyqp3KvDpBOiwAP3ksnDB/gMm2k1gPgGWQiB/7uL4px8IktGarFRa1FuL2RjV6J1GLUJFr1F5iWTmpLxf6WLLShFP9Xfb5DD44tp0TKtam3d9adyLcERbxitTZb7W4z13ZEbY6B3EC7UZRiyWckcUc1VKlSyfopxR3G4voXaz1m3udiK59bS8b41THtrw8xD8flIto3BC2sbEJDUrEt83I71ofjYmm+uuYV2jZOu8blWfbLKmt0mjQ4hmCtnR7yMpTdeDQAYf9C36QSPR7VY/OxyAhGZ4+N4+SaKebZxyg2VXc+a5bqeb7HNPH/rOfWvpbGmPgene76ol75nbWNR1N8xvMecxr1JlYeHh4eHh4eHh4dHBOeffz6efPJJCCFw9tlnY+utDYvrgw8+iIsvvhhBEGDGjBk477zzEATVt+7DsuGQT5O5hZiTjFKa2bSkbyUOnqLUkNpEgqUlAZImMFY90khBygPYe2IDUWnj3O/Ri1xalfLj6m2J2Z8nh2NtvsIxBYptkKsU3dxzFAVZbH+8MWnSDnjJtmpJS2UAe37hmLpNSmZgJIzGZCJGURdpinAeAoAUEh88QTlTr1OmOspZv3LNV5A/8U/RGwotEEFU66AYASj7mEYCAI47ZvcsrYtInbI4Vzv7WqEVWkejOOJl3yr6v3UMTJ+t/QlWK1/mW1AKs0tH0jhmVyszfn716Zui83yikkzzvGpLc5MIX/o7ms+4qW46G9E4GcocgL9JdmJs6nCYlEj9Dduag91/cEUkX/kyUeuKSdtBXvVFOv64imQfBEmaYBka8gSO86EiKkPk8LEf/zhT+6h+NfpMjyK8aHfHLUlIZQWw05EHqKqq75vrm29C//m7UpXPuIXOFducfbaWdP6gg7etXt9qYCphS7rrpPqNa+Jc3+LqV9DSHtUGp/2+U/XZsILWloxTac35onY9XOV/8bhdKFuOW8OO8y2jjFkMv3tZ0VpKaRO1uL4JAN1nb4vTP9FAjAru703JGB4u7YDLMVnXZbAXY9uCaJ0Tkefr51vtvctKGRn9/mtCMk16vtmtxXWcG3fJ45E66vcGiZ/vRtS6J1z/N7p/zGaO9gtdrn7XbAoZ5DDj8n9nb8dKiheDUZtYa6Wo9jH1Gq53ObpmkQl9gqyn1JuMDG8h+g4dpmwAICsYP7sBun+Pqnj44Yfx6quv4pprrsFLL72Es88+G9dcc42+fs455+C3v/0tJkyYgFNOOQX33nsvdt+9+rrLazg8PDw8PDw8PDw8hhPvcqfxBx54AHvvTVymM2fOxJo1a9Dd3Y32djK/vf766/XxqFGjsGrVqpr5DcuG49XzSVo/4w9JDUdWyMWPY/apRAeKbgpKFqF6LCiHOpbmS2kkY9K2Aa6yK6+UsNNpp2avGEsFAyPhTNDD5orOchNSF1lJUgQr+k5USiZa+6b76DQux2R9zNIutqkt9QKlGHVkGjjsTuMOcakhAqy4k6SmYz5MzuNsH5/bdh/9Hg21nsPXR+QSvh5aCoMcnrnpTgDAnFMaqN4Q2FZqOtDeFfTbagfAsqJZN4jMwcYA43PTs0w/75o0qw5w+s6LngWU1qb7LFKzti+8nxKV+5UmBxHJWJoyWPoVzNzbOGmnhZIKyiBnnMRzJmiUKsFy/eFvLoyRBqhzXcrzgMkudITgFmAKO6Pzt1pMSAFtyTFruAQ7fIoQyEC9nQo6aFxt3wQbr912OwBg+tFMDW60AU1f+L06UFLqBr73u//6LwDA3mdnuMnhaNwoROsY9CsfjqSvSfrnVBVBHkvWUr9L7VTdoOa0avkPE0HCxM8ojZCmIc9Zmo1ooFx9HSBtccIfjtA2bw+8/CfSOM9MVyNCw/4krrya0M/cw5pu3fLbSzl2VX2mQR49ZfelaqipJWtgISlfuYdu3exgdcK097jDtqBrHERR2D43OgdTboy2XsoQkiOd77ZZ6joJDlApw6SmmDW19fLg8a91NPIHqbUVj7Wqn7563CxMu/KNSPrqmvYYxbJeD+TR/JH0FPEe9bF8+XJsuaUJzTxq1CgsW7ZMbzL4d+nSpbjvvvvwpS/V9pf1Gg4PDw8PDw8PDw+P4cS7XMMRh0tAumLFCpx44olYsGABRo4cWfP+YdlwhBXlc1FDmpRa0vT2C8CMHeme5cRWJSZtR9dsG0CntEbEfu0HqGy0+1YCszPa/gOaMcrJIlJh0YnQdIC1bINFkAd2UsF/WNPRv4Z+W0YBj/0fHY+bTvlP3gpi1gf5Zjsn+tH+FepacxfE3o1IAqznxhSfTNGo/DCcQfOqYPTH1G6Yn9NismEVu5yC5SdvSmnOuYHOjdkUUdqyZJ3i2OIzn05dl3heWQPkuTC5U31erM1Y8iT9O3GuKTEiEc9SlsTMjga0Syw5LrRq/yDhCFpY2xdD2VKXB7SUq30BSef+fCAF5Tz0b0tIkwYjxSKfm/pVtNm8tGYvLSy7Zg4aGigtgqwlNRciQaeKIKcDjvJ3+M2PEcXLgscWQ+x6Ml1jjUVlMDkgWxoB0WpLJwFAQIyenapZ1ZBgaXqbqLzFlPmp/QOmfumbkbwwqPzGCs1Y+819AQCdFyha4oG1gINSuBb2POPkTOkBWH4kzQkNXO2ggEks3GkznHPHg/QP27hzcNfOjYxvVXOXKifjN1/uww5XPpztHpe2gS9xHXNNKesiMWF/oo82mnamRx2AvPnrVNQ+ZBkgB9ZCjJoZrQeEOeYuzHm8/0jM6F6doh7xaimKapkDYr6VtmY6lWa11I+NNqHvR3YrPzSWsBctHxGeh8NK6v6hbsCO07NpZGq/myS7Ut13qcYp3c95PMoVUTjgJDrHPjorX9QaJKEo5FEpGW060+jbRU7cEpnBASrtZ+kIHpoOAmt/djoAoPPi5yNXpv3KBECs5WtL16tr5N/6Pq1rJl/tA/8NBcaNG4fly5fr/5cuXYqxYw1TY3d3Nz772c/i1FNPxS677FI3v2HZcMw4nSNbJikLGdWcvBNO3bP2BJarzvo4qQjl7rSgEZ1TgbYxKkP+QGzzH8vJnM0t4lSVzSMg38g4eVjlOB3EFN2k7F+diCwrZag3RxHa1GeJwk5sTzFBNBWmEMDhP6R0q4lSM/zdycjPZsdPaxHK5h88uLNZRKUE+cQfKb+JDTh0AoYDmx3b2lUMhXJ/zajrGjJE7y2/BAC07UzOt4GiMZXrlmDMD56m4xdupfStYwC1mYksstQEK/k98iBU7sPym6mN4478Zbo22Y7wCZV09gjd0/Yjkzf5BzX47an4CMMyUKZ3I3nwlmEmaYcQAXY9eKfU6a07df1kjAY260JL5JsMZ7xarOmIy2vf0O9LmxwF2YIVkON31k2VcaBMOMbbixyX4zWPCXlTT/k6LVTF5O0BAOf8j1rYhSVD5MBjTalHxxyImDiqfOWa1yn52M3pWpCHfJnM/gSbbmVEQlixUfY+8cBCsnXa+WY11lj0uJ3f+w8d8PPK144b4vpOfrXwRwCA4w+9JH2lcmacTJhQZFlIAljwyGt46WPkTLrJH5UZbudGpr6NbjQY+Wbc8wnqH3vclbK/sumfyCXLVW1PXR8ZYu19RCDSuS/F49Cb4EILxP6K350Xj/YmPm5u5UKpF0vuJYHCpCzrOP7ew0pyns0q2S20oP3j1E8XnUh9fOPvkPOqmDTPmFLX2MjR9SrjuBCY8Jn/yVanGuB4PM6YJ6U+ayFvPRceT0wu6oYQYosP0aFa+4jJ2xtBoo5fFmLgRxQlvelL11M6bTYrdfyeTHBFrldzl4t6txaEEBhx0HG6rjZ6v7UrWhfyukvNUUHOufmoJbWaeHgDZDgeVbHzzjvjhz/8IY466ig8/fTTGDdunDajAoALLrgAxxxzDHbbbbdU+XmTKg8PDw8PDw8PD4/hxLvcpGru3LnYcsstcdRRR0EIgQULFuD6669HR0cHdtllF9xwww149dVXce21FJTzoIMOwpFHHlk1v+HZcDx9F/2ytN5CzQBgDrMjObAGYlOKmDlww8UAgOLrj1GirunaiZUdViOwpda5uCOd0i4MdGtpwrBBa16swHNbHBJNY1FVagmAauNzdzxiHKP5mYVl3Tb5H5J2BCo6uAQgtj1qaOquo5wqKXZYqRkYzzh4BWjd/9hIVlpb0tIFKDM0MZMYEh756Exsf+PSWL6hNtvR0jSm0RQ5jDnk4401ySXVaYeweUgAACAASURBVAD/vZG0M7N+v4hOsIo8yBupuI4muy7RZ4mqlc1H8rFrIf5xwwMAgH3PSF+nyDcVz9OickyNGJHA4f94i3PHRTuQhuOMe8j0EYW2pNlSyvzTIhIoTlMlx5xNpYQsGFIJAKb/Avq7EUEOkk1P2BxrT2UGWClBrnqJ0imTBrn0Zesdqnz7V0Oyw7kliVQFAKMyueGmh6ygHuUyY1JnjXS6nypN4to3jTlOSoxpboAYoRIjunAgrRmuXLUIE2ZP0veos5yJHjNliSlFhTEVTaPZLA9gq00ymploQgNHoMI6bUqYlEiJEXsfrusOwJBuBHkr2KoqMyxbJAAmUCmb40j1LWgH5X/8HBMOXI85UYiklL9vpSpjVMo8AojNaN7f+BtKAzR+Dl3L5bFwHmmsFjysKOODgtukUJNKJAlQxPuGaE60q+0KbFjqgSgk1ydiwtaJc3TBjGlipCKZCMvQfZjnQBnigmuJxnbBV6OmqEIIYMr8BhrgeIb5mDYJyfdbDeUn/gEAKO51VuR86zn/NBYZyildto3VfTdC8sEmhw4T2d57/wwAaP/IT1PVx6M+zjwzan6/2WaGdOA///lPpry8hsPDw8PDw8PDw8NjOOEjjW8AtLON7BCoj/ItkG+SrV/xSGWbqmxw5eA6LTEWBSXpzjcZ+369TYaR9ugAbCpATrEV8rX76HjLLDS+1YP7acmG09lKGCm3JVmXj10JAAj2+DL9r65JSyLH0vwtr1kUzY+h7K27r74AADBCaTggBOQ9ZFctjvhF7WZFGuIIVqV9YQrR3/h9fC/brgoB+Sw9Z+x6msqX/S8GjH2okma8/3ePG4kK26IW2yHfoMBzbJMtxqjdd5DHwL/IPr5l/dmYM9t3SymxaB3dM6toOU1DSc5ZE8PPr+gKjiUgZZVypcSi7uyDVU2H21xT8lytvGSoA81BB5rjOkmc8QD5LDz+ockAgLnXvuzUcFQNxhWWgW6lHWhzB7KrmVfMD8dQRlpt1+cKMGOBpZn7+/fo3Ie/T5dGUFuQKwKd0yJ5iNGbWBTYRuMmhHLgLMccemUI/GUhHZ/011TtqwebIjItXlxBfcJJ6crPh/0KRiZT1aN2fruvgUnVoWVsNOCYGD0LK18jeuV2m0oVgJQV46Sr/cDsfK2gatWcVfPNePkVGpNGIyXy7KeRbU6kZx1rd5DH0j8TffH4fb+uzlnjbE2yhMD8qvlCoClSR8w7BC9/9ysAgJmf/F36yrJGzzXWBOzjkI4qG2EJcqmSpjJpAQfZDCs45+S9KD/2uRq/VZSGPFGn2PchQ8jFZClBBCUbAqovOcZ6wNQdMz7gvg8w9Y98H6r/PvZbnHOp0hywP4hyrJcyhFykgpbOPS57na0+FydvyLJoDgcH3RfyLcYiYw35popiKwSv7ewxJl99npJh4zTzHhsew7PhmLpN3SSpBx4R6EWI2GRfvpl+g8B8aOyU+9ZjEBvtGE0nAkCpdLVzHX9E694C1rJZSHrUqruITXKJa1Y0XT733KUXAgA2VxsOVxRWqQYDYTl5anOjoKA/4I4Ln4nmnyvixh/8CgBwSJYNh2tgcaimeWKWrCKF0M60EczhgTU2SDR1mHfFC7f28XpjJh/5NZ2bvBXwhOIW/+iluizOs2m7veu3aQNBCIE+DvCrJ3X7Oi/kHCw1tsNxlX4lghyWNLKQS9FP60V9te6AfJAc8oUKshDp66ptc/+P2KLka/dBzNiDjlVsGJGvwcQjAsjbv02HR1+Voj61TcZs85xwpTKH6pquy3ISV+xxcjQvPYFLBF1To2lbRkH+hRZmq/5OTHKjzr9XC0TQEl0ACSEgD/hKqnatD+rFWZk1uroTdjIiu1loaEas0bMNY45jcbn1yOxsau6I0Y0JrIQI0N/jXujY9RWOzbBdZrXyhRB4uz/jt5h1o8GxlMKyHhMlC3dEgPEXqcVqvpbDuTpXGTSbKzVfyn/+CD/8Oo2hJ1+nxlRl4iO2Phwb/2KfTPXletGPY5nB40tY0eOEbe6mF7V6LJQYuJK+leav3afT8TXxqctVmaqNPcscEbsBqWJ3CfXtalIKKbHmR8QENfKyxkxxq8G0S5lCV1swK2FF8t1Z/6u1gm1u23/hfgCApg+doWN4xMsQIrDm4/Soua7JSN4AAGteISFU3FBSCGGEvuOITUtvaOg/TlmzTi89S+/3fZlr5jEc8CZVHh4eHh4eHh4eHsOJd7nT+FBjmEyqUiiaHRz9TmfjIK/NZ3RsCnaAs2jlpOJZR/cy6Iyl5bTJL5rzULzVYsQUyAQ93fCjWKzfEW2n7ISDdlhOqrQts6wRhaFRPbqd/ZV0ytI2ODF1XiS9DQk2wVKSmsEeQJkmifmfAwCs+9o8lPpI8jfqMHYytSQ7s9NRtW0oOJQXNaGlmKwur0OV2xRkf4epzFDy6aI8CyGAXU6qel1LqLhdT96Iyq1kmhR8/oY0JUAc9K1UdckK1mzUldI9/Fv6PeB/o/dX6fePX/4bAMDc619TJwOjrXPhnp/Tb8KMYgjBGonA7dw5ZmpqQyAA0jyz8VvRmX9eaui7m5NmoxO7MhIFIKWzdgaIIIVzecShPDp21dOuNGVdN9jmvWnA5eebjCTYGvPly3fRua2O1OfsNHRSjalh2eQ3SA66lX/fjZNPI8dwwTEcuHJhCPk8mfxVdWx2oRbhQ63nGVYggyjNvMwV0TSWYzylGfeEScdOxgDwFkW913EqWOspBFonb5Qi3wZQh6pX44V76XfW/sksarzPpi2IkhmT57nN1xjP3kW/O7xzMSpGzto4fWIhED5Jmm2xFZMi1H6GMzdLZ3rr8c7Aazg8PDw8PDw8PDw8hhNew7EB0Ox2kiJYfhV8Jk5dSAnop9gOrFJO0hwsS6WXpT5gsJtSq4A32GgHKwumcK0YByVOD47MKYA1b6Rp1QbF2r70ds/OqOW5gpP6j/9fPdiA5DCrVI6lR7ICyc78VkCkFRd8AgAw9scvJW5N2PAWW42UUfmDdJz3KOR/b6d0HFRt9Cxd5gtnfgwAsNlfq/NCb0gUMjxiOdijNRvGDycPl9MeI7+BxiohBGQPOdnq76gaiu1VL2lJOPsYHfIdBKpPXrg9BeU786G3EKf+taV5soqD5fpAShOFHJ1T1MlQS45tWlxsvIMjB5MP15Nuq2DTOSoApspfdEyEZJ8z1szaBAqzdmyo/na5daFJM9xUuU/+m+ye68eJjfk0cL67nGKc5R248zV6v1lcVYdKswEof5tc/fxcvnbSIvSo5QtTzqpsrNE+5/u1A/TFb5UVlG67DABQ3PIjdM6mE2UJP7+vXMEcq4CtuY8uNNTQMSpiuW4x+m/+GQCgdfcswfGkrkPifbL2vTKY8GF0LsKEADaZp+6NBe0VVn7su3nzuRBHKe0hj1GVQROMNF4fKVHYIvu3mAopF5WDT5GGo/lARxaOOZxpv8WBpAWWvcuqO8UD6H6YnMZHHJuqOpmw6lSad7suJgr0aprj/JHnZchVQMwhxhd57w/o1K5fcvsEKbR97tKq1zzeeQzPhuPFfwIA5Mx9kwOPg/nIqIodeS1/FmL6HnR9yRN0Ti+KBNC9hK6pQUw0d1kTrjWhcOTuGBuK7F8LMWu/1E3T96VYBDijjkqJuNpeSonJk1V08kQEWGGeGS9Gw4p2Eos4WjF3PqfTrB5lbDWmkQ0Hl5vRWUwExpxN51VBU4vafDiYk3S7ebLMFZMDd1iGmEpRZyWryjlGwGA3WtuHJp5GoyjVWIQkmH2kTDitEkuVOxMpJZqy++ylho5Qy46G+WZ33163mH47k+YIiW8iLOtv8cx7XwQALDlhBib8lCYpPYHawofVr9I5bebROIwDagWCWbX0gjLJUgUAWEr1hCI/07FiIBKc/uhdgbav/YMLo59HroDYlja+2uSTF4NBHljywnq3qx7qMVcNVNZzcS9ylrlmdFEupcRApeqdwwM7xkoKRMbpwBqjqkavDtGRz7jjsPpOGmd4oTcQgcVUpMaLSglhv/pO+1fTLy+y7c0Dj6lB3syx7Mg8aTtTp1UUy4IZgrBqEco96zI1L4oks5ZcQf3ejuli+qkjwrSUEDt+lo457gS3UVbw4seJPW2TbysTyN1OSI5X+SYyO1L52WXKMITY49QG2lYfuozIWJNE8f1JU6o4ImNqjBFPXn06xAnXRMvSi3OJ9g98pIHap0PX9ykCuiZ2qULCUbnuXKrWqWlY+aQeK8W8o+nMiheB0bPpnGNT03MZxUnquPSILNX3GCZ4kyoPDw8PDw8PDw+P4YSPw7EBMGF2jYvVH7gzSmdTp3lJj/6JfueR2k2Mm2OMsJRJCPHgs8bEcjyPqx5tTUIjdnWuGBXJRNGImfrXmGSok2gZEVMzRzhVWb1uIsdKqaQmoSXZsE0pAGBAOcjnmtDRlYb2NFn/LLBNFKTFu071rOCFF4iaeJ5T/aqej6JPTGp6QNI71mRNUY5zLEWulPDEW/QspibvbBhp6ZullDjoUx+kY66TjjReMNJExTle/vXnUTz99mRG7PDbT1GADbe8wEmfqm7uM1RgOmN6/o5vlaWgrnvjZgCWtFUoooeJv3gFl+9KZh2fvu1puqjaKGUI4eLST4mElFRLukUyzoNtUmVHuZ6yjbocc4APckDfKpWdkuavfVNTOqJE71xsc4TWvml6br6v0ArMeH/mdq2PuZFLE9tRg0CiHqUu5yVZ68k0qxx7BiK1BeaGg0Aun14d6DRRFTktNWYtl6H6FhlHRlh04i7639pzooyblAQ5rH6NqNwncKwn2zwzMYYmYyogF5h8uf9zPVYvxtJXaazOFk896vgduaJjXTioqIl7WV02ke7lTefQqY9cEq2fyGPWFU/SvWwq6aD1llIaDRBHlWeTynI/5PVkLiY+d2OaxmWGi3aZyW1E6xhg9u5V73VaUGhyD7oWnHidMQtRY90dB1L79vrr25Cb77k+1a8JrdHkcXVgDWRTZyyNMDTEGfIEANlKkdlF62jIJUTHjTFqXamegxAiFTmExzuH95bHioeHh4eHh4eHh4fHsGJ4NBxsp2xLERNISv+lHaHavo+lFGNIdi1GbUL/F1qM7bmmR3XQAopAH0slbRQcvRTSSCDZKT0F0gTBqScZt/N4+Tly5Ny61j1sk120XmM+HlIHkFJJ09Q1KSUeWUSSSId/WnXEaVsdcFL02lVmSUhlEKPb8/qYLip7zSBn3g/viS3aZJ1vud+0l+lzWYrUvQ6TW4Y+6mja4GNCCPzp17cCAI44Oq6tgom4rmyYndoNGKkYS3hsnH85RVn/xpmpqrReECIw30rLSKqTlFXtkbPg0/eQdPaiHcih88yHl3CpNftaPcT7obBooRMaTvu9Wv2291f0cNv/V/kIceCtSkk73OryWscYnyP+VlrHkH08YNrCQfKEwJqfnwEA6PrZsanbtT6Usa57lg3UyiddGTrfJvY9W79vb33aGL9X5AoIK9Ulq2nL0t9iDQf51OA8HON1Zgz2mAjLse8x4o9izX/x10Np1HMYydSl6r7BXoSVRt5ndcKLOEFEAorMhQNtIsgBK1VAXr0WsPJtI58s7esVVpLkMzIEAjUG3EkBRXEoBdhFroDyUiIeGU7PP2GN6yu+Q34KLhKVOKL+oJY/Dp9Sz3fPW5foe/77ZfIlm31rehKVrN+hTtfcpb8Te9ztff0VAI0/YyECYIIKJK0sNuwgtaveJE1cdSqTdxneYyxV763Wenh4eHh4eHh4eHgMK4ZHwzFIUj+XFqCWZkCIILEDlMU2TWWLvrV07j8UREzMO9ZIyUdMMfcwYwNLH0WgqVUFU/KxRFJKyBIxcmSRrdmsEG4mKm5T/FpoGDM0E0mI8eMtSarK17pL15X/d0qMFEWgZpBhO+uwhM1HN7DXZCas8oBhxUrYeBtGkpq2yE2G7lS+/gCdm7Yr/V/qtSh1Q+s3FvCq0GrsePn5sJ1+x0T0Z2THcbFlNQoppabFjTAbAarOscBWDgazWOUidZNhBSObqiffEGDNhv5fCIS/PoaOj1fsKFZ/SMfcZrSYZzxEEkzJLDT5FgxeQjq4pgVPpaqjtq8P8pA2wxmsb8PBCEZkcaqvWZrAlsmk5dT2+68SdaWYNNeSTvOLrujxRGqpmzRsOjxuqb4vwwpap6TXoprKsq/X+g/fUkq0ajZg1U9jgeUagWH/KaO5gc9pKMpmyFIfghpRONOWpf144lqJsJxMXA+5xj/eRH2bu3D9szR/naTnkhitOFCTTlSloL9Kq6C/3/cfhz+8THSm52apaAoadXcATWE0G/pkgIdvug8AMP/EZIcyQWbNfJCgORYBpPr2xCHfVWXxcwpx+1//DQA46Bs12mSXOYRaOABo6WxzpoHlIaR9IJyScWvuZw2axbLW1jkE2rSUkFJGNcqgtlbKbk2jzdbpWj9EtNU8xiv/QTtgbrmU3kfkXYH3mIZjWDYc/HG7UGsx7sxr3Jb43g5k/vTlh5ckExRcUZI50nYN8w++r2sqFm43GQBw7nMZ6BSrxLyoD5EYkIUIsGoZmTyNd9ZZRH6i91qO9rGI0SZGRxGLVtGHmSHup1FlB4H13mIO74Ch4U1sjGA2CGEFz6+ge6ZPi0UEz7eYe51Uida1eBRXiwTgv+vo3M5p26cWclIG1gCXdLQ0TYmnsdKVejGRTbpidK92n692zPnrc/F7y/2Y0jr0JmNchqsfuybJsJ/6KQ+bkTgNjo13wszJEiroc5aKPBwcyFR3e1ElqpmrVPtuHOae9/yFNsMf+Lwygds4neNlZLHDz8ThYP+nv9BG6hNfT5UtwWGG4lqcuMz/dF+18mjO8SK1+nRQW2ji6oem345syt5PnVTZFqmGOjDl1Pg+kStWJYOxSSBc/TsiUHHE6QAABDmM2wBruermqVEadSIN4apF33m9+SjSb5jKOb54zzehPSvtr6MuifLoRJ3Fs6lL7bhD8XZK12di8o0/13wzWrKyvafdqDrGdRd6VpCgsz2RziaMqVkjc09i7MthoC8bPXRaVP/+k+NEGLr7kU2eY6d3zsPxcUqb3aaLt+PxzuG9tb3y8PDw8PDw8PDw8BhWCLm+3n0eHh4eHh4eHh4eHqkR3pbSfm8DIdjvW8Nb3rCW5uHh4eHh4eHh4eHxnoKPNO7h4eHh4eHh4eExnHiPRRr3Gg4PDw8PDw8PDw8Pjw0Gv+Hw8PDw8PDw8PDw8Nhg8CZVHh4eHh4eHh4eHsOJ91gcjvdWaz08PDw8PDw8PDw8hhVew+Hh4eHh4eHh4eExnPAaDg8PDw8PDw8PDw8Pj6HBsGg45LrFAACRKyIeZ1BYtGB8TTiowvR9lUGsO+t9AICO8x6lc30r6TffDDSNoOMgpwoIALiox2LxDsv9dHbp08BNF1AWp91dr2kmtzWvmzqEZV1XABDFNtMGVQ5yRat+qi69K+i3ZSQWfXwKAGDjq96K1rc8AFFoMfkxuEwZqvwL5ph30QPd9FtoRs852wMA2r//Ruo2onc5FbH2TaBjYuSSUGXISknXVXAbLUiukwwhrzuV0h12If1a7XL1gSyQUkI++DMAQLBXuuA68q0nqB6dU/SzlUv/Q3mM30rna/oO1THSh8MKHVQG0PvNXQAAbd96PHJNvnALxKYHJcrnfOz3Gn8OOv/Bbsirv0h1+8KtqdoHALJnKR2EFSCIfv6m/FC/T32fDKG/I36HkLhu78kAgI/e8XbqOmRB99lbAwDaL1mcKr3sXmL+EbnINd2+sAKhxgd7zImPPzKs4HcfoPZ98m8vUSa5gsosgFDPL9W45bgupcTq02YDAEZetipV+wDzDu13pL8rR5/UacKKeXf87iuD+PFu0wAAX7x/STS99R3WamP0njBSN1kp4dVjpgMApl8/UK9pJh9+j91LgUIzHRdoHOWxTr79FMSE90XbU+oF1Diix8QgDwz20D3raDwVrWPoWrGNvgUACEvqtwK0dKmKmOcqV79CR13UHpT7TDmqr4nRs9K1j+cLEVCduS4AUKF6yFWLIEbNpHM8lnKbAKCsnme+CfLJayi7rY9Q19Q809Rh2qDyxcBaoGVUtH32WKDmLV2v/8fed8dbUpRpP9Un3nzv5BzJM8QhKSIooAgGEAOIAUTFNZE+M+KA6CqIuK6umFZX0TWgAiZ0FRFUBBQlo+QwTGDmTrr5nNP1/fHWW6G7zrndd2Yu8Jt6/jl9OlTorviG5wUgH72RbjvozEz1A6x3XSil5n09hsZ1S8qr2pYQpv3o5ySePJ3a6bxvPq5ONdw62PWI63r+0X0jbmga0mTfRWMUY5cdCwCoXHh3tgqq+dCpFX8fZ+xx5ws0RoFCRVXV9Kc/Hj8DAHDYz9e49RraAFRVe+TxR8q0dDyum3ei0rfLsPUDywAA3V96eryaGXjqOFFIKXHhijkAgJW3r05d2/Y5P8bDp1D6S6+tj3N3wDOBYFIVEBAQEBAQEBAQMJmIdi4jo8nZcNRJYiJFwZVGAHjkTSS1WPT1u4C2ProvVvfUhoGikuyw1GfwaXS++UIARnOgJUMAZN2VoiUltdYV558sklRMzFkBeezZGStmpWZrHbR2pWrO0ZG+z8mbr7PUSQh8519UvpV6168kjcWqydOWCLDkwylUQkpU6eQMsf5RkpJ2ZqlcMtnuueY9aykSazU85bCf5e8hIuClH6bjIkljWJrd8bEbICNKR1S6dPr5JCASYp/X5LgfkJsfo4OOaUZ6+NgtdG3Gcvp9+l6I6XvqPACl1dGSwpJK60m0LTvISZ+l6tj9+JZ1aXmNJfOVLmCPF2Sql5u2ahNCGmlYSpqYzt/pR4K1AzE213Zw4KKcNq5achk3IDeTJFR0kdSr9sVXAgCKb/s2JEu4FWTcSOclIpx62tF0yFpKS4IpldQcpXa65mmj433nnhPfmad6bpkdDSd/S9ZImHdhMox0HbXmolFDSvi8DUiPtxLlSv5pRn/Hjummb7FmYwtpZWvf+zDKZ11N10a30rV1d0MsOMxNq1BG/OT1AIBo12Pd0tmVt+Ym3Vf1fTFEN2mdhRqvpBr7hBCI19xJxxk1HHq+qA0D1R4+Sz81pZ0otUNYbYtuiY2moqxG7/oIBn5wCQCga9/X0znW9AuB1Aeu9uo6OHVkzQa32fap9DvcD/zlKjrOpeFQWsnuucDYgFNmufERyqrcBdE1y62jbNB7AfTcgIG1mPv8Q1TxVNuF0ipFxbQGJTLzkG6ThfRYotMqVFA6+GWZ65aE1jyrbyf7H6T0e+abdc36++meW74NcSStMSR/p2IVz/+PK53ySm4PxTbzzbXWrgBhrQUA9f34urYuYI1nhM5z/zd/vbhNQJg5UX1LWaPxT/QuSmmMIWPrnOm/J8xvNM8roxa1OQTm7rf7BJ8NmAxMzoaDBw17MhqjCWLRV/9O/8ud1qQZmee4A6nOJdp6cdfHyJRkn6tf52Qj44Z3IM0C23Sg9n0ywalceGLm511TAk6rkUjbNn3QOev7zTpPoCgmtgpo3Wn5XIyR4W1TOepJV030evLLgwpNtDzAdlxMpke3nzAHB/z4UUo/YSqSo4TORjQTfvdl+n3zfxvV/D//RKkd/Db6z+YUNqJS+n2XOyCnLfTnI2X+6qQggKlN0m+BdPuDNYnxJC3RqoCmjUUo7WABjShMcIgSAqJjOgBoU5jCcWfRNb3As++PvJuFJ35/AwBgwel8G5tixHqjsS2mAPIeMlXBRNY6tSEj3FD9UNfZ85295Sy3t/iGph1MuI5RCcVSYfz7kjnzwmVkk6kjv29etDYaxnSFBVND/WZBxBvC7rkQc1Y46XrrY5tgpeC5X71jiYgWljmgF6gyBgaV2YpamEKqctz+A8gXf4COecFZH9ZzIUaUGV65E51vv0wVU70PXuDbAi5+p2NbIbVpYMGUg/Pga7zYLHcBx7wnV/0AQHTOMmXnsYXf8U8+Stme9GmTjy6LJbTjdtw2BVi0j5u+5xtOtJ0KISBn7zWhZ6Vl3qSFHH2L6GKpXb93/T4Of5cRLqp6CiGw+t/fDgCY843H9DkAkIj1/NpcgAr1rVv0159/in7POjpH5dT7H91qzLTUvCq4vVp5mTJb+XO7G96EkSb7jW01p+KyikL+seYZRYg0HhAQEBAQEBAQEBAQsH0wORoOVo+WO4z2okymMtokZ8uTEB3kNKXvGd6opS3ayW54A5a/89+82dhqcMnSmqjY3IHSzosdBkUBpePelaNynCCrvJFS7/ocU41DdwHaqUynITC1qaLGSB190jrb8dekF7l5xg0Ui/l31t782OSJHQyVFDJTevddQ2nsd6oqF0m/DrhmDeSvldncS1fmLqdKHfKJmymNngXZHtnvpfQbFSE3kEocL/sQpWZ9G5OFccrX19kx/HvvAZ6X0MD50pgoxgbya3AAy+nUKkNSaiZjLaHyObLr22SMOW3b0R7HA91PJ/LsZjK9EVOW0Al2vHQcSg0ZBUt9jcNqjC1bxmDD1mT63klejPWTA2fa0LIFtDljSTtDp/qdjLOZo0mJ9mLyVHPTutyQ8cTeE2sPakPGdKSsxnclLS+/6mzjHM31n7GnmW8e/zP97nasNQaq/qllbQJ6/GVJe+STkkpTJmmZ/AIkWc7bpzmvQhmQW+hYa1iUtmbXF5rvy07v9WEzT9p9g89x2W0tTXL8L3cgadqMuEHzLQAoLZmBNBL5PGBNyegWUw8lqb/oyzcBAD5++hwYDb/1jRxtK4DGCITS9tgmyhrcJzwmy1kgpYTY5SW5nwF4XlfjgjIP07LcuA40aG6UI5vpnrYpVhtjgoo64rhJPylZ38syf02tazxEIE5ZeU2UBzyvV7q0FtWY65l2JVuNFVzmraswtW1HSvQlCpWJff+AyUFwGg8ICAgICAgICAiYTOxkcTgmZ8Nh2YSmKF2HiNJWVHvNLr5B0gq5dhhLhwAAIABJREFU8SGImWS3qekBK/tDXv81lUaCgtGW2rT4kF47dqY1jAqQT92Xo3KcXROpmJOvZRtvS520I7WRKm9yBav+PFtK1YSTHpWGnedKiBv5pY4taT+VxDh+8DeIMkqKxC6uLSlLMuXoAMTRH6HjVbfSxS1PQezV3KcmpX2REmL2/pnKodGubFJHt2qpuFx3rzq3WaerbVlZ4lMoG8ka5/+qC4Abv0rH+57ilG17SMZRagOeUvSNe2b3NXKpIpmmkSW7hqAgm320wNPZmU4nBFmbgFQOABo1RNN2c9Ni/4vvngm8+jN0kp3Bv/N2iLd9n44tH5Vl3/5H6lwS8cZH6VrPvJSjtrRs9EXnjNSz1XOuyV01PYbGxodBPnAdXeQ+4lDmNqfmFVEBe/V4xkOApJs2pbfn+XHLGhXQOWva+Dd6ngMAdM0xUlsmOWAb8iUvglRU34Kl/YNPaz8rsfQo+q32Iv4V+eWJY5XmlOcMiyrVpip3qFRV3uz0K+YcqO4v6rLGD/yGjg/ZJVv9tNP4kHHOZh+GhtIGdM020mT2YRzeaLTxrImQMfDHr9PxSZ9XZTP0zSz9l4o+XnRMd5yqAZBAnumHWdPCWqPGGHDHT+l44eGZ6gdAa4Dk0AaI7rmqXDR2fvwWRTM91A9orQBMmbXGXI2rxQqGP/lCAEDb+X9UZVbvXwhNpmJo16V3Tm5Gyy+EQHztB+n4jVdmqp7bF9TY/sRf6N/CF5q68PtUpCRyzR0Qe7zCTSsqYs47PuzPaGCNpqE3lhsFkye30fooUFD1T/iyCiEgj/ZbhrSEItORd/wAmKHG04dJcyhe8D76H9fN3NGwFi787XidNnU3LH0Pa6kSvpmNsUz+ty3JY0SE6CUTJ+EI2PEIGo6AgICAgICAgICAyUTQcOwAcICkaq/lO6GkD50zAQByyyqybQQMq8Pcg9N2h7UhDD1G9vWdiY/lBMICSzoagEgZKVsf2qOFWPXPHJWzngNJElK0fbZk2wqalk6koZ+rNvkywiO5dHxD7HS1z0CC8UNEENH2taXUkt1FL4Qc2UTH7KfjYQACAPkYSarEspNUcVV5S1WT3uwD6NroVpfZBSBpCkv+EtIsERUsqkpX0t0U7I8ysBZiKkkqmQqT6yI3Par9VjRbTKFk3rWSZoneRag9+QBVJ6GJ88EXbK8lRAFY80D2+5PYutpIVi1/JwDkH+JjclKwpYQHz9ixLBvVWXMn9mChnPb/YKaXk79obP9Hlf38i84w54pGC1v/BtFTlc75dSoL3aZ7mS1MprVXMgb3SZ+WQP76k3Quo1SV0rHsucuKuWnJi+ka+60IAcnBOXUQMmm5iRlfr2ndrrRbX7Pt4RXFqeyak0vLIWWMYmdX5vtTqA0ZKT9LQ3XwsxhCawcU3e+UpZDrafxmilrZGMPwn38BAGhnnzCtaLb8/rR2ytLwFVh7XoSYrliMEtSjUkrgvhvo3CHZ/P+Mz1dspMMJO33RNsU6p+6JCmbc0exOEdDgwK8Jvz07sN6QCvjWNSvtw2GzJfI70X6NwnN/BjADU+9Ck7bOTuXR1mveZ93yoUkGyJUN1LbQvNLuoV63mSB1mRXcPpmc760/m9aNVyMvHA3i0mN0eelEZObBTtUfb/k+wFYAWhNVBO76HR0/L6GJ6Jpj1SftX6XpZ6OCYWhLBN0VQgBPKEbQfd6QvXI8n+15vPYtYvY3eT1picULzzJaf247tRHTn/hdNEZTvitmrQTH79a+lqxHKkCr9ouV/vAAAc8aTM6GQ00YQkSpcUc3kN6F0J0pVgOqlDp2hB159Fu/p0icrYj6nHgP4OQsp12mrOVOYZVr1U20EJ6XiwnQDAQtTWaS0b+d+AbGzKrZ0tOnUnS55Fs4JluTUKG4fXbWKRONYlVPiByro6mq9JYf06/acOi0rA0ix/UQS435lZSqLWx8JGU24+CJ21X62eJxiBkUiRVtU80gyZM7q947Z5kFesOKTMzmCDzQR0Xc9Qfa8Kw4L/mufQNp+nu0NIURAkO33wAA6Dh1nIr5oGJTAEhxtvsojn3EBADwG8XlMAFlfSaMPk0RafNOI0IIJCONAzwxd+nNpRZutE3RGw27fpd97w4AwIfOGScvOvJcLEIqh17fBLr+t9cCAGa8sVVtUonq9LSZpHKI130tjnX9W5YPAlf9i9r6/0tesdskb15yQogI6+6hjXE+4lj1vL3x1d9T/RbKabPa7rnGfGdsSD1WxqW/oHa08hPNaTNbmeAIIawNmHqPakEuCmVsuJE2pNNOy1gx7UTtceDmfBqRcUzneyrdZs5kU6ktTwJ1prTliOQ18xwv4KbtodK1o3srNGrGfCxJYTs24DqoZ4WOU2LPwUmBmyAzOEDHrgAAMf955joAFEroetlbMmRqTB91rCg2zyqUzSaomIyRBeCA4zOkn4Y/HpYZseQICTW0EOv1X/amc9EVRMG+8h2J9O126WNn5jZaamvpNL/6O18CAMw9/tKm96TAAr1Kl+lrC1X8pyVksojhfr1BvOGViwAAR37/bxBVFhQo4p+oCHSTgDm9bmhtTqW/0+A6ywQxsTERBcgWgrKAZx7BpCogICAgICAgICBgMrGTxeF41mw4SFXGf1wnasCWREj0j0w8D0oLHumnQbS9zI2swFCcv0yqyGXDCiZm1OFZiiB878duwFZUUvt+GRURNyagIs8LdiRfdRuiuQelr+8/sciuut5TliC+g6KnRsox28Hi56XPtQCbYkQLDtPmcBzVWEcXF8KillXvPCoYMwiWtNZHsWB+E1MSGbdsfzaama9IKVGdk5HuN/Gc9c+Ux8bwRkgd9d5QVSbbqRACe/bu2HZUaJsA9S/GcS4EYESFytxpzT/IhBPGtEYIgb37ts3BX0ppUbempY+dM5MUpOPD1TYlosQrLQ2ZQKm6OfTJ6cBgfRlipdr55HUgr1S3j5mDz1xNrruHjpWjuBxYbQhGrHZ97tHj07o62gytxSQzH1msQG6ioG4RE0pYkvsph+ejVNWU1o0xM2aztp1NUhs1Y07WUBoG29yRx5zuuWhsJHMgPapwWlFJk7C4RCWJ8SeSZlxLlqfSBQxtzlc/wGhgYBFtcNpMHz60QUvCdf977CZgQJk3KZNr1EeNE7rPNDFBNy+lNFJzDlJr9UWvWU59ggQVFtLpxqhfQfTohbdcQTdVugxFt4KICrjgHYfmy4u/l/qVUhptlse0aPYp70idGxc+cgXWHPG73vyEnh+PvIqITH530nIc9Qul/tZWJBEwYw9/XSyTYp9mX7/PjukmUKaHhEObrQU8K7FzeawEBAQEBAQEBAQEBEwqJslpXO2Iy82zo12t6xDlOPRZu95koKqJoJV0rjZWb3qtKazAf8YRKi1NbO3UZJwHxzIIjn2O6pyG9AT+Mo6KNSSFhVmQV7Kp75tzIOKniWo4UpIQGdcBbeM5sTxFVASWEQ1ofCdRmoq9VbC9uA6/zXoLsBNoowbdFlk6zf+HNwHtSlqqaHHl4DqIvsXqLpXn2ACmHthEwyJjWLJIOtVEIp+0UbeuQOx2SJZaOXACQybej9ZgVHstjY1LgGCfA4AnBneszKK2leyf84ZzyqbdsM7MO0SPU7Z08O6NdO/ErLuRoOxMt+Whp0lalz1c5jjQjsQFo1GxHXATkm0hBGo5xhp+Jnmt1fsut018wPamrZ1Qa4aQRCrn1nIn9PctWo6zmZye2bkfaR8HKT1S4+2gCRfCov1l7TdLhD2UvXHd0kSoMtaGER3wUnPdvtYYcbQJAIgWl9sJ5xkVIdW7FPWERqQ2CCzxaKizoj5q+YklrBfW3AHseqy5D4CYe6CZT7UWuQgMb3XL3EpLHNc8ARzTTsWOVov9anLC0bwp3xnJZRvZhOKZ36NzV53HpYB4w1fSdTjaT+lq9wGHHIafLXBeG817Yz8HG8tfnrlOGlZAZJ02a4nuUj6YCw+xtGE0px/1ozvS1MuQiH9M9NTRe36euGQChGoneM8iRYgIsqMFzfYt36TfVx84TsWeJQgsVTsACcYEH1yTKnNeWjzo5t5tK46rik2zdZRabIyaJ+prOMlBIr1wlFKaOvIkUKxirImPXvbFfgQZ2YtLuJPLBHYceTn4nef6lBnCFmK7Ed1zgHkHb3OeOgqx4twfWkmL8I4LbwNaOZT70P8I/c450DgaPkWO3+CYHqU2057VoC7apkBueoyOmXWt/yFgSZM4IJ7FT/O4F+obxhbrCUALi/n7ZqyYlbVteqBNLNRCi81Hbr8SYgU5aOoNVNwwrCPMXFIo4bh90g7m2wp7EdC5Tz4zg6xIxaYB9AI9Vlz60fxDsc82mlTZeTF0xOFqD9qm9OVOz2sWlzCHkcMbaGFp3/n0feabM+MSJF5/kN/0z7+RsGIJgbMUKUYwe9HQvfc2TP6NUcjkwoUXhnENQpEfsGMubvoviJdeoIqqTD4KFXSecYkuD5dZ/0+Nj8ZE0l7oy81kUoVuD3PaC96as2LWd9MmmgkmIhFZ84r1zXkDqRecJaD/yXS6DHbeZjNJZ9FtEXR0KPOlpGO3iIBNT2Wrlg0n6nnCNIbrVe4wdVSMa5CWeMEy6dx8IzGN9Z54eSqrVH/2bEaE8z49ePA2+j26+S3jIjm2F6vGzOlVFwMALjlqBT5wiqfNTfObG1Ek70S0eA/zo6z0QK5VzIxqbnKEkhMhfuB+XSwCNbXReOpvdO5xItTAE3cC/E24bbX1IilAhowRHfs+fz7WfDTe2qQlm+N+J7V8NuCZxbPGhyMgICAgICAgICBgp0DQcOwA5DXB8XFMS/Nhtta2n2e/b7c8MpQhzHcS7LhlRQBN7fA9PPgAjIpySEUk7piOl8wxzm+p+3XZx3sPKjN2gLRU5bMWe1Su46WmJOAig8YqBXZc71YSydGtkL+8iM41oQnMBUVz2nEhSank8EbIW0m9Kl7275mSaFxP9xeXv9ZIBZe9in6d2ASJ9y4ERN8iOmbp/4xlGP4GSXPajzjPvX8CXOFJyk4ZlYDbfkJ/9ntz9oTsCMoFIz2kcqnv2tad7rNR0ZJOGs1ZbXQC5ofjwGnXY8O5nvWr4RN1iWuW1tXicOf7leZNSokHt27bWOMtj0XduHUN9fl8rvGG+lNrwLQWQF3rW2zaTKQkx9P3grznx/oYABA38NRjpHFJ6lp8VMjNhI96nFbaG3ZsFkJg+HGiO81jNqbHPdZg2tfYkTquGY0pO4of9X4jKbeiHo/+6FMAgLaPJSI8W/TJsmFLnQ2pBwCSUnPdkoQDUgJ//h86tWtGIoyBtfTbPs3RPAMw48PYoJYq6/5YajOmnKwp65gG9Kpo3Vqyb6XJz+qYVsKvxWATNJW+Q90+dWG2etmwzbuS0mturz3zTRm0tVDDzKdbic5YdM1B926slWuBZEyhBFIWDUw7XGoHej1OyC3gdV5PUoyLgnn/Svv9gVufsrRqpkNd9hLSzr//1jVOPi4tbmtaZzHb1arbZVt7LqU/68qBFrVKJUq/wxtN5Hk114rjaf6W913raF4AAFFJEzoIZd0AGQODG9Qhtw1Vn/qoIVKILef9FmsNmYzVAkEUvQHPWuxc26uAgICAgICAgICAgEnF5Gg4hmjXKW2bYh2J01C2ZpHmy6H1uODiM+iYd7Nsg75llaEvTUS0bAadLjvW/eDfsOisldnqZcPJj+1oVXBBJbGiYlqOiSp/qSQHbI+MqIA/PU17wRUpB83sEan1fUwLaNkDr3mEJA3ju21b6SlJwpp3LMLMKx5yr3mCZrnX3aYmKl3ACWRXHT95CwAgmkcSmPEpTT3pJyShoq0PeJ7fCa8ZCqd+np4VQr8rrXVqU/LfQiXlwOf4HynJndx4P0Y3bQRgJLtaurZ1dcoOPHed68PGiXIC2PLBfdB1NkllxbTd6ST3xd0sKS1XrDaknUp10LzaMGZeSkEyJUtFWUol5bhtIomkNkAIAfG6tL12SygKULn2bogZJBGVum2o9zu8EbJ9GmeifiMYSavRAr37LeT4Hz98Pd02S/nNVHuMk732rxGZ+qZdzxmv9NA5j58A/dSHDA0uSwVZclysmnxY0l8bAmbva45VWtNndqp6KO2crlc93W+bUJVrW2/WFlh+CeUp+al/eX6In7yNAnEC2j9KS8s3PGCCddptjevLfh3VHlTeRlpUdpzW37jU7kSzpucbJuCn5T8iFh3hpq+iOqPcCbyoiW16M3BdACQJJLQcMCpCqP7k+F5VVT9lx+D6KO793KcBAMsOVxEqrXboixzvo03V7Vhr4Li/9KP/ig8CAKYdnGNM5ffqk8oz3Wn3vPR1ERktuiUd//t1NwMAVqgoo0ZbYZOyJL4lEn4Myf7J7zduYPi23wIAOk7OUDcLMq6bOujvxPlLQHA0cZuWmLVHpoxdpXR5TT09BDS+sjRZQ0kpEUUTkC9z/y93Gq3/7APonFpXiP3emC6XKEDM3Jvy5rGgMYYfnH02AODk6092yinLHebbZLSgMMEbWVvUwMbPvQ0AMPUb2YL9PuMIJlU7AGqj4TRKkc46zcTgOdc2FZdf+HUAwHnHXewm0DalZRoMmxHLOKqrDn3yFXj01HkAgMUv/miW2nECnLo51uZglho3wb7hrLEs1Xcvj5ueQVUm0nBU5E4cjoZ7TnOgx+idlt/Zl/Od9dVHPWVJ/x9vcKx/+bUAgNLZ1znn/Yua8dNLlfcXxIghTv12pvu1WQasDZRin7JS9ZqVpDYhM5bhizfRIvxj+h61UOqclak8Pph4C1U8eSNFpl2QZ62jytD92fu9GycAkFHBDP48IZY7ICqdbhkKFfz4eHJ0fM31axP55CgTP+L5vps/djgAoPcrGVXliu9fzFnR/J4O3wLY6rcW+8u3vkuLnNPP/En6CStuTtNrkJZ0IX3f2msojszs132teXmT0IIEy7m2iQkJADOBF8oQHJHcWsxcdSdtPv6NNwtWeX1jTfo7WUxLZTf6r5QxHvszOZkuRQ7w5nfeIUDKRFXdMssiTeAyFavmWLPZCPzXScSE9O4/rU6Uz06TTZqaXOe4Dtw+lBBCiAhrPkxxOGZ/N6O5CrPPtfpuVvwUO5pykn1JliJc9TidW95ijBxfKOS/LtumYP1TNJa14AfyJOhntgOsec/Z+Mj0dcvkayRBpOLb3PvbLl9LM/PZ6cdj+UypzTcx39A4rduxtxIThoi8Zd/EBHMTJGdxzcXSxA5DA6P5E9XxXCyTWhaQNpmn0zB9c+Nos41Ttjq3zEdE2LqRhGL5DcYDJgPBaTwgICAgICAgICBgMrGtlKvPMUzOhkObT6Wzk8y9XTThbluaVMV1nPPBk1WyyozAcrqLb/oPuv+AU+lcqSMRaVchQeOoo6I+9H9YeOZ56fvHgVx/PxWBTbqoFOkbW7Qv41AHrNe+gklJiPAf+9TWhYSj3naixZVxHY+cOh8AsPhbD9A5lhSVOzNLaIrv/L7zn53SEZV0GhOV9gCAOO6inA9kMQHySXitq/yO6qP46HuP0ccAiM8egFz1V2AJ0fhqik8htKmSl/aUnRs5VsjNX8a8N04gciy/Y0ttnUXa5HUgHt2K7qZmAPnN4pKQUqJQyRAGu0k5kzFMNM9+qS2zWeJp71OqeaZYvYM49cXer4O8ilRL4qTLOUMtlXacrFMcA8aMc+Zbzs1YszTcuuojvprt/deG0ZOyYGC+/VEjgeZ32bLdeCSQY4No68j3Dd20PVJUHveHNxptlWUGJ+zI0iqtd13OYwFTQBszD00zy3PA5seBuSruBJuqjQ1A3kxaKPGi96tryrFZFDDzg1/JVz82W2uMpeMVcHlGtxgzMi59o2a08Tp+h8TL5zVMeoBOkyTsCQfluOZ3xk/OiTqWxyjmLBk/Uns6wYZbTjtpj3bAPzkaTdncHjcdyWNisWJp81WZ6yPaHM1Oy8SiUt+ftTrD/Sj1bQe5OL9rNdbIcnva9HJsALLU4eQvRIQPfOli9z42D113D6Ci3GPBYfRbLBuaY8boFsj+h+mYTQ2FMc9bsPKbE66PT/vF6y9RKLUeayzNaVkzsSdogSEgFUFC+rtZSXnz4Tl3BJW2/IQsAZOHncuALCAgICAgICAgICBgUjFJtLgtqNyKOaVfooA7/oekjPvpAEBKajE2BPE85VFm0V56JXG8Y2dJBzvUdc7AyA+JRrU9I50qANS/cxYAoHzub1rel8kxHhFOXOrfC07IoVqnyz4cDfTtlsui2k0vKmLxlY8BAB46hTQdS791j0q7mp32tez6kbD0VG5dA3RN3M9Bo5SW4rVEC00cI/O7FxGwcLlKV0nHtxLdYf2aS1A658VuGevGvtaWgGu60ZJ2PaefKfOB6UuylcVB8/aXzH9clNoxqFlx00GesmmMPCW0nfEb20K7a7RNvvNZ8OhPiEZ28Wu+QE/udQKlObAWeAk50rLfiM9Hww7emL4YYeymqwAA1Zfk1MYBlmM2DEFFQro/LopVzG1XUsbRBAlBfQSic4ZKL0ukbk/bKVZR7cxPo+36jiS0HZru16Ic1fc0UhJ9iQhiL0WHy1JxO4IyazaUlhpb1wKKwEJrH0REdNGAmTNYYxaVjCNt1vppR9qaodtlZ20mMLn5a8DRH3HOYXSL0YRaUeUPePtbTTkBQ7Vuz6/SfGcdxFM7OzeMBiSyzgFAoYzOV+Uj4HDS9qDVPOhNSkRYs4Xe2SIea+x5xgqaS7+m3t68khL7cgfikXwU3F5wOdR6wtHu8PsodXid9h+6lHxGd/mh+pbsozX4NMRSFY2QCTt8Y0q5C2LWPul8FcZ+TNTQlX3ekLEyMHOXNIEGja9fzuVjXEOFXzuPydpHJDLfbqIYG0S5+hzzEtjJnMZ3rtoGBAQEBAQEBAQEBEwqJmc7yDapFi1fEj7qN4dVgiVSjRqWvXAffkr9inT6HFip0m2kPFpyI4ExDm6kJMiDT9P/timo7v287HVTKB54rKdOPilOmhbX3G8kc09vJWlOkiMpK7zsTta50XUkbZ9ACD9KW0nlln5/FQBg9TtIYzL7Kw9kT2iYaGOTrEGia5ZjHzphsMQ2MyuUx158olL6gXXAQmUHzpKbAWLIKb3920ZiyZKesi8smkyzBCnJ6CMX/RsWX/6r/AVTEl5ZKBvfhqSvQ9xIS15sCbe27R7Dnj1u4Dzty5NDC5fsJ7aGp9iZh7g5AR53+P2L5toXH920lBKlcuK+p++lpOasIGkzp8fwtJdkkCvhlGMCfi5MaVsom2M19ukxJK6n6CW92tHaMBZNYSpdNU5yv6z2WO1vgr45tUFUenuz35+ETSGalOoWq6YfaYm1xQqW/P72OTvIHo8x/C7nrkhrhQsV8364rW8ke3kxdTe3DWQBt51i1VBvlxJjQHuv0ciossn190PMUsHdOM9yB3DASXSs2eV830ia+5Pn4obxjUnO0yICdj8mW708+ck4/W7k5ifooNcEFPT5f+n23BjDod8ltjMM0vuSQzRni56FkN8jDYx409dVBjLF3CZlnA4MqLQ4sv9htJ13bb7a+Wh5tbUFsZXJYsVoybh9RUVtYWHG3Dqq7TzXuaxsYtERVjC8FhrkuJb2B+KU4gZEaQKzPc9Ppchi9FRtTAc7LmYbE0Y2Y2aVtWj8/nnuKUEiZx9KIiqh3NF8jfmsxE6m4ZicDUcL1VtyQkvCqO+UE1yhjHtuvBMAsN9ZalFqmxHwBMSL2Liezt/umOqa0CY8AgO3Eed+15vGrZnBHi/NeGNyUyWthRYvABu4dQMdH5x8PIepSpob2zTuoU1kDtCF/BBC6PcrFdf97M/9mf5/70yIN7nOaVLGlvOZcqCTEnJkE51LbDikjE3sEjY3qHTnd0Ku5VSR+xyjeXHhiyrbyjyucybkrxRt8+z96Nwc2oDIofXA2rvonIoVgdqwpS5XGBsyG5FEP1l8/heBfxJvPJbmWAwkaFHp2B30vPSYvjZXatPxYvZIUnXmICVo9l1FVMDYZmojeaJUc/4isXCSXIfBpyE7Oaqwynt0AJIj6Fobn7kfuYKu8waYnYkBQDl+jn6GzOMqZ/3ExPfQmTYg772ant3rRJVlgSuI8hs+lbNmcGMzcB3rypSjbMbJ1HO+dlrpRMwRtu2NEAC56laIhYerc60pgJv2zUo3xAT4//1OxSo/VsyX2ox5k+T7I8sMUZm1iAjxI3+g42VqYc5zQlzTEZTFfCVoKpSNuZIaf+ToVoz8ipzG2w48je5nGu1yB+Saf9A5dtYdr37K4VeODZp0+N1yBPG9TzTRnTkGQu9i/Z2EirguZYzR/yYCg+pHb1L1sxavOuK6JRzwxL4w458a16x4LLhWmfy973eZ6pdMO9k+RN+i5rd7xiZZqOCBM4jqetcrKXK9mLor3RSVIN7yLVV0rmPNSsOY3dmxqOhZNf9PWYqxy2gOr1x4d7Zq+foCrz/4u0XF9GZ0ZHPKMVpERcx9L9P8J4Q4f/4vYL/X033aBLwDUqiNNG+atq7RjuY6tpLJAMVXfSBTvRzwJtteM2niEUNMkIR3TGibgt2mqb6pNpxCjZdydIummbaj0vtDGsDJV58rVVHq6ctZwYDJxM61vQoICAgICAgICAgImFRMktN4832Nz3zKSI59wXskZk5TkquEw7mIilpioc0Y6iPp6KOWE6ncRM7PQqt2Jaqz3CjQmfDEX+nXlnDpYHwtnOadwEDGFODYeU0kxLIBjkwrLSrAlATJR4fI94wNoGv2BKL/+spfVSYv6vfCy6/DyoRmSIgo1QaEEEZClUzTjpKrHCn9QZvGKVv3nFz3e6HU4LZUuRVlr/MtX6Cc/5gCUbW7/guOwdT/VKZnyrRCDvVT9HW4EptUGxpR5i7tU4Bl2aSpbgGV9BqmzSSpNNEYs6RXaWmTTZ+5X1+sr9P9bAJpTHrGcxBNmXRZpgqljuYUia3gleYryXe86jbEf5KUAAAgAElEQVSI3Y7jLAilqveZsZ9eCgCofujl6UzU+FP5IGlE5QPXQez+cid/GUtg3kFcqFQZ5V9/RH/2eFW2iiHxPvkda4m2MWPJEuldyhh/XUvvewmn1TWb0q/2GBMUNkEtdzkOuboezb5x3MDAWjJ9yaulaga7TSLpUA5PoDUAYokiaUjORVHJnOPI4XHDSKhV3xDFCv7z5zRXfOD82DwLNe+oqMpZoftQbQiQSlLN85S69vi7D8aCbz+RKK/5psZ8roHqeT/jk3yVfuzvoQgYZP+DEDOXu+kKkfqudj8Ub/hS1qql0Eoz7TNlbHInZi1RcxZr9RyHbOZbVdL+TY9BKIp6QwvfsNYAan3Ada6PoNSXK6yhH1pzYbRFQrUTOabGyh+dB7z1e85jQgiI3Y9Vx27QVTz/XcYsUNP5WmZMrAUsVnDHaWQTse/VbiBWIQQwe/+J1ysqmbbE6yfW/nVMS39DyxLDfv8LXkQaU9GmzCy5L296FHj493Ru+Ws5EXjn/NS6ymj/ywelTduf1djJ4nAEDUdAQEBAQEBAQEBAwA7DJHGIZbHn9geq8p1bt55s81vKr22Jh07DolNUkg3tu2FJhkbXk3Qgl7vyjN1Tp/wSxuZOmFoSUKzilnV0bpfk47ZTmCVVaGkzn3QaL3dguL8fwMSljqnAPaosF1x/C+JVpO0Rcw4w96jrtiQkvvnLAICIqYzHzdOV3tkaMF87ie+7htJfcUam9JN1AWAknQrj+ZHobykKwN9/QidZ+qnsa6d+8g9GUsUSnNu/D3nUh9R9SqpcHzb5W5ItgGx05f3KaZzt7LPAo51J2vvLQint12HfzzbyMsajg3T+AEtzYpfTSbeJNDOdl5FgN7YHVaXOX7WV3Y7LLFkaWv0UAIDdjh162GR7XHS4PnYCy60hnzN0qRHLCsgZP0X26PkkPx6/t6SdeG0YqLi0083wmnNOd09wW4trWuq+6SPk39DzkWuA6Xuo6+ywXdRlSpFhCIFpb3p/pnL40NpJXaY1FkKk3oWERO2KUwAApbN/SSe11tvSnGqncOv5ClPV1vD+719Jx+wbxhLcxhjkr8jHQbzxymwV43LbTuv8XVU55l9+vSkbzyWljhQZAESE4UtIY9f2sT+66cv0OxJTd/VbHfjGP1Uu+Z0z6dI5f8hQOZVMBs1F1gCcQkS48e/kLH58YmxxqKdV34JPey4i690l3mu5A4MPES3yRPwa7XICrvWBnrc47+M/DHsOY8jrL6ODV37GvVasGLID1kYn/f0AyOF+7PO6V6o/6W8p7/4pnTryQ9krpIMFF0zd2K9Qr2+aa5OpHkzAAAz8i/xjujhd1qT3LjKadtZCSWm+p69oyYDGjTFg1T8zVuxZguA0vgOQwck5j0PwL1fRR9qvZXqqc7RZ0UM5i2LVfOhqn5O/lDHWPkBsQtmma4WujOY7Qxson6RzKWWuyim0qUoSznuyBxVeJDATRrnTmEHwpGmxQJTbto3z2qh9E5NZ7wLInvlcQHWLv5mJvU/Knx+AX7yENonH/2Zty3YjFh+ROX16IFvnT5oAeREV8K1Lvw0AeOsrL6FzbJZVHzUbx4oacI/6cLocRcvpOcnWVaxi3dfJyXBWngkkg1lankXAumG1gUltrtMO+FnydtMXxqF5O8Asgj3jkcfJGgC+cBOZ2azUaVjvRlenoNPQk6COFdGOaI9XNC3Pqlv+DgBY6L3DD++mLfn+K52ZYh0IEeHCj/8PAGDlKz5N57itFQy7VO9lxM5lby7Mosla8KUyKODRL9BifMlRH2tajubla1H2qKgXdmYTbAlw2BwqKuCBPxFJw17nld10rbYgeZyUsRYOoK42F21TIFhwwP1YRX8WfYtRe4CcxjNHleIYHnpstuYgFddFREWH2QgAmVnyt9bO4MCGR2hjPC/BUkTmbuxkr+pcG0mbXjVq1iaIY3mYWAmj64nVcEdwAPkinPu++3EfOZcue01ZE+bDgCYckdcph/cVrzUO+oo8Qwt3hEDHyR+cYA3SsPujboeq3XCsjCS++dnvAADOeNWlicQKlpO5MTHVC3RlCiim7gac/F+p/BlrvvoJAMCcHPOFL06ans/LZl7XJuw6+ns1zTApCug8guIYaWdx/l7t0xAxEY0VBV63ja3UvuXPVgLLyTxS7HuKTpfz5GsBz07sXNurgICAgICAgICAgIBJxXMsLCOhllPoabQXfEZq7YBPEiCi/I48PhpKL8otnGAtySXT4rZ0C3akrarMbIIjhKF71CpNU9ctT5Oz8raoj5uC1b9sZlNq81MI/u97qHxv/2GmZFkactyvSdoma0Pamc6b/k1fpGuv+nyOwrvwSli1Jqrlk1jc2cTMzXL40xIcGRtufi0dtxxa7YjHABDX0d6dX0uVRZOYJ9bCRH3e8kYafq4gZR4JqeMQ+MaaUjn/ELy9353IYvJqm6iOQ2WeRDSB8TQrku/UfSfm+L7NdLyspbbHaD0kRygqWe+G89J9sKafqw+RxiKzhkNT9vqd3NVFq9/bZrFsDsRtR2DtRpJ2z0/Gr7BMlbW5YzwGrauw43awKQvX3apvTdFT75AoBx4yBR82XU0xNqa8yKeJEM6PHYdDHE33y/oIoKiETZ7GBLRxIzlyR4edk7sKLcFj/DiGHs3WNU37OWu/WMKfNR5GDmTR5jvtV8cLadITdmkS48ym+9dxi6JUbJhNt92I3oQWw4mZM2uvpuV8VmInM6nauWobEBAQEBAQEBAQEDCpeE5qOAoT3MQ7UmUPBSejWMofXVoqulITQLAJWgRBtJGljk4QOr13tIIGJYLWwbq7VN1xn14koh7Le38KqKBnjqTk8DyRFT2SnlK7dgwXe3poRffadoo8rzTZ03Z8kqXuUhPJ5ZgJMqe/V23ISN8YI5vMOZY0cUDEuAZRmIh0PIP/CdL1bhaEaVole4A/tx0aKXnWMu0I6DLVhlIaQQDYt4kvlS8Nb8CyuK6pt9G7iK5Z/gadU3N5iqXyaVqm+qiWULNms9lz3VmUs/xso278ibh/t9CICSEmrgbLAMnaVMsJXNuYW9qPlx7kpwH3knYA2hZd9qsgc1N2hbz353S84s302z1P3962Rz7KUZt4IaUpYv8RhxSAI4JbgWxZIiwb6Gu3qYLdZ7Wtv9aMlKH7X2z5PrDPRjFBOxvXUWjbXqTGLnz1l3Fd181uV23TlN1/wm9HnaRfbg+AaRNtym8nrkGuJ6di7cuhA9vVUejZDrS4HnCemmwhKjljAED1fM3+fusHXxsVNjkC+wPVhiDZN6Xoar+FEJh5QCuv1+xwaIYBhwIdZRrP5IYHAUVLbBNIYGBDuh6gbyrrSpuhtf51YxGi1hR9n/oz+j9MWpIpl7+ObtMaVGGsBJ4r2Mk0HM/JDUd7jlK7kaEtZiVLHZ14AqPDNeRGI/1McqHvDBJWZ2UnQePAN4bD5/obojNAcxo+J1jZ0A7q6JzJBaDf+gi6500g1kgGOExEvJBcegw2n0eDT+/n/mnum5eKo54bYvfjKT2OzqvjdkiIvsV5U0ufahVLRX/fGOmIyBL7n/EWlQR/a2ojcuMjxgGVHRtHNkF0q2/S4PseBjro2/FGVreosUF0vCobu5cDrd5vwXAmjROn3shKU0d7wlnW699wuO3UimSd2OBLxGahozeqZjKqTuOI4DsYpTbAw/60biSLk32Le2QM0aHqYDlD8nNjQyNNHpwYtMOlEGYyUwtOGRX1OZsk47V7+nccMm4YEwke32pDABNeWNG4JceaUQw2dpTqOQdMIF5MVqRYqux+ar7LQD851iZNSL1xkACzqeLYHI0RYOEhzcuxtT9zkQHrO9VHTbvnvskMZ7VBUw4eX2TD9CG9WG5g8Wtfb64DMDEQYjNOWU7mPO44AjAd64HnMmux/4LscWIYvqjQqWtx3WJeZFPVQuo+IQRKfVOd8kn729vMRgDkpkcgpu3BmdNvo5aeE7hdR0WgLf/mfzzIuK6jz8vVt1NxehZCqjnZfi9Tzv1as1QsH38VZ6RRB3jdoGIVyW++CdiForGLI5kZzhIYnbgyfwX4/djO41yGLasoh/ZpRkLKG0UfS5iUwDrawPNYKPmbj2yG/BOxVooXnqXzEYK/iapHqR1TLvmbOuURsm54LH8dAyYNO9f2KiAgICAgICAgICBgUvGc1HB0TLDU2szBdvAeIWc4rbpr1NDZOwHXOCUBaGVeIKWESKg6pSgYyaclnbp/PR2nZfRp0x7AMkexqDolm3dpXn2m7CxhcA1R/24bOW4aTpRwlp5Vu9H9bopUG199HhXhhMvSdIgTyY8lqfy7+Uk63zMvU/QXF63iG6R5xV1JXMI0rzYMLDlIlVFFuGbTmv6HAKZG5GjFbX1GYsXn5h5k0v8XxQ8Qi4jqV955FTB335z1wzjmLar+o1sMbaQvgrMlqd/lcDIlkbajH50wybKUuD4KtE1xk40b2vlRWg7xACBLbSgvOzRz1SYCH2WvVFI9USihL4fJmBdRCSgrCa6qV7yOKGajGXuhuJ2cxvW34TZpUy9zP6yPaHMdqejAAWDOK1+j0kjE1rGlz0oLJdffj4jbIDuGtvVBDqjIxg/9H/1qE0eB4sGeKO3bAK+TtcXFL4vJUU1g5se+q27jMdbqB6xhs9usmiPEVIqvJEc2469vpXof9ONHnDSkjIETLspXCT0mNwyVqC6uKluhpE1UdL+qDZvrVp+rPXgHAKCcpIcVwnx/rc2QQCFhluVQGyfiG8gYeOyufPUbB7ajbxbabCklRlbT2N6RkGwLEZmxg82sRre49QVojrffi53n2AAwuHniFdLlTNLzCiBSWsWZNObLzY9DMAUsF8eKDWbH8KCyDVrfXzmKl9qNGRPHVnrDl7DuXLIamHH42W7BCiW/ln48WBoOo9lW792Kfp6eEwt+zfn+Koo4a9GUOZgc7gdGaJ6QGx6ga/URiIUvcNMotaetRGzt39KcVPjPNHYgocazEUHDERAQEBAQEBAQEBCww/Cc1HD0j45/TyvYEaqTNHmiUMbWfpIizM6T6O0/oN9j0sGtvE6J1jVpOw4DQFTCCw6Y4TzjOqQmaX7HgR25GgCiMvr2Paj5/U2gnQ+jElLaAGmkMknnXyklxByyLcUSorSTQxsg/6iCFB13ce6ypMB28T3kyCnHBiHvUQ7lR3wgWxo+e1VfXTxaAjs6NqVRxd0fI9rfvX9KUh3RpVpUpQv6vXHUVtu+2iPxE7u+jA6UU6TY72RsOZ+k/z1fPiFL7bgCnnq4UW9loZJ24La1OZYz6d+vp8B1B53tOnnakkvJfaxsOfTaRVJaD9OuTXC2ob/+DgDQcUrW6o1PGSvjutGMOfb7PAaYgFW1ONF3rWCbth8E5dkkerN0NXli2u76udHBsfQz4yKtiUvZyBerKamjLLVrDYc9Jv31vyk69sEnfk7XDYBj38/fUCw4zOSp6ZsLQM8Cum+zCoZnvd8tPyHtZu9LcmoBmsA7nuqgdeV0XwSA237olN+5xv3d0TS4Ub9FsYwlC5UHCEt2bQ3AA9fT8aIXZqvEMJGMoGOGySvV5yIrmrsw5eFxShrn2tKMee6zejwumfGEHaqjotHqcL8uFM0xz0O2pmPWkmz1cgtBWbRyjPUG8fPQrEZFtC9nnz+PRoTzUE7DYvqeQFJTA1j1Vm3oYfpuYulRwOylrSqTCWlLAwPWvEXTdveuBS5/LQXCPe+WNe5z5Q6dIAcA9PnHyPapmHHFQ4lMzTd47DwiUVn00xxjjrbcsJ37TbA+AK6jvg32HbMDAD58E53j8aStT98rXkprJ8l9o2EWevYYK9mfY5XyiZnNzvACWHMnHc5r4W8V8IzhObnhCAgICAgICAgICHjOIrBU7QC0YvrJm5SUmNnGySbtxiWSLCzNkAqGpHbtcvBpdPZNwIejb176XMLG0GZEcTQWyhZTVohhCfURbHiSWE86uVyaEMnQUo5fRyXp42CALLFsjGHo4ftU+tnB5XzitPmY942H3ToWDL2gRN25XwgBKEmGlsxUeyFeTEwacpRsN0WlU9+TN4BR0jdGlDuAfV6XKw1kCN6YuV1JieXnfYROKgmQ3Ep+M+vPPxrTv3Cf+6BTZ0MViIQWgr+93Pw4us749LjlTRfQ8pPQ7cgd9EST4JQpKsPaMA66muyqL1wxBwBwwXWkkRCdMyDbpo6fhoy1FFeyZJWpFSXQvvKWfPVjqtahfqCd8098s7FBQLGZ2dpCH5POKacoifiP3k33v1xJ6Ss9jqYHUFodW5oHuBNKcnKJa5j+ipPz1M4tn0dTw3WwKUMdLY6SLBpGpDEsX7GIitNPfVr0LlTlq6ft4K066PE3rhupuNJWcZ9GfRjd+z8/f91smlNNA5t4f7URzfBkFcotF0B+cvOWOXXEqGK1m7KLQ58LgNpHu/I1KunJBlMvudWUCTBaikonMHVhvgo6mvUE246tuUpQjEPGKdt9yBh//+lvAAAHnJZmX9RtXNvcNzSbH4PaCGv9u0xe6v6nfkw+MPNe14xJKY1MNNeeNYHTD60x+YZvk8b6yJOvcMonbctwZoRrjJn5z+7/ibFN7M7+RRJbf0tasJ6Tvjx+ucfBeAxdvnPTteKM1zUcCM+m8bfKnmy3UppvlghSKWWMwgTo/nVwS2tck1Fi/dFkvhBFN/ifrI/ijs9cCADY74dvVuVUa6O+JVpzGKn2577DtL+qmE8afva5w9gABr5Jc273gW/PWsOAScTkbDi24y5OCIF1I5xsovgZ16j+yK7q4fZpGJmAmYM48HRO3fIXS8ZraFKGJC1joYTfPUYLgrcmH4yKHhpHYR3DnNOmHEI/S38FtqylyTLPhoPznf+tJ6yz7gJLjhNnxDbzqF1BVI6ls6/z34PWA/e45f3F+fTcG6/M+IBvY+y+12ZmcanrIsJFZ30GALDydnLgE1PILGH6lx70TMbpyKzOhJIwExHT9sAT5x0DAFhwVY7BVav807S49ibAt1hIccFXu/HdI4mYYOXta1L3W0+m8jRpRoait9CTur7xbKK27Pvqxhbp2wmqtNqnNTdlrHR5qVCTtwkh8NVv/wkA8I4/rvaUXaUflVLXEjf6z0clrLmaohvPybGQ0+YVPvM0j4mY81wivoEUVXzlOjKDOmelazYjxxm3he2MzItZ5Yyu0y+346FrifBg1zNbJpcuK2BiQrglo59yu/XREmMcYBy/RYTL33cBAOCcv7zdLZ9vLii1+69XmaI20vdxWqsvJfPJOf97OjJBvS+3nyW+l7WZEp5+y5Ayxs+epO+6osU302kUmvXtRP7Wgn1gc3765lbmjbaJYiqGlGciF0KgoT8Ft//0u9PpJjf+Ov1mc4hIpDcxtCR0aFEOIQQe3aqOE/1aZo0gLiI91qTHb4FGbQIkLd7+1xz++qt5tdyB36+hcu1f6Uo9l6xj5vme58lqLzasIkFCd4vbn1XYyTQcO1dtAwICAgICAgICAgImFUL6xf0BAQEBAQEBAQEBATsA8T/+5xnNP9rvLZOaX3AaDwgICAgICAgICJhMBJOqgICAgICAgICAgICA7YOg4QgICAgICAgICAiYTOQkwnmuI2g4AgICAgICAgICAgJ2GMKGIyAgICAgICAgICBghyGYVAUEBAQEBAQEBARMKoJJVUBAQEBAQEBAQEBAwHZB0HAEBAQEBAQEBAQETCYCLW5AQEBAQEBAQEBAQMD2waRoOOTgOgCAsHZzUsbqiG3YpHOd7pEQCdowGTew8Zw9AABT/uOBdF4qcDo/J+MGRFQY9z47fXnlGQCA6J0/H69q5rmBNZReZF4p19Gpd9xQ9xWs+1Sw98YYXStWsPKA2QCAlbevdvNp1CAKpab1AfjXqpekPBHXVfpVxF99DQAgOvuGDLVTGFpv5ed+P/s9+urNiFf/na7N2g+DH90XAND5qTvpufoo3VQop7+Lrp9VD+s98HX50G/o2qIjIX/+EQBA9IZs0Ty5nVKdJCdMZzxtqGVaMsZFK+YAAD5++xqnjMm6TQRSStS/cBwAoPTh27I/t/UpKkOh7JSVYLcZ9Q1Vve2+KEe3qtsjDF70AgBA58V/d/OJ64BIvzOdhvUuku1FbqEyyrEB4Ob/BgBEr/96xvqp/hIVdTvhfoVihX7HBoCxQTrumK7uqQGqb8pff4LK8/JPQa6+nY7nHkT31UfU/XWg3M6Vpd/hTUBbL53a/CQ91zWL0rbuk7/5FF077kJgaAMdz94/U/0AQPY/TM9Uu825Fn2uFeKNj2Lg348HAHRfco+bj4zNNxmn7Ta7Hvc/DNz/KwBAdPTHsxdMjTVSSmv8Ur9W29XfVp3zlU/GdTz9rt0AADOuoHcn+TkRpdtpfQSi1ObUSyWkHnHvlzKG/OcvAADRAadnqp5cp951pRt6rOFvx3NI3LDOWXnye+BrYwOQN19Bpw57D6W/ZRX975xp2j2Pr5BAQZ1rqHOlduj+Xxt28xzZDPnw7+nU88/KVD+g9bxvt6uWY705iwtXzAXgmRPtdqr6mm+OtPNP285LyN99GgAQvfJzzStlP6HmfIiCGWu4rvw7shFyeCOd6pxF5+I6RLWH0uBxqNQO8NjVOVNlEJu0OL1YjSUjm02epQ66fWQTMEzjSaTGEz1WlzuBEVWOqbtlqh8AyHX30kGlE4jUO030OTTGTJvktsZ9FtBzKEQEeePn6fCIc+gcpxnXTLvX7XAIUP0Q9nfjMZi/cWzGB/l/NLZmnS8CJhfBpCogICAgICAgICBgMrGTxeGYlA2Hlj7UhvWulDUBegceFc1uliHjlPRJRAX0vuwNdJkl7mqHL0ptJL0EIMuddG1sELAkgU7agFcKi6lzM9fNpCetQ1dCo6UYjTGAJRss9RzZZG7ka3ED5592QCJ5S0qgJAwszRXt0yDKHepOo9mRW5SUVaWLcpdJa8Ur8ldRlVVUe+2zOj91oCUVRpoELYUQHTPof2MUgxvo23fyPR4ppf0u9XmPZkPfM/8wU45N65ALXMZCCTLm9qHep5aqVDwSOVsSrJ4b3YLlfT5JnVvuvNoOXd/6MAqVaq5nAZhvUx8x0qWBtfTL0n4hHE0dPRCbb8x9uFjFo3eTJHVZsj6NGlDkvpXWurn1drUe6KQ2EkVzEHdNzVU9lmzKRs1IaCPSRMgfvZvuefkngLYp7oMjmyDvvpquv2ylTkuuuoOus4ajSGOUKAnEa0gzJ/oW07VKp36/om+RqRtrTp76q8r/kyr9MuLhB603kA/xw9cjWvJien6CtsCiew66PnMXAOA/DiUJ7PtuXp1Kc7x26usTACB65gOHnjmhsgEgaaZubyQ9ffzN8wAA87/xgNFE8DuGSL8LUcD0/7zHKZf+TiJC/MRf6Hjugep+f71lEwtkISJgl2Py1YvHQdlIaW60lnb+YRCVTucx2ahpKbcoqXZd7oQ45G3qBvXe21X7LnfgsTctAAAs+Pzv6FqlG6JbjeHWnGvGa/VbovkCxSrE7sflqx8AjG6hX2u+SI6Tvnbr1VJtXYu3LvdrmYWIzNiRHLeaIK2JewTofzLTsxo1NSeUO9L51pWWKG6Y8UFrpoSl2VDt9+HfAbfT+BO99osqEY8VBKNtSlrrXukCuueo+9X7rag5vzYEyeNVDg2HXjsVKmktDqNYMesobkMyMotpvr9Rwz+/dDkAYI8Xf4jOcZpR0UpXPVespuYh2RgzWhSt/TOal8ZD/6BT2WsYMImYXA1HsarVepIHIVbfWgOrvbj2Ysp8+lWLBrnxEfrfuxByuJ8eLbG5Qz31uJSyedpCALu/MENlErBMTzS0mrFqfq1BBwCp1BOTICDws2tocXLi+1KlN5Nl1+zEc4kFerfaONmqWb7vlp/QweHnZaoeAGMaAmviSA1Cwtp8pb+jXH8/nVp0BHqXLHSSt81t+Ng7+TRVi0O3L3TNBvpmjV8nG6oucnQzUFGbNN6EFFst7tOLZxQq+jVo8w29EUubD2aFfkfFKtAzPX8CWh1e0afkzy6gtN/wVXMuuYGQDdOOVJ+VhQr2es/Z7n38/MBaRGrRPf7mKrEhUaYfslQA5izLXDXA2vjSH/rlNvris815XgzZ7Xjuvup+tXhtjAG7Hu2moTdHjbTJ1tgY0GbMUADQuFYbcuql05cjpg/nqSNv/Bcdgfjen9LxHq4AIbVhTKZhCzBW0yT9vj89Qddu/RpdO+itljmaaf+ZzKxM44d8itIXu798nJp5UB82Y6sqy7zP30j/+x8EetRcwEuMYhkyYqEWL4IadC8AMcNtT1JKiFn70B9uOyObtFmLfk+28EQkTaok5KbH6VL3vIwV4/dfgFzzN/Us1UXM2k9VyWOCLGO96TXlEZAblZndHNo0yV99hq4deAoWfPaXdK1ngaqnZ060z1W6VbJmPJZPk2mNmLprxvqZdOTIZmNClJgvfCZVXrRPxdRFNJ7HD/2Wklh8JF3TpmLQi1FZHzNrAAf+vETXLMjFB45fDhuRLVBJpNtgc6d2t+3oDNW35T608HCM/YTMgSqJW6kNqvaiN6dN5g+eo3nzys8V24DVyoxv2WuaVimFwafpt2uOEeYkzcYAY+rF5YsK1jrACAqnz1GbWGXKx/OqT7AohUibisf15pvKYhX1QRJi+g3qno3YubZGO1dtAwICAgICAgICAgImFZOq4RBCQFb76I9WtypJcqHcxFksjYEffhYA0H2wUiP3LaHf2hBE1xyVmdpLsaQ6WY4meQkRQT5EKnbsc2qm8iTTZkglAXGdxvWN9FuIoCUA2ikxwoNbXYmJkTZZJgO282TyPgjo/aRwTVuEiDCylhxzO5ADFcskSzklsgoXY0qKW6zAlt4BUM6Pqj5lo8kq7XO4k7zcqhzvu2YZSbV+dx7NkS1FSarqa8PAwv3y1M58nGIb5KZHKR1lZmSkOgXHkdqUzdXIyGIFB5k8eGAAACAASURBVM8r6WcAGEflUpuR6U+YFk8AbV0TeMx+n+qbHHMu/bLZWFRKS5FkbO5nqXdcx9hffw0AqB79ETebzhnm/bBEPxJNpJmJd8ffoTEGbMlpFmdLgoeVuaJtcqmuaU2o+r6iUIGsqe8Tc/2K2ozHOIEqzUWpncyFACP999Utjk0/ZafRBCFBblgaS7HHK+nUXT+kU8tzSC9VGnosZs3pQW+lNP/6LYgDT6Nrus9lnDK0OUXBK1HPjEoPnL4PGBPRtl7IkS3qnNKYN+pGumtLWO8mAhBx1N7ONSGE1ogYrbOnX8nYKofHtEc562aGHuuLADv4/olMacTz3qmyMbJurVXyEGoIEUGuURoINv17EfXpwU8ehY7zb6Bz3E69EvdCc02DiICNj2eumgbnU2p3TW6Tt4HnybSG2zaVKrSrfjxbjets0iQbZuxS/RqVLkg2f7TTkmx6lzABKlaBnpwacTabiorpevG7rA+b8ZLHn0YdKKoxYVSNJ1KidMwZ6jDW53QZtXkotz3PGkbGllWFyp81D6IALH5+vvrBaBWFjE0VY1eLkioDXbTGKbMO6l6y1Dmn20VcN5pM+9vYawiAvnOJVy1Js3iJ8pQJaP0DJg3BaTwgICAgICAgICBgMhGcxncstBRFSanin5IkRrz8YuME55Ec2+g8/ZP+NMsdiB+5gc4tOkIl0YBXItUKex6d734ktBjSlWBKloQ0atoBUtN33nMtsNtRlMb0PXUa5/3s/5z0bWmGloAojQDap1rSBnXf8EbINcp2Wtn12pK79nN/kq+CgHbIR6UAtClNFWs6lL0uZJyWfMjYSPSmLDXnnvynW592chCWtSEj3ZNG02E0O9a7Tko/25STcbEMrH0wX/3Y/vf6S4DdVPuxqAwBABEgZUIrIWPHJpnPzXvvRXTIFJUs4SpWjaRNWLarOeibIWNg/xPy1Q/QdLjk6K7SnLZ7Ou1UQSxpk9JSiUIJN/yefHKO/VDi/mKbkSyOKxVX/YRfhfoOolCGfPKucZ51oe2MRzYZJ3imo1XtTcw9yNKEqjrVNgLrVHth4oFiFfIfP6DbDnsvnbOdzVmayjb1UdlIILl/AMCI6v+21B+AKFaMw7LV98dDpB3SrbT2PQUAmtJpJ2HakzC28Pqi0nQc/DbEd36fjpe9OvGcp0066Vt9dcFhLcsyTkFh5gDjHwUAaIwYemc1xsqxQYjyYrd8ogDZbhNdJMouEul77pNxDc38/oQQSqOcAyzhtfwPuI3Jp5RPx4zlkJp6WXWOsa2QrLHjth43gJlEFS9H1Rit3lHHhbdC/pDIEnCM8terDWstgU3JLm0qU8Dyg5TA/IPz1Q8w76s2ZFHzsk+Q+m6iACR9uKyxxtZOaJIMrf2u6fpoSwltOWDTftsU3NDHJn+QM/6mp/LVr2r6eMqHjce8Unu63ViaK7ucN3+UqGIP+4WyrNBJWs7jkfFOSPU/UYBU5DHC9x7+/C06WJqd4ED0sp+lsJzGjZYlCck039VeoORqfmVjDCNPkaasxOOk9jlJ10vabUPfXwcEO+t3OukjrkNMy+pDFfBMIPhwBAQEBAQEBAQEBATsMDzjJlXRiRRkRw6shVQ7f2Mjb+6zA+YNfpsCSHV95qR0ekpapyU3g+uNnwGnZQeTUtI8batYKAMbHs5dD1uK4vpRwNjOFsqGnaF9Gp1b8SZDkcf24e1TMXDZyQBMMC6TZmSOFX2oI0HhepfajGYjwf6DUhvkn75CV15zRfZKsiRXCFMn9pEpG2+QtLSnYLFrsHRH4NZrbgAAHPp2pR2IWOokUn4SzaSpKVtfW3K2aU32ugFGC/Oi9wMqWBNEwg/GZkFiiU99xLCuWXbjAz+4BADQdakKBuYEG3P9dmzmG4cZh6+zNNP2WfnHtXS854nZq6jfp9VmmCnO+oZpW2cJ8PexpFE9pSZ+CD5K66zqY2F9+9roODcnsuX3U0lTYYuFymcorkP+gegZxeFKc1GsGn8F7oflDgz9+tsAgM4XnpvOpzNh8x3XzLtkm/JSu5bOaYYkzfZWAe4hClQsf132Omq/hXQdP37zv6goP3wXxGu/RPd5bONNPWJdZlFOe3SJZWqMVW23/qUTUXzfL8yzgENLmkp/bND0iY789tU+Hyepy9kO+YfL6L4Vb6Tfao+/nVkMe6k8ktSbwqKKtZ9rpanLbRqh3kmh7FKDAhAzlZ/J1qcgpuzi3l+sukEAOe+H/0THPOZrf6wixCs+oY5Vv/rB2cDpV6pHVdsY22q+kxrLDBugBDY9RofzDslRRZVeud34oPA7ZK1/seSR1AtPe5L42+8ouOghZ89wr7X1Gdpy9tvqnJGeG5yiedrr+kez180prufbJwP12cdSGs0Df/tCGYtnJdjHfLA0/ulqSctfrmjOQfWjio+1axzYvpRcZrtdqLTZx1HwmiQqmvGfv3Whgr/dQ/Pqkdyv7D6VYOASUcGwmnFl23qh1zOabpcDBlfyU+E/0wgmVc8MROdMxD//MB2//N/p11bfW4Nsx+Lxqfn0s91zHLWxumi4mxn2/87t43jk44OXo2rDYZtcML3vZqKlFO1T8fnfkmryglSa9jux6HC5s/LGqdSuueF0/dlBrdQ2MYdjNnWx89V0xpazfJL6tzZsUQOrDUdUxG9X0/s5NFk3kY3S0xul2uLXl+UMA7gNptMUQseCSOcd68Fv5DNHAgCq7/+1+dbW+NF52sUmPcDU3YYnCj1/Q/nT8yBOJIIEHYvAJiU44j3Z6mVn5xvgkgtNYREO6IVRyTHHYvz6KeqXz0vmY/XX8WM4uP2E4x0IIfJ/Q6Q3VHrjX2DHyyLqd/weAFB60fvpnM0lr2IYCCHwxH1kZpE0eBJCpDZUUkSGApcnymLFwxtv9Z8n789ZP5gJFonNKaB59/Hqy3V/ZxMc34JZiAjxDWrR/pIL3DQhjdmO+p7RG7+gzXZ0HKSu2aZuyY1xqQ3yTmWW9rz35q+rB7Y5mJyvIrSrzV+ztrbxB18AAEx9QYpn3Jt+Kq5DVGzdjjfni+HgtPlkVGw2i5m6K4ZWkilT2/k3qXtNHCBbeCB7yJRO9zu7T3PkasZp39GmL/KXK+ncYWfoDbEx1aqrNItGoJIHei7y9OGiReCQNJmz5xJLgPP4EJ0/NLWRkEaQxfWuj0Jyv1OLW1koWeOtMuOMLfPR6UtyVA7WZiFNwmqbNOnxUgvbDC26XqgLga/+nd7XyhZZtqa7Fh66XKvNLty/xbPNoNpYoWQJu5jkwyZqSZhwNWpmvLUEyauG1bOJuVBKa6yxKLV1urFFLc9jLAuVrPhd0mcOHPCswbNmwxEQEBAQEBAQEBCwc2Dn8mp4dm04WOVrRRD3SkcOeXWuZJPOq3LjwyYIEkvtbemVL8+MeYwH+b130cFpZKohCpZK2XJIPf/LF+UrgNr1N75GzqOFd16lg+ro+tuBkPZ4ab70E0hSnpoLDSeAIQBS72tth5JOtE3FKU0UVa7ph3Eo9r5jDtanVfXGbCy3upKdixEZtbE2m1LB6IbWazrU6nuJilT+aiXECZfpsuu0klGCk0GMuGLqHEuvpFDf7dWfg/wnma9o0xYb9XzmRs2QMj2I68YEhTVTo1uNlNOS3i/tmiC1qwKZNzJFaIKqUhSA3nxUlX6a4TR95Ke+RzSiH3/3oMmT+we30WIVu19yZeuyJ9LV7YTNlGrDlvQ/YZJQrEAclyPwJqPdRF9POr/q85ZmRXLwrra+FAUpAGz8GY1FU5WGw9YgmHxUHfqWWCQdiuRjzR3GDCjxToSIIEcH89YwMy567ZsAAB//2yqdv68NTLnktolloM2dCmmHYwtil5fkStbV0rrmPfY7rL7+o3RuC2lQRM8Cr7YTI1sT6afN3ayr2myKo94PrDwMnRfeTJcLLp23lBJYdx+dy1FP39ydMrlDBPnXb9I1NmmEvx/v05tsWx5nf5u6nU152GS50m3M1nissRyz5YwcEbiBtGmbBWes57HU0Wpx4D9j2n3EzG2TztvSfW9wztUT0KY63yFBc15jOm1Lw8Nj6PBGM05Zabzmpe47tudLOaa+F7ebsSGjsbK0w7ovsLmhrWFauHfmqgVMPnau7VVAQEBAQEBAQEBAwKTiWaPhkFJCHHsh/WGp9erbIeYnrcMBDG/Jly7bHLJEomMG5M1fpnOHn51+KCm130Y40o5DKTCXpsXl4GEAIK0AWffeSL+HvCNb+iqP6DWfopOD6yG757o3aqcsAKuIMhdL81AAe5zwUlImaUk0jFOfluqyQy4k7u+ndJZmyLMpkr44LH2R0lAwZob1nbSzGlOqKulf2xSAaUm7lN30scbTxtBoxoh/9Z9UxF2PddOX0vgTRbZTdtFJA8UqsMcr/CWVMbDmTvqzIH9AJzetpATUIgXgb1nyOxw+mUN47XNaJkrRpNxj27Qm6bzYQd/4Fl1w7dV0zLbktSHdh+SQCgrYPg1D3/wAAKDzky/R6WpomsjI/FfSOdYuAjD0ueyMy9SRfYuA33+VruVw/E86V44Lppt+8lYdGM72k+lb+atxkzDSU4/WY+Y+ePSNNI4t+s6jdC6yKFt3yyf9zwopJS64WAVLe5ycpsWM5S4lMd+7TpFvLHxBprSNVlz1z61rgC7WtnnoQNV4LvoW50vf0yeMhq0CsdtxdMxz4hN/BhT5gaMFGCJn3HTAVAAjigCD2+TYoJE+Ky1sx7u/Bnn9Z+jcMaRV0T5tUQGyK+EHsp0ghABWvJnKznPygael/coA7Pa5H42bntM/tZ8InVv9zt0w+8tKUxOlxzMxY68cJQdazU3m+8Lj42FTMpvvtGYkZ/apPNPaLKdtjQ7lT5TnP2mFKOA+wXNCVLTmS9X+Omek1lFSSjx4C2lZlvkCJWptrXon5XbouWBUafAqXRBTE5oobZkggNU5qfCfaQSn8cmF49DNJgfsBDRtd8SbiLdZ9BC/shAR0D6+A5tJd9QMtIxKF7DLkQCA+LE/Urpq0SZE5Hfu3QY4C5+90nz2PEn0v58YRqb+54PAftlNnoQQADPWzNzbPW+lH3+NmK8K77oGmJszCjdgLbCEURNzbA7mxB4b1Gw0Ov9GXbOmyP6H6Fq1Fy/7wFl0zjblAYgTXkdlZUf/gt+UIBm1maNFF0rAspzxVOyNkla5qzZ5z2/pd+YuwOwDkg8aPn2L9zw65FWp+yibCHKsOWuOv0wJyBjoW+i/1gJ+NqEED35c133GWXgplhWOK4K+JXjZXL8ZgDdacJO4OsZsTt2nNqWybQrQ1pO5bjYcx1+uH7fZ2hBSzqP9D0L00mKxcSU5FkdnX4eOD/wsla4uK7cRHUlbQiuN62w6aPHw8333UtRrHPYeiJMuyV+5nI6RvDAV8w9F/Bg5H0fM2AVAdIy/mPSbqqlrUQGLrqRx+iuH0zh95h+f4gdNFPDtBL2ohgQOpsWqUJt/28zFMVv6pSJf+Lf0hsPHYqT7B1/rnOEwCqXw6J/pN+Pm33aMTZm5WfOVKCgzSxZGLDhMb54wnRbIIioAU5SJcILxCqLgOGjzb+p7zj0QULE57jiJ0tr3R4/QtXKHY8aXFeMxDOp7OObGoRRhXW55UvcZxyzrTjIvhYqx5ftudt/QTIcqjdn/dS/kd95GF1/3efrtmGYeXa/i9CQXtOOgKfsbQItuPrajhPM4yAQMlU4ctXt6k2XnQQdsbiqQHkuldb86YMf9YiX3uAHAmO2KQrr9C8sE1mpbXAD+dmxeKvsfxG4v3M8toJ1m0gRPROY+ew3HbYO/LzOUDa0D5mWPZxQw+XjGNxwBAQEBAQEBAQEBOxWChmNyYVOGJilO0TYFQjlSx6vI6U/MPQh4QJm37HdqhnSrnmuRjrTKiG8iiYc4/OwJOeOmyj5euexzaqfed4GRpl50BplyrLz9TbnK0Sr96NQvmZP/+DH95jGp4ngCUdFEPFVqVccxPRHjBFHBaEC0w3EV936R3vnyoz9C51j1bKugre/nVxcbrQHdo75DsQL58K1044ozstXP5gbXHqJKSvdyZaomG2mtQ6UTcsMDdB+TEURFDP6UTKo6D1ZSNZsjnR3KPZFWnfqqvOJ+ig0j2EwuKgL/up6Od31ZtvqhSftIRkn3SXBFZEwsLArShwYovaS+zC/VbDK4aqdxJc21tCvy0dubVcWflFK9Ez1sQgLKtL6D64zpC3/LSrc245ANZb4ipYkSr0wf3TbIUnBLmsnRdbkPyFibUum81j1s0h9YS5dm5nB2TMSoAeDRJqW/gZQSQsVR0GZd7VOx5aNkZtVz2T+dNBwtbIt0qWr07c68SWm/2GS0aw4k0+6++kup5yYCm3KZTVJ9DspyPZlviOl7oraR6utz821J+GHRlqbSt94hS6ozI9Hm3Sz5+8ZO/Ck6KAIzltP16z9Np476MHAbmQhyxHknvcS7kZ4xVURFrRXZ92qKXzR0AWncOz5xO/AkxcDIE6U6pTm1r/jaLpv59SzQZmU2Ne+1X/gfAMAJr7zEec5JjzXilpWCHt9K7RCKrAWjyixbj79l4EFlxry734w1BaXxFd6o5kzTDqCu5jytYJDa5NJEBo/Qv47akI+QPRUZ3SLb0IvWuK7nLaadlfa8MiV/FG45QG0hmra7FdvD1AMAmS4niTHiutHyMDHPjGX4141kyr38XKYotzWSiXSlNH3BNvFl2nhFrqLNqrtmQ/7rltx1DJg8BKfxgICAgICAgICAgIAdhknRcDgBhJpFpPXZlVqIlLPj8MXPR/V9P1LPuDba3qi0Pqc8X/rKeTy+4VJgv9ePe38qnzV30MHMfVwJnF1O2TD2qrZ0Rtkgsu1ofNt/4/2vIGlEUsLl2MZbDoJeCZztxwLoiOtybLC5b0ArsC1moWzRpipJirZnb2iJi7FjrZtoxrP2oWsbHsDSA3dxy8mRWKOSVUf2Gym0tgXm8rCDbG0Y6M4ZwFFLqqTWzpgATvy+0poIQEBM2z119rLfEL3zyk+7kdRtzUwrfyFHgzMlHZRq9JZfAgCqx1/aNI0k5ICKxNox3bxjpqFuMzSG6ei/VptmTUFcx4rZ/iFEjmyybPeb90/KLpEGS/9kDOQM/Cc0LaZlr5wkLFh7F8CO/NwfO2ZAbiTNQ+mc63S5P3fSCQCA/3erJ2o9azG4H0ZFI+1ss6Se3DbZnvqEz5rn71LR4vNoGoc3qjKn23erPiKEMG1c9ZP4L19B98V/GTeNrLTfeuxTmri7T5iBZf87ATpOBTk2aPzEkrTMca2lf0g0ney5ZW0I5Wnkp6LHzJpFwan9Ncx4laQU90HwWCNjYFGOCNxWHeTgOu3kLux2r8rj1YAwCcpRFCg3fuJm1LdQmyg++ge6NouCvIlqd2rOHS8oJ/fTjk+QdjG+75r8GhwLTmA4fZKl85E7PwJAbQRJ7aSUEkcfMjt1LlUHz3dz1glKK6U17nreqgGr/pWnWiagYJK4xM5TSotm2PhyJf0WpYwxa1GfPlYPpOti+y95AgujoXwlZclNXxQm5MNh+7Ok12zqwLEiUSejgh4XhaVluepxet/LW+fqpmWlIWOLdt+jYaoPkoa7OWHxsw07l8x/UjYc9qCZjpLKnSoyE3OSrQemo7Wd/2d84Xlk0vG+m3kRMD5bBOXpY1pxF8fRke9HvPofWatm0uGOaTuratOigvlVi39HPckdRi2IxIGn4/cfocnk+KS5i4zNIsnahKScymwHXU5fT7LtwNOP566jVo0WykZVqwdAi2GCWXk0D3fB8KM/RM7XYslRGO2nwb9NOyZ7IraymY2UzkSs07UioQIwMQfap+ZnG2M2l1K7y3YFuCZPPElpfnVhBkEuW20EF5xPG1ftUK4H/6gJh3uiHcc1cy75vuujKB/z1hyVU6iyE7Y0E0Yl6Zgtrb7I4erNZpkJABDXMGV2tz8fK03bZMtn5pPakPDGs1EH+ldnqxenx21cCMP2w2xlvMjoWQD5m0/QbS9QcXEaY/+fve+Ot6sq037WPvW2JDe9J6QAIdSAgIQqRbCiKCJFijRRQUfmU1EhdodRLKgMOooKIjgj9nFEdFBQFKSpINISSkgl9d7cds5e3x/velfZe51z9j5JLvhjPX/knuzyrrX2Xm2/5Xl14LF8hFwbxeyl+JebbrTqCfcjg9+JldE4taEtlk2f4HHIm7dSB8TBGd39bKgMu/azc8cEzHG6UJ2zPnTZpeeg87DynLkAgBnffMq9V9qbGo8rWZKwAUhdv/iWVZC3Uk6hdlyqRLlLB/2afmR9zHGOBYtFLbXGlDrxyP+Su8yi89WcqV1Z0oxB0trIuS50/g9nISLIDctztUuzWnVNMnMNfxAXjVueHFFjLbLeK9eJ3Qen7I3iKcq9ihUfPNcMbDRKIX5v9RFrPrPeoVZgcUCzkj/3MMhOkyMqM/SHuJmnkWTRimvmHPfdYtnMp3oeAgY3U326Eq61MrbcXCXP2yJ1DBBm/rNcf/S5kseVtBmYOalYSTPW6aDwzfQOAP1xLIe2EkMdYOaocjfGnKveYWLNd+ZN/XEM56PNbaclwwqwlnu8Ol/7AMjn1UdY9zRTthX8DoD2NPxePcHdsl8puQplXPFb+ojV49ZmpkqSa8QxpEys4bayU7mvsmsjhEDpvG/lbmPA6OEFj+EICAgICAgICAgIeEkhBI3vPMjNz2oaOjYfx4/fSv+feWBK0yprQ4Yi19Lmv+sGyv6rv5LZjBoV0hqv+pBlHm+kubK1CjGwRtEO5nFzsMrQspMaWxmnA3JlbMy8rH2pD+GYo3dPFGBMsPzVLzeRRlJ0T0tn5JQxMEwaGNb664DmuAbsbmgxM8MOsEtycttapE6mGrQ0q6xtnK4oZWuD6JxJtK4pWtyoYDTFtnZWW4osLarDEw6j9dmyElif04qjNMc+1z+dOdoOwLSsL6mgXTGEJ773PQDAwjd+QVXNsnRZvOJ2OQ4KpXQ9uO3DfajfeRNV47Ac2aq1ZlNYwZqesZDsp8I8E00UEBWw+knquz1s9WDN5OAmSPU8HarOhFybvlZywKemQqwACw/M3jaY9yPjGlDt1WUAQPwDomEWx38YYKICnle2rAQe+xWd3+P1SlYn5ONsYXG1sKJYRKxzP6iA73jEWDu4nwih37Vc81c6NOMAJaqEeHAT/c7RRrmOsqTXb/4giherTPQp1xvh/allsCl/YANm/AfJk3d9lS5/+UVKpq3Jb0IC4JPPXao2AHGEJ9dRRvjcceSfKHeJOOg8CA70b0ZtC2D379NcoF00LcvlwCeXAgCq/09RXw/3pam9nSDdtBuEWPS6XO1iEgIMbDDzaje5fW27gtyHOy76JiJFbqLbN9xnMi1PVGuErEOMm+3WjftfVNLuvmK8ynhULLt5YhRMGWoNZaphSFNWHuhcQx1m7VWuqdrN2kOW4TjEqPEUb3gCTz1N42hiM3dP4dnS2MeSlmV2Dx3aSpaVPOC9jBVwz9DW+GovIrUe6rncohgWyp1ODvcDD/2afi9iOnXb7VpZydm9UMZmnbU9N9R8okcDu+n1rQHWKddGX26zBhATFhrZ7JqNhLWlLoy7o/ZCsFyfug3t9sq3k2vwzOvUeLSzsGsLirpvZABC7Qkd9/AosQ/gfVNcA26mOR4Xtc4tFDD6eGk5kAUEBAQEBAQEBAQEjCpG16WqZxplSgWAWSpB0p9/oP+f0hIXyinNrxACkikt2YoxpBKFVcYan0g7C6YHjYIBhYiAXY7I0SgX0o4bYK0va3GiUsKvFfSVznXlL/aoiC0rngEATEpkvMbwVu2LLnqmK/llJGNRABiNvU1Py2C60RxN5eBiUe5KB667V7p1FpHxrd1Ivs5i6j4ozktQgeo4iDpSWhQhjFw7GWAygRvTmE7aHZieUyuntEhy+Hmt4ZSx8nX30G4yvH0pKqJ3Cr0nrQmfvr+6QaYsG43IDdL9X2kIK2NQmJkvQRUA3SeF5fMuN66gKk9sIY/fJ8chiQ4sePv5dKzgBr/Le74Ncfh76T9xIpYpJTdhERxWvuyVMcD4Wc3rlBJl9b2k5WQhafbsJH9aE9y/zlAV833D/RA8FyQIKmQcA2OYZjI217Omj7WBIoLuy6zp43dYGzQ0x3mgnnXx4p9D/uoTdOy4j6iqsOY4o7a2Og7yH5SIMDrknSSCKZ4nLMxM9924rmUTt6W1pTlQH9FzgGTr0iLli96/HpITnnL9RCHVz2RcM/PCOLKq6iSWUQHVd92kylL9etV9EAso8aqTwK3BmiGl1IH8pk+0AM/rHb3YdgVZ8TrOIrrXjncR/avonJS+XkSa0tnMHUL7sUcTlTad4+iiMoQKnjfJAFvY0xLxLVJKk7B1Uo7EamqOkd84BTibvBI05TwnNLTGR9MA/XFzsd9rj6R7BonSVqh378Rw2PfkcVcpdwG75bT6ZxkTnuzfdmyUbTWW65Xlyom5gvu76bi2aLY7xutjAGg929BG3KZF225IDdy5EKVOM9/xOKiOs8Ykx5EU0dGjLEo8r1gWrtRzsjwNdFnD/ekEgTrOsQLZvylvC19YtDuv/pPipdXagICAgICAgICAgIBRxejQ4loaTDGb/GXlJvW1fdz7VU0qHk1AnNJWAQB++W8k7q3ky8v+g3K4X2v1Ndr4gpQP/ZhunZZMZ9YaNuWgtGJL9LEUM50wX+wWffD//p0sNW9j9iV+NuUek9TH67fMmhPLSlBW2myLpm7bX8jS1JWnbSrhm5QxxeMAEIpqV2uuipW0RjuumbqwZl1KYF/X71k+/ku6bf6xFkVu0VyfogE0lhN9nbIcYGgrMPfgHK2Dq5FRmiIoTZG0aDR9tMeGbU1pBQtljF1IFgNmMNNawvELtC+u1NYti0EoC6ICMGlO9usZnvGk36vFUpPUaDuWMx1fE2PoXvI7rp7wcVfmklNNZ6+pZ9dIO6c1eRZjCUDWPGbpyQ0BjChfPw+N7QAAIABJREFUdGba6lXPq1DRzCli3Fw6Vuo0WmRmC7J8rXUdR0xbBLNeMYVyuRNIsfFY/Zbbr8dK1WLJyYH1lKAPkxZBHP0BAID8g4q/OOBMt6xWkLH2RWewr3+86WmIsX567gyCzd9uXyqzjCiULVkKyu8dIjLviud9ETssU3SsANymkg+eTM/JXGPFnLEmdqbl426zk3E1PFZzqd5J5gSOVlxT52VkAfr9m8giu/T7Koaw3GXmeG0Rr6a14yICVII2MEsVM5mtegBi2n5O+xqhsUVAmPkwD3huPuFSw1zIdd+s4go7J+jnr1m07LneiuvDPsfRb17/eIzVh83z4Xr2TIMvniNpsXOYKycsyNc+ZgJz2BUT458KU3+t+Ep9uWG9Ewe8zj2mLfnCjdPQxxIMkXEt/Y41a18EzNo/a8vS9ZNFSM5cmGL7lCYZn7KICCAR80kYu8Clj0+za8I8LxkDVuwqAJo7ec3UfcM8fzFheva2vSgQgsZ3ONyJTG3MFC3c5w+mzc57fv+MJ/C2gdvTG6/yHy93mY3wWDZt53+hYv8z89/jyf6rJ1d7sDB4Qx3XzOTEbjwyxuk3/MiVxwMyKmh58rl7qZhxu0Dy5ki5zMj+dcCGx+kYm+DVXxnX0PG+H+Zuo5ngyhA9U902WZv1ZACrlBYN8fgFuh1DNxD1b/UyoqwUHMxWG7QWHOOCpjfBtmk36XLUY004v/4y/c2aHZcXsrV/g5i0hyo34Y6W1XQvBAqHn0rybriQjr35Sk+hPKHGVq4P62yDvDWQMTAnX0A1AP2x6hyrNqC2ta8RkQkut4LGb7yVXOTO+XCCMrRnmlnMU7S7LcAb+UIZ+JvKpn7IJZluNXNI3ZTL9dhKmzIx7xXp9xgVzMaYN+BRCfKBm+mew1T5tivaZlKaiPHKVcjeZHBQbn0Ysk8Fnif6ryiUEKtAzjxBuWLRiaatTFW99F0AgE/sT+Pyw/d68obYMrSLRglQOY7MObXxHjcb8d/+m34vPilz/WwZKJSB6W1sdDQMvbchOeBCIqCkVCZWHiBvTqIZNJ4Ntbbqr1EEyR+m7A43tBnQrlpKVm3IymLtBlwLIZKfRK3BMgY26DKW/mgFVe0WRQJx9Hv1nK3Rt8Z8XLFyBRIYUMHCmkZWuZ9NX2KyvnP7okKqDU5QPNP/2vN4KR1k3hKs3OgYbzadnIuGXX6KFd1+/RGA2O2fCiN3kOtb6YCzlXxrI68DlKem625XyUNpzJCP304/MpLF+N0WVR+N3D7rwJnnC/rPPR8j5cHLfnJ2Sn6KEj6uO5TxAIBiB7kSA9Ym33pv9yjXwd1e07RdTmvYLU5Ks/4n585Ymk0/53SKisaF1Krn0Drqi116b5TOJWY+soumT7D8qGDc5jl/jf3xmDf3VsCoIrhUBQQEBAQEBAQEBATsNLzgeTguuYWyJctH/wfYnV1sVGDmqgcAZQ52NAWskeIgTxt2pmkAcvWDiGa5rjVSSh1o7s1Uy/LbgDfDqa3tY+uEDnwbB0hlJRiyjim3BqPVs8yxrBGatFgdszSrrIX703XAXkqTweZs1ipVxzlBw5lhZzllqw0HiXHWYxlr07jzzji51b0qIHLJGahceB2d4+fDdR8ZSLl52LC1/kkLkGSa5XlHAcf/v5wNVPWeuq9+F9qasvov6tzeKVO9jGt+jdoYopUU5xI9rlxL9KOICpDfJY20OO1aU38fRaTWqiYtE8J/fQv4goCZclFYrgppt7HYaNeY5rU6FmefdZQjT7+PvtWmTzAFsLDoE23ULWsfALlpBV3eNRk4+C252wiANG42RSOg6Wjl4GZAZRXXlqwVdwET5tL5p8ndMNrlSODgC1Tdk65E0iT71DSOMBY/PVZioyXkY+z2MXamcXfJA61pNH2O3+upu6r5YrhfU8Y2FSWlFchbTp0Tu72Kft/5RTq49F0NtceN5Q9nvj4Jh2acYVE7a22nms9RKHuCSgHsdpyWB8Clgk7Ohbb2n/vkqvsNCYcvwzv3o8ywCDC4b1XUcz1c9bma5U7I69m2dXo+F1NMvmYxVwU8aw04a4lLhjKd3c8gdSZ4jXgEcstzJKtDjVFrrHLm8nxQ83TP9PRc1aeSwZW703OHjPXbEZaXwKoH/g4AmJ3UsBfKZkwwgUzHeC/RgXb5SdRH2p4GWVtn9cuUJYP7TX04PXdLaWXLNq51B1z2Ua6MEmp5TSTcPeXgJkNQwG2pDUHqeU0F9/PcWqwC+52Yq30ADPlAtRfGdStBfFMoGbpedk8sd5l1na0elTHoOPqtpk12W61EtNpyIyLjymyTNySeJydWRKWnPUvcC4mXWB6OYOEICAgICAgICAgICNhpeMEtHKylkfGI0VaxX/Kk3VP0qyIqQP58Gf1+yzVpeZz4i7XgnOwO9ld1PaXVsjW/cuU99Hvy4sztaEYfafwQAa310clwpD6m4zoAyBuJolKc930lgy0dVnCx1tIUjQylHRAvP1fTZ8ptFKDH2g4hBOJHyLKUi+bQDrRkLaCm+jSBv2m/VWGsMkvO0LL6rnwjAKDnM2Q90D7JIjKaGSt2JZ1cSdr/oVs5XkMI4DdKK3vW0mzts2l8OY6A39fUvfW5lMa1QTBm7YefBgCU3ns4XcbPKq5DvJGID3SSLcDQSNqBxFpDlSQKkMBffko/d31167aZyqYPJZPxeQODhdE8qURLQkS48fr/AwCc9nZ1FffTzomW5la9Vw/JAQCLrlPFDtjJph6kxHbY+9TGTfLBfifcH5WFTwgBuXWle27e4eb9aE0wIO/5Jl229N3JAvxl8vtkitWBjbpf6XYNGupG+STFqIglZ2dtmekT1njgOWaX6ymmRj7yM2DxG51zjSDX/o2uSyQEE0KYeVK1v3b1a1C6hJJqNYwvcuo6DKliyThuLy9S8pU/txACcb+iAR2rCAEaaThXUrwbFPWzI5Mpna1EYrzeSBX3AyFM/JkHkuuBjJYOLqt7qukzPHer5zR01WtR4WSEHLDbMcFJHEdVixCrOI1IrVlmPRKQHOfC68FTd6TpmG1q2XLac0BTomdrnbqJ51Nh5gI+xmXUh027HWt+oiQRYfoBe7uHbG8CpTuVt1OSVfHKy/3ECc3GQteExuc8aErjy2QvUdETm2oliuVYGyHQ/1Paz3S//B2pcnQMA1vZSp1W0l0T86rfayqxsYB8mBKbYvGbsjeSx4asp+dvO2hb7bsE17NYsfoTW1olNv2QiH56D1PJQG1rTmJeE1EBEuod8t6oo9d6rwkiGgjguceyt+3FgH8CWtxPfepTePDBByGEwGWXXYa99zbjcGhoCJdffjkee+wx3HLLLS1lvfAfHJxJfOZBiG+mgRbxh4S9wbQH9zw3yNErl68fM11nH9cZt8fNTrsF6EGLtjqB/0MjnUNBc8kX7Gyqym2HXU8AfPwaCqS+4rwm5VimRROYrszwY2aYa9kFYOtz5t71K5o1xwvXRYM/elo/KyEE2P1Dm7SLFVx1G5lrl/HEaW90m5LhmABlPQfq9zmk6yqHBlrWzamnXX4GNh5jyh2TXh9FhOI4lYmWJ0iduRdADzN+2e5hyi1sDW0A8dNPQJz+H+qGbn0d/ShCLjw0Q6uS9Wqd6yNZr9Q11ljpG0neqU7ZLguO64/10U8Xmo9RtVjxeBXlLsi+9njVXddGaHlch9XLTgEATPua2gxXx3nfudjn5JQ8jYTLkixM1Dkt7r6UvsAO+u/HzTjlZ2KNYdFOQHWT+UlnkN7jDaZeTZUhAhuuJCXAhKsfb1IkPZvixT9HvOkpOqZcOiTSrGamPhXI3zB5Q7Zg3FZw3sVycn8TB7dgh9q8quGp1Ad2oWwyO69XQf27HJV2w7HHiMqDkRX6eQmYAHWGcmWqXnYHvnskzROn3qae+dgx0JON7W6ZcAey34eut/orFhwHqZR78ial2Dr1Ws04aIKRLSVcG+6bWoFgu6jxh0cToopGSrtPfvsBAMCyS3xzmLrn+I+qoi1mSOWWI0odDceOiIqQzz3csE7twu6r5rd9zNTnW7eTm9m7msnRiqey+0HHf5mQQis5rb491mT8zgw9eRYgosSHE+85hrZCblQMjJOVAnn1g0ZJZ73/cYe/SolL9yd2xdaupyPbIJSySuqs5iOQa4nFzQTIWy6WM3Pm3gpoirvvvhtPPfUUbr75ZjzxxBO47LLLcPPNN+vzV155JRYtWoTHHsv2offi/7wKCAgICAgICAgICBg13HXXXTjmGFIUzZ8/H5s3b0ZfX58+/973vlefz4IX3MLhYEtjLZGj9Rg/u+F1XrCG+R+U5wEHnZe6xNFy5aCozI3YpS20wdliRaUHJ81OJexoDg4q5Iy6nSbomrUj8WbKXi7GzAAmtJHDISO0tcXOw8EWCHZXKXXgnD0zWBFsMy4/O2UlkcWqCVZTJnp2lRGV3YCJGbP+5oSuE1uTfIHQAPCK81vKci0HLi3ljTfeidPOapLhfMreDc9tD2yKTE1WAGmNQfNODprYuJ+mssQCJhjf5lJnjRZbDGxO9rGNyQOaldnUzUfGmLhY8cFbdKqpOpa7IFcoyua9Ts5QeKw1kAccp6ywhZLRqnIGc50DZA7kOtKqam1gFmTJsSGE0YAnXNaS+MvjZFk9KlPhApHK1i25/0ellIXPwcR82eJzYddslNf1e2nujw7/l2xymcBj6j76UFOr36ZnssltIKsRHtuizmsLrnFZdbTXOQL5ARANNwCUlfY5Knkt8vp3KU/GJgXba6DhJdK4UrNbjk1oYdGm7jmu9Zpor+Px+kfpmHJRs/MlmfIt61/vzsnh4OQwotJSblZCCLx+YY53KOvpuXRws7EmKLdXxyV3XBvtY8INXz/lPlco6lxO5lzBXf8Bmie2bWlYVCq1gDNfqfJlDKxReWrY5Z3zlxXKkIUX15a2FVqN/xca69evx+LFJrRg/PjxWLduHbq7ab/V3d2NTZuyeyEEC0dAQEBAQEBAQEBAQEP4kk3nwYvrc7CigmaZNrI61vX/ZFhBnVmg/QU5C2/fGkgV1+CleGQquJ0Auep++jGTqHodH0/Lx3+Pz3wjn2Cl7bjhlOMBAGf8+qkUfZxNwSkWHp1PPvxauXQHlEhRiMo6dFBGh7EG/GUVaV+S9irXqsHWkiitqfUEI4qxljZ1y/pWTfKW622fnXiJ68GapS3PAmNdLa6UEti2sUk5CT/YuK4JD6D8YE/90d1Y+276PeXa5ak6SqZ2ztXKjEglGpSmIEvzdOc6+r2vVwRr8VhG7Ab4AfT+kr7B9pjvzJc0sFmMit2viguX0A+2AsR1QKjxUjH+5fKO79IPj4Uj1fdlrK000atUosChLaa/KH9k0WEFp/7910r+KU3blRdCRCY2TFGQyq6JXv/4GZ3ZFxFfsLXc/AygEq16YznG7zwLx9oPvAIAMPVrK5peVzjj6nyCmcDhefJNJvp0Tdaaulzsenw++ZasdPZrM+dcfiERXsj7rqdr9n4zJI8he5598g76O7l10LqUUvdTcRIFWcunfgdMJ9poH52y/Mf/0rkcljgf+YR3jmXSAhXPJC0LgC2jO2cYiU50y3E4kxcj+e6c/jqrHerfTDVRZaXnIRu/e4rWgtMyiSwoqlpA9yWmvbdh729WP5pFsgurL6Ti73julNJQUXPAd7cVr8hxq0Jg/R/JYjzpDLcYKWOHIljdYKxDlqVG7HI43+XIBwCsfzpX8wKaY/LkyVi/3uyj1q5di0mT2k+u+KL54JBSQpzyFfoPD8x7roNIMDYAMMxIWeXy5MULcFSE/PO36NiB56ZvymuezlOXMbT4SuXehLGzzORrl9vRwE3HKzfWQYCnn3sCHfvJ+4ETP0sX8ICMis49wI7YrPIkZGdYT0ywhbI5r4IxZaGMCRXlYuJzvdHijTlaBxVLa+HniY6v42DIQgXo6EEu2G5ECfcmbb4uVMxzZKaVYtX74SV/SzlHpGILgs0+1a8GcVG5x9SGgJJiGOFJu2caJl91N8lIyJdxDXhMZeGec1iuZrYCvbeE64FtDLVcJfobBI37YWWktY8lPQ5sF4TtyOGgxSWz9MoY2E0tWuyeN7ARgjONs0Kj3IXhjfSeko5tTmZmHr8yNlnHlWlfDm4xGzh+buxCUukG+tsIimfXwiauVY77iMppI2/9OHCEYoexPqoWXHN7yyJ9OQe0/LGzIK9T26Qzv0V/9XuWwPwjW8pvF5M/RXl3OK8RSlV/BujH1IedYqlqCXbv4U3rXdeYtSiVEweQW56l6zOz/okGv2F9BNcg3qYUT+qdy//9KMSrP0F32e2cc1DDklJBvnHNBBdzW6buh6fOoWcz53rVFvvjcge9w7RCQFLeIwCPv4Xmv/nX/gHonZe6/rhv/ypfYTyPjptLJT37J5PXy6PIFFNaEA9sJ1pph6dU82iPpTUPC3OsGeHNtnbmGpVPJJWHCGb+kbFxL9ZZwMdZrqSK8a0+gpFBJS+55jtuZxwgXzVtZFKGjnFAT8I1jOXHNaBewz8XXtxORkuXLsXVV1+NU045BQ899BAmT56s3anawYvmgyMgICAgICAgICAg4IXHkiVLsHjxYpxyyikQQuCKK67ALbfcgp6eHhx77LG4+OKLsXr1aixfvhxnnHEGTj75ZLz2ta9tKO8F/+AwJrNh88XMdHqLX4f4px+g36/5FP0VkUsv2rKAOpKZodE5AWIv0jrHTxDPuZh3tJIvnIDrHQkhhHE9sKuovuJHrqJsuOVLf22y/2aSG2mOf3k6u2JZgWlKi7v8dHJemve9Va5FIitY2xEVjBaONQqsqbddg3z3Mp97oYSDPv8ddU5pbLcpV7ZCyVgP2LIQFSDZhGxrcez+A0BuUPR8E3cD9n9zvvZZmlvW6OuAe0VZLMQ24xak6Qgj8zy4D8c1iGMuYMHu36gIuUVZuFjW1lWaQ12/NxEZyj8dWKmur48Au2TML2KhmduYbfJPB5BaWd1VHgnZMR7vOMJviaNxrWTw+x0ZgFDBjKbMCBKqDykCALmSrDqYeTAwtw3aWKcOMH1dZ+vdAgwpsgF22Vj/d6Cs3Gh+QBnqxdk3oHrpLxsXYFk2tHweB0q+6Jygc3Joa8pdxEUvjv0wxIn/nr9hHEjbhHPBpeNU4+W4yzHwUdKEd1zxJ3O+EelBA3nec2eT65m8/mw6dgblL4GUEJ68DjsKolO5kZR5TFoBx1a/ju+lfC6Rx2Lu1bayHNV3xYHnujS0sKzEIgJWPkjn5mcLYuey5OBmoKyoP1nbac9vbInisl55OR58I60h+/wXuVmKcpcmrkgSdtB7S7RPROn3WenBnG9Rlmr50/fTMV5zo6J/Tm/VQofyu8F6E9f0nLngJjUn9q3VGbO1q2pUBH6nKMJP/mrjMq0gcz2fsfvYzIMh71L3LlEWubLJ/SSfu49+TshoBcsAx33W509gWSuXHjq/iRxrjuG/dl4sAHauDFlUx5R7uOycAFTbGIfsyiQKVl0TFr64bo7pfYE0HiUjyqvh+ccw9XXsmipd+YUS0q7YVp/h/lcfMftDll/j/cPzwG75qeJfULzIg8YB4NJLL3X+v/vuhlTpS1/6Ui5ZL257TkBAQEBAQEBAQEDAPzVecAuHDpQrVIwWh7WOY2ZAvPYzAID4we/RuX3eCtyhNB2vb60dFE7cgtGq6qzDKhlV/KjKvL3rq0yQ2TRfKKwfWWkOvXVU9xTPMpqbr7/tdADABb9fnVdauh7qGcz9zPfNsXtvpL8zDsghmrX+JiA12YNkJD2JxoTR/Pep9vTOw7Mfp/iZ2d8hn2GpYnMc7Xthoj6WCuAWkdZQi4JK6qYsAKI6DvK+H9B1C0/I1j5twUlHJ+qM9TJOJ3AqRJCcPToy8TLrrqLszJP/41VpeTMUbSpreCYt8mcwVxpO+RQFhYqZyle7UAbuVQl4ciRUy5r4Lwnb6iHLJgnhbx6gwPgTrevonJWR3Yp5ScWiSMsXmeUyFWmpA3jkt/T7gDSVtbeeypfYTTbGliVOLNgHMV+RJrBWrWM8UFRaNBU3IqXUmlYOlnXrb5JR0cnYyNCXREBB1YUpG5c/YOT3UUK6XFm41VjyxVW0moc6PvJ7+qGIOUTPVKz5V7KUMTGBD63k6uNs2WDijY4JkH+g+VqcmE8blglKs+8l/1DZtzFmOobXU+C8b8FLjWfbSqKz03t0c4qCHGNnmWzhWcGa6srY9DPlLNVWIK0mPql0Y58fUgLXLx1Clo5L/rgaeJRiWcRBaSpuXXdPvzHXCBOb8tp/o+ueUUkVZy+FXPNX+t1mnEOajtbMDYbmV7VxzAwItugqi4UolHHtFymT8YVNLBz6eVlxNs67e/lF6jprzIL2HHLFPXQsI4EDW5O8fc+pk7G6qDuN14W11mxcTv3JZ0syifw4eWjBWE65/Ho9tYeSHb36ftnRONliQ6h4NjG2BzLVTrZSDJn1vWzoeLVVhpM7T90bT3z3lQCABbx388VDOXGbqh0cBycKOl5E96GC6Tf4k8p2fcgleVoZMEp4wT84AgICAgICAgICAl5S+CdwqdqRGJUPDp0gqlABfxWnKQDrTTUF0T5vBQDUrzkR4pgLs5ctY11WM01utCtpoeONKzLLdjBALAqyY0LDcmR9xGiqEnUEoOMwbjt+Cs77wqfdc9bzMtpMw7rDz85LvcfJe2YdQv/d8hyw8HDkBjNAlTpc30vA8nmu699Sa0CGgRolJhScrKdvNWYefbSqHvv4K/mVHmhto7pPFipWzIT1fJX2SNZIwyjGKVaZkQFgQk46Tkvb5IthoB9+x3mfH/zX/0x1/5AVu2FkMRNHWsOTkEz/zj0idWbwAdL+d2TISceQrLGqGrpZqfxuvew+XAuHDlVpYOMa5nYx01iin45sg2Bq0Tj2y2Fon3jVXyqqbrVBYCR7LBNg+fXK2FjCOB6Kk0Te+R/A8Vc4bUHnJGKxASDOJgpS1IdQu+G9AIDS+25TJVisbNqvWmkwZWw0ffY4Z2220oKLc2/S8rGOrKmY9fLsjexfR3J6pqZOtYy14Hqpewc+cQgmf+HelkVmtdzq96+of+V/vgXi9Gsz3euDjGvm/SWZheIRP20612UMsdnI+ghqfVuUPPWuWINuU8By/xvuh6iSNlg0iAkBoCm4pYyBaSY5ViZwX6zXTNyFjk00MUc+ilqe6y/5I1mL4xveDnSp2Ci2tLDl18fG1OJd6vbNJstX/NQdwPMrWjapEWRt0MybCeYiiIJZJ3Q83hBSsXnFKt7ysgysg545zHlvOtau5JQppQSeeTh7o+y2tEJyX8Ntp/+wMFS7/RYox7rPfUXWLfpY9WyLFW2R0P2c48ZEAS4TVEYwcx88ewuWVxkLJNkqCyXz22r/bStJxgLw5fY+xfJcAGCHvPDaJOO6y45FJ3X9BlZRHFAnAl6MGJUPDnfSa7SRK3pccQy4Yxbe8SMsW0IT87L7VrUuu8mC4ZMf9c5FPLy1pdwUKmlzZao9UdEK6rMDvtzN9dG/WI2P7U9tvPzesxsWqRd3+ILypJHLExJPtj3TgB9dRr8XnYjM4OA7KydGqtyokN4gREWziRxUuSm6p+Jb/0kBueecqSb/gifnAm9a7eA7mw6QF1aV40CqjOsoVoHVj2VvGwAMMR1qt9kk6wBdXqCk+eCxAzSTGatlHe85fqr+bZ+Tdv4Qmwo4+SztDO36g45p/yQqB2QNULXAGdntwHDPu+QFy2R/tYPAjYsAxyYmx6womSk/lUE2gVTQOterUMHwk5RVtvG2MiFLBzla9eF3w+43x30Ikjft7MbXPdls+JiooNKDz91E5X/gfbqy/MPILanaxTXzrnkMFMrWwtzjnitWgMeUi9OSxuM8Be7rzphwF1+nXzkQzvnqZXdg3UWk6Jh0zROee0XimEN3Yf00Acn2OfH2GyF/rILwT/tOwyY1goiKQLnBMhUV9Pv2KXK0jELJuP7xO6t4Nq8ce+s716yOIoLsz5fzR7tg2bkT+JhW4ghIRVbh5B5iMJHFyV+CfOI2OOC5Zrjf9FNY81ZyE2xTduv6KOKDKXtBbmwjvwGPBUvRqPsply9j497EfbdQAYqJDxNIrH6a5medfcIJ9ncJHKSE9XHDdOfCydju/JUxUGql/ElArTXkxpscH1ZQdHIjXR8xLkL8QVAoY2ib6svNcgnp+tuKL6tMtQ/Rcy7K+n7ZZeX/yQrO4dNp5/hI1C8eBHjPZJMccD9mMpjKGJz/A0VjzUpon0wrQF7PMNwPopKh4OVxyiQmxSqqb98Jbps7FS+tMOqXVmsDAgICAgICAgICAkYVQm5vrvKAgICAgICAgICAgMyQy29/QcsXuxw5quWFoPGAgICAgICAgICA0cRLLGg8uFQFBAQEBAQEBAQEBOw0BAtHQEBAQEBAQEBAwGgiWDgCAgICAgICAgICAgJ2DMIHR0BAQEBAQEBAQEDATkNwqQoICAgICAgICAgYVby0dP4vrdYGBAQEBAQEBAQEBIwqgoUjICAgICAgICAgYDTxEgsaH5UPjvjPXwcAiN1fp1PUy8d+Scd2PYEuqo8AxSodUy9ByhjgvITqWO2Lr0LhdEpfLyYsgLqQ/m59DhgzQx2LdfkiKqrLpJbfKN+h3LgcotJD101alL2R29ZnvzYD4p9/CAAQvfqTO1Qu4z+WTgUAXHh/LfM9cusq+hEVAKiBot4nRMH8X/+OzbGoRL/jmpYhv3ka3XrOjamyRGIgSim9xwB+p5E6xu9d4Pl3U/+Y+M3N2drXv9aRZcqwL4rTbYY0v+vD9LdQhvzhe+myN1zVWm48Aqh+iqEt9LfcTeMCAIoV+jvcr+SXsPUD+wIAxnw1e9+Ln70bABCNn2fVJU7VLwuklBj85FIAQMeH/5Dr3qzY+v49AQA9V6/OVqe+NQAAERW2q110n0D8vfMBANFbv5Y61y7sd3/d4dMAAOcR6g0zAAAgAElEQVTcm2Mcrvkr1aFnmunvAxvpWOeEVDnN6ho/80fg77cCAKLjLnfrF9cgCqWW7WgmX9YGUfvKGwAApQ/e0/C61H1qrhGFkqnPyDY6Vu4y5asxi47x9NeemwY30d/qWMgnbqN75x/ryMLgJqBnult4PGLGM89X/euArkn0u1Cm8tf+jWT2zoO8/7tU/FGXZWpf/PUT6d5TvwaMDJC8Z/9Ix2YfShfVR4BSh/qt5pVixbQvVnPDwEYgVnNSD83rPIfIwS0QPdTH9PxSHwIKFfc5RAVzDOb9AwAGzfyZZ02U/evonhZjJeuYim+6gKp6yrWZ62DL95Vhl/3n100GABxwW4wskKvupx9jZjScY3zrlg/xqgfSz5ZlRkWz5uh3X1N9HcDQVvpbGQP97rj/qvcrb/s0xCuvoFPcHzJAbn6Gfgz3mf5Qpv0RimVdJ7n1OfrVoeafQkmPE12/Ugfkc/fSdb27uG1Uey5qL885Eqm+HteBGo0XcFl6TSxCrqQ5Jlr0hsxtDBg9BJeqgICAgICAgICAgICdhlGxcIi5h9n/o3/nkGZUf52WOoxGXGuOob+AWUtQmLEAePhndO7QS9Q16iu4e6olg7+ljHbBp2lIaTwGNkDeriwop9/QunEJyKGt2kKij1la98ya0blL6F6l/RKs6dpBGMmmxHHA2k5HayOU9ag2SP8vlK1zBXV9lJIRb3oa2H2pK996NrI2pOSxtsP/7pKGAm3pqA9j09o+AMDEjO0z1rR0nbSWTERW37JQH3JkCCEgR4acOvmg28zaIACyMobPQpSKuj32daJYwX0PbwAAHNmqXXZ5XZNNe/RYcduYtY8KIVA55ky6lzWs3A+UtRIga0M7kDLG/91HWtLXZb3Jto56+ky+8iXQ0WN+Y/ssG5ZkJSvCYL3FpR6whpLGoerv1bGpemapq5h5IOTGFc695mTUtN2ZnkWhgk9++wEAwLIPtr7cLlvXiTWrbFngflsfBtiio60ydTM+1fwjRAT5DGmjxYLj6PJSJ11TrKbHeFQyGmUelz1TAT0nqTE4YVf6W6xgxReXAQB2yWjhEKd+zfxHze1izuFUj7/cRP/f51TLYqrm3rUPQfSSdVJberomQ95+JVX9qPfT9dVxdE11nJk7uN9ZY1M/B8BojtnTgNveNQnblh0IAOi88slM7SPh9AzjzSshlOcBexvoS+J60/nB6X8zF5O8+6+nY/sqC7lnfrXlNu2n1p5jyTlntGqRAzF2FokY2QbYzxFmPbSfdTNE0/bVv+tffT0AoHDRj/Wx+DtvJ3lnfFPJrZibO3obC1Z9RLzqE5nqkQLX324H9ydlwUL3FAi2EkaWpZ+fLdchKkD+4rP0+/z/doqh9cP1ZgEiMzb5HRfgrJWOfCHw0CUnAQD2vLWNDc4LgZzW9392jGoMh4gKkDUyjT113l4AgDnfeZpPeiYGz6I5e2+IRa/R8gBAqgVJPv5LRLvROeNO4Z9sGh4fOws4/sPZG5VEudsz2QhdZuaNy8qH6LrFNIB40RDJwdYmpnf6XcqywPkw4EW6YMyrre4BANE9BXL9U84559mohcl1lWrsUuUrd00/nVvQskUsIsOHAeyNmdk4SjYDWzLqq1cA8JsRmy+CZuI1ZfGHiXFxqcf5N79y4HmqU6Ubkmum3C+E2rTmwqbnVPV4QVD9oG81kMN03whHHzyjrfuEiBBveIL+M2YmHbMX6ab3Wu+6c6xzTG6j54fquLY/pOwN0itn5b9fsptdVDQfeuy+Yj3zTC5l9WFg3XL3GG/wh/sgmmxmMs1l8Qgu2D/bc3fAMm3XEaV8QaSUL4UKMLBBHVNLWanTrAvK5UPGdX0+WWcZx6BdTOK9x/wBozZXsq43Wnqjz+XIGJMXJNyyWkG5/qGj17hD8XOfeQD9rQ0aV0p+lx0T9ObPdn3DpjWmrVxfwFWQqOulrJmNv1aGydQcruXLOqoL9srXPkB/8EXj5qB+DbmQFd7xo8Q1zTdc9jsRe5GbDG/04+tOp/+f7VEMZt3I2dcd/s5s9yRFlDr1mNQuiG1sJPl5RxfQZjy+5T0k6g1XueMB1OeauTumZNcGM3/8uDdabtG8Bqn9jdzyLNWvc4J26QQry+I6UFH9SY0bKSJg0aG6/qly2L1PjS9b4aHHiBC6bwuhPsL5maCAPc48PX8bA0YNIWg8ICAgICAgICAgYFQRgsZ3PCLLFK6+/Gdf9mU6Z33pSv3whT5HGigYjfd+pyH++il06KKfONeLGQc67ktUprQ06Ebjl7xOa1KqYzH4mVcAADo+9WjupgohXPMj0Nqa4cPQNlfGDrJsMDYNt6Ed92g05YbHAQDRhIW5ZIliBXLNcu85Obg5pW33aWmpHg0sKoUy9lo8KVedjFbQDAufltj0J7tA1/UPAAaeXQEAyK6HaixLaouP6ddtWanYDD5ujjm2Pe56m1a79WJtdNfk3MHaaQgMbaGAw66sd9jvSbmecNCiVBpB0cJVyMFq14WEg7LlyDZI0ZFNRgJ22Wv7qX/Nz3O/CuQUvXPNMda+tygveQxRERhWc42t7QbIopCwcDhuV+p6ibR12rbMTTni6NaNSoHdMq0+5HGTTbpbeTXLIgK2ErFC6l016qP6OLcjMppXT12HNlNgddZ+inI33VmsardZDngV816h6mqtU1yfnmnpZy0i1JZTAHuZrS419VxG+ptbLoVZcxsHPgv0PfB7AMAY7xWtwZYN2aeIObonq+Kzjx35Q3JXE2eRS1WkLBv+9SL/+tb36VcDAMZ8ZV22+lhrg3Y3VhZQm7whO9Q4ZcvFqz8OAPjV8dNw7C9W0rkovWXLZmlsw3cTMMHatWErWJvmBDFWrSEyhnye9kqiV81kxQog1Whg60fXZGDRqxL1seYhbXWzPQiMlY3uiw3JgiY+UG68pSpwyNnttTNgVPDSciALCAgICAgICAgICBhVjI6Fg/1QOaAU0MF7mtawMtYJ/gHUh27CT1o+92eIhS+j/8QJKsmOXkMjqjUBUmusG2vJLQxuQTw8nLouK3w0eMOfJQ1f6X23Zde8LD2/4amYqd9mvKy9SgI4++I35b7HV3e2bGjfYV8sjox1vIATr7H3sX751bGpANZGz62Zn/p1vyNt0yWNGpSESPvkewMSEzEcFEqi6J6lsphFBXQfmF2z6/Qb1hxLaflf8zEj/8GN9DsHeTPElL2MbPat3USxNGCNeVTK3k9ffrqqnkUfCkA+9BPggLPomOVrnNfqMe74U3Jd7wVrfZ9W1L0zXpY5ngO7H+r814mpUUQB0tLMZfGrtp/tvBmZdeIaNqWxVvarOBVp+Tonn7UcGdDkE3ocIgKm70HH9FyrnlfPNI/21ASq2+MlGWNmLF5FfOUrPwcAvLvxlJaGpdnXslJWXmmCxhnxiA4wlZsoPjCauCvkuNbxRF5Nsa11VdTnUhEv2HEzHRMzU1MQlIVDjmwzQdrKsiGX307XzDk0QREKYNt6bUUUTAUsIhQPpkBjHd9jkUtIpgdmlLp0PzUxcpLqAuj3ymNERAVE1fxWUO8coqiFBz9NAfKVS3+ZmRBFnPIV/4nKGMTXk2ZbvPUavjrTGLfr2PPROzPVw9xrnh3LYctG/Oj/AACiXV+VXx7HgyjymeN+uQbr30WWgwmfVnXsnqKvdy2X/vVQNLTOtaiTEzTu2rdk9xT+BTGDSAVskhc9n9hxZT+k+Fhx3vfThSXq6Hin2BT0ifQJqHTrez56Au0rlz0y0rpxLwa8xPJwBAtHQEBAQEBAQEBAQMBOw+hYOFj7Weow/nnMOJKkNbQhRMKfD8C4uaj95WMk7tiPuNfLuo4XMb5/DerEjB3JC4pVrHmM/CXnJe/JAJ9Wp3zpr6msgY2QnFTQ44vp4PHf0N8D07VIWjZkXGstL4HrryYmjDPPzHVbQzRl7PFYPYQQDV8Nn99ebG7fUJWCV/vJ2uSoYPVT9oMvAIP9meU5ySg1I0ndiXGiG5WlQ0TYa1wbMRy2Vow1gFpjbrTYyWgAhyWMWUKiIvAX0l6LOUR9zVovccBZlnUyZ4wDzxeigMe/QUm+dj3pmiZ3NJREYlhzNvsQ+ju4SWvLW/az4W3ew7ZWVlsV4hHHmpMFtz5C82AeQk4uzxlzImlBjLW/fDJhHZ23+s42VwPeLFEaywZg9f9i41gISBwwoR2KSs94s8YK1QPpxJhWG0XF0siuW+EvxmZB8vUFO4Yiydhlabj719CzzmwH4DFU6nTZpgCI2YouPCoCQ4mkpbVB8z6NMOCJP9O9h7xT14n+xiZRm6YdtRKQ8lgb2gIUVe2TXgUyRm2bfxzkBT/j6gd/B4AsHfy7NfzznRAC4ozr6ArVJ2tXvwalS37h3t0qCV+9vQXDSxm9kBIay23rITpd61fLenjW8glfUvGkw0T1/uCJU7Hvj9cm5MaGta0zp8WtBVxWKWarUnu4QsUwVjE9bqHs0OED6jmVsjPWOc/ITrarx2T6nuOntxmrEjAqGCWXKjaBRWaDX1a81fZkqyg7pb3x4UWDF9Sxs1GcOF3JSwYAFtM0orUBHSyqF6d4xOT/0GZBhUo3pu7WBldlBoiOXp0tVbTKljopM5krRGS1e4CD1ppPOCcf187nVDbILYoqlTdlihM+hQlz/McbyeUNDr+74a1NqVfP3itn9/YEjTOcvqYmfRNsLSzXC2tiHmq8SHsXHF7wOGt91yTLpYLraOiep85qI4TTs5iluPGlhBxUmYr7Vebu3l30dezWBUBz42tZvAmz87HYsnkMsjvaxuU6n4ElRckSmLVfPjICX11Sdav2pikYG2HDswkZns0Fu2wJK79Bxizn27InGE+V16xeQkSIlQtrlMiLk7xOJj44mm2GhIh0kDOPA9E1KTVmzAd1hN52+C5sdy3LXdMtpG4yyyuqVMowz2tMazeShi6TPI6HVZbk6jg9FvUHj3JpEkJARPk+qu0xZ3LxpD9WYw6yVq6rwppL+WNBREUMPvUPAOaDx7xD4d2Y6WfKH6Xdk40rFbdPzbOi3IV1z1AfaTCTt43qB3+H+PdEIMMfSw3dZ58kJZzYo3EWaR4bpUt+YfoBrxfFSlOFwLbPUrafrn/3k5nkgW5D58TMgfJNc97wmFfB8fv+eC3i5+4DAKz8INV75rUPtVz388LOr6UVxzxm7FxLavzpcWu7QnK286iE2nraG2SdEowSzvRN6VWGAXkVWy8KvMTycLy0WhsQEBAQEBAQEBAQMKoYHQtHK1M1/62yyZrdrgZTGlkhBNb84Q4AwNSzfEUlkhYVq0YDydpiWYf+1kpo/YQQuPvuFQDyZXDOCnHSF6gKiqozmZVc4zmiOcQu2Wqh262eoRzcAlFtrAG/4w5KinZcJuk5wVYHh2bT0wfWqcRsGZXYRstD7ZLlLmMh84i/+e+kkbk0m3jtIiJFwTxPO+FQukJQF8HnrsBIJRvzJSyUsXEHYS1mVDTXJcYLaWGzNsyqstdFxqNZ4+BiVRe56SmI8fNVFdrPHJ4az5Ux2tqgXZ+sMRkPDbVVVjMIIXSG5ZaWiGq3/3gDueZ3OqDUh1UDO1Er9+vP0d+Tvpg65dSpa3w+uawlbkLVbcufe2AeWgOGNT4SxAla2xkVjfuPj7Za9WEpJTA84C/F836EEIYIQGeQ9iShtcbB00+RRbAdMtRmePo9RwEA5lz/bOqcbSWpvPFf8wnmZGmP3kqy9n9b+jlYFqL5F74nn/wcEIdcRHV5jFygRKNA60m755Or+wFp4mtfeg1K7/nfhtd3vOYdueRntWJKtoj3K8ttyiVOIXYD/luB5+MZ71xG5fzq08BrPq3q1GSezyRdlWFnpU9Y251g9UHl+qfa5tCTW1nYi+PyWmDUPMAEJ4VKQ8s1APxhHf0+OGcpLxhC0HhAQEBAQEBAQEBAQMCOwehYOHSCKPMbiZgDCsozPrEAIAtFS4PNGucaJuyxuxKRVPHKtPx4BIg6HbkQxcax5HENM7p23lensLVuAOKVf0Y044D0hTP2ak++ZQWIf0sazuiI96WuO/zI3dqS3wjs74tSh9Hugukpn9KJ5hzNy6z9G8vjeAZL253SrIsI8hFK/igWvT4l4/wjc+obPYHE2jpm9zWO3bCDZn2o11Ly+P+mWzM9qYDW5rCGK64bDYjHErdiB2lVhTW2zEGli2Bry+ZnEoGDSrM0dkZDmbI2lDyYjknpnKgTQ8kEPaYQEeLhHW/hcJOB0rF41f2Ipu2XvnjBoeljORH//ccAgMjTR8/bN3dayOyYvQ8Aa2wObYXomZq+bkr2ucAZB5zsbHCLCYJOBlYDKE+YkjqWB95Em/qk6qc6AaD1PIdofKDaC3Q1ij5oYH1VlMfa/78Mk3yTx/sQaa5lpQe7LdmldUPaQKGo5tAVv6UDcw/3a9Qn75FPsPLNF7sSNbl85i4TrO7Drkfmk58Duj0LXgkAiG/9GKLjLk9f1ygWsJV89b6KF/8cm95L8WLjPu9J6rvgFXklNzzjJAXs5b6h1vxGc02SIMQ+xf2Q+2W5G2ACmgOIEhhDWzDyuWPotCKqSUhpKD8bTFxW6szah+jc9CV0oNG7mjQ7Z5HqmdQNkUizOJyDJrZDUBEwWhidDw7eeEQF85uz//atAgCICbvpRUsqHnhIaXJ3cCcf2oLCMefSb15kdBZKi9XHY5Z0FsuiOq83tmryGNmG+e9+f/425oRmJ/J9bADA6kfo79wj2i7D96HBuPK/aMK9/ONti3dhmU0ZeqM9bg50kLDKHC+iArDij3R+VtoAmiWzuhAC2J0C5nyb4e/cSR+z78rUgMbBfMlz0p786IAJqB7kTc4YYGxj87EvyNdkf7b6Py9ckRvIiXIXqm14NiXzJbiVsALzeGzxJmvavtALLI+ZYgX4q2KC4RwCng2pf2E2WXVl0sXAci149lEKCm7HKacRHEYwfr5T9zF5CIpWroqVf6Fj87JtRrx5K3ZVjDU891kftr94lJ7lue02pgnEfqfRX8XgJK28BE49t67OLlMIcIC4cVvtcHIsJeVf9XViIbo0c0IcGCWVlCm3A903peHk5z4j4xFAZSCW991IdTjkIuDgsxoXpfuCFZyuCUb4I1yYD2PONr9pBZ2Zsheq5zbIEdGoTM61UCil8vrYzIwzr3mQfnJW5b61kJ3KBY6zWvdMA37yUTp2zo2N28eIazozt2ZN7OiFVHlFNBuXaq/omgT8/Eo6tttrc7UzF9QaHx13OaRiW9K5RgDI36ng8ld/MpdYE3gcYexVFFwvNysXtTGkMBFCYOPltFmf8I1NKRne6jZ1hzEuuSLB1iem7afnGtvdiN2XeJ5w8oiwS9HGFfT/wY2I5h1tiwU6elF6D+X/iH9MLnZCZSsXxWpL169moGeYHIdqvMg6xETl7sauqvURk33ddj1Tro2+PFsp12NHASb0X3vsJGWNbO831agjuFQFBAQEBAQEBAQEBATsEIyOhYM1h50TjeVBZVplqjgAFqe6+g6KCrqK8unf0/VzDwceUMFle3K2bPtLNxGMG9dcnnyA6qDdYNTX9BrSZIqp+0D+9f/o1PFttHU7IBUdquicCKx9YqfKP3luftNjM9o+Bmkq3GA60pizqxvzZBc0N75PbrNM4/b1qeBu1ooUqzh00k4yryptox0ErutrmXtrzz4OIDsFoENlqf4kucxtbW9/G5SqzWj4XFey5NRgPX+teaoAW9Y3lNFce9M4/4F+lwWgnvMVZgnkdKxhdv4K1s6xJaJYAVY/nqt8n+UqaW215U+s7hiVnHdsJsp13qlNC/xHysmDvd2s7q0C3p2s6sls0Zb8Y6a1MQ71+7P7pEWYAABC6izatiZc1/ngC4yM/6FgWrztW24xdk4KWGWy5pndsgolndtAy5+0SN85cO2FAICuz67I1j7LPdQ37nWdFC28fndWIHf8X2TBFufcCMzzuOik5MO0RReg+k2pM2Wljn/973TudVcC07LTtOeFb8zy+2QKZlHqAIYa5zVKy7Syf1vztH4WY2fSMWXFQUcvtm6gPrsjAv+N+7bfDM2WjXjDkwCASOdCgjdDOrsq+lwWnes4O/zr6d197VByoTz/zuxWTBvG6ls1VsfYJW+AKJjxynNbqSO1hksZY2TFwwCASoLcx8nzxG2Jipal3bL6e1yUWcZ+i5o/nxcdAi1uQEBAQEBAQEBAQEDAjsHoWDhYcyJj64uOfbhJCyelTCVWcrLGKnrYZUum4fJbf6vEmcRH+rrkF6OtzfFk5tX/VUFc8f3XQ7zuQ9nbtgPBSXtkXAPmLtl58uvDuHkFtf+KPPdniHEgLVL6O9b4saog7LgGzN43h9zYsZik5as+wLEU9RH8YiVdv29D6TsGZMFRWnmLRrC4qD1yPjuTtN1+Emo0ydy+A3MJ336Nuk2XiXHbFxDslW9py+9YS23cc0fKb6FV0jEP29YD3TteY6blD2xEsQ0X3qQPszrK0k05zcaVeofx2oeAQ0/LX4kWYPlycIum/s03DhvTeloHIMZyEKqJf9CWCtvC2usJlk+WaFs7uCidWNQideC6cXyTELjp/ygm4O0tS0kUINLxZ6bt0mNptK46+7tKlMS2X9Hv7iP/VR/juqWLtrT/TSiuo9ddqX9/7KM3AACWvfW6hteny8lGG9vsvCZZGdwCdGenb/auEb5jymIgh/vx/FbaT8zNWEbW9jUDWzbklpUQKp4kmeiyeaxIc7Bl4xuHTcXb72jDyqGs+YOfPBTV9/yIjnH8z9q/Uv3GztJkFE5sXJwwwdeGUD7/2wCAeD3FkAq27IhIB8TrUV4op9Z1u4w0aRDQ86FfpI4FvHgwSpnGlYlQ2JM2BxFz4FEMjJDJVJa6rOsVRigo8Yp7n8PQvx0JAKi8/3a63gqyS2Ual7E2BZqgwIKRzZMFL+L7no6H3kCbqD1vfWEYD0RUhFxBWUSTbg47RH6hjBnpGO+W0Fmio4JZMGWCqUlKy+TKC7+AvYiqg8ATf6Ij+5+ZOAd9fep9IrlAp83mXJ9z9k+4ebQAZ4RFxzjTZ+oJ9pu4ZkzLtpmXf3NW1WIHsJRcOrRZ2p4gE4GEiGvmWI0nXmHK5f7KZAuFEj7ym7tztQ9IblJ3ACbM2rHyElgzmG+xbZg5Gu7Cnck9sHMi5KY1ucrPA9HRixX9+TcT/nfYeIPerI3R5MWI71cuVU0yOLcLUR2DBzdS+SfkuS/DJksi0vOPfu+2O5I9neTIp0JyXMWYU4YuwLiWPJP3Paoge1nqMBVll187y7pUBA1JdjdLBopVlMcrZZL0ra+JtU7GJluzL4s7P3smhyiUcc6e+Rkqsm7Ek5mivR8G1TEmN0qbaOYiKMpdeGgz1bcxd2ICPBcXq+aZJcg2IGOgxu+wYP4mXZCq4yD71FyjSAk0iU4cp12z7A29naMpmZdGyT/nN09q+bmykat2VS+7I3VMTN3bqk7i40JEEJEbNI5yF+SN5Hoozro+UYyAbOaqbBMqaFe5hIslBL786kMAABc/2I6/8QuBEDQeEBAQEBAQEBAQEBCwQzA6Fg6m2+uanNLUahcUEQEcIOfL0lshDVW84rcoz1QBbMls5fVhowmyLBjGbNxEu6usMPH6f2D+Ae0HyMmtq4im0D5maZEym0cXKZpRRYPquLHsABTa+NTUNHQO1Z9ykeL3WepMaWCp/W4mbrn2b8DeblS+49LA1hQ7E73vuoR5VWtH6sMYGhjJ10ClWXLcDLThRsn3BPQBsAKBFaVqqcPQOFquCcl6m/5gybXcsvT1rEHizPSFCr5/AuVPOOVP9dT1jaC1TUNbobUrKhOv1rQWSpm0kzKuAx0q6zu3n9vTzM0xR13fPCd72/geVTnLJcHj4tdkHDqmepW/wXZz217Y8rdHmhNoyZa1ksk5lArG92h45cgAMP/gVL0AAPGIzi7cat5q5F4i4xrGtGFUs91l9fyp55CCrp+2hNpUufyOmEK6OgYou9ZOW6amtE5QJasK0N/6sCmDXf7YwlDuym8B0GNcWGNGza/PK3eTibsb1gS2JA9s0oHj2m1NShSXHJsowLJqqHwhepxHJUuLzHOkNM9L1c3OGdXdm89aDADxpqcAAFs/fhzG/PvDqjqJvEbxSFMKdGee3Pv1jtxI5XbKilZ9+PT/8uWuaCJPuRatuWAXTLl2OR1UbYmfJet9NPMg73yuYbed3RzXE3VvNNHkx5GcN0XLap6Pwsi0tnge6vrWMHS0KYIWFdCPcqe1PrNnST1lnRBREbLsfxayPmKsftwuh/bassQpOmimdNZrjSjgTXv41+cXLUKm8YCAgICAgICAgICAgB2D0bFwKIiooLVET5xJiWLm36wCmaygYH297+uvazLEiZ/S8uhW5ev4w0sRvfnLzrG8GknRNQnV9/0s1z0Ouqemgzrt4MWsgWbLyT9fLCQrwI7UsAJoK1hVQ0RG28ftsOJ0vImmOFCWNZdjZ0E++EP6rTIwO9qsZIxDI7APd1JLCehkeZmRkTJWF23X15fkbsOqXPKMXNen2fldMBqc4biNl8h+2aUOEytyDwWCigPO8tSlSZC5rJP2GLBiXNRY3PwshKKebBsyxqIL35n7HoD6Q/zEbXRszmF0rIF1Kgkn1mPiHCWP+mO8RgVKTlrUNKA3K848LmfmXdj+0sIkvHvkpwCAyIr50kkoK93mej7H73Vws2WNSmj4tz2fstY69aib+IKG81ltCBd+5LxWTfLAaFP1PG9TdKpy5ean6WrW3pY6IQpK829ZZDGeYo30/Msy68Np6ygiM074mQz3aV98bVnQMkYw80SmaM8G+SRp08Wcw6yYCY4XU/00HknHdQ1uguhKxGvEI8BktvpbMZEAjUfd7604AzsmjYSZ63iuZvm1QYw7K535uyWUxWTMZx/B8++m+k24+jFTHpCOTWgGZQFiy4bc8hyJGDM9f918aGaJ8ICfz2FTZFgAACAASURBVOSr7kZ8P8UkiH1Pp5N/ogSMctqSTHFzUkptpRQTKBu6Lzng9kByosg8MRzcP+rDFKtiHZP9FPMoImuOsOKEtPVGx34WgSMphkPTZnOfkzHk6gdIBCcRjIpmbuL4JikNVTJnM9eJaKuYuuzm7G0LGHWM6gdHQEBAQEBAQEBAQMBLy6VqdD44+Is4rmkN6Lwzz1Xn+Ms1hkwmQLKZGNTXshg7G/I2lcTphI+p65W25Ih3WPESNp2oy5AkRGT5erN/odK0lLogf/UJOnbSNbmb6mq7rWPmf9kEDfa5MnaQZYMxsh0EXEIITT0pH7qFji0+SZ9LQtrPg609Hb2QfRu88uXQVgj2z2xVl4SW2aZUnrx4YSYZGnGCkQqNfdOdshqcH15FvsZJvVmrhGrwsbUkNIEyrmNedxsvkcdHqaJHv9jrjfTDY6VpZtmBlMB9pFnXViplVRI907aLLpKEFLD5F0TH2fuGqzPfo3/OozgojiGTnDyrUG5JG6rPPU0JQXGIkjlZEfT2rYHsmkTH2hybQghseY60hLl0mOxrX+kxGvB1y9PXNY37Uu+wY5zRQJqa0RWbVjS1cNjzqlZYJ9niCiWjfc8Dnp+d8Z1YHwDtz41eNV/YViwrCSj6N6bv5SOeRGKaEYnjNOo1oDNBy8rsQ44Pe0aMJYuLKHVoS5S8i9YbccSlqiIR9HtiCt7x89LsPHEdWPkQnVfzsI6pGtjY3AKgNczSofm1IYUAVj6cr32ApnkVQmDC1ZRAU/7hK3TsELZcNqfpdupy303097jLlXxql+xfB6HG4nbhj4ry900HZbueLRBdk4DFNIfKG8maJ079mroo43ov65AbKNlvpNif2KKQZz1sKF5KyFX3U42s2JCWsNmn6hynqGJ8uF+JgkmQyTG0haJluVNW2GLV0FizhYNlVsZA9CqKXJ63pDTWC/VXDm4GthAFNVT/0vOhKED07pK9bQGjjtH54NCTgTAbzleoSdWiFkxutCSilBuEHNmG+j/uAQAUmWeRZTqdzeIyT37IWL85e61GXMOfv3otAODANj44pMc1rO9DxEDf/ckHsgeNH3aRkpfeGLFbRzRlr9z1Y5z1gcyM8SnYH3GRyvbO7hXSCkjUqI9A8gejvVjOdzNIGIIAM7m2ovZs9kHw1R/QIvnerMlGPBvHZjSr7kHux4ayr7yHf+FqxY2vaXclXBO1JR8iwsOKxvFQbykNwOb5wY16opaP/4aOzT+SRI+ZAQhPIG2CF14WSsCx/0K/9cKgNq+Dm/WG3HaPy8spP/a8z+S63vtxz6Z3RXssx0xvzv1vy+htsFnrnKDdV+xs8FnaZ1+z4jlaeFtnibDuZx58aeVpeMX76ZhDXc2B0ap+fWv0Iq3HYaFslDkJalkx6+VpdyO7n9quWEOb6ad61vr6qITffOj/AQCO+d37crQSjhwAVj2tukzeQ9WFA02tj3DOQl7pSdF2+ui23XPqPK8/hTIwzJTUyuVIbXzEpEUYeZRozLOGrGq3meF+7fImjqTnhK3kKoTuKZbrkWrzur8D4+bqdgFqYzqeNnIcXKzdsnqmGUIPfjalTivQlueXGuTmZ+g3f0iroHFRrKK+Qq05GdtHdbDyaPG8uPRdACifFgBccfcK9yOxCcRRl/qPd02C3LKS/qM+kNtRdohj8+Xf0q51256HYMKR0/4TAPCll9OIfvcfGrvVOrKiokMzC1htqPQ0yL2To65CAPOOzn9fqQlZAH8QSWnmWGuvpccuu90CiH/0QQBAdPo3UuJkz3RTVyiXMs47xWO/WAVUFnq9Hlkfm9cctw8A4B0P/JPQ4oag8YCAgICAgICAgICAgB2D0aXF7ZxotLfrHwFgND2ATAczydgkb2KN3K8+g8IZX6JjdsAbQJSBnDQHlrnZ51rA2klFTKm/qjc+iQOuvyd3Exk+zUrXJ+5X1ZRAMmixkXvN339O1x2SDprdHssG4/MfIw3D+958bRt3+77Kbbe1hEl+y0qI3rn0W2tMI8hK42RcqYQ/jUzTrL1TWl8bL5+4sxI3ct/iwPmCpVm16ChVssos8PYBO7hTl6n6T6GEhT1tZA1nrXehArACslcFd6sAXG2qtssd7ifXEQBSGqrERlSLsj4MoTTCKDN9obFiOtd6g+QVJi1q2py0LEsjnyQ2YG3dwEZIpSVr6VI1PpHY0LY06cBr0qSjUPbONT4rAf9/9pT8dKMMp+6sAbUDftnNQdGNy7iWfsIy1ufTp+rN3cUsq4H8B2X4Ffu8NSX/qNe6lsxM0NYZ2x3T0z+YmrNiJVpjEb/5PP048XPabTDtLtRA5yaTYzzSVhT9Hn/6cTp3zndRvuh7zduTBLsSO+NHlcnvY7jPWCq2rQcAjHznEpQv/rEjSggBLCTttfBRanOfHFZuLLVBYx3hdg5tAdSaDI8bXXTalzM2zILjmuoSElz+U3LFjK99Ewrv/KlbX4+XAACASRASc46Ukij3AcgnlbV29iH5g63ZPW9szmSmnjHy7l+RK+a6d87H5K8+6da3PtyUCtgHTUnP1LFbVyNSa2pSNl2flr89GcuB5vO0fEyN/7lH0oFSVe+tnPKPulDJShBUWPMpu2sTpbNyN/zH/9CxybuD3WbFlD1dWRC44NvfaqttAaODYOEICAgICAgICAgICNhpGB0LR0Vpnwtlo7lSGgmtrYjrKU0gRGRputS5E5Zh6HOvIrEf/J1zDpUeo1VxYkN8cAPJtavs+AXoX/ZyAED355/N0cjG0Bqx/nVGAxp1OudSaKL9T0LWBrVmq1XMA2N+O9pxBa/sJjS2FDzoCYRmjVXTMlpoZZposX67hr6nD24uYTtgafpFgloUQN5AUpnw14aMrY7JzzfW164byl9jLadY0b7TYvZSOrZNBfGPDECWu00dAAruqyf8h4tV4Ll76Tcn4eLA07EzXX96lpVMDCelm9TJLjMqAj/5KP2+OFtSLieQP5mAkn1/680fnNNH+9a55zzaTFlV5XDwYjN5if/f8neqy7ua1igDPBpN+cwfqbwFr6S/PZ54FBGZwOjUKf/cqeMh2LJT6YFY9Dp/vUSELY+R5ry3Ud39NzY85lo/3YSiiKzzh11o6vyXH9APj8VYX2PHZtlrEP9Vz1hf94oL+EbIn1GgmHj7Lc0aZeBJqKiL5znN0vCyv3zplefrudae67l94rD3eApTZbCFzx6DtpcA05FKM8do+Te+m879613Z2gc4hB5mTVfa6RkHAAC+/N178K7EK2kYf1H2r4kOUYuKU5DP/gliVs6ZP8ea696Xtq4zwcDtj/Tj5NTJBvGIVrLLRtAWrEaEEFHjGA9t/Wh4RXOkknraFPSKehxFNQ+JQppAAgDu+g6dftPL9HVaHhLrhSiY+J6Z1F9E5wRj1UR6Xt3wJZpJJ3zj9MztChg9jNIHhwqKE5EOsBRstuQOJwc1p7rOqlnupsA5QG9KRKUH5aPPUPLcTal0JiR1bqQPkjdKdeYy3wJsJEYIbQLk+bDUgc4TL96OxjaG6JqEeAOZV8X4eanzzsfCUF/qfEO5xSrkVpXPhBerqt9NgnHgnHyc4y3rYAWD67rw+yhVve4kGB5IyWkGbdLVHPnC72ah8Ma5OT+qdObULpfjHrAmcgko/nftemBvpPUHAnTdUu5hMjYLr529lWVwPZxNZNLFDHj1Utv1KRvMwm+GvuSPQWZ9kTHk84ovnxlJeqYZs74O1BeQ2za7cpHYqCWgWeR4o1Mb0iQBRoZZhPqXU9bdNrcCDrT8YrV5fhEbtdbZ6g3ZQbeVn8Z9v42wfnDHBA16mZZuV/lVdntNk/siHayZtRw5wvlbvgUAiI7816bXj33DhQ3PN4SXwCH5rATileT+GqmPKqffaVYpCQxs8cuwWc3s8crHa8wINiZ1HZjtRwiMPEssTJnzHDdjvbPqFP+CPrjF6/+d/u71FjeoFqA5f25jZiWfXO7/q84nJr9p/7kcUSpLfF3fJw4/I0urcuOdv19FijiAXK7hviPno+dRcqsRe6W38AwdnD7rYMR3UwC32P9MOhkVm47H4avJ7a5yxV9ytcEnkz8aTv7NGh1AfvFdtC4Kj6uZECKtoGlWZtck836kcbNtOt+k2Ohawxes7lNqSj2HN2dzHHmCyAfKnvk3dW+hBKlyTen9YlTUH3hmvKoP5DjGzx6kfdOZWRr3YkAIGg8ICAgICAgICAgICNgxGKU8HByoGpmvePuY+r9gCjT+Wx+2NMbWl2AiGNfVIPvcfTigUH2Zd/RCPvx3Ojb3CHMr1O0cXLczsEnx5XssHI52Yk5GLnCGsgTJG84hWWdc1/Tyqccen09+RgghILVZNa1Zc8B9ILv0hDzZ1IT8qGLqzJyNI/JoO5OWCAnINaQBEz2KzFRKq39aWqrOpCamSXZdm3PfDhRv4loWVXaQlSqllRKGajBFgQrLXTEC1rkBka3LYnns2lVOB/JaGrTnnybt546wcLhoQUbA0PkbssGX8b4ZOgrtuza2xNHnZ7tuLeVw8M1JTcFkAy1Qv+9WAEB0wqezy87w/Gw6Tq8rqbDcPPuez162DYe0IdFX9JgQqG/zu9M1BLsKixZL8L7squZ5HmzpLXVC/uzf6Pc7fpSr/KmfUNdLmR4KWuNeQP93KUC+56CLssnPCCEEYmXhEGxFsmhyXVrkfJSw4mVE/S6f+BX9f97RjdciALK2c6hUK9r4naZwd9qXyDnWElpOxjkk3jHtazf4XEoJUUq0zV5LPNfrPYJDY92gvbKO8s6cTwO2GyHTeEBAQEBAQEBAQMCo4qXlUjU6Hxy2P64O7uPEQ1YQKwe18Zd4XDP3Wv6N/T+lLJ49h1Ksha19NkkD1Yssd6fLrA9B7H2SWx8Ly6+mTOPzj12Wp5WZIGZTmjapYjREo0C1Z1QwrgquaymXtQ6nfZ3k962B4PgXD+78zg8BAIefk0l8LggVnGt8TOtak+doR6r59NYmizjHElhBo56Be896erevzl6Cr1CrLHVoNqedVn1T2MGrVl9/9hHn3lSGYAeWBtXOkJwgN7Dlr3uUYp5mN2tSDjiWQhULxMnN8NCPgSVvoyrY1o69X5tTfsJKNbgBskgxAZzQzu4js449Nl8jMoNjZ1oEavZM9B9vJd2KzWlGLTtQ34kLziO301+Ob2iEWk72ASYbWJjt3QyuoZinXPppX3B+woohhADGzHKOObD6kxzyW619tORCCI810pO0ssNkHr/rrhUAgKyp1ZoFBjt4Wq0DyhLvkCJwsjUAYr8TkAs8x/yCYkNw9g3pOloWhc458/PJzwExgWzQ8r/Vev6WBgl3W8QlpuTys5p/DMl/9u6mgeTlJfkS42UJ8gaA8675Al2vLC2Yf5yfjlvFIwn2umiBZDxeQzphlv/EbXT5+OzvMnOiQTt2sZEsIVDsSsTrNbE4CSGMLYOJPgplzz1sNS/hqb6X1gb+nw2j6lIlRWwFLqmu1MfBzh1AlXhMOLulrA+b65mZoz6CzrkL6LcnGFfW3UAqyFhnBNWDIeqEjGP3Ot4MxnVMntPeJiMLdOZ09Te+5T2I3viF9IXzD29PPk9+3VMQ306LiS+o89CL2s807oPUgc4lUwd+pivvBqYtoUP2BDa1cT6RNE935N0YxBufov8oliT7mrMW5Zx8PAuH+UiwOMjZDYJdGkTkZl3m68r+HAu0ofFxmicYd2pDxryeMp8LiGjHBhzrIFRRMGxT6rnKrod1eyWfiwpNzf86gNpioUoxR3VNBFSWYJ0vQGc0j1DfuAbAjg02szeZnFNE9q/TOXycRXvqrqn78yJeQ4GSYvKeSr55b8dMy+tWmAP7kVJFj017LrRRzpmvgJ+Pcn2VAxvA/Vh0pLmohrflI4cAYD6CnEzHLqOge8wopPT7GyDWNdkxAeh1c0vYa4fx2rUIHLjvWrlkZCLTObaqLNJjpuOoEw/J1bysqN1DgdLFGWqunHOoP7/EuOyB/wD0pk2cRnmY5MYVQO8udMwXBD1vv3zyM0JKaeaEk78KANhwyUKM/+Jj6TpM2K2tMnR/mHkQ4ts+BQCIjrksfeGBOUONm7j92XOfDnJXexn5h6/ojOsOBjblKz8BIaLmisyBrdslvymYkEdUdV28mJbzw1WNP7niDpI77yjnQ5vKMv31TbsEl6oXM4JLVUBAQEBAQEBAQMBo4iXGUjU6Hxw6UDzWnO9y3cN0jDP+Fqv6Op3JWFoWEamqOrIN4tVKY8+Bd6xBHdqiaTy1xtnOhmtrlSP1Bc5aag4+Hu5D13lXtd3UVki6BXitGwDwqMo7sB2ZlpvRVX5mGQWVX3by13PJbwhPhluteZi+v9YGacrjYhV48k46P23fxvc2MbkCRgPPYDcuERXw22fofc/N1AB43etMfcw5ye4elruf1o4OKo1MpQfoaZzNOqkBcuXbuWQSWaoHVSR8ZSwG+4eRFw7NZNJCWDAc6npcsFZ3oUUyoPJNyEoP8IDKOzDN1YA61KIFY7FIlunQsvJzYk1goYznHyJyh6m5W5qoC1xXnFTbOyearLYqeFUIATz/VFtlaQgBMWkP+q0sSNLSUK8e2HkLjuC5g8emnYW7TTpOJdn9b3WcxY2flv9f95LWNRc5rhV8nx4/bP2UlpvssD4mOSfRo7+kOuxzKsRBZzv18kHaQbVszVN9XRQ7jLa1gyxhOs/J4pOAkz+Xp3Xa6iRKHWlrroXSBd+lH2yZGtwMWad6sotOtPgkYKM/Z5TjCqotODXz3FRfFGOm62zmbGmUfWtJfu9coHtCrvYBaJr1uhl6v/Ao4n/8jMq2KJ3lHV8hea/JQT5gQQgBoSwbUlm/hOUWJ29SOUzed2c2gex1MdyfthyqHDXybz9CdOC5dIy9Gw65CPGTtL6LSYvpb89UCHb98tHO8nvkfl4f8Vox+Ji2VqtM9SIqQOz5xmztstAsr5czh9bZDdic81LhjyeiCUO5zHNylDqmc9AAxgorJWQfWb11Ljedq01gztIludsYMHoIFo6AgICAgICAgICAUUWwcOxwSOXrKsbNMQGHM1SmSecrma0NVkZU1lapjMZi5kHAHSqh1WkqQzJrwyKjrbADxTl5jQnGrRiLBn+FbyS6WjF+AXDn9XRu71PbbnMjNKOU4wRIomsSsG75jpevNFaiezLeumt+r3hbo9HIR9OOT+BrhP0O2De7WAVWP+bIbaZFadQux+8asBL1FXDoNO8t2w9OhufLpFowwfFyzXLnfFY6QRNk7LHuWBmKV/aT3My0v41LVOWa4EMk/NUpboO1bCrIW4yB7Pf7HbttFZ7j1rFEgLAOxixWcltxWlJPWtfQSYv2WAUZsxUOxSqwKu1LngUOkQVbU1lzbFn5xuRj+mwIX7s5/sFrLWQNaLkL+MNN9DuRINBvXbDaxtpxIdKZj+tmnO/Tm9eCAkvzafedBOmCAOQ2orsVrO2079nzTaYtP11G5077z0QxlmWRI4Wk1O0RnD2+WDHWAJa/qwrUlnXUvkX2m/KH78vWPiuWqZk1V6rkhdpKoBKsAcC9H3wHAOCAn5zUkKK4WVI6ACbpZ7GaslLLTSvoR+/clAUrE1pYpwEAcQ3SzuYOZYlQfdFZE7e2SW3sqxoTmwyR1UpUetC/nMZ69tB09ew8cVGCkxiydcM+JyKi6AWwbAktUsvuW5VaV9x71LFEDGjDmjHph46ZLSTioTLCY5WFj6iF53DeY0VF97xCfB9ZHQtLVeZ69mpBbCwbmvCnR3vEiOn7q1ORSZuQjHmUErWt5AGwg6bVgB2MkPgvICAgICAgICAgIGCnYVQsHJpVQ9ahv15Zc6QZB6RW6Gqtk+X7yXR2/Ve8DB2nfZTu4K9p1qTUh9NaAkuTpNl1hDAJB1m+ouaLf/9lYK9j2mnmdoOTrUkZA3N3vC+i6CYtoIzr+PYjpB1Yluf+JknofJpI5zy/h4LSIteHgdn7pO5tJrdp3RJxFTKu47tKOX15JgnZy/Ldp7WuRUuLNH1hW3J92mot39I03b6a2npkzvoyvBYaNHh/QkC/d91PpZe+dHthawxzx+Fkkd8kVgdQ2l6AGF/G5Yse8TGpNZTfvw5bRlKnW8Ibh8BWGat/NGsnP+N4zV+BpQ0suTJuqqVuOm65jYObMdSGgcM3x/iY3cSEBeqkFZ8CY93mesquNOUyyTRWHG1BgUVz3c1+4tKUoanc1VoTFfCTnz0IAHjThzM2jy295WI6nsiiwm4W/7D/j1erqsXo++YHAABjrnRpqhtZqRhN+4iiZ5VS4uoPUIzKJa/6t4bXp8B90seMxoiKzeun5pp409NA9/iG17ULoSxGsjaIof7BFlcnMEyxSbJsYinyzvXL7iPvj/imCyBeS3T8PL/uCPC6G//uKoh930IHe3KY/pUlRf75Ooi9lMWQ4ylU+wGhKc0ZUkodw6PZKiVQOFOxov3sg3TqlR8x5fCekMdBVDJrgdWHJI+d/9/elwdaUpT3/qr7nHvuPjuzMTAzDMOOIAoILkAQEUXgiYIxQtCn4K7RuMSNxETzYnxuiSYqRlGMxmA0iKhoVBBEREFkEZB1YFZmn3vn3nvu6Xp/VH1VX1fXObf7zNwLPr/fH/f07a6upbuquupbfh/9MlbMxuu+XL5tTwaI0/g0oD/icEaD1DnqacCqTzVRKyY1b3JlO3f/pTdDf+1ic+7AIFp2rcFUdMzxPOCZ1yphCyVyVLUObie8HjvefjAAYNaz3la+jSVRxrxGqQR6jaHSRNGfeo+hkhTLBqrTx+WcKsOPJI/mrj0Nqj0oqkuhgA15kyr/weXpojWZOp1SuOTYanSfpL5HY9gvLuiX+kuLkRBQm/jCmxyekxTqtPeY4zFyfqMFS80/D3pGrabfHNvnrHWrGEmd6Adbk/jAjfdUah8whbqeIbaojcYrWHZk5TqE5XQaCw/vanupDcqZ5ZUy42sMQj++pmoFpoTLf2ABtkxU/+BE61yrZi7h6rDP4dC/M5HAsfqMoKC9oABvDOP6DSafkyrcFotdEqWoRjCO8jf449nxmERcWJAnFaAU3myj8NzZ/3dvr/gemWDEleHmV98mrclElNFu6yB92oOB/U2UeB06nusMOqM5mkgu+NhmZinhhs7lX8fZB1YXLEQpmMM0EYrwWP9OZu+HbKi64zqQn8vamubWenHNHWbBe0HZjKnPNUf9+wyfoc68eaHyC2M319O648Uf9xsYMoNqY5ZqM/ZkObHvYUCao058AzDahUkamZwec2HxG97wxmeu3Zr1p/DbBUBf9QFz+eUhWY3K5WdTR0MfeCFy8Zl87PnGzP5tv52eqPGCPYM4jQsEAoFAIBAIBDOKPy4Nh9KdeAIFAoFAIBAIBALBXoV+vLqVwt6Emt9dMM1uIU7jAoFAIBAIBAKBYNogJlUCgUAgEAgEAsFM4o/MaVw0HAKBQCAQCAQCgWDaIBoOgUAgEAgEAoFgRiEaDoFAIBAIBAKBQCDYK5ANh0AgEAgEAoFAIJg2iEmVQCAQCAQCgUAwkxCncYFAIBAIBAKBQCDYO5gRDYce2QgAUMrvb7TOiudaE+ZApfZXIRa+HjseM4f988zvxIgtZxOw5X5z66rnmnNb7oeae4A53mXqgd9+E+r415jjrGV+W+Mmzdh2V36y9GnlGzn6eKlk1O58u/KxF5VK/LOguIyJfSatCSCzedQa5ndsG/RNnzfHzTHzu/xpUMuON1nYZ4KmeU5q+UmANu1WQ0tK1RuAa+PYh56Fnku+ZO5vDJky/uejJs1xF0DNWWHOrb/NpFl8NDC2w1y36dGagOqdFS1Gaw1ld/4Ul1IxSYB/hgA9RxVICrTOfN/qn1+qedntV5i8lp0AveX35uTG35lzh55t/h/fAfTNtUVTmQruHe7ean4nx6G3PWiu7neiLWDS/E6MAL2z84XrDMq+43ibbf7UL7JJ6Ed/AQBIDn9pqfYBgN65zvxufwSqb445mfp+BAD6oeuBX19tzj33Tea3fwHUwAJ7/aemfqueB2RNc1zrbV8ma49rB71DlYCeHb0vbcekSlLoSdOf1fC+5dpHc0NaB1rNfFk0Xpq7ob90kcn3gi/YcyNAYxgAcNnJpv++6vp15n0DbftqofxgPkGtL2grgN1bzL/986DtM1dzV5XKH7DzHIp9fsr72owresZ6/W9MQnrm+x4P/fPPmOOnm+elH74eauWf2PbYvB79BdSio8y5xHxS9K8uN0mediHcGJ21rHIboVv+e0Cg+tX7oMd3mnP1PlunxJVHcxy0xvpLDgQALPrX+1k6C/uuqA9rrf07o/GW1l09/DO0aSZGcd+frgQArL6Gz00d2rf9EduGfmiaF4J2xt6vzlqu7q4eI5v82LT9X9/zfZPmKedBj242x/S9TGqmPbmMtXuuLn/7TLO1v0ay5KnmWsm5lOpF9SzGF+b/q/y5rOW/dzyVrVf2+L22HfZ9DC3G5rceAQCY+/K3m2vzV0Dt/6x8BlnLz9nU1hr1GwXVM2COS7bR9dGd6/zzdN9pM/foWy6HOvp8c47m2d5hoDkKABj9++eZIt9/g1vHuHmZymHjNjxvyjJ9VNUaxeds+5ZK69D226TmHViqfaaC5pufbXkAamAfm6cdE/Te0h7oTXebM/bbj95Z/hlQX593IPTXXwsASM7/12pttN8ZJL7fxuay7LKXmWRvvLZ8G59QiIZDIBAIBAKBQCAQCPYKlC6KHvY6YhqOaDqSDlqoJC1oBJRSuOmFZqd93FXrc9dsLsE57Y9J4tWa8JINknRYCYVKavi3Zy8CAFx0y+QULWOl7tpAlfZSMWovST2ylq8fl3pSOpLs1hoYvfRYAEDfB24y16ykFUndSRFzeaU9tm1Nn1etJ19Je031DGDrW4yUY85nt5ZvI0l0dm0ASApjJTX0bPXoFi8VXni4uTayCRhcaI+tJqhn0EvKD3yebxtgIhVuuQAAIABJREFUpBnumUz69rj+Y9udNvz7q/fbNk64NPo+I+VLjnlVufZtfcjWrd+fJAkqaWZ0BpXkFYO5IeTeq8J955p+tPrKjfn0rQko+75i2oyOdWRlXX2ayf+FN7TaJS/ebyVR6J1dkGg6tMa9NM5pLrTXtjV3m996L0beb/rp4N/dZutn28+llDSuk9RrMXLTTjBmaZxCYfObVgMA5n9he7n20TjctR560kqu5yw31zYbrZUa3AdokMaCJGhsrLN+OP4RoyltvO17tu22v/fN8c+Bxp5Shb4BhG3NS8i3vuUgAMDcz1UYh1ZLpZiUOuxHU0oM6RmrFA+9fCkAYPnlD7p2mDSZn2tYG6hcrml079U+c4zZ91Xvw4/OMtqbU68r3087aYxjGrO22gCY78jPXmC+Gc+8emMhXbfg7/X/HLsYAPCuO0t+M0hyvObnSJY9o22+VbVYX7Lfrgt+utben/j35OamtHK+DhU0HNmjN5vieoeBnkFzcpI0RrZfpT1+/pmwYyute01xv9UmJ3Xo337d5Le/1Rg7LbFyEnA3NuYs9/m69ms3J+ktD5h0c1f6/G//mjk84c3lGmjfoXm+wfMkDadKc+OUkNfSm/f0w9PNN/LU720oV35FfOx40zfeenv5dY1r4/guqMZg7pK2c6Gq9xduA+Lz3q1nmXF49Lf33jjk6KqNTyCc9ckTBLL+mSnMkNN4xcnNfvy1SqKblG1N2nzENjChaQ37CINMtVK4hYZdUPEJePNYteraDHw+1nQjNBvTieq8CEi9GvkHP3sYAHA2tdFOru0WEg6RBY9D6jcgl91oFtJvb5+6CFqk9A4DtLCgidN+SFRjCJi9X74uA/v4RRx9QCZ2Qf/4MnPP6jPyeWkNIMvnoRKfh1ssqcL705Q+a6L1vU+bLEpuOFCnvJiZn91ouHORPpczGWDXH9hl6rQ6TM/eQ9UPP09/744uFg3WHEwlqdtkF/ppxsyjWHtUkn/GSin85GbzgX+hq1+xr+ukcz3DcUzjVCmFL99izAzeWrJ5rg8NLoKyJgqunCGzKETu+VN9I+NGJdhy30MAgMW0eaINR+8sf9zBnMxWijI0/zkzgRZ+/4BZmB87RQ45RM1NVMf/w/Ma3iznF+vMeFpBJmdUa77wTVThHGLmjPRsyXxHJbhxo7l+arv2TBMUe06T0yBW45u7yXKWVA5uM7T0WLQ+fRYAILnkm+YimcEE76Njfrbf00ZDf/Yl5sKr/8Nvpu276cYUz5k9VrhP9RoTRfQM+g0rCcFobCY1b0rGN0Zsc2RPYvyqTwEAet9t28YEE4BtGwm20oZvNxf8kbnY0CKbrsflsfubHwMADJTccHQ0j2b5RjfF7nvhz4V9NCYL7rR+iJrgMvTugT0L32y4vKeY92J13TzevgdVFb7FsKvZ9a2CGYCYVAkEAoFAIBAIBIJpw8xoOEjdmdS95ILg1PfaSyRIeq4SaCel8tK0I5dZ577ABAtKBVJye0hlUPrmCKCtSRU5OZIDmkrwomUVxVUsb52gIJ3hKm2nZeFOc2QGRBJ8leLAIV24l67lzFYAQCUFh+N83exzZc9y1fAeiPxqfUBKmg2rDiKTI5L68vpxiSyd652DtTfdAgBY9tq8xFYDRfOpnBSc9ZlQ40DS+lovfn/9rQCAQ8q1Chg1zrx6YIGXnlnVvmamBKGzWvycxqrAz9hLtiejavYy4CYxT53XRT+l99Uz4M1fbP/TpBrnY5SZJZBUnPfF5QP5fpTrrypv+mfMO3TbczHJ1uqq/ZT6eq0XIKkctbOHmQRQm5kk0jeCTHFq+M0a87wW09gcXurLiZiXxJwcC2Zk2o+L27eaa5U0HNYJGAMLcn0K4OZOXiodlUvb96ShcNziYJ4is0xoJ6nlzv7uFdKcpxI/xp0jq33maQ9OX9pFPy0L0oaz+ZvANXeNLkVrncyb+LXhwHp1SpCEvzGE5DXGVEg/fo8ph0g3mqNeikxjJEkLYwhQjkBC7WuIQtQrv2JSbLgdaoGdAelZQTOTziIBB2uhK1uv+blJccjZ5dvYYs7wiX0BobkxM+/S3JR1IBhb2SS2PvAoAGBxqP3IJvPjHjBrATe2fTvcesPNdX5Mrr/H5F/ayMQ5a/fG1yJUti1D5+YYeu70HIDZ9VDFwTQ4EVPUzpqAcN5UmNdb/Zvv2qWUf1bO4Z4sObQf76QpTmpw/ckRL/Rg1bzgW8/nS1qLOeWPiq51CqQJTNN0yqJpnGsEewyJwyEQCAQCgUAgEMwg9sR87A8RM7PhYDbfIfWft31XnlrO2V6qoh2mqmH3znF/3Ry4/N0ePvce7T9kS64SvxMn6SfZy6YJHhmJ2953biPTxDgphJeUAkbm4HblRG2rEv9MmHRuxPk85dvI6VORxhxwUUjHZCLmr1LY0eyio/N3R8+eJEqW6tPQRwaS8OZuRltp65Ck6B1sY6Oskrw9L7WHJCpc0k50xuTkzJ5hX39FLYKVqpl+at9ZSF/LqxmzmSWpapJi7S5zblWQXkds8KtCJTX8erNp60lVbrSSPc3tpJ3DvR0Lzd0AUUQ6gZEGdzQ2eWjMndfnjs0B08TR2CVputY5/wlfD9Lmuca5a/v3V5TKcSd3ys72Pb1jrW1vn6OezEnYmNSVrh2yT0jQYCXNaY+jmQTTDFFZvJ06mMPcfKA1FvV1oWmMaTP5M6b/A0lgri6WjheNWVh6zKHmmCSXuXfJHMgJ9IxTGmtsLpk0jvR6s6EuVXNW4LCj96vWvimQ+0jzb0sxpTsa71Lw2WlBwK9tHa+Yr/VvMGPCzF3JPuY93H+e8S9Y+bW1zHcsVi47aTUbDnauTBYfjewnHzGpn209odg83pnIRfkfovauAqKSTmp+7lPUd6zPBZ8L+feFaXTMvwmue8D0z/NCf6u0h5EbeK1WJ2f5mM/d99eY39eVbB6nAlfBnE4O1Uh74iQjpAEl8hoArWAq4Pd14vbp3Ed9+zbsrv7Nz78f+2u/DTnfvyTuOA7Ar60A3L/ZfEOWu/p5at1o+VT/tH0/5W28dp2pbxe9VTADEA2HQCAQCAQCgUAwoxANxzSgnJRIE2Vei2kdItL/tEaSuxhjUF7qr5QqBuOaHGdSOm5zaE911QdCSk9fZ24nW5BUcEkzvKR3Ti/5nQT0bjmqYJaNy5bbWgeaBpb/gkb3PhzmmQZ1IMn5ry6HOu417SqXe85zDzs8SMZYP6hfkD356GZoF/wtIoEt2KwCi59y0NSN4aDAd71z/DPbZamXieEomyywc3ApvfNX0i0cfcQ+xXQAML7LsV/x91uGPtHZzWdNnH9EeRYbnyHTUjBGLwDerrzVhJ7YZZJRQLGRTVDks1D373DR6WeadAH7EXTLt40oUqnNrB2A9tdJm8TG9aHnPL9qA219kuJYs+9Qb1/jn0OPrRMPMsp8OPoGiNnHjmXez6i+9I5GH/dsV4w6OPQz4pLBxd1oOKxtfG5uC32e2DyUsx2P2OnXBq0kmqSMFDw0Sf04pHm4ORqlOy7M0zRGshZ65panUq0MS8GtyT9HJeyb4n0cFk1FJBZA07xDGvB6X1GKzb4xq2d1O5/qwnds5RWGoVB/9iVQF19ZMpu8NJ/PJepZbzEHti36q5dA/fmX22cVZVWqQGlMsHMIar1FDSh9i8Z2MMpxH6gxF3DRpj/loIE2FWZzpA1uqycyP7adzxQbL6xuhBeuqrYc6sgi5YJ8bmX+f+wZZEXty5JZncrPa1iRNV3d8zTReZ8HzhB4SBd9NOfDEX5rqT+ldRP8EPDsdGnDpyNfl3o/nn5kfC7Q3IeXa4SoH1BYgMaw7xMh7W6S4jVPrepMJZhJzMyGo0Bx12ZSow5qP+R5B0Dfue7clFfLxZDLN1TbDSwoUnGy/H+12aQ/uUP+BeTMTYJFIh8YbkIqmgXwuj82Ys1xInSdMWcpf87n6c65SdubVG3qQE/XFm4j2Fv4+DoTnOMv9vVMrGNiUmemZH6BfcN3jCPis17rF4keNj2pnIeXomjWUi/0I22joGqd4OvfuQsAcMEHSrbPRprmpn/aOgm7uiU1aFqYW1WxSmp+IZpQPJcUd95puMbJ2MHlQXSRQC7ibywKd6Gf2g+ISuu45h7zPkqS/hqwyd9Twdp3SR/Gsa3AOEXK/ZI5d8gZLNYLc5ZcmnfJ9wtzBVWzpkP9tpxWk8VVsfmv+blzdPWbTBY5d79Dq7Qu97zaUcWq2fshu/+H5njFSeaXmy+wPjp71YpoHrljmtcGFyL72SfN8QoTW0EtPBIqDUwqmInVAQe0N9lrBxd93VTCnHQCAEYjHX7AdeY3CzSukjp23nsHAGCY2mHpodGaYJtS7pRbXNCHZbnI8PU+bLnbOEMvrNJGFkOjY7pHbjDpDrCku3y8MDrY1ee8oELpcI75+ldmYa5OfD206suncfnX8dKXBCZNU8DPWxGhGZmgvOYbyK79W3PuT95lfiNmNkopZ8KWLChSZLjNB/1e8EUXr0Y//DOT5pCzXD+ORqPfaOZSNbiofCMZZXlhcU5l9Q5D0+ac2h2ld1W4/j4z7744dN4H22QyE1hPgsDmUzLRcmsM/9387v3mfV5Stn12DlNpj+uvbnNPJr59NYSmqFCpj/XEnsuabaZ8mnHyDuKBwCPlUcX9M+K05PyaUgnWjHZjRh0xyyRrtLofD9kthqRAnfyXVGEo+oay+XTwwMPixXCTegZNAoV1vzbpVrBVmWLCLYv5h1SIoi6YcYhJlUAgEAgEAoFAMJMQp/FpQJtgaQUQjV4aMRVhWpKuA9iQejDmgOSkEDXUu8mfSYd0aLbAI/eGqtFcJHQSHQCzOvg7x8yyXJlM66Ej+dJBT9KFCUAHB0MvnffUg9HASEzL1R/0vlgEYXbRvyOid9R15kBr0zEn0uGQZnAqOBORmi+fIo1bB0itdZuAR6Tq9kErB9uMLp1NRiNSFyTm8bttPYBZVdsHeOrlDnkjbUDZQaYPMlHgVf98RoxgNRdZC9u+/nEAwNwTXt8hXyJtqDHaRDteFh3Fgjvac6QZ0RnWfMFoDPZ/wUdKNK4CrOZKr7nJ/L/sGV6azp5/a3QkvLMjFGn4yHn6/muBA41ZGOXP3/3OrSZdFT2H4qZp7tlG+kyg2QQ3B2PSxEcfMM7vh4UBUpO6v5ePffe+2DXH8kkalF537ZYHjQNtFR3DVJoNl27f41hdg+8K6+u/+c+rAQBHv6JkBQaNOaQ60fbrWm+RFpe9xyu+bmhpLygZSbUcO42COuWdpqxvv8OcOvPD0YCAygb0LFd44kxf1KrTTP4/+yQUjeFY/ta0shvEHZ6ZdoxIFMi5/TlvY1TxnozltKfl68DzjdFtIzintz8CzLIEBo5O2WsEzz+WaZ7LgPWvjrT0Uc1RcUzunIxrZKc6Fy0ytHgAsKUisUG78qLpaJww7bwDGyd3XP0TAMCRry5ZAeqny5/D8vWWGgCbrwB84yqjrX35+0rmL5hRiIZDIBAIBAKBQCCYUYiG44kDUcnZX90YzAfysXjaYRXtnkNHtXW3AQsONqcidGxnHdCFioMHoAol+0Q3mdR8bHeyW017vDSAtXH+7A7OT9HgiUHH1S34wGtFHNqFA5lqQ11nqlBGOp+XmNy30xw/rYwkR6VwvgYd6sGfw+1bzHHpUFUsXx+Myjij8qBtBZrDVtNL4lIvaW1Lxcl9mZhTXsyeW4dBIbkfzlgXk1VEs4JQO5ZNOhIANatoa+ykckph7cPG4TuUr0Z9qJivEZWp++Y4zVJRmqZw46NmXO3fsVHlkLOJtsHVYCW3k584A7W3fK9Qj97z24vKogQQ9HxJS7vP4Rj9a2Pf33+pDc7G3vPS15V1MGKI+JFoXZT6FSTy7Jyzq07qjII7hC6SATCnbJIsKqW8bxFdm7SaoXo/tnThL9YpGGQONGat9tMIjykIodeK93SY0uNlkebIjrtW09NZu/y9Vnx2T/ckHO2Qs2s/+6OmyCvfBHXup4r1jQSh7JgvfRtIW/ast+KxV5pRtu+/PVLMfyBPgFEFUfv/iARcHftKmyhj86hPd8NtmwAApwf5as36KXeXJK0D+W4NLgIPuhnmcfWtxl/g5V20KzzHaXdjvmSuzzFN3kQXfvnt6hG7trvtOO+uPEeskNS8D6cLmLsFus9qJ9g3Z3iwvYY9R7PryiKfFPJD08CE+eboxqxcfYAutf6CGcOTa8NBTkjUaXMOih6PP2o63KzClXbIL6jU0GK/IIhwmpdxSi/AMWx4lZ+/Rs5rEwDxkLu2ZjnnQ0LfEKm1meo5TE/5JmlhQZpb+LOox5Sk08BvB7c4zib9AptYXMiEZPdWr96nSWhixKizAaghG6l5YAHO+5fLzD3hwo0zc1H+nCWF2t0z4Dduro12AdIYwnu/8pmqDTQ/YB9Dytf1l+KGSqV158jrYkrUejuY5jGVesQpL5+3dbwPItlqJBjqhpDDRRv2f11daHOTNQGK00L9NGt6Ninqp72zsOo4s3EvOjDCm6i5hRnnxvebFmeCR874jumogTNPXNJFI9uB1W3kcfPbb/pq7ZWXeUEHObsmKTZ/8s0AgAX//MKps1fKt4s2UQML0P/ua825ILq5Ugnu+NBfAQCOPOld5ZsRRv9mbcu/h9hCJyDwaI5i/yWWVz/HSANAa9bvfF/xTF3evM/Vwi1C7GJvw+04bkH1IBidFlG5DQLFQImQBfBxvNFuzmMuqx0Xjh1NH308lYd3Ta+k0pX54k9i99+eAADoe++N/noJEzS+QIwtgpd+4WEAwMRHjQN+z9t+WCn/YqXtYjGbZO/H9hmaL7nwyBEUtKB/8TlzTGZeKsFpX7g6f6/LU/lzdg7RfbOLgr/mCJCasT35mXMBALU3/rdNkuJlV1xVqXkdTcXoWeuWIz7JbUZornGO1woHDMUXy7FxbvKLCKiYYMwk8u9txVAheWnwTZ0Djy/E5ngAhiHMmiNra2+pemejp9FpyZnfrOVMq3k8GmIfo/mUNpRpD+btAfumYPrx5NpwCAQCgUAgEAgE/79DnManASR9qPcVpAK5Xb9zKOVOuXlnXKQ92B3YquTp41ThnL9od8nEFW0S2vwtTWfa051q01HesQ7EJfGApZkM1P1JzUuCWQyRLeuNdIDV1MOZEZC0sl2njTuva63x0BbT3uXtWxQBORX3+GOS0JAJTmO46MBZ6wVI60F9IZvEHe80KvQj/mtDPn9O6cspPQMpmVJJUdprNS3QLXz3zYbg8IU3lPRQIwl/zmky5qQX6cNBpFtAo1lCsFvWKa/4jnV3jv85etNgbJG0OKnlpVeUnjQbo1tcdlsfMJqrxaGpVNZydJxas2mGc+0DRkpMY4ee+4SP0rvuHhMdvCzZYSdTHKeez1o+CjKZbA4v8XV/8Mcm/cpTkCTt30/BZCnLmMkGe27aO1ADgN7wW/PvoiO7i/kTkwC6CPf0rFVRY8HB3vnozvF8Oi4lVaQJZdLUgLzB9H2S+pO2zFKGLjwCW8aqNQ9g2q5ab0QLzTUr4RjQXuPCNFYr5nkigjwUy4M/p2BsJHVWrr3GzGJPXlhNi1OMn8JqxE1WnGTXp+t9j6GyzTbdbS7NPwj6tivM8VFkEFQ0u/NzVItppfh7M8f1vzAauWzdrebSwiOgbzba6OSU91Rqp6970VwyrKcz71IJ1LGW7JsooHsG8KNXGdqBU7+3ARxaa/+Ntb/RODzM8b/2RtJmeAn6FX9qYgq94pflbI86RbfX4Tc6RD1PsayUwlobCfzIjuV0WN+wtNyRms6vG41XpRNi86kr230nVdGKIoGPcca087tH4p7rnSKp5/JIakCgYeV9qK0Zs+BJAdFwCAQCgUAgEAgEMwrRcExDKV5j0dZWXSlvOx2jfmMUdL/dbq5TnOop6eMoAE0He3l+7TdbzfE50Zq2AUnHJya8/SZJbKj9E7sA60jFJVbaRulUsy1l38QIVp1lSSRJikbg0ncWVVQ55/oIPR9FcCepy9g2PPOMY6u0Ll/nkU3MSdpqGLY9ZJIM7+ujjnONSGCLqW//Gg5/3Rvy7XDIGKOh1wSF2iudizRuQc9h9xac8aaLqrUv1v8iFJFRqLxDaafgijHb2ymzj9xz17bq+eg1JtiiWnESk2TntXzGJ4gFkAM9E9u3iCJzbDt6+rwde1CSl+I6qX/dSeB1zFGcbOIbNmp01kJvf0VfoxzdZUxyDUCpgFrWVpO0BDa4lM5auO4uo2k8u2AbXQxSmQss5uiZE3fZ5b/wCJtVE1c+bPKJh8NqA9IS9gwwTWDkOZFtcyyAqvMrG8fgHPIn07n7jN9S4AzONY3sN6c9Arx2Ydd67DNU3f5f9bSJKs3akHN+9nf6Q5uHbjWxa5fVuETt3uHzC/PJEVQE/YjyH9uOa9ebfA9HSVDQOBaFu+BXEQmExuuhbJC/7Nt/CXXau20ezG/CZFLUyEaJIzwofbL4aABA6zNnI7nw81O1qAhWTkHy7pTVmR8zXCrvggB6rf8YTSesbb7SjPACdvwV3nX7AIQAYLvI3kUbp/GwL/H2xRD2kak047Hro104jccc80MHfaVUkSpeeQp0/swf2maODyhR39x5PhbonUX8in5qx+FJbVskeCLxhGk4opOsc9otfig5OjGOlCs767joSwpq+hJwpkV9aLtr7ZvH2svSWMYcN5Abw/jO5d8FAJz5Z/25LHLPyw3C4ocpl643zy+ue2fjU1+5BQDw1nfEqxoFLRIHPWOJe4/7RN0x/VFCkVLt/0f9GX5yholae/Jp7y/c6Z3FitHpY/mzROa3bx6+9JEvAgAuOq/kxzJ0qEWbfhpBzJSvNc3+awO1LpjGKLI2WwiHE3fUQTCXid2E9M3Bj+8wC/JzC6xifGqJONp3dEIlJ8A6Lvut6XOXdkgdrRv7qBei0euiSYtiLGHkXKvSOrZOBB/4DsxvvCxetgoW/rRgUmkdWTcSrr45JRO2F8RoZnryo7t2AQDOD/sBiv0gt3mJxL5wi8dBE1dcDS7C5SbQePl3OAVydegQkZy/x20dFpOlHdTDa9SHGkOo+klSJYRwpoy8aWNs85ic9RFM/pMhNai94Ts2eblPe8wcOZzL0td+C7s/+AwAQN+H7yuVr6mYj6HBSrSn+Bwbi9dUfKIkwCmwBGrtz0UisRfN7ng6f21DRda/TuZGHU27I+eVUh2/F6VNbzugg3XolOjURiA+n8bQ7Sex7He4twtuA8HMQUyqBAKBQCAQCASCmYQ4je99dC1BikhfAHSMBF6av71D+q4kATzCppNKBdoM3co7QVN60uxo78D3vFONutzFJiEKuKTmJUcZqZEngIbVYnAzFpIShaYSKsX/PqmLyLGWola3JrwKlVP+AkZy5JwUreYlmyy2uzmK57zGODhmWx8y1RpabAvSPsI3N2mgY2pXUssREtjKmd+Rjbjw4hJUprn2mbw0Vw/vNE7LutdKldN6NNJvSEeoswzPOmq+PZWnCsXurTkHSfotY2rFzZReelAX4hxroqdrvd6EyuXpKWs1OfzVyLlR+37Usv1ApTjtmLn2lsBUR2e+H3CHP+eYzLRJIfmBiySvcfExJU3aCKRF2LUBepdxLlULjaGL3mHeJRqzvJMptY45Njpn9+Yo/vxNhj4z22ac47H9YZNmvxM9tW6fjQukdZu+kZfgOrOy5ije96qnVWufqazNjvWXgPCgnRbXvycyS5zE/gMxSl2Y91Cg4NUu3/ic7c08qV6L+7pX9cXakdNqhNe45NhFf87Q12GodCYamFrTB52hW19VveMxKBv1Pl5E+080rxtpNi59qplDP/Crx2yaGBV1UtBmTJV/3/t+3qEVbeCIXrhZF+Xp/y+Y46nE9WfSBGmtceG/GJrzbOOd5hpFDa/1Qo9sNOnu+Ja59pSXMrNQp8r14+PXXzaXjrnQ1fX9X/xYpebFzI0IedrfqZ+x1hqzO1iPVl3X7G3osW1QvbNzddBEk94YjmsYC/0uxX6D+blmKs0JoWy751T8XAhmFqLhEAgEAoFAIBAIZhSi4ZhhMIrDgvQzLvXtikoyV6R2Ds2xnXNXGg4WhMjTgYb2nCyadMyRk0kOf3ndXQCAE9+Rp53NOdk5CUIfi6rqJQ2u3AIFo8a//8xEbX1NlTZqklj1OSm3GzC7LVVq31xG+RcE7TGVMr/NUaBhqXQpSi4L+KetZkfVWaRrp1Vh3nWcmo+XpTOgkacenBI1T1/qnt3wviVvDgMUJbj9TvOMT3QOxLaO/VGy4865h1oqpPjmfeY5vLVKRnUKQpU4ogQkQZCorIlwIsw5PysfyOrm35r3fqpzCo1EpHZ9UqPwnlC0v+e+Ad++y0gKLynbPupDfXN9oELq/1ZCFw8SxyVtfv5pPnAHAKDH+i3pNTf5PGt5utuy3w4u+dt5/70AgNnlbqVaF08VJOEqKjEstjdxdJyhMyiYI3rhWruaucjAlEcftsSZMEui+mRcrGs5iuquy4GqTKXuxvPQEmR3fxsAkBxylrlGPj4l/TA4PnDLGgDAbWcb/7ijvrXBE48QdfmMScmL/plurDgNQMPPO/RNSXvcmM1pB+8zgQ7V6jPMKdL+K+Vf9v6WDKXW568TRbXWQN2O2eUn2HP2xaV14N4bzPHxbyjVOk477d6n05R664ao5D4XgBWASlzQuoKmUeuCtUQ+unpMW1AMsHrA8B44FTa8H6ir3xT9yc0FzHNjvX0lq6epDz7eBQW3YOawh+7XAoFAIBAIBAKBQNAeM6LhKNj1AkW7bebLAE7pF1ADKpXgiEVWg5AFYiWlUPCHoLwBL81oTTipis/fpz9tSRfiMEbt6s1VA01E1CYahi4XcHb9WgNHHDzH15XnX+v1PgEkuUrSgk06oH0ZY9vMLwU7A3D6ym5evX1GPPAfvSvLSoOsVRSDcmkGtad/HnZc+x8AgFnP/xtbZU/P6vxJKlaIAAAcgUlEQVQ5OjGNcIk5Xbd+JhhciEf/+78AAPtdUKJpgKH7BYCBBf4Z0zmqT1vWrEA715rAYQfS8w4kV5PjcQreKmiO4pwDupAXtCjAZR0Yt/1uwvrLkOYlY/4/nMbWaSBte5Dg0H0DOkTt/UF0ROru01mpXJJEJHRe63nq0opSOU7B62imbTup/2vtAkRq8lHhdK9OItyHtb8xGojlqdGkKSuFRtb0fjjMHr0w1+WeQdBH0x7cfe9WAMAzqrTRahN1/3zfXvKvYppQ50/UKSgpgGcdZOld6RzVT2fQQYAy6Ay6Ffj8cBrlkGIZCucduJelmbl22DZGmND4e+mcXScfQy9hL9jpM7+EvqrTKUnf631QK08BAGRrf21ym7fa5K+UowfmYyRGQe3Gsk1/1FduN1fuvBJq1anmmg2mGMs3DuaXZe912ugyYP288A2OlZsSFa4K6IgBtCagH/2duUzfHBZQV4/vMMfcb6N/br6sbBLUFxRRe1PfaI6heb8JyFnaDcBRZNdz48mV79IEWkU2NnnA4H5XdbJMKK5D8tqPDn6vBY04OvqItAPPT5OnEgVmJX+YrOW/67Se4sEeiSK73o9ls+MDRWeTuW9rsfyi31powaGSFMfNn2ZqyL0NcRrf+/BqdlVQA+YHHm0+mHlKYfEKbNxmOveyiPOi+wQw/m6v+ibThzRigqDdfRSHo5I7JzOpChfHvv3FxYdSiVskcPOxtQ+ahchwWE+tvcmWi1oeGWR8AZXmOc0B4JePmQXBflO1i6PeXzxHZhc7jJMiBhYUB1HWKsTtgEpRGwg2SY5zu8c75NJHRWfFdmjtF4c9tm7MxKq3P/hoTQE1tIi1y24WA5OqmHmKqUrwzut9uPM+Y9LzzNCxlUW8LV23UH3eGMJXfmcm4fdWych9yDVQs8d1S3NMi8TxHWxxzj56bhHtY41EamrT1JEzfaP7g4WEzm3UiwvEsYmKHxDqIzpzHzBlaWSzzff5/ykeDt84k+kbLcayFuYsspsKt4C2z7/eD019npkEuvgRqZ9/ws2Hsot4nbUwp1oXNXDjkD0bMkdxhAfajc1c33Hv2G4yG0Oo9wTzCI3VVhadW5RzTI/RY1I8IloAD6JW756rslPEePNPewd2nq5biupORA68zC1tYu60vZcJiHSPiTuTLHkqACD70YdNmlPeFS2r2FYFHW5+7WJbHfZiPHqRmeX3/fzvzbW0aCrXppb2J4FuDHdI1waOepkL/6wAo16k+eXvUqOVv15rYMOtZhO1ONyM1PqghpeYY4o5Vev1i2DaoNR6/bxux65aaDclSmHNr0zk9lUlm6eY8NCbhdr8ScjXM1R0qFaJF5Yyc9sdzTBZjDqYqltcePNjWuvwa/ftNMfPL9O4SH4uLxsnydFOJymQ2DktRvTD+vra7eZdrAjLSWrRza+nI28/h/DndMNGk/5POjVK8IThSeDDIRAIBAKBQCAQ/DFBNBzThql24rplJREkkUhSL/VmO/pZjakp/Xg5BbVcc9Q7NpOEnqsee/ZELacLx14L6p3VvelXxihvfaCklccdautqvaBIGq01dKiW1i3ojDvmwpjOkGbFUgaq1EqBkhqe+9S51ZvGnfqDiMuazHI23wO1/Dm5NmJ8p5e8Ok1Hiv7XXVZoh8uT+seu9ebc+tuhDn5Rvo3ZpDcXSwKzkaSO+e/5aqXmae7ISHD0vKRVqhUiABvJse2zVrWv6/047lQTqZc7F5o0u6DJeZFMK9KGk3zn8w7MDJlZxJtOKRsALpdj5JQtY9Sa6my8w5lXqfkH23J3Ma2HNWNJG1h03qvzeTFiAvdeSQuVMtMD9wtG6cqk8zCSq0Nf9cpKreMSMa3z0mllA2xictxL4OldjmwCLD0pV9Hv2mrqPjtmCulonD0Kc03W9O23z89pBpIU91qp48HVGkkH4FLo3LUsc5rinOCQNKYkJZ6cQM/QUL7OuT4XaJ10VnBqNf+Hkkj7/+hm9A8/8VyV0x2Es6a6KyBGqapOuNgcTIw4aTJPDxY40t1D8yWZMTHN+NIPGtPVyX8+GwCQvvzjUPMPitQlb1LH56PuTD87PBNyCk+YtN59E8HmRRozGRb94/X5fFlQREd5/eNPmd9T/wJqwGpu6bvSbDkNC676W/P7qn93Za/87M0V2+cRpQUHgIkRp33ymkb2/XTBjhPM6Y9r07Rmc6kbhllRcxKrB9MMLOvfO4Mg1ESY+oXENCwdzadpHY0Oys4y4RNi6bmmdW5vIZngSQRxGhcIBAKBQCAQCATThieBSVXEkTELAqUF6eqNTtWO7ZIDZ00dcWxm983qwrmKSwBdeSRdYPaXnCLXnojUQaM1ZqWiREVLjLE9/b4d/D4n0WISDtLe0DXm2Dmxe6J808LSkpQ55Vtt05CVDk+OsXaTTXi9QF+rlIK++Qpz6qyj83VXqafcIwnQwALWNiY5IUmt03AwSe9P/tUcH1QyAGBg8w4AmnxDCgGrPJRSnmLWUaUqbLrDODkuDuzMdc+Af++c9jeKQIJNz1FrT0k7dcuK+QFFPybuKE4+DqTBGt/hbJZhfSKUUtDz9w/y534Fto81R+z/Ne+YziVxpG0I2qi1BhaE1r57AOqXzdG8XTfAHMCR07DOXtJeE8j9xAAYGl5yVKUxOjnGaKJJ+0D00gmOXNjFFEyauFqfnytDZ9Fsko0Fdm/ofKk1kl7S9trnM8Gl5KE/FnfuZNoPl8yeo76iMzQGI75fe4Cc78joZnOuz9vih4HkVJK2dVZtW4bTWHpfl5CmljQCSiU4cm530uOoVNfSN+vH74FacEgxfdrhA0V+S7yuS58OAKi97psAgK+dthwv+/GG3G05LVWtKCZ2z7x9yUW4cRR59nSuNeHnNO7PEfhpKJVAX/dP5p8Xftim8VptNWe5qedzXmvO3XkVcJwlfaf+WuvxPh7Hn5uvRwbg+39vji/6RqnmFbQJti6mXZb2ducjSBYfZdMzzaHVVKPXW1jse3RezxklCqD6N0fdXJobDy6Yqy1f0btUOOmILvxwXGUy7wNLY9yRa/R6chVOPU7vkOqU1nHQoQs7FFGk73f+b876oO6/SbS+Ydqccw7uxinuCcQfl0XVTLFURZwLY9zUgVNlPg/PFnL7BjOYlkfK6uQ86BhX6v0dGEe841GVhVxH56bIuZw6tMAwpXDNdQ8BAF78gQhDf+T5FMAn+VnL8nUE8E83GpX1pVPn5OEmyVnFxZZV/at5q/MmcYDZJNEk4vjVE2y4+koAwOKzP2quKVZnctyluBH98wtR19E723983YfDM5w9cNV3AQCrLi7ZvoiDaFkufBcHhXGT3/GomSSXhCYT/N2HDpBhvq5fWWdLWlAohZs2Ve+nuXwpTxqL5HjZNwfh5k7NWcneIS2Ye/DAh98OAFj1H6+w+bJn6N4hbapa+Y8zYPoDOV1zkz0AUCn+59L3mzae8p7u2hj7f2AB9JhltQnMHQAw00bgxlvWAQBOa5Nn7tzgPtCjRHZgNzB9cwqOx5rlcfldZjy8v1yTbJ2HfbkUiZk2tdQPwZjkwExWAvYxVWvgtp8bZ/pn0DVyCs1axRgpii1GueNrEEvILVp7BvHNG81i5MIqbeyA3Dug8RYxL3GLlJ4BjIxOFq93LsX82MVwdB5gpoA7m9VWDp2jKttnOP9gaFrUUT0iJitKKehd1myW4sWwayoQOJ3/P+tw9/8y6Q6+wghF0Bj2mxVu+mPOAGTa6tidSqBMlPRao0i4oVSBnRIAfvqZLwEATn7RP+TOa63N5huAWni4+V10pI+ETe+p3ucd2Q98Ht1t65riK582rIYXXDR103JtiDAX0nvgz8u9u7THMMwFeV1zrXFaP/udQf5A4R2ijbmbEwpFviufuM4Idi6dol0xqJjwt5fH5rCb78JaJj9nPHKP2egeNmUZphyt8gIMVYuYZ7Ln//37zLuuZogrmCmISZVAIBAIBAKBQCCYNjwJTKoYppuTmHH/7/WsKW5DR4m4Lko7tQYCx0ylFBa4qKNZ7hrPo3itbeXsncyBbKALE4CQ5hRcGmZPpD3MFM5LgFw61vydW4zGYlFMwlWgSM28VCfljv5FqYitLH5vBVxlaQ6dRJRJj7hZhjtHEjNnvpD6d2ElkrrehxVtNNhaZ1G6zSJVdFJ8145goNadGYdTb/f4PJ3jom3j+E5HUqA3GypNNWsZYOk7uUnHype8JN4GMMkblTk54SP9UpnbHoaaZcmZUyKi9/TVp7zs9OptLAMbhyNnSkVgWpgDZleck8iUiiTTQ0scUYQKNTgAVnZj5cA0jYXYGZz22IEdB/z8WmeY39dG25s1PY0oj1ycowBvAxaHYN8unFWj4y6mFbBj1mlClY/rwk1yBwcqfupIi0fjLW24dvvyfT2WD3QfyryTtsOZIxJpAYpO5oZ4ZOpw7ny8H3z5HQCA8Y+ZuDKNd/4I1B5vOuNNAMkZvdJooD7Z4599tK1hTC5OhsDqvv/s8NsZeebcuZrMiiyRh2oMB3EiKCGA5igOnVXtHfKI8G3n0pwpUu7uIH2CuW3IaqaOKh4xvYpgYW8X47CkdYrTxBXMouBiP6HWwMBAVXt1MiWz39W0jmIvpPokWDnU/Th8YvDHZVMlGg6BQCAQCAQCgUAwbZihwH8ld3E2gmVII2dOesnFUMVae4kdOVIlxToxu/HBbpzGWQCfaCRYKiP0feD0oUyKsaDgt+cl/oVoxjorOo3HnPGZ1KEr6t+kvb+B12DUOkpjuGR/zClMYnUOIrECTDLMnG9du+05pl3ZFQRSmhIx+9AYklgHCWy+VYJ1I6ZdB7ZLG54N/DWi9yReQnf/zi6kIzkNXGiDTP44Q57KkCLIN4YKgRe1zrDtZ98HAMx9SdCGWJk1VpYNjKd6Z3vJppPOey3VIz+4FgCw/LXlm1gGLjLxZhNJHMtOcNJ0xeyfV5x7brWMyWnS+htNfPS56HnbD8w5+155/uedfVTFmiMvPQxIBXKRhguaRnab9v9s3p3vp278qojdNs/DBYBMPN13oY4K68eq99OYT0a0b0W0Da7+jN56x0hFHw6ShGf+vkL57D3ev8u098SS2XcO5OfPkzO8vtf4o6lVz3N145oW3Tu7cG8xP9tHoBzxQ+Mvzfjd8Y4jMPxhG+k8tMNP60BfFzTqEW1/R0n9utvMweKj2Vzv+9ichXmKYG4REFI1Ayj49+gHfwy1ynpjhXNevR9LF1YkN4iOj8CxOqqRYg76dLdKMNaa4ruAdu+1/XX+nKsOgXblRdORr4oLkKp8gFA2lluTRSuJzhkHwYsZ3b8LgMxe+a6KvlRPOKbbqudJhieXSRUxMo2ZKNtaJd6Mgw3QI1dFzCA65kuTlzVT2vJ750gdWzyeuW8Xajm2mSkslskEIq15NiMXmyCyiFcKBz3/JHNMZj6UV5L6e/0NcBNci7HWhJ255hdyZ55xRMmGsVIa4YQfSVPiQ0qgyKdHlnHWji7CYx8uX+Yd20y+5ZeMsfoGpm1ZqxiHo7nbf1zZB/WwA9rYy2STfhJmjGMhWYLW2kdwd1HQvVnOiQuq99OcKVe4EaUNbHPEOz0Tl71SwULX1GXrBsPiE1uOFDbeuZgD9jkNLHDjJWZuePcmc235FO2qAq011JB1kCdGoB/8DdTpf52rt1IKWF12CWn7iGPcMe2rn/MujP+DiXvb+66fFvJPj3hO9Qa0OMMcmRyQOaBdNKkEfEFmymW3MWfkZfvkpRs5My3X1/1GhkxSO1hv+PxVgsMqmqpUQo3GjB2nWcvPlW4eriGrLF/JCxAAXWTRceQJXdEalqsFLa4s0152xf+Gevnniwl7pp6bfZ7K19n+Dv/j3Vj36hUAgCWff7h4U4m5vwBmShOtAx3TXLiAWJp0ZK4BfmdtZJ8R5JGfZ9gxlUsb46VP9+MkozE04O775UOmz74I5RCNK+aKjzBYRTMhk8VqHTQWv2WqOk52R6TWoQ6epc3FaKIxMr4jSrhQNZ6L2yTTu8xazMxwdiHP7X9oG44/MohJlUAgEAgEAoFAIJg2PGEajqhqNQ0kIfU+xKTOj60x5hDzyhdGB+bvwIKI+YbHTY+bfdjqsvnnykBRa5JGzFgcVWgWibQM7H70QQBAP0k+asQ5zUywwBzVKF5ErmybzkaQ9tJPhR0Pmvzz5HydoblkNZSKk4kK4Gl+qS6T49DbTHlqlo3b0BjCOV/9oa1+oOvNWvk4Bq4tZFZky+4ZdDz52Y5HTf6zl7v833/1dyu0jsog5RBpEpgTIlDQbgBGM+Ei/NLzqPVi++NGEjM3lIwy6XMYnySXr1LArH1NNVwMGfvOkxoeG60uzcmb4wVO6vQespZ3qiYJcqvpYh446WjvHOz3DBtNPWbSwB2N3alApa4USO6hSRPoTKwaOPmkSqOwI3gdteWNVxQt/LiLilGaVYqtl70PADDvky+rWpjJYv9novHG/zSnnGM39SGFWz/xMQDAMWd+tHze1uTUvMtgPolWJfJOLCkAmqNIa21IKNJ6UfIaMV/JC5gZVS4AvfnejtGF9xhEl8nNhaiqVAetsaOieaX7PhVIKdjc4Ew6NLZ1H9aoWr3+9HPIrjF9Mnn+B135ZSTHnmK3N0IZnWDx58wcveZCM+cs+9KjLP8uZJM9nfqkl467/sRiw7i5huhlARz/zQfz7XB9uQG97RFzTHPY4EJPo06mPWt/BbXkGADAb15m4ps85co1ruwzr36gchNNddnz57E2TAMLWnilVIHuGFpjQRun7qraD3MP0z7YPOaWtBhuVwdef8C/B532MIpsO17qfZ563FquqL65GBhuHwq8k4O6//anQN3Srbfyg06lPThoWJzGn8wQDYdAIBAIBAKBQCCYNsxM4L8cfVxg28idHZ2DEJPm0waQBRtrtdnwTynpoWs9gwXJtQtIl9SQdrPpdFSMCbi9MwCvuWhxMRuX6hQbtHOdkcD2h74MOvP2tG6Hr3xZuainFOQrkNLpDFs3mCBAVTQcaFptQ2PI15mivVIgI+1tqHOSwH0Ot3UmTcAk1r7rBQCApV94yJ5r+fTUbtKW6BbT3niphxqwz5YiWLtgT03cerEJ7nTMD0tKPUgCXetlUreib0gU9Iydu1CGySZpNjrdm5calc4fGvVuxAVMQ+Keo9PKsIjYjuaZ+Q4lxtEUu7fZZArb7zNO1/ND3wyd5STMVOcc0QHVIwnocCm6c9rA+jvvB9CdD0dMsmiKbkGRzTE54A4v9VL5e75jzh18JvoX5iPjdqQwzVo5J2NqA3opaJeViN9vHOFxwHOx7+JioKwp4d5N3T/Hlu+7poLcl4x1FEdVbK81hjE2wuYRgGmmuDaKtVcH75A7z7qy7Jy/4BBs70L67yTAfP4LpeysjdrxbXtNqGb+bysWWFrZUOvG5t98MFPyibOVTxvunOvO9MxViucurkip2oHS3PdTFi2e06JbX6Ns413myrxVyDYYmlu1OCQhYPm779AECz7JnJ1tnfb9N+PDkd36ZXPp0LORbbwTAJBYX5JSiPhw5EgNgDzZAJ8HbFA5optXaR03nmN8TE68emM+LwBq9n75slVS0PqpFSe5MfiUbxhtiRuv9T58/dSVAIDzf1HNsTnmj+J8NSM+HCZQodds0H2bx6d2Gs/lEUGoEeDzVTfjMFYHHa5rANB4976mbNw2/bpnbLR9JTp+A2MkMu6af8Zj1V7dE48/Mqdx0XAIBAKBQCAQCASCacPM+HAwJgZP/ZkPKKVU4pgIokHR2I6Z6ECPDotpSy9oJVdJSCfI8/dS0Id3TdmiImgHPj7qqeF651ANzE+r6aniuIBi3BZoJQb63u9in7/+L3NuxEhznH9L/zyfHwsU5/ONsBAFGhR95VtwwP/9VrX2AV4qM74dbq9K/ilkc9sYdowpenLC3+ck2lbKe/3HseTd/2LOkSTMSV8nveSJ2sjZvFi7nSSJ7iXWoTuvxNEf+nTF9vXarFj/aMOOVYCje/USmLXWhHh1QTLbhVF7IX+NW7eYelaQOUJb1is1tJhJsvNBBaEzH6CPS1gtlS36LSfV6Gb0zZllbwm0OVpD6+AdqsRLvpyNM5e+27ZRMD7dwqwF1TQAzh9IpV76DQrYxtrSGzCcpN6WnBiBdDaJn15nAh8+768Cau0kRUE6naQFempjo55//+qA55qD5ig+d4uR5L+3SiNJsl7v98+PKEC5b1ygpTP0oXkJv95yP/Y76Zn2XCCNZf5g0XzpWddqKAaoNGNaX/t3eOop1al/ibEtprX23wxVYPnj86qi+bS5G+O7raTczRfxe1y+zseKS3FD1j+b/8gm/Hi9yfeQqRrmCrVtYEHjwu9egakvqIPa51AAQPbDD0E9+825PLhmoIC29N/5eUkd/QoAwI63H4yhv7ulfVvaoQQtLnggW14+o/+m9EccYimC6XvJ56jdltnS/qrBRUXNflr33+lJrq0H0Gri2Svb+xfEkA88GGgHmQVD0V9G+T5Hz6PVxECNNGihP5xi/aXzt6Md7bDOJjG8B2RqUesRpp3JjxOYcUnBKmueffHBzaZf7l+y3ChVfAdWuOs3mP7yzJL5C2YWM+o0Ho9WyRfIpGaNTcCcgm7P6tEu0jOhnclWR9CHL6kXVf8Exm+eexaWw9qdW/0CXH2aoUF9wQ/W57LIDfqARrVwXQXmToRzP4VPnWCcZd/0mwrk3ETRyDjf3eQ4vLSYni/g6cNB8R2e/Rf4yRmmjSdfs6Fwq47GuqB6RCYhd6PtO4e9GF98jmnjRbdc0j6vyL18Q9DRhCZXpeJEPzK599SlxfIV+rrZt1g66E79v71pom0jbdwHFuCaG8wG5tzwQzhV0zt8NGjhoZIaPnGdWVxcOkV2vtyEHYYmXSwNi/ALBM+XTEDTOtbtDkzeOlKgcodq3x8KC2ZnxtGPSd1FHwk3S8gLZOganYvFwaHI3Mn81fjaV38KADj/1fl3qHV8seScpTvlb+c6dfrf4EPvMRTEl1baVbHyCuf8Ow4jkueeCS26631Yb2PirCpZhsujw/h3m+z+eWhW9FXNU4V2GIvBxjzXdlu35NS/QuufzwQApK+/yqQrSdUbW/iH14b/8XfY/UFDRtv34ftK5Ut5Uj5+fozNsRHzstAJWyW48mZjynlRQNGrtXZxIFw8iKBthfxdXBE/N3z2ViNwuXTqhuXyy/W5YD4p6/Ct0jrWhnNN7nkEm8GS3yP3f1LDprE2icvULzYO2ZwfUka3q99k1t03sex3OPmDs1D6g6vwHkFMqgQCgUAgEAgEAsG0QeluONcEAoFAIBAIBAKBoAREwyEQCAQCgUAgEAimDbLhEAgEAoFAIBAIBNMG2XAIBAKBQCAQCASCaYNsOAQCgUAgEAgEAsG0QTYcAoFAIBAIBAKBYNogGw6BQCAQCAQCgUAwbZANh0AgEAgEAoFAIJg2yIZDIBAIBAKBQCAQTBtkwyEQCAQCgUAgEAimDbLhEAgEAoFAIBAIBNMG2XAIBAKBQCAQCASCaYNsOAQCgUAgEAgEAsG0QTYcAoFAIBAIBAKBYNogGw6BQCAQCAQCgUAwbZANh0AgEAgEAoFAIJg2yIZDIBAIBAKBQCAQTBtkwyEQCAQCgUAgEAimDbLhEAgEAoFAIBAIBNMG2XAIBAKBQCAQCASCaYNsOAQCgUAgEAgEAsG0QTYcAoFAIBAIBAKBYNogGw6BQCAQCAQCgUAwbZANh0AgEAgEAoFAIJg2/D/y2fe7Smc7xQAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "visualize_all(rand_output_base, crop_len, title=\"Random {} example attention map: pre-trained model\".format(task))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 46,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAskAAALJCAYAAABGNb7tAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XlgVPW9///XyWTCkgWSsIhs1isoglQEjVotJRqSyI4UFQmKy630VuWCaK1oq6IIauUCol6sokWlCgiyBRFQEC01BKQFBeEqEChrFggJJJk5vz/8Oj85xGLD5D0heT7+ypxZXu8kkzMvDmc+47iu6woAAABASFSkBwAAAABqGkoyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwiI70ANWmpMgkxi0uNMlRtN8mpxYKfvMPk5yo87qY5LgH80xynLgkkxzFNrLJCZSbxDjRMSY5bjBok5O3xSRHkpzkFjZBDeJtcipsnnN2rw+OTUywwianYK9NTlyiTY4kxTSwyYny2eQU/NMkxmnVodLtHEkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8wlaS8/Ly1LFjR2VkZGjTpk36/PPPdeutt6pnz5669tprdcsttyg3Nzd0+/PPP19791b+aTevv/66+vTpo4yMDF1zzTV64IEHVFxcrBUrVigjI0Pnn39+uMYGAAAAThLWI8nNmzdXdna2oqKi9J//+Z8aOnSo3n//fX3wwQcaNmyY7rzzTn311Vf/8jFWrVqlt956S6+99pqys7O1ZMkSHTt2TBMnTlRqaqqys7PDOTIAAABwkujqeNAXXnhBN9xwg6699trQtmuuuUZTp05VcnLyv7zv1q1b1bZtWyUlJUmSYmJi9MQTT1THmAAAAEClquWc5M8++0zdu3c/afsVV1wRKr8/5Morr9THH3+sBx54QB999JGKi4sVFxenuLi46hgVAAAAOEm1lOSioiI1adKkSve98MIL9dZbbykYDOq3v/2tLr/8cv3Xf/2X9uzZE+YpAQAAgMpVS0lOTEzUvn37qnz/iy66SE8//bQ++eQTzZo1S2VlZfrv//7vME4IAAAA/LBqKckpKSl6//33T9o+Z84c/f3vf/+X983JyQkVbMdx1KlTJ913333aunVrdYwKAAAAnKRaSvKIESP03nvv6d133w1tW7ZsmZ599tlTnlu8YMEC/f73v1dxcbEkqaKiQosWLdKll15aHaMCAAAAJ6mW1S3atWunV155Rc8++6ymTp2qmJgYtW3bVjNmzNBPfvKT0O2ysrLk8/lCl8eNG6eHHnpIzz33nK6//npJ35bklJQUjR8/vjpGBQAAAE5SLSVZkjp16qRXX331B6/fsmXLD173wAMP6IEHHqiOsQAAAIBT4mOpAQAAAI+wluR9+/aFPpa6Onz3sdQAAABAdQrb6RatWrWqtnL8ndTUVKWmplZrBgAAAMDpFgAAAIAHJRkAAADwoCQDAAAAHo7rum6kh6gObnGBTc5um08CdOKTTHJkleMY/vssUG4S4/jrmeQAkuRW2DyvFeU79W3C5O1W55vkDM770iTHMfzZWXCDAaMkxyjGKMeQY/Q91bbnghOXWOl2jiQDAAAAHpRkAAAAwIOSDAAAAHhQkgEAAAAPSjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAeESvJU6ZM0UMPPfRv32/x4sUqLi6uhokAAACAb51xR5InT55MSQYAAEC1qraS3L17d+3YsUPSt0d/O3XqpNLSUknSq6++qqKiIpWVlWnUqFFKTU3V4MGDtW/fPknS//3f/+mmm25SZmam0tLStHDhQknSgw8+qK+//lpZWVnKycmprtEBAABQx1VbSU5JSdH69eslSZ999pk6duyojRs3SpJycnKUk5OjTz/9VKNHj9aKFSuUlJSk2bNnS5ImTpyoHj16aMmSJXryySf10EMPqby8XOPHj5ck/fnPf1a3bt2qa3QAAADUcdVakjds2CBJ+vzzzzVo0CDl5uaGLl922WXq2rWrWrZsKUm64IILQkeSp02bpttvv12S1LVrVx0/flwHDhyorlEBAACAE1R7SS4qKpLf79fll1+u3Nxcbd++XS1atFB8fLzi4uJCt/f5fAoEApKk1atX6+abb1Z6erquu+46ua6rYDBYXaMCAAAAJ4iurgdu1aqVSkpKtHr1al188cVq3bq18vLytG7dOl1xxRU/eL/y8nKNHDlSkyZNUvfu3VVWVqbOnTtX15gAAADASap1dYuuXbvq9ddf1yWXXCJJOvfcczVnzpx/WZJLS0tVUlKiTp06SZJee+01+f1+lZSUSJKio6N1+PDh6hwbAAAAdVy1luSUlBR9/vnn6tKliySpS5cu2rx5c6g0VyYhIUF33HGH+vfvr/79+6tNmza69tprddddd6mkpEQZGRm68cYbtXjx4uocHQAAAHWY47quG+khqoNbXGCTs3urSY4Tn2SSI6scx3CJ7kC5SYzjr2eSA0iSW2HzvFaUzyZH0tutzjfJGZz3pUmOY/izs+AGA0ZJjlGMUY4hx+h7qm3PBScusdLtZ9yHiQAAAADVjZIMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR7V94l6kOVFG/T/pLJscK5ZLswGoulq4fNXKolKTnMHBoEmO5fJ5Jqx+bj6bamK1XFrtVDeW6aMRAQAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAD0oyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHjW6JG/cuFG33357pMcAAABAHVOjS3Lnzp31pz/9KdJjAAAAoI6p0SV57dq1SktL09atW3XDDTeoV69e6tmzp2bOnBnp0QAAAFCL1eiS/J2pU6fqxhtv1KJFizRr1ix98sknKisri/RYAAAAqKXOiJKcnJyspUuXatOmTUpMTNS0adMUExMT6bEAAABQS50RJfm+++5T+/btNXLkSHXv3l1vvPFGpEcCAABALXZGlOTY2FiNGjVKy5Yt09SpUzV58mR9/fXXkR4LAAAAtdQZUZLvuusuffXVV5Kk9u3bKy4uTo7jRHgqAAAA1FbRkR7gxxg6dKhGjx6t8vJySdKQIUN0zjnnRHYoAAAA1FqO67pupIeoFiVFJjHu0UKTHDMNEiI9QfgFyk1iHH89kxxAktxAhU2QY/cfjiPi25jkvFBkc7qeE+03ybHiVtjsS+WzOX7H/0hXnRsM2gQZ/Y6c2MaVbj8jTrcAAAAALFGSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAByUZAAAA8Dgj1kmuCrfsmE1QlM8mJxiwyTlWbJNjuVzakUMmMW7j5iY5lktymTBahdKJqmU/t+MlNjn1GtrkSHo6/XybIKP9qesavcRareRactgmp2G8SYxrtS81XGrOseokFWU2OUbLAf6QWvaqAQAAAJw+SjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAI+IluS3335bkpSVlaX58+dHchQAAAAgJGIlORAIaOLEiZGKBwAAAH5QxEry8OHDdeTIEWVkZCgvL095eXnKysrS1VdfrVGjRikYDEqS1q1bp+uvv15paWkaPHiwdu3aFamRAQAAUEdErCQ/+eST8vl8ys7OVqtWrfS3v/1N06dPV3Z2ttauXavc3FwVFxdrxIgRGjVqlJYtW6Zhw4bp3nvvjdTIAAAAqCNqzBv3evbsqfr16ys2NlZt27bV3r17tW7dOjVv3lw/+9nPJEm9e/fWzp07tWfPnghPCwAAgNosOtIDfCcuLi70tc/nUyAQ0OHDh7Vr1y5lZGSErouJiVF+fr7OPvvsSIwJAACAOqDGlOTKNGvWTOeee67mzp0b6VEAAABQh0TsdAu/369gMKji4uIfvM1Pf/pTHThwQJ9//rkkadeuXRozZoxc17UaEwAAAHVQxI4kN23aVF27dlWPHj1UWlqqQYMGnXSb+vXra/LkyXr88cd19OhR+f1+3XvvvXIcJwITAwAAoK5w3Fp6WNYt3GcTFCi3yQkGbHIco/9c8NezyZGkI4dscho3t8mx+h1ZMdoFOVG16+fmlhy2CarX0CZHUvEv001y4mYtMskx289ZvYwXF9jkNIy3ybHalxoe2HOifCY5btkxkxz5bI7lOvHJlW6vXa8aAAAAQBhQkgEAAAAPSjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCo0Z+4d1qiY2xyyo2WQbFa4uf4D3+4S1hZrnVttCSOgkGbHJ/Nv22t1iN3XaOfW21j9TdkuEpog07n2AQFK2xyZLQEnNVzoazEJsdqCTgrtXGlXaslNSP8s+NIMgAAAOBBSQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAD0oyAAAA4GFaksvKyjRv3rzTeoysrCzNnz8/TBMBAAAAJzMtyZs3bz7tkgwAAABUt9Muye+8844yMzPVs2dP3Xzzzdq9e7dc19X48eOVmpqq9PR0vfzyyzp48KB+85vfaMOGDRoyZIjy8vJ04YUXhh7n+5eDwaAeffRRpaenKzU1VWPGjFF5efnpjgoAAAD8KKdVkg8dOqTHHntMr776qt5//321adNG06ZN03vvvaeNGzdq6dKlmjNnjmbOnKk9e/Zo1KhRuvjii/Xmm2/+y8ddtmyZcnJytHDhQi1ZskSbNm3S4sWLT2dUAAAA4EeLPp07Jycna926dYqJiZEkdevWTfPnz9exY8eUnp4uv98vv9+vxYsXq0GDBtq2bduPetz09HT16NFDfr9fknTRRRdp165dpzMqAAAA8KOdVkkOBAKaPHmyVqxYoUAgoKNHj+onP/mJCgoKlJCQELpdw4YN/63Hzc/P1+OPP67NmzfLcRwdPHhQt9xyy+mMCgAAAPxop1WSFy9erBUrVmjmzJlKSkrS22+/rQULFigxMVEFBQWh2x08eFD169c/4b4+n0/BYFCu68pxHB0+fDh03XPPPafo6GgtWLBAMTExGj169OmMCQAAAPxbTvuc5JYtWyopKUkFBQVasmSJjh49qtTUVC1atEhlZWUqKSnRkCFDtHXrVkVHR6u4uFiu6yoxMVE+n09btmyRpBNWvTh06JDat2+vmJgYffnll1q/fr1KSkpO7zsFAAAAfqTTKsm9e/dWYWGh0tLSNHr0aI0cOVJ79+7V3//+d1111VXq2bOnBgwYoEGDBumSSy5R165dtX//fl199dXy+/26++67dccdd2jgwIHq0KFD6HFvu+02zZo1S5mZmXrjjTf0wAMP6J133tGSJUtO+xsGAAAATsVxXdeN9BDVwS0uOPWNwqH08KlvEw5Wv6ayUpucBvE2OZJ03Oh/IeKSbHJ8p3WW1I/mOI5JjhsMmOQ4UT6THCtu6RGbIH/9U98mTAKP3WWS4/vd/5jkqF6sTY6VQ3k2OY2b2+Q4te9Dhx2j1we3oswkR7J5HXISmlS6vfY9QwAAAIDTREkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeNgsqBcJFcdtcvz1bHKs1nOMMVoT1XLt1RcfNcnx3f2kSY7ZmtlG6ySrcJ9JjJvYwiTHTEwDmxzD9aWdCy60CYqOscmx4gZtcho2sskxWjtd0UbP7dr4cRRRRvXR6nXoB3AkGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB7mJXnt2rVKS0s76fLWrVt1ww03qFevXurZs6dmzpwpSSorK9O4ceOUnp6u1NRUvfjii9YjAwAAoI6pMUeSp06dqhtvvFGLFi3SrFmz9Mknn6isrEzTp0/Xtm3btGDBAi1cuFBLly7VypUrIz0uAAAAarEaU5KTk5O1dOlSbdq0SYmJiZo2bZpiYmK0cuVKDRkyRDExMWrYsKH69eun999/P9LjAgAAoBarMSX5vvvuU/v27TVy5Eh1795db7zxhiTpyJEjGj9+vDIyMpSRkaHXX39dpaWlEZ4WAAAAtVm0daDP51MgEAhdPnz4sCQpNjZWo0aN0qhRo7Rx40bdeeeduvLKK9WsWTPddttt6tGjh/WoAAAAqKPMjyQ3bdpUBw4c0KFDhxQIBLRgwQJJ0l133aWvvvpKktS+fXvFxcXJcRxdc801eueddxQIBOS6rqZNm6ZVq1ZZjw0AAIA6xPxIctu2bXX99derf//+Ovvss9WvXz998cUXGjp0qEaPHq3y8nJJ0pAhQ3TOOedoyJAhysvLU69eveS6rjp16qRbbrnFemwAAADUIY7rum6kh6gObuHeSI8QXo7RQX83aJPjr2+TIykwaYxJju/uJ01yFGXzb1snyuY55+bvMclRYgubHCvBwKlvEw5RPpscScG3/miSEzX4bpMc+fw2OVb77WNHbXKijX5u0TE2OYY1y/HZvD64QaPnnOPYxMQ2rnR7jXnjHgAAAFBTUJIBAAAAD0oyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHuYfJmKm/LhNToMEmxyjtQIVqLDJsVrXU5LKymxyigtschKamMRYLaHuFheZ5DhW6yRbPbcD5TY5Vmu0S4rqf4dNkOH3ZMJqv11WapNT29YAt/r9SHKtuoLZ2s+R/VutZXsKAAAA4PRRkgEAAAAPSjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAelGQAAADA44wqybt27VJaWpr69esX6VEAAABQi51RJXndunVq2rSp5s+fH+lRAAAAUIvV2I+lXrJkiZ5//nlVVFSoWbNmuvfee/XMM8+ouLhYffv21XvvvRfpEQEAAFBL1cgjyXv27NHDDz+s559/XtnZ2frFL36hKVOmaNSoUbr44ospyAAAAKhWNbIkr1mzRikpKWrbtq0k6Ze//KXWrl2rQCAQ4ckAAABQF9TIklxQUKCEhITQ5fj4eLmuq4KCgghOBQAAgLqiRpbk5ORkFRYWhi4XFRUpKipKiYmJEZwKAAAAdUWNLMk/+9nPlJOTo127dkmSZs2apZ/97Gfy+XwRngwAAAB1QY1c3eKss87SuHHj9Otf/1rl5eVq1aqVHn/8cX366aeRHg0AAAB1QI0syZKUnp6u9PT0E7YNHDhQAwcOjNBEAAAAqCtq5OkWAAAAQCRRkgEAAAAPSjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAejuu6bqSHqA5ucYFJjhPFvzNgq+zewSY5/ufeMskx49j8rTqOY5JTG3fdVj87K24waJJT216HzJ7bFWUmMYXXpZrkSFLjJR/aBPlq7MdsVIkT27jS7bXrLwsAAAAIA0oyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHpRkAAAAwIOSDAAAAHhEvCSff/752rt370nbly1bpgcffDACEwEAAKCuq7EfmZKWlqa0tLRIjwEAAIA66EcdSX7xxRd1xRVX6Prrr9cbb7yh1NRUTZkyRWPHjtWgQYM0Y8YMBYNBPfroo0pPT1dqaqrGjBmj8vJySVJWVpamTJmiwYMH68orr9TYsWMVCARCj//RRx9p4MCBuuqqq/TKK69IkubOnatbb71VkpSfn6+77rpL11xzjfr06aOPP/44zD8GAAAA4P93yiPJX331lV5++WUtXrxYjRo10h133BG67qOPPtL8+fOVlJSkpUuXKicnRwsXLlQwGNSAAQO0ePFi9evXT5K0atUqvf7665Kk/v37a+XKlbr22mslSbt379bcuXP197//XTfffLOysrJOmOHZZ5/Vf/zHf+jFF1/U5s2bNXz4cK1evVoxMTFh+0EAAAAA3znlkeTPPvtMl112mZo1a6Z69erp+uuvD13305/+VElJSZKk9PR0zZkzR36/X/Xq1dNFF12kXbt2hW7bq1cvNWjQQA0aNNDVV1+t9evXh67r27evJOnCCy/U8ePHVVBQcMIMH330kXr37h26zfLlyynIAAAAqDanPJJ8+PBhNWrUKHS5efPmoa+/vz0/P1+PP/64Nm/eLMdxdPDgQd1yyy2V3rZRo0bav39/6HJcXJwkyefzSZKCweAJMxQWFio+Pv6k2wMAAADV4ZRHkuPi4lRSUhK6/P1y+33PPfecoqOjtWDBAmVnZ6t79+4nXP/9o8NFRUUnlOZTady48Qn3z8vLC53vDAAAAITbKUty586dtXbtWuXn56usrEzz5s2r9HaHDh1S+/btFRMToy+//FLr168/oVwvW7ZMZWVlKikp0apVq9StW7cfPWRqaqreffddSdK2bds0cODAE974BwAAAITTjyrJAwYM0IABAzRs2DD16NGj0tvddtttmjVrljIzM/XGG2/ogQce0DvvvKMlS5ZIkrp06aJhw4YpNTVVKSkp+vnPf/6jhxwzZoz27t2r1NRU/fd//7eeeeYZ1a9f/0ffHwAAAPh3OK7ruqe6keu6chxHkvThhx9q0qRJP3hEuTJZWVkaNGhQaKULC25xwalvFAZOVMQ/jwV1TNm9g01y/M+9ZZJjxrH5W/1uX1ndfsSu+4xj9bOz4nreX1NdatvrkNlzu6LMJKbwulSTHElqvORDmyBfjf2YjSpxYhtXuv2Uf1n5+fm6/PLLtXv3brmuqyVLlujiiy8O+4AAAABATXHKfwokJSVp5MiRuvXWW+U4js4991zdf//9FrMBAAAAEfGjjpffdNNNuummm6oc8uc//7nK9wUAAACs1a4TmQAAAIAwoCQDAAAAHpRkAAAAwIOSDAAAAHj8qHWSz0RW6ySrlq3rWSuVH7PJiY4xiXF3bDLJceIqXzcy7Bo3t8k5dtQmJ7aRTU7Q6FNHy4/b5EhS/Ti7LAtFB2xyrP5Wo3w2OUbrF7v5/zTJUdFBmxxJik80iXHOOtckR2WlJjFOUstKt3MkGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR40tyfPnz1dWVlakxwAAAEAdVGNLMgAAABAp0eF8sHnz5umFF16QJHXu3FlPPPGE5s+fr1deeUWBQEBNmzbVxIkT1bJlS82dO1crVqzQkSNH1LFjR913330aN26cVqxYoSZNmujSSy8NPe7f/vY3jR8/XsePH5frurrnnnuUmZkZztEBAACAkLAdSc7Ly9OECRP0+uuvKzs7W6WlpZoxY4Yee+wxvfrqq3r//ffVpk0bTZs2LXSfNWvW6NFHH9X999+v1atXa82aNVq0aJFmzpypnJyc0O0mTJigBx98UIsXL9YLL7ygDz74IFxjAwAAACcJW0les2aNunTpoubNm8txHD377LO67bbbtG7dOp111lmSpG7dumnXrl2h+5xzzjk655xzJEmfffaZunfvrtjYWNWvX/+EI8XJycmaN2+etm/frnPOOUfPPvtsuMYGAAAAThK2klxQUKCEhITQ5Xr16slxHE2ePFnXXXed0tPT9dxzz8l13dBtGjVqFPq6qKhI8fHxocvff6wnn3xSDRo00PDhw9WzZ09lZ2eHa2wAAADgJGEryYmJiSooKAhdLi4u1nvvvacVK1Zo5syZWrp0qe65554fvH9CQoKOHDkSupyfnx/6ukmTJnr44Ye1atUqPfLII3rwwQd19OjRcI0OAAAAnCBsJbl79+7Kzc1VXl6eXNfV73//e+3bt08tW7ZUUlKSCgoKtGTJkh8st126dNHHH3+s0tJSlZaWho4Wl5eXKysrS/v375ckdezYUdHR0YqKYmEOAAAAVI+wrW5x1lln6bHHHtMtt9win8+niy66SH369NHy5cuVlpam1q1ba+TIkRoxYoSeeuoptW/f/oT79+jRQx9++KEyMjLUpEkTde/eXTk5OfL7/Ro0aJBuvfVWSVJUVJTGjh2rBg0ahGt0AAAA4ASO+/2ThGsRt7jg1DcKB8exyUHVlR+zyYmOMYlxd2wyyXHiGpvkqHFzm5xjRqdoxTY69W3CIRiwySk/bpMjSfXj7LIsFB2wybH6W43y2eRUlJnEuPn/NMlR0UGbHEmKTzSJcc461yRHZaUmMU5Sy0q3c84CAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHpRkAAAAwIOSDAAAAHjU4nWS8099ozBwrNaNRJUFls00yYlKvdEkRwV7bXIaNzOJKbmln0lOwxnzTHKs1k53jNblRtW5waBVkk2MY3RczWoNcCtWn9sgSQ0TTGIcfz2THDMNK1/fniPJAAAAgAclGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR40tyRs3btTtt98e6TEAAABQB9XYkty5c2f96U9/ivQYAAAAqIOiIz3AD1m7dq3Gjh2rvn37qqCgQPv27dOXX36pxMRETZs2Tc2aNYv0iAAAAKilauyR5O/Lzs7W7373O33wwQdKTk7WnDlzIj0SAAAAarEaeyT5+7p166aWLVtKkjp06KB//vOfEZ4IAAAAtdkZcSQ5Pj4+9LXP51MgEIjgNAAAAKjtzoiSDAAAAFiiJAMAAAAelGQAAADAo8a+cS8lJUXLli07afvdd98dgWkAAABQl3AkGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAODhuK7rRnqI6uAW55vkOFE+kxxUndVT3HEckxw3UGGSo0C5TY6/vknM0ZsyTHJiZ8wxyVF0jE1OMGiTI8mJsXkuWAnMeMIkJ2roGJMc+fw2Oa7Rc67CaB9n9bcqSapllc7o9c5p1KzS7RxJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHpRkAAAAwIOSDAAAAHhQkgEAAAAPSjIAAADgQUkGAAAAPE6rJK9du1ZpaWlVvv/bb799OvEAAABAtYjYkeRAIKCJEydGKh4AAAD4QWEpyaWlpRo5cqTS09OVmpqqCRMmhK4Dg3myAAAgAElEQVRLTU3V9OnTNXDgQF1++eWaNGmSJGn48OE6cuSIMjIytGvXLu3Zs0e333670tPT1bt3b82bN0/St0er+/Tpo6eeeir0+Bs2bAjH2AAAAEClwlKS33rrLR09elTZ2dl69913NXfuXOXk5ISu37Bhg9555x0tWrRIb775pr788ks9+eST8vl8ys7OVuvWrfXwww/rsssu09KlS/XSSy9p3LhxysvLkyRt375dnTt31tKlSzVixAj94Q9/CMfYAAAAQKXCUpJvu+02TZs2TY7jqFGjRmrXrl2o4EpS//795fP5lJycrK5duyo3N/eE+5eXl+uTTz7RkCFDJEktW7ZUSkqK/vrXv0qSGjZsqMzMTElSz5499cUXX6i0tDQcowMAAAAnCUtJ/uabb3T33XerZ8+eysjI0D/+8Q8Fg8HQ9Y0aNTrh68OHD59w/8LCQrmuq/j4+NC2hIQE5efnh752HCf0taSTHgMAAAAIl7CU5Mcee0zt2rXTkiVLlJ2drQsuuOCE6wsKCkJfFxYWnlCaJSkxMVFRUVEqKio64XbJycmhr7/z3W0aN24cjtEBAACAk4SlJB86dEgdOnSQz+fTmjVrtGPHDpWUlISuX7x4sYLBoA4ePKjc3Fx169ZNfr9fwWBQxcXFio6O1lVXXaW//OUvkqSdO3cqJydHV155pSTp2LFj+uCDDyRJS5cuVadOnVSvXr1wjA4AAACcJDocDzJixAiNHz9e06ZN0zXXXKPf/OY3mjx5sjp06CBJateunQYNGqTdu3crKytL7dq1UzAYVNeuXdWjRw+99NJLevTRRzV27FjNnTtXfr9f48aNU4sWLbRz5061bNlS69at09NPP63y8vLQChkAAABAdXBc13WrMyA1NVUTJ05Ut27dqnT/tWvXauzYsVq2bNm/dT+3OL9Kef8uJ8pnkoOqq+aneMh3581XNzdQYZKjQLlNjr++SczRmzJMcmJnzDHJUXSMTc733l9S3ZwYm+eClcCMJ0xyooaOMcmRz2+T4xo95yqM9nFWf6uSJJvXOzNGr3dOo2aVbudjqQEAAAAPSjIAAADgEZZzkv+VFStWnNb9U1JS/u1TLQAAAIDTwZFkAAAAwIOSDAAAAHhQkgEAAACPal8CLlJYAq5qattyaZIU3L/DJMdp0sokR8GAUY7RMkyHD5rEBL9Ya5KjA/tMYqL63mmSI8t9nNF+wWy5RqPnttnvKKaBTY7VPs5qyUHDZRQVNFoiNLp2faCbE5dY6XaOJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB4RKcmPPPKIjh49Kkm67777JEnBYFD333+/JGnjxo167bXXIjEaAAAAEJmSfPToUcXGxqqiokJ+v1+StHPnTrVq9e3H+m7fvl3nnXdeJEYDAAAAIlOSXdeVJO3YsUNt2rSRJG3bti1UjL//9cyZM5WZmamMjAwNGjRIX331VSRGBgAAQB0SbRk2Y8YMrV69Wrt27dKvf/1r7du3T47jaOrUqWrbtq2aNm2qhQsXatOmTfr6669177336n/+53+0cuVKxcXFacmSJfrwww/Vrl07y7EBAABQx5iW5FtvvVWtWrXS8ePH1atXL02YMEHDhg1TixYtNGbMGD311FPy+XwaNWqU/vjHP6q8vFyO42j27Nnq3bu3MjMzLccFAABAHWV+usWmTZvUuXNnSdLevXvVokULSd++cc/n82nPnj2hbX6/XzNmzFBubq7S09M1ZMgQbdmyxXpkAAAA1DGO+90JwgYGDhyoHTt2qGnTppKk/fv3KzExUT6fT4WFhUpKSlJJSYmCwaDOOusszZ49O3TfsrIyvfzyy1q1apVmzZp1yiy3OL/avo/vc6J8JjlWrJ4OjuOY5EhScP8OkxynSSuTHAUDRjlBm5zDB01igl+sNcnRgX0mMVF97zTJkeU+zmi/YLX/cY2e22a/o5gGNjlW+7iY+jY5VvtSSQpW2ORE17PJMeLEJVa63fRI8ty5c9WjRw9lZ2frpZde0rBhw7R8+XKNHTtWv/vd75Sdna0BAwbozTff1OzZs7Vlyxbdc889KisrU0xMjDp16mRargAAAFA3mZ6T/M0334RWs8jJyVFKSookad26dRo0aJAkKS8vL3Sb9u3bq1WrVurdu7f8fr9iY2P1yCOPWI4MAACAOsj0dAtLnG5RNZxuUXWcblFFnG5RJZxucToxnG5RJZxuUTWcblHj1YjTLQAAAIAzASUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAj1q7BJxKikxi3EKb5Z4koyXT/DE2OdFGOZJUXmaT0zDeJqf8uE2Oz2gZ9SijHNdmGSbH6OfmlpXa5OT/0yRHktwtuSY5UVf2Mcmxes6Z7U+tljKLMjp+V3bMJsdqXyrZPRdKj9jklBw2iXHadKx0O0eSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAjzOiJH/88ccaNGiQMjMz1a9fPy1fvjzSIwEAAKAWM/ysxKrZs2eP7r//fs2YMUPt27fX1q1bNXToUM2fP18tWrSI9HgAAACohWr8keS9e/dqxIgRat++vSSpffv2atOmjTZt2hThyQAAAFBb1fgjyZdccokuueSS0OX8/Hx9/fXXOvvssyM4FQAAAGqzGn8k+fsOHDigO+64Q3379tWFF14Y6XEAAABQS9X4I8nfqaio0PDhw9WnTx/96le/ivQ4AAAAqMXOmCPJ69evV+PGjSnIAAAAqHZnTEm+9NJLNXPmzEiPAQAAgDrgjCnJy5Yt04MPPhjpMQAAAFAHnDHnJKelpSktLS3SYwAAAKAOOGOOJAMAAABWKMkAAACAByUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAD8d1XTfSQ1QHt3CvTVAwaJQTsMmJ8tnkWH0/kiSjp7i/vk2O49jkBCpscqJjbHIC5TY5Pr9NTuE+mxyrfYJk9zfkN3rOWb28Wv2OKspscqKN/obKjtvk1G9okyNJMnp9OHzQJsfo9c5peX6l2zmSDAAAAHhQkgEAAAAPSjIAAADgQUkGAAAAPCjJAAAAgAclGQAAAPCgJAMAAAAelGQAAADAIzocD5KXl6f09HS1bt1aTzzxhBYuXKhPPvlEjuPIdV31799fI0aMkCRlZWXp66+/Vlxc3AmPMXToUA0dOvSE613Xlc/n0w033KBbbrlFkyZN0rx589S6dWv9+c9/DsfoAAAAwEnCUpIlqXnz5srOztYzzzyjw4cPa8GCBYqJidHBgwd18803q02bNurVq5ckacyYMerXr98PPtb3rz9w4IBuuOEG/eQnP9HIkSN1xRVXaOrUqeEaGwAAADhJ2E+32Lp1qzp27KiYmG8/9rNJkyZ68803lZaWVqXHa9q0qTIyMrRmzZpwjgkAAAD8oLAdSf5O9+7d9cwzz6ioqEi/+MUvdNFFFyk5Ofm0HrOiokL16tUL04QAAADAvxb2knzzzTcrKSlJc+bM0YwZM+T3+zVw4ECNHj06VHSffvppvfDCCyfcb/r06WrduvVJj7dr1y5lZ2dzigUAAADMhL0kS1JmZqYyMzNVVlamTz/9VOPGjVO9evU0evRoSac+J/m7Eu26rhISEvTb3/5WnTt3ro5RAQAAgJOEtSSXl5fro48+Uo8ePeTz+RQTE6Pu3btr586d+vjjj3/045yqRAMAAADVKaxv3IuOjtZzzz2nF198UYFAQJJUXFysFStW6NJLLw1nFAAAAFBtwnok2XEcTZ8+XRMnTlRmZqYcx5Ek9e3bV8OHDw/drrJzkjt37qyJEyeGcxwAAACgSsJ+TvLZZ5+tSZMm/eD1p/oQED4kBAAAAJHGx1IDAAAAHmEryfv27VNGRoY2bdoUroc8yaRJk/TAAw9U2+MDAAAAkuS4rutGeojq4BbutQkKBo1yAjY5UT6bHKvvR5Jk9BT317fJ+X/n+le7QIVNTnSMTU6g3CbH57fJKdxnk2O1T5Ds/ob8Rs85q5dXq99RRZlNTrTR31DZcZuc+g1tciRJRq8Phw/a5Bi93jktz690O6dbAAAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAj9r7xr2jhTZBRQdscuo1sMmxetORL+xLdP+w/D02OU1am8Q4Vm/cQ43mWr2x0vCNe++2rvzNM+E2YPsGkxynnuUbtqqfa/RGdSeK43c1XW3b/zixjSuPN0kHAAAAziCUZAAAAMCDkgwAAAB4UJIBAAAAD0oyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwqDElecqUKXrooYciPQYAAABQc0oyAAAAUFOYleTu3btrx44dkqTFixerU6dOKi0tlSS9+uqrKioqUllZmUaNGqXU1FQNHjxY+/bt0xtvvKFf/epXoccJBoO68sor9cUXX1iNDgAAgDrGrCSnpKRo/fr1kqTPPvtMHTt21MaNGyVJOTk5ysnJ0aeffqrRo0drxYoVSkpK0uzZs5WRkaG//vWvKigokCTl5uYqISFBHTp0sBodAAAAdYxpSd6wYYMk6fPPP9egQYOUm5sbunzZZZepa9euatmypSTpggsu0L59+5ScnKxu3bpp6dKlkqRly5bpuuuusxobAAAAdZB5SS4qKpLf79fll1+u3Nxcbd++XS1atFB8fLzi4uJCt/f5fAoEApKkXr16aeHChZKk5cuXU5IBAABQrcxKcqtWrVRSUqLVq1fr4osvVuvWrZWXl6d169bpiiuu+Jf3TUtL0z/+8Q999NFHatCggc477zyjqQEAAFAXma5u0bVrV73++uu65JJLJEnnnnuu5syZc8qSHB8fr6uvvlqPPvqoMjMzLUYFAABAHWZaklNSUvT555+rS5cukqQuXbpo8+bNodL8r/Tq1Uu7d+/mVAsAAABUO8d1XTfSQ/wYGzdu1GOPPabZs2f/qNu7RwureaL/p+iATU69BjY5Pr9RTrRNjiTl77HJadLaJMZxHJMc1GxuoMImKMpnkyPp3dbnm+QM2L7BJMep19Akx4obDJrkOFF8hENNV9v2P05s48rjTdJPU0VFhZ5//nllZWVFehQAAADUATW+JG/evFlpaWlq1qyZ+vbtG+lxAAAAUAcY/p931Vx44YVauXJlpMcAAABAHVLjjyQDAAAA1ijJAAAAgAclGQAAAPCgJAMAAAAeNf6Ne1UWKDcKMlpm2mpNwrJjNjn+GJscSW7JEZMcx+g550bV3j/bamW0vrTZOtYlh21yoo3WTpfU739/axN0vNQkxvXXN8mRa7N+sY6XmMS4MTafC+AYPbet1peWDNeYLrP5GzJzJq+TDAAAAFiiJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR40tyTk5OUpNTY30GAAAAKiDamxJBgAAACKlRn2+7bRp0/SXv/xFiYmJoaPIx48f1xNPPKG1a9cqKipK3bt315gxY+Tz+SI8LQAAAGqrGnMkedu2bZoxY4bmzJmjOXPmaMuWLZKk1157TXv37tWiRYv07rvvKicnRwsXLozwtAAAAKjNakxJ/uyzz3TppZeqSZMm8vl86tu3ryTpww8/1ODBgxUdHa369eurT58+WrNmTYSnBQAAQG1WY0pyUVGR4uPjQ5cTEhIkSfn5+WrUqFFoe6NGjXTo0CHz+QAAAFB31JiSnJCQoCNHjoQuFxQUSJKaNGmiwsLC0PbCwkI1adLEfD4AAADUHTWmJHfp0kXr1q1Tfn6+AoGA3nvvPUnSL37xC82ePVuBQEAlJSWaP3++unfvHuFpAQAAUJvVmNUtOnTooBtvvFEDBgxQ48aN1atXL23dulVZWVnatWuXevXqJcdxlJGRoczMzEiPCwAAgFrMcV3XjfQQ1cE9fMAmqLjAJic6xiYnUGGT4zf6fiS5B/JMcpyW7UxyFFVj/m17ZnEcoxibHPdIvkmOov02OZKCK98xyYm6qp9JjuISbXLcoE3O8RKbnJgGJjGO0XPbDRr9fiQ5UTYnCLilR059ozOIk9yq0u015nQLAAAAoKagJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8au86yft3mOQ4cY1NcqxYPR2s1pKVJLfsmE2Q1Xqyx4ptcqx2DQ0STGICY4eb5Ph+/4JJjtnvR3YvEU79OLMsC8GvN5rkOK0vMMkxW6M9GLDJCZTb5Bi+3inKZ5Njtf8x+h05SS0r3c6RZAAAAMCDkgwAAAB4UJIBAAAAD0oyAAAA4EFJBgAAADwoyQAAAIAHJRkAAADwoCQDAAAAHqdVkteuXau0tLQq3//tt98+nXgAAACgWkTsSHIgENDEiRMjFQ8AAAD8oLCU5NLSUo0cOVLp6elKTU3VhAkTQtelpqZq+vTpGjhwoC6//HJNmjRJkjR8+HAdOXJEGRkZ2rVrl/bs2aPbb79d6enp6t27t+bNmyfp26PVffr00VNPPRV6/A0bNoRjbAAAAKBSYSnJb731lo4ePars7Gy9++67mjt3rnJyckLXb9iwQe+8844WLVqkN998U19++aWefPJJ+Xw+ZWdnq3Xr1nr44Yd12WWXaenSpXrppZc0btw45eXlSZK2b9+uzp07a+nSpRoxYoT+8Ic/hGNsAAAAoFJhKcm33Xabpk2bJsdx1KhRI7Vr1y5UcCWpf//+8vl8Sk5OVteuXZWbm3vC/cvLy/XJJ59oyJAhkqSWLVsqJSVFf/3rXyVJDRs2VGZmpiSpZ8+e+uKLL1RaWhqO0QEAAICThKUkf/PNN7r77rvVs2dPZWRk6B//+IeCwWDo+kaNGp3w9eHDh0+4f2FhoVzXVXx8fGhbQkKC8vPzQ187jhP6WtJJjwEAAACES1hK8mOPPaZ27dppyZIlys7O1gUXXHDC9QUFBaGvCwsLTyjNkpSYmKioqCgVFRWdcLvk5OTQ19/57jaNGzcOx+gAAADAScJSkg8dOqQOHTrI5/NpzZo12rFjh0pKSkLXL168WMFgUAcPHlRubq66desmv9+vYDCo4uJiRUdH66qrrtJf/vIXSdLOnTuVk5OjK6+8UpJ07NgxffDBB5KkpUuXqlOnTqpXr144RgcAAABOEh2OBxkxYoTGjx+vadOm6ZprrtFvfvMbTZ48WR06dJAktWvXToMGDdLu3buVlZWldu3aKRgMqmvXrurRo4deeuklPfrooxo7dqzmzp0rv9+vcePGqUWLFtq5c6datmypdevW6emnn1Z5eXlohQwAAACgOjiu67rVGZCamqqJEyeqW7duVbr/2rVrNXbsWC1btuzfup+7f0eV8v5dTlztOu2jmp8OId+dY27BLTtmExTtt8k5VmyTY/RcUIMEk5jA2OEmOb7fv2CSY/b7kVWO5NSPM8uyEPx6o0mO0/qCU98oHKLCclzt1IIBm5xAuU2O4eudonw2OVb7H6PfkZPUstLtfCw1AAAA4EFJBgAAADyq/f9OVqxYcVr3T0lJ+bdPtQAAAABOB0eSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR7Wvkxwpbv5um6DoGJscx+jfM27QJsfq+5EUXPe+SU7UxakmOTp21CbHZ7Tepr++TY7V+tIN4m1yyo/b5Fjt4yS7/Y/V91RWapNj9XPzGa0FH6iwyYlpYJNjte+RJJ/RWtZW6zEbdQWncfNKt3MkGQAAAPCgJAMAAAAelGQAAADAg5IMAAAAeFCSAQAAAA9KMgAAAOBBSQYAAAA8KMkAAACAR0RK8iOPPKKjR7/9QIT77rtPkhQMBnX//fdLkjZu3KjXXnstEqMBAAAAkSnJR48eVWxsrCoqKuT3f/sJPjt37lSrVq0kSdu3b9d5550XidEAAACAyJTk7z4Je8eOHWrTpo0kadu2baFi/P2vZ86cqczMTGVkZGjQoEH66quvIjEyAAAA6hDH/a6xGpgxY4ZWr16tXbt26bzzztO+ffvkOI6++OILtW3bVk2bNlVsbKw2bdqkjh076t5779XQoUO1cuVKxcXFacmSJcrLy9Odd955yiw3f7fBdyQpOsYmx+jzy+UGbXKsvh9JwXXvm+REXZxqkqNjR21yfD6bHH99m5xjxTY5DeJtcsqP2+RY7eMku/2P1fdUVmqTY/Vz8/ltcgIVNjkxDWxyrPY9kuSLtsmJMnp9MOoKTuPmlW43+ml+69Zbb1WrVq10/Phx9erVSxMmTNCwYcPUokULjRkzRk899ZR8Pp9GjRqlP/7xjyovL5fjOJo9e7Z69+6tzMxMy3EBAABQR5mfbrFp0yZ17txZkrR37161aNFC0rdv3PP5fNqzZ09om9/v14wZM5Sbm6v09HQNGTJEW7ZssR4ZAAAAdYzp6RYDBw7Ujh071LRpU0nS/v37lZiYKJ/Pp8LCQiUlJamkpETBYFBnnXWWZs+eHbpvWVmZXn75Za1atUqzZs06ZRanW1QRp1tUGadbVBGnW1QNp1tUHadbVA2nW1QNp1tUXYRPtzA9kjx37lz16NFD2dnZeumllzRs2DAtX75cY8eO1e9+9ztlZ2drwIABevPNNzV79mxt2bJF99xzj8rKyhQTE6NOnTrJcRzLkQEAAFAHmZ6T/M0334RWs8jJyVFKSookad26dRo0aJAkKS8vL3Sb9u3bq1WrVurdu7f8fr9iY2P1yCOPWI4MAACAOsj0dAtLnG5RRZxuUWWcblFFnG5RNZxuUXWcblE1nG5RNZxuUXV16XQLAAAA4ExASQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4mH6YiKkKo3UWrdZ4NVsn2WjZbKt1PSU5TVvZBAUDNjkN4mxySg7b5NSLtcmxWrO2vtH3E220Zq3lGq9W69Za7U+tcozWL3YL95nkOPUamuSovtG+1GrtdMlujekoo+d2qeH+pxIcSQYAAAA8KMkAAACAByUZAAAA8KAkAwAAAB6UZAAAAMCDkgwAAAB4UJIBAAAAD0oyAAAA4EFJBgAAADwoyQAAAIBHWD+Wet68eXrhhRckSZ07d9YTTzyh5cuX6/nnn1dFRYWaNWumcePGqU2bNpoyZYoOHjyovXv3atOmTbriiit03XXXacqUKdq/f78ef/xx9ejRQ2VlZZo4caJWr16t8vJyDR48WHfddVc4xwYAAABOELaSnJeXpwkTJmjevHlq1qyZ7r77bv3v//6vZsyYoTlz5qht27Z65ZVX9Mgjj2jGjBmSpA8//FBz585VVFSUfv7znysuLk5z587VzJkzNX36dPXo0UPTp0/Xtm3btGDBAlVUVOjmm2/W+eefrx49eoRrdAAAAOAEYTvdYs2aNerSpYuaN28ux3H07LPPqkmTJkpJSVHbtm0lSb/85S+1du1aVVRUSJK6dOmi5ORkJSYmqmnTpvr5z38uSWrfvr32798vSVq5cqWGDBmimJgYNWzYUP369dP7778frrEBAACAk4TtSHJBQYESEhJCl+vVq6fDhw+fsC0+Pl6u66qgoECSFBsbG7rO5/OpYcOGkqSoqCgFg0FJ0pEjRzR+/Hj98Y9/lCSVlZWpc+fO4RobAAAAOEnYSnJiYqLWr18fulxcXCzp/2PvzgNjuvr/gb9nJvtKVpHN1oSESGwRWyyxJLTWVhW19KmlnkfV2qIoVUsppVWqQq1FqkLFnkoEQRJiJ7EmIUtNZBHJJJnz+6PfzI+pliQzN5b36x9mcud8zr1z5sxnzj33XODBgwea53JyciCXy1G9evXnLtfBwQHDhw/n9AoiIiIikozOplsEBgYiISEBqampEEJg5syZUKlUiIuLQ0pKCgDgl19+QevWrWFg8Py5eadOnbB9+3aUlpZCCIEVK1YgOjpaV9UmIiIiIvobnY0k16hRA7Nnz8aQIUOgUCjQqFEjfPjhh3jjjTfw0Ucfobi4GC4uLpgzZ065yn3vvfeQmpqK7t27QwiBhg0bYsiQIbqqNhERERHR38iEEKKqK6EPIvO2NIFMLaSJI5NoSWt1qTRxIF2zEylXJIkjc6orSRwodLpy4z8ryJUmjnk1aeI8SJcmjpWdNHGkUvhQulhGpq9WHNUjaeLIFZKEEQ8yJIkjMzaTJA4sbaWJI9TSxAGA0hJp4sglykke5UsSRuZY+6nP82YiRERERERamCQTEREREWlhkkxEREREpIVJMhERERGRFibJRERERERamCQTEREREWlhkkxEREREpEWiBVergKFRVddAt2QyaeJItfahVOs+A9IdO6nWfpZqHcziQmnilBZLEkaUSrMGuEyqtcYlWhsXRibSxAEAQ2Np4kjVJ0i1Pq5EYWQW0qxpLvKyJYkjM7eWJA4g1XcQpFubW6p+oYpzOY4kExERERFpYZJMRERERKSFSTIRERERkRYmyUREREREWpgkExERERFpYZJMRERERKSFSTIRERERkRYmyUREREREWpgkExERERFpeWGS5MWLF2PLli0AgKNHj+Lu3bsAgOXLl2PatGlVWTUiIiIies28MEnyhAkTMGDAAADAunXrNEkyEREREZHUDKQKFBgYiPXr18Pd3R0RERGYPHkyTp8+DVNTU6xduxYnT56Ej48PVCoVYmNjcePGDUyaNAkAoFKpMH78eJw9exZ2dnZYvnw5HB0dpao6EREREb1mJBtJ9vf3x5kzZwAAp0+fhre3N86dOwcAiIuLQ3x8PABg3LhxcHR0xNdff42QkBAAwIkTJzBhwgRERkbCxsYGYWFhUlWbiIiIiF5DkibJZ8+eBQAkJiaiX79+SEhI0Dxu0aLFP762adOmcHZ2BgDUr18fGRkZ+q8wEREREb22JE+Sc3JyYGhoiJYtWyIhIQHXr1+Hk7JutCgAACAASURBVJMTLC0t//G1FhYWmv8rFAqUlpZKUWUiIiIiek1JliS7uLigoKAAR48eha+vL1xdXZGamor4+HgEBARIVQ0iIiIiomeSdHWLpk2bYv369WjSpAkAoE6dOvj111//liQbGBggLy9PyqoREREREWlImiT7+/sjMTERfn5+AAA/Pz9cunRJkzSX6dq1K8aPH4+1a9dKWT0iIiIiIgCATAghqroS+iCy70kTyMBImjhyhTRxhFqaODLpfp+J2xckiSOrUVuSOIBMmjD5SmniWNpKEkY8yJQkjszaTpI4kvUJagmvATE0kSaOVMeuMF+aOFL1p+oSScKIvGxJ4shsa0oSR7I+G5CuzRlJ9FmVqP+R2Tg/9fkX5mYiREREREQvCibJRERERERamCQTEREREWlhkkxEREREpIVJMhERERGRFibJRERERERaDKq6Anojk2bJFZniFTuEMmmWRpJy5UGZm5dUkSSJog5bLkkcecgQSeJAYShJGJmZlSRxYGQqTZyiAmniSLUsJF69/lRI1RYkWtJOJtH3KkwsJAmjTjgkSRyZvYskcQBA5uguTSCploZUS9f/PA1HkomIiIiItDBJJiIiIiLSwiSZiIiIiEgLk2QiIiIiIi1MkomIiIiItDBJJiIiIiLSwiSZiIiIiEgLk2QiIiIiIi1MkomIiIiItOg1SZ44cSICAwPh6emJixcv6qTMadOmYflyae44RkRERESvJ73eA3TPnj3Yv38/3Nzc9BmGiIiIiEin9DaSPHjwYKjVanzwwQcICAhAXFwcQkNDMWrUKM02w4cPx6ZNmwAAhw4dwptvvolOnTph+PDhUCqVAIDs7GwMHz4cHTt2xIgRI5CXl6evKhMRERERAdBjkrxhwwbNv6ampgCAIUOGIDMzEzExMTh06BAePnyIAQMGICUlBZMnT8bixYtx+PBh+Pv7Y9asWQCA1atXo3r16oiMjMSMGTMQExOjryoTEREREQHQ83QLbQqFAnPmzMGnn36KkpISfPvtt5DL5YiOjkaLFi3g4eEBAHj33XfRunVrlJaWIi4uDiNGjAAAuLi4oEWLFlJWmYiIiIheQ5ImyQDg7e0Nc3NzKBQKTVKcl5eHuLg4dOvWTbOdhYUFHjx4gJycHFhaWmqet7KykrrKRERERPSakTxJPnLkCAwMDFBUVISoqCgEBgbCwcEBrVq1wrJly/62vZWV1RPzkJVKJVxdXaWsMhERERG9ZiRdJ7mgoABz587F559/js8//xxffPEFCgoK0KZNG8TFxSElJQUAcO7cOXz55ZcAAF9fXxw6dAgAcOfOHcTHx0tZZSIiIiJ6DUk6krx8+XK0b98enp6eAICAgAAsXboUU6dOxZw5czBmzBgUFxfD3NwcU6dOBQCMHDkSn3zyCTp27Ii6deuiS5cuUlaZiIiIiF5DMiGEqOpK6IN4kC5JHJmRqSRxXjWSNjuhliiQTJIo6jBpbqYjDxkiSRyYmEsT55FEy0eaW0sTp6hAmjiSfX4AmZlEx04iorREmkByhSRhZDJp+jihLpUkjjrhkCRxZPYuksQBAJmju2SxJKGWpv+R2T19Gi9vS01EREREpIVJMhERERGRFibJRERERERamCQTEREREWlhkkxEREREpIVJMhERERGRlld3CbiHDySJI9WSOFRxQqolZOTS/Obc79ZAkjhdkhMkiQMDI2niyKR5fyRbJusV7Lpftf70Vet7pCJZ2y5RSRJGfXCTJHEAQN5lsDSBFBLdZkOitiCzqP7U51+tTxYRERERkQ4wSSYiIiIi0sIkmYiIiIhIC5NkIiIiIiItTJKJiIiIiLQwSSYiIiIi0sIkmYiIiIhIC5NkIiIiIiItTJKJiIiIiLRUeZLs6emJ9PT0vz1/8OBBfPbZZ1VQIyIiIiJ63Ul0X8Hy69y5Mzp37lzV1SAiIiKi19BzjSSvXLkSAQEB6Nu3LzZt2oSOHTti+fLlmD59Ovr164d169ZBrVbjiy++QNeuXdGxY0dMmjQJxcXFAIDBgwdj+fLleOedd9CqVStMnz4dpaWlmvKjoqLQp08ftGnTBqGhoQCAHTt2YOjQoQAApVKJUaNGoVOnTnjzzTcRExOj48NARERERPT/PXMkOSkpCT/99BMiIiJgbW2N//znP5q/RUVFITw8HDY2Nti/fz/i4uLw+++/Q61Wo3fv3oiIiEDPnj0BANHR0Vi/fj0AoFevXvjjjz8QFBQEAEhLS8OOHTtw/vx5DBw4EIMHD36iDosXL0bdunWxcuVKXLp0CcOGDcPRo0dhZGSkswNBRERERFTmmSPJp0+fRosWLeDg4ABjY2P07dtX87fGjRvDxsYGANC1a1f8+uuvMDQ0hLGxMRo1aoSUlBTNtt27d4epqSlMTU3Rtm1bnDlzRvO3t956CwDg5eWFoqIiZGdnP1GHqKgo9OjRQ7PN4cOHmSATERERkd48cyQ5NzcX1tbWmseOjo6a/z/+vFKpxJw5c3Dp0iXIZDL8+eefGDJkyFO3tba2RmZmpuaxhYUFAEChUAAA1Gr1E3V48OABLC0t/7Y9EREREZE+PHMk2cLCAgUFBZrHjye3j1uyZAkMDAywe/du7Nu3D4GBgU/8/fHR4ZycnCeS5mepVq3aE69PTU3VzHcmIiIiItK1ZybJPj4+OHnyJJRKJVQqFXbu3PnU7e7fvw8PDw8YGRnhypUrOHPmzBPJ9cGDB6FSqVBQUIDo6Gg0a9bsuSvZsWNH/PbbbwCA5ORk9OnT54kL/4iIiIiIdOm5kuTevXujd+/eeP/999GhQ4enbjd8+HD88ssvCA4OxqZNmzBlyhRs374de/fuBQD4+fnh/fffR8eOHeHv74927do9dyUnTZqE9PR0dOzYEZ988gkWLVoEExOT5349EREREVF5yIQQ4lkbCSEgk8kAAEeOHMHSpUv/cUT5aQYPHox+/fppVrqQgnj4QJI4ZceFXlxCa467vsjk0tybZ79bA0nidElOkCQODCS6CFcmzfsjVZ/wHF33S+dV609ftb5HKpK17RKVJGHUBzdJEgcA5F0GP3sjXVBIdJsNidqCzKL6U59/5idLqVSiZcuWSEtLgxACe/fuha+vr84rSERERET0onjmTwEbGxuMGzcOQ4cOhUwmQ506dTB58mQp6kZEREREVCWea7rFy4jTLajMq3bKk9MtKojTLV54r1p/+qr1PVLhdIuK43SLiqnwdAsiIiIiotcNk2QiIiIiIi1MkomIiIiItDBJJiIiIiLS8speuIeCnKquAb0gxKM8aQIZm0sSRh22XJI4aNBEkjDyBv6SxEFxkTRxTCwkCSPZBYKlJZLEAQCZVBcDSUQUFTx7I10wNJYmjkQXv0r2WVU9erXiAFCfPy5JHHm7vpLEgVqauyvLLG2f+jxHkomIiIiItDBJJiIiIiLSwiSZiIiIiEgLk2QiIiIiIi1MkomIiIiItDBJJiIiIiLSwiSZiIiIiEgLk2QiIiIiIi0vbJIcHh6OwYMHV3U1iIiIiOg19MImyUREREREVUWn9wDduXMnfvjhBwCAj48P5s6di/DwcISGhqK0tBT29vZYuHAhnJ2dsWPHDkRGRiIvLw/e3t6YOHEivvzyS0RGRsLOzg7NmzfXlHvq1CnMmzcPRUVFEEJg7NixCA4O1mXViYiIiIg0dDaSnJqaigULFmD9+vXYt28fHj16hHXr1mH27NlYu3YtDhw4ADc3N6xYsULzmmPHjuGLL77A5MmTcfToURw7dgx79uzBxo0bERcXp9luwYIF+OyzzxAREYEffvgBhw4d0lW1iYiIiIj+RmdJ8rFjx+Dn5wdHR0fIZDIsXrwYw4cPR3x8PGrUqAEAaNasGVJSUjSvqVWrFmrVqgUAOH36NAIDA2Fubg4TE5MnRoptbW2xc+dOXL9+HbVq1cLixYt1VW0iIiIior/RWZKcnZ0NKysrzWNjY2PIZDIsW7YMISEh6Nq1K5YsWQIhhGYba2trzf9zcnJgaWmpefx4WV999RVMTU0xbNgwdOnSBfv27dNVtYmIiIiI/kZnSXL16tWRnZ2teZyfn49du3YhMjISGzduxP79+zF27Nh/fL2VlRXy8vI0j5VKpeb/dnZ2+PzzzxEdHY0ZM2bgs88+w8OHD3VVdSIiIiKiJ+gsSQ4MDERCQgJSU1MhhMDMmTORkZEBZ2dn2NjYIDs7G3v37v3H5NbPzw8xMTF49OgRHj16pBktLi4uxuDBg5GZmQkA8Pb2hoGBAeRyLsxBRERERPqhs9UtatSogdmzZ2PIkCFQKBRo1KgR3nzzTRw+fBidO3eGq6srxo0bh9GjR2P+/Pnw8PB44vUdOnTAkSNH0K1bN9jZ2SEwMBBxcXEwNDREv379MHToUACAXC7H9OnTYWpqqquqExERERE9QSYenyT8KinIqeoa0AtCPMp79ka6YGwuSRh12HJJ4qBBE0nCyBv4SxIHxUXSxDGxkCSMTCaTJI4oLZEkDgDIFDpdlbTKiaICaQIZGksTRybRGVypPquqR69WHADq88cliSNv11eSOFCXShJGZmn71Oc5Z4GIiIiISAuTZCIiIiIiLUySiYiIiIi0MEkmIiIiItLCJJmIiIiISAuTZCIiIiIiLUySiYiIiIi0cJ3kShIPMiWJAwNDaeJI1RykWm8TAEpU0sSxevo6izon1ZqbConanIGRNHGEWpIwMrlCkjhConYgbl+SJA4AwKKaJGFkNk6SxJGMVOskS7VmtkSfIRQXShNHqj4OkK7flmgNcHE/TZI48jeaP/15SaITEREREb1EmCQTEREREWlhkkxEREREpIVJMhERERGRFibJRERERERamCQTEREREWlhkkxEREREpIVJMhERERGRlpciSY6JiUG/fv0QHByMnj174vDhw1VdJSIiIiJ6hRlUdQWe5e7du5g8eTLWrVsHDw8PXLt2DYMGDUJ4eDicnF6xuyYRERER0QvhhR9JTk9Px+jRo+Hh4QEA8PDwgJubGy5evFjFNSMiIiKiV9ULP5LcpEkTNGnSRPNYqVTi5s2bqFmzZhXWioiIiIheZS/8SPLjsrKy8J///AdvvfUWvLy8qro6RERERPSKeuFHksuUlJRg2LBhePPNNzFy5Miqrg4RERERvcJempHkM2fOoFq1akyQiYiIiEjvXpokuXnz5ti4cWNVV4OIiIiIXgMvTZJ88OBBfPbZZ1VdDSIiIiJ6Dbw0c5I7d+6Mzp07V3U1iIiIiOg18NKMJBMRERERSYVJMhERERGRFibJRERERERamCQTEREREWlhkkxEREREpIVJMhERERGRFibJRERERERaXpp1kstLFD6UJpCphTRxhFqaODKJfjepS6WJAwByE2niSLVPBkbSxCktkSaOVG27RCVJGKEwlCQOcrIkCSOzrSlJHACAkal0saQgWdsuliZOqURxFBKlJiUS9XFS9dmAdP12frYkYWSmlpLE+SccSSYiIiIi0sIkmYiIiIhIC5NkIiIiIiItTJKJiIiIiLQwSSYiIiIi0sIkmYiIiIhIC5NkIiIiIiItTJKJiIiIiLToZMXu1NRUdO3aFa6urpg7dy5+//13HD9+HDKZDEII9OrVC6NHjwYADB48GDdv3oSFxZM34Rg0aBAGDRr0xN+FEFAoFOjfvz+GDBmCpUuXYufOnXB1dcWGDRt0UXUiIiIior/R2W1tHB0dsW/fPixatAi5ubnYvXs3jIyM8Oeff2LgwIFwc3ND9+7dAQCTJk1Cz549/7Gsx/+elZWF/v37o3bt2hg3bhwCAgLw3Xff6araRERERER/o/PpFteuXYO3tzeMjP66DaOdnR02b96Mzp07V6g8e3t7dOvWDceOHdNlNYmIiIiI/pHOb5AeGBiIRYsWIScnB+3bt0ejRo1ga2tbqTJLSkpgbGysoxoSEREREf07nSfJAwcOhI2NDX799VesW7cOhoaG6NOnDyZMmKBJdL/++mv88MMPT7xu9erVcHV1/Vt5KSkp2LdvH6dYEBEREZFkdJ4kA0BwcDCCg4OhUqlw4sQJfPnllzA2NsaECRMAPHtOclkSLYSAlZUVPv30U/j4+OijqkREREREf6PTJLm4uBhRUVHo0KEDFAoFjIyMEBgYiDt37iAmJua5y3lWEk1EREREpE86vXDPwMAAS5YswcqVK1FaWgoAyM/PR2RkJJo3b67LUEREREREeqPTkWSZTIbVq1dj4cKFCA4OhkwmAwC89dZbGDZsmGa7p81J9vHxwcKFC3VZHSIiIiKiCtH5nOSaNWti6dKl//j3Z90EhDcJISIiIqKqxttSExERERFp0VmSnJGRgW7duuHixYu6KvJvli5diilTpuitfCIiIiIiAJAJIURVV0IfhPKuNIH+b9613gm1NHFkEp1cUJdKEwcApGriRibSxJFKaYk0cQwlulFQiUqaOApDaeJI1cdJtT8AYGQqTRxDI2niSNVvy/WymuvflRZLE0ch0f6oiqSJY2ImTRwAgEQ5yYMMaeLIFZKEkbnUf3p4SaITEREREb1EmCQTEREREWlhkkxEREREpIVJMhERERGRllf2wj0iIiIiooriSDIRERERkRYmyUREREREWpgkExERERFpYZJMRERERKSFSTIRERERkRYmyUREREREWpgkExERERFpYZL8D/S1fLRardb8v7i4WC8xiIhedWV9dGlpKYAn+1YifeLtJV4fTJIfExERgd9//x0AIJPJoFardf5huHTpEu7du4dt27bh3LlzOiv3dfnQ3r17F3fv3q3qalTIwYMHcfr06aquxhP02W5elzb5IsrPz8fhw4fx1Vdf4ejRoygqKpK8DrGxsXotXyaTISUlBQsWLAAAyOX6/zpjm9aNl/U4njhxAvn5+ZDJZACqbj9e5H77Vet7FLNmzZqls9JeYiqVCv3798fevXuxf/9++Pj4wNHRETKZDKWlpTrpgIUQSElJwciRI7F79268++67cHR01Pyt7INXEWWvjY6OxqlTp+Dh4QGFQlHpOpdHZffheUyePBmXL1+GiYkJrKysYGJiotd4gO72a9SoUfDy8sLly5dhbW0NU1NTyGQyvR+zx2nviz5j66rsss9faWkpZDIZCgsLYWhoqJOyK0oXbUKfn5epU6fi9u3buH//Pvbt24eMjAy0atVKL7GeZvPmzZg1axbOnz+P6tWrw8XFRecxhBB48OABlixZgtu3b8PX1xcmJiY6Oa5qtRoymQxKpRLZ2dlIS0uDtbW1pk+Voq/TVtmYVVHnMsXFxVCr1cjJydH0e7rw+D5lZGTg7t27MDExgZGRkU7Kf1xcXBx++OEHZGRkQC6Xw9nZ+YlkWZ/H9mXqt1+1vodJ8v+5ceMGWrVqhcWLFyMjIwNTp07F1atX0bZtW00iVtZxVpRMJkPNmjWRkZGBvLw8hIaGIi8vD02bNtV86efn55f7A15cXAyFQoFdu3Zh9erVcHBwQPPmzfHo0SMUFxfrpcMoo1arcenSJdjb20vSAdeoUQNnz55FVFQUsrOzYWJigmrVqsHAwEBnMVJTU3Hz5k3cvn0bAGBtbV3pMktKSgAAW7ZsQXh4ON59911Uq1ZNc8ZCimOnVqshl8uRm5uLHTt2YNasWejYsSPMzc11Un5ZMnv+/HkcOHAA2dnZqFWrlubvFf0iKfuBOmPGDOzZswdnzpxBcXExateurffjVrZPsbGxiIqKQmxsLHx8fHSSLJW99rfffsPx48eRl5f3xPGqaLtISEjAjh078NNPPyEkJAQNGzbEqlWrEBAQABsbmwrVtTxKS0uxbt06jB49GtWqVcMvv/yCxMRE1KlTRyefpTIymQzVqlWDv78/Tp06BRMTE9SpU0cnCXLZ5+TTTz/F9u3bcfXqVRw4cABGRkY6ifFv9NXm9NXenifu/PnzsWbNGly5cgWRkZFo1KiRTvqdsvpu2rQJs2fPxvnz53H+/HnIZLIn9k0XrK2tYWRkhKSkJMTExODBgwewsLCAjY2NXtvDy9Rvv4p9D5Nk/NUITUxMkJOTA2dnZwQEBOD999/HwYMHMX36dKjVavj7+1fqV+Pjr2ncuDGGDRuGFi1aYM2aNfjpp5/g4uKCCxcuIDY2Fk2bNi1X2WWd56effooZM2bgzTffRHh4OJYvX47vv/8e9vb2eOONN8pV5r8p+1BFRUXhyy+/xKFDh/Ddd9+hqKgIXl5eMDQ01Nsv65o1a6JLly4wMTHBoUOHEB8fD5VKBXNzc1hbW1c6ZnJyMqZOnYojR45AqVRiy5YtaNq0KapXr16pcuVyOdzd3bFjxw44OTnh5MmTUCqVaNy4sSYJ1NUZi38jk8kwZcoUPHz4EN27d4efnx+Av86kVObMgxACcrkcarUaQ4cOhaenJxo3bgwbGxvcunVL84OgvEpKSiCXy7Fv3z7s3bsXQUFByM3NRUJCAhITE2FpaYkaNWpUuN7/pmyfUlJSMGjQINSsWRORkZFYvHgxzMzM0Lhx4wq3t9LSUigUCly/fh0zZ85E+/bt0aFDB6hUKpw6dQrVq1ev8FmS+Ph4WFtbo2XLllCpVKhZsyb27NkDLy8vuLm56f1HmVwuh5+fHxo1agQfHx/Uq1cPN27cwNatW5GVlYVGjRrp9EetnZ0dhBCYM2cODAwM0Lhx40qVV/Y5nDVrFtzd3TFjxgw4OztDpVLh2LFjMDMz03kCVkZfbU6f7e1ZvvrqK2RlZWHixIkQQmD37t0YNGgQlEolLCwsKvxdcePGDaSlpcHBwQFz587F5MmTUbduXSiVSpw6dQoXL16EpaWl5mxtZRQXF2uOT1RUFDIyMnDjxg3cuXMHubm5cHBwgKmpaaXj/JOXpd9+FfseJskA5s+fj59//lmTDJeWlsLExATdu3dH69atsWrVKqxYsQKurq6oW7duhaZglDWMP/74AxEREbCwsICfnx8GDx4MIyMjzJs3D8eOHcOAAQPK1QH/9NNP8PHxQVpaGq5evYoWLVogISEBc+bMwccffww/Pz/s3bsXLVq00Nkvz7L9/uSTT9C/f39Mnz4dycnJSE1NRY8ePZCRkQFLS0udxCpTdrwfPHgAtVoNDw8P9OjRA7m5ufj999+RnJyMoqIi2NvbV6qzmjp1KoKCgjBnzhxkZmbi8uXLGDFiBOLi4lCzZs1K1b20tBT29vawt7dHw4YNERcXh507d8Lc3By1a9fWW4Jc9iUkk8lw8+ZNbNy4EatWrYKnp6dmm8WLF+P8+fNo1qxZhWKU7eM333yDatWqYdKkSSguLsakSZOwevVq7NixA126dIGZmVm5yi07Jhs2bMCYMWMQFBSEevXqwcDAACkpKThx4gSuX78OPz8/nU8vKvvMhoeHw9/fH2PHjsV7772H2rVrY+HChTh//jw6duxYoYSvbL+mTp2K/v37o3fv3jh+/DimTZuG+Ph4hIWFoX379rCwsCh32QUFBVi0aBGaNGkCZ2dnAH8lFLGxsQgODtbrl1RWVhZ27tyJvXv34tq1a5DL5fD19UXDhg3h7OyM2NhYhIWFQaFQwMPDo8JxDh06hEmTJuH+/ftQKpXw8PBAp06dEB4ejlq1asHBwaFCyVd+fj5MTEzw8OFD/Pzzzxg7diycnJzg4uKCOnXq4ObNm4iNjUVQUJBeprPpq83ps739m/z8fPz888+YNm0a6tatiw0bNsDPzw8tW7bE2rVr4evrW+GpU+vWrcOuXbuQlJQEW1tbDBw4EPXq1UPdunUhl8tx584d7N+/Hy4uLhXuu8uUvdejRo3C6NGjMWHCBE17iIyMxPXr16FSqXR6dutl7Ldfxb7ntU+SDx8+jJ07d2LmzJlo1aoVTE1NkZ+fj1OnTgEA6tevj4EDB6J69eqYNWsWDh06hKZNm8LW1va5Y5Q1xJ9//hnr1q1DUVGRZlqHh4cH2rRpg2HDhqFnz55o2LDhc5eblZWFa9euwcfHB1u2bEFhYSFOnDiBS5cu4c0330Tfvn1ha2uLDRs2YODAgTqdx3nlyhWcOnUKn332GQoLC7Fy5UpMmTIFDx48wLp16xAQEKCz0aKyX7tCCEyYMAGhoaFITExESUkJevbsiaCgIJw/fx6HDh1CSEhIhZNkpVKJffv2oUePHqhRowbmz5+PQYMGwc3NDd9//z08PT3LfcqmrO4AMHz4cNy/fx9RUVGIiIiAr68vfH19sXv3bhw/fhzt2rXTyxevSqXSvBdlp9XK5mqVdcSGhoaIj49H+/btKzUl4ujRo2jbti3y8vIwe/Zs2NnZITQ0FElJSTAwMEC9evWeu8zVq1ejadOmSExMREREBJKSkhAQEAA7Ozs0aNAALi4uyM3NRY0aNcr1uSmPy5cvY8WKFWjWrBm8vLwAAPXq1UNISAgOHToEX1/fcp9luH//PszMzKBSqRAfHw8nJycolUp88cUX6NOnD6ZMmYK7d+/CwsICbm5u5a5zjRo14ObmBnt7e00/5ezsjCNHjqBLly64ffu23k59jhs3DkIICCFQXFyMs2fPwtfXFzY2NnB1dUXjxo1hbGyMiIgI/PHHH3BwcICTk9Nzlf140puamoq6devi9OnTuHXrFkJDQ7F//37cuHED0dHRaNasGezt7ctV9/z8fPTs2RNZWVlo3749srOzER8fjwYNGsDExATm5uZo1aoVNm/ejBYtWuh06sjjdN3m9N3e/okQAsbGxjh9+jTu37+PoqIi7Ny5E99++y0UCgW+/fZb1KtXr8IJbNn7m5WVhf379wMAvLy8UL16dXh5ecHBwQFmZmbo2rWrTpKz1NRUREREIDAwEDVr1oSTkxPatWuHtLQ0HDhwAI0bN4a3t3el45R5GfvtV7Hvee2T5IkTJ2Ls2LFo3rw5TE1NERsbi88//xx79+7FmjVrIIRA8+bN0aBBA4wYMQKRkZH45ptv8OGHHz73yJ9cLodKpcKsWbOw/4Pd9QAAIABJREFUcOFCDB06FJmZmdizZw82bNgApVKJlStXon///uVKZM3NzeHn54fLly9j9erVuHHjBurWrYtFixZpTsf897//Rbdu3dC8efMKHZ9/YmhoiNDQUJiZmeHAgQNwcnJCnz598PDhQ2zevBm9evXSWVJe1iHMmzcPpaWlmDJlCpKTkxEWFoZbt27Bzc0NNWrUQOvWrcuVhGkzNTVFcnIyrl27hkuXLiEjIwMTJ06EgYEBli9fjqCgoAp9wGUyGZYsWaKZm/f++++jevXqWLNmDRQKBbp27QoXFxeddrCPW7FiBYyNjWFlZQVLS0skJydj9erVsLa2Rp06daBQKPDHH3/g/Pnz6NmzZ7nKLi4uxoQJE6BQKFC3bl0AwBdffIGrV6+iXr16mD17NgwMDLBp0ya4u7trvvSfJSsrC1evXkXDhg0xa9Ys+Pr6IicnB3FxcSgpKUHdunVha2uL5s2bo379+nobobh9+zZOnjyJ3377DZaWlvDx8QEAWFlZYc6cOejUqVO5vuTz8/MRFhYGMzMzODo6oqSkBNOnT0dWVhZ69uyJ999/H0ZGRliwYAG6detW4akktWrVgp2dHYC/fqTb2NhovgR/+ukndOjQoULl/ps//vgDx48fx7Jly9C+fXvUqlULYWFhyMnJQYsWLSCXy2FhYYF69erBy8sLQgjMmzcPHh4ecHV1fWb5JSUlUCgUiI2NRUFBAWrWrIkPPvgAHTt2xJAhQ+Dq6ooWLVrA1NQUx44d0/Tpz6ukpATOzs44fPgwfv75ZxgZGSElJQUPHjyAiYkJCgoKcPDgQVy9ehXDhg2rzKH6V7psc1K1t6cp+0waGBhg27Zt2LNnD9555x34+fnhyJEjiI2Nxbhx48pdrlqtRlZWFlxdXfHo0SO4u7ujWrVquHLlChISEqBQKODu7g4HBwf4+PjorG+wsrLC3bt3cfLkSbi7u8PU1BSGhobw9fVFfHw8RowYodMpKy9jvw28en2P7iaGvYSuXr0KOzs7tG/fHgBw8eJFjB8/HiEhIXj77beRkpKC33//XXPRlYGBAby9vdGyZctyj/gplUpYW1ujevXquHnzJhISEhAfH4/Zs2fjypUrePPNNys0AiqTydCwYUP88MMPOHjwIMLCwjBhwgTUrl0b2dnZcHZ21kuHbm1tjalTp+LHH3/EuXPnsGrVKgDAokWLEBgYqNP5WXK5HPn5+bh48SLmzZsHNzc35OXloUGDBkhPT8cnn3yCR48e4fDhwxWOUXbBZM+ePTF9+nRcvXoVEydOxIULF7B7927UrVu3Qgl42dScP//8E++++y4MDAygVqvRv39/+Pn5YeXKlWjXrp3efl2rVCoEBATA19cXH3/8Mby8vDBmzBjUqVMHs2fPxpYtW+Dk5IRr165hyZIlAP7//MXnkZ6eDkdHR2zZsgVHjx7FhAkTEBYWpplaUrYcUH5+Pvr27fvc9ba3t8cHH3yAy5cv488//8Thw4c1o0NHjx7FmTNn0LFjRzRr1kznCfLjI5bNmjXDqlWrcODAAYSFhWHr1q146623YGBggH79+qFp06blOl4WFhZ44403kJSUhJs3b8LIyAhnz56FUqmEjY0N4uLisG/fPnh4eFRqbu3jP+DL6latWjWo1Wr873//q3C5/2b58uUYO3YsgL/anaurKyZNmoSdO3c+cYyMjIzg4eGBoqIiGBkZPfc1GIaGhjh//jw+/vhjNGzYEEZGRqhRowY6d+6MVq1aoWPHjhBCIDg4GFOmTMHJkycRHBz8XNMuhBAwMzNDcHAw7t69iz179uDs2bNIT09HYmIikpKS8PDhQxQUFGDGjBmVO1D/EF8fbU6q9vY0x48fR2ZmJrp06QIbGxt88803OHXqFI4fP47CwkJMnDixQuXm5ORgx44duHTpEvLz8xEaGormzZsjJiZGc0o9KioK7733HurUqVOpfXj8fSkpKcGHH36I2bNnY/LkyejVqxdsbGzw+++/o379+jo9s/Cy9tvAq9f3vNYjySYmJti1axcuX76smR/q7u6O+fPnw87ODs7OzggLC0Pjxo1hZ2eHkpIS2NvbV+iXkImJCU6cOAF/f3+Eh4fDxsYGgYGBEELAz88Pb731VoX3QyaTwdLSEg0bNkSzZs2QnJyMo0ePQqVSYf78+ToZ0S2bcJ+amoqtW7ciLS0NtWrVQoMGDeDk5IS1a9di//79MDc31/mXSNkc8SNHjuDRo0cwMjLCL7/8gvXr18PT0xNmZmaYOXMmrKysKlR+Xl4evvrqK/z2228YMGAAevfuDQMDA5w8eRJnz56FoaEhPv300wqPEsjlcly7dg0rVqxA48aNNaM1tra2WLt2LerUqaPT05yPUygUcHZ2xsOHD6FUKhEfH4/w8HC0a9cOM2bMgKWlJRo3boyePXvCw8MDarW6XD8Ara2t4efnh+zsbOzfvx979uxBUVERGjVqBFNTU3z88ce4d+8eRowYUe7TqjKZDHZ2dmjXrh0MDQ1x8uRJXL9+HbVr1wYApKWlISAgQKdJctmV5CqVCtu3b8fKlSuhUqng4OCA4OBgqFQqbNy4Effu3cOYMWNgZ2enmQr0rHqcO3dOM6c0NTUVX3/9NczMzNC6dWvcv38fRkZGWL9+PQBg0qRJermQytTUVGfXJjxu165diIiIQK9evWBkZKS5JmHBggVwc3ODv7//345RWloaWrVq9cxEJi8vDytWrECNGjUQFhaGgQMHYvz48VAoFLh37x5OnTqFK1euaC7SMjAwwPXr15GYmIjOnTs/V/soe98XLFiAlJQUfPrppwgICIC9vT3S09NRUlKCjz76CB9++GG5p3E8b2xdt7mqbG9r1qzBt99+i8zMTPz4449wc3PDtGnTULt2bTRv3hzdunVDo0aNyl1uXl4eLC0tYWxsjNDQUCiVSjg5OcHZ2RleXl5o0KAB1Go17ty5A39//0olrmXTJNPS0rBp0yZs3boVCQkJGDduHGxtbREdHY20tDSYmZlh+vTpOu2HXuZ++5+8jH0PAMjEy7qqt46UXUgXGxuL0aNHo3PnzppOMDw8HFu3bsXmzZsrFaNsjUhjY2MAf63jd+TIEfz444/o168fhg4dih49elR6X8oUFhbi+PHjuHTpEv773/9WuryyThwABg0aBIVCgYyMDDRq1AitW7eGl5cX3N3dkZOTA1tbW53Nqy37BZiTkwNra2vcvXsX5ubmiImJwe7du7Fy5UocPnxYs3B5RU2fPh0mJiaYPHky1Go1Tp48CQCoU6cOatSoAblcrpN9+uabb5Camor69eujQ4cOSElJwXfffYcdO3ZUuuynKTt+xcXFePjwIapVq4Zbt24hIiICJ06cgKurKwYMGFChLyvgr5EVAwMDbNiwAadPn0bHjh3x4MEDxMfHIycnBx988AHatWunky+PkpISJCcnIzw8HNHR0TAyMsLSpUvh7u5e6bIfV9bWZ86cicLCQtSoUQP37t3T/Jjt168fLly4gB07duD06dMICgrSJGzPkpKSgs2bN6NLly64desWTp06pbm4Zfv27Vi/fj3c3d2f+Ly9LDIyMvDtt9/i7t278Pb2Ru/evXHnzh2sWLECYWFhAFDh/Tpx4gQ2bNgAW1tb5ObmYsiQIWjSpAkAaC7ejImJgaenJ8aMGQMhBLZv3665Uv95qVQqjBgxApMmTYK3tzfUajUePHiAsLAwrFmzBnZ2dti5c6fO1+jWV5urqvYmhMBXX32FPn36oG7dujh48CDWrVsHABg5ciSCgoIqXPZ3332HkJAQKBQKnDp1CsbGxvj+++/h4uKCoUOHom3btoiOjoahoSECAgJ0sj8jR46EjY0N3n33XcyYMQNWVlbYsGEDgIq36X/zKvXbUtBn3wMAECQePHggcnNzNY9zcnJEbm6u6N69u0hISBBCCFFSUlKhso8cOSIGDRok/ve//4lly5aJq1evipycHNG9e3fRoUMHMXr0aJ3sw9Oo1WqdlLNt2zZx8OBBcfToUTFy5EghxF/HbMmSJeK9994TM2fOFIcOHRI5OTk6iaetX79+YsCAASIjI0MIIcTdu3fFu+++KxYtWiQCAgJETExMhcu+ffu26NKli6bugwYNEkOHDhX+/v7i888/F0VFRTrZByGEyM3NFdu2bROff/65aNGihZgyZYo4fvy4zsrXVlpaKoQQYvbs2eKtt94SISEhIiYmRhQWForTp0+LuXPniu7du4u4uLgKx1CpVKJ3794iKSlJCCFEQUGBOHfunJg0aZLw9/cXH330kU6P4aNHj8Thw4fF6tWrdVamttTUVNGtWzdNvfPz88W2bdtESEiI2LZtmxDir/a/f/9+0bNnT7F3795nlpmfny+EEOL+/fti/fr1YtCgQaK4uFgIIcTIkSPF2LFj9bQ3+rd7925x//59IYQQUVFRYtSoUeLDDz8UQUFBYs2aNTqJcfHiRfH111+LTp06id69e4vffvvtiXaVmJioqYNara5w3zd37lzRo0cPcebMGc1zxcXF4quvvhIXL16s3E78C123uapsb2fOnBELFiwQkZGRmj4oPT1d/Pjjj6J58+YiLCysQuWWlJSI2NhYoVarxdy5c8XatWtFYWGhePjwofj6669Fu3btxIgRI0Tz5s1Fdna2TvYlMTFR9OjRQ/N4+PDh4sCBA+LixYsiNDRUs3+69Cr22/oiRd/zWk+3EP83BG9iYqIZ5b116xa++eYbhIWFoWnTpujXr98TKxQ8D6VSiUePHsHU1BRTpkxB+/btUaNGDaSlpSEmJgYymQyTJ09G37590a1bN73d7EMXvwQLCwuxbds2HD58GDKZDAqFAn5+fqhWrRpatmwJb29vxMfH48CBAwgJCSn3El/Po2XLlkhMTMSsWbOgVCrRvXt3ODg44P79+wgKCkK3bt0qXLZarYZarcbFixexceNG5OTkYOPGjfjPf/6DTZs2wd/fX2fL2RkbG8Pb2xtt27bFe++9h6CgIL2tt5qVlQULCwskJCTgxx9/xI8//ojCwkLMmTMHly5dQpcuXRAYGAhPT0/4+/tXOI5CoUBcXBzOnj2LRo0aoVq1anB0dISHhwfS09Px7rvv6nQfDQwMULt2bfj6+uplpKO0tBT5+fm4dOkSmjZtCktLSxgZGcHb2xuOjo44efKkZs597dq10blzZ82FVf+kqKgIM2fOxJ9//qlZG/3+/fuaKVxbtmxBkyZNcOvWrQqPDlWVCxcuYMGCBRg4cCAMDAzg7u6OHj16oLCwEGlpaVCpVCgsLISZmVmlTn3LZDIcP34cSqUSbdq0wZ07d3D8+HEYGxvDxcUFjo6OmusgKnMXy+bNm+PevXuIiYnBrVu34OjoiPPnz2Pbtm2aOY+6pus2V5XtLTk5GR9//DESExNx7tw5ODk5wd7eHjY2NmjSpAk6deqE1q1bV6hsuVwOFxcXREdHY82aNbh+/TpSUlJQWFiIwYMHo1OnTlCpVOjbt2+5LjT7N2q1Gvv27UNwcDD27NmDGzduYNy4ccjPz9dMf9Dl9/er2m/rg1R9z2udJD+tI1Wr1cjPz8eHH36ITp06QaFQlHu9zdDQUGzbtg337t2Dubk5PvnkEzRp0gT29vYoKipCQkICIiIi4O3trTn99aJKS0tDnz59cPnyZURHR2uWhLGwsICVlRUcHR0RFBSExo0bP9cV6s9De8Fxa2trBAcHo2nTppo5e+3bt8c777xTqRUhTp06haFDhyI1NRXm5uYIDAzEO++8AxsbG2zatAlJSUl4//33dbFLT5DL5TA2NtbpDRUel5mZiXbt2kGpVMLOzg4tW7ZEs2bNEBAQgMGDB2tuApOXl4f33nvvuefUlil7fzIyMjRrdMbHx+PWrVsoLi5GrVq1kJubi61bt2L8+PF6SWZ1XWZRUREMDAxQUFAACwsL7Nu3D+Hh4Wjbtq1m7dirV6/ixIkT6NOnj6YOz/Oj0MDAAE2bNkVSUhIOHDiAqKgorF+/HufPn8eMGTPQt29ffPzxx0hOTkaDBg10ul/6Nn36dPTs2RO+vr4A/hogAP66YVL79u2RnJysuciovPv2eD9gZmaG+vXr48qVKzh69Cg8PT1hYGCAqKgo3LlzR2cXcBoYGMDHxwdFRUW4cuUKFi1ahGvXrmHs2LE6Txr01eaqsr2tWLECXbt2xZIlS5CWloaff/4ZSUlJcHBwgI2NTaXmc5fNEZbJZCgoKMCpU6dgY2OD9PR0JCQkwNbWFiEhIZrVGirq8f7N0dERZ8+exa5du7Bjxw7MmTMHjo6OWLNmDVQqFUJCQioV63GvQ7+tS/rsex732iXJY8eOhZubGxwcHP72N/F/VzgLITQLkpd3VEKtVqOoqAiFhYVIT0/HiRMnYG9vj3r16sHR0VFzO87CwkJ06tRJr7eMriy1Wo3Y2Fj07dsXd+7cwebNm9GyZUts3boVFy5cgFwuh4mJCaytrTVLvuiKTCbDqlWrUFJSorn3uouLC9566y2cOHECP/zwA6ytrSu9AoC1tTUyMzORnp6OIUOGwMXFBYcPH8ayZcswb948nV+kIwVzc3O0bdsWW7du1Swx2KVLFxgZGcHY2BjBwcFo1aoVbt68iXbt2gEoX9JZtu3ixYthbW2tKePKlSu4cOECVq1ahd27d2PgwIF6W9ZOl1QqFbZs2YLIyEisWLECvXv3Rs+ePXHs2DFMnz4dd+7c0Szm/9lnn8HJyem5byZ07949zcWGPj4+WL9+Pdq1a4dOnTqhpKQEMTExaNu2LTw9PXU2+iWViIgInDlz5okLdUeNGgUjIyPUr18fpqamCAgIQPv27dGoUSMYGRmV60u9bLsffvgBJSUlMDc3R8+ePeHq6orc3Fz4+PjA0tIS9evXh7u7u87u8ll2xqdVq1bo1asXOnfurPMRV321uapsb7du3cKBAwfg7e2NN954A61bt0a7du1w4sQJrFy5ElZWVpU6jnK5HMXFxZolxaysrHDlyhXY2tqioKBAs4Z0ZVcKkslkePToEebOnYv27dujWbNmSEtLQ0lJCc6ePYvTp0/jxIkTWLhwoU7PnLLffn767nse99okyUIIqNVq7Nq1C9988w3u3r0LPz8/zSm64uJiKBQKREVFYfHixXj77bcrdEBlMhnc3d1RUFCAuLg4qFQq5Obm4uLFi7C1tYWDgwNq1aoFPz8/vVzpqUsymUyzfNDt27exc+dOtGrVCuPGjUN6ejp+//13nD9/XvNlpQtl7wMAxMbGYtq0abh27Rr8/f0161JWr14dXbt2Re/evSsVy9LSEvXq1UOjRo2gVCqxZMkSREREoKioCJ988oneblChT2Xt3MnJCf3794eHhwfCw8MRHh6O2rVra0b7nZyc0LZtWwAVux32zZs3sWHDBtSpU0dz+09fX184ODigWbNmCAwMrNQ0GCkpFAp4e3tj2rRpuHXrFurUqQNPT09069YNQUFBOHfuHBwdHREcHIyAgIByTb+6c+cO1q1bp/khdv36dTg4OGhOp06YMAFqtRoZGRkv/OlNbePHj8f48eM1beqPP/7AyZMnMX36dM02ixYtQr169TQrulTkDngLFixAXFwc5HI5tmzZgtTUVMhkMty9exejR4/WHDddj3wZGhpqzpjpmr7aXFW2t+joaFy4cAFRUVEoKipCzZo14eLigq5du6J27drw8vKqVAIbFRWluWgvKSkJzs7Omu/wXr16wcfHRyfJXWlpKQoLCxEWFobMzEwEBgaiQYMGeOONN1CzZk04ODhg6NChmsEbXWC/XT5S9D0aOpnZ/JJJTk4W/fr1E35+fn+bfD9o0CCxZ88eIUTFLtYruzgiOTlZzJ49W3To0EHMnz9fzJs3T0yePFmEhoaKBw8e6GZHJFJQUCBKS0vFli1bRPPmzcWAAQNEdna2+Omnn0Tbtm11FketVos9e/aIyMhIMWPGDJGVlSWSk5PFBx98IBo2bCgWLVoksrOzRe/evTXvUUUolUqxatWqJ57LyckRoaGhomvXrqJHjx5CpVJVdneqRFlbvn37tkhJSREFBQVCCCFWrFghGjRoIMaMGSNu3bpV6TixsbFi0qRJok2bNmL27Nnizp07lS6zqpQds/nz54vvv/9etG3bVnTt2lWcPHlSs82ff/6p+X9FLgr75ZdfRFBQkDh48KAQQoiPPvpIdO7cWSQnJwu1Wq3pN14Wv/76q+ai2bILavv37y+OHDmi2eb48eOid+/elYqjVqtFXFycGDJkiFi6dKm4e/eu+Oabb8SgQYOEp6enXi981Sd9t7mqam85OTli9erVYty4cWLKlCli9+7dOrsA7MaNG2LUqFHi008/FUeOHBETJkwQISEhokOHDqJ3796VjlP2npQd68TERDFp0iTNhWHFxcVCrVaLhw8fVm5H/iU2++1nk6rvKfPaLAGXlpaGbdu2oVGjRpolaPbu3Ys5c+bAwsICixcvRlJSEvbv36+5MUZFFRcXa5YJ2rhxIyIjI9GkSRPI5XLExcVhwoQJL/zpjH+Sm5uLJUuW4JdffoGVlRXmzp1bqSV9tKWkpGDMmDHIzMzEqlWrNNMpjhw5gjlz5kAmk8HLywvLli2rcIz4+HhMmzYN1tbW+OCDD9ClSxcAfy2N891336FHjx6VunNfVSlb5ub69euYNGkS8vPzUb9+fdSuXRuDBw+GoaEhxo4di5s3byI6Orrc5Qut01UlJSU4dOgQDh48iKysLHTs2BEDBgzQXAT7Mijbp5ycHJSUlMDW1hZZWVnYvHkzQkND0aZNG9y7dw/9+/dH//79y11+2Xty8OBB7Nq1C7169YKXlxdGjRqF//73v1AqlejWrZvebnGsLxkZGVi7di0uXryIJk2aIDs7G+np6fjxxx812wwcOBCDBw9Gt27dynWjg7JtU1NTYWNjA1NTU6SmpmLJkiWoW7cuxowZg6ysLNy/fx/169fX1y7qjT7bXFW0t0ePHmHfvn1ITEyEp6cnevfujevXr2Pv3r1ITk6Gi4sLpkyZUqGl87SX7rp69Sq+/PJL1KtXD9OmTcO5c+dw/PhxmJmZYfjw4TrZn2XLliE7OxvGxsZYt24dfH190aBBA5w4cQJ2dnaYOHGiZh6sLrDfLh999j1P81okyQkJCVi8eDFsbGzQqVMnBAcHP9Egvv/+e3z33XcQQmDnzp2oX79+hQ9sVFQURo4ciY8//hjW1taaC/PS0tLQvn17FBUVaW6G8DLLzMzEjRs30LJlS52WK4TAyJEjoVarIYRAnTp18P7772tOqzx8+BByubxSd/QTQiAzMxM7d+7Erl274O7ujvHjx+PevXv49ttvNWsrvqxGjRqFVq1aoWvXrjh69CjOnTuHR48eYezYsXB1dcWff/4JOzu7crXxso68oKAABw4cwL179+Dl5YXAwEBkZGTg0KFDiIyMhKGhIZYtW/ZCz7UvU7b/58+fx6xZs5CamopWrVqhcePG6NSpE4qLi7FixQqoVKpK/SgTQuDYsWNwdXXF9evXsWLFCjRo0ABz5sxBTEwM2rRpo8O9kkbZWqvx8fHYtGkTEhMTERQUhHfeeQd169bF9u3bcfDgwSe+uMqrX79+sLOzQ506deDs7Ky5q1r16tUrPdWqqkjR5qRub7NmzUJ6ejr8/PywY8cOFBQUYN68eWjTpg127twJW1tbzRSBilAqlVi6dCnatm2L2rVro1atWvjll19QUFCADz/8EDKZrNLz0bdv347g4GDk5eVh9+7dKCgoQNOmTXHx4kX89ttvGDZsGPz9/ZGVlYUWLVpUOM6/Yb/9fKToex73WiTJAwYMwNtvv625QriMWq1Gamoq3NzckJubi+joaPTo0aNSC0/fvHkTCxcuRLVq1dCtWzfs3r0bly9fRmFhIaysrLB169aXoiFKreyYl5aWIiUlBSYmJoiPj8eRI0eQmZmJkJAQXLhwAc2aNSv3fer/SWlpKZKSkrB582aEh4fD1dUVEydO1Nym/GVSdvzS0tKwevVq/O9//4OtrS2Avy7MWLFiBYC/LtowMDAo9xfK4zc8uHz5Mjw9PZGUlIQGDRpgwIAB8PDwwMGDB+Ho6PjMJdFeNEOGDEFISAg6dOiA8PBwXLhwAba2tvjkk09gaWmpOTNU2REJ4K+Lm95++20EBgaiS5cuCAoKeqluHJKUlIRNmzbh+vXr+PbbbzVzTA8dOoQtW7bA2Pj/tXenMVFdYR/A/wMjgqyjRERHy7QqGCBhs0WYivvYSgLFBkwxgorVYFqhCjQorValKqKmikhJnSrYFqGKisUFqSLGpVJwI8gmiiIgssnggOOc94Pv3FexfYsyMMzw/D7pDblz7s25zz33LM8ZDE9PT0ilUuzatatHHQ6MMZw/fx6tra0oLCzEuXPnYG1tjb/++gs7duzQirmT/6av6lxv17fy8nKEhobi1KlT3LH09HRs27YNGzZswPTp03t0fsYYmpqasHHjRgwbNgxNTU14+PAhXFxcUF1djXfeeQdhYWE9aiRfuHABS5YsQXFxMXJzc+Hg4PDKwv4ffvgBNjY2PdoV999Q3O6+vow9r1DLpI1+7Pjx42z+/PmMsdfndTU3N7Po6Gh29OjRV46/6fyvrgnFS0pK2Pz589natWvZs2fPWEFBAdu5c6faklvrGtX9a2lpYatWrWKTJ09mJSUljDHGSktLWUpKClu8eDF7//33WW1trdp/X6lUsrq6OlZeXq72c/e1VatWsUmTJrHVq1e/MoettbWVSSSSHt2/0tJSNmPGDO7/AQEBLCgoiPn5+bGwsDAWExPTo7L3JdV6gzt37rC4uLhXNsK5efMmCwgIYF999VWvJNR//PgxUyqV7MyZM2o/d28LCAhgSUlJ7MSJE4wxxuRyOevs7GSNjY3s+fPnLDU1lc2cOZOtX7+eMfZ6bOyJR48esezsbLZ3716tXDOc/gFeAAANxUlEQVSgqTrXm/WttLSULVu27LXjqampLD4+njGmvk2tHj9+zBoaGrj5yEFBQWzOnDk9Pm90dDTbu3cvO3z4MLO1tWXz5s1jJ0+eZI2NjYyxF/NbfX19WWlpaY9/699Q3P5vmoo9vZOotR/R19fn0s50/QozNzfHsGHDcP/+/VeOv+nXmp6e3mtDQlKpFL/99hv27t2LJUuWwMXFBUz3O+3fiup+r1u3DiNGjMDBgwdhZWUFABCJRNDT04Ovry9kMhl3XN2/P3z48H9MC6gt2P/2pERFRSErKwvZ2dn46aef4OnpCRsbG+Tm5mLcuHGwsrJ665ESuVzO9XLk5ORAqVRi48aNWLduHXg8HqZOnaruy+o1qt6F1atX4+bNm9DX18eiRYtgamoKe3t77Nq1CyEhIWhvb1f7yI+qB0TbRizS09MxZMgQfP7559yxxMREXLp0CYwxuLu7Izw8HF5eXmp/lpRKJSwtLTF79myurmvb9t2aqnO9Vd9Uo6OVlZVYuXIl5s+fD2dnZwAvhsRV71V1ZR1RXYeXlxe8vLzw6NGjHt8npVIJV1dXREdHY+jQoTh48CBu3LiBTZs2wcnJCUFBQZg4cSLc3d3Vtg/Ayyhud48mY49Op4CrqalBa2srEhIS4OHh8UrOW7lcDj6fj7q6OhQXF/do8RljDHK5HGfOnEFTUxPy8/ORmpoKMzMzVFRUoKSkhJu7298TdGuCahHL77//jmXLlnHBiDGGjIwMHDhwAD4+PmpLM6dLVAniFQoFamtrAQBisZhLFJ+VlYWsrCyMHDkSX375JUxMTN6oDqrSDHV0dECpVOLIkSP45JNPkJCQAH9/f27hxEcffdSjHaD6murlJJFIoKenh8zMTFRXV8PCwgJyuRy//vor93HWdXMbddGmWMAYQ1JSEvz9/SESiSCTyZCUlIS0tDTMnTsXYrEYp0+fhq2tLcaOHcs1CNV1jS+fR/Vvbbp/gObrnLrPFxERgZMnT3K70ubl5SE/Px/Nzc2QSqWIjo7ulTzzqntjbGwMQ0PDHp2Lx+NhwoQJKCsrQ21tLTIzM+Hl5YXly5fj+vXrkEqlqKurw6JFi9S62JHidvdpOvbobCO5qqoKkZGRWLlyJSorK1FUVARnZ2cu+bdqt7M1a9YgMDAQNjY2bx2YeDwejIyMIJFIYG9vD7FYjNGjR6OgoACNjY24efMmPvvsM60L6n2FMQZDQ0PU1dXh9OnTEIlEGDp0KHg8HhwcHJCWlgYPDw9qJP8D1Ys3Pj4eCQkJyM3NRUVFBcaOHYuAgADY29ujrKwMMpkMjDEuz3R36iJ7KTdrREQEcnNzER4ejjFjxuDq1au4du0ahEIhtm/fjo8//ljtG8r0BtUz/uzZM9TU1MDExITb9CAnJwf79u3Dn3/+CUdHR4SFhcHAwICeWwANDQ2ora1FWVkZDA0NkZmZiWPHjuG7776Dn58fxo8fj+LiYiiVSq3N3NNbdLHOxcXFob29HWFhYfjwww8xatQo6OnpoaCggNsmurcWuPXGvZkyZQqWLl2K8ePHY+PGjTh79ixCQ0MxdepUXL16Fb6+vmr9PYrb3afp2KOzjeSkpCRkZ2fD398fHh4eOHToEHbu3Ak+n4/m5mYUFhYiLS0NPB4PoaGhANTz8BkZGWHIkCGwsbHBrFmz4OnpCX9//x5/8eoi1dduQ0MD6urqUF9fj4aGBty7dw8tLS1oa2tDbm4url+/jsWLF2u6uP2Oavjt1q1bSExMREpKCoRCIfbv34/i4mI0Nzdj8ODBsLOz47a9LSgogKura7c/OHg8HuLi4iCTybBixQpugxVjY2MUFxfjjz/+gIuLC+bOndubl6pWPB4PW7ZswZ49e5Ceno7W1lY4Ojpi3rx5cHR0xJUrV3D37l0MHTqUS0E2kN25cweRkZHw9/dHdnY2srOz0djYiJCQEC59IvBicZFEIlHr7ne6QpfqnEKhwJEjRxAcHAw7OzsolUoIBAJ0dnaisbERa9aswbvvvqvpYr4RVXo6GxsbLFq0CE1NTYiOjkZ9fT22bt3KdaqpA8Xt7usPsUdn5yQvX74c5ubmCAwMxPTp05GcnIz8/HwkJSXB2NgY1dXVWLBgAddAVssqyJeoHgRt3Na4LzDGuPsdHh4OPp8PQ0NDPHnyBPfv34dcLsft27dhbW2NzZs3a7i0/ZOqt+D8+fOYOHEiLC0twefzYWpqikmTJuHHH3+EgYEBYmJiEBwcjBkzZiAnJwcjR47s1vlVw4F1dXVYuHAhxo4dy9VrfX19mJubIzIykpvv1p+p1gPweDwUFBQgLy8P6enpOHv2LGJjY3H16lX4+flBIBAgJiYGly5dwrZt2zBy5Ehs374dAoFAw1egOQcPHkRhYSFsbW2xa9cu1NfXw9LS8pUX9s8//wyhUMil+qIGsu7WOT6fD5FIhK+//hqxsbFcVgRnZ2ds2LABN27c4PLba6vQ0FAEBQXhwIEDas8fTHG7+/pD7NHZnmQDAwO4ublh2rRpyMvLQ1xcHOzt7fHNN9/Ax8eHm5ej2hpa3QtA6CXx33g8HrZt2waFQoHY2FgMHjwY58+fh4WFBQQCAcaMGYOZM2dqfcDtDS9/LfN4PGRlZcHHxwfR0dGIjIyEt7c32trasHLlSri4uECpVMLc3PyNk+Dr6emhrKwMe/bsgYODA7fFp7W1NXbu3AlXV9duB29NksvlXG+RVCqFlZUVJBIJ7t27h4qKCjg7O2P37t1cHllfX1/Mnj0bcrkcHh4eGi69Zjk5OWHw4MFYu3Yt7t27hylTpkAgEKCjowMZGRlIS0vDxYsX8e2330IgELzVdrm6SJfrnJubG2pqapCXl4eqqipYWFigqKgIFy5cQHh4uKaLpxYGBgZwdXVV6zkpbr+Z/hB7dLaRrGJmZoYZM2bA1tYWKSkpSElJgZmZGSZMmMD9DQX1vsfj8fD8+XMcOnQICxYsgEgkwr59+zBixAg4ODjg2LFjuH37NsLDwymvdBcvB1qpVAonJydMnz4dxsbGOHXqFJycnCAQCJCcnAyJRAIzM7MeJdz/txdifn6+1rwQ4+LisGPHDri5uUEkEuH69evw9PREREQE1q1bBx8fH7S2tuKLL76Aq6srFAoFTE1N4eLioumia5yBgQEmTpyIadOm4fLly9ixYwdkMhnc3d1x9+5dDBo0CMHBwVyPlTpH5LSZLtc5Ho8HR0dHdHR0oKSkBPHx8ejo6EBISAjGjBmj6eL1SxS331x/iD06O92iqw8++ABubm7IzMzEnj17kJGRgc2bN6OlpQUPHjyAl5eXpos44Ojr68PExARpaWkYNmwYrl27huPHj6OtrQ2mpqbw9vbmevrJ69LT01FaWoqFCxdyx8aPH4/U1FS0tLRAKBRCKBRyQfZtRzf09PSwdOlSnDhxAoWFhQgMDIRYLEZUVJS6LqXXffrppwBe7GplY2ODkJAQGBoawtramkvhWFJSgqCgIAAvhpRpXu2rhEIhYmNjceXKFSQmJiInJwdRUVHw9vbm/oY6HP6Prtc5MzMz+Pv7w8/PD6tXr8agQYO0ZmtjTaK4/eY0GXsGxI57XbW0tEAqleL48eOorq7G7t27MW3aNE0Xa8CSyWSoqKjA999/D6lUiqKiIiQkJCAlJUXTRet3VHPL5HI5EhMTce7cOWzatAl2dnYAgOrqamRnZ0MoFGLq1KkwMjJSaz5ZhULBDSNr2wuxs7MTRUVFyMjIwLVr1+Dr64vy8nIMGTIEVVVVGD16NGJjY7WqoaIpz58/x+HDh7Fv3z5YWlpiy5Yt1OHwD6jOEYDitjr1dewZkI1klbKyMly8eBELFizQdFEGvI6ODkRERODJkyeorKzE2rVrtSLJuabEx8dDJpPh6dOn6OzshJOTE+bMmcMl3Fehl+/r2trakJOTg6NHj6K8vBxGRkaIiorC5MmTwefztW6TCk2iDofuoTpHAIrb6tRXsWdAN5JJ/1JTU4Nbt26Bx+P1aHMXXaXKwNLS0oLIyEiUlZXBzs4OEyZMQFtbG54+fQoXFxe15/TURYwxPHz4EGfPnkVycjLWr18PsVhML6e3RB0O/43q3MBEcbt39XbsoUYyIVpm1apVsLKywv379yGTyVBVVQUnJyfY2Njgzp07CA0NxXvvvafpYmoFhUKB6upqiEQiTReFDBBU5wYmitvaacAs3CNEFxw7dgzl5eXYunUr2tra8PjxY5w+fRr79+9HZWUlJBIJBdo3oMr5SkhfoTo38FDc1l7USCZEi+jr68PQ0BDt7e0wMTGBiYkJvL29cevWLZiamiIrKwuGhoYIDAyEvr4+DeMSQoiGUdzWXrRSgBAt4uXlBQsLC8TExCA3NxcAMGLECAwfPhxisRgLFy7E5cuX8fTpUwq0hBDSD1Dc1l40J5kQLfPgwQP88ssvePjwIdrb2yESiZCdnY2cnBzw+Xz4+fkhMjIS7u7umi4qIYQQUNzWVjTdghAtM2rUKKxYsQJ///03CgoKUF9fj+TkZPD5fJw5cwYKhYICLSGE9CMUt7UT9SQTogM6Ojowa9YsjBs3DsHBwRCLxZouEiGEkP8Hxe3+jxrJhOiIxsZG8Pl8mJmZaboohBBCuoHidv9GjWRCCCGEEEK6oOwWhBBCCCGEdEGNZEIIIYQQQrqgRjIhhBBCCCFdUCOZEEIIIYSQLqiRTAghhBBCSBf/A9Jia2PasddvAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "rand_example = random.choice(data)\n",
+ "input_ids, input_mask, segment_ids = convert_to_bert_inputs(rand_example[1], rand_example[2], max_len)\n",
+ "_, rand_output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "rand_output_finetuned = rand_output_finetuned.squeeze(0).detach().numpy()\n",
+ "visualize_single(rand_output_finetuned, rand_example[1], rand_example[2], max_len, 9, 8)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 3. Cosine similarity map"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 47,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:44<00:00, 2.90it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "Text(0, 0.5, 'Layer id')"
+ ]
+ },
+ "execution_count": 47,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAIlCAYAAAB/4DRIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xl4VOUd9vE7K0sSMEGCgoBcVECDCAha9i0hYVUUSIosCkVwQS2KIEWiKAgUFwSsuBQriqKYIlokUAVEZBGrKKGK0tfIThISSAghmeR5/+BlXlPCDARmzjnD98OV62K25/xmMpP8cp/nPCfIGGMEAAAAywVbXQAAAABOoTEDAACwCRozAAAAm6AxAwAAsAkaMwAAAJugMQMAALAJGjOclTFGixYtUt++fZWYmKj4+Hg98cQTys/Pr/SYSUlJys7OvohVnunZZ5/VO++8c16POV1XWlqa7rzzzvPe5ogRI5SRkSFJeu+998778f6yfft2denSRWPHjr3gsd58803169dPSUlJSkxM1NSpU3XkyJFy9/niiy+UkpKixMRE9ezZU2PGjNEPP/zgvn3evHlq06aNkpKS3OOkpqbqxIkT51zH9OnTdfPNN+vw4cMX/Jz8acuWLUpISLC6DFt5/vnnNXXqVKvLACxFY4azmjNnjlauXKnXX39d6enpWrFihUpKSjRmzBhVdvm7VatW6fLLL7/IlZb38MMP6w9/+MN5PeZC6/r73/+uuLg4lZaWavbs2ZUex9e++OIL3XTTTXr55ZcvaJwXXnhBK1as0KuvvqpVq1bpn//8p6KiojRixAidPHnSva1JkybpkUceUXp6ulavXq1bbrlFd911l3bv3u0eKzExUatWrdKqVav00UcfKSsrSwsWLDinOkpKSrRhwwaNGDFCK1asuKDnBAB2QGOGCuXl5Wnx4sWaOXOm6tSpI0mqXr26pk6dqj/+8Y8yxujkyZOaOnWqEhMT1atXL82cOVOlpaWSpLfeeku9evVSUlKSBg4cqJ9++kmS1LRpUx08eFBbtmxRcnKynn32WfXq1Uvdu3fX1q1bJUnFxcV6+umnlZiYqO7du5+1idi1a5eSk5PVp08f9ezZU2+99ZYkadKkSXrppZckSd27d9fixYs1YMAAtW/fXqtXr9aTTz6p+Ph4DR48WEePHi1X129lZ2dr1KhRSkpKUvfu3bVo0SL3bd27d9f8+fOVmJio/fv3q3v37tq2bZvuuusu5efnKykpSW+//bbGjBnjfkxZWZnat2+v//znP+W2k5aWpgceeEAPP/ywunbtqrvuukvbtm1TSkqK2rdvr6VLl7of/+STT7pflwkTJqikpESSNGzYMM2bN0+DBw9W+/btNWXKFPf34rRVq1bpzTff1Nq1azV69GhJp1Kv3r17KykpSffcc4878Zo0aZKeeeYZ9evXT5988km5cY4ePao33nhDf/nLX3TFFVdIkkJDQzVhwgSFhYW5G6S5c+fqwQcfVJs2bdyP7d27t2699Vb99a9/rfB7Gh4eruTkZG3cuLHC2//X+vXr1bp1a91yyy3lGrNnnnlGM2bMcF/OyclRq1atVFBQoF27dumOO+5Qz5491a9fP+3cuVOS9OWXX2rIkCEaN26cHn30UUnSu+++q6SkJCUkJGjYsGE6cOCAJKmoqEjjxo1Tp06dNGrUKM2ePVt//vOfJUn79+/X3XffrcTERCUmJmrDhg3uOubPn68uXbpowIAB2rx581mf1z//+U/17dtXSUlJGjFihPbs2aMffvhBv//978t9X8eMGaP3339fJ0+e1LRp09zvjVdeecV9n86dO2vBggVKTEzUoUOHym3HGKN58+YpMTFR3bp104wZM1RWVqY9e/aoffv27vsvX75cKSkpMsZo9+7dSklJUa9evdSzZ0+tXLlSkuRyudS0aVO9//776tu3r7p27aotW7booYceUrdu3TRmzBiVlpYqMzNTN910k1555RX16dNHnTp10tq1a894Dc72OrpcLj322GNKSkpSfHy8HnjgAR0/fvysryXgOAaowLp160xCQoLH+yxcuNCMHj3alJSUmBMnTpjbb7/dLF++3OTn55s2bdqY/Px8Y4wxK1euNK+88ooxxpgmTZqYAwcOmM2bN5vmzZubNWvWGGOMefXVV82dd95pjDFm/vz5ZsSIEebkyZPm+PHj5tZbbzWfffbZGdsfN26cSUtLM8YYk5OTY+655x5z8uRJM3HiRLNgwQJjjDHdunUzjz/+uDHGmMWLF5sbbrjBbN682ZSVlZnbb7/dvPfee+Xq+uCDD8yIESOMMcZMmzbNTJ061RhjzK+//mri4uLM/v373eNOmTLFXUu3bt3MV199Zfbs2WOuvfZaY4wx2dnZpkWLFubIkSPGGGO++uork5iYeMbz+OCDD0zLli3Nf//7X3Py5EnTqVMnM2bMGONyucxnn31mOnfubIwxZtWqVaZv376muLjYFBUVmV69epnly5cbY4wZOnSoGThwoCksLDSFhYWmZ8+e7tf2t1588UUzefJkY4wx33zzjencubPJzs52P9/Tt02cONH069fPFBUVnTHGunXrKnwexhgzb9488/DDD5vjx4+bpk2bmsOHD59xny1btphOnTqdUc9pa9asMYMHD65w/P917733ms2bNxtjjLnjjjtMRkaGMcaYr7/+2sTHx7vvt3TpUnPfffcZl8tlevTo4X7fbN261XTu3Nm4XC6zceNGc/3115utW7caY4w5ePCguf76683BgweNMcZMmDDB/V5atGiRueOOO4zL5TK//vqrufnmm93P44477jDz5s0zxhjz3//+19x0000mLy/P/Oc//zE333yzyc7ONi6Xy4wZM6Zcjaf9+uuvpk2bNubXX381xpz6nI0cOdIYY0zPnj3NV199ZYwxJj8/37Rs2dLk5uaauXPnmpEjR5qTJ0+agoIC079/f7N+/XpjjDGdOnUyqampFb5+y5YtM3379jX5+fmmuLjYjBw50ixZssQYc+ozOWHCBFNQUGC6detmdu3aZYwxZtSoUea1114zxhjz5ZdfmpYtWxqXy2VKSkpMkyZNzKuvvmqMMebpp582bdu2NZmZmaaoqMh06NDBbNmyxfzyyy+madOmZtGiRcYYY9avX2/at29vXC6Xee6559yv8dlexzVr1piRI0easrIyU1ZWZp599lmzceNGz28UwEFIzFChvLw81apVy+N91q1bp8GDBys0NFRVq1ZVv379tHHjRlWpUkVBQUFatmyZsrOz1atXL3dC81sRERGKj4+XJMXFxWn//v2SpLVr12rIkCEKDw9X9erVdcstt2j16tVnPL5WrVpKT09XRkaGoqOj9dJLLyk8PPyM+/Xo0UOS1KRJE1WpUkU333yzgoKCdM0113iclzRlyhQ9/vjjkqT69eurdu3a2rt3r/v2rl27enx9atWqpTZt2ig9PV2StGbNGvXu3bvC+/7ud79To0aNFB4eroYNG6pjx44KCQlRkyZN3DUmJibqgw8+UFhYmKpUqaLrr79ee/bscY/Rp08fVatWTdWqVVOnTp30zTffeKxv3bp1SkxMdH+fBw0aVC6pateunapUqXLG444dO6aYmJizPudjx44pPz9fxhhFR0ef9T4VKSgo0JIlS85p7tWRI0e0a9cu3XTTTZKk/v37a/ny5ZKkVq1a6eTJk+6kds2aNerVq5d++ukn5efna8CAAZKktm3bKioqStu3b5d06j3Ztm1bSVKdOnW0bds2d2Lcpk0b9/f/66+/VlJSkkJCQlS/fn117txZkpSfn69t27a55yk2atRILVu21Oeff65t27bppptuUq1atRQSEqL+/ftX+Lw2btyodu3aqX79+pJOfV82b96ssrIyJSYm6tNPP5Ukff7552rdurUuu+yycp+ZiIgI9e/fX2vWrHGP2a1btwq3tXbtWg0aNEiRkZEKCwvToEGD3I+788479fPPP2v8+PHq37+/rrnmGknSK6+84n5+bdq0UWFhYbl5o6c/002aNNHVV1+tBg0aqEqVKmrQoIH7vWyM0cCBAyVJnTp1UmFhYbn3sqfXMSYmRj/++KM+/fRTnThxQuPHj1f79u0rfH6AE4VaXQDsKTo6+ozdHv/ryJEjqlmzpvtyzZo1lZOTo7CwML3xxht6+eWXNW/ePDVt2lSpqalq2rRpucdHRUW5/x8cHKyysjJJp34oP/PMM3ruueckndq12aJFizO2/8gjj2jhwoV66KGHdPLkSY0ZM0Z33HHHGfeLiIhwb+P0//93mxX5/vvv9eyzz+rAgQMKDg5WVlZWufv/9rmfTZ8+fZSWlqaUlBR9+umnZ90t+9u6QkJCVL16dff/T2/zyJEjeuqpp7Rz504FBQUpOztbI0aMqLCemjVrep0Mf+TIEcXGxrov16hRQzk5OV6fX506dc46dk5Ojq644gpdfvnlCg0NVVZWlq688soK73Naenq6vv76a0lSWFiYEhISzukAjI8//lgHDx50N1LGGFWtWlWPPvqoQkND1bNnT3322We68sortX37ds2dO1cZGRk6fvy4kpKS3OMUFBQoLy9PVatWLfecXS6XXnjhBa1du1ZlZWU6fvy4fve730k6tTv3t/etU6eOjhw54m5ITzcdklRYWKjOnTsrLy+v3Hu+Ro0aFT6vij5XLpdLeXl5SkxM1Pjx4zVx4kR9+umn7kb/2LFjevrpp/WXv/xF0qnPTKtWrcqNUZFjx47plVde0ZIlSyRJpaWlql27tqRTu6cHDhyoJ598Uqmpqe7HrF+/XgsXLlRubq6CgoLcr/1pp9/Lv30fn758ejdsaGioIiMjJUlBQUGKiooq16x7eh379eunyZMn64033tCECRPUo0cPpaamlnttASejMUOFWrZsqZycHGVkZCguLs59fUlJiebPn6+xY8fq8ssvV15envu2vLw89wT66667Ti+++KKKi4v12muvKTU1Ve++++45bTs2NlYjR44861/5p0VERGj8+PEaP368vvvuO40ePfqi/uU8YcIEjRgxQn/4wx8UFBSkTp06nfcYCQkJmjZtmtavX69q1aq5f7FXxvPPP6/Q0FB99NFHCg8P18MPP1zu9tzcXPf//7dxqIin758nzZo1U05Ojn744Qc1a9as3G1r167VnXfeqZCQEF1//fVas2aNhg8ffsZ9TjdT0qkkcPr06V63+7+WL1+uJUuW6Prrr3dfd/fdd2vDhg3q1q2bEhMTNWfOHNWvX1+///3vVb16dcXGxqpmzZpatWrVGeN9+eWX5S5//PHH2rBhg5YsWaLo6GgtWbLEndxGRkaqsLDQfd+srCyFhITo8ssvV3BwsJYvX66qVauWG2/x4sXatWuX+/L/HsF62uWXX15uHmJeXp7CwsJ02WWXKSYmRqWlpdq1a5c2btzobphiY2N1zz33uJO7cxUbG6tevXpVeLDM8ePHtWjRIg0bNkxz5szRc889p+LiYj344INasGCBOnXqpKKiIt1www3ntU3pVNObn5+vqKgoGWOUn59f7v3q6XWUTs1V7N27t3Jzc/XYY49p0aJFeuCBB867DsCO2JWJCtWoUUN//OMfNXHiRGVmZkqSTpw4oalTp2rnzp2qVq2aunbtqmXLlqm0tFSFhYX68MMP1aVLF/3444964IEHVFxcrPDwcDVv3tz9l/W56NGjh95//32VlpbKGKOXXnpJn3/++Rn3Gzt2rHtXVZMmTRQZGXle2/EmJyfHXfs//vEPnThxotwv44qEhYWprKxMBQUFkk6lgp06ddKTTz6pXr16XXA9TZo0UXh4uH744Qd988035epZs2aNiouLVVhYqM8//7zcpPuKdO3aVWvWrHE3dO+++666dOnitY4aNWrozjvv1IQJE9y7n1wul5599lm5XC7387z//vv10ksvlWt4PvnkE3300Uf64x//eN7P/7d+/PFH5eTklGvKpFO70U7vzrzxxht14MABrVixwl1TgwYNFBMTo3/961+STjVH48ePr3B5jpycHNWrV0/R0dE6cuSIVq1a5X69W7RoodWrV6usrEz79u3TF198IenUwQudOnVy/xFSWFioxx57TIcOHVLLli21bds25ebmyuVy6aOPPqrwuXXo0EFbtmzRvn37JEnvvPOOOnbsqODgUz+ue/bsqblz56pFixbu1O1/PzPz58931+RJjx499OGHH6qoqEiS9Pbbb+vDDz+UdOrI2969e2vSpEn66aeftH79ehUUFKi4uFjNmzeXMUZvvvmmwsLCznvyfVBQkPtgjc8//1yRkZFq0KCB+3ZPr+P777+vhQsXSjqV7Ddq1Oiifu4Bq5GY4azGjRunmjVr6p577lFpaamCg4PVo0cPPfHEE5JOHQm4Z88e9enTR0FBQUpKSnL/ArzqqqvUt29fhYWFKSIi4rzWJhoyZIj27t2rPn36yBij5s2bl9tld9rQoUP18MMPu49MHDJkiK6++uoLft6nPfjgg7rvvvt02WWXKSUlRcnJyXr88cfdu30qUrt2bd14443q1q2bFi5cqNatW6tPnz5avXr1WeeXnauRI0dq4sSJSktLU5s2bTRx4kT9+c9/du/mbdWqlYYPH65ffvlFCQkJXtOTFi1a6O6779Ydd9yhsrIyXXvtte7vrTcPPvigIiMjNWLECAUHB6u4uFht27bVm2++qdDQUz9WOnbsqL/85S965plndPz4cblcLjVs2FCvvfaaGjZseE7bWbVqlb744gs9/fTT5a5fvny5e+7gb/Xo0UMzZszQsWPHVKNGDXXv3l0ffvihnn/+eUmnGoLnn39eqampmjNnjkJCQjRy5EhVq1btjLH69eunlStXKiEhQQ0aNND48eN1zz33aPbs2Ro7dqy++uorxcfHq1mzZurVq5e7uXvqqaf0+OOPu5uKAQMGqE6dOqpTp44GDhyoW2+9VdHR0erdu7d++eWXM7Zbr149PfHEExo7dqxcLpeuuuqqcs8/KSlJgwYN0syZM93XDR8+XLNmzXJ/Zlq0aKFRo0Z5fX2TkpK0e/duDRgwQMYYXX311Zo+fbp27Nihzz77TB9//LFCQ0M1ZcoUPfbYY/r4449111136ZZbblGtWrV07733qnv37ho9evRZG82KhIWFqbCwUL1799axY8c0ffr0M5qrs72O8fHxeuyxx9SzZ0+FhISoUaNG5V4LwOmCjKnkglQAzsl3332nadOmadmyZT7bxrBhwzRw4EDdcsstPtvG2cyaNUvHjx/XtGnTznqftWvXaubMme4DIQKBMcbdTMyYMUMhISGaOHGixVXZX2Zmpvr27avvv//e6lIAW2JXJuBDLpdLCxYs0LBhw6wuxWf69OmjVatWeTzYoF27dsrPz9f69ev9WJnvrF69WoMGDVJxcbEKCgq0fv36cpPtAaCyaMwAH9m5c6cSEhIUGxt71qURAkHz5s01fPhw3XbbbZoyZUqF96lataqeeeYZPfHEE+rXr5+fK7z4unfv7t6FOWDAAHXr1o3TKwGXsF27dik+Pt690Plvffnllxo4cKCSk5PP6awm7MoEAACopMLCQo0ZM0ZXX321mjZtqqFDh5a7vXfv3nr99ddVp04dDR06VNOmTfN4hD6JGQAAQCWFh4fr1VdfLbcu5Gl79uxRzZo1deWVVyo4OFhdunTRpk2bPI5HYwYAAFBJp89+U5GsrKxyZ0qJiYlRVlaW5/EuanUX2eH8EqtL8MpVav89wTWq2frbrGLX2Vfft4tgB6yT5Cqz/3vxZEmp9zvBq+Bge78fnfCZrh115unG7KbMATONIqtYn+8UuXy/jap+/DVq/SsKAAAQgGJjY8udS/bQoUMV7vL8LRozAADgWMb4/quyrrrqKhUUFGjv3r1yuVxau3atOnTo4PEx9t7HBQAAYGM7duzQrFmztG/fPoWGhio9PV3du3fXVVddpYSEBD3xxBPucxv37t1bjRo18jierZfLYI7ZxcEcswvHHLOLgzlmFwdzzC4cc8wuDjvMMSss8f3rVD3Mf585619RAAAASGJXJgAAcDL7B4vnhcQMAADAJkjMAACAYwVYYEZiBgAAYBckZgAAwLEccPDqeSExAwAAsAkSMwAA4FjGL7PM/LeOGY0ZAABwLnZlAgAAwBdIzAAAgGMFWGBGYgYAAGAXfk/MZsyYoe3btysoKEiTJ09WixYt/F0CAAAIEIG2XIZfG7OtW7cqMzNTS5cu1e7duzV58mQtXbrUnyUAAADYll8bs02bNik+Pl6S1LhxYx09elQFBQWKjIz0ZxkAACBA+Ge5DP/x6xyz7OxsRUdHuy/HxMQoKyvLnyUAAADYlqVHZZpA2zEMAAD8KtBaCb8mZrGxscrOznZfPnz4sGrXru3PEgAAAGzLr41Zhw4dlJ6eLknKyMhQbGws88sAAAD+H7/uymzdurXi4uKUkpKioKAgpaam+nPzAAAAthZkbDzR63B+idUleOUqte3L51ajmr1P8FDsKrO6BK+Cg/x3AtvKcpXZ/714sqTU6hICQnCwvd+PTvhM146qYnUJXpXZ99ezW2QV69epP3Lc9z9XYiJCfL6N06x/RQEAACCJc2UCAAAHYx0zAAAA+ASJGQAAcCwHTMU7LyRmAAAANkFiBgAAHCvAAjMSMwAAALsgMQMAAM4VYJEZjRkAAHAslssAAACAT5CYAQAAx2K5DAAAAPgEiRkAAHCsAAvMSMwAAADswtaJWUQVW5cnSdqTU2h1CV6FhgRZXYJH1cJCrC7BqwN5RVaX4FVYqP3/zoqJCLO6BK9KSu3/93dBkcvqEjwKsvePHElSXmGJ1SV45YTXMbJKuNUlBFxkZv+f5AAAAJcI+0dSAAAAZ8E6ZgAAAPAJEjMAAOBYrGMGAAAAnyAxAwAAjhVggRmJGQAAgF2QmAEAAMdijhkAAAB8gsQMAAA4WGBFZiRmAAAANkFiBgAAHIs5ZgAAAPAJEjMAAOBYARaY0ZgBAADnYlcmAAAAfILEDAAAOJYJsJ2ZJGYAAAA2QWIGAACcK7ACMxIzAAAAuyAxAwAAjhVggRmJGQAAgF2QmAEAAMdiHTMAAAD4BIkZAABwLNYxAwAAgE+QmAEAAOcKrMCMxAwAAMAuSMwAAIBjBVhgRmIGAABgFyRmAADAsVjHDAAAAD5BYgYAAByLdcwAAADgEyRmAADAuQIrMCMxAwAAsAsSMwAA4FgBFpjRmAEAAOdiuQwAAAD4BIkZAABwrEBbLsPWjdmq/xy0ugSvft8gxuoSvAoNDrK6BI9yCoqtLsGr2jWqWF2CVydLSq0uwauDR09aXYJX0RFhVpfgVdUwe+/s+PA/+60uwavujWKtLsGrmIhwq0uABWzdmAEAAHgUWIEZc8wAAADsgsQMAAA4VoAFZiRmAAAAdkFiBgAAHIt1zAAAAOATJGYAAMCxAm0dMxIzAAAAmyAxAwAAzhVYgRmJGQAAgF2QmAEAAMcKsMCMxAwAAMAuSMwAAIBj2WEdsxkzZmj79u0KCgrS5MmT1aJFC/dtb7/9tlasWKHg4GA1b95cf/7znz2ORWIGAABQSVu3blVmZqaWLl2q6dOna/r06e7bCgoK9Prrr+vtt9/WO++8o927d+vbb7/1OB6NGQAAcCzjh3+ebNq0SfHx8ZKkxo0b6+jRoyooKJAkhYWFKSwsTIWFhXK5XDpx4oRq1qzpcTwaMwAAgErKzs5WdHS0+3JMTIyysrIkSVWqVNF9992n+Ph4devWTTfccIMaNWrkcTy/N2azZ89WcnKybr/9dq1evdrfmwcAAIHE+OHrfMr5zaS3goICLVy4UKtWrdKnn36q7du364cffvD4eL9O/t+8ebN++uknLV26VLm5uRowYIB69uzpzxIAAEAAsXruf2xsrLKzs92XDx8+rNq1a0uSdu/erfr16ysmJkaS1KZNG+3YsUPNmjU763h+Tczatm2ruXPnSpJq1KihEydOqLS01J8lAAAAXDQdOnRQenq6JCkjI0OxsbGKjIyUJNWrV0+7d+9WUVGRJGnHjh26+uqrPY7n18QsJCRE1atXlyQtW7ZMnTt3VkhIiD9LAAAAAcTq5TJat26tuLg4paSkKCgoSKmpqUpLS1NUVJQSEhI0atQoDR8+XCEhIWrVqpXatGnjcTxL1jH717/+pWXLlulvf/ubFZsHAAC4aB555JFyl3+7qzIlJUUpKSnnPJbfG7MNGzbo5Zdf1muvvaaoqCh/bx4AAAQQb8tZOI1fG7P8/HzNnj1bb7zxhi677DJ/bhoAAMD2/NqYrVy5Urm5uXrooYfc182aNUt169b1ZxkAACBQBFZg5t/GLDk5WcnJyf7cJAAAgGNwEnMAAOBYARaYcUomAAAAuyAxAwAAjmX1OmYXG4kZAACATZCYAQAAxwq0dcxIzAAAAGyCxAwAADhXYAVmJGYAAAB2QWIGAAAcK8ACMxIzAAAAuyAxAwAAjsU6ZgAAAPAJEjMAAOBYgbaOGY0ZAABwrsDqy+zdmHW/JtbqErzKL3JZXYLjlZSWWV2CVz/uz7e6BK/yS0qsLsGrVvWjrS7Bq0NHi6wuwavQEHvPQrm8WhWrS/CqaliI1SV4tfH/ZFtdglf9r7/C6hICjq0bMwAAAE8CLDBj8j8AAIBdkJgBAADHKguw9TJIzAAAAGyCxAwAADhWYOVlJGYAAAC2QWIGAAAcK8CmmJGYAQAA2AWJGQAAcKxAOyUTiRkAAIBNkJgBAADHKguswIzEDAAAwC5IzAAAgGMxxwwAAAA+QWIGAAAci3XMAAAA4BMkZgAAwLGYYwYAAACfIDEDAACOFWjrmNGYAQAAx2JXJgAAAHyCxAwAADgWy2UAAADAJ0jMAACAY5GYAQAAwCdIzAAAgGOVcVQmAAAAfIHEDAAAOBZzzAAAAOATJGYAAMCxWPkfAAAAPkFiBgAAHIs5ZgAAAPAJEjMAAOBYrGMGAAAAnyAxAwAAjsUcMwAAAPgEiRkAAHCsAAvMSMwAAADsgsQMAAA4lgmwSWa2bsyyjp20ugSvXtj4i9UleDWpW2OrS/Bo+4E8q0vw6j9ZhVaX4FXfJrFWl+BV7vFiq0vwKiQ4yOoSvKoeHmK0vjHVAAAd0klEQVR1CR79vkGM1SV4tT+3yOoSvNp1xP4/d3Dx2boxAwAA8KTM6gIuMhozAADgWIG2K5PJ/wAAADZBYgYAABwrsPIyEjMAAADbIDEDAACOxRwzAAAA+ASJGQAAcKxAWy6DxAwAAMAmSMwAAIBjMccMAAAAPkFiBgAAHCvAAjMSMwAAALsgMQMAAI4VYIEZiRkAAIBdkJgBAADHKguwSWYkZgAAADZhSWNWVFSk+Ph4paWlWbF5AAAQIIwfvvzJksbsr3/9q2rWrGnFpgEAAGzL73PMdu/erZ9//lldu3b196YBAECAYeX/CzRr1ixNmjTJ35sFAACwPb8mZsuXL1fLli1Vv359f24WAAAEqDKrC7jI/NqYrVu3Tnv27NG6det08OBBhYeH64orrlD79u39WQYAAAgQAbYn07+N2QsvvOD+/7x581SvXj2aMgAAgP+HBWYBAIBjBdoCs5Y1ZuPGjbNq0wAAALZEYgYAABwrwAIzTskEAABgFyRmAADAsZhjBgAAALcZM2Zo+/btCgoK0uTJk9WiRQv3bQcOHND48eNVUlKi6667TtOmTfM4FrsyAQCAY5UZ3395snXrVmVmZmrp0qWaPn26pk+fXu72mTNnauTIkVq2bJlCQkK0f/9+j+PRmAEAAFTSpk2bFB8fL0lq3Lixjh49qoKCAklSWVmZvv76a3Xv3l2SlJqaqrp163ocj8YMAAA4ljG+//IkOztb0dHR7ssxMTHKysqSJB05ckQRERF65pln9Ic//EHPPvus1+dDYwYAAHCRmN90csYYHTp0SMOHD9dbb72lnTt3at26dR4fT2MGAAAcq0zG51+exMbGKjs723358OHDql27tiQpOjpadevWVYMGDRQSEqJ27drpp59+8jgejRkAAEAldejQQenp6ZKkjIwMxcbGKjIyUpIUGhqq+vXr65dffnHf3qhRI4/jsVwGAABwLKuXMWvdurXi4uKUkpKioKAgpaamKi0tTVFRUUpISNDkyZM1adIkGWPUpEkT94EAZ0NjBgAAcAEeeeSRcpebNWvm/n/Dhg31zjvvnPNYNGYAAMCxvK0z5jTMMQMAALAJWydm+4+esLoEr0a3ucrqErw6WlhidQkeNY6OtLoEr66JibK6BK+qhNn/76wa1cKsLsGrPTmFVpfgVfUqtv7RrfSfDlpdglchQUFWl+DVsFb1rS7BEQLtXJn2/0kOAABwibD3n10AAAAeBFhgRmMGAACci8n/AAAA8AkSMwAA4FgmwPZlkpgBAADYBIkZAABwLOaYAQAAwCdIzAAAgGORmAEAAMAnSMwAAIBjGQVWZEZiBgAAYBMkZgAAwLGYYwYAAACfIDEDAACOFWAL/5OYAQAA2AWJGQAAcKyyAIvMSMwAAABsgsQMAAA4FkdlAgAAwCdIzAAAgGMF2BQzEjMAAAC7OGtiNn/+fI8PvP/++y96MQAAAOcj0I7KPGtj5nK5JEmZmZnKzMxUmzZtVFZWpq1bt+q6667zW4EAAABnE2B92dkbs4ceekiSNHbsWL3//vsKCQmRJJWUlOhPf/qTf6oDAAC4hHid/H/gwAGZ37SjQUFB2r9/v0+LAgAAOBdlVhdwkXltzLp27arExETFxcUpODhYO3fuVI8ePfxRGwAAwCXFa2P2pz/9SQMGDNCuXbtkjNH999+v3/3ud/6oDQAAwKNAm/x/1uUy1q9fL0latmyZtm3bpmPHjik/P1/ffvutli1b5rcCAQAALhVnTcx+/PFHdenSRV9//XWFtw8cONBnRQEAAJyLAAvMzt6Y3X333ZKkZ555xm/FAAAAXMo4JRMAAHAsTmIOAAAAn/DamJ0+CAAAAMBujDE+//Inr43ZG2+84T49EwAAAHzH6xyzqKgo9enTR9ddd53CwsLc18+ePdunhQEAAHgTaHPMvDZm3bp1U7du3fxRCwAAwCXNa2N2etX/X3/9VfHx8Tp27Jhq1Kjhj9oAAAA8uuQSszfeeEMff/yxiouLFR8fr5deekk1atTQvffe64/6AAAALhleJ/9//PHHeu+991SzZk1J0qOPPqp169b5ui4AAACvAu2oTK+JWUREhIKD/3//FhwcXO6yL9WOrOKX7VyI6lXsv0bvyZJSq0vwKMgBy+kdPVFidQlenSgOsroEr9bsPmx1CV51vbq21SV45Sots7oEj9rWrWV1CV5FVrX/z+7nN/zX6hK8mtmnqdUlBByv78wGDRpo/vz5OnbsmFavXq2VK1eqcePG/qgNAADAI3v/mXL+vEYVU6dOVbVq1VSnTh2tWLFCN9xwg1JTU/1RGwAAwCXFa2I2d+5cDRw4UKNGjfJHPQAAAOfM33PAfM1rY1azZk09/PDDql69um6//Xb16tVLVarYf+4XAACA03htzEaPHq3Ro0drz549+uSTTzRixAg1a9ZMw4YNY64ZAACwVIAFZud+ONzBgweVmZmp48ePKyIiQpMmTdKSJUt8WRsAAIBHZcb4/MufvCZm8+fP14oVK3T11Vdr8ODBmjZtmkJCQlRcXKyBAwdqyJAh/qgTAAAg4HltzEpKSrRo0SLVq1fPfV1mZqYaNmyoRx55xKfFAQAAeBJouzK9Nmbjxo3Txo0b9dVXX0mSiouL9fLLL+uzzz5T586dfV4gAADApcJrY/boo4/q6NGj+vHHH9W6dWtt375d48aN80dtAAAAHgXachleJ/8fPHhQr7/+uho1aqQXX3xRS5Ys0ffff++P2gAAAC4p53yyMJfLpZMnT6pevXr6+eeffVkTAADAOQmwwMx7Y/b73/9er776quLj43XbbbepXr16KisLtDNTAQAAWM9rY/bAAw+otLRUISEhatWqlXJyctSxY0d/1AYAAOCRv9cZ87VzWmA2JCREktS6dWslJCRowYIFPi0KAADgUnTOc8x+67vvvrvYdQAAAJy3wMrLzuOUTL91IYemrlixQv3799dtt92mdevWVXocAACAQFOpxCwoKKhSG8vNzdWCBQv0wQcfqLCwUPPmzVPXrl0rNRYAAECgrWN21sasS5cuFTZgxhjl5uZWamObNm1Su3btFBkZqcjISD311FOVGgcAACAQnbUxW7JkyUXf2N69e1VUVKSxY8fq2LFjGjdunNq1a3fRtwMAAC4NZYEVmJ29MfvtScsvpry8PM2fP1/79+/X8OHDtXbt2krvGgUAAAgklZpjVlm1atVSq1atFBoaqgYNGigiIkJHjhxRrVq1/FkGAAAIEIE2x6xSR2VWVseOHbV582aVlZUpNzdXhYWFio6O9mcJAAAAtuXXxKxOnTpKTEzU4MGDJUlTpkxRcLBfe0MAABBAAiww829jJkkpKSlKSUnx92YBAABsz++NGQAAwMUSaHPMaMwAAIBjBdpyGUzwAgAAsAkSMwAA4FiBtiuTxAwAAMAmSMwAAIBjBVZeRmIGAABgGyRmAADAscqYYwYAAABfIDEDAACOFWCBGYkZAACAXZCYAQAAx2IdMwAAAPgEiRkAAHCsAAvMSMwAAADswtaJWUmp/dvgE8WlVpfg1c85+VaX4FGTy6OsLsGrGZ/+ZHUJXk3vfa3VJXjVp+kVVpfgVU5+sdUleBVZ1dY/uhUcZHUF3mXmHLe6BK+Sm19pdQmOwDpmAAAA8Al7/9kFAADgQYAFZiRmAAAAdkFiBgAAHIt1zAAAAOA2Y8YMJScnKyUlRd99912F93n22Wc1bNgwr2ORmAEAAMcqszgw27p1qzIzM7V06VLt3r1bkydP1tKlS8vd5+eff9ZXX32lsLAwr+ORmAEAAMcyfvjnyaZNmxQfHy9Jaty4sY4ePaqCgoJy95k5c6b+9Kc/ndPzoTEDAACopOzsbEVHR7svx8TEKCsry305LS1NN910k+rVq3dO49GYAQAAxzLG91/nV8//f0BeXp7S0tJ01113nfPjacwAAAAqKTY2VtnZ2e7Lhw8fVu3atSVJmzdv1pEjR3THHXfo/vvvV0ZGhmbMmOFxPBozAADgWMYYn3950qFDB6Wnp0uSMjIyFBsbq8jISElSUlKSVq5cqffee0/z589XXFycJk+e7HE8jsoEAACopNatWysuLk4pKSkKCgpSamqq0tLSFBUVpYSEhPMej8YMAAA4ltXLZUjSI488Uu5ys2bNzrjPVVddpcWLF3sdi12ZAAAANkFiBgAAHItTMgEAAMAnSMwAAIBjBVhgRmIGAABgFyRmAADAscoCLDIjMQMAALAJEjMAAOBYARaYkZgBAADYBYkZAABwLNYxAwAAgE+QmAEAAMcKsMCMxAwAAMAuSMwAAIBjBdocMxozAADgWAHWl7ErEwAAwC5IzAAAgGMF2q5MEjMAAACbIDEDAACORWIGAAAAnyAxAwAAjhVggRmJGQAAgF2QmAEAAMdijhkAAAB8gsQMAAA4VoAFZiRmAAAAdkFiBgAAHCvQ5pjZujErdpVZXYJX9WOqWV2CV39Zv9vqEjy652b7v4aj29e3ugSvDhcUWV2CVzmFQVaX4FXtiCpWl+DV1j05VpfgUePLIq0uwauVP2VbXYJXtzWrY3UJsICtGzMAAABPAiwwY44ZAACAXZCYAQAAxwq0OWYkZgAAADZBYgYAABwrwAIzEjMAAAC7IDEDAACOFWhzzGjMAACAYwVYX8auTAAAALsgMQMAAI4VaLsyScwAAABsgsQMAAA4VoAFZiRmAAAAdkFiBgAAHIs5ZgAAAPAJEjMAAOBYARaY+bcxO378uCZOnKijR4+qpKRE9913nzp16uTPEgAAAGzLr43ZP/7xDzVq1EgPP/ywDh06pBEjRmjVqlX+LAEAAAQQ5phdgOjoaOXl5UmSjh07pujoaH9uHgAAwNb8mpj16dNHaWlpSkhI0LFjx7Rw4UJ/bh4AAAQYErML8OGHH6pu3bpas2aN/v73v2vatGn+3DwAAICt+TUx+/e//62OHTtKkpo1a6bDhw+rtLRUISEh/iwDAAAEiAALzPybmDVs2FDbt2+XJO3bt08RERE0ZQAAAP+PXxOz5ORkTZ48WUOHDpXL5dITTzzhz80DAIAAE2hzzPzamEVERGju3Ln+3CQAAIBjsPI/AABwrAALzDhXJgAAgF2QmAEAAMcqKwusyIzEDAAAwCZIzAAAgGMF2hwzGjMAAOBYgbZcBrsyAQAAbILEDAAAOFaABWYkZgAAAHZBYgYAAByLOWYAAADwCRIzAADgWAEWmJGYAQAA2AWJGQAAcKxAm2Nm68bsb//ea3UJXv3h+iutLsGr5Bb2rtEJH6r6NSKsLsGrHVl5VpfgVcahE1aX4FVyc3t/XiTp8/9z1OoSPKrRNMzqErxqXS/S6hK8qhYeYnUJsICtGzMAAABPnPDH/flgjhkAAIBNkJgBAADnCqzAjMQMAADALkjMAACAYzHHDAAAAD5BYgYAAByLxAwAAAA+QWIGAAAci8QMAAAAPkFiBgAAHCvQEjMaMwAA4FyB1ZexKxMAAMAuSMwAAIBjBdquTBIzAAAAmyAxAwAAjkViBgAAAJ8gMQMAAI5FYgYAAACfIDEDAADOFViBGYkZAACAXZCYAQAAx7LDHLMZM2Zo+/btCgoK0uTJk9WiRQv3bZs3b9Zzzz2n4OBgNWrUSNOnT1dw8NlzMRIzAACAStq6dasyMzO1dOlSTZ8+XdOnTy93+9SpU/Xiiy/q3Xff1fHjx7VhwwaP45GYAQAAx7I6Mdu0aZPi4+MlSY0bN9bRo0dVUFCgyMhISVJaWpr7/zExMcrNzfU4HokZAABAJWVnZys6Otp9OSYmRllZWe7Lp5uyw4cPa+PGjerSpYvH8UjMAACAY1mdmP2viurJycnR2LFjlZqaWq6JqwiJGQAAQCXFxsYqOzvbffnw4cOqXbu2+3JBQYFGjx6thx56SB07dvQ6Ho0ZAABwLGOMz7886dChg9LT0yVJGRkZio2Nde++lKSZM2dqxIgR6ty58zk9H3ZlAgAAVFLr1q0VFxenlJQUBQUFKTU1VWlpaYqKilLHjh21fPlyZWZmatmyZZKkvn37Kjk5+azj0ZgBAADnssEUs0ceeaTc5WbNmrn/v2PHjvMai12ZAAAANkFiBgAAHMtuR2VeKBozAADgWIHWmLErEwAAwCZIzAAAgGORmAEAAMAnSMwAAIBzBVZgRmIGAABgFyRmAADAsZhjBgAAAJ+wdWJ2U4NI73eymHHAzu3LqoRZXYJH4aH2//ugqKTM6hK8ahAVYXUJXl1dw/6f6Y9+PGR1CV4NvK6O1SV4lFN00uoSvHpg/karS/CqY/vGVpfgVdqoG60ugcQMAAAAvmHrxAwAAMATEjMAAAD4BIkZAABwLBIzAAAA+ASJGQAAcK7ACsxIzAAAAOyCxAwAADgWc8wAAADgEyRmAADAsUjMAAAA4BMkZgAAwLECLTGjMQMAAI4VaI0ZuzIBAABsgsQMAAA4V2AFZiRmAAAAduHTxmzXrl2Kj4/XW2+9JUk6cOCAhg0bpiFDhujBBx9UcXGxLzcPAAACnDHG51/+5LPGrLCwUE899ZTatWvnvu7FF1/UkCFDtGTJEjVs2FDLli3z1eYBAAAcx2eNWXh4uF599VXFxsa6r9uyZYt69OghSerWrZs2bdrkq80DAIBLQKAlZj6b/B8aGqrQ0PLDnzhxQuHh4ZKkWrVqKSsry1ebBwAAcBzLjsoMtHVHAACABQKsn/DrUZnVq1dXUVGRJOnQoUPldnMCAABc6vzamLVv317p6emSpNWrV6tTp07+3DwAAAg0psz3X37ks12ZO3bs0KxZs7Rv3z6FhoYqPT1dc+bM0aRJk7R06VLVrVtXt956q682DwAA4Dg+a8yaN2+uxYsXn3H9okWLfLVJAABwqWGOGQAAAHyBc2UCAADn8vMcMF8jMQMAALAJEjMAAOBczDEDAACAL5CYAQAA52KOGQAAAHyBxAwAADgXiRkAAAB8gcQMAAA4V4AdlUljBgAAnItdmQAAAPAFEjMAAOBcAbYrk8QMAADAJkjMAACAcwXYHDNbN2adGta2ugSvdh46ZnUJXv05PcPqEjx6cVBLq0vwatO+I1aX4FXnBrWsLsGr6lVs/SNHkpR8fV2rS/Dq81+zrC7BowNHS6wuwavPn+5jdQleFbsCq+HAubH/T0kAAICzYY4ZAAAAfIHEDAAAOFeAzTEjMQMAALAJEjMAAOBczDEDAACAL5CYAQAA52KOGQAAAHyBxAwAADgXc8wAAADgCyRmAADAuZhjBgAAAF8gMQMAAM5VxhwzAAAA+ACJGQAAcK4Am2NGYwYAAJwrwBozdmUCAADYBIkZAABwLhaYBQAAgC+QmAEAAOdijhkAAAB8gcQMAAA4F3PMAAAA4AskZgAAwLmYYwYAAABfIDEDAADOxRwzAAAA+AKJGQAAcC7mmAEAAMAXSMwAAIBzMccMAAAAvkBiBgAAnIs5ZgAAAPAFEjMAAOBczDEDAACAL5CYAQAA5wqwOWY0ZgAAwLlssCtzxowZ2r59u4KCgjR58mS1aNHCfduXX36p5557TiEhIercubPuu+8+j2OxKxMAAKCStm7dqszMTC1dulTTp0/X9OnTy93+9NNPa968eXrnnXe0ceNG/fzzzx7HozEDAADOZcp8/+XBpk2bFB8fL0lq3Lixjh49qoKCAknSnj17VLNmTV155ZUKDg5Wly5dtGnTJo/j0ZgBAABUUnZ2tqKjo92XY2JilJWVJUnKyspSTExMhbedja3nmF1Tp7rVJXjlhBpvaXGF1SU43o1X17C6BMDt2roRVpcA2MaJb+ZbXUI55gLnvJGYAQAAVFJsbKyys7Pdlw8fPqzatWtXeNuhQ4cUGxvrcTwaMwAAgErq0KGD0tPTJUkZGRmKjY1VZGSkJOmqq65SQUGB9u7dK5fLpbVr16pDhw4exwsyF5q5AQAAXMLmzJmjbdu2KSgoSKmpqdq5c6eioqKUkJCgr776SnPmzJEk9ezZU6NGjfI4Fo0ZAACATbArEwAAwCZozAAAAGzikmnMZsyYoeTkZKWkpOi7776zuhzHmj17tpKTk3X77bdr9erVVpfjWEVFRYqPj1daWprVpTjWihUr1L9/f912221at26d1eU40vHjx3X//fdr2LBhSklJ0YYNG6wuyVF27dql+Ph4vfXWW5KkAwcOaNiwYRoyZIgefPBBFRcXW1whnOiSaMy8nS4B52bz5s366aeftHTpUr322muaMWOG1SU51l//+lfVrFnT6jIcKzc3VwsWLNCSJUv08ssv69NPP7W6JEf6xz/+oUaNGmnx4sWaO3cuPxvPQ2FhoZ566im1a9fOfd2LL76oIUOGaMmSJWrYsKGWLVtmYYVwqkuiMfN0ugScu7Zt22ru3LmSpBo1aujEiRMqLS21uCrn2b17t37++Wd17drV6lIca9OmTWrXrp0iIyMVGxurp556yuqSHCk6Olp5eXmSpGPHjpVbvRyehYeH69VXXy23JtWWLVvUo0cPSVK3bt28nnoHqMgl0Zh5Ol0Czl1ISIiqVz91poNly5apc+fOCgkJsbgq55k1a5YmTZpkdRmOtnfvXhUVFWns2LEaMmQIvwArqU+fPtq/f78SEhI0dOhQTZw40eqSHCM0NFRVq1Ytd92JEycUHh4uSapVqxa/Z1Aptj4lk6+wQsiF+de//qVly5bpb3/7m9WlOM7y5cvVsmVL1a9f3+pSHC8vL0/z58/X/v37NXz4cK1du1ZBQUFWl+UoH374oerWravXX39dP/zwgyZPnsy8x4uE3zOorEuiMfN0ugScnw0bNujll1/Wa6+9pqioKKvLcZx169Zpz549WrdunQ4ePKjw8HBdccUVat++vdWlOUqtWrXUqlUrhYaGqkGDBoqIiNCRI0dUq1Ytq0tzlH//+9/q2LGjJKlZs2Y6fPiwSktLScIrqXr16ioqKlLVqlXP6dQ7QEUuiV2Znk6XgHOXn5+v2bNna+HChbrsssusLseRXnjhBX3wwQd67733NGjQIN177700ZZXQsWNHbd68WWVlZcrNzVVhYSHzoyqhYcOG2r59uyRp3759ioiIoCm7AO3bt3f/rlm9erU6depkcUVwoksiMWvdurXi4uKUkpLiPl0Czt/KlSuVm5urhx56yH3drFmzVLduXQurwqWoTp06SkxM1ODBgyVJU6ZMUXDwJfF35kWVnJysyZMna+jQoXK5XHriiSesLskxduzYoVmzZmnfvn0KDQ1Venq65syZo0mTJmnp0qWqW7eubr31VqvLhANxSiYAAACb4E9MAAAAm6AxAwAAsAkaMwAAAJugMQMAALAJGjMAAACboDED4NHevXvVuXPnM65v2rSpXC7XBY///PPPa968eWdcP2zYsArPxdq5c2ft3bv3grcLAHZ0SaxjBsB5Fi9ebHUJAOB3NGYALthzzz2nf//73yoqKlLbtm316KOPyhij1NRU/fe//1VxcbFuuOEGTZkyRdKplGzt2rW68sorVa1aNTVu3PiMMZs2baqMjAzl5eXpoYceUmlpqeLi4jgHIYCARmMG4IJ88sknOnTokN566y1J0n333ae1a9eqVatWatq0qZ566ilJUlJSknbt2qWwsDB99NFHWrVqlYKDgzVo0KAKG7PT3nzzTd1www2aMGGCMjIySNIABDQaMwBeHTlyRMOGDavwti1btujbb791356fn6+9e/eqS5cuOnDggJKTkxUeHq6srCzl5uYqLy9PcXFxCg8PlyS1adPG47Z37drlPvVSXFycoqKiLuIzAwB7oTED4FVMTMwZSVXTpk0lSeHh4Ro8eLBGjRpV7vYVK1bo+++/19tvv63Q0FDddtttkiRjjIKCgtz3Kysr87htY0y582BWdEAAAAQKjsoEcEFuvPFGrVmzxn2E5vz58/XLL78oJydHjRo1UmhoqHbs2KFff/1VxcXFaty4sXbu3Kni4mKVlJRo69atHsdv3LixvvnmG0nS9u3bVVhY6PPnBABWITEDcEF69uypb7/9VikpKQoJCdF1112n+vXrKykpSWPHjtXQoUPVunVrjRw5Uk8//bTee+89xcfHa/Dgwapbt66uvfZaj+OPGDFCDz74oIYPH65rrrlG9evX99MzAwD/CzIc4gQAAGAL7MoEAACwCRozAAAAm6AxAwAAsAkaMwAAAJugMQMAALAJGjMAAACboDEDAACwCRozAAAAm/i/dWVkkFToFQcAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "all_similarities = []\n",
+ "for example in tqdm(data):\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(example[1], example[2], max_len)\n",
+ " _, output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " _, output_base = model_base(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " \n",
+ " finetuned_vec = output_finetuned.squeeze(0).view(n_layers, n_heads, -1)\n",
+ " base_vec = output_base.squeeze(0).view(n_layers, n_heads, -1)\n",
+ "\n",
+ " sim_map = torch.nn.functional.cosine_similarity(base_vec, finetuned_vec, dim=-1).detach().numpy()\n",
+ " all_similarities.append(sim_map)\n",
+ "all_similarities = np.stack(all_similarities, axis=-1)\n",
+ "avg_sim = np.mean(all_similarities, axis=-1)\n",
+ "plt.figure(figsize=(12, 9))\n",
+ "plt.imshow(avg_sim, cmap='Blues_r', vmin=0, vmax=1)\n",
+ "plt.colorbar()\n",
+ "plt.grid(False)\n",
+ "plt.title('Cosine similarity map for {}. Averaged over examples'.format(task))\n",
+ "plt.xlabel('Head id')\n",
+ "plt.ylabel('Layer id')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.1 Target nouns"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 48,
+ "metadata": {
+ "scrolled": false
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:51<00:00, 2.85it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAsAAAJ1CAYAAAC/wnk8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XuY1HXdP/7n7iKBgMppxSOiJipGaoop3qgEipqWqIAHtLq70W7Ls0LoLZ4PVF4iHbRbze4sW0VMSfOUmocwwsqUTkZIeEBAYQUREJbfH353f07AsOCyszSPR9de187M5/CaYfM985rn+/2pWLly5coAAAAA/D+VpS4AAAAAaFk0CwAAAIACmgUAAABAAc0CAAAAoIBmAQAAAFBAswAAAAAooFkAAGVs/Pjx6dmzZ37zm9+UuhQAoAXRLACAj+C6665Lz549s88++2TJkiVr3O6JJ57IY489VnDfu+++m/Hjx+edd97Z0GUmSf72t7/l9ttvL7jv8MMPz7hx4/Lxj3+8WWpYndW9Nk2hf//+6dmzZ/7nf/5njdtMnDhxjc2S5cuX5957782IESNy4IEHZo899kifPn1y7LHH5sYbb8xbb7212mP27NkzgwYNKlpbfZPmgQceWOW+Aw44oOjfRM+ePTNq1Kiix0+S2bNnZ/z48WvdDgBWR7MAANbTsmXLcu+996aysjILFy7MQw89tMZtb7311lU+EL/44ov59re/3WzNgoceeij/93//V3DfzjvvnEGDBqVTp07NUsPqrO61aUp33313XnjhhXXaZ/bs2RkyZEhGjRqV2tranHLKKbnmmmty3nnnZeedd873v//9DBw4ML/85S+bvN633nor119//Uc+zrPPPptvf/vbTVARAOVIswAA1tMjjzyS+fPnZ9iwYamoqMhdd9212u3q6uoybdq0Ve5/8cUXN3SJJT1fY6zptWkqu+yyS7bYYotceumlWbFiRaP2Wbp0aU477bT8+c9/zhVXXJGampqMGDEiRx11VIYOHZrrrrsu999/f7bYYoucddZZ+cMf/tCkNe+3336pqan5yP9eLfHfG4CNh2YBAKynmpqaJMmpp56aT33qU3n++eczffr0gm0mTpyY3XbbLYsXL869996bnj17Zvz48enfv3+++c1vJkk+85nPpGfPngX7TZgwIccdd1w++clPZq+99soxxxyTH/3oR6mrq2vY5tVXX03Pnj1z0UUX5eWXX86Xv/zl7LPPPundu3dOOumkvPTSSwXbPfXUU3nttdfSs2fPDB8+PMma1yx46qmncuqpp2afffbJHnvskf79++fKK6/M22+/XbBd//79M3DgwCxcuDAXX3xx+vbtmz322COHH354Jk2aVPT1W9NrU2/69Ok599xz07dv3/Tq1SsHHHBAzjzzzPz1r39d679NvS222CLnnXde/vSnP+WOO+5o1D533313/vKXv+SUU07JkCFDVrvNjjvumHHjxmX58uW59tprG11PY5x//vnZbLPNMmbMmIJ/73XRs2fP3HnnnQ2/9+/fv+GxN998M1deeWUOOeSQ7LHHHtl///0zfPjwPPvss6scZ9GiRbniiity4IEHpnfv3jnuuOPy7LPP5u67707Pnj0zceLEhm3nzZuXa665Jocddlg++clPpk+fPhk6dGjuvffe9XoOAJSWZgEArIcZM2ZkypQp2WuvvbLDDjvk85//fJIPPmh+2H777ZcxY8YkSfr06ZNx48bl8MMPz5gxY9KnT58kyZgxYzJu3LiGfa699tpcdNFFqa6uzsUXX5wLL7wwXbt2zZVXXrna+fdz5szJF7/4xey000656KKLctJJJ+UPf/hDTj/99CxbtiydO3fOuHHj0qlTp3Tq1Cnjxo3L1772tTU+t/p5+nPnzs3pp5+eyy+/PIccckjuvPPOnHjiiVm8eHHB9nV1dRkxYkTeeeednHPOOTn77LOzYMGCXHjhhUVTA2t6bZIP1lcYMmRInn322Rx33HG56qqrcuKJJ+a3v/1thg0blj//+c9rPO6/Ou6447LXXnvlxhtvzJw5c9a6/c9+9rNUVFTkS1/6UtHtPvGJT2S//fbL73//+8yYMaPR9axNp06dcs4552TatGkNH/jX1bhx47Lzzjs3/F7/Oi9evDgnn3xyampqcvTRR+eaa67JV77ylbz11lv50pe+tMp0kPPPPz933HFHevfunYsvvjj/8R//kXPPPTe//vWvC7ZbsWJFvvCFL+QnP/lJBg4cmMsvvzznnXde2rZtm1GjRq0y/QWAlq9VqQsAgI1R/ZSDY489NskHCwVeddVV+dnPfpZzzz03rVu3TpJss8026devX8Pv9Qvf7bzzzvnFL36RJOnXr1+23XbbJMlf/vKX/OAHP8iJJ57Y8AEvSU444YSceeaZmTBhQk466aTsvvvuDY899dRTueGGGxo+aCdJbW1t7rnnnjz//PPZf//9M2jQoIwdOzZJii6+t2TJklxzzTXp1KlTfvrTn2azzTZLkgwePDhbbbVVvvGNb+SOO+7IiBEjGvZ59dVX069fv4J6u3TpkpEjR+axxx5Lr169VnuuNb02STJ27NgsWrQoP/3pT7PXXns13N+vX78cf/zx+da3vpVbbrlljc/jwyoqKnLppZdm8ODBufbaa4uuB/D+++/nz3/+c7p3754tt9xyrcfeb7/98txzz+UPf/hDevTo0ah6GmPIkCG55557csMNN+Swww5Lly5d1mn/QYMG5cc//nHD7/VmzpyZ7bbbLieeeGK++MUvNtx/4IEH5vDDD88dd9yRAQMGJEmmTZuWJ554Ivvuu2+++93vNmzbt2/fnHzyyQXn++tf/5qXX345J510Us4///yC53H++edn9uzZ61Q/AKUnWQAA66h+YcO2bds2fEBv3759Dj300MyfP/8jLdZX30A44ogj8s477xT8HHbYYUmSKVOmFOzTrVu3gkZB8sG33kkyd+7cdTr/lClTUltbmyOOOKKhUVBv8ODBSZInn3xylf0+/MHzo5w/+eDb72effTY9e/YsaBQkSe/evbPLLrtk8uTJWbp0aaOPueuuu+akk07KAw88sMq34h+2YMGCLF++PNXV1Y06brdu3ZKs3/MsprKyMpdeemkWLVrUpNMcdtttt9x2220N/17vvfde3nnnnXTp0iWtWrXKa6+91rBt/dSUz372swXH2GeffbL33nsX3FdVVZUk+dOf/pT33nuv4f6Kiop861vfyoUXXthkzwGA5qFZAADrqH5hw4EDB6Z9+/YN99d/mF7TQoeN8fe//z1JcvLJJ2ffffct+Dn33HOTJK+//nrBPttvv/0qx/nYxz6W5IPL/62Lf/zjH0k+WBjwX3Xq1ClbbLFFXnnllYL7q6qqss022zTJ+ZMPvv2uq6tb4+Uce/TokeXLl+fVV19dp+OeddZZ6dq1ay677LIsW7ZstdtUVn7w1mjlypWNOmb9dvUflptSr169csIJJ2TSpEmrvbTj+po8eXJOOeWUfOpTn8qee+7Z8Pe1fPnygkUg6xsH3bt3X+UYn/zkJwtu9+zZM4ceemh+//vf55BDDsno0aMzadKkVda4AGDjYRoCAKyj+oUN+/Tpk5kzZzbc361bt3Tp0iXPPfdcZs2ale22226dj/3uu+8mSa6//vo1Rs+7du1acLt+ykNTqF+PoG3btqt9vE2bNqtc6rGqqqpJPyzXvwabbrrpah+vb0T869oJa9O+ffuMGjUq5513Xm655Zb893//9yrbbLHFFmnduvUqDZk1eeONN5Kk0UmEevVNhoqKiqLbnX322Xn44Ydz2WWX5b777ssmm2yyTuf5V88880z+67/+Kx06dMgXv/jF7L777mnXrl2S5D//8z8Ltq1PCLRp02aV43To0GGV+66//vrce++9mTBhQiZOnJh77rknrVq1yuGHH55LLrlklaQKAC2bZgEArIN//OMfDdMALr744jVuN2HChJxzzjnrfPz6D27bbbddevfuvX5FfgT1H9DX9EH8vffea6hxQ6k/frEaPrzduvjsZz+bCRMm5Oabb85RRx21yuNVVVXp3bt3pk6dmpkzZ672W/UPq/9b2GeffRru69ChQxYtWlR0vwULFiRJOnbsWHS7zTbbLBdccEFGjhyZ2267LaeddlrR7dfm9ttvT11dXcaNG5f999+/4f4lS5ascmnJ+ibU6qZ7rO75bbLJJhkyZEiGDBmSt99+O88880xqamoyadKkzJs3L7fffvtHqh2A5mUaAgCsg/opBscff3zGjRu3ys/YsWNTVVWVe+65Z70i+PUr2P/ud79b5bF33313nebpr4/68//tb39b5bG5c+emtrY2O+644watYYcddkhVVdVqa0g+mKrRunXrhkUh19Ull1ySFStW5Morr1zt4/WLVt50001Fj/PSSy9lypQpOfDAA7PVVls13L/77rtn7ty5mTVr1mr3W7lyZZ577rm0atVqjYs/ftjnP//59OnTJ9/73vcK1hRYH6+++moqKyvz6U9/uuD+559/fpXLNNYv8Li6lMULL7xQ9DydOnXK0UcfnR/96Efp1atXJk+enIULF36k2gFoXpoFANBI9Qsbtm7dOueee24GDRq0ys/nPve5DBgwIHPnzm1YCLA+ov+vH/Tr58d/+P76hQrvvPPOLFmypGD7b3zjG/n0pz+df/7zn+tVf2Vl5VqbDX369EmnTp3y4IMPpra2tuCx+ukX9QstNoXVvTZt27bNQQcdlL/97W+ZOnVqwfZTpkzJjBkzcsghh6z39Isdd9wxX/rSl/Lkk0/m0UcfXeXxz33uc9lrr70yceLE3Hrrras9xsyZM3PWWWelTZs2GTVqVMFj9VcKuOaaa1bbMLr11lszffr0HHPMMY2O5l9yySVZvnz5Ghscq7O6v68uXbqkrq6uoAFQW1ubcePGpW3btgV/c/WLS9Yvullv6tSpqzSz7rrrrvTr1y/Tp09fpYZ27dqlqqqqoR4ANg6mIQBAIz388MNZsGBBBg8enE6dOq1xu5NPPjkPP/xw7r777gwYMCBdunRJmzZt8vTTT+fmm29O9+7dM2jQoIZvxr/xjW9k3333zec+97nsuuuuOfXUU/PDH/4wJ5xwQoYOHZpWrVrlV7/6VR555JEcffTRq13QsDG23XbbTJ48Oddcc0222mqrfOELX1hlm9atW+d//ud/cu655+akk07Kcccdlw4dOuSFF17I3XffnT333DPHH3/8ep1/ddb02lx44YWZOnVqzjjjjAwfPjzbbrttXnnllfzkJz9Jx44dc8EFF3yk837lK1/Jz3/+8zz++OOrPFZVVZXvfve7+epXv5qxY8fmwQcfzGGHHZatttoqixcvzh/+8Ic88MADadu2bW666aZVFmI89NBD86UvfSm33XZbBg8enEGDBmXrrbfOvHnz8utf/zrPPvts9tlnn4wePbrR9X784x/PKaecssbmxerU/32NGTMmO+64Y77whS/k8MMPz29/+9ucffbZOeGEE/LOO+/kJz/5SYYOHZrKysr8/ve/z/e///0MGDAg++23X/bYY4889dRTOe+883LAAQfktddey1133ZUjjzwykyZNajjXfvvtl7Fjx+aUU07JsGHD0r179yxbtizPPPNMpkyZkuOPP36DT18BoGlpFgBAI9V/s37qqacW3a5Pnz7ZZZdd8vTTT2f27Nnp1q1bRo0alRtuuCHf/e53M2TIkAwaNCjDhg3LM888k2eeeSYvvfRSDj300CTJ6NGj8/GPfzw1NTW55pprUldXlx122CEXXHDBaj/gN9ZZZ52V1157LT/+8Y/Ts2fPNR7riCOOyOabb56bb745N954Y5YtW5ZtttkmI0aMyGmnndakCypusskmq31tevTokbvuuis33nhjfvKTn6S2tjYdO3ZM//79c8YZZ6zX4pEf1rZt21x00UWrXeQw+SBGf8cdd+TBBx/Mz3/+8/zf//1fFixYkLZt22aHHXbIaaedlpNPPjmbb775avcfOXJkDjjggPz4xz/OHXfckdra2my66abp2bNnLr/88hx33HHrvCjkV7/61Tz44IMNiyquzZe//OW8+OKL+fnPf56uXbs2fJCfP39+Jk6cmMsuuyzdu3fPiBEjctxxx+XjH/94Ro8ene9973vZeuuts+OOO+amm27K1VdfnV/96ld54oknsscee+Tb3/52w9UZ6tMC3bt3T01NTW6++eZMmDAhb731Vlq1apUdd9wxo0ePbkhbALDxqFjZ2GsDAQBAkmuvvTY/+MEP8v3vfz8HHXRQqcsBYAMweQwAgFUsWbIk5513Xr7+9a8X3L906dI89NBD2WSTTfKJT3yiRNUBsKGZhgAAwCratGmTJJk4cWJqa2szYMCALFmyJBMmTMgbb7yRL3/5y0XX7gBg42YaAgAAq/X+++/ntttuy3333Zc33ngjdXV12XHHHXPcccflxBNPTEVFRalLBGAD0SwAAAAAClizAAAAACigWQAAAAAU0CwAAAAACmgWAAAAAAU0CwAAAIACmgUAAABAAc0CAAAAoIBmAQAAAFBAswAAAAAooFkAAAAAFNAsAAAAAApoFgAAAAAFNAsAAACAApoFAAAAQAHNAgAAAKCAZgEAAABQQLMAAAAAKKBZAAAAABTQLAAAAAAKaBYAAAAABTQLAAAAgAKaBQAAAEABzQIAAACggGYBAAAAUECzAAAAACigWQAAAAAU0CwAAAAACmgWAAAAAAU0CwAAAIACmgUAAABAAc0CAAAAoIBmAQAAAFBAswAAAAAooFkAAAAAFNAsAAAAAApoFgAAAAAFNAsAAACAApoFAAAAQAHNAgAAAKCAZgEAAABQQLMAAAAAKKBZAAAAABTQLAAAAAAKaBYAAAAABTQLAAAAgAKaBQAAAEABzQIAAACggGYBAAAAUECzAAAAACigWQAAAAAU0CwAAAAACrQqdQEAAADQklUM3LbZzrXy0Veb7VzFSBYAAAAABTQLAAAAgAKmIQAAAEAxFRWlrqDZSRYAAAAABSQLAAAAoJgy/Jq9DJ8yAAAAUIxkAQAAABRjzQIAAACg3EkWAAAAQDHlFyyQLAAAAAAKSRYAAABAMdYsAAAAAMqdZAEAAAAUU4Zfs5fhUwYAAACK0SwAAAAACpiGAAAAAMVY4BAAAAAod5IFAAAAUEz5BQskCwAAAIBCkgUAAABQTGX5RQskCwAAAIACkgUAAABQTPkFCyQLAAAAgEKSBQAAAFBMRflFCyQLAAAAgAKSBQAAAFBM+QULJAsAAACAQpIFAAAAUExl+UULJAsAAACAApIFAAAAUEz5BQskCwAAAIBCmgUAAABAAdMQAAAAoJiK8puHIFkAAAAAFJAsAAAAgGJcOhEAAAAod5IFAAAAUEz5BQskCwAAAIBCkgUAAABQjKshAAAAAOVOsgAAAACKKb9ggWQBAAAAUEiyAAAAAIqpLL9ogWQBAAAAUECyAAAAAIopv2CBZAEAAABQSLIAAAAAiqkov2iBZAEAAABQQLMAAAAAKGAaAgAAABRThl+zl+FTBgAAAIqRLAAAAIBiLHAIAAAAlDvJAgAAACim/IIFkgWwOj179szAgQMzaNCgHHbYYfnyl7+cWbNmlbqs9TJw4MD85je/KbrNqFGj8t3vfreZKgKgXH14fK3/+c///M8kyamnnppp06Y12bmefvrpvP766012vGIuuuiijB8/vlnOtfvuu+fVV18tus3w4cNz3333NUs9wL8vyQJYgx/96Efp1q1bkuRb3/pWrrrqqtx0000lrgoANm4fHl8/7Ic//GGTnuf222/PV77ylWy99dZNelygTJXhmgWaBdAIn/70p/P444833L777rtz2223ZcWKFenatWvGjh2bbbbZJm+++WYuvPDCzJ07N8uWLcuRRx6Zc845JytXrsx3vvOdTJo0KcuWLctnPvOZfP3rX09VVVXBeSZOnJgnn3wym2yySZ5//vn06NEjZ5xxRr75zW/mn//8Z84666wMHTo0dXV1GTduXB5++OEkyZ577plLLrkkm266aV566aWMHDkyy5cvz0EHHVRw/Mceeyzjxo3L4sWL071793zzm99Mp06dNvwLCABr0b9//4wdOzbdunXLsGHDMmLEiNx9991ZsGBBvv71r+eII45o9Hh6ww035Lnnnss//vGPXHDBBXnqqaey/fbb57//+7+TfJCoq7/dv3//jBgxIhMmTMjs2bPz2c9+NqNGjUqy5nFz/vz5Oe+88/LKK69k5513Tps2bVbbABk/fnzmzZuX2bNnZ9q0adl///1zxBFHZPz48ZkzZ06uuOKKHHLIIVm6dGmuuuqq/OY3v0llZWUOOuigXHDBBamqqsqvfvWrXHnllWnVqlWOPfbYguPX1NTkBz/4QZYtW5Y999wzV199ddq0abOB/oWAcmMaAqzFsmXLcv/996d///5JkrfeeiuXX355fvCDH+SRRx7J9ttv3xDhv/3227PvvvvmwQcfzKRJkzJr1qzMmTMn9913Xx566KFMmDAhjz76aGbNmpU777xzted7+umn89WvfjWPPPJIpk+fnltuuSU//vGPc9VVVzWc5xe/+EWeeuqpTJw4MQ888EDeeeed3H777UmSSy+9NKecckoefvjh7LXXXg1RxVmzZuXCCy/Mt771rfzyl7/Mfvvtl0svvXTDvngAsB7mz5+fysrKTJo0KaNHj84NN9yQJI0eT88+++xsueWW+cY3vpEjjjhiref77W9/m5qamtxzzz254447Mnv27KLj5v/+7/+mY8eOefzxx3PJJZfkmWeeWeOxn3zyyVx99dWZNGlSHnrooYbx+/TTT8///u//JvkgVTF79uw88MADuffeezN16tT8/Oc/z4oVK3LRRRdlzJgx+cUvfpHKysqsWLEiSTJ16tSMGzcuP/zhD/P444+nffv2GTdu3Lq+1EBjVTbjTwvRgkqBlmX48OEZNGhQ+vbtmxdffDGDBw9OknTu3DnPP/98wzcI++yzT8N6Bp07d84zzzyTqVOnpnXr1rn++utTXV2dJ554Iscee2w6dOiQVq1a5fjjj88jjzyy2vPuvPPO6dGjR1q3bp3u3bvnwAMPTFVVVXbZZZfMmTMnyQdvPD7/+c9n0003TVVVVQYPHpxnn302S5cuzYsvvtjwxmjQoEFp27ZtkuSpp55Knz59sssuuyRJhg0blscff7zhTQcANIf68bX+5+KLL15lm+XLlzeMu7169WpYe2BdxtN1cdRRR6WqqipbbrllOnfunDfeeKPouDl16tQcfvjhSZJtt902ffr0WeOx99prr3Tu3DkdO3ZM165d069fvyRZZVwfMmRIWrVqlTZt2uSoo47Ks88+m1deeSXLli3LgQcemCQ55phjGo77+OOP54gjjsiWW26ZJDnhhBOa5LUAqGcaAqzBh+dU/va3v83w4cMzceLEdO7cOTfeeGPDG4Z33303PXr0SJJ84QtfSF1dXS677LLMmTMnJ510Ur72ta9l4cKFufXWW1NTU5MkWbFixRrj/+3atWv4vaqqKptuumnD73V1dUmSt99+O5tvvnnDdptvvnneeuutLFiwIEnSvn37JElFRUU222yzJMnChQszderUDBo0qGG/9u3bN+wDAM1hTWsWfNiHx7/KysqG8W9N4+mbb76ZU089NUnSu3fvjB07dp1qqh8368+9YsWKouNmbW1tOnTo0HB//Vi7Omsa1z/8vNY0rtfW1hbU9uFtFi5cmEcffbQh1bBy5cq8//776/S8gXVgzQJgdfbdd99svfXWef7557N8+fI8/vjjueOOO9KpU6fcddddmTRpUpKkVatWGTFiREaMGJEZM2bkv/7rv/KpT30q1dXV6d+/f04++eQmqadLly4FH/IXLFiQLl26NLyJWLRoUTp06JC6urrU1tYmSaqrq3PAAQfkxhtvbJIaAKC5FRtPH3rooTXu9+EP5kkaxsa1nWtN4+Zmm22WhQsXNtx+++23s9122631mGtSbFxftGhRwXk+XN8xxxyTkSNHrvd5AYoxDQEaYcaMGZkxY0Z23HHHvPXWW9lmm20aFjj6xS9+kXfffTdJcskll+TZZ59Nkmy//fbp0qVLKioq8pnPfCb33Xdf3nvvvSTJT3/609x7773rXc/BBx+c+++/P++9916WL1+eCRMm5KCDDkqbNm2y66675tFHH02SPPDAA1m6dGmS5MADD8zUqVMbpkz88Y9/zJVXXrneNQBAc1uX8bRVq1YNH+i7du2av/zlL0k+WMPnd7/73VrPVWzc3HPPPfPYY48lSf75z3/m+eef/0jP6+CDD86ECROyYsWKLF68OPfdd18OOuigbL/99qmqqmq4BPLEiRNT8f++3ezfv38eeeSRhgbCY489lu9///sfqQ6giIpm/GkhJAtgDYYPH96wunLr1q1z2WWXpWfPnuncuXMeeOCBDBw4MNttt13OPvvsfOUrX8m1116bYcOG5ZJLLskVV1yRlStXpn///tl///2TJC+//HLDXMPtt98+V1111XrXNmjQoPz1r3/N4MGDs3Llyuy333455ZRTknywwOHo0aNz8803p1+/ftlpp52SfPANxBVXXJEzzjgj77//ftq1a5fRo0d/lJcIAJrVgAEDGj2eHnbYYTn33HNz5plnZsiQIfnqV7+aQw89NLvvvnsOO+ywtZ6r2Lh52mmn5Zxzzkn//v2z00475dBDD/1Iz2v48OGZNWtWjjzyyFRUVGTQoEE5/PDDU1FRkSuuuCKjR49O69atM3jw4IZpDL169crpp5+e4cOHp66uLp07d85ll132keoA+LCKlStXrix1EQAAANBSVXx1j2Y718pvv9Rs5yrGNAQAAADYiFx99dUZOnRohg0blj/+8Y8Fjy1dujQjR45suKrMhy1ZsiQDBgzIxIkT13oOzQIAAADYSEyZMiUzZ85MTU1NrrrqqlWmY40dOza77bbbavf93ve+V3BllWI0CwAAAKCYiorm+1mLyZMnZ8CAAUmSnXbaKbW1tQVXTjnnnHMaHv+w6dOn5+9//3sOPvjgRj1lzQIAAADYSMybNy8dO3ZsuN2pU6dzXWDfAAAgAElEQVTMnTu34Xb79u1Xu991112XUaNGNfo8zXI1hCdeX/N1b5vTYzOfKXUJSZITdz221CWkdtmCtW/UDFpVtowLcvyjdkapS0iSLF2xtNQlJEl6ddq91CUkSZ6f8/tSl5Ct2m1Z6hKSJJUVVaUuIUnysaqPlbqEJMnAbY8sdQkbve//6TulLiFJ8vL8f5a6hCTJF3c/odQl5PXFr5e6hCTJ+yuWlbqEJMkbi98sdQlJkrmL3y51CUmSg7Y9oNQlJEkenPHLUpeQT3Zdfby6uW3SQt7HtpT3CJ/tvur8+H8rLeiShv+qMdcs+NnPfpY999wz2223XaOP2zL+wgEAAIC1qq6uzrx58xpuz5kzJ127di26z5NPPplZs2blySefzOzZs9O6det069YtBxyw5kakZgEAAAAUUdGItQSaS9++fTN+/PgMGzYs06ZNS3V19RqnHtS74YYbGn4fP358ttlmm6KNgkSzAAAAADYae++9d3r16pVhw4aloqIiY8aMycSJE9OhQ4cMHDgwZ555ZmbPnp0ZM2Zk+PDhGTJkSI466qh1Po9mAQAAABTRkpIFSXL++ecX3N51110bfr/xxhuL7vu1r32tUedwNQQAAACggGQBAAAAFNHCggXNQrIAAAAAKCBZAAAAAEVUlmG0QLIAAAAAKCBZAAAAAEW0tKshNAfJAgAAAKCAZAEAAAAUIVkAAAAAlL31ThZcffXVeeGFF1JRUZHRo0end+/eTVkXALCOjM0AQFNZr2bBlClTMnPmzNTU1GT69OkZPXp0ampqmro2AKCRjM0AsOGYhtBIkydPzoABA5IkO+20U2pra7No0aImLQwAaDxjMwDQlNarWTBv3rx07Nix4XanTp0yd+7cJisKAFg3xmYA2HAqKprvp6VokgUOV65c2RSHAQCaiLEZAPgo1mvNgurq6sybN6/h9pw5c9K1a9cmKwoAWDfGZgDYcKxZ0Eh9+/bNww8/nCSZNm1aqqur0759+yYtDABoPGMzANCU1itZsPfee6dXr14ZNmxYKioqMmbMmKauCwBYB8ZmANhwyjFZsF7NgiQ5//zzm7IOAOAjMjYDAE1lvZsFAAAAUA4qUn7Jgia5GgIAAADw70OyAAAAAIooxzULJAsAAACAApIFAAAAUEQZBgskCwAAAIBCkgUAAABQRGUZRgskCwAAAIACkgUAAABQhKshAAAAAGVPswAAAAAoULFy5cqVG/okf6t9aUOfolGW1y0vdQlJklaVZn/UW1a3rNQlJEk222SzUpeQJNniY11KXUKS5I3Fs0pdQovRoYX8bby3fHGpS0iSzF78RqlLSJL07da/1CVs9KbOfbbUJSRJlqxYUuoSkiS1y94pdQnZtFXbUpeQJKmqaBnvU9q0+lipS0iSrKhbUeoSkrSc90x1K+tKXUKL+Rv9WFXrUpeQJJm9eHapS0iSfG6HIaUuYYPq/D8HNNu53rri1812rmIkCwAAAIACLaMtBwAAAC1UGa5vKFkAAAAAFJIsAAAAgCJcOhEAAAAoe5IFAAAAUIRkAQAAAFD2JAsAAACgCMkCAAAAoOxJFgAAAEARkgUAAABA2ZMsAAAAgCLKMFggWQAAAAAUkiwAAACAIqxZAAAAAJQ9zQIAAACggGkIAAAAUIRpCAAAAEDZkywAAACAIiolCwAAAIByJ1kAAAAARZRhsECyAAAAACgkWQAAAABFuBoCAAAAUPYkCwAAAKCIikgWAAAAAGVOsgAAAACKsGYBAAAAUPYkCwAAAKAIyQIAAACg7GkWAAAAQBEVFc330xhXX311hg4dmmHDhuWPf/xjwWNLly7NyJEjM3jw4IL7x44dm6FDh+bYY4/NI488stZzmIYAAAAAG4kpU6Zk5syZqampyfTp0zN69OjU1NQ0PD527Njstttuefnllxvue+655/Lyyy+npqYm8+fPzzHHHJNDDz206Hk0CwAAAGAjMXny5AwYMCBJstNOO6W2tjaLFi1K+/btkyTnnHNOFixYkPvvv79hn3333Te9e/dOkmy22WZ57733smLFilRVVa3xPM3SLHj3/UXNcZq16tB6s1KXkCTZvHXHUpeQ2mXzS11CkqTzx7qUuoQW5U/zXyh1CUmSrTbdutQlJEnatepQ6hLy6rszS11CkuTNxW+WuoQkScc2pf/vF01jxcoVpS4hSdJ+k/alLiFJskXrLUpdQuYumVvqEpIkbVu1KXUJSZK2rTYtdQlJklcXzSp1CUmS1pWtS11CkmSrdqV/j1C7bEGpS0iSTPrHo6UuIUnSffOtSl1CWWhJCxzOmzcvvXr1arjdqVOnzJ07t6FZ0L59+yxYUPj/k6qqqmy66Qf/XZ0wYUL69etXtFGQSBYAAADARmvlypWN3vaxxx7LhAkTctttt611W80CAAAAKKIlJQuqq6szb968httz5sxJ165d17rf008/nZtuuim33HJLOnRYe4LX1RAAAABgI9G3b988/PDDSZJp06alurq6YQrCmixcuDBjx47NzTffnC22aNzUO8kCAAAAKKIlJQv23nvv9OrVK8OGDUtFRUXGjBmTiRMnpkOHDhk4cGDOPPPMzJ49OzNmzMjw4cMzZMiQLF68OPPnz8/ZZ5/dcJzrrrsuW2+95nVINAsAAABgI3L++ecX3N51110bfr/xxhtXu8/QoUPX6RyaBQAAAFBECwoWNBtrFgAAAAAFJAsAAACgiJa0ZkFzkSwAAAAACkgWAAAAQBGSBQAAAEDZkywAAACAIiQLAAAAgLK33smCsWPH5vnnn8/y5ctz2mmn5dBDD23KugCAdWRsBoANowyDBevXLHjuuefy8ssvp6amJvPnz88xxxzjDQkAlJCxGQBoSuvVLNh3333Tu3fvJMlmm22W9957LytWrEhVVVWTFgcANI6xGQBoSuvVLKiqqsqmm26aJJkwYUL69evnzQgAlJCxGQA2nHJc4PAjXQ3hsccey4QJE3Lbbbc1VT0AwEdgbAYAmsJ6Nwuefvrp3HTTTbnlllvSoUOHpqwJAFgPxmYA2DAkCxpp4cKFGTt2bG6//fZsscUWTV0TALCOjM0AQFNar2bBgw8+mPnz5+fss89uuO+6667L1ltv3WSFAQCNZ2wGgA1HsqCRhg4dmqFDhzZ1LQDAejI2AwBN6SMtcAgAAAD/7sowWJDKUhcAAAAAtCySBQAAAFBEOa5ZIFkAAAAAFJAsAAAAgGIkCwAAAIByJ1kAAAAARVizAAAAACh7kgUAAABQRBkGCyQLAAAAgELNkixYmZXNcZq1WrJiSalLSJJ0rKgqdQl59/1FpS4hSTJr0T9LXUKSZIvWW5S6hCTJDu13LHUJSZK61JW6hCTJrEWvlLqEzF48u9QlJEnefG9OqUtIkvTq9IlSl0ATeb/u/VKXkCT558KWMQ7s1XXvUpeQBUtrS11CkmSz1puVuoQkyZzFb5a6hCTJjpvtVOoSkrSc99MzF75S6hIy7723Sl1CkuSFN18vdQlJkiN7fKbUJfBvyjQEAAAAKMIChwAAAEDZkywAAACAIiQLAAAAgLInWQAAAABFSBYAAAAAZU+yAAAAAIoow2CBZAEAAABQSLIAAAAAirBmAQAAAFD2JAsAAACgCMkCAAAAoOxJFgAAAEARkgUAAABA2ZMsAAAAgCIkCwAAAICyp1kAAAAAFDANAQAAAIoow1kIkgUAAABAIckCAAAAKMIChwAAAEDZkywAAACAIiQLAAAAgLInWQAAAABFSBYAAAAAZU+yAAAAAIoow2CBZAEAAABQSLIAAAAAirBmAQAAAFD2JAsAAACgGMkCAAAAoNxpFgAAAEARFRUVzfbTGFdffXWGDh2aYcOG5Y9//GPBY0uXLs3IkSMzePDgRu+zOpoFAAAAsJGYMmVKZs6cmZqamlx11VW56qqrCh4fO3Zsdtttt3XaZ3WaZc2C1959rTlOs1ZvLXm71CUkSY7psUOpS0jbVpuWuoQkSXXlJqUuIUky450ZpS4hSdJuk/alLiFJUreyrtQlJEl6dNi51CWkc5supS4hSdKzbnmpS0iSLFuxtNQl0ETmvDe31CUkSV5dNLvUJSRJdu24uNQlZKt23UpdQpLkrSVvlbqEJMmf33651CUkSbZqt3WpS0iSzF/aMt7H7tChR6lLSMePdSx1CUmSUfttX+oSkiTvLi/9f79oXpMnT86AAQOSJDvttFNqa2uzaNGitG//wWeJc845JwsWLMj999/f6H1WR7IAAAAAiqisaL6ftZk3b146dvz/m2adOnXK3Ln//5cAq2sArG2f1T7nRrwuAAAAQAu0cuXKDbKPSycCAABAEY1deLA5VFdXZ968eQ2358yZk65duzb5PpIFAAAAsJHo27dvHn744STJtGnTUl1dXXTtgfXdR7IAAAAAiqhsQcmCvffeO7169cqwYcNSUVGRMWPGZOLEienQoUMGDhyYM888M7Nnz86MGTMyfPjwDBkyJEcdddQq+6yNZgEAAABsRM4///yC27vuumvD7zfeeGOj9lkbzQIAAAAooiWtWdBcrFkAAAAAFJAsAAAAgCLK8Vv2cnzOAAAAQBGSBQAAAFBES7oaQnORLAAAAAAKSBYAAABAEa6GsI6WLFmSAQMGZOLEiU1VDwDwERibAYCm8JGSBd/73vey+eabN1UtAMBHZGwGgKZnzYJ1MH369Pz973/PwQcf3ITlAADry9gMADSV9W4WXHfddRk1alRT1gIAfATGZgCgqazXNISf/exn2XPPPbPddts1dT0AwHowNgPAhlOOCxyuV7PgySefzKxZs/Lkk09m9uzZad26dbp165YDDjigqesDABrB2AwANKX1ahbccMMNDb+PHz8+22yzjTcjAFBCxmYA2HA+0mUEN1Ll+JwBAACAIj7SpROT5Gtf+1pT1AEANBFjMwA0LZdOBAAAAMreR04WAAAAwL+zcrwagmQBAAAAUECyAAAAAIqwZgEAAABQ9iQLAAAAoIjyyxVIFgAAAAD/QrIAAAAAirBmAQAAAFD2JAsAAACgCMkCAAAAoOw1S7Kgum11c5xmrfbp2qfUJSRJ3luxuNQlZIvWnUpdQpKkbuWKUpeQJKneslupS0iSVLaQ/t077y8odQlJkpmL/lHqEvL6u6+XuoQkyXYdti91CUmSbTZtGXXw0d35pydKXUKSZMQnjy51CUmS5XXLS11CXpj7YqlLSJJ8okuvUpeQJNm2/ValLiFJ8rHKj5W6hCTJvPfmlbqEJMmkfzxW6hLSrV3nUpeQJHml9o1Sl5AkObxHv1KXwL8p0xAAAACgiArTEAAAAIByJ1kAAAAARVjgEAAAACh7kgUAAABQRPnlCiQLAAAAgH8hWQAAAABFWLMAAAAAKHuSBQAAAFCEZAEAAABQ9iQLAAAAoIgKyQIAAACg3EkWAAAAQBHWLAAAAADKnmQBAAAAFFF+uQLJAgAAAOBfaBYAAAAABUxDAAAAgCIscAgAAACUPckCAAAAKEKyAAAAACh7kgUAAABQRIVkAQAAAFDuJAsAAACgiHL8lr0cnzMAAABQhGQBAAAAFGHNAgAAAKDsSRYAAABAEZWSBQAAAEC5kywAAACAIiQLAAAAgLInWQAAAABFlOPVEDQLAAAAYCNy9dVX54UXXkhFRUVGjx6d3r17Nzz261//Otdff32qqqrSr1+/nHHGGXn33XczcuTI1NbW5v33388ZZ5yR//iP/yh6jmZpFnRv36M5TrNWm1S2LnUJSZIVK1eUuoRUVVSVuoQkyZvvvV7qEpIkby99u9QlJEl23myXUpeQJJm3ZG6pS0iSfHzz3UpdQjq36VLqEpIkrSvblLqEJMnfav9U6hKSJPtV9yt1CRu9L3xiUKlLSJJ023TLUpeQJKlsAePi4d1bxr/JuU+OLXUJSZLtNt+81CUkSdq0ahn//f3c1ZeUuoQkybTr7ix1CVm+8v1Sl5AkeX/rllHHwzOfKHUJSZJDtz2q1CVsUJVpOcmCKVOmZObMmampqcn06dMzevTo1NTUNDx+5ZVX5tZbb82WW26Zk08+OYcddliee+659OjRI+edd17efPPNnHrqqXnooYeKnseaBQAAALCRmDx5cgYMGJAk2WmnnVJbW5tFixYlSWbNmpXNN988W221VSorK3PQQQdl8uTJ6dixYxYsWJAkeeedd9KxY8e1nkezAAAAADYS8+bNK/iw36lTp8yd+0EyeO7cuenUqdMqjx155JF5/fXXM3DgwJx88skZOXLkWs+jWQAAAABFVFRUNNvPulq5cuVat7nvvvuy9dZb59FHH80Pf/jDXH755WvdR7MAAAAANhLV1dWZN29ew+05c+aka9euq33szTffTHV1dX73u9/lwAMPTJLsuuuumTNnTlasKL6WnmYBAAAAFFFZUdFsP2vTt2/fPPzww0mSadOmpbq6Ou3bt0+SbLvttlm0aFFeffXVLF++PE888UT69u2b7t2754UXXkiSvPbaa2nXrl2qqoov7uvSiQAAALCR2HvvvdOrV68MGzYsFRUVGTNmTCZOnJgOHTpk4MCBufTSS3PeeeclSY444oj06NEj1dXVGT16dE4++eQsX748l1566VrPo1kAAAAARVS0oEsnJsn5559fcHvXXXdt+H3fffctuJRikrRr1y7jxo1bp3OYhgAAAAAUkCwAAACAItbnKgUbO8kCAAAAoIBkAQAAABTRmKsU/LuRLAAAAAAKSBYAAABAERVl+D37ej/j+++/P0cffXQGDx6cJ598sglLAgDWh7EZAGgq65UsmD9/fr7zne/knnvuyeLFizN+/PgcfPDBTVwaANBYxmYA2HDKcc2C9WoWTJ48Ofvvv3/at2+f9u3b54orrmjqugCAdWBsBgCa0npNQ3j11VezZMmSnH766TnxxBMzefLkpq4LAFgHxmYA2HAqKiqa7aelWO8FDhcsWJBvf/vbef3113PKKafkiSeeaFFPDADKjbEZAGgq65Us6Ny5c/baa6+0atUq22+/fdq1a5e33367qWsDABrJ2AwANKX1ahYceOCBee6551JXV5f58+dn8eLF6dixY1PXBgA0krEZADacimb8X0uxXtMQttxyyxx22GEZMmRIkuTiiy9OZWX5XXcSAFoKYzMA0JTWe82CYcOGZdiwYU1ZCwDwERibAWDDKMdLJ/rKAQAAACiw3skCAAAAKAfleHUhyQIAAACggGQBAAAAFFFZht+zl98zBgAAAIqSLAAAAIAirFkAAAAAlD3JAgAAAChCsgAAAAAoe5IFAAAAUERlJAsAAACAMtcsyYKqiqrmOM1avV+3rNQlJEkWvv9OqUvINu26l7qEJEndyrpSl5Ak6dKma6lLSJJs3rpjqUtIknRuU13qEpIkC9+vLXUJSQvpIv/orz8udQlJklN2PbnUJdBEPtl5z1KXkCR57d1ZpS4hSTJ/6fxSl5B9uu5X6hKSJEfv0qfUJSRJ/jRveqlLSJLs3nH3UpeQJHlr/K9KXUKSZPbi10tdQlpVbFLqEpIkwydcWeoSkiS3HTey1CWUBWsWAAAAAGVPswAAAAAoYIFDAAAAKKLSNAQAAACg3EkWAAAAQBEVLWTR6+YkWQAAAAAUkCwAAACAIioryu979vJ7xgAAAEBRkgUAAABQRIWrIQAAAADlTrIAAAAAinA1BAAAAKDsSRYAAABAEZXWLAAAAADKnWQBAAAAFGHNAgAAAKDsSRYAAABAEdYsAAAAAMqeZgEAAABQwDQEAAAAKKKiovy+Zy+/ZwwAAAAUJVkAAAAARbh0IgAAAFD2JAsAAACgCJdOBAAAAMqeZAEAAAAUUSFZAAAAAJQ7yQIAAAAootLVEAAAAIByJ1kAAAAARVizAAAAACh7kgUAAABQREVF+X3PXn7PGAAAACiqWZIFlZUtI8BQsbJl9EY2qdyk1CWkdtnbpS4hSbJd+x1KXUKSZPHyd0tdQovy85k/K3UJSZJ9uu5T6hJSVVFV6hL+v/buPcbL+t4T+PsHI1UYRGbqiBTlsJzNckLXVXfphpuX41C2utvYmzOZjJrsH1u3trUJJpKJG2lREowxrWisQe02Ne1Og3jZtBHbDZyYOuhGdzGStlbOOYSLwszKpTBwFJ39w3TMz8vD5Qzz/Mbn9Wom4Tc/9HlPU+ZDv/N+Pk+S5Kymz5QdIUnyT+8eLTsCI+RX//irsiMkSa6YcXnZEZIkz+3aXHaE/PWUf1l2hCSN8b03ST4zvjG+7zWfcXbZEZIki9f+57IjJEle27az7Aj5T1d9oewISZJXNm0tO0KSZNd/2FV2hCTJv/1s2QlOr0Z7GsKqVauyZcuW1Gq19PT05KKLLhp+7/nnn8+9996b8ePH57LLLsvNN9+cJHn66afz8MMPp6mpKd/97ndzxRVXFF6jMf5fPAAAAHBcL774YrZv357e3t5s27YtPT096e3tHX7/zjvvzCOPPJLzzjsv3d3dWbp0aVpbW/PAAw/k8ccfz+DgYNasWeOwAAAAAD4t+vr60t7eniSZPXt2Dhw4kEOHDqW5uTk7duzIlClTcv755ydJLr/88vT19aW1tTXz589Pc3Nzmpubs3LlyuNepzF6+QAAANCgarXaqH0cz8DAQKZOnTr8uqWlJf39/UmS/v7+tLS0fOS9nTt35ujRo7npppvS1dWVvr6+415HswAAAADGqKGhoRP6ffv378/999+f3bt354YbbsjGjRsLDyccFgAAAECBWgMtOGxra8vAwMDw67179+bcc8/92Pf27NmTtra2nHXWWbnkkkvS1NSUCy+8MJMmTcpbb72V1tbWT7yO2xAAAABgjFi4cGE2bNiQJNm6dWva2trS3NycJJkxY0YOHTqUnTt35tixY9m4cWMWLlyYRYsWZfPmzXnvvfeyb9++DA4O1t3K8HE0CwAAAKDAiewSGC2XXnpp5s6dm87OztRqtdxxxx1Zv359Jk+enCVLlmTFihVZtmxZkuTqq6/OrFmzkiRLly7NddddlyS5/fbbM25ccXfAYQEAAACMIbfeemvd6zlz5gz/et68eXWPUvyLzs7OdHZ2nvA1HBYAAABAgXENtLNgtNhZAAAAANTRLAAAAIACtVr1fs5eva8YAAAAKKRZAAAAAAVqFdxZcEqHBYcPH85tt92WAwcO5J133snNN9+cxYsXj3Q2AOAEmc0AwEg6pcOCJ554IrNmzcqyZcuyZ8+e3HjjjXnmmWdGOhsAcILMZgA4fWq16jULTmlnwdSpU7N///4kycGDBzN16tQRDQUAnByzGQAYSafULLjmmmuyfv36LFmyJAcPHsxDDz000rkAgJNgNgPA6VPFnQWn1Cx46qmnMn369PzmN7/JT3/60/zgBz8Y6VwAwEkwmwGAkXRKhwUvv/xyFi1alCSZM2dO9u7dm3fffXdEgwEAJ85sBgBG0ikdFsycOTNbtmxJkuzatSuTJk3K+PHjRzQYAHDizGYAOH1qtdqofTSKU9pZ0NHRkZ6ennR3d+fYsWNZsWLFCMcCAE6G2QwAjKRTOiyYNGlSfvSjH410FgDgFJnNAHD6jLPgEAAAAKi6U2oWAAAAQFU00i6B0aJZAAAAANTRLAAAAIACtQr+nL16XzEAAABQSLMAAAAACthZAAAAAFSeZgEAAAAUqEWzAAAAAKg4zQIAAAAoMK6COwtG5bDgpf4XRuMyx3XGuDPKjpAkOWfCOWVHyPRJF5QdIUly8O39ZUdIkvz09z8vO0KS5L/+6/9SdoSG0renr+wI+dvPXVV2hCTJ/GlfKDtCkuTwsUNlR2CE/MOBXWVHSJI8t+PBsiMkSbrnLi07Qg6+faDsCEmSAw2So/NHd5cdIUnyv277bNkRkiTXfeHflR0hSfJ301rKjpDbvtBddoQkye//fmfZEZIkv/r735UdIUny5b/6RtkRGGGaBQAAAFDAzgIAAACg8hwWAAAAAHXchgAAAAAFahVccKhZAAAAANTRLAAAAIACtQr+nL16XzEAAABQSLMAAAAACthZAAAAAFSeZgEAAAAUGBfNAgAAAKDiNAsAAACggJ0FAAAAQOVpFgAAAECBmp0FAAAAQNVpFgAAAEABOwsAAACAytMsAAAAgAK1Cv6cvXpfMQAAAFDIYQEAAABQx20IAAAAUGCcBYcAAABA1WkWAAAAQIFaNAsAAACAitMsAAAAgAI1OwsAAACAqtMsAAAAgAJ2FgAAAACVp1kAAAAABewsAAAAACpPswAAAAAKjKvgz9mr9xUDAAAAhTQLAAAAoEAVdxaMymHBrMn/YjQuc1znfKal7AhJkqGhobIj5J/ePVJ2hCTJsaFjZUdIktxy8XfKjpAkeeX//Z+yIyRJ/nrK7LIjJEn+xx+fLDtCFp+/uOwISZK7Nj9adoQkyYqFN5UdgRHyucltZUdIkqz49/+t7AhJkv4jb5QdIWc1TSo7QpLkzPETy46QJDl8b1/ZEZI0zt9VNu1sjP8+2mf9q7Ij5G+mXlR2hCTJ2Wc3xp/Z//6rvys7QpLkob8tOwEjTbMAAAAACtRSvWaBnQUAAAAwhqxatSodHR3p7OzMK6+8Uvfe888/n69//evp6OjIAw88UPfe0aNH097envXr1x/3Gg4LAAAAYIx48cUXs3379vT29uauu+7KXXfdVff+nXfemTVr1uQXv/hFfve73+X1118ffu/BBx/MlClTTug6DgsAAACgQK1WG7WP4+nr60t7e3uSZPbs2Tlw4EAOHTqUJNmxY0emTJmS888/P+PGjcvll1+evr73d55s27Ytr7/+eq644ooT+podFgAAAMAYMTAwkKlTpw6/bmlpSX9/f5Kkv78/LS0tH/ve6tWrs3z58hO+jgWHAAAAUKCRFxyeyNP2nnzyyVx88cW54IILTvjf67AAAAAAxoi2trYMDAwMv967d2/OPffcj31vz549aWtry6ZNm7Jjx45s2oj6m4cAAAxuSURBVLQpb775ZiZMmJBp06ZlwYIFn3gdhwUAAABQoJGaBQsXLsyaNWvS2dmZrVu3pq2tLc3NzUmSGTNm5NChQ9m5c2emTZuWjRs35p577kl3d/fwP79mzZp87nOfKzwoSBwWAAAAwJhx6aWXZu7cuens7EytVssdd9yR9evXZ/LkyVmyZElWrFiRZcuWJUmuvvrqzJo165Su47AAAAAAipzAUwpG06233lr3es6cOcO/njdvXnp7ez/xn/3Od75zQtfwNAQAAACgjmYBAAAAFGiknQWjRbMAAAAAqHNChwWvvfZa2tvb89hjjyVJ3njjjVx//fXp6urKLbfckrfffvu0hgQA6pnNADB6arXaqH00iuMeFgwODmblypWZP3/+8Ofuu+++dHV15ec//3lmzpyZdevWndaQAMAHzGYA4HQ77mHBhAkTsnbt2rS1tQ1/7oUXXshVV12VJLnyyivT19d3+hICAHXMZgAYXbVR/E+jOO6Cw6ampjQ11f+2I0eOZMKECUmS1tbW9Pf3n550AMBHmM0AwOn2z34awtDQ0EjkAABGiNkMACOrkX7iP1pO6WkIEydOzNGjR5Mke/bsqatBAgCjz2wGAEbSKR0WLFiwIBs2bEiSPPvss1m8ePGIhgIATo7ZDACMpOPehvDqq69m9erV2bVrV5qamrJhw4bcc889Wb58eXp7ezN9+vRce+21o5EVAIjZDACjrZEeaThajntY8PnPfz4/+9nPPvL5n/zkJ6clEABQzGwGAE63f/aCQwAAAPg0s+AQAAAAqDzNAgAAACigWQAAAABUnmYBAAAAFKji0xA0CwAAAIA6mgUAAABQwM4CAAAAoPI0CwAAAKCAnQUAAABA5WkWAAAAQIEq7iyoDQ0NDZ3ui+w6/I+n+xInpFGqI4ffOVR2hHz2zPPKjpAkOfjOvrIjJEneGNxddoQkyYXNf1V2hCTJZ8afVXaEJMmrb/3fsiPkgkkXlh0hSTLxjOayIyRJ3ht6t+wISZLpE2eWHWHM+/M7+8uOkCR55713yo6QJGn9j5eUHSF7/+f/LjtCkmTPkTfKjpAkmTGpMf6cn1E7o+wISZJarTEKwQ+++mDZEfLSm/9QdoQkydr21WVHSJIca5Dvo1MmtJQd4bT6/f4to3atvznn34zatYpoFgAAAECBKjYLGuOIEgAAAGgYmgUAAABQoFFuaR9NmgUAAABAHYcFAAAAQB23IQAAAEABCw4BAACAytMsAAAAgAKaBQAAAEDlaRYAAABAAY9OBAAAACpPswAAAAAKaRYAAAAAFadZAAAAAAXsLAAAAAAqT7MAAAAACtTsLAAAAACqTrMAAAAACmgWAAAAAJWnWQAAAAAFPA0BAAAAqDyHBQAAAEAdtyEAAABAAQsOAQAAgMrTLAAAAIACmgUAAABA5WkWAAAAQAGPTgQAAAAqT7MAAAAACthZAAAAAFSeZgEAAAAUsLMAAAAAqDzNAgAAAChQxZ0FtaGhoaGyQwAAAECj2j24fdSuNX3izFG7VhHNAgAAAChUvWaBwwIAAAAYQ1atWpUtW7akVqulp6cnF1100fB7zz//fO69996MHz8+l112WW6++eYkyd13352XXnopx44dyze/+c188YtfLLyGwwIAAAAo0Ei9ghdffDHbt29Pb29vtm3blp6envT29g6/f+edd+aRRx7Jeeedl+7u7ixdujQDAwP505/+lN7e3uzbty9f+cpXHBYAAADAp0VfX1/a29uTJLNnz86BAwdy6NChNDc3Z8eOHZkyZUrOP//8JMnll1+evr6+dHV1DbcPzj777Bw5ciTvvvtuxo8f/4nX8ehEAAAAGCMGBgYyderU4dctLS3p7+9PkvT396elpeUj740fPz4TJ05Mkqxbty6XXXZZ4UFBolkAAAAAhWq1RroRod7JPODwt7/9bdatW5dHH330uL/XYQEAAACMEW1tbRkYGBh+vXfv3px77rkf+96ePXvS1taWJHnuuefy4x//OA8//HAmT5583OuMidsQVq1alY6OjnR2duaVV14pO07p7r777nR0dORrX/tann322bLjlO7o0aNpb2/P+vXry45Suqeffjpf/vKX89WvfjWbNm0qO06pDh8+nG9/+9u5/vrr09nZmeeee67sSKV47bXX0t7ensceeyxJ8sYbb+T6669PV1dXbrnllrz99tslJ2SsMpvrmc31zOYPmM0fMJvfZzaPVbVR/Ci2cOHCbNiwIUmydevWtLW1pbm5OUkyY8aMHDp0KDt37syxY8eycePGLFy4MH/+859z991356GHHso555xzQl9xwzcLjrfpsWo2b9580lssP+0efPDBTJkypewYpdu3b18eeOCBPP744xkcHMyaNWtyxRVXlB2rNE888URmzZqVZcuWZc+ePbnxxhvzzDPPlB1rVA0ODmblypWZP3/+8Ofuu+++dHV15Utf+lLuvfferFu3Ll1dXSWmZCwym+uZzR9lNr/PbK5nNpvNjIxLL700c+fOTWdnZ2q1Wu64446sX78+kydPzpIlS7JixYosW7YsSXL11Vdn1qxZwzPqe9/73vC/Z/Xq1Zk+ffonXqfhDwuKNj1W0bx58056i+Wn2bZt2/L6669XevD+RV9fX+bPn5/m5uY0Nzdn5cqVZUcq1dSpU/PHP/4xSXLw4MG6JTBVMWHChKxduzZr164d/twLL7yQ73//+0mSK6+8Mo8++qi/kHDSzOZ6ZnM9s/kDZnM9s9lsHssabWPBrbfeWvd6zpw5w7+eN2/eRw7xOzo60tHRcVLXaPjbEIo2PVbRqWyx/DRbvXp1li9fXnaMhrBz584cPXo0N910U7q6utLX11d2pFJdc8012b17d5YsWZLu7u7cdtttZUcadU1NTTnzzDPrPnfkyJFMmDAhSdLa2lrp76ecOrO5ntlcz2z+gNlcz2w2mxlbGr5Z8GEns+nx0+xktlh+Wj355JO5+OKLc8EFF5QdpWHs378/999/f3bv3p0bbrghGzdubOjNrafTU089lenTp+eRRx7JH/7wh/T09Lh39kN8P2Wk+N/S+8xms/njmM0fMJuPz/fTRla9P7cNf1hQtOmxqk52i+Wn1aZNm7Jjx45s2rQpb775ZiZMmJBp06ZlwYIFZUcrRWtray655JI0NTXlwgsvzKRJk/LWW2+ltbW17GilePnll7No0aIk79ey9u7dW+la8F9MnDgxR48ezZlnnlm3HRdOhtn8UWbz+8zmemZzPbP545nNNKqGvw2haNNjFZ3KFstPqx/+8Id5/PHH88tf/jLf+MY38q1vfauyfxlJkkWLFmXz5s157733sm/fvgwODlbyXsC/mDlzZrZs2ZIk2bVrVyZNmlT5v4wkyYIFC4a/pz777LNZvHhxyYkYi8zmembzB8zmemZzPbP545nNY0OtVhu1j0bR8M2Cj9v0WGW//vWvT3qLJdVw3nnnZenSpbnuuuuSJLfffnvGjWv488DTpqOjIz09Penu7s6xY8eyYsWKsiONuldffTWrV6/Orl270tTUlA0bNuSee+7J8uXL09vbm+nTp+faa68tOyZjkNlcz2zmk5jN9cxms5mxpTbkxhgAAAD4RHuP7h61a7Wd2RiHzdU92gQAAAA+VsPfhgAAAABlqlXwaQiaBQAAAEAdzQIAAAAooFkAAAAAVJ7DAgAAAKCOwwIAAACgjsMCAAAAoI4FhwAAAFCgVrPgEAAAAKg4hwUAAABAHYcFAAAAQB07CwAAAKBALXYWAAAAABWnWQAAAACFNAsAAACAitMsAAAAgALV6xVoFgAAAAAfolkAAAAABWq16nULNAsAAACAOpoFAAAAUEizAAAAAKg4hwUAAABAHbchAAAAQIHq3YSgWQAAAAB8iGYBAAAAFKpet0CzAAAAAKijWQAAAAAFajXNAgAAAKDiHBYAAAAAdRwWAAAAAHXsLAAAAIACNU9DAAAAAKpOswAAAAAKaRYAAAAAFadZAAAAAAWq1yvQLAAAAAA+RLMAAAAACtRq1esWaBYAAAAAdRwWAAAAAHXchgAAAACF3IYAAAAAVJxmAQAAABSoXq9AswAAAAD4EM0CAAAAKFS9boFmAQAAAFBHswAAAAAK1GqaBQAAAEDFOSwAAAAA6jgsAAAAAOrYWQAAAAAFap6GAAAAAFRdbWhoaKjsEAAAAEDj0CwAAAAA6jgsAAAAAOo4LAAAAADqOCwAAAAA6jgsAAAAAOo4LAAAAADq/H9X92xwIV7HXQAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "pos = 'NOUN'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=pos)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=pos)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to NOUN tags')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.2 Target verbs"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 49,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:51<00:00, 2.84it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAsAAAJ1CAYAAAC/wnk8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xu4lGWhN/7vYiEiBwmQBSqiiAGJmmiihqESCGLWFhXwgJa50TItzUOBiSXoVqutor1aabJrZyBiSB4wXzVPKEK/rcauVFQCDYFAjnJevz98WZcTMBxcMMvm8/Fa17XWzDPzfOcBvNfc833up6K6uro6AAAAAP9PvVIHAAAAAOoWkwUAAABAAZMFAAAAQAGTBQAAAEABkwUAAABAAZMFAAAAQAGTBQBARo0alU6dOuWFF14odRQAoA6oqK6uri51CAD4uLv++utz1113pWnTpnnmmWfSsGHDjW73xBNPZO3atenVq1fNbcuWLctdd92Vs88+O7vuuut2z/rqq6/mueeey5e//OWa215//fW8/vrr6datW1q0aLHdM2zMxo7NR/Hyyy/n1FNPzRFHHJHRo0cX3Xbs2LH53ve+l3POOSdXXHFFRo0alVtvvXWz+2jatGmmTp2aJHnhhRdy1llnbXS7+vXrp1WrVunWrVv+/d//PZ/85CcL7h88eHCmTJlScFtlZWWaNWuW/fffPyeffHL69eu32TxJ7R9HAMpT/VIHAICPu1WrVuX+++9PvXr1smTJkjzyyCP5t3/7t41ue+edd6Zt27YFb+ReeeWV3HrrrTnppJN2yGTBI488kt/+9rcFkwX77bdf9ttvv+2+72I2dmw+ioMOOij7779/XnjhhcyaNSt77bXXJrcdN25ckmTAgAEFt5999tk55JBDNvm4nXbaaYPbunfvvsHzLFmyJNOnT8/48ePzyCOP5O67797o844YMSJNmzZN8sHfq7fffjv3339/Lr744syYMSMXXnjhpl/w/1PbxxGA8mSyAAA+okcffTQLFy7M6aefnnvuuSdjx47d6GTBunXrMn369LRt27bg9ldeeWVHRS3J/rbEpo7NRzVgwIBcffXVGT9+fL75zW9udJvXXnstL730Urp165b27dsX3PfpT386ffv23ap9tm3bdqOPOfXUU9O3b998+ctfznXXXZd77713g22OOeaYtGrVquC2gQMHplevXrnrrrvy9a9/PZWVlZvc9/Y6jgCUH2sWAMBHNGbMmCQffAp96KGHZtq0aZkxY0bBNuPHj8+nPvWpLF++PPfff386deqUUaNGpWfPnvnhD3+YJPn85z+fTp06FTxu3LhxOeWUU/LpT386Xbt2zUknnZRf/vKXWbduXc02s2fPTqdOnTJs2LC89tprOffcc/OZz3wmBx10UM4444z86U9/Ktjuqaeeyttvv51OnTpl8ODBSTa9ZsFTTz2Vs88+O5/5zGdywAEHpGfPnhkxYkQWLFhQsF3Pnj3Tu3fvLFmyJFdeeWW6d++eAw44IMcff3wmTpxY9Pht6tisN2PGjFxyySXp3r17unTpks9+9rO56KKL8te//nWzfzYnnnhiGjVqlPvvv7/gmH3YfffdlyQZNGjQZp/vozriiCOy99575+WXX87777+/RY9p0aJF2rdvn+XLlxd9zOaO47PPPpuvfvWr6datWw444ID06tUrw4YNy7vvvrvBc73yyis566yz0rVr13Tr1i2XXnppFixYkNNPP32Dv6NTpkzJkCFD8rnPfS4HHnhgevTokYsvvjivvvrqFh4VAOoikwUA8BG8+eabmTJlSrp27Zp99tmnplHwz58aH3744Rk+fHiSpFu3brn55ptz/PHHZ/jw4enWrVuSZPjw4bn55ptrHvMf//EfGTZsWKqqqnLllVfm8ssvT6tWrTJixIh873vf2yDL3Llz85WvfCUdOnTIsGHDcsYZZ+R//ud/cv7552fVqlVp2bJlbr755rRo0SItWrTIzTffXLTWfv/992fIkCGZN29ezj///PzgBz/Isccem3vuuSenn356li9fXrD9unXrMmTIkCxevDgXX3xxvvWtb+W9997L5ZdfnunTp29yP5s6NskH6ysMGDAgzz77bE455ZSMHDkyp59+el588cUMGjQof/7znzf5vEnSpEmTnHDCCfn73/+eZ555ZoP7V69enQkTJqR58+bp3bt30eeqLQ0aNEhFRUUqKiq2aPsVK1Zk9uzZadu2bZo0abLJ7Yodxz/84Q8599xz8+677+aiiy7Ktddemz59+mTixIkZNGhQli1bVvM8s2bNytlnn51XXnklZ555Zi677LKsWbMm55xzThYtWlSwzz/+8Y/58pe/nNmzZ+fcc8/NtddeW/Pnc8YZZ+Sdd97Z2sMDQB3hNAQA+AjGjh2bJDn55JOTJMcff3xGjhyZ3/72t7nkkkvSoEGDJMmee+6ZHj161Hy/vqa+33775eGHH06S9OjRo6Y+/pe//CW/+MUvcvrpp9e8AUyS0047LRdddFHGjRuXM844I/vvv3/NfU899VRuuummmjeISbJo0aLcd999mTZtWo488sj07ds3N9xwQ5IUrdevWLEi1113XVq0aJHf/OY3NWsp9O/fP7vvvntuvPHG/OpXv8qQIUNqHjN79uz06NGjIO9uu+2WK664Io899li6dOmy0X1t6tgkyQ033JClS5fmN7/5Tbp27Vpze48ePXLqqafmRz/6UX7+859v8nUkH9T477333owbN65mP+s98cQTWbBgQc4555yaP6sPe//997N48eJNPnfDhg03+rhNeeONN/LGG2+kY8eOG10Ec+nSpdl5552TJGvXrs3MmTMzatSoLF++PNdff33R5y52HGfMmJGuXbtm5MiRBada1KtXLz/96U/z2GOP5Utf+lKSZPTo0Vm2bFmuvfbamr/Xp556aq688soNJsEeeuihrF27Ntdff30OPPDAmtuPOeaYXH/99Xn99dezxx57bPHxAaDu0CwAgG20fmHDXXbZpeYNepMmTXLcccdl4cKFeeyxx7b5uddPIPTr1y+LFy8u+OrTp0+SbLB6fps2bQomCpLUvIGbN2/eVu1/ypQpWbRoUfr167fBoov9+/dPkjz55JMbPO4rX/lKrew/SZYvX55nn302nTp1KpgoSD5YvLBjx46ZPHlyVq5cWfR5DjzwwHTp0iWPP/74BqdPrF/YcODAgRt97LBhw3LYYYdt8ut3v/vdBo9ZvXr1Bn9m77zzTiZNmpTzzjsv69at2+T6CX379q157iOOOCIDBw7M//7v/+aGG27IMcccU/R1FnPOOefk17/+ddq3b5/q6uosXbo0ixcvrln08e23367Z9oUXXkhlZeUGf5c+PDG0Xv36H3zuNG3atILbO3funF/84hcbTM4A8PGhWQAA22j9woZf/OIXC+rh/fv3z4QJEzJ27NgtvtzdP3v99deTJGeeeeYmt/nnine7du022Gb9p9Rr1qzZqv2/8cYbSZKOHTtucF+LFi3yiU98Im+99VbB7ZWVldlzzz1rZf9JMnPmzKxbt26Dywyu1759+7z66quZPXt2OnToUPS5BgwYkOHDh+eBBx6ouQrEu+++m2eeeSaHH3549tlnn40+7utf/3qOOOKITT7vvvvuu8Ft48ePz/jx4ze6/T777JPbbrstPXv23Oj9o0aNSrNmzZJ8cFrH/Pnz89xzz+Wyyy7LAw88kP/8z//c5GU5i1m9enV+9rOfZeLEiZk1a1ZWr15dcP/atWtrvp89e3ZatWqVRo0aFWzTrl27NG/ePAsXLqy57bTTTsuECRNy3XXXZcKECenRo0c++9nP5tBDD62ZSADg48n/xQFgG61f2LBbt26ZOXNmze1t2rTJbrvtlueff36zl+zblPXnkP/4xz/ObrvtttFt/nnV/K2pw2/O+vUIdtlll43e37Bhww3q+ZWVlUVX6t9a64/BP79pXW/9RMQ/r52wMV/4whdy/fXXZ9y4cTWTBffff3/Wrl1bdGHD/fbbL4cffvhW5T722GMLGhbV1dUZOnRoFi1alF/+8pepqqra5GO7du26wZ/riSeemE9/+tMZPnx47rjjjk22EooZOnRoHnjggZrn2WOPPVK/fv08//zz+clPflKw7YoVKzaZsWnTpgWTBXvvvXfuv//+3HnnnZk0aVJuv/323H777WnZsmUuvPDCnHbaaVudFYC6wWQBAGyDN954o+Y0gCuvvHKT240bNy4XX3zxVj9/48aNkyR77bVXDjrooG0L+RGsf4O+qTfi77//fk3G7WX98xfL8OHtimnSpEm+8IUvZOzYsXnppZfy6U9/OuPHj0+LFi3Sq1ev2gudpKqqaoMJhu9973s5//zzc/XVV2/w5nxLnHzyybn66qvz9NNPb/Vkwdy5czNx4sTss88++a//+q+CZsKbb765wfYNGjTIqlWrNvpcS5cu3eC2Nm3aZNiwYRk2bFj+8pe/5IknnsivfvWrXH311WnUqFHNWggAfLyYLACAbbB+YcNTTz01Rx111Ab3r1y5Mt/97ndz33335cILL9zqSvZ+++2Xxx57LH/84x83mCxYtmxZ6tevX/PJ+vaw3377JclGL383b968LFq0aKs/cd9a++yzTyorKzd5Cb7XX389DRo0qFkUcnMGDhyYsWPH5ne/+12qq6szc+bMfPWrX63VRsamHHvssenTp08mTZqUBx98MCeccMJWPX7t2rWprq7e7PoMG/POO++kuro6hxxyyAanMLz44osbbN+6deu88847WbVqVcGxmTVr1gZrPvyzzp07p3Pnzjn22GPzpS99KY8++qjJAoCPKQscAsBWWr+wYYMGDXLJJZekb9++G3x96UtfSq9evTJv3ryahQDXV/T/+Q1fvXr1Nrh9/eJy99xzT1asWFGw/Y033pgjjjgif/vb37Ypf7169Tb7prNbt25p0aJFHnrooQ0ul7f+9Iv1Cy3Who0dm1122SVHH310Xn311UydOrVg+ylTpuTNN9/Mscceu8Vv9g844IB06dIlDz/8cB5++OFUVFRkwIABtfYaNmfYsGFp0qRJrrnmms2+6f5n6yenDj300KLbbew4tmzZMskHaxF82OTJk2suJ/nhv2Ndu3bN6tWrN1ig8/bbb99gf0OGDMlXvvKVrFu3ruD29Wt47IiJGAC2D80CANhKkyZNynvvvZf+/funRYsWm9zuzDPPzKRJk3LvvfemV69e2W233dKwYcM8/fTTueOOO7L33nunb9++NZ+M33jjjTnssMPypS99KZ07d87ZZ5+d0aNH57TTTsvAgQNTv379/OEPf8ijjz6aL37xixtd0HBLtG3bNpMnT851112X3XffveYc/g9r0KBBvve97+WSSy7JGWeckVNOOSVNmzbNSy+9lHvvvTcHH3xwTj311G3a/8Zs6thcfvnlmTp1ai644IIMHjw4bdu2zVtvvZVf//rXad68eS677LKt2s/AgQNz1VVX5Z577im6sOF6L7300mbXYejatWtat2692X23bt06F198ca655ppcc801+c///M8NtnnyySfTtGnTmp8XLVqUadOm5cEHH8xee+2Vb3zjG0X3sbHj2KdPnxx00EGZMmVKRowYkQMPPDB//vOfM2HChFx//fU577zz8uijj6Zjx445/vjjc9ZZZ+XBBx/M8OHD88Ybb6RNmzZ55plnMn/+/Oyzzz4FC1t269YtN954Y84666wcf/zxadasWebPn5+xY8emfv36RdeDAKBuM1kAAFtp/SfrZ599dtHtunXrlo4dO+bpp5/OnDlz0qZNm3znO9/JTTfdlJ/85CcZMGBA+vbtm0GDBuWZZ57JM888kz/96U857rjjknywKN0nP/nJjBkzJtddd13WrVuXffbZJ5dddtlG3+BvqW9+85t5++2389///d/p1KnTJp+rX79+adasWe64447ccsstWbVqVfbcc88MGTIk5513Xq1+arzTTjtt9Ni0b98+Y8eOzS233JJf//rXWbRoUZo3b56ePXvmggsu2OrFI9cvdLhs2bJNXi7xw0aPHp3Ro0cX3ea2227bosmCJDn99NMzYcKEPPTQQznhhBM2WC/hn9e/+MQnPpE999wz5557bs4555yaKyVsyqaO40033ZRrrrkmDzzwQCZOnJiuXbtm9OjR6dixYwYOHJgHHnggP/7xj9O7d+906dIlt912W2666abccccdadasWT7/+c9nxIgROfnkk2uaMEly7rnnpqqqKmPGjMktt9ySpUuXZtddd83BBx+cESNG5JBDDtmi4wJA3VNRXV1dXeoQAADUfYcffngqKyvz3HPPlToKANuZNQsAAKgxefLkDBkyJA899FDB7X/4wx/y3nvvbXbdBAD+NTgNAQCAGvvuu29eeeWVvPjii3n11Vez7777ZubMmbn77ruz884752tf+1qpIwKwAzgNAQCAAm+99VZuvfXWTJkyJQsWLEjjxo1zyCGH5IILLsgBBxxQ6ngA7AAmCwAAAIAC1iwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAKmCwAAAAACpgsAAAAAAqYLAAAAAAK1C91AAAAAKjLKnq33WH7qv797B22r2I0CwAAAIACJgsAAACAAk5DAAAAgGIqKkqdYIfTLAAAAAAKaBYAAABAMWX4MXsZvmQAAACgGM0CAAAAKMaaBQAAAEC50ywAAACAYsqvWKBZAAAAABTSLAAAAIBirFkAAAAAlDvNAgAAACimDD9mL8OXDAAAABRjsgAAAAAo4DQEAAAAKMYChwAAAEC50ywAAACAYsqvWKBZAAAAABTSLAAAAIBi6pVftUCzAAAAACigWQAAAADFlF+xQLMAAAAAKKRZAAAAAMVUlF+1QLMAAAAAKKBZAAAAAMWUX7FAswAAAAAopFkAAAAAxdQrv2qBZgEAAABQQLMAAAAAiim/YoFmAQAAAFDIZAEAAABQwGkIAAAAUExF+Z2HoFkAAAAAFNAsAAAAgGJcOhEAAAAod5oFAAAAUEz5FQs0CwAAAIBCmgUAAABQjKshAAAAAOVOswAAAACKKb9igWYBAAAAUEizAAAAAIqpV37VAs0CAAAAoIBmAQAAABRTfsUCzQIAAACgkGYBAAAAFFNRftUCzQIAAACggMkCAAAAoIDTEAAAAKCYMvyYvQxfMgAAAFCMZgEAAAAUY4FDAAAAoNxpFgAAAEAx5Vcs0CyAjenUqVN69+6dvn37pk+fPjn33HMza9asUsfaJr17984LL7xQdJvvfOc7+clPfrKDEgFQrj48vq7/+upXv5okOfvsszN9+vRa29fTTz+dd955p9aer5hhw4Zl1KhRO2Rf+++/f2bPnl10m8GDB2fChAk7JA/wr0uzADbhl7/8Zdq0aZMk+dGPfpSRI0fm9ttvL3EqAPh4+/D4+mGjR4+u1f3cfffd+drXvpY99tijVp8XKFNluGaByQLYAkcccUQef/zxmp/vvffe3HXXXVm7dm1atWqVG264IXvuuWfefffdXH755Zk3b15WrVqVE044IRdffHGqq6tz2223ZeLEiVm1alU+//nP57vf/W4qKysL9jN+/Pg8+eST2WmnnTJt2rS0b98+F1xwQX74wx/mb3/7W775zW9m4MCBWbduXW6++eZMmjQpSXLwwQfnqquuSqNGjfKnP/0pV1xxRdasWZOjjz664Pkfe+yx3HzzzVm+fHn23nvv/PCHP0yLFi22/wEEgM3o2bNnbrjhhrRp0yaDBg3KkCFDcu+99+a9997Ld7/73fTr12+Lx9Obbropzz//fN54441cdtlleeqpp9KuXbt8/etfT/JBo279zz179syQIUMybty4zJkzJ1/4whfyne98J8mmx82FCxfm29/+dt56663st99+adiw4UYnQEaNGpX58+dnzpw5mT59eo488sj069cvo0aNyty5c3PNNdfk2GOPzcqVKzNy5Mi88MILqVevXo4++uhcdtllqayszB/+8IeMGDEi9evXz8knn1zw/GPGjMkvfvGLrFq1KgcffHCuvfbaNGzYcDv9CQHlxmkIsBmrVq3KAw88kJ49eyZJ/vGPf+QHP/hBfvGLX+TRRx9Nu3btair8d999dw477LA89NBDmThxYmbNmpW5c+dmwoQJeeSRRzJu3Lj8/ve/z6xZs3LPPfdsdH9PP/10vvGNb+TRRx/NjBkz8vOf/zz//d//nZEjR9bs5+GHH85TTz2V8ePH58EHH8zixYtz9913J0muvvrqnHXWWZk0aVK6du1aU1WcNWtWLr/88vzoRz/K//2//zeHH354rr766u178ABgGyxcuDD16tXLxIkTM3To0Nx0001JssXj6be+9a20bt06N954Y/r167fZ/b344osZM2ZM7rvvvvzqV7/KnDlzio6bP/vZz9K8efM8/vjjueqqq/LMM89s8rmffPLJXHvttZk4cWIeeeSRmvH7/PPPz89+9rMkH7Qq5syZkwcffDD3339/pk6dmt/97ndZu3Zthg0bluHDh+fhhx9OvXr1snbt2iTJ1KlTc/PNN2f06NF5/PHH06RJk9x8881be6iBLVVvB37VEXUoCtQtgwcPTt++fdO9e/e88sor6d+/f5KkZcuWmTZtWs0nCJ/5zGdq1jNo2bJlnnnmmUydOjUNGjTIj3/841RVVeWJJ57IySefnKZNm6Z+/fo59dRT8+ijj250v/vtt1/at2+fBg0aZO+9985RRx2VysrKdOzYMXPnzk3ywS8e//Zv/5ZGjRqlsrIy/fv3z7PPPpuVK1fmlVdeqfnFqG/fvtlll12SJE899VS6deuWjh07JkkGDRqUxx9/vOaXDgDYEdaPr+u/rrzyyg22WbNmTc2426VLl5q1B7ZmPN0aJ554YiorK9O6deu0bNkyf//734uOm1OnTs3xxx+fJGnbtm26deu2yefu2rVrWrZsmebNm6dVq1bp0aNHkmwwrg8YMCD169dPw4YNc+KJJ+bZZ5/NW2+9lVWrVuWoo45Kkpx00kk1z/v444+nX79+ad26dZLktNNOq5VjAbCe0xBgEz58TuWLL76YwYMHZ/z48WnZsmVuueWWml8Yli1blvbt2ydJvvzlL2fdunX5/ve/n7lz5+aMM87IhRdemCVLluTOO+/MmDFjkiRr167dZP2/cePGNd9XVlamUaNGNd+vW7cuSbJgwYI0a9asZrtmzZrlH//4R957770kSZMmTZIkFRUV2XXXXZMkS5YsydSpU9O3b9+axzVp0qTmMQCwI2xqzYIP+/D4V69evZrxb1Pj6bvvvpuzzz47SXLQQQflhhtu2KpM68fN9fteu3Zt0XFz0aJFadq0ac3t68fajdnUuP7h17WpcX3RokUF2T68zZIlS/L73/++ptVQXV2d1atXb9XrBraCNQuAjTnssMOyxx57ZNq0aVmzZk0ef/zx/OpXv0qLFi0yduzYTJw4MUlSv379DBkyJEOGDMmbb76Zf//3f8+hhx6aqqqq9OzZM2eeeWat5Nltt90K3uS/99572W233Wp+iVi6dGmaNm2adevWZdGiRUmSqqqqfPazn80tt9xSKxkAYEcrNp4+8sgjm3zch9+YJ6kZGze3r02Nm7vuumuWLFlS8/OCBQuy1157bfY5N6XYuL506dKC/Xw430knnZQrrrhim/cLUIzTEGALvPnmm3nzzTez77775h//+Ef23HPPmgWOHn744SxbtixJctVVV+XZZ59NkrRr1y677bZbKioq8vnPfz4TJkzI+++/nyT5zW9+k/vvv3+b8xxzzDF54IEH8v7772fNmjUZN25cjj766DRs2DCdO3fO73//+yTJgw8+mJUrVyZJjjrqqEydOrXmlImXX345I0aM2OYMALCjbc14Wr9+/Zo39K1atcpf/vKXJB+s4fPHP/5xs/sqNm4efPDBeeyxx5Ikf/vb3zJt2rSP9LqOOeaYjBs3LmvXrs3y5cszYcKEHH300WnXrl0qKytrLoE8fvz4VPy/Tzd79uyZRx99tGYC4bHHHstPf/rTj5QDKKJiB37VEZoFsAmDBw+uWV25QYMG+f73v59OnTqlZcuWefDBB9O7d+/stdde+da3vpWvfe1r+Y//+I8MGjQoV111Va655ppUV1enZ8+eOfLII5Mkr732Ws25hu3atcvIkSO3OVvfvn3z17/+Nf379091dXUOP/zwnHXWWUk+WOBw6NChueOOO9KjR4906NAhyQefQFxzzTW54IILsnr16jRu3DhDhw79KIcIAHaoXr16bfF42qdPn1xyySW56KKLMmDAgHzjG9/Icccdl/333z99+vTZ7L6KjZvnnXdeLr744vTs2TMdOnTIcccd95Fe1+DBgzNr1qyccMIJqaioSN++fXP88cenoqIi11xzTYYOHZoGDRqkf//+NacxdOnSJeeff34GDx6cdevWpWXLlvn+97//kXIAfFhFdXV1dalDAAAAQF1V8Y0Ddti+qm/90w7bVzFOQwAAAAAKmCwAAAAAClizAAAAAIopw0snahYAAAAABXZIs+Ce10fviN1s1v+8++dSR0iSnNLxC6WOkCdmP1PqCEmS3Ru3KnWEJMny1e+XOkKSpEmDxqWOkCRpWNmw1BGSJCvWrih1hOzRePdSR0iS/G3JrFJHSJK0blRV6ghJkr57fanUET72/vD3R0sdIUmvs2MuAAAgAElEQVTywpzNX8JuR+jf4YuljpDpC+rGglbr6sja14tXLS51hCRJ850/UeoISZL368CYmCTLVi8vdYQcVnVoqSMkSd5Y/EapIyRJdq8jv6t0a/W5UkfYvsqvWKBZAAAAABSyZgEAAAAUUWHNAgAAAKDcaRYAAABAEZoFAAAAQJ127bXXZuDAgRk0aFBefvnlgvtWrlyZK664Iv3799/ix2yMyQIAAAAooqJix31tzpQpUzJz5syMGTMmI0eOzMiRIwvuv+GGG/KpT31qqx6zMSYLAAAA4GNi8uTJ6dWrV5KkQ4cOWbRoUZYuXVpz/8UXX1xz/5Y+ZmNMFgAAAEAR9SoqdtjX5syfPz/Nmzev+blFixaZN29ezc9NmjTZ6sds9DVvyYEBAAAA6p7q6urt8hhXQwAAAIAi6tLVEKqqqjJ//vyan+fOnZtWrVrV+mM0CwAAAOBjonv37pk0aVKSZPr06amqqtroqQcf9TGaBQAAAFBEXWoWHHLIIenSpUsGDRqUioqKDB8+POPHj0/Tpk3Tu3fvXHTRRZkzZ07efPPNDB48OAMGDMiJJ564wWM2x2QBAAAAfIxceumlBT937ty55vtbbrllix6zOds8WXDttdfmpZdeSkVFRYYOHZqDDjpoW58KAKgFxmYAoLZs02TBlClTMnPmzIwZMyYzZszI0KFDM2bMmNrOBgBsIWMzAGw/dek0hB1lmxY4nDx5cnr16pUk6dChQxYtWpSlS5fWajAAYMsZmwGA2rRNkwXz589P8+bNa35u0aJF5s2bV2uhAICtY2wGgO2nomLHfdUVtXLpxOrq6tp4GgCglhibAYCPYpvWLKiqqsr8+fNrfp47d25atWpVa6EAgK1jbAaA7ceaBVuoe/fumTRpUpJk+vTpqaqqSpMmTWo1GACw5YzNAEBt2qZmwSGHHJIuXbpk0KBBqaioyPDhw2s7FwCwFYzNALD9lGOzYJsmC5Lk0ksvrc0cAMBHZGwGAGrLNk8WAAAAQDmoSPk1C2rlaggAAADAvw7NAgAAACiiHNcs0CwAAAAACmgWAAAAQBFlWCzQLAAAAAAKaRYAAABAEfXKsFqgWQAAAAAU0CwAAACAIlwNAQAAACh7JgsAAACAAjvkNIRuVd12xG42a/dGbUodIUmycOXCUkfIF9v3K3WEJMmc5X8vdYQkyT5N25c6QpKkTaO2pY6QJHl72cxSR0iSrF63utQRskv9XUodIUnSqH6jUkdIkry5+K1SR6CW7NW4XakjJEka7tmw1BGSJEtXLyl1hByy26GljpAkmb1sVqkjJEm67ta11BGSJEtWLy51hCRJRUXd+IyvunpdqSNklzoyJu7ReI9SR0iS/G1J3fg3261VqRNsX05DAAAAAMqeBQ4BAACgiDIsFmgWAAAAAIU0CwAAAKAIaxYAAAAAZU+zAAAAAIrQLAAAAADKnmYBAAAAFKFZAAAAAJQ9zQIAAAAoQrMAAAAAKHuaBQAAAFBEGRYLNAsAAACAQpoFAAAAUIQ1CwAAAICyZ7IAAAAAKOA0BAAAACjCaQgAAABA2dMsAAAAgCLqaRYAAAAA5U6zAAAAAIoow2KBZgEAAABQSLMAAAAAinA1BAAAAKDsaRYAAABAERXRLAAAAADKnGYBAAAAFGHNAgAAAKDsaRYAAABAEZoFAAAAQNnTLAAAAIAiyrBYoFkAAAAAFDJZAAAAABTYIachvL7otR2xm83auXLnUkdIknRrdWSpI2TO+++UOkKSZM/GbUsdIUmyYu2KUkdIkkya9btSR0iS7Nl4z1JHSJK0b7pfqSNk6rwXSh0hSdKiYYtSR0iS7LPr3qWOQC35+/K6MQ6sWreq1BGSJB127VjqCJm9bGapIyRJWjRsWeoISZLla5aXOkKS5Ol3nit1hCRJl5adSx0hSdKxWelzvLH49VJHSJIsXb201BGSJA0qdyp1hLJggUMAAACg7FngEAAAAIrQLAAAAADKnmYBAAAAFKFZAAAAAJQ9zQIAAAAoogyLBZoFAAAAQCHNAgAAACjCmgUAAABA2dMsAAAAgCI0CwAAAICyp1kAAAAARWgWAAAAAGVvm5sFN9xwQ6ZNm5Y1a9bkvPPOy3HHHVebuQCArWRsBoDtowyLBds2WfD888/ntddey5gxY7Jw4cKcdNJJfiEBgBIyNgMAtWmbJgsOO+ywHHTQQUmSXXfdNe+//37Wrl2bysrKWg0HAGwZYzMAUJu2abKgsrIyjRo1SpKMGzcuPXr08MsIAJSQsRkAtp9yXODwI10N4bHHHsu4ceNy11131VYeAOAjMDYDALVhmycLnn766dx+++35+c9/nqZNm9ZmJgBgGxibAWD70CzYQkuWLMkNN9yQu+++O5/4xCdqOxMAsJWMzQBAbdqmyYKHHnooCxcuzLe+9a2a266//vrssccetRYMANhyxmYA2H40C7bQwIEDM3DgwNrOAgBsI2MzAFCbPtIChwAAAPCvrgyLBalX6gAAAABA3aJZAAAAAEWU45oFmgUAAABAAc0CAAAAKEazAAAAACh3mgUAAABQhDULAAAAgLKnWQAAAABFlGGxQLMAAAAAKLRDmgUtGrbYEbvZrHXV60odIUlSr6Ky1BGyZt3qUkdIksx+f3apIyRJlq9eVuoISZLDq44sdYQkyYKV/yh1hCTJhLd+W+oI2b1R61JHSJLMWlo3/q0sWbWk1BGSJEdUHV3qCB97zXb+RKkjJEmWrq4bf6ca129a6ghZtGpRqSMkSVo1rBv/33tmztOljpAkGfjJQaWOkCRZXkf+rby5ZEapI+QfK+rG7ylT332l1BGSJJ/d49BSR+BflNMQAAAAoAgLHAIAAABlT7MAAAAAitAsAAAAAMqeZgEAAAAUoVkAAAAAlD3NAgAAACiiDIsFmgUAAABAIc0CAAAAKMKaBQAAAEDZ0ywAAACAIjQLAAAAgLKnWQAAAABF1LVmwbXXXpuXXnopFRUVGTp0aA466KCa+5577rn8+Mc/TmVlZXr06JELLrggy5YtyxVXXJFFixZl9erVueCCC/K5z32u6D5MFgAAAMDHxJQpUzJz5syMGTMmM2bMyNChQzNmzJia+0eMGJE777wzrVu3zplnnpk+ffrk+eefT/v27fPtb3877777bs4+++w88sgjRffjNAQAAAAooqKiYod9bc7kyZPTq1evJEmHDh2yaNGiLF26NEkya9asNGvWLLvvvnvq1auXo48+OpMnT07z5s3z3nvvJUkWL16c5s2bb3Y/JgsAAADgY2L+/PkFb/ZbtGiRefPmJUnmzZuXFi1abHDfCSeckHfeeSe9e/fOmWeemSuuuGKz+zFZAAAAAB9T1dXVm91mwoQJ2WOPPfL73/8+o0ePzg9+8IPNPsaaBQAAAFBEXVrfsKqqKvPnz6/5ee7cuWnVqtVG73v33XdTVVWVP/7xjznqqKOSJJ07d87cuXOzdu3aVFZWbnI/mgUAAADwMdG9e/dMmjQpSTJ9+vRUVVWlSZMmSZK2bdtm6dKlmT17dtasWZMnnngi3bt3z957752XXnopSfL222+ncePGRScKEs0CAAAAKKouXTrxkEMOSZcuXTJo0KBUVFRk+PDhGT9+fJo2bZrevXvn6quvzre//e0kSb9+/dK+fftUVVVl6NChOfPMM7NmzZpcffXVm92PyQIAAAD4GLn00ksLfu7cuXPN94cddljBpRSTpHHjxrn55pu3ah8mCwAAAKCIutQs2FGsWQAAAAAU0CwAAACAIjQLAAAAgLKnWQAAAABFlGGxQLMAAAAAKKRZAAAAAEVYswAAAAAoe5oFAAAAUIxmAQAAAFDuNAsAAACgCGsWAAAAAGVvhzQL5r4/b0fsZrMWrFhQ6ghJkg67dix1hFSnutQR6pSG9RuWOkKS5MV5L5Q6QpLk4JZdSx0hSdKvXb9SR8j8lXXj/1+LVy0qdYQkyf7N9y91BGrJXxb+pdQRkiSdm3cudYQkyYq175c6Qto02r3UEZIkk2ZNKnWEJMkndm5W6ghJkneW/a3UEZIkFakbn2o23alpqSNkn6b7ljpCkmT3xnXj32zLnXcrdQT+RTkNAQAAAIqoVzfm63YopyEAAAAABTQLAAAAoAgLHAIAAABlT7MAAAAAiqinWQAAAACUO80CAAAAKMKaBQAAAEDZ0ywAAACAIsrxU/ZyfM0AAABAEZoFAAAAUISrIQAAAABlT7MAAAAAinA1hK20YsWK9OrVK+PHj6+tPADAR2BsBgBqw0dqFvyf//N/0qxZs9rKAgB8RMZmAKh91izYCjNmzMjrr7+eY445phbjAADbytgMANSWbZ4suP766/Od73ynNrMAAB+BsRkAqC3bdBrCb3/72xx88MHZa6+9ajsPALANjM0AsP2U4wKH2zRZ8OSTT2bWrFl58sknM2fOnDRo0CBt2rTJZz/72drOBwBsAWMzAFCbtmmy4Kabbqr5ftSoUdlzzz39MgIAJWRsBoDt5yNdRvBjqhxfMwAAAFDER7p0YpJceOGFtZEDAKglxmYAqF0unQgAAACUvY/cLAAAAIB/ZeV4NQTNAgAAAKCAZgEAAAAUYc0CAAAAoOxpFgAAAEAR5dcr0CwAAAAA/olmAQAAABRhzQIAAACg7GkWAAAAQBGaBQAAAEDZ2yHNgqpdWu2I3WzW4VVHljpCkuRvS98sdYQsXb2s1BGSJG0atSl1hCTJzvV2LnWEJMmKte+XOkKS5K+L/lrqCEmS5jt/otQR8tp7r5c6QpKk8U6NSh0hSbJbw7rx/3P+dbRqWFXqCEmShSvnlzpC/rrwL6WOkCTp3LxjqSMkSSorKksdIUmyat2qUkdIkvz05bGljpAkOWHf7qWOkOkL/rfUEZIkjervUuoISZI5y+eUOkKSZN9dO5c6ArXMaQgAAABQRIXTEAAAAIByp1kAAAAARVjgEAAAACh7mgUAAABQRPn1CjQLAAAAgH+iWQAAAABFWLMAAAAAKHuaBQAAAFCEZgEAAABQ9jQLAAAAoIgKzQIAAACg3GkWAAAAQBHWLAAAAADKnmYBAAAAFFF+vQLNAgAAAOCfmCwAAAAACjgNAQAAAIqwwCEAAABQ9jQLAAAAoAjNAgAAAKDsaRYAAABAERWaBQAAAEC50ywAAACAIsrxU/ZyfM0AAABAEZoFAAAAUIQ1CwAAAICyp1kAAAAARdTTLAAAAADKnWYBAAAAFKFZAAAAAJQ9zQIAAAAowtUQAAAAgLK3Q5oFFakbszDL1iwpdYQkydLVy0odIe2atCt1hCTJynUrSx0hSTJt3rRSR0iStG5UVeoISZIDmh9Y6ghJknoVlaWOkFv/v3tKHSFJ8vWDB5Y6QpJkzvI5pY6QJOnY7IBSR/jYa1Bvp1JHSJKsXreq1BGSJMvXlH5sPqJ191JHSJLMff/vpY6QJBn5ws9LHSFJcs6BXyh1hCTJhV3PKnWEJEmjykaljpA/LXil1BGSJO13bV/qCEmSlWvrxu/T/+rq1ZH3tDuSZgEAAABQwGQBAAAAUMAChwAAAFCEBQ4BAACAsqdZAAAAAEXU0ywAAAAAyp1mAQAAABRR4dKJAAAAQLnTLAAAAIAiXA0BAAAAKHuaBQAAAFCEqyEAAAAAZU+zAAAAAIqoKMPP2bf5FT/wwAP54he/mP79++fJJ5+sxUgAwLYwNgMAtWWbmgULFy7Mbbfdlvvuuy/Lly/PqFGjcswxx9RyNABgSxmbAWD7Kcc1C7ZpsmDy5Mk58sgj06RJkzRp0iTXXHNNbecCALaCsRkAqE3bdBrC7Nmzs2LFipx//vk5/fTTM3ny5NrOBQBsBWMzAGw/FRUVO+yrrtjmBQ7fe++93HrrrXnnnXdy1lln5YknnqhTLwwAyo2xGQCoLdvULGjZsmW6du2a+vXrp127dmncuHEWLFhQ29kAgC1kbAYAatM2TRYcddRRef7557Nu3bosXLgwy5cvT/PmzWs7GwCwhYzNALD9VOzA/7bEtddem4EDB2bQoEF5+eWXC+577rnncsopp2TgwIG57bbbam7f2qsmbdNpCK1bt06fPn0yYMCAJMmVV16ZevXK77qTAFBXGJsBoDxMmTIlM2fOzJgxYzJjxowMHTo0Y8aMqbl/xIgRufPOO9O6deuceeaZ6dOnT1q2bLnVV03a5jULBg0alEGDBm3rwwGAWmZsBoDtoy5dOnHy5Mnp1atXkqRDhw5ZtGhRli5dmiZNmmTWrFlp1qxZdt999yTJ0UcfncmTJ6dly5ZbfdUkHzkAAADAx8T8+fMLTjVs0aJF5s2blySZN29eWrRoscF923LVpG1uFgAAAEA5qMtXF6qurt6i7bb2qkmaBQAAAPAxUVVVlfnz59f8PHfu3LRq1Wqj97377rupqqrapqsmmSwAAACAIurtwP82p3v37pk0aVKSZPr06amqqkqTJk2SJG3bts3SpUsze/bsrFmzJk888US6d+++TVdNchoCAAAAfEwccsgh6dKlSwYNGpSKiooMHz4848ePT9OmTdO7d+9cffXV+fa3v50k6devX9q3b58kW33VJJMFAAAAUERdW7Pg0ksvLfi5c+fONd8fdthhBZdSXG9rr5rkNAQAAACggGYBAAAAFFHXmgU7gmYBAAAAUECzAAAAAIqoF80CAAAAoMztkGbBnxZM3xG72az9mq0odYQkSdUuVaWOkEb1m5Q6QpLkN/87rtQRkiRVjVuWOkKSZN9dO5Q6QpJkl/qNSx0hSfL2sr+VOkKOatul1BGSJO8se6fUEZIkx+zRq9QRqCWdmnfe/EY7wJ3T/6vUEZIkJ7TvXeoI2bmyYakjJEkWr15c6ghJks4tW5c6QpLkU80/VeoISZKd6jUodYQkycq1pf99umXDuvF727z355U6QpLkiNafK3WEsmDNAgAAAKDsmSwAAAAACljgEAAAAIqo5zQEAAAAoNxpFgAAAEARFS6dCAAAAJQ7zQIAAAAool5F+X3OXn6vGAAAAChKswAAAACKqHA1BAAAAKDcaRYAAABAEa6GAAAAAJQ9zQIAAAAoop41CwAAAIByp1kAAAAARVizAAAAACh7mgUAAABQhDULAAAAgLJnsgAAAAAo4DQEAAAAKKKiovw+Zy+/VwwAAAAUpVkAAAAARbh0IgAAAFD2NAsAAACgCJdOBAAAAMqeZgEAAAAUUaFZAAAAAJQ7zQIAAAAoop6rIQAAAADlTrMAAAAAirBmAQAAAFD2NAsAAACgiIqK8vucvfxeMQAAAFBURXV1dfX23sn8FXO29y62SP16O5U6QpJk/op3Sx0hu1Q2KnWEJMm66rWljpAkWb52eakjJEla7tyq1BGSJDOXvlHqCEmSlWtXljpCdmtYN/5Mlq9ZVuoISZJZS2eVOkKS5MS9Tyl1hI+9N5f8tdQRkiStGrYpdYQkyd+Xzy51hOzVpH2pIyRJVq5dUeoISerGn0mStGuyb6kjJEleXjCt1BGSJKOn/7bUEXJU24NKHSFJsnjV0lJHSJI0a9C01BGSJKd/8suljrBd/fLVO3fYvgZ3/OoO21cxmgUAAABAAZMFAAAAQAELHAIAAEARLp0IAAAAlD3NAgAAACiiIpoFAAAAQJnTLAAAAIAirFkAAAAAlD3NAgAAACiinjULAAAAgHKnWQAAAABFVFSU3+fs5feKAQAAgKI0CwAAAKCIijJcs2CbJguWLVuWK664IosWLcrq1atzwQUX5P9v7/5jrKzvPYG/DzOdcoERBy4jTrUsl9xc9trbWDdul5/qddBNTXrtzyETaHfTzW3TH7Ebmy0hTWCLkAwh3So1woo2XQ3NNFPUJtsUagMJqaO0ei9Emq5C9rKACjMtP0QYXejsH81ijj8ekA7znOnzepFJmDkDz3uMnM/kM+/nexYsWDDS2QCAi2Q2AwAj6ZKWBY899lhmzpyZu+++O0eOHMnnP//5/OxnPxvpbADARTKbAeDyqdWq1yy4pDML2tracvz48STJyZMn09bWNqKhAID3xmwGAEbSJTUL7rjjjmzZsiWLFi3KyZMns3HjxpHOBQC8B2YzAFw+VTyz4JKaBU888UQ6Ojry85//PD/4wQ/y7W9/e6RzAQDvgdkMAIykS1oWPPfcc5k/f36SZPbs2Tl69GjOnTs3osEAgItnNgMAI+mSlgUzZszI7t27kySHDx/OxIkT09TUNKLBAICLZzYDwOVTq9VG7a1RXNKZBV1dXVm+fHmWLFmSs2fPZuXKlSMcCwB4L8xmAGAkXdKyYOLEibn33ntHOgsAcInMZgC4fMY54BAAAACouktqFgAAAEBVNNJZAqNFswAAAACoo1kAAAAABWoV/Dl79b5iAAAAoJBmAQAAABRwZgEAAABQeZoFAAAAUKAWzQIAAACg4jQLAAAAoMC4Cp5ZMCrLgl8PPDMal7mgv227ruwISZJXTr9cdoS8v+n9ZUdIkvxV61+XHSFJ8s+/+6eyIyRJ/nVbU9kRkiQHXv0/ZUdIkjz5L+U/d/zXOf+l7AhJktb3XVF2hCTJX45vLzsCI2TP73aXHSFJ8jdXvl52hCTJ/pP7y46Q/Sf3lR0hSdL6vtayIyRJvv3U/yg7QpLkngX/qewISZLnjjbGv9mhs2fLjpC//8CtZUdIkmza+4OyIyRJ/qK5Mb6v58+PZgEAAAAUcGYBAAAAUHmWBQAAAEAdtyEAAABAgVoFDzjULAAAAADqaBYAAABAgVoFf85eva8YAAAAKKRZAAAAAAWcWQAAAABUnmYBAAAAFBgXzQIAAACg4jQLAAAAoIAzCwAAAIDK0ywAAACAAjVnFgAAAABVp1kAAAAABZxZAAAAAFSeZgEAAAAUqFXw5+zV+4oBAACAQpYFAAAAQB23IQAAAECBcQ44BAAAAKpOswAAAAAK1KJZAAAAAFScZgEAAAAUqDmzAAAAAKg6zQIAAAAo4MwCAAAAoKGtWbMmXV1dWbx4cfbs2VP32FNPPZVPf/rT6erqyv3331/32NDQUDo7O7Nly5YLXsOyAAAAAArUarVRe7uQXbt25cCBA+nt7c3q1auzevXqusfvueeerF+/Pj/84Q/zy1/+Mvv27Tv/2AMPPJDJkydf1NdsWQAAAABjRH9/fzo7O5Mks2bNyokTJ3Lq1KkkycGDBzN58uRcffXVGTduXG666ab09/cnSfbv3599+/bl5ptvvqjrWBYAAABAgXGj+OtCBgcH09bWdv79KVOmZGBgIEkyMDCQKVOmvONjPT09WbZs2Xv4mgEAAIAxaXh4+IKf8/jjj+f666/Ptddee9F/r1dDAAAAgAIXc5bAaGlvb8/g4OD5948ePZpp06a942NHjhxJe3t7duzYkYMHD2bHjh155ZVX0tLSkunTp2fu3Lnvep1RWRa0vq91NC5zQU21xtiNXDvpg2VHaJj/Fq/+3xNlR0iS/N2UD5cdIUny/qbxZUdIkny0/d+VHSFJcvWE6WVHyIsnflt2hCRJ7//6n2VHSJJ86q//fdkRkiTXTJxZdoQx7xP/bVXZEZIkw0Pnyo6QJGme0FJ2hPzt38woO0KSZM+zL5QdIUkyccrEsiMkSRb231V2hCTJ0MHG+J4pvxsqO0Ee+7e/KjtCkmTo9TfKjpAkef3EmbIjJEn+w4P/WHaEypg3b17Wr1+fxYsXZ+/evWlvb8+kSZOSJNdcc01OnTqVQ4cOZfr06dm+fXvWrVuXJUuWnP/z69evzwc+8IHCRUGiWQAAAACFammcZsENN9yQ6667LosXL06tVsuKFSuyZcuWtLa2ZtGiRVm5cmXuvvvuJMnHPvaxzJx5aT9ksSwAAACAMeQb3/hG3fuzZ88+//sbb7wxvb297/pnv/a1r13UNRxwCAAAANTRLAAAAIACjXTA4WjRLAAAAADqaBYAAABAgUY64HC0aBYAAAAAdTQLAAAAoIBmAQAAAFB5mgUAAABQxKshAAAAAFWnWQAAAAAFnFkAAAAAVN5FLQteeOGFdHZ25tFHH02SvPzyy1m6dGm6u7tz11135Y033risIQGAemYzAIyeWq02am+N4oLLgtOnT2fVqlWZM2fO+Y/dd9996e7uzubNmzNjxoz09fVd1pAAwJvMZgDgcrvgsqClpSUPPvhg2tvbz3/smWeeya233pokueWWW9Lf33/5EgIAdcxmABhdtVH81SgueMBhc3NzmpvrP+3MmTNpaWlJkkydOjUDAwOXJx0A8DZmMwBwuf3Jr4YwPDw8EjkAgBFiNgPAyGqkn/iPlkt6NYQJEyZkaGgoSXLkyJG6GiQAMPrMZgBgJF3SsmDu3LnZunVrkmTbtm1ZsGDBiIYCAN4bsxkAGEkXvA3h+eefT09PTw4fPpzm5uZs3bo169aty7Jly9Lb25uOjo7ceeedo5EVAIjZDACjrZFe0nC0XHBZ8KEPfSiPPPLI2z7+/e9//7IEAgCKmc0AwOX2Jx9wCAAAAH/OHHAIAAAAVJ5mAQAAABTQLAAAAAAqT7MAAAAAClTx1RA0CwAAAIA6mgUAAABQwJkFAAAAQOVpFgAAAEABZxYAAAAAladZAAAAAAWqeGbBqCwLZrT+q9G4zAUNDh0tO0KS5PU/vF52hHRMuKbsCEmS186+VnaEJMm54bNlR0iStGa47AhJkonNrWVHSJL89z0/LjtC/uOH/qHsCEmS//yRfyw7QpKkedz7yo7ACPn7hTeUHSFJcvpM+TMxSf7lf79cdoS0tDTGz3A+Ov/vyo6QJGm7YmLZEZIkr587V3aEJMn75zaVHSFJ0jyu/GLys3v2lR0hSfLRfzO77AhJknPDjfH9I39+GmMqAQAAQIOqYrOg/NUgAAAA0FA0CwAAAKCAV0MAAAAAKs+yAAAAAKjjNgQAAAAo4IBDAAAAoPI0CwAAAKCAZgEAAABQeZoFAAAAUMBLJwIAAACVp1kAAAAAhTQLAAAAgIrTLAAAAIACzrGATFgAAAlbSURBVCwAAAAAKk+zAAAAAArUnFkAAAAAVJ1mAQAAABTQLAAAAAAqT7MAAAAACng1BAAAAKDyLAsAAACAOm5DAAAAgAIOOAQAAAAqT7MAAAAACmgWAAAAAJWnWQAAAAAFvHQiAAAAUHmaBQAAAFDAmQUAAABA5WkWAAAAQAFnFgAAAACVp1kAAAAABap4ZkFteHh4uOwQAAAA0KheOn1g1K7VMWHGqF2riGYBAAAAFKpes8CZBQAAAEAdzQIAAAAoUL1egWYBAAAA8BaWBQAAAEAdtyEAAABAgVqtejciaBYAAAAAdcbEsmDNmjXp6urK4sWLs2fPnrLjlG7t2rXp6urKpz71qWzbtq3sOKUbGhpKZ2dntmzZUnaU0v3kJz/Jxz/+8Xzyk5/Mjh07yo5Tqtdeey1f/epXs3Tp0ixevDg7d+4sO1IpXnjhhXR2dubRRx9Nkrz88stZunRpuru7c9ddd+WNN94oOSFjldlcz2yuZza/yWx+k9n8R2bzWFUbxbfG0PDLgl27duXAgQPp7e3N6tWrs3r16rIjlerpp5/Oiy++mN7e3mzatClr1qwpO1LpHnjggUyePLnsGKU7duxY7r///mzevDkbNmzIL37xi7Ijleqxxx7LzJkz88gjj+Tee++t5HPH6dOns2rVqsyZM+f8x+677750d3dn8+bNmTFjRvr6+kpMyFhlNtczm9/ObP4js7me2Ww2M7Y0/LKgv78/nZ2dSZJZs2blxIkTOXXqVMmpynPjjTfm3nvvTZJcccUVOXPmTM6dO1dyqvLs378/+/bty80331x2lNL19/dnzpw5mTRpUtrb27Nq1aqyI5Wqra0tx48fT5KcPHkybW1tJScafS0tLXnwwQfT3t5+/mPPPPNMbr311iTJLbfckv7+/rLiMYaZzfXM5npm85vM5npms9k8llWvVzAGlgWDg4N1TyRTpkzJwMBAiYnK1dTUlAkTJiRJ+vr6snDhwjQ1NZWcqjw9PT1ZtmxZ2TEawqFDhzI0NJQvfelL6e7urvygueOOO/LSSy9l0aJFWbJkSb75zW+WHWnUNTc3Z/z48XUfO3PmTFpaWpIkU6dOrfTzKZfObK5nNtczm99kNtczm81mxpYx92oIw8PDZUdoCE8++WT6+vry8MMPlx2lNI8//niuv/76XHvttWVHaRjHjx/P9773vbz00kv53Oc+l+3bt1fy5NYkeeKJJ9LR0ZGHHnoov/3tb7N8+XL3zr6F51NGiv+X/shsNpvfidn8JrP5wjyfNrLq/btt+GVBe3t7BgcHz79/9OjRTJs2rcRE5du5c2c2bNiQTZs2pbW1tew4pdmxY0cOHjyYHTt25JVXXklLS0umT5+euXPnlh2tFFOnTs1HPvKRNDc354Mf/GAmTpyY3//+95k6dWrZ0Urx3HPPZf78+UmS2bNn5+jRozl37lylf9qXJBMmTMjQ0FDGjx+fI0eO1NUg4WKZzW9nNv+R2VzPbK5nNr8zs5lG1fC3IcybNy9bt25Nkuzduzft7e2ZNGlSyanK8+qrr2bt2rXZuHFjrrzyyrLjlOq73/1ufvzjH+dHP/pRPvOZz+TLX/5yZb8ZSZL58+fn6aefzh/+8IccO3Ysp0+fruS9gP/fjBkzsnv37iTJ4cOHM3HixMp/M5Ikc+fOPf+cum3btixYsKDkRIxFZnM9s/lNZnM9s7me2fzOzOaxoVarjdpbo2j4ZsENN9yQ6667LosXL06tVsuKFSvKjlSqn/70pzl27Fi+/vWvn/9YT09POjo6SkxFI7jqqqty++2357Of/WyS5Fvf+lbGjWv4feBl09XVleXLl2fJkiU5e/ZsVq5cWXakUff888+np6cnhw8fTnNzc7Zu3Zp169Zl2bJl6e3tTUdHR+68886yYzIGmc31zGbejdlcz2w2mxlbasNujAEAAIB3dXTopVG7Vvv4xlg2V3e1CQAAALyjhr8NAQAAAMpUq+CrIWgWAAAAAHU0CwAAAKCAZgEAAABQeZYFAAAAQB3LAgAAAKCOZQEAAABQxwGHAAAAUKBWc8AhAAAAUHGWBQAAAEAdywIAAACgjjMLAAAAoEAtziwAAAAAKk6zAAAAAAppFgAAAAAVp1kAAAAABarXK9AsAAAAAN5CswAAAAAK1GqN1S1Ys2ZNdu/enVqtluXLl+fDH/7w+ceeeuqpfOc730lTU1MWLlyYr3zlK0mStWvX5tlnn83Zs2fzxS9+MbfddlvhNSwLAAAAYIzYtWtXDhw4kN7e3uzfvz/Lly9Pb2/v+cfvueeePPTQQ7nqqquyZMmS3H777RkcHMyLL76Y3t7eHDt2LJ/4xCcsCwAAAOBP0zjNgv7+/nR2diZJZs2alRMnTuTUqVOZNGlSDh48mMmTJ+fqq69Oktx0003p7+9Pd3f3+fbBFVdckTNnzuTcuXNpamp61+s4swAAAADGiMHBwbS1tZ1/f8qUKRkYGEiSDAwMZMqUKW97rKmpKRMmTEiS9PX1ZeHChYWLgkSzAAAAAMas4eHhi/7cJ598Mn19fXn44Ycv+LmWBQAAAFCgcW5CSNrb2zM4OHj+/aNHj2batGnv+NiRI0fS3t6eJNm5c2c2bNiQTZs2pbW19YLXcRsCAAAAjBHz5s3L1q1bkyR79+5Ne3t7Jk2alCS55pprcurUqRw6dChnz57N9u3bM2/evLz66qtZu3ZtNm7cmCuvvPKirlMbfi+dBQAAAKiYE2/8ftSuNbllygU/Z926dfn1r3+dWq2WFStW5De/+U1aW1uzaNGi/OpXv8q6deuSJLfddlu+8IUvpLe3N+vXr8/MmTPP/x09PT3p6Oh412tYFgAAAECBRlsWjAZnFgAAAECBWq2RTi0YHc4sAAAAAOpYFgAAAAB1LAsAAACAOs4sAAAAgAK1OLMAAAAAqDjNAgAAACikWQAAAABUnGYBAAAAFKher0CzAAAAAHgLzQIAAAAoUKtVr1ugWQAAAADUsSwAAAAA6rgNAQAAAAq5DQEAAACoOM0CAAAAKFC9XoFmAQAAAPAWmgUAAABQqHrdAs0CAAAAoI5mAQAAABSo1TQLAAAAgIqzLAAAAADqWBYAAAAAdZxZAAAAAAVqXg0BAAAAqLra8PDwcNkhAAAAgMahWQAAAADUsSwAAAAA6lgWAAAAAHUsCwAAAIA6lgUAAABAHcsCAAAAoM7/AwM4dpeIHfiQAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "pos = 'VERB'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=pos)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=pos)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to VERB tags')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.3 Target pronouns"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 50,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:46<00:00, 2.88it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABBAAAAJ1CAYAAABgjAMjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xl4VOX9///nJEgBE5AEwiKIiIoIpW5AEQ0SdtqqIJso4opWtEUExGABlUVRKRGrWBW1IhqJUIoblIKyGEFov1RpK4qIwbKExUBkkyS/P/LLfEyBmRBCDrTPx3XNdWXmnDn3O5PAybzndd8nVFBQUIAkSZIkSVIEMUEXIEmSJEmSTnw2ECRJkiRJUlQ2ECRJkiRJUlQ2ECRJkiRJUlQ2ECRJkiRJUlQ2ECRJkiRJUlQ2ECRJ+h83ZcoUGjduzPLly4MuRZIkncBCBQUFBUEXIUnSyezRRx9l2rRpxMfHs3TpUipVqnTY/RYtWkReXh4dOnQIP/bdd98xbdo0BgwYQNWqVY97rWvXruXDDz/kxhtvDD/2xRdf8MUXX9CyZUsSEhKOew2Hc7jXpiykpKTwzTffHPJ45cqVqV+/Pp07d+bmm2+mSpUq4W0jRoxg9uzZhz1e5cqVadCgQfh5R/pZ/+Mf/2DGjBl8/PHHbN26lfz8fGrUqEGLFi3o1asXF1988SHPmTJlCk899RSJiYm89957R/x9aNy4Md27d+eRRx6J+L1v3ryZmTNncvfdd0fcT5KkkqoQdAGSJJ3MDhw4wOzZs4mJiWH37t289957XH311Yfd94UXXqBevXrF3iR/8sknPPXUU3Tv3r1cGgjvvfcef/zjH4s1EM4++2zOPvvs4z52JId7bcrKKaecwuOPPx6+X1BQQHZ2Nh988AFTpkzhL3/5CzNmzKBy5crFnjdkyBAaNGhQ7Hlbt25lwYIFpKWlsXDhQmbMmEHFihWL7fP444/z/PPPk5iYyC9+8QvOPfdcYmNjWb9+PXPnzmX27Nn07NmTMWPGcMoppxxS7/bt25k0aRJjxow5pu972bJlPPXUUzYQJEllxgaCJEnHYP78+ezcuZN+/frx2muv8cYbbxy2gZCfn8+aNWuoV69escc/+eST8io1kPFK4kivTVkJhUJ06dLlkMdvuOEGHnroIV599VXS09OLNVUAWrVqxQUXXHDI8wYMGMD999/PrFmzyMjIoF+/fuFtzz33HM8//zzJycn89re/JS4urthzBw0axOjRo8nIyKBy5co88MADhxy/VatWpKenc8011/DjH/+4lN/1ifmzliSd3FwDQZKkY5Ceng4Uvqm8+OKLWbVqFevWrSu2z6xZs2jSpAl79uxh9uzZNG7cmClTppCSkhL+ZLx9+/Y0bty42PMyMjLo2bMnP/nJT7jwwgvp3r07r7zyCvn5+eF9Nm7cSOPGjRk5ciSff/45t956K5dccgnNmzfnuuuu49NPPy223+LFi/nmm29o3Lgx/fv3B468BsLixYsZMGAAl1xyCc2aNSMlJYWxY8eyY8eOYvulpKTQsWNHdu/ezQMPPECbNm1o1qwZXbt2Ze7cuRFfvyO9NkXWrVvHkCFDaNOmDU2bNuXSSy/lV7/6FZ999lnUn01JXHPNNQBHvf5Dr169AFixYkX4sR07dvC73/2OOnXqMHny5EOaBwAVK1bk4YcfpkmTJkyfPv2Q3xWAoUOHUrVqVUaPHl3sZ300GjduzGuvvRb+OiUlJbxty5YtjB07lnbt2tGsWTNat25N//79WbZs2SHHyc3N5eGHH+ayyy6jefPm9OzZk2XLljFz5kwaN27MrFmzwvtu27aNCRMm0LlzZ37yk5/QsmVL+vTpc8TpIJKkk48NBEmSSmn9+vWsWLGCCy+8kDPPPDOcPJg5c2ax/Vq1asXo0aMBaNmyJWlpaXTt2pXRo0fTsmVLAEaPHk1aWlr4OY888ggjR44kKSmJBx54gOHDh1OzZk3Gjh3Lb37zm0Nq2bp1KzfddBONGjVi5MiRXHfddfy///f/uOOOOzhw4ACJiYmkpaWRkJBAQkICaWlpEaPts2fPZuDAgWRnZ3PHHXfw0EMP0a5dO1577TX69evHnj17iu2fn5/PwIED2bVrF/fccw+DBw/m22+/Zfjw4axZs+aI4xzptYHC9Rp69+7NsmXL6NmzJ+PGjaNfv358/PHH9O3bl3/+859HPG5JFa1hcPDgwaN6XtG0hZiY//tT6t1332Xfvn1ce+21nHrqqUd8boUKFbjxxhspKChgzpw5h2xPSEjgnnvuYc2aNeEmwNFKS0sLT0tJS0sLv8Z79uzh+uuvJz09nSuvvJIJEybwy1/+ku3bt3PzzTezYMGCYscZOnQo06dPp3nz5jzwwANcfvnlDBkyhA8//LDYfnl5edx4443MmDGDjh078tBDD3HvvfdSuXJlRowYwR/+8IdSfR+SpBOLUxgkSSqlN954A/i/T7G7du3KuHHj+OMf/8iQIUPCbzJPP/10kpOTw18XxenPPvts3n33XQCSk5PDEf5//etfvPjii/Tr1y/8xg/g2muv5Ve/+hUZGRlcd911nH/++eFtixcvZvLkyeE33wA5OTm8+eabrFq1itatW9OlSxcmTpwIcNhIf5F9+/YxYcIEEhISeP3118NrM/To0YM6derw2GOPMX36dAYOHBh+zsaNG0lOTi5Wb40aNbjvvvtYsGABTZs2PexYR3ptACZOnEhubi6vv/46F154Yfjx5ORkevXqxRNPPMHzzz9/xO+jJIo+dW/evPlRPe8vf/nLIc/7+9//DhBuCkXy05/+FIC//e1vh93eu3dv3nzzTSZPnkznzp2pUaPGUdXXpUsXXn311fDXRTZs2ED9+vXp168fN910U/jxyy67jK5duzJ9+vTwOhRr1qxh0aJFtGjRgqeffjq8b5s2bbj++uuLjffZZ5/x+eefc9111zF06NBi38fQoUPZvHnzUdUvSToxmUCQJKkUihZPrFy5cvhNe1xcHJ06dWLnzp2HfJJ7NIqaCt26dWPXrl3Fbp07dwaKR+cBateuXax5AITnz2dnZx/V+CtWrCAnJ4du3bodsrBjjx49AHj//fcPed4P35Aey/hQ+En5smXLaNy4cbHmARS+aT/33HPJzMxk//79JTref76O69at46WXXuK3v/0tNWrUKLaOwQ9r+OFzcnJy+Oyzz5gyZQrPPfcc9erVC09lgMIIP0BSUlLUepKSkoiJiQk/5z/FxMQwZswYcnNzo15t4Wg0adKEadOmhX9We/fuZdeuXdSoUYMKFSoUu2JF0bSOn//858WOcckll3DRRRcVeyw2NhYovPrE3r17w4+HQiGeeOIJhg8fXmbfgyQpOCYQJEkqhaLFE6+88spic9179OjBnDlzeOONN+jWrVupjv3FF18AHPIp7w/9+9//Lnb/jDPOOGSfH/3oR8DRx/O//PJLAM4999xDtiUkJHDaaafx1VdfFXs8NjaW008/vUzGh8JPyvPz8znnnHMOu71hw4asXbuWjRs30qhRo4jHOnDgAC1atDjsttatWzNmzBgSExMP2fafDZEiFSpU4PLLL+fhhx8uNlWhaDpDSdctKCgoCL/xPpymTZty7bXX8uqrr9KrVy9atWpVouNGk5mZyTPPPMOaNWvIzc0tti0vLy/8dVEz4YdXoijyk5/8hFWrVoXvN27cmE6dOjF//nzatWtHSkoKrVu3pk2bNoFdGlSSVPZsIEiSVApFiye2bNmSDRs2hB+vXbs2NWrU4KOPPiIrK4v69esf9bG/++47ACZNmnTE6HrNmjWL3f/hpQSPVdH6Bv95WcMilSpVYteuXcUei42Njfhm+GgVvQZVqlQ57Pai5sR/rsVwOKeccgovvPBCsccqVapE/fr1I765HTVqVLHLW37xxRc89NBDtG/fnieffPKQ/YuSB5s2bYr6c9+8eTMFBQVR0wqDBw9m3rx5PPjgg8yZM+ewl308GkuXLuW2224jPj6em266ifPPPz/cBLnllluK7VuUJChaJ+KH4uPjD3ls0qRJzJ49m4yMDGbNmsWbb75JhQoV6Nq1K6NGjSqXy5RKko4vGwiSJB2lL7/8MjyF4HCX4SuSkZHBPffcc9THL3pDV79+/aOem18Wit60H+nN+d69eyMuElgWio4fqYYf7hdJKBQq1af3TZs2LXYZx1atWrF8+XLmzZvHggULwmsFFLnwwgvJyMjgww8/jLoOQtHvz8UXXxxxv6pVqzJs2DDuu+8+pk2bxu23337U38cPvfTSS+Tn55OWlkbr1q3Dj+/bt69Y+gD+ryl1uGki/5lcgMJGTe/evenduzc7duxg6dKlpKenM3fuXLZt28ZLL710TLVLkoLnGgiSJB2losUTe/XqRVpa2iG3iRMnEhsby5tvvlmq+H7Rp95//etfD9n23XfflXjef2kVjb927dpDtmVnZ5OTk8NZZ511XGs488wziY2NPWwNUJgGqFixYnjhyfIycuRI4uLiGDNmDDk5OcW2denShbi4OF5//XV27tx5xGPk5eXx8ssvU6FCBbp37x51zKuvvpqWLVvyzDPPFFujoDQ2btxITExMeBHHIqtWrTpk6kWtWrWAQ6fLAKxevTriOAkJCVx55ZW88sorNG3alMzMTHbv3n1MtUuSgmcDQZKko1C0eGLFihUZMmQIXbp0OeR21VVX0aFDB7Kzs8OLDRbF+//zzX/RvPkfPl60GOJrr73Gvn37iu3/2GOP8dOf/pSvv/66VPXHxMREbUC0bNmShIQE3nnnnUPeJBdN3ShazLEsHO61qVy5Mm3btmXt2rWsXLmy2P4rVqxg/fr1tGvXrkynbpRErVq1uOeee8jOzmbChAnFtsXFxTF06FB27tzJXXfdxbfffnvI8w8cOMCoUaP4xz/+wR133EHdunVLNO6oUaM4ePAgY8eOLXGth/vdqlGjBvn5+cWaAjk5OaSlpVG5cuViv29Fi1cWLepZZOXKlYc0t9544w2Sk5NZt27dITWceuqpxMbGFrvkpSTp5OQUBkmSjsK8efP49ttv6dGjR8T589dffz3z5s1j5syZdOjQgRo1alCpUiWWLFnCs88+S4MGDejSpUv4E/THHnuMFi1acNVVV3HeeecxYMAAXn75Za699lr69OlDhQoV+OCDD5g/fz5XXnnlYRdNLIl69eqRmZnJhAkTqFOnDjfeeOMh+1SsWJHf/OY3DBkyhOuuu46ePXsSHx/P6tWrmTlzJhdccEGxqw8cqyO9NsOHD2flypUMGjSI/v37U69ePb766itmzJhB9erVGTZsWJnVcDT69evHnDlzmD17Nt26dQtfhhIKL7VZ9Ia8c+fOXHnllTRp0oSYmBg2bNjAW2+9xddff83NN9/MoEGDSjzmOeecww033HDIWg6RFP1ujR49mrPOOosbb7yRrl278vHHHzN48GCuvfZadu3axYwZM+jTpw8xMTH87W9/4/e//z0dOnSgVatWNGvWjMWLF3Pvvfdy6aWX8s033/DGG2/ws5/9jLlz54bHatWqFRMnTuSGG26gb9++NGjQgAMHDrB06VJWrFhBr169jvu0F0nS8WcDQZKko1D0CfyAAQMi7teyZUvOPfdclixZwubNm6lduzYjRoxg8uTJPP300/Tu3ZsuXbrQt29fli5dytKlS/n000/p1KkTAKmpqZxzzjmkp6czYcIE8vPzOfPMMxk2bNhh3/SX1K9//Wu++eYbXn31VRo3bnzEY3Xr1o1q1arx7LPP8uSTT3LgwAFOP/10Bg4cyO23316mn/yfcsoph31tGjZsyBtvvMGTTz7JjBkzyMnJoXr16qSkpDBo0KBSLVBZFmJiYnjooYfo2bMno0aN4q233ip2JY477riD9u3b8+qrr7J48WIyMjLIz88nKSmJSy65hN/+9rc0a9bsqMe96667eOedd9i0aVOJ9r/11lv55JNPeOutt6hZs2b4zf3OnTuZNWsWDz74IA0aNGDgwIH07NmTc845h9TUVJ555hnq1q3LWWedxdSpUxk/fjwffPABixYtolmzZjz11FPhSzwWpQoaNGhAeno6zz77LBkZGWzfvp0KFSpw1llnkZqaGvGKIpKkk0eooKCgIOgiJEmSdPJ45JFHePHFF/n9739P27Ztgy5HklROnIwmSZKkQ+zbt497772X+++/v9jj+/fv57333uOUU07hxz/+cUDVSZKC4BQGSZIkHaJSpUoAzJo1i5ycHDp06MC+ffvIyMhg06ZN3HrrrRHXAZEk/fdxCoMkSZIO6/vvv2fatGnMmTOHTZs2kZ+fz1lnnUXPnj3p168foVAo6BIlSeXIBoIkSZIkSYrKNRAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUNhAkSZIkSVJUFYIuQJIkSZKkE1moY71yG6vgzxvLbayjZQJBkiRJkiRFZQNBkiRJkiRF5RQGSZIkSZIiCYWCruCEYAJBkiRJkiRFZQJBkiRJkqRI/Ogd8GWQJEmSJEklYAJBkiRJkqRIXAMBMIEgSZIkSZJKwASCJEmSJEmRGEAATCBIkiRJkqQSMIEgSZIkSVIkroEAmECQJEmSJEklYAJBkiRJkqRI/Ogd8GWQJEmSJEklYANBkiRJkiRF5RQGSZIkSZIicRFFwASCJEmSJEkqARMIkiRJkiRFYgABMIEgSZIkSZJKwASCJEmSJEmRxBhBABMIkiRJkiSpBEwgSJIkSZIUiQEEwASCJEmSJEkqARMIkiRJkiRFEjKCACYQJEmSJElSCZhAkCRJkiQpEgMIgAkESZIkSZJUAiYQJEmSJEmKJMYIAphAkCRJkiRJJWACQZIkSZKkSAwgACYQJEmSJElSCdhAkCRJkiRJUTmFQZIkSZKkSELOYQATCJIkSZIkqQRMIEiSJEmSFImXcQRMIEiSJEmSpBIwgSBJkiRJUiQnWABh/PjxrF69mlAoRGpqKs2bNw9v+/DDD5k0aRKxsbEkJyczaNAg9u7dy4gRI9i+fTv79+/nzjvvpF27dowYMYI1a9Zw2mmnAXDLLbdwxRVXHHFcGwiSJEmSJJ0kVqxYwYYNG0hPT2fdunWkpqaSnp4e3j527FheeOEFatWqxfXXX0/nzp1Zu3YtzZo147bbbuObb77h5ptvpl27dgAMGTIk/HU0NhAkSZIkSYrkBLoKQ2ZmJh06dACgUaNG5OTkkJubS1xcHFlZWVSrVo06deoA0LZtWzIzM+nfv3/4+Zs2baJWrVqlGtsGgiRJkiRJJ4lt27bRtGnT8P2EhASys7OJi4sjOzubhISEYtuysrLC9/v27cvmzZuZOnVq+LHp06fz4osvkpiYyG9+85tiz/9PLqIoSZIkSVIkoXK8HaWCgoIS7/v666/zzDPPMGzYMAoKCrjqqqsYOnQof/jDH2jSpAlPPfVUxOfbQJAkSZIk6SSRlJTEtm3bwve3bt1KzZo1D7tty5YtJCUl8emnn7Jp0yYAmjRpQl5eHjt27KB169Y0adIEgJSUFNauXRtxbBsIkiRJkiRFEhMqv1sUbdq0Yd68eQCsWbOGpKQk4uLiAKhXrx65ubls3LiRgwcPsmjRItq0acPKlSuZNm0aUDgFYs+ePVSvXp277747PMVh+fLlnHPOORHHDhUcTd5BkiRJkqT/MaEBjcttrIKXP4u6z+OPP87KlSsJhUKMHj2af/zjH8THx9OxY0c+/vhjHn/8cQA6derELbfcwr59+xg5ciSbNm1i37593HXXXaSkpPDRRx/x2GOPUblyZapUqcKECRNITEw84rg2ECRJkiRJiiB0Yzk2EF6K3kAIilMYJEmSJElSVF7GUZIkSZKkSEKluDzCfyETCJIkSZIkKSobCJIkSZIkKSqnMEiSJEmSFIkfvQO+DJIkSZIkqQRMIEiSJEmSFImLKAImECRJkiRJUgmYQJAkSZIkKRIDCIAJBOmwGjduTMeOHenSpQudO3fm1ltvJSsrK+iySqVjx44sX7484j4jRozg6aefLqeKJEn/q354fi263XLLLQAMGDCANWvWlNlYS5Ys4d///neZHS+SkSNHMmXKlHIZ6/zzz2fjxo0R9+nfvz9z5swpl3ok/W8xgSAdwSuvvELt2rUBeOKJJxg3bhxTp04NuCpJkk5uPzy//tDLL79cpuO89NJL/PKXv6Ru3bplelxJ/6NcAwGwgSCVyE9/+lMWLlwYvj9z5kymTZtGXl4eNWvWZOLEiZx++uls2bKF4cOHk52dzYEDB/jZz37GPffcQ0FBAb/73e+YO3cuBw4coH379tx///3ExsYWG2fWrFm8//77nHLKKaxatYqGDRsyaNAgHn/8cb7++mt+/etf06dPH/Lz80lLS2PevHkAXHDBBYwaNYoqVarw6aefct9993Hw4EHatm1b7PgLFiwgLS2NPXv20KBBAx5//HESEhKO/wsoSVIUKSkpTJw4kdq1a9O3b18GDhzIzJkz+fbbb7n//vvp1q1bic+nkydP5qOPPuLLL79k2LBhLF68mDPOOIM777wTKEzeFd1PSUlh4MCBZGRksHnzZn7+858zYsQI4MjnzZ07d3Lvvffy1VdfcfbZZ1OpUqXDNkWmTJnCtm3b2Lx5M2vWrKF169Z069aNKVOmsHXrVh5++GHatWvH/v37GTduHMuXLycmJoa2bdsybNgwYmNj+eCDDxg7diwVKlTgmmuuKXb89PR0XnzxRQ4cOMAFF1zA+PHjqVSp0nH6CUmSUxikqA4cOMCf/vQnUlJSANi+fTsPPfQQL774IvPnz+eMM84Ix/9feuklWrRowTvvvMPcuXPJyspi69atzJkzh/fee4+MjAz+/Oc/k5WVxWuvvXbY8ZYsWcJdd93F/PnzWbduHc8//zyvvvoq48aNC4/z7rvvsnjxYmbNmsXbb7/Nrl27eOmllwAYM2YMN9xwA/PmzePCCy8MxxyzsrIYPnw4TzzxBH/5y19o1aoVY8aMOb4vniRJpbBz505iYmKYO3cuqampTJ48GaDE59PBgwdTq1YtHnvsMbp16xZ1vI8//pj09HTefPNNpk+fzubNmyOeN5977jmqV6/OwoULGTVqFEuXLj3isd9//33Gjx/P3Llzee+998Ln7zvuuIPnnnsOKExfbN68mbfffpvZs2ezcuVK3nrrLfLy8hg5ciSjR4/m3XffJSYmhry8PABWrlxJWloaL7/8MgsXLiQuLo60tLSjfakllVRMOd5OYCd4eVJw+vfvT5cuXWjTpg2ffPIJPXr0ACAxMZFVq1aFP2m45JJLwusjJCYmsnTpUlauXEnFihWZNGkSSUlJLFq0iGuuuYb4+HgqVKhAr169mD9//mHHPfvss2nYsCEVK1akQYMGXHbZZcTGxnLuueeydetWoPCPkauvvpoqVaoQGxtLjx49WLZsGfv37+eTTz4J/7HUpUsXKleuDMDixYtp2bIl5557LgB9+/Zl4cKF4T9EJEkqD0Xn16LbAw88cMg+Bw8eDJ93mzZtGl7L4GjOp0fjF7/4BbGxsdSqVYvExEQ2bdoU8by5cuVKunbtCkC9evVo2bLlEY994YUXkpiYSPXq1alZsybJyckAh5zXe/fuTYUKFahUqRK/+MUvWLZsGV999RUHDhzgsssuA6B79+7h4y5cuJBu3bpRq1YtAK699toyeS0kKRKnMEhH8MM5mh9//DH9+/dn1qxZJCYm8uSTT4b/iPjuu+9o2LAhADfeeCP5+fk8+OCDbN26leuuu467776b3bt388ILL5Ceng5AXl7eEacOnHrqqeGvY2NjqVKlSvjr/Px8AHbs2EG1atXC+1WrVo3t27fz7bffAhAXFwdAKBSiatWqAOzevZuVK1fSpUuX8PPi4uLCz5EkqTwcaQ2EH/rh+S8mJiZ8/jvS+XTLli0MGDAAgObNmzNx4sSjqqnovFk0dl5eXsTzZk5ODvHx8eHHi861h3Ok8/oPv68jnddzcnKK1fbDfXbv3s2f//zncPqhoKCA77///qi+b0lHwTUQABsIUom0aNGCunXrsmrVKg4ePMjChQuZPn06CQkJvPHGG8ydOxeAChUqMHDgQAYOHMj69eu57bbbuPjii0lKSiIlJYXrr7++TOqpUaNGsTf+3377LTVq1Aj/YZGbm0t8fDz5+fnk5OQAkJSUxKWXXsqTTz5ZJjVIklTeIp1P33vvvSM+74dv1oHwuTHaWEc6b1atWpXdu3eH7+/YsYP69etHPeaRRDqv5+bmFhvnh/V1796d++67r9TjStLRcgqDVALr169n/fr1nHXWWWzfvp3TTz89vIjSu+++y3fffQfAqFGjWLZsGQBnnHEGNWrUIBQK0b59e+bMmcPevXsBeP3115k9e3ap67niiiv405/+xN69ezl48CAZGRm0bduWSpUqcd555/HnP/8ZgLfffpv9+/cDcNlll7Fy5crwdIu///3vjB07ttQ1SJJU3o7mfFqhQoXwm/yaNWvyr3/9CyhcE+ivf/1r1LEinTcvuOACFixYAMDXX3/NqlWrjun7uuI2kkCPAAAgAElEQVSKK8jIyCAvL489e/YwZ84c2rZtyxlnnEFsbGz4csyzZs0i9P9/CpqSksL8+fPDTYUFCxbw+9///pjqkBRBqBxvJzATCNIR9O/fP7yqc8WKFXnwwQdp3LgxiYmJvP3223Ts2JH69eszePBgfvnLX/LII4/Qt29fRo0axcMPP0xBQQEpKSm0bt0agM8//zw8d/GMM85g3Lhxpa6tS5cufPbZZ/To0YOCggJatWrFDTfcABQuopiamsqzzz5LcnIyjRo1Ago/qXj44YcZNGgQ33//PaeeeiqpqanH8hJJklSuOnToUOLzaefOnRkyZAi/+tWv6N27N3fddRedOnXi/PPPp3PnzlHHinTevP3227nnnntISUmhUaNGdOrU6Zi+r/79+5OVlcXPfvYzQqEQXbp0oWvXroRCIR5++GFSU1OpWLEiPXr0CE+BaNq0KXfccQf9+/cnPz+fxMREHnzwwWOqQ5KiCRUUFBQEXYQkSZIkSSeq0F3Nym2sgqc+LbexjpZTGCRJkiRJUlQ2ECRJkiRJUlSugSBJkiRJUiRexhEwgSBJkiRJkkqgXBIIHd4cUB7DRHXnRV2DLgGAzH//LegSiDlBrg/SoFrdoEsA4NzqZwddAgBjPngl6BIAWLb070GXAMCI/lcHXQK1T00MugQATpT1bhtUrRd0CQB0b9gn6BJOekOWDAu6BAB+flZK0CUAkFeQH3QJrN/1VdAlALDn+31BlwDAFfUuC7oEAO5fPDXoEgBYsDj4vx8Bbr2yXdAlcGWjtkGXAEDOgV1BlwDAGfH1gy4BgEtrBf+7cVydGG+fAmcCQZIkSZIkReUaCJIkSZIkRRByDQTABIIkSZIkSSoBEwiSJEmSJEVgAqGQCQRJkiRJkhSVCQRJkiRJkiIwgFDIBIIkSZIkSYrKBIIkSZIkSRHEGEEATCBIkiRJkqQSMIEgSZIkSVIEXoWhkAkESZIkSZIUlQkESZIkSZIiMIFQyASCJEmSJEmKqtQJhPHjx7N69WpCoRCpqak0b968LOuSJElHyXOzJEk6nkrVQFixYgUbNmwgPT2ddevWkZqaSnp6elnXJkmSSshzsyRJx49TGAqVagpDZmYmHTp0AKBRo0bk5OSQm5tbpoVJkqSS89wsSZKOt1I1ELZt20b16tXD9xMSEsjOzi6zoiRJ0tHx3CxJ0vETCpXf7URWJosoFhQUlMVhJElSGfHcLEmSylqp1kBISkpi27Zt4ftbt26lZs2aZVaUJEk6Op6bJUk6flwDoVCpEght2rRh3rx5AKxZs4akpCTi4uLKtDBJklRynpslSdLxVqoEwkUXXUTTpk3p27cvoVCI0aNHl3VdkiTpKHhuliTp+DGBUKhUDQSAoUOHlmUdkiTpGHluliRJx1OpGwiSJEmSJP0vCGECAcroKgySJEmSJOm/mwkESZIkSZIicA2EQiYQJEmSJElSVCYQJEmSJEmKwABCIRMIkiRJkiQpKhMIkiRJkiRFEGMEATCBIEmSJEmSSsAEgiRJkiRJEXgVhkImECRJkiRJUlQ2ECRJkiRJUlTlMoVhwI+Ty2OYqBZuWBF0CQDkFRQEXQJDLx4YdAkAvLluTtAlAFCzUo2gSwDg3V7PBF0CAGMbPBJ0CQD0OLtr0CWwP29/0CUA8PGWvwVdAgCfbPss6BIA6N4w6ApOfvWr1gq6BABmf7Eg6BIAOLNa7aBLoM851wRdAgCvfpYedAkAZOVmBV0CACN+em3QJQBwc/P2QZcAwGk/qh50CazZ/q+gSwCgQdV6QZcAwLAFvw+6BACWXdcu6BKOK6cwFDKBIEmSJEmSonIRRUmSJEmSIjCAUMgEgiRJkiRJisoEgiRJkiRJEbgGQiETCJIkSZIkKSoTCJIkSZIkRXCiJRDGjx/P6tWrCYVCpKam0rx58/C2Dz/8kEmTJhEbG0tycjKDBg1i7969jBgxgu3bt7N//37uvPNO2rVrx6ZNmxg+fDh5eXnUrFmTxx57jIoVKx5xXBMIkiRJkiSdJFasWMGGDRtIT09n3LhxjBs3rtj2sWPHMmXKFF577TWWLVvGF198waJFi2jWrBnTp09n8uTJPPJI4WXbn3zySfr168eMGTNo0KABGRkZEce2gSBJkiRJUgShUKjcbtFkZmbSoUMHABo1akROTg65ubkAZGVlUa1aNerUqUNMTAxt27YlMzOTbt26cdtttwGwadMmatWqBcDy5ctp3749AO3atSMzMzPi2E5hkCRJkiTpJLFt2zaaNm0avp+QkEB2djZxcXFkZ2eTkJBQbFtWVlb4ft++fdm8eTNTp04FYO/eveEpC4mJiWRnZ0cc2waCJEmSJEkRnGhrIPxQQUFBifd9/fXX+ec//8mwYcP405/+dNTHcQqDJEmSJEkniaSkJLZt2xa+v3XrVmrWrHnYbVu2bCEpKYlPP/2UTZs2AdCkSRPy8vLYsWMHVapUYd++fcX2jcQGgiRJkiRJEYRC5XeLpk2bNsybNw+ANWvWkJSURFxcHAD16tUjNzeXjRs3cvDgQRYtWkSbNm1YuXIl06ZNAwqnQOzZs4fq1atz6aWXho81f/58Lr/88ohjO4VBkiRJkqSTxEUXXUTTpk3p27cvoVCI0aNHM2vWLOLj4+nYsSNjxozh3nvvBaBbt240bNiQOnXqMHLkSPr168e+ffsYNWoUMTEx3H333dx3332kp6dTt25drr766ohj20CQJEmSJCmCE20NhKFDhxa7f95554W/btGiBenp6cW2V6pUiSeeeOKQ4yQlJfHiiy+WeFynMEiSJEmSpKhsIEiSJEmSpKicwiBJkiRJUgQn2hSGoJhAkCRJkiRJUZlAkCRJkiQpghgTCIAJBEmSJEmSVAImECRJkiRJisAAQiETCJIkSZIkKSoTCJIkSZIkReBVGAqZQJAkSZIkSVGZQJAkSZIkKYIQJhDABIIkSZIkSSoBEwiSJEmSJEXgGgiFTCBIkiRJkqSoTCBIkiRJkhSBCYRCJhAkSZIkSVJUJhAkSZIkSYrAAEIhEwiSJEmSJCkqGwiSJEmSJCmqcpnCsDjr7+UxTFQDml4ZdAkANKp6TtAlsGLrR0GXAECThLODLgGAdbu+DLoEAOZ9vTDoEgD4ad0Lgy4BgJ37dwZdAjM/WxB0CQCkNLgo6BIAOPWU3KBLUBn5ZOuGoEsA4M4L+gZdAgAJP0oMugRWbF0RdAkAnJfQKOgSANh94MT4/2bSijlBlwDAjc2Tgy4BgEoVKgVdAqs2nxh/tx3I/z7oEgB48Ir+QZfwP8FFFAuZQJAkSZIkSVG5iKIkSZIkSRGYQChkAkGSJEmSJEVlAkGSJEmSpAhMIBQygSBJkiRJkqIygSBJkiRJUgQGEAqZQJAkSZIkSVGZQJAkSZIkKQLXQChkAkGSJEmSJEVlAkGSJEmSpAhMIBQygSBJkiRJkqIygSBJkiRJUgQmEAqZQJAkSZIkSVGVOoEwceJEVq1axcGDB7n99tvp1KlTWdYlSZKOkudmSZKODwMIhUrVQPjoo4/4/PPPSU9PZ+fOnXTv3t0/UiRJCpDnZkmSdLyVqoHQokULmjdvDkDVqlXZu3cveXl5xMbGlmlxkiSpZDw3S5Kk461UDYTY2FiqVKkCQEZGBsnJyf6BIklSgDw3S5J0/LiIYqFjugrDggULyMjIYNq0aWVVjyRJOgaemyVJ0vFS6gbCkiVLmDp1Ks8//zzx8fFlWZMkSSoFz82SJB0fJhAKlaqBsHv3biZOnMhLL73EaaedVtY1SZKko+S5WZIkHW+laiC888477Ny5k8GDB4cfe/TRR6lbt26ZFSZJkkrOc7MkScePCYRCpWog9OnThz59+pR1LZIkqZQ8N0uSpOPtmBZRlCRJkiTpv50BhEIxQRcgSZIkSZJOfCYQJEmSJEmKwDUQCplAkCRJkiRJUZlAkCRJkiQpEhMIgAkESZIkSZJUAiYQJEmSJEmKwDUQCplAkCRJkiRJUZlAkCRJkiQpAgMIhUwgSJIkSZKkqMolgXBTs6vLY5ioPvz3x0GXAEDtKnWCLoGD+XlBlwDA/rw9QZcAwLpvs4IuAYA+53YPugQAvj2wM+gSAPjDmtlBl0Cbek2DLgGAxqedG3QJAHy1e0PQJaiM/KRWw6BLAGD9rvVBlwBAftX8oEvg2/05QZcAwCVJFwVdAgBTV88IugQAJqfcHXQJALz/zdKgSwBg14HcoEvgotonxv9fbeq2CroEADZ/tznoEvQ/xCkMkiRJkiRF4CKKhZzCIEmSJEmSojKBIEmSJElSBCYQCplAkCRJkiRJUZlAkCRJkiQpAhMIhUwgSJIkSZKkqEwgSJIkSZIUgQGEQiYQJEmSJElSVCYQJEmSJEmKwDUQCplAkCRJkiRJUZlAkCRJkiQpAhMIhWwgSJIkSZJ0Ehk/fjyrV68mFAqRmppK8+bNw9s+/PBDJk2aRGxsLMnJyQwaNAiAiRMnsmrVKg4ePMjtt99Op06dGDFiBGvWrOG0004D4JZbbuGKK6444rg2ECRJkiRJiuBESiCsWLGCDRs2kJ6ezrp160hNTSU9PT28fezYsbzwwgvUqlWL66+/ns6dO7Nt2zY+//xz0tPT2blzJ927d6dTp04ADBkyhHbt2pVobBsIkiRJkiSdJDIzM+nQoQMAjRo1Iicnh9zcXOLi4sjKyqJatWrUqVMHgLZt25KZmUm/fv3CKYWqVauyd+9e8vLyjnpsF1GUJEmSJCmCUChUbrdotm3bRvXq1cP3ExISyM7OBiA7O5uEhIRDtsXGxlKlShUAMjIySE5OJjY2FoDp06dzww03cM8997Bjx46IY9tAkCRJkiTpJFVQUFDifRcsWEBGRgajRo0C4KqrrmLo0KH84Q9/oEmTJjz11FMRn28DQZIkSZKkk0RSUhLbtm0L39+6dSs1a9Y87LYtW7aQlJQEwJIlS5g6dSrPPfcc8fHxALRu3ZomTZoAkJKSwtq1ayOObQNBkiRJkqQIQqHyu0XTpk0b5s2bB8CaNWtISkoiLi4OgHr16pGbm8vGjRs5ePAgixYtok2bNuzevZuJEyfy7LPPhq+4AHD33XeTlZUFwPLlyznnnHMiju0iipIkSZIknSQuuugimjZtSt++fQmFQowePZpZs2YRHx9Px44dGTNmDPfeey8A3bp1o2HDhuGrLwwePDh8nEcffZTrrruOwYMHU7lyZapUqcKECRMijm0DQZIkSZKkCE6kyzgCDB06tNj98847L/x1ixYtil3WEaBPnz706dPnkOPUrVuXN998s8TjOoVBkiRJkiRFZQJBkiRJkqQITrQEQlBMIEiSJEmSpKhMIEiSJEmSFIEJhEImECRJkiRJUlQmECRJkiRJisAAQiETCJIkSZIkKSoTCJIkSZIkReAaCIVMIEiSJEmSpKhMIEiSJEmSFIkJBMAEgiRJkiRJKgETCJIkSZIkReAaCIVMIEiSJEmSpKjKJYFQq3Lt8hgmqjpxNYMuAYCD+QeDLoEz4usHXQIA/9jxz6BLAKDPud2DLgGAbfuygy4BgPNOaxp0CQDcc9FtQZdwwvxM1n77edAlABBfMT7oElRGrj2nb9AlAPC7vz8bdAkA1KhcI+gSOKvqmUGXAECIE+NTtl9fdHPQJQDwff73QZcAQL8T5N/sJztWB10C1X5ULegSANh1YFfQJQAweeXcoEsA4MozewVdgsqBUxgkSZIkSYog5sTorQbOKQySJEmSJCkqEwiSJEmSJEXgIoqFTCBIkiRJkqSoTCBIkiRJkhRBjAkEwASCJEmSJEkqARMIkiRJkiRF4BoIhUwgSJIkSZKkqEwgSJIkSZIUgZ+8F/J1kCRJkiRJUZlAkCRJkiQpAq/CUMgEgiRJkiRJisoEgiRJkiRJEXgVhkLHlEDYt28fHTp0YNasWWVVjyRJOgaemyVJ0vFyTAmEZ555hmrVqpVVLZIk6Rh5bpYkqey5BkKhUicQ1q1bxxdffMEVV1xRhuVIkqTS8twsSZKOp1I3EB599FFGjBhRlrVIkqRj4LlZkiQdT6WawvDHP/6RCy64gPr165d1PZIkqRQ8N0uSdPy4iGKhUjUQ3n//fbKysnj//ffZvHkzFStWpHbt2lx66aVlXZ8kSSoBz82SJOl4K1UDYfLkyeGvp0yZwumnn+4fKJIkBchzsyRJx88xXb7wv4ivgyRJkiRJiuqYLuMIcPfdd5dFHZIkqYx4bpYkqWx5GcdCJhAkSZIkSVJUx5xAkCRJkiTpv5lXYShkAkGSJEmSJEVlAkGSJEmSpAhcA6GQCQRJkiRJkhSVCQRJkiRJkiIwf1DIBIIkSZIkSYrKBIIkSZIkSRG4BkIhEwiSJEmSJCkqEwiSJEmSJEVgAqGQCQRJkiRJkhRVuSQQVmavLI9houp6xs+DLgGA3/19atAl0Lxm46BLAKB2lVpBlwDApzs+DboEAC6v0zboEgDIObAz6BIAWL5ledAlMG/9ifH/V9KppwZdAgD3XnhX0CWojHy6Y3XQJQBw4/n9gi4BgPW7vgy6BOrF1Q+6BAB2fZ8TdAkAfPPdN0GXAECzhOZBlwBA7sHdQZcAwOlx9YIugadXvxx0CQBUij0xwtwTkm8NugT9DzkxfuslSZIkSTpBhZzCADiFQZIkSZIklYAJBEmSJEmSInARxUImECRJkiRJUlQmECRJkiRJisD8QSETCJIkSZIkKSoTCJIkSZIkReAaCIVMIEiSJEmSpKhMIEiSJEmSFIEJhEImECRJkiRJUlQmECRJkiRJiiBkAgEwgSBJkiRJkkrABIIkSZIkSRG4BkIhEwiSJEmSJCkqEwiSJEmSJEVg/qCQDQRJkiRJkk4i48ePZ/Xq1YRCIVJTU2nevHl424cffsikSZOIjY0lOTmZQYMGATBx4kRWrVrFwYMHuf322+nUqRObNm1i+PDh5OXlUbNmTR577DEqVqx4xHGdwiBJkiRJ0klixYoVbNiwgfT0dMaNG8e4ceOKbR87dixTpkzhtddeY9myZXzxxRd89NFHfP7556Snp/P8888zfvx4AJ588kn69evHjBkzaNCgARkZGRHHtoEgSZIkSVIEMaFQud2iyczMpEOHDgA0atSInJwccnNzAcjKyqJatWrUqVOHmJgY2rZtS2ZmJi1atCAtLQ2AqlWrsnfvXvLy8li+fDnt27cHoF27dmRmZkZ+HY7lRZQkSZIkSeVn27ZtVK9ePXw/ISGB7OxsALKzs0lISDhkW2xsLFWqVAEgIyOD5ORkYmNj2bt3b3jKQmJiYvg4R+IaCJIkSZIkRXAiX8axoKCgxPsuWLCAjIwMpk2bVqrjmECQJEmSJOkkkZSUxLZt28L3t27dSs2aNQ+7bcuWLSQlJQGwZMkSpk6dynPPPUd8fDwAVapUYd++fYfseyQ2ECRJkiRJiiAUCpXbLZo2bdowb948ANasWUNSUhJxcXEA1KtXj9zcXDZu3MjBgwdZtGgRbdq0Yffu3UycOJFnn32W0047LXysSy+9NHys+fPnc/nll0cc2ykMkiRJkiSdJC666CKaNm1K3759CYVCjB49mlmzZhEfH0/Hjh0ZM2YM9957LwDdunWjYcOGpKens3PnTgYPHhw+zqOPPsrdd9/NfffdR3p6OnXr1uXqq6+OOLYNBEmSJEmSIjjRovtDhw4tdv+8884Lf92iRQvS09OLbe/Tpw99+vQ57LFefPHFEo97or0OkiRJkiTpBGQCQZIkSZKkCEqyNsH/AhMIkiRJkiQpKhMIkiRJkiRFEGMCATCBIEmSJEmSSsAEgiRJkiRJEZhAKGQCQZIkSZIkRWUCQZIkSZKkCLwKQyETCJIkSZIkKapySSDUrFSjPIaJ6u0Nfwq6BAAuSGoSdAm0qNkq6BIAWLp5SdAlAFA/rl7QJQDwde76oEsAoFJs5aBLAOD8hOD/rSRVSQq6BACqVDgxfiYPLZ8UdAkAPN3ut0GXcNK76ZXHgy4BgNFX9w66BAAuqNE86BKoUuHUoEsAYG3O2qBLACCxUkLQJQDwec5nQZcAQI0T5O/pf3/376BLoPvZHYMuAYAvd30VdAkAzPjXifEep1VSctAlHFcxmEAAEwiSJEmSJKkEbCBIkiRJkqSoXERRkiRJkqQIXESxkAkESZIkSZIUlQkESZIkSZIiiDGBAJhAkCRJkiRJJWACQZIkSZKkCEJexhEwgSBJkiRJkkrABIIkSZIkSRF4FYZCJhAkSZIkSVJUJhAkSZIkSYrAqzAUMoEgSZIkSZKiMoGg/6+9+4+xs673BP4+02FumU4tM7VT2kW6E2SF1Hilu97YHwiFaerCFQminS1tza67e41KQCFL0yXStVBTYoxAiSKiqzQ1o6Uo2aiDettdIkPZBBeW3utWMbf2B7Yz9gcObSmF2T8mW3YEnnMYO/Ocsa/XZBLOOXPO86Yw59N8z/v5PgAAABSo+Ow9yZ/RQHjkkUdy1VVX5ZprrsnWrVtPYSQAYCTMZgBgNI2ogXDw4MHce++9eeihh3LkyJHcc889ufTSS09xNACgVmYzAIweeyAMGdECQm9vb+bOnZuWlpa0tLRkzZo1pzoXAPAWmM0AwGgb0SkMu3fvzrFjx/LJT34yS5cuTW9v76nOBQC8BWYzAIyeSqUyZt/1bMSbKB46dCjr16/P3r17s2LFimzZsqXu/2UB4C+Z2QwAjKYRNRCmTp2aiy66KI2NjTn33HMzadKkHDhw4FRnAwBqZDYDAKNtRAsICxYsyBNPPJFXX301Bw8ezJEjR9La2nqqswEANTKbAWD0VMbwq56N6BSG6dOnZ/HixfnYxz6WJLn11lvT0OC6mABQFrMZABhtI94DoaurK11dXacyCwDwZzCbAWB0uIzjEB9NAAAAAFWNuIEAAAAApwNXNRqigQAAAABUpYEAAAAABRp89p5EAwEAAACogQYCAAAAFLAHwhANBAAAAKAqDQQAAAAooIEwRAMBAAAAqEoDAQAAAAo0RAMh0UAAAAAAajAmDYRn+v9hLA5T1Yc6rig7QpKk+9eby46Q+WdfUnaEJEmlTlbyDr10qOwISZL3T19QdoQkyYTKhLIjJEl2Dvy27Ah559veWXaEJMmBl/5QdoQkyUVnn192BE6Rzf9+TdkRkiRvnzit7AhJkt0Du8qOkImNzWVHSJL8j13byo6QJPnUX3+i7AhJkkmNk8uOkCSpVOrjc78JlfILzC1ntJQdIUkyvfnssiMkSX783FNlRzgt2ANhSH28EwEAAAB1zQICAAAAUFX5HSQAAACoYw1OYUiigQAAAADUQAMBAAAACtTL5u9l00AAAAAAqtJAAAAAgAINdXIp1bL5UwAAAACq0kAAAACAAhVXYUiigQAAAADUQAMBAAAACrgKwxANBAAAAKAqDQQAAAAo0GAPhCQaCAAAAEANNBAAAACggD0QhmggAAAAAFVpIAAAAEABeyAM0UAAAAAAqrKAAAAAAFTlFAYAAAAoUKn47D3RQAAAAIBxZe3atVmyZEm6urryzDPPDHvs8ccfz7XXXpslS5bk3nvvPXn/jh070tnZmQ0bNpy8b+XKlfnQhz6U5cuXZ/ny5dm6dWvhcTUQAAAAoEA9XcbxySefzM6dO9Pd3Z3nnnsuq1atSnd398nHb7/99jzwwAOZPn16li1blsWLF2fmzJlZs2ZN5s6d+7rX+9znPpeFCxfWdGwNBAAAABgnent709nZmSQ577zzcvjw4QwMDCRJdu3alSlTpmTGjBlpaGjIJZdckt7e3jQ1NeX+++9Pe3v7n3VsCwgAAABQoKFSGbPvavr7+9Pa2nrydltbW/r6+pIkfX19aWtre91jjY2NmThx4hu+3oYNG7JixYp89rOfzYEDB4r/HGr5wwIAAADqz+Dg4Iif++EPfzg333xzvvOd7+TCCy/M+vXrC3/eAgIAAAAUqFQqY/ZdTXt7e/r7+0/e3r9/f6ZNm/aGj+3bt6/wtIW5c+fmwgsvTJJcdtll2bFjR+GxLSAAAADAODF//vz09PQkSbZv35729va0tLQkSc4555wMDAxk9+7dOXHiRLZs2ZL58+e/6Wtdf/312bVrV5Jk27ZtOf/88wuP7SoMAAAAUKChjq7CMGfOnMyePTtdXV2pVCq57bbbsnnz5kyePDmLFi3K6tWrc9NNNyVJrrjiinR0dOTZZ5/NunXrsmfPnjQ2Nqanpyf33HNPrrvuutx4440588wz09zcnC9+8YuFx7aAAAAAAOPIzTffPOz2BRdccPKf3/e+9w27rGOSvPvd786DDz74utd5//vfn4ceeqjm41pAAAAAgAK17E1wOrAHAgAAAFCVBgIAAAAUqFR89p5oIAAAAAA1GJMGwt92/OuxOExV/cf6yo6QJJk341+VHSFT//aisiMkSf7nxu7qPzQGzm6eWXaEJMm+o3vLjpAk+e0Lz5UdIUly/JXjZUfIjEkzyo6QJLly/S1lR0iSrFj05pcBYnx56ZWXyo6QJNk18LuyIyRJnjv827IjZMHf/buyIyRJfvnN+pjNjZUzyo6QJPnDS/Xx98fjdfI7+9jex8uOkBdeGig7QpLkkX98puwISZKb519ddoTTQj1dhaFMGggAAABAVRYQAAAAgKpsoggAAAAFXMZxiAYCAAAAUJUGAgAAABSo2EQxiQYCAAAAUAMNBAAAAChgD4QhGggAAABAVRoIAAAAUKDBHghJNBAAAACAGmggAAAAQIFKxWfviQYCAAAAUAMNBAAAAChQsQdCkhEuILz44ou55ZZbcvjw4bz88sv59Kc/nYsvvvhUZwMAamQ2AwCjbUQLCA8//HA6Ojpy0003Zd++ffn4xz+en/zkJ6c6GwBQI7MZAEZPpaKBkIxwD4TW1tYcOnQoSfLCCy+ktbX1lIYCAN4asxkAGG0jaiBceeWV2bx5cxYtWpQXXngh991336nOBQC8BV39GiMAABL9SURBVGYzAIweeyAMGVED4Yc//GFmzpyZn/70p/n2t7+dL3zhC6c6FwDwFpjNAMBoG9ECwlNPPZUFCxYkSS644ILs378/r7zyyikNBgDUzmwGAEbbiBYQZs2alaeffjpJsmfPnkyaNCkTJkw4pcEAgNqZzQAweiqVyph917MR7YGwZMmSrFq1KsuWLcuJEyeyevXqUxwLAHgrzGYAYLSNaAFh0qRJueuuu051FgBghMxmABg9DTZRTDLCUxgAAACA08uIGggAAABwuqj3vQnGigYCAAAAUJUGAgAAABSo+Ow9iQYCAAAAUAMNBAAAAChgD4QhGggAAABAVRoIAAAAUKASDYREAwEAAACogQYCAAAAFGiwB0KSMVpAeHXw1bE4zLjxy77/XXaE/GrTj8uOkCS575nvlB0hSXLh2zvKjpAkOavpbWVHSJJcOvOysiMkSa75wc1lR8h9H/xPZUdIkvzshq+UHSFJcvClg2VH4BQ5o+GMsiMkSQ69dLjsCEmSfzq8p+wI+T/f+W9lR0iSvPjyQNkRkiTnXP/BsiMkSX665stlR0iSXDT1X5YdIUnyh6Pl/85efu6CsiMkSX73Ql/ZEZIkA3XyO8vpQQMBAAAACtgDYYg9EAAAAICqLCAAAAAAVTmFAQAAAApUbKKYRAMBAAAAqIEGAgAAABSo+Ow9iQYCAAAAUAMNBAAAAChgD4QhGggAAABAVRoIAAAAUKAhGgiJBgIAAABQAw0EAAAAKGAPhCEaCAAAAEBVGggAAABQoGIPhCQaCAAAAEANNBAAAACggD0QhmggAAAAAFVpIAAAAECBis/ek2ggAAAAADWwgAAAAABU5RQGAAAAKNBgE8UkGggAAABADTQQAAAAoEAlGgiJBgIAAABQAw0EAAAAKFCxB0ISDQQAAAAYV9auXZslS5akq6srzzzzzLDHHn/88Vx77bVZsmRJ7r333pP379ixI52dndmwYcPJ+55//vksX748S5cuzQ033JDjx48XHtcCAgAAABSojOFXNU8++WR27tyZ7u7u3HHHHbnjjjuGPX777bfnnnvuyXe/+9384he/yG9+85scOXIka9asydy5c4f97N13352lS5dm48aNmTVrVjZt2lR4bAsIAAAAME709vams7MzSXLeeefl8OHDGRgYSJLs2rUrU6ZMyYwZM9LQ0JBLLrkkvb29aWpqyv3335/29vZhr7Vt27ZcfvnlSZKFCxemt7e38NgWEAAAAKBApVIZs+9q+vv709raevJ2W1tb+vr6kiR9fX1pa2t73WONjY2ZOHHi617r6NGjaWpqSpJMnTr15Ou8GQsIAAAAME4NDg6O2eu4CgMAAAAUaKijz97b29vT399/8vb+/fszbdq0N3xs3759rztt4f/X3NycY8eOZeLEiVV/NtFAAAAAgHFj/vz56enpSZJs37497e3taWlpSZKcc845GRgYyO7du3PixIls2bIl8+fPf9PXmjdv3snXevTRR3PxxRcXHlsDAQAAAArUsjfBWJkzZ05mz56drq6uVCqV3Hbbbdm8eXMmT56cRYsWZfXq1bnpppuSJFdccUU6Ojry7LPPZt26ddmzZ08aGxvT09OTe+65J9dff31uueWWdHd3Z+bMmbn66qsLj10ZPFUnTBQ4+78Ur2KMlX+6tafsCEmSfzj4dNkRMv3MGWVHSJL8rz/8suwISZKL3j6n7AhJkis33FB2hCTJf/3IqrIjJEl2DewuO0LeOeWdZUdIkvxs15ayIyRJ/ubs+vhd+Ztp9TFXxrOF319edoQkyY+vua/sCEmSF1/+Y9kRMphR/ytZTc6c0Fx2hCRJpVIfRdl6+e/Sf+z3ZUdIkvzhWH/1Hxpl/2LK7LIjJEkee/7vy46QJNl/tHjTu7Hy8Xf9h7IjjKon9v/3MTvW+9svGbNjvVUaCAAAAFCgkvppIJSpPpZ2AQAAgLpmAQEAAACoyikMAAAAUKCeNlEskwYCAAAAUJUGAgAAABSwieIQDQQAAACgKg0EAAAAKKCBMEQDAQAAAKhKAwEAAACKuApDEg0EAAAAoAYaCAAAAFDAHghDNBAAAACAqmpaQNixY0c6OzuzYcOGJMnzzz+f5cuXZ+nSpbnhhhty/PjxUQ0JAAxnNgPA2KlUKmP2Xc+qLiAcOXIka9asydy5c0/ed/fdd2fp0qXZuHFjZs2alU2bNo1qSADgNWYzAFCGqgsITU1Nuf/++9Pe3n7yvm3btuXyyy9PkixcuDC9vb2jlxAAGMZsBoCxVRnDr3pWdRPFxsbGNDYO/7GjR4+mqakpSTJ16tT09fWNTjoA4HXMZgCgDH/2VRgGBwdPRQ4A4BQxmwHg1Kr3ZsBYGdFVGJqbm3Ps2LEkyb59+4ZVKAGAsWc2AwCjbUQLCPPmzUtPT0+S5NFHH83FF198SkMBAG+N2QwAjLaqpzA8++yzWbduXfbs2ZPGxsb09PTkS1/6UlauXJnu7u7MnDkzV1999VhkBQBiNgPAWKv3yyuOlaoLCO9+97vz4IMPvu7+b33rW6MSCAAoZjYDAGX4szdRBAAAgL9kNlEcMqI9EAAAAIDTiwYCAAAAFNBAGKKBAAAAAFSlgQAAAAAFXIVhiAYCAAAAUJUGAgAAABSwB8IQDQQAAACgKg0EAAAAKGAPhCEaCAAAAEBVGggAAABQwB4IQyqDg4ODo32Qgy/1j/YhanLo+B/KjpAkef7I3rIjpKmhqewISZKJEyaWHSFJcnbzPys7QpJkz4u/KztCkuSdUy4sO0KS5Oe7f1J2hLRNbCs7QpLkXXXy32RCwxllR0iStP3VtLIjjHt/fPlQ2RGSJC+/+nLZEZIkf/ezW8uOkLsX/ueyIyRJ/n7Pz8uOkCS58twPlR0hSdLQUB+ftzVW6iPH4u//x7IjpG1Sc9kRkiTf+eCdZUdIkrz86vGyIyRJ3j7x7LIjjKp/PPT0mB3rwrP+esyO9VbVxzsRAAAA1CkNhCH2QAAAAACq0kAAAACAAq7CMEQDAQAAAKjKAgIAAABQlVMYAAAAoIBNFIdoIAAAAABVaSAAAABAAQ2EIRoIAAAAQFUaCAAAAFDAZRyHaCAAAAAAVWkgAAAAQCENhEQDAQAAAKiBBgIAAAAUsAfCEA0EAAAAoCoNBAAAAChQsQdCEg0EAAAAoAYaCAAAAFBAA2GIBgIAAABQlQYCAAAAFHAVhiEaCAAAAEBVFhAAAACAqpzCAAAAAAVsojhEAwEAAACoSgMBAAAACmggDNFAAAAAAKrSQAAAAIACLuM4RAMBAAAAqEoDAQAAAArYA2GIBQQAAAAYR9auXZunn346lUolq1atynve856Tjz3++OP58pe/nAkTJuQDH/hAPv3pT7/pc1auXJnt27fnrLPOSpJ84hOfyKWXXvqmx7WAAAAAAAXqaQ+EJ598Mjt37kx3d3eee+65rFq1Kt3d3Scfv/322/PAAw9k+vTpWbZsWRYvXpwDBw686XM+97nPZeHChTUd2wICAAAAjBO9vb3p7OxMkpx33nk5fPhwBgYG0tLSkl27dmXKlCmZMWNGkuSSSy5Jb29vDhw48IbPeatsoggAAAAFKmP4VU1/f39aW1tP3m5ra0tfX1+SpK+vL21tba97rOg5GzZsyIoVK/LZz342Bw4cKDz2mDQQWv/q7WNxmKrqJUfH5HeVHYE69faJZ5cdoa5c9c8/WnYE+Is1+Yyzyo5QV75/5X1lR6gb153/b8uOQB177N9sLDsC8CcGBwdH/JwPf/jDOeuss3LhhRfm61//etavX5/Pf/7zb/o8pzAAAABAofrZA6G9vT39/f0nb+/fvz/Tpk17w8f27duX9vb2nHHGGW/4nI6OjpP3XXbZZVm9enXhsZ3CAAAAAOPE/Pnz09PTkyTZvn172tvb09LSkiQ555xzMjAwkN27d+fEiRPZsmVL5s+f/6bPuf7667Nr164kybZt23L++ecXHlsDAQAAAArUT/8gmTNnTmbPnp2urq5UKpXcdttt2bx5cyZPnpxFixZl9erVuemmm5IkV1xxRTo6OtLR0fG65yTJddddlxtvvDFnnnlmmpub88UvfrHw2JXBkZwwAQAAAKeJ54/8bsyONaP53DE71lvlFAYAAACgKqcwAAAAQIFKpZ5OYiiPBgIAAABQ1bhYQFi7dm2WLFmSrq6uPPPMM2XHKd2dd96ZJUuW5CMf+UgeffTRsuOU7tixY+ns7MzmzZvLjlK6Rx55JFdddVWuueaabN26tew4pXrxxRfzmc98JsuXL09XV1cee+yxsiOVYseOHens7MyGDRuSJM8//3yWL1+epUuX5oYbbsjx48dLTsh4ZTYPZzYPZza/xmx+jdk8xGwerypj+F2/6n4B4cknn8zOnTvT3d2dO+64I3fccUfZkUr1xBNP5Ne//nW6u7vzjW98I2vXri07Uum++tWvZsqUKWXHKN3Bgwdz7733ZuPGjfna176Wn//852VHKtXDDz+cjo6OPPjgg7nrrrtOy/eOI0eOZM2aNZk7d+7J++6+++4sXbo0GzduzKxZs7Jp06YSEzJemc3Dmc2vZzYPMZuHM5vNZsa/ul9A6O3tTWdnZ5LkvPPOy+HDhzMwMFByqvK8733vy1133ZUkedvb3pajR4/mlVdeKTlVeZ577rn85je/yaWXXlp2lNL19vZm7ty5aWlpSXt7e9asWVN2pFK1trbm0KFDSZIXXnghra2tJScae01NTbn//vvT3t5+8r5t27bl8ssvT5IsXLgwvb29ZcVjHDObhzObhzObX2M2D2c2m83jmf7BkLpfQOjv7x/25tLW1pa+vr4SE5VrwoQJaW5uTpJs2rQpH/jABzJhwoSSU5Vn3bp1WblyZdkx6sLu3btz7NixfPKTn8zSpUtP++Fz5ZVXZu/evVm0aFGWLVuWW265pexIY66xsTETJ04cdt/Ro0fT1NSUJJk6depp/X7KyJnNw5nNw5nNrzGbhzObzWbGv3F3FYbBwcGyI9SFn/3sZ9m0aVO++c1vlh2lND/4wQ/y3ve+N+94xzvKjlI3Dh06lPXr12fv3r1ZsWJFtmzZctruGPvDH/4wM2fOzAMPPJBf/epXWbVqlXNx/4T3U04V/y8NMZvN5jdiNr/GbK7O+2k9Oz1/b/9U3S8gtLe3p7+//+Tt/fv3Z9q0aSUmKt9jjz2Wr33ta/nGN76RyZMnlx2nNFu3bs2uXbuydevW/P73v09TU1POPvvszJs3r+xopZg6dWouuuiiNDY25txzz82kSZNy4MCBTJ06texopXjqqaeyYMGCJMkFF1yQ/fv355VXXjmtPxVMkubm5hw7diwTJ07Mvn37hlUooVZm8+uZzUPM5uHM5uHM5jdmNjOe1P0pDPPnz09PT0+SZPv27Wlvb09LS0vJqcrzxz/+MXfeeWfuu+++nHXWWWXHKdVXvvKVPPTQQ/ne976Xj370o/nUpz512v4FJUkWLFiQJ554Iq+++moOHjyYI0eOnJbnFv4/s2bNytNPP50k2bNnTyZNmnTa/wUlSebNm3fyPfXRRx/NxRdfXHIixiOzeTiz+TVm83Bm83Bm8xszm8eHSqUyZt/1rO4bCHPmzMns2bPT1dWVSqWS2267rexIpfrRj36UgwcP5sYbbzx537p16zJz5swSU1EPpk+fnsWLF+djH/tYkuTWW29NQ0PdrxGOmiVLlmTVqlVZtmxZTpw4kdWrV5cdacw9++yzWbduXfbs2ZPGxsb09PTkS1/6UlauXJnu7u7MnDkzV199ddkxGYfM5uHMZt6M2Tyc2Ww2M/5VBp1oAwAAAG9q/7G9Y3as9on1uwB9+i6BAgAAADWr+1MYAAAAoEwVV2FIooEAAAAA1EADAQAAAApoIAzRQAAAAACqsoAAAAAAVGUBAQAAAKjKAgIAAABQlU0UAQAAoEClYhPFRAMBAAAAqIEFBAAAAKAqCwgAAABAVfZAAAAAgAKV2AMh0UAAAAAAaqCBAAAAAIU0EBINBAAAAKAGGggAAABQQP9giAYCAAAAUJUGAgAAABSoVHQQEg0EAAAAoAYaCAAAAFBIAyHRQAAAAABqYAEBAAAAqMopDAAAAFDACQxDNBAAAACAqjQQAAAAoJAOQqKBAAAAANRAAwEAAAAKVCoaCIkGAgAAAFADCwgAAABAVRYQAAAAgKrsgQAAAAAFKq7CkEQDAQAAAKiBBgIAAAAU0kBINBAAAACAGmggAAAAQAH9gyEaCAAAAEBVGggAAABQoFLRQUg0EAAAAIAaWEAAAAAAqnIKAwAAABRyCkOigQAAAADUQAMBAAAACugfDNFAAAAAAKrSQAAAAIBCOgiJBgIAAABQAw0EAAAAKFCpaCAkGggAAABADSwgAAAAAFVZQAAAAACqsgcCAAAAFKi4CkMSDQQAAACgBpXBwcHBskMAAAAA9U0DAQAAAKjKAgIAAABQlQUEAAAAoCoLCAAAAEBVFhAAAACAqiwgAAAAAFX9XwJoDzC3JyFEAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "pos = 'PRON'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=pos)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=pos)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to PRON tags')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.4 Target negations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 51,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [00:34<00:00, 28.66it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAsAAAJ1CAYAAAC/wnk8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs/XmYVOWdN/6/uxsQtBEBaVSQiEtjRoNiBKPwuCAIkphExgV382RGUWNccCUqGpfE9VHRGR2XaBKNKGLcxThC3FAGM2MMGZOIiuAGKCIIgkD9/vBL/6wIRYMNhanXK1ddV1fVqXO/6zTm7vrU59ynqlAoFAIAAADw/6kudwAAAABg3aJYAAAAABRRLAAAAACKKBYAAAAARRQLAAAAgCKKBQAAAEARxQIAqDAjR45Mt27d8sILL5Q7CmvAEUcckW7dupU7BgBfcYoFALCKLr300nTr1i0777xzPvnkkxVuN27cuDzxxBNFj3388ccZOXJkPvroozUdM0ny17/+NbfddlvRY/vuu2+uueaabLPNNmslw/Is79g0hb59+6Zbt24599xzV7jNmDFjvlAsWVZAWdlt5513Xu4+J0yYkNNPPz177713dthhh2y//fbZfffdM3To0DzwwAMpFAqNfg/333+/Qg4AZdes3AEA4Ktk0aJFue+++1JdXZ25c+fmsccey/e///3lbnvLLbekc+fO6devX8NjL7/8cq677rrsv//+2XDDDdd43sceeyy//e1vc/TRRzc8tvXWW2frrbde42OXsrxj05TuueeeHHDAAdlhhx1W6XVHHXVUdtpppxU+37x586L7CxcuzPDhw/PQQw9liy22yHe/+9187Wtfy6JFi/L666/n4Ycfzrhx4zJ27NhcccUVadWq1UozXH311Rk8eHB22WWXVcoOAE1JsQAAVsHjjz+e2bNn59BDD81vfvOb3H333cstFixdujSTJ09O586dix5/+eWX11bUsozXGCs6Nk2lvr4+M2fOzPnnn5/Ro0enpqam0a/dYYcdMnDgwEZv/9Of/jQPPfRQDjvssAwfPjzNmhX/aXXSSSdl2LBheeKJJ3LllVfmnHPOKbm/999/P2+//XajxweANcVpCACwCkaNGpXks2+gv/nNb+bFF1/MlClTirYZM2ZMvv71r2f+/Pm577770q1bt4wcOTJ9+/bNFVdckSTZe++9v3Be+ejRoxu+De/Ro0f233///OpXv8rSpUsbtpk+fXq6deuWn/zkJ/nb3/6Wf/mXf8nOO++c7t2757DDDsuf/vSnou2eeuqpvPXWW+nWrVuOOOKIJCtes+Cpp57KUUcdlZ133jnbb799+vbtm4suuigffPBB0XZ9+/ZN//79M3fu3Jxzzjnp3bt3tt9+++y777558MEHSx6/FR2bZaZMmZJTTz01vXv3znbbbZfddtstP/7xj/OXv/xlpb+bZTbaaKMMGzYsf/7zn/PrX/+60a9bVa+88kpGjx6dHXfcMeecc84XCgVJ0rJly1x66aUZNmxYDjjggJL7GzlyZHbbbbckyXXXXZdu3bplzJgxDc+/9NJLGTp0aHbZZZeG0xzOPvvsTJ8+faVZFy1alEMPPTTbb799nn322YbHp0+fnrPPPjt9+vTJ9ttvn969e2fYsGFf+Dd91llnpVu3bnnnnXfyH//xH+nXr1+23377/J//839y1VVXZcmSJQ3bFgqF3HvvvTnooIOyyy67ZIcddkj//v1zySWXZM6cOSvNCsC6QWcBADTS66+/nokTJ6ZHjx7ZYost8v3vfz+TJk3KPffck7POOqthu1122SUjRozIBRdckF69euWwww7L1ltvne7du+fmm2/OxIkTM2LEiLRr167hNT//+c/zi1/8InvvvXcOPvjgLF68OOPGjctFF12UV155JRdffHFRlhkzZuQHP/hBvv3tb+fb3/52/vrXv+aXv/xlhg4dmieffDLt27fPNddckwsuuCBJvjDe37vvvvty9tlnZ8stt8zQoUPTrl27TJ48Ob/5zW/yzDPPZMyYMVl//fUbtl+6dGmOOeaYdOjQIaeccko+/PDD3HLLLTnjjDOy5ZZbZrvttlvuOCs6Nsln6ysccsghadasWYYMGZKuXbtm+vTpueOOOzJkyJDceeed+frXv96o39UBBxyQe++9N9dee2323Xff1NXVNep1q+KBBx5Ikhx99NGprl7x9y+1tbU55phjVrq/fffdN1VVVRk5cmQGDhyYfffdN9/4xjeSfLYmwr/+67+mffv2Oeqoo7LZZpvl1VdfzR133JHf//73ue+++9KxY8fl7rdQKOTMM8/MH/7wh1x55ZXp3bt3kmTatGk58MADG453586d8+abb+aOO+7I+PHjc9ddd31hXYv/9//+X6ZMmZKjjz46zZo1y1133ZUbb7yxIVeS3HTTTbnyyivTp0+fnHbaaWnRokX+/Oc/54477sikSZNy7733pqqqauUHGIDyKgAAjfLzn/+8UF9fX7j77rsLhUKhMHfu3MIOO+xQ2GWXXQoLFy4s2nbatGmF+vr6wplnnln0+Jlnnlmor68vTJs2reGx//3f/y3U19cXzj///C+MeeKJJxbq6+sLkydPLtpvfX194ZFHHina9uyzzy7U19cXnnvuuYbH9tprr8Jee+1VtN21115bqK+vLzz//POFQqFQWLBgQaFnz56FXXfdtTBnzpyibW+66aZCfX194cYbbyza5/Ly3nfffYX6+vrC1VdfvZyjt/Jj88Mf/rBQX19f+MMf/lD0+EsvvVSor68v/PCHPyy532XZDj/88EKh8Nlx/frXv1445ZRTira59957i95/ofD/Pyb33HNPYc6cOSu8ff73fNhhhxXq6+sLM2bMWGmuxnr++ecL9fX1hWuvvbbo8YEDBxa+8Y1vFN58882ixx955JFCfX194bzzzmt47PDDDy/U19c33L/88ssL9fX1hdtvv73otSeccEKhR48ehalTpxY9vuy4DR06tOGxZf9uv/e97xUdg+nTpxe6detWOPTQQxse22+//Qo77bRTYfHixUX7vffeewvHHnts4a233mrs4QCgjJyGAACNsGxhw1atWmXfffdN8tk3xvvss09mz579pVb2f/TRR5MkgwYNykcffVR0GzBgQJJk4sSJRa/ZZJNNGnIss+xb6JkzZ67S+BMnTsycOXMyaNCgLyy6OHjw4CTJ+PHjv/C6H/zgB00yfpLMnz8/zz77bLp165YePXoUPde9e/fU19dnwoQJWbhwYaP3ue222+awww7Lww8/nOeee65Rr/nJT36Snj17rvD20EMPNWw7a9asVFdXZ+ONN/7CfhYsWPCF3+VHH32UxYsXNzr/MlOmTMlrr72W3r17Z/PNNy96bp999knr1q2X+/tJPjtt5qabbsqxxx6bI488sijf+PHj881vfjMbbbRRUcbNNtss22yzzRf+zSXJYYcdlhYtWjTc79SpU9q3b1/0O2/WrFnmz5+fV155pei1gwcPzg033JDNNttslY8BAGuf0xAAoBGWLWz43e9+N7W1tQ2PDx48OPfff3/uvvvuDBo0aLX2/eqrryZJDj/88BVu8/eL3nXp0uUL26y33npJssofSF977bUkny0M+PfatWuXjTbaKG+88UbR4zU1NenUqVOTjJ8kU6dOzdKlS1d4OceuXbvmr3/9a6ZPn56tttqq0fs96aST8uijj+aCCy7Igw8+WPRBd3mOP/74fOtb31rh81tuuWXDzzU1NSkUCikUCl9oqz/33HOXu37DL3/5y1W+ysHrr7+eZPm/n5qamnTp0iWTJ0/OJ598kpYtWzY899RTT+WnP/1pBgwYkFNPPbXodVOnTs2nn36ap556Kj179lzh2HPnzk3r1q0b7q/o393nf+fHHXdcTjrppBx44IHZdddd07t37/Tu3fsLa3QAsG5TLACARli2sGGvXr0yderUhsc32WSTbLzxxnn++eczbdq0L3zz2xgff/xxkuSqq65a7rfUSdKhQ4ei+yv70Lsq5s+fnyQrvKxfy5Yt89FHHxU9VlNTs0pXGViZZcfg8+sifN6yQsSyrI1VW1ubs846K8OGDcvNN9+c448/vuT2W2+9daM/zNfV1eXVV1/NW2+99YXf+3HHHZcDDzyw4f59992X++67b5WyL7Ps2JT6/SSfHZvPFwtOPvnkLF68OC+99FLmzJmTNm3aNDw3b968JEmfPn1Krqew7Liv6P7y9O/fP3fddVduvfXWPPXUU3nmmWeSfFbsOPfcc9OrV6+V7gOA8lMsAICVeO211xpasktd+m706NE55ZRTVnn/G2ywQZJk8803T/fu3Vcv5Jew7AP6ij6IL1iwoCHjmrJs/6UyfH67VfGd73wno0ePzo033pj99ttv9UP+nW9+85t57rnn8uyzz2bIkCFFz2211VZFHRDLa+lvrNU9NnvttVf69OmTs846K+eee26uvfbahueWdcdUV1evcqdDY3Tv3j1XX311Pv300/zP//xPHnvssYwaNSo//OEP8/DDDy+3QwGAdYs1CwBgJe6+++4kyYEHHphrrrnmC7fLLrssNTU1uffee1erBX/Z1QD+8Ic/fOG5jz/+eJXO018dn78awd+bOXNm5syZU9R+vyZsscUWqampWW6G5LNTNVq0aJHOnTuv1v7PO++8LFmyJBdddNGXiVlk//33T01NTW699dZ88sknTbbfv7es6LC8Y7N48eJMnTo1nTt3/sK3/ldeeWX233//fP/738/YsWMbumOSz4538+bN8/LLL+fTTz/9wn7//nKZq6t58+bp2bNnzj333JxxxhlZtGhRxo0b1yT7BmDNUiwAgBKWLWzYokWLnHrqqRk4cOAXbt/73vfSr1+/zJw5s2GhuWUt+n//QX/ZJfY+//iyhQp/85vffOFD5+WXX55vfetbefPNN1crf3V19UqLDb169Uq7du3yyCOPZM6cOUXPLfuAuWyhxaawvGPTqlWr7LHHHvnrX/+aSZMmFW0/ceLEvP7669lrr71W+/SLLbfcMv/3//7fjB8/Pr/73e9WP/zndOrUKccee2ymTp2ak08+uaG1/+9NmDCh4Th+/jSB5VnesenatWu6deuWZ599NtOmTSva/sEHH8zHH3+cffbZZ4X7PPfcc7P55pvnZz/7WaZMmdKQY88998zs2bPz29/+tmj7adOmpW/fvhkxYkTJrMvz3nvvZb/99svVV1/9heeWdTM05Sk0AKw5TkMAgBLGjh2bDz/8MIMHD067du1WuN3hhx+esWPH5p577km/fv2y8cYbp2XLlnn66adz44035mtf+1oGDhzY8M345Zdfnp49e+Z73/tett122xx11FG5/fbbc8ghh+Tggw9Os2bN8vvf/z6PP/54vvvd765223bnzp0zYcKE/OxnP8umm26ao48++gvbtGjRIueee25OPfXUHHbYYTnggAPSunXrvPTSS7nnnnuy4447Fp1//2Wt6NicccYZmTRpUk444YQcccQR6dy5c954443ceeedadu2bU4//fQvNe5xxx2Xhx56KE8++eQKt3nppZdWuhZDjx490rFjxyTJiSeemIULF+bWW2/NgAEDst9++6W+vj7V1dV555138vvf/z7//d//nbZt2+byyy/PDjvsUHLfnTp1SlVVVR588MG0bds23bp1S58+fXLeeeflBz/4QY488sgccsgh6dChQ/7yl7/kzjvvTJcuXTJ06NAV7rO2tjaXX355DjvssAwbNix33313WrRo0XC8L7jggrz22mv5+te/nrfeeit33HFHqqqqcvDBB5fMujwdO3bMpptumhtuuCFvv/12evXqlfXWWy+vv/56fvWrX6VDhw5NWngCYM1RLACAEpZ9I3zUUUeV3K5Xr16pr6/P008/nXfffTebbLJJzjrrrFx99dX5t3/7txx00EEZOHBghgwZkmeeeSbPPPNM/vSnPzV8Izx8+PBss802GTVqVH72s59l6dKl2WKLLXL66acv9wN+Y5100kkNHwC7deu2wn0NGjQobdq0yY033phrr702ixYtSqdOnXLMMcfk2GOPbdJvg5s3b77cY9O1a9fcfffdufbaa3PnnXdmzpw5adu2bfr27ZsTTjhhtRaP/LxWrVrlJz/5SclFDm+//fbcfvvtJfdz/fXXNxQLqqurc8YZZ+Q73/lO7rrrrowbNy6jRo3K0qVL065du9TX12fEiBH53ve+16j1FjbddNMMHTo0v/rVr3LdddflxBNPTJ8+fbLzzjvnzjvvzHXXXZebb7458+fPT11dXQ466KAcf/zxRYsXLk+PHj1y/PHHZ+TIkbniiisyfPjwdOnSJffcc0+uv/76PPjgg/nlL3+Z1q1bp1evXjn++OOz7bbbrjTv8lx33XW56aab8thjj+WJJ57IwoUL07FjxwwcODDHH398yaIbAOuOqkKhUCh3CAAAAGDdYc0CAAAAoIhiAQAAAFBEsQAAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIooFAAAAQBHFAgAAAKCIYgEAAABQRLEAAAAAKKJYAAAAABRRLAAAAACKKBYAAAAARRQLAAAAgCKKBQAAAEARxQIAAACgiGIBAAAAUESxAAAAACiiWAAAAAAUUSwAAAAAiigWAAAAAEUUCwAAAIAiigUAAABAEcUCAAAAoIhiAQAAAFBEsQAAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIooFAAAAQBHFAgAAAKCIYgEAAABQRLEAAAAAKKJYAAAAABRRLAAAAACKKBYAAAAARRQLAAAAgCKKBQAAAEARxQIAAACgiGIBAAAAUESxAAAAACiiWAAAAAAUUSwAAAAAiigWAAAAAEUUCwAAAIAiigUAAABAEcUCAAAAoIhiAQAAAFBEsQAAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIs3KHQAAAADWZVX9O6+1sQq/m77WxipFZwEAAABQRLEAAAAAKOI0BAAAACilqqrcCdY6nQUAAABAEZ0FAAAAUEoFfs1egW8ZAAAAKEVnAQAAAJRizQIAAACg0uksAAAAgFIqr7FAZwEAAABQTGcBAAAAlLKOrVlwySWX5KWXXkpVVVWGDx+e7t27Nzz33HPP5aqrrkpNTU123333nHDCCbnnnnvywAMPNGzzpz/9Kf/93/9dcgzFAgAAAPiKmDhxYqZOnZpRo0ZlypQpGT58eEaNGtXw/EUXXZRbbrklHTt2zOGHH54BAwbkwAMPzIEHHtjw+kcffXSl4zgNAQAAAEqpXou3lZgwYUL69euXJNlqq60yZ86czJs3L0kybdq0tGnTJptuummqq6uzxx57ZMKECUWvv/7663P88cc36i0DAAAAXwGzZs1K27ZtG+63a9cuM2fOTJLMnDkz7dq1W+5zSfLHP/4xm266aTp06LDScRQLAAAA4CuqUCg0etvRo0dn//33b9S2igUAAABQSlXV2rutRF1dXWbNmtVwf8aMGQ2dAn//3HvvvZe6urqG+y+88EJ69OjRqLesWAAAAABfEb17987YsWOTJJMnT05dXV1qa2uTJJ07d868efMyffr0LF68OOPGjUvv3r2TfFY42GCDDdKiRYtGjeNqCAAAAFDKOnTlxJ122inbbbddhgwZkqqqqowYMSJjxoxJ69at079//5x//vkZNmxYkmTQoEHp2rVrki+uZ7AyVYVVOcEBAAAAKkzV97ZYa2MV7n9jrY1Vis4CAAAAKKV6HWotWEusWQAAAAAU0VkAAAAApVReY4HOAgAAAKCYzgIAAAAoparyWgt0FgAAAABFdBYAAABAKZXXWKCzAAAAACimswAAAABKqa681gKdBQAAAEARnQUAAABQSuU1FugsAAAAAIopFgAAAABFnIYAAAAApVRV3nkIOgsAAACAIjoLAAAAoBSXTgQAAAAqnc4CAAAAKKXyGgt0FgAAAADFdBYAAABAKa6GAAAAAFQ6nQUAAABQSuU1FugsAAAAAIrpLAAAAIBSqiuvtUBnAQAAAFBEZwEAAACUUnmNBToLAAAAgGI6CwAAAKCUqsprLdBZAAAAABRRLAAAAACKOA0BAAAASqnAr9kr8C0DAAAApegsAAAAgFIscAgAAABUOp0FAAAAUErlNRboLIDl6datW/r375+BAwdmwIAB+Zd/+ZdMmzat3LFWS//+/fPCCy+U3Oass87Kv/3bv62lRABUqs/Pr8tuP/zhD5MkRx11VCZPntxkYz399NN5++23m2x/pfzkJz/JyJEj18pY//RP/5Tp06eX3OaII47I/fffv1byAP+4dBbACvzqV7/KJptskiS58sorc/HFF+eGG24ocyoA+Gr7/Pz6ebfffnuTjnPbbbfluOOOy2abbdak+wUqVAWuWaBYAI3wrW99K08++WTD/XvuuSe33nprlixZkg4dOuSyyy5Lp06d8t577+WMM87IzJkzs2jRonz729/OKaeckkKhkOuvvz4PPvhgFi1alL333jtnn312ampqisYZM2ZMxo8fn+bNm+fFF19M165dc8IJJ+SKK67Im2++mZNOOikHH3xwli5dmmuuuSZjx45Nkuy4444577zzsv766+dPf/pTzjzzzCxevDh77LFH0f6feOKJXHPNNZk/f36+9rWv5Yorrki7du3W/AEEgJXo27dvLrvssmyyySYZMmRIjjnmmNxzzz358MMPc/bZZ2fQoEGNnk+vvvrqPP/883nttddy+umn56mnnkqXLl1y/PHHJ/mso27Z/b59++aYY47J6NGj8+677+Y73/lOzjrrrCQrnjdnz56dYcOG5Y033sjWW2+dli1bLrcAMnLkyMyaNSvvvvtuJk+enF133TWDBg3KyJEjM2PGjFx44YXZa6+9snDhwlx88cV54YUXUl1dnT322COnn356ampq8vvf/z4XXXRRmjVrln/+538u2v+oUaPyi1/8IosWLcqOO+6YSy65JC1btlxDvyGg0jgNAVZi0aJFeeCBB9K3b98kyfvvv5+f/vSn+cUvfpHHH388Xbp0aWjhv+2229KzZ8888sgjefDBBzNt2rTMmDEj999/fx577LGMHj06v/vd7zJt2rT85je/We54Tz/9dH70ox/l8ccfz5QpU3LzzTfnjjvuyMUXX9wwzqOPPpqnnnoqY8aMycMPP5yPPvoot912W5Lk/PPPz5FHHpmxY8emR48eDa2K06ZNyxlnnJErr7wy//mf/5lddtkl559//po9eACwGmbPnp3q6uo8+OCDGT58eK6++uokafR8evLJJ6djx465/PLLM2jQoJWO91//9V8ZNWpU7r333vz617/Ou+++W3LevOmmm9K2bds8+eSTOe+88/LMM8+scN/jx4/PJZdckgcffDCPPfZYw/w9dOjQ3HTTTUk+66p499138/DDD+e+++7LpEmT8tBDD2XJkiX5yU9+khEjRuTRRx9NdXV1lixZkiSZNGlSrrnmmtx+++158sknU1tbm2uuuWZVDzXQWNVr8baOWIeiwLrliCOOyMCBA9O7d++8/PLLGTx4cJKkffv2efHFFxu+Qdh5550b1jNo3759nnnmmUyaNCktWrTIVVddlbq6uowbNy7//M//nNatW6dZs2Y58MAD8/jjjy933K233jpdu3ZNixYt8rWvfS19+vRJTU1N6uvrM2PGjCSf/eHx/e9/P+uvv35qamoyePDgPPvss1m4cGFefvnlhj+MBg4cmFatWiVJnnrqqfTq1Sv19fVJkiFDhuTJJ59s+KMDANaGZfPrsts555zzhW0WL17cMO9ut912DWsPrMp8uir222+/1NTUpGPHjmnfvn3eeeedkvPmpEmTsu+++yZJOnfunF69eq1w3z169Ej79u3Ttm3bdOjQIbvvvnuSfGFeP+igg9KsWbO0bNky++23X5599tm88cYbWbRoUfr06ZMk2X///Rv2++STT2bQoEHp2LFjkuSQQw5pkmMBsIzTEGAFPn9O5X/913/liCOOyJgxY9K+fftce+21DX8wfPzxx+natWuS5Oijj87SpUtzwQUXZMaMGTnssMNy4oknZu7cubnlllsyatSoJMmSJUtW2P6/wQYbNPxcU1OT9ddfv+HnpUuXJkk++OCDtGnTpmG7Nm3a5P3338+HH36YJKmtrU2SVFVVZcMNN0ySzJ07N5MmTcrAgQMbXldbW9vwGgBYG1a0ZsHnfX7+q66ubpj/VjSfvvfeeznqqKOSJN27d89ll122SpmWzZvLxl6yZEnJeXPOnDlp3bp1w+PL5trlWdG8/vn3taJ5fc6cOUXZPr/N3Llz87vf/a6hq6FQKOTTTz9dpfcNrAJrFgDL07Nnz2y22WZ58cUXs3jx4jz55JP59a9/nXbt2uXuu+/Ogw8+mCRp1qxZjjnmmBxzzDF5/fXX86//+q/55je/mbq6uvTt2zeHH354k+TZeOONiz7kf/jhh9l4440b/oiYN29eWrdunaVLl2bOnDlJkrq6uuy222659tprmyQDAKxtpebTxx57bIWv+/wH8yQNc+PKxlrRvLnhhhtm7ty5Dfc/+OCDbL755ivd54qUmtfnzZtXNM7n8+2///4588wzV3tcgFKchgCN8Prrr+f111/Plltumffffz+dOnVqWODo0Ucfzccff5wkOe+88/Lss88mSbp06ZKNN944VVVV2XvvvXP//fdnwYIFSZK77ror991332rn2XPPPfPAAw9kwYIFWbx4cUaPHp099tgjLVu2zLbbbpvf/e53SZKHH344CxcuTJL06dMnkyZNajhl4o9//GMuuuii1c4AAGvbqsynzZo1a/hA36FDh7zyyitJPlvD5w9/+MNKxyo1b+6444554oknkiRvvvlmXnzxxS/1vvbcc8+MHj06S5Ysyfz583P//fdnjz32SJcuXVJTU9NwCeQxY8ak6v/7drNv3755/PHHGwoITzzxRP7jP/7jS+UASqhai7d1hM4CWIEjjjiiYXXlFi1a5IILLki3bt3Svn37PPzww+nfv38233zznHzyyTnuuOPy85//PEOGDMl5552XCy+8MIVCIX379s2uu+4uvJ8LAAAgAElEQVSaJPnb3/7WcK5hly5dcvHFF692toEDB+Yvf/lLBg8enEKhkF122SVHHnlkks8WOBw+fHhuvPHG7L777tlqq62SfPYNxIUXXpgTTjghn376aTbYYIMMHz78yxwiAFir+vXr1+j5dMCAATn11FPz4x//OAcddFB+9KMfZZ999sk//dM/ZcCAASsdq9S8eeyxx+aUU05J3759s9VWW2Wfffb5Uu/riCOOyLRp0/Ltb387VVVVGThwYPbdd99UVVXlwgsvzPDhw9OiRYsMHjy44TSG7bbbLkOHDs0RRxyRpUuXpn379rngggu+VA6Az6sqFAqFcocAAACAdVXVj7Zfa2MVrvvTWhurFKchAAAAAEUUCwAAAIAi1iwAAACAUirw0ok6CwAAAIAia6Wz4PfvPL42hlmpdWUtxzYt2pQ7Qv7y4V/LHSFJUteqQ7kjJElmLJhZ7ghJkm3bdit3hCTJnz/433JHSJJss9HW5Y6QeZ/OW/lGa8HshR+ufKO1YMHiBeWOkCQ5bJsflDvCV96fZ/9PuSMkSWYseK/cEZIk27Qp/////u+Hfy53hCRJq5qW5Y6QJFmw5JNyR0iSbNyyfbkjJEnenf9uuSMkSTZZf5NyR8h668i/0Tc+er3cEZIkGzSvLXeEJMnum/Yrd4Q1q/IaC3QWAAAAAMWsWQAAAAAlVFmzAAAAAKh0OgsAAACgBJ0FAAAAQMXTWQAAAAAlVGBjgc4CAAAAoJjOAgAAACihugJbC3QWAAAAAEV0FgAAAEAJroYAAAAAVDydBQAAAFCCzgIAAACg4q12Z8Ell1ySl156KVVVVRk+fHi6d+/elLkAgFVkbgYAmspqFQsmTpyYqVOnZtSoUZkyZUqGDx+eUaNGNXU2AKCRzM0AsOY4DaGRJkyYkH79+iVJttpqq8yZMyfz5s1r0mAAQOOZmwGAprRaxYJZs2albdu2DffbtWuXmTNnNlkoAGDVmJsBYM2pqlp7t3VFkyxwWCgUmmI3AEATMTcDAF/Gaq1ZUFdXl1mzZjXcnzFjRjp06NBkoQCAVWNuBoA1x5oFjdS7d++MHTs2STJ58uTU1dWltra2SYMBAI1nbgYAmtJqdRbstNNO2W677TJkyJBUVVVlxIgRTZ0LAFgF5mYAWHMqsbNgtYoFSXLaaac1ZQ4A4EsyNwNAZbjkkkvy0ksvpaqqKsOHD0/37t0bnnvuuedy1VVXpaamJrvvvntOOOGEJMkDDzyQm2++Oc2aNcuPf/zj7LnnniXHWO1iAQAAAFSCqqw7nQUTJ07M1KlTM2rUqEyZMiXDhw/PqFGjGp6/6KKLcsstt6Rjx445/PDDM2DAgLRv3z7XX3997r333syfPz8jR45ULAAAAIB/FBMmTEi/fv2SJFtttVXmzJmTefPmpba2NtOmTUubNm2y6aabJkn22GOPTJgwIe3bt8+uu+6a2tra1NbW5sILL1zpOE1y6UQAAAD4R1VVVbXWbisza9astG3btuF+u3btMnPmzCTJzJkz065duy88N3369HzyyScZOnRoDj300EyYMGGl4+gsAAAAgK+oQqHQqO0+/PDDXHfddXn77bdz5JFHZty4cSWLE4oFAAAAUMK6dDGEurq6zJo1q+H+jBkz0qFDh+U+995776Wuri6tWrVKjx490qxZs3Tp0iUbbLBBPvjgg7Rv336F4zgNAQAAAL4ievfunbFjxyZJJk+enLq6utTW1iZJOnfunHnz5mX69OlZvHhxxo0bl969e6dPnz55/vnns3Tp0syePTvz588vOpVheXQWAAAAQAnV61BrwU477ZTtttsuQ4YMSVVVVUaMGJExY8akdevW6d+/f84///wMGzYsSTJo0KB07do1STJgwIAcdNBBSZJzzjkn1dWlewcUCwAAAOAr5LTTTiu6v+222zb83LNnz6JLKS4zZMiQDBkypNFjKBYAAABACY25SsE/GmsWAAAAAEUUCwAAAIAia+U0hK9vtP3aGGalXp/7arkjJEnqWm1S7gipbd663BGSJO/Mf6fcEZIke2y2Z7kjJEnarrdxuSMkSTZdv3O5IyRJ5n36UbkjZPPaLcodIUny0aIPyx0hSTLv07nljkATabteu3JHSJJM/3hauSMkST5ePK/cEbJg8fxyR0iStKxZr9wRkiStm9eWO0KSZP1mG5Q7QpJksw02K3eEJI2/nvua1Ly6ebkjJEk6tOpQ7ghJkikfvVbuCBXBaQgAAABAxbPAIQAAAJRQgY0FOgsAAACAYjoLAAAAoARrFgAAAAAVT2cBAAAAlKCzAAAAAKh4OgsAAACgBJ0FAAAAQMXTWQAAAAAl6CwAAAAAKp7OAgAAACihAhsLdBYAAAAAxXQWAAAAQAnWLAAAAAAqnmIBAAAAUMRpCAAAAFCC0xAAAACAiqezAAAAAEqo1lkAAAAAVDqdBQAAAFBCBTYW6CwAAAAAiuksAAAAgBJcDQEAAACoeDoLAAAAoISq6CwAAAAAKpzOAgAAACjBmgUAAABAxdNZAAAAACXoLAAAAAAqns4CAAAAKKECGwt0FgAAAADFFAsAAACAImvlNIT/njVpbQyzUl1af63cEZIkbVq0LXeENK9uUe4ISZL2LevKHSFJsnDJgnJHSJJMnv0/5Y6QJOnYapNyR0iSbNSiXbkjZNLMF8odIUnSY+Odyx0hSVKVCuzB+wf1v7MnlztCkmTT9Tctd4QkScdWm5U7QtZvtkG5IyRJWtWsX+4ISZKFSz8pd4Qkyb1TxpQ7QpLku12/U+4ISZI268DcPHHGc+WOkCTZru03yh0hSdKypmW5I1QECxwCAAAAFc8ChwAAAFCCzgIAAACg4uksAAAAgBJ0FgAAAAAVT2cBAAAAlFCBjQU6CwAAAIBiOgsAAACgBGsWAAAAABVPZwEAAACUoLMAAAAAqHg6CwAAAKAEnQUAAABAxVvtzoLLLrssL774YhYvXpxjjz02++yzT1PmAgBWkbkZANaMCmwsWL1iwfPPP5+//e1vGTVqVGbPnp3999/fHyQAUEbmZgCgKa1WsaBnz57p3r17kmTDDTfMggULsmTJktTU1DRpOACgcczNAEBTWq1iQU1NTdZff/0kyejRo7P77rv7YwQAysjcDABrTiUucPilrobwxBNPZPTo0bn11lubKg8A8CWYmwGAprDaxYKnn346N9xwQ26++ea0bt26KTMBAKvB3AwAa4bOgkaaO3duLrvsstx2223ZaKONmjoTALCKzM0AQFNarWLBI488ktmzZ+fkk09ueOzSSy/NZptt1mTBAIDGMzcDwJqjs6CRDj744Bx88MFNnQUAWE3mZgCgKX2pBQ4BAADgH10FNhakutwBAAAAgHWLzgIAAAAooRLXLNBZAAAAABTRWQAAAACl6CwAAAAAKp3OAgAAACjBmgUAAABAxdNZAAAAACVUYGOBzgIAAACg2FrpLPhmh15rY5iVWrz003JHSJJUV9WUO0IWLf2k3BGSJDXV60Zzy6SZE8sdIUmy8zry38qSwuJyR0iSfLp0UbkjpOuGW5Y7wjqlRU2LckegiWy54VbljpAkWVJYUu4ISZL1alqWO0KmzptS7ghJki1b15c7QpLkoakPlDtCkmSvTruXO0KSZO6ij8odIUlS27xNuSOkY6uO5Y6QJGlevW7MievCZwvWvksuuSQvvfRSqqqqMnz48HTv3r3hueeeey5XXXVVampqsvvuu+eEE07ICy+8kJNOOinbbLNNkqS+vj7nnntuyTHWjU9qAAAAsI5alxY4nDhxYqZOnZpRo0ZlypQpGT58eEaNGtXw/EUXXZRbbrklHTt2zOGHH54BAwYkSXr16pVrr7220eM4DQEAAAC+IiZMmJB+/folSbbaaqvMmTMn8+bNS5JMmzYtbdq0yaabbprq6urssccemTBhwmqNo1gAAAAAJVRVVa2128rMmjUrbdu2bbjfrl27zJw5M0kyc+bMtGvXbrnPvfrqqxk6dGgOOeSQPPvssysdx2kIAAAA8BVVKBRWus0WW2yRH/3oR9l3330zbdq0HHnkkXn88cfTosWK197QWQAAAAAlrEudBXV1dZk1a1bD/RkzZqRDhw7Lfe69995LXV1dOnbsmEGDBqWqqipdunTJxhtvnPfee6/kOIoFAAAA8BXRu3fvjB07NkkyefLk1NXVpba2NknSuXPnzJs3L9OnT8/ixYszbty49O7dOw888EBuueWWJJ+dqvD++++nY8fSVxZxGgIAAACUsA5dDCE77bRTtttuuwwZMiRVVVUZMWJExowZk9atW6d///45//zzM2zYsCTJoEGD0rVr13To0CGnnXZa/vM//zOffvppzj///JKnICSKBQAAAPCVctpppxXd33bbbRt+7tmzZ9GlFJOktrY2N9xwwyqNoVgAAAAAJTRmLYF/NNYsAAAAAIroLAAAAIASdBYAAAAAFU9nAQAAAJSgswAAAACoeDoLAAAAoASdBQAAAEDFUywAAAAAijgNAQAAAEqowLMQdBYAAAAAxXQWAAAAQAkWOAQAAAAqns4CAAAAKEFnAQAAAFDxdBYAAABACToLAAAAgIqnswAAAABKqMDGAp0FAAAAQDGdBQAAAFCCNQsAAACAiqezAAAAAErRWQAAAABUOp0FAAAAUII1CwAAAICKt1Y6C6qyblRh3po/rdwRkiTrN29d7ghp3XyjckdIksxY8E65IyRJtm/3jXJHSJIsXvppuSMkSRYtXVTuCEmS2pZtyh0hhRTKHSFJUsjSckdIkrSsaVXuCDSRdut1KHeEJMkrc/5U7ghJkrbrtS93hHTZoGu5IyRZd+bmAZsPLHeEJOvO/+/97aNXyh0hybrxt8om63cqd4QkyZLC4nJHSJK0a7lxuSPwD8ppCAAAAFBC9brx/fda5TQEAAAAoIjOAgAAACjBAocAAABAxdNZAAAAACVU6ywAAAAAKp3OAgAAACjBmgUAAABAxdNZAAAAACVU4rfslfieAQAAgBJ0FgAAAEAJroYAAAAAVDydBQAAAFCCqyGsok8++ST9+vXLmDFjmioPAPAlmJsBgKbwpToL/v3f/z1t2rRpqiwAwJdkbgaApmfNglUwZcqUvPrqq9lzzz2bMA4AsLrMzQBAU1ntYsGll16as846qymzAABfgrkZAGgqq3Uawm9/+9vsuOOO2XzzzZs6DwCwGszNALDmVOICh6tVLBg/fnymTZuW8ePH5913302LFi2yySabZLfddmvqfABAI5ibAYCmtFrFgquvvrrh55EjR6ZTp07+GAGAMjI3A8Ca86UuI/gVVYnvGQAAACjhS106MUlOPPHEpsgBADQRczMANC2XTgQAAAAq3pfuLAAAAIB/ZJV4NQSdBQAAAEARnQUAAABQgjULAAAAgIqnswAAAABKqLy+Ap0FAAAAwN/RWQAAAAAlWLMAAAAAqHg6CwAAAKAEnQUAAABAxVsrnQWfLFmwNoZZqW5tti93hCTJp0sXlTtCCimUO0KSpLZ563JHSJKsV9Oy3BGSJEsLS8sdIUny1kfTyx0hSbJBs9pyR8j0j98sd4QkSbv12pc7QpKkVbP1yx2BJvLBwpnljpAk2bL1NuWOkCRZUlhS7gipqlo3vsNp3XzDckdYp3y8eF65IyRJ5iz6qNwRkiQffzq33BGyuLC43BGSJIvWgb/pk2Tj9TqUOwL/oJyGAAAAACVUOQ0BAAAAqHQ6CwAAAKAECxwCAAAAFU9nAQAAAJRQeX0FOgsAAACAv6OzAAAAAEqwZgEAAABQ8XQWAAAAQAk6CwAAAICKp7MAAAAASqjSWQAAAACsyy655JIcfPDBGTJkSP74xz8WPffcc8/lgAMOyMEHH5zrr7++6LlPPvkk/fr1y5gxY1Y6hs4CAAAAKGFdWrNg4sSJmTp1akaNGpUpU6Zk+PDhGTVqVMPzF110UW655ZZ07Ngxhx9+eAYMGJCtt946SfLv//7vadOmTaPG0VkAAAAAXxETJkxIv379kiRbbbVV5syZk3nz5iVJpk2bljZt2mTTTTdNdXV19thjj0yYMCFJMmXKlLz66qvZc889GzWOYgEAAACUULUWbysza9astG3btuF+u3btMnPmzCTJzJkz065du+U+d+mll+ass85q9HtWLAAAAICvqEKhsNJtfvvb32bHHXfM5ptv3uj9WrMAAAAAviLq6uoya9ashvszZsxIhw4dlvvce++9l7q6uowfPz7Tpk3L+PHj8+6776ZFixbZZJNNsttuu61wHMUCAAAAKGFdWuCwd+/eGTlyZIYMGZLJkyenrq4utbW1SZLOnTtn3rx5mT59ejbZZJOMGzcuV1xxRQ4//PCG148cOTKdOnUqWShIFAsAAADgK2OnnXbKdtttlyFDhqSqqiojRozImDFj0rp16/Tv3z/nn39+hg0bliQZNGhQunbtulrjKBYAAABACetSZ0GSnHbaaUX3t91224afe/bsWXQpxb934oknNmoMCxwCAAAARXQWAAAAQAlV61hnwdqgswAAAAAoorMAAAAASqjEb9kr8T0DAAAAJegsAAAAgBKsWQAAAABUPJ0FAAAAUEK1zgIAAACg0uksAAAAgBJ0FgAAAAAVT2cBAAAAlOBqCAAAAEDFWyudBW1atF0bw6zU4sKn5Y6QJFm49JNyR0hV1o3KWIualuWOkCRZWlha7ghJko8WfVjuCEmSrTfsVu4ISdaNf6cdWtaVO0KSpO16G5c7QpLknfnTyx0hSbLZ+uVO8NXXqtkG5Y6QJCmkUO4ISZJPly4qd4QsrVpS7ghJkvcWvFPuCEmSDZrVljtCkuTDRbPLHSFJ8sni8v/9mCStm29Y7giZ++lH5Y6QJOm4wWbljpAkefWjV8odIUmyyfqblzvCGlW9DvxdurbpLAAAAACKKBYAAAAARSxwCAAAACVY4BAAAACoeDoLAAAAoIRqnQUAAABApdNZAAAAACWsC5f0Xtt0FgAAAABFdBYAAABACa6GAAAAAFQ8nQUAAABQgqshAAAAABVPZwEAAACUUFWB37Ov9jt+4IEH8t3vfjeDBw/O+PHjmzASALA6zM0AQFNZrc6C2bNn5/rrr8+9996b+fPnZ+TIkdlzzz2bOBoA0FjmZgBYcypxzYLVKhZMmDAhu+66a2pra1NbW5sLL7ywqXMBAKvA3AwANKXVOg1h+vTp+eSTTzJ06NAceuihmTBhQlPnAgBWgbkZANacqqqqtXZbV6z2Aocffvhhrrvuurz99ts58sgjM27cuHXqjQFApTE3AwBNZbU6C9q3b58ePXqkWbNm6dKlSzbYYIN88MEHTZ0NAGgkczMA0JRWq1jQp0+fPP/881m6dGlmz56d+fPnp23btk2dDQBoJHMzAKw5VWvxf+uK1ToNoWPHjhkwYEAOOuigJMk555yT6urKu+4kAKwrzM0AQFNa7TULhgwZkiFDhjRlFgDgSzA3A8CaUYmXTvSVAwAAAFBktTsLAAAAoBJU4tWFdBYAAAAARXQWAAAAQAnVFfg9e+W9YwAAAKAknQUAAABQgjULAAAAgIqnswAAAABK0FkAAAAAVDydBQAAAFBCdXQWAAAAABVurXQWvLvgrbUxzEo1q1o3Gilqm29Y7gh5Y+6UckdIkrRqtn65IyRJCoWl5Y6QJOnSeqtyR0iy7vy3smDJ/HJHSPPqFuWOkCRZUlhc7ghJkk4bdCl3BJrI0sKSckdIkox5bUy5IyRJDt764HJHyAcLZ5U7QpJkg2a15Y6QJJm/+ONyR0iSbLPh18sdIUmyReutyx0hSbJkafnno/WbbVDuCEmSf/nd2eWOkCT5970vKHeEimDNAgAAAKDiKRYAAAAARdaNXmMAAABYR1U7DQEAAACodDoLAAAAoIQql04EAAAAKp3OAgAAACihuqryvmevvHcMAAAAlKSzAAAAAEqocjUEAAAAoNLpLAAAAIASXA0BAAAAqHg6CwAAAKCEamsWAAAAAJVOZwEAAACUYM0CAAAAoOLpLAAAAIASrFkAAAAAVDzFAvj/tXf/MVbW957A34dhKR0GcaAOOuuVS91uMbhuy8bsIoo1gjSadO0PC84O7aZN0652Vzfayp02la1KAktNsTWFKHYbje00FKvZbYq1wRt3HbXWXLrY9mrZXAKIMFN+C2gHZ/8wO+5R+wzSmXnO3Of1IpN4zhl93sfIfPB73s/3CwAAQB23IQAAAECBWq16n7NX7x0DAAAAhTQLAAAAoICjEwEAAIDK0ywAAACAAo12dOKKFSuyZcuW1Gq1dHV15YILLhh87cknn8ydd96ZpqamzJ8/P9dff32OHTuWZcuW5Y9//GNeffXVXHfddbnssssKr2GxAAAAAMaIZ555Jtu3b093d3e2bduWrq6udHd3D75+++23Z/369Zk+fXo6OzuzaNGivPDCCzn//PPzhS98Ibt27crnPvc5iwUAAADwl6g1ULOgp6cnCxYsSJKce+65OXjwYI4cOZKWlpbs2LEjU6ZMyVlnnZUkufTSS9PT05OlS5cO/v27d+/O9OnTh7yOxQIAAAAYI/r6+jJ79uzBx1OnTk1vb29aWlrS29ubqVOn1r22Y8eOwcdLlizJyy+/nLVr1w55HYsFAAAAUGBcA5+GMDAwcNLf+6Mf/Si/+93v8pWvfCWPPPJIYWPCaQgAAAAwRrS1taWvr2/w8d69e3PGGWe842t79uxJW1tbtm7dmt27dydJzjvvvJw4cSL79u0rvI7FAgAAAChQq9VG7Wso8+bNy6ZNm5Ikzz//fNra2tLS0pIkOfvss3PkyJHs3Lkz/f392bx5c+bNm5dnn3029913X5I3bmM4evRoWltbC6/jNgQAAAAYI+bMmZPZs2dnyZIlqdVqufXWW7Nx48ZMnjw5CxcuzPLly3PTTTclSa688srMnDkzZ511Vr72ta+lo6Mjx48fzze+8Y2MG1fcHbBYAAAAAAVqtcYq5d988811j2fNmjX41xdeeGHdUYpJMnHixHzrW996V9dorHcMAAAAlG5UmgVtE88ajcsMaSAnv0vkSDp+4mjZEfLPTvtg2RGSNM4K3YmB/rIjvOFd7GQ6kvYef6nsCEmSvz/w+7IjZHrz0GfQjob3N8jv2e2Ht5UdIUly/tR/VXaEMW/rvv9ddoQkyb/7QGfZEZIkx08cKztC2pvPKTtCksb4d5Ekre+ZVnaEJElTransCEmSl47uGPqbRsHfPPGdsiNk2nvfW3aEJMl/nHNN2RGSJPte7Rv6m0bB+yaeWXaEEdXIpyGMlMb4PzUAAACgYVgsAAAAAOrY4BAAAAAKnMyRhv/YaBYAAAAAdTQLAAAAoEDNBocAAABA1WkWAAAAQAF7FgAAAACVp1kAAAAABcbZswAAAACoOs0CAAAAKFCrVe9z9uq9YwAAAKCQZgEAAAAUqFVwz4JTWix45ZVXcsstt+TgwYP505/+lOuvvz6XXHLJcGcDAE6S2QwADKdTWix46KGHMnPmzNx0003Zs2dPPvvZz+bnP//5cGcDAE6S2QwAI6dWq16z4JT2LGhtbc2BAweSJIcOHUpra+uwhgIA3h2zGQAYTqfULLjqqquycePGLFy4MIcOHcq6deuGOxcA8C6YzQAwcqq4Z8EpNQsefvjhtLe35xe/+EV+8IMf5Jvf/OZw5wIA3gWzGQAYTqe0WPDcc8/l4osvTpLMmjUre/fuzYkTJ4Y1GABw8sxmAGA4ndJiwYwZM7Jly5Ykya5duzJp0qQ0NTUNazAA4OSZzQAwcmq12qh9NYpT2rNg8eLF6erqSmdnZ/r7+7N8+fJhjgUAvBtmMwAwnE5psWDSpElZs2bNcGcBAE6R2QwAI2ecDQ4BAACAqjulZgEAAABURSPtJTBaNAsAAACAOpoFAAAAUKBWwc/Zq/eOAQAAgEKaBQAAAFDAngUAAABA5WkWAAAAQIFaNAsAAACAitMsAAAAgALjKrhnwagsFhx8bd9oXGZIE8c3lx0hSXL4T4fKjpBn9j9VdoQkyb9uu6jsCEmSAw3y3+jeY3vKjpAk+cCUD5YdIUkyvXl62RHS9t4zy46QJNlzdFfZEZIk42pNZUdgmJwz+ZyyIyRJDv3pQNkRkiTN41vKjpCdr/xD2RGSJNMmtpUdIUny73/+N2VHSJJ86yP/uewISZLz/tOnyo6QJLm2Y0HZEbJi3rKyIyRJbvzb28uOkCT5tx+YW3aEJMk/n3J+2REYZpoFAAAAUMCeBQAAAEDlWSwAAAAA6rgNAQAAAArUKrjBoWYBAAAAUEezAAAAAArUKvg5e/XeMQAAAFBIswAAAAAK2LMAAAAAqDzNAgAAACgwLpoFAAAAQMVpFgAAAEABexYAAAAAladZAAAAAAVq9iwAAAAAqk6zAAAAAArYswAAAACoPM0CAAAAKFCr4Ofs1XvHAAAAQCGLBQAAAEAdtyEAAABAgXE2OAQAAACqTrMAAAAACtSiWQAAAABUnGYBAAAAFKjZswAAAACoOs0CAAAAKGDPAgAAAKDyNAsAAACggD0LAAAAgMrTLAAAAIAC4yr4OXv13jEAAABQSLMAAAAAClRxz4LawMDAwEhf5Jb/tWykL3FSVt37UNkR3jDpn5SdIB94/z8tO0KS5OXe/WVHSJIc3nWg7AhJkn8xd1bZEZIkvb2N8e/jYxfPKTtC/kfP35UdIUnys//wX8uOkCS59sH/UnaEJMnzX/7vZUcY8z7y486yIyRJHrjqtrIjJEl2H91VdoRMe8/7yo6QJBnIiP/R8KQ01ZrKjpCkcXL8fMemsiMkSV4+8seyI+Tzsz9TdoQkyYdWdpQdIUmy7+CRsiMkSfq/vaXsCCPqqb1/O2rX+jdtl47atYpoFgAAAECBWqrXLLBYAAAAAGPIihUrsmXLltRqtXR1deWCCy4YfO3JJ5/MnXfemaampsyfPz/XX399kmTVqlX59ZTVV+0AAA2mSURBVK9/nf7+/nzxi1/MFVdcUXgNiwUAAAAwRjzzzDPZvn17uru7s23btnR1daW7u3vw9dtvvz3r16/P9OnT09nZmUWLFqWvry8vvvhiuru7s3///nz84x+3WAAAAAB/iUba4LCnpycLFixIkpx77rk5ePBgjhw5kpaWluzYsSNTpkzJWWedlSS59NJL09PTk46OjsH2wWmnnZZjx47lxIkTaWr68/uyODoRAAAAxoi+vr60trYOPp46dWp6e3uTJL29vZk6derbXmtqakpzc3OSZMOGDZk/f37hQkGiWQAAAACFGnmDw3dzwOFjjz2WDRs25L777hvyey0WAAAAwBjR1taWvr6+wcd79+7NGWec8Y6v7dmzJ21tbUmSJ554ImvXrs29996byZMnD3kdtyEAAABAgdoo/hrKvHnzsmnTpiTJ888/n7a2trS0tCRJzj777Bw5ciQ7d+5Mf39/Nm/enHnz5uXw4cNZtWpV1q1bl9NPP/2k3rNmAQAAAIwRc+bMyezZs7NkyZLUarXceuut2bhxYyZPnpyFCxdm+fLluemmm5IkV155ZWbOnDl4CsKNN944+M9ZuXJl2tvb/+x1LBYAAABAkQY6DSFJbr755rrHs2bNGvzrCy+8sO4oxSRZvHhxFi9e/K6u4TYEAAAAoI5mAQAAABRo5NMQRopmAQAAAFDnpBYLXnjhhSxYsCAPPPBAkmT37t1ZunRpOjo6csMNN+S1114b0ZAAQD2zGQBGT61WG7WvRjHkYsHRo0dz2223Ze7cuYPP3XXXXeno6MiDDz6YGTNmZMOGDSMaEgB4k9kMAIy0IRcLJkyYkHvuuSdtbW2Dzz399NO5/PLLkySXXXZZenp6Ri4hAFDHbAaA0VUbxV+NYsgNDsePH5/x4+u/7dixY5kwYUKSZNq0aent7R2ZdADA25jNAMBI+4tPQxgYGBiOHADAMDGbAWB4NdIn/qPllE5DaG5uzvHjx5Mke/bsqatBAgCjz2wGAIbTKS0WXHTRRdm0aVOS5NFHH80ll1wyrKEAgHfHbAYAhtOQtyFs3bo1K1euzK5duzJ+/Phs2rQpq1evzrJly9Ld3Z329vZcffXVo5EVAIjZDACjrZGONBwtQy4WnH/++bn//vvf9vz3v//9EQkEABQzmwGAkfYXb3AIAAAA/5jZ4BAAAACoPM0CAAAAKKBZAAAAAFSeZgEAAAAUqOJpCJoFAAAAQB3NAgAAAChgzwIAAACg8jQLAAAAoIA9CwAAAIDK0ywAAACAAlXcs6A2MDAwMOIXWXzuSF/ipEz669ayIyRJzp/9/rIj5FfP/b7sCEmSlub3lh0hSTJ5UmPk+OOBw2VHSJIc/7s9ZUdIkvy3795adoSs/uUjZUdIkmx96u/LjvCGo/1lJ0iSDGz4P2VHGPP+4fALZUdIkrzS/0rZEZIkW/dtLTtCZp3+wbIjJEleff3VsiMkSaa9531lR0iSjB/XGJ+tjWuQQvDvDvy27Aj5n7t+VXaEJMlnz7u27AhJktcHXi87QpLkA1Nmlx1hRP3uwJZRu9Z5p//LUbtWkcb46QcAAAANqorNgsZYogQAAAAahmYBAAAAFHAaAgAAAFB5FgsAAACAOm5DAAAAgAI2OAQAAAAqT7MAAAAACmgWAAAAAJWnWQAAAAAFHJ0IAAAAVJ5mAQAAABTSLAAAAAAqTrMAAAAACtizAAAAAKg8zQIAAAAoULNnAQAAAFB1mgUAAABQQLMAAAAAqDzNAgAAACjgNAQAAACg8iwWAAAAAHXchgAAAAAFbHAIAAAAVJ5mAQAAABTQLAAAAAAqT7MAAAAACjg6EQAAAKg8zQIAAAAoYM8CAAAAoPI0CwAAAKCAPQsAAACAytMsAAAAgAJV3LOgNjAwMFB2CAAAAGhULx3dPmrXam+eMWrXKqJZAAAAAIWq1yywZwEAAABQR7MAAAAAClSvV6BZAAAAALyFxQIAAACgjtsQAAAAoECtVr0bETQLAAAAgDpjYrFgxYoVWbx4cZYsWZLf/OY3Zccp3apVq7J48eJ88pOfzKOPPlp2nNIdP348CxYsyMaNG8uOUrpHHnkkH/vYx/KJT3wijz/+eNlxSvXKK6/ky1/+cpYuXZolS5bkiSeeKDtSKV544YUsWLAgDzzwQJJk9+7dWbp0aTo6OnLDDTfktddeKzkhY5XZXM9srmc2v8lsfpPZ/AazeayqjeJXY2j4xYJnnnkm27dvT3d3d+64447ccccdZUcq1VNPPZUXX3wx3d3duffee7NixYqyI5Xue9/7XqZMmVJ2jNLt378/d999dx588MGsXbs2v/zlL8uOVKqHHnooM2fOzP333581a9ZU8mfH0aNHc9ttt2Xu3LmDz911113p6OjIgw8+mBkzZmTDhg0lJmSsMpvrmc1vZza/wWyuZzabzYwtDb9Y0NPTkwULFiRJzj333Bw8eDBHjhwpOVV5LrzwwqxZsyZJctppp+XYsWM5ceJEyanKs23btvzhD3/IRz7ykbKjlK6npydz585NS0tL2tractttt5UdqVStra05cOBAkuTQoUNpbW0tOdHomzBhQu655560tbUNPvf000/n8ssvT5Jcdtll6enpKSseY5jZXM9srmc2v8lsrmc2m81jWfV6BWNgsaCvr6/uB8nUqVPT29tbYqJyNTU1pbm5OUmyYcOGzJ8/P01NTSWnKs/KlSuzbNmysmM0hJ07d+b48eP50pe+lI6OjsoPmquuuiovvfRSFi5cmM7Oztxyyy1lRxp148ePz8SJE+ueO3bsWCZMmJAkmTZtWqV/nnLqzOZ6ZnM9s/lNZnM9s9lsZmwZc6chDAwMlB2hITz22GPZsGFD7rvvvrKjlOanP/1pPvShD+Wv/uqvyo7SMA4cOJDvfve7eemll/KZz3wmmzdvruTOrUny8MMPp729PevXr8/vf//7dHV1uXf2Lfw8Zbj4b+kNZrPZ/E7M5jeZzUPz87SRVe/3bcMvFrS1taWvr2/w8d69e3PGGWeUmKh8TzzxRNauXZt77703kydPLjtOaR5//PHs2LEjjz/+eF5++eVMmDAhZ555Zi666KKyo5Vi2rRp+fCHP5zx48fnnHPOyaRJk7Jv375Mmzat7GileO6553LxxRcnSWbNmpW9e/fmxIkTlf60L0mam5tz/PjxTJw4MXv27KmrQcLJMpvfzmx+g9lcz2yuZza/M7OZRtXwtyHMmzcvmzZtSpI8//zzaWtrS0tLS8mpynP48OGsWrUq69aty+mnn152nFJ9+9vfzk9+8pP8+Mc/zjXXXJPrrruusn8YSZKLL744Tz31VF5//fXs378/R48ereS9gP/PjBkzsmXLliTJrl27MmnSpMr/YSRJLrroosGfqY8++mguueSSkhMxFpnN9czmN5nN9czmembzOzObx4ZarTZqXyej6FSiJ598Mp/61KeyePHi3H333YPPv/UkjqE0fLNgzpw5mT17dpYsWZJarZZbb7217Eil+tnPfpb9+/fnxhtvHHxu5cqVaW9vLzEVjWD69OlZtGhRPv3pTydJvv71r2fcuIZfDxwxixcvTldXVzo7O9Pf35/ly5eXHWnUbd26NStXrsyuXbsyfvz4bNq0KatXr86yZcvS3d2d9vb2XH311WXHZAwym+uZzfw5ZnM9s9lsZnj8/6cSbdu2LV1dXenu7h58/fbbb8/69eszffr0dHZ2ZtGiRWlvb3/bSRxDqQ24MQYAAAD+rL3HXxq1a7VNLF5sXrNmTdrb23PNNdckST760Y9mw4YNaWlpyY4dO/LVr341P/zhD5Mk69atS3Nzc6699tr09/fnnnvuSWtrazo7O4fMUd2lTQAAABhjik4l6u3tzdSpU9/22judxDGUhr8NAQAAAMpUa+DTEEbqZgHNAgAAABgjik4leutrf8kJGxYLAAAAoEBtFH8NpehUorPPPjtHjhzJzp0709/fn82bN2fevHmn9p5tcAgAAAB/Xt/xl0ftWu+beOaQ37N69eo8++yzg6cS/fa3v83kyZOzcOHC/OpXv8rq1auTJFdccUU+//nPv+0kjunTp+c73/lO4ZG/FgsAAACgQKMtFowGtyEAAAAAdSwWAAAAAHUcnQgAAAAFarXGPTpxpGgWAAAAAHUsFgAAAAB1LBYAAAAAdexZAAAAAAVqsWcBAAAAUHGaBQAAAFBIswAAAACoOM0CAAAAKFC9XoFmAQAAAPAWmgUAAABQoFarXrdAswAAAACoo1kAAAAAhTQLAAAAgIqzWAAAAADUcRsCAAAAFKjeTQiaBQAAAMBbaBYAAABAoep1CzQLAAAAgDqaBQAAAFCgVtMsAAAAACrOYgEAAABQx2IBAAAAUMeeBQAAAFCg5jQEAAAAoOo0CwAAAKCQZgEAAABQcZoFAAAAUKB6vQLNAgAAAOAtNAsAAACgQK1WvW6BZgEAAABQx2IBAAAAUMdtCAAAAFDIbQgAAABAxWkWAAAAQIHq9Qo0CwAAAIC30CwAAACAQtXrFmgWAAAAAHU0CwAAAKBAraZZAAAAAFScxQIAAACgjsUCAAAAoI49CwAAAKBAzWkIAAAAQNXVBgYGBsoOAQAAADQOzQIAAACgjsUCAAAAoI7FAgAAAKCOxQIAAACgjsUCAAAAoI7FAgAAAKDO/wUfF1N8E/VpVQAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "feature = 'NEG'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " if not detect_all_negations(example[1]) and not detect_all_negations(example[2]):\n",
+ " continue\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=feature)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=feature)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to NEG tokens')\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.5 Target subjects"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 52,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [06:03<00:00, 2.75it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAsAAAJ1CAYAAAC/wnk8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xu4lGW9P/73WgvJA0icFqiIErZQMUTzsA0TQZBD3zTxtFJRy1Lb6vZ8QgVLpDTzEsnK1L0tS/dSBA/bPG4lTyBhOw/ULiU1UDkpIoiKwPz+cLN+jsCAuGCweb261nUx8zwz92eeNXav+cz7uZ+qQqFQCAAAAMD/qS53AQAAAMCGRbMAAAAAKKJZAAAAABTRLAAAAACKaBYAAAAARTQLAAAAgCKaBQBQAcaMGZNu3brlqaeeKncprGczZsxIt27dct5555W7FAA+QzQLAGANXHbZZenWrVt22223vPfee6vc75FHHslDDz1UdN8777yTMWPG5O23317XZSZJ/va3v+XGG28sum/QoEEZPXp0vvjFL66XGlZmZcemqbz88ssZMWJEvv71r2fPPfdM9+7ds+eee+bII4/Mr3/963zwwQdF+w8dOjTdunXLnDlzVvmcTz31VLp165bhw4evcN/Hf3beeecMHjw4I0eOzKxZs1b5nOedd17R4wYOHLja17a+3z8AkCTNyl0AAGzoFi9enPHjx6e6ujoLFizIfffdl2984xsr3feGG25Ip06d0q9fv8b7nnvuufz0pz/NQQcdlM0333yd13vffffljjvuyLHHHtt433bbbZfttttunY9dysqOTVOYMmVKvvOd7+Rzn/tchgwZkrq6ulRXV+f111/Pf/3Xf+XSSy/N73//+1x//fWpqqpqkjF79eqVww47rPH2W2+9lT/+8Y/57W9/m/vuuy/jx49P+/btV3jckUcemX333TdJcuqpp67RWOv7/QMAiWYBAKzWAw88kHnz5uWII47ILbfckltvvXWlzYJly5Zl6tSp6dSpU9H9zz333PoqtSzjrYlVHZum8KMf/Sjvv/9+Ghoa0q1bt6Jt3/72t/Pd7343jz/+eB577LHss88+TTJmp06dVkgF1NfXp2PHjrn22mszfvz4HH/88Ss87ktf+lK+9KUvfaKxNsTfJwD//JyGAACr0dDQkCQ55phj8uUvfzlPP/10pk2bVrTPuHHjssMOO2TRokUZP358unXrljFjxqRv37654oorkiT77bffCh9mx44dm0MOOSQ777xzdtlllxx00EG56aabsmzZssZ9lp9zfsEFF+SFF17Id77zney2227p0aNHjjzyyDz//PNF+z366KN59dVX061btwwdOjTJqtcsePTRR3PMMcdkt912y0477ZS+fftm5MiRefPNN4v269u3b/r3758FCxbkwgsvTK9evbLTTjtl0KBBufvuu0sev1Udm+WmTZuWM844I7169Ur37t3zla98Jf/2b/+Wv/71r6v93SQfnnbRoUOHFY5tkjRv3jwXX3xxfvnLX37iD+lrY9ddd02SvPHGG03yfKXeP8uWLcuvf/3rfOMb38jOO++cnXfeOQcccEBuuOGGLFmyZLXP/eyzz6Znz5458MADs2DBgsb7H3rooRx11FHZZZdd0qNHjwwePDjXXHNN3n///aLHd+vWLccee2xmzZqVU089NXvuuWd22mmnDBkyJI8//njRvnPnzs0Pf/jDDBgwIDvvvHP22GOPHH744Rk/fvynOTwArEOaBQBQwksvvZTJkydnl112ybbbbtuYKLjtttuK9ttzzz0zYsSIJMkee+yR0aNHZ9CgQRkxYkT22GOPJMmIESMyevToxsf86Ec/ygUXXJDa2tpceOGFOeecc9K+ffuMHDkyF1100Qq1zJ49O9/61rfStWvXXHDBBTnyyCPzpz/9KSeeeGIWL16ctm3bZvTo0WnTpk3atGmT0aNH55RTTlnla1v+7fecOXNy4okn5gc/+EH69OmTW265JUcccUQWLVpUtP+yZcty/PHH5+23387pp5+e0047LW+99VbOOeecTJ06dZXjrOrYJB9+0D/ssMPyxBNP5JBDDsmll16aI444In/4wx9SX1+fv/zlL6t83uXat2+fWbNmrXLxxi5duqR3795p3br1ap/r03rhhReSJDvssEOTPF+p98+FF16YSy+9NO3atcs555yT888/P1tttVUuv/zy1S5mOGPGjHzve99L27Ztc/3116dly5ZJkptuuiknnXRSqqqqcs455+Siiy7KDjvskDFjxuR73/teCoVC0fO8++67Ofroo9OiRYuce+65OeGEE/LSSy/l5JNPzuzZs5MkS5cuzbHHHpubb745/fv3zw9+8IOceeaZ2WSTTXLeeefl17/+dZMcKwCaWAEAWKUf/ehHhbq6usKtt95aKBQKhQULFhR23nnnwp577ll4//33i/adPn16oa6urnDuuecW3X/uuecW6urqCtOnT2+87y9/+Uuhrq6ucPHFF68w5imnnFKoq6srTJ06teh56+rqCr/73e+K9j3//PMLdXV1hSeffLLxvj59+hT69OlTtN/VV19dqKurK0yaNKlQKBQK7777bmH33Xcv7LXXXoX58+cX7XvdddcV6urqCtdee23Rc66s3vHjxxfq6uoKV1111UqO3uqPzXHHHVeoq6sr/PGPfyy6/5lnninU1dUVjjvuuJLPWygUCr/5zW8KdXV1hR133LFw2mmnFe65557CzJkzSz7mqKOOKtTV1RVmz569yn0mTZpUqKurK1x00UUr3HfeeecV5s+f3/jz6quvFn7zm98UevbsWRg6dGhh8eLFq627rq6uMGDAgNXut7L3z5/+9KdCXV1d4dvf/nZh2bJlRfsff/zxhbq6usKf/vSnQqGw4rGfP39+YdCgQYW99tqr8PLLLzc+bs6cOYUvfelLhRNOOGGF51z+38EDDzxQVH9dXV3huuuuK9r3pz/9aaGurq5w2223FQqFQmHq1KmFurq6wve///2i/ZYtW1Y444wzCpdddtlqjwEA659kAQCswvKFDTfZZJPGb8JbtGiR/fffP/PmzftUK/vfe++9SZLBgwfn7bffLvoZMGBAkmTy5MlFj+nYsWNjHcstj9aXWtV/ZSZPnpz58+dn8ODBKyyaN2TIkCTJhAkTVnjct771rSYZP0kWLVqUJ554It26dcsuu+xStK1Hjx6pq6vLxIkTV4i/f9yRRx6ZUaNGpU2bNvnd736X008/Pfvss0/222+/DB8+PFOmTPnEta3OuHHjsvvuuzf+9OnTJz/4wQ/y1a9+NVdddVU22mijJh/zox588MEkH66T8PFFG5f//h555JEVHrd48eKcdNJJmTlzZn75y19mm222adz28MMP5/3338+gQYOyYMGCovdk//79k2SF9EazZs1y9NFHF9338fdETU1NkuTPf/5z3n333cb9qqqq8pOf/CTnnHPOJz8AAKxzFjgEgFVYvrDhAQcckBYtWjTeP2TIkNx555259dZbM3jw4LV67hdffDFJctRRR61yn9dee63odufOnVfY53Of+1ySrNE56h/197//PUlSV1e3wrY2bdrk85//fF5++eWi+2tqarLVVls1yfhJ8sorr2TZsmWrvJxjly5d8re//S0zZsxI165dSz7XwQcfnAMPPDBPPfVUnnrqqTz99NN59tln09DQkIaGhvTv3z8/+clPGuv9tPr06VPUOHn33Xfz0ksvpaGhIQMHDszll1/eeNWDdWH5729lx65Lly5JssLvL0mGDRuWyZMn5xe/+EV22mmnom3L35OlPry//vrrRbc7dOiQ5s2bF9338fdEt27dsv/+++eBBx5Inz590rdv3+y1117p1atX2rRpU+plAlBGmgUAsArLFzbcY4898sorrzTe37Fjx7Rr1y6TJk3K9OnTs/XWW3/i537nnXeSJFdeeWXatWu30n0+fum9j38o+zSWr0ewySabrHT7xhtvnLfffrvovpqamsZviZvC8mOw6aabrnT78g+dH187YVWaNWuWXr16pVevXkk+/AA/YcKEXHXVVXnwwQdz3XXX5eSTT17j+gr/d35+dfWKQcza2trsueeeRfftu+++OfTQQ3PwwQfn7LPPzoMPPpjPf/7zazzeJ1Hq97fxxhsnSdG3+MmHza/lx/zRRx9Nnz59irYv33b++eevcs2FVq1aFd1e0/fklVdemfHjx2fs2LEZN25cbr/99jRr1iyDBg3K8OHDXRISYAOkWQAAK/H3v/+98TSACy+8cJX7jR07Nqeffvonfv7NNtssSbL11lunR48ea1fkp7D8A/qqPoi/++67jTWuK8ufv1QNH93vk1p++kj37t3Tv3//PProo43NguUL+i1cuHCFpsxyb731VpJ8og/8LVq0yKBBg/Lzn/88zzzzTHr37l1y/4+fQrCmSv3+lt/38eO2ZMmSXHnllbnjjjty8803Z++9985+++3XuH35/itrhHxaG220UQ477LAcdthhefPNN/P444+noaEhd999d+bOnZsbb7yxSccD4NOzZgEArMStt96aJDn00EMzevToFX4uv/zy1NTU5Pbbb1+rCP52222XJPnjH/+4wrZ33nlntefpf1rLx//b3/62wrY5c+Zk/vz5+cIXvrBOa9h2221TU1Oz0hqSD2PxzZs3T6dOnVb5HJMmTcrw4cPz9NNPr3KfLbfcMlVVVUXftO+4445JVn78l3vyySeTJDvvvHPJ1/Fxy98P7733XqZPn57TTz89P/vZz4r2WX5pyo9/U7+mSv3+ll/W8+O/v8GDB+drX/tafvjDH6Zt27a54IILMmvWrMbty09pWNkxWbx4cRYuXLhWtX5cmzZtcsABB+Smm25K9+7dM3HixKJLNwKwYdAsAICPWb6wYfPmzXPGGWdk4MCBK/wceOCB6devX+bMmdO4EODyiP7HP+gvj7F/9P7lCxXecsstee+994r2//GPf5x/+Zd/yT/+8Y+1qr+6unq1zYY99tijcUHA+fPnF21bfvrF8oUWm8LKjs0mm2yS3r17529/+9sKixBOnjw5L730Uvr06VMy6v7OO++koaEho0aNWuWH2V//+tcpFAqNpyckH65xsMkmm+RnP/tZ3njjjRUeM2XKlIwfPz5du3bNV7/61TV+nQsWLMi9996bmpqa9OzZM+3bt8+ECRNy8803F30gXv6eWX5ZxFJW9v5Z/rtpaGgoupxhoVBobHTtv//+K32+du3aZdSoUZk3b17OPffcxsf37ds3zZs3z913373CMfnVr36VvfbaK3/4wx9WW+/H3Xrrrdlnn30amxgffV2bbbZZampqVnqqBwDl5TQEAPiY+++/P2+99VaGDBlScgG2o446Kvfff39uu+229OvXL+3atcvGG2+cxx57LNdee2222WabDBw4sPGb8R//+MfZfffdc+CBB2b77bfPMccck1/96lf55je/mcMPPzzNmjXL73//+zzwwAM54IADVrqg4Zro1KlTJk6cmB/+8IfZYostcuyxx66wT/PmzXPRRRfljDPOyJFHHplDDjkkLVu2zDPPPJPbbrstPXv2zKGHHrpW46/Mqo7NOeeckylTpuSkk07K0KFD06lTp7z88su5+eab07p165x99tkln3e//fbLkUcemd/+9rfZf//9c+CBB6Zbt26pqanJ3Llz8+ijj+bJJ5/MjjvumH/9139tfNwWW2yRyy67LGeddVYOOOCAfOMb30jXrl2zaNGiPPfcc7nnnnvSunXrjBkzJs2arfjn0owZM3Lfffc13v7ggw/y6quvZvz48ZkxY0bOPvvsdOjQIUly8skn5/LLL8/RRx+dI444IrNnz87111+ftm3bZujQoas9dit7/3Tv3j1HHHFEbr755px44onp27dvlixZkocffjiTJk3Kt771rZUuXrncvvvu2/j46667Lscff3zatm2bs846K6NGjUp9fX2GDh2ali1bZvLkybnjjjuy2267rXDVijWx5557Nr7++vr6bLPNNlm8eHEef/zxTJ48OYceeug6P+UFgE9OswAAPmb5N+vHHHNMyf322GOP1NXV5bHHHsvMmTPTsWPHnHfeebnqqqvys5/9LIcddlgGDhyY+vr6PP7443n88cfz/PPPN37jO2zYsHzxi19MQ0NDfvjDH2bZsmXZdtttc/bZZ6/0A/6aOvXUU/Pqq6/mt7/9bbp167bK5xo8eHBatWqVa6+9NldffXUWL16crbbaKscff3xOOOGEJl1QcaONNlrpsenSpUtuvfXWXH311bn55pszf/78tG7dOn379s1JJ520RotHDh8+PL17987tt9+e+++/PzfddFMKhUI233zz1NXVZcSIETnkkENWeD0DBgzIdtttlxtuuCH33ntvZs+enY022iidO3fOd7/73RxzzDGrXK/giSeeyBNPPFH0+jp06JC6urpcdNFF2XvvvRu3HXfccWnTpk1uuummjBo1Ks2aNcvee++ds846a5XrJXzUqt4/w4cPT9euXXPrrbfm0ksvTXV1dbbbbruMHDlyjRo95557biZPnpyrr746e+21V770pS/lmGOOyVZbbZUbb7wxV1111QrviZU1TlZnm222SUNDQ6699tqMHTs2b7zxRpo1a5YvfOELGTZsWMkrggBQPlWFj2bXAAAAgIrnBDEAAACgiGYBAAAAUESzAAAAACiiWQAAAAAU0SwAAAAAimgWAAAAAEU0CwAAAIAimgUAAABAEc0CAAAAoIhmAQAAAFBEswAAAAAoolkAAAAAFNEsAAAAAIpoFgAAAABFNAsAAACAIpoFAAAAQBHNAgAAAKCIZgEAAABQRLMAAAAAKKJZAAAAABTRLAAAAACKaBYAAAAARTQLAAAAgCKaBQAAAEARzQIAAACgiGYBAAAAUESzAAAAACiiWQAAAAAU0SwAAAAAimgWAAAAAEU0CwAAAIAimgUAAABAEc0CAAAAoIhmAQAAAFBEswAAAAAoolkAAAAAFNEsAAAAAIpoFgAAAABFNAsAAACAIpoFAAAAQBHNAgAAAKCIZgEAAABQRLMAAAAAKKJZAAAAABTRLAAAAACKaBYAAAAARTQLAAAAgCKaBQAAAEARzQIAAACgiGYBAAAAUESzAAAAACiiWQAAAAAU0SwAAAAAimgWAAAAAEU0CwAAAIAizcpdAAAAAGzIqvp3Wm9jFR6csd7GKkWyAAAAACiiWQAAAAAUcRoCAAAAlFJVVe4K1jvJAgAAAKCIZAEAAACUUoFfs1fgSwYAAABKkSwAAACAUqxZAAAAAFQ6yQIAAAAopfKCBZIFAAAAQDHJAgAAACjFmgUAAABApZMsAAAAgFIq8Gv2CnzJAAAAQCmaBQAAAEARpyEAAABAKRY4BAAAACqdZAEAAACUUnnBAskCAAAAoJhkAQAAAJRSXXnRAskCAAAAoIhkAQAAAJRSecECyQIAAACgmGQBAAAAlFJVedECyQIAAACgiGQBAAAAlFJ5wQLJAgAAAKCYZAEAAACUUl150QLJAgAAAKCIZAEAAACUUnnBAskCAAAAoJhmAQAAAFDEaQgAAABQSlXlnYcgWQAAAAAUkSwAAACAUlw6EQAAAKh0kgUAAABQSuUFCyQLAAAAgGKSBQAAAFCKqyEAAAAAlU6yAAAAAEqpvGCBZAEAAABQTLIAAAAASqmuvGiBZAEAAABQRLIAAAAASqm8YIFkAQAAAFBMsgAAAABKqaq8aIFkAQAAAFBEswAAAAAo4jQEAAAAKKUCv2avwJcMAAAAlCJZAAAAAKVY4BAAAACodJIFAAAAUErlBQskC2BlunXrlv79+2fgwIEZMGBAvvOd72T69OnlLmut9O/fP0899VTJfc4777z87Gc/W08VAVCpPjq/Lv857rjjkiTHHHNMpk6d2mRjPfbYY3nttdea7PlKueCCCzJmzJj1MtaOO+6YGTNmlNxn6NChufPOO9dLPcA/L8kCWIWbbropHTt2TJL85Cc/yaWXXppf/OIXZa4KAD7bPjq/ftSvfvWrJh3nxhtvzPe+971sueWWTfq8QIXawNYsGDVqVJ555plUVVVl2LBh6dGjR+O2J598MldeeWVqamqyzz775KSTTsptt92Wu+66q3Gf559/Pv/zP/9TcgzNAlgD//Iv/5KHH3648fZtt92Wf//3f8/SpUvTvn37XH755dlqq60ya9asnHPOOZkzZ04WL16cr33tazn99NNTKBRyzTXX5O67787ixYuz33775fzzz09NTU3ROOPGjcuECROy0UYb5emnn06XLl1y0kkn5Yorrsg//vGPnHrqqTn88MOzbNmyjB49Ovfff3+SpGfPnhk+fHg23XTTPP/88zn33HOzZMmS9O7du+j5H3rooYwePTqLFi3KNttskyuuuCJt2rRZ9wcQAFajb9++ufzyy9OxY8fU19fn+OOPz2233Za33nor559/fgYPHrzG8+lVV12VSZMm5e9//3vOPvvsPProo+ncuXP+9V//NcmHibrlt/v27Zvjjz8+Y8eOzcyZM/P//t//y3nnnZdk1fPmvHnzcuaZZ+bll1/Odtttl4033nilDZAxY8Zk7ty5mTlzZqZOnZq99torgwcPzpgxYzJ79uxccskl6dOnT95///1ceumleeqpp1JdXZ3evXvn7LPPTk1NTX7/+99n5MiRadasWQ4++OCi529oaMh//Md/ZPHixenZs2dGjRqVjTfeeB39hoANxeTJk/PKK6+koaEh06ZNy7Bhw9LQ0NC4feTIkbnhhhvSoUOHHHXUURkwYEAOPfTQHHrooY2Pv/fee1c7jtMQYDUWL16cu+66K3379k2SvPHGG/nBD36Q//iP/8gDDzyQzp07N0b4b7zxxuy+++753e9+l7vvvjvTp0/P7Nmzc+edd+a+++7L2LFj8+CDD2b69Om55ZZbVjreY489lpNPPjkPPPBApk2bluuvvz6//e1vc+mllzaOc++99+bRRx/NuHHjcs899+Ttt9/OjTfemCS5+OKLc/TRR+f+++/PLrvs0hhVnD59es4555z85Cc/yX//939nzz33zMUXX7xuDx4ArIV58+aluro6d999d4YNG5arrroqSdZ4Pj3ttNPSoUOH/PjHP87gwYNXO94f/vCHNDQ05Pbbb89vfvObzJw5s+S8ed1116V169Z5+OGHM3z48Dz++OOrfO4JEyZk1KhRufvuu3Pfffc1zt8nnnhirrvuuiQfpipmzpyZe+65J+PHj8+UKVPyX//1X1m6dGkuuOCCjBgxIvfee2+qq6uzdOnSJMmUKVMyevTo/OpXv8rDDz+cFi1aZPTo0Z/0UANrqno9/qzGxIkT069fvyRJ165dM3/+/CxcuDDJh3/zt2rVKltssUVj83HixIlFj7/mmmsam6ere8nASgwdOjQDBw5Mr1698txzz2XIkCFJkrZt2+bpp59u/AZht912a1zPoG3btnn88cczZcqUNG/ePFdeeWVqa2vzyCOP5OCDD07Lli3TrFmzHHrooXnggQdWOu52222XLl26pHnz5tlmm22y9957p6amJnV1dZk9e3aSD//w+MY3vpFNN900NTU1GTJkSJ544om8//77ee655xr/MBo4cGA22WSTJMmjjz6aPfbYI3V1dUmS+vr6PPzww41/dADA+rB8fl3+c+GFF66wz5IlSxrn3e7duzeuPfBJ5tNP4utf/3pqamrSoUOHtG3bNq+//nrJeXPKlCkZNGhQkqRTp07ZY489Vvncu+yyS9q2bZvWrVunffv22WeffZJkhXn9sMMOS7NmzbLxxhvn61//ep544om8/PLLWbx4cfbee+8kyUEHHdT4vA8//HAGDx6cDh06JEm++c1vNsmxADZ8c+fOTevWrRtvt2nTJnPmzEmSzJkzpyg5/NFtSfLss89miy22SPv27Vc7jtMQYBU+ek7lH/7whwwdOjTjxo1L27Ztc/XVVzf+wfDOO++kS5cuSZJjjz02y5Yty/e///3Mnj07Rx55ZE455ZQsWLAgN9xwQ2M8aOnSpauM/2+22WaN/66pqcmmm27a+O9ly5YlSd588820atWqcb9WrVrljTfeyFtvvZUkadGiRZKkqqoqm2++eZJkwYIFmTJlSgYOHNj4uBYtWjQ+BgDWh1WtWfBRH53/qqurG+e/Vc2ns2bNyjHHHJMk6dGjRy6//PJPVNPyeXP52EuXLi05b86fPz8tW7ZsvH/5XLsyq5rXP/q6VjWvz58/v6i2j+6zYMGCPPjgg42phkKhkA8++OATvW7gE9jA1iz4qEKhsMb7jh07tqjxWIpmAayB3XffPVtuuWWefvrpLFmyJA8//HB+85vfpE2bNrn11ltz9913J0maNWuW448/Pscff3xeeumlfPe7382Xv/zl1NbWpm/fvjnqqKOapJ527doVfch/66230q5du8Y/IhYuXJiWLVtm2bJlmT9/fpKktrY2X/nKV3L11Vc3SQ0AsL6Vmk/vu+++VT7uox/MkzTOjasba1Xz5uabb54FCxY03n7zzTez9dZbr/Y5V6XUvL48Wrx8nI/Wd9BBB+Xcc89d63GBz6ba2trMnTu38fbs2bMbkwIf3zZr1qzU1tY23n7qqadWmuhaGachwBp46aWX8tJLL+ULX/hC3njjjWy11VaNCxzde++9eeedd5Ikw4cPzxNPPJEk6dy5c9q1a5eqqqrst99+ufPOO/Puu+8mSf7zP/8z48ePX+t69t1339x111159913s2TJkowdOza9e/fOxhtvnO233z4PPvhgkuSee+7J+++/nyTZe++9M2XKlMZTJp599tmMHDlyrWsAgPXtk8ynzZo1a/xA3759+/zv//5vkg/P5/3jH/+42rFKzZs9e/bMQw89lCT5xz/+kaeffvpTva599903Y8eOzdKlS7No0aLceeed6d27dzp37pyamprGSyCPGzcuVf/37Wbfvn3zwAMPNDYQHnroofzyl7/8VHUAJVStx5/V6NWrV+NC51OnTk1tbW1jCqlTp05ZuHBhZsyYkSVLluSRRx5Jr169knzYONhss83SvHnzNXrJkgWwCkOHDm1cXbl58+bWbNluAAAgAElEQVT5/ve/n27duqVt27a555570r9//2y99dY57bTT8r3vfS8/+tGPUl9fn+HDh+eSSy5JoVBI3759s9deeyVJXnjhhcbIT+fOnXPppZeudW0DBw7MX//61wwZMiSFQiF77rlnjj766CQfLnA4bNiwXHvttdlnn33StWvXJB92GS+55JKcdNJJ+eCDD7LZZptl2LBhn+YQAcB61a9fvzWeTwcMGJAzzjgj//Zv/5bDDjssJ598cvbff//suOOOGTBgwGrHKjVvnnDCCTn99NPTt2/fdO3aNfvvv/+nel1Dhw7N9OnT87WvfS1VVVUZOHBgBg0alKqqqlxyySUZNmxYmjdvniFDhjSextC9e/eceOKJGTp0aJYtW5a2bdvm+9///qeqA/hs2HXXXdO9e/fU19enqqoqI0aMyLhx49KyZcv0798/F198cc4888wkyeDBgxtPmf74egarU1X4JCc4AAAAQIWpOnmn9TZW4afPr7exSnEaAgAAAFBEswAAAAAoYs0CAAAAKGUDvnTiuiJZAAAAABRZL8mCUx89a30Ms1rT5r25+p3Wg8N2+Eq5S8jTM/+33CUkSbq16VzuEpIkr8x/rdwlJEkm/t/lmcqtWbOacpeQJNl6883LXUK2bdW23CVsUD6/cctyl5AkOX3nDWNe+Sz71oOnlLuEJMn8/7u8a7mdsuuQcpeQR2dMKncJSZJm1RtG8HTJsiXlLiFJcuczz5S7hCRJz207lbuEJMmfX5tV7hLyjZ16lruEJGm8jGW5bdmittwlJEmG1h1X7hLWrQ3j171eSRYAAAAARTaM1jEAAABsoDaUJMn6JFkAAAAAFJEsAAAAgBIkCwAAAICKJ1kAAAAAJVRgsECyAAAAACgmWQAAAAAlVFdgtECyAAAAACgiWQAAAAAluBoCAAAAUPEkCwAAAKAEyQIAAACg4q11smDUqFF55plnUlVVlWHDhqVHjx5NWRcA8AmZmwGAprJWzYLJkyfnlVdeSUNDQ6ZNm5Zhw4aloaGhqWsDANaQuRkA1h2nIayhiRMnpl+/fkmSrl27Zv78+Vm4cGGTFgYArDlzMwDQlNaqWTB37ty0bt268XabNm0yZ86cJisKAPhkzM0AsO5UVa2/nw1FkyxwWCgUmuJpAIAmYm4GAD6NtVqzoLa2NnPnzm28PXv27LRv377JigIAPhlzMwCsO9YsWEO9evXK/fffnySZOnVqamtr06JFiyYtDABYc+ZmAKAprVWyYNddd0337t1TX1+fqqqqjBgxoqnrAgA+AXMzAKw7lZgsWKtmQZKcddZZTVkHAPApmZsBgKay1s0CAAAAqARVqbxkQZNcDQEAAAD45yFZAAAAACVU4poFkgUAAABAEckCAAAAKKECgwWSBQAAAEAxyQIAAAAooboCowWSBQAAAEARyQIAAAAowdUQAAAAgIqnWQAAAAAUWS+nIRzYdb/1McxqzVw0q9wlJEneWbKo3CXkxB5Dy11CkuTRVx8rdwlJkiN3OLjcJSRJhu/RrdwlJEnuePn2cpeQJNm25TblLiEbN9u43CUkSf785l/KXUKS5IV5r5S7BJpIv213LXcJSZLZi94odwlJkj/Ofq7cJWS/zl8tdwlJkoenP1HuEpIku9R2L3cJSZIv79+j3CUkSd5e/Ha5S0iS9NlmablLSCGFcpeQJNms2WblLiFJ8ss/3V/uEpIkQ+uOK3cJ65TTEAAAAICKZ4FDAAAAKKECgwWSBQAAAEAxyQIAAAAowZoFAAAAQMWTLAAAAIASJAsAAACAiidZAAAAACVIFgAAAAAVT7IAAAAASpAsAAAAACqeZAEAAACUUIHBAskCAAAAoJhkAQAAAJRgzQIAAACg4mkWAAAAAEWchgAAAAAlOA0BAAAAqHiSBQAAAFBCtWQBAAAAUOkkCwAAAKCECgwWSBYAAAAAxSQLAAAAoARXQwAAAAA2aKNGjcrhhx+e+vr6PPvss0XbnnzyyRxyyCE5/PDDc8011zTef9ddd+WAAw7IkCFDMmHChNWOIVkAAAAAJVRlw0kWTJ48Oa+88koaGhoybdq0DBs2LA0NDY3bR44cmRtuuCEdOnTIUUcdlQEDBqRt27a55pprcvvtt2fRokUZM2ZM9t1335LjaBYAAADAZ8TEiRPTr1+/JEnXrl0zf/78LFy4MC1atMj06dPTqlWrbLHFFkmS3r17Z+LEiWnbtm322muvtGjRIi1atMgll1yy2nGchgAAAAAlVFVVrbef1Zk7d25at27deLtNmzaZM2dOkmTOnDlp06bNCttmzJiR9957LyeeeGKOOOKITJw4cbXjSBYAAADAZ1ShUFij/d5666389Kc/zWuvvZajjz46jzzySMnmhGYBAAAAlLAhXQ2htrY2c+fObbw9e/bstG/ffqXbZs2aldra2myyySbZZZdd0qxZs3Tu3DmbbbZZ3nzzzbRt23aV4zgNAQAAAD4jevXqlfvvvz9JMnXq1NTW1qZFixZJkk6dOmXhwoWZMWNGlixZkkceeSS9evXK3nvvnUmTJmXZsmWZN29eFi1aVHQqw8pIFgAAAEAJG1CwILvuumu6d++e+vr6VFVVZcSIERk3blxatmyZ/v375+KLL86ZZ56ZJBk8eHC6dOmSJBkwYEAOO+ywJMmFF16Y6urS2QHNAgAAAPgMOeuss4pub7/99o3/3n333YsupbhcfX196uvr13gMpyEAAAAARdZLsuDZuX9eH8Os1s7tu5e7hCTJDp8vfx1/nf+XcpeQJNm9w5fLXUKS5O3Fb5e7hCTJ9X+5vtwlJEl2aPPFcpeQJFlaWFbuEvL4a5PKXUKSpN0mpc8pW1926/ClcpdAE5n21vRyl5Ak2bV2p3KXkCTp3HLrcpeQ/5nzp3KXkCTZse125S4hyYazmNjNf3mo3CUkSb669Q7lLiFJ8oXNtyl3CXnytafLXUKS5NnZr5W7hCTJoK7m5vVhQ/n/pPVJsgAAAAAoYs0CAAAAKEGyAAAAAKh4kgUAAABQgmQBAAAAUPEkCwAAAKCECgwWSBYAAAAAxSQLAAAAoARrFgAAAAAVT7IAAAAASpAsAAAAACqeZAEAAACUIFkAAAAAVLy1ThZcfvnlefrpp7NkyZKccMIJ2X///ZuyLgDgEzI3A8C6UYHBgrVrFkyaNCkvvPBCGhoaMm/evBx00EH+IAGAMjI3AwBNaa2aBbvvvnt69OiRJNl8883z7rvvZunSpampqWnS4gCANWNuBgCa0lo1C2pqarLpppsmScaOHZt99tnHHyMAUEbmZgBYdypxgcNPdTWEhx56KGPHjs2///u/N1U9AMCnYG4GAJrCWjcLHnvssfziF7/I9ddfn5YtWzZlTQDAWjA3A8C6IVmwhhYsWJDLL788N954Yz7/+c83dU0AwCdkbgYAmtJaNQt+97vfZd68eTnttNMa77vsssuy5ZZbNllhAMCaMzcDwLojWbCGDj/88Bx++OFNXQsAsJbMzQBAU/pUCxwCAADAP7sKDBakutwFAAAAABsWyQIAAAAooRLXLJAsAAAAAIpIFgAAAEApkgUAAABApZMsAAAAgBKsWQAAAABUPMkCAAAAKKECgwWSBQAAAECx9ZIs6NJq6/UxzGrNfndOuUtIknT7/LJyl7DBeO6N58tdQpJkzqJ55S4hSbL/Nn3KXUKS5N0l75a7hCTJxNf/UO4SUlO9YfRUp7/9erlLSJJ0b1dX7hJoIs2qa8pdQpLkj7M3jHmg3SZty13CBnM+bNuN25S7hCTJrX+9v9wlJEmO3LF/uUtIsuHMzc+/8ddyl5C3Fy8qdwlJkpdf3zA+W/Tddsdyl8A/KachAAAAQAkbSkN3fdowvjIDAAAANhiSBQAAAFCCZAEAAABQ8SQLAAAAoATJAgAAAKDiSRYAAABACRUYLJAsAAAAAIpJFgAAAEAJ1iwAAAAAKp5kAQAAAJQgWQAAAABUPMkCAAAAKEGyAAAAAKh4kgUAAABQgmQBAAAAUPE0CwAAAIAiTkMAAACAEirwLATJAgAAAKCYZAEAAACUYIFDAAAAoOJJFgAAAEAJkgUAAABAxZMsAAAAgBIkCwAAAICKJ1kAAAAAJVRgsECyAAAAACgmWQAAAAAlbGhrFowaNSrPPPNMqqqqMmzYsPTo0aNx25NPPpkrr7wyNTU12WeffXLSSSflqaeeyqmnnpovfvGLSZK6urpcdNFFJcfQLAAAAIDPiMmTJ+eVV15JQ0NDpk2blmHDhqWhoaFx+8iRI3PDDTekQ4cOOeqoozJgwIAkyR577JGrr756jcdxGgIAAACUUlW1/n5WY+LEienXr1+SpGvXrpk/f34WLlyYJJk+fXpatWqVLbbYItXV1endu3cmTpy4Vi9ZswAAAAA+I+bOnZvWrVs33m7Tpk3mzJmTJJkzZ07atGmz0m0vvvhiTjzxxHzzm9/ME088sdpxnIYAAAAAJWxoaxZ8VKFQWO0+2267bU4++eQMGjQo06dPz9FHH50HHnggzZs3X+VjJAsAAADgM6K2tjZz585tvD179uy0b99+pdtmzZqV2tradOjQIYMHD05VVVU6d+6cdu3aZdasWSXHWS/JgsVLF6+PYVZr5jtzyl1CkqS6qqbcJeT9pe+Xu4QkyabNNil3CUmSndtvWe4SkiTT5v+93CUkSbZvvX25S0iSHLrdweUuITPe+Ue5S0iSTJo5pdwlJEmWLFta7hJoIju337HcJSRJnp71XLlLSJIsWbak3CWs0TdD68Mb771Z7hKSJD1qv1DuEpIkHTapLXcJSZL3lr5X7hKSJFtutkW5S8h77TeMY9Fxszar32k9+Osb08tdAutZr169MmbMmNTX12fq1Kmpra1NixYtkiSdOnXKwoULM2PGjHTs2DGPPPJIrrjiitx1112ZM2dOjjvuuMyZMydvvPFGOnToUHIcpyEAAABACdUb0FkIu+66a7p37576+vpUVVVlxIgRGTduXFq2bJn+/fvn4osvzplnnpkkGTx4cLp06ZL27dvnrLPOyn//93/ngw8+yMUXX1zyFIREswAAAAA+U84666yi29tv//8ng3ffffeiSykmSYsWLfKLX/ziE42hWQAAAAAlbMgLHK4rFjgEAAAAikgWAAAAQAnVkgUAAABApZMsAAAAgBKsWQAAAABUPMkCAAAAKKESv2WvxNcMAAAAlCBZAAAAACW4GgIAAABQ8SQLAAAAoARXQ/iE3nvvvfTr1y/jxo1rqnoAgE/B3AwANIVPlSz4+c9/nlatWjVVLQDAp2RuBoCmZ82CT2DatGl58cUXs++++zZhOQDA2jI3AwBNZa2bBZdddlnOO++8pqwFAPgUzM0AQFNZq9MQ7rjjjvTs2TNbb711U9cDAKwFczMArDuVuMDhWjULJkyYkOnTp2fChAmZOXNmmjdvno4dO+YrX/lKU9cHAKwBczMA0JTWqllw1VVXNf57zJgx2WqrrfwxAgBlZG4GgHXnU11G8DOqEl8zAAAAUMKnunRikpxyyilNUQcA0ETMzQDQtFw6EQAAAKh4nzpZAAAAAP/MKvFqCJIFAAAAQBHJAgAAACjBmgUAAABAxZMsAAAAgBIqL1cgWQAAAAB8jGQBAAAAlGDNAgAAAKDiSRYAAABACZIFAAAAQMVbL8mCtxa/vT6GWa1Dtzu43CUkSZ5943/KXUKWFZaVu4QkyRabbVHuEjYo27bcttwlJEneWbKw3CUkSf7yzqvlLiE3Pvdf5S4hSdKxRYtyl5AkObLuiHKXQBMZ+fv/LHcJSZJL+h5T7hKSJPM3gL9VHpv+XLlLSJIc+MV9yl1CkmTx0g/KXUKSpKaqptwlJElef+f1cpeQJPlczefKXUKumHRHuUtIksx6/Y1yl5AkOXfQQeUugX9STkMAAACAEqqchgAAAABUOskCAAAAKMEChwAAAEDFkywAAACAEiovVyBZAAAAAHyMZAEAAACUYM0CAAAAoOJJFgAAAEAJkgUAAABAxZMsAAAAgBKqJAsAAACASidZAAAAACVYswAAAACoeJIFAAAAUELl5QokCwAAAICP0SwAAAAAijgNAQAAAEqwwCEAAABQ8SQLAAAAoATJAgAAAKDiSRYAAABACVWSBQAAAEClkywAAACAEirxW/ZKfM0AAABACZIFAAAAUII1CwAAAICKJ1kAAAAAJVRLFgAAAACVTrIAAAAASpAsAAAAADZoo0aNyuGHH576+vo8++yzRduefPLJHHLIITn88MNzzTXXFG1777330q9fv4wbN261Y2gWAAAAQAlVVVXr7Wd1Jk+enFdeeSUNDQ259NJLc+mllxZtHzlyZMaMGZNbbrklTzzxRF588cXGbT//+c/TqlWrNXrNmgUAAADwGTFx4sT069cvSdK1a9fMnz8/CxcuTJJMnz49rVq1yhZbbJHq6ur07t07EydOTJJMmzYtL774Yvbdd981Gme9rFlQ9/nt1scwq/XS29PKXUKSpGoD6NHs0Hr7cpeQJJnw6oRyl5Akqd2kfblLSJJs3nzzcpeQJHl/6fvlLiFJsv3ny/8+Pa5H+f97TZKFHywsdwlJkksmX1HuEpIko/fZMOr4LKup2TDe2y+9/XK5S0iSbNNy63KXkDO+/O1yl5AkGfXUL8tdQpKk0+Zr9s3Xuvb2B2+Xu4QkyfNzXyh3CUmSPlt/pdwl5MK968tdQpLkDzOfKXcJSZLfvfh0uUtIkny7/H+2rVPV2XDWLJg7d266d+/eeLtNmzaZM2dOWrRokTlz5qRNmzZF26ZPn54kueyyy3LRRRfljjvuWKNxNoy/FAAAAIBPrFAorHafO+64Iz179szWW695c9zVEAAAAOAzora2NnPnzm28PXv27LRv336l22bNmpXa2tpMmDAh06dPz4QJEzJz5sw0b948HTt2zFe+suq0kGYBAAAAlLAmCw+uL7169cqYMWNSX1+fqVOnpra2Ni1atEiSdOrUKQsXLsyMGTPSsWPHPPLII7niiity1FFHNT5+zJgx2WqrrUo2ChLNAgAAAPjM2HXXXdO9e/fU19enqqoqI0aMyLhx49KyZcv0798/F198cc4888wkyeDBg9OlS5e1GkezAAAAAEqo3oCSBUly1llnFd3efvv/f4XJ3XffPQ0NDat87CmnnLJGY1jgEAAAACgiWQAAAAAlVG1Al05cXyQLAAAAgCKSBQAAAFDChnQ1hPVFsgAAAAAoIlkAAAAAJWxoV0NYHyQLAAAAgCKSBQAAAFBCVQV+z77Wr/iuu+7KAQcckCFDhmTChAlNWBIAsDbMzQBAU1mrZMG8efNyzTXX5Pbbb8+iRYsyZsyY7Lvvvk1cGgCwpszNALDuVOKaBWvVLJg4cWL22muvtGjRIi1atMgll1zS1HUBAJ+AuRkAaEprdRrCjBkz8t577+XEE0/MEUcckYkTJzZ1XQDAJ2BuBoB1p6qqar39bCjWeoHDt956Kz/96U/z2muv5eijj84jjzyyQb0wAKg05mYAoKmsVbKgbdu22WWXXdKsWbN07tw5m222Wd58882mrg0AWEPmZgCgKa1Vs2DvvffOpEmTsmzZssybNy+LFi1K69atm7o2AGANmZsBYN2pWo//21Cs1WkIHTp0yIABA3LYYYclSS688MJUV1fedScBYENhbgYAmtJar1lQX1+f+vr6pqwFAPgUzM0AsG5U4qUTfeUAAAAAFFnrZAEAAABUgkq8upBkAQAAAFBEsgAAAABKqK7A79kr7xUDAAAAJUkWAAAAQAnWLAAAAAAqnmQBAAAAlCBZAAAAAFQ8yQIAAAAooTqSBQAAAECFWy/Jgj+/+df1Mcxq7b91v3KXkCT587yp5S4hrZu3K3cJSZKNqjcqdwlJko2bbVzuEpIk27b8YrlLSJI0r25e7hKSJK+9849yl5BNm21a7hKSJA+98ni5S0iSHNh1v3KXQBMZvf8p5S4hSVIoFMpdQpLkhfkvlruE7Nx213KXkCR5b8mScpeQJNmk2efKXUKSpOvmG8bc/KWeu5S7hCTJPxb+vdwl5P2l75e7hCTJQy9tGJ9x+nXpVu4SKoI1CwAAAICKp1kAAAAAFLHAIQAAAJRQ7TQEAAAAoNJJFgAAAEAJVS6dCAAAAFQ6yQIAAAAoobqq8r5nr7xXDAAAAJQkWQAAAAAlVLkaAgAAAFDpJAsAAACgBFdDAAAAACqeZAEAAACUUG3NAgAAAOD/a+9ug7Ss7/2Af292QwksIlAXQQyHmk7xYG1iywtZhVhBTnEmMeZhtzswmWaOJ5mYqXYwI2JGGUUaGLVB4ojHh8SjNdk5BGNepFkfRmds3GgSO6Q4PfUhqQcQYTciuALi4vaF07W34sVDlr12vT4fhxmWG7y+94j8mN/9vf5X1WkWAAAAQAFnFgAAAACVp1kAAAAABZxZAAAAAFSeZQEAAABQx20IAAAAUKBWq97n7NV7xwAAAEAhzQIAAAAo4NGJAAAAQOVpFgAAAEABj04EAAAAKk+zAAAAAArUNAsAAACAqtMsAAAAgAKjPA0BAAAAqDrNAgAAACjgzAIAAACg8jQLAAAAoECtVr3P2av3jgEAAIBCQ9Is+MoZXxmKyxzRvr7esiMkSSaMPrnsCPnPv7u57AhJki99enHZEZIkn2wcW3aEJMneg7vLjpAkeegPPys7QpLkX0z8dNkRctq46WVHSJL85L8/U3aEJMmkhRPKjpAk+ben/buyI4x43fu7y46QJJnyySllR0iS/OPeV8uOkDt331N2hCTJ5edcWnaEJEnP/p6yIyRJRg2TTxNveW5d2RGSJP9jZ/n/r1w0c3bZEZIkT/7tE2VHSJJ84ebPlB2hEobb0xBWr16dzZs3p1arZcWKFTn77LMHXnv66adz6623pqGhIfPmzcvll1+e/fv3Z/ny5fnTn/6Ut99+O9/61rdywQUXFF7DbQgAAAAwQjz77LN55ZVX0tHRkZdffjkrVqxIR0fHwOurVq3KPffckylTpmTJkiVZtGhRXnjhhZx11lm57LLLsn379nz961+3LAAAAICPi66urixYsCBJcsYZZ2TPnj3p7e1NU1NTtm7dmgkTJmTq1KlJkvnz56erqytLly4d+PU7duzIlClHbvZZFgAAAECB4fToxJ6ensye/f7tOJMmTUp3d3eamprS3d2dSZMm1b22devWga/b2try2muvZcOGDUe8zvC4CQsAAAA4Zv39/Uf9c3/yk5/kjjvuyHe+850j/jrLAgAAAChQG8J/jqS5uTk9Pe8fArtr166ccsoph31t586daW5uzpYtW7Jjx44kyZlnnplDhw7l9ddfL7yOZQEAAACMEC0tLens7EySPP/882lubk5TU1OSZPr06ent7c22bdvS19eXJ554Ii0tLfntb3+be++9N8l7tzHs27cvEydOLLyOMwsAAACgwHA6s+Ccc87J7Nmz09bWllqtluuvvz6bNm3K+PHjs3DhwqxcuTLLli1LkixevDgzZ87M1KlTc+2116a9vT0HDhzIddddl1GjirsDlgUAAAAwglx11VV1X8+aNWvg+3PmzKl7lGKSjBkzJrfccssxXcOyAAAAAAqMOoqzBD5unFkAAAAA1NEsAAAAgAK1WvU+Z6/eOwYAAAAKaRYAAABAgVoFzyw4rmXBW2+9lauvvjp79uzJO++8k8svvzznn3/+YGcDAI6S2QwADKbjWhY89NBDmTlzZpYtW5adO3fma1/7Wn75y18OdjYA4CiZzQBw4tRq1WsWHNeZBRMnTswbb7yRJNm7d28mTpw4qKEAgGNjNgMAg+m4mgUXX3xxNm3alIULF2bv3r258847BzsXAHAMzGYAOHGqeGbBcTULHn744UybNi2PPvpo7rvvvtxwww2DnQsAOAZmMwAwmI5rWfDcc8/lvPPOS5LMmjUru3btyqFDhwY1GABw9MxmAGAwHdeyYMaMGdm8eXOSZPv27Rk3blwaGhoGNRgAcPTMZgA4cWq12pB9Gy6O68yC1tbWrFixIkuWLElfX19Wrlw5yLEAgGNhNgMAg+m4lgXjxo3LunXrBjsLAHCczGYAOHFGOeAQAAAAqLrjahYAAABAVQynswSGimYBAAAAUEezAAAAAArUKvg5e/XeMQAAAFBIswAAAAAKOLMAAAAAqDzNAgAAAChQi2YBAAAAUHGaBQAAAFBgVAXPLBiSZcHj2x8dissc0dSxU8uOkCQ5+O7BsiPkP/zlvy87QpJk+1vbyo6QJLlny8ayIyRJ/uZftpYdIUnyzyb8RdkRkiRrnyn/v8vdF11bdoQkyU+/vqrsCEmSfX37yo7AIHlu15ayIyRJTh33WtkRkiRnTvp02REyfvT4siMkSZo+0VR2hCTJX//XW8qOkCT5L1/5J2VHSJKcd9q/KTtCkuTxFx4sO0IuPH1+2RGSJH9zzdayIyRJdvR2lx2BjynNAgAAACjgzAIAAACg8iwLAAAAgDpuQwAAAIACtQoecKhZAAAAANTRLAAAAIACtQp+zl69dwwAAAAU0iwAAACAAs4sAAAAACpPswAAAAAKjIpmAQAAAFBxmgUAAABQwJkFAAAAQOVpFgAAAECBmjMLAAAAgKrTLAAAAIACziwAAAAAKk+zAAAAAArUKvg5e/XeMQAAAFDIsgAAAACo4zYEAAAAKDDKAYcAAABA1WkWAAAAQIFaNAsAAACAitMsAAAAgAI1ZxYAAAAAVadZAAAAAAWcWQAAAABUnmYBAAAAFHBmAQAAAFB5mgUAAABQYFQFP2ev3jsGAAAACmkWAAAAQIEqnlkwJMuC//36H4fiMke06PTFZUdIkvS+s6fsCHm3/92yIyRJxjaOLTtCkmTV3GvKjpAk+bt/+LuyIyRJFpz+ubIjJEn+9alTy46Q3r7esiMkST7/t8Pj9+hP//qGsiMwSB79wz+UHSFJ8t8uvbPsCEmSngOvlU83XRoAAA4ESURBVB0hnxg1uuwISZJxjePLjpAk+Z//6aGyIyRJ3u0/VHaEJMmmP/592RGSJP9xbvl/n545/p+XHSFJcujd4fH36d/seLXsCHxMaRYAAABAgVqq1yxwZgEAAABQx7IAAAAAqOM2BAAAACgw3A44XL16dTZv3pxarZYVK1bk7LPPHnjt6aefzq233pqGhobMmzcvl19+eZJk7dq1+d3vfpe+vr584xvfyEUXXVR4DcsCAAAAGCGeffbZvPLKK+no6MjLL7+cFStWpKOjY+D1VatW5Z577smUKVOyZMmSLFq0KD09PXnxxRfT0dGR3bt354tf/KJlAQAAAPw5htMBh11dXVmwYEGS5IwzzsiePXvS29ubpqambN26NRMmTMjUqe89VWz+/Pnp6upKe3v7QPvgpJNOyv79+3Po0KE0NDR85HWcWQAAAAAjRE9PTyZOnDjw9aRJk9Ld3Z0k6e7uzqRJkz70WkNDQ8aOfe+x9Rs3bsy8efMKFwWJZgEAAAAUGk7Ngg/q7+8/6p/72GOPZePGjbn33nuP+HMtCwAAAGCEaG5uTk9Pz8DXu3btyimnnHLY13bu3Jnm5uYkyVNPPZUNGzbk7rvvzvjx4494HbchAAAAQJFabei+HUFLS0s6OzuTJM8//3yam5vT1NSUJJk+fXp6e3uzbdu29PX15YknnkhLS0vefPPNrF27NnfeeWdOPvnko3rLmgUAAAAwQpxzzjmZPXt22traUqvVcv3112fTpk0ZP358Fi5cmJUrV2bZsmVJksWLF2fmzJkDT0G48sorB/49a9asybRp0z7yOpYFAAAAUGC4nVlw1VVX1X09a9asge/PmTOn7lGKSdLa2prW1tZjuobbEAAAAIA6R7UseOGFF7JgwYI88MADSZIdO3Zk6dKlaW9vzxVXXJGDBw+e0JAAQD2zGQCGTq1WG7Jvw8URlwX79u3LjTfemHPPPXfgx2677ba0t7fnwQcfzIwZM7Jx48YTGhIAeJ/ZDACcaEdcFowePTp33XXXwOMWkuSZZ57JhRdemCS54IIL0tXVdeISAgB1zGYAGFq1IfxnuDjiAYeNjY1pbKz/afv378/o0aOTJJMnT053d/eJSQcAfIjZDACcaH/20xD6+/sHIwcAMEjMZgAYXMPpE/+hclxPQxg7dmwOHDiQJNm5c2ddDRIAGHpmMwAwmI5rWTB37tx0dnYmSR555JGcf/75gxoKADg2ZjMAMJiOeBvCli1bsmbNmmzfvj2NjY3p7OzMzTffnOXLl6ejoyPTpk3LJZdcMhRZAYCYzQAw1IbTIw2HyhGXBWeddVbuv//+D/34D3/4wxMSCAAoZjYDACfan33AIQAAAHycOeAQAAAAqDzNAgAAACigWQAAAABUnmYBAAAAFKji0xA0CwAAAIA6mgUAAABQwJkFAAAAQOVpFgAAAEABZxYAAAAAladZAAAAAAWqeGZBrb+/v/9EX2TX/ldP9CWOypvv7Ck7QpJka+8/lh0hs07+y7IjJEn+9HZP2RGSJIf6+8qOkCQZ2ziu7AhJklM/eVrZEZIkG7bcWXaEzJ8+t+wISZJPNf1F2RGSJKNqDWVHSJKcMmZq2RFGvD0HXy87QpLk4KEDZUdIkpx27cVlR8gfbvxp2RGSJL/Z9ZuyIyRJ5k27oOwISZJP1D5RdoQkScOo4fEZ31/9/TfKjpD/88cdZUdIkjx/1aayIyRJDr77dtkRkiT/dMypZUc4of7XG5uH7FpnnvyvhuxaRYbHnzoAAAAwTFWxWeDMAgAAAKCOZgEAAAAU8DQEAAAAoPIsCwAAAIA6bkMAAACAAg44BAAAACpPswAAAAAKaBYAAAAAladZAAAAAAU8OhEAAACoPM0CAAAAKKRZAAAAAFScZgEAAAAUcGYBAAAAUHmaBQAAAFCg5swCAAAAoOo0CwAAAKCAZgEAAABQeZoFAAAAUMDTEAAAAIDKsywAAAAA6rgNAQAAAAo44BAAAACoPM0CAAAAKKBZAAAAAFSeZgEAAAAU8OhEAAAAoPI0CwAAAKCAMwsAAACAytMsAAAAgALOLAAAAAAqT7MAAAAAClTxzIJaf39/f9khAAAAYLh6dd8rQ3ataWNnDNm1imgWAAAAQKHqNQucWQAAAADU0SwAAACAAtXrFWgWAAAAAB9gWQAAAADUsSwAAACAArVabci+HY3Vq1entbU1bW1t+f3vf1/32tNPP50vf/nLaW1tze233z7w4y+88EIWLFiQBx544KiuYVkAAAAAI8Szzz6bV155JR0dHbnpppty00031b2+atWqrF+/Pj/+8Y/zq1/9Ki+99FL27duXG2+8Meeee+5RX2dELAuKtiZVtHbt2rS2tuZLX/pSHnnkkbLjlO7AgQNZsGBBNm3aVHaU0v385z/P5z//+Vx66aV58skny45Tqrfeeivf/va3s3Tp0rS1teWpp54qO1IpPrhB3rFjR5YuXZr29vZcccUVOXjwYMkJGanM5npmcz2z+X1m8/vM5veYzSNVbQi/Fevq6sqCBQuSJGeccUb27NmT3t7eJMnWrVszYcKETJ06NaNGjcr8+fPT1dWV0aNH56677kpzc/NRv+Nhvyw40takan7961/nxRdfTEdHR+6+++6sXr267Eilu+OOOzJhwoSyY5Ru9+7duf322/Pggw9mw4YNefzxx8uOVKqHHnooM2fOzP33359169ZV8s+Ow22Qb7vttrS3t+fBBx/MjBkzsnHjxhITMlKZzfXM5g8zm99jNtczm81mBkdPT08mTpw48PWkSZPS3d2dJOnu7s6kSZM+9FpjY2PGjBlzTNcZ9suCoq1JFc2ZMyfr1q1Lkpx00knZv39/Dh06VHKq8rz88st56aWX8rnPfa7sKKXr6urKueeem6ampjQ3N+fGG28sO1KpJk6cmDfeeCNJsnfv3ro/UKvicBvkZ555JhdeeGGS5IILLkhXV1dZ8RjBzOZ6ZnM9s/l9ZnM9s9lsHsmGT6/gw/r7+4/zXRUb9suCoq1JFTU0NGTs2LFJko0bN2bevHlpaGgoOVV51qxZk+XLl5cdY1jYtm1bDhw4kG9+85tpb2+v/KC5+OKL8+qrr2bhwoVZsmRJrr766rIjDbnDbZD379+f0aNHJ0kmT55c6T9POX5mcz2zuZ7Z/D6zuZ7ZbDYzOJqbm9PT0zPw9a5du3LKKacc9rWdO3ce060H/79hvyz4oBO1NRlpHnvssWzcuDHXXXdd2VFK87Of/Syf+cxncvrpp5cdZdh444038oMf/CDf+973cs0111T6/5eHH34406ZNy6OPPpr77rsvN9xwQ9mRhp0q//5gcPm99B6z2Ww+HLP5fWbzkVX598fwN3y6BS0tLens7EySPP/882lubk5TU1OSZPr06ent7c22bdvS19eXJ554Ii0tLcf1jhuP61cNoaKtSVU99dRT2bBhQ+6+++6MHz++7DilefLJJ7N169Y8+eSTee211zJ69OiceuqpmTt3btnRSjF58uR89rOfTWNjYz71qU9l3Lhxef311zN58uSyo5Xiueeey3nnnZckmTVrVnbt2pVDhw5V+tO+JBk7dmwOHDiQMWPG/FmbZqrNbP4ws/k9ZnM9s7me2Xx4ZjPH6pxzzsns2bPT1taWWq2W66+/Pps2bcr48eOzcOHCrFy5MsuWLUuSLF68ODNnzsyWLVuyZs2abN++PY2Njens7Mz69etz8sknf+R1hv2yoKWlJevXr09bW9uHtiZV9Oabb2bt2rX50Y9+VPgftgq+//3vD3x//fr1Oe200yr7l5EkOe+887J8+fJcdtll2bNnT/bt21fJewH/nxkzZmTz5s1ZtGhRtm/fnnHjxlX+LyNJMnfu3HR2duYLX/hCHnnkkZx//vllR2IEMpvrmc3vM5vrmc31zObDM5tHhlrteE4TOHGuuuqquq9nzZo18P05c+ako6Oj7vWzzjor999//zFdY9gvCw63NamyX/ziF9m9e3euvPLKgR9bs2ZNpk2bVmIqhoMpU6Zk0aJF+epXv5ok+e53v5tRo0bcnUaDprW1NStWrMiSJUvS19eXlStXlh1pyB1ug3zzzTdn+fLl6ejoyLRp03LJJZeUHZMRyGyuZzbzUczmemaz2czIUut3YwwAAAB8pF0HXh2yazWPGR7L5uquNgEAAIDDGva3IQAAAECZakfxlIKPG80CAAAAoI5mAQAAABTQLAAAAAAqz7IAAAAAqGNZAAAAANSxLAAAAADqOOAQAAAACtRqDjgEAAAAKs6yAAAAAKhjWQAAAADUcWYBAAAAFKjFmQUAAABAxWkWAAAAQCHNAgAAAKDiNAsAAACgQPV6BZoFAAAAwAdoFgAAAECBWq163QLNAgAAAKCOZgEAAAAU0iwAAAAAKs6yAAAAAKjjNgQAAAAoUL2bEDQLAAAAgA/QLAAAAIBC1esWaBYAAAAAdTQLAAAAoECtplkAAAAAVJxlAQAAAFDHsgAAAACo48wCAAAAKFDzNAQAAACg6jQLAAAAoJBmAQAAAFBxmgUAAABQoHq9As0CAAAA4AM0CwAAAKBArVa9boFmAQAAAFDHsgAAAACo4zYEAAAAKOQ2BAAAAKDiNAsAAACgQPV6BZoFAAAAwAdoFgAAAECh6nULNAsAAACAOpoFAAAAUKBW0ywAAAAAKs6yAAAAAKhjWQAAAADUcWYBAAAAFKh5GgIAAABQdbX+/v7+skMAAAAAw4dmAQAAAFDHsgAAAACoY1kAAAAA1LEsAAAAAOpYFgAAAAB1LAsAAACAOv8XU1KOsUg17SIAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "feature = 'SUBJ'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=feature)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=feature)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to SUBJ tokens')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.6 Target objects"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 53,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [06:05<00:00, 2.74it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAsAAAJ1CAYAAAC/wnk8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3X/81fP9P/7bq1daVEu/XkWJlr1C5McWS94ipR/eb5vmR0PY23vkjZnf5PckY+YrzcYbm43xfpH8+mwSoyER9hnTe+/5MayYflApRarz/cOn18VRnX541cnO9brLuVw65/k853E7P+zxOvdzfz6eVYVCoRAAAACA/6dRuQMAAAAAGxfFAgAAAKCIYgEAAABQRLEAAAAAKKJYAAAAABRRLAAAAACKKBYAQAUbM2ZMunXrlmeeeabcUficpk+fnm7duuWcc84pdxQA/gk0LncAAPgiu+KKK/KLX/wiLVq0yJNPPpmmTZuudL/HHnssS5cuTb9+/epv++CDD/KLX/wiRx99dL785S+v96wvv/xynnrqqRxzzDH1tw0aNChf/epX89WvfnW9j78qK3ttGtLMmTNz22235cknn8y0adOyaNGibL755tlxxx1zwAEH5IADDkijRsW/nzzzzDM56qijVnispk2bpmPHjtlzzz3zve99L+3bt1/pmOecc07uueee+utdunTJ+PHjS+bc0J8HAChFsQAA1tHixYtzzz33pFGjRpk/f37Gjx+fb33rWyvd9+abb06nTp2KvhD/+c9/zk9/+tMcdNBBG+TL4fjx43PvvfcWFQu23XbbbLvttut97FJW9to0lPHjx+ecc87J0qVLM2jQoBx66KFp1qxZ3nnnnTzyyCM544wzcuutt+a6665Lu3btVrh/7969c+ihh9Zfnzt3bv74xz/mN7/5TcaPH5977rlnpfc74ogjss8++yRJTjnllDXKuqE/DwBQimIBAKyjCRMmZM6cOTn88MNzxx135M4771xpsWDZsmWZOnVqOnXqVHT7n//85w0VtSzjrYlVvTYN4bnnnstpp52WLbfcMjfddFO22Wabou3f+973cscdd+SSSy7J8OHDc8cdd6RJkyZF+3Tq1CkDBw4sum3o0KHp0KFDbrjhhtxzzz057rjjVhh7p512yk477bRWeTfG9weAymXNAgBYR3V1dUmSo48+Ol/72tfy/PPP57XXXivaZ9y4cdl+++2zcOHC3HPPPenWrVvGjBmTvn375qqrrkqS7LfffunWrVvR/caOHZuDDz44O++8c3bdddccdNBBufXWW7Ns2bL6fZYfo37eeefllVdeyX/8x3/k61//enr06JEjjjgiL730UtF+jz/+eN56661069Ytw4YNS7LqNQsef/zxHH300fn617+eHXfcMX379s3IkSPz3nvvFe3Xt2/f9O/fP/Pnz8/555+f3r17Z8cdd8ygQYPywAMPlHz9VvXaLPfaa6/ltNNOS+/evdO9e/fsueee+f73v5+//vWvq31vkuTyyy9PoVDItddeu0KhYLnvfOc79a/VuHHj1uhxk2S33XZLkrz77rtrfJ9SSn0eli1bll//+tf51re+lZ133jk777xzDjzwwNx8881ZsmTJah/7xRdfzC677JJvfvObmT9/fv3tjzzySI488sjsuuuu6dGjRwYPHpzrrrsuH330UdH9u3XrlmOOOSYzZszIKaeckj322CM77rhjhgwZkieffLJo39mzZ+fyyy/PgAEDsvPOO2f33XfPYYcdVnRIBgBfDIoFALAOXn/99UyZMiW77rprttlmm/qOgrvuuqtovz322CMXXXRRkmT33XfP6NGjM2jQoFx00UXZfffdkyQXXXRRRo8eXX+fH/3oRznvvPNSU1OT888/P2eddVbatWuXkSNH5oILLlghy8yZM/Pd7343Xbt2zXnnnZcjjjgif/rTnzJ8+PAsXrw4bdq0yejRo9O6deu0bt06o0ePzsknn7zK57b81/JZs2Zl+PDh+eEPf5h99903d9xxRw4//PAsXLiwaP9ly5bluOOOy/vvv59TTz01P/jBDzJ37tycddZZmTp16irHWdVrk3yyvsKhhx6aSZMm5eCDD85ll12Www8/PM8++2yGDh2av/zlL6t83CR59dVX89JLL2XPPffMDjvsUHLfY489Nkly7733ltzv01555ZUkyfbbb7/G9yml1Ofh/PPPz2WXXZa2bdvmrLPOyrnnnpuOHTvmyiuvXO1ihtOnT88JJ5yQNm3a5KabbkqLFi2SJLfeemtOPPHEVFVV5ayzzsoFF1yQ7bffPmPGjMkJJ5yQQqFQ9DiLFi3KUUcdlebNm+fss8/O8ccfn9dffz0nnXRSZs6cmSRZunRpjjnmmNx+++3p379/fvjDH+b000/PpptumnPOOSe//vWvG+S1AmADKQAAa+1HP/pRoba2tnDnnXcWCoVCYf78+YWdd965sMceexQ++uijon2nTZtWqK2tLZx99tlFt5999tmF2trawrRp0+pv+8tf/lKora0tXHzxxSuMefLJJxdqa2sLU6dOLXrc2trawu9+97uifc8999xCbW1t4amnnqq/bd999y3su+++Rftde+21hdra2sLTTz9dKBQKhUWLFhV69uxZ6NWrV2HevHlF+954442F2trawg033FD0mCvLe8899xRqa2sL11xzzUpevdW/Nscee2yhtra28Mc//rHo9hdeeKFQW1tbOPbYY0s+7tixYwu1tbWF66+/vuR+y/Xr16+www47FD7++ONCoVAoPP3004Xa2trCOeecU5g3b1795a233ircdttthV122aUwbNiwwuLFi1f72LW1tYUBAwasdr+VfR7+9Kc/FWprawv//u//Xli2bFnR/scdd1yhtra28Kc//alQKKz4Ws6bN68waNCgQq9evQpvvPFG/f1mzZpV2GmnnQrHH3/8Co+5/HM9YcKEovy1tbWFG2+8sWjfn/70p4Xa2trCXXfdVSgUCoWpU6cWamtrC5dccknRfsuWLSucdtpphSuuuGK1rwEAGw+dBQCwlpYvbLjpppvW/xLevHnz7L///pkzZ04eeeSRdX7sBx98MEkyePDgvP/++0WXAQMGJEmmTJlSdJ8OHTrU51hu+fHys2bNWqvxp0yZknnz5mXw4MErLLI3ZMiQJMnEiRNXuN93v/vdBhk/SRYuXJhJkyalW7du2XXXXYu29ejRI7W1tZk8efIK7fKftvzwgJqamjUas0OHDlmyZEnmzp1bdPu4cePSs2fP+su+++6bH/7wh/mXf/mXXHPNNdlkk03W8tmtnYcffjjJJ+skVFVVFW1b/n489thjK9xv8eLFOfHEE/POO+/kv/7rv7L11lvXb3v00Ufz0UcfZdCgQZk/f37RZ6x///5JssJhKY0bN17h7BCffY+rq6uTJP/zP/+TRYsW1e9XVVWVn/zkJznrrLPW/gUAoGwscAgAa2n5woYHHnhgmjdvXn/7kCFDct999+XOO+/M4MGD1+mxX3311STJkUceucp93n777aLrnTt3XmGfL33pS0myRse0f9rf/va3JEltbe0K21q3bp3NN988b7zxRtHt1dXV6dixY4OMnyRvvvlmli1btsrTOXbp0iUvv/xypk+fnq5du650n+VfrD+9xkMphf/Xdv/ZUyjuu+++RYWQRYsW5fXXX09dXV0GDhyYK6+8sv6sB+vD8vdjZa9Fly5dkmSF9yNJRowYkSlTpuT666/PjjvuWLRt+Wes1Jf3f/zjH0XX27dvv8Lij599j7t165b9998/EyZMyL777pu+ffumV69e6d27d1q3bl3qaQKwEVIsAIC1tHxhw9133z1vvvlm/e0dOnRI27Zt8/TTT2fatGnZaqut1vqxP/jggyTJ1VdfnbZt2650n8+equ+zX+I+j+XrEWy66aYr3d60adO8//77RbdVV1fX/6rcEJa/BpttttlKty//kvrZtRM+bXlHwWe/9K7KP/7xj3zpS19Ky5YtV3icPfbYo+i2ffbZJ4cccki+/e1v58wzz8zDDz+czTfffI3GWVul3o+mTZsmSdGv+Mknxazlr+Hjjz+efffdt2j78m3nnnvuKtdc+OzrsKafsauvvjr33HNPxo4dm3HjxuXuu+9O48aNM2jQoFx44YVOCQnwBaJYAABr4W9/+1v9YQDnn3/+KvcbO3ZsTj311LV+/GbNmiVJttpqq/To0WPdQn4Oy7+gr+qL+KJFi+ozri/LH79Uhk/vtzLLz1bw1FNP5aSTTio53ltvvZXp06fnG9/4xhoXPZo3b55Bgwbl5z//eV544YX06dOn5P6fPYRgTZV6P5bf9tnXYcmSJbn66qtz77335vbbb89ee+2V/fbbr3778v1XVgj5vDbZZJMceuihOfTQQ/Pee+/lySefTF1dXR544IHMnj07t9xyS4OOB8D6Y80CAFgLd955Z5LkkEMOyejRo1e4XHnllamurs7dd9+9Ti342267bZLkj3/84wrbPvjgg5LH6TeE5eO//PLLK2ybNWtW5s2bl6985SvrNcM222yT6urqlWZIPmmjb9KkSTp16rTKx9hqq62y++675/nnn8+zzz5bcrxf/vKXST55T9fG8vf3ww8/zLRp03LqqafmZz/7WdE+y081+dlf6tdUqfdj+Wk6P/t+DB48OAcccEAuv/zytGnTJuedd15mzJhRv335IQ0r+4wtXrw4CxYsWKesn9W6desceOCBufXWW9O9e/dMnjy56NSNAGzcFAsAYA0tX9iwSZMmOe200zJw4MAVLt/85jfTr1+/zJo1q34hwOW/Vn/2i/7y4+M/ffvyhQrvuOOOfPjhh0X7//jHP843vvGN/P3vf1+n/I0aNVptsWH33XdP69at87vf/S7z5s0r2rb88IvlCy02hJW9Nptuumn69OmTl19+Oc8991zR/lOmTMnrr7+efffdd7Wt8SNGjEiTJk1y6qmn5n//939Xus+dd96Z3/zmN9ljjz3Wap2J+fPn58EHH0x1dXV22WWXtGvXLhMnTsztt99e9IV4+Wdg+WkRS1nZ52H5a11XV1d0OsNCoVBfuNp///1X+nht27bNqFGjMmfOnJx99tn19+/bt2+aNGmSBx54oH4hyOV+9atfpVevXqstsKzMnXfemb333ru+iPHp59WsWbNUV1evsCYEABsvhyEAwBp66KGHMnfu3AwZMqTkgm1HHnlkHnroodx1113p169f2rZtm6ZNm+aJJ57IDTfckK233joDBw6s/2X8xz/+cXr27JlvfvOb2W677XL00UfnV7/6Vb7zne/ksMMOS+PGjfOHP/whEyZMyIEHHrjSBQ3XRKdOnTJ58uRcfvnl2WKLLXLMMcessE+TJk1ywQUX5LTTTssRRxyRgw8+OC1atMgLL7yQu+66K7vsssta/wJfyqpem7POOivPPfdcTjzxxAwbNiydOnXKG2+8kdtvvz2tWrXKmWeeudrH3n777fOzn/0sp556ag4++OAMGDAgPXv2TLNmzTJz5sz8/ve/z/PPP59evXpl9OjRK/0iO3369IwfP77++scff5y33nor99xzT6ZPn54zzzwz7du3T5KcdNJJufLKK3PUUUfl8MMPz8yZM3PTTTelTZs2GTZs2Grzruzz0L179xx++OG5/fbbM3z48PTt2zdLlizJo48+mqeffjrf/e53V7oY5XL77LNP/f1vvPHGHHfccWnTpk3OOOOMjBo1KkOHDs2wYcPSokWLTJkyJffee2++/vWvr3AWijWxxx571D//oUOHZuutt87ixYvz5JNPZsqUKTnkkEPW+yEsADQcxQIAWEPLf1k/+uijS+63++67p7a2Nk888UTeeeeddOjQIeecc06uueaa/OxnP8uhhx6agQMHZujQoXnyySfz5JNP5qWXXqr/hXjEiBH56le/mrq6ulx++eVZtmxZttlmm5x55pkr/YK/pk455ZS89dZb+c1vfpNu3bqt8rEGDx6cli1b5oYbbsi1116bxYsXp2PHjjnuuONy/PHHN+iCiptssslKX5suXbrkzjvvzLXXXpvbb7898+bNS6tWrdK3b9+ceOKJa7x45L/8y79kwoQJue222zJx4sQ8/vjjWbRoUTbffPP06NEj1157bfbff/9VrikwadKkTJo0qShv+/btU1tbmwsuuCB77bVX/bZjjz02rVu3zq233ppRo0alcePG2WuvvXLGGWessCjlyqzq83DhhRema9euufPOO3PZZZelUaNG2XbbbTNy5Mg1KtycffbZmTJlSq699tr06tUrO+20U44++uh07Ngxt9xyS6655poV3uPGjdf+T8Stt946dXV1ueGGGzJ27Ni8++67ady4cb7yla9kxIgRJc/wAcDGp6rw6Z42AAAAoOI5cAwAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIooFAAAAQBHFAgAAAKCIYgEAAABQRLEAAAAAKKJYAAAAABRRLAAAAACKKBYAAAAARRQLAAAAgCKKBQAAAEARxQIAAACgiGIBAAAAUESxAAAAACiiWAAAAAAUUSwAAAAAiigWAAAAAEUUCwAAAIAiigUAAABAEcUCAAAAoIhiAQAAAFBEsQAAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIooFAAAAQBHFAgAAAKCIYgEAAABQRLEAAAAAKKJYAAAAABRRLAAAAACKKBYAAAAARRQLAAAAgCKKBQAAAEARxQIAAACgiGIBAAAAUESxAAAAACiiWAAAAAAUUSwAAAAAiigWAAAAAEUUCwAAAIAiigUAAABAEcUCAAAAoIhiAQAAAFBEsQAAAAAoolgAAAAAFFEsAAAAAIooFgAAAABFFAsAAACAIooFAAAAQBHFAgAAAKBI43IHAAAAgI1ZVf9OG2yswsPTN9hYpegsAAAAAIooFgAAAABFHIYAAAAApVRVlTvBBqezAAAAACiiswAAAABKqcCf2SvwKQMAAACl6CwAAACAUqxZAAAAAFQ6nQUAAABQSuU1FugsAAAAAIrpLAAAAIBSrFkAAAAAVDqdBQAAAFBKBf7MXoFPGQAAAChFsQAAAAAo4jAEAAAAKMUChwAAAECl01kAAAAApVReY4HOAgAAAKCYzgIAAAAopdHG1VowatSovPDCC6mqqsqIESPSo0eP+m1PPfVUrr766lRXV2fvvffOiSeemLvuuiv3339//T4vvfRS/u///b8lx1AsAAAAgC+IKVOm5M0330xdXV1ee+21jBgxInV1dfXbR44cmZtvvjnt27fPkUcemQEDBuSQQw7JIYccUn//Bx98cLXjOAwBAAAASqnagJfVmDx5cvr165ck6dq1a+bNm5cFCxYkSaZNm5aWLVtmiy22SKNGjdKnT59Mnjy56P7XXXdd/vM//3O14ygWAAAAwBfE7Nmz06pVq/rrrVu3zqxZs5Iks2bNSuvWrVe6LUlefPHFbLHFFmnXrt1qx3EYAgAAAJRStXGtWfBphUJhjfcdO3ZsDjrooDXaV2cBAAAAfEHU1NRk9uzZ9ddnzpxZ3ynw2W0zZsxITU1N/fVnnnkmu+666xqNo1gAAAAApWxEaxb07t07Dz30UJJk6tSpqampSfPmzZMknTp1yoIFCzJ9+vQsWbIkjz32WHr37p3kk8JBs2bN0qRJkzV6yg5DAAAAgC+I3XbbLd27d8/QoUNTVVWViy66KOPGjUuLFi3Sv3//XHzxxTn99NOTJIMHD06XLl2SrLiewepUFdbmAAcAAACoMFWHdt1gYxXufG2DjVWKwxAAAACAIg5DAAAAgFI23pMhrDc6CwAAAIAiigUAAABAEYchAAAAQClVlXccgs4CAAAAoIjOAgAAACilkc4CAAAAoMLpLAAAAIBSKq+xQGcBAAAAUExnAQAAAJTibAgAAABApdNZAAAAAKVUXmOBzgIAAACgmM4CAAAAKKVR5bUW6CwAAAAAiugsAAAAgFIqr7FAZwEAAABQTGcBAAAAlFJVea0FOgsAAACAIooFAAAAQBGHIQAAAEApFfgzewU+ZQAAAKAUnQUAAABQigUOAQAAgEqnswAAAABKqbzGAp0FsDLdunVL//79M3DgwAwYMCD/8R//kWnTppU71jrp379/nnnmmZL7nHPOOfnZz362gRIBUKk+Pb8uvxx77LFJkqOPPjpTp05tsLGeeOKJvP322w32eKWcd955GTNmzAYZa4cddsj06dNL7jNs2LDcd999GyQP8M9LZwGswq233poOHTokSX7yk5/ksssuy/XXX1/mVADwxfbp+fXTfvWrXzXoOLfccktOOOGEbLnllg36uECFqsA1CxQLYA184xvfyKOPPlp//a677sovfvGLLF26NO3atcuVV16Zjh07ZsaMGTnrrLMya9asLF68OAcccEBOPfXUFAqFXHfddXnggQeyePHi7Lfffjn33HNTXV1dNM64ceMyceLEbLLJJnn++efTpUuXnHjiibnqqqvy97//PaecckoOO+ywLFu2LKNHj85DDz2UJNlll11y4YUXZrPNNstLL72Us88+O0uWLEmfPn2KHv+RRx7J6NGjs3Dhwmy99da56qqr0rp16/X/AgLAavTt2zdXXnllOnTokKFDh+a4447LXXfdlblz5+bcc8/N4MGD13g+veaaa/L000/nb3/7W84888w8/vjj6dy5c/7zP/8zyScddcuv9+3bN8cdd1zGjh2bd955J//6r/+ac845J8mq5805c+bk9NNPzxtvvJFtt902TZs2XWkBZMyYMZk9e3beeeedTJ06Nb169crgwYMzZsyYzJw5M5deemn23XfffPTRR7nsssvyzDPPpFGjRunTp0/OPPPMVFdX5w9/+ENGjhyZxo0b59vf/nbR49fV1eWXv/xlFi9enF122SWjRo1K06ZN19M7BFQahyHAaixevDj3339/+vbtmyR5991388Mf/jC//OUvM2HChHTu3Lm+hf+WW25Jz54987vf/S4PPPBApk2blpkzZ+a+++7L+PHjM3bs2Dz88MOZNm1a7rjjjpWO98QTT+Skk07KhAkT8tprr+Wmm27Kb37zm1x22WX14zz44IN5/PHHM27cuPz2t7/N+++/n1tuuSVJcvHFF+eoo47KQw89lF133bW+VXHatGk566yz8pOf/CS///3vs8cee+Tiiy9evy8eAKyDOXPmpFGjRnnggQcyYsSIXHPNNUmyxvPpD37wg7Rv3z4//vGPM3jw4NWO9+yzz6auri533313brvttrzzzjsl580bb7wxrVq1yqOPPpoLL7wwTz755Cofe+LEiRk1alQeeOCBjB8/vn7+Hj58eG688cYkn3RVvPPOO/ntb3+be+65J88991z+z//5P1m6dGnOO++8XHTRRXnwwQfTqFGjLF26NEny3HPPZfTo0fnVr36VRx99NM2bN8/o0aPX9qUG1lSjDXjZSGxEUWDjMmzYsAwcODC9e/fOn//85wwZMiRJ0qZNmzz//PP1vyB8/etfr1/PoE2bNnnyySfz3HPPpUmTJrn66qtTU1OTxx57LN/+9rfTokWLNG7cOIccckgmTJiw0nG33XbbdOnSJU2aNMnWW2+dvfbaK9XV1amtrc3MmTOTfPKHx7e+9a1sttlmqa6uzpAhQzJp0qR89NFH+fOf/1z/h9HAgQOz6aabJkkef/zx7L777qmtrU2SDB06NI8++mj9Hx0AsCEsn1+XX84///wV9lmyZEn9vNu9e/f6tQfWZj5dG//2b/+W6urqtG/fPm3atMk//vGPkvPmc889l0GDBiVJOnXqlN13332Vj73rrrumTZs2adWqVdq1a5e99947SVaY1w899NA0btw4TZs2zb/9279l0qRJeeONN7J48eLstddeSZKDDjqo/nEfffTRDB48OO3bt0+SfOc732mQ1wJgOYchwCp8+pjKZ599NsOGDcu4cePSpk2bXHvttfV/MHzwwQfp0qVLkuSYY47JsmXLcskll2TmzJk54ogjcvLJJ2f+/Pm5+eabU1dXlyRZunTpKtv/mzVrVv/v6urqbLbZZvX/XrZsWZLkvffeS8uWLev3a9myZd59993MnTs3SdK8efMkSVVVVb785S8nSebPn5/nnnsuAwcOrL9f8+bN6+8DABvCqtYs+LRPz3+NGjWqn/9WNZ/OmDEjRx99dJKkR48eufLKK9cq0/J5c/nYS5cuLTlvzps3Ly1atKi/fflcuzKrmtc//bxWNa/PmzevKNun95k/f34efvjh+q6GQqGQjz/+eK2eN7AWrFkArEzPnj2z5ZZb5vnnn8+SJUvy6KOP5rbbbkvr1q1z55135oEHHkiSNG7cOMcdd1yOO+64vP766/ne976Xr33ta6mpqUnfvn1z5JFHNkietm3bFn3Jnzt3btq2bVv/R8SCBQvSokWLLFu2LPPmzUuS1NTUZM8998y1117bIBkAYEMrNZ+OHz9+lff79BfzJPVz4+rGWtW8+eUvfznz58+vv/7ee+9lq622Wu1jrkqpeX3BggVF43w630EHHZSzzz57nccFKMVhCLAGXn/99bz++uv5yle+knfffTcdO3asX+DowQcfzAcffJAkufDCCzNp0qQkSefOndO2bdtUVVVlv/32y3333ZdFixYlSf77v/8799xzzzrn2WeffXL//fdn0aJFWbJkScaOHZs+ffqkadOm2W677fLwww8nSX7729/mo48+SpLstddeee655+oPmXjxxRczcuTIdc4AABva2synjRs3rv9C365du/zv//5vkk/W8PnjH/+42rFKzZu77LJLHnnkkSTJ3//+9zz//POf63nts88+GTt2bJYuXZqFCxfmvvvuS58+fdK5c+dUV1fXnwJ53Lhxqfp/v2727ds3EyZMqC8gPPLII/mv//qvz5UDKKFqA142EjoLYBWGDRtWv7pykyZNcskll6Rbt25p06ZNfvvb36Z///7Zaqut8oMf/CAnnHBCfvSjH2Xo0KG58MILc+mll6ZQKKRv377p1atXkuSVV16pP9awc+fOueyyy9Y528CBA/PXv/41Q4YMSaFQyB577JGjjjoqyScLHI4YMSI33HBUiZcsAAAgAElEQVRD9t5773Tt2jXJJ79AXHrppTnxxBPz8ccfp1mzZhkxYsTneYkAYIPq16/fGs+nAwYMyGmnnZbvf//7OfTQQ3PSSSdl//33zw477JABAwasdqxS8+bxxx+fU089NX379k3Xrl2z//77f67nNWzYsEybNi0HHHBAqqqqMnDgwAwaNChVVVW59NJLM2LEiDRp0iRDhgypP4yhe/fuGT58eIYNG5Zly5alTZs2ueSSSz5XDoBPqyoUCoVyhwAAAICNVdVJO26wsQo/fWmDjVWKwxAAAACAIooFAAAAQBFrFgAAAEApFXjqRJ0FAAAAQJEN0lnwo+dHbYhhVuvXT00qd4QkyaUHrHhu4A1t/OtPlztCkqTL5jXljpAk+ducGeWOkCSZ++GH5Y6QJNmubftyR0iSPPPW38sdIV/bYstyR0iSNKnepNwRkiSzF85f/U4bwM/2/f/KHeEL78jx/1nuCEmSF1+fXu4ISZKrDxxe7gh5Yvoz5Y6QJFm8bEm5IyRJZi9cUO4ISZK/zppd7ghJkm/v8LVyR0iS3PjEo+WOkEG7dC93hCTJpo2/VO4ISZI2m25e7ghJklN6nFbuCOtX5TUW6CwAAAAAilmzAAAAAEqosmYBAAAAUOl0FgAAAEAJOgsAAACAiqezAAAAAEqowMYCnQUAAABAMZ0FAAAAUEKjCmwt0FkAAAAAFNFZAAAAACU4GwIAAABQ8XQWAAAAQAk6CwAAAICKt86dBaNGjcoLL7yQqqqqjBgxIj169GjIXADAWjI3AwANZZ2KBVOmTMmbb76Zurq6vPbaaxkxYkTq6uoaOhsAsIbMzQCw/jgMYQ1Nnjw5/fr1S5J07do18+bNy4IFCxo0GACw5szNAEBDWqdiwezZs9OqVav6661bt86sWbMaLBQAsHbMzQCw/lRVbbjLxqJBFjgsFAoN8TAAQAMxNwMAn8c6rVlQU1OT2bNn11+fOXNm2rVr12ChAIC1Y24GgPXHmgVrqHfv3nnooYeSJFOnTk1NTU2aN2/eoMEAgDVnbgYAGtI6dRbstttu6d69e4YOHZqqqqpcdNFFDZ0LAFgL5mYAWH8qsbNgnYoFSXLGGWc0ZA4A4HMyNwMADWWdiwUAAABQCapSeZ0FDXI2BAAAAOCfh84CAAAAKKES1yzQWQAAAAAUUSwAAACAEqqqNtxlTYwaNSqHHXZYhg4dmhdffLFo21NPPZWDDz44hx12WK677rr62++///4ceOCBGTJkSCZOnLjaMRyGAAAAAF8QU6ZMyZtvvpm6urq89tprGTFiROrq6uq3jxw5MjfffHPat2+fI488MgMGDEibNm1y3XXX5e67787ChQszZsyY7LPPPiXHUSwAAACAEhptRGsWTJ48Of369UuSdO3aNfPmzcuCBQvSvHnzTJs2LS1btswWW2yRJOnTp08mT56cNm3apFevXmnevHmaN2+eSy+9dLXjOAwBAAAAviBmz56dVq1a1V9v3bp1Zs2alSSZNWtWWrduvcK26dOn58MPP8zw4cNz+OGHZ/LkyasdR2cBAAAAlLAxnw2hUCis0X5z587NT3/607z99ts56qij8thjj5V8XjoLAAAA4AuipqYms2fPrr8+c+bMtGvXbqXbZsyYkZqamrRp0ya77rprGjdunM6dO6dZs2Z57733So6jWAAAAABfEL17985DDz2UJJk6dWpqamrSvHnzJEmnTp2yYMGCTJ8+PUuWLMljjz2W3r17Z6+99srTTz+dZcuWZc6cOVm4cGHRoQwrU1VY056Fz+HpmX9Y30OskQ3wVNfIWx+8Xe4I+WrLbcsdIUnyp9kvlDtCkqRbq9pyR0iS7NRq13JHSJI88tb4ckdIkjTbpHm5I2SzxpuWO0KSZOp7fyl3hCTJG3PL//9fSXJZr5HljvCF98CbY8sdIUny/uL55Y6QJHl/8YJyR0jNpm3LHSFJMvXdV8odIUnSvc1Xyx0hSfKl6i+VO0KS5L2PSv8CuKEsWLyw3BHSvMlm5Y6QJGnSqEm5IyRJ7v7rpHJHSJLcdcAN5Y6wXrW5YM8NNta7lz612n2uuuqqPPfcc6mqqspFF12U//mf/0mLFi3Sv3//PPvss7nqqquSJPvvv3+OPfbYJMl///d/Z+zYT+b/E044Ifvtt1/JMaxZAAAAAF8gZ5xxRtH17bbbrv7fPXv2LDqV4nJDhw7N0KFD13gMxQIAAAAoYSNe33C9sWYBAAAAUERnAQAAAJSwMZ86cX3RWQAAAAAU0VkAAAAAJegsAAAAACqezgIAAAAoQWcBAAAAUPF0FgAAAEAJOgsAAACAiqezAAAAAEqowMYCnQUAAABAMZ0FAAAAUII1CwAAAICKp1gAAAAAFHEYAgAAAJTgMAQAAACg4uksAAAAgBIa6SwAAAAAKp3OAgAAACihAhsLdBYAAAAAxXQWAAAAQAnOhgAAAABUPJ0FAAAAUEJVdBYAAAAAFU5nAQAAAJRgzQIAAACg4uksAAAAgBJ0FgAAAAAVT2cBAAAAlFCBjQU6CwAAAIBiigUAAABAkQ1yGMIrc1/dEMOsVqfmHcsdIUnSu8Ne5Y6Qv72/cbwne3bYs9wRkiTzFs8td4Qkye2v3lbuCEmSbVpsXe4ISZIPl3xY7giZuXBmuSMkSaqrqssdIUnytQ47ljsCDeTVuW+UO0KSZItmNeWOkCTZpe1O5Y6QF2e/VO4ISZKd221X7ghJkqbVTcsdIUny8JtPljtCkqRji43jv5Wd2u5Q7gj58+z/KXeEJMmEv00td4QkSccWLcodoSJY4BAAAACoeBY4BAAAgBJ0FgAAAAAVT2cBAAAAlKCzAAAAAKh4OgsAAACghApsLNBZAAAAABTTWQAAAAAlWLMAAAAAqHg6CwAAAKAEnQUAAABAxdNZAAAAACXoLAAAAAAq3jp3Flx55ZV5/vnns2TJkhx//PHZf//9GzIXALCWzM0AsH5UYGPBuhULnn766bzyyiupq6vLnDlzctBBB/mDBADKyNwMADSkdSoW9OzZMz169EiSfPnLX86iRYuydOnSVFdXN2g4AGDNmJsBgIa0TsWC6urqbLbZZkmSsWPHZu+99/bHCACUkbkZANafSlzg8HOdDeGRRx7J2LFj84tf/KKh8gAAn4O5GQBoCOtcLHjiiSdy/fXX56abbkqLFi0aMhMAsA7MzQCwfugsWEPz58/PlVdemVtuuSWbb755Q2cCANaSuRkAaEjrVCz43e9+lzlz5uQHP/hB/W1XXHFFttxyywYLBgCsOXMzAKw/OgvW0GGHHZbDDjusobMAAOvI3AwANKTPtcAhAAAA/LOrwMaCNCp3AAAAAGDjorMAAAAASqjENQt0FgAAAABFdBYAAABAKToLAAAAgEqnswAAAABKsGYBAAAAUPF0FgAAAEAJFdhYoLMAAAAAKLZBOguqq6o3xDCrtbSwrNwRkiTVVeVv6Fi09MNyR0iSvPjui+WOkCSZsXBWuSMkSbb+8lbljpAkadu0TbkjJEmenvFsuSPkoyWLyx0hSfLYm38pd4QkyX/sPLDcEWggG8tn++U5b5Q7QpKkdvOvljvCRvN3SotNWpQ7QpLkwTcmljtCkmTvTj3LHSFJ0rpp63JHSJJMentKuSPk/Y8+KHeEJMmEPzxf7ghJkptPOKncEfgnVf5vrQAAALARq8QFDhULAAAA4Atk1KhReeGFF1JVVZURI0akR48e9dueeuqpXH311amurs7ee++dE088Mc8880xOOeWUfPWrn3TS1dbW5oILLig5hmIBAAAAlLAxdRZMmTIlb775Zurq6vLaa69lxIgRqaurq98+cuTI3HzzzWnfvn2OPPLIDBgwIEmy++6759prr13jcSxwCAAAAF8QkydPTr9+/ZIkXbt2zbx587JgwYIkybRp09KyZctsscUWadSoUfr06ZPJkyev0ziKBQAAAFBCVVXVBruszuzZs9OqVav6661bt86sWZ8s2D5r1qy0bt16pdteffXVDB8+PN/5zncyadKk1Y7jMAQAAAD4gioUCqvdZ5tttslJJ52UQYMGZdq0aTnqqKMyYcKENGnSZJX30VkAAAAAJVRVbbjL6tTU1GT27Nn112fOnJl27dqtdNuMGTNSU1OT9u3bZ/Dgwamqqkrnzp3Ttm3bzJgxo+Q4igUAAADwBdG7d+889NBDSZKpU6empqYmzZs3T5J06tQpCxYsyPTp07NkyZI89thj6d27d+6///7cfPPNST45VOHdd99N+/btS47jMAQAAAAoYWM6G8Juu+2W7t27Z+jQoamqqspFF12UcePGpUWLFunfv38uvvjinH766UmSwYMHp0uXLmnXrl3OOOOM/P73v8/HH3+ciy++uOQhCIliAQAAAHyhnHHGGUXXt9tuu/p/9+zZs+hUiknSvHnzXH/99Ws1hmIBAAAAlLAxdRZsKNYsAAAAAIroLAAAAIASdBYAAAAAFU9nAQAAAJSgswAAAACoeIoFAAAAQBGHIQAAAEAJFXgUgs4CAAAAoJjOAgAAACjBAocAAABAxdNZAAAAACXoLAAAAAAqns4CAAAAKEFnAQAAAFDxdBYAAABACRXYWKCzAAAAACimswAAAABKsGYBAAAAUPF0FgAAAEApOgsAAACASqezAAAAAEqwZgEAAABQ8TZIZ8EHSxZuiGFWa9JbU8odIUmyW9uvlztCliz7uNwRkiSNG1WXO0KS5KOli8sdIUny7ofvljtCkqRt0zbljpAkGdh5/3JHyLQF08odIUnSpHqTckdIkixeunH8fwefX+tNW5Y7QpKkumrjmAc2BtVVG8dvOPMWv1/uCEmSv8+bV+4ISZIOtR3KHSFJMnPhrHJHSJL03epfyh0hMxbOLHeEJEnd93cod4QkyYKPF5Q7Av+kHIYAAAAAJTSqvKMQHIYAAAAAFNNZAAAAACVY4BAAAACoeDoLAAAAoIRGOgsAAACASqezAAAAAEqwZgEAAABQ8XQWAAAAQAmV+Ct7JT5nAAAAoASdBQAAAFCCsyEAAAAAFU9nAQAAAJTgbAhr6cMPP0y/fv0ybty4hsoDAHwO5mYAoCF8rs6Cn//852nZsmVDZQEAPidzMwA0PGsWrIXXXnstr776avbZZ58GjAMArCtzMwDQUNa5WHDFFVfknHPOacgsAMDnYG4GABrKOh2GcO+992aXXXbJVltt1dB5AIB1YG4GgPWnEhc4XKdiwcSJEzNt2rRMnDgx77zzTpo0aZIOHTpkzz33bOh8AMAaMDcDAA1pnYoF11xzTf2/x4wZk44dO/pjBADKyNwMAOvP5zqN4BdUJT5nAAAAoITPderEJDn55JMbIgcA0EDMzQDQsJw6EQAAAKh4n7uzAAAAAP6ZVeLZEHQWAAAAAEV0FgAAAEAJ1iwAAAAAKp7OAgAAACih8voKdBYAAAAAn6GzAAAAAEqwZgEAAABQ8XQWAAAAQAk6CwAAAICKt0E6C5pv0mxDDLNax+xwRLkjJElmfziz3BGytLCs3BGSJK2+1LrcEZIke3RoV+4ISZKFSxaVO0KSZPaH75Y7QpKNI8f4158sd4QkyVZf3jg+o4ds0afcEWggP3nogXJHSJKc3G9QuSMkSeZ+NLfcEfLCzNfKHSFJMvgrW5Y7QpLkK63alDtCkuT9xfPLHSFJMnPRrHJHSJKMe/mxckdI08YbR3P0C2+/U+4ISZJ/383czPqxcfyXBgAAABupKochAAAAAJVOZwEAAACUYIFDAAAAoOLpLAAAAIASKq+vQGcBAAAA8Bk6CwAAAKAEaxYAAAAAFU9nAQAAAJSgswAAAADYqI0aNSqHHXZYhg4dmhdffLFo21NPPZWDDz44hx12WK677rqibR9++GH69euXcePGrXYMxQIAAAAooaqqaoNdVmfKlCl58803U1dXl8suuyyXXXZZ0faRI0dmzJgxueOOOzJp0qS8+uqr9dt+/vOfp2XLlmv0nBULAAAA4Ati8uTJ6devX5Kka9eumTdvXhYsWJAkmTZtWlq2bJktttgijRo1Sp8+fTJ58uQkyWuvvZZXX301++yzzxqNo1gAAAAAJTSqqtpgl9WZPXt2WrVqVX+9devWmTVrVpJk1qxZad269Uq3XXHFFTnnnHPW/Dmv8Z4AAADARqVQKKx2n3vvvTe77LJLttpqqzV+XGdDAAAAgBI2pnMh1NTUZPbs2fXXZ86cmXbt2q1024wZM1JTU5OJEydm2rRpmThxYt555500adIkHTp0yJ577rnKcRQLAAAA4Auid+/eGTNmTIYOHZqpU6empqYmzZs3T5J06tQpCxYsyPTp09OhQ4c89thjueqqq3LkkUfW33/MmDHp2LFjyUJBolgAAAAAXxi77bZbunfvnqFDh6aqqioXXXRRxo0blxYtWqR///65+OKLc/rppydJBg8enC5duqzTOIoFAAAAUMKaLDy4IZ1xxhlF17fbbrv6f/fs2TN1dXWrvO/JJ5+8RmNY4BAAAAAoorMAAAAAStjYOgs2BJ0FAAAAQBGdBQAAAFBClc4CAAAAoNLpLAAAAIASKvFX9kp8zgAAAEAJOgsAAACgBGsWAAAAABVPZwEAAACU0EhnAQAAAFDpdBYAAABACToLAAAAgIqnswAAAABKcDYEAAAAoOJtkM6C7q132BDDfGHMXTyn3BHSpUWXckdIkrz1wVvljpAkeWXutHJHSJJ8vWa3ckdIkixauqjcEZIk27ToWu4Iadu0TbkjJEne/uAf5Y6QJLnt5dvLHSFJcurOZ5Q7whfeyAOPLHeEJMlObXqUO0KSZOGSD8odIYdv/6/ljpAkGf/GxHJHSJL88R8bx98I+3RaUu4ISZJX5vy93BGSJN/r8e1yR0ijqo3j986/d9443pNfv/RYuSMkSY7dfni5I6xXjaKzAAAAAKhwigUAAABAEQscAgAAQAkWOAQAAAAqns4CAAAAKKGRzgIAAACg0uksAAAAgBKqnDoRAAAAqHQ6CwAAAKAEZ0MAAAAAKp7OAgAAACjB2RAAAACAiqezAAAAAEqoqsDf2df5Gd9///058MADM2TIkEycOLEBIwEA68LcDAA0lHXqLJgzZ06uu+663H333Vm4cGHGjBmTffbZp4GjAQBrytwMAOtPJa5ZsE7FgsmTJ6dXr15p3rx5mjdvnksvvbShcwEAa8HcDAA0pHU6DGH69On58MMPM3z48Bx++OGZPHlyQ+cCANaCuRkA1p+qqqoNdtlYrPMCh3Pnzs1Pf/rTvP322znqqKPy2GOPbVRPDAAqjbkZAGgo69RZ0KZNm+y6665p3LhxOnfunGbNmuW9995r6GwAwBoyNwMADWmdigV77bVXnn766Sxbtixz5szJwoUL06pVq4bOBgCsIXMzAKw/VRvwfxuLdToMoX379hkwYEAOPfTQJMn555+fRo0q77yTALCxMDcDAA1pndcsGDp0aIYOHdqQWQCAz8HcDADrRyWeOtFPDgAAAECRde4sAAAAgEpQiWcX0lkAAAAAFNFZAAAAACU0qsDf2SvvGQMAAAAl6SwAAACAEqxZAAAAAFQ8nQUAAABQgs4CAAAAoOLpLAAAAIASGkVnAQAAAFDhNkhnwTMznt0Qw6xWny33LneEJEmhUCh3hHRs1rncEZIkT894ptwRkiTd22xf7ghJkq5f7lbuCEmSxo02KXeEJMm0Ba+XO0JmLZpd7ghJkjHP/7bcEZIkvx50Sbkj0EB2bbdbuSMkSd6Y/0a5IyRJ3lrwVrkjZP+tBpQ7QpKkuurxckdIkuxUU1PuCEmSTs23KneEJMkZu32j3BGSJK/M+0u5I6Rxo42jOfr7v/mvckdIktw7fGS5I1QEaxYAAAAAFU+xAAAAACiycfTwAAAAwEaqkcMQAAAAgEqnswAAAABKqHLqRAAAAKDS6SwAAACAEhpVVd7v7JX3jAEAAICSdBYAAABACVXOhgAAAABUOp0FAAAAUIKzIQAAAAAVT2cBAAAAlNDImgUAAABApdNZAAAAACVYswAAAACoeDoLAAAAoIRKXLNAsQAAAAC+QEaNGpUXXnghVVVVGTFiRHr06FG/7amnnsrVV1+d6urq7L333jnxxP+/vbuPsbq+9wT+PsCyXBjEgTLILJWl7ANeSLc1lzSIDzWCppjt1T4NO0JNumna1WY1i43caVNIRROINVVriteHbqupnV7E6iatqBtM3DpqWhO9sHZV7i7lSZhRHhwBcXD2D3PHHrU/EIf5nenv9TKTcOaM/t7HOPMZv+f9+36vzKFDh7J8+fK8+uqrefPNN3PFFVfk/PPPL7yGxQIAAAAYJp555pls3bo1nZ2d2bJlSzo6OtLZ2Tnw/KpVq3LXXXdlypQpWbJkSS666KK8+OKLmTNnTr7+9a9nx44d+drXvmaxAAAAAD6KWq1xtvvr6urKggULkiQzZ87M/v3709vbm6ampmzbti0TJkzI1KlTkyTnnXdeurq6snTp0oG/f9euXZkyZcoxr2OxAAAAAIaJnp6ezJ49e+DxxIkT093dnaampnR3d2fixIl1z23btm3g8eLFi/PKK69k7dq1x7yOxQIAAAAo0MhHJ/b39x/31/7iF7/ICy+8kG9/+9t56KGHUivYuLFxuhQAAABAoZaWlvT09Aw83rNnTyZPnvyBz+3evTstLS3ZtGlTdu3alSQ544wzcvTo0bz22muF17FYAAAAAAVG1GpD9nEs8+fPz4YNG5IkmzdvTktLS5qampIk06ZNS29vb7Zv356+vr5s3Lgx8+fPz+9+97vcfffdSd65jeHgwYNpbm4uvI7bEAAAAGCYOPPMMzN79uwsXrw4tVotK1asyPr16zN+/PgsXLgwK1euzLJly5IkixYtyowZMzJ16tR85zvfSXt7ew4fPpzvfe97GTGiuDtgsQAAAAAKFN3bX4Zrrrmm7vGsWbMG/jx37ty6oxSTZMyYMfnBD37woa7hNgQAAACgjmYBAAAAFBjRwKchnCyaBQAAAEAdzQIAAAAo0Gh7FgwFzQIAAACgjmYBAAAAFKjVqvc+e/VeMQAAAFBoSJoFl8y4dCguc0xv979ddoQkyc43dpQdITsPbis7QpJk/tSzyo6QJNn75mtlR0iS9L51oOwISZL//sI9ZUdIksxr/ZuyI+TjTdPKjpAkuXDGrGN/0RD47Sv/q+wISZK2mTPKjjDsbX5tU9kRkiT/dsK/KztCkmTXG7vKjpBX3+wpO0KS5KtnLC47QpJkwx8fLTtCkuRjY1rKjpAkWfqb5WVHSJJMHju27Aj564/9q7IjJEkmnDKu7AhJkk2v/u+yIyRJPtNybtkRTiqnIQAAAACVZ7EAAAAAqGODQwAAACjg6EQAAACg8jQLAAAAoEDNBocAAABA1WkWAAAAQAF7FgAAAACVp1kAAAAABUbYswAAAACoOs0CAAAAKFCrVe999uq9YgAAAKCQZgEAAAAUqFVwz4ITWix44403cu2112b//v156623cuWVV+acc84Z7GwAwHEymwGAwXRCiwUPPPBAZsyYkWXLlmX37t25/PLL8/DDDw92NgDgOJnNAHDy1GrVaxac0J4Fzc3N2bdvX5LkwIEDaW5uHtRQAMCHYzYDAIPphJoFF198cdavX5+FCxfmwIEDuf322wc7FwDwIZjNAHDyVHHPghNqFjz44INpbW3No48+mp/+9Kf5/ve/P9i5AIAPwWwGAAbTCS0WPPvsszn77LOTJLNmzcqePXty9OjRQQ0GABw/sxkAGEwntFgwffr0PPfcc0mSHTt2ZNy4cRk5cuSgBgMAjp/ZDAAnT61WG7KPRnFCexa0tbWlo6MjS5YsSV9fX1auXDnIsQCAD8NsBgAG0wktFowbNy4333zzYGcBAE6Q2QwAJ88IGxwCAAAAVXdCzQIAAACoikbaS2CoaBYAAAAAdTQLAAAAoECtgu+zV+8VAwAAAIU0CwAAAKCAPQsAAACAytMsAAAAgAK1aBYAAAAAFadZAAAAAAVGVHDPgiFZLHh8x/8cissc0ycmfKLsCEmSGaeUn+PI0SNlR0iSNP/LSWVHSJKsff5nZUdIknx9zmVlR0iSnNb0sbIjJEl+9fKjZUfIf/v0N8uOkCT525kXlx0hSfLm0cNlR2CQ/H73prIjJEl633qj7AhJklnN/77sCOk51F12hCTJ5L9qKTtCkuS+f3yq7AhJktPHTys7QpLkP/31/LIjJEmu/826siPkv17WXnaEJMkfZu8sO0KS5J/2bS87An+hNAsAAACggD0LAAAAgMqzWAAAAADUcRsCAAAAFKhVcINDzQIAAACgjmYBAAAAFKhV8H326r1iAAAAoJBmAQAAABSwZwEAAABQeZoFAAAAUGBENAsAAACAitMsAAAAgAL2LAAAAAAqT7MAAAAACtTsWQAAAABUnWYBAAAAFLBnAQAAAFB5mgUAAABQoFbB99mr94oBAACAQhYLAAAAgDpuQwAAAIACI2xwCAAAAFSdZgEAAAAUqEWzAAAAAKg4zQIAAAAoULNnAQAAAFB1mgUAAABQoNH2LLjhhhvy3HPPpVarpaOjI5/85CcHnnvyySdz0003ZeTIkTn33HNz5ZVXJknWrFmT3//+9+nr68s3vvGNXHjhhYXXsFgAAAAAw8QzzzyTrVu3prOzM1u2bElHR0c6OzsHnl+1alXuuuuuTJkyJUuWLMlFF12Unp6evPTSS+ns7MzevXtz6aWXWiwAAACAj6KR9izo6urKggULkiQzZ87M/v3709vbm6ampmzbti0TJkzI1KlTkyTnnXdeurq60t7ePtA+OOWUU3Lo0KEcPXo0I0eO/LPXsWcBAAAADBM9PT1pbm4eeDxx4sR0d3cnSbq7uzNx4sT3PTdy5MiMHTs2SbJu3bqce+65hQsFiWYBAAAAFBrRwO+z9/f3H/fXPvbYY1m3bl3uvvvuY36txQIAAAAYJnFVS+oAAA2bSURBVFpaWtLT0zPweM+ePZk8efIHPrd79+60tLQkSZ544omsXbs2d955Z8aPH3/M6zTu8ggAAAA0gFqtNmQfxzJ//vxs2LAhSbJ58+a0tLSkqakpSTJt2rT09vZm+/bt6evry8aNGzN//vy8/vrrWbNmTW6//faceuqpx/Wah6RZMHXc1KG4zDFNb/pE2RGSJIePHio7QkYX354yZHrfOlB2hCTJys/8XdkRkiQv7X+h7AhJks9M+UzZEZIkv9nybNkRcqgBvl+T5Oyb/kvZEZIk/+PK68uOwCDpOfh62RGSJB1/01Z2hCTJKwe3lx0hpzeNKztCkmTcqGO/2zQUNnz578uOkCTp6+8rO0KS5IF/+oeyIyRJvreo/O/Zf3PKrLIjJEn+8Y87yo6QJNk8YlfZEZIk188rO0F1nHnmmZk9e3YWL16cWq2WFStWZP369Rk/fnwWLlyYlStXZtmyZUmSRYsWZcaMGQOnIFx99dUD/5zVq1entbX1z17HbQgAAABQoJbGOQ0hSa655pq6x7NmvbuINnfu3LqjFJOkra0tbW0fbrHPbQgAAABAHYsFAAAAQB23IQAAAECB49l48C+NZgEAAABQR7MAAAAACjTaBodDQbMAAAAAqKNZAAAAAAU0CwAAAIDK0ywAAACAIk5DAAAAAKpOswAAAAAK2LMAAAAAqLzjWix48cUXs2DBgtx7771Jkl27dmXp0qVpb2/PVVddlSNHjpzUkABAPbMZAIZOrVYbso9GcczFgoMHD+a6667LvHnzBj53yy23pL29PT//+c8zffr0rFu37qSGBADeZTYDACfbMRcLRo8enTvuuCMtLS0Dn3v66adzwQUXJEnOP//8dHV1nbyEAEAdsxkAhlZtCP9qFMfc4HDUqFEZNar+yw4dOpTRo0cnSSZNmpTu7u6Tkw4AeB+zGQA42T7yaQj9/f2DkQMAGCRmMwAMrkZ6x3+onNBpCGPHjs3hw4eTJLt3766rQQIAQ89sBgAG0wktFpx11lnZsGFDkuSRRx7JOeecM6ihAIAPx2wGAAbTMW9D2LRpU1avXp0dO3Zk1KhR2bBhQ2688cYsX748nZ2daW1tzSWXXDIUWQGAmM0AMNQa6UjDoXLMxYI5c+bknnvued/nf/KTn5yUQABAMbMZADjZPvIGhwAAAPCXzAaHAAAAQOVpFgAAAEABzQIAAACg8jQLAAAAoEAVT0PQLAAAAADqaBYAAABAAXsWAAAAAJWnWQAAAAAF7FkAAAAAVJ5mAQAAABSo4p4Ftf7+/v6TfZGdB7ee7Escl7633yo7QpLkj73l//uY3fwfyo6QJNlzeFfZEZIkB47sLztCkmTauNPLjpAk+auRY8uOkCS59fkflx0hi2ZcUHaEJMnHx/3rsiMkSfr6G+PnaOvY6WVHGPb2H3mt7AhJkiNvv1l2hCRJy3+cW3aE7HzwybIjJEle2v9i2RGSJHMmfrLsCEmSfzFidNkRkiQja43xHt8F9/3nsiPk//3fxvj98f/83UNlR0iSHDl6uOwISZKPjTmt7Agn1Qv7nhuya51xamP8v1pj/NQBAACABlXFZoE9CwAAAIA6mgUAAABQwGkIAAAAQOVZLAAAAADquA0BAAAACtjgEAAAAKg8zQIAAAAooFkAAAAAVJ5mAQAAABRwdCIAAABQeZoFAAAAUEizAAAAAKg4zQIAAAAoYM8CAAAAoPI0CwAAAKBAzZ4FAAAAQNVpFgAAAEABzQIAAACg8jQLAAAAoIDTEAAAAIDKs1gAAAAA1HEbAgAAABSwwSEAAABQeZoFAAAAUECzAAAAAKg8zQIAAAAo4OhEAAAAoPI0CwAAAKCAPQsAAACAytMsAAAAgAL2LAAAAAAqT7MAAAAAClRxz4Jaf39/f9khAAAAoFHtPLh1yK7VOnb6kF2riNsQAAAAoFBtCD+O7YYbbkhbW1sWL16c559/vu65J598Ml/60pfS1taW2267beDzL774YhYsWJB77733uK5hsQAAAACGiWeeeSZbt25NZ2dnrr/++lx//fV1z69atSq33npr7rvvvvz2t7/Nyy+/nIMHD+a6667LvHnzjvs6FgsAAACgQCP1Crq6urJgwYIkycyZM7N///709vYmSbZt25YJEyZk6tSpGTFiRM4777x0dXVl9OjRueOOO9LS0nLcr9liAQAAAAwTPT09aW5uHng8ceLEdHd3J0m6u7szceLE9z03atSojBkz5kNdx2IBAAAADFMn68wCRycCAABAgVqtcY5ObGlpSU9Pz8DjPXv2ZPLkyR/43O7duz/UrQd/SrMAAAAAhon58+dnw4YNSZLNmzenpaUlTU1NSZJp06alt7c327dvT19fXzZu3Jj58+ef0HWGxWJB0bEQVbRmzZq0tbXli1/8Yh555JGy45Tu8OHDWbBgQdavX192lNI99NBD+fznP58vfOELefzxx8uOU6o33ngj3/rWt7J06dIsXrw4TzzxRNmRSvHeI3J27dqVpUuXpr29PVdddVWOHDlSckKGK7O5ntlcz2x+l9n8LrP5HWbzcNU4WxyeeeaZmT17dhYvXpxVq1ZlxYoVWb9+fR599NEkycqVK7Ns2bJcdtllWbRoUWbMmJFNmzZl6dKleeCBB/Kzn/0sS5cuzb59+wqv0/C3IfzpsRBbtmxJR0dHOjs7y45VmqeeeiovvfRSOjs7s3fv3lx66aW58MILy45Vqh//+MeZMGFC2TFKt3fv3tx22225//77c/Dgwdx666357Gc/W3as0jzwwAOZMWNGli1blt27d+fyyy/Pww8/XHasIfVBR+TccsstaW9vz+c+97ncdNNNWbduXdrb20tMyXBkNtczm9/PbH6H2VzPbDabGTzXXHNN3eNZs2YN/Hnu3Lnvm8tz5szJPffc86Gu0fDNgqJjIapo7ty5ufnmm5Mkp5xySg4dOpSjR4+WnKo8W7Zsycsvv1zpwfvPurq6Mm/evDQ1NaWlpSXXXXdd2ZFK1dzcPLBaeuDAgbodY6vig47Iefrpp3PBBRckSc4///x0dXWVFY9hzGyuZzbXM5vfZTbXM5vN5uGscXoFQ6fhFwuKjoWoopEjR2bs2LFJknXr1uXcc8/NyJEjS05VntWrV2f58uVlx2gI27dvz+HDh/PNb34z7e3tlR80F198cXbu3JmFCxdmyZIlufbaa8uONOQ+6IicQ4cOZfTo0UmSSZMmVfrnKSfObK5nNtczm99lNtczm81mhpeGvw3hvU7WsRDDzWOPPZZ169bl7rvvLjtKaX71q1/lU5/6VD7+8Y+XHaVh7Nu3Lz/60Y+yc+fOfPWrX83GjRsbaufWofTggw+mtbU1d911V/7whz+ko6PDvbPv4ecpg8V/S+8wm83mD2I2v8tsPjY/TxtZ9b5vG36xoOhYiKp64oknsnbt2tx5550ZP3582XFK8/jjj2fbtm15/PHH88orr2T06NE57bTTctZZZ5UdrRSTJk3Kpz/96YwaNSqnn356xo0bl9deey2TJk0qO1opnn322Zx99tlJ3rmHa8+ePTl69Gil3+1LkrFjx+bw4cMZM2bMRzpKh2ozm9/PbH6H2VzPbK5nNn8ws5lG1fC3IRQdC1FFr7/+etasWZPbb789p556atlxSvXDH/4w999/f375y1/my1/+cq644orK/jKSJGeffXaeeuqpvP3229m7d28OHjxYyXsB/9n06dPz3HPPJUl27NiRcePGVf6XkSQ566yzBn6mPvLIIznnnHNKTsRwZDbXM5vfZTbXM5vrmc0fzGweHmq12pB9NIqGbxb86bEQtVotK1asKDtSqX79619n7969ufrqqwc+t3r16rS2tpaYikYwZcqUXHTRRfnKV76SJPnud7+bESMafj3wpGlra0tHR0eWLFmSvr6+rFy5suxIQ27Tpk1ZvXp1duzYkVGjRmXDhg258cYbs3z58nR2dqa1tTWXXHJJ2TEZhszmemYzf47ZXM9sNpsZXmr9bowBAACAP2vP4Z1Ddq2WMY2x2FzdpU0AAADgAzX8bQgAAABQploFT0PQLAAAAADqaBYAAABAAc0CAAAAoPIsFgAAAAB1LBYAAAAAdSwWAAAAAHVscAgAAAAFajUbHAIAAAAVZ7EAAAAAqGOxAAAAAKhjzwIAAAAoUIs9CwAAAICK0ywAAACAQpoFAAAAQMVpFgAAAECB6vUKNAsAAACA99AsAAAAgAK1WvW6BZoFAAAAQB3NAgAAACikWQAAAABUnMUCAAAAoI7bEAAAAKBA9W5C0CwAAAAA3kOzAAAAAApVr1ugWQAAAADU0SwAAACAArWaZgEAAABQcRYLAAAAgDoWCwAAAIA69iwAAACAAjWnIQAAAABVp1kAAAAAhTQLAAAAgIrTLAAAAIAC1esVaBYAAAAA76FZAAAAAAVqtep1CzQLAAAAgDoWCwAAAIA6bkMAAACAQm5DAAAAACpOswAAAAAKVK9XoFkAAAAAvIdmAQAAABSqXrdAswAAAACoo1kAAAAABWo1zQIAAACg4iwWAAAAAHUsFgAAAAB17FkAAAAABWpOQwAAAACqrtbf399fdggAAACgcWgWAAAAAHUsFgAAAAB1LBYAAAAAdSwWAAAAAHUsFgAAAAB1LBYAAAAAdf4/S9Gi+iqaSCQAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "feature = 'OBJ'\n",
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_target_attention(example[1], example[2], max_len, model_finetuned, feature=feature)\n",
+ " weights_base = analyze_target_attention(example[1], example[2], max_len, model_base, feature=feature)\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to OBJ tokens')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.7 Attention so SEP"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 54,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:42<00:00, 2.92it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAUAAAJ1CAYAAAChC0mPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XuYlWWhN/7vmgEEAYkziqBsTEyKraaU4YuGoGjbUioFDcl9IPfWDlqaokVpkJn2ilTvtoPplWUo4SlL1I3mYUMIts14K88mmgLKGeU08/vDl/lJwgLGYZ5hrc/Ha10Xa+ZZz/Ndi5F71r2+z/2U6uvr6wMAAABUnZqiAwAAAADFMCkAAAAAVcqkAAAAAFQpkwIAAABQpUwKAAAAQJUyKQAAAABVyqQAALDTTZ06NQMGDMjvfve7oqM0qYULF2bAgAENtyFDhhQdqaynn356s7zDhg0rOhIABWtVdAAAYNu+9a1v5dprr03Hjh3z0EMPpW3btlvc7r777svGjRszfPjwhq+tXr061157bcaNG5c99thjp2d94okn8t///d/59Kc/3fC14447Lu9+97vz7ne/e6cff2u29No0lcGDB+e0007Lbrvt1vC1jRs3ZsaMGbnzzjvz17/+NYsXL06pVEqvXr1yyCGH5IwzzsiAAQMatl+4cGGOPvro7Tre9773vYbnMWzYsLz44otv26Zdu3bp06dPjj322PzzP/9zdt999/Ts2TNTpkxJknz9619/J08ZgAphUgAAWrh169bllltuSU1NTVauXJm77rorJ5544ha3/fGPf5y99957sze+jz/+eL773e/mpJNOapZJgbvuuiu33nrrZpMC++23X/bbb7+dfuxytvTaNJXevXtn5MiRDffr6upy9tlnZ9asWTn88MMzbty4dOvWLStWrMgf/vCH/OpXv8qdd96ZH/3oR/nABz6w2b7e85735Mwzzyx7vPe9732b3W/dunWuuOKKhvv19fVZvHhxfvvb32bq1Kn5r//6r/z85z9Phw4dGnJefvnl7/RpA1ABTAoAQAt39913Z+nSpTn11FNz44035qabbtripEBdXV0WLFiQvffee7OvP/74480VtZDjbY+tvTY7y4MPPphZs2bl2GOPzdVXX73Z98aMGZMTTjghZ5xxRq688srcdNNNm32/a9eum00wbI9SqbTFx5x++um55JJL8rOf/SzTpk3bbKIGABJrCgBAizdt2rQkybhx4/L+978/8+fPz9NPP73ZNjNmzMh73vOerFmzJrfccksGDBiQqVOnZtiwYQ2fIB999NGb1dWTZPr06fnEJz6Rf/zHf8zBBx+ck046KT/96U9TV1fXsM2m8+YvuuiiPPnkk/nXf/3XHHrooRk0aFBOO+20/PGPf9xsuwceeCAvvvhiBgwYkLFjxybZ+poCDzzwQMaNG5dDDz00733vezNs2LB84xvfyGuvvbbZdsOGDcuIESOycuXKXHzxxRkyZEje+9735rjjjssdd9xR9vXb2muzydNPP51zzz03Q4YMycCBA/OhD30on/vc5/KXv/xlm383W7PpsUOHDt3i9z/0oQ/le9/7XiZOnNjoY2yvj3/840lSces5ANA0TAoAQAv27LPPZu7cuTn44IOz7777NjQEbr755s22+8AHPtDwBnPw4MGZMmVKjjvuuEycODGDBw9OkkycOLHhfPIkueyyy3LRRRelR48eufjii3P++eene/fu+cY3vpGvfOUrb8uyaNGinHHGGenfv38uuuiinHbaafmf//mfnHnmmVm3bl26du2aKVOmpEuXLunSpUumTJmSz372s1t9brfcckvGjx+fxYsX58wzz8wll1ySD3/4w7nxxhtz6qmnZs2aNZttX1dXl/Hjx2fFihU555xz8oUvfCHLli3L+eefnwULFmz1OFt7bZI31z84+eST8/DDD+cTn/hEJk2alFNPPTWPPPJIRo8enT/96U9b3W85PXr0SPJmy2PdunVb3Gb48OEZOHBgo/a/IzatP7Fhw4adfiwAdj1OHwCAFmxTtXzTp73HHXdcJk2alFtvvTXnnntu2rRpk+TNc9o3fSr91vPb99tvv/zmN79J8uan1pvq83/+85/zk5/8JKeeeupmn1aPGTMmn/vc5zJ9+vScdtppOfDAAxu+98ADD+Sqq65qeEOdJMuXL88vf/nLzJ8/P4cffnhGjhzZcK56uQr8G2+8kW9+85vp0qVLfvGLXzSsdTBq1Kjsueee+fa3v50bbrgh48ePb3jMwoULM3To0M3yduvWLV/+8pdz7733bvUN9tZem+TN8+pXrVqVX/ziFzn44IMbvj506NB88pOfzJVXXpkf/ehHW30eWzNixIhcffXV+e1vf5sTTjghJ554Yg4//PAMHDgwrVu3LvvYjRs3ZsWKFVv9fm1tbdq3b7/dWR5++OEkyaBBg7b7MQBUD00BAGihNi0w2K5du4Y34h06dMgxxxyTpUuX5t577230vjdNFBx//PFZsWLFZrdjjz02STJ37tzNHtOrV6/NJgSS/3/Bu8WLF+/Q8efOnZvly5fn+OOPf9vih6NGjUqS3H///W973BlnnNEkx0+SNWvW5OGHH86AAQM2mxBI3nwDvf/++2f27NlZu3btDu+7ffv2ufHGG3PMMcfk+eefz1VXXZVTTjklhx56aMaNG5frr79+q2/8Z8+encMOO2yrt60tQvj3f49PP/10rrvuuvzv//2/061bt5x66qk7/DwAqHyaAgDQQm1aYPCjH/1oOnTo0PD1UaNG5bbbbstNN92U448/vlH7fuqpp5Ikn/rUp7a6zUsvvbTZ/b59+75tm02X4NvRavozzzyTJNl///3f9r0uXbrkXe96V5577rnNvl5bW5vevXs3yfGT5Pnnn09dXd1WL5PYr1+/PPHEE1m4cGH69++/w/vv2bNnpk6dmpdffjn33XdfHn300cybNy9z5szJnDlzMnXq1Fx11VU54ogjNnvc+973vpx33nlb3e+WriCxbt26HHbYYVvc/vDDD8/Xvva1dO3adYefAwCVz6QAALRQmxYYHDx4cJ5//vmGr/fq1SvdunXLnDlz8sILL6RPnz47vO/Vq1cnSb7zne+kW7duW9yme/fum93fdKpCU9i0XkC7du22+P22bdu+7ZP02tra1NbWNlmGTa/B7rvvvsXvb5pw+Pu1DXZUr169MmbMmIwZMybJmwsbTps2Lddff32++MUv5u67706nTp0atu/UqdPbLlO4La1bt86Pf/zjzb7Wtm3b9OnTJ126dHlH+QGobCYFAKAFeuaZZxrq+xdffPFWt5s+fXrOOeecHd7/pnPS+/TpU8i55pveiG/tDffrr7++Q+fNN8am/ZfL8Nbtmkr//v0zYcKELF26NLfffnsee+yxrV6lYHuVSqUdnkgAgMSkAAC0SJsWGPzkJz/5tnp5kqxduzYXXnhhfvnLX+azn/1sWrXasSF9v/32y7333ptHH330bZMCq1evTqtWrRo+Kd8Z9ttvvyRvrv7/9xYvXpzly5fv9De5++67b2pra7eYIXnzFIs2bdo0LM64vdatW5drrrkmq1atyoUXXrjV7Tbt94033tih/QNAU7LQIAC0MJsWGGzTpk3OPffcjBw58m23j33sYxk+fHgWL17csCDfpmr93y+MV1NT87avb1ow8MYbb3zbm9Jvf/vb+eAHP5i//vWvjcpfU1OzzcX5Bg8enC5duuTXv/51li9fvtn3Np02sWnBw6awpdemXbt2OfLII/PEE09k3rx5m20/d+7cPPvss/nwhz+8w6dNtGnTJvfdd1+uu+663HHHHVvc5tVXX82vfvWr7Lbbbjn00EN38NkAQNPRFACAFmbmzJlZtmxZRo0aVfZ88E996lOZOXNmbr755gwfPjzdunVL27Zt8+CDD+aaa67JPvvsk5EjRzZ8Iv3tb387hx12WD72sY/lgAMOaFgFf8yYMTnllFPSqlWr/Pa3v83dd9+dj370o1tcWHB77L333pk9e3a++c1vZs8998ynP/3pt23Tpk2bfOUrX8m5556b0047LZ/4xCfSsWPHPPbYY7n55ptz0EEH5ZOf/GSjjr8lW3ttzj///MybNy9nnXVWxo4dm7333jvPPfdcfv7zn6dz585lF/wr57LLLssZZ5yR8847L7feemuOOuqodOnSJatXr85TTz2VX/3qV1m2bFkuueSSt/0dv/rqq7nrrrvK7r9nz55vu2ICADSGSQEAaGE2fVI+bty4stsNHjw4+++/fx588MG8/PLL6dWrVy644IJcddVV+f73v5+TTz45I0eOzOjRo/PQQw/loYceyh//+Mccc8wxSZIJEybk3e9+d6ZNm5ZvfvObqaury7777pvzzjtvi2/kt9fnP//5vPjii/nZz36WAQMGbHVfxx9/fDp16pRrrrkmV199ddatW5fevXtn/Pjx+cxnPtOkCxu2bt16i69Nv379ctNNN+Xqq6/Oz3/+8yxfvjydO3fOsGHDctZZZzVqEcfkzasq3HHHHbnhhhvywAMPZMqUKVmzZk3atGmTPffcM8OHD8+pp56aAw444G2P/dOf/pTPf/7zZfd/9NFH5/vf/36jsgHAW5Xq6+vriw4BALArWrhwYY4++uicdNJJueyyy4qOs0OGDRuWJJk1a1bBSQAokjUFAAAAoEo5fQAA4B168cUXc9ddd2W33XbLhz/84aLjbNWqVavy0EMPJXnzkovt2rUrOBEARTMpAADwDs2dOzdz585Nt27d8vDDDxcdZ6teeeWVzdYr6N27d4FpAGgJrCkAAAAAVcqaAgAAAFClTAoAAABAlTIpAAAAAFXKpAAAAABUKZMCAAAAUKVMCgAAAECVMikAAAAAVcqkAAAAAFQpkwIAAABQpUwKAAAAQJUyKQAAAABVyqQAAAAAVCmTAgAAAFClTAoAAABAlTIpAAAAAFXKpAAAAABUKZMCAAAAUKVMCgAAAECVMikAAAAAVcqkAAAAAFQpkwIAAABQpUwKAAAAQJUyKQAAAABVyqQAAAAAVCmTAgAAAFClTAoAAABAlTIpAAAAAFXKpAAAAABUKZMCAAAAUKVMCgAAAECVMikAAAAAVcqkAAAAAFQpkwIAAABQpUwKAAAAQJUyKQAAAABVyqQAAAAAVCmTAgAAAFClTAoAAABAlTIpAAAAAFXKpAAAAABUKZMCAAAAUKVMCgAAAECVMikAAAAAVcqkAAAAAFQpkwIAAABQpUwKAAAAQJUyKQAAAABVyqQAAAAAVCmTAgAAAFClTAoAAABAlTIpAAAAAFWqVdEBAAAAoCUrjdi72Y5Vf8/CZjtWoikAAAAAVcukAAAAAFQppw8AAABAOaVS0Ql2Gk0BAAAAqFKaAgAAAFBOBX+cXsFPDQAAAChHUwAAAADKsaYAAAAAUGk0BQAAAKCcyi0KaAoAAABAtdIUAAAAgHKsKQAAAABUGk0BAAAAKKeCP06v4KcGAAAAlGNSAAAAAKqU0wcAAACgHAsNAgAAAJVGUwAAAADKqdyigKYAAAAAVCtNAQAAACinpnKrApoCAAAAUKU0BQAAAKCcyi0KaAoAAABAtdIUAAAAgHJKlVsV0BQAAACAKqUpAAAAAOVUblFAUwAAAACqlaYAAAAAlFNTuVUBTQEAAACoUpoCAAAAUE7lFgU0BQAAAKBamRQAAACAKuX0AQAAACinVLnnD2gKAAAAQJXSFAAAAIByXJIQAAAAqDSaAgAAAFBO5RYFNAUAAACgWmkKAAAAQDkVfPUBkwIAAACwC5k8eXIee+yxlEqlTJgwIYMGDWr43t/+9rece+65Wb9+fQ488MBccsklZffl9AEAAAAop9SMt22YO3dunn/++UybNi2TJk3KpEmTNvv+ZZddln/+53/O9OnTU1tbm5deeqns/kwKAAAAwC5i9uzZGT58eJKkf//+Wb58eVatWpUkqaury/z58zNs2LAkycSJE7PXXnuV3Z9JAQAAACinptR8t21YsmRJOnfu3HC/S5cuWbx4cZLktddeS/v27fPNb34zY8aMyZVXXrntp9b4VwUAAAAoUn19/WZ/fuWVV3L66afnhhtuyP/9v/83999/f9nHmxQAAACAclrQmgI9evTIkiVLGu4vWrQo3bt3T5J07tw5e+21V/r27Zva2tocfvjhefLJJ8vuz6QAAAAA7CKGDBmSmTNnJkkWLFiQHj16pEOHDkmSVq1apU+fPnnuuecavt+vX7+y+3NJQgAAACintB0f4TeTQw45JAMHDszo0aNTKpUyceLEzJgxIx07dsyIESMyYcKEXHDBBamvr8/+++/fsOjg1pTq33oCAgAAALCZ0hkHNNux6n/y52Y7VuL0AQAAAKhaTh8AAACAcir44/QKfmoAAABAOZoCAAAAUE4LWmiwqWkKAAAAQJXSFAAAAIByKrcooCkAWzJgwICMGDEiI0eOzLHHHpt//dd/zQsvvFB0rEYZMWJEfve735Xd5oILLsj3v//9ZkoEQLV66/i66fYv//IvSZJx48ZlwYIFTXasBx98MC+99FKT7a+ciy66KFOnTm2WYx144IFZuHBh2W3Gjh2b2267rVnyALs+TQHYip/+9Kfp1atXkuTKK6/MpEmT8p//+Z8FpwKAXdtbx9e3uv7665v0ONddd13+/d//PXvttVeT7heoUhW8poBJAdgOH/zgBzNr1qyG+zfffHOuvfbabNy4Md27d8/ll1+e3r1755VXXsn555+fxYsXZ926dfnIRz6Sc845J/X19fne976XO+64I+vWrcvRRx+dCy+8MLW1tZsdZ8aMGbn//vvTunXrzJ8/P/369ctZZ52VK664In/961/z+c9/Pqecckrq6uoyZcqUzJw5M0ly0EEH5atf/Wp23333/PGPf8yXv/zlbNiwIUceeeRm+7/33nszZcqUrFmzJvvss0+uuOKKdOnSZee/gACwDcOGDcvll1+eXr16ZfTo0Rk/fnxuvvnmLFu2LBdeeGGOP/747R5Pr7rqqsyZMyfPPPNMzjvvvDzwwAPp27dv/uM//iPJmw25TfeHDRuW8ePHZ/r06Xn55ZfzT//0T7nggguSbH3cXLp0ab74xS/mueeey3777Ze2bdtucaJj6tSpWbJkSV5++eUsWLAghx9+eI4//vhMnTo1ixYtyqWXXpoPf/jDWbt2bSZNmpTf/e53qampyZFHHpnzzjsvtbW1+e1vf5tvfOMbadWqVT7+8Y9vtv9p06blJz/5SdatW5eDDjookydPTtu2bXfS3xBQqZw+ANuwbt263H777Rk2bFiS5NVXX80ll1ySn/zkJ7n77rvTt2/fhur9ddddl8MOOyy//vWvc8cdd+SFF17IokWLctttt+Wuu+7K9OnTc8899+SFF17IjTfeuMXjPfjggzn77LNz99135+mnn86PfvSj/OxnP8ukSZMajvOb3/wmDzzwQGbMmJE777wzK1asyHXXXZck+drXvpbTTz89M2fOzMEHH9xQMXzhhRdy/vnn58orr8x//dd/5QMf+EC+9rWv7dwXDwAaYenSpampqckdd9yRCRMm5KqrrkqS7R5Pv/CFL6Rnz5759re/neOPP36bx3vkkUcybdq0/PKXv8wNN9yQl19+uey4+cMf/jCdO3fOrFmz8tWvfjUPPfTQVvd9//33Z/Lkybnjjjty1113NYzfZ555Zn74wx8mebMl8fLLL+fOO+/MLbfcknnz5uVXv/pVNm7cmIsuuigTJ07Mb37zm9TU1GTjxo1Jknnz5mXKlCm5/vrrM2vWrHTo0CFTpkzZ0Zca2F41zXhrZiYFYCvGjh2bkSNHZsiQIXn88cczatSoJEnXrl0zf/78hk8EDj300Ib1Brp27ZqHHnoo8+bNS5s2bfKd73wnPXr0yH333ZePf/zj6dixY1q1apVPfvKTufvuu7d43P322y/9+vVLmzZtss8+++SII45IbW1t9t9//yxatCjJm79gnHjiidl9991TW1ubUaNG5eGHH87atWvz+OOPN/wCNHLkyLRr1y5J8sADD2Tw4MHZf//9kySjR4/OrFmzGn65AIDmsGl83XS7+OKL37bNhg0bGsbdgQMHNqwNsCPj6Y444YQTUltbm549e6Zr167529/+VnbcnDdvXo477rgkyd57753Bgwdvdd8HH3xwunbtms6dO6d79+4ZOnRokrxtXD/55JPTqlWrtG3bNieccEIefvjhPPfcc1m3bl2OOOKIJMlJJ53UsN9Zs2bl+OOPT8+ePZMkY8aMaZLXAqg+Th+ArXjrOY+PPPJIxo4dmxkzZqRr1665+uqrG34xWL16dfr165ck+fSnP526urp8/etfz6JFi3Laaafls5/9bFauXJkf//jHmTZtWpJk48aNW63tt2/fvuHPtbW12X333Rv+XFdXlyR57bXX0qlTp4btOnXqlFdffTXLli1LknTo0CFJUiqVssceeyRJVq5cmXnz5mXkyJENj+vQoUPDYwCgOWxtTYG3euv4V1NT0zD+bW08feWVVzJu3LgkyaBBg3L55ZfvUKZN4+amY2/cuLHsuLl8+fJ07Nix4eubxtot2dq4/tbntbVxffny5Ztle+s2K1euzD333NPQUqivr8/69et36HkDO8CaAlDdDjvssOy1116ZP39+NmzYkFmzZuWGG25Ily5dctNNN+WOO+5IkrRq1Srjx4/P+PHj8+yzz+bf/u3f8v73vz89evTIsGHD8qlPfapJ8nTr1m2zN/PLli1Lt27dGn5ZWLVqVTp27Ji6urosX748SdKjR4986EMfytVXX90kGQCguZUbT++6666tPu6tb8CTNIyN2zrW1sbNPfbYIytXrmy4/9prr6VPnz7b3OfWlBvXV61atdlx3prvpJNOype//OVGHxcgcfoAbJdnn302zz77bP7hH/4hr776anr37t2w0NBvfvObrF69Okny1a9+NQ8//HCSpG/fvunWrVtKpVKOPvro3HbbbXn99deTJL/4xS9yyy23NDrPUUcdldtvvz2vv/56NmzYkOnTp+fII49M27Ztc8ABB+See+5Jktx5551Zu3ZtkuSII47IvHnzGk51+MMf/pBvfOMbjc4AAM1tR8bTVq1aNbxx7969e/785z8neXONnUcffXSbxyo3bh500EG59957kyR//etfM3/+/Hf0vI466qhMnz49GzduzJo1a3LbbbflyCOPTN++fVNbW9twaeEZM2ak9P8+rRw2bFjuvvvuhomCe++9Nz/4wQ/eUQ6gjFIz3pqZpgBsxdixYxtWM27Tpk2+/vWvZ8CAAenatWvuvPPOjBgxIn369MkXvvCF/Pu//3suu+yyjB49Ol/96ldz6aWXpr6+PsOGDcvhhx+eJHnyyScbzgXs27dvJk2a1OhsI0eOzF/+8peMGjUq9fX1+cAHPpDTTz89yZsLDU6YMCHXXHNNhg4dmv79+yd58xOFSy+9NGeddVbWr1+f9u3bZ8KECe/kJQKAZjV8+PDtHk+PPfbYnHvuufnc5z6Xk08+OWeffXaOOeaYHHjggTn22GO3eaxy4+ZnPvOZnHPOORk2bFj69++fY4455h09r7Fjx+aFF17IRz7ykZRKpYwcOTLHHXdcSqVSLr300kyYMCFt2rTJqFGjGk4/GDhwYM4888yMHTs2dXV16dq1a77+9a+/oxxAdSrV19fXFx0CAAAAWqrS2e9ttmPVf/ePzXasxOkDAAAAULVMCgAAAECVsqYAAAAAlFPBlyTUFAAAAIAq1SxNgaVrlzTHYbapTU2boiMkSdbWrS06QkpFXOtiC9ZufL3oCEmSNrW7FR0hSbJbTduiIyRJ1mxYte2NmkFNqbboCC3mZ+ONDWuKjpAkqa1pGQWzLrv1KDrCLm9ZCxmb27bavegISZJ1deuKjpC6ug1FR0iStJQVqFvXtC46wv/TMn5n2lC/vugISZLaFjA2txRrN75RdIQkSesW8h6nU5suRUfYuVrGPwU7haYAAAAAVKmW8ZEPAAAAtFAlawoAAAAAlUZTAAAAAMrQFAAAAAAqjqYAAAAAlFHBRQFNAQAAAKhWmgIAAABQRk0FVwU0BQAAAKBKaQoAAABAGa4+AAAAAFQcTQEAAAAoQ1MAAAAAqDiNbgpMnjw5jz32WEqlUiZMmJBBgwY1ZS4AYAcZmwGAHdWoSYG5c+fm+eefz7Rp0/L0009nwoQJmTZtWlNnAwC2k7EZAHYepw/8ndmzZ2f48OFJkv79+2f58uVZtWpVkwYDALafsRkAaIxGTQosWbIknTt3brjfpUuXLF68uMlCAQA7xtgMADtPqdR8t+bWJAsN1tfXN8VuAIAmYmwGALZHo9YU6NGjR5YsWdJwf9GiRenevXuThQIAdoyxGQB2HmsK/J0hQ4Zk5syZSZIFCxakR48e6dChQ5MGAwC2n7EZAGiMRjUFDjnkkAwcODCjR49OqVTKxIkTmzoXALADjM0AsPNUclOgUZMCSfKlL32pKXMAAO+QsRkA2FGNnhQAAACAalBK5TYFmuTqAwAAAMCuR1MAAAAAyqjkNQU0BQAAAKBKaQoAAABAGRVcFNAUAAAAgGqlKQAAAABl1FRwVUBTAAAAAKqUpgAAAACU4eoDAAAAQMUxKQAAAABVqllOH1i+7rXmOMw2dd6tW9ERkiQ1Kb56Upe6oiMkSTq27lR0hCTJivXLio6QJGlX277oCEmS1jVtio6QJFm5fnnREbJqw4qiIyRJ2rfqWHSEJMn6uvVFR6CJrK1bW3SEJEnNxtqiIyRJakrF56hrIdXUNqXWRUdIkqzZsKroCEmS3VrtXnSEJMmGFvLv78ZsKDpC6lNfdIQkLePfjSSpLTkjvDk4fQAAAACoOKaVAAAAoIwKLgpoCgAAAEC10hQAAACAMqwpAAAAAFQcTQEAAAAoQ1MAAAAAqDiaAgAAAFCGpgAAAABQcTQFAAAAoAxNAQAAAKDiaAoAAABAGRVcFNAUAAAAgGqlKQAAAABlWFMAAAAAqDgmBQAAAKBKOX0AAAAAynD6AAAAAFBxNAUAAACgjBpNAQAAAKDSaAoAAABAGRVcFNAUAAAAgGqlKQAAAABlVPLVB0yCwS1KAAAgAElEQVQKAAAAwC5k8uTJeeyxx1IqlTJhwoQMGjSo4XvDhg1Lr169UltbmyS54oor0rNnz63uy6QAAAAAlFFKy2kKzJ07N88//3ymTZuWp59+OhMmTMi0adM22+aHP/xh2rdvv137s6YAAAAA7CJmz56d4cOHJ0n69++f5cuXZ9WqVY3en0kBAAAAKKNUKjXbbVuWLFmSzp07N9zv0qVLFi9evNk2EydOzJgxY3LFFVekvr6+7P5MCgAAAMAu6u/f9H/uc5/LhRdemJ/+9Kd58sknM3PmzLKPNykAAAAAZbSkpkCPHj2yZMmShvuLFi1K9+7dG+6feOKJ6dq1a1q1apWhQ4fmiSeeKLs/kwIAAACwixgyZEjDp/8LFixIjx490qFDhyTJypUr8y//8i9Zt25dkuSRRx7Ju9/97rL7c/UBAAAAKGM7PsBvNoccckgGDhyY0aNHp1QqZeLEiZkxY0Y6duyYESNGZOjQoTnllFOy22675cADD8zIkSPL7s+kAAAAAOxCvvSlL212/4ADDmj487hx4zJu3Ljt3pfTBwAAAKBKNUtToPNu3ZrjMNu0cv2yoiMkSVrXtCk6QtbXrSs6QpJkQ82GoiO0KK+ufaXoCElaxs9okrSu3a3oCGlXu3vREZIk9fV1RUdI0nJ+NnjndqttW3SEJMn6+vVFR0iS1NetLTpCi1FXahn/3rQULeXf39pSyyj4toTfIdu2ahljc20L+Xx1XQv4O6kG27MA4K6qZfwkAwAAAM2uZUw5AgAAQAulKQAAAABUHE0BAAAAKENTAAAAAKg4mgIAAABQRgUXBTQFAAAAoFppCgAAAEAZ1hQAAAAAKo6mAAAAAJShKQAAAABUHE0BAAAAKENTAAAAAKg4jW4KXH755Zk/f342bNiQz3zmMznmmGOaMhcAsIOMzQCwc1RwUaBxkwJz5szJk08+mWnTpmXp0qU56aST/OIBAAUyNgMAjdGoSYHDDjssgwYNSpLsscceef3117Nx48bU1tY2aTgAYPsYmwGAxmjUpEBtbW123333JMn06dMzdOhQv3QAQIGMzQCw81TyQoPv6OoD9957b6ZPn55rr722qfIAAO+AsRkA2BGNnhR48MEH85//+Z/50Y9+lI4dOzZlJgCgEYzNALBzaAr8nZUrV+byyy/Pddddl3e9611NnQkA2EHGZgCgMRo1KfDrX/86S5cuzRe+8IWGr33rW9/KXnvt1WTBAIDtZ2wGgJ2nkpsCpfr6+vqdfZDl617b2YfYLivXLys6QpKkdU2boiNkfd26oiMkSdrUtC06QpJkY/2GoiMkaTk5WsLPaJLs9H+ctkO72t2LjpAkqa+vKzpCkqRUqik6QpKkU5suRUfY5S1b92rREZIkG+s3Fh0hScv5f6wlqCm1jAUqN9atLzpCkqRNbcv4XaWuhfyMtoTfIdu2ahljc21axpi4rgX8nSRJ5926FR1hpxr0/Y8227H+8B+3N9uxkne40CAAAABUugouCrSQ6S0AAACg2WkKAAAAQBmVvKaApgAAAABUKU0BAAAAKEdTAAAAAKg0mgIAAABQhjUFAAAAgIqjKQAAAABlVHBRQFMAAAAAqlWzNAX++Nr/NMdhtmm32t2KjpAkeWn1S0VHyJ7t9yw6QpJk3cZ1RUdIkry/+weLjpAkmb94TtERkiRtatsUHSFJUluqLTpC9ty9d9ERkiQvrn6h6AhJknatdi86QpKkU5suRUfY5T25/E9FR0iS7N6qfdERkiSLX19UdIRsqN9YdIQkyYurXiw6QpLk4/9wStERkiSPv/Zo0RGSJB3b7FF0hCTJwlXFj0f7dNy36AhJkvV1LeP32KVrlxYdIUly5J7HFB2BRnL6AAAAAJRhoUEAAACg4mgKAAAAQBmaAgAAAEDF0RQAAACAMjQFAAAAgIqjKQAAAABlVHBRQFMAAAAAqpWmAAAAAJRhTQEAAACg4mgKAAAAQBmaAgAAAEDF0RQAAACAMjQFAAAAgIqjKQAAAABlaAoAAAAAFcekAAAAAFQppw8AAABAGRV89oCmAAAAAFQrTQEAAAAow0KDAAAAQMXRFAAAAIAyNAUAAACAiqMpAAAAAGVoCgAAAAAVR1MAAAAAyqjgooCmAAAAAFQrTQEAAAAow5oCAAAAQMXRFAAAAIByNAUAAACASqMpAAAAAGVYUwAAAACoOM3SFJj7yqPNcZht+kT/k4qOkCS57HfXFx0hw/Z9b9ERkiS9O/QqOkKS5No//ajoCEmSuvr6oiMkSR5e+OeiIyRJOrRpU3SEHNX3oKIjJEmeX/FS0RGSJEvfWFF0hCTJt4ccXHSEXd5tT88sOkKSZPT+JxYdIUky95XfFx0hXdu+q+gISZKV61YXHSFJ8vMnf1p0hCTJ/JefLDpCkqRz23ZFR0iS7LFbh6Ij5KXVLxcdIUny/PKWMTb/w7v2LjpCkuTIPYtOQGM5fQAAAADKqKncswecPgAAAADVSlMAAAAAyrDQIAAAAFBxNAUAAACgjBpNAQAAAKDSaAoAAABAGdYUAAAAACqOSQEAAAAoo6YZb9tj8uTJOeWUUzJ69Oj84Q9/2OI2V155ZcaOHbtdzw0AAADYBcydOzfPP/98pk2blkmTJmXSpElv2+app57KI488sl37MykAAAAAZdSUSs1225bZs2dn+PDhSZL+/ftn+fLlWbVq1WbbXHbZZTnnnHO277nt+MsBAAAAFGHJkiXp3Llzw/0uXbpk8eLFDfdnzJiRwYMHp3fv3tu1P1cfAAAAgDJa8tUH6uvrG/68bNmyzJgxIz/5yU/yyiuvbNfj31FT4I033sjw4cMzY8aMd7IbAKCJGJsBoLL16NEjS5Ysabi/aNGidO/ePUkyZ86cvPbaaznttNNy9tlnZ8GCBZk8eXLZ/b2jSYH/83/+Tzp16vROdgEANCFjMwA0vZa0psCQIUMyc+bMJMmCBQvSo0ePdOjQIUkycuTI/PrXv85NN92U7373uxk4cGAmTJhQdn+NPn3g6aefzlNPPZWjjjqqsbsAAJqQsRkAKt8hhxySgQMHZvTo0SmVSpk4cWJmzJiRjh07ZsSIETu8v0ZPCnzrW9/KV77yldx6662N3QUA0ISMzQBQHb70pS9tdv+AAw542zZ77713fvrTn25zX42aFLj11ltz0EEHpU+fPo15OADQxIzNALDztOSFBt+pRk0K3H///XnhhRdy//335+WXX06bNm3Sq1evfOhDH2rqfADAdjA2AwCN0ahJgauuuqrhz1OnTk3v3r390gEABTI2A8DO845W6G/hKvm5AQAAAGU0eqHBTT772c82RQ4AoIkYmwGgaW3PpQJ3VZoCAAAAUKXecVMAAAAAKlklX31AUwAAAACqlKYAAAAAlGFNAQAAAKDiaAoAAABAGZXbE9AUAAAAgKqlKQAAAABlWFMAAAAAqDiaAgAAAFCGpgAAAABQcZqlKdC6pnVzHGabTr5lQtERkiQnDjyo6Ah5YeUrRUdIktSWWsa81O9febroCEmSpa+/XnSEJEmvDh2KjpAk2bdTr6Ij5H1d31t0hCRJu1btio6QJLnliQeLjkATOXLvDxYdIUnyiyduLTpCkqRD6+L/H3v1jWVFR0iS7Nm+e9ERkiSL1rxWdIQkSZva2qIjJEne3XmfoiMkSXZvAf+vvLfLwKIjJEkO6/H+oiMkSW5/5q6iI7CLc/oAAAAAlFFy+gAAAABQaTQFAAAAoAwLDQIAAAAVR1MAAAAAyqjcnoCmAAAAAFQtTQEAAAAow5oCAAAAQMXRFAAAAIAyNAUAAACAiqMpAAAAAGWUNAUAAACASqMpAAAAAGVYUwAAAACoOJoCAAAAUEbl9gQ0BQAAAKBqmRQAAACAKuX0AQAAACjDQoMAAABAxdEUAAAAgDI0BQAAAICKoykAAAAAZZQ0BQAAAIBKoykAAAAAZVTyp+mV/NwAAACAMjQFAAAAoAxrCgAAAAAVR1MAAAAAyqjRFAAAAAAqjaYAAAAAlKEpAAAAAFQcTQEAAAAow9UHAAAAgIrTLE2B/37xL81xmG167169io6QJHljw9qiI+SRlxYWHSFJsmZ98a9FkuzbqXvREZIkffeoLzpCkpbxM5okf3ntxaIj5D1dW8b/K7966uGiIyRJxr33I0VHoIn8duGcoiMkSV5auazoCEmS9m3WFB0hC1esKDpCkuTofdsVHSFJ0rVdp6IjJEnatdqt6AhJkl7texYdIUny+OI/FR0hy9a2jP9XHl/0TNERkiRjD/xY0RGqQk00BQAAAIAKY1IAAAAAqpSFBgEAAKAMCw0CAAAAFUdTAAAAAMqo0RQAAAAAKo2mAAAAAJRRcklCAAAAoNJoCgAAAEAZrj4AAAAAVBxNAQAAACjD1QcAAACAiqMpAAAAAGWUKvjz9EY/s9tvvz0f/ehHM2rUqNx///1NGAkAaAxjMwCwoxrVFFi6dGm+973v5Ze//GXWrFmTqVOn5qijjmriaADA9jI2A8DOU8lrCjRqUmD27Nk5/PDD06FDh3To0CGXXnppU+cCAHaAsRkAaIxGnT6wcOHCvPHGGznzzDNz6qmnZvbs2U2dCwDYAcZmANh5SqVSs92aW6MXGly2bFm++93v5qWXXsrpp5+e++67r5AnAAC8ydgMAOyoRjUFunbtmoMPPjitWrVK37590759+7z22mtNnQ0A2E7GZgCgMRo1KXDEEUdkzpw5qaury9KlS7NmzZp07ty5qbMBANvJ2AwAO0+pGf9rbo06faBnz5459thjc/LJJydJLr744tTUVO51GwGgpTM2AwCN0eg1BUaPHp3Ro0c3ZRYA4B0wNgPAzlHJlyT0EQIAAABUqUY3BQAAAKAaVPLVfDQFAAAAoEppCgAAAEAZNRX8eXrlPjMAAACgLE0BAAAAKKOlrSkwefLkPPbYYymVSpkwYUIGDRrU8L2bbrop06dPT01NTQ444IBMnDixbH5NAQAAANhFzJ07N88//3ymTZuWSZMmZdKkSQ3fe/3113PnnXfmZz/7WX7xi1/kmWeeye9///uy+9MUAAAAgDJaUlNg9uzZGT58eJKkf//+Wb58eVatWpUOHTqkXbt2uf7665O8OUGwatWqdO/evez+NAUAAABgF7FkyZJ07ty54X6XLl2yePHizbb5wQ9+kBEjRmTkyJHp06dP2f2ZFAAAAIAyalJqttuOqq+vf9vXxo8fn3vvvTcPPvhg5s+fv43nBgAAAOwSevTokSVLljTcX7RoUcMpAsuWLcsjjzySJGnbtm2GDh2aRx99tOz+mmVNgbMP/mRzHGabbntqZtERkiRr1r9RdIQsX76q6AhJkpMOP7XoCEmS3y9+vOgISZLr5zxYdIQkSU1Ny5gvPHPI8KIj5NFX/lh0hCTJP/bsV3SEJMnXH7ih6AhJkuFjji86wi5vQ92GoiMkSfq9q0fREZIkazeuKzpCXiytLDpCkqRvx72KjpAkWbV+ddERkiQ/+8vDRUdIkrSqKb9QWHP5YO99io6QF1cuKjpCkmRon4OLjpAkufaPtxQdIUlyeM+jio6wU7WkNQWGDBmSqVOnZvTo0VmwYEF69OiRDh06JEk2bNiQCy64ILfffnvat2+fxx9/PB/96EfL7s9CgwAAALCLOOSQQzJw4MCMHj06pVIpEydOzIwZM9KxY8eMGDEiZ511Vk4//fS0atUqAwYMyNFHH112fyYFAAAAYBfypS99abP7BxxwQMOfR40alVGjRm33vkwKAAAAQBk1Lej0gabWMk4cBgAAAJqdpgAAAACUUWrEpQJ3FZoCAAAAUKU0BQAAAKCMmlLlfp5euc8MAAAAKEtTAAAAAMooufoAAAAAUGk0BQAAAKAMVx8AAAAAKo6mAAAAAJRRY00BAAAAoNJoCgAAAEAZ1hQAAAAAKo6mAAAAAJRhTQEAAACg4pgUAAAAgCrl9AEAAAAoo1Sq3M/TK/eZAQAAAGVpCgAAAEAZLkkIAAAAVBxNAQAAACjDJQkBAACAiqMpAAAAAGWUNAUAAACASqMpAAAAAGXUuPoAAAAAUGk0BQAAAKAMawoAAAAAFUdTAAAAAMoolSr38/TKfWYAAABAWc3SFDjv3h80x2G2qV3bNkVHSJJ0btu26Ag54X3/WHSEJMn5M39YdIQkyW4t5Gejtra26AhJkk8d9qGiIyRJ1m5cV3SErNnwRtERkiSL1iwrOkKS5Jj93lN0BJpIfX190RGSJH959W9FR0iS7L1Hp6IjpHVNy/is5srf3VZ0hCRJr44dio6QJFmxck3REZIkYw7+YNERkiQHdNmv6AhZtX510RGSJO1atSs6QpLkwG59io5QFVx9AAAAAKg4JgUAAACgSlloEAAAAMpwSUIAAACg4mgKAAAAQBklCw0CAAAAlUZTAAAAAMqwpgAAAABQcTQFAAAAoIwaawoAAAAAlUZTAAAAAMoolSr38/TKfWYAAABAWZoCAAAAUEapgtcUaNSkwOrVq/PlL385y5cvz/r163PWWWflf/2v/9XU2QCA7WRsBgAao1GTArfcckv69euXL37xi3nllVcybty43HXXXU2dDQDYTsZmANh5SqXKbQo0ak2Bzp07Z9myZUmSFStWpHPnzk0aCgDYMcZmAKAxGtUU+MhHPpIZM2ZkxIgRWbFiRa655pqmzgUA7ABjMwDsPJW8pkCjmgK33XZb9tprr9xzzz25/vrrc8kllzR1LgBgBxibAYDGaNSkwKOPPpojjjgiSXLAAQdk0aJF2bhxY5MGAwC2n7EZAGiMRk0K7LPPPnnssceSJC+++GLat2+f2traJg0GAGw/YzMA7DylUqnZbs2tUWsKnHLKKZkwYUI+9alPZcOGDfna177WxLEAgB1hbAYAGqNRkwLt27fPlClTmjoLANBIxmYA2HlqLDQIAAAAVJpGNQUAAACgWhRxrn9z0RQAAACAKqUpAAAAAGWUKvjz9Mp9ZgAAAEBZmgIAAABQhjUFAAAAgIqjKQAAAABllKIpAAAAAFQYTQEAAAAoo6aC1xRolkmBf9yrV3McZpt+v/CloiMkSfbt1KnoCJn17BNFR0iSLFu6sugISZIvHntC0RGSJI++8lTREZIkv1wwv+gISZIfHPfloiNkxPc+X3SEJMm+/fYqOkKS5B+6vVF0BJrIGxs3FB0hSdK6trboCEmSVevWFh0h72rbtugISZK6+vqiIyRJzj301KIjJEnueOaeoiMkSRYs+WvREZIkp+4/pugIOfmO84uOkCQ56/0ji46QJOnbsXfREdjFaQoAAABAGdYUAAAAACqOSQEAAACoUk4fAAAAgDJKFbzQoKYAAAAAVClNAQAAACijVMGfp1fuMwMAAADK0hQAAACAMqwpAAAAAFQcTQEAAAAooyaaAgAAAECF0RQAAACAMqwpAAAAAFQcTQEAAAAoo9TC1hSYPHlyHnvssZRKpUyYMCGDBg1q+N6cOXPyne98JzU1NenXr18mTZqUmpqt9wE0BQAA4P9r725jrKzPNIBfR2anFMEKlKOdRInLZpeNJGvdmF1etHUZt8nWNizdOpOJ6KduSNNUTZqVsE0xIdBC+FBIGzGCftCQjjviy6bNDmm3bNhmkDRmNdA1IB+IgjgzdkB5mdLi7AfTYU/VB7TDPDPP+f3IJJwz5JxrCMxNbq7n/wBMEvv27cuRI0fS3d2ddevWZd26dQ2f/853vpMtW7bkRz/6UU6fPp09e/YUvp6mAAAAABSYSGcK9PX1pb29PUkyb968nDx5MqdOncr06dOTJDt37hz9+axZszI0NFT4epoCAAAAMEkMDg5m5syZo49nzZqVgYGB0ce/Xwj09/fnF7/4RT73uc8Vvp6mAAAAABSoTeD/Tx8ZGXnfc2+99VZWrlyZNWvWNCwQPsjE/coAAACABvV6PYODg6OP+/v7M2fOnNHHp06dyte+9rXcf//9WbJkyUVfz1IAAAAAJonFixent7c3SXLgwIHU6/XRSwaS5Hvf+17uvffe3HbbbZf0ei4fAAAAgAJXTKCDBm+++ebceOON6ezsTK1Wy5o1a7Jz587MmDEjS5YsybPPPpsjR46kp6cnSXLnnXemo6PjQ1/PUgAAAAAmkW9961sNj+fPnz/68/3793+k17IUAAAAgAK1TJymwFhzpgAAAAA0KU0BAAAAKFCbQGcKjDVNAQAAAGhSmgIAAABQwJkCAAAAQOVoCgAAAEABZwoAAAAAlaMpAAAAAAWuqPD/p1f3KwMAAAAKaQoAAABAgSqfKTAuS4E/vfoz4/E2F/Xw+n8rO0KS5H9uuqbsCPnMnJllR0iSDL19quwISZKV6zaXHSFJcu38ifF3ZdonP1F2hCRJ98Fnyo6QT3/66rIjJEn+/Z8mxp/RZc88UHYExkjnX9xZdoQkSefja8uOkCS5ZcG8siPkrbNny46QJBnoHyo7QpLk3kPfLTtCkqTlT6aUHSFJsmLh4rIjJEm2vPRw2REy0H+i7AhJkr+55m/LjpAkeee375QdgUlOUwAAAAAK1FLdpoAzBQAAAKBJWQoAAABAk3L5AAAAABSo8kGDmgIAAADQpDQFAAAAoICDBgEAAIDK0RQAAACAApoCAAAAQOVoCgAAAEARdx8AAAAAqkZTAAAAAAo4UwAAAAConEtaChw8eDDt7e158sknkyRvvPFGVqxYka6urtx33305d+7cZQ0JADQymwFg/NRqtXH7GG8XXQqcOXMma9euzcKFC0ef27JlS7q6urJjx47MnTs3PT09lzUkAHCB2QwAjJWLLgVaW1vz6KOPpl6vjz73wgsvZOnSpUmS22+/PX19fZcvIQDQwGwGgPFVG8cf4+2iBw22tLSkpaXxl509ezatra1JktmzZ2dgYODypAMA3sdsBgDGyh9994GRkZGxyAEAjBGzGQDGlrsP/IFp06ZleHg4SfLmm2821BcBgPFnNgMAH8fHWgosWrQovb29SZJdu3bl1ltvHdNQAMBHYzYDAB/HRS8f2L9/fzZs2JCjR4+mpaUlvb292bRpU1atWpXu7u60tbVl2bJl45EVAIjZDADjrYxbBY6Xiy4FFixYkCeeeOJ9zz/++OOXJRAAUMxsBgDGyh990CAAAABUmYMGAQAAgMrRFAAAAIACmgIAAABA5WgKAAAAQIEq331AUwAAAACalKYAAAAAFHCmAAAAAFA5mgIAAABQwJkCAAAAQOVoCgAAAECBKp8pUBsZGRm53G/y19uWX+63uCR/1lYvO0KS5O9vuKnsCHnsxf8qO0KS5M/rny47QpLk3cv/1+CSnBgeLjtCkuSVw6+XHSFJ8p//vLXsCPnxkR+XHSFJsn3f7rIjJEmGh8+VHSFJ8tLK58qOMOn9Xc89ZUdIksyaOrXsCEmSh5c+VHaE/Mt/f7fsCEmSBXOuLztCkmTv0VfLjpAkGTh9uuwISZLf/vZ82RGSJE//48ayI+SK2pSyIyRJ7v2Pfy07QpLkqk98ouwISZLufyj/322X0/+eeGnc3usvr/6rcXuvRFMAAAAAClW5KeBMAQAAAGhSmgIAAABQwN0HAAAAgMqxFAAAAIAm5fIBAAAAKOCgQQAAAKByNAUAAACggKYAAAAAUDmaAgAAAFDALQkBAACAytEUAAAAgEKaAgAAAEDFaAoAAABAAWcKAAAAAJWjKQAAAAAFas4UAAAAAKpGUwAAAAAKaAoAAAAAlaMpAAAAAAXcfQAAAACoHEsBAAAAaFIuHwAAAIACDhoEAAAAKkdTAAAAAApoCgAAAACVoykAAAAABdySEAAAAKgcTQEAAAAo4EwBAAAAoHI0BQAAAKCAMwUAAACAyrEUAAAAgAK1cfxxKdavX5+Ojo50dnbm5Zdfbvjcb37zmzz44INZvnz5pX1tIyMjIx/5dwQAAACaxLEzR8btvdqmzS38/L59+7J9+/Y88sgjOXz4cFavXp3u7u7Rz69duzbXXXddnn/++ezcufOi76cpAAAAAIVq4/hRrK+vL+3t7UmSefPm5eTJkzl16tTo5x944IHRz18KSwEAAACYJAYHBzNz5szRx7NmzcrAwMDo4+nTp3+k13P3AQAAACgwke898MeeCKApAAAAAJNEvV7P4ODg6OP+/v7MmTPnY7+epQAAAABMEosXL05vb2+S5MCBA6nX6x/5koH/z90HAAAAoMDxs6+N23td+8nrLvprNm3alF/+8pep1WpZs2ZNfvWrX2XGjBm544478s1vfjPHjx/PoUOHsmDBgtx111350pe+9KGvZSkAAAAABSbaUmAsTYrLB9avX5+Ojo50dnbm5ZdfLjtO6TZu3JiOjo585Stfya5du8qOU7rh4eG0t7df0j04q+7555/Pl7/85Sxfvjy7d+8uO06pTp8+nW984xtZsWJFOjs7s2fPnrIjleLgwYNpb2/Pk08+mSR54403smLFinR1deW+++7LuXPnSk7IZGU2NzKbG5nNF5jNF5jN7zGbJ6uJc0vCsTbhlwL79u3LkSNH0t3dnXXr1mXdunVlRyrV3r17c+jQoXR3d2fbtm1Zv3592ZFK9/DDD+dTn/pU2TFKNzQ0lB/+8IfZsWNHtm7dmp/97GdlRyrVM888kxtuuCFPPPFENm/e3JTfO86cOZO1a9dm4cKFo89t2bIlXV1d2bFjR+bOnZuenp4SEzJZmc2NzOb3M5vfYzY3MpvNZiamCb8U6OvrS3t7e5Jk3rx5OXnyZE6dOlVyqvLccsst2bx5c5LkqquuytmzZ3P+/PmSU5Xn8OHDefXVV/P5z3++7Cil6+vry8KFCzN9+vTU6/WsXbu27EilmjlzZk6cOJEkefvttxvu5dosWltb8+ijj6Zer48+98ILL2Tp0qVJkttvvz19fX1lxWMSM5sbmc2NzOYLzOZGZrPZPJlVtycwCZYCg4ODDd8wZs2alYGBgRITlWvKlCmZNm1akqSnp8quKYAAAASzSURBVCe33XZbpkyZUnKq8mzYsCGrVq0qO8aE8Prrr2d4eDgrV65MV1dX0w+UL37xizl27FjuuOOO3H333XnwwQfLjjTuWlpaMnXq1Ibnzp49m9bW1iTJ7Nmzm/r7KR+f2dzIbG5kNl9gNjcym81mJqaWsgN8VM5FfM9Pf/rT9PT05LHHHis7SmmeffbZ3HTTTbnuuvE9iGMiO3HiRH7wgx/k2LFjueeee/Lzn/88tVoZ+8byPffcc2lra8v27dvzyiuvZPXq1a5t/QO+nzJW/Fl6j9lsNn8Qs/kCs/nifD+dyKr793bCLwXq9XoGBwdHH/f392fOnDklJirfnj17snXr1mzbti0zZswoO05pdu/enddeey27d+/O8ePH09rammuvvTaLFi0qO1opZs+enc9+9rNpaWnJ9ddfnyuvvDK//vWvM3v27LKjleLFF1/MkiVLkiTz589Pf39/zp8/39T/e5ck06ZNy/DwcKZOnZo333yzob4Il8psfj+z+T1mcyOzuZHZ/MHMZso24S8fWLx4cXp7e5MkBw4cSL1ez/Tp00tOVZ533nknGzduzCOPPJKrr7667Dil+v73v5+nn346Tz31VL761a/m61//etP+oyNJlixZkr179+bdd9/N0NBQzpw505TX6v3e3Llz89JLLyVJjh49miuvvLLp/9GRJIsWLRr9nrpr167ceuutJSdiMjKbG5nNF5jNjczmRmbzBzObJ4darTZuH+NtwjcFbr755tx4443p7OxMrVbLmjVryo5Uqp/85CcZGhrK/fffP/rchg0b0tbWVmIqJoJrrrkmX/jCF3LXXXclSb797W/niism/N7vsuno6Mjq1atz991353e/+10eeuihsiONu/3792fDhg05evRoWlpa0tvbm02bNmXVqlXp7u5OW1tbli1bVnZMJiGzuZHZzIcxmxuZzWYzE1NtxIUrAAAA8KH6h4+N23vVp47vUrl5V5UAAADQ5Cb85QMAAABQplqF7z6gKQAAAABNSlMAAAAACmgKAAAAAJVjKQAAAABNylIAAAAAmpSlAAAAADQpBw0CAABAgVrNQYMAAABAxVgKAAAAQJOyFAAAAIAm5UwBAAAAKFCLMwUAAACAitEUAAAAgEKaAgAAAEDFaAoAAABAger2BDQFAAAAoGlpCgAAAECBWq26XQFNAQAAAGhSmgIAAABQSFMAAAAAqBhLAQAAAGhSLh8AAACAAtW9eEBTAAAAAJqWpgAAAAAUqm5XQFMAAAAAmpSmAAAAABSo1TQFAAAAgIqxFAAAAIAmZSkAAAAATcqZAgAAAFCg5u4DAAAAQNVoCgAAAEAhTQEAAACgYjQFAAAAoEB1ewKaAgAAANC0NAUAAACgQK1W3a6ApgAAAAA0KUsBAAAAaFIuHwAAAIBCLh8AAAAAKkZTAAAAAApUtyegKQAAAABNS1MAAAAAClW3K6ApAAAAAE1KUwAAAAAK1GqaAgAAAEDFWAoAAABAk7IUAAAAgCblTAEAAAAoUHP3AQAAAKBqaiMjIyNlhwAAAADGn6YAAAAANClLAQAAAGhSlgIAAADQpCwFAAAAoElZCgAAAECTshQAAACAJvV/rRcT4SAelGsAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = np.zeros((n_layers, n_heads))\n",
+ " weights_base = np.zeros((n_layers, n_heads))\n",
+ "\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(example[1], example[2], max_len)\n",
+ " crop_len = torch.sum(input_mask != 0)\n",
+ " sep2_idx = crop_len - 1\n",
+ " sep1_idx = torch.sum(input_mask - segment_ids) - 1\n",
+ " \n",
+ " _, output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " _, output_base = model_base(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " \n",
+ " output_finetuned = output_finetuned.squeeze(0).detach().numpy()\n",
+ " output_base = output_base.squeeze(0).detach().numpy()\n",
+ " \n",
+ " for l in range(n_layers):\n",
+ " for h in range(n_heads): \n",
+ " weights_finetuned[l, h] = get_max_target_weight(output_finetuned[l, h, :crop_len, :crop_len], [sep1_idx, sep2_idx])\n",
+ " weights_base[l, h] = get_max_target_weight(output_base[l, h, :crop_len, :crop_len], [sep1_idx, sep2_idx])\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'Attention to [SEP]')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 4.8 Attention to CLS"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 55,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [05:36<00:00, 2.97it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAUAAAJ1CAYAAAChC0mPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3XuY1WW9N/73mhkQBTRAwERRQsVH2jxqaimGiqBou4PkATWkrMydZmlmbjTxEGSm+5J092Rnt5mOIp62Ceo28hCKYNuUZ5dGimcBBeSkCLN+f/hjHlFYwjgza7HW68U118Wa+c73/qwFM/dan/W+72+hWCwWAwAAANScunIXAAAAAJSHpgAAAADUKE0BAAAAqFGaAgAAAFCjNAUAAACgRmkKAAAAQI3SFAAA2twVV1yRAQMG5OGHHy53Ka3q+eefz4ABA5o/Bg8eXO6S1mvOnDlr1Tp06NBylwRABWgodwEAwPv74Q9/mF/96lfp2rVrHnjggXTq1Gmdx/3hD3/I6tWrM2zYsObPLVu2LL/61a8yZsyYbLnllm1e65NPPpk//elP+eIXv9j8ucMOOyw777xzdt555zYff33W9di0ln322SfHH398Nttss7U+XywWc+edd+a2227LE088kUWLFqVr16758Ic/nKFDh+aoo45K79691/qeAQMGpF+/fpkyZcoGjf3MM8/k17/+dR599NHMmzcvS5cuTZcuXbLTTjvl0EMPzbHHHpsOHTqkd+/emThxYpLkggsuaJ07DsAmT1MAACrcypUrc/PNN6euri5LlizJlClT8rnPfW6dx/7yl7/Mdtttt9YL38cffzxXXnlljjjiiHZpCkyZMiW33HLLWk2BnXbaKTvttFObj13Kuh6b1tKnT5+MGDFirc8tXrw43/jGN/Lwww9nt912ywknnJBtttkmr776ah5++OFceeWVueaaazJx4sR84hOfaNG4M2fOzFe+8pVsttlmGTlyZHbZZZfU1dXlpZdeyn/+539m/Pjx+eMf/5hf/OIX6dKlS3ONl1xyyQe+zwBUB00BAKhwd911VxYuXJjjjjsu1113XW644YZ1NgWampoye/bsbLfddmt9/vHHH2+vUssy3oZY32PTVorFYs4444w8/PDDOf300/O1r30thUKh+etf+tKXcv/99+fUU0/NaaedljvvvDM9evTY6HEuvvjivPnmm2lsbMyAAQPW+tqJJ56Yr371q3nggQdy//33Z8iQIR/4fgFQfewpAAAVrrGxMUkyZsyYfOxjH8usWbMyZ86ctY6ZPHly/tf/+l9Zvnx5br755gwYMCBXXHFFhg4dmksvvTRJcvDBB7/nheOkSZNy5JFH5n//7/+dPfbYI0cccUSuueaaNDU1NR+zZt38Oeeck6eeeipf+cpXstdee2XQoEE5/vjj88QTT6x13H333ZcXXnghAwYMyOjRo5Osf0+B++67L2PGjMlee+2Vj370oxk6dGi+//3v57XXXlvruKFDh2b48OFZsmRJzj333AwePDgf/ehHc9hhh+X2228v+fit77FZY86cOTnjjDMyePDgDBw4MPvtt19OO+20/O1vf3vff5v1mTZtWh544IEccsghOfnkk9dqCKzxyU9+MmeccUYOO+ywLF26tEXjPPnkk+ndu/d7/l2TpGPHjjn//PPzs5/9LP/0T//UovMDUP00BQCggj399NOZMWNG9thjj+y4447NCYEbb7xxreM+/vGPZ9y4cUneXt8+ceLEHHbYYRk3blz22WefJMm4ceOa15Qnb7/LfM4556RXr14599xzc9ZZZ6Vnz575/ve/n+9973vvqWXevHn50pe+lP79++ecc87J8ccfn//+7//OySefnJUrV6ZHjx6ZOHFiunfvnu7du2fixIn5xje+sd77dvPNN+ekk07K/Pnzc/LJJ+fCCy/MQQcdlOuuuy7HHXdcli9fvtbxTU1NOemkk/L666/n9NNPz7e+9a0sWrQoZ511VmbPnr3ecdb32CRvv6g++uij8+CDD+bII4/M+PHjc9xxx+WRRx7JqFGj8j//8z/rPW8pt9xyS5K3EwGljBkzJhdccEF22GGHFo3Ts2fPvPLKK+vdwLFfv3454IAD0q1btxadH4DqZ/kAAFSwG264IUny+c9/PsnbG/aNHz8+t9xyS84444x07Ngxydtr2tfEw9+5vn2nnXbKnXfemSQZMmRIc3z+r3/9a37961/nuOOOa37BnCTHHntsTjvttEyaNCnHH398dtttt+av3Xfffbn88subX1Anb6+bv+mmmzJr1qzsu+++GTFiRPN69XevsX+nN954Iz/4wQ/SvXv3XH/99c17HYwcOTIf/vCH86Mf/Si//e1vc9JJJzV/z/PPP58hQ4asVe/WW2+d7373u7nnnnsycODAdY61vscmeXtt/dKlS3P99ddnjz32aP78kCFDctRRR+Wyyy7LL37xi/Xej/X5y1/+kk6dOmXQoEEb/b0b48QTT8yFF16YE088MYccckiGDx+ej33sY+/ZvBAA1kdSAAAq1JoNBjfffPPmF+JdunTJIYcckoULF+aee+5p8bnXNAoOP/zwvP7662t9HHrooUmSGTNmrPU922yzzVoNgSTNsfT58+dv1PgzZszI4sWLc/jhh79n88ORI0cmeTuC/27vfue9peMnyfLly/Pggw9mwIABazUEkmTQoEHZZZddMn369Lz55psbfe4FCxakR48eaWho2/dfjj/++EyYMCHdu3fP73//+5x++ukZMmRIDj744Jx33nmZOXNmm44PwKZPUgAAKtSaDQY/85nPpEuXLs2fHzlyZG699dbccMMNOfzww1t07r///e9Jki984QvrPebFF19c63bfvn3fc8yaS/CtWrVqo8b/xz/+kSTZZZdd3vO17t2750Mf+lCeeeaZtT5fX1+fPn36tMr4STJ37tw0NTWt9zKJ/fr1y5NPPpnnn38+/fv336hz19XVpVgsbnRNLfH5z38+n/3sZ/Pwww/n4YcfzqxZs/KXv/wljY2NaWxszPDhw3PZZZe953KJAJBoCgBAxVqzweA+++yTuXPnNn9+m222ydZbb52HHnoozz33XLbffvuNPveyZcuSJP/2b/+Wrbfeep3H9OzZc63ba5YqtIY1+wVsvvnm6/x6p06d8vrrr6/1ufr6+tTX17daDWsegy222GKdX1/zIvrdextsiF69euWll17KypUrW/VxW5+GhoYMHjw4gwcPTpKsWLEi06ZNy+WXX5677747P//5z3Pqqae2eR0AbHo0BQCgAv3jH/9oju+fe+656z1u0qRJOf300zf6/J07d06SbL/99m2+7n1d1rwQX98L7hUrVjTX2FbWnL9UDe88bmPsscceefbZZzNjxozsv//+JY9duHBhq28EuGbJycCBAzN8+PDcd999mgIArJM9BQCgAq3ZYPCoo47KxIkT3/NxySWXpL6+PjfddFOLovM77bRTkuTRRx99z9eWLVvWonX0LRn/ySeffM/X5s+fn8WLF+cjH/lIm9aw4447pr6+fp01JG8vsejYsWPz5owb44gjjkiS/PSnPy25jOCmm27K0KFDc/fdd2/0GA899FDOO++8zJo1a73HbLvttikUCs0NDgB4N00BAKgwazYY7NixY84444yMGDHiPR+f/exnM2zYsMyfP795Q7410fp3v6Cvq6t7z+fXbBh43XXX5Y033ljr+B/96Ef5xCc+kWeffbZF9dfV1b1vU2GfffZp3hxv8eLFa31tzbKJNRsetoZ1PTabb755DjjggDz55JPv2ZBvxowZefrpp3PQQQe1KP6/7777ZtiwYXnkkUdy4YUX5q233nrPMX/84x9z4YUXZosttshee+210WMsW7YsjY2NmTBhQpYuXbrOY/7jP/4jxWKxeVkBALyb5QMAUGGmTp2aRYsWZeTIkenevft6j/vCF76QqVOn5sYbb8ywYcOy9dZbp1OnTrn//vtz1VVXZYcddsiIESOa3+n+0Y9+lL333juf/exns+uuu2bMmDG5+uqrc+yxx+aYY45JQ0ND/vjHP+auu+7KZz7zmXVuLLghtttuu0yfPj0/+MEP8uEPfzhf/OIX33NMx44d873vfS9nnHFGjj/++Bx55JHp2rVrHnvssdx4443Zfffdc9RRR7Vo/HVZ32Nz1llnZebMmTnllFMyevTobLfddnnmmWfyu9/9Lt26dct3vvOdFo95ySWX5Iwzzsjvfve7/OlPf8qnP/3p9O3bN6+99lqmT5+eadOmpW/fvvnpT3/6nuUDy5Yty5QpU9Z53i233DL77bdfDj744Bx//PG59tprc8ghh+Szn/1sBgwYkPr6+ixYsCD33Xdf/vSnP2W33XbL17/+9RbfDwCqm6YAAFSYNe+UjxkzpuRx++yzT3bZZZfcf//9efnll7PNNtvk7LPPzuWXX56f/OQnOfroozNixIiMGjUqDzzwQB544IE88cQTOeSQQ5IkY8eOzc4775zGxsb84Ac/SFNTU3bcccd85zvfWecL+Q31zW9+My+88EKuvfbaDBgwYL3nOvzww7PVVlvlqquuyo9//OOsXLkyffr0yUknnZSvfe1rrbpBX4cOHdb52PTr1y833HBDfvzjH+d3v/tdFi9enG7dumXo0KE55ZRTWrSJ4xqdO3fOVVddlbvvvju33HJLrr/++ixatCgdO3bMzjvvnPPPPz+f+9zn1rnZ4rx58/LNb35znefdddddc+uttyZJzjvvvBxwwAG56aabMnXq1FxzzTUpFovZcssts8suu2TcuHE58sgj22WzQwA2TYVie10vBwCgyjz//PM5+OCDc8QRR+Tiiy8udzkbbOjQoUmSe++9t8yVAFBu9hQAAACAGmX5AADAB/TCCy9kypQp2WyzzXLQQQeVu5x1Wrp0aR544IEkb19ucV3LFgCoPZoCAAAf0IwZMzJjxoxsvfXWefDBB8tdzjq98sora+1T0KdPnzJWA0ClsKcAAAAA1Ch7CgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADVKUwAAAABqlKYAAAAA1ChNAQAAAKhRmgIAAABQozQFAAAAoEZpCgAAAECN0hQAAACAGqUpAAAAADWqodwFAAAAQCUrDN+u3cYq3v18u42VSAoAAABAzdIUAAAAgBpl+QAAAACUUiiUu4I2IykAAAAANUpSAAAAAEqp4rfTq/iuAQAAAKVICgAAAEAp9hQAAAAAqo2kAAAAAJRSvUEBSQEAAACoVZICAAAAUIo9BQAAAIBqIykAAAAApVTx2+lVfNcAAACAUjQFAAAAoEZZPgAAAACl2GgQAAAAqDaSAgAAAFBK9QYFJAUAAACgVkkKAAAAQCl11RsVkBQAAACAGiUpAAAAAKVUb1BAUgAAAABqlaQAAAAAlFKo3qiApAAAAADUKEkBAAAAKKV6gwKSAgAAAFCrJAUAAACglLrqjQpICgAAAECNkhQAAACAUqo3KCApAAAAALVKUwAAAABqlOUDAAAAUEqhetcPSAoAAABAjZIUAAAAgFJckhAAAACoNpICAAAAUEr1BgUkBQAAAKBWSQoAAABAKa4+AAAAAFQbSQEAAAAopXqDApICAAAAUKskBQAAAKCUuuqNCkgKAAAAQI2SFAAAAIBSqjcoICkAAAAAtUpSAAAAAEopVG9UQFIAAAAAapSkAAAAAGxCJkyYkMceeyyFQiFjx47NoEGDmr927bXX5rbbbktdXV0++tGP5pxzzil5Lk0BAAAAKKWCMvYzZszI3Llz09jYmDlz5mTs2LFpbGxMkixdujS//OUvc9ddd6WhoSEnnnhi/vu//zu77777es9XQXcNAAAAKGX69OkZNmxYkqR///5ZvHhxli5dmiTp0KFDOnTokOXLl2fVqlVZsWJFttpqq5LnkxQAAACAUipoo8EFCxZk4MCBzbe7d++e+fPnp0uXLtlss81yyimnZNiwYdlss83yqU99Kv369St5PkkBAAAA2EQVi8Xmvy9dujRXXXVVpkyZkv/6r//KY489lr/+9a8lv19TAAAAAEoptOPH++jVq1cWLFjQfHvevHnp2bNnkmTOnDnZfvvt071793Ts2DF77bVXnnjiiZLn0xSAdRgwYECGDx+eESNG5NBDD81XvvKVPPfcc+Uuq0WGDx+ehx9+uOQxZ599dn7yk5+0U0UA1Kp3zq9rPr785S8nScaMGZPZs2e32lj3339/XnzxxVY7XynnnHNOrrjiinYZa7fddsvzzz9f8pjRo0fn1ltvbZd6gPY3ePDgTJ06NUkye/bs9OrVK126dEmS9OnTJ3PmzMkbb7yRJHniiSey4447ljyfPQVgPa655ppss802SZLLLrss48ePz09/+tMyVwUAm7Z3zq/vdPXVV7fqOL/5zW/yL//yL9l2221b9bxAjaqgPQX23HPPDBw4MKNGjUqhUMi4ceMyefLkdO3aNcOHD8+Xv/zlnHDCCamvr88ee+yRvfbaq+T5NAVgA3ziE5/Ivffe23z7xhtvzK9+9ausXr06PXv2zCWXXJI+ffrklVdeyVlnnZX58+dn5cqV+dSnPpXTTz89xWIx//7v/57bb789K1euzMEHH5x//dd/TX19/VrjTJ48OdOmTUuHDh0ya9as9OvXL6ecckouvfTSPPvss/nmN7+ZY445Jk1NTZk4cWJzh3D33XfPeeedly222CJPPPFEvvvd72bVqlU54IAD1jr/Pffck4kTJ2b58uXZYYcdcumll6Z79+5t/wACwPsYOnRoLrnkkmyzzTYZNWpUTjrppNx4441ZtGhR/vVf/zWHH374Bs+nl19+eR566KH84x//yHe+853cd9996du3b77+9a8neTsht+b20KFDc9JJJ2XSpEl5+eWX88///M85++yzk6x/3ly4cGG+/e1v55lnnslOO+2UTp06rbPRccUVV2TBggV5+eWXM3v27Oy77745/PDDc8UVV2TevHm56KKLctBBB+XNN9/M+PHj8/DDD6euri4HHHBAvvOd76S+vj5//OMf8/3vfz8NDQ35/Oc/v9b5Gxsb8+tf/zorV67M7rvvngkTJqRTp05t9C8EVJIzzzxzrdu77rpr899HjRqVUaNGbfC5LB+A97Fy5crcdtttGTp0aJLk1VdfzYUXXphf//rXueuuu9K3b9/m6P1vfvOb7L333vn973+f22+/Pc8991zmzZuXW2+9NVOmTMmkSZNy991357nnnst11123zvHuv//+nHrqqbnrrrsyZ86c/OIXv8i1116b8ePHN49z55135r777svkyZNzxx135PXXX89vfvObJMn555+fE044IVOnTs0ee+zRHDF87rnnctZZZ+Wyyy7Lf/3Xf+XjH/94zj///LZ98ACgBRYuXJi6urrcfvvtGTt2bC6//PIk2eD59Fvf+lZ69+6dH/3oRzn88MPfd7xHHnkkjY2Nuemmm/Lb3/42L7/8csl58+c//3m6deuWe++9N+edd14eeOCB9Z572rRpmTBhQm6//fZMmTKlef4++eST8/Of/zzJ2ymJl19+OXfccUduvvnmzJw5M//5n/+Z1atX55xzzsm4ceNy5513pq6uLqtXr06SzJw5MxMnTszVV1+de++9N126dMnEiRM39qEGNlRdO360M00BWI/Ro0dnxIgRGTx4cB5//PGMHDkySdKjR4/MmjWr+R2Bvfbaq3m/gR49euSBBx7IzJkz07Fjx/zbv/1bevXqlT/84Q/5/Oc/n65du6ahoSFHHXVU7rrrrnWOu9NOO6Vfv37p2LFjdthhh+y///6pr6/PLrvsknnz5iV5+wnG5z73uWyxxRapr6/PyJEj8+CDD+bNN9/M448/3vwEaMSIEdl8882TJPfdd1/22Wef7LLLLkne7iDee++9zU8uAKA9rJlf13yce+657zlm1apVzfPuwIEDm/cG2Jj5dGN8+tOfTn19fXr37p0ePXrkpZdeKjlvzpw5M4cddliSZLvttss+++yz3nPvscce6dGjR7p165aePXtmyJAhSfKeef3oo49OQ0NDOnXqlE9/+tN58MEH88wzz2TlypXZf//9kyRHHHFE83nvvffeHH744endu3eS5Nhjj22VxwKoPZYPwHq8c83jI488ktGjR2fy5Mnp0aNHfvzjHzc/MVi2bFnztT+/+MUvpqmpKRdccEHmzZuX448/Pt/4xjeyZMmS/PKXv0xjY2OSZPXq1euN7Xfu3Ln57/X19dliiy2a/97U1JQkee2117LVVls1H7fVVlvl1VdfzaJFi5KkeaORQqGQLbfcMkmyZMmSzJw5MyNGjGj+vi5dujR/DwC0h/XtKfBO75z/6urqmue/9c2nr7zySsaMGZMkGTRoUC655JKNqmnNvLlm7NWrV5ecNxcvXpyuXbs2f37NXLsu65vX33m/1jevL168eK3a3nnMkiVLcvfddzenFIrFYt56662Nut/ARqigPQVam6YAbIC999472267bWbNmpVVq1bl3nvvzW9/+9t07949N9xwQ26//fYkSUNDQ0466aScdNJJefrpp/PVr341H/vYx9KrV68MHTo0X/jCF1qlnq233nqtF/OLFi3K1ltv3fxkYenSpenatWuampqyePHiJG9fumS//fbLj3/841apAQDaW6n5dMqUKev9vne+AE/SPDe+31jrmze33HLLLFmypPn2a6+9lu233/59z7k+peb1pUuXrjXOO+s74ogj8t3vfrfF4wIklg/ABnn66afz9NNP5yMf+UheffXV9OnTp3mjoTvvvDPLli1Lkpx33nl58MEHkyR9+/bN1ltvnUKhkIMPPji33nprVqxYkSS5/vrrc/PNN7e4ngMPPDC33XZbVqxYkVWrVmXSpEk54IAD0qlTp+y66665++67kyR33HFH3nzzzSTJ/vvvn5kzZzYvdfjLX/6S73//+y2uAQDa28bMpw0NDc0v3Hv27Jm//vWvSd7eY+fRRx9937FKzZu777577rnnniTJs88+m1mzZn2g+3XggQdm0qRJWb16dZYvX55bb701BxxwQPr27Zv6+vrmSwtPnjw5hf//3cqhQ4fmrrvuam4U3HPPPfnZz372geoASii040c7kxSA9Rg9enTzbsYdO3bMBRdckAEDBqRHjx654447Mnz48Gy//fb51re+lX/5l3/JxRdfnFGjRuW8887LRRddlGKxmKFDh2bfffdNkjz11FPNawH79u2b8ePHt7i2ESNG5G9/+1tGjhyZYrGYj3/84znhhBOSvL3R4NixY3PVVVdlyJAh6d+/f5K331G46KKLcsopp+Stt95K586dM3bs2A/yEAFAuxo2bNgGz6eHHnpozjjjjJx22mk5+uijc+qpp+aQQw7JbrvtlkMPPfR9xyo1b37ta1/L6aefnqFDh6Z///455JBDPtD9Gj16dJ577rl86lOfSqFQyIgRI3LYYYelUCjkoosuytixY9OxY8eMHDmyefnBwIEDc/LJJ2f06NFpampKjx49csEFF3ygOoDaVCgWi8VyFwEAAACVqnDqR9ttrOKVT7TbWInlAwAAAFCzNAUAAACgRtlTAAAAAEqp4ksSSgoAAABAjWqXpMDCNxe0xzDvq5jK2FPx/pf+UO4S8skPH1juEpIkq5pWlbuEJMnqYmXU0bmha7lLSJK8suLFcpeQJOm9+bblLiEvr3ih3CUkSbpvtnW5S0iS/HXR7HKXkCTZf5uDy13CJu/ZpX8vdwlJkg51HctdQpLkrPt/WO4Scs7Hv1buEpIkH9qse7lLSJLMnDej3CUkqZznTK+vXFjuEpIkW3Us//+PGfOnl7uEJMmOXXcsdwlJkvtevL/cJSRJTty1Mn6HtZnqDQpICgAAAECtsqcAAAAAlFCwpwAAAABQbSQFAAAAoARJAQAAAKDqSAoAAABACVUcFJAUAAAAgFolKQAAAAAl1FVxVEBSAAAAAGqUpAAAAACU4OoDAAAAQNWRFAAAAIASJAUAAACAqtPipMCECRPy2GOPpVAoZOzYsRk0aFBr1gUAbCRzMwCwsVrUFJgxY0bmzp2bxsbGzJkzJ2PHjk1jY2Nr1wYAbCBzMwC0HcsH3mX69OkZNmxYkqR///5ZvHhxli5d2qqFAQAbztwMALREi5oCCxYsSLdu3Zpvd+/ePfPnz2+1ogCAjWNuBoC2Uyi030d7a5WNBovFYmucBgBoJeZmAGBDtGhPgV69emXBggXNt+fNm5eePXu2WlEAwMYxNwNA27GnwLsMHjw4U6dOTZLMnj07vXr1SpcuXVq1MABgw5mbAYCWaFFSYM8998zAgQMzatSoFAqFjBs3rrXrAgA2grkZANpONScFWtQUSJIzzzyzNesAAD4gczMAsLFa3BQAAACAWlBI9SYFWuXqAwAAAMCmR1IAAAAASqjmPQUkBQAAAKBGSQoAAABACVUcFJAUAAAAgFolKQAAAAAl1FVxVEBSAAAAAGqUpAAAAACU4OoDAAAAQNXRFAAAAIAaVSgWi8W2HuTWZ25o6yE2yF9fm1PuEpIkW27WpdwlZMaLT5a7hCRJvw/1LHcJSZLeW3QvdwlJkpO/dXG5S0iSTP7ZZeUuIUllbOjy9OLnyl1CkqRrx87lLiFJ8s87/nO5S0iS9N58u3KXsMm794U7y11CkuS1NxeWu4Qkyesrl5S7hDz3+kvlLiFJ8sk+Hy93CUmSt5reKncJSZKrZ08pdwlJkq/806fk4X26AAAgAElEQVTLXUKSZKvNtip3CWkqNpW7hCTJU4v+Xu4SkiSHbD+i3CUkSbbutE25S2hTPb63X7uN9epFf2q3sRJJAQAAAKhZNhoEAACAEiogwNpmJAUAAACgRkkKAAAAQAkuSQgAAABUHUkBAAAAKEFSAAAAAKg6kgIAAABQQqUlBSZMmJDHHnsshUIhY8eOzaBBg5Ikr7zySs4888zm45577rl8+9vfzqc//en1nktTAAAAADYRM2bMyNy5c9PY2Jg5c+Zk7NixaWxsTJL07t0711xzTZJk1apVGT16dIYOHVryfJoCAAAAUEIlJQWmT5+eYcOGJUn69++fxYsXZ+nSpenSpctax91888059NBD07lz55Lns6cAAAAAbCIWLFiQbt26Nd/u3r175s+f/57jbrzxxhx55JHvez5JAQAAACihgoIC71EsFt/zuT//+c/5yEc+8p70wLpICgAAAMAmolevXlmwYEHz7Xnz5qVnz55rHTNt2rTsu+++G3Q+TQEAAAAooVAotNvH+xk8eHCmTp2aJJk9e3Z69er1nkTA448/nl133XWD7pvlAwAAALCJ2HPPPTNw4MCMGjUqhUIh48aNy+TJk9O1a9cMHz48STJ//vz06NFjg86nKQAAAACbkDPPPHOt2+9OBdx+++0bfC5NAQAAACihki5J2NrsKQAAAAA1SlIAAAAASqiTFAAAAACqjaQAAAAAlFDFQQFJAQAAAKhVkgIAAABQgqsPAAAAAFVHUgAAAABKKERSAAAAAKgykgIAAABQgj0FAAAAgKojKQAAAAAlSAoAAAAAVUdSAAAAAEqo4qCApAAAAADUKk0BAAAAqFHtsnzgE733a49h3tf/vPr3cpeQJPnVw9PKXUK+sf9h5S4hSfL4/KfKXUKSZKcP9S13CUmSl26dXu4SkiR3zL2j3CUkSea+/mK5S8jX/+mr5S4hSfIff7u23CUkSaY8e2e5S0iSjBlQGf8um7I9e+5T7hKSJLc8fVO5S0iSvLZicblLyOA+lfFvMm/F/HKXkCTp0alHuUtIkvzkoAvLXUKS5OF5fyp3CUmSuUueLXcJOXDboeUuIUny2huvlbuEJMmfF8wqdwlJkuHbfarcJbQpGw0CAAAAVcdGgwAAAFCCpAAAAABQdSQFAAAAoARJAQAAAKDqSAoAAABACVUcFJAUAAAAgFolKQAAAAAl2FMAAAAAqDqSAgAAAFCCpAAAAABQdSQFAAAAoARJAQAAAKDqtDgpcMkll2TWrFlZtWpVvva1r+WQQw5pzboAgI1kbgaAtlHFQYGWNQUeeuihPPXUU2lsbMzChQtzxBFHeOIBAGVkbgYAWqJFTYG99947gwYNSpJsueWWWbFiRVavXp36+vpWLQ4A2DDmZgCgJVrUFKivr88WW2yRJJk0aVKGDBniSQcAlJG5GQDaTjVvNPiBrj5wzz33ZNKkSfnVr37VWvUAAB+AuRkA2Bgtbgrcf//9+elPf5pf/OIX6dq1a2vWBAC0gLkZANqGpMC7LFmyJJdcckl+85vf5EMf+lBr1wQAbCRzMwDQEi1qCvz+97/PwoUL861vfav5cz/84Q+z7bbbtlphAMCGMzcDQNuRFHiXY445Jsccc0xr1wIAtJC5GQBoiQ+00SAAAABUuyoOCqSu3AUAAAAA5SEpAAAAACVU854CkgIAAABQoyQFAAAAoBRJAQAAAKDaSAoAAABACfYUAAAAAKqOpAAAAACUUMVBAUkBAAAAqFWFYrFYbOtBnlo8u62H2KQ8v+y5cpeQbpt1K3cJSZKuHbYsdwlJkodeeajcJSRJdvnQzuUuIUlSTJv/WtggW2/Ws9wl5L6X7it3CUmST374k+UuIUny98VPlbuEJMmh23+m3CVs8l5Z8Xy5S0iSNBWbyl1CkmT+G6+Uu4T02aJvuUtIkrz+1qJyl5AkWV1cXe4SkiQrV68sdwlJkoa6ygj4btHQudwl5NU35pe7hCRJ3y4fKXcJSZKFby4odwlJko9suWu5S2hTe/78iHYb69Gv3txuYyWWDwAAAEBJ1bzRoKYAAAAAbEImTJiQxx57LIVCIWPHjs2gQYOav/bSSy/ljDPOyFtvvZXddtstF154Yclz2VMAAAAASigUCu328X5mzJiRuXPnprGxMePHj8/48ePX+vrFF1+cE088MZMmTUp9fX1efPHFkufTFAAAAIBNxPTp0zNs2LAkSf/+/bN48eIsXbo0SdLU1JRZs2Zl6NChSZJx48Zl2223LXk+TQEAAAAooZKSAgsWLEi3bv9v4/ju3btn/vy3N+B87bXX0rlz5/zgBz/Isccem8suu+x9z6cpAAAAAJuod15QsFgs5pVXXskJJ5yQ3/72t/m///f/Ztq0aSW/X1MAAAAASigU2u/j/fTq1SsLFvy/S1HOmzcvPXu+fSnvbt26Zdttt03fvn1TX1+ffffdN089VfqS0poCAAAAsIkYPHhwpk6dmiSZPXt2evXqlS5duiRJGhoasv322+eZZ55p/nq/fv1Kns8lCQEAAKCEDVnr31723HPPDBw4MKNGjUqhUMi4ceMyefLkdO3aNcOHD8/YsWNz9tlnp1gsZpdddmnedHB9NAUAAABgE3LmmWeudXvXXXdt/vsOO+yQ6667boPPpSkAAAAAJVRSUqC12VMAAAAAapSkAAAAAJQgKQAAAABUHUkBAAAAKEFSAAAAAKg6mgIAAABQoywfAAAAgBKqePWApAAAAADUKkkBAAAAKMFGgwAAAEDVkRQAAACAEiQFAAAAgKojKQAAAAAlSAoAAAAAVUdSAAAAAEqo4qCApAAAAADUKkkBAAAAKMGeAgAAAEDVkRQAAACAUiQFAAAAgGojKQAAAAAl2FMAAAAAqDqFYrFYbOtBlq9a2tZDbJBKqaMSdKrfvNwlJEkKhcroS61uWlXuEpJUdweyJdr8l9MGWNW0stwlJEka6jqWu4QkSV0q4/9olw5blbuETV6lzInLVi0pdwlJkg4V8DPWodCh3CUkqYzfvYk58d0KFfL7t1hsKncJeaPpjXKXkCTZvH6LcpdQUbZo6FLuEtrUQTee0G5j/eGo/2i3sRLLBwAAAKCkusroy7WJynibFgAAAGh3kgIAAABQQjUvaZIUAAAAgBolKQAAAAAl1EkKAAAAANVGUgAAAABKsKcAAAAAUHUkBQAAAKCEan43vZrvGwAAAFCCpAAAAACU4OoDAAAAQNWRFAAAAIASXH1gPd54440MGzYskydPbq16AIAPwNwMAGyMD5QU+D//5/9kq622aq1aAIAPyNwMAK3PngLrMGfOnPz973/PgQce2IrlAAAtZW4GADZWi5sCP/zhD3P22We3Zi0AwAdgbgYANlaLlg/ccsst2X333bP99tu3dj0AQAuYmwGg7VTzRoMtagpMmzYtzz33XKZNm5aXX345HTt2zDbbbJP99tuvtesDADaAuRkAaIkWNQUuv/zy5r9fccUV6dOnjycdAFBG5mYAaDsf6LJ9Fa6a7xsAAABQwge6JGGSfOMb32iNOgCAVmJuBoDW5ZKEAAAAQNX5wEkBAAAAqGbVfPUBSQEAAACoUZICAAAAUII9BQAAAICqIykAAAAAJVRvTkBSAAAAAGqWpAAAAACUUGl7CkyYMCGPPfZYCoVCxo4dm0GDBjV/bejQodlmm21SX1+fJLn00kvTu3fv9Z5LUwAAAAA2ETNmzMjcuXPT2NiYOXPmZOzYsWlsbFzrmJ///Ofp3LnzBp1PUwAAAABKqKSkwPTp0zNs2LAkSf/+/bN48eIsXbo0Xbp0adH57CkAAAAAm4gFCxakW7duzbe7d++e+fPnr3XMuHHjcuyxx+bSSy9NsVgseb52SQoUU7qI9rJFQ8s6J62tUh6PSrC6aVW5S0iS1BUqoz9WqJA6KkWhAvZ5XbMWi7c1FVeXuwSqTJeGLctdQpJkdbH881GlzAGV8FgkSYNAa0WqhP+nneo6lbuEJJXznL4Sni9RXu9+0X/aaaflk5/8ZLbaaquccsopmTp1akaMGLHe7y//TzUAAABUsEKh0G4f76dXr15ZsGBB8+158+alZ8+ezbc/97nPpUePHmloaMiQIUPy5JNPljyfpgAAAABsIgYPHpypU6cmSWbPnp1evXo17yewZMmSfPnLX87KlSuTJI888kh23nnnkueTywIAAIASKmmjwT333DMDBw7MqFGjUigUMm7cuEyePDldu3bN8OHDM2TIkBxzzDHZbLPNsttuu5VcOpAkheL77TrQCpatWtLWQ2yQSllvUynrjyqBPQXWVgnr9CpJJfzM+nldW6XsKdC1w4fKXcImb/mqpeUuIUll/JwnlbGOvq5QGXuYrKqAxyJJGgreu2LdisWmcpfwtgp5kVgpv0crZf+2tnLCXae221j/cciV7TZWIikAAAAAJVVG66VteFsSAAAAapSkAAAAAJRQSXsKtDZJAQAAAKhRkgIAAABQgqQAAAAAUHUkBQAAAKCEgqQAAAAAUG0kBQAAAKAEewoAAAAAVUdSAAAAAEqo3pyApAAAAADULE0BAAAAqFGWDwAAAEAJNhoEAAAAqo6kAAAAAJQgKQAAAABUHUkBAAAAKKEgKQAAAABUG0kBAAAAKKGa302v5vsGAAAAlCApAAAAACXYUwAAAACoOpICAAAAUEKdpAAAAABQbSQFAAAAoARJAQAAAKDqSAoAAABACa4+AAAAAFSddkkK1FVI72FV8a1yl5CkMh6PukJ9uUtIktTXb1buEipKU7Gp3CVUlKbi6nKXkKYUy11CkqShUBnBroZCh3KXQJUpVsjPWCXMi4VUxrtQHes6lruEilIsVsb/0Ur5WakIFfKObX0F/N5IktUV8HypFtRVyO/otlD+V6cAAABAWWgKAAAAQI2qjDwqAAAAVCgbDQIAAABVR1IAAAAASqiTFAAAAACqjaQAAAAAlFApl41tC5ICAAAAUKMkBQAAAKAEVx8AAAAAqo6kAAAAAJTg6gMAAABA1ZEUAAAAgBIKVfx+eovv2W233ZbPfOYzGTlyZKZNm9aKJQEALWFuBgA2VouSAgsXLsy///u/56abbsry5ctzxRVX5MADD2zl0gCADWVuBoC2U817CrSoKTB9+vTsu+++6dKlS7p06ZKLLrqotesCADaCuRkAaIkWLR94/vnn88Ybb+Tkk0/Occcdl+nTp7d2XQDARjA3A0DbKRQK7fbR3lq80eCiRYty5ZVX5sUXX8wJJ5yQP/zhD2W5AwDA28zNAMDGalFSoEePHtljjz3S0NCQvn37pnPnznnttddauzYAYAOZmwGAlmhRU2D//ffPQw89lKampixcuDDLly9Pt27dWrs2AGADmZsBoO0U2vFPe2vR8oHevXvn0EMPzdFHH50kOffcc1NXV73XbQSASmduBgBaolAsFottPciKVcvaeogNsqr4VrlLSJLUtSyg0bo1FOrLXUKSWOv6Lk3FpnKXUFGaiqvLXUKa0ua/IjdIQ6HFW8C0qnJ0r9dl84bO5S5hk7d81dJyl8C7VMrPl7l5be3wVHmDFCtkPqoElfJY1FfI8+nVFfB8KUk6N3Qtdwlt6vsz2++qPufu9b12Gytp4fIBAAAAoDwmTJiQY445JqNGjcpf/vKXdR5z2WWXZfTo0e97rsp46wkAAAAqVCWlqGbMmJG5c+emsbExc+bMydixY9PY2LjWMX//+9/zyCOPpEOHDu97PkkBAAAA2ERMnz49w4YNS5L0798/ixcvztKlay8LvPjii3P66adv0Pk0BQAAAKCEunb8834WLFiw1hWGunfvnvnz5zffnjx5cvbZZ5/06dNnA+8bAAAAsEl654aoixYtyuTJk/OlL31pg7/fngIAAABQQiXtKdCrV68sWLCg+fa8efPSs2fPJMlDDz2U1157Lccff3xWrlyZZ599NhMmTMjYsWPXez5JAQAAANhEDB48OFOnTk2SzJ49O7169UqXLl2SJCNGjMjvf//73HDDDbnyyiszcODAkg2BRFIAAAAASqqkpMCee+6ZgQMHZtSoUSkUChk3blwmT56crl27Zvjw4Rt9vkLxnQsQ2siKVcvaeogNsqr4VrlLSJIN2jyizWso1Je7hCSV9cNVCZqKTeUuoaI0FVeXu4Q0pc1/RW6QhkJl9HALqYyf2c0bOpe7hE3e8lVL3/8g2lWl/HyZm9fWDk+VN0ixQuajSlApj0V9hTyfXl0Bz5eSpHND13KX0KZ++OeL222s7+5xdruNlUgKAAAAQEl1FdK4bQvlf8saAAAAKIt2SQq8/tbC9hjmfXXt8KFyl8C7iMuvrVKio5USy6uEOFyHuo7lLiFJUleojB6un1laW6X8364EFRNTr5A6LGNYW6X8rFTC/49Keb5UKSplGUO1q+bfSZXx2wUAAABod5oCAAAAUKNsNAgAAAAl1Fk+AAAAAFQbSQEAAAAooZo3uJQUAAAAgBolKQAAAAAlVMplQdtC9d4zAAAAoCRJAQAAACih4OoDAAAAQLWRFAAAAIASXH0AAAAAqDqSAgAAAFBCnT0FAAAAgGojKQAAAAAl2FMAAAAAqDqSAgAAAFCCPQUAAACAqqMpAAAAADXK8gEAAAAooVCo3vfTq/eeAQAAACVJCgAAAEAJLkkIAAAAVB1JAQAAACjBJQkBAACAqiMpAAAAACUUJAUAAACAaiMpAAAAACXUufoAAAAAUG0kBQAAAKAEewoAAAAAVUdSAAAAAEooFKr3/fTqvWcAAABASe2SFOjSYav2GOZ9vbF6RblLSJJsXr9FuUvI8tXLyl1CkqRT/eblLiFJUl+oL3cJSZJisVjuEpIkhQrZXbVDXcdyl5CmYlO5S0iS1FVId7pSHg8+OP+n1lYJj0cx5oBKVM3riDdV/k3WVinPH6udqw8AAAAAVUdTAAAAAGqUjQYBAACghGpetiIpAAAAADVKUgAAAABKqOZNWCUFAAAAoEZJCgAAAEAJ9hQAAAAAqo6kAAAAAJRQZ08BAAAAoNpICgAAAEAJhUL1vp9evfcMAAAAKElSAAAAAEooVPGeAi1qCixbtizf/e53s3jx4rz11ls55ZRT8slPfrK1awMANpC5GQBoiRY1BW6++eb069cv3/72t/PKK69kzJgxmTJlSmvXBgBsIHMzALSdQqF6kwIt2lOgW7duWbRoUZLk9ddfT7du3Vq1KABg45ibAYCWaFFS4FOf+lQmT56c4cOH5/XXX89VV13V2nUBABvB3AwAbafS9hSYMGFCHnvssRQKhYwdOzaDBg1q/toNN9yQSZMmpa6uLrvuumvGjRtXMunQoqTArbfemm233TZ33313rr766lx44YUtOQ0A0ErMzQBQG2bMmJG5c+emsbEx48ePz/jx45u/tmLFitxxxx259tprc/311+cf//hH/vznP5c8X4uaAo8++mj233//JMmuu+6aefPmZfXq1S05FQDQCszNAFAbpk+fnmHDhiVJ+vfvn8WLF2fp0qVJks033zxXX311OnTokBUrVmTp0qXp2bNnyfO1qCmwww475LHHHkuSvPDCC+ncuXPq6+tbcioAoBWYmwGg7RQKhXb7eD8LFixYa++g7t27Z/78+Wsd87Of/SzDhw/PiBEjsv3225c8X4uaAsccc0xeeOGFfOELX8i3v/3tnH/++S05DQDQSszNAFCbisXiez530kkn5Z577sn999+fWbNmlfz+Fm002Llz50ycOLEl3woAtAFzMwC0nboK2miwV69eWbBgQfPtefPmNS8RWLRoUZ566qnsvffe6dSpU4YMGZJHH300H/vYx9Z7vhYlBQAAAID2N3jw4EydOjVJMnv27PTq1StdunRJkqxatSpnn312li1bliR5/PHH069fv5Lna1FSAAAAAGrFhqz1by977rlnBg4cmFGjRqVQKGTcuHGZPHlyunbtmuHDh+eUU07JCSeckIaGhgwYMCAHH3xwyfMViutagNDKlq1a0tZDbJA3V79R7hKSJJvXb1HuErJ89bJyl5Ak6VS/eblLSJLUFypjM652+HHcpBRT/sejqdhU7hKSJA11ldHDXdW0qtwlJEm6dNiy3CVs8t5YvbzcJSSpnJ+xukL5w5OV8lhUyrW4K+kJOP9PJTxX8X9jbZXwb5Ikmzd0LncJbeqWZxrbbazP7XhMu42V/6+9Owyt6677AP49SUzztJ2zzbNsFLSMvZkMfHSwF+266VjEF4qMibaEVV8JQ4abMlgpYoWaYssQVxQnm75xFCO1TgWhVbFQRroJg0oHMpVRtJ3NsmWdXZvVNvd5Mc28up10XXrPyTmfzwj03lvO/d0u5/4uv/v9/08kBQAAAKBU0eCV9819ZQAAAEApSQEAAAAo0eRlK5ICAAAA0FKSAgAAAFCiLpuwXg6SAgAAANBSkgIAAABQoq/Bewr0ZChQl2vAL+tbVnUJSZJ/zJ2ruoQM9Q1VXUKS+lxXdS71uDb0+bl/VF1CkmSwvx7nytzchapLqMW1y5Pk/Nz5qktIksx1qv9/QrPU5Ryb61TfB+rybwFlOqn+s1tdYtx1+Rxbh/8nLG2SAgAAAFCiLsOoy8FIGgAAAFrKUAAAAABayvIBAAAAKFE0eKNBSQEAAABoKUkBAAAAKFE0+Pv05r4yAAAAoJSkAAAAAJSwpwAAAADQOJICAAAAUKIvkgIAAABAw0gKAAAAQAl7CgAAAACNIykAAAAAJQp7CgAAAABNIykAAAAAJewpAAAAADSOpAAAAACUKBr8fXpzXxkAAABQylAAAAAAWsryAQAAACjRZ6NBAAAAoGkkBQAAAKBEEUkBAAAAoGEkBQAAAKBEYU8BAAAAoGkkBQAAAKCEPQUAAACAxpEUAAAAgBL2FAAAAAAaR1IAAAAASvQ1+Pv05r4yAAAAoJSkAAAAAJRo8p4CPRkKvHZhthdPs6Bl/UNVl5Ak6ev0V11COulUXcI/1aOOvqIeoZmBvndVXUKSZK4zV3UJSZK5VF/HQKo/X19Xj3OlLr+jvHPnLrxWdQlJksH+ZVWXkKQel5rqdOpxnjf5gy/vXB3Olbp48bWpqktIkgwvG6m6BJY4SQEAAAAo0eSBWD2+HgUAAAB6zlAAAAAAWsryAQAAACjR5P1WJAUAAACgpSQFAAAAoISNBgEAAIDGkRQAAACAEpICAAAAQONICgAAAECZBl99wFAAAAAAlpAdO3bkyJEjKYoiW7duzQc+8IH5xw4fPpxvfvOb6evry7XXXpvx8fH09b31IgHLBwAAAKBE0cP/FvLUU0/l2LFjmZiYyPj4eMbHx7se/+pXv5rdu3fnRz/6UV599dUcOnSo9HiGAgAAALBETE5OZnR0NEly3XXX5dSpUzl9+vT84/v27cs111yTJFm9enVmZmZKj3dRQ4Fnn302o6Ojeeyxx5Ikzz//fDZv3pyxsbHce++9OXfu3CW9GADg0ujNANA7RVH07Gch09PTWbVq1fzt1atX54UXXpi/vXLlyiTJ1NRUnnjiiXz4wx8uPd6CQ4EzZ85k+/btWbdu3fx9u3fvztjYWPbs2ZO1a9dm7969CxYOACwOvRkA+JdOp/Nf97344ou5++67s23btq4BwptZcCgwODiYRx55JCMjI/P3Pfnkk7n99tuTJLfddlsmJyffbt0AwCXSmwGgt+q0p8DIyEimp6fnb09NTeWqq66av3369Ol8/vOfz3333ZcNGzYseLwFhwIDAwMZGhrquu/s2bMZHBxMkgwPD3dFFQCAy0tvBoD2uvnmm7N///4kyTPPPJORkZH5JQNJ8o1vfCOf+9zncuutt17U8d7xJQnfLKoAAFRHbwaAxXUx3+D3yo033pgbbrghmzZtSlEU2bZtW/bt25crrrgiGzZsyOOPP55jx47NLyX8xCc+kY0bN77l8S5pKLB8+fLMzs5maGgoJ0+e7IovAgC9pzcDQHvcf//9Xbevv/76+T8fPXr0bR3rki5JuH79+vm4woEDB3LLLbdcymEAgEWiNwMAl2LBpMDRo0ezc+fOHD9+PAMDA9m/f38efPDBbNmyJRMTE1mzZk3uuOOOXtQKAERvBoBeu5hLBS5VRacHCw9PnXvpcj/FRVnWP7TwX+qBOqz17KT6GpL61NFf9FddQpJkrjNXdQm1cqFzvuoSMlC8q+oSktTj3yJJ+mpyriwfWLnwX6LUK+dmqi4hSTLYv6zqEpLUozfXRZM/+PLO1eFcqcvv6PTsyapLSJIML6vHcrH/GVhRdQmX1ZGXftez5/q/1Tf17LmSRdhoEAAAAJqsThsNLrZL2lMAAAAAWPokBQAAAKCEpAAAAADQOJICAAAAUKIuG1xeDpICAAAA0FKSAgAAAFDCngIAAABA40gKAAAAQAl7CgAAAACNIykAAAAAJZq8p0BPhgLL+od68TQL6nQ6VZeQJJnrXKi6hKQm8Zcmn1yXopN6/I72F/1Vl/BP1c8tL3TOV11CkmSg711Vl0DDDPYvq7oE/kOTo6mwmOY6c1WXkCT536Grqy4BFkX1n7gBAACgxpr8ZaY9BQAAAKClJAUAAACgRJOXeEkKAAAAQEsZCgAAAEBLWT4AAAAAJWw0CAAAADSOpAAAAACUkBQAAAAAGkdSAAAAAEq4JCEAAADQOJICAAAAUEpSAAAAAGgYSQEAAAAoYU8BAAAAoHEkBQAAAKBEYU8BAAAAoGkkBQAAAKCEpAAAAADQOJICAAAAUMLVBwAAAIDGMRQAAACAlrJ8AAAAAErYaBAAAABoHEkBAAAAKCEpAAAAADSOpAAAAACUcElCAAAAoHEkBQAAAKCEPQUAAACAxpEUAAAAgBL2FAAAAAAaR1IAAAAASjR5T4GeDAWG+pf34mkAgIukNwMAiaQAAAAALKC5SQF7CgAAAEBLSQoAAABAiebmBCQFAAAAYEnZsWNHNm7cmE2bNuX3v/9912OvvfZaHnjggTbpmt4AAAc3SURBVNx5550XdSxDAQAAAFginnrqqRw7diwTExMZHx/P+Ph41+O7du3K+9///os+nqEAAAAAlCiKomc/C5mcnMzo6GiS5LrrrsupU6dy+vTp+ce/9KUvzT9+MQwFAAAAYImYnp7OqlWr5m+vXr06L7zwwvztlStXvq3jLYmhQNl6iTbatWtXNm7cmE996lM5cOBA1eVUbnZ2NqOjo9m3b1/VpVTu5z//eT75yU/mzjvvzMGDB6sup1Kvvvpq7rnnnmzevDmbNm3KoUOHqi6pEs8++2xGR0fz2GOPJUmef/75bN68OWNjY7n33ntz7ty5iitkqdKbu+nN3fTmN+jNb9CbX6c3L1VFD3/enk6nc8mvKlkCQ4GF1ku0zeHDh/PHP/4xExMTefTRR7Njx46qS6rcd7/73Vx55ZVVl1G5mZmZfOc738mePXvy8MMP5ze/+U3VJVXqpz/9aa699tr88Ic/zEMPPdTK944zZ85k+/btWbdu3fx9u3fvztjYWPbs2ZO1a9dm7969FVbIUqU3d9Ob/5ve/Dq9uZverDezOEZGRjI9PT1/e2pqKlddddUlH6/2Q4GF1ku0zU033ZSHHnooSfLud787Z8+ezYULFyquqjp//vOf86c//Skf+chHqi6lcpOTk1m3bl1WrlyZkZGRbN++veqSKrVq1aq8/PLLSZJXXnmlK2LVFoODg3nkkUcyMjIyf9+TTz6Z22+/PUly2223ZXJysqryWML05m56cze9+Q16cze9WW9eyuqUE7j55puzf//+JMkzzzyTkZGRt71k4N/Vfiiw0HqJtunv78/y5cuTJHv37s2tt96a/v7+iquqzs6dO7Nly5aqy6iFv/71r5mdnc3dd9+dsbGx1jeUj3/84zlx4kQ++tGP5q677soDDzxQdUk9NzAwkKGhoa77zp49m8HBwSTJ8PBwq99PuXR6cze9uZve/Aa9uZverDezOG688cbccMMN2bRpU77+9a9n27Zt2bdvX371q18lSb74xS/my1/+cp577rls3rw5v/jFL0qPN9CLohfTO10v0RS//vWvs3fv3vzgBz+oupTKPP744/ngBz+Y9773vVWXUhsvv/xyvv3tb+fEiRP57Gc/m9/+9rcXtYNpE/3sZz/LmjVr8v3vfz9/+MMfsnXrVmtb/4P3UxaL36XX6c1685vRm9+gNy/M+2md1eu8vf/++7tuX3/99fN/3r1799s6Vu2HAou9XqIJDh06lIcffjiPPvporrjiiqrLqczBgwfzl7/8JQcPHszf/va3DA4O5pprrsn69eurLq0Sw8PD+dCHPpSBgYG8733vy4oVK/LSSy9leHi46tIq8fTTT2fDhg1JXn+TnJqayoULF1r97V2SLF++PLOzsxkaGsrJkye74otwsfTm/6Y3v05v7qY3d9Ob35zeTNVqv3xgsddLLHV///vfs2vXrnzve9/Le97znqrLqdS3vvWt/OQnP8mPf/zjfPrTn84XvvCF1n7oSJINGzbk8OHDmZuby8zMTM6cOdPKtXr/snbt2hw5ciRJcvz48axYsaL1HzqSZP369fPvqQcOHMgtt9xScUUsRXpzN735DXpzN725m9785vTmpaEoip799FrtkwL/vl6iKIps27at6pIq9ctf/jIzMzO577775u/buXNn1qxZU2FV1MHVV1+dj33sY/nMZz6TJPnKV76Svr7az/0um40bN2br1q256667cv78+Xzta1+ruqSeO3r0aHbu3Jnjx49nYGAg+/fvz4MPPpgtW7ZkYmIia9asyR133FF1mSxBenM3vZm3ojd305v1Zuqp6Fi4AgAAAG9pavZEz55rZKi3Q+X2jioBAACg5Wq/fAAAAACqVNTs6gOLSVIAAAAAWkpSAAAAAEpICgAAAACNYygAAAAALWUoAAAAAC1lKAAAAAAtZaNBAAAAKFEUNhoEAAAAGsZQAAAAAFrKUAAAAABayp4CAAAAUKKIPQUAAACAhpEUAAAAgFKSAgAAAEDDSAoAAABAiebmBCQFAAAAoLUkBQAAAKBEUTQ3KyApAAAAAC0lKQAAAAClJAUAAACAhjEUAAAAgJayfAAAAABKNHfxgKQAAAAAtJakAAAAAJRqblZAUgAAAABaSlIAAAAAShSFpAAAAADQMIYCAAAA0FKGAgAAANBS9hQAAACAEoWrDwAAAABNIykAAAAApSQFAAAAgIaRFAAAAIASzc0JSAoAAABAa0kKAAAAQImiaG5WQFIAAAAAWspQAAAAAFrK8gEAAAAoZfkAAAAA0DCSAgAAAFCiuTkBSQEAAABoLUkBAAAAKNXcrICkAAAAALSUpAAAAACUKApJAQAAAKBhDAUAAACgpQwFAAAAoKXsKQAAAAAlClcfAAAAAJqm6HQ6naqLAAAAAHpPUgAAAABaylAAAAAAWspQAAAAAFrKUAAAAABaylAAAAAAWspQAAAAAFrq/wGMPixGVJLlnAAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = np.zeros((n_layers, n_heads))\n",
+ " weights_base = np.zeros((n_layers, n_heads))\n",
+ "\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(example[1], example[2], max_len)\n",
+ " crop_len = torch.sum(input_mask != 0)\n",
+ " \n",
+ " _, output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " _, output_base = model_base(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " \n",
+ " output_finetuned = output_finetuned.squeeze(0).detach().numpy()\n",
+ " output_base = output_base.squeeze(0).detach().numpy()\n",
+ " \n",
+ " for l in range(n_layers):\n",
+ " for h in range(n_heads): \n",
+ " weights_finetuned[l, h] = np.mean(output_finetuned[l, h, :crop_len, 0])\n",
+ " weights_base[l, h] = np.mean(output_base[l, h, :crop_len, 0])\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned_cls = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base_cls = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base_cls, avg_weights_finetuned_cls, 'Attention to [CLS]')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5.1 SUBJ-VERB relation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [06:11<00:00, 2.69it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAUAAAJ1CAYAAAChC0mPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs/XmYVOWdN/6/iyasjQQQcFRMCEaMGFQSVwwqghKXPJFEQRNwycRxYhY1cQkaURG3aCKiecbEMTJqIkjafTfquAyKaAaN82RUEheMCoggm6z9/cMf/bMjFNA0XU3V6+VV10VVnTrn0wfpu+pT7/s+hdra2toAAAAAFadFqQsAAAAASkNTAAAAACqUpgAAAABUKE0BAAAAqFCaAgAAAFChNAUAAACgQmkKAACUuWeeeSa9e/fO+PHjS10KAM1Moba2trbURQBAuVi5cmVqampyzz335I033sjs2bNTKBSy1VZbpV+/fjn++OPTu3fvuu2feeaZjBw5MsOGDcsFF1yw1v2OGDEiU6dOzZNPPpmuXbvWe+zjqqqq0rFjx+y00075xje+kUMOOWSN+5s5c2YOPPDAeo/94he/yKGHHrrWGl544YUceeSR2WuvvTJhwoSi52HSpEn52c9+lhNOOCFnnnlmxo8fn6uvvrroa5KkQ4cOmTZtWpL//7lZk5YtW6Zr167ZY4898t3vfjef//zn6z2/MedmY7z88sv5r//6rxx33HH1Hr/pppvypS99KV/4whca/Zj/aNGiRbn++utz7LHHZosttkiSzJ07N1OnTs3222+f7bfffpPXAMDmo2WpCwCAcrFq1ap8//vfzyOPPJK99947xx57bLbccst88MEHeeGFF3L33XfnnnvuyXXXXZc999yz0Y574YUXpkOHDkmSZcuW5a233sptt92WU089NTNmzMgPfvCDT7ymS5cuGTduXJLkvvvuy/3337/O4/Tt2zc77bRTnnnmmbz55pvp0aPHWredPHlykuSoo46q9/ixxx6bfv36rfV1n/rUpz7xWP/+/T+xnwULFuSll15KTU1N7r///txwww1r3G9Dzs3GuP/++3P77bfXawosW7Ysl1xySS644IImaQq8+OKLufrqq3PEEUfUNQU6d+6cIUOGbPJjA7D50RQAgEbyxBNP5JFHHsnBBx+cq666qt5zRx99dA4//PAcf/zxueKKKzJp0qRGO+7+++9flx5YbdiwYRk0aFCuv/76fO9730tVVVW959u2bVv3IfGVV15Z72MdddRROe+881JTU5Mf/ehHa9zmlVdeyfTp07PHHnukZ8+e9Z7bZZddNvjD6bbbbrvG1xx55JEZMmRIjjvuuFx88cW59dZbP7FNQ87NxnjxxRc/8dhf/vKXLF++vNGO0ZAaAGBtrCkAAI3kf//3f5MkAwYMWOPz++yzT6655pqMHj16k9fSuXPn9OzZM4sXL86SJUsabb+HH3542rVrl9tuuy2rVq1a4zZ/+MMfkiTDhw9vtOOuzV577ZXPfOYzeeGFF9b759zQc/PBBx/kl7/8ZQ466KDsvPPO2WOPPXLUUUflnnvuqdtm5syZ6d27dx5//PG89dZb6d27d0aMGJGzzjorRx55ZJLkpz/9aXr37p1nnnmm7nXTpk3Ld7/73ey+++754he/mMGDB+fSSy/N/Pnz69UwcODADB48OAsWLMg555yT/v37Z+edd85Xv/rV3HXXXfW2u/zyy5MkBx54YN1UlbWtKTBjxoycdtpp6d+/f/r06ZN99tknP/zhD+v+X17trLPOSu/evfP222/n17/+dQYNGpSdd945X/nKV/KLX/wiK1euXJ9TD0AzJCkAAI2kW7duSZIHH3wwX/va19KqVatPbDNo0KAmqeXDDz/MzJkzs+2226a6urrR9ltdXZ1DDz00t956a5588slPNECWL1+eO+64I506dcrgwYMb7bjFtGrVKoVCIYVCYb2239Bzc+KJJ2b69Ok55phjsssuu2TRokW57bbbctppp+W9997LyJEj66ZjnH/++UmS0aNHp3Pnzmnbtm3atWuXm2++Od/61reyxx571K1/8PDDD+eHP/xhdthhh/zgBz9IdXV1/vu//zs33nhjnnzyydx6661p06ZNXR2rVq3KiSeemK5du+bUU0/NvHnz8u///u8544wz8rnPfS59+vTJ6NGjc91112Xq1Kl1NazNyy+/nKOPPjotW7bM8OHD07Nnz8ycOTM333xzhg8fnt/97nefmO7wy1/+MjNmzMhxxx2Xli1b5pZbbsm1116bLl265Nhjj12v8w9A86IpAACNZPDgwbnqqqvyn//5nzn88MPz9a9/PXvvvXf69OmzxrnyjWXhwoVp3bp1ko8WOnz99dczfvz4LF68OJdeemmjH2/YsGG59dZbM3ny5E80BR599NHMnTs3J5xwwhqbIkuWLMkHH3yw1n23adNmja9bm7/+9a/561//mh122KHeB+jVNvbcvPvuu6murs7xxx+fM844o+7xww47LPvss09uvPHGjBw5sm46xmWXXZYk9aY7rJ6esfPOO9c9vmzZspx33nnZcccd8/vf/76uxqFDh2aHHXbImDFjcsstt9Rbm2DmzJkZMGBAvaTJlltumTPPPDMPP/xw+vTpk/322y/33Xdfko8SK9tuu+1af7bLLrssCxcuzC233JLddtut7vEBAwbkyCOPzBVXXJHrrruu3mtefvnlTJo0qe7v6Ctf+UoOPPDAPPjgg5oCAJspTQEAaCTt27fP73//+1x44YV56KGHcuWVV+bKK69MmzZtsuuuu2bgwIH1Fn9rLGuab9+5c+dcdtll2X///Rv1WEnyxS9+MX369MkjjzySuXPn1vs2evUCg8OGDVvja88+++ycffbZa933xRdfnKFDh9Z7bPny5Z9oJCxcuDAvvvhiLr/88qxatWqt6xts7Lnp3r17vQ/GS5cuzdKlS+uee+utt9a5jzV59tlnM3v27Hz729+ut8/koykAY8eOzTPPPPOJqxgcf/zx9e5/8YtfTJLMnj17g46/ePHiPPXUU+ndu3e9hkDy0YKSO+ywQ6ZMmZKlS5fWNSyS5Fvf+la9ps0222yTLl26bPDxAWg+NAUAoBF1794948ePzzvvvJNHH300zz//fKZNm5ann346Tz/9dMaPH58rr7wy++67b6Mdc/z48enYsWOSjyLmc+bMyX/913/l9NNPz5133plf/vKXa/wWfW2WL1+eBQsW1HusdevWad++fd39o446KqNHj86dd95Z98H13XffzZNPPpk999wzn/3sZ9e47+9973vZa6+91nrsz33uc594rKamJjU1NWvc/rOf/WyuueaaDBw4cI3PN8a5+fOf/5yrr746f/rTnzJv3ryi266vV199NclHcfxf/vKXa9zm7bffrne/qqoq22yzTb3HVn9gX7FixQYd//XXX8+qVas+cSnH1Xr27JmXX345M2fOTK9eveoe32677T6xbevWrTf4+AA0H5oCALAJbLXVVjn66KNz9NFHJ/loQbeJEydmwoQJ+fGPf5wHH3wwHTt2XO958LW1tUmSFi0+uUbwbrvt9okV9g8//PDssssuGT16dK699tq1fpO+Js8//3xGjhxZ77Ejjjgil1xySd39ww47LJdeemkmT55c1xS47bbbsnLlyqILDG6//fYbfDnGAw44oN435LW1tRk1alTmz5+fG2+8sW4thzXZ2HPz8ssv55hjjkmSfPvb306/fv3qLnF45plnfuKD+/patGhRkuS73/1uvvKVr6xxm39sVlRVVTXalRJWH79du3ZrfH51s2Hx4sVrfByA8qEpAABNoFevXhk1alTef//93HnnnZk+fXoGDBhQ9wFz9Ye0tZk3b14KhcIGTT34xje+kfPOOy9PPPHEejUFVjcovvCFL+Q//uM/6j235ZZb1rtfXV2dww47LJMmTcr06dOzyy67pKamJp07d270xRS7dev2iUbCz372s5x00kk577zz8qtf/WqD97m+5+bmm2/O0qVLM3bs2Hzzm9+s99zGfDu+OnXRsWPHDW6SNIbVx//HD/2rrb4qw8fTIQCUJ5ckBIBGsGzZsowfPz4XX3xx0e1WL/z24YcfJvkopt22bdv86U9/Wusl/t59993MmDEjO+200wYtWLhy5crU1tbWzVe/4YYbMnLkyE/M/547d26S1MXst9hii+y55571bh+PkK+2et2Au+++O//93/+d119/PUccccQGLRTYUAcccEAOPvjg/PGPf6x3acD19Y/nZm1mzpyZJNl7773rPf7aa69t1Dz61bH9559/fo3Pr/472VQ++9nPpqqqKi+//PIan3/11VfTqlWrogsVAlAeNAUAoBG0atUqjz76aG644YZ6143/uPfeey933313WrdunS9/+ctJPoqIf/Ob38xbb72V66+//hOvWb58ec4///ysWrXqE4vOrcukSZOSJF/60peSfLQ43zPPPJPbb7+9bpsVK1bkySefTOvWrdO3b98N2v/OO++cPn365L777st9992XQqGQo446aoP2sTHOPvvsVFdXZ8yYMRv8Ifofz83arE5IfHxBwaVLl+bCCy+sS22sbvAkH03v+MdGw+opHx9//Mtf/nK6dOmSxx9/PDNmzKi3/b333pt99913rf8frcuajveP2rZtm/322y8vv/xypk2bVu+5qVOn5m9/+1sOOOCAJmnwAFBapg8AQCO55JJLcvzxx+f000/P7bffnv333z+dO3fOokWL8uqrr+buu+/OvHnzcsEFF9Rbsf+0007LX/7yl/z85z/Ps88+m7322iudOnXKzJkzc++992bGjBk5/vjj87WvfW2Nx33sscfqpiEkyfz58/Pcc8/lnnvuSY8ePfL9738/SXLMMcekpqYm48aNy9y5c9OrV6/cfffdeeONN/K9732v3j7W17Bhw3Luuefm97//fdEFBlebPn36OufF77bbbunevfs6j929e/eceuqpGTNmTMaMGbPGBfvW99yszSGHHJLbb789P/vZz/Kd73wnK1asyOTJk7PrrrumY8eOufvuuzNu3Lgcdthh6dOnT7bddttMmTIlF198cf7pn/4pxx13XN237TfffHM+/PDD9OvXL7vsskvOO++8nHLKKRk5cmSOO+64dO3aNX/+858zceLE9OzZMwcccMA6z8GarD7ez3/+8+y+++75P//n/6xxuzPOOCPTpk3LySefnBEjRmTbbbfNa6+9lt/97nfp1KlTTj/99AYdH4DNi6YAADSSHXbYIXfddVduuummPP744xk3blwWL16cVq1a5Z/+6Z8yaNCgHHPMMdlxxx3rva5du3aZMGFCJk2alDvvvDPXXHNNlixZkk6dOqVv3745++yz079//7Ue95xzzql3/9Of/nS22Wab/PM//3NOOOGEumkBnTt3zo033pirrroq9957b95777306NEj55xzTkaMGNGgn3n1goOLFi1a62UIP27ChAmZMGFC0W2uueaa9WoKJB81Ou64447ce++9OfTQQz+xnsH6npu12W+//TJ69OhMmDAhY8eOzVZbbZUjjzwyJ5xwQv7f//t/mT59em6++eZ07tw5ffr0yY9+9KO89dZbufnmm9O7d+8cd9xx+fKXv5yhQ4fmvvvuy69+9aucf/752WWXXXLQQQdlwoQJufbaa/PrX/86ixcvTrdu3XLUUUfl5JNPTnV19Xqdg380fPjwPPnkk3nyySfz5z//OQcddNAat+vZs2cmTZqUq666Kr/73e8yf/78dOrUKQMHDszJJ5+cHj16NOj4AGxeCrWrlzMGAAAAKoo1BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUJoCAAAAUKE0BQAAAKBCaQoAAABAhdIUAAAAgAqlKQAAAAAVSlMAAAAAKpSmAAAAAFQoTQEAAACoUC1LXQAAAAA0Z/sXzm2yYz1We0GTHSuRFAAAAICKpSkAAAAAFcr0AQAAACiiUCiUuoR6LrrookyfPj2FQiGjRo1K37596557++23c9ppp2X58uXZaaedcsEFxacjSAoAAADAZmLq1Kl5/fXXM3HixIwdOzZjx46t9/wll1ySE044IZMnT05VVVX+/ve/F92fpgAAAAAUU2jC2zpMmTIlgwYNSpL06tUr8+fPz8KFC5Mkq1atynPPPZeBAwcmSUaPHp2tt9666P40BQAAAGAzMWfOnHTq1KnufufOnTN79uwkydy5c9O+fftcfPHFOfroo3PFFVesc3+aAgAAAFBEoUWhyW4bqra2tt6f33333YwcOTI33XRT/ud//iePPfZY0ddrCgAAAMBmolu3bpkzZ07d/VmzZqVr165Jkk6dOmXrrbfOdtttl6qqquy999555ZVXiu5PUwAAAACKKBSa7rYu/fv3zwMPPJAkeemll9KtW7dUV1cnSVq2bJkePXrktddeq3u+Z8+eRffnkoQAAACwmejXr1/69OmT4cOHp1AoZPTo0ampqUmHDh0yePDgjBo1KmeddVZqa2uzww471C06uDaF2o9PQAAAAADqObD1+U12rD8uHd1kx0pMHwAAAICKZfoAAAAAFLE+c/03V5ICAAAAUKE0BQAAAKBCmT4AAAAARRRalO/8AUkBAAAAqFCSAgAAAFBMGa80KCkAAAAAFUpSAAAAAIoo46CApAAAAABUKkkBAAAAKKJQxlEBSQEAAACoUJICAAAAUEz5BgUkBQAAAKBSSQoAAABAEYUW5RsVkBQAAACACiUpAAAAAEWU8cUHJAUAAACgUkkKAAAAQDFlHBWQFAAAAIAKpSkAAAAAFcr0AQAAACiijGcPSAoAAABApZIUAAAAgCIKLco3KiApAAAAABVKUgAAAACKKeNFBSQFAAAAoEJJCgAAAEARZRwUkBQAAACASiUpAAAAAEUUyjgqICkAAAAAFUpSAAAAAIop36CApAAAAABUKkkBAAAAKKLQonyjApICAAAAUKEkBQAAAKCY8g0KSAoAAABApdIUAAAAgApl+gAAAAAUUSiU7/wBSQEAAACoUJICAAAAUISkAAAAAFB2JAUAAACgmDL+Or2MfzRouN69e2fw4MEZMmRIDj744PzzP/9z3nzzzVKX1SCDBw/OM888U3Sbs846K7/61a+aqCIAKtXHx9fVt+985ztJkmOPPTYvvfRSox3riSeeyN///vdG218xZ599dsaPH98kx9ppp50yc+bMotuMGDEid9xxR5PUA2z+JAVgLW688cZstdVWSZIrrrgiY8eOzb/927+VuCoA2Lx9fHz9uAkTJjTqcW644Yb867/+a7beeutG3S9Qmcp5TQFNAVgPe+21Vx555JG6+7feemuuv/76rFy5Ml27ds1ll12WbbbZJu+++27OOOOMzJ49O8uWLcuhhx6aU089NbW1tbnmmmty1113ZdmyZTnwwAPz05/+NFVVVfWOU1NTk8ceeyyf+tSn8txzz6Vnz545+eSTc/nll+eNN97Ij370owwbNiyrVq3KuHHj8sADDyRJdt1115x77rlp165d/vznP+fMM8/MihUrst9++9Xb/8MPP5xx48Zl8eLF+cxnPpPLL788nTt33vQnEADWYeDAgbnsssuy1VZbZfjw4TnxxBNz6623Zt68efnpT3+aQw45ZL3H0yuvvDJPP/10/vrXv+b000/P448/nu222y7f+973knyUkFt9f+DAgTnxxBMzefLkvPPOOznssMNy1llnJVn7uPn+++/nxz/+cV577bVsv/32adOmzRobHePHj8+cOXPyzjvv5KWXXsree++dQw45JOPHj8+sWbMyZsyYHHDAAVm6dGnGjh2bZ555Ji1atMh+++2X008/PVVVVfnP//zPXHjhhWnZsmW+8Y1v1Nv/xIkT89vf/jbLli3Lrrvumosuuiht2rTZRH9DQLkyfQDWYdmyZbnzzjszcODAJMl7772XCy64IL/97W/z4IMPZrvttquL3t9www3Zfffdc++99+auu+7Km2++mVmzZuWOO+7I/fffn8mTJ+ehhx7Km2++md///vdrPN4TTzyR73//+3nwwQczY8aMXHfddbn55pszduzYuuPcd999efzxx1NTU5N77rknH3zwQW644YYkyXnnnZeRI0fmgQceyG677VYXMXzzzTdzxhln5Iorrsgf//jH7LnnnjnvvPM27ckDgAZ4//3306JFi9x1110ZNWpUrrzyyiRZ7/H0lFNOSffu3fPzn/88hxxyyDqP9+yzz2bixIn5wx/+kJtuuinvvPNO0XHzN7/5TTp16pRHHnkk5557bp588sm17vuxxx7LRRddlLvuuiv3339/3fh90kkn5Te/+U2Sj1IS77zzTu65557cdtttmTZtWu6+++6sXLkyZ599dkaPHp377rsvLVq0yMqVK5Mk06ZNy7hx4zJhwoQ88sgjqa6uzrhx4zb0VAPrqVBoultT0xSAtRgxYkSGDBmS/v3758UXX8zQoUOTJF26dMlzzz1X943Al7/85br1Brp06ZInn3wy06ZNS6tWrfKLX/wi3bp1y6OPPppvfOMb6dChQ1q2bJkjjzwyDz744BqPu/3226dnz55p1apVPvOZz2TfffdNVVVVdthhh8yaNSvJR28wvv71r6ddu3apqqrK0KFD89RTT2Xp0qV58cUX694ADRkyJG3btk2SPP7449ljjz2yww47JEmGDx+eRx55pO7NBQA0hdXj6+rbOeec84ltVqxYUTfu9unTp25tgA0ZTzfE4YcfnqqqqnTv3j1dunTJ22+/XXTcnDZtWr761a8mSbbddtvssccea933brvtli5duqRTp07p2rVrBgwYkCSfGNePOuqotGzZMm3atMnhhx+ep556Kq+99lqWLVuWfffdN0lyxBFH1O33kUceySGHHJLu3bsnSY4++uhGORdA5TF9ANbi43Men3322YwYMSI1NTXp0qVLrrrqqro3BosWLUrPnj2TJMcdd1xWrVqV888/P7Nmzcq3vvWt/OAHP8iCBQvy7//+75k4cWKSZOXKlWuN7bdv377uz1VVVWnXrl3dn1etWpUkmTt3bjp27Fi3XceOHfPee+9l3rx5SZLq6uokH8192mKLLZIkCxYsyLRp0zJkyJC611VXV9e9BgCawtrWFPi4j49/LVq0qBv/1jaevvvuuzn22GOTJH379s1ll122QTWtHjdXH3vlypVFx8358+enQ4cOdY+vHmvXZG3j+sd/rrWN6/Pnz69X28e3WbBgQR566KG6lEJtbW2WL1++QT83sAGsKQCVbffdd8/WW2+d5557LitWrMgjjzySm266KZ07d86kSZNy1113JUlatmyZE088MSeeeGL+9re/5bvf/W6+9KUvpVu3bhk4cGC+/e1vN0o9W265Zb0P8/PmzcuWW25Z92Zh4cKF6dChQ1atWpX58+cnSbp165Z99tknV111VaPUAABNrdh4ev/996/1dR//AJ6kbmxc17HWNm5uscUWWbBgQd39uXPnpkePHuvc59oUG9cXLlxY7zgfr++II47ImWee2eDjAiSmD8B6+dvf/pa//e1v+dznPpf33nsv22yzTd1CQ/fdd18WLVqUJDn33HPz1FNPJUm22267bLnllikUCjnwwANzxx13ZMmSJUmSW265JbfddluD69l///1z5513ZsmSJVmxYkUmT56c/fbbL23atMmOO+6Yhx56KElyzz33ZOnSpUmSfffdN9OmTaub6vDCCy/kwgsvbHANANDUNmQ8bdmyZd0H965du+Yvf/lLko/W2Hn++efXeaxi4+auu+6ahx9+OEnyxhtv5Lnnntuon2v//ffP5MmTs3LlyixevDh33HFH9ttvv2y33Xapqqqqu7RwTU1N3QroAwcOzIMPPljXKHj44Yfz61//eqPqANaunNcUkBSAtRgxYkTdasatWrXK+eefn969e6dLly655557Mnjw4PTo0SOnnHJK/vVf/zWXXHJJhg8fnnPPPTdjxoxJbW1tBg4cmL333jtJ8sorr9TNBdxuu+0yduzYBtc2ZMiQ/O///m+GDh2a2tra7Lnnnhk5cmSSjxYaHDVqVK699toMGDAgvXr1SvLRNwpjxozJySefnOXLl6d9+/YZNWrUxpwiAGhSgwYNWu/x9OCDD85pp52WH/7whznqqKPy/e9/PwcddFB22mmnHHzwwes8VrFx81/+5V9y6qmnZuDAgenVq1cOOuigjfq5RowYkTfffDOHHnpoCoVChgwZkq9+9aspFAoZM2ZMRo0alVatWmXo0KF10w/69OmTk046KSNGjMiqVavSpUuXnH/++RtVB1CZCrW1tbWlLgIAAACaq2/u8MsmO9bkl09tsmMlpg8AAABAxdIUAAAAgAplTQEAAAAopowvSSgpAAAAABWqSZIC8+YuborDrNOHS1eUuoQkSZvWpQ9oLFy0rNQlJEm22KJ1qUtIkjSX5TY/1bJ59OlWrly17o2awJIPS/9vdost2pS6hCTJqO/dUeoSkiQ/vXRIqUtIknTtVl3qEjZ7C+Z/WOoSkiQrVqwsdQlJkhZVpf/9W7uqeQxGLT9V+nORNKfzUVXqEpIky5c1j38ri5csL3UJ6dSpbalLSJJcPPbRUpeQJDnl1H1LXUKS5NOd25W6hE2qjIMCkgIAAABQqUr/lTUAAAA0Y4UyjgpICgAAAECFkhTrW1ePAAAgAElEQVQAAACAYsr46/Qy/tEAAACAYiQFAAAAoAhrCgAAAABlR1IAAAAAipAUAAAAAMqOpAAAAAAUUSjjr9PL+EcDAAAAipEUAAAAgGKsKQAAAACUmwYnBS666KJMnz49hUIho0aNSt++fRuzLgBgAxmbAYAN1aCmwNSpU/P6669n4sSJmTFjRkaNGpWJEyc2dm0AwHoyNgPAplPGswcaNn1gypQpGTRoUJKkV69emT9/fhYuXNiohQEA68/YDAA0RIOaAnPmzEmnTp3q7nfu3DmzZ89utKIAgA1jbAaATafQotBkt6bWKAsN1tbWNsZuAIBGYmwGANZHg9YU6NatW+bMmVN3f9asWenatWujFQUAbBhjMwBsQmW8qECDkgL9+/fPAw88kCR56aWX0q1bt1RXVzdqYQDA+jM2AwAN0aCkQL9+/dKnT58MHz48hUIho0ePbuy6AIANYGwGgE2njIMCDWsKJMlPfvKTxqwDANhIxmYAYEM1uCkAAAAAlaAUVwVoKpoCAAAAsBm56KKLMn369BQKhYwaNSp9+/ate27gwIHZaqutUlVVlSS5/PLL071797XuS1MAAAAAimlGiwpMnTo1r7/+eiZOnJgZM2Zk1KhRmThxYr1tfvOb36R9+/brtb8GXX0AAAAAaHpTpkzJoEGDkiS9evXK/Pnzs3DhwgbvT1MAAAAAiigUmu62LnPmzEmnTp3q7nfu3DmzZ8+ut83o0aNz9NFH5/LLL09tbW3R/WkKAAAAwGbqHz/0//CHP8xPf/rT3HjjjXnllVfywAMPFH29pgAAAAAUUWhRaLLbunTr1i1z5sypuz9r1qx07dq17v7Xv/71dOnSJS1btsyAAQPy8ssvF92fpgAAAABsJvr371/37f9LL72Ubt26pbq6OkmyYMGCfOc738myZcuSJM8++2w+//nPF92fqw8AAABAMc3n4gPp169f+vTpk+HDh6dQKGT06NGpqalJhw4dMnjw4AwYMCDDhg1L69ats9NOO2XIkCFF96cpAAAAAJuRn/zkJ/Xu77jjjnV/PvbYY3Pssceu975MHwAAAIAK1SRJgaqq5pG1+NW4/yp1CUmSM885oNQl5NOfalPqEpIky5etLHUJSZI2bT9V6hKSJC1bVpW6hI8sW1HqCpIkHTq0LnUJabE+14VpAqN/eWipS0iStG0m/1bYeOu6PFFTGXvm/aUuIUlywbjDS11Cs/k7aS5jc8uWzeO7q0ULl5W6hCRJmzbNI+DbHMbmZvJPJT86pX+pS0iSVDeDv5NKUGgm7wk3hebx2xYAAABocs2j5QgAAADN1PpcKnBzJSkAAAAAFUpSAAAAAIoo4yUFJAUAAACgUkkKAAAAQDFlHBWQFAAAAIAKJSkAAAAARbj6AAAAAFB2JAUAAACgiDJeUkBSAAAAACqVpAAAAAAUU8ZRAUkBAAAAqFCSAgAAAFBEQVIAAAAAKDeaAgAAAFChTB8AAACAIgpl/HV6Gf9oAAAAQDGSAgAAAFCMhQYBAACAciMpAAAAAEWUcVBAUgAAAAAqlaQAAAAAFFFoUb5RAUkBAAAAqFCSAgAAAFBMGS8qICkAAAAAFUpSAAAAAIoo46CApAAAAABUKkkBAAAAKMLVBwAAAICyIykAAAAAxZTxogKSAgAAAFChNAUAAACgQjXJ9IHDP31xUxxmnW75+09KXUKSpFWrqlKXkFWraktdQpKkqkUz6Us1j9ORZlRIs1BoBjGtg9peUOoSkiT3LTyn1CUkST6Y/2GpS0iSbNmtutQlbPa+1ql5jM2//esppS4hSdKyZenHo5UrVpW6hCRJi2aymFZzqePTndqWuoQkyYdLlpe6hGZjSLsxpS4hSXLPB2eXuoQkyTvvLCh1CUmSbXt8utQlbFLN4G3pJlP6ERAAAAAoCQsNAgAAQBEuSQgAAACUHUkBAAAAKKI5rHW1qUgKAAAAQIWSFAAAAIBiyjcoICkAAAAAlUpSAAAAAIpw9QEAAACg7EgKAAAAQBGuPgAAAACUHUkBAAAAKMaaAgAAAEC5aXBS4LLLLstzzz2XFStW5F/+5V9y0EEHNWZdAMAGMjYDwKZRxksKNKwp8PTTT+eVV17JxIkT8/777+eII47wxgMASsjYDAA0RIOaArvvvnv69u2bJNliiy2yZMmSrFy5MlVVVY1aHACwfozNAEBDNKgpUFVVlXbt2iVJJk+enAEDBnjTAQAlZGwGgE2nnC9JuFFXH3j44YczefLkXH/99Y1VDwCwEYzNAMCGaHBT4Iknnsi//du/5brrrkuHDh0asyYAoAGMzQCwiZTxJQkb1BRYsGBBLrvsstxwww359Kc/3dg1AQAbyNgMADREg5oC9957b95///2ccsopdY9deuml2XrrrRutMABg/RmbAWDTKeMlBRrWFBg2bFiGDRvW2LUAAA1kbAYAGmKjFhoEAACAclco4zUFWpS6AAAAAKA0JAUAAACgmDJeVEBSAAAAACqUpAAAAAAUUZAUAAAAAMqNpAAAAAAUUSjjr9PL+EcDAAAAipEUAAAAgCKsKQAAAACUnSZJCvxx+XlNcZh1+vDD5aUu4f+n9F2md99dUOoSkiRdu7YvdQlJkoULlpa6hCRJx0+3LXUJSZIWVc2jX7hs6YpSl5Cvjj2g1CUkaU6/vygXzWVsXrSoefz+bQ5e/J93S11CkmTnL3QrdQlJkjlzF5e6hCRJty2bx3uVqpbNY2xesXxlqUvITl//fKlLSJLMnrOo1CUkSTp3ah7vH9l8mT4AAAAAxZg+AAAAAJQbSQEAAAAowiUJAQAAgLIjKQAAAABFuCQhAAAAUHYkBQAAAKCYFpICAAAAQJmRFAAAAIAirCkAAAAAlB1JAQAAACiijIMCkgIAAABQqTQFAAAAoJgWhaa7rYeLLroow4YNy/Dhw/PCCy+scZsrrrgiI0aMWPePtkEnAgAAACiZqVOn5vXXX8/EiRMzduzYjB079hPbvPrqq3n22WfXa3+aAgAAAFBEoVBostu6TJkyJYMGDUqS9OrVK/Pnz8/ChQvrbXPJJZfk1FNPXa+fTVMAAAAANhNz5sxJp06d6u537tw5s2fPrrtfU1OTPfbYI9tss8167U9TAAAAADZTtbW1dX+eN29eampqcvzxx6/3612SEAAAAIpoTpck7NatW+bMmVN3f9asWenatWuS5Omnn87cuXPzrW99K8uWLcsbb7yRiy66KKNGjVrr/iQFAAAAYDPRv3//PPDAA0mSl156Kd26dUt1dXWSZMiQIbn33nszadKkXH311enTp0/RhkAiKQAAAADFreelAptCv3790qdPnwwfPjyFQiGjR49OTU1NOnTokMGDB2/w/jQFAAAAYDPyk5/8pN79HXfc8RPbbLvttrnxxhvXuS9NAQAAAChifS4VuLmypgAAAABUKEkBAAAAKKLQjNYUaGySAgAAAFChJAUAAACgmPINCkgKAAAAQKWSFAAAAIAiXH0AAAAAKDuSAgAAAFCEqw8AAAAAZUdSAAAAAIqwpgAAAABQdpokKbB06YqmOMw6rVi+stQlJElaty59QGObbTqWuoQkyYcfLi91CUmSZc3k/43Fi5aVuoQkSatWVaUuIUnSps2nSl1CfnRK/1KXkCSprS11BR95b+7iUpdAI1mypHn8vqlq4fuJ1b602zalLiFJsuCDD0tdQpJk4cLm8f/olp3blbqEJMnKlatKXUKSpKqq9P9mf/G74aUuIUmyckXz+Dt5+tmZpS4hSXLgwF6lLoEGKv2nUwAAAGjOynf2gOkDAAAAUKkkBQAAAKAICw0CAAAAZUdSAAAAAIoo46CApAAAAABUKkkBAAAAKEJSAAAAACg7kgIAAABQhKsPAAAAAGVHUgAAAACKKOOggKQAAAAAVCpJAQAAACjCmgJr8eGHH2bQoEGpqalprHoAgI1gbAYANsRGJQX+7//9v+nYsWNj1QIAbCRjMwA0vjIOCjQ8KTBjxoy8+uqr2X///RuxHACgoYzNAMCGanBT4NJLL81ZZ53VmLUAABvB2AwAbKgGTR+4/fbbs+uuu6ZHjx6NXQ8A0ADGZgDYdMp5ocEGNQUee+yxvPnmm3nsscfyzjvvpFWrVtlqq62yzz77NHZ9AMB6MDYDAA3RoKbAlVdeWffn8ePHZ5tttvGmAwBKyNgMAJtOGQcFNu6ShAAAAMDma6MuSZgkP/jBDxqjDgCgkRibAaBxFVK+UQFJAQAAAKhQG50UAAAAgHJmTQEAAACg7EgKAAAAQBGSAgAAAEDZkRQAAACAIgplHBWQFAAAAIAKJSkAAAAARZRxUEBSAAAAACqVpAAAAAAUU8ZRAUkBAAAAqFBNkhRY+uHypjjMOrVr36rUJSRJVqxYVeoSmk2jq6pF8yikW7cOpS4hSbJi+cpSl5AkmTd/SalLSJJs0aFNqUvI9w/7j1KXkCQZc9ORpS4hSdK27adKXQKN5O9vLyh1CUmS7Xp8utQlJElWriz92FxbW1vqEpIkrVo3jyDpDp/fstQlJEmWN5Oxedq0t0tdQpJk935bl7qEHLfHr0pdQpJk/CMnlLqEJMnOX+ha6hLYzDWP3/oAAADQTDWXL1U3BdMHAAAAoEJJCgAAAEARhTKOCkgKAAAAQIWSFAAAAIAiyjgoICkAAAAAlUpSAAAAAIqwpgAAAABQdiQFAAAAoIgyDgpICgAAAEClkhQAAACAIso4KCApAAAAAJVKUgAAAACKcPUBAAAAoOxICgAAAEARZRwUkBQAAACASqUpAAAAABXK9AEAAAAowkKDAAAAQNmRFAAAAIAiyjgoICkAAAAAlUpSAAAAAIqwpgAAAABQdiQFAAAAoIgyDgpICgAAAEClkhQAAACAIiQFAAAAgLIjKQAAAABFuPoAAAAAUHYkBQAAAKCIMg4KSAoAAABApZIUAAAAgCKsKQAAAACUnUJtbW3tpj7IiuUrN/Uh1suqlZv8R10vy5vB+ahqqR/0ce+8s6DUJSRJ/v5286jjy/22KXUJzcbMt+aXuoQkybbbdCx1CUmS5StK//srSdpXty51CZu9ZUtXlLqEJMmqlatKXUKSpNCi9N8ArWwm71Oy6d8arpe/vvZ+qUtIkiz5sHn8W/lin+6lLiFJ0gQfHdZp5Yrm8XujVevmEbpevqyZjM0dynts/u11zzTZsY7/5z2b7FiJ6QMAAACwWbnooosyffr0FAqFjBo1Kn379q17btKkSZk8eXJatGiRHXfcMaNHjy46/cHXxQAAALCZmDp1al5//fVMnDgxY8eOzdixY+ueW7JkSe65557cfPPNueWWW/LXv/41f/rTn4ruT1IAAAAAimhOCw1OmTIlgwYNSpL06tUr8+fPz8KFC1NdXZ22bdtmwoQJST5qECxcuDBdu3Ytuj9JAQAAANhMzJkzJ506daq737lz58yePbveNr/+9a8zePDgDBkyJD169Ci6P00BAAAAKKJQKDTZbUOtaQHQE088MQ8//HCeeOKJPPfcc0VfrykAAAAAm4lu3bplzpw5dfdnzZpVN0Vg3rx5efbZZ5Mkbdq0yYABA/L8888X3Z+mAAAAABRRKDTdbV369++fBx54IEny0ksvpVu3bqmurk6SrFixImeddVYWLVqUJHnxxRfTs2fPovuz0CAAAABsJvr165c+ffpk+PDhKRQKGT16dGpqatKhQ4cMHjw4J598ckaOHJmWLVumd+/eOfDAA4vuT1MAAAAAimhOVx9Ikp/85Cf17u+44451fx46dGiGDh263vsyfQAAAAAqlKQAAAAAFNHMggKNSlIAAAAAKpSkAAAAABTR3NYUaEwNTgrceeed+drXvpahQ4fmsccea8SSAICGMDYDABuqQUmB999/P9dcc03+8Ic/ZPHixRk/fnz233//Ri4NAFhfxmYA2HTKOSnQoKbAlClTsvfee6e6ujrV1dUZM2ZMY9cFAGwAYzMA0BANmj4wc+bMfPjhhznppJNyzDHHZMqUKY1dFwCwAYzNALDpFApNd2tqDV5ocN68ebn66qvz97//PSNHjsyjjz5a1pEKAGjujM0AwIZqUFKgS5cu2W233dKyZctst912ad++febOndvYtQEA68nYDAA0RIOaAvvuu2+efvrprFq1Ku+//34WL16cTp06NXZtAMB6MjYDwKZTKBSa7NbUGjR9oHv37jn44INz1FFHJUnOOeectGjR4KsbAgAbydgMADREg9cUGD58eIYPH96YtQAAG8HYDACbRqFF+a7R4ysEAAAAqFANTgoAAABAJSjni/lICgAAAECFkhQAAACAIkpxVYCmIikAAAAAFUpSAAAAAIoo46CApAAAAABUKkkBAAAAKMKaAgAAAEDZkRQAAACAIiQFAAAAgLLTJEmBD5csb4rDrNPf31lQ6hKSJDNmzC11CTlgv56lLiFJMv+DpaUuIUnScYvWpS4hSdLqU1WlLiFJsmLFylKXkCR5860PSl1CturWvtQlJEnemdU8fn9t/U9blLoEGkltbW2pS0iSzPjb+6UuIUky9/0lpS4he35521KXkCRZsXJVqUtIknTv2jx+/y5c3Dzexy5buqLUJSRJljaDOjp0aB7v296fu7jUJSRJOnVuV+oSKkIZBwUkBQAAAKBSaQoAAABAhbLQIAAAABRTxvMHJAUAAACgQkkKAAAAQBEuSQgAAACUHUkBAAAAKKKMgwKSAgAAAFCpJAUAAACgiEKL8o0KSAoAAABAhZIUAAAAgCKsKQAAAACUHUkBAAAAKKJQxlEBSQEAAACoUJICAAAAUISkAAAAAFB2JAUAAACgiDIOCkgKAAAAQKXSFAAAAIAKZfoAAAAAFGGhQQAAAKDsSAoAAABAEZICAAAAQNmRFAAAAIAiyjgoICkAAAAAlUpSAAAAAIqwpgAAAABQdiQFAAAAoAhJAQAAAKDsSAoAAABAEWUcFJAUAAAAgEolKQAAAABFFFqUb1RAUgAAAAAqVJMkBVq1ah6BhJ6f7VTqEpIkW2/VodQlNJtJMZ07tyt1CUmShQs+LHUJSZIuWzaP87F82cpSl5Ak6bFtx1KXkLf+/kGpS0jSTH5vJFmyeFmpS0iSdOjYttQlbPZWrawtdQlJks9v36XUJSRJ5s9vHuNAc9CyZfP4zqhVq6pSl5Ak2ebTzeP3zYrlzWNsLhQ+VeoSMue9xaUuIUnStm3pz0WSLFmyvNQlJElatW4en/k2lWby8WmTaB6/9QEAAIAmpykAAAAAFaq8Mx4AAACwkQop3/kDkgIAAABQoSQFAAAAoJjyDQpICgAAAEClkhQAAACAIgplfE1CSQEAAACoUJICAAAAUEQZBwUkBQAAAKBSSQoAAABAEdYUAAAAAMqOpAAAAAAUUcZBgYY1BRYtWpQzzzwz8+fPz/Lly3PyySfnK1/5SmPXBgCsJ2MzANAQDWoK3HbbbenZs2d+/OMf5913382xxx6b+++/v7FrAwDWk7EZADYdawr8g06dOmXevHlJkg8++CCdOnVq1KIAgA1jbAYAGqJBSYFDDz00NTU1GTx4cD744INce+21jV0XALABjM0AsOmUcVCgYUmBO+64I1tvvXUeeuihTJgwIRdccEFj1wUAbABjMwDQEA1qCjz//PPZd999kyQ77rhjZs2alZUrVzZqYQDA+jM2AwAN0aCmwGc+85lMnz49SfLWW2+lffv2qaqqatTCAID1Z2wGgE2nUCg02a2pNWhNgWHDhmXUqFH59re/nRUrVuS8885r5LIAgA1hbAYAGqJBTYH27dtn3LhxjV0LANBAxmYA2HTKeaHBBjUFAAAAgNK46KKLMn369BQKhYwaNSp9+/ate+7pp5/OL37xi7Ro0SI9e/bM2LFj06LF2lcOaNCaAgAAAFApCoWmu63L1KlT8/rrr2fixIkZO3Zsxo4dW+/5c889N1dddVVuueWWLFq0KE888UTR/WkKAAAAwGZiypQpGTRoUJKkV69emT9/fhYuXFj3fE1NTbbaaqskSefOnfP+++8X3Z+mAAAAABRRaML/1mXOnDnp1KlT3f3OnTtn9uzZdferq6uTJLNmzcpTTz2V/fbbr+j+NAUAAABgM1VbW/uJx957772cdNJJGT16dL0GwppYaBAAgP+vvbsLkeuu+wD+nezsZrtJbNM8GSVY80hvKrnRgBd5abF0ixe+UCp2l6XRK0FEbAWhoQgRQoIJvbBFsdLqjSWwssbqhbBBMdCLbXshtCQgVZGg6Uuyj2naNLtJdrPPRXg2z9j2JE1355w95/MpC53Z7cyv83J+w2++//8BoECVzj7Q6XQyPT29ePnUqVPZuHHj4uVz587lG9/4Rh5++OHs3LnzmrcnKQAAAAArxI4dOzI5OZkkOX78eDqdzuKSgST54Q9/mK9//eu56667ruv2JAUAAACgQKtCUYGtW7dmy5YtGR0dTavVyp49e3L48OGsW7cuO3fuzLPPPpsTJ05kYmIiSfLFL34xIyMj73t7hgIAAACwgnzve9/runzHHXcs/vuxY8c+0G0ZCgAAAECBCgUFllxPhgILefduiKVYqMYzuaqv/K0cLsxeKruEJMlNNw2UXUKSZHCwv+wSklQnlvReO5iWoQqPxn9vvqXsEpIkly7Ol11CkmTwpmq8V6iPqhxv1q0tvx/NzVXjfV7rT743oCqv0fn5y2WXkCS5XIGHo7Nx7bX/qAdmZi6WXUKSZPVAX9klsMJJCgAAAECBqnx5txzK/8oaAAAAKIWhAAAAADSU5QMAAABQoMarByQFAAAAoKkkBQAAAKCAjQYBAACA2pEUAAAAgCL1DQpICgAAAEBTSQoAAABAAXsKAAAAALUjKQAAAAAFahwUkBQAAACAppIUAAAAgAL2FAAAAABqR1IAAAAACtQ3JyApAAAAAI0lKQAAAAAF7CkAAAAA1I6hAAAAADSU5QMAAABQoMarByQFAAAAoKkkBQAAAKCAjQYBAACA2pEUAAAAgAI1DgpICgAAAEBTSQoAAABAAUkBAAAAoHYkBQAAAKCAsw8AAAAAtSMpAAAAAAVqHBSQFAAAAICmkhQAAACAAnXeU6BRQ4FVq6rxRFahjqE1A2WXkCRZWCi7giva/VUJzZT/2kiS1YP9ZZeQJFmoxAukGs9Ju91XdglJktmZS2WXkCTpH2hU+1oWl+bmyy4hSTLYrsbxZr4Cx5t2fzXe5yn/oUiStIeq8VmlOqrRj/qr8JmpGg9FVq2qwGORZPp/zpddQpLk496zK1Y1XskAAABAzxkKAAAAQEMZCgAAAEBDWZQJAAAABeq80aCkAAAAADSUpAAAAAAUqHFQQFIAAAAAmspQAAAAABrKUAAAAAAayp4CAAAAUMCeAgAAAEDtSAoAAABAgVbqGxWQFAAAAICGuq6hwCuvvJLh4eE888wzSZLXXnstu3btytjYWB566KFcvHhxWYsEALrpzQDQQ60e/vTYNYcC58+fz969e7Nt27bF65544omMjY3l0KFD2bx5cyYmJpa1SADgKr0ZAFgq1xwKDAwM5Kmnnkqn01m87oUXXsg999yTJLn77rszNTW1fBUCAF30ZgDorVardz+9ds2NBtvtdtrt7j+bmZnJwMBAkmTDhg05ffr08lQHALyL3gwALJUPffaBhYWFpagDAFgiejMALC1nH/gPQ0NDmZ2dTZK88cYbXfFFAKD39GYA4Ebc0FBg+/btmZycTJIcOXIkd95555IWBQB8MHozAHAjrrl84NixYzlw4EBOnjyZdrudycnJPPbYY9m9e3fGx8ezadOm3Hfffb2oFQCI3gwAPVff1QNpLfRg4eGF2UvLfRfXpa/vhoIRS25u7nLZJaSvrxqv6qosey1jl8/3Vo1CqrIeuQp1rFpVjePGwuXyH4skuXChGsfzdTffVHYJK95bZ2fKLiFJMjjYX3YJSarRm9vtahxvUo3DTVVaYmVcmJ0ru4QkSV8FXqdVea9U5Tk5+9Zs2SUkST5+2y1ll7CsTpw407P72rx5fc/uK1mCjQYBAACgzuo8p6zGmA0AAADoOUkBAAAAKNCqznrjJScpAAAAAA0lKQAAAABF6hsUkBQAAACAppIUAAAAgAI1DgpICgAAAEBTSQoAAABAAWcfAAAAAGrHUAAAAAAaqifLB1atqkbUYm7uctklXLGwUHYFWbhcjefkcgUeiySpShpIHd3m5sp/fVyeny+7hCRJX7saM9zVg/1ll8ASGVxdjedyoSJ9YGbmUtklZOimajwnVenN/e2+sku4oiI9cfVgNVb9nn/nYtkl5OKFsiu4YmjNQNklJEk2rl5TdgmscNU4ugAAAEBFVeVLs+VQja+eAAAAgJ6TFAAAAIACzj4AAAAA1I6hAAAAADSUoQAAAAA0lKEAAAAANJSNBgEAAKBAjfcZlBQAAACAppIUAAAAgAKt1DcqICkAAAAADWUoAAAAAEVaPfy5Dvv378/IyEhGR0fz8ssvd/3uwoULeeSRR3L//fdf120ZCgAAAMAK8eKLL+bEiRMZHx/Pvn37sm/fvq7fHzx4MJ/61Keu+/YMBQAAAKBAq9W7n2uZmprK8PBwkuT222/P2bNnc+7cucXff/e73138/fUwFAAAAIAVYnp6OuvXr1+8fOutt+b06dOLl9euXfuBbs/ZBwAAAKBAlc89sLCw8KH+e0kBAAAAWCE6nU6mp6cXL586dSobN2684dszFAAAAIAiFdpUYMeOHZmcnEySHD9+PJ1O5wMvGfj/LB8AAACAFWLr1q3ZsmVLRkdH02q1smfPnhw+fDjr1q3Lvffem+985zt5/fXX849//CO7du3KAw88kC996Uvve3uthQ+7AOE6XLo4t9x3cV3m55f9f/X6LP9Dfk2rVlUjJHK5Ao9Fcn27fPZCqyKFVKSMXLp0uewSKrN+rK9djfdsVQysNtP+sC7OVqM3L6QafeDcuYtll5Chm5Mc9RUAAAfSSURBVPrLLiFJdXpzf7uv7BKuqEojqIjz75T/XqmKoTUDZZeQ5MOvJ18qqwercQxbLtOnzl37j5bIf3Vu/Fv/G+FTJgAAADSUoQAAAAA0lPwlAAAAFKjK8trlICkAAAAADSUpAAAAAEVqHBWQFAAAAICGkhQAAACAAvXNCUgKAAAAQGNJCgAAAECBGm8pICkAAAAATSUpAAAAAIXqGxWQFAAAAICGkhQAAACAAnXeU6AnQ4H+gWrMHvrLLgD4QPoHyq4A6mtgsBq9uSpWD/qUANdjYLVjB9SN5QMAAADQUIYCAAAA0FDyPwAAAFCgznsKSAoAAABAQxkKAAAAQENZPgAAAACF6rt+QFIAAAAAGmpFDAX279+fkZGRjI6O5uWXXy67nNIdPHgwIyMj+cpXvpIjR46UXU7pZmdnMzw8nMOHD5ddSul+97vf5ctf/nLuv//+HD16tOxySvXOO+/k29/+dnbt2pXR0dE899xzZZdUildeeSXDw8N55plnkiSvvfZadu3albGxsTz00EO5ePFiyRWyUunN3fTmbnrzVXrzVXrzFXrzytRq9e6n1yo/FHjxxRdz4sSJjI+PZ9++fdm3b1/ZJZXq+eefz1//+teMj4/n6aefzv79+8suqXQ//elPc/PNN5ddRunOnDmTn/zkJzl06FCefPLJ/PGPfyy7pFL95je/ySc/+cn88pe/zOOPP97IY8f58+ezd+/ebNu2bfG6J554ImNjYzl06FA2b96ciYmJEitkpdKbu+nN76Y3X6E3d9Ob9WaqqfJDgampqQwPDydJbr/99pw9ezbnzp0ruaryfPazn83jjz+eJPnIRz6SmZmZzM/Pl1xVef7+97/nb3/7Wz73uc+VXUrppqamsm3btqxduzadTid79+4tu6RSrV+/Pm+++WaS5K233sr69etLrqj3BgYG8tRTT6XT6Sxe98ILL+See+5Jktx9992ZmpoqqzxWML25m97cTW++Sm/upjfrzVRT5YcC09PTXQeMW2+9NadPny6xonL19fVlaGgoSTIxMZG77rorfX19JVdVngMHDmT37t1ll1EJ//rXvzI7O5tvfvObGRsba3xD+cIXvpBXX3019957bx588ME88sgjZZfUc+12O4ODg13XzczMZGBgIEmyYcOGRh9PuXF6cze9uZvefJXe3E1v1pupphV39oGFhYWyS6iEP/zhD5mYmMgvfvGLskspzbPPPptPf/rTue2228oupTLefPPN/PjHP86rr76ar33ta/nTn/6UVhkLkyrgt7/9bTZt2pSf//zn+ctf/pJHH33U2tb/4HjKUvFaukJv1pvfi958ld58bY6nFVbjt23lhwKdTifT09OLl0+dOpWNGzeWWFH5nnvuuTz55JN5+umns27durLLKc3Ro0fzz3/+M0ePHs3rr7+egYGBfOxjH8v27dvLLq0UGzZsyGc+85m02+184hOfyJo1a/Lvf/87GzZsKLu0Uvz5z3/Ozp07kyR33HFHTp06lfn5+UZ/e5ckQ0NDmZ2dzeDgYN54442u+CJcL7353fTmK/TmbnpzN735venNlK3yywd27NiRycnJJMnx48fT6XSydu3akqsqz9tvv52DBw/mZz/7WW655ZayyynVj370o/z617/Or371q3z1q1/Nt771rcZ+6EiSnTt35vnnn8/ly5dz5syZnD9/vpFr9f7P5s2b89JLLyVJTp48mTVr1jT+Q0eSbN++ffGYeuTIkdx5550lV8RKpDd305uv0pu76c3d9Ob3pjevDK0e/tNrlU8KbN26NVu2bMno6GharVb27NlTdkml+v3vf58zZ87k4YcfXrzuwIED2bRpU4lVUQUf/ehH8/nPfz4PPPBAkuT73/9+Vq2q/Nxv2YyMjOTRRx/Ngw8+mLm5ufzgBz8ou6SeO3bsWA4cOJCTJ0+m3W5ncnIyjz32WHbv3p3x8fFs2rQp9913X9llsgLpzd30Zt6P3txNb9abqabWgoUrAAAA8L7ePjvbs/tad/Pgtf9oCTV3VAkAAAANV/nlAwAAAFCmOp80RFIAAAAAGspQAAAAABrKUAAAAAAaylAAAAAAGspGgwAAAFCkxjsNSgoAAABAQ0kKAAAAQIH65gQkBQAAAKCxJAUAAACgSI2jApICAAAA0FCSAgAAAFCgxkEBSQEAAABoKkkBAAAAKNKqb1ZAUgAAAAAaylAAAAAAGspQAAAAABrKngIAAABQoL47CkgKAAAAQGNJCgAAAECRGkcFJAUAAACgoQwFAAAAoKEsHwAAAIACrRqvH5AUAAAAgIaSFAAAAIAi9Q0KSAoAAABAU0kKAAAAQIEaBwUkBQAAAKCpJAUAAACgSI2jApICAAAA0FCSAgAAAFCovlEBSQEAAABoKEkBAAAAKFDfnICkAAAAADSWpAAAAAAUqXFUQFIAAAAAGkpSAAAAAArUOCggKQAAAABNZSgAAAAADWX5AAAAABRp1XcBgaQAAAAANJShAAAAADSUoQAAAAA0lD0FAAAAoECNtxSQFAAAAICmMhQAAACAhjIUAAAAgIaypwAAAAAUaNV4UwFJAQAAAGgoQwEAAABoKEMBAAAAaKjWwsLCQtlFAAAAAL0nKQAAAAANZSgAAAAADWUoAAAAAA1lKAAAAAANZSgAAAAADWUoAAAAAA31v0uKPWD7ixiOAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_cross_attention(example[1], example[2], max_len, model_finetuned, relation=\"SUBJ-VERB\")\n",
+ " weights_base = analyze_cross_attention(example[1], example[2], max_len, model_base, relation=\"SUBJ-VERB\")\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'SUBJ-VERB attention', cmap=\"Purples\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5.2 VERB-SUBJ relation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [06:09<00:00, 2.71it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAABAUAAAJ1CAYAAAChC0mPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xm01XW9N/D35iAgHEQOMjj2EF4x6ZKaQ4qCISiZ+hgOoAZqXc2bWlo5hCZOmJnmgFmW13QZJUQ4D6jhnBPaY+q6XZKrJqYyCMg8nfP84eIsCdjA8XD2Ye/Xi7XXYu/927/fZ5/pu/dnv7/fX6Gurq4uAAAAQMVpUeoCAAAAgNLQFAAAAIAKpSkAAAAAFUpTAAAAACqUpgAAAABUKE0BAAAAqFCaAgAAZe75559Pz549M3r06FKXAkAzU6irq6srdREAsKk75ZRT8sQTT2TMmDHZY4891rpdbW1t+vfvn1mzZuWJJ55ITU1NevbsuV7H+OEPf5gTTzwxSTJs2LC88MILa9yuXbt22XHHHXP44Yfn2GOPTVVVVf19zz//fIYPH77aY9q0aZNtt902++67b04++eR07dp1vWpKkhUrVmTChAm5//77849//CMzZsxIoVBIt27dsvvuu+ekk05a5TmurGHIkCG55JJL1rrflc/x6aefTufOndf6vKuqqtKhQ4fssssuOfLII3PIIYescX/Tpk3LgQceuMptP/vZz/LVr351vZ/rukyZMiV//vOf679PK/32t7/NF7/4xXzuc59rtGOtzYIFC3LLLbfkhBNOyBZbbJEk+fDDD/PCCy9kxx13zI477rjRawBg09Gy1AUAQDkYMmRInnjiifzxj38s2hR4+umn89577+XQQw9NTU1N/e1dunTJ+eefX/QYa3pDedlll6V9+/b112tra/P+++/nnnvuyaWXXppnnnkmv/jFL1Z7XJ8+fXLMMcfUX58zZ05efvnljBkzJg899FDuvPPO+jfixdTW1ub000/PpEmTss8+++SEE07IVlttlY8++ih//etfc9999+X+++/PzTffnL333nud+1tfn3zeS5cuzbvvvps777wzZ511VqZOnZozzjhjtcd06tQp1113XZLkwQcfzEMPPdRo9az00EMP5a677lqlKbB06dJcccUVueSSS5qkKfDqq6/mhhtuyNe+9rX6pkBNTU0GDRq00Y8NwKZHUwAAGsEBBxyQrl275qGHHsr555+f6urqNW43fvz4JB83ET6pXbt2DXrTdsABB6zxzfuJJ56Yb3zjG5k0aVKeeOKJ9OvXb5X7t9tuu9WON3To0HTr1i033XRT7rzzzpxyyinrPP5TTz2VSZMm5eCDD87111+/yn3HHntsDjvssJx00km5+uqrM27cuA1+fmuzpuc9ZMiQDBgwILfccku+/e1vr5KQSJLNN9+8/jn//e9/b7RaPunVV19d7ba//e1vWbZs2UY53vrWAABrY00BAGgEVVVVOeqoo7Jw4cI88MADa9xm9uzZmTRpUj772c9mr7322qj1tGjRIkceeWSSrHWawZrsvvvuSZJZs2at1/b/8z//kyTp27fvGu/fd9998/Of/zwjR45c7xoaqqamJt27d8/ChQuzaNGiRtvvRx99lGuuuSYHHXRQPv/5z2evvfbKMccck/vvv79+m2nTpqVnz5558skn8+6776Znz54ZNmxYzjvvvBx99NFJPp7+0bNnzzz//PP1j5s8eXJOPvnk7Lnnnvn3f//3DBw4MD/5yU8yd+7cVWro379/Bg4cmHnz5uWCCy5Inz598vnPfz5f+cpXcu+9966y3VVXXZUkOfDAA+unbaxtTYGpU6fme9/7Xvr06ZNevXpl3333zXe+85367+tK5513Xnr27Jn33nsvv/rVrzJgwIB8/vOfz/7775+f/exnWbFiRSN8pQEoBUkBAGgkRx99dH7xi19k/Pjxq0TzV7r77ruzbNmyNd63MbRq1SrJxw2C9bXyE/T1jbl36dIlSfLwww/n8MMPrz/mJw0YMGC9j/9pLF68ONOmTct222231qRGQ5xyyil55ZVXctxxx+ULX/hCFixYkDvvvDPf+973MmvWrAwfPrx+asLFF1+cJBk5cmRqamqy+eabp23bthkzZkyOP/747LXXXvm3f/u3JMmjjz6a73znO9lpp51yxhlnpLq6Ov/v//2/3H777Xn66afzhz/8IW3atKmvo7a2Nqeccko6d+6cs846K3PmzMl//dd/5ZxzzslnP/vZ9OrVKyNHjszNN9+cF154ob6GtZkyZUqOPfbYtGzZMkOHDk337t0zbdq0jBkzJkOHDs3vfve71X4OrrnmmkydOjUnnnhiWrZsmTvuuCM33XRTOnXqlBNOOKHRvuYANB1NAQBoJFtvvXX69u2bxx9/PH//+9/r3/yt9Mc//jGtWrXKEUccsdpj6+rq8tFHH6113y1atNjgN7p/+tOfkiS9e/de7b5ly5atcrz58+fnsccey4033pi99957vRffGzhwYK6//vo88cQTOeyww3LEEUdkn332Sa9evbLZZpttUL0bYv78+WndunWSjxc6fPvttzN69OgsXLgwP/nJTxrtOB988EGqq6tz0kkn5Zxzzqm//dBDD82+++6b22+/PcOHD6+fmnDllVcmySpTM1Y2Wj7/+c/X37506dJcdNFF2XnnnfP73/++/rkMHjw4O+20Uy699NLccccdq6xNMG3atPTt23eV1MVWW22Vc889N48++mh69eqVfv365cEHH0zycXpju+22W+tzu/LKKzN//vzccccd2W233epv79u3b44++uhcffXVufnmm1d5zJQpUzJu3Lj65s/++++fAw88MA8//LCmAMAmSlMAABrRkCFD8vjjj2f8+PH54Q9/WH/7X//610yZMiWHHnpoOnbsuNrj3nrrrey5555r3e+2226bSZMmrXb7J98cJx9/mvzuu+9mwoQJufvuu9O7d+/VVtxPkgkTJmTChAmr3X7wwQfnoosuWu839O3atcvvf//7XHbZZXnkkUdy7bXX5tprr02bNm2y6667pn///qsseNdY1rT+Qk1NTa688soccMABjXacrl27rvLGeMmSJVmyZEn9fe+++26D9vviiy9mxowZ+frXv77KPpOPpwCMGjUqzz///GpnMTjppJNWuf7v//7vSZIZM2Zs0PEXLlyYZ555Jj179lylIZB83ETaaaed8uyzz2bJkiWr/Hwdf/zxq6RBtt1223Tq1GmDjw9A86EpAACNqF+/funWrVvuvvvufP/7369/A7VygcGhQ4eu8XFbb7110U+4P/nG7JPWtjhh69atc8QRR+RHP/rRGqcPfPnLX17lDeaiRYvy5ptvZuzYsfWfeK98cz1v3rzVFsrbcsst6/fbtWvXjB49Ou+//34ee+yxvPzyy5k8eXKee+65PPfccxk9enSuvfba7Lfffmt9fhtq9OjR6dChQ5KPGyEzZ87Mn//855x99tm55557cs0116wSvf80Xnvttdxwww35y1/+kjlz5jTKPt94440kH8fxr7nmmjVu8957761yvaqqKttuu+0qt638uVi+fPkGHf/tt99ObW3tammWlbp3754pU6Zk2rRp6dGjR/3tO+yww2rbtm7deoOPD0DzoSkAAI1o5YKDN9xwQyZNmpRBgwZl8eLFeeCBB/LZz352rWmANm3aNOiUfZ98c5wkzz33XG688cYcf/zxOffcc9f6uC5duqx2vAMOOCBHH310jjzyyJx99tl55JFHsuWWW+bb3/72aosV/ulPf1otmt6tW7cce+yxOfbYY5N8vIjd2LFjc9ttt+X73/9+Hn744XTo0CGFQmG9nltdXV2SNa+JsNtuu6129oHDDjssX/jCFzJy5MjcdNNN+e53v7texylmypQpOe6445IkX//617P77rvXnwrx3HPPXe2N+/pasGBBkuTkk0/O/vvvv8Zt/rWpUVVVtdoZFRpq5fHbtm27xvtXNhsWLly4xtsBKB+aAgDQyI466qj6BQcHDRqUiRMnZt68eTn99NMb/Vj/+uZ4zz33zNNPP53bbrsthxxySH28fH1VV1fnK1/5Sn7xi1/klVdeSb9+/TJixIjV1jtY02kQ/1WPHj0yYsSIzJ49O/fcc09eeeWV9O3bt/5N9co3pmszZ86cFAqFDZp6cOSRR+aiiy7KU089tV5NgXU1KMaMGZMlS5Zk1KhROeqoo1a579N8Ot6uXbskSYcOHRrUDPq0Vh7/X9/0r7Ty7A0rtwOgfDklIQA0spULDv75z3/Ohx9+mHvuuac+zr+xtWjRIpdcckkKhUJGjBixWux/fax8s7t48eIkH5+JYO+9917l0rp16yxdujSjR4/Oj3/846L7W5koWLm/7t27Z/PNN89f/vKX1NbWrvExH3zwQaZOnZpddtllgxYsXLFiRerq6urn6N96660ZPnz4anPeP/zwwyRZJWWxJtOmTUuS7LPPPqvc/tZbb32qefQrY/svv/zyGu9fWd/G8n/+z/9JVVVVpkyZssb733jjjbRq1aroQoUAlAdNAQDYCI455pisWLEi48aNy3PPPZeDDjooW265ZZMc+3Of+1yGDx+eKVOm5Je//OUGPXbevHl58MEHU1VVlV133bXotq1atcpjjz2WW2+9Nffee+8at5k1a1buu+++tG7dOnvssUeSj2PxRx11VN59993ccsstqz1m2bJlufjii1NbW7vaQnvrMm7cuCTJF7/4xSQfL8T4/PPP56677qrfZvny5Xn66afTunXrNZ6Z4ZO22mqrJFllQcElS5bksssuq08wrGx2JB83ZT65aODK21Y+bqU99tgjnTp1ypNPPpmpU6eusv0DDzyQ/fbbb61f03VZ0/H+1eabb55+/fplypQpmTx58ir3vfDCC3nzzTfz5S9/eY2nmASgvJg+AAAbwcoFB2+88cYsX758rQsMrrRgwYI89NBDRbfZYostsu+++67X8c8444xMnDgxN910Uw466KD07NlzlfunTZu2yvGWLVuWd999N3feeWemTZuWs88+O127dl3nca644oqcdNJJOfvss3PXXXflgAMOSE1NTRYsWJA33ngj9913X+bMmZNLLrkkNTU19Y/73ve+l7/97W/56U9/mhdffDFf+tKX0rFjx0ybNi0PPPBApk6dmpNOOimHH374Go/7+OOP109DSJK5c+fmpZdeyv3335/tt9++fqrGcccdlwkTJuS6667Lhx9+mB49euS+++7LP/7xj3z7299eZR9rcsghh+Suu+7Kj370o3zzm9/M8uXLM378+Oy6667p0KFD7rvvvlx33XU59NBD06tXr2y33XZ59tln8+Mf/zhbb711TjzxxPpP28eMGZPFixdn9913zxe+8IVcdNFFOfPMMzN8+PCceOKJ6dy5c1577bWMHTs23bt3z5e//OV1fv3XZOXxfvrTn2bPPffM//2//3eN251zzjmZPHlyTjvttAwbNizbbbdd3nrrrfzud79Lx44dc/bZZzfo+ABsWjQFAGAjqKqqytFHH53Ro0enR48e9Z+Sr8306dPXOQd+5513zt13371ex2/btm1+9KMf5dRTT82IESMybty4VRape+aZZ/LMM8/UX99ss83StWvX7LTTTvnRj3603mcK2GmnnXLvvffmt7/9bZ588slcd911WbhwYVq1apWtt946AwYMyHHHHZedd955tfpuu+22jBs3Lvfcc09+/vOfZ9GiRenYsWN69+6d888/P3369FnrcS+44IJVrm+55ZbZdttt8x//8R/5xje+UT8toKamJrfffnuuv/76PPDAA5k1a1a23377XHDBBRk2bNg6n1+/fv0ycuTI3HbbbRk1alS6deuWo48+Ot/4xjfy3//933nllVcyZsyY1NTUpFevXvnud7+bd999N2PGjEnPnj1z4oknZo899sjgwYPz4IMP5sYbb8zFF1+cL3zhCznooINy22235aabbsqvfvWrLFy4MF26dMkxxxyT0047LdXV1ev1PfhXQ4cOzdNPP52nn346r732Wg466KA1bte9e/eMGzcu119/fX73u99l7ty56dixY/r375/TTjst22+/fYOOD8CmpVC3cmlfAAAAoKJYUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACqUpAAAAABVKUwAAAAAqlKYAAAAAVChNAQAAAKhQmgIAAABQoTQFAAAAoEJpCgAAAECF0hQAAACACtWy1AUAAABAc3ZA4cImO9bjdZc02bESSQEAAACoWJoCAAAAUKFMHwAAAIAiCoVCqUvYaCQFAAAAoEJJCgAAAEAx5RsUkBQAAACASiUpAAAAAEUUWpRvVEBSAAAAACqUpAAAAAAUUcYnH5AUAAAAgEolKQAAAADFlHFUQFIAAAAAKpSkAAAAABRRxkEBSQEAAACoVJoCAAAAUKFMHwAAAIAiCi3Kd/6ApAAAAABUKEkBAAAAKKaMVxqUFAAAAIAKJSkAAAAARZRxUEBSAAAAACqVpAAAAAAUUSjjqICkAAAAAFQoSQEAAAAopnyDApICAAAAUKkkBQAAAKCIQovyjQpICgAAAECFkhQAAACAIsr45AOSAgAAAFCpJAUAAACgmDKOCkgKAAAAQIXSFAAAAIAKZfoAAAAAFFHGswckBQAAAKBSSQoAAABAEYUW5RsVkBQAAACACiUpAAAAAMWU8aICkgIAAABQoSQFAAAAoIgyDgpICgAAAEClkhQAAACAIgrNLCpw+eWX55VXXkmhUMiIESPSu3fv+vv69++fbt26paqqKkly1VVXpWvXrmvdl6YAAAAAbCJeeOGFvP322xk7dmymTp2aESNGZOzYsats8+tf/zrt2rVbr/2ZPgAAAADFFJrwsg7PPvtsBgwYkCTp0aNH5s6dm/nz5zf4qWkKAAAAwCZi5syZ6dixY/31mpqazJgxY5VtRo4cmWOPPTZXXXVV6urqiu7P9AEAAAAootCiea0p8En/+qb/O9/5Tvbff/906NAhp512WiZOnJhBgwat9fGSAgAAALCJ6NKlS2bOnFl/ffr06encuXP99SOOOCKdOnVKy5Yt07dv30yZMqXo/jQFAAAAoJhmtKZAnz59MnHixCTJ66+/ni5duqS6ujpJMm/evHzzm9/M0qVLkyQvvvhi/u3f/q3o/kwfAAAAgE3E7rvvnl69emXo0KEpFAoZOXJkJkyYkPbt22fgwIHp27dvhgwZktatW2eXXXYpOnUgSQp161p1AAAAACrY4dtd2WTHumfaOU12rERSAAAAAIoqFJrvQoOfljUFAAAAoEJJCgAAAEARkgIAAABA2ZEUAAAAgGLK+OP0Mn5q0HA9e/bMwIEDM2jQoBx88MH5j//4j7zzzjulLqtBBg4cmOeff77oNuedd15uvPHGJqoIgEr1yfF15eWb3/xmkuSEE07I66+/3mjHeuqpp/LPf/6z0fZXzPnnn5/Ro0c3ybF22WWXTJs2reg2w4YNy913390k9QCbPkkBWIvbb7893bp1S5JcffXVGTVqVH75y1+WuCoA2LR9cnz9pNtuu61Rj3PrrbfmP//zP7PNNts06n6BylTOawpoCsB6+NKXvpRJkybVX//DH/6QW265JStWrEjnzp1z5ZVXZtttt80HH3yQc845JzNmzMjSpUvz1a9+NWeddVbq6ury85//PPfee2+WLl2aAw88MD/84Q9TVVW1ynEmTJiQxx9/PJtttlleeumldO/ePaeddlquuuqq/OMf/8h3v/vdDBkyJLW1tbnuuusyceLEJMmuu+6aCy+8MG3bts1rr72Wc889N8uXL0+/fv1W2f+jjz6a6667LgsXLsxnPvOZXHXVVampqdn4X0AAWIf+/fvnyiuvTLdu3TJ06NCccsop+cMf/pA5c+bkhz/8YQ455JD1Hk+vvfbaPPfcc/nf//3fnH322XnyySezww475Nvf/naSjxNyK6/3798/p5xySsaPH5/3338/hx56aM4777wkax83Z8+ene9///t56623suOOO6ZNmzZrbHSMHj06M2fOzPvvv5/XX389++yzTw455JCMHj0606dPz6WXXpovf/nLWbJkSUaNGpXnn38+LVq0SL9+/XL22WenqqoqTzzxRC677LK0bNkyRx555Cr7Hzt2bH7zm99k6dKl2XXXXXP55ZenTZs2G+k7BJQr0wdgHZYuXZp77rkn/fv3T5LMmjUrl1xySX7zm9/k4Ycfzg477FAfvb/11luz55575oEHHsi9996bd955J9OnT8/dd9+dhx56KOPHj88jjzySd955J7///e/XeLynnnoqp59+eh5++OFMnTo1N998c8aMGZNRo0bVH+fBBx/Mk08+mQkTJuT+++/PRx99lFtvvTVJctFFF2X48OGZOHFidtttt/qI4TvvvJNzzjknV199df70pz9l7733zkUXXbRxv3gA0ACzZ89OixYtcu+992bEiBG59tprk2S9x9MzzzwzXbt2zU9/+tMccsgh6zzeiy++mLFjx+aPf/xjfvvb3+b9998vOm7++te/TseOHTNp0qRceOGFefrpp9e678cffzyXX3557r333jz00EP14/epp56aX//610k+Tkm8//77uf/++3PnnXdm8uTJue+++7JixYqcf/75GTlyZB588MG0aNEiK1asSJJMnjw51113XW677bZMmjQp1dXVue666zb0Sw2sp0Kh6S5NTVMA1mLYsGEZNGhQ+vTpk1dffTWDBw9OknTq1CkvvfRS/ScCe+yxR/16A506dcrTTz+dyZMnp1WrVvnZz36WLl265LHHHsuRRx6Z9u3bp2XLljn66KPz8MMPr/G4O+64Y7p3755WrVrlM5/5TPbbb79UVVVlp512yvTp05N8/ALjiCOOSNu2bVNVVZXBgwfnmWeeyZIlS/Lqq6/WvwAaNGhQNt988yTJk08+mb322is77bRTkmTo0KGZNGlS/YsLAGgKK8fXlZcLLrhgtW2WL19eP+726tWrfm2ADRlPN8Rhhx2WqqqqdO3aNZ06dcp7771XdNycPHlyvvKVryRJtttuu+y1115r3fduu+2WTp06pWPHjuncuXP69u2bJKuN68ccc0xatmyZNm3a5LDDDsszzzyTt956K0uXLs1+++2XJPna175Wv99JkyblkEMOSdeuXZMkxx57bKN8LYDKY/oArMUn5zy++OKLGTZsWCZMmJBOnTrl+uuvr39hsGDBgnTv3j1JcuKJJ6a2tjYXX3xxpk+fnuOPPz5nnHFG5s2bl//6r//K2LFjkyQrVqxYa2y/Xbt29f+vqqpK27Zt6/9fW1ubJPnwww/ToUOH+u06dOiQWbNmZc6cOUmS6urqJB/Pfdpiiy2SJPPmzcvkyZMzaNCg+sdVV1fXPwYAmsLa1hT4pE+Ofy1atKgf/9Y2nn7wwQc54YQTkiS9e/fOlVdeuUE1rRw3Vx57xYoVRcfNuXPnpn379vW3rxxr12Rt4/onn9faxvW5c+euUtsnt5k3b14eeeSR+pRCXV1dli1btkHPG9gA1hSAyrbnnntmm222yUsvvZTly5dn0qRJ+e1vf5uampqMGzcu9957b5KkZcuWOeWUU3LKKafkzTffzMknn5wvfvGL6dKlS/r375+vf/3rjVLPVltttcqb+Tlz5mSrrbaqf7Ewf/78tG/fPrW1tZk7d26SpEuXLtl3331z/fXXN0oNANDUio2nDz300Fof98k34Enqx8Z1HWtt4+YWW2yRefPm1V//8MMPs/32269zn2tTbFyfP3/+Ksf5ZH1f+9rXcu655zb4uACJ6QOwXt588828+eab+exnP5tZs2Zl2223rV9o6MEHH8yCBQuSJBdeeGGeeeaZJMkOO+yQrbbaKoVCIQceeGDuvvvuLFq0KElyxx135M4772xwPQcccEDuueeeLFq0KMuXL8/48ePTr1+/tGnTJjvvvHMeeeSRJMn999+fJUuWJEn222+/TJ48uX6qw1//+tdcdtllDa4BAJrahoynLVu2rH/j3rlz5/ztb39L8vEaOy+//PI6j1Vs3Nx1113z6KOPJkn+8Y9/5KWXXvpUz+uAAw7I+PHjs2LFiixcuDB33313+vXrlx122CFVVVX1pxaeMGFC/Qro/fv3z8MPP1zfKHj00Ufzq1/96lPVAaxdOa8pICkAazFs2LD61YxbtWqViy++OD179kynTp1y//33Z+DAgdl+++1z5pln5j//8z9zxRVXZOjQobnwwgtz6aWXpq6uLv37988+++yTJPn73/9ePxdwhx12yKhRoxpc26BBg/I///M/GTx4cOrq6rL33ntn+PDhST5eaHDEiBG56aab0rdv3/To0SPJx58oXHrppTnttNOybNmytGvXLiNGjPg0XyIAaFIDBgxY7/H04IMPzve+97185zvfyTHHHJPTTz89Bx10UHbZZZccfPDB6zxWsXHzW9/6Vs4666z0798/PXr0yEEHHfSpntewYcPyzjvv5Ktf/WoKhUIGDRqUr3zlKykUCrn00kszYsSItGrVKoMHD66fftCrV6+ceuqpGTZsWGpra9OpU6dcfPHFn6oOoDIV6urq6kpdBAAAADRXR+10TZMda/yUs5rsWInpAwAAAFCxNAUAAACgQllTAAAAAIop41MSSgoAAABAhWqSpMCM6fPXvVETWLGMHNW5AAAgAElEQVS8dt0bNYHf3DK51CVki/atSl1CkuSEb+xR6hKSJMuWrih1CUmSzTarKnUJSZIlzeTrsWTJ8lKXkC233LzUJSRJ3nt/3ro3agJLlzWPn43P7dyl1CVs8uZ8uLDUJSRJamubx3rHL7/yXqlLSHNZ+rnPl7YvdQlJkuXN5HVbixbN49PB5vLzMXvuolKXkC5btSt1CUmS9z9oHu9x2rVrHq/rO3epLnUJG1UZBwUkBQAAAKBSWVMAAAAAiiiUcVRAUgAAAAAqlKQAAAAAFFPGH6eX8VMDAAAAipEUAAAAgCKsKQAAAACUHUkBAAAAKEJSAAAAACg7kgIAAABQRKGMP04v46cGAAAAFCMpAAAAAMVYUwAAAAAoNw1OClx++eV55ZVXUigUMmLEiPTu3bsx6wIANpCxGQDYUA1qCrzwwgt5++23M3bs2EydOjUjRozI2LFjG7s2AGA9GZsBYOMp49kDDZs+8Oyzz2bAgAFJkh49emTu3LmZP39+oxYGAKw/YzMA0BANagrMnDkzHTt2rL9eU1OTGTNmNFpRAMCGMTYDwMZTaFFosktTa5SFBuvq6hpjNwBAIzE2AwDro0FrCnTp0iUzZ86svz59+vR07ty50YoCADaMsRkANqIyXlSgQUmBPn36ZOLEiUmS119/PV26dEl1dXWjFgYArD9jMwDQEA1KCuy+++7p1atXhg4dmkKhkJEjRzZ2XQDABjA2A8DGU8ZBgYY1BZLkBz/4QWPWAQB8SsZmAGBDNbgpAAAAAJWgFGcFaCqNcvYBAAAAYNMjKQAAAADFlPGiApICAAAAUKEkBQAAAKCIMg4KSAoAAABApZIUAAAAgCKcfQAAAAAoO5ICAAAAUEz5BgUkBQAAAKBSaQoAAABAhWqS6QNt227WFIdZpwULlpa6hCTJGd/dt9QlpGVV8+gHnTnk96UuIUly9e+HlrqEJEmrVs1jRs+KFbWlLiFJsvmWm5e6hFRVNY+s2BbVrUpdQpJky45tS10CjaSqmYwDixYvKXUJSZJ99t6+1CU0m+/J2Sf8odQlJEku/eURpS4hSVJXV1fqEpI0n0XOajqWfmyurW0e35Pm8LVIkjabN4/3WuWuUMbnJGweow8AAADQ5JrHx5IAAADQTDWXtM7GICkAAAAAFUpSAAAAAIoo4yUFJAUAAACgUkkKAAAAQDFlHBWQFAAAAIAKJSkAAAAARTj7AAAAAFB2JAUAAACgiDJeUkBSAAAAACqVpAAAAAAUU8ZRAUkBAAAAqFCSAgAAAFBEQVIAAAAAKDeaAgAAAFChTB8AAACAIgpl/HF6GT81AAAAoBhJAQAAACjGQoMAAABAuZEUAAAAgCLKOCggKQAAAACVSlIAAAAAiii0KN+ogKQAAAAAVChJAQAAACimjBcVkBQAAACACiUpAAAAAEWUcVBAUgAAAAAqlaQAAAAAFOHsAwAAAEDZkRQAAACAYsp4UQFJAQAAAKhQmgIAAABQoZpk+sCHHy5qisOsU11dXalLSJLU1LQtdQlZvry21CUkSa6549hSl5Ck+fxsNBctN6sqdQlJkqqq0se0pk9fUOoSkiRbbVX6vxtJsmD+klKXkCTZshn8Hd3UzZm7uNQlJEmWLF1e6hKSJJ06tSt1CVm+bEWpS0iS/PjmwaUuIUlSaCZR3dZtmsds20ULl5a6hCRJy6rSf6Y4Y+bCUpeQJOnQoXWpS0iSzJvXPMbmTq2bx+/KxtJM/iRtFKX/rQYAAABKorzbOQAAAPApOSUhAAAAUHY0BQAAAKCIQqHQZJf1cfnll2fIkCEZOnRo/vrXv65xm6uvvjrDhg1b5740BQAAAGAT8cILL+Ttt9/O2LFjM2rUqIwaNWq1bd544428+OKL67U/TQEAAAAoptCEl3V49tlnM2DAgCRJjx49Mnfu3MyfP3+Vba644oqcddZZ6/XUNAUAAABgEzFz5sx07Nix/npNTU1mzJhRf33ChAnZa6+9su22267X/px9AAAAAIpozmcfqKurq///nDlzMmHChPzmN7/JBx98sF6PlxQAAACATUSXLl0yc+bM+uvTp09P586dkyTPPfdcPvzwwxx//PE5/fTT8/rrr+fyyy8vuj9NAQAAACiiOZ19oE+fPpk4cWKS5PXXX0+XLl1SXV2dJBk0aFAeeOCBjBs3LjfccEN69eqVESNGFN2f6QMAAACwidh9993Tq1evDB06NIVCISNHjsyECRPSvn37DBw4cIP3pykAAAAAxTSzNQV+8IMfrHJ95513Xm2b7bbbLrfffvs692X6AAAAAFSoBicFrrzyyrz00ktZvnx5vvWtb+Wggw5qzLoAgA1kbAaAjWM9pvpvshrUFHjuuefy97//PWPHjs3s2bPzta99zQsPACghYzMA0BANagrsueee6d27d5Jkiy22yKJFi7JixYpUVVU1anEAwPoxNgMADdGgpkBVVVXatm2bJBk/fnz69u3rRQcAlJCxGQA2nvU5VeCm6lOdfeDRRx/N+PHjc8sttzRWPQDAp2BsBgA2RIObAk899VR++ctf5uabb0779u0bsyYAoAGMzQCwkTSzUxI2pgY1BebNm5crr7wyt956a7bccsvGrgkA2EDGZgCgIRrUFHjggQcye/bsnHnmmfW3/eQnP8k222zTaIUBAOvP2AwAG08ZLynQsKbAkCFDMmTIkMauBQBoIGMzANAQn2qhQQAAACh3hTJeU6BFqQsAAAAASkNSAAAAAIop40UFJAUAAACgQkkKAAAAQBEFSQEAAACg3EgKAAAAQBGFMv44vYyfGgAAAFCMpAAAAAAUYU0BAAAAoOw0SVJgm222aIrDrNPCBUtLXUKSpK6u1BUkixctK3UJSZI2m29W6hKSJCd88RelLiFJ8pvnTy11CUmSqqrm0Qldvqy21CVk2bIVpS4hSfLee/NKXUKSpLq6ValLoJE0l7H5o7mLSl1CkqSuGQzOS5c2j783bdo0jyDpoHaXlbqEJMl9H40odQlJkpabVZW6hCRJbW3pf1dWrCj964Mkee316aUuIUnStWt1qUtIknTaql2pS6CBmsdffQAAAGiuTB8AAAAAyo2kAAAAABThlIQAAABA2ZEUAAAAgCKckhAAAAAoO5ICAAAAUEwLSQEAAACgzEgKAAAAQBHWFAAAAADKjqQAAAAAFFHGQQFJAQAAAKhUkgIAAABQjLMPAAAAAOVGUgAAAACKcPYBAAAAoOxoCgAAAECFMn0AAAAAiijj2QOSAgAAAFCpJAUAAACgGKckBAAAAMqNpAAAAAAU4ZSEAAAAQNmRFAAAAIAiCtYUAAAAAMqNpAAAAAAUU75BAUkBAAAAqFSSAgAAAFCEsw8AAAAAZUdSAAAAAIpw9gEAAACg7EgKAAAAQBHWFAAAAADKTpMkBZYtW9EUh1mn2rq6UpfQbLTfok2pS0jSfH42rn3kxFKXkCRZvqK21CUkSeYvWF7qEpIkHTtuXuoSss02W5S6hCTJG1NnlbqEJEmnTm1LXQKNZPny5vH3t6qlzydWajZj89LmMQbc8sZ3S11Ckubz6eCMGQtKXUKSpGvX6lKXkG7d2pe6hCTN53Vbq82qSl0CmzjTBwAAAKCY5tEf3Ci05wEAAKBCSQoAAABAEc1lKtHGICkAAAAAFUpSAAAAAIoo46CApAAAAABUKkkBAAAAKEJSAAAAACg7kgIAAABQhLMPAAAAAGVHUgAAAACKKOOggKQAAAAAVCpJAQAAACjCmgJrsXjx4gwYMCATJkxorHoAgE/B2AwAbIhPlRT4xS9+kQ4dOjRWLQDAp2RsBoDGV8ZBgYYnBaZOnZo33ngjBxxwQCOWAwA0lLEZANhQDW4K/OQnP8l5553XmLUAAJ+CsRkA2FANmj5w1113Zdddd83222/f2PUAAA1gbAaAjaecFxpsUFPg8ccfzzvvvJPHH38877//flq1apVu3bpl3333bez6AID1YGwGABqiQU2Ba6+9tv7/o0ePzrbbbutFBwCUkLEZADaeMg4KfLpTEgIAAACbrk91SsIkOeOMMxqjDgCgkRibAaBxFVK+UQFJAQAAAKhQnzopAAAAAOXMmgIAAABA2ZEUAAAAgCIkBQAAAICyIykAAAAARRTKOCogKQAAAAAVSlIAAAAAiijjoICkAAAAAFQqSQEAAAAopoyjApICAAAAUKGaJCnwwQfzm+Iw67TVVm1LXUKSZMWK2lKXkEWLlpe6hCRJy5ZVpS4hSdK5c3WpS0iSvPvPuaUuIUnStZl8PVYsL/3vyjvvNo/vyf33/HepS0iSnPm9/UpdAo3k7X/MKXUJSZKaLTcvdQlJklatSh+eXLRwaalLSJJUtWwenxltv/2WpS4hSTJ79sJSl5AkaV/dqtQlJEnqautKXUJe++/ppS4hSTLp4b+XuoQkyXe+26fUJbCJK/0ICAAAAM1YGc8eMH0AAAAAKpWkAAAAABRRaGZRgcsvvzyvvPJKCoVCRowYkd69e9ffN27cuIwfPz4tWrTIzjvvnJEjRxatX1IAAAAANhEvvPBC3n777YwdOzajRo3KqFGj6u9btGhR7r///owZMyZ33HFH/vd//zd/+ctfiu5PUgAAAACKaE5BgWeffTYDBgxIkvTo0SNz587N/PnzU11dnc033zy33XZbko8bBPPnz0/nzp2L7k9SAAAAADYRM2fOTMeOHeuv19TUZMaMGats86tf/SoDBw7MoEGDsv322xfdn6YAAAAAFFEoFJrssqHq6lY/Vegpp5ySRx99NE899VReeumloo/XFAAAAIBNRJcuXTJz5sz669OnT6+fIjBnzpy8+OKLSZI2bdqkb9++efnll4vuT1MAAAAAiigUmu6yLn369MnEiROTJK+//nq6dOmS6urqJMny5ctz3nnnZcGCBUmSV199Nd27dy+6PwsNAgAAwCZi9913T69evTJ06NAUCoWMHDkyEyZMSPv27TNw4MCcdtppGT58eFq2bJmePXvmwAMPLLo/TQEAAAAoohmdfCBJ8oMf/GCV6zvvvHP9/wcPHpzBgwev975MHwAAAIAKJSkAAAAARTTkrACbCkkBAAAAqFCSAgAAAFBEGQcFJAUAAACgUmkKAAAAQIUyfQAAAACKsNAgAAAAUHYkBQAAAKCIMg4KSAoAAABApZIUAAAAgCKsKQAAAACUHUkBAAAAKKKMgwKSAgAAAFCpJAUAAACgCEkBAAAAoOxICgAAAEARzj4AAAAAlB1JAQAAACiijIMCkgIAAABQqSQFAAAAoAhrCgAAAABlp0mSAltv3b4pDrNOCxcsLXUJSZJCq1JXkCxcsKzUJSRJZsyaW+oSkiQ7bN+h1CUkSbaqaVvqEj7WTBqhVVWl71tu2aFNqUtIkhw+uFepS0iSHPf50aUuIUnyxze+V+oSNnmf2WHLUpeQJFm8qHmMR6mrK3UF+WjeklKXkCRZvHh5qUtIkrSvbgYvmJK03XyzUpeQJCm0aCaDczP4tLR7M/n7ddSQ3qUuIUny3aN+V+oSkiQ3PXBiqUvYuEr/o7/RlP4VNwAAAFASmgIAAABQoSw0CAAAAEVYaBAAAAAoO5ICAAAAUISkAAAAAFB2JAUAAACgiDIOCkgKAAAAQKWSFAAAAIAirCkAAAAAlB1JAQAAACiijIMCkgIAAABQqSQFAAAAoAhrCqzBPffck8MPPzyDBw/O448/3oglAQANYWwGADZUg5ICs2fPzs9//vP88Y9/zMKFCzN69OgccMABjVwaALC+jM0AsPGUc1KgQU2BZ599Nvvss0+qq6tTXV2dSy+9tLHrAgA2gLEZAGiIBk0fmDZtWhYvXpxTTz01xx13XJ599tnGrgsA2ADGZgDYeAqFprs0tQYvNDhnzpzccMMN+ec//5nhw4fnscceK+tIBQA0d8ZmAGBDNSgp0KlTp+y2225p2bJldthhh7Rr1y4ffvhhY9cGAKwnYzMA0BANagrst99+ee6551JbW5vZs2dn4cKF6dixY2PXBgCsJ2MzAGw8hUKhyS5NrUHTB7p27ZqDDz44xxxzTJLkggsuSIsWDT67IQDwKRmbAYCGaPCaAkOHDs3QoUMbsxYA4FMwNgPAxlFoUb5r9PgIAQAAACpUg5MCAAAAUAnK+WQ+kgIAAABQoSQFAAAAoIhSnBWgqUgKAAAAQIWSFAAAAIAiyjgoICkAAAAAlUpSAAAAAIqwpgAAAABQdiQFAAAAoAhJAQAAAKDsNElS4KO5i5viMOvUpk3zCEasWFFb6hLSsWbzUpeQJPndmL+UuoQkySnf2rvUJSRJWrVuHj+jzaUT2hx+V1o3k+/JwoXLSl1CkuRH44eWugQaSW1tXalLSJJUVTWPzycWL15e6hKyVad2pS4hSXLD6GdKXUKS5Jsn71XqEpIkm7WqKnUJSZrP2Nwc/na0biav6ef9c16pS0iSHHdev1KXUBGaya/gRtE8RmIAAACgyWkKAAAAQIVqHtkbAAAAaK7KeP6ApAAAAABUKEkBAAAAKKK5LPa5MUgKAAAAQIWSFAAAAIAiyjgoICkAAAAAlUpSAAAAAIootCjfqICkAAAAAFQoSQEAAAAowpoCAAAAQNmRFAAAAIAiCmUcFZAUAAAAgAolKQAAAABFSAoAAAAAZUdSAAAAAIoo46CApAAAAABUKk0BAAAAqFCmDwAAAEARFhoEAAAAyo6kAAAAABQhKQAAAACUHUkBAAAAKKKMgwKSAgAAAFCpJAUAAACgCGsKAAAAAGVHUgAAAACKkBQAAAAAyo6kAAAAABRRxkEBSQEAAACoVJICAAAAUEShRflGBSQFAAAAoEI1SVKg/Ratm+Iw61RIM+nuNIMyFi5YWuoSkiTfOvVLpS4hSVJXV1fqEpI0n+9Lc9GqdenDTK02qyp1CUmSN9+YVeoSkiT9+nUvdQk0khXLa0tdQpJks1bN43esZcvSf06yePGyUpeQJPnWfzaPsXn+/OYxJm7WDH42kqS2tnm8VmkOL5laVDWDF9NpPj+jvXt1KXUJFcGaAgAAAEDZKf3HcAAAAMB6u/zyy/PKK6+kUChkxIgR6d27d/19zz33XH72s5+lRYsW6d69e0aNGpUWLdaeB5AUAAAAgCIKTfhvXV544YW8/fbbGTt2bEaNGpVRo0atcv+FF16Y66+/PnfccUcWLFiQp556quj+NAUAAABgE/Hss89mwIABSZIePXpk7ty5mT9/fv39EyZMSLdu3ZIkNTU1mT17dtH9aQoAAABAMYUmvKzDzJkz07Fjx/rrNTU1mTFjRv316urqJMn06dPzzDPPpF+/fkX3pykAAAAAm6g1nUlt1qxZOfXUUzNy5MhVGghrYqFBAAAAKKLQjM5J2KVLl8ycObP++vTp09O5c+f66/Pnz8/JJ5+cM888M/vtt9869ycpAAAAAJuIPn36ZOLEiUmS119/PV26dKmfMpAkV1xxRU444YT07dt3vfYnKQAAAABFNKOgQHbffff06tUrQ4cOTaFQyMiRIzNhwoS0b98+++23X+666668/fbbGT9+fJLk0EMPzZAhQ9a6P00BAAAA2IT84Ac/WOX6zjvvXP//1157bYP2pSkAAAAARTSnNQUamzUFAAAAoEJJCgAAAEARZRwUaFhTYMGCBTn33HMzd+7cLFu2LKeddlr233//xq4NAFhPxmYAoCEa1BS48847071793z/+9/PBx98kBNOOCEPPfRQY9cGAKwnYzMAbDzWFPgXHTt2zJw5c5IkH330UTp27NioRQEAG8bYDAA0RIOSAl/96lczYcKEDBw4MB999FFuuummxq4LANgAxmYA2HjKOCjQsKTA3XffnW222SaPPPJIbrvttlxyySWNXRcAsAGMzQBAQzSoKfDyyy9nv/32S5LsvPPOmT59elasWNGohQEA68/YDAA0RIOaAp/5zGfyyiuvJEnefffdtGvXLlVVVY1aGACw/ozNALDxFAqFJrs0tQatKTBkyJCMGDEiX//617N8+fJcdNFFjVwWALAhjM0AQEM0qCnQrl27XHfddY1dCwDQQMZmANh4LDQIAAAAlJ0GJQUAAACgUkgKAAAAAGVHUgAAAACKKKR8owKSAgAAAFChJAUAAACgCGsKAAAAwP9v7+5C7CrPPYD/d2bPzjiZqMkcR0+gDcUbi4dDK/QiUUvFkV70A7HUDIPpdSmltlcGKVgICSR4UaWlFm1vKoEpqf24KExoacCLUS8KFQWxLSVoTEyCScz3ZD7OhZyk09qVGDP7XbPW7xf2xewJez3Zs/d6Vp75v++mcSQFAAAAoEKnwVEBSQEAAABoKUkBAAAAqNDgoEB/hgIL84v9OMwVXZybK11CkmRVDV5RvdU1mQct1uO1UZMysmakV7qEJMnF2fnSJSRJgz/45aP78pfvKF1CkqSzyk+lKeoSg5y9UI/e3O0OlC4hg4Pla0hSm5Pv6l49no9VA/UI1nY69bhYmZ0t/57trR4sXUKS5H//59bSJSRJBgZq8qZlxarJ/wwBAACgnuoyTF8O9Rh9AgAAAH1nKAAAAAAtZfkAAAAAVGjw6gFJAQAAAGgrSQEAAACoYKNBAAAAoHEkBQAAAKBKc4MCkgIAAADQVpICAAAAUMGeAgAAAEDjSAoAAABAhQYHBSQFAAAAoK0kBQAAAKCCPQUAAACAxpEUAAAAgArNzQlICgAAAEBrSQoAAABABXsKAAAAAI1jKAAAAAAtZfkAAAAAVGjw6gFJAQAAAGgrSQEAAACoYKNBAAAAoHEkBQAAAKBCg4MCkgIAAADQVpICAAAAUEFSAAAAAGgcSQEAAACo4NMHAAAAgMaRFAAAAIAKDQ4KSAoAAABAW0kKAAAAQIUm7ynQl6HA4uJiPw5zRcPDvdIlJElq8nTUwuzsXOkSkiSrhwZLl5Akmbs4X7qEJMlAtx4hoouz5Z+PoRvq8dqYn18oXUKSZGHeCYzra3jN6tIlJKnHe2zVquZecF6LXq8ev7uqy3XshQv1uGaqQ1/sdgdKl5CkPtf0Z87Mli4hSTI8Uo/zOR9dPa78AQAAgL4zFAAAAICWMhQAAACAlqrHYi0AAACoqSZvNCgpAAAAAC0lKQAAAAAVGhwUkBQAAACAtjIUAAAAgJYyFAAAAICWsqcAAAAAVLCnAAAAANA4kgIAAABQoZPmRgUkBQAAAKClrmoo8Oabb2Z8fDzPP/98kuTQoUPZunVrJicn8+ijj2Z2dnZZiwQAltKbAaCPOn289dkVhwJnz57N9u3bs2nTpkv3Pf3005mcnMyePXuycePG7N27d1mLBAAu05sBgOvlikOBXq+XZ599NmNjY5fue/nll3P//fcnSe67777MzMwsX4UAwBJ6MwD0V6fTv1u/XXGjwW63m2536V87d+5cer1ekmR0dDRHjx5dnuoAgH+jNwMA18vH/vSBxcXF61EHAHCd6M0AcH359IF/MTw8nPPnzydJ3n333SXxRQCg//RmAOBaXNNQYPPmzZmenk6S7Nu3L/fee+91LQoA+Gj0ZgDgWlxx+cBrr72WXbt25eDBg+l2u5mens6TTz6Zbdu2ZWpqKhs2bMiDDz7Yj1oBgOjNANB3zV09kM5iHxYenj19YbkPcVV6qz/2FgrXhaWel83OzpUuIUnS69XjtTF3cb50CUmSVQP1OOtdnC3/fAzdMFi6hCTJ/PxC6RKSJAvz9TiB3bCmV7qEFe/MqXr0Zu+xy1atqse5ty46Jbbg/hB12aPjwoV6XDN1u9cUNL7ONQyULiFJcrEm121nzsyWLiFJMvpfa0qXsKwOHDjet2Nt3Liub8dKrsNGgwAAANBk9RhTLo/yoz4AAACgCEkBAAAAqFCXJU3LQVIAAAAAWkpSAAAAAKo0NyggKQAAAABtJSkAAAAAFRocFJAUAAAAgLaSFAAAAIAKPn0AAAAAaBxDAQAAAGipviwfGOzVY5XCwsJi6RKSJHNzC6VLyODgQOkSktSnjsXFerw2uoP1mNPVJR7VWV2+jjq8X5NkYKD8c5EkAwP1eI3y8a0eqktvrsd77NSpC6VLyMhIr3QJSZL5mpz36nL9WJOWmKGhwdIlJEnOnCn/Xpm9MF+6hCTJDcP1+JnceONQ6RJY4VzdAQAAQIVOp3+3q7Fz585s2bIlExMTefXVV5d878KFC3nsscfy0EMPXdVjGQoAAADACvHKK6/kwIEDmZqayo4dO7Jjx44l39+9e3c+/elPX/XjGQoAAABAhU6n07fblczMzGR8fDxJcvvtt+fkyZM5ffr0pe9/73vfu/T9q2EoAAAAACvEsWPHsm7duktfr1+/PkePHr309cjIyEd6PEMBAAAAWKE+7qbphgIAAACwQoyNjeXYsWOXvj5y5EhuueWWa348QwEAAABYIe6+++5MT08nSV5//fWMjY195CUD/6weHwALAAAANXW1HxXYD3fddVfuvPPOTExMpNPp5IknnsgLL7yQtWvX5oEHHsh3vvOdHD58OP/4xz+ydevWPPzww/nKV77yHx+vs/hxFyBchYuz88t9iKvSh3/qVZmbWyhdQgYHB0qXkKQ+P5O6qMvJ5mp2Pe2HhYXyr4861JAkAwP1+JnU5bXRrck5bCWbu6g3/7P3379QuoSMjPRKl5Akma/BdUqSDPbq8burmpz2anP+PXOm/Hulk3o8FzcMD5YuIUlSk9NoBnvN7s2HD73ft2Pd9t839u1YiaQAAAAAVKrLMGo52FMAAAAAWkpSAAAAAKo0NyggKQAAAABtJSkAAAAAFWqy1+eykBQAAACAlpIUAAAAgAoNDgpICgAAAEBbSQoAAABAlQZvKiApAAAAAC0lKQAAAAAVmpsTkBQAAACA1jIUAAAAgJayfAAAAAAqNHifQUkBAAAAaCtJAQAAAKjS4KiApAAAAAC0lKQAAAAAVGhuTkBSAAAAAFpLUgAAAAAqNHhLAUkBAAAAaCtJAf3gelkAAAcxSURBVAAAAKjU3KiApAAAAAC0lKQAAAAAVGjyngKdxcXFxdJFAAAAQF2deO9s34518/rhvh0rsXwAAAAAWstQAAAAAFrKngIAAABQocl7CkgKAAAAQEsZCgAAAEBLWT4AAAAAlZq7fkBSAAAAAFpqRQwFdu7cmS1btmRiYiKvvvpq6XKK2717d7Zs2ZKvfe1r2bdvX+lyijt//nzGx8fzwgsvlC6luN/97nf56le/moceeij79+8vXU5RZ86cybe//e1s3bo1ExMTefHFF0uXVMSbb76Z8fHxPP/880mSQ4cOZevWrZmcnMyjjz6a2dnZwhWyUunNS+nNS+nNl+nNl+nNH9CbV6ZOp3+3fqv9UOCVV17JgQMHMjU1lR07dmTHjh2lSyrqpZdeyl//+tdMTU3lueeey86dO0uXVNxPfvKT3HTTTaXLKO748eP58Y9/nD179uSZZ57JH//4x9IlFfXrX/86n/rUp/KLX/wiTz31VCvPHWfPns327duzadOmS/c9/fTTmZyczJ49e7Jx48bs3bu3YIWsVHrzUnrzv9ObP6A3L6U3683UU+2HAjMzMxkfH0+S3H777Tl58mROnz5duKpyPve5z+Wpp55Kktx44405d+5c5ufnC1dVzt///vf87W9/yxe+8IXSpRQ3MzOTTZs2ZWRkJGNjY9m+fXvpkopat25dTpw4kSR5//33s27dusIV9V+v18uzzz6bsbGxS/e9/PLLuf/++5Mk9913X2ZmZkqVxwqmNy+lNy+lN1+mNy+lN+vN1FPthwLHjh1bcsJYv359jh49WrCisgYGBjI8PJwk2bt3bz7/+c9nYGCgcFXl7Nq1K9u2bStdRi28/fbbOX/+fL75zW9mcnKy9Q3lS1/6Ut5555088MADeeSRR/LYY4+VLqnvut1uhoaGltx37ty59Hq9JMno6Girz6dcO715Kb15Kb35Mr15Kb1Zb6aeVtynDywuLpYuoRb+8Ic/ZO/evfn5z39eupRifvOb3+Qzn/lMPvGJT5QupTZOnDiRH/3oR3nnnXfyjW98I3/605/SKbEwqQZ++9vfZsOGDfnZz36WN954I48//ri1rf/C+ZTrxWvpA3qz3vxh9ObL9OYrcz6tsQa/bWs/FBgbG8uxY8cufX3kyJHccsstBSsq78UXX8wzzzyT5557LmvXri1dTjH79+/PW2+9lf379+fw4cPp9Xq57bbbsnnz5tKlFTE6OprPfvaz6Xa7+eQnP5k1a9bkvffey+joaOnSivjzn/+ce+65J0lyxx135MiRI5mfn2/1b++SZHh4OOfPn8/Q0FDefffdJfFFuFp687/Tmz+gNy+lNy+lN384vZnSar984O6778709HSS5PXXX8/Y2FhGRkYKV1XOqVOnsnv37vz0pz/NzTffXLqcon74wx/mV7/6VX75y1/m61//er71rW+19qIjSe6555689NJLWVhYyPHjx3P27NlWrtX7fxs3bsxf/vKXJMnBgwezZs2a1l90JMnmzZsvnVP37duXe++9t3BFrER681J682V681J681J684fTm1eGTh//9FvtkwJ33XVX7rzzzkxMTKTT6eSJJ54oXVJRv//973P8+PF897vfvXTfrl27smHDhoJVUQe33nprvvjFL+bhhx9Oknz/+9/PqlW1n/stmy1btuTxxx/PI488krm5ufzgBz8oXVLfvfbaa9m1a1cOHjyYbreb6enpPPnkk9m2bVumpqayYcOGPPjgg6XLZAXSm5fSm/lP9Oal9Ga9mXrqLFq4AgAAAP/RqZPn+3astTcNXfkvXUftHVUCAABAy9V++QAAAACU1OQPDZEUAAAAgJYyFAAAAICWMhQAAACAljIUAAAAgJay0SAAAABUafBOg5ICAAAA0FKSAgAAAFChuTkBSQEAAABoLUkBAAAAqNLgqICkAAAAALSUpAAAAABUaHBQQFIAAAAA2kpSAAAAAKp0mpsVkBQAAACAljIUAAAAgJYyFAAAAICWsqcAAAAAVGjujgKSAgAAANBakgIAAABQpcFRAUkBAAAAaClDAQAAAGgpywcAAACgQqfB6wckBQAAAKClJAUAAACgSnODApICAAAA0FaSAgAAAFChwUEBSQEAAABoK0kBAAAAqNLgqICkAAAAALSUpAAAAABUam5UQFIAAAAAWkpSAAAAACo0NycgKQAAAACtJSkAAAAAVRocFZAUAAAAgJaSFAAAAIAKDQ4KSAoAAABAWxkKAAAAQEtZPgAAAABVOs1dQCApAAAAAC1lKAAAAAAtZSgAAAAALWVPAQAAAKjQ4C0FJAUAAACgrQwFAAAAoKUMBQAAAKCl7CkAAAAAFToN3lRAUgAAAABaylAAAAAAWspQAAAAAFqqs7i4uFi6CAAAAKD/JAUAAACgpQwFAAAAoKUMBQAAAKClDAUAAACgpQwFAAAAoKUMBQAAAKCl/g8lgLzX1C3qigAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "all_weights_finetuned = []\n",
+ "all_weights_base = []\n",
+ "for example in tqdm(data):\n",
+ " weights_finetuned = analyze_cross_attention(example[1], example[2], max_len, model_finetuned, relation=\"VERB-SUBJ\")\n",
+ " weights_base = analyze_cross_attention(example[1], example[2], max_len, model_base, relation=\"VERB-SUBJ\")\n",
+ "\n",
+ " all_weights_finetuned.append(weights_finetuned)\n",
+ " all_weights_base.append(weights_base)\n",
+ "\n",
+ "all_weights_finetuned = np.stack(all_weights_finetuned, axis=-1)\n",
+ "all_weights_base = np.stack(all_weights_base, axis=-1)\n",
+ "\n",
+ "avg_weights_finetuned = np.mean(all_weights_finetuned, axis=-1)\n",
+ "avg_weights_base = np.mean(all_weights_base, axis=-1)\n",
+ "\n",
+ "visualize_before_and_after(avg_weights_base, avg_weights_finetuned, 'VERB-SUBJ attention', cmap=\"Purples\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### 5.3 CLS"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 65,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def cls_last_layer(task):\n",
+ " \n",
+ " if task == \"MNLI\":\n",
+ " mnli=True\n",
+ " else:\n",
+ " mnli=False\n",
+ " \n",
+ " if task == \"STS-B\":\n",
+ " sts = True\n",
+ " else:\n",
+ " sts = False\n",
+ " \n",
+ " data = read_dataset(task, path_to_data)\n",
+ " model_finetuned = restore_model(path_to_model, sts=sts, mnli=mnli)\n",
+ " if len(data) > 1000: \n",
+ " data = random.sample(data, 1000)\n",
+ "\n",
+ " all_pron_scores = []\n",
+ " all_noun_scores = []\n",
+ " all_verb_scores = []\n",
+ " all_obj_scores = []\n",
+ " all_subj_scores = []\n",
+ " all_neg_scores = []\n",
+ " \n",
+ " all_cls_scores = []\n",
+ " all_sep_scores = []\n",
+ " for example in tqdm(data):\n",
+ " bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(example[1], example[2], max_len)\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(example[1], example[2], max_len)\n",
+ " _, output_finetuned = model_finetuned(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " output_finetuned = output_finetuned.squeeze(0).detach().numpy()\n",
+ " \n",
+ " crop_len = torch.sum(input_mask != 0)\n",
+ " sep2_idx = crop_len - 1\n",
+ " sep1_idx = torch.sum(input_mask - segment_ids) - 1\n",
+ "\n",
+ " last_layer = output_finetuned[11]\n",
+ "\n",
+ " prons = detect_all_pos(example[1], 'PRON').union(detect_all_pos(example[2], 'PRON'))\n",
+ " nouns = detect_all_pos(example[1], 'NOUN').union(detect_all_pos(example[2], 'NOUN'))\n",
+ " verbs = detect_all_pos(example[1], 'VERB').union(detect_all_pos(example[2], 'VERB'))\n",
+ " objs = detect_all_dep(example[1], 'OBJ').union(detect_all_dep(example[2], 'OBJ'))\n",
+ " subjs = detect_all_dep(example[1], 'SUBJ').union(detect_all_dep(example[2], 'SUBJ'))\n",
+ " negs = detect_all_negations(example[1]).union(detect_all_negations(example[2]))\n",
+ "\n",
+ " prons_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(pron) for pron in prons]))\n",
+ " nouns_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(noun) for noun in nouns]))\n",
+ " verbs_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(verb) for verb in verbs]))\n",
+ " objs_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(obj) for obj in objs]))\n",
+ " subjs_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(subj) for subj in subjs]))\n",
+ " negs_tokens = list(itertools.chain.from_iterable([tokenizer.tokenize(neg) for neg in negs]))\n",
+ "\n",
+ " target_prons_ids = [i for i, token in enumerate(bert_tokens) if token in prons_tokens]\n",
+ " target_nouns_ids = [i for i, token in enumerate(bert_tokens) if token in nouns_tokens]\n",
+ " target_verbs_ids = [i for i, token in enumerate(bert_tokens) if token in verbs_tokens]\n",
+ " target_objs_ids = [i for i, token in enumerate(bert_tokens) if token in objs_tokens]\n",
+ " target_subjs_ids = [i for i, token in enumerate(bert_tokens) if token in subjs_tokens]\n",
+ " target_negs_ids = [i for i, token in enumerate(bert_tokens) if token in negs_tokens]\n",
+ "\n",
+ " pron_scores = []\n",
+ " noun_scores = []\n",
+ " verb_scores = []\n",
+ " subj_scores = []\n",
+ " obj_scores = []\n",
+ " neg_scores = []\n",
+ " \n",
+ " cls_scores = []\n",
+ " sep_scores = []\n",
+ "\n",
+ " for h in range(n_heads):\n",
+ " if target_prons_ids: \n",
+ " pron_score = max(last_layer[h, 0, target_prons_ids])\n",
+ " pron_scores.append(pron_score)\n",
+ "\n",
+ " if target_nouns_ids: \n",
+ " noun_score = max(last_layer[h, 0, target_nouns_ids])\n",
+ " noun_scores.append(noun_score)\n",
+ "\n",
+ " if target_verbs_ids: \n",
+ " verb_score = max(last_layer[h, 0, target_verbs_ids]) \n",
+ " verb_scores.append(verb_score)\n",
+ "\n",
+ " if target_objs_ids: \n",
+ " obj_score = max(last_layer[h, 0, target_objs_ids]) \n",
+ " obj_scores.append(obj_score)\n",
+ "\n",
+ " if target_subjs_ids: \n",
+ " subj_score = max(last_layer[h, 0, target_subjs_ids]) \n",
+ " subj_scores.append(subj_score)\n",
+ "\n",
+ " if target_negs_ids:\n",
+ " neg_score = max(last_layer[h, 0, target_negs_ids]) \n",
+ " neg_scores.append(neg_score) \n",
+ " \n",
+ " cls_scores.append(last_layer[h, 0, 0])\n",
+ " sep_scores.append(max([last_layer[h, 0, sep1_idx], last_layer[h, 0, sep2_idx]]))\n",
+ "\n",
+ " pron_scores = np.array(pron_scores)\n",
+ " noun_scores = np.array(noun_scores)\n",
+ " verb_scores = np.array(verb_scores)\n",
+ " obj_scores = np.array(obj_scores)\n",
+ " subj_scores = np.array(subj_scores)\n",
+ " neg_scores = np.array(neg_scores)\n",
+ " \n",
+ " cls_scores = np.array(cls_scores)\n",
+ " sep_scores = np.array(sep_scores)\n",
+ "\n",
+ " if target_prons_ids:\n",
+ " all_pron_scores.append(pron_scores)\n",
+ " if target_nouns_ids:\n",
+ " all_noun_scores.append(noun_scores)\n",
+ " if target_verbs_ids: \n",
+ " all_verb_scores.append(verb_scores)\n",
+ " if target_objs_ids: \n",
+ " all_obj_scores.append(obj_scores)\n",
+ " if target_subjs_ids: \n",
+ " all_subj_scores.append(subj_scores)\n",
+ " if target_negs_ids: \n",
+ " all_neg_scores.append(neg_scores)\n",
+ " \n",
+ " all_cls_scores.append(cls_scores)\n",
+ " all_sep_scores.append(sep_scores)\n",
+ "\n",
+ " all_pron_scores = np.mean(np.stack(all_pron_scores), axis=0)\n",
+ " all_noun_scores = np.mean(np.stack(all_noun_scores), axis=0)\n",
+ " all_verb_scores = np.mean(np.stack(all_verb_scores), axis=0)\n",
+ " all_obj_scores = np.mean(np.stack(all_obj_scores), axis=0)\n",
+ " all_subj_scores = np.mean(np.stack(all_subj_scores), axis=0)\n",
+ " all_neg_scores = np.mean(np.stack(all_neg_scores), axis=0)\n",
+ " \n",
+ " all_cls_scores = np.mean(np.stack(all_cls_scores), axis=0)\n",
+ " all_sep_scores = np.mean(np.stack(all_sep_scores), axis=0)\n",
+ " \n",
+ " return (all_pron_scores, all_noun_scores, all_verb_scores, all_obj_scores, \n",
+ " all_subj_scores, all_neg_scores, all_cls_scores, all_sep_scores)\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 66,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def visualize_cls(data, task):\n",
+ " all_pron_scores, all_noun_scores, all_verb_scores, all_obj_scores, all_subj_scores, all_neg_scores, all_cls_scores, all_sep_scores = data\n",
+ " fig, ((ax1, ax2), (ax3, ax4), (ax5, ax6), (ax7, ax8)) = plt.subplots(4, 2, sharex=True, sharey=True, \n",
+ " figsize=(12, 6))\n",
+ " plt.subplots_adjust(wspace=0.04, hspace=0)\n",
+ "\n",
+ " ax1.imshow(np.expand_dims(all_pron_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax2.imshow(np.expand_dims(all_noun_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax3.imshow(np.expand_dims(all_verb_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax4.imshow(np.expand_dims(all_obj_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax5.imshow(np.expand_dims(all_subj_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax6.imshow(np.expand_dims(all_neg_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " ax7.imshow(np.expand_dims(all_cls_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ " im = ax8.imshow(np.expand_dims(all_sep_scores, 0), vmin=0, vmax=0.5, cmap=\"YlGn\")\n",
+ "\n",
+ " ax1.grid(False)\n",
+ " ax2.grid(False)\n",
+ " ax3.grid(False)\n",
+ " ax4.grid(False)\n",
+ " ax5.grid(False)\n",
+ " ax6.grid(False)\n",
+ " ax7.grid(False)\n",
+ " ax8.grid(False)\n",
+ "\n",
+ " ax1.set_title(\"PRON\", fontsize=28)\n",
+ " ax2.set_title(\"NOUN\", fontsize=28)\n",
+ " ax3.set_title(\"VERB\", fontsize=28)\n",
+ " ax4.set_title(\"OBJ\", fontsize=28)\n",
+ " ax5.set_title(\"SUBJ\", fontsize=28)\n",
+ " ax6.set_title(\"NEG\", fontsize=28)\n",
+ " ax7.set_title(\"[CLS]\", fontsize=28)\n",
+ " ax8.set_title(\"[SEP]\", fontsize=28)\n",
+ " \n",
+ " ax1.set_yticks([0])\n",
+ " ax7.set_xticks(range(12))\n",
+ "# ax8.set_xticks(range(12))\n",
+ "\n",
+ " ax1.set_yticklabels([], fontsize=12)\n",
+ " ax3.set_yticklabels([], fontsize=12)\n",
+ " ax5.set_yticklabels([], fontsize=12)\n",
+ " ax7.set_yticklabels([], fontsize=12)\n",
+ "\n",
+ " ax7.set_xticklabels(range(1, 13), fontsize=14)\n",
+ " ax8.set_xticklabels(range(1, 13), fontsize=14)\n",
+ "\n",
+ "# fig.colorbar(im, ax=[ax1, ax2, ax3, ax4, ax5, ax6, ax7, ax8], pad=0.05)\n",
+ "\n",
+ "\n",
+ " fig.suptitle(task, fontsize=32, weight='bold')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 67,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 1000/1000 [03:23<00:00, 4.92it/s]\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqsAAAF5CAYAAAC4BWNIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xd4VFX+x/HPpFdIAgQC0pEqHRHpRRDpzaUtSFHUpQgWpAsYKSKCoCAsIkhbkI6wsPxAXUQEA4TeawgtEAiQmGSSzO8PnswypE0Kk4u+X8+zzzO5c+4538nyXD+5c+45JovFYhEAAABgQE65XQAAAACQFsIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMNyye0CACArwsLCtHPnTv3666+6ePGi7t69q+joaOXNm1cBAQGqUKGC6tevryZNmsjX1zfT/Z8+fVo//fST9uzZoytXrujOnTuKjY2Vn5+f8uXLp6pVq6pevXpq3Lix3N3d0+3rypUratasmV3jenl5yc/PT+XKlVOdOnXUoUMH+fn5Zbp+APizMLEpAICnydWrV/XFF19o06ZNSkxMzLB9njx51KdPH/Xv318eHh4Ztj9z5ow+//xz7dy50656ChQooDfffFM9evSQs7Nzqm0yE1Yf5+npqQ8++EA9e/bM0vkA8LQjrAJ4auzdu1dDhgzR3bt3U33fyclJSUlJqb5XqVIlzZ07VwULFkyz/y1btmjkyJGKjY3NdP8NGjTQjBkzUr2Lm1pYdXFxkZOT7UyspKQkJSQkpNr/4MGDNWjQoDRrB4A/K8IqgKdCaGio/v73v8tsNluPubq6qkePHmrTpo3KlCkjT09P3blzR7/99pu+++47HTx40KaPokWLau3atcqTJ0+K/rdt26YhQ4bYHPPy8lLfvn318ssvq3jx4nJzc1NERIR2796tBQsW6Ny5czbtq1atqqVLl8rNzc3meGph9b333tOAAQNS1HHz5k395z//0YwZM/TgwQPrcScnJ61fv17lypXL4DcFAH8uzuPHjx+f20UAQHru3bun1157TVFRUdZj/v7+Wrp0qTp16qSCBQvK1dVVJpNJnp6eevbZZ9W5c2fFx8dr//79Nv2cP39erVu3tuk/LCxMb7zxhuLj463HihUrplWrVql58+bKnz+/XFxcZDKZ5OPjowoVKujVV19VeHi4Tp06ZT3nxo0bio6OVsOGDVPU/91339kcq1u3rmrWrJnis3p7e6tKlSqqXLmyNmzYYD1usVhksVjUpEmTTP72AODpxmoAAAxvyZIlunbtms2xL774QhUrVkzzHJPJpPfee08vv/yyzfEdO3YoJCTE5tjcuXNt7mK6ublp3rx5KlKkSJr9u7m5acqUKapatarN8eXLl+vKlSsZfqaM1K1bV+XLl7c59vvvv2e7XwB42hBWARhafHy8li5danOsefPmeuGFF+w6f8SIESnmhi5cuND6+tatW9q4caPN+z169FCpUqUy7NvZ2VkffvihzbGEhAQtXrzYrtoyUrp0aZufIyIicqRfAHiaEFYBGNrhw4cVGRlpc6xDhw52n1+4cGHVqVPH5tivv/5q/cp/9+7dNvNgJaljx45291+zZk0VL17c5thPP/1k9/npuX//vs3Pj8+FBYC/AsIqAEN7dM5pstTmeqbn8fZ//PGHjhw5kmr/Pj4+mX6I6fH+L1++rBs3bmSqj9Q8/gDX46EYAP4KCKsADO3SpUs2P/v5+cnf3z9TfaT2lX5YWFiq/RcvXlwmkynH+s+qrVu3Kjw83OZY48aNs9UnADyN2MEKgKE9vqZqVnajSm2pquR+n3T/mRUREaEffvhBM2bMSDFG165ds9QnADzNCKsADO2PP/6w+TmtRfnTk9pOV8kL/z+p/h/v93FffPGFZs+enWLs1DYFcHZ21qeffqq8efNmujYAeNoRVgEYmre3t83PWblj+fiDStL/7qA+qf5Tu9v6qLR2qnpcwYIFNXXqVL344ouZrgsA/gwIqwAM7fH5qdHR0bp69aoKFy5sdx9nz55NcaxAgQKp9n/x4kWZzWa5urpmq//8+fPbff6jPD095e/vr4oVK6px48Zq3749qwAA+EsjrAIwtNSezN+7d2+mlpdKbUWB5557ztr/7t27rcfj4+N18OBB1a5dO8v9u7m56dlnn033nLS2WwUA2GI1AACG9vgOUZK0cuVKu88PCwtLsWNV0aJFrXdms9v/vn37Ujy1X6NGDe6GAkAOIawCMLTKlSurRIkSNscOHjyoTZs22XX+1KlTUzw09ehd2UaNGsnHx8fm/c2bN6cIuKlJSEjQp59+muJ4Zu76AgDSR1gFYHjdu3dPcWzMmDE2X98/LikpSZ9++qm2b99uczxPnjw2/Xl6eqpTp042bSwWiwYNGqQTJ06k2X98fLw+/PBD6+YCyYoWLapWrVql+3kAAPYjrAIwvJ49e6aYAxobG6v+/ftr1KhRCgkJ0f3795WYmKgbN25o06ZN6tatm7755psUfb333nsKCAiwOTZo0KAUD0TduXNHr776qiZPnqwjR44oOjpaZrNZV69e1apVq9S+fXv98MMPNueYTCaNGzeOKQAAkINMFovFkttFAEBGTp48qd69eysqKirLffTq1UtjxoxJ9b1ff/1Vb775puLj47Pc/4gRI9S3b98Ux69cuaJmzZrZHOMBKwCwD3dWATwVypcvrwULFigwMDDT57q4uOi9997T6NGj02xTt25dffnllxmuj5oaT09PTZ06NdWgCgDIHsIqgKdGlSpVtHnzZnXt2lUeHh4Ztnd1dVXr1q21fv16DRgwQCaTKd32jRo10g8//KCWLVvKxSXjlf28vLzUtWtXbd68WR06dLD7cwAA7Mc0AABPpXv37mn79u06evSo9u/fr1OnTtm8379/fw0cODDFDlX2unXrlrZt26YTJ05o7969unz5ss37o0aNUvfu3ZmfCgBPGGEVwFPv+vXratKkic0SVTVq1NCKFStypP+DBw+qW7duNsfatGmj6dOn50j/AIC0MQ0AwFOvUKFCql+/vs2xAwcOaPLkydl6YCpZ9erVVbp0aZtjP/zwg7799tsUa7gCAHIWd1YB/Cn88ssv6t+/f4rjAQEBql27tvz8/BQbG6u//e1vqlmzZqb7X7lypcaNG5fieFBQkGrUqKE8efLo/v37Gjx4cIpNDAAAWUdYBfCn8d5776VY+/RxX3zxhVq2bJnpvpOSktSrV68Md7ZavXq1KleunOn+AQCpYxoAgD+NTz75RE2bNn0ifTs5OWn27NmqXr36E+kfAJA65/Hjx4/P7SIAICe4uLiodevWKlOmjKKjo/XgwQOZzWZ5eXkpMDBQNWrUUMuWLVWgQIEs9e/p6akOHTooKChI0dHRio6OVkJCgnx8fBQUFKTnn39eLVu2lK+vbw5/MgD462IaAAAAAAyLaQAAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADMsltwuA8fTq1Uv79u1L831nZ2f5+vqqePHiqlOnjrp06aJixYrZtLly5YqaNWuW4VgeHh4KCAhQhQoV1KJFC7Vu3Vqurq521blnzx7t3LlTISEhunnzpqKiomz6q1evnlq3bi1vb+80+3i0TpPJpOXLl6tGjRoZjp38Oxo0aJAGDx5sV70A/poevaZ27dpVEydOzPCctWvXauTIkSpSpIh27tyZZrvw8HBt3rxZv/32my5evKg7d+4oMTFR/v7+euaZZ1SnTh21atVKpUuXTne8pk2bKjw8XI0bN9a8efPs+lwTJ07UsmXLJEmnTp2yeW/EiBFat26dJKl+/fr65ptvMuxv79696t27d6r94a+NsIo0ubu7q2TJkimOm81m3bhxQ4cOHdKhQ4e0cOFCDR8+3HqReVxgYKACAgJSHLdYLLp165auXbumq1evaseOHVqwYIG+/vprPfPMM2nWdfToUY0fP15HjhyxHsuXL5+KFSummJgYXb16VZcuXdLWrVs1ffp0DRo0SL169crw81osFo0bN07r1q2zOzADQGasWrVK7dq1U61atbLVT0xMjKZMmaK1a9fKbDZLkry8vJQ/f345OTnp1q1bCgkJUUhIiObMmaOWLVtq7NixqV6Ln7RffvlFGzduVLt27Rw+Nv4cCKtIU7FixbRhw4ZU37NYLAoNDdWkSZN0+PBhffLJJwoKClLz5s1TtO3Tp4/69++f5jgPHjzQmjVrNH36dJ05c0ZvvPGGNm7cmGpg3L59u9577z3FxcXJ399fAwYM0CuvvKKgoCBrm5iYGO3evVtz587VsWPHFBwcrCNHjmjKlClyckp75ouTk5POnDmj+fPna+DAgen9agAg05ycnJSUlKRx48Zp/fr1cnNzy1I/t27dUr9+/XTq1Ck5OTmpXbt26tmzp6pUqWK9xlksFh0/flwrV67U6tWrtWXLFuvNhRIlSuTgp0pf8meePHmyGjRoIH9/f4eNjT8P5qwiS0wmk6pXr65FixZZg+KcOXOy1JePj49ee+01jR8/XpJ0/vx5bdmyJUW7kydP6v3331dcXJwqVaqkzZs3q1+/fjZBVXp4d6F58+Zas2aN9Y7qhg0b9OWXX6ZbR8+ePSVJX3/9tc6fP5+lzwIAaenYsaO8vLx07tw5u79qf1xiYqKGDRumU6dOycPDQ1999ZWmTZumatWq2fwxbjKZVKlSJU2cOFELFy6Ut7e3wsPDNXDgQEVHR+fUR8rQSy+9pEKFCikyMlJTp0512Lj4cyGsIlu8vb318ssvS5KOHz+umJiYLPfVpk0beXh4SFKqc2aDg4MVGxsrPz8/zZ07V/ny5Uu3P5PJpDFjxqhBgwaSHobQsLCwNNv37t1bzz33nOLj4zVu3DhZLJYsfxYAeFxQUJCGDh0qSZo3b57OnTuX6T7Wr19vvT6OHDlSTZs2zfCcOnXqKDg4WJJ09uxZffvtt5keN6u8vb01duxYSdK6deu0Z88eh42NPw/CKrLt0a91svMXu5ubm/LkySNJioqKsnnv8OHD+v333yVJ/fr1U8GCBe3ud8SIEZIe3pFYtGhRmu2cnJz08ccfy9nZWb///ru+//77TH4CAEhfr169VLlyZZnNZo0dOzbTfxQnP6hUsmRJde3a1e7zWrVqpcqVK0uSlixZovj4+EyNmx0vvfSSWrRoIUn66KOPFBcX57Cx8edAWEW2Xbp0SdLDJ/uzMx8pOjpakZGRkh4+lPWo//73v9bXHTp0yFS/ZcqUsV6kf/7553TbVqxYUa+99pokadq0aYqIiMjUWACQHicnJwUHB8vFxUX79+/XypUr7T43PDzceje2Q4cOMplMmRq7Y8eOkqS7d+/q8OHDmTo3u8aMGSNfX19dunQpwylZwOMIq8iWsLAwbd26VZLUuHFjubhk/Zm95cuXKyEhQdLDpU4eFRoaKkkqUqRIpu6qJqtZs6a13tu3b6fbdsiQIXrmmWd0794961dnAJBTypcvrz59+kiSPvvsM928edOu8w4dOmR9Xb169UyPm3wdlKSDBw9m+vzsKFiwoN59911J0sKFC3Xy5EmHjo+nG2EVmWY2m3X16lV9//336tGjh2JiYuTr66thw4Zlqb+oqCh9+eWXmjFjhiTp+eefV+PGjW3aJAfMQoUKZWmMRx/CunXrVrptPT09rQ97bd26VT/++GOWxgSAtAwePFhFixbV/fv37f6j+NFr1+MPltrj0XMy+qP9SejevbuqV6+uhIQEjR07VklJSQ6vAU8nlq5Cms6cOaNy5cpl2K5EiRKaPn16msuhLFq0SBs3bkz1vejoaIWHh1svWi+88IJmzZqVYomp5Lmwnp6emfgE//PoxgD2zKtt0KCB2rZtq02bNmnChAmqXbt2upsLAEBmeHh4aMKECerXr5+2bdumHTt2ZLiRyqPXrqxcCzN7HcxpJpNJwcHB6tChgw4fPqwlS5ZYp10B6SGsIk1pbQrg5OQkHx8flSxZUvXq1VOzZs3S/fr/5s2bGX7N1bp1a7Vt21aNGzdOdR6Wj4+PpIdrsmbFo+flzZvXrnNGjRqlXbt26dq1a5oxY4bGjBmTpbEBIDX16tVT+/bttWHDBk2cOFEvvPCC9VqXGl9fX+vr+/fvq0CBApka79HrYPLDrI5WpkwZvfHGG5ozZ45mzpypFi1aZOkuMf5aCKtIU3qbAmTG8OHDU90U4PDhw+ratauSkpJUsGBBNWnSJM0+AgMDdezYMV2+fDlLNYSHh0t6+Je9vRf4gIAAffjhhxo5cqSWLVumdu3aqUqVKlkaHwBSM2LECP33v//V9evX9fnnn2vcuHFptn30wdOwsDCVKlUqU2MlXwcf7yu7Mvug19tvv61///vfunDhgiZMmKCvv/46x2rBnxNzVpFrqlSpoh49ekiSFi9enO7TqckPE0RGRmZpbcLkhwnKli2bqTsKnTp1Up06dZSUlKQxY8ZYHwADgJwQEBBgXV5vxYoV1odJU1OtWjXr69TWos7Io30///zzNu8lTxHIzFrZ9+7dk2T/t1XJ3NzcNHHiRJlMJv3444+pbgIDPIqwilw1bNgwBQYGKjExUSNHjkxz7b9HH7jK7Pqn586d09GjRyXJrgW0Hzdx4kS5u7vr1KlT1jUOASCndOjQQXXr1lVSUpLGjh0rs9mcarvAwEBVqlRJ0sNd+TK7Vur69eslSYULF1aFChVS9C3Z3n3NSPJOf88880ym6pCk2rVrq3PnzpKkSZMmWYMvkBrCKnKVj4+PdXeTs2fP6quvvkq1Xbly5azLWS1fvjxTd1cnT54s6eEDCb179850jcWLF9c//vEPSdJXX31lXVcWAHLKhAkT5OHhodOnT2vBggVptkueUhUREZGpr883b95s/fbq9ddfT/HVffL1NTw8XCdOnMiwv7Nnz+r48eOSlO4UrvQMHz5c+fPnV0REhD799NMs9YG/BsIqcl2LFi2sF7sFCxakeaEcO3asvL29FRcXpzfffDPdrVMlyWKx6JNPPtGuXbskSR988IECAgKyVGP//v1VtmxZxcXFpTunDACyolixYho4cKAkae7cuWn+UdyqVSs1bNjQ2s6eb5r27dtnvSlQvXp1/e1vf0vRpn379tav84ODg9O9axsXF6cxY8bIYrHI19dX3bt3z7CG1OTNm1ejRo2SJK1evdq6SyHwOMIqDOGjjz6Sl5eXEhISNHLkyFTnhpYoUUIzZsyQl5eXwsLC1L59e82dOzfF11axsbHasWOHXn31VX333XeSHm7R2rNnzyzX5+rqqo8//lhOTk767bff0p1XBgBZ0a9fP5UrV05xcXFauHBhqm1MJpM+/fRTVa5c2TqXftCgQdq/f3+KdUtPnjypCRMmqE+fPoqOjlbZsmU1a9Ysubq6pug3ICBAEyZMkJOTk0JCQtSrVy+FhITY9Gk2m/XTTz/p1Vdf1cGDB+Xs7Kzg4GDly5cvy5+5devWatiwoSwWi+bNm5flfvDnxmoAMISgoCC98847mjx5sk6cOKH58+dbv3p/VKNGjbRixQp99NFHCg0N1cyZMzVz5kzly5dP/v7+io6O1q1bt6xzvgIDA/XBBx+oXbt22a6xWrVq6t69u5YtW+bQfbUB/DW4uLgoODhYXbt2Tfca4+/vr6VLl+qzzz7Tv/71L23fvl3bt2+Xl5eXChQoIJPJpFu3blmXqnJxcVHXrl314Ycfprte9CuvvCJXV1eNHTtWoaGh6tmzp7y9vVWoUCElJibq2rVriouLkyQVKFBAH3/8cZanADxq/PjxatOmTaYe7sJfC2EVhtGrVy9t3LhRx44d05w5c9SiRQuVKVMmRbvy5ctr5cqV+vXXX7Vz506FhITo5s2bunTpkjw8PFSkSBE999xzql+/vlq1aiV3d/ccq/Hdd9/Vjh07dP369RzrEwCSValSRT179tSSJUvSbefh4aExY8aob9++2rx5s3799VddunRJN2/elMVikZ+fnypWrKgXX3xRbdu2VdGiRe0a/6WXXlKdOnW0YcMG/fTTTzp58qTCwsJkMpmUL18+VahQQU2aNFHr1q3l5eWVEx9ZRYoUsd6sAFJjslgsltwuAgAAAEgNc1YBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWOkuXWWxXHFUHbIozmFjSVJC0g2HjWUyeTpsLElKtNxy2FiuTsUdNpaTcmaZFCOKT7ro0PFMppSLgj+xseTmsLEczdWpeqbPceR11Wy56rCxJGlSyEaHjTWyZguHjSVJUfGRDhsrj1teh43l7pRyecAny5RxkxxyN/6gw8Z6ON4Dh41V1LuSw8aSpERLlMPGcndukOIYd1YBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGJbJYrFYcrsIAAAAIDXcWQUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhueR2ATCeIUOGaNu2bXJ1ddVPP/2k/Pnz233u999/rzFjxkiSZsyYoSpVqqhZs2ZZqmPQoEEaPHiw9efZs2fryy+/TPcck8kkHx8fBQUFqWbNmurUqZOqVKmSatu1a9dq5MiRafbl6empgIAAPfvss2rQoIE6deokLy+vLH0WAMius2fPauvWrfrtt98UFhamu3fvymQyyd/fXyVLllSdOnXUpk0bFS5cOM0+mjZtqvDw8FTfc3FxkY+Pj4oUKaJq1aqpffv2qlq1ql211apVS/fv309x/IsvvlDLli3t+4BAGgirSKF79+7atm2bzGaz1q5dqwEDBth97qpVqyRJBQoUUPPmzXXjxg3re4GBgQoICLC7r/RCctmyZeXklPKLgfj4eF2/fl2nT5/W6dOntWLFCv3973/XmDFjZDKZ0uyvRIkS8vDwsP5ssVgUGRmp69evKzw8XD/99JPmzZunefPmqWLFinZ/BgDIrsjISE2YMEHbtm2TxWKRJPn4+KhQoUJKTEzU7du3tXv3bu3evVuzZs1Sly5d9OGHH8rT0zPNPn18fPTMM8/YHIuNjVVERISOHTumY8eOadmyZerRo4c++uijDGssW7asoqOjJUkxMTG6fPlyNj4xYIuwihRefPFFlSpVSufPn9f333+vN954I92gl+zkyZM6fPiwJKlLly5ydXW1eb9Pnz7q379/jtS4ePHidIPv+fPnNWPGDP3nP//R0qVLFRAQoIEDB6bZ/rPPPlPlypVTHL9//77WrVun6dOn6+bNmxo8eLD+85//yNnZOUc+BwCk5/z58+rfv7+uXr0qV1dXdevWTV26dFH58uWtbRITExUaGqrFixdr27ZtWrFihUJDQ7Vw4cI0r5O1atXSvHnzUn3v0KFDmjFjhvbs2aPly5erUqVK6tKlS7p1Ll++3Pp679696t27dxY+LZA65qwiVd27d5ckXb58WXv27LHrnOS7qs7OzurWrdsTq80epUqV0qxZs1S7dm1J0oIFCxQXF5fpfnx9fdW7d29r0L1y5YoOHDiQo7UCQGoePHiggQMH6urVq/Lz89OyZcs0ZswYm6AqPbzm1qxZU7NmzdL06dPl6uqqEydOaOjQoUpMTMz0uFWrVtWcOXPk5+cnSdq4cWOOfB4gqwirSFXHjh2t8zNXrlyZYfvY2Fht2rRJktSkSRMVKlToidZnD5PJpM6dO0t6+LXUkSNHstxXjRo1rK8jIiKyXRsAZGThwoU6f/68JGnatGl2zR9t06aNhgwZIunhHc7k63JmeXl5qUKFCpK45iH3EVaRKl9fX7Vp00aStGPHDkVGRqbbfsuWLbp3754kqUePHk+8PnsVKFDA+joqKirL/dy6dcv6Or2HFwAgJ8THx2vp0qWSpPr166thw4Z2n9uvXz8FBgZKkr755pss15B83eOah9xGWEWakkOn2WzWmjVr0m2bPAWgRIkSqlu37hOvzV5hYWHW18kX76zYunWrpIef77nnnst2XQCQntDQUOsf2B06dMjUuS4uLmrbtq0k6fTp07p27Vqmxz916pT1rm5yX0BuIawiTRUqVFD16tUlSatXr7Y+hfq4M2fO6ODBg5IeznW152EsR4iPj9eKFSskPVxZIPkrLXs9ePBAISEhGjJkiP7973/L399fn376qVxceC4RwJMVGhpqfZ18Hc6MmjVrWl8nX58zYjabFR4eruXLl6tv375KTExUhw4d1K5du0yPD+Qk/quLdHXv3l0HDx7UxYsX9dtvv+nFF19M0SZ5TquHh4c6deqUZl+LFi2ye6L+Cy+8oFGjRmWtaD1cmWDy5Mk6efKkJGno0KHphsz0nnR1cXFRz549NWDAAEPMxQXw53f79m3r66xcd4KCglLtK9lPP/2kcuXKpXl+qVKlNG7cONZIhSEQVpGuV155RZMnT9adO3e0atWqFGE1Li7OGkBbt26tPHnypNnXzZs3dfPmTbvGLVKkSLrvv/baa6mus5qUlKRbt25Z59i6uLho2LBhevXVV9Pt7/F1VqWHD43dvHlTMTExWr58uU6fPq1BgwapTp06dn0GAMiq5DVL3dzcsvRtjre3d4q+HpXaOqtJSUmKjIxUZGSkzp8/rwkTJujo0aN666235OPjk+kagJxCWEWwWogHAAAgAElEQVS63Nzc1LlzZy1YsEDbt29XZGSkzbp9W7dutc6rSl7uKi3Dhw/PsXVWT58+ne77fn5+6tatmzp37qxixYpl2F9a66xK0sWLFzVv3jytXbtWffv21aRJk9SxY8cs1Q0A9vD19ZX0cDpTfHy83NzcMnX+gwcPrK9Tu4mQ3jqrcXFx2rFjh6ZMmaJ//vOf2rVrl5YtW0ZgRa5hzioy1K1bNzk5OclsNmvdunU27yU/WFW5cuU0w96TsGfPHp06dSrF/3r27Cnp4YW6ZcuWdgXVjJQoUUKTJ09Wp06dlJSUpIkTJ2a4OgIAZMejD4RmZTeoR7dUzezDpe7u7mrVqpW+++47ubm56eTJk5o7d26mawByCmEVGSpatKgaNGgg6X/hVJLOnTunkJAQScZZrurdd99VYGCgEhISNHLkSCUkJORY38lBOCYmRv/9739zrF8AeFy1atWsr/ft25fp85MfqjKZTDYPW2VGiRIlVK9ePUn/WxEFyA2EVdglOYxevHjRGlCTl7Py8/NT69atc622R/n4+GjMmDGSpBMnTmj+/Pk51veja7beuHEjx/oFgMc999xz1mvO6tWrM3VuQkKCfvjhB0kPNzTx9/fPch3JNTx+zYuOjtadO3dSPSc+Pt76OrPTF4DUEFZhl4YNG1on42/evFkWi0WbN2+WJHXq1Enu7u65WZ6Nl19+WU2aNJEkzZkzR2fPns2Rfh+9WGfn4g8AGXF1dVXv3r0lSceOHctwretHLVy40Pow64ABA7JVx/Xr1yX975oXFxen5s2bq2bNmpoxY0aq5zy6rmt21rcGkhFWYRcnJyd17dpVkrRt2zbt27dP169fl8lkyvDBqtwwbtw4eXl5yWw2a9SoUVnaH/tx33//vaSHX6uxIgCAJ61Pnz7W9aEnTpyon3/+OcNztmzZolmzZkmSWrZsqcaNG2d5/LCwMO3du1eSrCvBuLu7KzAwUBaLRVu3bk11Wazt27dLevhgV/ny5bM8PpCMsAq7denSRW5ubrp9+7Y++eQTSVK9evVy5CGmnFa4cGG98847kqRDhw5p0aJFWe4rIiJCkyZNss7XtXeFAQDIDjc3N82ePVvFixdXbGys3nrrLY0ePVrHjh2zaZeUlKSDBw/q3Xff1bBhw2Q2m1W7dm1NmjQpS+MmJiZq165d6tu3r+Li4uTp6am33nrL+n7y3dqoqCi988471ruvSUlJmj9/vnVO/2uvvcYmKsgRJkta2xIBqfjggw9sFvafM2eOmjVrlmb7K1euWN8PDAy0WfbKHo8ulzJ79mx9+eWXkh6uBpBRX4mJiXr11Vd17NgxeXh4aMOGDSpRooT1/bVr12rkyJGSUl9nNSEhQbdu3VJUVJR1964OHTro448/Zh4WAIe5c+eOgoODtWXLFiUlJUl6OD8/f/781rWlY2JiJEmenp7q3bu3hgwZkmpQbNq0qcLDw1NdZ1WS7t+/r4iICOu80/z582vGjBmqXbu2TbuZM2daVwhwdnZWkSJFbOpo1qyZZs2aRVhFjuBfETKlR48e1rBauHBh69xQe2RmU4BkyRfmrHB2dtbEiRP1t7/9TbGxsRo9erSWLl2a6nawFy9eTHHMZDLJ29tb5cuXV/Xq1dW+fXubJ3QBwBH8/f01ffp0vfXWW9q6dav27NmjK1eu6Nq1a3JycpK/v79q1qypunXrql27dsqfP3+GfT548MC6w9+jXF1dlS9fPj377LNq3LixOnbsaLPBQLKhQ4eqTp06WrZsmQ4ePKirV6/Ky8tLtWvXVufOndWuXbtUN24BsoI7qwAAADAs/uwBAACAYRFWAQAAYFiEVQAAABgWYRUAAACGle5qALdjN6b3do6KSzI7bCxJuh8f47Cx3J1dHTaWJCU58Jk5b1ePjBvlEBeTs8PGcrQkZX3Vg6yITYjPuFEOcTY59m/iRIvjfpdFff6W6XMslitPoJLUJVqiHDaWJIVFH3bYWM94l3HYWJKUaHngsLFcTAUybpRDnE1+DhvL0eKTzjt0vESLI3NFCYeNJUlJlvsOG8vV6fkUx7izCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLJPFYrHkdhEAAABAarizCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMMirAIAAMCwCKsAAAAwLMIqAAAADIuwCgAAAMNyye0C8HSJiYnRhg0b9N///ldnzpzR7du3FRsbK3d3dwUEBKhUqVKqX7++OnfuLF9fX5tz9+7dq969e0uSvvjiC7Vs2dKuMatXr66YmBh17NhRU6ZMsXmvXLlyaZ7n6uoqHx8fFS9eXDVq1FCXLl1UunTpDMe7d++enn/++VTfW716tSpXrmxX3QCQWb169dK+ffskSV27dtXEiRMzPGft2rUaOXKkihQpop07d1qPjxgxQuvWrctSHadOnUrzPYvFol27dunnn3/W/v37FRERoaioKDk7O8vX19d6zW3durXKly+fpfGBRxFWYbfdu3frgw8+0O3btyU9DINBQUHy8PBQdHS0rl27pvDwcO3atUtz587VzJkz9eKLLzqkNn9/fxUsWNDmWExMjCIiIhQaGqrQ0FAtWrRIw4YN04ABA9Lty8nJyeYCGxUVpWvXrj2RugEgLatWrVK7du1Uq1atbPXj5OSksmXL5khNBw8e1Pjx43Xy5ElJkslkUr58+VSiRAnFxcXp2rVrioiIUEhIiObPn69WrVppwoQJypMnT46Mj78mwirscurUKb399tuKi4tTmTJl9MEHH6hevXpydXW1tnnw4IF++OEHTZkyRXfv3tXAgQO1adMmFSlS5InX16pVK40bNy7FcYvFoj179mjatGk6fvy4pk+frsqVK6cbon18fLRhwwbrz8l3LQDAUZycnJSUlKRx48Zp/fr1cnNzy3JfHh4eNte0rNq4caNGjRols9msggUL6q233lKLFi2UP39+axuz2WwNqr/++qu2bNmiixcv6l//+pfc3d2zXQP+mpizCrssXLhQcXFx8vLy0pIlS9S4cWOboCo9DHndunXT3LlzJUnR0dHW17nFZDKpbt26+vrrr+Xs7Czp4QUXAIysY8eO8vLy0rlz5zRv3rzcLkeHDx+2BtVatWpp8+bN6tGjh01QlR5+4/biiy9q4cKFevPNNyVJx48f18yZM3OjbPxJcGcVdjl+/LgkqUKFCgoICEi37YsvvqhBgwbJ29vbMPM7CxYsqCJFiujy5cuKiIjI7XIAIF1BQUEaOnSoJk2apHnz5qlVq1Z2zbl/UoKDg2U2m5U/f359+eWXKZ5JeJzJZNKwYcN07949+fn5qUmTJg6qFH9GhFXYJSkpSZIUGRlpV/vBgwc/yXKyJHmubeHChXO5EgDIWK9evbRp0yYdOXJEY8eO1bJly2QymRxex4EDB3To0CFJ0uuvvy5/f3+7zjOZTBo/fvwTrAx/FUwDgF2qVasmSbpw4YKmTJmi+Pj4XK4oc37++WdFR0dLktq2bZvL1QBAxpycnBQcHCwXFxft379fK1euzJU6du3aZX3N9RO5gbAKuwwcOFBeXl6SpG+//VaNGzdWcHCwdu3apXv37uVydamLj4/XxYsXNX/+fA0dOlSS9NZbb6W5LBUAGE358uXVp08fSdJnn32mmzdvOryGw4cPS5KeeeaZFHNUAUdgGgDsUrhwYa1YsUJDhw7VhQsXdPv2bS1ZskRLliyRyWRS6dKlVaNGDdWrV0/169eXj4+PQ+tbtmyZli1blub7VatW1bBhwxy2lBYA5JTBgwdr27ZtCgsLU3BwsGbNmpWp82NjY9W+fXu728+YMUOlSpWy/pw8/atQoUKZGhfIKYRV2K18+fLavHmztm3bpk2bNmnPnj36448/ZLFYdPbsWZ09e1arVq2Sp6en2rVrp8GDB6tAgQIOqS21dVaTkpIUERGhu3fv6tChQxo5cqR69Oihvn37pljJAACMysPDQxMmTFC/fv20bds27dixQ82aNbP7/KSkJOu6qPaIi4uz+Tl5CpWHh0e65w0fPjzdzQSGDBmSqbqBZIRVZIqzs7NatWqlVq1ayWw26+jRozpw4IBCQ0O1f/9+3b59W3/88YdWrlypnTt3asGCBQ7ZwSStdValh+u/bt68WdOmTdP06dO1d+9ezZs3Ty4u/PMH8HSoV6+e2rdvrw0bNmjixIl64YUX7P4Gy8vLSwcPHszy2MlP/j948CDddhcvXkw3FEdFRWW5Bvy1MWcVWebq6qrq1aurf//+mj17tnbv3q0VK1aoYcOGkqSIiAiNGjVKFotFkrL9FGtWz/fx8VHXrl2ta77+8ssvWrVqVbZqAQBHGzFihPz9/XX9+nV9/vnnDhs3MDBQknT58uV0261atUqnTp1K8T8guwiryDEmk0k1atTQ/Pnz1aJFC0nSsWPHdOLECUmyPqAlSX/88YddfZrNZsXExEiS/Pz8slXf888/b91ycOvWrdnqCwAcLSAgQCNGjJAkrVixQqGhoQ4Zt0aNGpIezl09d+6cQ8YEHkVYhd3sXa7KZDKpY8eO1p+vXbsmSTZzSq9cuWJXX+fPn7e+fuaZZ+w6Jz3Jc2hv3Lhhc/zevXu6f/9+quc8+rmzs+UhAGRXhw4dVLduXSUlJWns2LEym81PfMymTZtav9niWynkBsIqMvTPf/5TLVu2VN26dTOcs5QsNjbW+jp5AekCBQpY72zu2LHDrn4e3Rq1UaNG9pacpuvXr9vUdPXqVTVs2FDPP/+8li9fnuo5yWFbksMeGAOAtEyYMEEeHh46ffq0FixY8MTHK126tBo3bixJWr58uc6cOWP3uQkJCU+oKvyVEFaRoQIFCujChQu6f/++XUumJCYmWoOfn5+fzZarvXv3liSdOHFCK1asSLefQ4cOacmSJZKkNm3aZPvO6oEDB6xfYSUvYRUUFCRnZ2dJ0po1a2xCtvTwKdqdO3dKksqWLZvhVrMA8KQVK1ZMAwcOlCTNnTtXly5deuJjjh49Wnny5FF8fLzefPNNm2+90hIZGak333zT+rOTE5EDWcO/HGSoXbt2qlevniRp8eLFevvttxUSEmLdgjWZ2WzWr7/+ql69eun333+X9PCBgEeXierSpYt16ZKJEydqypQp1rudySIjIzV//ny99tpriouLU4kSJTRq1Kgs1x8fH68tW7ZYL+4FChRQr169JD2csvD6669Lki5duqThw4fr7t271vMmTZqk06dPS5L69euX5RoAICf169dP5cqVU1xcnBYuXPjExytatKhmzZolHx8fhYeHq1OnTpoxY4YuXrxo085isejo0aP67LPP9NJLL+mXX36RJL366qtq1arVE68Tf04mS/Kj2kA6YmJiNGHCBK1fv956zN3dXYUKFZKbm5vu3r2rO3fuWL/yyZs3r0aOHGkzdzVZfHy8Jk+erBUrVlhXCggMDFTevHn14MEDm6/d69evr0mTJqVYQzVZuXLlJKW+zqrFYlFUVJRu375tnddVrFgxffXVV9bpCMntRowYYf1srq6uKly4sK5du2adr9qzZ880l8YCgJzSq1cv7du3T4MGDdLgwYPTbXv48GF17drVeuOgSJEi1m+CpIc3C9atWycnJyeba5492rZta/1D/lHnzp3T+PHjtW/fPusxPz8/BQQEKDY2Vrdu3bKZ51+pUiW98847OTKNC39dLDQJu3h5eWnq1Knq37+/Nm/erP379+vixYu6fv26EhIS5OnpqcKFC6ts2bKqX7++WrdurTx58qTal5ubmz766CP16NFDa9asUUhIiC5fvqwLFy7I09NTZcuWVfXq1dW2bVu7t0a9c+eO7ty5k+K4u7u7AgMDVaFCBTVr1kxt2rRJ8ZCUyWTS1KlT1axZM61atUpHjx5VeHi4fH19VadOHfXo0UNNmjTJ/C8NAJ6gKlWqqGfPntbpUmnJ7KYAklS7du1Uj5cuXVpLlizR77//rp07d2rfvn26ceOGwsLCrDcwChUqpNq1a6tRo0aqUqVKpsYFUsOdVQAAABgWc1YBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhpbsagMUS5qg6lGi557CxJOlW7CmHjRXgkf1tQjMjLvGWw8bydC7msLGcTXkdNtZDJoeNFJeUuSd1sys2ITbjRjnE27WIw8aSJIue/PaTydycUn9iOj2JFsddexItdx02liSZHLjAjEWO3hnJcdcDR/4enUzeDhvL0Uxyduh4jswxjrzOOZqb0wspjnFnFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhmSwWiyW3iwAAAABSw51VAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWC65XQCeDuXKlUtxrHfv3ho9enQuVPPktW/fXidPnrQ5Vrt2bS1ZsiSXKgLwZ/JXu6ZK0okTJ9ShQ4cUxydPnqxOnTrlQkV4WhBWkSlBQUHKmzevJKlgwYKptrFYLNq5c6d+/vlnHThwQBEREXrw4IHc3NxUqFAhVaxYUU2aNFGLFi3k5uaWah9NmzZVeHi4GjdurHnz5mW53j179ujf//63jhw5ovDwcEVHR8vJyUk+Pj4qWrSoqlevro4dO6p8+fI255UsWdL6+tq1a4qKispyDQCQloyuqdeuXdPatWu1d+9eXbhwQVFRUUpISJCnp6cCAwNVvnx5vfTSS2rZsqWcnZ1TnL927VqNHDkyS7V99913euGFF6w/9+rVS/v27UuzvbOzs3x9fVW8eHHVqVNHXbp0UbFixazvu7u721xrH78hAKSFsIpMGTJkSLp/AYeEhGj8+PE6c+aMJMnJyUn58uVTyZIldf/+fZ0/f17nz5/XDz/8oMKFC+vjjz9W/fr1c7zOyMhIvf/++9q9e7f1WGBgoAIDA5WQkKAbN27o0KFDOnTokBYtWqTXXntNI0eOlMlkkiTNnDnTet6IESO0bt26HK8RANK7pv7zn//UF198IbPZLEny9vZWUFCQXF1ddffuXev1dMuWLZo/f77mzp2rwoULpzlWiRIl5OHhYXdtXl5eqR53d3e3+YM+mdlstrm2Lly4UMOHD1fv3r0lSaVKldKGDRus7VO7uwykhrCKHLNx40aNGjVKZrNZ+fPn1z/+8Q+1bNlS+fLls7a5ffu21q1bp6+//lpXr17VgAEDNG3aNLVu3TrH6rBYLBo0aJD2798vNzc3DR48WJ07d7apIykpSQcOHNCkSZN07NgxLV68WEFBQerbt2+O1QEAWbVq1Sp99tlnkqQmTZpo8ODBqlixovUPakm6fv26Fi5cqMWLF+vkyZMaOHCgVq9eneodVkn67LPPVLly5WzXVqxYMZvQ+SiLxaLQ0FBNmjRJhw8f1ieffKKgoCA1b9482+Pir4sHrJAjjhw5otGjR8tsNqtSpUratGmTevbsaRMQJSlfvnx6/fXXtXbtWgUGBioxMVGjR49WWFhYjtWyd+9e7d+/X9LDu6IDBgxIUYeTk5Nq1aqlJUuWqHTp0pKkr776SjExMTlWBwBk1dy5cyVJVapU0dy5c1WpUiWboCpJhQoV0qhRo/T2229Lko4fP67Nmzc7vNZHmUwmVa9eXYsWLVJQUJAkac6cOblaE55+3FlFjggODlZ8fLz8/Pz09ddfKyAgIN32xYoVU3BwsAYMGKAyZcro8uXLKlq0aI7UcuLECevrBg0apNvW29tbH374oQ4cOKBy5copKSkpR2oAgKy6e/eurl69KkmqV69eipD6uNdff133799XmTJlVKlSJUeUmCFvb2+9/PLLWrRokY4fP66YmJg0pxUAGSGsItsOHDig0NBQSVLfvn0VGBho13mNGjXSrl277G5vr8TEROvr27dv20zwT6uORo0a5WgNAJBVj/7RHBkZmWF7Hx8fjR079kmWlCX+/v7W19HR0YRVZBnTAJBtu3btsr7u2LFjps7N6aAqSdWqVbO+Hj9+fI5OMQCAJy0gIMD6R/batWu1adOmXK4oay5duiRJ8vDwsAmuQGYRVpFthw8fliQVKVIkzeWsHKlWrVpq2rSppIdLo7zyyit66623tG7dOuvFEwCM7IMPPpD08An7999/X61atdLcuXN18OBBxcfH53J1GQsLC9PWrVslSY0bN5aLC1/kIuv414NsS/6aqlChQrlcyf/MmDFDEyZM0Lp162Q2m/Xjjz/qxx9/lPTwq6kaNWqodu3aatKkiYoXL57L1QKArRYtWujzzz/X+PHjde/ePZ07d866pJ6rq6sqVaqkmjVrqmHDhqpVq5ZdYfD999+3e+mq/v37q127dpmq2Ww2KyIiQrt379asWbMUExMjX19fDRs2LFP9AI8jrCLboqOjJUmenp65XMn/eHh4aPLkyerbt69WrVqlnTt3Kjw8XJJ0584d7dixQzt27NDkyZNVq1Yt/eMf/1C9evVyuWoA+J/WrVurUaNGWr16tbZt26ZDhw4pMTFRZrNZoaGhCg0N1TfffKMCBQro73//u/r165fmRiuSdPHiRbvHTm+u7JkzZ+xaI7VEiRKaPn26SpQoYfe4QGoIq8g2Hx8fSdKDBw9yuZKUypYtqzFjxmjMmDG6fPmy9u/fr4MHD+rgwYM6c+aMLBaLQkJC1L9/f7377rsaMGBAbpcMAFY+Pj7q06eP+vTpowcPHlivYaGhoTp06JBiYmIUERGhGTNmaOfOnVq4cKH1mvy41atX58g6q2ltCpC8O2DJkiVVr149NWvWjK//kSP4V4RsCwwM1LFjxww/H7RYsWIqVqyY9SGwGzduaOXKlVqwYIHi4uI0c+ZM1a9fXxUrVszlSgEgJR8fH5vVS+Lj4/Xjjz9q5syZOn/+vA4dOqTZs2dneXtVe6W3KQDwJPCAFbKtatWqkh5+vX7q1KlcrsZ+BQsW1JAhQzRjxgxJD5e8Wrt2bS5XBQD2cXNz08svv6xVq1apSJEikqQ1a9bIYrHkcmVAziKsItsaN25sff2vf/0rU+cuWrRIGzZssFkbNSckJSXZ3WezZs3k5+cn6eH2hQBgBPY+9e/r66sWLVpIku7fv299jgD4syCsItsqVKigF154QZL0/fff6/jx43add/ToUU2bNk3Dhw/X559/niO1HDt2TN26dVONGjW0fv16u85JTEy0/keBtQAB5LZ3331XDRo0UP/+/e0+JzY2VtLDu63e3t5PqjQgVxBWkSPGjRsnDw8Pmc1mvf322zp37ly67U+cOKE33nhDCQkJKlSokN58880cqaN48eI6d+6c/vjjD3399de6c+dOhuesWbNGMTExkjLenhUAnjR3d3fdvHlT+/bt0//93/9l2P7mzZvWNU3t2Z4VeNoQVpEjypQpo2nTpsnd3V3Xr19X586d9eWXX1qXi0p248YNffrpp+rRo4ciIyOVJ08ezZ49W3ny5MmROnx8fDR69GhJ0uXLl9W5c2etWbNG9+/fT9H20qVLmjp1qsaPHy9JqlOnjpo3b54jdQBAVr377rvWDVbeeecdTZkyRefPn0/R7sGDB1q/fr26du2qO3fuyMvLS++//76jywWeOFYDQI5p0aKFlixZoo8++kgnTpzQ7NmzNXv2bOXLl0/+/v6Kjo7WtWvXrO1r1aqlSZMmpbsof0hIiNq3b5/h2N26dVP37t0lSR06dJDJZFJwcLDCw8M1atQojR49WgULFpSvr6/++OMPRUZGWu+mSlK7du00YcIE7kgAyHUFChTQ0qVL9f777+vQoUP69ttv9e2338rX11f58+eXk5OTbt++raioKOvDVKVKldLUqVNVpkyZNPvNzKYAklSyZEnrRgRAbiKsIkdVrVpV69ats+4YdeDAAd26dUsXLlyQp6enypcvr2rVqql169aqXbt2hv09ePBAJ0+ezLDdrVu3bH5u3769mjVrpk2bNmn37t06c+aMbt68qZs3b8rNzU2+vr6qVKmSqlevrjZt2ti1wDUAOEqxYsW0atUq/fLLL9q+fbuOHj2qK1eu6MqVK7JYLPL29lbZsmVVoUIFvfTSS2rSpEmGa5pmZlMAwEgIq8hxJpNJTZs2VdOmTbPcx86dO7Ndh4+Pj7p372694woAT5v69eurfv36WT6/U6dO6tSpU47UsmTJkhzpB8gs5qwCAADAsAirAAAAMCzCKgAAAAyLOavIlLW7lnMAAAxsSURBVFmzZmnx4sWSpLZt2+r111/P5YqejKFDh+rChQuSZLOCAQDkpL/KNVWSzp8/r2HDhuV2GXgKZRBWrzqmilxgsZgdNpbJ5OqwsZ60a9euWcNb7dqV9Gf9N3LhwmmdPPn4xgZxelKfN8kS90T6TYtJjvs3aTI59gsciyXJYWOZTM9k+hzPofWeQCWpa/XScw4bS5K+bz3cYWON+PWfDhtLkpoVy/z/1/Z49JpasKK7iobF6quDoU9krNT8+FPGq63kFJeoBAWeTLlRy5srPlGvXdNzfLxfxzg2+Jf0DXTYWG/t3OywsSSpeF4/h401+cVJKY5xZxV2OXXqx9wuwaE2bFiQ2yUA+BOb9X+Dc7sEh0vI66IrrfLndhl4CjFnFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZhFQAAAIZFWAUAAIBhEVYBAABgWIRVAAAAGBZh9f/buf+YqOs/DuBPlCOaoDEwTFq13MAK1DtAcChDjen0rNQiJRStDX8MFI6VVlP8x4jS0vQSybacOcBSQiHgn7bUjeqw5fjh3ZKc60AQDEJ2ykG8+8PBVyzr63i/7z5+ej42/uD9x/t5B/s8ed37PgcRERERaRaHVSIiIiLSLA6rRERERKRZHFaJiIiISLM4rBIRERGRZnFYJSIiIiLN4rBKRERERJrFYZWIiIiINIvDKhERERFpFodVIiIiItIsDqtEREREpFkcVomIiIhIszisEhEREZFmcVglIiIiIs3isEpEREREmsVhlYiIiIg0i8MqEREREWkWh1UiIiIi0iwfIYTw9oMgIiIiIvo7PFklIiIiIs3isEpEREREmsVhlYiIiIg0i8MqEREREWkWh1UiIiIi0iwOq0RERESkWRxWiYiIiEizpA6rDocDZrMZ8+bNk7nt32ppaUFWVhbi4+MRHx+PzZs3o729XUnWTz/9hLS0NJhMJiQkJMBisaCjo0NJ1u3eeecdREREKM2IiIhAZGQkoqKihr/y8vKU5X366adITEzEjBkzkJqaiosXLyrJsdlsI57T0FdERARaWlqk5124cAHp6emIjY3FrFmzsGnTJrS2tkrPAYCGhgakp6cjJiYGs2fPRmFhodT973Yd//DDD0hJSYHJZMLChQtRXFysNA8AampqMHPmTGzdulVpls1mw4oVK2AymZCUlIT33nsPAwMDUjJHQ6+dCui3Vz3dqYBnelXPnQroq1d12alCksrKSjF79myxceNGMXfuXFnb3pXZbBa5ubni+vXrorOzU6xevVpkZGRIz+nu7hZGo1F89tlnwu12i87OTpGWliY2bNggPet2TU1NYubMmSI8PFxpTnh4uPjuu++UZgwpLi4WycnJwuFwiN7eXrF7926Rm5vrkeyh/JdfflkMDg5K3be/v18kJCSI999/X/T19Ymenh6RlZUlVq5cKTVHCCG6urpEXFycKCgoEC6XS1y8eFEkJyeLsrIyKfvf7Tq+evWqMBqN4ujRo+LGjRvi3LlzwmQyiW+//VZJnhBC7Ny5U5jNZrF8+XKxZcuWUeX8U1ZLS4uYMWOGOHz4sHC73cJut4uEhARx6NChUWeOhl47VQh996onO1UI7/aqHjpVCH31ql47VdrJqsvlQmlpKWbNmiVry7vq6elBZGQkXn/9dQQEBCA4OBgpKSmw2WzSs9xuN95++22kp6fDYDAgODgYycnJsNvt0rOGDA4OIi8vD2vXrlWW4Q2ffPIJNm/ejPDwcIwbNw4WiwW7du3ySPZvv/2GvXv3Ii8vDz4+PlL3vnLlCjo6OrB06VL4+fkhMDAQixYtwoULF6TmALdOo3p7e5GTk4MHH3wQU6ZMQUZGBkpLS6Xsf7fr+OTJkwgLC0Nqair8/f1hMpnw/PPPo6SkREkeAISEhOCLL77A5MmTR5Xxb1mdnZ1YtmwZVq9eDYPBgIiICMybN09Jn9wLvXYqwF6VyVu9qpdOBfTVq3rtVGnD6osvvijtB/Bvxo8fj/z8fISGhg6vXblyZcT3skycOBHLly8HAAgh0NzcjLKyMixevFh61pCSkhL4+/vDbDYry7jd4cOHMX/+fERHR+ONN95AT0+P9Iz29nY4nU64XC4sWbIEsbGxWLduHdra2qRn/R2r1Yq5c+fiqaeekr53WFgYpk6dipKSEvT29qK3txdff/21R966BYAJEyZI+yN/t+u4sbERzzzzzIi1p59+GvX19UryACAjIwP+/v6j2v//yZo2bRq2bds2Yq2trU1Jn9wLvXYqoP9e9USnAt7tVT13KnD/9qpeO1UXH7D65ZdfcODAAWzcuFFZht1uR2RkJMxmM6KiopCdna0kp7OzE1arFTt27FCy/52mT5+OmJgYVFZW4sSJE3A4HNi+fbv0nKHyrKioQFFREaqqqtDf3w+LxSI9607t7e04ceIE1q9fr2T/MWPGYP/+/fjmm28QHR2N6OhotLa2KrlPzWg0Yty4cfjwww9x48YNOJ1OfP7553C5XHC73dLzhnR3d2P8+PEj1h566CF0dXUpy/SWiooK2Gw23Z3A3QtPdCqgz171VKcC3utVPXUqwF5VTUan3vfDan19PdLS0rB27VosWbJEWc7UqVPR0NCAiooKXLp0SVkZ5Ofn46WXXsKTTz6pZP87HTt2DK+++ir8/f3x+OOPw2KxoLq6Gjdv3pSaI4QAALz22mt45JFHEBISAovFgnPnzik/BThy5AjmzJmDxx57TMn+brcb69evx4IFC1BXV4fTp0/j4YcfRm5urvSsCRMmwGq1oq6uDgkJCcjOzsYLL7wAABg7dqz0vNsN/Q717Pjx49i+fTs++ugjPPHEE95+OF7hqU4F9NmrnupUwHu9qqdOBdirKsnq1Pt6WD1z5gzWrFmDzMxMZGZmKs/z8fHBlClThstH9idXa2trUV9fjw0bNkjd9148+uijEEJIf24hISEAbr1qHBIWFgYAuHr1qtSsO1VVVeHZZ59Vtn9tbS0uX76MnJwcBAYGIjQ0FJs2bcLp06dx7do16XkxMTE4duwYfvzxR3z55ZcICgrCxIkTlZZqUFAQuru7R6x1dXUhODhYWaanffzxx9i1axcOHTqEOXPmePvheIWnOxXQf6+q6lTAe72qt04F2KsqyOzU+3ZYPX/+PHJyclBQUIDU1FRlOVVVVVi2bNmItTFjbv3YfH19pWadPHkS7e3tSExMRFxc3HBuXFwcKisrpWYBQFNTE959990Ra83NzTAYDJg0aZLUrEmTJiEwMBBNTU3Da06nEwCU3pdnt9vhdDqRmJioLOOPP/74y6tjVf/2qK+vD1999RWuX78+vHb27FlER0cryRsSFRWFhoaGEWv19fWYPn260lxPOXLkCEpKSlBcXAyTyeTth+MVnupUQL+96slOBbzTq3rrVIC9qoLsTr0vh9WBgQG89dZbyMrKUvrqDgBMJhMuX74Mq9WKmzdv4tq1a9i3bx9MJhOCgoKkZm3duhU1NTUoLy9HeXk5ioqKAADl5eVKbiwPDg5GaWkpioqK4Ha7cenSJezduxcpKSkwGAxSs3x9fbFy5UoUFhaiubkZv//+O/bs2YOkpKTh0wEVGhsbERgYOOLkQTaj0YiAgADs2bMHLpcLXV1dKCwshNFolP4K2WAwYN++fbBarRgYGEBtbS2OHz+ONWvWSM2503PPPYeOjg4cPXoUfX19+P7773Hq1CmsWrVKaa4n/Prrr/jggw9w4MCB/+xb/57sVEC/verJTgW806t661SAvSqbik71EZJumFiwYAFaW1sxODiIgYEB+Pn5AQCqq6uH35aQpa6uDq+88spwxu1U5J0/fx75+floampCQEAA4uPjsWXLFuWfFnY6nZg/fz4cDoeyDJvNht27d8PhcMDPzw9Lly5FTk4OHnjgAelZ/f39KCgowKlTp9DX14ekpCTs2LFDaekdPHgQZWVlqK6uVpYB3PqH0gUFBbDb7TAYDIiNjcWbb76p5DSlsbER27ZtQ3NzM0JDQ5GdnY1FixZJ2fufruO2tjbs3LkTP//8M0JDQ5GZmTl8X5eKvIULFwL434mKr68vJk+ejJqaGqlZ69atw/79+/8yTIwmSwY9dyqg3171ZKcCnu9VPXYqoJ9e1WunShtWiYiIiIhkuy9vAyAiIiKi/wYOq0RERESkWRxWiYiIiEizOKwSERERkWZxWCUiIiIizeKwSkRERESaxWGViIiIiDSLwyoRERERaRaHVSIiIiLSrD8Bt4CrUbsH9moAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "data_qqp = cls_last_layer(\"QQP\")\n",
+ "visualize_cls(data_qqp, \"QQP\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## 7. BaseBERT study"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 72,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def exceeds_token_limit(span, n_max=3):\n",
+ " tokens = word_tokenize(span)\n",
+ " return len(tokens) > n_max"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 73,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def is_close(sentence, target, relations):\n",
+ " tokens = word_tokenize(sentence)\n",
+ " \n",
+ " target_token = sentence[target[0]:target[1]]\n",
+ " try:\n",
+ " target_token_idx = tokens.index(target_token)\n",
+ " except Exception:\n",
+ " return True\n",
+ " \n",
+ " relation_spans = [sentence[relation[0]:relation[1]] for relation in relations]\n",
+ " relation_tokens = list(itertools.chain.from_iterable([word_tokenize(span) for span in relation_spans]))\n",
+ "\n",
+ " # make sure we don't go beyond list bounds\n",
+ " padded_tokens = ['', ''] + tokens + ['', '']\n",
+ " for token in padded_tokens[target_token_idx: target_token_idx + 5]:\n",
+ " if token in relation_tokens:\n",
+ " return True \n",
+ " return False"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 74,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 13572/13572 [05:06<00:00, 44.35it/s]\n"
+ ]
+ }
+ ],
+ "source": [
+ "annotations = []\n",
+ "for lu in tqdm(fn.lus()):\n",
+ " core_list = list(itertools.chain.from_iterable([[fe['name'] for fe in item] for item in lu['frame']['FEcoreSets']]))\n",
+ " corpora = lu['subCorpus']\n",
+ " for corpus in corpora:\n",
+ " sentences = corpus['sentence']\n",
+ " for sentence in sentences:\n",
+ " text = sentence['text']\n",
+ " \n",
+ " tokens = word_tokenize(text)\n",
+ " if len(tokens) > 12:\n",
+ " continue\n",
+ " \n",
+ " try:\n",
+ " target = sentence['Target']\n",
+ " except Exception:\n",
+ " continue\n",
+ " \n",
+ " # if target word appears many times\n",
+ " if len(target) > 1:\n",
+ " continue\n",
+ " \n",
+ " # filter cases with more than 3 tokens\n",
+ " fes = sentence['FE'][0]\n",
+ " flag = True\n",
+ " for span in fes:\n",
+ " if exceeds_token_limit(text[span[0]:span[1]]):\n",
+ " flag = False\n",
+ " if not flag:\n",
+ " continue\n",
+ " \n",
+ " # filter relations too close to the target\n",
+ " target_ind = target[0]\n",
+ " if is_close(text, target_ind, fes):\n",
+ " continue\n",
+ " \n",
+ " core_fes = [(fe[0], fe[1]) for fe in fes if fe[2] in core_list]\n",
+ " \n",
+ " # filter no core fes\n",
+ " if not core_fes:\n",
+ " continue\n",
+ " annotations.append({\"sentence\": text,\n",
+ " \"target\": target[0],\n",
+ " \"links\": core_fes})"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 56,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "annotations_reformatted = []\n",
+ "for annotation in annotations:\n",
+ " sentence = annotation['sentence']\n",
+ " target = sentence[annotation['target'][0]:annotation['target'][1]]\n",
+ " links = []\n",
+ " for link in annotation['links']:\n",
+ " links.append(sentence[link[0]:link[1]])\n",
+ " annotations_reformatted.append({'sentence': sentence, 'target': target, 'links':links})"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 57,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Mitch 's chin dipped an inch and then lifted again . lifted [\"Mitch 's chin\"] \n",
+ "\n",
+ "His head sags , nods once , jerks up again . nods ['His head'] \n",
+ "\n",
+ "He guessed that a reaction was expected and shrugged . shrugged ['He'] \n",
+ "\n",
+ "He sat up and stretched , then stripped off his gloves . stretched ['He'] \n",
+ "\n",
+ "Caterina shut her eyes and twitched . twitched ['Caterina'] \n",
+ "\n",
+ "This is a small island and tongues are beginning to wag . wag ['tongues'] \n",
+ "\n",
+ "Peach tried and tried , but the toes would n't wiggle . wiggle ['the toes'] \n",
+ "\n",
+ "He caught Carol 's eye , and winked . winked ['He'] \n",
+ "\n",
+ "He tried to wink but , naturally , failed . wink ['He'] \n",
+ "\n",
+ "There are some rolls a film star just has to grab … grab ['a film star'] \n",
+ "\n",
+ "'' I do n't know . know ['I'] \n",
+ "\n",
+ "( Do n't say Stephanopoulos did n't know . know ['Stephanopoulos'] \n",
+ "\n",
+ "I 'm not given to making guesses . \" guesses ['I'] \n",
+ "\n",
+ "He made a guess . guess ['He'] \n",
+ "\n",
+ "For the present we need to proceed by inferences . inferences ['we'] \n",
+ "\n",
+ "I ca n't learn again now . learn ['I'] \n",
+ "\n",
+ "Only I do want to learn . learn ['I'] \n",
+ "\n",
+ "An improvement is not expected until next summer . expected ['An improvement'] \n",
+ "\n",
+ "The discovery should have been predicted a decade earlier by Einstein . predicted ['The discovery'] \n",
+ "\n",
+ "Yet his comments were curiously approving . approving ['his comments'] \n",
+ "\n",
+ "In profile the man appeared haughty , contemptuous almost . contemptuous ['the man'] \n",
+ "\n",
+ "Annie tried not to sound reproachful . reproachful ['Annie'] \n",
+ "\n",
+ "Rory tried to look reproachful even through her laughter . reproachful ['Rory'] \n",
+ "\n",
+ "But , there again , actresses were known to be broad-minded . broad-minded ['actresses'] \n",
+ "\n",
+ "Surere suddenly looked crafty . crafty ['Surere'] \n",
+ "\n",
+ "We could see right through your crafty but loose Propaganda . crafty ['Propaganda'] \n",
+ "\n",
+ "But here Microsoft has been cunning . cunning ['Microsoft'] \n",
+ "\n",
+ "You 'll have to be cunning , devious , smart . cunning ['You'] \n",
+ "\n",
+ "Sarin is both cynical and shrewd . \" cynical ['Sarin'] \n",
+ "\n",
+ "` You are being cynical and evasive . \" cynical ['You'] \n",
+ "\n",
+ "That seemed to me to be daft , bordering on snobbery . daft ['That'] \n",
+ "\n",
+ "He was shrewd , diligent , methodical , and impartial . diligent ['He'] \n",
+ "\n",
+ "` Robin-Anne 's not dim , \" I said . dim ['Robin-Anne'] \n",
+ "\n",
+ "The audience grew restless and inattentive . inattentive ['audience'] \n",
+ "\n",
+ "He 'd always believed that humans were quite intelligent . intelligent ['humans'] \n",
+ "\n",
+ "` I think our guest is very intelligent . \" intelligent ['our guest'] \n",
+ "\n",
+ "I said that this was becoming ludicrous . ludicrous ['this'] \n",
+ "\n",
+ "He may be just naive and well meaning . naive ['He'] \n",
+ "\n",
+ "He was too narrow-minded and prejudiced -- and bigoted . narrow-minded ['He'] \n",
+ "\n",
+ "His sister was at once more realistic and more perceptive . perceptive ['His sister'] \n",
+ "\n",
+ "That would be ridiculous at my age anyway . ridiculous ['That'] \n",
+ "\n",
+ "They were often very sage and experienced . \" sage ['They'] \n",
+ "\n",
+ "He 's supposed to be sensible . \" sensible ['He'] \n",
+ "\n",
+ "You 're way too smart to be driving a truck . smart ['You'] \n",
+ "\n",
+ "Right I 'll just see who 's looking smart to start . smart ['who'] \n",
+ "\n",
+ "` People are too smart now . ` smart ['People'] \n",
+ "\n",
+ "He 's supposed to be smart . smart ['He'] \n",
+ "\n",
+ "oh really he must be smart yeah smart ['he'] \n",
+ "\n",
+ "SHe decided to be blunt . blunt ['SHe'] \n",
+ "\n",
+ "This blunt and simple question was the crux . blunt ['question'] \n",
+ "\n",
+ "They are competitive , bright , candid , and dedicated . candid ['They'] \n",
+ "\n",
+ "` I 'm not being coy , \" she protested . coy ['I'] \n",
+ "\n",
+ "Buckingham Palace had promised his remarks would be discreet . discreet ['his remarks'] \n",
+ "\n",
+ "This perspective is now being made more explicit amongst linguists . explicit ['This perspective'] \n",
+ "\n",
+ "The Genn study is even more forthright : forthright ['The Genn study'] \n",
+ "\n",
+ "` Those involved have needed to become secretive , for protection . secretive ['Those involved'] \n",
+ "\n",
+ "And he had promised Frances that he would always be truthful . truthful ['he'] \n",
+ "\n",
+ "A pleasingly curt , truthful and enigmatic reply . truthful ['reply'] \n",
+ "\n",
+ "Environmental assessments are important and they should be truthful and comprehensive . truthful ['they'] \n",
+ "\n",
+ "The girls were required to take the Official Secrets oath . oath ['The girls'] \n",
+ "\n",
+ "But , remember , I ca n't promise yet . \" promise ['I'] \n",
+ "\n",
+ "She certainly made threats against Angela and has bothered her recently . threats ['She'] \n",
+ "\n",
+ "Such detail will not be volunteered as a general rule . volunteered ['Such detail'] \n",
+ "\n",
+ "And let 's make a vow . vow [\"'s\"] \n",
+ "\n",
+ "… Everybody was talking and gesticulating at the same time . ” gesticulating ['Everybody'] \n",
+ "\n",
+ "Stephanie lowered the book and murmured . murmured ['Stephanie'] \n",
+ "\n",
+ "` I must n't keep rattling on like this ! \" rattling ['I'] \n",
+ "\n",
+ "Vi took a step forward , assuming command . command ['Vi'] \n",
+ "\n",
+ "anything else you wanted to add add ['anything else', 'you'] \n",
+ "\n",
+ "Defence counsel stated that Adams had made a full admission . admission ['Adams'] \n",
+ "\n",
+ "On November 5 her pregnancy is officially announced . announced ['her pregnancy'] \n",
+ "\n",
+ "Obviously something important was about to be announced . announced ['something important'] \n",
+ "\n",
+ "War had been declared . declared ['War'] \n",
+ "\n",
+ "They issued a denial through their solicitor which reads : denial ['They'] \n",
+ "\n",
+ "She gave an exclamation and hurried forward . exclamation ['She'] \n",
+ "\n",
+ "She 's come to gloat , he thought . gloat ['She'] \n",
+ "\n",
+ "`` Rabin today made this proposal . proposal ['Rabin'] \n",
+ "\n",
+ "Alternative approaches were subsequently proposed . proposed ['Alternative approaches'] \n",
+ "\n",
+ "I saw report something the other night about it . report ['about it'] \n",
+ "\n",
+ "You used to say . say ['You'] \n",
+ "\n",
+ "I felt stimulated -- mentally alert , talkative and sociable . talkative ['I'] \n",
+ "\n",
+ "He 'll be cross , but I 'll go . cross ['He'] \n",
+ "\n",
+ "There was really no reason why Star should be cross . cross ['Star'] \n",
+ "\n",
+ "Her mother would be cross and he would feel stupid . cross ['Her mother'] \n",
+ "\n",
+ "He should n't be disappointed . disappointed ['He'] \n",
+ "\n",
+ "\" Debbie would be pleased , \" she said . pleased ['Debbie'] \n",
+ "\n",
+ "She dreams of being adored , the centre of attention . adored ['She'] \n",
+ "\n",
+ "He is therefore to be hated and despised . despised ['He'] \n",
+ "\n",
+ "No wonder he was so universally detested . detested ['he'] \n",
+ "\n",
+ "He was a man to be envied . envied ['a man'] \n",
+ "\n",
+ "This led to skinheads being both hated and feared . hated ['skinheads'] \n",
+ "\n",
+ "Within journalism , MacKenzie is either loved or loathed . loathed ['MacKenzie'] \n",
+ "\n",
+ "She loved the little house and she loved Thrush Green . loved ['she', 'Thrush Green'] \n",
+ "\n",
+ "They will not be mourned . mourned ['They'] \n",
+ "\n",
+ "Lucy is much to be pitied . pitied ['Lucy'] \n",
+ "\n",
+ "Such people are to be pitied . pitied ['Such people'] \n",
+ "\n",
+ "Later , Ronni lay and seethed in her bed . seethed ['Ronni'] \n",
+ "\n",
+ "I 'm very positive that Natalie will be cured . cured ['Natalie'] \n",
+ "\n",
+ "` Nothing that your absence would n't cure . cure ['your absence'] \n",
+ "\n",
+ "When treated early , most people recover . treated ['most people'] \n",
+ "\n",
+ "Stephanie said that he had been seriously ill and was convalescing . convalescing ['he'] \n",
+ "\n",
+ "It was a treat to be savoured … savoured ['a treat'] \n",
+ "\n",
+ "I , I had a little taste . taste ['I'] \n",
+ "\n",
+ "Robert 's eyes were starring to ache . ache [\"Robert 's eyes\"] \n",
+ "\n",
+ "It had n't hurt before . hurt ['It'] \n",
+ "\n",
+ "Her foot began to tingle and she shifted her position . tingle ['Her foot'] \n",
+ "\n",
+ "Her skin seemed to be tingling . tingling ['Her skin'] \n",
+ "\n",
+ "He served there for many years , ultimately becoming chairman . chairman ['He'] \n",
+ "\n",
+ "This use may become authoritarian and overpowering for subordinates . authoritarian ['for subordinates'] \n",
+ "\n",
+ "Sartori had been indulgent for the weeks they 'd been together . indulgent ['Sartori'] \n",
+ "\n",
+ "Father Duryea appeared to be hopeful and indulgent . indulgent ['Father Duryea'] \n",
+ "\n",
+ "He later described his three-month suspension as too lenient . lenient ['three-month suspension'] \n",
+ "\n",
+ "In the mid 1970s Soviet emigration policies became more lenient . lenient ['Soviet emigration policies'] \n",
+ "\n",
+ "TODAY believes the sentence is too lenient . lenient ['the sentence'] \n",
+ "\n",
+ "Regulators should resist pressure to be lenient lenient ['Regulators'] \n",
+ "\n",
+ "Communicative acts are multiplied endlessly but could become fleeting and ephemeral . ephemeral ['Communicative acts'] \n",
+ "\n",
+ "I want a poem to be monumental and eternal . eternal ['a poem'] \n",
+ "\n",
+ "Probably human cruelty is fixed and eternal . eternal ['Probably human cruelty'] \n",
+ "\n",
+ "Reloading was always lengthy . lengthy ['Reloading'] \n",
+ "\n",
+ "That means lengthy and usually uncomfortable treatment . lengthy ['treatment'] \n",
+ "\n",
+ "The songs are mostly short and flash . short ['The songs'] \n",
+ "\n",
+ "We 're going to be late . late ['We'] \n",
+ "\n",
+ "His examination results were now overdue . overdue ['His examination results'] \n",
+ "\n",
+ "The firing is both preceded and followed by blatant cronyism . preceded ['by blatant cronyism'] \n",
+ "\n",
+ "You are wonderfully punctual . punctual ['You'] \n",
+ "\n",
+ "Renwick was always punctual . punctual ['Renwick'] \n",
+ "\n",
+ "Idealists could claim that British policy had been tardy and hesitant . tardy ['British policy'] \n",
+ "\n",
+ "Eating out is convenient and economical . economical ['Eating out'] \n",
+ "\n",
+ "` She was rather frugal , \" admitted the rector . frugal ['She'] \n",
+ "\n",
+ "Their Christmas had been too frugal , he decided . frugal ['Their Christmas'] \n",
+ "\n",
+ "There are many other ways of being thrifty . thrifty ['many other ways'] \n",
+ "\n",
+ "She had been brought up to be thrifty and careful . thrifty ['She'] \n",
+ "\n",
+ "He or she is urban , trendy and affluent . affluent ['He or she'] \n",
+ "\n",
+ "As they bicker , Panama is flat broke . broke ['Panama'] \n",
+ "\n",
+ "` I 'm going to get really , really rich . \" rich ['I'] \n",
+ "\n",
+ "The man wants to get rich . rich ['The man'] \n",
+ "\n",
+ "He could grow immensely rich . rich ['He'] \n",
+ "\n",
+ "Interest rates have been cautiously lowered . lowered ['Interest rates'] \n",
+ "\n",
+ "His cigarette gIowed and faded and glowed again . glowed ['His cigarette'] \n",
+ "\n",
+ "The cottage had not been inhabited for a while . inhabited ['The cottage'] \n",
+ "\n",
+ "a state where she has never even lived ? lived ['a state', 'where'] \n",
+ "\n",
+ "I 'll beat you , just as my father beat me . beat ['my father', 'me'] \n",
+ "\n",
+ "Ms Guy worked as a part-time teacher from 1969 to 1977 . teacher ['Ms Guy'] \n",
+ "\n",
+ "\" Something has been killed and something has been burned . burned ['something'] \n",
+ "\n",
+ "Remember her , the schoolmistress ? \" schoolmistress ['her'] \n",
+ "\n",
+ "John Knee is a quietly-spoken 40-year-old schoolteacher from Malvern in Worcestershire . schoolteacher ['John Knee'] \n",
+ "\n",
+ "You are a strait-laced schoolteacher . schoolteacher ['You'] \n",
+ "\n",
+ "One was naked , and had been speared . speared ['One'] \n",
+ "\n",
+ "She was to be ritually speared for having broken a taboo . speared ['She'] \n",
+ "\n",
+ "But new equipment will have to comply straight away . comply ['new equipment'] \n",
+ "\n",
+ "Maybe he 'd been mauled . mauled ['he'] \n",
+ "\n",
+ "Advanced weapon techniques 1 : the attacker attempts to strike . strike ['the attacker'] \n",
+ "\n",
+ "I still do n't understand why she was all bashed up . bashed ['she'] \n",
+ "\n",
+ "Four children were scolded on buses and smacked . smacked ['Four children'] \n",
+ "\n",
+ "` You probably were n't smacked enough as a child . smacked ['You'] \n",
+ "\n",
+ "These problems do not arise if the referee is an expert . expert ['the referee'] \n",
+ "\n",
+ "We have the expertise to help you meet your ideal partner . expertise ['We'] \n",
+ "\n",
+ "They 're more experienced . \" experienced ['They'] \n",
+ "\n",
+ "` He 's not a connoisseur like you . \" connoisseur ['He'] \n",
+ "\n",
+ "\" We were all very inexperienced when the situation arose . inexperienced ['We'] \n",
+ "\n",
+ "` In those days I was young and relatively inexperienced . inexperienced ['I'] \n",
+ "\n",
+ "We must be confident . confident ['We'] \n",
+ "\n",
+ "I 'm not convinced . convinced ['I'] \n",
+ "\n",
+ "Her military prowess should not have been doubted . doubted ['Her military prowess'] \n",
+ "\n",
+ "The pistols lay in a row , black and gleaming . gleaming ['The pistols'] \n",
+ "\n",
+ "Huge , blue-black , shining , nightmare lobster . shining ['lobster'] \n",
+ "\n",
+ "When she was still unsure he patted her shoulder . unsure ['she'] \n",
+ "\n",
+ "` I 've got a few doubts , sir . doubts ['I'] \n",
+ "\n",
+ "Even the Belgians have their doubts . doubts ['the Belgians'] \n",
+ "\n",
+ "His behaviour became anti-social or unpredictable at times . unpredictable ['His behaviour'] \n",
+ "\n",
+ "The second is that investment is volatile and unpredictable . unpredictable ['investment'] \n",
+ "\n",
+ "She had been completely naked . naked ['She'] \n",
+ "\n",
+ "You will enter naked . naked ['You'] \n",
+ "\n",
+ "Neither man is completely nude . nude ['Neither man'] \n",
+ "\n",
+ "` He was found dressed as some kind of animal . \" dressed ['He'] \n",
+ "\n",
+ "I 'm not walking barefoot . \" barefoot ['I'] \n",
+ "\n",
+ "The old man was wearing a plastic raincoat but was hatless . hatless ['The old man'] \n",
+ "\n",
+ "The inquiry recommended 10 officials should be disciplined . disciplined ['10 officials'] \n",
+ "\n",
+ "She took the punishments meted out to her without a murmur . punishments ['to her'] \n",
+ "\n",
+ "Inevitably they would be in an arrestable state of undress . undress ['they'] \n",
+ "\n",
+ "My belly is writhing and wriggling ! wriggling ['My belly'] \n",
+ "\n",
+ "He was , in fact , a spendthrift . spendthrift ['He'] \n",
+ "\n",
+ "The station was never built . built ['The station'] \n",
+ "\n",
+ "It was a fiasco . fiasco ['It'] \n",
+ "\n",
+ "` It was a complete fiasco , \" said his lawyer . fiasco ['It'] \n",
+ "\n",
+ "His uniform would be ruined . ruined ['His uniform'] \n",
+ "\n",
+ "The flesh was tender but not particularly flavoursome . flavoursome ['The flesh'] \n",
+ "\n",
+ "Caledor was famously brusque and curt . curt ['Caledor'] \n",
+ "\n",
+ "But she was not silenced for long . silenced ['she'] \n",
+ "\n",
+ "It tickled , and the patient had to be shushed . shushed ['the patient'] \n",
+ "\n",
+ "Kiah phoned Adai and Rilla phoned Commander Zadak in Australia . phoned ['Commander Zadak', 'in Australia'] \n",
+ "\n",
+ "The wound was then cleansed and redressed aseptically . cleansed ['The wound'] \n",
+ "\n",
+ "He 's been sweating again . sweating ['He'] \n",
+ "\n",
+ "He 's going to spew . \" spew ['He'] \n",
+ "\n",
+ "I turned and retched . retched ['I'] \n",
+ "\n",
+ "The Circumcision and Presentation of Jesus ( Luke 2:21-40 ) Circumcision ['of Jesus'] \n",
+ "\n",
+ "She was pronounced dead on arrival at Brighton General . \" dead ['She'] \n",
+ "\n",
+ "One suspect , having survived the shooting , was burned alive . alive ['One suspect'] \n",
+ "\n",
+ "It 's Anya the Apostate , irate and undead . undead ['Anya the Apostate'] \n",
+ "\n",
+ "Morenz was taller , burly , grey-haired . grey-haired ['Morenz'] \n",
+ "\n",
+ "One thing is for sure : they are not bearded . bearded ['they'] \n",
+ "\n",
+ "The argument is both dangerous and disingenuous . disingenuous ['The argument'] \n",
+ "\n",
+ "You 're being evasive , I can feel it . \" evasive ['You'] \n",
+ "\n",
+ "So in July 1982 I set off on a similar journey . journey ['I'] \n",
+ "\n",
+ "The food these children get is tasteless and un-nutritous . tasteless ['The food'] \n",
+ "\n",
+ "` We should never have toured , for one thing . \" toured ['We'] \n",
+ "\n",
+ "Counsellors have to be impartial . impartial ['Counsellors'] \n",
+ "\n",
+ "It tasted sweet and sharp and sensational . sensational ['It'] \n",
+ "\n",
+ "She smiled and chewed and chewed and smiled chewed ['She'] \n",
+ "\n",
+ "Mrs Snell bought clothes there and she always looked terrible . terrible ['she'] \n",
+ "\n",
+ "People have been tremendous with sponsorship . tremendous ['People'] \n",
+ "\n",
+ "The 2 pain studies are of fair to poor quality . fair ['quality'] \n",
+ "\n",
+ "In 1937 anything British was suspect , second-rate , cheapo . second-rate ['anything British'] \n",
+ "\n",
+ "You may feel that the collection is very incomplete and second-rate . second-rate ['the collection'] \n",
+ "\n",
+ "Hohnfeldt 's leadership left much to be desired . desired ['much'] \n",
+ "\n",
+ "Why was he insisting that her behaviour had been dishonourable ? dishonourable ['her behaviour'] \n",
+ "\n",
+ "` You 're a vile , stupid woman . vile ['woman'] \n",
+ "\n",
+ "` Sex is not only sinful , it 's disgusting . \" sinful ['Sex'] \n",
+ "\n",
+ "His wicked , mocking eyes gleamed down at her . wicked ['eyes'] \n",
+ "\n",
+ "` You wicked , cruel boy ! \" wicked ['boy'] \n",
+ "\n",
+ "The self-confident leader can be insensitive and inconsiderate . inconsiderate ['The self-confident leader'] \n",
+ "\n",
+ "These are issues which future studies may seek to address . address ['future studies'] \n",
+ "\n",
+ "You are gentle and considerate . considerate ['You'] \n",
+ "\n",
+ "\" I knew you were being thoughtless , rather than mean . thoughtless ['you'] \n",
+ "\n",
+ "His expression was calculating , perhaps cruel . cruel ['His expression'] \n",
+ "\n",
+ "But the culture can be not only cruel but stupid . cruel ['the culture'] \n",
+ "\n",
+ "To Helen , the reaction seemed unnecessarily cruel and impolite . impolite ['the reaction'] \n",
+ "\n",
+ "But refusing a drink would be churlish . churlish ['refusing a drink'] \n",
+ "\n",
+ "Athelstan felt like refusing , but that would have been churlish . churlish ['that'] \n",
+ "\n",
+ "He was n't being unkind , just generous ! unkind ['He'] \n",
+ "\n",
+ "This time her eyes were troubled , compassionate . compassionate ['her eyes'] \n",
+ "\n",
+ "Caroline tried to sound compassionate . compassionate ['Caroline'] \n",
+ "\n",
+ "` We believe it is a fair and generous price . \" fair ['price'] \n",
+ "\n",
+ "His eyes were watchful , unfriendly . unfriendly ['His eyes'] \n",
+ "\n",
+ "Dolly was obviously set on being sociable for the whole evening . sociable ['Dolly'] \n",
+ "\n",
+ "Jackson 's genial , reassuring tone snaked out past clenched teeth . genial ['tone'] \n",
+ "\n",
+ "Theodora smiled and was civil . civil ['Theodora'] \n",
+ "\n",
+ "She 's intelligent as well as tactful and efficient . \" tactful ['She'] \n",
+ "\n",
+ "You should be polite , tactful and helpful whenever possible . tactful ['You'] \n",
+ "\n",
+ "George decided that he had better be diplomatic . diplomatic ['he'] \n",
+ "\n",
+ "He seems to have been corrupt from an early age . corrupt ['He'] \n",
+ "\n",
+ "It was n't ethical , but Celia seemed so terrified . ethical ['It'] \n",
+ "\n",
+ "But I do n't want to be virtuous . virtuous ['I'] \n",
+ "\n",
+ "He could n't have been more chatty and pleasant . pleasant ['He'] \n",
+ "\n",
+ "` You 'd prefer to be villainous ? \" villainous ['You'] \n",
+ "\n",
+ "She began screaming and in desperation threw ornaments and bottles . desperation ['She'] \n",
+ "\n",
+ "She thought with envy of Jasper in his cell . envy ['She'] \n",
+ "\n",
+ "Rachel felt the envy and anger again and was confused . envy ['Rachel'] \n",
+ "\n",
+ "He felt no pity . pity ['He'] \n",
+ "\n",
+ "She sighed without regret . regret ['She'] \n",
+ "\n",
+ "She was bursting with adoration . adoration ['She'] \n",
+ "\n",
+ "Hindenburg regarded such a prospect with abhorrence . abhorrence ['Hindenburg'] \n",
+ "\n",
+ "He looked at her with detestation , and so did April . detestation ['He'] \n",
+ "\n",
+ "He is n't grateful or contrite . contrite ['He'] \n",
+ "\n",
+ "Leonora bit her lip in remorse . remorse ['Leonora'] \n",
+ "\n",
+ "` He does n't seem to be remorseful in any way . remorseful ['He'] \n",
+ "\n",
+ "Holmes watched the fog and began to look worried . worried ['Holmes'] \n",
+ "\n",
+ "You rang me , \" Sly replied , suddenly feeling nervous . nervous ['Sly'] \n",
+ "\n",
+ "He 'd sounded nervous on the phone last night . nervous ['He'] \n",
+ "\n",
+ "The cats are fine , although nervous . nervous ['The cats'] \n",
+ "\n",
+ "Freebase is a compound that can be inhaled or smoked . smoked ['Freebase'] \n",
+ "\n",
+ "He turned a livid , stubbled face to the crowd . livid ['face'] \n",
+ "\n",
+ "Collingridge was clearly in a relaxed and very trusting mood . relaxed ['mood'] \n",
+ "\n",
+ "Andrew gives a relaxed , reflective smile . relaxed ['smile'] \n",
+ "\n",
+ "She was simultaneously pleased , revolted and delighted . revolted ['She'] \n",
+ "\n",
+ "But why does Mr Ross always look sad ? sad ['Mr Ross'] \n",
+ "\n",
+ "She 's feeling stressed . stressed ['She'] \n",
+ "\n",
+ "If this happened they would be terror-stricken . terror-stricken ['they'] \n",
+ "\n",
+ "Stiarkoz slowly stood up , looking neither unsettled nor surprised . unsettled ['Stiarkoz'] \n",
+ "\n",
+ "No , she 's not vexed . vexed ['she'] \n",
+ "\n",
+ "I 'm getting vexed . vexed ['I'] \n",
+ "\n",
+ "Please help Goodwill and help people find jobs . help ['find jobs'] \n",
+ "\n",
+ "Please help Goodwill and help people find jobs . help ['find jobs'] \n",
+ "\n",
+ "Mr Hollins looked surprised , but he was not ruffled . ruffled ['he'] \n",
+ "\n",
+ "Lucker wakes from his half-sleep , looking ruffled and resentful . ruffled ['Lucker'] \n",
+ "\n",
+ "The powder is usually sniffed . sniffed ['The powder'] \n",
+ "\n",
+ "If so , I am not abashed . abashed ['I'] \n",
+ "\n",
+ "He mumbled unintelligibly and looked abashed . abashed ['He'] \n",
+ "\n",
+ "George was not abashed . abashed ['George'] \n",
+ "\n",
+ "She sounded suddenly flustered and evaded an answer . flustered ['She'] \n",
+ "\n",
+ "Rachel had been nonplussed . nonplussed ['Rachel'] \n",
+ "\n",
+ "Sheila Williams was feeling miserable . miserable ['Sheila Williams'] \n",
+ "\n",
+ "He must be miserable . \" miserable ['He'] \n",
+ "\n",
+ "He felt increasingly irritated and despondent . despondent ['He'] \n",
+ "\n",
+ "By comparison , she felt drained and despondent . despondent ['she'] \n",
+ "\n",
+ "` You must n't be downcast , \" said Lydia . downcast ['You'] \n",
+ "\n",
+ "He felt too let-down and disgruntled . disgruntled ['He'] \n",
+ "\n",
+ "Yet they were not downhearted . downhearted ['they'] \n",
+ "\n",
+ "I am not downhearted . downhearted ['I'] \n",
+ "\n",
+ "The woman was not affronted . affronted ['The woman'] \n",
+ "\n",
+ "He was feeling sleepy . sleepy ['He'] \n",
+ "\n",
+ "I 'm warm and sleepy and there 's this girl . sleepy ['I'] \n",
+ "\n",
+ "` I 'm not hungry . \" hungry ['I'] \n",
+ "\n",
+ "Christine woke up feeling horny . horny ['Christine'] \n",
+ "\n",
+ "He was unshaven and looked exhausted . exhausted ['He'] \n",
+ "\n",
+ "He looks a little weary . weary ['He'] \n",
+ "\n",
+ "He still looked cheerful , but very weary . weary ['He'] \n",
+ "\n",
+ "I 'm not frightened or bored or weary . weary ['I'] \n",
+ "\n",
+ "She was numb with cold , and bone-weary . bone-weary ['She'] \n",
+ "\n",
+ "The boy was broken-hearted and inconsolable . inconsolable ['The boy'] \n",
+ "\n",
+ "He was not inconsolable . inconsolable ['He'] \n",
+ "\n",
+ "The Germans still seemed stunned . stunned ['The Germans'] \n",
+ "\n",
+ "Rory watched for a while , befuddled . befuddled ['Rory'] \n",
+ "\n",
+ "They stumbled on , half unconscious with fatigue . fatigue ['They'] \n",
+ "\n",
+ "His face was grey with fatigue . fatigue ['His face'] \n",
+ "\n",
+ "His eyes felt heavy with fatigue . fatigue ['His eyes'] \n",
+ "\n",
+ "I started to feel drowsy . drowsy ['I'] \n",
+ "\n",
+ "Now I 'm getting drowsy , the fire is burning low drowsy ['I'] \n",
+ "\n",
+ "She looked at him with drowsy , yearning eyes . drowsy ['eyes'] \n",
+ "\n",
+ "With some tuna mousse , if you 're really feeling peckish . peckish ['you'] \n",
+ "\n",
+ "He felt faint and nauseous . nauseous ['He'] \n",
+ "\n",
+ "I was just parched and thirsty -- parched ['I'] \n",
+ "\n",
+ "First-time heroin users may be sick . sick ['First-time heroin users'] \n",
+ "\n",
+ "Leonora shook her head in revulsion . revulsion ['Leonora'] \n",
+ "\n",
+ "Erika stamped her foot in vexation . vexation ['Erika'] \n",
+ "\n",
+ "Frau Nordern bit her lip in vexation . vexation ['Frau Nordern'] \n",
+ "\n",
+ "Sister Cecilia was flushed with vexation . vexation ['Sister Cecilia'] \n",
+ "\n",
+ "His face darkened with anger . anger ['His face'] \n",
+ "\n",
+ "The guard stiffened in anger . anger ['The guard'] \n",
+ "\n",
+ "Mr Jones exploded in anger . anger ['Mr Jones'] \n",
+ "\n",
+ "Her voice shook , her face was white with anguish . anguish ['her face'] \n",
+ "\n",
+ "` I am practical , \" Jessamy insisted with some annoyance . annoyance ['Jessamy'] \n",
+ "\n",
+ "Dotty flushed with annoyance . annoyance ['Dotty'] \n",
+ "\n",
+ "My heart was thumping with disquiet . disquiet ['My heart'] \n",
+ "\n",
+ "She realised with astonishment that he was actually pleading . astonishment ['She'] \n",
+ "\n",
+ "Elaine Morris stared out of the window in astonishment . astonishment ['Elaine Morris'] \n",
+ "\n",
+ "Constance looked at him in bewilderment . bewilderment ['Constance'] \n",
+ "\n",
+ "He paused as his mother stared at him in bewilderment . bewilderment ['his mother'] \n",
+ "\n",
+ "Mrs Alderley frowned in perplexity . perplexity ['Mrs Alderley'] \n",
+ "\n",
+ "Her brow creased in puzzlement . puzzlement ['Her brow'] \n",
+ "\n",
+ "Downes shook his head in stupefaction . stupefaction ['Downes'] \n",
+ "\n",
+ "` It 's stuck ! \" she cried in dismay . dismay ['she'] \n",
+ "\n",
+ "She jumped up , frowning in dismay . dismay ['She'] \n",
+ "\n",
+ "Edouard looked at him , and felt a sickening dismay . dismay ['Edouard'] \n",
+ "\n",
+ "Cleo stared at the contents with dismay . dismay ['Cleo'] \n",
+ "\n",
+ "Melissa stared at her friend in dismay . dismay ['Melissa'] \n",
+ "\n",
+ "They were young but they walked bent in sorrow . sorrow ['they'] \n",
+ "\n",
+ "Jasper 's sometimes funny when he has had a disappointment . \" disappointment ['he'] \n",
+ "\n",
+ "` Damn ! \" she muttered in exasperation . exasperation ['she'] \n",
+ "\n",
+ "Irina hung up in exasperation . exasperation ['Irina'] \n",
+ "\n",
+ "Parkin sighed with exasperation . exasperation ['Parkin'] \n",
+ "\n",
+ "Emily rose to her feet in agitation . agitation ['Emily'] \n",
+ "\n",
+ "A thin hand gestured in agitation . agitation ['A thin hand'] \n",
+ "\n",
+ "By now the adjutant was trembling with distress . distress ['the adjutant'] \n",
+ "\n",
+ "The crowd was wild with excitement . excitement ['The crowd'] \n",
+ "\n",
+ "She was tingling with excitement . excitement ['She'] \n",
+ "\n",
+ "Robyn was almost ill with excitement and outrage . excitement ['Robyn'] \n",
+ "\n",
+ "Athelstan gasped with excitement and went round to rouse the coroner . excitement ['Athelstan'] \n",
+ "\n",
+ "Maggie felt an odd excitement stirring inside . excitement ['Maggie'] \n",
+ "\n",
+ "Tremayne was pink with gratification . gratification ['Tremayne'] \n",
+ "\n",
+ "Some of us contemplate this prospect with horror . horror ['Some of us'] \n",
+ "\n",
+ "She went red with mortification . mortification ['She'] \n",
+ "\n",
+ "When he wondered this he felt a deep sadness . sadness ['he'] \n",
+ "\n",
+ "I asked , feeling suddenly bilious . bilious ['I'] \n",
+ "\n",
+ "She was getting amorous , and he was n't . amorous ['She'] \n",
+ "\n",
+ "` And Glyn is not amorous . amorous ['Glyn'] \n",
+ "\n",
+ "Tories in the region say recent royal coverage has been deplorable . deplorable ['recent royal coverage'] \n",
+ "\n",
+ "The author has no intention of making accusations or unmasking society . accusations ['The author'] \n",
+ "\n",
+ "` I never intended to belittle or humiliate her in public . belittle ['I'] \n",
+ "\n",
+ "“ I do not criticise . criticise ['I'] \n",
+ "\n",
+ "I 'm not criticising . criticising ['I'] \n",
+ "\n",
+ "` I 'm not criticising , \" he said . criticising ['I'] \n",
+ "\n",
+ "Out he leaps from his ministerial hot-rod to remonstrate . remonstrate ['he'] \n",
+ "\n",
+ "When Denis started to remonstrate , Joshua interrupted him . remonstrate ['Denis'] \n",
+ "\n",
+ "Jeffrey Archer bravely attempted to remonstrate . remonstrate ['Jeffrey Archer'] \n",
+ "\n",
+ "Many such drugs were , in fact , illicit now . illicit ['Many such drugs'] \n",
+ "\n",
+ "Huge rats roamed the streets slaying then devouring their victims . slaying ['Huge rats'] \n",
+ "\n",
+ "He was becoming agitated . agitated ['He'] \n",
+ "\n",
+ "She was getting agitated . agitated ['She'] \n",
+ "\n",
+ "His voice was tetchy , but not alarmed . alarmed ['His voice'] \n",
+ "\n",
+ "Kate 's voice was husky and bewildered . bewildered [\"Kate 's voice\"] \n",
+ "\n",
+ "Michael frowned at him , bewildered . bewildered ['Michael'] \n",
+ "\n",
+ "Her whole face drooped and she blushed with chagrin . chagrin ['she'] \n",
+ "\n",
+ "He 's being covetous the way I was . covetous ['He'] \n",
+ "\n",
+ "Then they walked away , humbled and dejected . dejected ['they'] \n",
+ "\n",
+ "I felt pierced to the heart , dejected and lonely . dejected ['I'] \n",
+ "\n",
+ "She glanced down at the linoleum in despair . despair ['She'] \n",
+ "\n",
+ "Bishop did n't seem discomfited . discomfited ['Bishop'] \n",
+ "\n",
+ "But Rachel was not discouraged . discouraged ['Rachel'] \n",
+ "\n",
+ "He does n't get discouraged easily . discouraged ['He'] \n",
+ "\n",
+ "Vincent was irritated but not dismayed . dismayed ['Vincent'] \n",
+ "\n",
+ "When they began to overlap women felt confused and disoriented . disoriented ['women'] \n",
+ "\n",
+ "Her expression became dreamy , ecstatic . ecstatic ['Her expression'] \n",
+ "\n",
+ "He felt both elated and frightened . elated ['He'] \n",
+ "\n",
+ "Oliver was not displeased . displeased ['Oliver'] \n",
+ "\n",
+ "He was not displeased . displeased ['He'] \n",
+ "\n",
+ "Is this the statement of an embittered , biased man ? embittered ['man'] \n",
+ "\n",
+ "She began to feel exasperated . exasperated ['She'] \n",
+ "\n",
+ "No wonder Superintendent Miller had become exasperated . exasperated ['Superintendent Miller'] \n",
+ "\n",
+ "Lili was getting exasperated . exasperated ['Lili'] \n",
+ "\n",
+ "He shook his head , exasperated and cynical . exasperated ['He'] \n",
+ "\n",
+ "You must have been heartbroken . \" heartbroken ['You'] \n",
+ "\n",
+ "The kids will be heartbroken . heartbroken ['The kids'] \n",
+ "\n",
+ "Daisy leaned forward , at once horrified and fascinated . horrified ['Daisy'] \n",
+ "\n",
+ "She was irritated , indignant . indignant ['She'] \n",
+ "\n",
+ "It 'll be very loud and you might get frightened . \" frightened ['you'] \n",
+ "\n",
+ "Now she was mad with grief . grief ['she'] \n",
+ "\n",
+ "Pilger was immediately sympathetic . sympathetic ['Pilger'] \n",
+ "\n",
+ "Japan is also unsympathetic . unsympathetic ['Japan'] \n",
+ "\n",
+ "He hoped she was n't offended , as a Times reporter . offended ['she'] \n",
+ "\n",
+ "Nutty was not offended . offended ['Nutty'] \n",
+ "\n",
+ "I laughed -- I did n't feel offended . offended ['I'] \n",
+ "\n",
+ "His smile was patient , peeved , and very sad . peeved ['His smile'] \n",
+ "\n",
+ "I 'm not surprised he 's feeling peeved . peeved ['he'] \n",
+ "\n",
+ "Zambia ordered Nathan , who was looking up perplexed . perplexed ['Nathan', 'who'] \n",
+ "\n",
+ "Eva was excited and perplexed and intimidated by the crowd . perplexed ['Eva'] \n",
+ "\n",
+ "She did n't seem perturbed . perturbed ['She'] \n",
+ "\n",
+ "Lavinia , still scolding , wiped the crumbs from their cardigans . scolding ['Lavinia'] \n",
+ "\n",
+ "Milton is paying to Eden the highest compliment in his power . compliment ['Milton'] \n",
+ "\n",
+ "People can be very shy about giving compliments . compliments ['People'] \n",
+ "\n",
+ "The incompetent or misguided general had been dismissed . incompetent ['general'] \n",
+ "\n",
+ "Another accuses his dad of being a bit stingy . stingy ['his dad'] \n",
+ "\n",
+ "Sir Ralph , he thought , was probably a miser . miser ['Sir Ralph'] \n",
+ "\n",
+ "Isabel had eventually hushed . hushed ['Isabel'] \n",
+ "\n",
+ "` They wo n't tell , \" said Dagmar . tell ['They'] \n",
+ "\n",
+ "Léonie shut her eyes and frowned . frowned ['Léonie'] \n",
+ "\n",
+ "Madeleine stiffened and frowned . frowned ['Madeleine'] \n",
+ "\n",
+ "He grinned , and the porter grinned back . grinned ['the porter'] \n",
+ "\n",
+ "I 'm not going to pout or be upset at anybody . pout ['I'] \n",
+ "\n",
+ "Major Vanavskaya put down the phone and scowled . scowled ['Major Vanavskaya'] \n",
+ "\n",
+ "Edward saw that he was teased , and smiled . smiled ['Edward'] \n",
+ "\n",
+ "She , naturally enough , smiled back . smiled ['She'] \n",
+ "\n",
+ "She smiled , and the girl smiled back . smiled ['the girl'] \n",
+ "\n",
+ "She sat there smirking for a long time . smirking ['She'] \n",
+ "\n",
+ "With humidity soaring he was already perspiring . perspiring ['he'] \n",
+ "\n",
+ "` I was arrested , imprisoned and tortured . tortured ['I'] \n",
+ "\n",
+ "Pulse : elevated ; normal . normal ['Pulse'] \n",
+ "\n",
+ "Everything is completely back to normal now ! normal ['Everything'] \n",
+ "\n",
+ "` I 've been forbidden to divulge , \" she said . divulge ['I'] \n",
+ "\n",
+ "She hinted of an important secret still to be divulged . divulged ['an important secret'] \n",
+ "\n",
+ "Whatever he was about to confess was crucifying him with guilt . confess ['Whatever', 'he'] \n",
+ "\n",
+ "HF is a dangerous chemical with a poisonous and corrosive vapour . poisonous ['vapour'] \n",
+ "\n",
+ "Baynes flushed with pleasure . pleasure ['Baynes'] \n",
+ "\n",
+ "Was she really going to be caned ? caned ['she'] \n",
+ "\n",
+ "\" That calf has been poisoned . \" poisoned ['That calf'] \n",
+ "\n",
+ "“ They may have been poisoned . ” poisoned ['They'] \n",
+ "\n",
+ "His rise to power ushered in the Hellenistic period . power ['His'] \n",
+ "\n",
+ "Thornton was a strong , if irregular , family man . strong ['family man'] \n",
+ "\n",
+ "You ca n't complain . complain ['You'] \n",
+ "\n",
+ "What are you trying to convey ? convey ['you'] \n",
+ "\n",
+ "` The accounts have obviously been falsified . \" falsified ['The accounts'] \n",
+ "\n",
+ "They were jailed after their visas were found to be forged . forged ['their visas'] \n",
+ "\n",
+ "` How do I know this is n't a forgery ? \" forgery ['this'] \n",
+ "\n",
+ "` Admit it , you were reading ! \" Admit ['you were reading'] \n",
+ "\n",
+ "He escaped as you have just described . described ['as', 'you'] \n",
+ "\n",
+ "He or she is urban , trendy and affluent . trendy ['He or she'] \n",
+ "\n",
+ "I was born cool … \" cool ['I'] \n",
+ "\n",
+ "The operations had been mainly exploratory . exploratory ['The operations'] \n",
+ "\n",
+ "Such people are to be pitied . pitied ['Such people'] \n",
+ "\n",
+ "` Monsieur , your room has been ransacked . ransacked ['your room'] \n",
+ "\n",
+ "But somehow she has managed to remain cool and calm . cool ['she'] \n",
+ "\n",
+ "` It 's the one time when Tara becomes almost quiet . quiet ['Tara'] \n",
+ "\n",
+ "` I 'll keep quiet for now , but later -- ` quiet ['I'] \n",
+ "\n",
+ "( Gingrich even warned his GOP underlings to keep quiet . ) quiet ['his GOP underlings'] \n",
+ "\n",
+ "We know that greenhouse effects can be enormous , however . enormous ['greenhouse effects'] \n",
+ "\n",
+ "The small , lovely cloister has curious , rather Arab-style arches . curious ['arches'] \n",
+ "\n",
+ "In other manifestations of this curious , cumbersome style are curious ['style'] \n",
+ "\n",
+ "they 've got to be curious curious ['they'] \n",
+ "\n",
+ "However , it was seen at the time as a success . success ['it'] \n",
+ "\n",
+ "However , they view the program as a success . success ['the program'] \n",
+ "\n",
+ "The early results have been poor to middling . poor ['The early results'] \n",
+ "\n",
+ "But the need is still great . great ['the need'] \n",
+ "\n",
+ "This makes no sense . sense ['This'] \n",
+ "\n",
+ "Because it does n't even make sense . sense ['it'] \n",
+ "\n",
+ "Day by day I watched Peter grow and develop . develop ['Peter'] \n",
+ "\n",
+ "Intramuscular injections are not recommended . recommended ['Intramuscular injections'] \n",
+ "\n",
+ "A dark , mysterious voice answered him . dark ['voice'] \n",
+ "\n",
+ "Senecio pulcher needs good soil and a sunny , sheltered corner sunny ['corner'] \n",
+ "\n",
+ "These are all true - probably . true ['These'] \n",
+ "\n",
+ "And we grew up puny , with bad chests . puny ['we'] \n",
+ "\n",
+ "She had no energy and tended to be listless and lethargic . energy ['She'] \n",
+ "\n",
+ "Then we wonder why we have no energy . energy ['we'] \n",
+ "\n",
+ "The Scottish are n't lazy like the English . lazy ['The Scottish'] \n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "for example in annotations_reformatted:\n",
+ " print(example['sentence'], example[\"target\"], example['links'], '\\n')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 58,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# saving annotations (indices)\n",
+ "with open('framenet_examples_indices.txt', 'w') as csvfile:\n",
+ " writer = csv.writer(csvfile, delimiter=';')\n",
+ " for example in annotations:\n",
+ " links = example['links']\n",
+ " sentence = example['sentence']\n",
+ " target = example['target']\n",
+ " writer.writerow((sentence, target, links))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 59,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# saving annotations\n",
+ "with open('framenet_examples_words.txt', 'w') as csvfile:\n",
+ " writer = csv.writer(csvfile, delimiter=';')\n",
+ " for example in annotations_reformatted:\n",
+ " links = example['links']\n",
+ " sentence = example['sentence']\n",
+ " target = example['target']\n",
+ " writer.writerow((sentence, target, links))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 61,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def analyze_sem_links(model, example):\n",
+ " \"\"\"\n",
+ " Analyze weights of semantic links\n",
+ " \"\"\"\n",
+ " bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(example['sentence'], \"\", max_len=20)\n",
+ " input_ids, input_mask, segment_ids = convert_to_bert_inputs(example['sentence'], \"\", max_len=20)\n",
+ " crop_len = torch.sum(input_mask != 0)\n",
+ " \n",
+ " _, output = model(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ " output = output.squeeze(0).detach().numpy()\n",
+ " \n",
+ " target_start, target_end = example['target'][0], example['target'][1]\n",
+ " target_tokens = tokenizer.tokenize(example['sentence'][target_start:target_end])\n",
+ " target_idx = [bert_tokens.index(token) for token in target_tokens]\n",
+ " \n",
+ " all_source_idx = []\n",
+ " for link in example['links']:\n",
+ " source_start, source_end = link[0], link[1]\n",
+ " source_tokens = tokenizer.tokenize(example['sentence'][source_start:source_end])\n",
+ " source_idx = [bert_tokens.index(token) for token in source_tokens]\n",
+ " all_source_idx.append(source_idx)\n",
+ " all_source_idx = list(itertools.chain.from_iterable(all_source_idx))\n",
+ " \n",
+ " weights = np.zeros((n_layers, n_heads))\n",
+ " for l in range(n_layers):\n",
+ " for h in range(n_heads): \n",
+ " weights[l, h] = get_max_source_target_weight(output[l, h, :crop_len, :crop_len], all_source_idx, target_idx)\n",
+ "\n",
+ " return weights\n",
+ " "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 62,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "100%|██████████| 473/473 [00:19<00:00, 24.06it/s]\n"
+ ]
+ }
+ ],
+ "source": [
+ "all_weights = []\n",
+ "for example in tqdm(annotations):\n",
+ " weights = analyze_sem_links(model_base, example)\n",
+ " all_weights.append(weights)\n",
+ " \n",
+ "all_weights = np.mean(np.stack(all_weights), axis=0)\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 63,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "([,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ],\n",
+ " )"
+ ]
+ },
+ "execution_count": 63,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAqcAAAJNCAYAAAAMOtzgAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3Xu0XWV57/Hv2vEChAYjCAHMMaUyHgVaT7lU0GKCgKLFcpSAPSKIYqsIFK036oVrq6OixRtVOEVRrAoIoaCAESxgT3SIqVJU+ggcuUiCiaKYoASS7PPHnMFF3Nl7T0rW+67s72eMNVhrrjnXfDLZYT/83vedqzc6OookSZJUg5HSBUiSJEnr2JxKkiSpGjankiRJqobNqSRJkqphcypJkqRq2JxKkiSpGk8oXYAkSZLgtF4M7P6ep4xmb1Dn6srkVJIkSdWwOZUkSVI1HNaXJEmqgIlhY6ia07X/vP/QfdfqP/3lT0qX0Mnxq84pXUJ3D/+qdAWdrfmnD5QuoZv7Hi5dQXc/ebB0BZ2NnLhP6RI6Wfv33yhdQmcjb/7j0iV01tv95aVL6GT0h1eXLqGz3nP/T7XzL6eioWpOJUmSNlUmpw2vgyRJkqphcipJklQBE8OG10GSJEnVMDmVJEmqgIlhw+sgSZKkapicSpIkVcDEsOF1kCRJUjVMTiVJkirgNwE0TE4lSZJUDZtTSZIkVcNhfUmSpAqYGDa8DpIkSaqGyakkSVIFTAwbXgdJkiRVw+RUkiSpAiaGDZtTSZIkPUpEnAXsDYwCJ2bmjX3v7Qe8H1gDJPD6zFw71jERMRu4AJgGLAWOzMxV4527iiY9InaLiNsj4vjStUiSJJUwMsDHeCJiLrBzZu4DHAN8dL1dzgXmZ+bzgd8DDhrnmNOBszNzX+A24HWTuQ5FRcR04GPAtaVrkSRJEvsDlwFk5i3AzIiY0ff+Hpn5k/b5cmDrcY6ZB1ze7nsFcMBEJy/enAKrgJcCS0oXIkmSVEotySkwi6bpXGd5uw2AzPwVQERsD7wIuHKcY6b3DeMvA7afzHUoKjNXZ+ZvStchSZKkMfXW3xAR29IkoW/KzJ9P5pgNbPsdxZtTSZIkVZWcLqEvKQV2oFnMBEA7XH8V8J7MXDjBMSsjYvN2245MYqTc5lSSJEn9FgLzASJid2BJZq7oe/9DwFmZefUkjrkGOLTd51Cg/5gxeSspSZKkCtSSGGbmoohYHBGLgLXAcRFxNHA/8FXgKGDniHh9e8jnM/Pc9Y9p3zsF+GxEvAG4E/jMROcv3pxGxB40Hfgc4OGImA+8IjPvK1qYJEnSFJWZJ6236aa+50+e5DFk5lLgwC7nLt6cZuZimtsMSJIkTVm1JKeleR0kSZJUDZtTSZIkVaP4sL4kSZJMDNfxOkiSJKkaJqeSJEkVMDFseB0kSZJUDZNTSZKkCpgYNrwOkiRJqobJqSRJUgV6pQuohMmpJEmSqmFyKkmSVAETw4bXQZIkSdUwOZUkSaqAiWHD6yBJkqRqmJxKkiRVwMSw4XWQJElSNUxOJUmSKmBi2PA6SJIkqRq90dHR0jVM3i8uGKJiWzP+oHQF3ay4o3QFnY0uuqh0CZ31Xvre0iV0c//tpSvobqsh+7sHjC5bXLqEblYuLV1BdzN2LF1BZ73NtildQjdrHy5dQXczDqviy5ku6cXA+pxDR7OKP/NYTE4lSZJUDZtTSZIkVcMFUZIkSRUwMWx4HSRJklQNk1NJkqQKmBg2vA6SJEmqhsmpJElSBUwMG14HSZIkVcPkVJIkqQImhg2vgyRJkqphcipJklQBE8OG10GSJEnVMDmVJEmqQK90AZUwOZUkSVI1TE4lSZIqYGLY8DpIkiSpGiankiRJFTAxbBRvTiPiA8C+bS3vz8xLC5ckSZKkQoo26RGxH7BbZu4DHAR8uGQ9kiRJKqt0gnwDcFj7/JfA9IiYVrAeSZKkIkYG+KhZ0WH9zFwDPNC+PAa4st0mSZKkKaj4nFOAiDiEpjl9UelaJEmSSuh5F36gguY0Il4MvBs4KDPvL12PJEmSyinanEbEVsCZwAGZeV/JWiRJkkoa6Y2WLqEKpZPTVwLbABdFxLptR2XmXeVKkiRJUimlF0SdC5xbsgZJkqQaOOe0UfvdBCRJkjSFlB7WlyRJEmBw2jA5lSRJUjVMTiVJkirQc7U+YHIqSZKkipicSpIkVcDV+g2TU0mSJFXD5FSSJKkCJqcNk1NJkiRVw+ZUkiRJ1XBYX5IkqQIjFd1KKiLOAvYGRoETM/PGvvc2A84Bds3MPdttxwBH9n3Enpm5ZURcB0wHHmi3vzUzF493bptTSZIkPSIi5gI7Z+Y+EfFs4FPAPn27nAl8D9h13YbMPA84r+/4w/v2f21mfn+y53dYX5IkqQK9AT4msD9wGUBm3gLMjIgZfe+/C1gwzvEnA2dMfJqx2ZxKkiSp3yxged/r5e02ADJzxYYOjIi9gLsz896+zadHxA0RcU5EbD7RyW1OJUmSKtDrDe7RtbQO+74eOL/v9UeAt2fmC4C1wHETfYBzTiVJktRvCX1JKbADsHSSx84DTlj3IjP7h/+vAF450QeYnEqSJFWgouR0ITAfICJ2B5aMN5S/TkTsAKzMzIfa172IuCYintLuMg+YcGHUcCWnT3lW6Qq6+/Vk/0ejEr3h+pEA4A+fV7qC7lbcUbqCTkZX/7p0CZ31hrBmHvx56Qq6+cW9E+9Tmd7vv7R0CZ2N3nND6RI6Gb1hYekSOht51WGlS6hKZi6KiMURsYh2KD4ijgbuz8wFEXExMBuI9lZR52bm54HtgWV9nzMaEecC10bEA8A9wKkTnX8IOxFJkqRNT6+i+5xm5knrbbqp770xu/n2/qUvWW/bRcBFXc7tsL4kSZKqYXIqSZJUgZHuq+g3SSankiRJqobJqSRJUgUew/1HN0kmp5IkSaqGyakkSVIFetSzWr8kk1NJkiRVw+ZUkiRJ1XBYX5IkqQIuiGqYnEqSJKkaJqeSJEkVMDltmJxKkiSpGiankiRJFRjpeSspMDmVJElSRUxOJUmSKuCc04bJqSRJkqphcipJklQBg9NG0eY0IrYAzge2AzYDzsjML5esSZIkSeWUHtZ/GfCdzJwLHA78Y+F6JEmSiuj1Rgf2qFnR5DQzL+x7ORv4SalaJEmSVF4Vc04jYhHwdODg0rVIkiSV4Gr9RulhfQAy83nAnwOfiwj/1UiSJE1RRZvTiNgjImYDZOb3aJLcp5WsSZIkqYSR3uAeNSudnL4AeCtARGwHbAn8rGhFkiRJKqZ0c/pJYNuI+AbwFeC4zFxbuCZJkiQVUnq1/m+AV5WsQZIkqQa13+JpUEonp5IkSdIjqriVlCRJ0lRX+TqlgTE5lSRJUjVMTiVJkirgTfgbJqeSJEmqhsmpJElSBVyt3zA5lSRJUjVMTiVJkipQ+9eKDorJqSRJkqphcipJklQBV+s3TE4lSZJUDZNTSZKkCpicNkxOJUmSVA2TU0mSpAr08D6nYHIqSZKkiticSpIkqRoO60uSJFXABVGN4WpOe8MX9I7+/JbSJXTSm/UnpUvobPSLnyhdQmejQ/Y1ICN/eWbpErp78GelK+hu821KV9DNk35auoLufvXj0hV0d/t3S1fQycjh7yldgobccDWnkiRJm6jekAUXG8vwRZGSJEnaZJmcSpIkVWAIZy9uFF4GSZIkVcPkVJIkqQKu1m+YnEqSJKkaJqeSJEk1cLU+YHIqSZKkipicSpIkVcDV+g0vgyRJkqphcipJklSBnsv1AZNTSZIkVcTkVJIkqQI1zTmNiLOAvYFR4MTMvLHvvc2Ac4BdM3PPdts84GLgB+1uN2fmCRExG7gAmAYsBY7MzFXjnbuiyyBJkqTSImIusHNm7gMcA3x0vV3OBL43xqHXZ+a89nFCu+104OzM3Be4DXjdROe3OZUkSVK//YHLADLzFmBmRMzoe/9dwIJJftY84PL2+RXAARMdYHMqSZJUg15vcI/xzQKW971e3m4DIDNXbOC4XSLi8oj494g4sN02vW8Yfxmw/UQnd86pJEmSxjOZ2wjcCpwGXATsBPxbRDzzMXxOHc1pRGwOfB84IzPPL1yOJEnSwFW0IGoJfUkpsAPNYqYNysx7gAvbl7dHxL3AjsDKiNg8M3/Tvl4y0clruQzvAe4rXYQkSZJYCMwHiIjdgSXjDOXT7ndERLytfT4L2A64B7gGOLTd7VDg6olOXjw5jYhnAbsAXyldiyRJUim9kTpuwp+ZiyJicUQsAtYCx0XE0cD9mbkgIi4GZgMREdcB59Isevp8RBwCPAk4NjMfiohTgM9GxBuAO4HPTHT+4s0p8CHgeOA1pQuRJEkSZOZJ6226qe+9wzZw2MvG+JylwIFj7LtBRYf1I+Io4JuZ+eOSdUiSJJVWz2L9skonp38G7BQRBwNPB1ZFxE8y85rCdUmSJKmAos1pZr5y3fOIOBW4w8ZUkiRNRRWt1i/KyyBJkqRqlB7Wf0Rmnlq6BkmSpGIqWa1fmsmpJEmSqlFNcipJkjSV1b6KflBMTiVJklQNk1NJkqQK1PINUaWZnEqSJKkaNqeSJEmqhsP6kiRJFfAm/A0vgyRJkqphcipJklSBnveSAkxOJUmSVBGTU0mSpBoYGQJeBkmSJFXE5FSSJKkCTjltmJxKkiSpGiankiRJFfDrSxsmp5IkSaqGyakkSVIF/IaohpdBkiRJ1eiNjo6WrmHS1py45/AUu85Da0tX0Mm0D51RuoTutphVuoLORm+9pHQJnay96LrSJXQ28kdbly6hs94Bx5UuoZtV95WuoLPRB4ev5t5Tdy1dQjfDGP89cW4Vkz0fmBsD63OmX59V/JnHMoQ/QZIkSdpUOedUkiSpAsMYOm8MXgZJkiRVw+ZUkiRJ1XBYX5IkqQLehL9hcipJkqRqmJxKkiRVoGdwCpicSpIkqSImp5IkSRVwzmnD5FSSJEnVMDmVJEmqgcEpYHIqSZKkipicSpIkVcCvL214GSRJklQNk1NJkqQKuFq/YXIqSZKkapicSpIkVcBviGoUbU4jYh5wMfCDdtPNmXlCuYokSZJUUg3J6fWZOb90EZIkSSU557ThnFNJkiRVo4bkdJeIuBx4KnBaZn6tdEGSJEkqo3RyeitwGnAI8BrgvIh4UtmSJEmSChgZ4KNiRZPTzLwHuLB9eXtE3AvsCPy4XFWSJEkqpfRq/SOA7TPzgxExC9gOuKdkTZIkSUW4IAooP+f0cuDzEXEI8CTg2Mx8qHBNkiRJKqT0sP4K4GUla5AkSapC5XNBB8XLIEmSpGqUHtaXJEkSOOe0ZXIqSZKkapicSpIk1cDIEPAySJIkqSImp5IkSTVwzilgcypJkqT1RMRZwN7AKHBiZt7Y995mwDnArpm5Z9/2DwD70vSX78/MSyPifGAP4Oftbmdm5lfGO7fNqSRJUg0qSU4jYi6wc2buExHPBj4F7NO3y5nA94Bd+47ZD9itPWZr4LvApe3bf5uZX57s+Z1zKkmSpH77A5cBZOYtwMyImNH3/ruABesdcwNwWPv8l8D0iJj2WE5ucypJklSDkQE+xjcLWN73enm7DXjkGz4fJTPXZOYD7ctjgCszc037+viI+HpEfDEitpno5DankiRJGs+k5xtExCE0zenx7aYLgJMy84U0UwFOnegznHMqSZJUg0rmnAJL6EtKgR2ApRMdFBEvBt4NHJSZ9wNk5rV9u1wOfGKizzE5lSRJUr+FwHyAiNgdWDLWUH6/iNiKZqHUwZl5X9/2SyJip/blPOD7E53c5FSSJEmPyMxFEbE4IhYBa4HjIuJo4P7MXBARFwOzgYiI64BzgS2BbYCLImLdRx0FfBy4MCJ+DawEXjvR+W1OJUmSalDReHZmnrTeppv63juMsZ07xra7gL26nLuiyyBJkqSpzuRUkiSpBvUsiCpqqJrTkZNeU7qEznrbPbd0CZ2M3vql0iV01nvGgaVL6Ky37XNKl9DJtLf8aekSunvyU0pX0NnozydcJ1CVted9pnQJnU078V2lS+huZKh+VTN60+dLl9BZb4+5pUtQn+H6iZckSdpUGZwCzjmVJElSRUxOJUmSauCcU8DkVJIkSRUxOZUkSaqBySlgcipJkqSKmJxKkiTVwMgQ8DJIkiSpIiankiRJNXDOKWByKkmSpIqYnEqSJFWgZ2QImJxKkiSpIjankiRJqobD+pIkSTVwQRRgcipJkqSKmJxKkiTVwMgQ8DJIkiSpIiankiRJNXDOKVBBcxoRRwDvAFYDJ2fmVwqXJEmSpEKKNqcRsTVwCrAHsCVwGmBzKkmSph6TU6B8cnoAcE1mrgBWAH9VuB5JkiQVVLo5nQNsERGXAzOBUzPz2rIlSZIkFeAydaD8ZegBWwOvAI4GPh0RZtqSJElTVOnk9KfAosxcDdweESuApwHLypYlSZI0YM45BconpwuBF0bESLs4akvgZ4VrkiRJUiFFk9PMvCcivgR8q910QmauLVmTJElSEaUjw0qUHtYnM88BzildhyRJksor3pxKkiQJ55y2DJAlSZJUDZtTSZIkVWPSzWlE5MYsRJIkaUobGeCjYl3KuzsiDt5olUiSJGnK67Ig6i6ab3C6C7gTWN3/ZmYe/ngWJkmSNKW4IAro1pz2gC9vrEIkSZKkSTenmfnajVmIJEnSlFb5XNBB6XQZIuLAiPiXiLiuff2EiDh6YxQmSZKkqafLav0TgAuBnwHPbTdvC5wcEe/cCLVJkiRNHSO9wT0q1iU5fQvwksw8cd2GzFwCHAy84fEuTJIkSVNPlwVR2wLfbp+P9m2/FZj1uFUkSZI0FVWeaA5Kl+T0R8CBY2x/DfDjx6ccSZIkTWVdktO/By6NiCuBJ0bE2cAf0cw/feXGKE6SJGnKcLU+0OEyZOYlwFxgGXANsD3wTWDXzFywccqTJEnSVDLp5DQi3gr8S2YevxHrkSRJmpqccwp0C5CPBO6OiK9FxJERMX1jFSVJkqSpqTc6OjrxXq2IeCbwCuBQYBfgCuBzwNWZuXajVNhvzaLJF1uJ0V/8sHQJnfS22rl0Cd1Ne3LpCrp74J7SFXRy2ox3lS6hs1NW/EPpEjpbe+U5pUvoZOTQ95Yuobu1D5euYNP3hC1KV9Bdb68qIsu1Z+07sD5n5C3fqOLPPJZOU28z87bM/EBmPpemOf0eTXO6NCL+ISJ23BhFSpIkaWrovC4sInoRcQBwGvBO4H7gHGBr4D8j4iWPb4mSJEmaKrosiPqfwKuBvwBmAJcAh2bmdX37LADOBnZ6fMuUJEnaxPWqHWkfqC7J6WLgOcDfArMy87X9jSlAZn4FWPX4lSdJkqSppMtN+Odk5t1jvRERZ2TmewEy89mPS2WSJElTicEp0KE5zcy7I2J/YE+gf3n0jsCrgCFctilJkqSadJlz+rfAKUACuwL/CTwTuAvwxvySJEn/Hc45BbrNOX0j8PzMfA7wcGbuDjwduJnmK00lSZKk/5Yuc06fmpmL2+drI2IkM38VEW8HvgZc9fiXJ0mSNEVUFJxGxFnA3sAocGJm3tj33mY0txHdNTP3HO+YiJgNXABMA5YCR2bmuIvnuySnP46Ig9rnS4D92ue/pklQJUmSNOQiYi6wc2buAxwDfHS9Xc6k+SKmyRxzOnB2Zu4L3Aa8bqLzd2lO3wdcHhFbAZ8GFkTEVcCNwPUdPkeSJEnr6/UG9xjf/sBlAJl5CzAzImb0vf8uYMEkj5kHXN7ucwVwwEQnn3RzmplfpOmI78/M9wFvAe6kiXWH6wuhJUmStCGzgOV9r5e32wDIzBUdjpneN4y/DNh+opN3mXNKZt7Z9/w84DyAiPg1sEWXz5IkSVKfzl8qPzCPZTbsWMdM6nMer8tQ0RReSZIk/TcsoS8pBXagWcz0WI5ZGRGbt9t2bPcb1+PVnI4+Tp8jSZI0NdUz53QhMB8gInYHlmxgKH8yx1wDHNrucyhw9UQn7zSsL0mSpE1bZi6KiMURsQhYCxwXEUcD92fmgoi4GJgNRERcB5ybmZ9f/5j2404BPhsRb6BZq/SZic4/YXMaEW+axJ9j2iT2kSRJ0oZUNEkyM09ab9NNfe8dNsljyMylwIFdzj2Z5PTtk9hnwvkDY4mIY4Aj+zbtmZlbPpbPkiRJ0vCbsDnNzN/fWCdfb8X/XODwjXUuSZIk1a+mOacnA0eULkKSJKmIiRcqTQlV3FErIvYC7s7Me0vXIkmSpHJqSU5fD5xfughJkqRiDE6BSpJTmu9dXVS6CEmSJJVVPDmNiB2AlZn5UOlaJEmSinHOKVBHcro9sKx0EZIkSSqveHKamYuBl5SuQ5IkqagaIsMKeBkkSZJUjeLJqSRJknDOacvkVJIkSdUwOZUkSaqBwSlgcipJkqSKmJxKkiTVwDmngMmpJEmSKmJyKkmSVAGD04bJqSRJkqphcypJkqRqOKwvSZJUA8f1AZNTSZIkVcTkVJIkqQYGp4DJqSRJkipicipJklSDEaNTMDmVJElSRUxOJUmSamBwCpicSpIkqSLDlZw+/EDpCrr77ldLV9DJ6E53lS6hu63mlK6gs96Ws0uX0MnJVzyrdAmdjf7s5tIldDbyv95ZuoRO1n725NIldNbbb7/SJXQ2uvDa0iV0MvL695UuobtaEkvvcwqYnEqSJKkiw5WcSpIkbaoMTgGTU0mSJFXE5FSSJKkGzjkFTE4lSZJUEZNTSZKkGhicAiankiRJqojNqSRJkqrhsL4kSVINRhzXB5NTSZIkVcTkVJIkqQYGp4DJqSRJkipicipJklQDb8IPmJxKkiSpIiankiRJNTA4BUxOJUmSVBGTU0mSpBo45xQo3JxGxJbAZ4GZwJOB0zLzqyVrkiRJUjmlh/WPBjIz9wPmAx8pW44kSVIhvQE+Kla6Of0ZsHX7fGb7WpIkSVNU0eY0M78I/I+IuA24AXhbyXokSZKKGekN7lGxos1pRLwauCsznwm8EPh4yXokSZJUVulh/ecDXwXIzJuAHSJiWtmSJEmSCuj1BveoWOnm9DbguQAR8QxgZWauKVuSJEmSSil9n9NzgE9FxPVtLW8sXI8kSZIKKtqcZuZK4PCSNUiSJFWh8uH2QSk9rC9JkiQ9ovSwviRJksDktGVzKkmSpEeJiLOAvYFR4MTMvLHvvQOA9wFrgCsz84yIOAY4su8j9szMLSPiOmA68EC7/a2ZuXi8c9ucSpIk1aBXx2zLiJgL7JyZ+0TEs4FPAfv07fJR4MXAPcD1EXFJZp4HnNd3fP+aotdm5vcne/46roIkSZJqsT9wGUBm3gLMjIgZABGxE3BfZt6dmWuBK9v9+50MnPFYT25yKkmSVIN6vlZ0FtA/9L683far9p/L+95bBvzBuhcRsRdwd2be27fP6RGxDXAL8ObM/M14Jzc5lSRJ0njG65rXf+/1wPl9rz8CvD0zXwCsBY6b6GQmp5IkSTWoZ7X+EpqEdJ0dgKUbeG/Hdts684AT1r3IzAV9710BvHKik5ucSpIkqd9CYD5AROwOLMnMFQCZeQcwIyLmRMQTgIPb/YmIHWi+iv6h9nUvIq6JiKe0nzsPmHBhlMmpJElSDSpZrZ+ZiyJicUQsoh2Kj4ijgfvbJPRY4Avt7hdm5o/a59vTzEFd9zmjEXEucG1EPECzuv/Uic5vcypJkqRHycyT1tt0U997N/DoW0ut274YeMl62y4CLupybptTSZKkGtQz57SoOvJjSZIkCZNTSZKkOtRzn9OiTE4lSZJUDZtTSZIkVcNhfUmSpBpUciup0nqjo6Ola5i8+y8aomJbmz+tdAXdTNusdAXdrfhx6Qq6+705pSvoZtUvS1fQ3WbblK6gs9Fbv1S6hG6G6fdHa+2CRaVL6GzkqPmlS+jm4ZWlK+isN+e9VUz2HP23Vw3sL1Vvv89X8Wcei8mpJElSDbyVFOCcU0mSJFXE5FSSJKkGJqeAyakkSZIqYnIqSZJUA1frAyankiRJqojJqSRJUg38+lLA5FSSJEkVMTmVJEmqgav1AZNTSZIkVcTkVJIkqQau1gdMTiVJklQRk1NJkqQaOOcUMDmVJElSRWxOJUmSVA2H9SVJkmrgTfgBk1NJkiRVxORUkiSpBt5KCijcnEbECPBJYDfgIeCNmflfJWuSJElSOaVb9EOArTLzecAxwAcL1yNJklRGrze4R8VKN6c7A98GyMzbgWdExLSyJUmSJKmU0s3pzcCLI2JaRASwE7BN4ZokSZIGz+QUKNycZuZVNMnpDcCbgVuAuq+YJEmSNpriq/Uz8z3rnkfE7cCyguVIkiSVUXmiOShFk9OIeE5EfKp9fhDwH5m5tmRNkiRJKqd0cnozMBIR3wYeBI4oXI8kSVIZI6WXAtWhaHPapqRHl6xBkiRJ9SidnEqSJAmcc9oyP5YkSVI1TE4lSZJqYHIKmJxKkiSpIiankiRJNeiZGYLJqSRJkipicypJkqRqOKwvSZJUgxEXRIHJqSRJkipicipJklQDbyUFmJxKkiSpIiankiRJNfBWUoDJqSRJkipicipJklQD55wCJqeSJEmqiMmpJElSDUxOAZNTSZIkVWSoktPR1Q+ULqGzHk8rXUI3q+4rXUFnoyt+UrqEznoM1/8dr1342dIldDZy6HtLl9DdvUtKV9BJ748PKV1CZ9PeckDpEjobveWK0iV00tvpBaVLGF4jZoZgcipJkqSKDFVyKkmStOkarlG1jcXkVJIkSdUwOZUkSaqBq/UBm1NJkiStJyLOAvYGRoETM/PGvvcOAN4HrAGuzMwzImIecDHwg3a3mzPzhIiYDVwATAOWAkdm5qrxzu2wviRJkh4REXOBnTNzH+AY4KPr7fJR4FDg+cCLImKXdvv1mTmvfZzQbjsdODsz9wVuA1430fltTiVJkmrQGxncY3z7A5cBZOYtwMyImAEQETsB92Xm3Zm5Friy3X9D5gGXt8+vACa8n5vNqSRJkvrNApb3vV7ebhvrvWXA9u3zXSLi8oj494g4sN02vW8Yv3/fDXLOqSRJUhWqXRA1XmHr3rsVOA24CNgJ+LeIeGaHz3mEzakkSZL6LeG3SSnADjSLmcZ6b0dgSWbeA1zYbrs9Iu5t31sZEZvEd25FAAAPqUlEQVRn5m/W7TvRyR3WlyRJqkGvN7jH+BYC8wEiYnea5nMFQGbeAcyIiDkR8QTgYGBhRBwREW9rj5kFbAfcA1xDs3iK9p9XT3Rym1NJkiQ9IjMXAYsjYhHNyvzjIuLoiHh5u8uxwBeAbwAXZuaPaBY9zY2IbwD/ChybmQ8BpwCvabc/FfjMROd3WF+SJKkGE6+iH5jMPGm9TTf1vXcDsM96+68AXjbG5ywFDlx/+3jquQqSJEma8kxOJUmSqlDtav2BMjmVJElSNUxOJUmSajDxKvopYaDNaUTsRrOC66zM/HhEzAYuAKbR3D/ryL5vEZAkSdIUM7Bh/YiYDnwMuLZv8+nA2Zm5L3Ab8LpB1SNJklSXkQE+6jXI6lYBL+XR3wwwj+a+WABXAAcMsB5JkiRVZmDD+pm5GlgdEf2bp/cN4y8Dth9UPZIkSVVxzilQV67rvxFJkqQprnRzujIiNm+f78ijh/wlSZI0xZRuTq8BDm2fHwpcXbAWSZKkcnq9wT0qNrA5pxGxB/AhYA7wcETMB44Azo+INwB3Ap8ZVD2SJEmqzyAXRC2mWZ2/vgMHVYMkSVK96k40B6X0sL4kSZL0CL++VJIkqQY9M0MwOZUkSVJFTE4lSZJqUPkq+kExOZUkSVI1TE4lSZKqYHIKJqeSJEmqiMmpJElSDVytD5icSpIkqSImp5IkSRXouVofMDmVJElSRUxOJUmSqmByCiankiRJqojNqSRJkqrhsL4kSVINvJUUYHIqSZKkipicSpIkVcEFUWByKkmSpIoMV3L60MrSFWzyRv/rK6VL6Ky3y8tLl9Dd6JrSFXQy8vKTSpfQ3UO/Kl1Bd7vtX7qCbnrTSlfQ3ROml66gs96zDyldQidrF36kdAmdjfz54aVLaHgTfsDkVJIkSRUZruRUkiRpU+VqfcDkVJIkSRUxOZUkSaqCc07B5FSSJEkVMTmVJEmqgav1AZNTSZIkVcTkVJIkqQau1gdMTiVJklQRk1NJkqQqOOcUTE4lSZJUEZtTSZIkVcNhfUmSpBp4KynA5FSSJEkVMTmVJEmqgpkheBUkSZJUEZNTSZKkGjjnFBhwchoRu0XE7RFxfN+2v46IhyNiy0HWIkmSpPoMLDmNiOnAx4Br+7YdBWwHLBlUHZIkSVUyOQUGm5yuAl7KoxvRBZn5bmB0gHVIkiSpUgNLTjNzNbA6Ivq3rRjU+SVJkurmOnXwKkiSJKkirtaXJEmqgXNOAZNTSZIkVWSQq/X3AD4EzAEejoj5wNeAA4FZwFUR8c3MfMegapIkSapHPclpRJwF7E2zaP3EzLyx770DgPcBa4ArM/OMdvsHgH1p+sv3Z+alEXE+sAfw8/bwMzPzK+Ode5ALohYD88Z46+8HVYMkSZLGFxFzgZ0zc5+IeDbwKWCfvl0+CrwYuAe4PiIuobk16G7tMVsD3wUubff/28z88mTP77C+JElSDXojg3uMb3/gMoDMvAWYGREzACJiJ+C+zLw7M9cCV7b73wAc1h7/S2B6REx7LJfB5lSSJEn9ZgHL+14vb7eN9d4yYPvMXJOZD7TbjqEZ7l/Tvj4+Ir4eEV+MiG0mOrnNqSRJksYz3mTYR70XEYfQNKfrvqr+AuCkzHwh8D3g1IlO5q2kJEmSalDPraSW8NukFGAHYOkG3tux3UZEvBh4N3BQZt4PkJnX9u17OfCJiU5ucipJkqR+C4H5ABGxO7Bk3bd6ZuYdwIyImBMRTwAOBhZGxFbAmcDBmXnfug+KiEvaearQLIz//kQnNzmVJEmqQh3JaWYuiojFEbEIWAscFxFHA/dn5gLgWOAL7e4XZuaPIuKvgG2Ai/q+qv4o4OPAhRHxa2Al8NqJzm9zKkmSpEfJzJPW23RT33s38OhbS5GZ5wLnjvFRdwF7dTm3zakkSVINJr7F05TgVZAkSVI1TE4lSZKqUMec09JMTiVJklQNk1NJkqQa1HOf06JMTiVJklQNk1NJkqQqmBmCV0GSJEkVMTmVJEmqgXNOAZNTSZIkVWSoktPeU3cpXUJ3a1eXrqCbp80pXUF3a35TuoLu1qwqXUE3q+4rXUF3T9qqdAWd9WbsVLqETkbv+2HpErq79zulK+is94wDS5fQyciL31y6hOHlN0QBJqeSJEmqiM2pJEmSqjFUw/qSJEmbLhdEgcmpJEmSKmJyKkmSVANvJQWYnEqSJKkiJqeSJElVMDMEr4IkSZIqYnIqSZJUA+ecAiankiRJqojJqSRJUhXMDMGrIEmSpIqYnEqSJNXAOaeAyakkSZIqYnIqSZJUA5NTwORUkiRJFTE5lSRJqoKZIXgVJEmSVJGBJqcRsRvwr8BZmfnxiJgNfBp4IvAw8OrMvHeQNUmSJKkeA0tOI2I68DHg2r7Nfwecm5lzgQXA3wyqHkmSpKr0eoN7VGyQw/qrgJcCS/q2vQm4pH2+HNh6gPVIkiSpMgMb1s/M1cDqiOjf9gBAREwDjgNOH1Q9kiRJdak70RyU4gui2sb0AuDrmXntRPtLkiRp01XDraQ+DdyamaeVLkSSJKmYXvHMsApFr0JEHAE8lJmnlKxDkiRJdRhYchoRewAfAuYAD0fEfGBb4MGIuK7d7YeZ+aZB1SRJklQP55zCYBdELQbmDep8kiRJGj41zDmVJEmSc06BClbrS5IkSeuYnEqSJFXBOadgcipJkqSKmJxKkiTVoPLvvB8Uk1NJkiRVw+RUkiSpBq7WB0xOJUmSVBGbU0mSJFXDYX1JkqQquCAKTE4lSZJUEZNTSZKkGngrKcDkVJIkSRUxOZUkSaqCmSF4FSRJklQRk1NJkqQaOOcUMDmVJElSRXqjo6Ola5AkSdLodwbXlPX2rDamNTmVJElSNZxzKkmSVAPnnAImp5IkSaqIc04lSZJUDZNTSZIkVcPmVJIkSdVwQRQQEbsB/wqclZkfL13PRCLiA8C+NP/+3p+ZlxYuaVwRsQVwPrAdsBlwRmZ+uWhRkxARmwPfp6n3/MLljCsi5gEXAz9oN92cmSeUq2hyIuII4B3AauDkzPxK4ZI2KCKOAY7s27RnZm5Zqp7JiIgtgc8CM4EnA6dl5lfLVjW+iBgBPgnsBjwEvDEz/6tsVWNb/3dHRMwGLgCmAUuBIzNzVcka+431uy4i/hr4EDAzM1eWrG8sG7jGnwaeCDwMvDoz7y1Zox5/Uz45jYjpwMeAa0vXMhkRsR+wW2buAxwEfLhwSZPxMuA7mTkXOBz4x8L1TNZ7gPtKF9HB9Zk5r30MQ2O6NXAK8KfAwcAhZSsaX2aet+760tT9mcIlTcbRQGbmfsB84CNly5mUQ4CtMvN5wDHABwvXM6YN/O44HTg7M/cFbgNeV6K2sYxVb0QcRRMaLClV13g2cI3/Dji3/X2yAPibErVp45ryzSmwCngplf7lHMMNwGHt818C0yNiWsF6JpSZF2bmB9qXs4GflKxnMiLiWcAuQLVJ3ibgAOCazFyRmUsz869KF9TBycAZpYuYhJ8BW7fPZ7ava7cz8G2AzLwdeEal/40b63fHPODy9vkVND/jtRir3gWZ+W6g1pXRY9X8JuCS9vlyfvvzrU3IlB/Wz8zVwOqIKF3KpGTmGuCB9uUxwJXttupFxCLg6TQpWe0+BBwPvKZ0IR3sEhGXA0+lGb79WumCJjAH2KKteSZwamZWP4IREXsBdw/DUGJmfjEijo6I22iu8Z+VrmkSbgbeEhEfBp4J7ARsA/y0aFXr2cDvjul9w/jLgO0HXtgGjFVvZq4oV9HENlDzAwDt/7AcR5NWaxNjcjqkIuIQmub0+NK1TFY7TPfnwOcioto7DbdDXd/MzB+XrqWDW4HTaIZEXwOcFxFPKlvShHo0qccraIafP13zz0Wf19PMoa5eRLwauCsznwm8EKh+Tn1mXkWTnN4AvBm4heZnZdgMY81DoW1MLwC+Pgz/Q6vubE6HUES8GHg38JLMvL90PROJiD3aSexk5vdoEvunla1qXH8GHBIR36JpRN4bETUNz/2OzLynnT4x2g6F3gvsWLquCfwUWJSZq9uaV1D3z8U684BFpYuYpOcDXwXIzJuAHSodIn+UzHxPZj4/M4+lSXyXla5pkla2Cymh+fs3LNPFhs2ngVsz87TShWjjsDkdMhGxFXAmcHBmDstinRcAbwWIiO2ALal47ltmvjIz98rMvYF/plmtf03pusYTEUdExNva57NoFjncU7aqCS0EXhgRI+3iqKp/LgAiYgdgZWY+VLqWSboNeC5ARDyDpvaqpwFFxHMi4lPt84OA/8jMtYXLmqxrgEPb54cCVxesZZPU3uHjocw8pXQt2nim/JzTiNiDZn7hHODhiJgPvKLixu+VNPOvLuqbh3NUZt5VrqQJfZJmmPkbwObAcUP0y2ZYXA58vp3u8STg2NobqMy8JyK+BHyr3XTCEPxcbM/wpHgA5wCfiojraf57/8bC9UzGzcBIRHwbeBA4onA9Y9rA744jgPMj4g3AnVR0R4cN1Ps14EBgFnBVRHwzM99RrspH20DN2wIPRsR17W4/zMw3lalQG4tfXypJkqRqOKwvSZKkaticSpIkqRo2p5IkSaqGzakkSZKqYXMqSZKkaticSlKfiLguIj5Yug5JmqpsTiVVKyLuiIjf+Yre9vviq75hvyTpsbE5lSRJUjWm/DdESRpuETEb+DjwPJpvx7qK5lvIft6+/xfAu4HfB34BfCIz39d3/HuAN7XHnj3Y6iVJ6zM5lTS0IqJH89WtS2mazz8AZtA0q0TEHOBzwEmZuSXwCuCUiDiwff9FNI3rK4GnA6uAvQb7p5Ak9fPrSyVVKyLuAHYE1qz31jTgfuAlwLeAmZn5q/aY3YFvA1tl5gMRMTMzf9H3mf8JfCEz3x8RnwB2yMxD2vemAUuACzLzbRv1DydJGpPD+pJq95bM/Hj/hog4GvggTVI6AiyLiPWP2wG4FTg2Il5H0+T2aIbvn9zu83Tg9nUHZOaaiLhtI/wZJEmTZHMqaZj9BngwMzcf682IOAZ4F81w/tczc3VEfLdvlyfzu/8ddLqTJBXkf4QlDbPbgM0iYud1GyJii4jYtn35J8D/zcyFbWM6A3hm3/FLgNl9xz4B2BlJUjEmp5KGVmb+ICK+AXw0Io4CHgQ+TNNgvgD4f8CLImJrYHOaqQB30wzxQ7Oy/9MR8TzgP4B38tshf0lSATankobdETS3gPoxzWr764H/3b73SWAucCdwD/A3wNOAsyNiGfAe4A+BS4EntvtfN8DaJUnrcbW+JEmSquGcU0mSJFXD5lSSJEnVsDmVJElSNWxOJUmSVA2bU0mSJFXD5lSSJEnVsDmVJElSNWxOJUmSVA2bU0mSJFXj/wNEE1PaRVZTGwAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plt.figure(figsize=(12, 10))\n",
+ "plt.imshow(all_weights, cmap='YlOrRd')\n",
+ "plt.colorbar()\n",
+ "plt.grid(False)\n",
+ "plt.xlabel(\"Head\", fontsize=14)\n",
+ "plt.ylabel(\"Layer\", fontsize=14)\n",
+ "plt.xticks(ticks=range(12), labels=range(1, 13))\n",
+ "plt.yticks(ticks=range(12), labels=range(1, 13))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 332,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "He was becoming agitated . \n",
+ "Target: agitated\n",
+ "He\n"
+ ]
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAtUAAALUCAYAAADE787OAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzs3Xmc7mVd//H3sIjKIrhiuBt+wAWTLHPJndTcckFyCxcM99T8aRrumhSaG2pmWWiJW+5amgumqYmopSafcCFzg0QEjAThzO+P7/foxOMIHK5zzn1m5vl8PM6Dme/c98w1w5z7vO7rvr7Xd2l5eTkAAMAlt8OiBwAAAKudqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEE7LXoA24sNX/ucvQW3saXL7L7oIaw/l91j0SNYn846bdEjWHc2nPCRRQ9h3dnhDvdf9BDWpQ1vf/Wih7Du7HjoEUubOm6mGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABu20qC9cVddKclKSryV5QJKdk7wgyTUzxf7JSZ7R3Z+cb7+c5Ord/a1NfK7HJ3nE/DkuleTjSR6b5LZJjkpS3b20Vb8hAADWrUXPVH+7u/dLsiHJ+5Mc3d37dvd1k7wsyd9X1Q0u7BNU1Z2TPCrJ7ebPtX+SyyQ5qrvfMx8DAICtZtFRvdEfJnlNd79z44HufneSeyc59SLue6MkX+3u78/3OyfJYUn+31YaKwAA/B8LW/5xAbdJ8tILHuzuD1+M+34oyfOr6pgkb0ryie4+cwuPDwAAfq7tZab68klOuSR37O7PJ7llpu/lmCSnVdU7quoaW3B8AADwc20vUf39JPtc0jt392e7+8FJrpLk5kl2SfLmLTQ2AAC4UNtLVH80yX0ueLCqHlpVN72wO1bVrapqnyTp7uXuPiHJUzOttQYAgK1ue4nq5yd5UFUduvFAVd0ryZFJLmp99AOTvLqq9pjvt1OS+yf52FYaKwAA/B/bxYmK3f3lqjooyZFV9awk5yT5apI7dPd/rLjpcVV13or3D0vyhEz7Wx9fVcm0V/VHkzx0mwweAIB1b7uI6iSZl20cdCEfv7CLtzx5/gMAANvc9rL8AwAAVq1FR/U+VXViVR24NT55Vd29qk7cGp8bAAA2Wtjyj+4+OdP65635Nd6T5D1b82sAAMCiZ6oBAGDVE9UAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMWlpeXl70GLYPZ5/hB7GN+d1j3fjfMxc9gnXn2F+86aKHsO484DsnLXoI69Ly2WcsegjrztIVr7G0qeNmqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQas6qqvqWlV13qLHAQDA+raqoxoAALYHOy16AFtCVT0syROS7JXkKUnelOQZSR6Y5NJJ3pnkSd19/sIGCQDAmrUWZqp3SHKp7j4gyROTPD/Jg5LcL8mvJrnu/OdRCxshAABr2lqI6qUkr5/f/nySqyW5e5LXdfcZ3X1ekr9Icu8FjQ8AgDVuLSz/OL+7z974dpIdk+yZ5MlV9bvz8Z2S/PciBgcAwNq3FqJ6U76T5N3dffSiBwIAwNq3FpZ/bMq7kjy4qi6bJFV1eFUduuAxAQCwRq3Vmep3JrlBks9VVZJ8LcnDFzoiAADWrKXl5eVFj2H7cPYZfhDbmN891o3/PXPRI1h3jv3Fmy56COvOA75z0qKHsC4tn33Gooew7ixd8RpLmzq+Vpd/AADANiOqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAbttOgBbC+Wzz9v0UNYf/zMt72dd1n0CNans89c9AjWnXveft9FD2Hd8e/oYmz4/EcXPYR1Z8eDDt3kcTPVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwSFQDAMAgUQ0AAINENQAADBLVAAAwaKdFD+DCVNV/Jbldd3+1qg5J8oYke3b32VX1pCT7Jblckl9Kcqkkf9fdT57ve3CSZyXZMclPkjy+u49bwLcBAMAat73PVH80yc3nt2+d5IQkvzq//+tJvp1k90xxfWCSh1TVreaPvyrJXbt7/ySPTnKPbTVoAADWl9UU1b+W5C+T3HLF+y9Ncs/uXu7u05N8Ocl15o+fmuSRVXXN7v5Edz9pG44bAIB1ZFVEdVXtleTcJB9Jcsuq2i/JN5NcOcnbq+qkqjoxyU3zs+/pHkn2TnJCVX2+qm6z7YcPAMB6sF1HdXefnGS3JHdO8qnu/nqSa2da+vHhJK9M8qUk+3X3fkm+sOK+X+vuh2YK75cleeO2HT0AAOvFdh3Vs48n+b0k/zy/f2KSh2WK6isn+Xx3n19VByXZN8luVXWlqvrHqtqjuzck+XSS5QWMHQCAdWA1RPVHk9wsySfn9z+Z5CaZIvv5SV5cVV9Kcpskz5n/XC/JPyQ5vqr+Pcmbkjx8G48bAIB1Yml52QRukiyfdZofxLZ2/nmLHsH6s/Muix7B+nTatxY9gnXn7CccvughrDuXPebdix7CurTh0+9d9BDWnR0POnRpU8dXw0w1AABs10Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwKCdFj2A7cYOOy56BLDVLS0tLXoI69Ly5fdZ9BDWnUvf5XaLHsL649/RhTjr+X+66CGsO3sedOgmj5upBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGiWoAABgkqgEAYJCoBgCAQaIaAAAGXWRUV9Vtq+qr22IwF6Wq7lVVr1v0OAAAYKWdFj2AzdHd70jyjkWPAwAAVrrYUV1VL0pyjyQbkjwsyQlJjkpy5ySXSvLn3f1H821/OcmfJ9k9yXeTPKS7v1FVByR5dZIrJPlxkqd29weq6rZJXpjkX+av8YMkj0lyZJL9k7ymu59VVQ9J8qDuvmNV/XWS/0xyiyTXS/IfSe7Z3WdX1Z2S/EWSHyV5SZIXJTmgu0++BD8jAAC4UBd3TfW1kny2u6+X5MVJXpnkKUmun+RGSW6Q5L5Vdbf59m9KcsR8+3ckObqqdpiPH93d+yU5LMmxVbX7fJ8Dk7wzyXUzhfvRSe6a5I5Jnl5Vl97EuA5Ocsh8nysluVdV7ZjkmCS/2937J9k3ya4X8/sEAIDNdnGj+sdJ3jK//ZYkv5Tk7kle1d3ndPf/JHl9kntX1fWSXLG7/36+/dFJ7pPk2kn2zhTW6e7PZppp/pX5dj/s7uO6eznJl5N8rLvPnt/eMVM0X9D7uvsH3X1eki8muUamWetdVnz9V2zG9wkAAJvt4sbmad29YX77zPm/eyV5SVWdWFUnJvm9TDPCV0xyxsY7dvd53f3jTFH8wzmaNzo9yZXnt89acfz8TEs3Mt9+Q6awvqAzLnCfHedxnb7i+Hcu5vcIAACXyMVdU73Xirf3nP/7/SQv6O73rrzhPFN9+araobs3VNXOSfZJcsp8fGlFWF9hPr4lnZlktxXv772FPz8AAPwfF3em+rJVda/57fsmOT7TMpDDqmrHqlqqqiOq6s5JTkryrST3nm//8EwnLZ48Hz8kSarqFpmC9zNb4htZ4aQkO88nPybJI5Ms//ybAwDAmIsb1Scmufm8zOOJmXbmeGWmNdFfnj++f5JPzLPQByf5w6o6KckDkjxqPv7bSR5bVV9J8vIkB8/rsbeY7j4nyaOS/HVVfSHTriAbIqwBANhKlpaX13ZrVtWumdZn79ndZ/y82y3/zw/X9g9ie7Th/EWPYN1Z2nFVbU2/Zixv2HDRN2KL2nDsixc9hHVnhwc8edFDWJfO+M3bLHoI686eH/vXpU0dX5O7YlTV8VV1yPzuIUm+cmFBDQAAI9bqtNUTk7yyqp6X6cTFQxc8HgAA1rA1GdXd/YkkN170OAAAWB/W5PIPAADYlkQ1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg5aWl5cXPYbtw9ln+EEArBHLGzYsegjrztIO5ukW4TG7Xn3RQ1h3Xrl85tKmjvsbAAAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMWlhUV9WHq+rA+e1HXMz7HFJVe2zm17laVS1fkjECAMDFsdOivnB33yFJqmrHJEclee3FuNtzkvxzkjO34tAAAGCzbLGorqrDkvz+/Dm/m+TBSU5J8vokt0zy5SSfS7J3dz+kqk5O8qAkz01yuao6McldklwqyV8muUKSnZM8o7uPrarXJakkx1XVQ5J8Kckrktxs/prP6+6/msfysCTPyhTff7ulvkcAANiULbL8o6qunOToJAd1975JvprkGUkOS/ILSa6Z5BFJHrqJuz8syfndvV93fyPJi5K8t7v3nz/2l1W1c3c/bL79bbv7E0lenGRDkv0yhfVzquqGVbVXkpcnuXN332j++gAAsNVskaju7lOT7NHd35oPfTzJdZL8epK3dfd53f2fSd53MT7dPTMtB0mSTyS5dJKrbuJ2d0/ysu7e0N3/neTtSe6dKbBP6u6vzLc75pJ8TwAAcHFtkeUf87ro51bVPZLsmGT3JP+RZK8kP1hx028nufpFfLo7JTmiqq6UaSZ6KZuO/z2TvKWqzpvfv0yStya5fJIzVtzu9M37bgAAYPNsqTXVhyS5R5Jbd/f35908HphpTfNuK263qRnnn6qqnTOF8f26+/1VtUuS//05N/9Okt/q7i9d4HPcJcnlVhy60mZ9JwAAsJm21JZ6V05y8hzUV0hyv0wx/Zkk96mqHarq6plORLygnyTZoap2T7Lr/Oez88d+L8m5+VmYn5dphjpJ3pXkkUlSVTtV1UvmLfo+Ox2qfefbHbqFvkcAANikLRXVxya5QlV9dX77iEzLPK6a5MdJvpbklUnelOSCe0Z/N9Pa6W8muX6SP0ny+ar6/Hy/dyZ5b1XtmuQtST5ZVffLdCLk5aqqM+0ssmOSf5vXV/9+kg9V1ZeS9Bb6HgEAYJOWlpe37nVRqmqpu5fnt49KslN3P3GrftFL4uwzXCAGYI1Y3rBh0UNYd5Z2cJHmRXjMrhd1qhpb2iuXz1za1PGt+jdgPnHx+Krapap2S3LXJJ/aml8TAAC2ta19RcX3JfnNJF/JtJPHe5O8bSt/TQAA2Ka2alR39/mZTyYEAIC1ygIoAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYtNOiB7C9WD77jEUPYf259G6LHsH6s2HDokewPi37uW9zPzln0SNYd5Yvs/uih7Au3fXyuy56CMzMVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg9ZFVFfVh6vqwEWPAwCAtWmnRQ9gW+juOyx6DAAArF3rYqYaAAC2JlENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAIFENAACDRDUAAAwS1QAAMEhUAwDAoJ0WPYDtxdJlL7foIcDWt8OOix4BbBs777LoEcA28e4f/GjRQ1h3fvPnHDdTDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg0Q1AAAMEtUAADBIVAMAwCBRDQAAg3ZaxBetqmslOSnJ15I8PMkDktwxyXKm0D+mu18w3/a4JNdLcuYFPs3R3X30BT6+lOT8JK/p7pdV1fOSHJrk69192637XQEAsF4tJKpn3+7u/arqyCR7JrlRd59bVVdJ8vGq+lp3v2m+7VO6+28u5HP99ONVtXeST1VVd/czqurDSZ69Nb8RAADWt+1h+ceNknyuu89Nku4+Jcmtkrz9knyy7v5ekrcm+Y0tNkIAALgQi5yp3uj9Sf64qvZK8r4kx3f3qYOfc+ckPx4eGQAAXAwLj+rufmVVnZrkYUmelOTcqvqrJE/r7o1h/CdVdcQF7nqX7v7GBT9fVV0nycFJ7r01xw0AABstPKqTpLvfmuStVbVLktsneUWmmeanzTe5qDXVG6N7KckPk/x+d39ma44ZAAA2WmhUV9XOSe6a5D3dfX53n5Pk76vqZUnutBmf6qKiGwAAtppFn6h4XpIXJHl6Ve2YJFW1R5J7JPnYIgcGAAAX10Jnqrt7uarukuSoJCdW1fL8oTck+dMVN93UmurPdPfvbItxAgDAhVn4muru/maSQy7k47e9iPtf6McBAGBrW/TyDwAAWPUWGdX7VNWJVXXg1voC82XKX7+1Pj8AACTJ0vLy8kXfaj04+ww/CABgVXnkrldb9BDWnT9bPmtpU8ct/wAAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABolqAAAYJKoBAGCQqAYAgEGiGgAABi0tLy8vegwAALCqmakGAIBBohoAAAaJagAAGCSqAQBgkKgGAIBBohoAAAaJagAAGCSq15iqWlr0GADYcqrKv9WwCviLugZU1f2q6v5J0t3LVbWDuAa2lqracf6vx5ltoLs3JH7erD1VdbtFj2FLckXFVa6qdklyepJLJ/lSkod29wnzx3bs7vMXOb61rqp22PgPHltXVd08yTWSvDPJufMTSL/jC1RVRyf5dHf/zaLHshZV1V5JvpjkSd39lvnYjkk2dLd/vFnVqupRSZ6X5Lgkr+zujy52ROPMVK9+v5jkft29Q5J3J/l0Vb21qnbfGBteOhy3YmbuylV1naq6TVXtIqi3jap6dJIXJdmtu89JsnOSCOptb8Xfhb2TXDvJ66vq41X1a4sd2Zp0ZpJXJ3lNVX2kqvbt7vM3PqFc9ODgkpp/f2+T5GFJPpLkGVX1qqq67mJHNsZM9So2x/JuSW7S3R+bj10uyV8nuVuSF3T3s1fcfsnsxubbOBtdVXsmeUuSnZJ8J8mNkxzc3ScudIBrXFVdOskXkjy4u4+vqocmOSjJHZM8srvfvtABrlNV9fFMwXd+kjsleUiSY5Ic0d3fnm/jMecSWvmzq6rfTfLcJFdO8vokh89PLr1axqpVVfuseKy4aZIHJDkwyQeSvKS7f7zI8V0SZjBXtxdlein89slPl3uc0d33SvLrSe5XVd+rqnsnP11vbXZj822Mgpcl+XJ33z7J3ybZJck3quqWVXWphY1u7fvlJJ+ag/reSV6S6ff+D5Lcsap2Wujo1qGq+q0kl+/uN3b3m7v7YZn+P90jyeeq6snJ9JizyHGucktJUlV/nORmSe6e6Ynk9ZKcXlWHJz9bb824+Qk8W1lV7V1Vj8k0O/20qrpFd382yQuS/EmS6yd518ZzxVYTUb1KVdU9Mr108qQkr5gP715Vd6qqX+zuT3f39TOFx+uq6tNVVV4u33zzk5HdkuyZ5I3z4cOTvHqeLTo4yVUXNb514D+T3KqqTso0k3HEvL70xEyx4Xd62/v3JBuq6tYbD3T355M8Lcnrkjy5qj5aVXssaoCr3fzq2KWT3CrJUd19fHd/JMktkxyd5NVV9SUTJVtGVd0o01IEtr7XJakkG5JcKcnDq+rK3X1aplnqpyV5c5IHVdWx8/k0q4KoXr2emeSZ3f2F7v7+fAbte5K8Msm/VtUfJkl3/3V375nklCSf9QB8yXT3j5L8U5LHVtUTkly1u18yf/hWSa61qLGtdd39rSQ3yPQE8eFJXjV/6I+SvNxs6LbX3f+R5G+SvLmqXjE/6UySWyf5YJKrJzk7yXUWNMS1YkOSryV5VVVdJfnp7P9RmZaDPKG7z3fezBZxYqaQYyuqqrsl2bO7H9/dj870BPF6SR6VTOfJzI/5f5PkqUk+nuRNVXXHRY15c/iLuArNz6hP6e73ze8fmORNST6X5F5JHpjkgKraacVL4ydkmuEzq3cxbVzSUVUHVNUjklwtyXlJDknyofmExcclOWPjmna2nKq6SVX9aVW9KMlNk3ygu89I8ltVdVySk7v7mIUOcp1YcXLi1atekn2UAAAQs0lEQVTqQVX1m0nemunvwhWTnFpVH8p0nsFx3f2TJHsnueaixrwWdPe5SR6bKayPnV8yT5KbJDmouz80384SkEHd/ZN5ppSt69mZlnlkPtn/60mekmTvlZN+3X1ud38pyWeSnJPkEwsY62ZzouIqVFW7Z1pT+m9J/ifJryY5q7vvM3/80knem+Sx3X3iHNb7dvdXFjXm1WqeAfrPJB/LdALFfkk+n+QfkhyQ6ef/HD/bLa+qvpzprPDrJrlcppcF35Xkq0n2T3Li/AoC20hV/WOmiD4701Zvn8m0HdZlM81OfyjT35GHJvnV7r7VYka6+lXVAUlulOS/klwlyeWT3C/TE8wvJ3lRd7/dtpKsFlX1wCQvzTT5d1J3nzIff3OSL3X38y54cnNV3SLTrk8fXMigN5MTfFah7j6rql6SaZbofkmen+QdK25y3yS7bNyVorvPSyL6NkNVvTTT/pm/kOQ93f3oec/Yw5M8OdPuKg/u7n9b4DDXrKo6JMnXu/tx85PC72U6ceugJJdJ8nfziS1sZSt2v7lNkp90902q6pczPf78ZqalOR/MFNc7JjkyyWeTPHJBQ161NgbyvNvHgzOdJH1Skr2SPCHTS+L7J/lhd381sa0kq8pHM239++wkJ1TVMZkmTa7d3YfMt1nKzzYHSHd/clsPcoTlH6tUd783yeOTXL+7X93d36uqPect9f4g08spKWuoN9u8PvSAJN9K8sJMJ2Ul0zKPI5P8SpJvJPm4XT+2vPmVlttlColk2u3jtZlO0Dor084rf7aY0a0/K5YW3CDJ1+djJ3T3U5L8VaZXER6XZI/u/nF337W7nzW/dMtmmIN6KdP60sO7+9aZlpyd1t3fTPIr3f3ZjUENq8W8k8e53f3wTDuX7Zfp3ICXZMVa9tW+lElUr0Lzg266+/R5jWmqat9MW769Lck/dPen5pdRzGJspu7+0bxt3l0yhd2fVtVDNv5l7+5vdPdvJbnhvOaRLev2SXZP8vL5/Vskefv8kuAXk3y8u09f1ODWk/ml11TVDTK9ZHu3eQusGybJfF7HE5Mc2d2nVNXOixvtmnHtJGckObOmC+zcMskLq+p6mU6UvtpCRwebaX5l62mZlkumu/+hu++Z5O2ZJq8OqKoHVNW1FzjMLcKa6jWiqq6UadnHezKdxPgTFwUYN6+p/t1MS2y+neSw7j5+saNa26qqMm0Ved0kh2VacvPDJD9I8vjuvtECh7duzCdA3zPJczKdpHtKprXSd0ry35nOyv/ofKY+A6pqp+4+b8WJ5a9N8s1M69T/q7ufVVXXyLTM7+aezLOaVNX7kry7u18zv3+lJP/T3WfPbz8xyX2SvLC7/3pxIx1npnqVqOnS4wf+nI8tdfd/Jzlh/gfuvGT1v4yyPejuDd39Z5kuB//RTEs+PuCCI1tHVe3ck8OTfDhTXFwzyYMyXVjk2Qsc3nrzjUzro++b6QTEx2W6YuITk5yeaeb6mWthdmk7cJt5V6cXZNpe7M8ynS/zkCT/XlV3z3Sl3Nd397mW9bFaVNX9kuy6Mahnb0qy8aJ0/93dT8/0yvDb5vssbfOBbiFmqrdz8y/XDpl2PTgo0yVq/2Dj1j9zhPxk3uLqmd39a4sb7do3/8N3cHc/c9FjWWuq6i5J3pfkGZlmpb+e6Xf/Ckk+OW+9xDZQ//cS2VfJNDt9j0zrp/+8u986X/jlLt39tAUOdU2Yf5ZvzXQhjP27u+fjT0py10zndXyvu1+wuFHC5quqf0vyxO7+8Pz+3ZI8vbtvseI2RyZ5Q3d/eUHD3GJE9SpSVftnmq3YP9OM3Us3zkbP+/a+qrvfYoslVqN5zehRmYL6LZn2W79Jpu3azkpys+7+38WNcP2pqt/I9MTmg5lerblnpif3pyV5WXd/er6dpWaDqurPM50MerNMJ4A+sbt/NL8qtmd3f3++nZ81q0JVPSTJH2d6pfFL3f3dqvpkkud39/vn29whyR93900XN9ItR1Rv56rqmkkekeT47n7XfOzgTFchOjPJ/ZPcMMl9u/tuCxsobAHzKwEvzzQz93uZ9mC/Y5IfdfefLnJs68WKLfQemWmpxxUznTj3/zLtFX6DTGurz5x3AOESusArAhtfdbxpkr/MtOzp9zNdlfKU7n75hXwq2O5U1S9k+h3+5ST/nOmx5GrdfdcVt/mnTFfGfdtamBC0pno7Np95/4ZMM9N7zFuNpbvf2t1XybQU5F+SvC7JH873sdaOVaWqrlRVr6mqeyU5P9NM6ImZHow/1d3PzbTtEtvAilnQWyc5pLuvkOnS8MdkuujUTzJt2/mc5Kcn87KZ5icvy1V1qZqu2Pq6qrrjvGXejTM9oXlmpv3A3zPfZ9WuNWVdOrW7fz9Tn1wn02N7V9V+SVJVD8904bq3JWtjz3UPhtu3P0nyuu6+T3e/obt/nEwPxlV13e5+Xqb1pg/s7n+dH6RX/S8l68eKSNgtU8Q9LdMJilfNtPTjuQsa2rpUP7sc+X5JPpdpyUe6+0WZHmu+menCLnfs7o3bY1mKMOalSQ7O9GTl/VV1TFVdr7v/Ksm+SW7a3d+YZ/G8tMx2r6puUFWvSvKPVXXF7v7n7r5/pieKN0hyZFU9JsnT5z9rZkLQ8o/t1HxFuUd19203cdnOvTKtPf1Id79xxfElD7qsVvPWSkuZLsP8gCR7J9m7u2+40IGtExsfP+bdPP4lyY8z7QBybKat8zaePLdfpqtd2tZt0Pyz/vskv9zd/1NV7850afJfyHTluX1WntAFq0FV/XOm39+vdvffza+yn59kj0w7Bz0q0xVC39/dv7eWzhOwLdj267wkn0mSC4Zyd59eVadmukjAyuOCmlVr3hYySd6facZu7yTnLHBI68qKx4+DM+3L/uYkj0lyhyQ3rKpPJDmuu0+05GPc/DPcMcl3k+ww735zle6+dlW9NlOA/HSJzVqJDta2qjos0x7Uf7zi8B9mehxZSvLh7j5i3rv6u4sY49bkgXE7NG/y/79JDq2qX7rAxy4zv/mVzC/NwlqyMdi6+3uunLhtzSeK3iPTDNMp89aRr800y3Ropr2TLfkYsOIxfNck30/yX5kmUe6ZadebJPlkktd09wcSP29Wh3k5352SvGJ+f7eqenamC6i9I9NVn29VVQd298ndfU6ytn6/RfV2pqbLjb9x3m7mPUkeX1VX3vjxFVuKPS7zA7BZI9aStfQAuwrtMf/3b6rq0UnS3R9M8tQkf5dpaz0nzF1CVXWpJI+sqj/K9LM8u7t/Z35c/1aSO86P989P8qMFDhUuiask+WKm3+PbZDqh+cFJHtrdR3X3m5L8W6bdQNYka6q3M1X1okyXaN4nyYYkf5vkgCQvTPIfmbakuVmSq3b3vRY1TmBt2MQ5G5fN/A9hphnUZ3f3hxY1vrWmqnZO8tVMj/EP7u5j5+PXzLRl4blJPtjdT3aeDKvFfJ2Bv8h0svlzM50Tc1qSP7vAuV9fSPLU7v7AWvz9tqZ6+/OcTBe/+HimqyjeJdPLKU/LNHNxnUwvoTwvmc6YteMHcEms2JN65yR3S3L3TNvmfWr+c98kr62qz2Q6eXTDWvtHcFuZr3q7U3e/u6remeR7SV5cVUdkOin9n5LsV1UHZprtS6Y1qH7erAaPSHKLTL+79850su33Mu1xnySpqicm+caKZU1r7nfbsoHtTHef1d1/lOmCF1fItF/vlbv7lt19pyQ36u6ju/u78+0FNTDqJUl+O9Ms6VPnP/vMa6ofmuRtHmuGXTrJF6vq/pl2U3lhpgt3vT/JB6rqHVV1QpIbd/dPEkuhWFWem+lKz5/LtB3wafM5MbtU1eHzFUN/O9Njy5rZQu+CLP/YzlXVbZM8K8nlkhzZ3W9Z8TGz1MCQqrpOpm3dDpy3dbtcpiVohyV5bHe/Y8Vt19zLtdvCBa6c+IIkv5PkuCQv7e4Tqqoy7de7W3ffZ3EjhTFVda0kz0hyu0wXpntBppjeJ8kHuvuLa3k3G1G9CszP6H4n076Op8xvXz7JteYTGgEukaq6fpLndvd9L3D8sCT7dvdTFzOytWe+gtzfJdkl02zeryd5Y5JXd/e3N06UmDBhtZtPVDwiyV5Jntzdx6342M4bX41Zayz/WAW6+/z56lq3zXRRhk8k+VKmPU4BRpyc5OpV9a6VOw1lusKfC+8MWnGVyn0zbUn4rkyXa35opnXqt0zyqaq628aQFtSsdt39sSR3TvKqJK+oqn+sqr3nJ/EHLXZ0W4+oXkW6+/Tufkam/Uyf0N3vWfSYgNVl5XZ4VbVrkhtnOiH6x0k+XVUvq6qNax9/ekL0Isa62s3LPjYG8osz7fpxXqYL7Lwh04mLt8u0FrUXMkjYSuYJwdcluXWST2cdTAha/gGwjqxYYvC4TE/QT+vuQ6pqtyQ3T/KwJN9J8tnuPtY66nFV9QdJfqW771NVu2R61fHeSX4j08z1Md39+QUOEba6qrpBkjt098sXPZatRVQDrDNVtWeS4zMtP+juPnM+fvMk/97dK7fBEtWD5qi+TZKHdPcp87FrJXlNphnqvZK8Yb7QDrBKWf4BsP7cMMkXu/v47j6zqjZes+DgTCdC/5Sg3iLememKiQ+sql+tqt26++RMlyj/TJKzMl09d5cFjhEY5OIvAOtPJ9m/qv6ou5/e3efNx/8900lEr1jc0Fa/FUtsbphpO7GrJLlGkqsnuVaS86pqryS/1N2HZbos/GeSXDvTtQmAVcjyD4B1qKoOyHR11stl2t7tC0n+NtN++G9dy3vJbitVdVyS7yb510xXm9styTmZrph740wng/4g06sDv93dt1rMSIEtwUw1wBq3Yub0BpkuyrBrkjMzXXDkVkmOzLRd50e6+62Jq/mNqqp7Jtmhu+9fVbtnuozz3ye5SaaTQf8iyalJPpbp/8XhixorsGVYUw2whl1gW7e3Jbl+kstmmh29ZXcf1d3XynQS3f+b7+PfhnFnJrni/PZTM20pdkSmtdWvTXJsd5+d5D5J7tPdX17IKIEtxkw1wBq24vLYj0ry5e5+dFVdJVNU/21V/UaSb3T3SSvuY5Z63L9k2i4vSW6W5Cnd/cOq+k6myzX/Z5J09zcXNUBgyzIbAbA+/CTJzvPbz0nyru7+bqblH/df2KjWqO4+u7ufNr/79STPq6rDM11l7iuLGxmwtYhqgPXhX5OcWlVPSXL7TEsRkumEuVOT/3u1RbaoVyU5JdNSjxd29zmuUglrj90/ANaYOY6XuntDVf1Kkkcm+aX/394doloVRXEc/kXR4gCMD3YxPUVQdAQKTsIqBhE0OgKDyEOwCQ7CYFF4TfMGHYJNQdRgOFe4/Ty4cN73xbNPWPHPYu21Wy7Jfa2+VB+qa9XVOeetgxV7TowxLlS/jdbAdulUA2zPxb3w9rL6Xj2p3lVH1d3qcvW5elDLhpAD1HluzDl/CdSwbTrVABszxnhR3WwJ1Mdzzsd7Z3d230/mnK8PVCLA5uhUA2zPm+q0ZZXb/THG9f8Hc86PLaH69t7z5ACsJFQDbMxu5/HT6mH1rXo/xni298uN6sec86/LiQBnw/gHwIaNMS61bJ14VF2pPrWsdHs+5/zjOXKAs6FTDbBhc86f1dvqXvWqOq5OBWqAs6VTDXBO7DZ8HM0556FrAdgaoRoAAFYy/gEAACsJ1QAAsJJQDQAAKwnVAACwklANAAArCdUAALCSUA0AACv9A1yGefem49xSAAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "random_annotation = random.choice(annotations)\n",
+ "sentence = random_annotation['sentence']\n",
+ "target = sentence[random_annotation['target'][0]:random_annotation['target'][1]]\n",
+ "print(sentence)\n",
+ "print(\"Target:\", target)\n",
+ "for link in random_annotation['links']:\n",
+ " print(sentence[link[0]:link[1]])\n",
+ "bert_tokens, tokens_1, tokens_2 = convert_to_bert_tokens(random_annotation['sentence'], \"\", max_len=20)\n",
+ "input_ids, input_mask, segment_ids = convert_to_bert_inputs(random_annotation['sentence'], \"\", max_len=20)\n",
+ "crop_len = torch.sum(input_mask != 0)\n",
+ " \n",
+ "_, output = model_base(input_ids.unsqueeze(0), segment_ids.unsqueeze(0), input_mask.unsqueeze(0))\n",
+ "output = output.squeeze(0).detach().numpy()\n",
+ "visualize_single(output, random_annotation['sentence'], \"\", max_len, 6, 5)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 344,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['[CLS]', 'he', 'was', 'becoming', 'agitated', '.', '[SEP]']"
+ ]
+ },
+ "execution_count": 344,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "bert_tokens[:crop_len]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 348,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "([,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ],\n",
+ " )"
+ ]
+ },
+ "execution_count": 348,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAEqCAYAAABZbejwAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XecZEW5xvHfzC5BwkpyZcmC8EpQ4kVQMogiXkUEFS8SRARBkoKgBEEFkSyIJOEiICg5CbKiZJAkrgF8EJAkSJKcl537x1t9txl3d2Z2uk+f6X2+fPrDdJ/Tp6pne96urnqrqqevrw8zM2u/3k5XwMxsRuGAa2ZWEQdcM7OKOOCamVXEAdfMrCIOuGZmFRnd6QqMJJPu/0PHcuh63jFnp4qG2cZ0ruwXn+lc2cCkO3/XsbJ719+iY2VPuvCEjpU9auv9eobz/B17xgz67/TEvheGVdZQOeCaWVep89d2B1wz6yq9PZU2WofEAdfMuopbuGZmFRld3wauA66ZdRd3KZiZVcRdCmZmFemtbwPXAdfMuotbuGZmFelxH+7AIqIPeBO4R9LyFZZ7HLA9MAuwrqRrqyrbzFrPWQqDt2FzwIuIJYH9gY8AcwFPAlcAB0n6VznnWuAOSXtO6YIRMQdwELAJMA54C/g9cICkWyTtAuxSAr6ZjXB17lKobd0i4gPA7cDTwErA7MDGwGLAbREx9yAvdVZ5/kbAHMAiwJ3AbyJiXIurbWYd1tvTM+hb1erWwm12LHCNpK83PfaXiPgU8BNgIeDZQVzno8B2ku4t95+NiG8DdwOTWllhM+u82rYiqWnAjYh3AWsDG/Q/JukN4MtDuNzdwNci4o5G0JU0CTijFXU1s3pxWtjQLV7+rxZca2vgbEARcT9wPXAlcLGkN1twfTOrkTq3cOtcN4BRw72ApL9I+gDZj3sCMB/Zup1QWtJm1kVG9/QM+la1ugbce4E+YNlWXVDSXZKOlPRJYClgXmCnVl3fzOqhdwi3TtStdiQ9C/wW+Eb/YxExU0TcGBEfH+g6EbFcRBwXEW9rKUt6hOzb7eBWBmbWDr09g79Vra59uAC7AzdFxHnAnsAjwDLAEWR61/WDuMYTwBbA7BHxXeAhMr1sc2B1Mj/XzLpIL/UdNatlCxdA0l+B/yJnn90GvARcDPwRWEvSS02n7x4Rr/W7fVnSU8CHyVlkNwKvAI+RA2mbeFaZWfdxC3c6Sfo78IUBzllngOMC/qeF1TKzGvPUXjOzirhLYfDGR8SEKgssg2qvVVmmmbWPuxQGQVJHPpYai9d0omwza736tm9rFHDNzFrBU3vNzCpS5z5cB1wz6yrOUjAzq0iN460Drpl1l04sLD5YPX193llm0F55vmO/rBn23+nVFzpa/DnvXaVjZX/hsb93rOy+V57vWNk98y0yrIh57tzvHvQfy2effaLS6OwWrpl1lfq2bx1wzazLOOCamVWkzn24Drhm1lXqtl5BMwdcM+sqNW7gOuCaWXfpqXEvrgOumXWV+oZbB1wz6zKjWhhxI+JoYDVyU9vdJN3edGxnYEvgLeAOSbsPdL069y+bmQ1ZzxD+m5aIWBtYUtLqwHbAsU3HxgB7AWtKWgNYJiJWG6huDrhm1lV6hnAbwPrkPopIugeYuwRagDfKbY6IGA3MBvx7oAt2ZcCNiL6I+ESn62Fm1evpGfxtAPMDTzXdf6o8hqTXyF2/HyB3A79V0r0DXbArA66Zzbha2MKd0qWB/+9S+DawFPAe4IMRsfxAF3DANbOu0kvPoG8DeIzSoi0WAB4vPy8NPCDpaUlvADcAKw90wW7OUhgbEVcBawEPA5+XdBdARLwfOJr8Bb0FnA/sXr4mmNkI1sItdsaT3QYnRcRKwGOSXizHHgSWjoh3SHoVWAW4YsC6taxq9bMDsCvwLrKP5YcAETEb8GvgOuDdwErkL+s7nammmbVSq7oUJN0M3BkRN5MZCjtHxDYR8WlJTwCHA9dExI3AXZJuGKhu3dzC/bkkAUTERcC+5fGNgZklfa/cfzgiDgGOB75VfTXNrJVaOdNM0j79HprQdOwk4KShXK+bA+4/mn5+FZi1/LwEMG9E9O8+GBURs0h6vZLamVlbeNfezpjaqu+vApK0dJWVMbNq1DjednUf7tTcByzWlMBMRMwdEe/sYJ3MrEXamBY2bN3cwp2aq8jUjqMj4uvAzMDpwDPAVh2sl5m1QJ0XIJ/hWriSJgKfAhYnA+9fgCeBr3WyXmbWGr1DuFWtK1u4knr63T+dbMU27v8ZWLfaWplZFerbvu3SgGtmM66eGncpOOCaWVepb7h1wDWzLlPnQTMHXDPrKp74YGZWkZ4aR1wHXDPrKjXuUXDANbPu4oBrZlYRp4V1ib63Jnau8E6WPdMsnSv7lRc6VzbwqfWW7FjZnXy/Tbrrmo6VPeojWw/r+b3uwzUzq0aNG7gOuGbWXZyHa2ZWkRrHWwdcM+suHjQzM6tI7ygHXDOzStS4geuAa2bdxV0KZmYVqXG8dcA1s+7itDAzs4rUON464JpZd/HUXjOziriFa2ZWES9A3kIR8TCwm6SLyv0rgUUlLVPuLw/cCqwIHA6sBowCrgd2lPR4Oe+bwE7AWOBfwNGSjqv45ZhZi9W5hdvb6QpMh2uADwFExChgZeAdETFvOb4mcDPwY+A5YEFgMWAMcER53oeAg4D/ljQb8DngoIh4f3Uvw8zaobenZ9C3yutWeYnD9zvgw+XnlYC/A7cBa5TH1gR+C3wC2F7S65KeBy4DVinnzFX+/xKApNuB+ST9uf3VN7N26ukZ/K1qI65LgQy4J0fErMDawA3AY2TAvYQMuEeRLd9DShfDLGS3wj/LNX4LXA0oIq4FrgJOB56p7FWYWVvUOUthxLVwJT0CPEy2VtcCbiS7ENaMiCWA2YD7gCuAO8j+3VmBPZuu8bqk/wb+iwzY2wL3RMR7qnwtZtZ6PT09g75VbSS2cCFbuWuRXQtbAy8CAWwIXAssBcwJHC7pufKclRtPjojRwBySJgATIuIHwD3ApsCRFb0GM2sDD5q13u+ArYB/SnpW0kRgAvA1srvgIWAS8KGImC0ivkIG5Lkj4h3AXsC1EbFYuV4A85ItYzMbwdzCbb1rgHOAE5seuwn4NnC1pMciYq9yfBTwM2Az4DpykG0JYCHg1oiYE3gc+KGkS6p7CWbWDj0tbEZGxNFkamkfmY56e9Oxhck4NDPwB0k7DnS9ERlwJT1Jv9a5pH2BfZvuH0UOnjVbvOnnncvNzLpIz6jWRNyIWBtYUtLqEbE0cBqwetMpRwJHSrooIo6PiEUkPTyta47ULgUzsylrXV7Y+sDFAJLuIbskxwBERC+ZEXVpOb7zQMEWHHDNrMu0sA93fuCppvtPlccA3kUO1h8dETeWgfcBOeCaWXfp7Rn8bWh6+v28IPAjcj7AihGx8YBVG2qJZma11rouhceY3KIFWIAcYAd4GnhI0v2S3iKzo5Yd6IIOuGbWVXp6ewZ9G8B4MruJiFgJeEzSiwAlFfWBiFiynLsyoIEuOCKzFMzMpqZVWQqSbo6IOyPiZjKvf+eI2AZ4vqxWuDtwehlA+zO5Xss0OeCaWXdp4YQGSfv0e2hC07H7mLxo1qA44JpZd6nx4jUOuEPRO6rTNeiITkyBbOibZ8GOlQ0w60brdq7wDr7fXvx+/zlD1ZnrI1sP6/mdfL8OxAHXzLqLW7hmZhVxC9fMrBo9oxxwzcwq4V17zcyq4i4FM7OKuIVrZlYNp4WZmVWlRVN728EB18y6ilu4ZmZVGcl9uGVn238A75f0l7bXaDpExKLk0mgrSbq70/Uxs85xC7fNJD0EzNrpephZDYzkFq6Z2YjSJS3clSLi58ASwJ3AVpIeKlsJ/wBYDngZOBU4QNIkgIjYA9gNmBe4DfiqpHvLse2AbwDvAf4JHC7ppHLsdODVUvb/AM8C2wIrAHuTu1XsI+nU/t0eEfEgcDCwCbnf0OPAVyRdU669MfATYD7gIuBvwCaSVhnC78PMaqhVC5C3w1BqtgPwSWAh4A3gjIhYCLgc+CkwD7mt8JbA9gARsQmwL/BpMuDeA1xYjm1MbsC2MzAnuXr6cRGxXlOZnwV+Te6Q+QfgzFLOIsCPgaPKautTsidwYDn/NnIPeSJiHHABcFw5Nh7YYwi/BzOrs/ZtIjn8qg3h3BMkPSTpOeAIck/2bQFJOk3SxDJgdSywTXnOtsAvJd0l6Q3gO8B3I2Im4MvALyRdU557OXA18LmmMh+QdImk14GrgHHAD8r9XwFjgLFTqe8Vkm4v5V4CLF0eXw94DThG0uuSzgD+NITfg5nVWAu3SW+5oXQpNI/+309uE7wuuT3wa03HeoAnys9LADc0Dkh6BjgXICIWbz5W3AdE0/1Hmn5+DXhO0stN92Hqg2X/aPr5labzxgGPlk3gGm4jW+dmNtJ1yaDZpKafG6/oIWC8pI2m8ZyptaJnmcrjfVMpc0r3p2Vq5/YCbw7jumZWZzUeNBtKl0Jzy3MJ4C3gj8Byzf2oETE2It5R7j7Q/LyImCsi9oyIOchW8tK83fvIVm47PQks1K/vd9U2l2lmVenpGfytYkMJuF+NiHERMYYcZPoVcAbZj3pgRMwWEYsAVwDfKs85Ddg8ItaIiJnJAbQvSXoJOB3YohwbHRGfJvtXz2jJK5u668jBsp0iYuaI2BJYts1lmllVRo0a/K1iQxo0A34DPEZ2Rewk6Vkyc2Ej4BngFuB64HsAki4Fvgn8ohxfkcxYQNJ55CDaqWTK1/7ApyTdNuxXNQ2S/kEO5u1PtnZXL3Vwt4JZN6hxC7enr69v4LO6TMmSmCipr9w/CRgr6dPTel7fy8917pc16a2OFd0zqnPzY/omdfZzcNI5R3as7N4v7Nmxsp//+NodK3uu6yYMKxJO/MZnBv13OvrICyqNuvXNEG6TiJgdeBrYNSJGRcQKwOZkV4iZjXQ1buHOcAG3pJVtDmwNvEDm6B5P9jeb2UjX2zv4W8VmyLUUJI0nZ5iZWbfpQCAdrBky4JpZF6txHq4Drpl1F7dwzcwq4haumVlFHHDNzCrigGtmVo2eDkzZHSwH3CHo6OZ0HZzt1Uk9HR4A6d3iGx0ru5Pvt32v/8fAJ7XJ8cO9gFu4ZmYVccA1M6tIC78VRcTRwGrkOt27Sbp9Cuf8AFhd0joDVq1lNTMzq4MWraVQNshdUtLqwHbk9mH9z1kGWGuwVXPANbPu0rrFa9YHLgaQdA8wd1kPvNmR5Drfg+IuBTPrLq3LUpgfuLPp/lPlsRcAImIbckODBwd7QQdcM+su7Rs0+/8LR8Q85EYGGwALDvYC7lIws+7Sui6Fx8gWbcMCwOPl5/WAd5E7j18ErFQG2KbJLVwz6y6ty1IYDxwEnBQRKwGPSXoRQNL5wPkAEbEYcLqkPQasWqtqZmZWCy1q4Uq6GbgzIm4mMxR2johtyoa306UWLdyI+CLwA0kLdaj8B4EjJP24E+WbWQu1MA9X0j79HpowhXMeBNYZzPVq0cKVdGZzsI2ITSJiqcE+v3zqjG1P7cxsRKnxFju1CLhT8D1gUAE3IkYBRwMOuGZW600kW9qlUDqWjwKWByYCVwI7S3oxIjYGfgLMR47q/Q3YRNIqJZ/tCEnzRcRfgWWACyPiF5K2iogNgEOA9wEvA+cCX5f0FpkTNxvZ13KYpP3LDJEfAMuV808FDpA0KSJGlzp+AXiDDO5m1i1qvJZCq1u45wK/J4PqcsAqwD4RMQ64ADgOmIcc/ZviiJ6kZcuPm5Zg+w4yQJ8OvBNYgwyWXyrnNc5fuQTbhYDLgZ+WstYHtgS2L+d9Cfg8sDbw3vL8dw/3hZtZTfT0Dv5WsVaXuALZknxL0hPA1WTQXQ94DThG0uuSzgD+NJgLSnoVWAg4SVKfpPuBW8p1p2SLfJpOkzRR0t3kCOM25fingXMk/VXSK8B+wEzT82LNrIZ6ewZ/q1irsxQ2APaLiCCD2GjgRmAc8KikiU3n3ka2Pgdjc2CPku82qlz7zKmcuwSwYkS81vRYD/BE+Xkh4KrGAUnPRcQTmFl36J0BFiCPiPcB5wH7ACdIeqXMvFiRbEm/2e8pkwZ53fWBE4GtgPMlvRERF03jKa8C4yVtNJXjs/Cfr7uug4dmNlQ13rW3lTVbkRwoO6p8VQdYufz/SWChiGgub9VBXndV4H5JZ5dgO4oclJua+4DlmsuKiLGlLxhyut7CTcfmxRkOZt1jBslSeIBsPa4cEfcCuwOzl9sN5ADWThFxMvBZcrDq0alc6zVgybIU2gPAuNKd8BKZVfAcOa8ZskULsFREPAScTWY0HBgRh5IDeBcCVwAHkJkTe0TEicAj5dzXW/ELMLMa6MBg2GC1rGaSbiXzYa8G7iW7ELYF5gZ+Vn7en2ztrk6mak2tW+EEMq3rHDK74TLgz+RSabcBewOrRsRZZXDugnLuDyU9C3wS2Ah4hhxgu57J6V9HA5eQfct/Jwfv7m/F78DMaqDGLdyevr6+SgqKiJmAiZL6yv2TgLGSpntecuVeeb6aX5bVRt+kQQ01tEUnN9DcefaFBz6pTY7ve2FYkfCtM74/6L/TUVvtV2nUreRfNCJmB54Gdo2IURGxApl5cEUV5ZvZDGQGysOdIkkvkwF2a3Jm2CXkbsinVVG+mc1AZqA83KmSNJ6cYWZm1j41HjSrxfKMZmYtU+O1FBxwzay7uIVrZlaRUQ64ZmbVcAvXzKwiHcg+GCwHXDPrLm7hdoe+V57vXOGzztG5sjs424q+DpYN8Gbnltnoe8ecHSt743lm71jZw+YsBTOziriFa2ZWkVEzwALkZma14C4FM7OK1HjHBwdcM+subuGamVXEg2ZmZhXxoJmZWUXcpWBmVhF3KZiZVcQtXDOziriFa2ZWEa8WZmZWkV5nKZiZVcNdCmZm1ehp4aBZRBwNrAb0AbtJur3p2LrAD4C3AAFfljTN9UTr+1FgZjY9enoHf5uGiFgbWFLS6sB2wLH9TjkZ2EzSh4E5gY8NVDUHXDPrLi0KuMD6wMUAku4B5o6IMU3HV5b0aPn5KWDegS7ogGtm3aW3Z/C3aZufDKQNT5XHAJD0AkBEjAM2BK4Y6ILuwzWz7tK+LIX/iNARMRa4DNhJ0jMDXcAtXCAizoiIb3a6HmbWAq3rUniMphYtsADweONO6V64EthP0vjBVM0BNy0CjO10JcysBXp6Bn+btvHAZgARsRLwmKQXm44fCRwt6deDrlpfX99QX84Mq+/phzv3y/KuvZ3RwV176eCuvVcusnTHyv74M48PK69r0l1XD/rvtHfFDaZZVkQcCqwFTAJ2BlYEngeuAp4Fbmk6/WxJJ0/reu7DNbPu0sKpvZL26ffQhKafZxnq9Rxwzay7eGqvmVlFPLXXzKwiXg/XzKwibuGamVWk1wHXzKwSrVwtrNUccM2suzhLoTv0zPbOTlehM2r8Bm67mYacatkVLv33Sx0r++PDvYD7cM3MKuIuBTOzinjQzMysIm7hmplVxH24ZmYVccA1M6uIuxTMzCrigGtmVhUHXDOzariFa2ZWEQfcaYuIPuBN4B5Jy1dc9nPA7MA/JS1WZdlm1gYOuIOyoaRrASJiNPBtYAtgYbJTZgJwsKRflXMOBA4A3pjCte6U9OGIWAe4BmjeCfBp4LfA3pL+JWmuiNgGOLD1L8nMKueAO2RHAOuTAffPwKzAdsDFEfFBSX8o5/1B0iqDuN58kl4CiIiFgVOA84E1Wl5zM+swB9yh+ihwpqQ/lvsvA8dGxL/IrYmnm6RHImJf4I6IGCvpyWHW1czqpMYt3LpOybgb2Doi3tZ6lXSupH+04PqN9QYntuBaZlYrPUO4VauuLdzdgHOA2yPiUeAG4CrgfEkvN523UkS8NoXn7yjp9CldOCIWAb4PXCXp362ttpl1nFcLGxpJjwJrRkQAGwBrAccBh0TEepJUTh1sH+7TeSkAngR+DezT4mqbWS3Ut0uhlgG3oQRWAcdHxNzATWSg3HaIl/r/QTMz62513tOsdm3viFgoIn4SEW/bz0bSs8CtwJjO1MzMRoSensHfKlbHFu6TwEeAhSPim8C9wExk5sKmwC4drJuZ1V59W7i1C7iS3oiItciJCFcCY4FJwF+BnSWd1XT61AbNABZsa0XNrJ48aDY0kh4HdhjgnAMZeHbYtdT5487M2qC+f/K1DLhmZtPNg2aDMj4iJlRdaFm85pSqyzWzNqnvvId6tHAldewjSdJcnSrbzNqhvi3cWgRcM7OWqXGXggOumXUX79prZlYRt3DNzCrigGtmVpX6Btyevr6+TtfBzGyGUN/eZTOzLuOAa2ZWEQdcM7OKOOCamVXEAdfMrCIOuGZmFXHANTOriAOumVlFHHArFhHzRcS7Ol0P64yIWDcivP3TDMoBt3pXAxdExFoRMUuVBUdEfec8zgAi4sPAZcB+EbF+RHTtDtQRsUZEnBIR4zpdlzrx1N4KRcQuwJ7AncAngf8FjgTulTSpk3Vrp4hYBvgysADwRUlvdrhKQH4ASeqLiPcDPwL2lnR7m8vcBtiVbOxcAFwO/EnSW+0st2oRcS8wO/AycLqkQzpcpVpwC7daE4EPSdoU+ASwDnAzsFdEzN/KgiJiVPn/+yPicxFxUER8spVlDMGF5HvtMklvRsTYiFggIhbuUH0AkNRobSwFLEd+8zgmIhZtdVkR0VMC/OnA6sAfgW+Q2zttFxGLt7rMTomIjYE+YGXgNODzEXFXRGzW2Zp1nlu4HRYRuwKHAA+RuxD/WtKLw7xmr6RJETEPcBPwPDAB2B7YRdLxw6v1kOqyB7C1pBXK/Y8AxwGzADcCe0h6uqr6NNVrtKSJEfEl4KPAK8BYYD5gZuBnwPGtbI1HxChJb0XEYcAk8newFBmYbgLOAG5q5e+jUWarrjeEcmeT9EpEzAssDXyWbGTcA+wn6a6q61QHDrht1vS1dSZgCeBTwOnAs5LeKOfMDJwAbAtcCXwPuHN6/9ibyjwbeFXSdhGxKtl/vIik5yLiY8BVQE+7ujNKn/GJwF8lHVs+XD4FPEx+nT4R+Lqkc9tR/iDqNzPwAvARSTeUx9YF/gfYlOz6OVnSeS0oq/FvshxwKzCPpNebyjwSeA8ZdC+WdE0rygT2AO6XdMlwrzeY8hrfGiJiEeAlSf8u98cBqwJfBD4IXEoG3mfbXa86cZdC+zUGqr4D/BTYBviApDciohdA0huStgOWBcYA1wFHRMTiETHkNYvLH/Y8wHuBn5SHTwCOKcF2feAAYHQ7+47LH98fgWMi4jzgcOA84NuSLicDfie/Sr8XuJf8AACgBLq9gT8AcwF7RMSBETH7cApq6r5YtpQ5uunYNZJWAu4n3x+tymKYi/wQX71F15uqpg+UWSLiGLIb6aGIOLW0sh8vQX8v4LvA8sBtEbFbu+tWJw64bVTehJMi4gPAbuSbbXlJvymnLB8Rnyp9rO+WdI+kNYEvABsBdwCbTE/ZpWXxJLBqRKwNjJV0QDl8N/kHv9z0v7pB1+ME4OvAS8BXJJ0o6fGIeDfwOaAjrdvicWA24GcRMV/jQUnPkK3vy4GLyaD1gRaVeR35uz8oIpbod+znwI6SzmpFQaX1uBLZZdVujYbFT8gP0WOALchxil82MnIk/YPsrtkZOAvYISJuLv2+Xc87PrRRU6vmR8BRkm4BiIg5yEB6IvAW2X+7WEQcBUyUdAE5gDMJeH0YVbiQbFXODezS9PhywPzt7EcrAfV9wCOSjul37FBgTeBnkh5oVx0GIunZiPgMcDLw24g4TdKPyuGtgBskHR4RawJrA7e0oMx/lT7cvYFxEXEx2X30APmhvDu8/ev5MMt7E2hrVki/hsXngQUkPV+OjQG+AswSEW9I6itdaRMi4lHyG89PgUOBX7WznnXggNtmEbEU2Yr6cdPDuwP/Tfbl7kGOVh8AXCDpvvK8TwGXS7pskOU095/1An2STit9x1sDu0TESsCrZD/q/i14eVOry0bA94FZyT+0c4EfSno+IpYlW0DnkR9ElWkaKFuQ/NCZWdJlEfF54Gtka+tg4D5y8Oyz5anjgGems8zGAOa4cp25gUvILovjyX+HmSKij0wPvBje9mFde011PRk4rPw7z1L6qK8BvgnMLemFfs97BngmIv6L/FbX9dyl0H6PA68BH4mIMRGxBRlwLwX2KS2QY4DfAc0z0G4kA+VgNdLAtgFOBcZHxOaSTiIHKn5KDtrNBXyzpCe1yynltgnwV/ID5faI2F7SX4GLJB1TdVCRNLH8eCVwNHB2RNxJBoNvAJuR3T6nkCPr74uIY4G5JJ0y1PKagu1Y4BfAReTv4mlysHIdYD/gWDIobVmeN2r6X2VnRMQngFUlHdTv0AHAQ5Iein4TbyJidBmj+GYrBiZHArdw2+8N4CngCPJr5PvI1t8xkl4u58wP/Bdl8Kb8oQ6qRVXSbl4sg3DjyGBxHtkiOzYivkpmAhxBDsTNKum11r28/6jP/wBPSzqx/IGtR7YeVwdOiogjycyAc9pVh6nUqzGos1sp/wtkS3tn4I4yqPe10udMRMxGtkR/TX5Nnh6ND5SfAvdJWrv8ftZk8kDdzf3TwEboJIg5gOcj4j7gu5LOKB802wBLlnN6mPw7af4APLrKinaSW7htJul1SZuRX58vAzaT9L2mYAs50HCJpH+WEd1BZQ5ExALkV9NtI2Iust/xcElfIPsDtyeDyzURcXpELNDOYFv8EXixpFydDFxRWocnAleQ/Xlrt7kOb1M+wPpKV0svcISkh4EbgB3IFLDFgccj4kcAkl4BQtIOku6YnnJLmYuTH7J7lof3Ag6W9EL5tnPMVC9Qc40sm4iYXdIvyEbDr4BTIuJ3wG/ILoZHSxfDf7yv+7d6u50Dbps1vh5K+qGk/Rp9shExb0S8LyK+BQTw1fKUQadpSXqMzABopFu9TGYmIOmJknq1G9lPvAJwSwmEbRERJwNnklNV3yCT+hv5n3cAD5ItuvvbVYcpafpDP4CcYvyViFhE0lslm+NSsvtjfyZ3zYwa7odTCSb/JieevLv0y79T0qHllGeBRcsA44jSNFA2L3B1RIwp4w97kZkJLwDLAAtHToLU25rfAAAOhElEQVSY4uDvSOqrbgUH3DZTzizqaf4kLyky3yZbpx8Adipv3lFDfQOWjIbGjLJjgb0jFw6ZtRx/iEw3+hKweQmE7XIuIOBf5Sv588CXSktoRbKP9OVpPL+lSqu+OQf1RuARYDXgWxGxdkTMIWmSpH+Skw+GnRcak3Onx0p6juzHPpPsx92j6dSZyAD8xHDL7IDG+/lUcrDvBfj/nPJbyK6ELciupMcjYr+O1LJmPNOsjaaW2lMyB8YAy0q6voXlzUX+Ya9BBo+zyQGLSvoEI+IdZG7t9qUe15F5rIuSrduLJe1TUV1Gk33l3y4fZu+X9OdybFuyJfYm2Zd8BTkbbti/p5IGtTLwN7JLZXsysB4MfIjsxriWbEkfQk5tPic6NAV3ejT1h69GTl6ZX9Kr5dgXgTsk3VPujyMHbQ8FNm1kYcyoHHBbpGlE+v3AvJKubTr2tpStxlfciFhY0iOtyrlsKm95Mmn/RfKP+jftbEWV/MtnJT1S7q9HJrVfRra6g5zmOb5ddRigfmuRQe5MYH9JD5cPh++QHxBPkHnSw56EETmB4jjgM+RA2TLl8YXIbos1yIFEAVdqBK+iFRF/As6SdFi5vwI5HXqekhrW/L5fuPH+mJE54LZYRFxOpl5dRObRqulYD9Bbuhl+BPyz8WZtQz16yDzSM4FDNXmWWTvKupNMOTuS7LO8m+xL3hJ4pmRIdFREbA4cBCwEHEb+TiZG5kn/GDi29Hm3oqx3kUH8TTIAfUdldmHpwnhpah/CdReTF+DZnsx+WbqpxXsLMF7Sdxp5uKU7aV/gkJHSgm8nB9wWK0nc2wOrkAn0VwC/kvRU0zlLki2chUtmQktbuP3qMwswZ//UoxZdu4f8unwAmVt8K/l6lwfWJ/tLVwa2b3Pe75Tq1vjGMQYYpbJISkTsSea+Pg18q9X5n+X3PZF8/TcBvySnaZ9LLtZyf0nVu1rS31tZdpUi4inghMYHeUl3O0zSguX+TMqlOE8ju84+2MHq1oYDbptEzg3fEXg3OTh2KfA7Sa9FxLVkP9eeI6nvblrKCPzG5Ff388kUoc3J4Ltro/+0oro0gu27yBzYVcgBs/0k/b1kBXyPzFj4PbB+ow+yBWU2WoDzquRSR67MdhiZj3oDub7BQhWk6LVF5IpnxwDzku/tg8l/969JujQmL834XjJNcClJj3XLe304HHCHqenr1ELkzLAx5Cf/g2XgZgcycb6HTJHqAfaUNLb5+R2q/rBErjHwFvCXkle6A5kBcKzKOg3NgacD9buYTMi/hFxycQWyT/m75Wv9h8gF4Y9o1b9D5NTgtYBHyRbuuZKeLMd2JvtwL5J07kgOQBERwMfINW6XAWaRNF+/c64j14n4+kh+ra3kgDsMTa2a1YAfAnOSf2RHKVdFapy3ADmjaQOy5belpLOjzO3vRN2HK3K90zvJBV1mJVtuZ5FTVDclP1TO7GD9Zif/Tb4r6cmSprYF2Y/bB3xfOe25cf5096M2tWr3LmX8jPzgWZ18P5wv6YKR1Fc7LRExj6R/ly6l1ciguyG5iPtZkk6JiDWAC7uhYdFKDrgtEBH3kAvRnKKy4HJ5vBGQ51auTPVhYPU6DCK1Qkn5WZGcmrwjmep0ETlYtwTZepzQobotT/Yr/75fYB1bHt+H8sHXovJmAR4gU59ujYifAh8mV4JbCriLDE4f79TvZHo1vY/fRy5VuQ2wjaQry/E5yenKnyEXF/8buYvGrpL+dyQ3LFrNAXc6NXUlbE++sd7f/1j5eWUyH3QLZRJ845yuaO00i1yNbHlyEZ4NJG1YcfmNluZO5O/8FXKFrkMk7d/v3PeqrMzWorLfw+QBskbf5liyNX0jubXMg5JG7ASAiLgLuI38ZvPrkl7X+J3PTE6k+gw5aPySpE90sLq15IA7TKUl82dJP4qImTWFmVwR8RC5ItIvq6/hjCcibiW7Ex4hswW+RPadf6dVLdpSTiPYND587ydzbT8HLCapsfrX4WSK4HXl/oj5sG1q3e4K7CBp2aZj7yQnNKxEzio8VtLlJS/7MUlPu+/27Ty1dzrF5CX0nie/VtM/2JYcRMideYe1RctIUPr0OrIgSUxeSGUdcmT8MuWW5z8hA+5vgMMi4k8RseRw61jKW7xc55DSVbFkycZ4DVgiyvRqciufRRrPHSnBFrKu5bV+maZF7Mu3mePJD5ebKWl2ETFW0p8aaYgOtm/ngDsdSku28UYS8KHIhbX7B53G14eFyNzMrtboRunE4Eijr5zsn90cODxyKcoXJN1IDpbtDPwTmNSCOs5frnc92af576ZA+huyW+WWiPg5mbXQspZ1lUqwXYRcYrT5PXwMuYbH5yTtQX6jmJf8cLGpcMCdPuMjZyhBLmY9iVx7dtHy1bJXuZVIX+QSfItJOqNz1e1+5QPuebIf9XJy1Px/I+K/IVdPI3OhtyyTD6brvV9aso2V2k4kMxHmAfaJiGUi18m4jQz8N5Az77YoXQ8jav3p0rCYJOlBck+6bSJi98ilF8eSfeO/KV0qE8jZhWM6WOXacx/uEEXEt8kR2qWaHluSXAlqKXJ33GuBmck58xuSOztc7NHa1ptSf2gZwNmcXGR8HnIG3NmSbmtBefeRrbo7Sx/md5i8Xc6/yZbeL0tWysrAH0ZqOlTkBJ0dJf0tIj4LHEhmojzeeLzp3M+T2ygt2om6jhQOuEMUEceRXyV3I1tMjzZmFpFfLXcBZiG/PdwOnCbpwg5Vd4YREVuRI+T/IjMCfkb+G+xAfvC9i/ygnO6NMyNXY1td0pXl589KOrkcG00uMr8jGeDPI9fXXUfSn6b7hXXIVBoW7yT3ZHtCTTPzSqv/DuAbks5zw2LqHHCnQ+lOuLTcPYgcgX6x6fgK5BYqLyr3LHPidxs0ZQnsRS4BeCc5s2wc2d94tKRLImID4IOSDm5h2Z8kg+oD5A4OZ5XHFyenEy9KrqGx60j8t+/XsLiMbFhM7HfO5mQa3GLkQkxfrLqeI40D7jCU/sFzyTzLg8hpjMOak29DUzIBHgU+Kenm0pe7EdmlsBSwXflK3EjdGs6MssauvzMr95BbiWzRbkouMr5Xo9sicibe48oFXEZMGlizKTQsrtDk7c9nJqdLbwkcBdyoXJLRaWDT4IA7TOWr5L7kClSnkKO392nymrcjrnUzkkROqz5Y0vr9Hp8PuBq4XdL2LS7zFDLIH6pcgvAT5EaZHwQuIPvsW746W6c0NSxuAL4L3DaVfHO/1wfgLIVhkjRRuTX0/MB7yLnzu0fEguWUlSNisU7VbwbwCLBMROwfOb228Yf/NNmHOkfJHBiWRt51mVHWWJLy6oj4rHId3c3IXZnXJTNXuoZyH77GOiHXAMdExFLNv1cH28FxwG0RSc9I2ohcoGYP4PLIJRovA5ad5pNtuin3IjuMnLv/tcitdBp/+DuSAzxvDmeiQwkmja/JxzE5/ew14ICIuJBcgvDkUo9ty/NGTel6I9FUGha7uGExNO5SaIPyx/15cqT8T5JW6XCVukbTQNk7yQ+yeYHfkt06nyMXfe8h80YD+ECZFDHdLbCm/t8fAmtJWr08vgiZ9rcrmQXxc+AYSY8O71XWX0zexulpsjvtp8CXJf2qoxWrOQfcNioDC7Or7DZgwxNvXxTod+R06fuB3ZVLMH6AXKlsLnJG2WWS/tKKgZzyIXokue7rzv2ObUkuaP538lvjgZJu7Pav2W5YDJ0Dro0YMXkhlQPJrptNyJbsm6XVuwTwnNq04HkJrGeQSf+N/NtG6/fHZLbKDmR2whfaUYc6csNi8NyHayNGCbajya6Eo8vAWPNo+YbANWVSwrDF5AVxRpWfrwC+Dnw1Ii6OiFVLsF0L+JikX5ADde8tg2szBElvONgOzoia221W8mDvB/aNiAkqa9qWgHgVuePCWOC5aVxmsGU1cmcPJXfqeJ3sNjiTzPG9KHIniWfLYwDvBOZS044fZg1u4dpIdBrwArkc4EcjYkwJjpOApYFhdyk0pYHtSk6k+AU5ueHTwCfJTIWPkQvYbAEcHBHbkstB/mC45Vt3ch+ujSgRMQe5k8PK5IaQM5GBcE5gQeAGtWg35BJ07wW2knRTRPwEWIfsq92QnNa7DNmXvByZDniipOOGU651L3cpWK01TaddE9gOWJhcZ/hEch+t3cj1Wt8CzpF0XnlqK6bSzkcuOfhCRCxKrtewBNm6Hg88AVwv6ffA7yPiF5JeakG51qUccK22SgZAY8GUc4H/JQeu9gZ2knQ+mao1pee14qvb82S322gy3ewaTd7y/Apy6cXx5f5oB1sbiPtwrbaacm73zbv6NvBrckLD18qxb5R1Z//jeS0o/zVgtbKk44vAok3TWVci+4sb53o5QhuQA66NBI2dbwHOAi6UdHfkNu2fJ7/mt0VT8L6enFDxUEScA3wEOAk6s4ebjUwOuDYS3E+uk/AtYF1JWwFIehx4kNwzrq0k3Q18HDiVnM66qaTXSleCR55tUJylYLXTNFA2C7lH1nxkJsAOwGNMTsv6KLmIzIKSnuv2qbQ28nnQzGqnqT/0Z2Q/6eLAm8DfyMVpfk8G4fuAfUuw9cLXVnsOuFYbEbGUpHvLz7sAywN7kssgbkamhc0GbE/urTVB0v3l6SNuRwWb8bhLwWohIj5N7pZwFJnqtQtwraTxTQvErEnO5LpT0jadq63Z9PGgmdWCpIuAL5E7795OLlCzXtk/rNEquIlcbHy5iGj7QJlZqzngWm1IOp2cInsq8GHgK8DHG2lXZb2EK8k+XbMRx10KVktlu5ZDyDzbC4GTyXUTPgcgaauRuhuuzbjcwrVakvRgWcR7A2AxcobZWeSqXdt2sGpm080B12pN0u+AVcmFY+YAZmukf7l1ayONuxRsxIiIOSW92Ol6mE0vB1wzs4q4S8HMrCIOuGZmFXHANTOriAOumVlFHHDNzCrigGtmVhEHXDOzivwfgMMmLsOJz4cAAAAASUVORK5CYII=\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plt.imshow(output[6, 5, :crop_len, :crop_len], cmap='Reds')\n",
+ "plt.grid(False)\n",
+ "plt.colorbar()\n",
+ "plt.xticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], rotation=60, fontsize=14)\n",
+ "plt.yticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], fontsize=14)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 349,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "([,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ,\n",
+ " ],\n",
+ " )"
+ ]
+ },
+ "execution_count": 349,
+ "metadata": {},
+ "output_type": "execute_result"
+ },
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVwAAAEuCAYAAADC/KrmAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XecXFX5x/HPbkKREnqkg2D4SlEkID+iNAVRxAIKKooIIoIEKYqK0lVQqQIqTREBBalBECSiICUiIWhs+EVAEAhCaKEHkuzvj+cMDGuSnc3enZmdPO/Xa1/szL1zz5ll8sy5z2ldPT09pJRSGnzdra5ASinNLzLgppRSk2TATSmlJsmAm1JKTZIBN6WUmiQDbkopNcnwVlcgpZTalaSTgE2AHmB/2xPrjn0IOBSYDlxo+/t9XS9buCmlNBuStgBG2R4D7AGcUnesG/g+8D5gc+ADklbu65oZcFNKafa2AsYB2L4TWErSiHJsWeAp21NtzwJ+C2zd1wUzpZBSSrO3PDCp7vHU8tzT5ffFJY0C7gPeCdzQ1wUz4PbD3l2Lt2we9GnT/t2qoqGrq3Vldw9rXdmtNv351pW94OtaVnTXYksN6AO3d9eIhv+dnt7zdH/KeuVc2z2SPg2cDUwD/l1/fE4y4KaUOkqFedIpRIu2ZkXg4doD278HNgOQ9G2ipTtXGXBTSh2lu7o7svHAUcAZkkYDU2w/Uzso6Rrg08BzwAeAE/q6YAbclFJHqaqFa3uCpEmSJgCzgLGSdgOm2b4cOIsIyj3At20/1tc1M+CmlDrK8Aq7HGwf3OupyXXHLgMu68/1MuCmlDpKhSmFymXATSl1lHaeXJABN6XUUbrbt4GbATel1FmyhZtSSk3SlTncvknqAV4G7rS9fhPLPRXYE1gIeKftG5pVdkqpelWOUqha2wTcYpv6gFfmKR8GvBtYEngUuBo4yvZ/yzk3ALfbPmh2F5S0GDF4eXtgBWAmcCtwuO0/2P4C8IUS8FNKQ1w7pxTatm6S3gJMBB4DRgOLAtsBqwO3SVqqwUudX16/LbAYsCqxIMVvJK1QcbVTSi3W3dXV8E+ztVsLt94pwPW2v1j33N/Kor8/BFYGnmzgOu8B9rB9V3n8pKSvA/8gZo+klDpI27YiadOAK2k5YAtms76k7ZeAz/bjcv8A9pV0ey3olvUrz62iriml9pLDwvpvjfJfV3CtTwM/ByzpHuBG4BpgnO2XK7h+SqmNtHMLt53rBjDgxVBt/832W4g87mnESu3nApNLSzql1EGGd3U1/NNs7Rpw7yJW4Fm3qgva/pPtE2x/EFgLWAbYp6rrp5TaQ3c/flpRt7Zj+0lij6Av9T4maQFJN0t6X1/XkbSepFMlvaalbPsBIrc7YvavTCkNVd1djf80W7vmcAEOAG6RdDFwEPAAsA5wPDG868YGrvEIsDOwqKRvAPcTw8t2AsYQ43NTSh2ku++dblqmbQOu7b9LehsRFG8DFie2vLgEOMb2s3WnHyBp316X2Nf2jyS9AzgcuJlII7wM3AFsn7PKUuo8VbZcJZ0EbEKkOPe3PbHu2FhgF2Iy1e22D+jrem0bcAFs/wv4RB/nbNnHcQOfrLBaKaU2VtXUXklbAKNsj5G0NrFh5JhybATwZeCNtmdIGi9pE9u3zu2abZnDTSmledVNV8M/fdgKGAdg+05gqRJoAV4qP4tJGg4sAjzRd93ay3hJk/s+rTqlU+3FZpaZUho8FXaaLQ9MrXs8tTyH7ReJdOe9RN/QH+tms85R26QUbLck011bvKYVZaeUqjeIgeSVS5eW7teJIaZPA7+TtL7tuTYY262Fm1JKA1JhC3cKpUVbrAg8XH5fG7jX9mNluYGbgA37rFv/305KKbWvCnO444EdASSNBqbYfqYcuw9YW9LryuONgH/1dcG2SSmklFIVqhqlYHuCpEmSJhArC46VtBswzfblko4Drpc0A5hg+6a+rtnV05Prbjdq767FW/bHOm3av1tVNLRyy5LuAS+nMXRNf751ZS/4ur7PGSRdiy01oA/cBUuObPjf6c5PPdrUD3e2cFNKHaUVC4s3KgNuP5z29H2tK7yVH6IWtrRmXXpay8oGmDnx9paVvcCJP29Z2T2P3teysrsWa3Qzlzm8vqJ6DIYMuCmljpIBN6WUmiQDbkopNUnmcFNKqUnaeXJBBtyUUkdp4wZuBtyUUmfpauMsbgbclFJHad9wmwE3pdRhhrVxxM2Am1LqKJlSSCmlJmnfcNveIyjmmaQeSe9vdT1SSs3X1dX4T7NlCzel1FHauYWbATel1FEaWFi8ZTo54I6UdC2wOfAf4OO2/wQg6c3AScSWGDOBS4ADysZwKaUhrIGtc1qmkwPuXsCuwEPAZcB3gW0kLQL8GjgdeB+xZ9FlwBHA11pT1ZRSVaqMt5JOAjYBeoD9bU8sz68E/Kzu1DWAg23PdU3NTg64P7NtAEmXA4eU57cDFrT9zfL4P5KOAX5ABtyUhryqhoVJ2gIYZXuMpLWBs4ExALYfArYs5w0HbgB+2dc1Ozng1u9J8wKwcPl9TWAZSb3TB8MkLWR7elNql1IaFBWmFLYCxgHYvlPSUpJG2H6613m7AZfafravC3ZywJ3TvkYvALa9djMrk1JqjgpTCssDk+oeTy3P9Q64nwW2aeSCHTkOtw93A6tLGlF7onxzLdHCOqWUKtLVj595uPRrSBoD/HM2rd7Z6uQW7pxcCzwMnCTpi8CCwDnA40QnW0ppCKtwAfIpRIu2ZkUidtR7P3Bdoxec71q4tmcAHyJ6FR8G/gY8CuzbynqllKrR3Y+fPowHdgSQNBqYYvuZXue8DZjcaN06soVru6vX43OIVmzt8V+Bdza3VimlZqiqfWt7gqRJkiYAs4CxknYDptm+vJy2AtFga0hHBtyU0vyrq8JFEmwf3Oupyb2Ov7k/18uAm1LqKG080SwDbkqps+SuvSml1CS5lkJKKTVJVxtH3Ay4KaWO0sYZhQy4KaXOkgE3pZSapMphYVXLgNsf3cNaV/asma0ru4V6/v63lpbfvdgiLS2/VbqWfH2rqzDPujOHm1JKzdHGDdwMuCmlzpLjcFNKqUnaON5mwE0pdZbsNEsppSbpHpYBN6WUmqKNG7gZcFNKnSVTCiml1CRVxltJJwGbEJvS7m97Yt2xVYALiG267rC9d1/Xm++22Ekpdbburq6Gf+ZG0hbAKNtjgD2AU3qdcgJwgu2NgZmSVu2zbvP4nlJKqS11dTX+04etgHEAtu8Elqrt9i2pG9gM+GU5Ptb2f/q6YAbclFJH6e7uavinD8sDU+seT+XVXXyXA54hdv++WdK3G6pbf99MSim1swpbuP9z6V6/rwScDGwBbCBpu74ukAE3pdRRurq7Gv7pwxRebdECrAg8XH5/DLjf9j22ZwK/Bdbt64JDbpSCpP8QvYWXl8fXAKvZXqc8Xh/4I7ABcBzRwzgMuBHY2/bD5byvAPsAI4H/AifZPrXJbyelVLEKRymMB44CzpA0Gphi+xkA2zMk3StplO1/ARsSIxbmaii2cK8H3g4gaRjxRl8naZlyfDNgAvB94Cmi2b86MAI4vrzu7cQf8gO2FwE+BhwlqV9bHqeU2k9VoxRsTwAmSZpAjFAYK2k3STuUUw4AflKOTwOu7KtuQ66FC/wO2Kv8Phr4F/AgsClwBRFwfwucCGB7OjBd0pV1r1uy/PfZcs5EScvantWUd5BSGjRVjsO1fXCvpybXHbubiDsNG6oB90xJCxPJ6puIXEt9wD2RaPkeU1IMCxFphYfKNX4LXAdY0g3AtcA5wONNexcppUHRzguQD7mUgu0HgP8AGwGbAzcTKYTNJK0JLALcDVwN3E7kdxcGDqq7xnTbHwDeRgTs3YE7Jb2hme8lpVS9rq6uhn+abSi2cCFauZsD7wA+TYyHE7ANcAOwFrA4cJztp8prNqy9WNJwYDHbk4HJZQzdncCHidkjKaUhqo2XUhh6Ldzid8CuwEO2n7Q9g8it7EukC+4HZgFvl7SIpM8RAXkpSa8DvgzcIGn1cj0ByxAt45TSENbOLdyhGnCvJ1qxN9c9dwuwDnCd7SlEUD2d6FBbG9iRyNH+i8jx3gL8UdLzxPS879q+omnvIKU0KLq6G/9pet16enqaX+oQ1fPcU637Y7Vy196XXmhZ0TO/MbZlZQPQ3bo2ybBv/KhlZfPy9JYV3bX0igNqek7bcv2G/50uccPkpjZzh2oON6WUZq+Nk7gZcFNKHSUXIE8ppWZp43G4GXBTSp0lW7gppdQcDawC1jIZcFNKHaVrWPuOds2Am1LqLJlSSCmlJsmUQoeYOaN1Zfe0cOXIGS+3rOiuXT/fsrIBnj/0sJaVvVgLJ7v0/PfelpXdtfSKA3t9tnBTSqlJKmzhSjqJ2DWmh9hpZmLdsfuAB4DaN+MnbT/U+xr1MuCmlDpLRS1cSVsAo2yPkbQ2cDYwptdp29p+ttFrtm93XkopzYOuYV0N//RhK2AcgO07idUGRwykbtnCTSl1lArH4S4PTKp7PLU893Tdc6eXZV5vBr5me64L52QLN6XUWbq6Gv/p55V7PT4c+CKwJbAe8JG+LpAt3JRSZ6muhTuFaNHWrAg8XHtg+9za75KuBt4MXDLXqlVVs5RSagcV7vgwnti4AEmjgSm2nymPl5B0raQFy7lbAH/r64LZwk0pdZaKpvbaniBpkqQJxJZdYyXtBkyzfXlp1d4q6QXgT/TRuoUMuCmlDlPlxAfbB/d6anLdsZOBk/tzvQy4KaXOMpSn9pYhD/8G3my7zxxFK0haDTAw2vY/Wl2flFLr5NTeQWb7fmDhVtcjpdQGhnILN6WUhpQOaeGOlvQzYE1i9sWutu8v842/TQz8fQ74MXC47VkAkg4E9geWAW4DPm/7rnJsD+BLwBuAh4DjbJ9Rjp0D1Pbn/iTwJLA78Fbgq8SQtoNt/7h32qMsKnE0sD0xXONh4HO2ry/X3g74IbAscDnwT2B72xv14++RUmpD7bwAeX9qthfwQWBl4CXgXEkrA1cBPwKWJuYe7wLsCSBpe+AQYAci4N4JXFaObUf08I0FFgcOAE6V9K66Mj8K/BpYDrgDOK+UsyrwfeBESXN6DwcBR5bzbwNOKOWuAFwKnFqOjQcO7MffIaXUzrq7Gv9pdtX6ce5ptu+3/RRwPLAZ0eK07bNtzygdVqcAu5XX7A78wvafbL8EHAF8Q9ICwGeBC21fX157FXAd8LG6Mu+1fYXt6cC1wArAt8vjXwEjgJFzqO/VtieWcq8A1i7Pvwt4Efie7elltshf+vF3SCm1sQonPlSuPymF+t7/e4h5xe8ENpD0Yt2xLuCR8vuawE21A7YfBy4CkLRG/bHibkB1jx+o+/1F4Cnbz9U9hjl3lv277vfn685bAXjQdv1q4rcRrfOU0lDXIZ1m9VsO1N7R/cB429vO5TVzakUvNIfn61fb6b3NQX+2PZjTud1A7y0MWridQkqpUm3cadaflEJ9y3NNYpXzPwPr1edRJY2U9Lry8N7610laUtJBkhYjWslr81pvIlq5g+lRYOVeud+NB7nMlFKzDN5qYQPWn4D7eUkrlAV4DyRyqOcSedQjJS0iaVXgauBr5TVnAztJ2rQs8nAI8JmyQvo5wM7l2HBJOxD51XMZXL8nOsv2kbSgpF2AdQe5zJRSswwb1vhPk/Wr0wz4DbFk2XBgH9tPEiMXtgUeB/4A3Ah8E8D2L4GvABeW4xsQIxawfTHRifZjYsjXYcCHbN824Hc1F7b/TXTmHUa0dseUOmRaIaVO0MYt3K6enrkuUN6RyiiJGbXV2SWdAYy0vcPcXtfz9GOt+2O1ctfeF5/r+5xBMutBt6xsaPGuvRf+qmVl9zx0V8vK7l5n0wFFwhlf+kjD/06Hn3BpU6Nu+44QHiSSFgUeA/aTNEzSW4GdiFRISmmoa+MW7nwXcMuwsp2ATxN7E10B/IDIN6eUhrru7sZ/mmy+XEvB9nhihllKqdNUGEglnQRsQgxX3d/2xNmc821gjO0t+6xaZTVLKaV2UFFKoawTM8r2GGAPYhZt73PWATZvtGoZcFNKnaW6lMJWwDgA23cCS5VhsfVOIIa7Nla1/ryPlFJqe9V1mi0PTK17PJW6XXzL/ma/B+5rtGrzZQ43pdTBBm/0wSsXlrQ0MZ5/a2ClRi+QLdyUUmeproU7hboWLbAisbY2xKzY5YgFuC4n1gs/qa8LZgs3pdRRuqqbsjseOAo4Q9JoYIrtZwBsX0LZFr1sgHCO7T7X1c6A2x8tGLf3iukv9n3OYJk5o+9zBq3sma0rG1j0mGNaV/j051tW9KwzjmtZ2d0nbzqwC1SUUrA9QdIkSROIqf9jS952mu3L5+WaGXBTSp2lwhyu7YN7PTV5NufcB2zZyPUy4KaUOksr70T7kAE3pdRZ2ngB8gy4KaXOkgE3pZSapAULizcqA25KqbNkCzellJokA25KKTVJjlJIKaUmaeMWblt8FUj6lKQHW1j+fZL2bVX5KaUKtfGOD20RcG2fZ3vl2mNJ20taq9HXS9pN0sjBqV1KaUjJgNtv3wQaCriShgEnARlwU0ptvYlkpTncsqLOicD6wAzgGmCs7WckbQf8EFiWWM7sn8D2tjcqC0Icb3tZSX8H1gEuk3Sh7V0lbQ0cA7wJeA64CPii7ZnERpCLAJMkHWv7sLI1xreB9cr5PwYOtz1L0vBSx08ALxHBPaXUKeajHO5FwK1EUF0P2Ag4WNIKwKXAqcDSxLJns13KzPa65dcPl2D7OiJAnwMsAWxKBMvPlPNq529Ygu3KwFXAj0pZWwG7AHuW8z4DfBzYAnhjef3rB/rGU0ptoqu78Z8mq7rEtxItyZm2HwGuI4Luu4AXge/Znm77XOAvjVzQ9gvAysAZtnts3wP8oVx3dnaOl/ls2zNs/4PY/G23cnwH4ALbf7f9PHAosMC8vNmUUhvq7mr8p8mqHha2NXCoJBFBbDhwM7AC8KDt+oVVbyNan43YCTiwLPQ7rFz7vDmcuyawgaT6BWS7gEfK7ysD19YO2H5K0iOklDpD93wwtVfSm4CLgYOB02w/X7ac2IBoSb/c6yWzGrzuVsDpwK7AJbZfkjS3xX9fAMbb3nYOxxfif993u3YeppT6q8LRByWGbQL0APvbnlh3bE9i+/SZxDq5Y233zLVqldUsAusM4MRyqw6wYfnvo8DKkurL27jB624M3GP75yXYDiM65ebkbmC9+rIkjSy5YIh9ilapO7YMOcIhpc5R0SiF0vk+yvYYIrCeUndsEaIvaDPb7yA69Mf0VbUqUwr3Eq3HDSXdBRwALFp+biI6sPaRdCbwUaKzak6THV4ERpU94O8FVijphGeJUQVPERu6QbRoAdaSdD/wc2JEw5GSvkN04F0GXA0cToycOFDS6cAD5dzpVfwBUkptoLrOsK2AcQC275S0lKQRtp8ujcqt4JXguwTw374uWFnNbP+RGA97HXAXkULYHVgK+Gn5/TCitTuGGKo1p7TCacSwrguI0Q1XAn8FJhG5368CG0s6v3TOXVrO/a7tJ4EPAtsCjxMdbDfy6vCvk4AriNzyv4jOu3uq+BuklNpAdeNwlwem1j2eymt38UXSwUT8uMj2vX1WradnrimHykhaAJhRy3FIOgMYaXuHplSgAj3PPtGcP9bstHBDQaa/0Pc5g2TWg3e1rGyArgUWbF3ZK41qWdkzj96/ZWUPP/mKAQ0fmHnutxr+dzps10PnWFa5G/+V7SvK45uBz9i+q9d5ryPuoA+1fcvcymtKZ5GkRYHHgP0kDZP0VmLkwdXNKD+lNB+pbhzuFF7bol0ReBhA0tKSNodXhq5eA7yjrws2JeDafo4IsJ8mZoZdAfwAOLsZ5aeU5iPVjcMdD+wIr8yinWL7mXJsAeAcSYuVxxsD7uuCTVue0fZ44g2klNLgqajTzPYESZMkTSD6m8aWZQim2b5c0jeA6yXNIIaF/bKva+Z6uCmlzlLhWgq2D+711OS6Y+cQSw40LANuSqmztGCNhEZlwE0pdZZhGXBTSqk5soWbUkpN0oJVwBqVATel1FmyhdsZulq47NvMq3/asrJ7prZu9cphu3yxZWUDsNAiLSt61s3jWlb2sKN/0rKyB6yNd3zIgJtS6izZwk0ppSYZNh8sQJ5SSm0hUwoppdQkFe74ULUMuCmlzpIt3JRSapLsNEsppSbJTrOUUmqSTCmklFKTZEohpZSapMIWrqSTgE2AHmB/2xPrjr2T2Ox2JrHbw2dtz2ljXKBJW+yklFLTVLSnmaQtgFG2xwB7AKf0OuVMYEfb7wAWB97bV9Uy4KaUOkt1e5ptBYwDsH0nsJSkEXXHN7T9YPl9KrBMn1Wbh7eTUkrtq3tY4z9ztzwRSGumUreLr+2nASStAGxDA7uQZw43pdRZBq/T7H+axJJGAlcC+9h+vK8LZMBNKXWUruo6zaZQ16IFVgQerj0o6YVrgEPKruR9ypRCSqmzVNRpBowHdgSQNBqYYvuZuuMnACfZ/nWjVcsWbkqps1SUUrA9QdIkSROAWcBYSbsB04BrgV2BUZI+W17yc9tnzu2aGXBTSp2lwj3NbB/c66nJdb8v1N/rZcAFJJ0L/M32sa2uS0ppgFq4FVZfMuCGVYFHW12JlFIFcmpve7O9ZavrkFKqSC5ek1JKTZIt3JRSapIKO82qlgE3pdRZstMspZSaJFMKKaXUJNlpllJKTZIt3JRSapLuDLgppdQUFa4WVrkMuCmlzpKjFDpDz/TnW1f4zJktK7r7vR9tWdks0O/1QarVwtZSz02/a1nZvP0DrSt7oDKHm1JKTZIphZRSapLsNEsppSbJFm5KKTVJhTlcSScBmwA9wP62J9YdWxg4A1jX9kaNXK99294ppTQvKtrTTNIWwCjbY4A9gFN6nXIc8Of+VC0Dbkqps3R1Nf4zd1sB4wBs3wksVXbqrfk6cHl/qpYBN6XUWaoLuMsDU+seT6Vu2/ReO/g2JHO4KaUOM2idZgO+cAbclFJnqW6UwhTqWrTAisDDA7lgWwRcST3Ay8CdttdvctlPAYsCD9levZllp5QGQXUBdzxwFHCGpNHAlHlJI9Rri4BbbGP7BgBJw4mE9M7AKkRTfjJwtO1flXOOBA4HXprNtSbZfoekLYHrgel1xx4Dfgt81fZ/bS8paTfgyOrfUkqp6SoKuLYnSJokaQIwCxhbYsU025dLupiIT5J0A3Cm7Z/P7ZrtFHDrHU/0EO4M/BVYmBiWMU7S/9m+o5x3R4Pj35a1/SyApFWAs4BLgE0rr3lKqbUqnPhg++BeT02uO7ZTf6/XrgH3PcB5tmtj3J4DTpH0X+DJgVzY9gOSDgFulzTS9qMDrGtKqa2070yzdh0W9g/g05Je03q1fZHtf1dw/dr6bTMquFZKqZ1UNyyscu3awt0fuACYKOlB4CbgWuAS28/VnTda0ouzef3ets+Z3YUlrQp8C7jW9hPVVjul1Hrt28Jty4Br+0FgM0kCtgY2B04FjpH0Ltsupzaaw30sLgXAo8Cvgd65mZRSJ8jVwuZNCawGfiBpKeAWIlDu3s9LvdJpllLqdO3bwm27rwJJK0v6oaQl6p+3/STwR2DE7F+ZUkqxp1mjP83Wji3cR4F3A6tI+gpwF7AAMXLhw8AXWli3lFK7y/VwG2f7JUmbExMRrgFGEoOO/w6MtX1+3elz6jQDWGlQK5pSalMZcPvF9sPAXn2ccyR9zw67gXb+66eUqpedZiml1Czt28Zqp6+C8ZIm931atcriNWc1u9yU0iDJiQ9zZ7tlX0m2l2xV2SmlQdC+Ddz2CLgppVSd9o24GXBTSp0lh4WllFKTVLhNetUy4KaUOku2cFNKqUky4KaUUrO0b8Dt6unpaXUdUkppvtC+2eWUUuowGXBTSqlJMuCmlFKTZMBNKaUmyYCbUkpNkgE3pZSaJANuSik1SQbclFJqkgy4TSZpWUnLtajs9p2CM5+Q9E5JHb/fnqRNJZ0laYVW16Wd5EyzJpP0Z+Bp4FDgj7ant7hKqUkkvQO4FjgPuASYaPvp1tZqcEi6C1gUeA44x/YxLa5SW8iA20SSvgAcBEwCPgj8BDgBuMv2rIrLGmZ7pqQ3A+uUn0m2f1llOQ3WZR3gs8CKwKdsv9zsOsyOpC7bPeVvdDLwVdsTB7nM3YD9iLvLS4GrgL/YnjmY5TaTpO2AE4EtgN2ATwAzgaNtX9LCqrVcphSaawbwdtsfBt4PbAlMAL4safmqCpHUXYLt0sBFwIHA8sA4SWOrKqcfLiM+a1faflnSSEkrSlqlBXV5he1aa2MtYD3gUknfk7Ra1WVJ6ioB/hxgDPBn4EvEfnp7SFqj6jJbxfavgA1s/5d4f/sANwHHSvqVpA1aWsEWyhZui0naDzgGuJ/Y9v3Xtp8Z4DVrLbefAy/Y3kPSxsB1wKq2n5L0XuL2tqvq1nWvuhwIfNr2W8vjdwOnAgsBNwMH2n5ssMqfS72G254h6TPAe4DngZHAssCCwE+BH1TZGq+76zgWmEX8DdYCNgRuAc4Fbqny71Ers6rr9VFWV+1LTNKqwLO2nyiPVwA2Bj4F/B/wS+BQ2082o27tIgPuIKsLfgsAawIfAs4BnrT9UjlnQeA0YHfgGuCbxO3/PP9jL63bXwOftz1J0iTgV7YPl7RVKWOLwby9L510pwN/t31K+XL5EPAf4nb6dOCLti8arDr0Ub8FiXz6u23fVJ57J/BJ4MNE6udM2xdXUFbtc7Ae8Edg6Vr+vpR5AvAGIuiOs319FWUSdzf32L5ioNfrq6zy/hYCvgtsCoi4w/pcLehLegOwNfBp4PXA922fPJh1ayeZUhh8tZEBRwA/InJab7H9kqRuANsv2d4DWBcYAfweOF7SGpLmac3i0rJ4FNhY0hbASNuHl8P/INZCXm8e31Ojdeghbp2/J+li4DjgYuDrtq8iWtitvJV+I3AX8QUAQAl0XwXuAJYEDpR0pKRFB1JQXfpi3VLm8Lpj19seDdxDfD6qGsWwJPElPqai681N7XP+Q+L/6feAnYm02S9KIMb2v4m7h7HA+cBekiaUvG/Hy4A7iMq3/ixJbwH2B74MrG/7N+WU9SXpwXYrAAAS5UlEQVR9SNLHJL3e9p22NyM6GbYFbge2H0AVLgO+BVwPfKfu+fWA5W3/aQDXbojt04AvAs8SLZ3TbT8s6fXAx4gWUKs8DCwC/FTSsrUnbT9OtL6vAsYRQestFZX5eyLYHiVpzV7Hfgbsbfv8Kgoqt+ujiZTVoOn1Of840TF6fvlSPYxI0yxUG5ZYGhiTge8DO5bLfGd21+40mVJoAknXAzfaPqI8XowIpKcTvbf3E//YTgRm1OXBZgEfsn1lA2XU58+6gZ5yi7cXcfu2NJEnfIG4rT+sdOAMihJQ3wQ8YPveXse+A2wG/Nl2Kzrx6uuyPnAmsDBwdu32VtIvgZtsHyfpSiK3WklQkPQpohU9mQjok2zfK+k+4ADb4+r/fw4Vkm4FrrF9lKSFbE8vudtriM/x/XN43cvAJ6pI3bS73GJnkElai2hFfb/u6QOADxC53AOJ3urDgUtt311e9yHgqkaCbTEMmFGGHW0BrCzpTNtnSLoO2IEYGfEg8BXbFwz0vc2JpG2JlvXCRMvmIuC7tqdJWpe45byYGIrVNHUdZSsRrfwFbV8p6ePAvsTt7dHA3USr7KPlpSsAj89jmd2l9bdCuc5SwBVEyuIHRAtwAUk9xPDAcfCaFMSQIOn9wMa2N+l16HDgftv39/4SqUuXfWV+CLaQLdxBJ2lx4tb0jPLf7Yhe+pOAk20/K2lhIgAdY/sP5XXLALP66sUt5z1TcsIrEPnIi4l00RbAnUTH1J/L+QvbfnEQ3mp9nR4kAu5vgeOB9xKt+ONsnyVp58EM+A3U7y9EY2MVIp+6p+07ynjhLYi/3blEJ+dngPfZfuM8lFMLtiOJ/yerA38nOo1G2/6bpA8SOdsHgNtsP9rMkQVVKV9apxFfTN+wfW553/cDo2w/WPt7tLSiLZY53MH3EjCVCDw3Ea3ak4Hv2X62nLM88DZK5035YD7eQLBdkWgp7S5pSWBXIqh9gsgZ70n0wl8v6RxJKzYh2H4SeMz26URL8V1E6/Fm4AxJTxOdZ01Vyx9K2p/4m7yXuMu4F7hd0i+AqbZPs/0DItVzBTF06+PzWGytNfMj4G7bqxGpo+m82lE3oZR5le1HAYZKsK11+kpa1PaFxGf4V8BZkn4H/AY4tgTbhWYXbDWfTTfPgDvIbE+3vSMRZK8EdrT9TdvP1Z32Q+AK2w+V1k1DrQDbU4gOqVrv/3PEyARsP1I6LfYn0hZvBf5QhkINpj8Dz5RyzgSutn0Wka++Gvgc0YpsmvIF1lMCRDdwvO3/EF+AexFDwNYAHpZ0MoDt5wHZ3sv27fNSbilzDSKXfVB5+svEjKunJe1M9OYPOXUdZcsA10kaUdJhXyZGJjxNzG5cRdIinsMU9qGWOhmozOEOstrtoe3v9np+GWA5IrcqIr8KMSC+YbYvlXQFsTbDKcAjkm4Hbrf9Ysmd/Qz4CzDcZezvYJB0JrAR8IeS4liLSKVAjLi4j2jR/WcOlxgUdV9ghwM7AfdLuqPU44nSQTaRuENYCV75/zagu4HSensCmAa8XtLmwBJ1nW9PAquVESqPDKSsFugiWvA/JnLPT0OMQCC+2HcjUiffJL7IjrP9rVZVtl1kC3eQOWYWddXfOpUxiV8n0gFvAfYprYVh8/KNb3uG7SOJkQi3Ey3pgxTjeIfZftn2HbZvq+RNzdlFgIH/SlqECDSfKS3LDYghQM/N5fWVKmmU+jGoNxO50k2Ar0naQtJitmfZfoiYfLB/BeXWGjIjbT9F5G3PAy4k7jZqFiAC8JAKtnWt202AdwJ71x37lKS1bT/lWDfhXcDRwDckDWSIY0fITrNBNKehPYpZZyOAdW3fOAjlrk900D1DjMH8TTP+UUt6HTG2dk8iwPyeGPa0GtG6HWf74MGuR6nLcKLj7uslOLzZ9l/Lsd2JW9+XgQuIVMffq8idShpBTNX9J5FS2ZMIrEcDbyfSGDcQo0qOIaY2XzBEO8r+Apxv+9jy+K3E7Lyly4iU+qGKq9h+oIXVbQsZcCtS1yP9ZmAZ2zfUHXvNGNnaLW7tQzgYYy5Li/qjROD7jl+dZVY5xYD3J2v/oCS9i5hFdCWR5hAxr378YNWhj/ptTgS584jxx/8pXw5HEF8QjwAnuoIpxooJFKcCHyE6ytYpz69MjL3elGj1mRizOqSWLdSr60HsSYx+WVuvTuv9AzDe9hF6dRxuN3AIMQJnSH2hDIYMuBWTdBUxpfJyYhyt6451AbWVvE4GHqq1DgaxPgsBi3sQF4hRrNOwJnFL/gQxdfhRYBfgcdvHD1bZjZK0E3AUsDJwLPElNKPkmb8PnFI6GasoazkiiL9MtPiOcJldWFIYz87pS3iokDQVOK32RV5Gpxxru5YDX8CxMtzZxJ3c/7Wwum0jA27FJL2NuI3ciBgWdTWxaMzUunNGES2cVcrIhCE3qwhe+QJZgOiMOoBYlOVqYH1gKyJfuiExzvWcJtetdscxAhhWG2In6SCig/Ex4GuueMB9+YKbQbz/W4BfENO0LyJWx7pH0ueB62z/q8qym0WxAM/3gGWIfoijiTuIfW3/soxKeF7SG4lRK2vZnjIU0yZVy4A7SBSLcexNrIh0B7Ec3e9svyjpBmIUwUGd8iFUzIzbjviHdwkxJnMnIvjuV8ufNqkutWC7HDEGdiOiw+xQ2/9STDv+JrEo+q3AVrZfqKjM2i33Mo41GVAshXksMIrI4Y4GVh7oKIhWkiRiLPP7ieFfC9lettc5vyemLX+xUz7nA5UBd4Dq8lcrE2sWjCBute4rHTd7EQPnu4iB9F3AQbZH1r++RdUfEEmbERME/lbGle5FjAA4xWVhnPrA04L6jQMWI/7unyTGIp9CzIR6VtLbiQXhj6/q/4NiavDmxBTqW4CLXCY0KBZ/3xS43PZFQzUISVra9hPlDmcTIuhuQ6wpfL5jNuGmwGWd8DmvUgbcAahr1WxCrAG6OPGP7ETHMnS181YklqPbmmj57WL75ypz+1tR94FSLDA9CfgDsWbCTURH2VeIiQQH2T6vhfVblPh/8g3HdNlFiOUCjyLGj37L9hl1589zHrWuVfvVUsZPiUA0hvg8XOIYLz3kcrXwms/5m4iV03YDdrN9TTm+OLEY0UeIxcX/SSzqvp/tnwzlz3nVMuBWQNKdxJTds1xWuC/P1z6oS9l+UrGJ4Jh26ESqgmLthg2Iqcl7E0OdLidGR6xJtB4nt6hu6xN55Vt7BdaR5fmDKV98FZW3EDFN+MO2/yjpR8A7iLUE1gL+RATh97XqbzJQkv4E3EZ80f66jPaofdksSIzr/wjRh/Gs7ffP5XLzpQy486gulbAn8U3+5t7Hyu8bEuNBd3YMgq+dMyRbO3MjaTSRs10O2Nr2Nk0uv/aPfx/ib/48sULXMbYP63XuG11WZquo7DfwagdZrTNpJNGavplYROg+24dWVWYz1DUa9gP2sr1u3bEliHVsRxOTXE6xfVUZJjjF9mNDNW0yWDLgDlBpyfzV9smSFvRsps5Kup9Ygu4Xza/h/EfSH4l0wgPEaIHPELnzI6pq0ZZyagG+9uV7DzHW9mPA6rZ3KecdRwwR/H15PKS+bMtY2j8Ta/X+rjw3mljH431ECuX1xASXHWo56/S/cmrvPJI0rPw6jbitpnewLR9UiJ15B7RFy1CgV1fkavoKUHp15aotieBwpWPL8x8SAfc3xK6xf5E0aqB1LOWtUa5zTElVjCqjMV4E1lQsuwmxlc+qtdcOwWC7KrHiXX0e9nvEVPKP2T6Q+IJbhnivaQ4y4M6D0pKt3SYZeLtiYe3eQad2+7Ayr/2wdqRaGqUVvdG1XDmRn90JOE6x9u/Ttm8mOsvGAg8R6wwPtI7Ll+vdSHQiPVEXSH9DpFX+oFg4aHOgspZ1s5TP+Szb9xFbJO0m6QDF0osjiVTNb0oLfzIx2WVEC6vc9jLgzpvxihlKENuHzAJOkbRaubXstl3b4mZn4vby3NZVt/OVL7hpRB71KmKY0k8kfQBiuUpiLPQuZfLBPH32S0u2tjTm6cRIhKWBgyWto1gn4zYi8N9EzLzbuaQehtrqfOPLyASIdXw3AT5PxI3tyxdZbRnKjwOr2f51a6o6NGQOt58kfZ0YErNW3XOjiJWg1iJWvb8BWJCYM78NcLBjn6ocHlOx2eVDS4/5TsRmnEsTM+B+7gpWS5N0N3EbPal0Gh3Bq9vlPEHcWv+ijErZELhjKI4/ncPnfAlii6BH6ieKlC+h24Ev2b44P+dzlgG3nySdStxK7k+0mB6szSwibi2/QOwS0E2ssXq27ctaVN35hqRdiSFJ/yVGBPyU+H+wF/HFtxwRQOZ5p2LFrhpjbF9Tfv+o7TPLseHEIvN7EwH+YmK/si1t/2We31iL9PqcX0l8zmf0OmcnYlTG6sS6IJ9qdj2Hmgy486CkE35ZHh5F9EA/U3f8rcQWKs/Yfrk8lzNtKlY3SuDLwKeI8aGLEUPBZgAn2b5C0tbA/9k+usKyP0gE1XuJHRzOL8+vQUwnXo1YQ2O/ofr/fjaf86ttTyvHFiRm7+1C7DZ9s2NJxhwGNhcZcAeg5AcvIsZZHkXMGx/QnPzUP2UkwIPAB21PKLncbYmUwlrAHrb/WTd0ayAzymq7/i7o2NFiNNGi/TCxyPiXa2kLxUy8hx0rZg2pYWC91X3ObwK+QWx2Obvhj0Pyi6WZMuAOULmVPIRYgeosYrjM3X51zdv8EA4ixbTqo21v1ev5ZYHrgIm296y4zLOIIP8dx5qv7yc2yvw/4FIiZz9oy2G2whw+5//OO7j+yVEKA+TY3uYoYpjQG4i58wdIWqmcsqGk1VtVv/nAA8A6kg4r02tr//gfI3Koi5WRAwNSG3ddZpTVlqS8TtJHHevo7gh8ldhy5pqBltdu5vA5/0J+zvsnA25FHNuab0ssUHMgcJViicYrgXXn+uI0zxx7kR1LLJayr2IrnVpLa2+iR/3lgUx0KAG8lpc8lVeHn70IHC7pMmLN1zNLPXYvrxs2u+sNZfk5H5hMKQyC8o/740RP+V9sb9TiKnWMuo6yJYh/4MsAvyVudz9GLPreRQzUF/CWMilinm956/K/3wU2tz2mPL8qMexvP2IUxM+A79l+cGDvcmjIz3n/ZcAdRKUnd1GX3QbSwOi1iwL9jpgufQ8xx/9RxaIpHyW2OHqImN77typ6zktwOYFYaHtsr2O7EAua/4u4azzS9s3zS14zP+eNy4Cbhgy9unLVkcQt7fZES/bl0updE3jKg7TgeQms5wJ7142/rbV+v0+MVtmLGJ3wicGoQxraMoebhowSbIcTqYSTSsdY/fCkbYDry6SEAdOrC+IMK79fTayQ9XlJ4yRtXILt5sB7bV9IdNS9sXSupfQaQ21ud5rPlXGw9wCHSJrssqZtCYjXEjsujASemstlGi2rNnb2O8ROHdOJtMF5xBjfyxU7STxZngNYAljSdTt+pFSTLdw0FJ0NPA18TdJ7JI0owXEWsDYw4JRC3TCw/YiJFBcSkxt2AD5IjFR4L7GAzc7A0ZJ2J5aD/PZAy0+dKXO4aUiRtBixk8OGxIaQCxCBcHFgJeAmV7Qbcgm6dwG72r5F0g+BLYlc7TbEtN51iFzyesQwqdNtnzqQclPnypRCamt102k3A/YAViHWGT6d2Lhwf2KB7JnABbYvLi+tYirtssQar09LWo1Yr2FNonU9HngEuNH2rcCtki60/WwF5aYOlQE3ta0yAqC2QtVFwE+IjquvAvvYvoQYqjW711Vx6zaNSLsNJ4abXe9Xtzy/mlh6cXx5PDyDbepL5nBT26obc3tIPPTXgV8TExr2Lce+VNad/Z/XVVD+i8AmZUnHZ4DV6qYJjybyxbVzc/3X1KcMuGkoqO18C3A+cJntfyi2af84cZs/KOqC943EhIr7JV0AvBs4A1qzh1samjLgpqHgHmKdhK8B77S9K4Dth4H7iD3jBpXtfxA71P4YeAz4sO0XSyohe55TQ3KUQmo7dR1lCxGbEi5LjATYC5jCq8Oy3kMsIrOS7afml6m0aejKTrPUduryoT8l8qRrAC8D/yQWp7mVCMJ3A4eUYJs7DaS2lwE3tQ1Ja9m+q/z+BWB94CBiGcQdiWFhiwB7EpsZTrZ9T3n5kN1RIc0/MqWQ2oKkHYjdEk4khnp9AbjB9vi6BWI2I2ZyTbK9W+tqm9K8yU6z1BZsXw58hth5dyKxQM27yv5htVbBLcRi4+tJGvSOspSqlgE3tQ3b5xBTZH8MvAP4HPC+2rCrsl7CNURON6UhJ1MKqS2V/bGOIcbZXgacSayb8DEA27sO9d1w0/wnW7ipLdm+ryzivTWwOjHD7Hxi1a7dW1i1lOZZBtzU1mz/DtiYWDhmMWCR2vCvbN2moSZTCmnIkLS47WdaXY+U5lUG3JRSapJMKaSUUpNkwE0ppSbJgJtSSk2SATellJokA25KKTVJBtyUUmqSDLgppdQk/w972u+tb9C0bwAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "plt.imshow(output[0, 1, :crop_len, :crop_len], cmap='Reds')\n",
+ "plt.grid(False)\n",
+ "plt.colorbar()\n",
+ "plt.xticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], rddotation=60, fontsize=14)\n",
+ "plt.yticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], fontsize=14)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 370,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAACACAYAAAD5y4mRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJztnXmczeUXx9+Yoexbxk7CKVRSaVNRkrT9KtrzaxMVStuvVSpalKJooU27FhUVlaRdi1Ytp0zZtwzGMHbz++M8d1wiy5j53nud9+vlZeY+33vvmXu/38/3POec5zzF8vLycBzHcZKb4lEb4DiO4xQcF3PHcZwUwMXccRwnBXAxdxzHSQFczB3HcVKAtMjeOTfby2giJG/poqhN2G6mtmkXtQkF4p5f5kRtQoF4JGd61CYUiHXfjI3ahAJRovVZxTb1uHvmjuM4KYCLueM4TgrgYu44jpMCuJg7juOkAC7mjuM4KYCLueM4TgrgYu44jpMCuJg7juOkAC7mjuM4KYCLueM4TgrgYu44jpMCuJg7juOkAC7mjuM4KYCLueM4TgrgYu44jpMCuJg7juOkAC7mjuM4KYCLueM4TgrgYu44jpMCbFHMRSRPRE4oCmO2lp7XXEelWvWQ5gfw+cQvNxh79/0P2KNZcyrXrs8tt/cFYO3atVzY7XIq1KhLvT2b8fJrr0dhdj7baj/AyDdHUbFmXVq3P76ozf0HPW/sQ+XG+7LnoUfx+deTNhh798OPaNjySKpIc265ewAAT7/0KsUzds//V6PZgVGYnc9tc+fT/LdMjp4ylUm5yzcYe27hYg7+/U/2/S2TK2fOZVVeHkvXrqPrjNns/dsU2k6ZyhfLciOy3Dh9UH8GLJxOn9++pcEhB/1jvER6Orf//h39/pqc/9hRPS9lQNY0+s/N5IAzOxaluRvQ89rrqVxnd/bcr+U/z/1xH9BwnxZUqduAW+7oB4Rr99LuVKxVj/pN9uHlkdFeu1c+OJyqJ15Mk85X8/nk3zcYm/l3Fq2630pam7OZOvdvAHJyl3PqzQOo2OFCmna+mg+/+7nQbEs6z3zc+AkMfnQY7745klaHHMxlV16dP5aXl8fFl/fk5OM78PIzT9P3nvv4/ocfefaFlxjx2ut8OWEcZ3XqyIWXdmfNmjVJY/+48RO48robqVmjRiQ2xzPuo08Z8uQzjH1pOIe1PJDL/3dL/lheXh5drrqBk9ofw4ihg+n3wGC+n/wLAHVr12TJn5NZ8udkMr/6OCrz+XRpLs8szGZ4vVocUHpXes+Znz82Y9Vqbp/7N9dVq8qjdWowakkOY5bk8HjWIiYvX8nYPeqx9667cEvcc4qaPY9uTevuXXno2FPI/PQLznr4/n8cc3Sv7pSuXDn/95pN96LjA3fz5DkX88nQp2jTo1tRmpzPuA8nMOSxYYx9/VUOO+QgLr/q2vyxvLw8unS/gpM6HMeI4U/Sr/8Avv/xJ559cQQvj3yDiePf58yOp3HRZT2ju3Yn/cSQN97jnf7Xc1gzocfApzYYP7LHbZQuVXKDx+5/+W2++2MqPz7Vn/2lAd0feLLQ7Es6MZ/wyafUqV2LlgfsT4djj+GHnyazcKHtNJ/551/MnDWLEzu0p+1RrSlbtiwffvwJx7U7hh8mfsqe0pj69eqybNkycnKWJo39devU5rvPP6Zxwz0isTmeCZ9PpE6tGrRs0ZwObVvzw8+/snDRYgAyp05j5uw5nNjuaNoe2YqyZcrw4adfAFCsWDHKlilD2TJlKF1618js/zI3lxrpaey76y60LluaX1euYvHatQDUKZnO700acWrF8lQuUQKASiVKcHql8rxcvza10tOpnpbG/DVrI7O/cevDWTRjJlO/nsTkd96lTvN9KF2pUv54+YxqtL2qO+MHPZz/2N4nHEfWX1P5eez7jO7dl3sPaxuF6Uz45LMtnPuz7dxvE3/ttuX7zz9OiGv3o+9/pU61KrTcqyHHHdScHzKnsXDJelueu6U7l5/SboPnXNihDRMG9aZuRlVq7VaZOQsXF5p9Wyvm1UTkXRFZLiIqIvvFBkRkbxEZJyKLRGSBiDwqIrsUkr3Mmz+fCuXLA1CpYkUA5s6blz8GbDA+Z+48MjKq0XCPBmRlLeSe+wfSfJ+9qVSpYmGZ+K9sj/2NGzWkSpXKm3i1omfe3wuoUK4cAJUqVABg7vy/88cAKpQP4xUrMCf8TYsWL+Hg9v+hSau2PPH8iKI2O58Fa9ZSrrid9hWCYP+9kafXZ858TvhzOpdUqcQRZctQMz2dWiXTmbZqFS8uyubQMtHdjMpnVGN59hIAcsNNtEL1avnjp9x9O5889iSLps/Mf6xK/bqsWbmKy0a/wu1/fM8R3S4qWqMDmzz352/u3K9g12619ddu/wcGRXrtzl+UTYXw3VcsVwaAuXHifEjTxv94Tp1qVahXfTcyZ81j2OgPaLNf00Kzb2vFvCvQE9gNmAbcAyAipYGxwEdABtACOAC4dYdbWkBycnJo/5/T+HtBFsOGPBi1OTsVDXevR6eTOjDknjs4pcOxdLv2JqbPnBW1WZulV7UqDKpdncezFvHZUouPz1u9hs7TZlGqeDFuztgtYgs3Tb0DWtC4zeGMvXvD0EteXh6V6tZmTN/+fD9yNGc8eC/lqiXm37ApcnJyOO7UTvy9IIuhDw2M2pxtZvaCRbS/9k52KVmSAZefV2jvs7Vi/rwaS4HXgSbh8eOBkqp6h6quUtXpwJ3A+TveVKNG9QwWB89kQdZCAGrWqB7G7P/F2dnk5eWRtXAhNWtUJy8vj3Mu7MIfmZmMe+sNDmix36ZfvAjYHvsTiRoZ1Vi8JAeABQuD/dUzbKyaeYiLs5cE+xdRMyODVgcdyLD772b/fffmvE6nsnbtWn7P/CsS+6ulpbFk7ToAFoVwSUZaGgAzV63myaxFlC5enA7ly1GpRAm+yM1l5bp1dJkxmzV58GK92tQumR6J7QDZc+ZSuqLNiMpWrQLA4tlzAWjWoR2Vatfi/oXTOXfYQ1SpX48+v31L9py5ZP01jb++/JpvX3uTEunpVKlXt8htt3M/G4AFWVkA1KxeI4xtfO4vyr92z72oK39kZvL+qJGRXrvVK1dkcbi5Z2XbNVCzaqV/eworV63mlJvuY/WatYwfeAv1qxfeTXRrxTz+ylsOxMIoewBVRGRF7B/wAlBVRErtQDvzadumNTNmzmTiV1/z1pixHLh/C0qVKsWKFStosHt9Guxen1Fvj2HMu++Tm5vLMUe1YeSboxj9zliGPjSIZk32YunSpaxdG03cc3vsz8nJYUrmnyxblsvy5SuYkvkny5cv39JbFY79RxzGjFmzmfjNd7z9/ngObL4PpUqWZMWKlTSoX5cG9eoy6t1xjPlgArnLl3PMka24tf8DNDqoNdNmzGTk22NJT0+niTSKxP7DypRmzpo1fJe7nPFLl7HPLqUoWawYK9etY9m6dfSbt4A3s3P4cfkKFq9dy+4lS/Lsomx+XbGSB2tXJyM9jWXr1pGXlxeJ/b+Om0DlunXY/aAD2fuE9kz96hvWrFxJWqlSTBgyjNuatqRv88MY1bsfi2fNZnCH0/hp9BgypBH1Wx5As+OOYdXy5cyf8meR2962dWtmzJzFxK++5u2x74Vzv+SG5/47Yxjz3rhw7rdm5KjRjB4zlscefCDya/fo/ZsxY34WE3/5g7e/+JYD9mxAqfR0VqxaBcCUWXOZu9BuVtPmLmD+omyGvPEe30+Zxgu9e1KzaiWWLl9RaOdO2lYet7l3Xw6oqu61g+zZIke0OoyrelxOh1M7UbN6DZ4e+jDHndKRihUq8MaIFxg2eBCX9LiS519+hX59bqFpk724b9BDAJzR+YL81/lwzGhaH3F4UZldIPuffvZ5Luh2ef5rNNqnRXT2H3IQvbpdxPHnXEDNjAyeevBeOpx9PhXLl+f14UMZOuAuul5zIy+MfJO+N1xD0z0b07Pabnzz/Y80PbwdGbtV5cmB/fO9+aKmZZlduahyRS6cPpuM9DT618zggumzKV+iOI/VqckNGVW5b/4CVqzL49SK5Tm5Qjk6T5vFWqDT1PVx6I8b1o/EQ5/yyWe8P+Ahur/zKotnz2X4+d3oMWYkuYuzefSUs1gWPN4lc+aydvVqFvxpftibN97GpW++xLo1a3i+6xXkLlpU5LYf0epQenW/jOM7nkHN6tV56tEhdDj1dCpWqMDrLz3H0IcG0rVnL14Y8Sp9e99M0732YsCgwQCc+d/1cf7x74yi9eGtit7+fffiyk4dOPH6/tSsUoknru/G8f+7h4plSzOy79Xsee5V+cce3esOOh97BNPmLWDtunUc0aNP/tiUFwcViodebEt3CRHJA05U1bfC7+cD96lqVRE5HngVyFDVJWG8ErBOVbP/9YVzs6NxbRwA8pYW/cW8o5japt2WD0pg7vllTtQmFIhHcqZHbUKBWPfN2KhNKBAlWp9VbFOPb61nvjneBeYAD4jIVUBJ4GkgC+hcwNd2HMdxtpIC1Zmr6hrgZKABJuqTgflA94Kb5jiO42wtW/TMVbXYRr8/jXnfsd9/AtrsaMMcx3GcrSfpVoA6juM4/8TF3HEcJwVwMXccx0kBXMwdx3FSABdzx3GcFMDF3HEcJwVwMXccx0kBXMwdx3FSABdzx3GcFMDF3HEcJwVwMXccx0kBXMwdx3FSABdzx3GcFMDF3HEcJwVwMXccx0kBXMwdx3FSgC3uAeo4juMkPu6ZO47jpAAu5o7jOCmAi7njOE4K4GLuOI6TAriYO47jpAAu5o7jOCmAi7njOE4K4GLuOI6TAriYO47jpAApLeYiUlVEdovaju1FRIpFbYOTGohIGxGpFbUdOwsi0kpEholIjaJ6z5Rezi8i3wNLgJuBL1V1ZcQm7TSISBPgYqAmcJ6qro7YpJ0WETkMeBd4FngV+FpVl0RrVWojIr8DZYBlwNOqemdhv2fKeuYi0gOoBCwAxgODRWRPEUnIv1lESoT/9xaRM0TkNhE5KWq7CsBI7PwaraqrRaSaiNQUkTpRG1ZQYjOm8F2NF5EDo7bp31DVz4DuwEHAAOAKEdkvds45OxYROR7IA/YHngTOFJHvRKRjYb5vynrmInIpMEpVZ4lIe+AhoApwDzBcVedGamAcIlJcVdeJSGXgMyAb+AHoAvRQ1SGRGriNiEgv4L+q2jz8fgz2+ZcCPgV6qeqCCE3cIYjIacAjwArs5vWAqk6L1qoNid14VDVPREoBjwH/AaYAQ4FxqvpnhCamJCJSWlVzRaQKsBdwOnAC8Ctws6p+t6PfMyG91B2Bqj6iqrPCz2NVtRHQB7gF+EBEOolIuShtjCN2Rx0MfK6qBwNPAEuB5wFEpL2IFEvUmUWMIB57Yh4JItITuB74AugBtAGOiszAAiIiaeH/C7EL9G3gJ+AQ4A0RuVJE0iM0cQNUNY/11/kdwFzgKWAecDtwr4icLCJVIzJxsyTbzGGjHFdVEamsqlmq+ilwF9ALWA6MEpEhIlJpR75/SnnmIlIseCDpwB7AycDTwCJVXRWOKYl5UxcAY7ATfFLUMd3glY8FLlXVSSIyCXhbVXuLyNHBziOjtnNrCLOiIcBrwEnAFcCbqjpHRJ4A/lDVu6O0sSCEc2gJcIyqfhIeawOcA5wKTAKGquor0VlpxF0TzYAvgcqx3FGweQCwO/AM8IaqfhidtesJwtgLyFTVN6O2Z0vEfc6lsNl/K0CAl4FLVHVtOG53oC3wXyADGKyqg3aEDQnt5W0HsTvjrcDjwPnAPqq6KubRquoqVb0IaAqUBz4C7hORBjGvKwpUdSEwH2gpIkcC1VS1dxj+BUgDmkVl37agqo8AV2Ezi0tU9dEg5BnAGdgJnsw0BH4HpsceCCL4P+BboCLQS0T6iEiZaEzMtyvmrTXFbE6LG/tQVVsAmdi1kkjVLhUxh+uQqA3ZSmLa8zDQABgInAW0BkYEkUdV/wKGA5cDzwFdReTzEGcvEJGJ144m3BnXicg+mCfYDvO4V4VD9hWRusAuwARV/RU4PMQ97wLOAy7Bsv1RMRK4F0vc9oh7vBlQvTDibDuSINZ7AjNUdeBGY3cDh2P5imSP0c4BSgPDRaRjLP6vqlki8igmnMuxC/ZdLMQUNR9hVV23icgjqpoZN/Y8MFdVX4zGtH+iqotEpAWwa9S2bImNtOdMoKaqZoex8piulBKRVaqaFzTpBxGZCbyCOZ53YyG77SZlPPM4D2QQcL+qfhE88rIici7wCTaVvAk4X0TSw5fwmqo2xjyBIitdjI+viUjxYMuTwI3ARKCHiDwhIoOx+HPvzbxUQiAixwHvYJ7JeyJyp4hUCGNNMW/lFayqIqlR1UXAaZjQfCAiV8QNdwZyVbU/8CNwZAQm/oOQ8O8PtAduDzmjBmH4Cuzmk1BrG1R1dTKUUMZpz1Cgv6pmxzxx4ENMWyrFHRd7Xpaq/gIciOUvCkTKeOYAItIY85gGxz18JXAiFjvvBVyNCeNrqjolPO9k4C1VHV2E5pYA1ojI+dgFX1tEhqrqYyIyDjgFy37PBK5LJK9pMwwD+gIfAPdhn3NHEblXVYeJyOtJ8DdsEhFJU9U1YotumgElVXW0iJyJ3Zy6ikg/rEKkKpYYBagBZEVkc6xCqkawoxLwJhYGGoIVAqSLSB7wu6q+ARsIk7MNiMgJQMtQvBBPb2Caqk6LxdXjnhPT3+t2RH4l1RKg5YC3sPKrt4DjsZK4B4BBqrpURHbBPMQ7VfWL8LwqwLrgcRW2jVWAnDBrqIHFXV/BZklHYqVLV6nq9+H4XVR1RWHbVRBE5BzgWlVtHjy7JVjM/BAsFrsUWKKqtaOzsuCIyI+YA1QHiz93UdVvxRZIHYl9h89gyfcLgQ6q2jACO2NCXg07t+oDP2OJtxaqOllsDUMtYAbwlarOF5ESsUSds22EG/sj2M37dlV9Jnz+04BGqjoz9r0Ulg0pE2YJrAL+xjzDTzBvfBAwUFWXhmOqY9Oa6ZB/4mcVkZDXxDyjC0SkIjYlv1dVz8amul0wIfxQRJ4WkZqJLuSB74GcUOUxFHhHVYcBj2Khl0tIkHDDtiLrFwhdgX037bGZ3p/ANyIyAvg7lMIOAdZiHnApLH4aBTEP7XFgiqrWw+LiK1mftP082PyWqs4HcCHfNmJFFSJSRlVfwnTlbWCYiIwH3sfCLjNFpNSmhHxHhrVSSsxVdaWqdsQEfDTQUVXvUNVlcYc9jJXJzQqeSKHdKTdh32zMY70X85iWYRUsqOo8VX0LE/VeQHPgiyCQCYuIDMWWif8YEjuNMTED+AaYiglH5qZfIXEJN/q8cNEWB+5T1emYo9AVK0NsAMwRkUEAqpoLiKp2VdVvorA72NwAS0ZfEx6+FuinqktE5Cys2sLZTuKSnlWAcSJSPoRtr8UqWJYATYA6YguINpmP25FhrVQLs2xymhg+8N2wOPTF2LRn3cYxrCK0Mw2rLOiNLd7oBHwT88LF6uT3BtJU9auitm9bEJG2wEVY+eQA4CUsb9EO2A/zVJqqaiSx4x2BiPTBvqNpQLcg6DHPrAY2w6qlqt0TIVQRvL0KmGd4HlbvPFBVdw/j7bFCgI6qOi8yQ5OYuFDWG0C2qv53o/GKWFjrDqw/0b2q2rcwbUqpBKiqrpW45csAIat8I3Ap5jFeFr6EyC46VV0D9BGRgZhXOxoYICIvYMmS1Vg4Jhn4DKiNhYj+xmqt3wByMa/86WQTchF5GngsllPBWhAcHP7dICIvYWWvS4FZIjIAC69ESixRi61RmCciP2PnVzOs5jlGOlDBhXz7iPPKD8ZWNFePGzsPc8x+BV4Vkc+wG+rdIjI5lmguDFLGM9+clx283PKYd/hx0Vu2ZURkXyxhmwPcCbyf6BdaqKldpKozwu9HYYsgRgMPYt7gUlV9Lzort50wa+oL3Bgu2L1V9acwdgE2jV4NvIjlA36O2hOH/Hrm/YHfsLxFF0y0+wGHYqGhCVgV1Z1Yf5wXE2EmkayEhPhzoQwVEWmOrf6tHMoTi8U5lXVi10phkZQx87jEw94i0ho28MQ3qN8OtapZwF8bjycKqvoDUBe4Daspvzxai7aKp4CfROQWEbkcS7odAywEjlPVkckm5GCzJlW9Pgj5EdjijuEiUldVn8IEcwwWMx+K1ZsnAiWxRPM0YA9VnRvE40YsPr4rlqu5AHgwVibqQr5tyPrupl2AdFXtH6cpjwB9g5CXiuVbROQWYHZh25aUYh6XtLwL6CsiV4uIhLF4UY9VIgwiTDMTtY5WbWXYCKAc5tkmJGLNvkpigpaGJXt2wUTiPWzaeZdY/XxSE2ZyZ2BVCpNF5GZgtapeDxyLJblyIzQxH7VVqD2x72QPsSXix6jqTFUdDFysqtWAIzT01pYEb9qWiMTd/O7EihhiCedzgLqqemsYj2nU48AJRXHTTOowi1gf6S7AAdiCjXew5lR/xx3TCFCgTqhgiSTpmYqILbY6Hpu+v4qJXidgX6BnLDyRTMQltsoDJWIlqyJyDZa0XgDcoAnQRCuekBtaAxyN5TFGAMdhfXBuVtVMsQZo41T1j+gsTX7EmpYNxFpqf4uFsiYA3VV1lKxvf9sQK9ttrKqzCzukldRiHkOsSU03rAvZt8AoYLyqrhCRCVhC4hqPDxYcETkcS/ZNDmVuXbHE4IMaeseISJVkS3rCBkK+G+ZRHYAlP29W1T/Ees/cgVVETQSOVtXl0Vm8gc0lQgFA/mcfqlb6A42wmHkLoLYmx9qFhCZEAtpjq7SbAKVUtepGx3yEJcqvKgrtSSoxl/VtJmtjLSTLA4+o6tSQuOqKLdQohlWuFAOuCdPLzSZJna1DrFHZJKxx1C6YQDwHXIfVXF+jqs9GZ+GOIZSblcXOoXOwmv8HsZV9S0XkUOBQVb0vUc4psXYCR2DtHz4DXtawGCjkNFoBr6vqy+7UFAyxPuULQyj3YEzQ22Eht+fU2le0AkYWpfYkjZjHeSAHY/2Cy2En7f1qbSVjx9XEEohtsWn/uar6QlzZllMAxFoQ7IeVY3XDqiNex/qR7IGJ3A/RWVgwxFrW3oMJ93wRKY3lW27Dkrx9VfWxuOMLdYn2FmyNeeP/CzYOx8TlEOzaeFVVX4vSxlQgTnv2xHJD5wPnq+qYMF4O6wh6GrY1329YTqWnqj5VVNqTNGIeQ0R+xZbpD1PrAR57PPaBV1Jrn3kYcIiq3heVrTsDYm1K98UWZbVV1XYRm1QgQpnolcDEjUS7Wnj8eoKDEJGJGxBi5X8Cp6rqlyLyOHAYVtXSGPgOE/gOyXyTTQRE5DvgK2x2OlZVp8fdUEtiBSWnYXm8pap6QlHalxRiHhde6YLd7fbeeCz8vD9WI3yWqi6OO8Y9E2ezxF2Ql2HnTy62svNOVb1lo2Mbaui2mQiI7VwTS3bGEnLVsFnEp1jjtqmqenNkRiYxcU5iT6CrqjaNG6uA9SFvge3b+6CqvhXWYMxW1QVFGdJKitKkuFjTQVhiinAn3KDUUFUnYcmIYzd6vgt5ISHrG1ElXP3+1hJ3sf0XS26egi13P0NE/hCRs+OOjVzI42qdi4UQYxVsyXhnbOu33JCYHY/NYG8OxyfF9Z5IBCEvjp0X+RvGhBnpEKx09XNClZOIVFPVH3X9hiVFlptIii9X1m/smo3Fa9H1OwjFjon9LZ8DkW7VtTMRu5kmQhJwe5D1C9BaY2Vko1X1a6wh24WEznci8qOINIr6phXsbRDsuDOEfxqFMtAVWI35LuHwhthiNMCdmu0hfN51sVYV8XHvgUBl4AxV7YXlWapgn3kkJLyYi0jJuLubAoeK7VyzsVcYE5PabPihO85mieVZsHh4J2y3+l1UdYnaruq3YQn1WVjP+6hvWtWDPR9jibiFcSL9Ppa7+EJEnseqWxIitp+MBO1Zp6pTsZ7854vIlWLtbathYbj3wwzpB6wDavmo7E14Mce2IGscfh6Drax6UETqhTh6cbXVk3lirT3rq+oz0ZnrJBPBEcjG4s5vYSVmT4nIiWCtibF1C+eGhTeRXDPBA4+1UX4Uq1ipDFwvIk3EehB9hd2UPsHaKpwVcgEp1VCvCHkvVLCA9YM/GGvYVxz4T7jZx1aAngnUU9Wx0Zia4AlQEbkRKwFqHPdYI6zNamOsF8IErC/FUdiFeL2qvuGliM6/samkeMjDdALOxoTyS+AFTYA2xCIyBZvSTwqJt1tZvwXcQmyaPyJUcu0PfJsAs4ikZTPaUwHbfm9e/GKxcKP9BrhaVV+JSnsSXcwfwqaUV2De0czYKjdsitkD29GlOPA18KSqjozIXCcJEZHOWDnZXKzyYzh2PnXFHITdsIv6uwhtrIiV2Y4JP5+uqkPDWBq2GUs37ObzCra/Z2tV/TEqm5OdjbRnNKY9azY6phNWRVQfmKWq5xW1nfEktJgDhBDLqPDrbdjGyzlx482xrbBy1PqA+0pP51+JK0W8Fus1PQlb8VkDy7c8oKpvim28cZCq9ovQ3A0Q27vzFay2vJ+qPhceb4BVetXD+hP19OugYGxCe95R1ewwVhJbHXwucD/wqVq3xMhW1ya8mMcIMcyXsdrZ27CeB5H2xXCSl1DxMRM4SVU/D7Hz47AwS2PgIlX9LW6NQ5QrPdNUdU1IyK0KZXHdsBYKP2ObaX8Vjq0LzFHV1b6+YscQpz2fALdjG2Cv2sRxkd48k0bMIX9KeRPWvW4YVh40JXbCRv1hOsmDWFuIfqp69EaPVwXGAV+rapdIjNsMIjIMuwHdraorReQEoDu2/uI1LF+0IEobU5XNaM9fiRQNSIZqlnzUNg64DSvP2h3rP3GliNQKh+wvIvWjss9JKmYATcQ21ygF+RfkAizmXDZUiERK3AKh3bGdg3pjGwifrrYBeEdsq742WLWXUwhsRnt6JJL2JJWYx1DVLFU9Dmum1Qt4S6wN7mig6b8+2XEAVZ2FtYc9Fugutj1czLPqhlUsrI5ykVC4ucTirw+xvoRyBdBbREZivbKHYn/HBeF5JTb1ek7BSWTtSaowy6YIF9uZWBXCj6p6QMQmOQlIXNKzAnbRVQE+ICzbxzY3KYYtDhFgn7CgKLLpc1y8/h5sh6BDwuN1sTLcnli1zfPAQFVA1DEgAAAD/0lEQVSdGYWdOyuJpj1JL+YxQna5jIadYRwnhmzYjG081u4hE7hSrc3tPlgL34rYSs/Rqjo5ysqEGEEwBmCbH1y+0di52GYZf2Cz7D6q+mkixG93JhJFe1JGzB1nc8j6znd9sOnxfzAPfHXw1vcAFmuC7o4URPsZoFtcfXnMax+MVXh1xapYzv6Xl3JSmKSMmTvOthCEPA0LrzwQkpzxpWXtgA/DgpzIkfXNv0qEn98BrgIuFZE3RKRlEPIjgPaq+hKWtG0YEqXOToj3bHB2CkKddiZwk4j8oKGVbRDLd7GdeqoBi//lZYqEuNrwu7HdslZioZRnsRr418V2QFoUHgOoAFTUuF23nJ0L98ydnYkngSVY3+ljRaR8EM51wF5A5GGWuFLEntgippewhUGnACdhFS3tsWZbZwH9ROQCrGXvXVHY7CQGHjN3dgpEpCy2g9D+2ObM6ZhIlgNqAZ+o6jUJkvQsAfwOdFbVz0TkYaA1Fhtvhy3lb4LF/pthJXKPqupD0VjsJAIeZnFSkrgl8IcDFwF1sJ73j2Kb716BbTqwFnhRVV8JT02E5e9Vsd7YS0SkHtY/Zg9sVvEeMA/4WFUnAhNF5CVVXRqZtU5C4GLupByh0iPW4e5l4Cksifg/4DJVfRUr99vU8xJhqpqNhUDTsJLJD1V1PoCIvIO1t30v/J7mQu6Ax8ydFCSupvwm+1VvBMZii4G6h7GrQ9/vfzwvalR1BXBwaLubA9SLay3QAovvx471nv0O4GLupDaxHeoBngNGquovIlIDW7m3R2SWbYG4G8vH2GKmaSLyInAM8Bgk9ybazo7HxdxJZTKxvis3AG1UtTOAqs4BpmL7xSY0qvoL0AF4AtsB/lRVXRHCKwkxk3ASA69mcVKGuKRnKWxj3apYxUdXYDbrS/uOxRpW1VLVxQkUK3ec7cYToE7KEBc/Ho7FlRsAq4HfsEZaEzGBnwLcFIQ88lJEx9kRuJg7SY+INFbV38PPPYB9gWuwVrEdsdLE0kAXbEPeH1Q1Mzw9EUoRHafAeJjFSWpE5BRsl537sXLDHsAEVX0vrhnV4dgKyUmqen501jpO4eEJUCepUdXXgQuB04CvsWZaR4X9MmOeymfYRhTNRCThk56Osz24mDtJj6o+jS1rfwI4DLgE6BAr3Qv9V8ZgMXTHSUk8zOKkFGEfxjuxOvKRwFCsD8sZAKra2Xetd1IR98ydlEJVp4YNGtoC9bGVn89h3QcviNA0xylUXMydlERVxwMtsSZVZYHSsRJE98qdVMTDLE7KIyLlVDUnajscpzBxMXccx0kBPMziOI6TAriYO47jpAAu5o7jOCmAi7njOE4K4GLuOI6TAriYO47jpAD/B68b83Ai51y4AAAAAElFTkSuQmCC\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "to_show = np.expand_dims(output[0, 1, 1, :crop_len], 0)\n",
+ "plt.imshow(to_show, cmap='Reds')\n",
+ "plt.grid(False)\n",
+ "plt.xticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], rotation=60, fontsize=14)\n",
+ "plt.yticks(ticks=range(1), labels=['he'], fontsize=14)\n",
+ "colors = ['black', 'black', 'black', 'black', 'white', 'black', 'black']\n",
+ "for i, value in enumerate(to_show[0]):\n",
+ " plt.text(i, 0, \"{:.2f}\".format(value), ha=\"center\", va=\"center\", weight='bold', color=colors[i])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 371,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXMAAACACAYAAAD5y4mRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAHxdJREFUeJztnXeYVdXVh190BD9FiiBNAQFhGbvYY8OOWBJji7GbIBjBbmKsGAWNJbFGxNh7wwKKBRUbYsGe6E9BRUFABSkKKMh8f6x94YKAMANz7j2s93l4mLn7nJk1957zO2uvtfbatSorKwmCIAjKmxWyNiAIgiCoPiHmQRAEOSDEPAiCIAeEmAdBEOSAEPMgCIIcUJHZb542OcpogipR+f2krE2oFsc12SBrE6pF3+9HZ21CtSj366fWGq1rLej18MyDIAhyQIh5EARBDggxD4IgyAEh5kEQBDkgxDwIgiAHhJgHQRDkgBDzIAiCHBBiHgRBkANCzIMgCHJAiHkQBEEOCDEPgiDIASHmQRAEOSDEPAiCIAeEmAdBEOSAEPMgCIIcEGIeBEGQA0LMgyAIckCIeRAEQQ4IMQ+CIMgBvyjmZlZpZnvXhDGLywmn/YWGa7bGNtmcocNenWfsyaefod0Gm7D6Wmtzzt8vXKxzapqq2N//kUdp0KIVnTrvVdPm/oyyt//MXqzefiPW3WZnhr42fJ6xJ599nnW22IFGHTbmnIsvn/P6lf1uolGHjWm23ubc3f+RmjZ5Hg668hIun/g5vT58k7bbbDXP2JaHHkyfUf/jiqljOe6Re6m9yioceXNf+lZOnedfo9atMrG97O/dEr52alVWLnpfZTOrBPaRNHCp/uYqbug8+Nkh7L7vfgwbMpjrb7yZ4W+9zdvDXgKgsrKSVrYBB+73G7rssTu77fNb3hr6At9MmLjQc2qaqtp/zHE9qFt3VZqs0ZghTzyWie2lYn91NuQd/PxL7HHQ4bwy6CGuv+0u3nznPd56btAc+1tv+msO2KcLXXbdid0PPIw3n3mMiooKNu7UmYF33cTQ19/k2Rde5qXHHqRWrQXuq/uLVGdD53V36cQJTz3CJVvvzPbdjqHVZpvQe9NtAai1wgpcOXUsj5x9AR8Ofo6z3h7KPcefwtCbbmfFlVYCYMfjj2W7Px1Br19tzuyffqqSDVXd0LlU7t2qXj+lcO1AjjZ0HvLiS7Rca0223HwzuuyxG++89z4TJ34LwMhPPmX0mDHs06Uzu+7cibp16/LcCy8u8pxysL9Vy7V4a+gLdFinXSY2F1P29g8dRss1W7Blx03osstOvPPfD5j4rd/cIz8dxegvx7LPHruw647bUXfVVXnu5VcY+PQztGnVkj132YkLzjiVlx/vX62bsTp06LQ9334xms9eH877jz9Jy002YpWGDX2wspKZM2bw/YSJTPpyHJWzZ/PDd98z68cf+eH771mhooLdTz+Rgb0uqrKQV4eyv3dL/NpZXDFvYmZPmtl0M5OZbVoYMLMNzWywmX1rZt+YWV8zW3mZWAuM/+or6terB0DDBg0AGDd+/JwxYJ7xsePGL/KcmqYq9ndovw6NGq2egbU/p+zt//ob6q+2GgANG9QHYNxXX88Zg2L76zN2/Nd89vlo6tSpzT6HHkP7LXfkuptvz8Byp17TJkyfPAWAaUlI6jdrArh3eFf3kzjshqu5eIwY8eJQXrvrvjnndupxLDOmTuX1ex6oecPJwb1b4tfO4op5N+AEYA1gFPAPADNbBXgCeB5oCnQENgfOW+qWBkFG1KpVi89Hf8lZJ/dgv706c+JZ5zM+3cSlRJ1VV+Wgqy5lwHl9uHLXfWmz1eZs1/UowEMwnXp044W+N1E5e3a2hi5H1OS1s7hifqec74CHgPXS63sBtSVdIOlHSZ8DfYCjlr6pTvNmTZmUPJNvJkwEoEXzZmnM/580eTKVlZVMmDiRFs2bLfKcmqYq9pcSZW9/kyZMmpLsn5jsT55t86b+/6TJU9z+b7+lRbMmNG/ahDatW7L15h3Zf+/OzJo1i1Gjx2Ri/+Sx41gleYV1Gzdye78c5/av/yvqN2vKG/c8wIiXhvLVxyNpv4PH09ttuw31mzXlvQGPZ2I35ODeLfFrZ3HF/NOir6cDhTBKO6CRmc0o/APuAhqbWZ2laOccdt2pE1+MHs2w115n4KAn2GKzjtSpU4cZM2bQts3atG2zNo8+NohBTz7NtGnT2G3nnRZ4ToM0ZatpqmL/1KlTGTHyE77/fhrTp89gxMhPmD59ethfFft33JYvxnzJsDfe5LGnnmWLTTemTu1k/9qtaNu6FY8++TSDnhnCtGnT2W3H7dlnj13QiE94dfhbDHpmCCuvXId12qydif0fDB7C6q1a0marLdhw78589tobzPrhByrq1GHiqM/5aeZM1tt9Z5paBxq3a8NXH40AoN22WzNj6lTGfqBM7IYc3Lslfu1ULOZxC6s8mQ5I0q+Wkj2/yA7bbcspPY+ny+8OpEWz5tzS79/sud8BNKhfn4fvvYsbrrmSY3uexJ333U/vXuew/npu2vznZEVV7L/l9js5uvvxc35G+4068tygAXTaYfuwf0nt32YrTu7+J/b6w9G0aNaUm6+6jC6HHEmDevV46LYb6PfPi+l26t+468FHuPDM01l/3Q4A9D7zdH57RFcqKiroe2kfVm+YjaCMePFlnr78ano8/gCTvhzHrUd1p+eg/kybNJm++x3Cnd1OZO9ef2P/y3rz4eAhPHOFX+sNWjRjyrhsYs0Fyv7eLfFrZ4lLE83sKOAySY3NbC/gAaCppClpvCEwW9LkRf7gKpYmBkF1ShNLgeqUJpYCVS1NLBXK/fpZWGni4nrmC+NJYCzwLzM7BagN3AJMAI6o5s8OgiAIFpNq1ZlLmgX8BmiLi/r7wFdAj+qbFgRBECwuvxhmWWZEmCWoIuU+TY4wS7aU+/WTmxWgQRAEwc8JMQ+CIMgBIeZBEAQ5IMQ8CIIgB4SYB0EQ5IAQ8yAIghwQYh4EQZADQsyDIAhyQIh5EARBDggxD4IgyAEh5kEQBDkgxDwIgiAHhJgHQRDkgBDzIAiCHBBiHgRBkANCzIMgCHJAdptTBEEQBEuN8MyDIAhyQIh5EARBDggxD4IgyAEh5kEQBDkgxDwIgiAHhJgHQRDkgBDzIAiCHBBiHgRBkANCzIMgCHJArsXczBqb2RpZ21FVzKxW1jYE+cDMdjKzNbO2Y3nBzLYzsxvMrHlN/c5cL+c3s7eBKcDZwKuSfsjYpOUGM1sP+BPQAjhc0syMTVpuMbNtgSeB24EHgNclTcnWqnxjZh8BqwLfA7dI6rOsf2duPXMz6wk0BL4BngWuMbN1zawk/2YzWzH9v6GZHWxm55vZvlnbVQ3649fXAEkzzayJmbUws5ZZG1ZdCjOm9Fk9a2ZbZG3TopD0MtAD2Aq4HDjRzDYtXHPB0sXM9gIqgc2Am4Dfm9lbZnbAsvy9ufXMzew44FFJY8ysM3A10Aj4B3CrpHGZGliEma0gabaZrQ68DEwG3gG6Aj0lXZupgUuImZ0MHClpk/T9bvj7Xwd4CThZ0jcZmrhUMLP9geuAGfjD61+SRmVr1bwUHjySKs2sDnA98FtgBNAPGCzpkwxNzCVmtoqkaWbWCPgVcBCwN/ABcLakt5b27yxJL3VpIOk6SWPS109Iag/0As4BnjGzA81stSxtLKLwRL0GGCppa+BG4DvgTgAz62xmtUp1ZlEgice6uEeCmZ0AnAG8AvQEdgJ2zszAamJmFen/Y/Ab9DHgPWAb4GEzO8nMVsrQxHmQVMnc+/wCYBxwMzAe+DtwqZn9xswaZ2TiQim3mcN8Oa7GZra6pAmSXgIuAk4GpgOPmtm1ZtZwaf7+XHnmZlYreSArAe2A3wC3AN9K+jEdUxv3po4GBuEX+PCsY7rJK38COE7ScDMbDjwm6Vwz2yXZuWPWdi4OaVZ0LfAgsC9wIvCIpLFmdiPwsaSLs7SxOqRraAqwm6QX02s7AYcCvwOGA/0k3Z+dlU7RPbEB8CqweiF3lGy+HGgD3AY8LOm57KydSxLGk4GRkh7J2p5fouh9roPP/rcDDLgPOFbST+m4NsCuwJFAU+AaSVcuDRtK2surAoUn43nAf4CjgI0k/VjwaCX9KOmPwPpAPeB54DIza1vwurJA0kTgK2BLM9sRaCLp3DT8P6AC2CAr+5YESdcBp+Azi2Ml9U1C3hQ4GL/Ay5l1gI+AzwsvJBH8K/Am0AA42cx6mdmq2Zg4x66Ct7Y+bnNF0dhzkjoCI/F7pZSqXRrgDtc2WRuymBS0599AW+AK4BCgE3BvEnkkfQrcChwP3AF0M7OhKc5eLTITr6VNejLONrONcE9wd9zj/jEdsrGZtQJWBoZI+gDYPsU9LwIOB47Fs/1Z0R+4FE/c9ix6fQOg2bKIsy1NklivC3wh6Yr5xi4GtsfzFeUeox0LrALcamYHFOL/kiaYWV9cOKfjN+yTeIgpa57Hq7rON7PrJI0sGrsTGCfp7mxM+zmSvjWzjsD/ZW3LLzGf9vweaCFpchqrh+tKHTP7UVJl0qR3zGw0cD/ueF6Mh+yqTG488yIP5Ergn5JeSR55XTM7DHgRn0qeBRxlZiulD+FBSR1wT6DGSheL42tmtkKy5SbgTGAY0NPMbjSza/D487kL+VElgZntCTyOeyZPmVkfM6ufxtbHvZX78aqKskbSt8D+uNA8Y2YnFg0fAUyTdAnwLrBjBib+jJTwvwToDPw95YzapuET8YdPSa1tkDSzHEooi7SnH3CJpMkFTxx4DteWhkXHFc6bIOl/wBZ4/qJa5MYzBzCzDrjHdE3RyycB++Cx85OBU3FhfFDSiHTeb4CBkgbUoLkrArPM7Cj8hl/LzPpJut7MBgP74dnv0cBfSslrWgg3ABcCzwCX4e/zAWZ2qaQbzOyhMvgbFoiZVUiaZb7oZgOgtqQBZvZ7/OHUzcx64xUijfHEKEBzYEJGNhcqpJonOxoCj+BhoGvxQoCVzKwS+EjSwzCPMAVLgJntDWyZiheKORcYJWlUIa5edE5Bf/+yNPIreUuArgYMxMuvBgJ74SVx/wKulPSdma2Me4h9JL2SzmsEzE4e17K2sREwNc0amuNx1/vxWdKOeOnSKZLeTsevLGnGsrarOpjZocDpkjZJnt0UPGa+DR6L/Q6YImmt7KysPmb2Lu4AtcTjz10lvWm+QGpH/DO8DU++HwN0kbROBnYWhLwJfm2tDfwXT7x1lPS++RqGNYEvgNckfWVmKxYSdcGSkR7s1+EP779Lui29/6OA9pJGFz6XZWVDbsIsiR+Br3HP8EXcG78SuELSd+mYZvi05nOYc+FPqCEhb4F7RkebWQN8Sn6ppD/gU92uuBA+Z2a3mFmLUhfyxNvA1FTl0Q94XNINQF889HIsJRJuWFJs7gKhE/HPpjM+0/sEeMPM7gW+TqWw1wI/4R5wHTx+mgUFD+0/wAhJrfG4+A/MTdoOTTYPlPQVQAj5klEoqjCzVSXdg+vKY8ANZvYs8DQedhltZnUWJORLM6yVKzGX9IOkA3ABHwAcIOkCSd8XHfZvvExuTPJEltmTcgH2fYl7rJfiHtP3eAULksZLGoiL+snAJsArSSBLFjPrhy8TfzcldjrgYgbwBvAZLhwjF/wTSpf0oK9MN+0KwGWSPscdhW54GWJbYKyZXQkgaRpgkrpJeiMLu5PNbfFk9Gnp5dOB3pKmmNkheLVFUEWKkp6NgMFmVi+FbU/HK1imAOsBLc0XEC0wH7c0w1p5C7MscJqY3vA18Dj0n/Bpz+z5Y1g1aGcFXllwLr5440DgjYIXbl4nvyFQIem1mrZvSTCzXYE/4uWTlwP34HmL3YFNcU9lfUmZxI6XBmbWC/+MRgHdk6AXPLPm+AxrTUk9SiFUkby9+rhneDhe73yFpDZpvDNeCHCApPGZGVrGFIWyHgYmSzpyvvEGeFjrArw/0aWSLlyWNuUqASrpJytavgyQsspnAsfhHuOf04eQ2U0naRbQy8yuwL3aAcDlZnYXniyZiYdjyoGXgbXwENHXeK31w8A03Cu/pdyE3MxuAa4v5FTwFgRbp39/M7N78LLX74AxZnY5Hl7JlEKiFl+jMN7M/otfXxvgNc8FVgLqh5BXjSKvfGt8RXOzorHDccfsA+ABM3sZf6BebGbvFxLNy4LceOYL87KTl1sP9w5fqHnLfhkz2xhP2E4F+gBPl/qNlmpqv5X0Rfp+Z3wRxADgKtwb/E7SU9lZueSkWdOFwJnpht1Q0ntp7Gh8Gj0TuBvPB/w3a08c5tQzbwZ8iOctuuKi3Rv4NR4aGoJXUfXB++PcXQoziXIlJcTvSGWomNkm+Orf1VN5Yq0ip7Jl4V5ZVpRlzLwo8bChmXWCeTzxeeq3U63qBODT+cdLBUnvAK2A8/Ga8uOztWixuBl4z8zOMbPj8aTbbsBEYE9J/ctNyMFnTZLOSEK+A76441YzayXpZlwwB+Ex8354vXkpUBtPNI8C2kkal8TjTDw+/n94ruZo4KpCmWgI+ZJhc7ubdgVWknRJkaZcB1yYhLxOId9iZucAXy5r28pSzIuSlhcBF5rZqWZmaaxY1AuVCFeSppmlWkcrXxl2L7Aa7tmWJObNvmrjglaBJ3tWxkXiKXzaeZF5/XxZk2ZyB+NVCu+b2dnATElnAHvgSa5pGZo4B/kq1BPwz6Sd+RLx3SSNlnQN8CdJTYAdlHprW4k3bStFih5+ffAihkLC+VCglaTz0nhBo/4D7F0TD82yDrOY95HuCmyOL9h4HG9O9XXRMe0BAS1TBUsmSc88Yr7Yai98+v4ALnoHAhsDJxTCE+VEUWKrHrBioWTVzE7Dk9bfAH9TCTTRKiblhmYBu+B5jHuBPfE+OGdLGmneAG2wpI+zs7T8MW9adgXeUvtNPJQ1BOgh6VGb2/52Hbxst4OkL5d1SKusxbyAeZOa7ngXsjeBR4FnJc0wsyF4QuK0iA9WHzPbHk/2vZ/K3LrhicGrlHrHmFmjckt6wjxCvgbuUW2OJz/PlvSxee+ZC/CKqGHALpKmZ2fxPDavmAoA5rz3qWrlEqA9HjPvCKyl8li7UNKkSEBnfJX2ekAdSY3nO+Z5PFF+Sk1oT1mJuc1tM7kW3kKyHnCdpM9S4qobvlCjFl65Ugs4LU0vF5okDRYP80Zlw/HGUSvjAnEH8Be85vo0SbdnZ+HSIZWb1cWvoUPxmv+r8JV935nZr4FfS7qsVK4p83YCO+DtH14G7lNaDJRyGtsBD0m6L5ya6mHep3xiCuVujQv67njI7Q55+4rtgP41qT1lI+ZFHsjWeL/g1fCL9p/ytpKF41rgCcRd8Wn/YZLuKirbCqqBeQuCTfFyrO54dcRDeD+SdrjIvZOdhdXDvGXtP3Dh/srMVsHzLefjSd4LJV1fdPwyXaL9C7YWvPG/JhtvxcVlG/zeeEDSg1namAeKtGddPDd0FHCUpEFpfDW8I+j++NZ8H+I5lRMk3VxT2lM2Yl7AzD7Al+nfIO8BXni98IY3lLfP3BbYRtJlWdm6PGDepnRjfFHWrpJ2z9ikapHKRE8Chs0n2k3S62eQHISMTJyHFCv/BPidpFfN7D/AtnhVSwfgLVzgu5TzQ7YUMLO3gNfw2ekTkj4veqDWxgtK9sfzeN9J2rsm7SsLMS8Kr3TFn3Ybzj+Wvt4MrxE+RNKkomPCMwkWStEN+Wf8+pmGr+zsI+mc+Y5dR6nbZilgvnNNIdlZSMg1wWcRL+GN2z6TdHZmRpYxRU7iCUA3SesXjdXH+5B3xPftvUrSwLQG40tJ39RkSKssSpOKYk1b4Ykp0pNwnlJDScPxZMQe850fQr6MsLmNqEqufn9xKbrZjsSTm/vhy90PNrOPzewPRcdmLuRFtc61UoixEb5k/Ah867dpKTH7LD6DPTsdXxb3eymRhHwF/LqYs2FMmpFei5euDiVVOZlZE0nvau6GJTWWmyiLD9fmbuw6GY/Xork7CBWOKfwtQ4FMt+panig8TEshCVgVbO4CtE54GdkASa/jDdmOIXW+M7N3zax91g+tZG/bZEefFP5pn8pAZ+A15iunw9fBF6MB4dRUhfR+t8JbVRTHva8AVgcOlnQynmdphL/nmVDyYm5mtYuebgJ+bb5zzfxeYUFM1mLeNz0IFkohz4LHww/Ed6tfWdIU+a7q5+MJ9TF4z/usH1rNkj0v4Im4iUUi/TSeu3jFzO7Eq1tKIrZfjiTtmS3pM7wn/1FmdpJ5e9smeBju6TRDegfvgFovK3tLXszxLcg6pK8H4SurrjKz1imOvoJ89WSleWvPtSXdlp25QTmRHIHJeNx5IF5idrOZ7QPemhhft3BYWniTyT2TPPBCG+W+eMXK6sAZZraeeQ+i1/CH0ot4W4VDUi4gVw31apCnUgULeD/4rfGGfSsAv00P+8IK0N8DrSU9kY2pJZ4ANbMz8RKgDkWvtcfbrHbAeyEMwftS7IzfiGdIejhKEYNFsaCkeMrDHAj8ARfKV4G7VAJtiM1sBD6lH54Sb+cxdwu4ifg0/95UybUZ8GYJzCLKloVoT318+73xxYvF0oP2DeBUSfdnpT2lLuZX41PKE3HvaHRhlRs+xeyJ7+iyAvA6cJOk/hmZG5QhZnYEXk42Dq/8uBW/nrrhDsIa+E39VoY2NsDLbAelrw+S1C+NVeCbsXTHHz734/t7dpL0blY2lzvzac8AXHtmzXfMgXgV0drAGEmH17SdxZS0mAOkEMuj6dvz8Y2XpxaNb4JvhTVV3gc8VnoGi6SoFPF0vNf0cHzFZ3M83/IvSY+Yb7yxlaTeGZo7D+Z7d96P15b3lnRHer0tXunVGu9PdELcB9VjAdrzuKTJaaw2vjr4MOCfwEvybomZra4teTEvkGKY9+G1s+fjPQ8y7YsRlC+p4mM0sK+koSl2viceZukA/FHSh0VrHLJc6VkhaVZKyP2YyuK64y0U/otvpv1aOrYVMFbSzFhfsXQo0p4Xgb/jG2D/uIDjMn14lo2Yw5wp5Vl497ob8PKgEYULNus3MygfzNtC9Ja0y3yvNwYGA69L6pqJcQvBzG7AH0AXS/rBzPYGeuDrLx7E80XfZGljXlmI9nxaStGAcqhmmYN844Dz8fKsNnj/iZPMbM10yGZmtnZW9gVlxRfAeuaba9SBOTfkN3jMuW6qEMmUogVCbfCdg87FNxA+SL4B+AH4Vn074dVewTJgIdrTs5S0p6zEvICkCZL2xJtpnQwMNG+DOwBYf5EnBwEgaQzeHnYPoIf59nAFz6o7XrEwM8tFQunhUoi/Xs3cEsoZwLlm1h/vld0P/zuOTuetuKCfF1SfUtaesgqzLIh0s/0er0J4V9LmGZsUlCBFSc/6+E3XCHiGtGwf39ykFr44xICN0oKizKbPRfH6f+A7BG2TXm+Fl+GegFfb3AlcIWl0FnYur5Sa9pS9mBdI2eVVlXaGCYICNm8ztmfxdg8jgZPkbW43wlv4NsBXeg6Q9H6WlQkFkmBcjm9+cPx8Y4fhm2V8jM+ye0l6qRTit8sTpaI9uRHzIFgYNrfzXS98evxb3AOfmbz1dsAklejuSEm0bwO6F9WXF7z2a/AKr254FcsfFvGjghxTljHzIFgSkpBX4OGVf6UkZ3Fp2e7Ac2lBTubY3OZfK6avHwdOAY4zs4fNbMsk5DsAnSXdgydt10mJ0mA5JHo2BMsFqU57JHCWmb2j1Mo2ieWT+E49TYBJi/gxNUJRbfjF+G5ZP+ChlNvxGviHzHdA+ja9BlAfaKCiXbeC5YvwzIPliZuAKXjf6T3MrF4SztnAr4DMwyxFpYgn4IuY7sEXBu0H7ItXtHTGm20dAvQ2s6Pxlr0XZWFzUBpEzDxYLjCzuvgOQpvhmzOvhIvkasCawIuSTiuRpOeKwEfAEZJeNrN/A53w2Pju+FL+9fDY/wZ4iVxfSVdnY3FQCkSYJcglRUvgtwf+CLTEe973xTffPRHfdOAn4G5J96dTS2H5e2O8N/YUM2uN949ph88qngLGAy9IGgYMM7N7JH2XmbVBSRBiHuSOVOlR6HB3H3AznkT8K/BnSQ/g5X4LOq8UpqqT8RBoBV4y+ZykrwDM7HG8ve1T6fuKEPIAImYe5JCimvKz/FudCTyBLwbqkcZOTX2/f3Ze1kiaAWyd2u5OBVoXtRboiMf3C8dGz/4ACDEP8k1hh3qAO4D+kv5nZs3xlXvtMrPsFyh6sLyAL2YaZWZ3A7sB10N5b6IdLH1CzIM8MxLvu/I3YCdJRwBIGgt8hu8XW9JI+h/QBbgR3wH+d5JmpPBKScwkgtIgqlmC3FCU9KyDb6zbGK/46AZ8ydzSvj3whlVrSppUQrHyIKgykQANckNR/PhWPK7cFpgJfIg30hqGC/wI4Kwk5JmXIgbB0iDEPCh7zKyDpI/S1z2BjYHT8FaxB+CliasAXfENed+RNDKdXgqliEFQbSLMEpQ1ZrYfvsvOP/Fyw57AEElPFTWj2h5fITlc0lHZWRsEy45IgAZljaSHgGOA/YHX8WZaO6f9Mgueysv4RhQbmFnJJz2DoCqEmAdlj6Rb8GXtNwLbAscCXQqle6n/yiA8hh4EuSTCLEGuSPsw9sHryPsD/fA+LAcDSDoidq0P8kh45kGukPRZ2qBhV2BtfOXnHXj3waMzNC0Ilikh5kEukfQssCXepKousEqhBDG88iCPRJglyD1mtpqkqVnbEQTLkhDzIAiCHBBhliAIghwQYh4EQZADQsyDIAhyQIh5EARBDggxD4IgyAEh5kEQBDng/wF79PTiILihmwAAAABJRU5ErkJggg==\n",
+ "text/plain": [
+ ""
+ ]
+ },
+ "metadata": {
+ "needs_background": "light"
+ },
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "to_show = np.expand_dims(output[6, 5, 1, :crop_len], 0)\n",
+ "plt.imshow(to_show, cmap='Reds')\n",
+ "plt.grid(False)\n",
+ "plt.xticks(ticks=range(crop_len), labels=bert_tokens[:crop_len], rotation=60, fontsize=14)\n",
+ "plt.yticks(ticks=range(1), labels=['he'], fontsize=14)\n",
+ "colors = ['black', 'black', 'black', 'black', 'white', 'black', 'black']\n",
+ "for i, value in enumerate(to_show[0]):\n",
+ " plt.text(i, 0, \"{:.2f}\".format(value), ha=\"center\", va=\"center\", weight='bold', color=colors[i])"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.2"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/darkbert-main/darkbert-main.gitattributes b/darkbert-main/darkbert-main.gitattributes
new file mode 100644
index 0000000000000000000000000000000000000000..637fa167e56685c01bc97f08a420ea76330cf6df
--- /dev/null
+++ b/darkbert-main/darkbert-main.gitattributes
@@ -0,0 +1,32 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
diff --git a/darkbert-main/darkbert-main.gitignore b/darkbert-main/darkbert-main.gitignore
new file mode 100644
index 0000000000000000000000000000000000000000..0348ea97130c017c407fcfb6fd4003859f17b84c
--- /dev/null
+++ b/darkbert-main/darkbert-main.gitignore
@@ -0,0 +1 @@
+checkpoint-*/
\ No newline at end of file
diff --git a/darkbert-main/darkbert-main/README.md b/darkbert-main/darkbert-main/README.md
new file mode 100644
index 0000000000000000000000000000000000000000..2dfce8ec93ae875fe6b61e627070ad4d74f17d91
--- /dev/null
+++ b/darkbert-main/darkbert-main/README.md
@@ -0,0 +1,91 @@
+---
+license: apache-2.0
+tags:
+- generated_from_trainer
+datasets:
+- conll2003
+metrics:
+- precision
+- recall
+- f1
+- accuracy
+model-index:
+- name: dark-bert-finetuned-ner
+ results:
+ - task:
+ name: Token Classification
+ type: token-classification
+ dataset:
+ name: conll2003
+ type: conll2003
+ config: conll2003
+ split: train
+ args: conll2003
+ metrics:
+ - name: Precision
+ type: precision
+ value: 0.928300642821823
+ - name: Recall
+ type: recall
+ value: 0.9478290138000673
+ - name: F1
+ type: f1
+ value: 0.9379631942709634
+ - name: Accuracy
+ type: accuracy
+ value: 0.9859009831047272
+---
+
+
+
+# dark-bert-finetuned-ner
+
+This model is a fine-tuned version of [bert-base-cased](https://huggingface.co/bert-base-cased) on the conll2003 dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.0639
+- Precision: 0.9283
+- Recall: 0.9478
+- F1: 0.9380
+- Accuracy: 0.9859
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 2e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: linear
+- num_epochs: 3
+
+### Training results
+
+| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1 | Accuracy |
+|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
+| 0.0881 | 1.0 | 1756 | 0.0716 | 0.9172 | 0.9322 | 0.9246 | 0.9817 |
+| 0.0375 | 2.0 | 3512 | 0.0610 | 0.9275 | 0.9455 | 0.9364 | 0.9857 |
+| 0.0207 | 3.0 | 5268 | 0.0639 | 0.9283 | 0.9478 | 0.9380 | 0.9859 |
+
+
+### Framework versions
+
+- Transformers 4.22.1
+- Pytorch 1.10.0
+- Datasets 2.5.1
+- Tokenizers 0.12.1
diff --git a/darkbert-main/darkbert-main/config.json b/darkbert-main/darkbert-main/config.json
new file mode 100644
index 0000000000000000000000000000000000000000..6c6b65a39c5c124209f0c9de62436be4327b28ef
--- /dev/null
+++ b/darkbert-main/darkbert-main/config.json
@@ -0,0 +1,48 @@
+{
+ "_name_or_path": "bert-base-cased",
+ "architectures": [
+ "BertForTokenClassification"
+ ],
+ "attention_probs_dropout_prob": 0.1,
+ "classifier_dropout": null,
+ "gradient_checkpointing": false,
+ "hidden_act": "gelu",
+ "hidden_dropout_prob": 0.1,
+ "hidden_size": 768,
+ "id2label": {
+ "0": "O",
+ "1": "B-PER",
+ "2": "I-PER",
+ "3": "B-ORG",
+ "4": "I-ORG",
+ "5": "B-LOC",
+ "6": "I-LOC",
+ "7": "B-MISC",
+ "8": "I-MISC"
+ },
+ "initializer_range": 0.02,
+ "intermediate_size": 3072,
+ "label2id": {
+ "B-LOC": "5",
+ "B-MISC": "7",
+ "B-ORG": "3",
+ "B-PER": "1",
+ "I-LOC": "6",
+ "I-MISC": "8",
+ "I-ORG": "4",
+ "I-PER": "2",
+ "O": "0"
+ },
+ "layer_norm_eps": 1e-12,
+ "max_position_embeddings": 512,
+ "model_type": "bert",
+ "num_attention_heads": 12,
+ "num_hidden_layers": 12,
+ "pad_token_id": 0,
+ "position_embedding_type": "absolute",
+ "torch_dtype": "float32",
+ "transformers_version": "4.22.1",
+ "type_vocab_size": 2,
+ "use_cache": true,
+ "vocab_size": 28996
+}
diff --git a/darkbert-main/darkbert-main/pytorch_model.bin b/darkbert-main/darkbert-main/pytorch_model.bin
new file mode 100644
index 0000000000000000000000000000000000000000..0298414e68fb5f62450583d539ff08d9f933ce3c
--- /dev/null
+++ b/darkbert-main/darkbert-main/pytorch_model.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f81344e728f5a18bf527fc1c141e71c6554ae790d2c2fc5e068fee0713ac1e19
+size 134
diff --git a/darkbert-main/darkbert-main/special_tokens_map.json b/darkbert-main/darkbert-main/special_tokens_map.json
new file mode 100644
index 0000000000000000000000000000000000000000..a8b3208c2884c4efb86e49300fdd3dc877220cdf
--- /dev/null
+++ b/darkbert-main/darkbert-main/special_tokens_map.json
@@ -0,0 +1,7 @@
+{
+ "cls_token": "[CLS]",
+ "mask_token": "[MASK]",
+ "pad_token": "[PAD]",
+ "sep_token": "[SEP]",
+ "unk_token": "[UNK]"
+}
diff --git a/darkbert-main/darkbert-main/tokenizer.json b/darkbert-main/darkbert-main/tokenizer.json
new file mode 100644
index 0000000000000000000000000000000000000000..de7ac9b6bda3ed337df237120512a771c1c3f519
--- /dev/null
+++ b/darkbert-main/darkbert-main/tokenizer.json
@@ -0,0 +1,29151 @@
+{
+ "version": "1.0",
+ "truncation": {
+ "direction": "Right",
+ "max_length": 512,
+ "strategy": "LongestFirst",
+ "stride": 0
+ },
+ "padding": null,
+ "added_tokens": [
+ {
+ "id": 0,
+ "content": "[PAD]",
+ "single_word": false,
+ "lstrip": false,
+ "rstrip": false,
+ "normalized": false,
+ "special": true
+ },
+ {
+ "id": 100,
+ "content": "[UNK]",
+ "single_word": false,
+ "lstrip": false,
+ "rstrip": false,
+ "normalized": false,
+ "special": true
+ },
+ {
+ "id": 101,
+ "content": "[CLS]",
+ "single_word": false,
+ "lstrip": false,
+ "rstrip": false,
+ "normalized": false,
+ "special": true
+ },
+ {
+ "id": 102,
+ "content": "[SEP]",
+ "single_word": false,
+ "lstrip": false,
+ "rstrip": false,
+ "normalized": false,
+ "special": true
+ },
+ {
+ "id": 103,
+ "content": "[MASK]",
+ "single_word": false,
+ "lstrip": false,
+ "rstrip": false,
+ "normalized": false,
+ "special": true
+ }
+ ],
+ "normalizer": {
+ "type": "BertNormalizer",
+ "clean_text": true,
+ "handle_chinese_chars": true,
+ "strip_accents": null,
+ "lowercase": false
+ },
+ "pre_tokenizer": {
+ "type": "BertPreTokenizer"
+ },
+ "post_processor": {
+ "type": "TemplateProcessing",
+ "single": [
+ {
+ "SpecialToken": {
+ "id": "[CLS]",
+ "type_id": 0
+ }
+ },
+ {
+ "Sequence": {
+ "id": "A",
+ "type_id": 0
+ }
+ },
+ {
+ "SpecialToken": {
+ "id": "[SEP]",
+ "type_id": 0
+ }
+ }
+ ],
+ "pair": [
+ {
+ "SpecialToken": {
+ "id": "[CLS]",
+ "type_id": 0
+ }
+ },
+ {
+ "Sequence": {
+ "id": "A",
+ "type_id": 0
+ }
+ },
+ {
+ "SpecialToken": {
+ "id": "[SEP]",
+ "type_id": 0
+ }
+ },
+ {
+ "Sequence": {
+ "id": "B",
+ "type_id": 1
+ }
+ },
+ {
+ "SpecialToken": {
+ "id": "[SEP]",
+ "type_id": 1
+ }
+ }
+ ],
+ "special_tokens": {
+ "[CLS]": {
+ "id": "[CLS]",
+ "ids": [
+ 101
+ ],
+ "tokens": [
+ "[CLS]"
+ ]
+ },
+ "[SEP]": {
+ "id": "[SEP]",
+ "ids": [
+ 102
+ ],
+ "tokens": [
+ "[SEP]"
+ ]
+ }
+ }
+ },
+ "decoder": {
+ "type": "WordPiece",
+ "prefix": "##",
+ "cleanup": true
+ },
+ "model": {
+ "type": "WordPiece",
+ "unk_token": "[UNK]",
+ "continuing_subword_prefix": "##",
+ "max_input_chars_per_word": 100,
+ "vocab": {
+ "[PAD]": 0,
+ "[unused1]": 1,
+ "[unused2]": 2,
+ "[unused3]": 3,
+ "[unused4]": 4,
+ "[unused5]": 5,
+ "[unused6]": 6,
+ "[unused7]": 7,
+ "[unused8]": 8,
+ "[unused9]": 9,
+ "[unused10]": 10,
+ "[unused11]": 11,
+ "[unused12]": 12,
+ "[unused13]": 13,
+ "[unused14]": 14,
+ "[unused15]": 15,
+ "[unused16]": 16,
+ "[unused17]": 17,
+ "[unused18]": 18,
+ "[unused19]": 19,
+ "[unused20]": 20,
+ "[unused21]": 21,
+ "[unused22]": 22,
+ "[unused23]": 23,
+ "[unused24]": 24,
+ "[unused25]": 25,
+ "[unused26]": 26,
+ "[unused27]": 27,
+ "[unused28]": 28,
+ "[unused29]": 29,
+ "[unused30]": 30,
+ "[unused31]": 31,
+ "[unused32]": 32,
+ "[unused33]": 33,
+ "[unused34]": 34,
+ "[unused35]": 35,
+ "[unused36]": 36,
+ "[unused37]": 37,
+ "[unused38]": 38,
+ "[unused39]": 39,
+ "[unused40]": 40,
+ "[unused41]": 41,
+ "[unused42]": 42,
+ "[unused43]": 43,
+ "[unused44]": 44,
+ "[unused45]": 45,
+ "[unused46]": 46,
+ "[unused47]": 47,
+ "[unused48]": 48,
+ "[unused49]": 49,
+ "[unused50]": 50,
+ "[unused51]": 51,
+ "[unused52]": 52,
+ "[unused53]": 53,
+ "[unused54]": 54,
+ "[unused55]": 55,
+ "[unused56]": 56,
+ "[unused57]": 57,
+ "[unused58]": 58,
+ "[unused59]": 59,
+ "[unused60]": 60,
+ "[unused61]": 61,
+ "[unused62]": 62,
+ "[unused63]": 63,
+ "[unused64]": 64,
+ "[unused65]": 65,
+ "[unused66]": 66,
+ "[unused67]": 67,
+ "[unused68]": 68,
+ "[unused69]": 69,
+ "[unused70]": 70,
+ "[unused71]": 71,
+ "[unused72]": 72,
+ "[unused73]": 73,
+ "[unused74]": 74,
+ "[unused75]": 75,
+ "[unused76]": 76,
+ "[unused77]": 77,
+ "[unused78]": 78,
+ "[unused79]": 79,
+ "[unused80]": 80,
+ "[unused81]": 81,
+ "[unused82]": 82,
+ "[unused83]": 83,
+ "[unused84]": 84,
+ "[unused85]": 85,
+ "[unused86]": 86,
+ "[unused87]": 87,
+ "[unused88]": 88,
+ "[unused89]": 89,
+ "[unused90]": 90,
+ "[unused91]": 91,
+ "[unused92]": 92,
+ "[unused93]": 93,
+ "[unused94]": 94,
+ "[unused95]": 95,
+ "[unused96]": 96,
+ "[unused97]": 97,
+ "[unused98]": 98,
+ "[unused99]": 99,
+ "[UNK]": 100,
+ "[CLS]": 101,
+ "[SEP]": 102,
+ "[MASK]": 103,
+ "[unused100]": 104,
+ "[unused101]": 105,
+ "!": 106,
+ "\"": 107,
+ "#": 108,
+ "$": 109,
+ "%": 110,
+ "&": 111,
+ "'": 112,
+ "(": 113,
+ ")": 114,
+ "*": 115,
+ "+": 116,
+ ",": 117,
+ "-": 118,
+ ".": 119,
+ "/": 120,
+ "0": 121,
+ "1": 122,
+ "2": 123,
+ "3": 124,
+ "4": 125,
+ "5": 126,
+ "6": 127,
+ "7": 128,
+ "8": 129,
+ "9": 130,
+ ":": 131,
+ ";": 132,
+ "<": 133,
+ "=": 134,
+ ">": 135,
+ "?": 136,
+ "@": 137,
+ "A": 138,
+ "B": 139,
+ "C": 140,
+ "D": 141,
+ "E": 142,
+ "F": 143,
+ "G": 144,
+ "H": 145,
+ "I": 146,
+ "J": 147,
+ "K": 148,
+ "L": 149,
+ "M": 150,
+ "N": 151,
+ "O": 152,
+ "P": 153,
+ "Q": 154,
+ "R": 155,
+ "S": 156,
+ "T": 157,
+ "U": 158,
+ "V": 159,
+ "W": 160,
+ "X": 161,
+ "Y": 162,
+ "Z": 163,
+ "[": 164,
+ "\\": 165,
+ "]": 166,
+ "^": 167,
+ "_": 168,
+ "`": 169,
+ "a": 170,
+ "b": 171,
+ "c": 172,
+ "d": 173,
+ "e": 174,
+ "f": 175,
+ "g": 176,
+ "h": 177,
+ "i": 178,
+ "j": 179,
+ "k": 180,
+ "l": 181,
+ "m": 182,
+ "n": 183,
+ "o": 184,
+ "p": 185,
+ "q": 186,
+ "r": 187,
+ "s": 188,
+ "t": 189,
+ "u": 190,
+ "v": 191,
+ "w": 192,
+ "x": 193,
+ "y": 194,
+ "z": 195,
+ "{": 196,
+ "|": 197,
+ "}": 198,
+ "~": 199,
+ "¡": 200,
+ "¢": 201,
+ "£": 202,
+ "¥": 203,
+ "§": 204,
+ "¨": 205,
+ "©": 206,
+ "ª": 207,
+ "«": 208,
+ "¬": 209,
+ "®": 210,
+ "°": 211,
+ "±": 212,
+ "²": 213,
+ "³": 214,
+ "´": 215,
+ "µ": 216,
+ "¶": 217,
+ "·": 218,
+ "¹": 219,
+ "º": 220,
+ "»": 221,
+ "¼": 222,
+ "½": 223,
+ "¾": 224,
+ "¿": 225,
+ "À": 226,
+ "Á": 227,
+ "Â": 228,
+ "Ä": 229,
+ "Å": 230,
+ "Æ": 231,
+ "Ç": 232,
+ "È": 233,
+ "É": 234,
+ "Í": 235,
+ "Î": 236,
+ "Ñ": 237,
+ "Ó": 238,
+ "Ö": 239,
+ "×": 240,
+ "Ø": 241,
+ "Ú": 242,
+ "Ü": 243,
+ "Þ": 244,
+ "ß": 245,
+ "à": 246,
+ "á": 247,
+ "â": 248,
+ "ã": 249,
+ "ä": 250,
+ "å": 251,
+ "æ": 252,
+ "ç": 253,
+ "è": 254,
+ "é": 255,
+ "ê": 256,
+ "ë": 257,
+ "ì": 258,
+ "í": 259,
+ "î": 260,
+ "ï": 261,
+ "ð": 262,
+ "ñ": 263,
+ "ò": 264,
+ "ó": 265,
+ "ô": 266,
+ "õ": 267,
+ "ö": 268,
+ "÷": 269,
+ "ø": 270,
+ "ù": 271,
+ "ú": 272,
+ "û": 273,
+ "ü": 274,
+ "ý": 275,
+ "þ": 276,
+ "ÿ": 277,
+ "Ā": 278,
+ "ā": 279,
+ "ă": 280,
+ "ą": 281,
+ "Ć": 282,
+ "ć": 283,
+ "Č": 284,
+ "č": 285,
+ "ď": 286,
+ "Đ": 287,
+ "đ": 288,
+ "ē": 289,
+ "ė": 290,
+ "ę": 291,
+ "ě": 292,
+ "ğ": 293,
+ "ġ": 294,
+ "Ħ": 295,
+ "ħ": 296,
+ "ĩ": 297,
+ "Ī": 298,
+ "ī": 299,
+ "İ": 300,
+ "ı": 301,
+ "ļ": 302,
+ "Ľ": 303,
+ "ľ": 304,
+ "Ł": 305,
+ "ł": 306,
+ "ń": 307,
+ "ņ": 308,
+ "ň": 309,
+ "ŋ": 310,
+ "Ō": 311,
+ "ō": 312,
+ "ŏ": 313,
+ "ő": 314,
+ "Œ": 315,
+ "œ": 316,
+ "ř": 317,
+ "Ś": 318,
+ "ś": 319,
+ "Ş": 320,
+ "ş": 321,
+ "Š": 322,
+ "š": 323,
+ "Ţ": 324,
+ "ţ": 325,
+ "ť": 326,
+ "ũ": 327,
+ "ū": 328,
+ "ŭ": 329,
+ "ů": 330,
+ "ű": 331,
+ "ų": 332,
+ "ŵ": 333,
+ "ŷ": 334,
+ "ź": 335,
+ "Ż": 336,
+ "ż": 337,
+ "Ž": 338,
+ "ž": 339,
+ "Ə": 340,
+ "ƒ": 341,
+ "ơ": 342,
+ "ư": 343,
+ "ǎ": 344,
+ "ǐ": 345,
+ "ǒ": 346,
+ "ǔ": 347,
+ "ǫ": 348,
+ "Ș": 349,
+ "ș": 350,
+ "Ț": 351,
+ "ț": 352,
+ "ɐ": 353,
+ "ɑ": 354,
+ "ɔ": 355,
+ "ɕ": 356,
+ "ə": 357,
+ "ɛ": 358,
+ "ɡ": 359,
+ "ɣ": 360,
+ "ɨ": 361,
+ "ɪ": 362,
+ "ɲ": 363,
+ "ɾ": 364,
+ "ʀ": 365,
+ "ʁ": 366,
+ "ʂ": 367,
+ "ʃ": 368,
+ "ʊ": 369,
+ "ʋ": 370,
+ "ʌ": 371,
+ "ʐ": 372,
+ "ʑ": 373,
+ "ʒ": 374,
+ "ʔ": 375,
+ "ʰ": 376,
+ "ʲ": 377,
+ "ʳ": 378,
+ "ʷ": 379,
+ "ʻ": 380,
+ "ʼ": 381,
+ "ʾ": 382,
+ "ʿ": 383,
+ "ˈ": 384,
+ "ː": 385,
+ "ˡ": 386,
+ "ˢ": 387,
+ "ˣ": 388,
+ "́": 389,
+ "̃": 390,
+ "̍": 391,
+ "̯": 392,
+ "͡": 393,
+ "Α": 394,
+ "Β": 395,
+ "Γ": 396,
+ "Δ": 397,
+ "Ε": 398,
+ "Η": 399,
+ "Θ": 400,
+ "Ι": 401,
+ "Κ": 402,
+ "Λ": 403,
+ "Μ": 404,
+ "Ν": 405,
+ "Ο": 406,
+ "Π": 407,
+ "Σ": 408,
+ "Τ": 409,
+ "Φ": 410,
+ "Χ": 411,
+ "Ψ": 412,
+ "Ω": 413,
+ "ά": 414,
+ "έ": 415,
+ "ή": 416,
+ "ί": 417,
+ "α": 418,
+ "β": 419,
+ "γ": 420,
+ "δ": 421,
+ "ε": 422,
+ "ζ": 423,
+ "η": 424,
+ "θ": 425,
+ "ι": 426,
+ "κ": 427,
+ "λ": 428,
+ "μ": 429,
+ "ν": 430,
+ "ξ": 431,
+ "ο": 432,
+ "π": 433,
+ "ρ": 434,
+ "ς": 435,
+ "σ": 436,
+ "τ": 437,
+ "υ": 438,
+ "φ": 439,
+ "χ": 440,
+ "ψ": 441,
+ "ω": 442,
+ "ό": 443,
+ "ύ": 444,
+ "ώ": 445,
+ "І": 446,
+ "Ј": 447,
+ "А": 448,
+ "Б": 449,
+ "В": 450,
+ "Г": 451,
+ "Д": 452,
+ "Е": 453,
+ "Ж": 454,
+ "З": 455,
+ "И": 456,
+ "К": 457,
+ "Л": 458,
+ "М": 459,
+ "Н": 460,
+ "О": 461,
+ "П": 462,
+ "Р": 463,
+ "С": 464,
+ "Т": 465,
+ "У": 466,
+ "Ф": 467,
+ "Х": 468,
+ "Ц": 469,
+ "Ч": 470,
+ "Ш": 471,
+ "Э": 472,
+ "Ю": 473,
+ "Я": 474,
+ "а": 475,
+ "б": 476,
+ "в": 477,
+ "г": 478,
+ "д": 479,
+ "е": 480,
+ "ж": 481,
+ "з": 482,
+ "и": 483,
+ "й": 484,
+ "к": 485,
+ "л": 486,
+ "м": 487,
+ "н": 488,
+ "о": 489,
+ "п": 490,
+ "р": 491,
+ "с": 492,
+ "т": 493,
+ "у": 494,
+ "ф": 495,
+ "х": 496,
+ "ц": 497,
+ "ч": 498,
+ "ш": 499,
+ "щ": 500,
+ "ъ": 501,
+ "ы": 502,
+ "ь": 503,
+ "э": 504,
+ "ю": 505,
+ "я": 506,
+ "ё": 507,
+ "і": 508,
+ "ї": 509,
+ "ј": 510,
+ "њ": 511,
+ "ћ": 512,
+ "Ա": 513,
+ "Հ": 514,
+ "ա": 515,
+ "ե": 516,
+ "ի": 517,
+ "կ": 518,
+ "մ": 519,
+ "յ": 520,
+ "ն": 521,
+ "ո": 522,
+ "ս": 523,
+ "տ": 524,
+ "ր": 525,
+ "ւ": 526,
+ "ְ": 527,
+ "ִ": 528,
+ "ֵ": 529,
+ "ֶ": 530,
+ "ַ": 531,
+ "ָ": 532,
+ "ֹ": 533,
+ "ּ": 534,
+ "א": 535,
+ "ב": 536,
+ "ג": 537,
+ "ד": 538,
+ "ה": 539,
+ "ו": 540,
+ "ז": 541,
+ "ח": 542,
+ "ט": 543,
+ "י": 544,
+ "כ": 545,
+ "ל": 546,
+ "ם": 547,
+ "מ": 548,
+ "ן": 549,
+ "נ": 550,
+ "ס": 551,
+ "ע": 552,
+ "פ": 553,
+ "צ": 554,
+ "ק": 555,
+ "ר": 556,
+ "ש": 557,
+ "ת": 558,
+ "،": 559,
+ "ء": 560,
+ "آ": 561,
+ "أ": 562,
+ "إ": 563,
+ "ئ": 564,
+ "ا": 565,
+ "ب": 566,
+ "ة": 567,
+ "ت": 568,
+ "ث": 569,
+ "ج": 570,
+ "ح": 571,
+ "خ": 572,
+ "د": 573,
+ "ذ": 574,
+ "ر": 575,
+ "ز": 576,
+ "س": 577,
+ "ش": 578,
+ "ص": 579,
+ "ض": 580,
+ "ط": 581,
+ "ظ": 582,
+ "ع": 583,
+ "غ": 584,
+ "ف": 585,
+ "ق": 586,
+ "ك": 587,
+ "ل": 588,
+ "م": 589,
+ "ن": 590,
+ "ه": 591,
+ "و": 592,
+ "ى": 593,
+ "ي": 594,
+ "َ": 595,
+ "ِ": 596,
+ "ٹ": 597,
+ "پ": 598,
+ "چ": 599,
+ "ک": 600,
+ "گ": 601,
+ "ہ": 602,
+ "ی": 603,
+ "ے": 604,
+ "ं": 605,
+ "आ": 606,
+ "क": 607,
+ "ग": 608,
+ "च": 609,
+ "ज": 610,
+ "ण": 611,
+ "त": 612,
+ "द": 613,
+ "ध": 614,
+ "न": 615,
+ "प": 616,
+ "ब": 617,
+ "भ": 618,
+ "म": 619,
+ "य": 620,
+ "र": 621,
+ "ल": 622,
+ "व": 623,
+ "श": 624,
+ "ष": 625,
+ "स": 626,
+ "ह": 627,
+ "ा": 628,
+ "ि": 629,
+ "ी": 630,
+ "ु": 631,
+ "े": 632,
+ "ो": 633,
+ "्": 634,
+ "।": 635,
+ "॥": 636,
+ "আ": 637,
+ "ই": 638,
+ "এ": 639,
+ "ও": 640,
+ "ক": 641,
+ "খ": 642,
+ "গ": 643,
+ "চ": 644,
+ "ছ": 645,
+ "জ": 646,
+ "ট": 647,
+ "ত": 648,
+ "থ": 649,
+ "দ": 650,
+ "ধ": 651,
+ "ন": 652,
+ "প": 653,
+ "ব": 654,
+ "ম": 655,
+ "য": 656,
+ "র": 657,
+ "ল": 658,
+ "শ": 659,
+ "স": 660,
+ "হ": 661,
+ "়": 662,
+ "া": 663,
+ "ি": 664,
+ "ী": 665,
+ "ু": 666,
+ "ে": 667,
+ "ো": 668,
+ "্": 669,
+ "য়": 670,
+ "க": 671,
+ "த": 672,
+ "ப": 673,
+ "ம": 674,
+ "ய": 675,
+ "ர": 676,
+ "ல": 677,
+ "வ": 678,
+ "ா": 679,
+ "ி": 680,
+ "ு": 681,
+ "்": 682,
+ "ร": 683,
+ "་": 684,
+ "ག": 685,
+ "ང": 686,
+ "ད": 687,
+ "ན": 688,
+ "བ": 689,
+ "མ": 690,
+ "ར": 691,
+ "ལ": 692,
+ "ས": 693,
+ "ི": 694,
+ "ུ": 695,
+ "ེ": 696,
+ "ོ": 697,
+ "ა": 698,
+ "ე": 699,
+ "ი": 700,
+ "ლ": 701,
+ "ნ": 702,
+ "ო": 703,
+ "რ": 704,
+ "ს": 705,
+ "ᴬ": 706,
+ "ᴵ": 707,
+ "ᵀ": 708,
+ "ᵃ": 709,
+ "ᵇ": 710,
+ "ᵈ": 711,
+ "ᵉ": 712,
+ "ᵍ": 713,
+ "ᵏ": 714,
+ "ᵐ": 715,
+ "ᵒ": 716,
+ "ᵖ": 717,
+ "ᵗ": 718,
+ "ᵘ": 719,
+ "ᵢ": 720,
+ "ᵣ": 721,
+ "ᵤ": 722,
+ "ᵥ": 723,
+ "ᶜ": 724,
+ "ᶠ": 725,
+ "ḍ": 726,
+ "Ḥ": 727,
+ "ḥ": 728,
+ "Ḩ": 729,
+ "ḩ": 730,
+ "ḳ": 731,
+ "ṃ": 732,
+ "ṅ": 733,
+ "ṇ": 734,
+ "ṛ": 735,
+ "ṣ": 736,
+ "ṭ": 737,
+ "ạ": 738,
+ "ả": 739,
+ "ấ": 740,
+ "ầ": 741,
+ "ẩ": 742,
+ "ậ": 743,
+ "ắ": 744,
+ "ế": 745,
+ "ề": 746,
+ "ể": 747,
+ "ễ": 748,
+ "ệ": 749,
+ "ị": 750,
+ "ọ": 751,
+ "ố": 752,
+ "ồ": 753,
+ "ổ": 754,
+ "ộ": 755,
+ "ớ": 756,
+ "ờ": 757,
+ "ợ": 758,
+ "ụ": 759,
+ "ủ": 760,
+ "ứ": 761,
+ "ừ": 762,
+ "ử": 763,
+ "ữ": 764,
+ "ự": 765,
+ "ỳ": 766,
+ "ỹ": 767,
+ "ἀ": 768,
+ "ἐ": 769,
+ "ὁ": 770,
+ "ὐ": 771,
+ "ὰ": 772,
+ "ὶ": 773,
+ "ὸ": 774,
+ "ῆ": 775,
+ "ῖ": 776,
+ "ῦ": 777,
+ "ῶ": 778,
+ "‐": 779,
+ "‑": 780,
+ "‒": 781,
+ "–": 782,
+ "—": 783,
+ "―": 784,
+ "‖": 785,
+ "‘": 786,
+ "’": 787,
+ "‚": 788,
+ "“": 789,
+ "”": 790,
+ "„": 791,
+ "†": 792,
+ "‡": 793,
+ "•": 794,
+ "…": 795,
+ "‰": 796,
+ "′": 797,
+ "″": 798,
+ "⁄": 799,
+ "⁰": 800,
+ "ⁱ": 801,
+ "⁴": 802,
+ "⁵": 803,
+ "⁶": 804,
+ "⁷": 805,
+ "⁸": 806,
+ "⁹": 807,
+ "⁺": 808,
+ "⁻": 809,
+ "ⁿ": 810,
+ "₀": 811,
+ "₁": 812,
+ "₂": 813,
+ "₃": 814,
+ "₄": 815,
+ "₅": 816,
+ "₆": 817,
+ "₇": 818,
+ "₈": 819,
+ "₉": 820,
+ "₊": 821,
+ "₍": 822,
+ "₎": 823,
+ "ₐ": 824,
+ "ₑ": 825,
+ "ₒ": 826,
+ "ₓ": 827,
+ "ₕ": 828,
+ "ₖ": 829,
+ "ₘ": 830,
+ "ₙ": 831,
+ "ₚ": 832,
+ "ₛ": 833,
+ "ₜ": 834,
+ "₤": 835,
+ "€": 836,
+ "₱": 837,
+ "₹": 838,
+ "ℓ": 839,
+ "№": 840,
+ "ℝ": 841,
+ "⅓": 842,
+ "←": 843,
+ "↑": 844,
+ "→": 845,
+ "↔": 846,
+ "⇌": 847,
+ "⇒": 848,
+ "∂": 849,
+ "∈": 850,
+ "−": 851,
+ "∗": 852,
+ "∘": 853,
+ "√": 854,
+ "∞": 855,
+ "∧": 856,
+ "∨": 857,
+ "∩": 858,
+ "∪": 859,
+ "≈": 860,
+ "≠": 861,
+ "≡": 862,
+ "≤": 863,
+ "≥": 864,
+ "⊂": 865,
+ "⊆": 866,
+ "⊕": 867,
+ "⋅": 868,
+ "─": 869,
+ "│": 870,
+ "■": 871,
+ "●": 872,
+ "★": 873,
+ "☆": 874,
+ "☉": 875,
+ "♠": 876,
+ "♣": 877,
+ "♥": 878,
+ "♦": 879,
+ "♭": 880,
+ "♯": 881,
+ "⟨": 882,
+ "⟩": 883,
+ "ⱼ": 884,
+ "、": 885,
+ "。": 886,
+ "《": 887,
+ "》": 888,
+ "「": 889,
+ "」": 890,
+ "『": 891,
+ "』": 892,
+ "〜": 893,
+ "い": 894,
+ "う": 895,
+ "え": 896,
+ "お": 897,
+ "か": 898,
+ "き": 899,
+ "く": 900,
+ "け": 901,
+ "こ": 902,
+ "さ": 903,
+ "し": 904,
+ "す": 905,
+ "せ": 906,
+ "そ": 907,
+ "た": 908,
+ "ち": 909,
+ "つ": 910,
+ "て": 911,
+ "と": 912,
+ "な": 913,
+ "に": 914,
+ "の": 915,
+ "は": 916,
+ "ひ": 917,
+ "ま": 918,
+ "み": 919,
+ "む": 920,
+ "め": 921,
+ "も": 922,
+ "や": 923,
+ "ゆ": 924,
+ "よ": 925,
+ "ら": 926,
+ "り": 927,
+ "る": 928,
+ "れ": 929,
+ "ん": 930,
+ "ア": 931,
+ "ィ": 932,
+ "イ": 933,
+ "ウ": 934,
+ "エ": 935,
+ "オ": 936,
+ "カ": 937,
+ "ガ": 938,
+ "キ": 939,
+ "ク": 940,
+ "グ": 941,
+ "コ": 942,
+ "サ": 943,
+ "シ": 944,
+ "ジ": 945,
+ "ス": 946,
+ "ズ": 947,
+ "タ": 948,
+ "ダ": 949,
+ "ッ": 950,
+ "テ": 951,
+ "デ": 952,
+ "ト": 953,
+ "ド": 954,
+ "ナ": 955,
+ "ニ": 956,
+ "ハ": 957,
+ "バ": 958,
+ "パ": 959,
+ "フ": 960,
+ "ブ": 961,
+ "プ": 962,
+ "マ": 963,
+ "ミ": 964,
+ "ム": 965,
+ "ャ": 966,
+ "ュ": 967,
+ "ラ": 968,
+ "リ": 969,
+ "ル": 970,
+ "レ": 971,
+ "ロ": 972,
+ "ン": 973,
+ "・": 974,
+ "ー": 975,
+ "一": 976,
+ "三": 977,
+ "上": 978,
+ "下": 979,
+ "中": 980,
+ "事": 981,
+ "二": 982,
+ "井": 983,
+ "京": 984,
+ "人": 985,
+ "亻": 986,
+ "仁": 987,
+ "佐": 988,
+ "侍": 989,
+ "光": 990,
+ "公": 991,
+ "力": 992,
+ "北": 993,
+ "十": 994,
+ "南": 995,
+ "原": 996,
+ "口": 997,
+ "史": 998,
+ "司": 999,
+ "吉": 1000,
+ "同": 1001,
+ "和": 1002,
+ "囗": 1003,
+ "国": 1004,
+ "國": 1005,
+ "土": 1006,
+ "城": 1007,
+ "士": 1008,
+ "大": 1009,
+ "天": 1010,
+ "太": 1011,
+ "夫": 1012,
+ "女": 1013,
+ "子": 1014,
+ "宀": 1015,
+ "安": 1016,
+ "宮": 1017,
+ "宿": 1018,
+ "小": 1019,
+ "尚": 1020,
+ "山": 1021,
+ "島": 1022,
+ "川": 1023,
+ "州": 1024,
+ "平": 1025,
+ "年": 1026,
+ "心": 1027,
+ "愛": 1028,
+ "戸": 1029,
+ "文": 1030,
+ "新": 1031,
+ "方": 1032,
+ "日": 1033,
+ "明": 1034,
+ "星": 1035,
+ "書": 1036,
+ "月": 1037,
+ "木": 1038,
+ "本": 1039,
+ "李": 1040,
+ "村": 1041,
+ "東": 1042,
+ "松": 1043,
+ "林": 1044,
+ "正": 1045,
+ "武": 1046,
+ "氏": 1047,
+ "水": 1048,
+ "氵": 1049,
+ "江": 1050,
+ "河": 1051,
+ "海": 1052,
+ "版": 1053,
+ "犬": 1054,
+ "王": 1055,
+ "生": 1056,
+ "田": 1057,
+ "白": 1058,
+ "皇": 1059,
+ "省": 1060,
+ "真": 1061,
+ "石": 1062,
+ "社": 1063,
+ "神": 1064,
+ "竹": 1065,
+ "美": 1066,
+ "義": 1067,
+ "花": 1068,
+ "藤": 1069,
+ "西": 1070,
+ "谷": 1071,
+ "車": 1072,
+ "辶": 1073,
+ "道": 1074,
+ "郎": 1075,
+ "郡": 1076,
+ "部": 1077,
+ "野": 1078,
+ "金": 1079,
+ "長": 1080,
+ "門": 1081,
+ "陽": 1082,
+ "青": 1083,
+ "食": 1084,
+ "馬": 1085,
+ "高": 1086,
+ "龍": 1087,
+ "龸": 1088,
+ "사": 1089,
+ "씨": 1090,
+ "의": 1091,
+ "이": 1092,
+ "한": 1093,
+ "fi": 1094,
+ "fl": 1095,
+ "！": 1096,
+ "（": 1097,
+ "）": 1098,
+ "，": 1099,
+ "－": 1100,
+ "／": 1101,
+ "：": 1102,
+ "the": 1103,
+ "of": 1104,
+ "and": 1105,
+ "to": 1106,
+ "in": 1107,
+ "was": 1108,
+ "The": 1109,
+ "is": 1110,
+ "for": 1111,
+ "as": 1112,
+ "on": 1113,
+ "with": 1114,
+ "that": 1115,
+ "##s": 1116,
+ "his": 1117,
+ "by": 1118,
+ "he": 1119,
+ "at": 1120,
+ "from": 1121,
+ "it": 1122,
+ "her": 1123,
+ "He": 1124,
+ "had": 1125,
+ "an": 1126,
+ "were": 1127,
+ "you": 1128,
+ "be": 1129,
+ "In": 1130,
+ "she": 1131,
+ "are": 1132,
+ "but": 1133,
+ "which": 1134,
+ "It": 1135,
+ "not": 1136,
+ "or": 1137,
+ "have": 1138,
+ "my": 1139,
+ "him": 1140,
+ "one": 1141,
+ "this": 1142,
+ "me": 1143,
+ "has": 1144,
+ "also": 1145,
+ "up": 1146,
+ "their": 1147,
+ "first": 1148,
+ "out": 1149,
+ "who": 1150,
+ "been": 1151,
+ "they": 1152,
+ "She": 1153,
+ "into": 1154,
+ "all": 1155,
+ "would": 1156,
+ "its": 1157,
+ "##ing": 1158,
+ "time": 1159,
+ "two": 1160,
+ "##a": 1161,
+ "##e": 1162,
+ "said": 1163,
+ "about": 1164,
+ "when": 1165,
+ "over": 1166,
+ "more": 1167,
+ "other": 1168,
+ "can": 1169,
+ "after": 1170,
+ "back": 1171,
+ "them": 1172,
+ "then": 1173,
+ "##ed": 1174,
+ "there": 1175,
+ "like": 1176,
+ "so": 1177,
+ "only": 1178,
+ "##n": 1179,
+ "could": 1180,
+ "##d": 1181,
+ "##i": 1182,
+ "##y": 1183,
+ "what": 1184,
+ "no": 1185,
+ "##o": 1186,
+ "where": 1187,
+ "This": 1188,
+ "made": 1189,
+ "than": 1190,
+ "if": 1191,
+ "You": 1192,
+ "##ly": 1193,
+ "through": 1194,
+ "we": 1195,
+ "before": 1196,
+ "##r": 1197,
+ "just": 1198,
+ "some": 1199,
+ "##er": 1200,
+ "years": 1201,
+ "do": 1202,
+ "New": 1203,
+ "##t": 1204,
+ "down": 1205,
+ "between": 1206,
+ "new": 1207,
+ "now": 1208,
+ "will": 1209,
+ "three": 1210,
+ "most": 1211,
+ "On": 1212,
+ "around": 1213,
+ "year": 1214,
+ "used": 1215,
+ "such": 1216,
+ "being": 1217,
+ "well": 1218,
+ "during": 1219,
+ "They": 1220,
+ "know": 1221,
+ "against": 1222,
+ "under": 1223,
+ "later": 1224,
+ "did": 1225,
+ "part": 1226,
+ "known": 1227,
+ "off": 1228,
+ "while": 1229,
+ "His": 1230,
+ "re": 1231,
+ "...": 1232,
+ "##l": 1233,
+ "people": 1234,
+ "until": 1235,
+ "way": 1236,
+ "American": 1237,
+ "didn": 1238,
+ "University": 1239,
+ "your": 1240,
+ "both": 1241,
+ "many": 1242,
+ "get": 1243,
+ "United": 1244,
+ "became": 1245,
+ "head": 1246,
+ "There": 1247,
+ "second": 1248,
+ "As": 1249,
+ "work": 1250,
+ "any": 1251,
+ "But": 1252,
+ "still": 1253,
+ "again": 1254,
+ "born": 1255,
+ "even": 1256,
+ "eyes": 1257,
+ "After": 1258,
+ "including": 1259,
+ "de": 1260,
+ "took": 1261,
+ "And": 1262,
+ "long": 1263,
+ "team": 1264,
+ "season": 1265,
+ "family": 1266,
+ "see": 1267,
+ "right": 1268,
+ "same": 1269,
+ "called": 1270,
+ "name": 1271,
+ "because": 1272,
+ "film": 1273,
+ "don": 1274,
+ "10": 1275,
+ "found": 1276,
+ "much": 1277,
+ "school": 1278,
+ "##es": 1279,
+ "going": 1280,
+ "won": 1281,
+ "place": 1282,
+ "away": 1283,
+ "We": 1284,
+ "day": 1285,
+ "left": 1286,
+ "John": 1287,
+ "000": 1288,
+ "hand": 1289,
+ "since": 1290,
+ "World": 1291,
+ "these": 1292,
+ "how": 1293,
+ "make": 1294,
+ "number": 1295,
+ "each": 1296,
+ "life": 1297,
+ "area": 1298,
+ "man": 1299,
+ "four": 1300,
+ "go": 1301,
+ "No": 1302,
+ "here": 1303,
+ "very": 1304,
+ "National": 1305,
+ "##m": 1306,
+ "played": 1307,
+ "released": 1308,
+ "never": 1309,
+ "began": 1310,
+ "States": 1311,
+ "album": 1312,
+ "home": 1313,
+ "last": 1314,
+ "too": 1315,
+ "held": 1316,
+ "several": 1317,
+ "May": 1318,
+ "own": 1319,
+ "##on": 1320,
+ "take": 1321,
+ "end": 1322,
+ "School": 1323,
+ "##h": 1324,
+ "ll": 1325,
+ "series": 1326,
+ "What": 1327,
+ "want": 1328,
+ "use": 1329,
+ "another": 1330,
+ "city": 1331,
+ "When": 1332,
+ "2010": 1333,
+ "side": 1334,
+ "At": 1335,
+ "may": 1336,
+ "That": 1337,
+ "came": 1338,
+ "face": 1339,
+ "June": 1340,
+ "think": 1341,
+ "game": 1342,
+ "those": 1343,
+ "high": 1344,
+ "March": 1345,
+ "early": 1346,
+ "September": 1347,
+ "##al": 1348,
+ "2011": 1349,
+ "looked": 1350,
+ "July": 1351,
+ "state": 1352,
+ "small": 1353,
+ "thought": 1354,
+ "went": 1355,
+ "January": 1356,
+ "October": 1357,
+ "##u": 1358,
+ "based": 1359,
+ "August": 1360,
+ "##us": 1361,
+ "world": 1362,
+ "good": 1363,
+ "April": 1364,
+ "York": 1365,
+ "us": 1366,
+ "12": 1367,
+ "2012": 1368,
+ "2008": 1369,
+ "For": 1370,
+ "2009": 1371,
+ "group": 1372,
+ "along": 1373,
+ "few": 1374,
+ "South": 1375,
+ "little": 1376,
+ "##k": 1377,
+ "following": 1378,
+ "November": 1379,
+ "something": 1380,
+ "2013": 1381,
+ "December": 1382,
+ "set": 1383,
+ "2007": 1384,
+ "old": 1385,
+ "2006": 1386,
+ "2014": 1387,
+ "located": 1388,
+ "##an": 1389,
+ "music": 1390,
+ "County": 1391,
+ "City": 1392,
+ "former": 1393,
+ "##in": 1394,
+ "room": 1395,
+ "ve": 1396,
+ "next": 1397,
+ "All": 1398,
+ "##man": 1399,
+ "got": 1400,
+ "father": 1401,
+ "house": 1402,
+ "##g": 1403,
+ "body": 1404,
+ "15": 1405,
+ "20": 1406,
+ "18": 1407,
+ "started": 1408,
+ "If": 1409,
+ "2015": 1410,
+ "town": 1411,
+ "our": 1412,
+ "line": 1413,
+ "War": 1414,
+ "large": 1415,
+ "population": 1416,
+ "named": 1417,
+ "British": 1418,
+ "company": 1419,
+ "member": 1420,
+ "five": 1421,
+ "My": 1422,
+ "single": 1423,
+ "##en": 1424,
+ "age": 1425,
+ "State": 1426,
+ "moved": 1427,
+ "February": 1428,
+ "11": 1429,
+ "Her": 1430,
+ "should": 1431,
+ "century": 1432,
+ "government": 1433,
+ "built": 1434,
+ "come": 1435,
+ "best": 1436,
+ "show": 1437,
+ "However": 1438,
+ "within": 1439,
+ "look": 1440,
+ "men": 1441,
+ "door": 1442,
+ "without": 1443,
+ "need": 1444,
+ "wasn": 1445,
+ "2016": 1446,
+ "water": 1447,
+ "One": 1448,
+ "system": 1449,
+ "knew": 1450,
+ "every": 1451,
+ "died": 1452,
+ "League": 1453,
+ "turned": 1454,
+ "asked": 1455,
+ "North": 1456,
+ "St": 1457,
+ "wanted": 1458,
+ "building": 1459,
+ "received": 1460,
+ "song": 1461,
+ "served": 1462,
+ "though": 1463,
+ "felt": 1464,
+ "##ia": 1465,
+ "station": 1466,
+ "band": 1467,
+ "##ers": 1468,
+ "local": 1469,
+ "public": 1470,
+ "himself": 1471,
+ "different": 1472,
+ "death": 1473,
+ "say": 1474,
+ "##1": 1475,
+ "30": 1476,
+ "##2": 1477,
+ "2005": 1478,
+ "16": 1479,
+ "night": 1480,
+ "behind": 1481,
+ "children": 1482,
+ "English": 1483,
+ "members": 1484,
+ "near": 1485,
+ "saw": 1486,
+ "together": 1487,
+ "son": 1488,
+ "14": 1489,
+ "voice": 1490,
+ "village": 1491,
+ "13": 1492,
+ "hands": 1493,
+ "help": 1494,
+ "##3": 1495,
+ "due": 1496,
+ "French": 1497,
+ "London": 1498,
+ "top": 1499,
+ "told": 1500,
+ "open": 1501,
+ "published": 1502,
+ "third": 1503,
+ "2017": 1504,
+ "play": 1505,
+ "across": 1506,
+ "During": 1507,
+ "put": 1508,
+ "final": 1509,
+ "often": 1510,
+ "include": 1511,
+ "25": 1512,
+ "##le": 1513,
+ "main": 1514,
+ "having": 1515,
+ "2004": 1516,
+ "once": 1517,
+ "ever": 1518,
+ "let": 1519,
+ "book": 1520,
+ "led": 1521,
+ "gave": 1522,
+ "late": 1523,
+ "front": 1524,
+ "find": 1525,
+ "club": 1526,
+ "##4": 1527,
+ "German": 1528,
+ "included": 1529,
+ "species": 1530,
+ "College": 1531,
+ "form": 1532,
+ "opened": 1533,
+ "mother": 1534,
+ "women": 1535,
+ "enough": 1536,
+ "West": 1537,
+ "must": 1538,
+ "2000": 1539,
+ "power": 1540,
+ "really": 1541,
+ "17": 1542,
+ "making": 1543,
+ "half": 1544,
+ "##6": 1545,
+ "order": 1546,
+ "might": 1547,
+ "##is": 1548,
+ "given": 1549,
+ "million": 1550,
+ "times": 1551,
+ "days": 1552,
+ "point": 1553,
+ "full": 1554,
+ "service": 1555,
+ "With": 1556,
+ "km": 1557,
+ "major": 1558,
+ "##7": 1559,
+ "original": 1560,
+ "become": 1561,
+ "seen": 1562,
+ "II": 1563,
+ "north": 1564,
+ "six": 1565,
+ "##te": 1566,
+ "love": 1567,
+ "##0": 1568,
+ "national": 1569,
+ "International": 1570,
+ "##5": 1571,
+ "24": 1572,
+ "So": 1573,
+ "District": 1574,
+ "lost": 1575,
+ "run": 1576,
+ "couldn": 1577,
+ "career": 1578,
+ "always": 1579,
+ "##9": 1580,
+ "2003": 1581,
+ "##th": 1582,
+ "country": 1583,
+ "##z": 1584,
+ "House": 1585,
+ "air": 1586,
+ "tell": 1587,
+ "south": 1588,
+ "worked": 1589,
+ "woman": 1590,
+ "player": 1591,
+ "##A": 1592,
+ "almost": 1593,
+ "war": 1594,
+ "River": 1595,
+ "##ic": 1596,
+ "married": 1597,
+ "continued": 1598,
+ "Then": 1599,
+ "James": 1600,
+ "close": 1601,
+ "black": 1602,
+ "short": 1603,
+ "##8": 1604,
+ "##na": 1605,
+ "using": 1606,
+ "history": 1607,
+ "returned": 1608,
+ "light": 1609,
+ "car": 1610,
+ "##ra": 1611,
+ "sure": 1612,
+ "William": 1613,
+ "things": 1614,
+ "General": 1615,
+ "##ry": 1616,
+ "2002": 1617,
+ "better": 1618,
+ "support": 1619,
+ "100": 1620,
+ "among": 1621,
+ "From": 1622,
+ "feet": 1623,
+ "King": 1624,
+ "anything": 1625,
+ "21": 1626,
+ "19": 1627,
+ "established": 1628,
+ "district": 1629,
+ "2001": 1630,
+ "feel": 1631,
+ "great": 1632,
+ "##ton": 1633,
+ "level": 1634,
+ "Cup": 1635,
+ "These": 1636,
+ "written": 1637,
+ "games": 1638,
+ "others": 1639,
+ "already": 1640,
+ "title": 1641,
+ "story": 1642,
+ "##p": 1643,
+ "law": 1644,
+ "thing": 1645,
+ "US": 1646,
+ "record": 1647,
+ "role": 1648,
+ "however": 1649,
+ "By": 1650,
+ "students": 1651,
+ "England": 1652,
+ "white": 1653,
+ "control": 1654,
+ "least": 1655,
+ "inside": 1656,
+ "land": 1657,
+ "##C": 1658,
+ "22": 1659,
+ "give": 1660,
+ "community": 1661,
+ "hard": 1662,
+ "##ie": 1663,
+ "non": 1664,
+ "##c": 1665,
+ "produced": 1666,
+ "George": 1667,
+ "round": 1668,
+ "period": 1669,
+ "Park": 1670,
+ "business": 1671,
+ "various": 1672,
+ "##ne": 1673,
+ "does": 1674,
+ "present": 1675,
+ "wife": 1676,
+ "far": 1677,
+ "taken": 1678,
+ "per": 1679,
+ "reached": 1680,
+ "David": 1681,
+ "able": 1682,
+ "version": 1683,
+ "working": 1684,
+ "young": 1685,
+ "live": 1686,
+ "created": 1687,
+ "joined": 1688,
+ "East": 1689,
+ "living": 1690,
+ "appeared": 1691,
+ "case": 1692,
+ "High": 1693,
+ "done": 1694,
+ "23": 1695,
+ "important": 1696,
+ "President": 1697,
+ "Award": 1698,
+ "France": 1699,
+ "position": 1700,
+ "office": 1701,
+ "looking": 1702,
+ "total": 1703,
+ "general": 1704,
+ "class": 1705,
+ "To": 1706,
+ "production": 1707,
+ "##S": 1708,
+ "football": 1709,
+ "party": 1710,
+ "brother": 1711,
+ "keep": 1712,
+ "mind": 1713,
+ "free": 1714,
+ "Street": 1715,
+ "hair": 1716,
+ "announced": 1717,
+ "development": 1718,
+ "either": 1719,
+ "nothing": 1720,
+ "moment": 1721,
+ "Church": 1722,
+ "followed": 1723,
+ "wrote": 1724,
+ "why": 1725,
+ "India": 1726,
+ "San": 1727,
+ "election": 1728,
+ "1999": 1729,
+ "lead": 1730,
+ "How": 1731,
+ "##ch": 1732,
+ "##rs": 1733,
+ "words": 1734,
+ "European": 1735,
+ "course": 1736,
+ "considered": 1737,
+ "America": 1738,
+ "arms": 1739,
+ "Army": 1740,
+ "political": 1741,
+ "##la": 1742,
+ "28": 1743,
+ "26": 1744,
+ "west": 1745,
+ "east": 1746,
+ "ground": 1747,
+ "further": 1748,
+ "church": 1749,
+ "less": 1750,
+ "site": 1751,
+ "First": 1752,
+ "Not": 1753,
+ "Australia": 1754,
+ "toward": 1755,
+ "California": 1756,
+ "##ness": 1757,
+ "described": 1758,
+ "works": 1759,
+ "An": 1760,
+ "Council": 1761,
+ "heart": 1762,
+ "past": 1763,
+ "military": 1764,
+ "27": 1765,
+ "##or": 1766,
+ "heard": 1767,
+ "field": 1768,
+ "human": 1769,
+ "soon": 1770,
+ "founded": 1771,
+ "1998": 1772,
+ "playing": 1773,
+ "trying": 1774,
+ "##x": 1775,
+ "##ist": 1776,
+ "##ta": 1777,
+ "television": 1778,
+ "mouth": 1779,
+ "although": 1780,
+ "taking": 1781,
+ "win": 1782,
+ "fire": 1783,
+ "Division": 1784,
+ "##ity": 1785,
+ "Party": 1786,
+ "Royal": 1787,
+ "program": 1788,
+ "Some": 1789,
+ "Don": 1790,
+ "Association": 1791,
+ "According": 1792,
+ "tried": 1793,
+ "TV": 1794,
+ "Paul": 1795,
+ "outside": 1796,
+ "daughter": 1797,
+ "Best": 1798,
+ "While": 1799,
+ "someone": 1800,
+ "match": 1801,
+ "recorded": 1802,
+ "Canada": 1803,
+ "closed": 1804,
+ "region": 1805,
+ "Air": 1806,
+ "above": 1807,
+ "months": 1808,
+ "elected": 1809,
+ "##da": 1810,
+ "##ian": 1811,
+ "road": 1812,
+ "##ar": 1813,
+ "brought": 1814,
+ "move": 1815,
+ "1997": 1816,
+ "leave": 1817,
+ "##um": 1818,
+ "Thomas": 1819,
+ "1996": 1820,
+ "am": 1821,
+ "low": 1822,
+ "Robert": 1823,
+ "formed": 1824,
+ "person": 1825,
+ "services": 1826,
+ "points": 1827,
+ "Mr": 1828,
+ "miles": 1829,
+ "##b": 1830,
+ "stop": 1831,
+ "rest": 1832,
+ "doing": 1833,
+ "needed": 1834,
+ "international": 1835,
+ "release": 1836,
+ "floor": 1837,
+ "start": 1838,
+ "sound": 1839,
+ "call": 1840,
+ "killed": 1841,
+ "real": 1842,
+ "dark": 1843,
+ "research": 1844,
+ "finished": 1845,
+ "language": 1846,
+ "Michael": 1847,
+ "professional": 1848,
+ "change": 1849,
+ "sent": 1850,
+ "50": 1851,
+ "upon": 1852,
+ "29": 1853,
+ "track": 1854,
+ "hit": 1855,
+ "event": 1856,
+ "2018": 1857,
+ "term": 1858,
+ "example": 1859,
+ "Germany": 1860,
+ "similar": 1861,
+ "return": 1862,
+ "##ism": 1863,
+ "fact": 1864,
+ "pulled": 1865,
+ "stood": 1866,
+ "says": 1867,
+ "ran": 1868,
+ "information": 1869,
+ "yet": 1870,
+ "result": 1871,
+ "developed": 1872,
+ "girl": 1873,
+ "##re": 1874,
+ "God": 1875,
+ "1995": 1876,
+ "areas": 1877,
+ "signed": 1878,
+ "decided": 1879,
+ "##ment": 1880,
+ "Company": 1881,
+ "seemed": 1882,
+ "##el": 1883,
+ "co": 1884,
+ "turn": 1885,
+ "race": 1886,
+ "common": 1887,
+ "video": 1888,
+ "Charles": 1889,
+ "Indian": 1890,
+ "##ation": 1891,
+ "blood": 1892,
+ "art": 1893,
+ "red": 1894,
+ "##able": 1895,
+ "added": 1896,
+ "rather": 1897,
+ "1994": 1898,
+ "met": 1899,
+ "director": 1900,
+ "addition": 1901,
+ "design": 1902,
+ "average": 1903,
+ "minutes": 1904,
+ "##ies": 1905,
+ "##ted": 1906,
+ "available": 1907,
+ "bed": 1908,
+ "coming": 1909,
+ "friend": 1910,
+ "idea": 1911,
+ "kind": 1912,
+ "Union": 1913,
+ "Road": 1914,
+ "remained": 1915,
+ "##ting": 1916,
+ "everything": 1917,
+ "##ma": 1918,
+ "running": 1919,
+ "care": 1920,
+ "finally": 1921,
+ "Chinese": 1922,
+ "appointed": 1923,
+ "1992": 1924,
+ "Australian": 1925,
+ "##ley": 1926,
+ "popular": 1927,
+ "mean": 1928,
+ "teams": 1929,
+ "probably": 1930,
+ "##land": 1931,
+ "usually": 1932,
+ "project": 1933,
+ "social": 1934,
+ "Championship": 1935,
+ "possible": 1936,
+ "word": 1937,
+ "Russian": 1938,
+ "instead": 1939,
+ "mi": 1940,
+ "herself": 1941,
+ "##T": 1942,
+ "Peter": 1943,
+ "Hall": 1944,
+ "Center": 1945,
+ "seat": 1946,
+ "style": 1947,
+ "money": 1948,
+ "1993": 1949,
+ "else": 1950,
+ "Department": 1951,
+ "table": 1952,
+ "Music": 1953,
+ "current": 1954,
+ "31": 1955,
+ "features": 1956,
+ "special": 1957,
+ "events": 1958,
+ "character": 1959,
+ "Two": 1960,
+ "square": 1961,
+ "sold": 1962,
+ "debut": 1963,
+ "##v": 1964,
+ "process": 1965,
+ "Although": 1966,
+ "Since": 1967,
+ "##ka": 1968,
+ "40": 1969,
+ "Central": 1970,
+ "currently": 1971,
+ "education": 1972,
+ "placed": 1973,
+ "lot": 1974,
+ "China": 1975,
+ "quickly": 1976,
+ "forward": 1977,
+ "seven": 1978,
+ "##ling": 1979,
+ "Europe": 1980,
+ "arm": 1981,
+ "performed": 1982,
+ "Japanese": 1983,
+ "1991": 1984,
+ "Henry": 1985,
+ "Now": 1986,
+ "Dr": 1987,
+ "##ion": 1988,
+ "week": 1989,
+ "Group": 1990,
+ "myself": 1991,
+ "big": 1992,
+ "UK": 1993,
+ "Washington": 1994,
+ "ten": 1995,
+ "deep": 1996,
+ "1990": 1997,
+ "Club": 1998,
+ "Japan": 1999,
+ "space": 2000,
+ "La": 2001,
+ "directed": 2002,
+ "smile": 2003,
+ "episode": 2004,
+ "hours": 2005,
+ "whole": 2006,
+ "##de": 2007,
+ "##less": 2008,
+ "Why": 2009,
+ "wouldn": 2010,
+ "designed": 2011,
+ "strong": 2012,
+ "training": 2013,
+ "changed": 2014,
+ "Society": 2015,
+ "stage": 2016,
+ "involved": 2017,
+ "hadn": 2018,
+ "towards": 2019,
+ "leading": 2020,
+ "police": 2021,
+ "eight": 2022,
+ "kept": 2023,
+ "Institute": 2024,
+ "study": 2025,
+ "largest": 2026,
+ "child": 2027,
+ "eventually": 2028,
+ "private": 2029,
+ "modern": 2030,
+ "Court": 2031,
+ "throughout": 2032,
+ "getting": 2033,
+ "originally": 2034,
+ "attack": 2035,
+ "##E": 2036,
+ "talk": 2037,
+ "Great": 2038,
+ "longer": 2039,
+ "songs": 2040,
+ "alone": 2041,
+ "##ine": 2042,
+ "wide": 2043,
+ "dead": 2044,
+ "walked": 2045,
+ "shot": 2046,
+ "##ri": 2047,
+ "Oh": 2048,
+ "force": 2049,
+ "##st": 2050,
+ "Art": 2051,
+ "today": 2052,
+ "friends": 2053,
+ "Island": 2054,
+ "Richard": 2055,
+ "1989": 2056,
+ "center": 2057,
+ "construction": 2058,
+ "believe": 2059,
+ "size": 2060,
+ "White": 2061,
+ "ship": 2062,
+ "completed": 2063,
+ "##B": 2064,
+ "gone": 2065,
+ "Just": 2066,
+ "rock": 2067,
+ "sat": 2068,
+ "##R": 2069,
+ "radio": 2070,
+ "below": 2071,
+ "entire": 2072,
+ "families": 2073,
+ "league": 2074,
+ "includes": 2075,
+ "type": 2076,
+ "lived": 2077,
+ "official": 2078,
+ "range": 2079,
+ "hold": 2080,
+ "featured": 2081,
+ "Most": 2082,
+ "##ter": 2083,
+ "president": 2084,
+ "passed": 2085,
+ "means": 2086,
+ "##f": 2087,
+ "forces": 2088,
+ "lips": 2089,
+ "Mary": 2090,
+ "Do": 2091,
+ "guitar": 2092,
+ "##ce": 2093,
+ "food": 2094,
+ "wall": 2095,
+ "Of": 2096,
+ "spent": 2097,
+ "Its": 2098,
+ "performance": 2099,
+ "hear": 2100,
+ "##P": 2101,
+ "Western": 2102,
+ "reported": 2103,
+ "sister": 2104,
+ "##et": 2105,
+ "morning": 2106,
+ "##M": 2107,
+ "especially": 2108,
+ "##ive": 2109,
+ "Minister": 2110,
+ "itself": 2111,
+ "post": 2112,
+ "bit": 2113,
+ "groups": 2114,
+ "1988": 2115,
+ "##tion": 2116,
+ "Black": 2117,
+ "##ng": 2118,
+ "Well": 2119,
+ "raised": 2120,
+ "sometimes": 2121,
+ "Canadian": 2122,
+ "Paris": 2123,
+ "Spanish": 2124,
+ "replaced": 2125,
+ "schools": 2126,
+ "Academy": 2127,
+ "leaving": 2128,
+ "central": 2129,
+ "female": 2130,
+ "Christian": 2131,
+ "Jack": 2132,
+ "whose": 2133,
+ "college": 2134,
+ "onto": 2135,
+ "provided": 2136,
+ "##D": 2137,
+ "##ville": 2138,
+ "players": 2139,
+ "actually": 2140,
+ "stopped": 2141,
+ "##son": 2142,
+ "Museum": 2143,
+ "doesn": 2144,
+ "##ts": 2145,
+ "books": 2146,
+ "fight": 2147,
+ "allowed": 2148,
+ "##ur": 2149,
+ "beginning": 2150,
+ "Records": 2151,
+ "awarded": 2152,
+ "parents": 2153,
+ "coach": 2154,
+ "##os": 2155,
+ "Red": 2156,
+ "saying": 2157,
+ "##ck": 2158,
+ "Smith": 2159,
+ "Yes": 2160,
+ "Lake": 2161,
+ "##L": 2162,
+ "aircraft": 2163,
+ "1987": 2164,
+ "##ble": 2165,
+ "previous": 2166,
+ "ft": 2167,
+ "action": 2168,
+ "Italian": 2169,
+ "African": 2170,
+ "happened": 2171,
+ "vocals": 2172,
+ "Act": 2173,
+ "future": 2174,
+ "court": 2175,
+ "##ge": 2176,
+ "1986": 2177,
+ "degree": 2178,
+ "phone": 2179,
+ "##ro": 2180,
+ "Is": 2181,
+ "countries": 2182,
+ "winning": 2183,
+ "breath": 2184,
+ "Love": 2185,
+ "river": 2186,
+ "matter": 2187,
+ "Lord": 2188,
+ "Other": 2189,
+ "list": 2190,
+ "self": 2191,
+ "parts": 2192,
+ "##ate": 2193,
+ "provide": 2194,
+ "cut": 2195,
+ "shows": 2196,
+ "plan": 2197,
+ "1st": 2198,
+ "interest": 2199,
+ "##ized": 2200,
+ "Africa": 2201,
+ "stated": 2202,
+ "Sir": 2203,
+ "fell": 2204,
+ "owned": 2205,
+ "earlier": 2206,
+ "ended": 2207,
+ "competition": 2208,
+ "attention": 2209,
+ "1985": 2210,
+ "lower": 2211,
+ "nearly": 2212,
+ "bad": 2213,
+ "older": 2214,
+ "stay": 2215,
+ "Saint": 2216,
+ "##se": 2217,
+ "certain": 2218,
+ "1984": 2219,
+ "fingers": 2220,
+ "blue": 2221,
+ "try": 2222,
+ "fourth": 2223,
+ "Grand": 2224,
+ "##as": 2225,
+ "king": 2226,
+ "##nt": 2227,
+ "makes": 2228,
+ "chest": 2229,
+ "movement": 2230,
+ "states": 2231,
+ "moving": 2232,
+ "data": 2233,
+ "introduced": 2234,
+ "model": 2235,
+ "date": 2236,
+ "section": 2237,
+ "Los": 2238,
+ "deal": 2239,
+ "##I": 2240,
+ "skin": 2241,
+ "entered": 2242,
+ "middle": 2243,
+ "success": 2244,
+ "Texas": 2245,
+ "##w": 2246,
+ "summer": 2247,
+ "island": 2248,
+ "##N": 2249,
+ "Republic": 2250,
+ "length": 2251,
+ "husband": 2252,
+ "1980": 2253,
+ "##ey": 2254,
+ "reason": 2255,
+ "anyone": 2256,
+ "forced": 2257,
+ "via": 2258,
+ "base": 2259,
+ "500": 2260,
+ "job": 2261,
+ "covered": 2262,
+ "Festival": 2263,
+ "Roman": 2264,
+ "successful": 2265,
+ "rights": 2266,
+ "cover": 2267,
+ "Man": 2268,
+ "writing": 2269,
+ "Ireland": 2270,
+ "##F": 2271,
+ "related": 2272,
+ "goal": 2273,
+ "takes": 2274,
+ "buildings": 2275,
+ "true": 2276,
+ "weeks": 2277,
+ "1983": 2278,
+ "Because": 2279,
+ "opening": 2280,
+ "novel": 2281,
+ "ISBN": 2282,
+ "meet": 2283,
+ "gold": 2284,
+ "##ous": 2285,
+ "mid": 2286,
+ "km²": 2287,
+ "standing": 2288,
+ "Football": 2289,
+ "Chicago": 2290,
+ "shook": 2291,
+ "whom": 2292,
+ "##ki": 2293,
+ "1982": 2294,
+ "Day": 2295,
+ "feeling": 2296,
+ "scored": 2297,
+ "boy": 2298,
+ "higher": 2299,
+ "Force": 2300,
+ "leader": 2301,
+ "heavy": 2302,
+ "fall": 2303,
+ "question": 2304,
+ "sense": 2305,
+ "army": 2306,
+ "Second": 2307,
+ "energy": 2308,
+ "meeting": 2309,
+ "themselves": 2310,
+ "kill": 2311,
+ "##am": 2312,
+ "board": 2313,
+ "census": 2314,
+ "##ya": 2315,
+ "##ns": 2316,
+ "mine": 2317,
+ "meant": 2318,
+ "market": 2319,
+ "required": 2320,
+ "battle": 2321,
+ "campaign": 2322,
+ "attended": 2323,
+ "approximately": 2324,
+ "Kingdom": 2325,
+ "runs": 2326,
+ "active": 2327,
+ "##ha": 2328,
+ "contract": 2329,
+ "clear": 2330,
+ "previously": 2331,
+ "health": 2332,
+ "1979": 2333,
+ "Arts": 2334,
+ "complete": 2335,
+ "Catholic": 2336,
+ "couple": 2337,
+ "units": 2338,
+ "##ll": 2339,
+ "##ty": 2340,
+ "Committee": 2341,
+ "shoulder": 2342,
+ "sea": 2343,
+ "systems": 2344,
+ "listed": 2345,
+ "##O": 2346,
+ "caught": 2347,
+ "tournament": 2348,
+ "##G": 2349,
+ "northern": 2350,
+ "author": 2351,
+ "Film": 2352,
+ "Your": 2353,
+ "##men": 2354,
+ "holding": 2355,
+ "offered": 2356,
+ "personal": 2357,
+ "1981": 2358,
+ "southern": 2359,
+ "artist": 2360,
+ "traditional": 2361,
+ "studio": 2362,
+ "200": 2363,
+ "capital": 2364,
+ "##ful": 2365,
+ "regular": 2366,
+ "ask": 2367,
+ "giving": 2368,
+ "organization": 2369,
+ "month": 2370,
+ "news": 2371,
+ "Are": 2372,
+ "read": 2373,
+ "managed": 2374,
+ "helped": 2375,
+ "studied": 2376,
+ "student": 2377,
+ "defeated": 2378,
+ "natural": 2379,
+ "industry": 2380,
+ "Year": 2381,
+ "noted": 2382,
+ "decision": 2383,
+ "Government": 2384,
+ "quite": 2385,
+ "##id": 2386,
+ "smiled": 2387,
+ "1972": 2388,
+ "Maybe": 2389,
+ "tracks": 2390,
+ "##ke": 2391,
+ "Mark": 2392,
+ "al": 2393,
+ "media": 2394,
+ "engine": 2395,
+ "hour": 2396,
+ "Their": 2397,
+ "relationship": 2398,
+ "plays": 2399,
+ "property": 2400,
+ "structure": 2401,
+ "1976": 2402,
+ "ago": 2403,
+ "Hill": 2404,
+ "Martin": 2405,
+ "1978": 2406,
+ "ready": 2407,
+ "Many": 2408,
+ "Like": 2409,
+ "Bay": 2410,
+ "immediately": 2411,
+ "generally": 2412,
+ "Italy": 2413,
+ "Greek": 2414,
+ "practice": 2415,
+ "caused": 2416,
+ "division": 2417,
+ "significant": 2418,
+ "Joseph": 2419,
+ "speed": 2420,
+ "Let": 2421,
+ "thinking": 2422,
+ "completely": 2423,
+ "1974": 2424,
+ "primary": 2425,
+ "mostly": 2426,
+ "##field": 2427,
+ "##K": 2428,
+ "1975": 2429,
+ "##to": 2430,
+ "Even": 2431,
+ "writer": 2432,
+ "##led": 2433,
+ "dropped": 2434,
+ "magazine": 2435,
+ "collection": 2436,
+ "understand": 2437,
+ "route": 2438,
+ "highest": 2439,
+ "particular": 2440,
+ "films": 2441,
+ "lines": 2442,
+ "network": 2443,
+ "Science": 2444,
+ "loss": 2445,
+ "carried": 2446,
+ "direction": 2447,
+ "green": 2448,
+ "1977": 2449,
+ "location": 2450,
+ "producer": 2451,
+ "according": 2452,
+ "Women": 2453,
+ "Queen": 2454,
+ "neck": 2455,
+ "thus": 2456,
+ "independent": 2457,
+ "view": 2458,
+ "1970": 2459,
+ "Angeles": 2460,
+ "Soviet": 2461,
+ "distance": 2462,
+ "problem": 2463,
+ "Board": 2464,
+ "tour": 2465,
+ "western": 2466,
+ "income": 2467,
+ "appearance": 2468,
+ "access": 2469,
+ "Mexico": 2470,
+ "nodded": 2471,
+ "street": 2472,
+ "surface": 2473,
+ "arrived": 2474,
+ "believed": 2475,
+ "Old": 2476,
+ "1968": 2477,
+ "1973": 2478,
+ "becoming": 2479,
+ "whether": 2480,
+ "1945": 2481,
+ "figure": 2482,
+ "singer": 2483,
+ "stand": 2484,
+ "Following": 2485,
+ "issue": 2486,
+ "window": 2487,
+ "wrong": 2488,
+ "pain": 2489,
+ "everyone": 2490,
+ "lives": 2491,
+ "issues": 2492,
+ "park": 2493,
+ "slowly": 2494,
+ "la": 2495,
+ "act": 2496,
+ "##va": 2497,
+ "bring": 2498,
+ "Lee": 2499,
+ "operations": 2500,
+ "key": 2501,
+ "comes": 2502,
+ "fine": 2503,
+ "cold": 2504,
+ "famous": 2505,
+ "Navy": 2506,
+ "1971": 2507,
+ "Me": 2508,
+ "additional": 2509,
+ "individual": 2510,
+ "##ner": 2511,
+ "Zealand": 2512,
+ "goals": 2513,
+ "county": 2514,
+ "contains": 2515,
+ "Service": 2516,
+ "minute": 2517,
+ "2nd": 2518,
+ "reach": 2519,
+ "talking": 2520,
+ "particularly": 2521,
+ "##ham": 2522,
+ "movie": 2523,
+ "Director": 2524,
+ "glass": 2525,
+ "paper": 2526,
+ "studies": 2527,
+ "##co": 2528,
+ "railway": 2529,
+ "standard": 2530,
+ "Education": 2531,
+ "45": 2532,
+ "represented": 2533,
+ "Chief": 2534,
+ "Louis": 2535,
+ "launched": 2536,
+ "Star": 2537,
+ "terms": 2538,
+ "60": 2539,
+ "1969": 2540,
+ "experience": 2541,
+ "watched": 2542,
+ "Another": 2543,
+ "Press": 2544,
+ "Tom": 2545,
+ "staff": 2546,
+ "starting": 2547,
+ "subject": 2548,
+ "break": 2549,
+ "Virginia": 2550,
+ "nine": 2551,
+ "eye": 2552,
+ "##age": 2553,
+ "evidence": 2554,
+ "foot": 2555,
+ "##est": 2556,
+ "companies": 2557,
+ "Prince": 2558,
+ "##V": 2559,
+ "gun": 2560,
+ "create": 2561,
+ "Big": 2562,
+ "People": 2563,
+ "guy": 2564,
+ "Green": 2565,
+ "simply": 2566,
+ "numerous": 2567,
+ "##line": 2568,
+ "increased": 2569,
+ "twenty": 2570,
+ "##ga": 2571,
+ "##do": 2572,
+ "1967": 2573,
+ "award": 2574,
+ "officer": 2575,
+ "stone": 2576,
+ "Before": 2577,
+ "material": 2578,
+ "Northern": 2579,
+ "grew": 2580,
+ "male": 2581,
+ "plant": 2582,
+ "Life": 2583,
+ "legs": 2584,
+ "step": 2585,
+ "Al": 2586,
+ "unit": 2587,
+ "35": 2588,
+ "except": 2589,
+ "answer": 2590,
+ "##U": 2591,
+ "report": 2592,
+ "response": 2593,
+ "Edward": 2594,
+ "commercial": 2595,
+ "edition": 2596,
+ "trade": 2597,
+ "science": 2598,
+ "##ca": 2599,
+ "Irish": 2600,
+ "Law": 2601,
+ "shown": 2602,
+ "rate": 2603,
+ "failed": 2604,
+ "##ni": 2605,
+ "remains": 2606,
+ "changes": 2607,
+ "mm": 2608,
+ "limited": 2609,
+ "larger": 2610,
+ "Later": 2611,
+ "cause": 2612,
+ "waiting": 2613,
+ "Time": 2614,
+ "##wood": 2615,
+ "cost": 2616,
+ "Bill": 2617,
+ "manager": 2618,
+ "activities": 2619,
+ "likely": 2620,
+ "allow": 2621,
+ "operated": 2622,
+ "retired": 2623,
+ "##ping": 2624,
+ "65": 2625,
+ "directly": 2626,
+ "Who": 2627,
+ "associated": 2628,
+ "effect": 2629,
+ "hell": 2630,
+ "Florida": 2631,
+ "straight": 2632,
+ "hot": 2633,
+ "Valley": 2634,
+ "management": 2635,
+ "girls": 2636,
+ "expected": 2637,
+ "eastern": 2638,
+ "Mike": 2639,
+ "chance": 2640,
+ "cast": 2641,
+ "centre": 2642,
+ "chair": 2643,
+ "hurt": 2644,
+ "problems": 2645,
+ "##li": 2646,
+ "walk": 2647,
+ "programs": 2648,
+ "Team": 2649,
+ "characters": 2650,
+ "Battle": 2651,
+ "edge": 2652,
+ "pay": 2653,
+ "maybe": 2654,
+ "corner": 2655,
+ "majority": 2656,
+ "medical": 2657,
+ "Joe": 2658,
+ "Summer": 2659,
+ "##io": 2660,
+ "attempt": 2661,
+ "Pacific": 2662,
+ "command": 2663,
+ "Radio": 2664,
+ "##by": 2665,
+ "names": 2666,
+ "municipality": 2667,
+ "1964": 2668,
+ "train": 2669,
+ "economic": 2670,
+ "Brown": 2671,
+ "feature": 2672,
+ "sex": 2673,
+ "source": 2674,
+ "agreed": 2675,
+ "remember": 2676,
+ "Three": 2677,
+ "1966": 2678,
+ "1965": 2679,
+ "Pennsylvania": 2680,
+ "victory": 2681,
+ "senior": 2682,
+ "annual": 2683,
+ "III": 2684,
+ "Southern": 2685,
+ "results": 2686,
+ "Sam": 2687,
+ "serving": 2688,
+ "religious": 2689,
+ "Jones": 2690,
+ "appears": 2691,
+ "##der": 2692,
+ "despite": 2693,
+ "claimed": 2694,
+ "Both": 2695,
+ "musical": 2696,
+ "matches": 2697,
+ "fast": 2698,
+ "security": 2699,
+ "selected": 2700,
+ "Young": 2701,
+ "double": 2702,
+ "complex": 2703,
+ "hospital": 2704,
+ "chief": 2705,
+ "Times": 2706,
+ "##ve": 2707,
+ "Championships": 2708,
+ "filled": 2709,
+ "Public": 2710,
+ "Despite": 2711,
+ "beautiful": 2712,
+ "Research": 2713,
+ "plans": 2714,
+ "Province": 2715,
+ "##ally": 2716,
+ "Wales": 2717,
+ "##ko": 2718,
+ "artists": 2719,
+ "metal": 2720,
+ "nearby": 2721,
+ "Spain": 2722,
+ "##il": 2723,
+ "32": 2724,
+ "houses": 2725,
+ "supported": 2726,
+ "piece": 2727,
+ "##no": 2728,
+ "stared": 2729,
+ "recording": 2730,
+ "nature": 2731,
+ "legal": 2732,
+ "Russia": 2733,
+ "##ization": 2734,
+ "remaining": 2735,
+ "looks": 2736,
+ "##sh": 2737,
+ "bridge": 2738,
+ "closer": 2739,
+ "cases": 2740,
+ "scene": 2741,
+ "marriage": 2742,
+ "Little": 2743,
+ "##é": 2744,
+ "uses": 2745,
+ "Earth": 2746,
+ "specific": 2747,
+ "Frank": 2748,
+ "theory": 2749,
+ "Good": 2750,
+ "discovered": 2751,
+ "referred": 2752,
+ "bass": 2753,
+ "culture": 2754,
+ "university": 2755,
+ "presented": 2756,
+ "Congress": 2757,
+ "##go": 2758,
+ "metres": 2759,
+ "continue": 2760,
+ "1960": 2761,
+ "isn": 2762,
+ "Awards": 2763,
+ "meaning": 2764,
+ "cell": 2765,
+ "composed": 2766,
+ "separate": 2767,
+ "Series": 2768,
+ "forms": 2769,
+ "Blue": 2770,
+ "cross": 2771,
+ "##tor": 2772,
+ "increase": 2773,
+ "test": 2774,
+ "computer": 2775,
+ "slightly": 2776,
+ "Where": 2777,
+ "Jewish": 2778,
+ "Town": 2779,
+ "tree": 2780,
+ "status": 2781,
+ "1944": 2782,
+ "variety": 2783,
+ "responsible": 2784,
+ "pretty": 2785,
+ "initially": 2786,
+ "##way": 2787,
+ "realized": 2788,
+ "pass": 2789,
+ "provides": 2790,
+ "Captain": 2791,
+ "Alexander": 2792,
+ "recent": 2793,
+ "score": 2794,
+ "broke": 2795,
+ "Scott": 2796,
+ "drive": 2797,
+ "financial": 2798,
+ "showed": 2799,
+ "Line": 2800,
+ "stories": 2801,
+ "ordered": 2802,
+ "soldiers": 2803,
+ "genus": 2804,
+ "operation": 2805,
+ "gaze": 2806,
+ "sitting": 2807,
+ "society": 2808,
+ "Only": 2809,
+ "hope": 2810,
+ "actor": 2811,
+ "follow": 2812,
+ "Empire": 2813,
+ "Yeah": 2814,
+ "technology": 2815,
+ "happy": 2816,
+ "focus": 2817,
+ "policy": 2818,
+ "spread": 2819,
+ "situation": 2820,
+ "##ford": 2821,
+ "##ba": 2822,
+ "Mrs": 2823,
+ "watch": 2824,
+ "Can": 2825,
+ "1963": 2826,
+ "Commission": 2827,
+ "touch": 2828,
+ "earned": 2829,
+ "troops": 2830,
+ "Under": 2831,
+ "1962": 2832,
+ "individuals": 2833,
+ "cannot": 2834,
+ "19th": 2835,
+ "##lin": 2836,
+ "mile": 2837,
+ "expression": 2838,
+ "exactly": 2839,
+ "suddenly": 2840,
+ "weight": 2841,
+ "dance": 2842,
+ "stepped": 2843,
+ "places": 2844,
+ "appear": 2845,
+ "difficult": 2846,
+ "Railway": 2847,
+ "anti": 2848,
+ "numbers": 2849,
+ "kilometres": 2850,
+ "star": 2851,
+ "##ier": 2852,
+ "department": 2853,
+ "ice": 2854,
+ "Britain": 2855,
+ "removed": 2856,
+ "Once": 2857,
+ "##lo": 2858,
+ "Boston": 2859,
+ "value": 2860,
+ "##ant": 2861,
+ "mission": 2862,
+ "trees": 2863,
+ "Order": 2864,
+ "sports": 2865,
+ "join": 2866,
+ "serve": 2867,
+ "Major": 2868,
+ "poor": 2869,
+ "Poland": 2870,
+ "mainly": 2871,
+ "Theatre": 2872,
+ "pushed": 2873,
+ "Station": 2874,
+ "##it": 2875,
+ "Lady": 2876,
+ "federal": 2877,
+ "silver": 2878,
+ "##ler": 2879,
+ "foreign": 2880,
+ "##ard": 2881,
+ "Eastern": 2882,
+ "##den": 2883,
+ "box": 2884,
+ "hall": 2885,
+ "subsequently": 2886,
+ "lies": 2887,
+ "acquired": 2888,
+ "1942": 2889,
+ "ancient": 2890,
+ "CD": 2891,
+ "History": 2892,
+ "Jean": 2893,
+ "beyond": 2894,
+ "##ger": 2895,
+ "El": 2896,
+ "##les": 2897,
+ "growing": 2898,
+ "championship": 2899,
+ "native": 2900,
+ "Parliament": 2901,
+ "Williams": 2902,
+ "watching": 2903,
+ "direct": 2904,
+ "overall": 2905,
+ "offer": 2906,
+ "Also": 2907,
+ "80": 2908,
+ "Secretary": 2909,
+ "spoke": 2910,
+ "Latin": 2911,
+ "ability": 2912,
+ "##ated": 2913,
+ "safe": 2914,
+ "presence": 2915,
+ "##ial": 2916,
+ "headed": 2917,
+ "regional": 2918,
+ "planned": 2919,
+ "1961": 2920,
+ "Johnson": 2921,
+ "throat": 2922,
+ "consists": 2923,
+ "##W": 2924,
+ "extended": 2925,
+ "Or": 2926,
+ "bar": 2927,
+ "walls": 2928,
+ "Chris": 2929,
+ "stations": 2930,
+ "politician": 2931,
+ "Olympics": 2932,
+ "influence": 2933,
+ "share": 2934,
+ "fighting": 2935,
+ "speak": 2936,
+ "hundred": 2937,
+ "Carolina": 2938,
+ "die": 2939,
+ "stars": 2940,
+ "##tic": 2941,
+ "color": 2942,
+ "Chapter": 2943,
+ "##ish": 2944,
+ "fear": 2945,
+ "sleep": 2946,
+ "goes": 2947,
+ "Francisco": 2948,
+ "oil": 2949,
+ "Bank": 2950,
+ "sign": 2951,
+ "physical": 2952,
+ "##berg": 2953,
+ "Dutch": 2954,
+ "seasons": 2955,
+ "##rd": 2956,
+ "Games": 2957,
+ "Governor": 2958,
+ "sorry": 2959,
+ "lack": 2960,
+ "Centre": 2961,
+ "memory": 2962,
+ "baby": 2963,
+ "smaller": 2964,
+ "charge": 2965,
+ "Did": 2966,
+ "multiple": 2967,
+ "ships": 2968,
+ "shirt": 2969,
+ "Assembly": 2970,
+ "amount": 2971,
+ "leaves": 2972,
+ "3rd": 2973,
+ "Foundation": 2974,
+ "conditions": 2975,
+ "1943": 2976,
+ "Rock": 2977,
+ "Democratic": 2978,
+ "Daniel": 2979,
+ "##at": 2980,
+ "winner": 2981,
+ "products": 2982,
+ "##ina": 2983,
+ "store": 2984,
+ "latter": 2985,
+ "Professor": 2986,
+ "civil": 2987,
+ "prior": 2988,
+ "host": 2989,
+ "1956": 2990,
+ "soft": 2991,
+ "vote": 2992,
+ "needs": 2993,
+ "Each": 2994,
+ "rules": 2995,
+ "1958": 2996,
+ "pressure": 2997,
+ "letter": 2998,
+ "normal": 2999,
+ "proposed": 3000,
+ "levels": 3001,
+ "records": 3002,
+ "1959": 3003,
+ "paid": 3004,
+ "intended": 3005,
+ "Victoria": 3006,
+ "purpose": 3007,
+ "okay": 3008,
+ "historical": 3009,
+ "issued": 3010,
+ "1980s": 3011,
+ "broadcast": 3012,
+ "rule": 3013,
+ "simple": 3014,
+ "picked": 3015,
+ "firm": 3016,
+ "Sea": 3017,
+ "1941": 3018,
+ "Elizabeth": 3019,
+ "1940": 3020,
+ "serious": 3021,
+ "featuring": 3022,
+ "highly": 3023,
+ "graduated": 3024,
+ "mentioned": 3025,
+ "choice": 3026,
+ "1948": 3027,
+ "replied": 3028,
+ "percent": 3029,
+ "Scotland": 3030,
+ "##hi": 3031,
+ "females": 3032,
+ "constructed": 3033,
+ "1957": 3034,
+ "settled": 3035,
+ "Steve": 3036,
+ "recognized": 3037,
+ "cities": 3038,
+ "crew": 3039,
+ "glanced": 3040,
+ "kiss": 3041,
+ "competed": 3042,
+ "flight": 3043,
+ "knowledge": 3044,
+ "editor": 3045,
+ "More": 3046,
+ "Conference": 3047,
+ "##H": 3048,
+ "fifth": 3049,
+ "elements": 3050,
+ "##ee": 3051,
+ "##tes": 3052,
+ "function": 3053,
+ "newspaper": 3054,
+ "recently": 3055,
+ "Miss": 3056,
+ "cultural": 3057,
+ "brown": 3058,
+ "twice": 3059,
+ "Office": 3060,
+ "1939": 3061,
+ "truth": 3062,
+ "Creek": 3063,
+ "1946": 3064,
+ "households": 3065,
+ "USA": 3066,
+ "1950": 3067,
+ "quality": 3068,
+ "##tt": 3069,
+ "border": 3070,
+ "seconds": 3071,
+ "destroyed": 3072,
+ "pre": 3073,
+ "wait": 3074,
+ "ahead": 3075,
+ "build": 3076,
+ "image": 3077,
+ "90": 3078,
+ "cars": 3079,
+ "##mi": 3080,
+ "33": 3081,
+ "promoted": 3082,
+ "professor": 3083,
+ "et": 3084,
+ "bank": 3085,
+ "medal": 3086,
+ "text": 3087,
+ "broken": 3088,
+ "Middle": 3089,
+ "revealed": 3090,
+ "sides": 3091,
+ "wing": 3092,
+ "seems": 3093,
+ "channel": 3094,
+ "1970s": 3095,
+ "Ben": 3096,
+ "loved": 3097,
+ "effort": 3098,
+ "officers": 3099,
+ "Will": 3100,
+ "##ff": 3101,
+ "70": 3102,
+ "Israel": 3103,
+ "Jim": 3104,
+ "upper": 3105,
+ "fully": 3106,
+ "label": 3107,
+ "Jr": 3108,
+ "assistant": 3109,
+ "powerful": 3110,
+ "pair": 3111,
+ "positive": 3112,
+ "##ary": 3113,
+ "gives": 3114,
+ "1955": 3115,
+ "20th": 3116,
+ "races": 3117,
+ "remain": 3118,
+ "kitchen": 3119,
+ "primarily": 3120,
+ "##ti": 3121,
+ "Sydney": 3122,
+ "easy": 3123,
+ "Tour": 3124,
+ "whispered": 3125,
+ "buried": 3126,
+ "300": 3127,
+ "News": 3128,
+ "Polish": 3129,
+ "1952": 3130,
+ "Duke": 3131,
+ "Columbia": 3132,
+ "produce": 3133,
+ "accepted": 3134,
+ "00": 3135,
+ "approach": 3136,
+ "minor": 3137,
+ "1947": 3138,
+ "Special": 3139,
+ "44": 3140,
+ "Asian": 3141,
+ "basis": 3142,
+ "visit": 3143,
+ "Fort": 3144,
+ "Civil": 3145,
+ "finish": 3146,
+ "formerly": 3147,
+ "beside": 3148,
+ "leaned": 3149,
+ "##ite": 3150,
+ "median": 3151,
+ "rose": 3152,
+ "coast": 3153,
+ "effects": 3154,
+ "supposed": 3155,
+ "Cross": 3156,
+ "##hip": 3157,
+ "Corps": 3158,
+ "residents": 3159,
+ "Jackson": 3160,
+ "##ir": 3161,
+ "Bob": 3162,
+ "basketball": 3163,
+ "36": 3164,
+ "Asia": 3165,
+ "seem": 3166,
+ "Bishop": 3167,
+ "Book": 3168,
+ "##ber": 3169,
+ "ring": 3170,
+ "##ze": 3171,
+ "owner": 3172,
+ "BBC": 3173,
+ "##ja": 3174,
+ "transferred": 3175,
+ "acting": 3176,
+ "De": 3177,
+ "appearances": 3178,
+ "walking": 3179,
+ "Le": 3180,
+ "press": 3181,
+ "grabbed": 3182,
+ "1954": 3183,
+ "officially": 3184,
+ "1953": 3185,
+ "##pe": 3186,
+ "risk": 3187,
+ "taught": 3188,
+ "review": 3189,
+ "##X": 3190,
+ "lay": 3191,
+ "##well": 3192,
+ "council": 3193,
+ "Avenue": 3194,
+ "seeing": 3195,
+ "losing": 3196,
+ "Ohio": 3197,
+ "Super": 3198,
+ "province": 3199,
+ "ones": 3200,
+ "travel": 3201,
+ "##sa": 3202,
+ "projects": 3203,
+ "equipment": 3204,
+ "spot": 3205,
+ "Berlin": 3206,
+ "administrative": 3207,
+ "heat": 3208,
+ "potential": 3209,
+ "shut": 3210,
+ "capacity": 3211,
+ "elections": 3212,
+ "growth": 3213,
+ "fought": 3214,
+ "Republican": 3215,
+ "mixed": 3216,
+ "Andrew": 3217,
+ "teacher": 3218,
+ "turning": 3219,
+ "strength": 3220,
+ "shoulders": 3221,
+ "beat": 3222,
+ "wind": 3223,
+ "1949": 3224,
+ "Health": 3225,
+ "follows": 3226,
+ "camp": 3227,
+ "suggested": 3228,
+ "perhaps": 3229,
+ "Alex": 3230,
+ "mountain": 3231,
+ "contact": 3232,
+ "divided": 3233,
+ "candidate": 3234,
+ "fellow": 3235,
+ "34": 3236,
+ "Show": 3237,
+ "necessary": 3238,
+ "workers": 3239,
+ "ball": 3240,
+ "horse": 3241,
+ "ways": 3242,
+ "questions": 3243,
+ "protect": 3244,
+ "gas": 3245,
+ "activity": 3246,
+ "younger": 3247,
+ "bottom": 3248,
+ "founder": 3249,
+ "Scottish": 3250,
+ "screen": 3251,
+ "treatment": 3252,
+ "easily": 3253,
+ "com": 3254,
+ "##house": 3255,
+ "dedicated": 3256,
+ "Master": 3257,
+ "warm": 3258,
+ "Night": 3259,
+ "Georgia": 3260,
+ "Long": 3261,
+ "von": 3262,
+ "##me": 3263,
+ "perfect": 3264,
+ "website": 3265,
+ "1960s": 3266,
+ "piano": 3267,
+ "efforts": 3268,
+ "##ide": 3269,
+ "Tony": 3270,
+ "sort": 3271,
+ "offers": 3272,
+ "Development": 3273,
+ "Simon": 3274,
+ "executive": 3275,
+ "##nd": 3276,
+ "save": 3277,
+ "Over": 3278,
+ "Senate": 3279,
+ "1951": 3280,
+ "1990s": 3281,
+ "draw": 3282,
+ "master": 3283,
+ "Police": 3284,
+ "##ius": 3285,
+ "renamed": 3286,
+ "boys": 3287,
+ "initial": 3288,
+ "prominent": 3289,
+ "damage": 3290,
+ "Co": 3291,
+ "##ov": 3292,
+ "##za": 3293,
+ "online": 3294,
+ "begin": 3295,
+ "occurred": 3296,
+ "captured": 3297,
+ "youth": 3298,
+ "Top": 3299,
+ "account": 3300,
+ "tells": 3301,
+ "Justice": 3302,
+ "conducted": 3303,
+ "forest": 3304,
+ "##town": 3305,
+ "bought": 3306,
+ "teeth": 3307,
+ "Jersey": 3308,
+ "##di": 3309,
+ "purchased": 3310,
+ "agreement": 3311,
+ "Michigan": 3312,
+ "##ure": 3313,
+ "campus": 3314,
+ "prison": 3315,
+ "becomes": 3316,
+ "product": 3317,
+ "secret": 3318,
+ "guess": 3319,
+ "Route": 3320,
+ "huge": 3321,
+ "types": 3322,
+ "drums": 3323,
+ "64": 3324,
+ "split": 3325,
+ "defeat": 3326,
+ "estate": 3327,
+ "housing": 3328,
+ "##ot": 3329,
+ "brothers": 3330,
+ "Coast": 3331,
+ "declared": 3332,
+ "happen": 3333,
+ "titled": 3334,
+ "therefore": 3335,
+ "sun": 3336,
+ "commonly": 3337,
+ "alongside": 3338,
+ "Stadium": 3339,
+ "library": 3340,
+ "Home": 3341,
+ "article": 3342,
+ "steps": 3343,
+ "telling": 3344,
+ "slow": 3345,
+ "assigned": 3346,
+ "refused": 3347,
+ "laughed": 3348,
+ "wants": 3349,
+ "Nick": 3350,
+ "wearing": 3351,
+ "Rome": 3352,
+ "Open": 3353,
+ "##ah": 3354,
+ "Hospital": 3355,
+ "pointed": 3356,
+ "Taylor": 3357,
+ "lifted": 3358,
+ "escape": 3359,
+ "participated": 3360,
+ "##j": 3361,
+ "drama": 3362,
+ "parish": 3363,
+ "Santa": 3364,
+ "##per": 3365,
+ "organized": 3366,
+ "mass": 3367,
+ "pick": 3368,
+ "Airport": 3369,
+ "gets": 3370,
+ "Library": 3371,
+ "unable": 3372,
+ "pull": 3373,
+ "Live": 3374,
+ "##ging": 3375,
+ "surrounding": 3376,
+ "##ries": 3377,
+ "focused": 3378,
+ "Adam": 3379,
+ "facilities": 3380,
+ "##ning": 3381,
+ "##ny": 3382,
+ "38": 3383,
+ "##ring": 3384,
+ "notable": 3385,
+ "era": 3386,
+ "connected": 3387,
+ "gained": 3388,
+ "operating": 3389,
+ "laid": 3390,
+ "Regiment": 3391,
+ "branch": 3392,
+ "defined": 3393,
+ "Christmas": 3394,
+ "machine": 3395,
+ "Four": 3396,
+ "academic": 3397,
+ "Iran": 3398,
+ "adopted": 3399,
+ "concept": 3400,
+ "Men": 3401,
+ "compared": 3402,
+ "search": 3403,
+ "traffic": 3404,
+ "Max": 3405,
+ "Maria": 3406,
+ "greater": 3407,
+ "##ding": 3408,
+ "widely": 3409,
+ "##burg": 3410,
+ "serves": 3411,
+ "1938": 3412,
+ "37": 3413,
+ "Go": 3414,
+ "hotel": 3415,
+ "shared": 3416,
+ "typically": 3417,
+ "scale": 3418,
+ "1936": 3419,
+ "leg": 3420,
+ "suffered": 3421,
+ "yards": 3422,
+ "pieces": 3423,
+ "Ministry": 3424,
+ "Wilson": 3425,
+ "episodes": 3426,
+ "empty": 3427,
+ "1918": 3428,
+ "safety": 3429,
+ "continues": 3430,
+ "yellow": 3431,
+ "historic": 3432,
+ "settlement": 3433,
+ "400": 3434,
+ "Come": 3435,
+ "Corporation": 3436,
+ "enemy": 3437,
+ "content": 3438,
+ "picture": 3439,
+ "evening": 3440,
+ "territory": 3441,
+ "method": 3442,
+ "trial": 3443,
+ "solo": 3444,
+ "driver": 3445,
+ "Here": 3446,
+ "##ls": 3447,
+ "entrance": 3448,
+ "Prize": 3449,
+ "spring": 3450,
+ "whatever": 3451,
+ "##ent": 3452,
+ "75": 3453,
+ "##ji": 3454,
+ "reading": 3455,
+ "Arthur": 3456,
+ "##cy": 3457,
+ "Our": 3458,
+ "clothes": 3459,
+ "Prime": 3460,
+ "Illinois": 3461,
+ "Kong": 3462,
+ "code": 3463,
+ "##ria": 3464,
+ "sit": 3465,
+ "Harry": 3466,
+ "Federal": 3467,
+ "chosen": 3468,
+ "administration": 3469,
+ "bodies": 3470,
+ "begins": 3471,
+ "stomach": 3472,
+ "Though": 3473,
+ "seats": 3474,
+ "Hong": 3475,
+ "density": 3476,
+ "Sun": 3477,
+ "leaders": 3478,
+ "Field": 3479,
+ "museum": 3480,
+ "chart": 3481,
+ "platform": 3482,
+ "languages": 3483,
+ "##ron": 3484,
+ "birth": 3485,
+ "holds": 3486,
+ "Gold": 3487,
+ "##un": 3488,
+ "fish": 3489,
+ "combined": 3490,
+ "##ps": 3491,
+ "4th": 3492,
+ "1937": 3493,
+ "largely": 3494,
+ "captain": 3495,
+ "trust": 3496,
+ "Game": 3497,
+ "van": 3498,
+ "boat": 3499,
+ "Oxford": 3500,
+ "basic": 3501,
+ "beneath": 3502,
+ "Islands": 3503,
+ "painting": 3504,
+ "nice": 3505,
+ "Toronto": 3506,
+ "path": 3507,
+ "males": 3508,
+ "sources": 3509,
+ "block": 3510,
+ "conference": 3511,
+ "parties": 3512,
+ "murder": 3513,
+ "clubs": 3514,
+ "crowd": 3515,
+ "calling": 3516,
+ "About": 3517,
+ "Business": 3518,
+ "peace": 3519,
+ "knows": 3520,
+ "lake": 3521,
+ "speaking": 3522,
+ "stayed": 3523,
+ "Brazil": 3524,
+ "allowing": 3525,
+ "Born": 3526,
+ "unique": 3527,
+ "thick": 3528,
+ "Technology": 3529,
+ "##que": 3530,
+ "receive": 3531,
+ "des": 3532,
+ "semi": 3533,
+ "alive": 3534,
+ "noticed": 3535,
+ "format": 3536,
+ "##ped": 3537,
+ "coffee": 3538,
+ "digital": 3539,
+ "##ned": 3540,
+ "handed": 3541,
+ "guard": 3542,
+ "tall": 3543,
+ "faced": 3544,
+ "setting": 3545,
+ "plants": 3546,
+ "partner": 3547,
+ "claim": 3548,
+ "reduced": 3549,
+ "temple": 3550,
+ "animals": 3551,
+ "determined": 3552,
+ "classes": 3553,
+ "##out": 3554,
+ "estimated": 3555,
+ "##ad": 3556,
+ "Olympic": 3557,
+ "providing": 3558,
+ "Massachusetts": 3559,
+ "learned": 3560,
+ "Inc": 3561,
+ "Philadelphia": 3562,
+ "Social": 3563,
+ "carry": 3564,
+ "42": 3565,
+ "possibly": 3566,
+ "hosted": 3567,
+ "tonight": 3568,
+ "respectively": 3569,
+ "Today": 3570,
+ "shape": 3571,
+ "Mount": 3572,
+ "roles": 3573,
+ "designated": 3574,
+ "brain": 3575,
+ "etc": 3576,
+ "Korea": 3577,
+ "thoughts": 3578,
+ "Brian": 3579,
+ "Highway": 3580,
+ "doors": 3581,
+ "background": 3582,
+ "drew": 3583,
+ "models": 3584,
+ "footballer": 3585,
+ "tone": 3586,
+ "turns": 3587,
+ "1935": 3588,
+ "quiet": 3589,
+ "tower": 3590,
+ "wood": 3591,
+ "bus": 3592,
+ "write": 3593,
+ "software": 3594,
+ "weapons": 3595,
+ "flat": 3596,
+ "marked": 3597,
+ "1920": 3598,
+ "newly": 3599,
+ "tight": 3600,
+ "Eric": 3601,
+ "finger": 3602,
+ "Journal": 3603,
+ "FC": 3604,
+ "Van": 3605,
+ "rise": 3606,
+ "critical": 3607,
+ "Atlantic": 3608,
+ "granted": 3609,
+ "returning": 3610,
+ "communities": 3611,
+ "humans": 3612,
+ "quick": 3613,
+ "39": 3614,
+ "48": 3615,
+ "ranked": 3616,
+ "sight": 3617,
+ "pop": 3618,
+ "Swedish": 3619,
+ "Stephen": 3620,
+ "card": 3621,
+ "analysis": 3622,
+ "attacked": 3623,
+ "##wa": 3624,
+ "Sunday": 3625,
+ "identified": 3626,
+ "Jason": 3627,
+ "champion": 3628,
+ "situated": 3629,
+ "1930": 3630,
+ "expanded": 3631,
+ "tears": 3632,
+ "##nce": 3633,
+ "reaching": 3634,
+ "Davis": 3635,
+ "protection": 3636,
+ "Emperor": 3637,
+ "positions": 3638,
+ "nominated": 3639,
+ "Bridge": 3640,
+ "tax": 3641,
+ "dress": 3642,
+ "allows": 3643,
+ "avoid": 3644,
+ "leadership": 3645,
+ "killing": 3646,
+ "actress": 3647,
+ "guest": 3648,
+ "steel": 3649,
+ "knowing": 3650,
+ "electric": 3651,
+ "cells": 3652,
+ "disease": 3653,
+ "grade": 3654,
+ "unknown": 3655,
+ "##ium": 3656,
+ "resulted": 3657,
+ "Pakistan": 3658,
+ "confirmed": 3659,
+ "##ged": 3660,
+ "tongue": 3661,
+ "covers": 3662,
+ "##Y": 3663,
+ "roof": 3664,
+ "entirely": 3665,
+ "applied": 3666,
+ "votes": 3667,
+ "drink": 3668,
+ "interview": 3669,
+ "exchange": 3670,
+ "Township": 3671,
+ "reasons": 3672,
+ "##ised": 3673,
+ "page": 3674,
+ "calls": 3675,
+ "dog": 3676,
+ "agent": 3677,
+ "nose": 3678,
+ "teaching": 3679,
+ "##ds": 3680,
+ "##ists": 3681,
+ "advanced": 3682,
+ "wish": 3683,
+ "Golden": 3684,
+ "existing": 3685,
+ "vehicle": 3686,
+ "del": 3687,
+ "1919": 3688,
+ "develop": 3689,
+ "attacks": 3690,
+ "pressed": 3691,
+ "Sports": 3692,
+ "planning": 3693,
+ "resulting": 3694,
+ "facility": 3695,
+ "Sarah": 3696,
+ "notes": 3697,
+ "1933": 3698,
+ "Class": 3699,
+ "Historic": 3700,
+ "winter": 3701,
+ "##mo": 3702,
+ "audience": 3703,
+ "Community": 3704,
+ "household": 3705,
+ "Netherlands": 3706,
+ "creation": 3707,
+ "##ize": 3708,
+ "keeping": 3709,
+ "1914": 3710,
+ "claims": 3711,
+ "dry": 3712,
+ "guys": 3713,
+ "opposite": 3714,
+ "##ak": 3715,
+ "explained": 3716,
+ "Ontario": 3717,
+ "secondary": 3718,
+ "difference": 3719,
+ "Francis": 3720,
+ "actions": 3721,
+ "organizations": 3722,
+ "yard": 3723,
+ "animal": 3724,
+ "Up": 3725,
+ "Lewis": 3726,
+ "titles": 3727,
+ "Several": 3728,
+ "1934": 3729,
+ "Ryan": 3730,
+ "55": 3731,
+ "Supreme": 3732,
+ "rolled": 3733,
+ "1917": 3734,
+ "distribution": 3735,
+ "figures": 3736,
+ "afraid": 3737,
+ "rural": 3738,
+ "yourself": 3739,
+ "##rt": 3740,
+ "sets": 3741,
+ "barely": 3742,
+ "Instead": 3743,
+ "passing": 3744,
+ "awards": 3745,
+ "41": 3746,
+ "silence": 3747,
+ "authority": 3748,
+ "occupied": 3749,
+ "environment": 3750,
+ "windows": 3751,
+ "engineering": 3752,
+ "surprised": 3753,
+ "flying": 3754,
+ "crime": 3755,
+ "reports": 3756,
+ "Mountain": 3757,
+ "powers": 3758,
+ "driving": 3759,
+ "succeeded": 3760,
+ "reviews": 3761,
+ "1929": 3762,
+ "Head": 3763,
+ "missing": 3764,
+ "Song": 3765,
+ "Jesus": 3766,
+ "opportunity": 3767,
+ "inspired": 3768,
+ "ends": 3769,
+ "albums": 3770,
+ "conversation": 3771,
+ "impact": 3772,
+ "injury": 3773,
+ "surprise": 3774,
+ "billion": 3775,
+ "learning": 3776,
+ "heavily": 3777,
+ "oldest": 3778,
+ "union": 3779,
+ "creating": 3780,
+ "##ky": 3781,
+ "festival": 3782,
+ "literature": 3783,
+ "letters": 3784,
+ "sexual": 3785,
+ "##tte": 3786,
+ "apartment": 3787,
+ "Final": 3788,
+ "comedy": 3789,
+ "nation": 3790,
+ "orders": 3791,
+ "##sen": 3792,
+ "contemporary": 3793,
+ "Power": 3794,
+ "drawn": 3795,
+ "existence": 3796,
+ "connection": 3797,
+ "##ating": 3798,
+ "Post": 3799,
+ "Junior": 3800,
+ "remembered": 3801,
+ "message": 3802,
+ "Medal": 3803,
+ "castle": 3804,
+ "note": 3805,
+ "engineer": 3806,
+ "sounds": 3807,
+ "Beach": 3808,
+ "crossed": 3809,
+ "##dy": 3810,
+ "ear": 3811,
+ "scientific": 3812,
+ "sales": 3813,
+ "##ai": 3814,
+ "theme": 3815,
+ "starts": 3816,
+ "clearly": 3817,
+ "##ut": 3818,
+ "trouble": 3819,
+ "##gan": 3820,
+ "bag": 3821,
+ "##han": 3822,
+ "BC": 3823,
+ "sons": 3824,
+ "1928": 3825,
+ "silent": 3826,
+ "versions": 3827,
+ "daily": 3828,
+ "Studies": 3829,
+ "ending": 3830,
+ "Rose": 3831,
+ "guns": 3832,
+ "1932": 3833,
+ "headquarters": 3834,
+ "reference": 3835,
+ "obtained": 3836,
+ "Squadron": 3837,
+ "concert": 3838,
+ "none": 3839,
+ "du": 3840,
+ "Among": 3841,
+ "##don": 3842,
+ "prevent": 3843,
+ "Member": 3844,
+ "answered": 3845,
+ "staring": 3846,
+ "Between": 3847,
+ "##lla": 3848,
+ "portion": 3849,
+ "drug": 3850,
+ "liked": 3851,
+ "association": 3852,
+ "performances": 3853,
+ "Nations": 3854,
+ "formation": 3855,
+ "Castle": 3856,
+ "lose": 3857,
+ "learn": 3858,
+ "scoring": 3859,
+ "relatively": 3860,
+ "quarter": 3861,
+ "47": 3862,
+ "Premier": 3863,
+ "##ors": 3864,
+ "Sweden": 3865,
+ "baseball": 3866,
+ "attempted": 3867,
+ "trip": 3868,
+ "worth": 3869,
+ "perform": 3870,
+ "airport": 3871,
+ "fields": 3872,
+ "enter": 3873,
+ "honor": 3874,
+ "Medical": 3875,
+ "rear": 3876,
+ "commander": 3877,
+ "officials": 3878,
+ "condition": 3879,
+ "supply": 3880,
+ "materials": 3881,
+ "52": 3882,
+ "Anna": 3883,
+ "volume": 3884,
+ "threw": 3885,
+ "Persian": 3886,
+ "43": 3887,
+ "interested": 3888,
+ "Gallery": 3889,
+ "achieved": 3890,
+ "visited": 3891,
+ "laws": 3892,
+ "relief": 3893,
+ "Area": 3894,
+ "Matt": 3895,
+ "singles": 3896,
+ "Lieutenant": 3897,
+ "Country": 3898,
+ "fans": 3899,
+ "Cambridge": 3900,
+ "sky": 3901,
+ "Miller": 3902,
+ "effective": 3903,
+ "tradition": 3904,
+ "Port": 3905,
+ "##ana": 3906,
+ "minister": 3907,
+ "extra": 3908,
+ "entitled": 3909,
+ "System": 3910,
+ "sites": 3911,
+ "authorities": 3912,
+ "acres": 3913,
+ "committee": 3914,
+ "racing": 3915,
+ "1931": 3916,
+ "desk": 3917,
+ "trains": 3918,
+ "ass": 3919,
+ "weren": 3920,
+ "Family": 3921,
+ "farm": 3922,
+ "##ance": 3923,
+ "industrial": 3924,
+ "##head": 3925,
+ "iron": 3926,
+ "49": 3927,
+ "abandoned": 3928,
+ "Out": 3929,
+ "Holy": 3930,
+ "chairman": 3931,
+ "waited": 3932,
+ "frequently": 3933,
+ "display": 3934,
+ "Light": 3935,
+ "transport": 3936,
+ "starring": 3937,
+ "Patrick": 3938,
+ "Engineering": 3939,
+ "eat": 3940,
+ "FM": 3941,
+ "judge": 3942,
+ "reaction": 3943,
+ "centuries": 3944,
+ "price": 3945,
+ "##tive": 3946,
+ "Korean": 3947,
+ "defense": 3948,
+ "Get": 3949,
+ "arrested": 3950,
+ "1927": 3951,
+ "send": 3952,
+ "urban": 3953,
+ "##ss": 3954,
+ "pilot": 3955,
+ "Okay": 3956,
+ "Media": 3957,
+ "reality": 3958,
+ "arts": 3959,
+ "soul": 3960,
+ "thirty": 3961,
+ "##be": 3962,
+ "catch": 3963,
+ "generation": 3964,
+ "##nes": 3965,
+ "apart": 3966,
+ "Anne": 3967,
+ "drop": 3968,
+ "See": 3969,
+ "##ving": 3970,
+ "sixth": 3971,
+ "trained": 3972,
+ "Management": 3973,
+ "magic": 3974,
+ "cm": 3975,
+ "height": 3976,
+ "Fox": 3977,
+ "Ian": 3978,
+ "resources": 3979,
+ "vampire": 3980,
+ "principal": 3981,
+ "Was": 3982,
+ "haven": 3983,
+ "##au": 3984,
+ "Walter": 3985,
+ "Albert": 3986,
+ "rich": 3987,
+ "1922": 3988,
+ "causing": 3989,
+ "entry": 3990,
+ "##ell": 3991,
+ "shortly": 3992,
+ "46": 3993,
+ "worry": 3994,
+ "doctor": 3995,
+ "composer": 3996,
+ "rank": 3997,
+ "Network": 3998,
+ "bright": 3999,
+ "showing": 4000,
+ "regions": 4001,
+ "1924": 4002,
+ "wave": 4003,
+ "carrying": 4004,
+ "kissed": 4005,
+ "finding": 4006,
+ "missed": 4007,
+ "Earl": 4008,
+ "lying": 4009,
+ "target": 4010,
+ "vehicles": 4011,
+ "Military": 4012,
+ "controlled": 4013,
+ "dinner": 4014,
+ "##board": 4015,
+ "briefly": 4016,
+ "lyrics": 4017,
+ "motion": 4018,
+ "duty": 4019,
+ "strange": 4020,
+ "attempts": 4021,
+ "invited": 4022,
+ "kg": 4023,
+ "villages": 4024,
+ "5th": 4025,
+ "Land": 4026,
+ "##mer": 4027,
+ "Christ": 4028,
+ "prepared": 4029,
+ "twelve": 4030,
+ "check": 4031,
+ "thousand": 4032,
+ "earth": 4033,
+ "copies": 4034,
+ "en": 4035,
+ "transfer": 4036,
+ "citizens": 4037,
+ "Americans": 4038,
+ "politics": 4039,
+ "nor": 4040,
+ "theatre": 4041,
+ "Project": 4042,
+ "##bo": 4043,
+ "clean": 4044,
+ "rooms": 4045,
+ "laugh": 4046,
+ "##ran": 4047,
+ "application": 4048,
+ "contained": 4049,
+ "anyway": 4050,
+ "containing": 4051,
+ "Sciences": 4052,
+ "1925": 4053,
+ "rare": 4054,
+ "speech": 4055,
+ "exist": 4056,
+ "1950s": 4057,
+ "falling": 4058,
+ "passenger": 4059,
+ "##im": 4060,
+ "stands": 4061,
+ "51": 4062,
+ "##ol": 4063,
+ "##ow": 4064,
+ "phase": 4065,
+ "governor": 4066,
+ "kids": 4067,
+ "details": 4068,
+ "methods": 4069,
+ "Vice": 4070,
+ "employed": 4071,
+ "performing": 4072,
+ "counter": 4073,
+ "Jane": 4074,
+ "heads": 4075,
+ "Channel": 4076,
+ "wine": 4077,
+ "opposition": 4078,
+ "aged": 4079,
+ "1912": 4080,
+ "Every": 4081,
+ "1926": 4082,
+ "highway": 4083,
+ "##ura": 4084,
+ "1921": 4085,
+ "aired": 4086,
+ "978": 4087,
+ "permanent": 4088,
+ "Forest": 4089,
+ "finds": 4090,
+ "joint": 4091,
+ "approved": 4092,
+ "##pur": 4093,
+ "brief": 4094,
+ "doubt": 4095,
+ "acts": 4096,
+ "brand": 4097,
+ "wild": 4098,
+ "closely": 4099,
+ "Ford": 4100,
+ "Kevin": 4101,
+ "chose": 4102,
+ "shall": 4103,
+ "port": 4104,
+ "sweet": 4105,
+ "fun": 4106,
+ "asking": 4107,
+ "Be": 4108,
+ "##bury": 4109,
+ "sought": 4110,
+ "Dave": 4111,
+ "Mexican": 4112,
+ "mom": 4113,
+ "Right": 4114,
+ "Howard": 4115,
+ "Moscow": 4116,
+ "Charlie": 4117,
+ "Stone": 4118,
+ "##mann": 4119,
+ "admitted": 4120,
+ "##ver": 4121,
+ "wooden": 4122,
+ "1923": 4123,
+ "Officer": 4124,
+ "relations": 4125,
+ "Hot": 4126,
+ "combat": 4127,
+ "publication": 4128,
+ "chain": 4129,
+ "shop": 4130,
+ "inhabitants": 4131,
+ "proved": 4132,
+ "ideas": 4133,
+ "address": 4134,
+ "1915": 4135,
+ "Memorial": 4136,
+ "explain": 4137,
+ "increasing": 4138,
+ "conflict": 4139,
+ "Anthony": 4140,
+ "Melbourne": 4141,
+ "narrow": 4142,
+ "temperature": 4143,
+ "slid": 4144,
+ "1916": 4145,
+ "worse": 4146,
+ "selling": 4147,
+ "documentary": 4148,
+ "Ali": 4149,
+ "Ray": 4150,
+ "opposed": 4151,
+ "vision": 4152,
+ "dad": 4153,
+ "extensive": 4154,
+ "Infantry": 4155,
+ "commissioned": 4156,
+ "Doctor": 4157,
+ "offices": 4158,
+ "programming": 4159,
+ "core": 4160,
+ "respect": 4161,
+ "storm": 4162,
+ "##pa": 4163,
+ "##ay": 4164,
+ "##om": 4165,
+ "promotion": 4166,
+ "der": 4167,
+ "struck": 4168,
+ "anymore": 4169,
+ "shit": 4170,
+ "Region": 4171,
+ "receiving": 4172,
+ "DVD": 4173,
+ "alternative": 4174,
+ "##ue": 4175,
+ "ride": 4176,
+ "maximum": 4177,
+ "1910": 4178,
+ "##ious": 4179,
+ "Third": 4180,
+ "Affairs": 4181,
+ "cancer": 4182,
+ "Executive": 4183,
+ "##op": 4184,
+ "dream": 4185,
+ "18th": 4186,
+ "Due": 4187,
+ "##ker": 4188,
+ "##worth": 4189,
+ "economy": 4190,
+ "IV": 4191,
+ "Billboard": 4192,
+ "identity": 4193,
+ "subsequent": 4194,
+ "statement": 4195,
+ "skills": 4196,
+ "##back": 4197,
+ "funding": 4198,
+ "##ons": 4199,
+ "Round": 4200,
+ "Foreign": 4201,
+ "truck": 4202,
+ "Please": 4203,
+ "lights": 4204,
+ "wondered": 4205,
+ "##ms": 4206,
+ "frame": 4207,
+ "yes": 4208,
+ "Still": 4209,
+ "districts": 4210,
+ "fiction": 4211,
+ "Colonel": 4212,
+ "converted": 4213,
+ "150": 4214,
+ "grown": 4215,
+ "accident": 4216,
+ "critics": 4217,
+ "fit": 4218,
+ "Information": 4219,
+ "architecture": 4220,
+ "Point": 4221,
+ "Five": 4222,
+ "armed": 4223,
+ "Billy": 4224,
+ "poet": 4225,
+ "functions": 4226,
+ "consisted": 4227,
+ "suit": 4228,
+ "Turkish": 4229,
+ "Band": 4230,
+ "object": 4231,
+ "desire": 4232,
+ "##ities": 4233,
+ "sounded": 4234,
+ "flow": 4235,
+ "Norwegian": 4236,
+ "articles": 4237,
+ "Marie": 4238,
+ "pulling": 4239,
+ "thin": 4240,
+ "singing": 4241,
+ "Hunter": 4242,
+ "Human": 4243,
+ "Battalion": 4244,
+ "Federation": 4245,
+ "Kim": 4246,
+ "origin": 4247,
+ "represent": 4248,
+ "dangerous": 4249,
+ "weather": 4250,
+ "fuel": 4251,
+ "ex": 4252,
+ "##sing": 4253,
+ "Last": 4254,
+ "bedroom": 4255,
+ "aid": 4256,
+ "knees": 4257,
+ "Alan": 4258,
+ "angry": 4259,
+ "assumed": 4260,
+ "plane": 4261,
+ "Something": 4262,
+ "founding": 4263,
+ "concerned": 4264,
+ "global": 4265,
+ "Fire": 4266,
+ "di": 4267,
+ "please": 4268,
+ "Portuguese": 4269,
+ "touched": 4270,
+ "Roger": 4271,
+ "nuclear": 4272,
+ "Register": 4273,
+ "Jeff": 4274,
+ "fixed": 4275,
+ "royal": 4276,
+ "lie": 4277,
+ "finals": 4278,
+ "NFL": 4279,
+ "Manchester": 4280,
+ "towns": 4281,
+ "handle": 4282,
+ "shaped": 4283,
+ "Chairman": 4284,
+ "Dean": 4285,
+ "launch": 4286,
+ "understanding": 4287,
+ "Children": 4288,
+ "violence": 4289,
+ "failure": 4290,
+ "sector": 4291,
+ "Brigade": 4292,
+ "wrapped": 4293,
+ "fired": 4294,
+ "sharp": 4295,
+ "tiny": 4296,
+ "developing": 4297,
+ "expansion": 4298,
+ "Free": 4299,
+ "institutions": 4300,
+ "technical": 4301,
+ "Nothing": 4302,
+ "otherwise": 4303,
+ "Main": 4304,
+ "inch": 4305,
+ "Saturday": 4306,
+ "wore": 4307,
+ "Senior": 4308,
+ "attached": 4309,
+ "cheek": 4310,
+ "representing": 4311,
+ "Kansas": 4312,
+ "##chi": 4313,
+ "##kin": 4314,
+ "actual": 4315,
+ "advantage": 4316,
+ "Dan": 4317,
+ "Austria": 4318,
+ "##dale": 4319,
+ "hoped": 4320,
+ "multi": 4321,
+ "squad": 4322,
+ "Norway": 4323,
+ "streets": 4324,
+ "1913": 4325,
+ "Services": 4326,
+ "hired": 4327,
+ "grow": 4328,
+ "pp": 4329,
+ "wear": 4330,
+ "painted": 4331,
+ "Minnesota": 4332,
+ "stuff": 4333,
+ "Building": 4334,
+ "54": 4335,
+ "Philippines": 4336,
+ "1900": 4337,
+ "##ties": 4338,
+ "educational": 4339,
+ "Khan": 4340,
+ "Magazine": 4341,
+ "##port": 4342,
+ "Cape": 4343,
+ "signal": 4344,
+ "Gordon": 4345,
+ "sword": 4346,
+ "Anderson": 4347,
+ "cool": 4348,
+ "engaged": 4349,
+ "Commander": 4350,
+ "images": 4351,
+ "Upon": 4352,
+ "tied": 4353,
+ "Security": 4354,
+ "cup": 4355,
+ "rail": 4356,
+ "Vietnam": 4357,
+ "successfully": 4358,
+ "##red": 4359,
+ "Muslim": 4360,
+ "gain": 4361,
+ "bringing": 4362,
+ "Native": 4363,
+ "hers": 4364,
+ "occurs": 4365,
+ "negative": 4366,
+ "Philip": 4367,
+ "Kelly": 4368,
+ "Colorado": 4369,
+ "category": 4370,
+ "##lan": 4371,
+ "600": 4372,
+ "Have": 4373,
+ "supporting": 4374,
+ "wet": 4375,
+ "56": 4376,
+ "stairs": 4377,
+ "Grace": 4378,
+ "observed": 4379,
+ "##ung": 4380,
+ "funds": 4381,
+ "restaurant": 4382,
+ "1911": 4383,
+ "Jews": 4384,
+ "##ments": 4385,
+ "##che": 4386,
+ "Jake": 4387,
+ "Back": 4388,
+ "53": 4389,
+ "asks": 4390,
+ "journalist": 4391,
+ "accept": 4392,
+ "bands": 4393,
+ "bronze": 4394,
+ "helping": 4395,
+ "##ice": 4396,
+ "decades": 4397,
+ "mayor": 4398,
+ "survived": 4399,
+ "usual": 4400,
+ "influenced": 4401,
+ "Douglas": 4402,
+ "Hey": 4403,
+ "##izing": 4404,
+ "surrounded": 4405,
+ "retirement": 4406,
+ "Temple": 4407,
+ "derived": 4408,
+ "Pope": 4409,
+ "registered": 4410,
+ "producing": 4411,
+ "##ral": 4412,
+ "structures": 4413,
+ "Johnny": 4414,
+ "contributed": 4415,
+ "finishing": 4416,
+ "buy": 4417,
+ "specifically": 4418,
+ "##king": 4419,
+ "patients": 4420,
+ "Jordan": 4421,
+ "internal": 4422,
+ "regarding": 4423,
+ "Samuel": 4424,
+ "Clark": 4425,
+ "##q": 4426,
+ "afternoon": 4427,
+ "Finally": 4428,
+ "scenes": 4429,
+ "notice": 4430,
+ "refers": 4431,
+ "quietly": 4432,
+ "threat": 4433,
+ "Water": 4434,
+ "Those": 4435,
+ "Hamilton": 4436,
+ "promise": 4437,
+ "freedom": 4438,
+ "Turkey": 4439,
+ "breaking": 4440,
+ "maintained": 4441,
+ "device": 4442,
+ "lap": 4443,
+ "ultimately": 4444,
+ "Champion": 4445,
+ "Tim": 4446,
+ "Bureau": 4447,
+ "expressed": 4448,
+ "investigation": 4449,
+ "extremely": 4450,
+ "capable": 4451,
+ "qualified": 4452,
+ "recognition": 4453,
+ "items": 4454,
+ "##up": 4455,
+ "Indiana": 4456,
+ "adult": 4457,
+ "rain": 4458,
+ "greatest": 4459,
+ "architect": 4460,
+ "Morgan": 4461,
+ "dressed": 4462,
+ "equal": 4463,
+ "Antonio": 4464,
+ "collected": 4465,
+ "drove": 4466,
+ "occur": 4467,
+ "Grant": 4468,
+ "graduate": 4469,
+ "anger": 4470,
+ "Sri": 4471,
+ "worried": 4472,
+ "standards": 4473,
+ "##ore": 4474,
+ "injured": 4475,
+ "somewhere": 4476,
+ "damn": 4477,
+ "Singapore": 4478,
+ "Jimmy": 4479,
+ "pocket": 4480,
+ "homes": 4481,
+ "stock": 4482,
+ "religion": 4483,
+ "aware": 4484,
+ "regarded": 4485,
+ "Wisconsin": 4486,
+ "##tra": 4487,
+ "passes": 4488,
+ "fresh": 4489,
+ "##ea": 4490,
+ "argued": 4491,
+ "Ltd": 4492,
+ "EP": 4493,
+ "Diego": 4494,
+ "importance": 4495,
+ "Census": 4496,
+ "incident": 4497,
+ "Egypt": 4498,
+ "Missouri": 4499,
+ "domestic": 4500,
+ "leads": 4501,
+ "ceremony": 4502,
+ "Early": 4503,
+ "camera": 4504,
+ "Father": 4505,
+ "challenge": 4506,
+ "Switzerland": 4507,
+ "lands": 4508,
+ "familiar": 4509,
+ "hearing": 4510,
+ "spend": 4511,
+ "educated": 4512,
+ "Tennessee": 4513,
+ "Thank": 4514,
+ "##ram": 4515,
+ "Thus": 4516,
+ "concern": 4517,
+ "putting": 4518,
+ "inches": 4519,
+ "map": 4520,
+ "classical": 4521,
+ "Allen": 4522,
+ "crazy": 4523,
+ "valley": 4524,
+ "Space": 4525,
+ "softly": 4526,
+ "##my": 4527,
+ "pool": 4528,
+ "worldwide": 4529,
+ "climate": 4530,
+ "experienced": 4531,
+ "neighborhood": 4532,
+ "scheduled": 4533,
+ "neither": 4534,
+ "fleet": 4535,
+ "1908": 4536,
+ "Girl": 4537,
+ "##J": 4538,
+ "Part": 4539,
+ "engines": 4540,
+ "locations": 4541,
+ "darkness": 4542,
+ "Revolution": 4543,
+ "establishment": 4544,
+ "lawyer": 4545,
+ "objects": 4546,
+ "apparently": 4547,
+ "Queensland": 4548,
+ "Entertainment": 4549,
+ "bill": 4550,
+ "mark": 4551,
+ "Television": 4552,
+ "##ong": 4553,
+ "pale": 4554,
+ "demand": 4555,
+ "Hotel": 4556,
+ "selection": 4557,
+ "##rn": 4558,
+ "##ino": 4559,
+ "Labour": 4560,
+ "Liberal": 4561,
+ "burned": 4562,
+ "Mom": 4563,
+ "merged": 4564,
+ "Arizona": 4565,
+ "request": 4566,
+ "##lia": 4567,
+ "##light": 4568,
+ "hole": 4569,
+ "employees": 4570,
+ "##ical": 4571,
+ "incorporated": 4572,
+ "95": 4573,
+ "independence": 4574,
+ "Walker": 4575,
+ "covering": 4576,
+ "joining": 4577,
+ "##ica": 4578,
+ "task": 4579,
+ "papers": 4580,
+ "backing": 4581,
+ "sell": 4582,
+ "biggest": 4583,
+ "6th": 4584,
+ "strike": 4585,
+ "establish": 4586,
+ "##ō": 4587,
+ "gently": 4588,
+ "59": 4589,
+ "Orchestra": 4590,
+ "Winter": 4591,
+ "protein": 4592,
+ "Juan": 4593,
+ "locked": 4594,
+ "dates": 4595,
+ "Boy": 4596,
+ "aren": 4597,
+ "shooting": 4598,
+ "Luke": 4599,
+ "solid": 4600,
+ "charged": 4601,
+ "Prior": 4602,
+ "resigned": 4603,
+ "interior": 4604,
+ "garden": 4605,
+ "spoken": 4606,
+ "improve": 4607,
+ "wonder": 4608,
+ "promote": 4609,
+ "hidden": 4610,
+ "##med": 4611,
+ "combination": 4612,
+ "Hollywood": 4613,
+ "Swiss": 4614,
+ "consider": 4615,
+ "##ks": 4616,
+ "Lincoln": 4617,
+ "literary": 4618,
+ "drawing": 4619,
+ "Marine": 4620,
+ "weapon": 4621,
+ "Victor": 4622,
+ "Trust": 4623,
+ "Maryland": 4624,
+ "properties": 4625,
+ "##ara": 4626,
+ "exhibition": 4627,
+ "understood": 4628,
+ "hung": 4629,
+ "Tell": 4630,
+ "installed": 4631,
+ "loud": 4632,
+ "fashion": 4633,
+ "affected": 4634,
+ "junior": 4635,
+ "landing": 4636,
+ "flowers": 4637,
+ "##he": 4638,
+ "Internet": 4639,
+ "beach": 4640,
+ "Heart": 4641,
+ "tries": 4642,
+ "Mayor": 4643,
+ "programme": 4644,
+ "800": 4645,
+ "wins": 4646,
+ "noise": 4647,
+ "##ster": 4648,
+ "##ory": 4649,
+ "58": 4650,
+ "contain": 4651,
+ "fair": 4652,
+ "delivered": 4653,
+ "##ul": 4654,
+ "wedding": 4655,
+ "Square": 4656,
+ "advance": 4657,
+ "behavior": 4658,
+ "Program": 4659,
+ "Oregon": 4660,
+ "##rk": 4661,
+ "residence": 4662,
+ "realize": 4663,
+ "certainly": 4664,
+ "hill": 4665,
+ "Houston": 4666,
+ "57": 4667,
+ "indicated": 4668,
+ "##water": 4669,
+ "wounded": 4670,
+ "Village": 4671,
+ "massive": 4672,
+ "Moore": 4673,
+ "thousands": 4674,
+ "personnel": 4675,
+ "dating": 4676,
+ "opera": 4677,
+ "poetry": 4678,
+ "##her": 4679,
+ "causes": 4680,
+ "feelings": 4681,
+ "Frederick": 4682,
+ "applications": 4683,
+ "push": 4684,
+ "approached": 4685,
+ "foundation": 4686,
+ "pleasure": 4687,
+ "sale": 4688,
+ "fly": 4689,
+ "gotten": 4690,
+ "northeast": 4691,
+ "costs": 4692,
+ "raise": 4693,
+ "paintings": 4694,
+ "##ney": 4695,
+ "views": 4696,
+ "horses": 4697,
+ "formal": 4698,
+ "Arab": 4699,
+ "hockey": 4700,
+ "typical": 4701,
+ "representative": 4702,
+ "rising": 4703,
+ "##des": 4704,
+ "clock": 4705,
+ "stadium": 4706,
+ "shifted": 4707,
+ "Dad": 4708,
+ "peak": 4709,
+ "Fame": 4710,
+ "vice": 4711,
+ "disappeared": 4712,
+ "users": 4713,
+ "Way": 4714,
+ "Naval": 4715,
+ "prize": 4716,
+ "hoping": 4717,
+ "values": 4718,
+ "evil": 4719,
+ "Bell": 4720,
+ "consisting": 4721,
+ "##ón": 4722,
+ "Regional": 4723,
+ "##ics": 4724,
+ "improved": 4725,
+ "circle": 4726,
+ "carefully": 4727,
+ "broad": 4728,
+ "##ini": 4729,
+ "Fine": 4730,
+ "maintain": 4731,
+ "operate": 4732,
+ "offering": 4733,
+ "mention": 4734,
+ "Death": 4735,
+ "stupid": 4736,
+ "Through": 4737,
+ "Princess": 4738,
+ "attend": 4739,
+ "interests": 4740,
+ "ruled": 4741,
+ "somewhat": 4742,
+ "wings": 4743,
+ "roads": 4744,
+ "grounds": 4745,
+ "##ual": 4746,
+ "Greece": 4747,
+ "Champions": 4748,
+ "facing": 4749,
+ "hide": 4750,
+ "voted": 4751,
+ "require": 4752,
+ "Dark": 4753,
+ "Matthew": 4754,
+ "credit": 4755,
+ "sighed": 4756,
+ "separated": 4757,
+ "manner": 4758,
+ "##ile": 4759,
+ "Boys": 4760,
+ "1905": 4761,
+ "committed": 4762,
+ "impossible": 4763,
+ "lip": 4764,
+ "candidates": 4765,
+ "7th": 4766,
+ "Bruce": 4767,
+ "arranged": 4768,
+ "Islamic": 4769,
+ "courses": 4770,
+ "criminal": 4771,
+ "##ened": 4772,
+ "smell": 4773,
+ "##bed": 4774,
+ "08": 4775,
+ "consecutive": 4776,
+ "##ening": 4777,
+ "proper": 4778,
+ "purchase": 4779,
+ "weak": 4780,
+ "Prix": 4781,
+ "1906": 4782,
+ "aside": 4783,
+ "introduction": 4784,
+ "Look": 4785,
+ "##ku": 4786,
+ "changing": 4787,
+ "budget": 4788,
+ "resistance": 4789,
+ "factory": 4790,
+ "Forces": 4791,
+ "agency": 4792,
+ "##tone": 4793,
+ "northwest": 4794,
+ "user": 4795,
+ "1907": 4796,
+ "stating": 4797,
+ "##one": 4798,
+ "sport": 4799,
+ "Design": 4800,
+ "environmental": 4801,
+ "cards": 4802,
+ "concluded": 4803,
+ "Carl": 4804,
+ "250": 4805,
+ "accused": 4806,
+ "##ology": 4807,
+ "Girls": 4808,
+ "sick": 4809,
+ "intelligence": 4810,
+ "Margaret": 4811,
+ "responsibility": 4812,
+ "Guard": 4813,
+ "##tus": 4814,
+ "17th": 4815,
+ "sq": 4816,
+ "goods": 4817,
+ "1909": 4818,
+ "hate": 4819,
+ "##ek": 4820,
+ "capture": 4821,
+ "stores": 4822,
+ "Gray": 4823,
+ "comic": 4824,
+ "Modern": 4825,
+ "Silver": 4826,
+ "Andy": 4827,
+ "electronic": 4828,
+ "wheel": 4829,
+ "##ied": 4830,
+ "Deputy": 4831,
+ "##bs": 4832,
+ "Czech": 4833,
+ "zone": 4834,
+ "choose": 4835,
+ "constant": 4836,
+ "reserve": 4837,
+ "##lle": 4838,
+ "Tokyo": 4839,
+ "spirit": 4840,
+ "sub": 4841,
+ "degrees": 4842,
+ "flew": 4843,
+ "pattern": 4844,
+ "compete": 4845,
+ "Dance": 4846,
+ "##ik": 4847,
+ "secretary": 4848,
+ "Imperial": 4849,
+ "99": 4850,
+ "reduce": 4851,
+ "Hungarian": 4852,
+ "confused": 4853,
+ "##rin": 4854,
+ "Pierre": 4855,
+ "describes": 4856,
+ "regularly": 4857,
+ "Rachel": 4858,
+ "85": 4859,
+ "landed": 4860,
+ "passengers": 4861,
+ "##ise": 4862,
+ "##sis": 4863,
+ "historian": 4864,
+ "meters": 4865,
+ "Youth": 4866,
+ "##ud": 4867,
+ "participate": 4868,
+ "##cing": 4869,
+ "arrival": 4870,
+ "tired": 4871,
+ "Mother": 4872,
+ "##gy": 4873,
+ "jumped": 4874,
+ "Kentucky": 4875,
+ "faces": 4876,
+ "feed": 4877,
+ "Israeli": 4878,
+ "Ocean": 4879,
+ "##Q": 4880,
+ "##án": 4881,
+ "plus": 4882,
+ "snow": 4883,
+ "techniques": 4884,
+ "plate": 4885,
+ "sections": 4886,
+ "falls": 4887,
+ "jazz": 4888,
+ "##ris": 4889,
+ "tank": 4890,
+ "loan": 4891,
+ "repeated": 4892,
+ "opinion": 4893,
+ "##res": 4894,
+ "unless": 4895,
+ "rugby": 4896,
+ "journal": 4897,
+ "Lawrence": 4898,
+ "moments": 4899,
+ "shock": 4900,
+ "distributed": 4901,
+ "##ded": 4902,
+ "adjacent": 4903,
+ "Argentina": 4904,
+ "crossing": 4905,
+ "uncle": 4906,
+ "##ric": 4907,
+ "Detroit": 4908,
+ "communication": 4909,
+ "mental": 4910,
+ "tomorrow": 4911,
+ "session": 4912,
+ "Emma": 4913,
+ "Without": 4914,
+ "##gen": 4915,
+ "Miami": 4916,
+ "charges": 4917,
+ "Administration": 4918,
+ "hits": 4919,
+ "coat": 4920,
+ "protected": 4921,
+ "Cole": 4922,
+ "invasion": 4923,
+ "priest": 4924,
+ "09": 4925,
+ "Gary": 4926,
+ "enjoyed": 4927,
+ "plot": 4928,
+ "measure": 4929,
+ "bound": 4930,
+ "friendly": 4931,
+ "throw": 4932,
+ "musician": 4933,
+ "##lon": 4934,
+ "##ins": 4935,
+ "Age": 4936,
+ "knife": 4937,
+ "damaged": 4938,
+ "birds": 4939,
+ "driven": 4940,
+ "lit": 4941,
+ "ears": 4942,
+ "breathing": 4943,
+ "Arabic": 4944,
+ "Jan": 4945,
+ "faster": 4946,
+ "Jonathan": 4947,
+ "##gate": 4948,
+ "Independent": 4949,
+ "starred": 4950,
+ "Harris": 4951,
+ "teachers": 4952,
+ "Alice": 4953,
+ "sequence": 4954,
+ "mph": 4955,
+ "file": 4956,
+ "translated": 4957,
+ "decide": 4958,
+ "determine": 4959,
+ "Review": 4960,
+ "documents": 4961,
+ "sudden": 4962,
+ "threatened": 4963,
+ "##ft": 4964,
+ "bear": 4965,
+ "distinct": 4966,
+ "decade": 4967,
+ "burning": 4968,
+ "##sky": 4969,
+ "1930s": 4970,
+ "replace": 4971,
+ "begun": 4972,
+ "extension": 4973,
+ "##time": 4974,
+ "1904": 4975,
+ "equivalent": 4976,
+ "accompanied": 4977,
+ "Christopher": 4978,
+ "Danish": 4979,
+ "##ye": 4980,
+ "Besides": 4981,
+ "##more": 4982,
+ "persons": 4983,
+ "fallen": 4984,
+ "Rural": 4985,
+ "roughly": 4986,
+ "saved": 4987,
+ "willing": 4988,
+ "ensure": 4989,
+ "Belgium": 4990,
+ "05": 4991,
+ "musicians": 4992,
+ "##ang": 4993,
+ "giant": 4994,
+ "Six": 4995,
+ "Retrieved": 4996,
+ "worst": 4997,
+ "purposes": 4998,
+ "##bly": 4999,
+ "mountains": 5000,
+ "seventh": 5001,
+ "slipped": 5002,
+ "brick": 5003,
+ "07": 5004,
+ "##py": 5005,
+ "somehow": 5006,
+ "Carter": 5007,
+ "Iraq": 5008,
+ "cousin": 5009,
+ "favor": 5010,
+ "islands": 5011,
+ "journey": 5012,
+ "FIFA": 5013,
+ "contrast": 5014,
+ "planet": 5015,
+ "vs": 5016,
+ "calm": 5017,
+ "##ings": 5018,
+ "concrete": 5019,
+ "branches": 5020,
+ "gray": 5021,
+ "profit": 5022,
+ "Russell": 5023,
+ "##ae": 5024,
+ "##ux": 5025,
+ "##ens": 5026,
+ "philosophy": 5027,
+ "businesses": 5028,
+ "talked": 5029,
+ "parking": 5030,
+ "##ming": 5031,
+ "owners": 5032,
+ "Place": 5033,
+ "##tle": 5034,
+ "agricultural": 5035,
+ "Kate": 5036,
+ "06": 5037,
+ "southeast": 5038,
+ "draft": 5039,
+ "Eddie": 5040,
+ "earliest": 5041,
+ "forget": 5042,
+ "Dallas": 5043,
+ "Commonwealth": 5044,
+ "edited": 5045,
+ "66": 5046,
+ "inner": 5047,
+ "ed": 5048,
+ "operates": 5049,
+ "16th": 5050,
+ "Harvard": 5051,
+ "assistance": 5052,
+ "##si": 5053,
+ "designs": 5054,
+ "Take": 5055,
+ "bathroom": 5056,
+ "indicate": 5057,
+ "CEO": 5058,
+ "Command": 5059,
+ "Louisiana": 5060,
+ "1902": 5061,
+ "Dublin": 5062,
+ "Books": 5063,
+ "1901": 5064,
+ "tropical": 5065,
+ "1903": 5066,
+ "##tors": 5067,
+ "Places": 5068,
+ "tie": 5069,
+ "progress": 5070,
+ "forming": 5071,
+ "solution": 5072,
+ "62": 5073,
+ "letting": 5074,
+ "##ery": 5075,
+ "studying": 5076,
+ "##jo": 5077,
+ "duties": 5078,
+ "Baseball": 5079,
+ "taste": 5080,
+ "Reserve": 5081,
+ "##ru": 5082,
+ "Ann": 5083,
+ "##gh": 5084,
+ "visible": 5085,
+ "##vi": 5086,
+ "notably": 5087,
+ "link": 5088,
+ "NCAA": 5089,
+ "southwest": 5090,
+ "Never": 5091,
+ "storage": 5092,
+ "mobile": 5093,
+ "writers": 5094,
+ "favorite": 5095,
+ "Pro": 5096,
+ "pages": 5097,
+ "truly": 5098,
+ "count": 5099,
+ "##tta": 5100,
+ "string": 5101,
+ "kid": 5102,
+ "98": 5103,
+ "Ross": 5104,
+ "row": 5105,
+ "##idae": 5106,
+ "Kennedy": 5107,
+ "##tan": 5108,
+ "Hockey": 5109,
+ "hip": 5110,
+ "waist": 5111,
+ "grandfather": 5112,
+ "listen": 5113,
+ "##ho": 5114,
+ "feels": 5115,
+ "busy": 5116,
+ "72": 5117,
+ "stream": 5118,
+ "obvious": 5119,
+ "cycle": 5120,
+ "shaking": 5121,
+ "Knight": 5122,
+ "##ren": 5123,
+ "Carlos": 5124,
+ "painter": 5125,
+ "trail": 5126,
+ "web": 5127,
+ "linked": 5128,
+ "04": 5129,
+ "Palace": 5130,
+ "existed": 5131,
+ "##ira": 5132,
+ "responded": 5133,
+ "closing": 5134,
+ "End": 5135,
+ "examples": 5136,
+ "Marshall": 5137,
+ "weekend": 5138,
+ "jaw": 5139,
+ "Denmark": 5140,
+ "lady": 5141,
+ "township": 5142,
+ "medium": 5143,
+ "chin": 5144,
+ "Story": 5145,
+ "option": 5146,
+ "fifteen": 5147,
+ "Moon": 5148,
+ "represents": 5149,
+ "makeup": 5150,
+ "investment": 5151,
+ "jump": 5152,
+ "childhood": 5153,
+ "Oklahoma": 5154,
+ "roll": 5155,
+ "normally": 5156,
+ "Ten": 5157,
+ "Operation": 5158,
+ "Graham": 5159,
+ "Seattle": 5160,
+ "Atlanta": 5161,
+ "paused": 5162,
+ "promised": 5163,
+ "rejected": 5164,
+ "treated": 5165,
+ "returns": 5166,
+ "flag": 5167,
+ "##ita": 5168,
+ "Hungary": 5169,
+ "danger": 5170,
+ "glad": 5171,
+ "movements": 5172,
+ "visual": 5173,
+ "subjects": 5174,
+ "credited": 5175,
+ "soldier": 5176,
+ "Norman": 5177,
+ "ill": 5178,
+ "translation": 5179,
+ "José": 5180,
+ "Quebec": 5181,
+ "medicine": 5182,
+ "warning": 5183,
+ "theater": 5184,
+ "praised": 5185,
+ "municipal": 5186,
+ "01": 5187,
+ "commune": 5188,
+ "churches": 5189,
+ "acid": 5190,
+ "folk": 5191,
+ "8th": 5192,
+ "testing": 5193,
+ "add": 5194,
+ "survive": 5195,
+ "Sound": 5196,
+ "devices": 5197,
+ "residential": 5198,
+ "severe": 5199,
+ "presidential": 5200,
+ "Mississippi": 5201,
+ "Austin": 5202,
+ "Perhaps": 5203,
+ "Charlotte": 5204,
+ "hanging": 5205,
+ "Montreal": 5206,
+ "grin": 5207,
+ "##ten": 5208,
+ "racial": 5209,
+ "partnership": 5210,
+ "shoot": 5211,
+ "shift": 5212,
+ "##nie": 5213,
+ "Les": 5214,
+ "downtown": 5215,
+ "Brothers": 5216,
+ "Garden": 5217,
+ "matters": 5218,
+ "restored": 5219,
+ "mirror": 5220,
+ "forever": 5221,
+ "winners": 5222,
+ "rapidly": 5223,
+ "poverty": 5224,
+ "##ible": 5225,
+ "Until": 5226,
+ "DC": 5227,
+ "faith": 5228,
+ "hundreds": 5229,
+ "Real": 5230,
+ "Ukraine": 5231,
+ "Nelson": 5232,
+ "balance": 5233,
+ "Adams": 5234,
+ "contest": 5235,
+ "relative": 5236,
+ "ethnic": 5237,
+ "Edinburgh": 5238,
+ "composition": 5239,
+ "##nts": 5240,
+ "emergency": 5241,
+ "##van": 5242,
+ "marine": 5243,
+ "reputation": 5244,
+ "Down": 5245,
+ "pack": 5246,
+ "12th": 5247,
+ "Communist": 5248,
+ "Mountains": 5249,
+ "pro": 5250,
+ "stages": 5251,
+ "measures": 5252,
+ "##ld": 5253,
+ "ABC": 5254,
+ "Li": 5255,
+ "victims": 5256,
+ "benefit": 5257,
+ "Iowa": 5258,
+ "Broadway": 5259,
+ "gathered": 5260,
+ "rating": 5261,
+ "Defense": 5262,
+ "classic": 5263,
+ "##ily": 5264,
+ "ceiling": 5265,
+ "##ions": 5266,
+ "snapped": 5267,
+ "Everything": 5268,
+ "constituency": 5269,
+ "Franklin": 5270,
+ "Thompson": 5271,
+ "Stewart": 5272,
+ "entering": 5273,
+ "Judge": 5274,
+ "forth": 5275,
+ "##sk": 5276,
+ "wanting": 5277,
+ "smiling": 5278,
+ "moves": 5279,
+ "tunnel": 5280,
+ "premiered": 5281,
+ "grass": 5282,
+ "unusual": 5283,
+ "Ukrainian": 5284,
+ "bird": 5285,
+ "Friday": 5286,
+ "tail": 5287,
+ "Portugal": 5288,
+ "coal": 5289,
+ "element": 5290,
+ "Fred": 5291,
+ "guards": 5292,
+ "Senator": 5293,
+ "collaboration": 5294,
+ "beauty": 5295,
+ "Wood": 5296,
+ "chemical": 5297,
+ "beer": 5298,
+ "justice": 5299,
+ "signs": 5300,
+ "##Z": 5301,
+ "sees": 5302,
+ "##zi": 5303,
+ "Puerto": 5304,
+ "##zed": 5305,
+ "96": 5306,
+ "smooth": 5307,
+ "Bowl": 5308,
+ "gift": 5309,
+ "limit": 5310,
+ "97": 5311,
+ "heading": 5312,
+ "Source": 5313,
+ "wake": 5314,
+ "requires": 5315,
+ "Ed": 5316,
+ "Constitution": 5317,
+ "factor": 5318,
+ "Lane": 5319,
+ "factors": 5320,
+ "adding": 5321,
+ "Note": 5322,
+ "cleared": 5323,
+ "pictures": 5324,
+ "pink": 5325,
+ "##ola": 5326,
+ "Kent": 5327,
+ "Local": 5328,
+ "Singh": 5329,
+ "moth": 5330,
+ "Ty": 5331,
+ "##ture": 5332,
+ "courts": 5333,
+ "Seven": 5334,
+ "temporary": 5335,
+ "involving": 5336,
+ "Vienna": 5337,
+ "emerged": 5338,
+ "fishing": 5339,
+ "agree": 5340,
+ "defensive": 5341,
+ "stuck": 5342,
+ "secure": 5343,
+ "Tamil": 5344,
+ "##ick": 5345,
+ "bottle": 5346,
+ "03": 5347,
+ "Player": 5348,
+ "instruments": 5349,
+ "Spring": 5350,
+ "patient": 5351,
+ "flesh": 5352,
+ "contributions": 5353,
+ "cry": 5354,
+ "Malaysia": 5355,
+ "120": 5356,
+ "Global": 5357,
+ "da": 5358,
+ "Alabama": 5359,
+ "Within": 5360,
+ "##work": 5361,
+ "debuted": 5362,
+ "expect": 5363,
+ "Cleveland": 5364,
+ "concerns": 5365,
+ "retained": 5366,
+ "horror": 5367,
+ "10th": 5368,
+ "spending": 5369,
+ "Peace": 5370,
+ "Transport": 5371,
+ "grand": 5372,
+ "Crown": 5373,
+ "instance": 5374,
+ "institution": 5375,
+ "acted": 5376,
+ "Hills": 5377,
+ "mounted": 5378,
+ "Campbell": 5379,
+ "shouldn": 5380,
+ "1898": 5381,
+ "##ably": 5382,
+ "chamber": 5383,
+ "soil": 5384,
+ "88": 5385,
+ "Ethan": 5386,
+ "sand": 5387,
+ "cheeks": 5388,
+ "##gi": 5389,
+ "marry": 5390,
+ "61": 5391,
+ "weekly": 5392,
+ "classification": 5393,
+ "DNA": 5394,
+ "Elementary": 5395,
+ "Roy": 5396,
+ "definitely": 5397,
+ "Soon": 5398,
+ "Rights": 5399,
+ "gate": 5400,
+ "suggests": 5401,
+ "aspects": 5402,
+ "imagine": 5403,
+ "golden": 5404,
+ "beating": 5405,
+ "Studios": 5406,
+ "Warren": 5407,
+ "differences": 5408,
+ "significantly": 5409,
+ "glance": 5410,
+ "occasionally": 5411,
+ "##od": 5412,
+ "clothing": 5413,
+ "Assistant": 5414,
+ "depth": 5415,
+ "sending": 5416,
+ "possibility": 5417,
+ "mode": 5418,
+ "prisoners": 5419,
+ "requirements": 5420,
+ "daughters": 5421,
+ "dated": 5422,
+ "Representatives": 5423,
+ "prove": 5424,
+ "guilty": 5425,
+ "interesting": 5426,
+ "smoke": 5427,
+ "cricket": 5428,
+ "93": 5429,
+ "##ates": 5430,
+ "rescue": 5431,
+ "Connecticut": 5432,
+ "underground": 5433,
+ "Opera": 5434,
+ "13th": 5435,
+ "reign": 5436,
+ "##ski": 5437,
+ "thanks": 5438,
+ "leather": 5439,
+ "equipped": 5440,
+ "routes": 5441,
+ "fan": 5442,
+ "##ans": 5443,
+ "script": 5444,
+ "Wright": 5445,
+ "bishop": 5446,
+ "Welsh": 5447,
+ "jobs": 5448,
+ "faculty": 5449,
+ "eleven": 5450,
+ "Railroad": 5451,
+ "appearing": 5452,
+ "anniversary": 5453,
+ "Upper": 5454,
+ "##down": 5455,
+ "anywhere": 5456,
+ "Rugby": 5457,
+ "Metropolitan": 5458,
+ "Meanwhile": 5459,
+ "Nicholas": 5460,
+ "champions": 5461,
+ "forehead": 5462,
+ "mining": 5463,
+ "drinking": 5464,
+ "76": 5465,
+ "Jerry": 5466,
+ "membership": 5467,
+ "Brazilian": 5468,
+ "Wild": 5469,
+ "Rio": 5470,
+ "scheme": 5471,
+ "Unlike": 5472,
+ "strongly": 5473,
+ "##bility": 5474,
+ "fill": 5475,
+ "##rian": 5476,
+ "easier": 5477,
+ "MP": 5478,
+ "Hell": 5479,
+ "##sha": 5480,
+ "Stanley": 5481,
+ "banks": 5482,
+ "Baron": 5483,
+ "##ique": 5484,
+ "Robinson": 5485,
+ "67": 5486,
+ "Gabriel": 5487,
+ "Austrian": 5488,
+ "Wayne": 5489,
+ "exposed": 5490,
+ "##wan": 5491,
+ "Alfred": 5492,
+ "1899": 5493,
+ "manage": 5494,
+ "mix": 5495,
+ "visitors": 5496,
+ "eating": 5497,
+ "##rate": 5498,
+ "Sean": 5499,
+ "commission": 5500,
+ "Cemetery": 5501,
+ "policies": 5502,
+ "Camp": 5503,
+ "parallel": 5504,
+ "traveled": 5505,
+ "guitarist": 5506,
+ "02": 5507,
+ "supplies": 5508,
+ "couples": 5509,
+ "poem": 5510,
+ "blocks": 5511,
+ "Rick": 5512,
+ "Training": 5513,
+ "Energy": 5514,
+ "achieve": 5515,
+ "appointment": 5516,
+ "Wing": 5517,
+ "Jamie": 5518,
+ "63": 5519,
+ "novels": 5520,
+ "##em": 5521,
+ "1890": 5522,
+ "songwriter": 5523,
+ "Base": 5524,
+ "Jay": 5525,
+ "##gar": 5526,
+ "naval": 5527,
+ "scared": 5528,
+ "miss": 5529,
+ "labor": 5530,
+ "technique": 5531,
+ "crisis": 5532,
+ "Additionally": 5533,
+ "backed": 5534,
+ "destroy": 5535,
+ "seriously": 5536,
+ "tools": 5537,
+ "tennis": 5538,
+ "91": 5539,
+ "god": 5540,
+ "##ington": 5541,
+ "continuing": 5542,
+ "steam": 5543,
+ "obviously": 5544,
+ "Bobby": 5545,
+ "adapted": 5546,
+ "fifty": 5547,
+ "enjoy": 5548,
+ "Jacob": 5549,
+ "publishing": 5550,
+ "column": 5551,
+ "##ular": 5552,
+ "Baltimore": 5553,
+ "Donald": 5554,
+ "Liverpool": 5555,
+ "92": 5556,
+ "drugs": 5557,
+ "movies": 5558,
+ "##ock": 5559,
+ "Heritage": 5560,
+ "##je": 5561,
+ "##istic": 5562,
+ "vocal": 5563,
+ "strategy": 5564,
+ "gene": 5565,
+ "advice": 5566,
+ "##bi": 5567,
+ "Ottoman": 5568,
+ "riding": 5569,
+ "##side": 5570,
+ "Agency": 5571,
+ "Indonesia": 5572,
+ "11th": 5573,
+ "laughing": 5574,
+ "sleeping": 5575,
+ "und": 5576,
+ "muttered": 5577,
+ "listening": 5578,
+ "deck": 5579,
+ "tip": 5580,
+ "77": 5581,
+ "ownership": 5582,
+ "grey": 5583,
+ "Claire": 5584,
+ "deeply": 5585,
+ "provincial": 5586,
+ "popularity": 5587,
+ "Cooper": 5588,
+ "##á": 5589,
+ "Emily": 5590,
+ "##sed": 5591,
+ "designer": 5592,
+ "Murray": 5593,
+ "describe": 5594,
+ "Danny": 5595,
+ "Around": 5596,
+ "Parker": 5597,
+ "##dae": 5598,
+ "68": 5599,
+ "rates": 5600,
+ "suffering": 5601,
+ "considerable": 5602,
+ "78": 5603,
+ "nervous": 5604,
+ "powered": 5605,
+ "tons": 5606,
+ "circumstances": 5607,
+ "wished": 5608,
+ "belonged": 5609,
+ "Pittsburgh": 5610,
+ "flows": 5611,
+ "9th": 5612,
+ "##use": 5613,
+ "belt": 5614,
+ "81": 5615,
+ "useful": 5616,
+ "15th": 5617,
+ "context": 5618,
+ "List": 5619,
+ "Dead": 5620,
+ "Iron": 5621,
+ "seek": 5622,
+ "Season": 5623,
+ "worn": 5624,
+ "frequency": 5625,
+ "legislation": 5626,
+ "replacement": 5627,
+ "memories": 5628,
+ "Tournament": 5629,
+ "Again": 5630,
+ "Barry": 5631,
+ "organisation": 5632,
+ "copy": 5633,
+ "Gulf": 5634,
+ "waters": 5635,
+ "meets": 5636,
+ "struggle": 5637,
+ "Oliver": 5638,
+ "1895": 5639,
+ "Susan": 5640,
+ "protest": 5641,
+ "kick": 5642,
+ "Alliance": 5643,
+ "components": 5644,
+ "1896": 5645,
+ "Tower": 5646,
+ "Windows": 5647,
+ "demanded": 5648,
+ "regiment": 5649,
+ "sentence": 5650,
+ "Woman": 5651,
+ "Logan": 5652,
+ "Referee": 5653,
+ "hosts": 5654,
+ "debate": 5655,
+ "knee": 5656,
+ "Blood": 5657,
+ "##oo": 5658,
+ "universities": 5659,
+ "practices": 5660,
+ "Ward": 5661,
+ "ranking": 5662,
+ "correct": 5663,
+ "happening": 5664,
+ "Vincent": 5665,
+ "attracted": 5666,
+ "classified": 5667,
+ "##stic": 5668,
+ "processes": 5669,
+ "immediate": 5670,
+ "waste": 5671,
+ "increasingly": 5672,
+ "Helen": 5673,
+ "##po": 5674,
+ "Lucas": 5675,
+ "Phil": 5676,
+ "organ": 5677,
+ "1897": 5678,
+ "tea": 5679,
+ "suicide": 5680,
+ "actors": 5681,
+ "lb": 5682,
+ "crash": 5683,
+ "approval": 5684,
+ "waves": 5685,
+ "##ered": 5686,
+ "hated": 5687,
+ "grip": 5688,
+ "700": 5689,
+ "amongst": 5690,
+ "69": 5691,
+ "74": 5692,
+ "hunting": 5693,
+ "dying": 5694,
+ "lasted": 5695,
+ "illegal": 5696,
+ "##rum": 5697,
+ "stare": 5698,
+ "defeating": 5699,
+ "##gs": 5700,
+ "shrugged": 5701,
+ "°C": 5702,
+ "Jon": 5703,
+ "Count": 5704,
+ "Orleans": 5705,
+ "94": 5706,
+ "affairs": 5707,
+ "formally": 5708,
+ "##and": 5709,
+ "##ves": 5710,
+ "criticized": 5711,
+ "Disney": 5712,
+ "Vol": 5713,
+ "successor": 5714,
+ "tests": 5715,
+ "scholars": 5716,
+ "palace": 5717,
+ "Would": 5718,
+ "celebrated": 5719,
+ "rounds": 5720,
+ "grant": 5721,
+ "Schools": 5722,
+ "Such": 5723,
+ "commanded": 5724,
+ "demon": 5725,
+ "Romania": 5726,
+ "##all": 5727,
+ "Karl": 5728,
+ "71": 5729,
+ "##yn": 5730,
+ "84": 5731,
+ "Daily": 5732,
+ "totally": 5733,
+ "Medicine": 5734,
+ "fruit": 5735,
+ "Die": 5736,
+ "upset": 5737,
+ "Lower": 5738,
+ "Conservative": 5739,
+ "14th": 5740,
+ "Mitchell": 5741,
+ "escaped": 5742,
+ "shoes": 5743,
+ "Morris": 5744,
+ "##tz": 5745,
+ "queen": 5746,
+ "harder": 5747,
+ "prime": 5748,
+ "Thanks": 5749,
+ "indeed": 5750,
+ "Sky": 5751,
+ "authors": 5752,
+ "rocks": 5753,
+ "definition": 5754,
+ "Nazi": 5755,
+ "accounts": 5756,
+ "printed": 5757,
+ "experiences": 5758,
+ "##ters": 5759,
+ "divisions": 5760,
+ "Cathedral": 5761,
+ "denied": 5762,
+ "depending": 5763,
+ "Express": 5764,
+ "##let": 5765,
+ "73": 5766,
+ "appeal": 5767,
+ "loose": 5768,
+ "colors": 5769,
+ "filed": 5770,
+ "##isation": 5771,
+ "gender": 5772,
+ "##ew": 5773,
+ "throne": 5774,
+ "forests": 5775,
+ "Finland": 5776,
+ "domain": 5777,
+ "boats": 5778,
+ "Baker": 5779,
+ "squadron": 5780,
+ "shore": 5781,
+ "remove": 5782,
+ "##ification": 5783,
+ "careful": 5784,
+ "wound": 5785,
+ "railroad": 5786,
+ "82": 5787,
+ "seeking": 5788,
+ "agents": 5789,
+ "##ved": 5790,
+ "Blues": 5791,
+ "##off": 5792,
+ "customers": 5793,
+ "ignored": 5794,
+ "net": 5795,
+ "##ction": 5796,
+ "hiding": 5797,
+ "Originally": 5798,
+ "declined": 5799,
+ "##ess": 5800,
+ "franchise": 5801,
+ "eliminated": 5802,
+ "NBA": 5803,
+ "merely": 5804,
+ "pure": 5805,
+ "appropriate": 5806,
+ "visiting": 5807,
+ "forty": 5808,
+ "markets": 5809,
+ "offensive": 5810,
+ "coverage": 5811,
+ "cave": 5812,
+ "##nia": 5813,
+ "spell": 5814,
+ "##lar": 5815,
+ "Benjamin": 5816,
+ "##ire": 5817,
+ "Convention": 5818,
+ "filmed": 5819,
+ "Trade": 5820,
+ "##sy": 5821,
+ "##ct": 5822,
+ "Having": 5823,
+ "palm": 5824,
+ "1889": 5825,
+ "Evans": 5826,
+ "intense": 5827,
+ "plastic": 5828,
+ "Julia": 5829,
+ "document": 5830,
+ "jeans": 5831,
+ "vessel": 5832,
+ "SR": 5833,
+ "##fully": 5834,
+ "proposal": 5835,
+ "Birmingham": 5836,
+ "le": 5837,
+ "##ative": 5838,
+ "assembly": 5839,
+ "89": 5840,
+ "fund": 5841,
+ "lock": 5842,
+ "1893": 5843,
+ "AD": 5844,
+ "meetings": 5845,
+ "occupation": 5846,
+ "modified": 5847,
+ "Years": 5848,
+ "odd": 5849,
+ "aimed": 5850,
+ "reform": 5851,
+ "Mission": 5852,
+ "Works": 5853,
+ "shake": 5854,
+ "cat": 5855,
+ "exception": 5856,
+ "convinced": 5857,
+ "executed": 5858,
+ "pushing": 5859,
+ "dollars": 5860,
+ "replacing": 5861,
+ "soccer": 5862,
+ "manufacturing": 5863,
+ "##ros": 5864,
+ "expensive": 5865,
+ "kicked": 5866,
+ "minimum": 5867,
+ "Josh": 5868,
+ "coastal": 5869,
+ "Chase": 5870,
+ "ha": 5871,
+ "Thailand": 5872,
+ "publications": 5873,
+ "deputy": 5874,
+ "Sometimes": 5875,
+ "Angel": 5876,
+ "effectively": 5877,
+ "##illa": 5878,
+ "criticism": 5879,
+ "conduct": 5880,
+ "Serbian": 5881,
+ "landscape": 5882,
+ "NY": 5883,
+ "absence": 5884,
+ "passage": 5885,
+ "##ula": 5886,
+ "Blake": 5887,
+ "Indians": 5888,
+ "1892": 5889,
+ "admit": 5890,
+ "Trophy": 5891,
+ "##ball": 5892,
+ "Next": 5893,
+ "##rated": 5894,
+ "##ians": 5895,
+ "charts": 5896,
+ "kW": 5897,
+ "orchestra": 5898,
+ "79": 5899,
+ "heritage": 5900,
+ "1894": 5901,
+ "rough": 5902,
+ "exists": 5903,
+ "boundary": 5904,
+ "Bible": 5905,
+ "Legislative": 5906,
+ "moon": 5907,
+ "medieval": 5908,
+ "##over": 5909,
+ "cutting": 5910,
+ "print": 5911,
+ "##ett": 5912,
+ "birthday": 5913,
+ "##hood": 5914,
+ "destruction": 5915,
+ "Julian": 5916,
+ "injuries": 5917,
+ "influential": 5918,
+ "sisters": 5919,
+ "raising": 5920,
+ "statue": 5921,
+ "colour": 5922,
+ "dancing": 5923,
+ "characteristics": 5924,
+ "orange": 5925,
+ "##ok": 5926,
+ "##aries": 5927,
+ "Ken": 5928,
+ "colonial": 5929,
+ "twin": 5930,
+ "Larry": 5931,
+ "surviving": 5932,
+ "##shi": 5933,
+ "Barbara": 5934,
+ "personality": 5935,
+ "entertainment": 5936,
+ "assault": 5937,
+ "##ering": 5938,
+ "talent": 5939,
+ "happens": 5940,
+ "license": 5941,
+ "86": 5942,
+ "couch": 5943,
+ "Century": 5944,
+ "soundtrack": 5945,
+ "shower": 5946,
+ "swimming": 5947,
+ "cash": 5948,
+ "Staff": 5949,
+ "bent": 5950,
+ "1885": 5951,
+ "bay": 5952,
+ "lunch": 5953,
+ "##lus": 5954,
+ "dozen": 5955,
+ "vessels": 5956,
+ "CBS": 5957,
+ "greatly": 5958,
+ "critic": 5959,
+ "Test": 5960,
+ "symbol": 5961,
+ "panel": 5962,
+ "shell": 5963,
+ "output": 5964,
+ "reaches": 5965,
+ "87": 5966,
+ "Front": 5967,
+ "motor": 5968,
+ "ocean": 5969,
+ "##era": 5970,
+ "##ala": 5971,
+ "maintenance": 5972,
+ "violent": 5973,
+ "scent": 5974,
+ "Limited": 5975,
+ "Las": 5976,
+ "Hope": 5977,
+ "Theater": 5978,
+ "Which": 5979,
+ "survey": 5980,
+ "Robin": 5981,
+ "recordings": 5982,
+ "compilation": 5983,
+ "##ward": 5984,
+ "bomb": 5985,
+ "insurance": 5986,
+ "Authority": 5987,
+ "sponsored": 5988,
+ "satellite": 5989,
+ "Jazz": 5990,
+ "refer": 5991,
+ "stronger": 5992,
+ "blow": 5993,
+ "whilst": 5994,
+ "Wrestling": 5995,
+ "suggest": 5996,
+ "##rie": 5997,
+ "climbed": 5998,
+ "##els": 5999,
+ "voices": 6000,
+ "shopping": 6001,
+ "1891": 6002,
+ "Neil": 6003,
+ "discovery": 6004,
+ "##vo": 6005,
+ "##ations": 6006,
+ "burst": 6007,
+ "Baby": 6008,
+ "peaked": 6009,
+ "Brooklyn": 6010,
+ "knocked": 6011,
+ "lift": 6012,
+ "##try": 6013,
+ "false": 6014,
+ "nations": 6015,
+ "Hugh": 6016,
+ "Catherine": 6017,
+ "preserved": 6018,
+ "distinguished": 6019,
+ "terminal": 6020,
+ "resolution": 6021,
+ "ratio": 6022,
+ "pants": 6023,
+ "cited": 6024,
+ "competitions": 6025,
+ "completion": 6026,
+ "DJ": 6027,
+ "bone": 6028,
+ "uniform": 6029,
+ "schedule": 6030,
+ "shouted": 6031,
+ "83": 6032,
+ "1920s": 6033,
+ "rarely": 6034,
+ "Basketball": 6035,
+ "Taiwan": 6036,
+ "artistic": 6037,
+ "bare": 6038,
+ "vampires": 6039,
+ "arrest": 6040,
+ "Utah": 6041,
+ "Marcus": 6042,
+ "assist": 6043,
+ "gradually": 6044,
+ "qualifying": 6045,
+ "Victorian": 6046,
+ "vast": 6047,
+ "rival": 6048,
+ "Warner": 6049,
+ "Terry": 6050,
+ "Economic": 6051,
+ "##cia": 6052,
+ "losses": 6053,
+ "boss": 6054,
+ "versus": 6055,
+ "audio": 6056,
+ "runner": 6057,
+ "apply": 6058,
+ "surgery": 6059,
+ "Play": 6060,
+ "twisted": 6061,
+ "comfortable": 6062,
+ "##cs": 6063,
+ "Everyone": 6064,
+ "guests": 6065,
+ "##lt": 6066,
+ "Harrison": 6067,
+ "UEFA": 6068,
+ "lowered": 6069,
+ "occasions": 6070,
+ "##lly": 6071,
+ "##cher": 6072,
+ "chapter": 6073,
+ "youngest": 6074,
+ "eighth": 6075,
+ "Culture": 6076,
+ "##room": 6077,
+ "##stone": 6078,
+ "1888": 6079,
+ "Songs": 6080,
+ "Seth": 6081,
+ "Digital": 6082,
+ "involvement": 6083,
+ "expedition": 6084,
+ "relationships": 6085,
+ "signing": 6086,
+ "1000": 6087,
+ "fault": 6088,
+ "annually": 6089,
+ "circuit": 6090,
+ "afterwards": 6091,
+ "meat": 6092,
+ "creature": 6093,
+ "##ou": 6094,
+ "cable": 6095,
+ "Bush": 6096,
+ "##net": 6097,
+ "Hispanic": 6098,
+ "rapid": 6099,
+ "gonna": 6100,
+ "figured": 6101,
+ "extent": 6102,
+ "considering": 6103,
+ "cried": 6104,
+ "##tin": 6105,
+ "sigh": 6106,
+ "dynasty": 6107,
+ "##ration": 6108,
+ "cabinet": 6109,
+ "Richmond": 6110,
+ "stable": 6111,
+ "##zo": 6112,
+ "1864": 6113,
+ "Admiral": 6114,
+ "Unit": 6115,
+ "occasion": 6116,
+ "shares": 6117,
+ "badly": 6118,
+ "longest": 6119,
+ "##ify": 6120,
+ "Connor": 6121,
+ "extreme": 6122,
+ "wondering": 6123,
+ "girlfriend": 6124,
+ "Studio": 6125,
+ "##tions": 6126,
+ "1865": 6127,
+ "tribe": 6128,
+ "exact": 6129,
+ "muscles": 6130,
+ "hat": 6131,
+ "Luis": 6132,
+ "Orthodox": 6133,
+ "decisions": 6134,
+ "amateur": 6135,
+ "description": 6136,
+ "##lis": 6137,
+ "hips": 6138,
+ "kingdom": 6139,
+ "##ute": 6140,
+ "Portland": 6141,
+ "whereas": 6142,
+ "Bachelor": 6143,
+ "outer": 6144,
+ "discussion": 6145,
+ "partly": 6146,
+ "Arkansas": 6147,
+ "1880": 6148,
+ "dreams": 6149,
+ "perfectly": 6150,
+ "Lloyd": 6151,
+ "##bridge": 6152,
+ "asleep": 6153,
+ "##tti": 6154,
+ "Greg": 6155,
+ "permission": 6156,
+ "trading": 6157,
+ "pitch": 6158,
+ "mill": 6159,
+ "Stage": 6160,
+ "liquid": 6161,
+ "Keith": 6162,
+ "##tal": 6163,
+ "wolf": 6164,
+ "processing": 6165,
+ "stick": 6166,
+ "Jerusalem": 6167,
+ "profile": 6168,
+ "rushed": 6169,
+ "spiritual": 6170,
+ "argument": 6171,
+ "Ice": 6172,
+ "Guy": 6173,
+ "till": 6174,
+ "Delhi": 6175,
+ "roots": 6176,
+ "Section": 6177,
+ "missions": 6178,
+ "Glasgow": 6179,
+ "penalty": 6180,
+ "NBC": 6181,
+ "encouraged": 6182,
+ "identify": 6183,
+ "keyboards": 6184,
+ "##zing": 6185,
+ "##ston": 6186,
+ "disc": 6187,
+ "plain": 6188,
+ "informed": 6189,
+ "Bernard": 6190,
+ "thinks": 6191,
+ "fled": 6192,
+ "Justin": 6193,
+ "##day": 6194,
+ "newspapers": 6195,
+ "##wick": 6196,
+ "Ralph": 6197,
+ "##zer": 6198,
+ "unlike": 6199,
+ "Stars": 6200,
+ "artillery": 6201,
+ "##ified": 6202,
+ "recovered": 6203,
+ "arrangement": 6204,
+ "searching": 6205,
+ "##pers": 6206,
+ "##tory": 6207,
+ "##rus": 6208,
+ "deaths": 6209,
+ "Egyptian": 6210,
+ "diameter": 6211,
+ "##í": 6212,
+ "marketing": 6213,
+ "corporate": 6214,
+ "teach": 6215,
+ "marks": 6216,
+ "Turner": 6217,
+ "staying": 6218,
+ "hallway": 6219,
+ "Sebastian": 6220,
+ "chapel": 6221,
+ "naked": 6222,
+ "mistake": 6223,
+ "possession": 6224,
+ "1887": 6225,
+ "dominated": 6226,
+ "jacket": 6227,
+ "creative": 6228,
+ "Fellow": 6229,
+ "Falls": 6230,
+ "Defence": 6231,
+ "suspended": 6232,
+ "employment": 6233,
+ "##rry": 6234,
+ "Hebrew": 6235,
+ "Hudson": 6236,
+ "Week": 6237,
+ "Wars": 6238,
+ "recognize": 6239,
+ "Natural": 6240,
+ "controversial": 6241,
+ "Tommy": 6242,
+ "thank": 6243,
+ "Athletic": 6244,
+ "benefits": 6245,
+ "decline": 6246,
+ "intention": 6247,
+ "##ets": 6248,
+ "Lost": 6249,
+ "Wall": 6250,
+ "participation": 6251,
+ "elevation": 6252,
+ "supports": 6253,
+ "parliament": 6254,
+ "1861": 6255,
+ "concentration": 6256,
+ "Movement": 6257,
+ "##IS": 6258,
+ "competing": 6259,
+ "stops": 6260,
+ "behalf": 6261,
+ "##mm": 6262,
+ "limits": 6263,
+ "funded": 6264,
+ "discuss": 6265,
+ "Collins": 6266,
+ "departure": 6267,
+ "obtain": 6268,
+ "woods": 6269,
+ "latest": 6270,
+ "universe": 6271,
+ "alcohol": 6272,
+ "Laura": 6273,
+ "rush": 6274,
+ "blade": 6275,
+ "funny": 6276,
+ "Dennis": 6277,
+ "forgotten": 6278,
+ "Amy": 6279,
+ "Symphony": 6280,
+ "apparent": 6281,
+ "graduating": 6282,
+ "1862": 6283,
+ "Rob": 6284,
+ "Grey": 6285,
+ "collections": 6286,
+ "Mason": 6287,
+ "emotions": 6288,
+ "##ugh": 6289,
+ "literally": 6290,
+ "Any": 6291,
+ "counties": 6292,
+ "1863": 6293,
+ "nomination": 6294,
+ "fighter": 6295,
+ "habitat": 6296,
+ "respond": 6297,
+ "external": 6298,
+ "Capital": 6299,
+ "exit": 6300,
+ "Video": 6301,
+ "carbon": 6302,
+ "sharing": 6303,
+ "Bad": 6304,
+ "opportunities": 6305,
+ "Perry": 6306,
+ "photo": 6307,
+ "##mus": 6308,
+ "Orange": 6309,
+ "posted": 6310,
+ "remainder": 6311,
+ "transportation": 6312,
+ "portrayed": 6313,
+ "Labor": 6314,
+ "recommended": 6315,
+ "percussion": 6316,
+ "rated": 6317,
+ "Grade": 6318,
+ "rivers": 6319,
+ "partially": 6320,
+ "suspected": 6321,
+ "strip": 6322,
+ "adults": 6323,
+ "button": 6324,
+ "struggled": 6325,
+ "intersection": 6326,
+ "Canal": 6327,
+ "##ability": 6328,
+ "poems": 6329,
+ "claiming": 6330,
+ "Madrid": 6331,
+ "1886": 6332,
+ "Together": 6333,
+ "##our": 6334,
+ "Much": 6335,
+ "Vancouver": 6336,
+ "instrument": 6337,
+ "instrumental": 6338,
+ "1870": 6339,
+ "mad": 6340,
+ "angle": 6341,
+ "Control": 6342,
+ "Phoenix": 6343,
+ "Leo": 6344,
+ "Communications": 6345,
+ "mail": 6346,
+ "##ette": 6347,
+ "##ev": 6348,
+ "preferred": 6349,
+ "adaptation": 6350,
+ "alleged": 6351,
+ "discussed": 6352,
+ "deeper": 6353,
+ "##ane": 6354,
+ "Yet": 6355,
+ "Monday": 6356,
+ "volumes": 6357,
+ "thrown": 6358,
+ "Zane": 6359,
+ "##logy": 6360,
+ "displayed": 6361,
+ "rolling": 6362,
+ "dogs": 6363,
+ "Along": 6364,
+ "Todd": 6365,
+ "##ivity": 6366,
+ "withdrew": 6367,
+ "representation": 6368,
+ "belief": 6369,
+ "##sia": 6370,
+ "crown": 6371,
+ "Late": 6372,
+ "Short": 6373,
+ "hardly": 6374,
+ "grinned": 6375,
+ "romantic": 6376,
+ "Pete": 6377,
+ "##ken": 6378,
+ "networks": 6379,
+ "enemies": 6380,
+ "Colin": 6381,
+ "Eventually": 6382,
+ "Side": 6383,
+ "donated": 6384,
+ "##su": 6385,
+ "steady": 6386,
+ "grab": 6387,
+ "guide": 6388,
+ "Finnish": 6389,
+ "Milan": 6390,
+ "pregnant": 6391,
+ "controversy": 6392,
+ "reminded": 6393,
+ "1884": 6394,
+ "Stuart": 6395,
+ "##bach": 6396,
+ "##ade": 6397,
+ "Race": 6398,
+ "Belgian": 6399,
+ "LP": 6400,
+ "Production": 6401,
+ "Zone": 6402,
+ "lieutenant": 6403,
+ "infantry": 6404,
+ "Child": 6405,
+ "confusion": 6406,
+ "sang": 6407,
+ "resident": 6408,
+ "##ez": 6409,
+ "victim": 6410,
+ "1881": 6411,
+ "channels": 6412,
+ "Ron": 6413,
+ "businessman": 6414,
+ "##gle": 6415,
+ "Dick": 6416,
+ "colony": 6417,
+ "pace": 6418,
+ "producers": 6419,
+ "##ese": 6420,
+ "agencies": 6421,
+ "Craig": 6422,
+ "Lucy": 6423,
+ "Very": 6424,
+ "centers": 6425,
+ "Yorkshire": 6426,
+ "photography": 6427,
+ "##ched": 6428,
+ "Album": 6429,
+ "championships": 6430,
+ "Metro": 6431,
+ "substantial": 6432,
+ "Standard": 6433,
+ "terrible": 6434,
+ "directors": 6435,
+ "contribution": 6436,
+ "advertising": 6437,
+ "emotional": 6438,
+ "##its": 6439,
+ "layer": 6440,
+ "segment": 6441,
+ "sir": 6442,
+ "folded": 6443,
+ "Roberts": 6444,
+ "ceased": 6445,
+ "Hampshire": 6446,
+ "##ray": 6447,
+ "detailed": 6448,
+ "partners": 6449,
+ "m²": 6450,
+ "##pt": 6451,
+ "Beth": 6452,
+ "genre": 6453,
+ "commented": 6454,
+ "generated": 6455,
+ "remote": 6456,
+ "aim": 6457,
+ "Hans": 6458,
+ "credits": 6459,
+ "concerts": 6460,
+ "periods": 6461,
+ "breakfast": 6462,
+ "gay": 6463,
+ "shadow": 6464,
+ "defence": 6465,
+ "Too": 6466,
+ "Had": 6467,
+ "transition": 6468,
+ "Afghanistan": 6469,
+ "##book": 6470,
+ "eggs": 6471,
+ "defend": 6472,
+ "##lli": 6473,
+ "writes": 6474,
+ "Systems": 6475,
+ "bones": 6476,
+ "mess": 6477,
+ "seed": 6478,
+ "scientists": 6479,
+ "Shortly": 6480,
+ "Romanian": 6481,
+ "##zy": 6482,
+ "Freedom": 6483,
+ "muscle": 6484,
+ "hero": 6485,
+ "parent": 6486,
+ "agriculture": 6487,
+ "checked": 6488,
+ "Islam": 6489,
+ "Bristol": 6490,
+ "Freyja": 6491,
+ "Arena": 6492,
+ "cabin": 6493,
+ "Germans": 6494,
+ "electricity": 6495,
+ "ranks": 6496,
+ "viewed": 6497,
+ "medals": 6498,
+ "Wolf": 6499,
+ "associate": 6500,
+ "Madison": 6501,
+ "Sorry": 6502,
+ "fort": 6503,
+ "Chile": 6504,
+ "detail": 6505,
+ "widespread": 6506,
+ "attorney": 6507,
+ "boyfriend": 6508,
+ "##nan": 6509,
+ "Students": 6510,
+ "Spencer": 6511,
+ "##ig": 6512,
+ "bite": 6513,
+ "Maine": 6514,
+ "demolished": 6515,
+ "Lisa": 6516,
+ "erected": 6517,
+ "Someone": 6518,
+ "operational": 6519,
+ "Commissioner": 6520,
+ "NHL": 6521,
+ "Coach": 6522,
+ "Bar": 6523,
+ "forcing": 6524,
+ "Dream": 6525,
+ "Rico": 6526,
+ "cargo": 6527,
+ "Murphy": 6528,
+ "##fish": 6529,
+ "##ase": 6530,
+ "distant": 6531,
+ "##master": 6532,
+ "##ora": 6533,
+ "Organization": 6534,
+ "doorway": 6535,
+ "Steven": 6536,
+ "traded": 6537,
+ "electrical": 6538,
+ "frequent": 6539,
+ "##wn": 6540,
+ "Branch": 6541,
+ "Sure": 6542,
+ "1882": 6543,
+ "placing": 6544,
+ "Manhattan": 6545,
+ "attending": 6546,
+ "attributed": 6547,
+ "excellent": 6548,
+ "pounds": 6549,
+ "ruling": 6550,
+ "principles": 6551,
+ "component": 6552,
+ "Mediterranean": 6553,
+ "Vegas": 6554,
+ "machines": 6555,
+ "percentage": 6556,
+ "infrastructure": 6557,
+ "throwing": 6558,
+ "affiliated": 6559,
+ "Kings": 6560,
+ "secured": 6561,
+ "Caribbean": 6562,
+ "Track": 6563,
+ "Ted": 6564,
+ "honour": 6565,
+ "opponent": 6566,
+ "Virgin": 6567,
+ "Construction": 6568,
+ "grave": 6569,
+ "produces": 6570,
+ "Challenge": 6571,
+ "stretched": 6572,
+ "paying": 6573,
+ "murmured": 6574,
+ "##ata": 6575,
+ "integrated": 6576,
+ "waved": 6577,
+ "Nathan": 6578,
+ "##ator": 6579,
+ "transmission": 6580,
+ "videos": 6581,
+ "##yan": 6582,
+ "##hu": 6583,
+ "Nova": 6584,
+ "descent": 6585,
+ "AM": 6586,
+ "Harold": 6587,
+ "conservative": 6588,
+ "Therefore": 6589,
+ "venue": 6590,
+ "competitive": 6591,
+ "##ui": 6592,
+ "conclusion": 6593,
+ "funeral": 6594,
+ "confidence": 6595,
+ "releases": 6596,
+ "scholar": 6597,
+ "##sson": 6598,
+ "Treaty": 6599,
+ "stress": 6600,
+ "mood": 6601,
+ "##sm": 6602,
+ "Mac": 6603,
+ "residing": 6604,
+ "Action": 6605,
+ "Fund": 6606,
+ "##ship": 6607,
+ "animated": 6608,
+ "fitted": 6609,
+ "##kar": 6610,
+ "defending": 6611,
+ "voting": 6612,
+ "tend": 6613,
+ "##berry": 6614,
+ "answers": 6615,
+ "believes": 6616,
+ "##ci": 6617,
+ "helps": 6618,
+ "Aaron": 6619,
+ "##tis": 6620,
+ "themes": 6621,
+ "##lay": 6622,
+ "populations": 6623,
+ "Players": 6624,
+ "stroke": 6625,
+ "Trinity": 6626,
+ "electoral": 6627,
+ "paint": 6628,
+ "abroad": 6629,
+ "charity": 6630,
+ "keys": 6631,
+ "Fair": 6632,
+ "##pes": 6633,
+ "interrupted": 6634,
+ "participants": 6635,
+ "murdered": 6636,
+ "Days": 6637,
+ "supporters": 6638,
+ "##ab": 6639,
+ "expert": 6640,
+ "borders": 6641,
+ "mate": 6642,
+ "##llo": 6643,
+ "solar": 6644,
+ "architectural": 6645,
+ "tension": 6646,
+ "##bling": 6647,
+ "Parish": 6648,
+ "tape": 6649,
+ "operator": 6650,
+ "Cultural": 6651,
+ "Clinton": 6652,
+ "indicates": 6653,
+ "publisher": 6654,
+ "ordinary": 6655,
+ "sugar": 6656,
+ "arrive": 6657,
+ "rifle": 6658,
+ "acoustic": 6659,
+ "##uring": 6660,
+ "assets": 6661,
+ "##shire": 6662,
+ "SS": 6663,
+ "sufficient": 6664,
+ "options": 6665,
+ "HMS": 6666,
+ "Classic": 6667,
+ "bars": 6668,
+ "rebuilt": 6669,
+ "governments": 6670,
+ "Beijing": 6671,
+ "reporter": 6672,
+ "screamed": 6673,
+ "Abbey": 6674,
+ "crying": 6675,
+ "mechanical": 6676,
+ "instantly": 6677,
+ "communications": 6678,
+ "Political": 6679,
+ "cemetery": 6680,
+ "Cameron": 6681,
+ "Stop": 6682,
+ "representatives": 6683,
+ "USS": 6684,
+ "texts": 6685,
+ "mathematics": 6686,
+ "innings": 6687,
+ "civilian": 6688,
+ "Serbia": 6689,
+ "##hill": 6690,
+ "practical": 6691,
+ "patterns": 6692,
+ "dust": 6693,
+ "Faculty": 6694,
+ "debt": 6695,
+ "##end": 6696,
+ "##cus": 6697,
+ "junction": 6698,
+ "suppose": 6699,
+ "experimental": 6700,
+ "Computer": 6701,
+ "Food": 6702,
+ "wrist": 6703,
+ "abuse": 6704,
+ "dealing": 6705,
+ "bigger": 6706,
+ "cap": 6707,
+ "principle": 6708,
+ "##pin": 6709,
+ "Muhammad": 6710,
+ "Fleet": 6711,
+ "Collection": 6712,
+ "attempting": 6713,
+ "dismissed": 6714,
+ "##burn": 6715,
+ "regime": 6716,
+ "Herbert": 6717,
+ "##ua": 6718,
+ "shadows": 6719,
+ "1883": 6720,
+ "Eve": 6721,
+ "Lanka": 6722,
+ "1878": 6723,
+ "Performance": 6724,
+ "fictional": 6725,
+ "##lock": 6726,
+ "Noah": 6727,
+ "Run": 6728,
+ "Voivodeship": 6729,
+ "exercise": 6730,
+ "broadcasting": 6731,
+ "##fer": 6732,
+ "RAF": 6733,
+ "Magic": 6734,
+ "Bangladesh": 6735,
+ "suitable": 6736,
+ "##low": 6737,
+ "##del": 6738,
+ "styles": 6739,
+ "toured": 6740,
+ "Code": 6741,
+ "identical": 6742,
+ "links": 6743,
+ "insisted": 6744,
+ "110": 6745,
+ "flash": 6746,
+ "Model": 6747,
+ "slave": 6748,
+ "Derek": 6749,
+ "Rev": 6750,
+ "fairly": 6751,
+ "Greater": 6752,
+ "sole": 6753,
+ "##lands": 6754,
+ "connecting": 6755,
+ "zero": 6756,
+ "bench": 6757,
+ "##ome": 6758,
+ "switched": 6759,
+ "Fall": 6760,
+ "Owen": 6761,
+ "yours": 6762,
+ "Electric": 6763,
+ "shocked": 6764,
+ "convention": 6765,
+ "##bra": 6766,
+ "climb": 6767,
+ "memorial": 6768,
+ "swept": 6769,
+ "Racing": 6770,
+ "decides": 6771,
+ "belong": 6772,
+ "##nk": 6773,
+ "parliamentary": 6774,
+ "##und": 6775,
+ "ages": 6776,
+ "proof": 6777,
+ "##dan": 6778,
+ "delivery": 6779,
+ "1860": 6780,
+ "##ów": 6781,
+ "sad": 6782,
+ "publicly": 6783,
+ "leaning": 6784,
+ "Archbishop": 6785,
+ "dirt": 6786,
+ "##ose": 6787,
+ "categories": 6788,
+ "1876": 6789,
+ "burn": 6790,
+ "##bing": 6791,
+ "requested": 6792,
+ "Guinea": 6793,
+ "Historical": 6794,
+ "rhythm": 6795,
+ "relation": 6796,
+ "##heim": 6797,
+ "ye": 6798,
+ "pursue": 6799,
+ "merchant": 6800,
+ "##mes": 6801,
+ "lists": 6802,
+ "continuous": 6803,
+ "frowned": 6804,
+ "colored": 6805,
+ "tool": 6806,
+ "gods": 6807,
+ "involves": 6808,
+ "Duncan": 6809,
+ "photographs": 6810,
+ "Cricket": 6811,
+ "slight": 6812,
+ "Gregory": 6813,
+ "atmosphere": 6814,
+ "wider": 6815,
+ "Cook": 6816,
+ "##tar": 6817,
+ "essential": 6818,
+ "Being": 6819,
+ "FA": 6820,
+ "emperor": 6821,
+ "wealthy": 6822,
+ "nights": 6823,
+ "##bar": 6824,
+ "licensed": 6825,
+ "Hawaii": 6826,
+ "viewers": 6827,
+ "Language": 6828,
+ "load": 6829,
+ "nearest": 6830,
+ "milk": 6831,
+ "kilometers": 6832,
+ "platforms": 6833,
+ "##ys": 6834,
+ "territories": 6835,
+ "Rogers": 6836,
+ "sheet": 6837,
+ "Rangers": 6838,
+ "contested": 6839,
+ "##lation": 6840,
+ "isolated": 6841,
+ "assisted": 6842,
+ "swallowed": 6843,
+ "Small": 6844,
+ "Contemporary": 6845,
+ "Technical": 6846,
+ "Edwards": 6847,
+ "express": 6848,
+ "Volume": 6849,
+ "endemic": 6850,
+ "##ei": 6851,
+ "tightly": 6852,
+ "Whatever": 6853,
+ "indigenous": 6854,
+ "Colombia": 6855,
+ "##ulation": 6856,
+ "hp": 6857,
+ "characterized": 6858,
+ "##ida": 6859,
+ "Nigeria": 6860,
+ "Professional": 6861,
+ "duo": 6862,
+ "Soccer": 6863,
+ "slaves": 6864,
+ "Farm": 6865,
+ "smart": 6866,
+ "Attorney": 6867,
+ "Attendance": 6868,
+ "Common": 6869,
+ "salt": 6870,
+ "##vin": 6871,
+ "tribes": 6872,
+ "nod": 6873,
+ "sentenced": 6874,
+ "bid": 6875,
+ "sample": 6876,
+ "Drive": 6877,
+ "switch": 6878,
+ "instant": 6879,
+ "21st": 6880,
+ "Cuba": 6881,
+ "drunk": 6882,
+ "Alaska": 6883,
+ "proud": 6884,
+ "awareness": 6885,
+ "hitting": 6886,
+ "sessions": 6887,
+ "Thai": 6888,
+ "locally": 6889,
+ "elsewhere": 6890,
+ "Dragon": 6891,
+ "gentle": 6892,
+ "touching": 6893,
+ "##lee": 6894,
+ "Springs": 6895,
+ "Universal": 6896,
+ "Latino": 6897,
+ "spin": 6898,
+ "1871": 6899,
+ "Chart": 6900,
+ "recalled": 6901,
+ "Type": 6902,
+ "pointing": 6903,
+ "##ii": 6904,
+ "lowest": 6905,
+ "##ser": 6906,
+ "grandmother": 6907,
+ "Adelaide": 6908,
+ "Jacques": 6909,
+ "spotted": 6910,
+ "Buffalo": 6911,
+ "restoration": 6912,
+ "Son": 6913,
+ "Joan": 6914,
+ "farmers": 6915,
+ "Lily": 6916,
+ "1879": 6917,
+ "lucky": 6918,
+ "##dal": 6919,
+ "luck": 6920,
+ "eldest": 6921,
+ "##rant": 6922,
+ "Market": 6923,
+ "drummer": 6924,
+ "deployed": 6925,
+ "warned": 6926,
+ "prince": 6927,
+ "sing": 6928,
+ "amazing": 6929,
+ "sailed": 6930,
+ "##oon": 6931,
+ "1875": 6932,
+ "Primary": 6933,
+ "traveling": 6934,
+ "Masters": 6935,
+ "Sara": 6936,
+ "cattle": 6937,
+ "Trail": 6938,
+ "gang": 6939,
+ "Further": 6940,
+ "desert": 6941,
+ "relocated": 6942,
+ "##tch": 6943,
+ "##ord": 6944,
+ "Flight": 6945,
+ "illness": 6946,
+ "Munich": 6947,
+ "ninth": 6948,
+ "repair": 6949,
+ "Singles": 6950,
+ "##lated": 6951,
+ "Tyler": 6952,
+ "tossed": 6953,
+ "boots": 6954,
+ "Work": 6955,
+ "sized": 6956,
+ "earning": 6957,
+ "shoved": 6958,
+ "magazines": 6959,
+ "housed": 6960,
+ "dam": 6961,
+ "researchers": 6962,
+ "Former": 6963,
+ "spun": 6964,
+ "premiere": 6965,
+ "spaces": 6966,
+ "organised": 6967,
+ "wealth": 6968,
+ "crimes": 6969,
+ "devoted": 6970,
+ "stones": 6971,
+ "Urban": 6972,
+ "automatic": 6973,
+ "hop": 6974,
+ "affect": 6975,
+ "outstanding": 6976,
+ "tanks": 6977,
+ "mechanism": 6978,
+ "Muslims": 6979,
+ "Ms": 6980,
+ "shots": 6981,
+ "argue": 6982,
+ "Jeremy": 6983,
+ "connections": 6984,
+ "Armenian": 6985,
+ "increases": 6986,
+ "rubbed": 6987,
+ "1867": 6988,
+ "retail": 6989,
+ "gear": 6990,
+ "Pan": 6991,
+ "bonus": 6992,
+ "jurisdiction": 6993,
+ "weird": 6994,
+ "concerning": 6995,
+ "whisper": 6996,
+ "##gal": 6997,
+ "Microsoft": 6998,
+ "tenure": 6999,
+ "hills": 7000,
+ "www": 7001,
+ "Gmina": 7002,
+ "porch": 7003,
+ "files": 7004,
+ "reportedly": 7005,
+ "venture": 7006,
+ "Storm": 7007,
+ "##ence": 7008,
+ "Nature": 7009,
+ "killer": 7010,
+ "panic": 7011,
+ "fate": 7012,
+ "Secret": 7013,
+ "Wang": 7014,
+ "scream": 7015,
+ "drivers": 7016,
+ "belongs": 7017,
+ "Chamber": 7018,
+ "clan": 7019,
+ "monument": 7020,
+ "mixing": 7021,
+ "Peru": 7022,
+ "bet": 7023,
+ "Riley": 7024,
+ "Friends": 7025,
+ "Isaac": 7026,
+ "submarine": 7027,
+ "1877": 7028,
+ "130": 7029,
+ "judges": 7030,
+ "harm": 7031,
+ "ranging": 7032,
+ "affair": 7033,
+ "prepare": 7034,
+ "pupils": 7035,
+ "householder": 7036,
+ "Policy": 7037,
+ "decorated": 7038,
+ "Nation": 7039,
+ "slammed": 7040,
+ "activist": 7041,
+ "implemented": 7042,
+ "Room": 7043,
+ "qualify": 7044,
+ "Publishing": 7045,
+ "establishing": 7046,
+ "Baptist": 7047,
+ "touring": 7048,
+ "subsidiary": 7049,
+ "##nal": 7050,
+ "legend": 7051,
+ "1872": 7052,
+ "laughter": 7053,
+ "PC": 7054,
+ "Athens": 7055,
+ "settlers": 7056,
+ "ties": 7057,
+ "dual": 7058,
+ "dear": 7059,
+ "Draft": 7060,
+ "strategic": 7061,
+ "Ivan": 7062,
+ "reveal": 7063,
+ "closest": 7064,
+ "dominant": 7065,
+ "Ah": 7066,
+ "##ult": 7067,
+ "Denver": 7068,
+ "bond": 7069,
+ "boundaries": 7070,
+ "drafted": 7071,
+ "tables": 7072,
+ "##TV": 7073,
+ "eyed": 7074,
+ "Edition": 7075,
+ "##ena": 7076,
+ "1868": 7077,
+ "belonging": 7078,
+ "1874": 7079,
+ "Industrial": 7080,
+ "cream": 7081,
+ "Ridge": 7082,
+ "Hindu": 7083,
+ "scholarship": 7084,
+ "Ma": 7085,
+ "opens": 7086,
+ "initiated": 7087,
+ "##ith": 7088,
+ "yelled": 7089,
+ "compound": 7090,
+ "random": 7091,
+ "Throughout": 7092,
+ "grades": 7093,
+ "physics": 7094,
+ "sank": 7095,
+ "grows": 7096,
+ "exclusively": 7097,
+ "settle": 7098,
+ "Saints": 7099,
+ "brings": 7100,
+ "Amsterdam": 7101,
+ "Make": 7102,
+ "Hart": 7103,
+ "walks": 7104,
+ "battery": 7105,
+ "violin": 7106,
+ "##born": 7107,
+ "explanation": 7108,
+ "##ware": 7109,
+ "1873": 7110,
+ "##har": 7111,
+ "provinces": 7112,
+ "thrust": 7113,
+ "exclusive": 7114,
+ "sculpture": 7115,
+ "shops": 7116,
+ "##fire": 7117,
+ "VI": 7118,
+ "constitution": 7119,
+ "Barcelona": 7120,
+ "monster": 7121,
+ "Devon": 7122,
+ "Jefferson": 7123,
+ "Sullivan": 7124,
+ "bow": 7125,
+ "##din": 7126,
+ "desperate": 7127,
+ "##ć": 7128,
+ "Julie": 7129,
+ "##mon": 7130,
+ "##ising": 7131,
+ "terminus": 7132,
+ "Jesse": 7133,
+ "abilities": 7134,
+ "golf": 7135,
+ "##ple": 7136,
+ "##via": 7137,
+ "##away": 7138,
+ "Raymond": 7139,
+ "measured": 7140,
+ "jury": 7141,
+ "firing": 7142,
+ "revenue": 7143,
+ "suburb": 7144,
+ "Bulgarian": 7145,
+ "1866": 7146,
+ "##cha": 7147,
+ "timber": 7148,
+ "Things": 7149,
+ "##weight": 7150,
+ "Morning": 7151,
+ "spots": 7152,
+ "Alberta": 7153,
+ "Data": 7154,
+ "explains": 7155,
+ "Kyle": 7156,
+ "friendship": 7157,
+ "raw": 7158,
+ "tube": 7159,
+ "demonstrated": 7160,
+ "aboard": 7161,
+ "immigrants": 7162,
+ "reply": 7163,
+ "breathe": 7164,
+ "Manager": 7165,
+ "ease": 7166,
+ "##ban": 7167,
+ "##dia": 7168,
+ "Diocese": 7169,
+ "##vy": 7170,
+ "##ía": 7171,
+ "pit": 7172,
+ "ongoing": 7173,
+ "##lie": 7174,
+ "Gilbert": 7175,
+ "Costa": 7176,
+ "1940s": 7177,
+ "Report": 7178,
+ "voters": 7179,
+ "cloud": 7180,
+ "traditions": 7181,
+ "##MS": 7182,
+ "gallery": 7183,
+ "Jennifer": 7184,
+ "swung": 7185,
+ "Broadcasting": 7186,
+ "Does": 7187,
+ "diverse": 7188,
+ "reveals": 7189,
+ "arriving": 7190,
+ "initiative": 7191,
+ "##ani": 7192,
+ "Give": 7193,
+ "Allied": 7194,
+ "Pat": 7195,
+ "Outstanding": 7196,
+ "monastery": 7197,
+ "blind": 7198,
+ "Currently": 7199,
+ "##war": 7200,
+ "bloody": 7201,
+ "stopping": 7202,
+ "focuses": 7203,
+ "managing": 7204,
+ "Florence": 7205,
+ "Harvey": 7206,
+ "creatures": 7207,
+ "900": 7208,
+ "breast": 7209,
+ "internet": 7210,
+ "Artillery": 7211,
+ "purple": 7212,
+ "##mate": 7213,
+ "alliance": 7214,
+ "excited": 7215,
+ "fee": 7216,
+ "Brisbane": 7217,
+ "lifetime": 7218,
+ "Private": 7219,
+ "##aw": 7220,
+ "##nis": 7221,
+ "##gue": 7222,
+ "##ika": 7223,
+ "phrase": 7224,
+ "regulations": 7225,
+ "reflected": 7226,
+ "manufactured": 7227,
+ "conventional": 7228,
+ "pleased": 7229,
+ "client": 7230,
+ "##ix": 7231,
+ "##ncy": 7232,
+ "Pedro": 7233,
+ "reduction": 7234,
+ "##con": 7235,
+ "welcome": 7236,
+ "jail": 7237,
+ "comfort": 7238,
+ "Iranian": 7239,
+ "Norfolk": 7240,
+ "Dakota": 7241,
+ "##tein": 7242,
+ "evolution": 7243,
+ "everywhere": 7244,
+ "Initially": 7245,
+ "sensitive": 7246,
+ "Olivia": 7247,
+ "Oscar": 7248,
+ "implementation": 7249,
+ "sits": 7250,
+ "stolen": 7251,
+ "demands": 7252,
+ "slide": 7253,
+ "grandson": 7254,
+ "##ich": 7255,
+ "merger": 7256,
+ "##mic": 7257,
+ "Spirit": 7258,
+ "##°": 7259,
+ "ticket": 7260,
+ "root": 7261,
+ "difficulty": 7262,
+ "Nevada": 7263,
+ "##als": 7264,
+ "lined": 7265,
+ "Dylan": 7266,
+ "Original": 7267,
+ "Call": 7268,
+ "biological": 7269,
+ "EU": 7270,
+ "dramatic": 7271,
+ "##hn": 7272,
+ "Operations": 7273,
+ "treaty": 7274,
+ "gap": 7275,
+ "##list": 7276,
+ "Am": 7277,
+ "Romanized": 7278,
+ "moral": 7279,
+ "Butler": 7280,
+ "perspective": 7281,
+ "Furthermore": 7282,
+ "Manuel": 7283,
+ "absolutely": 7284,
+ "unsuccessful": 7285,
+ "disaster": 7286,
+ "dispute": 7287,
+ "preparation": 7288,
+ "tested": 7289,
+ "discover": 7290,
+ "##ach": 7291,
+ "shield": 7292,
+ "squeezed": 7293,
+ "brushed": 7294,
+ "battalion": 7295,
+ "Arnold": 7296,
+ "##ras": 7297,
+ "superior": 7298,
+ "treat": 7299,
+ "clinical": 7300,
+ "##so": 7301,
+ "Apple": 7302,
+ "Syria": 7303,
+ "Cincinnati": 7304,
+ "package": 7305,
+ "flights": 7306,
+ "editions": 7307,
+ "Leader": 7308,
+ "minority": 7309,
+ "wonderful": 7310,
+ "hang": 7311,
+ "Pop": 7312,
+ "Philippine": 7313,
+ "telephone": 7314,
+ "bell": 7315,
+ "honorary": 7316,
+ "##mar": 7317,
+ "balls": 7318,
+ "Democrat": 7319,
+ "dirty": 7320,
+ "thereafter": 7321,
+ "collapsed": 7322,
+ "Inside": 7323,
+ "slip": 7324,
+ "wrestling": 7325,
+ "##ín": 7326,
+ "listened": 7327,
+ "regard": 7328,
+ "bowl": 7329,
+ "None": 7330,
+ "Sport": 7331,
+ "completing": 7332,
+ "trapped": 7333,
+ "##view": 7334,
+ "copper": 7335,
+ "Wallace": 7336,
+ "Honor": 7337,
+ "blame": 7338,
+ "Peninsula": 7339,
+ "##ert": 7340,
+ "##oy": 7341,
+ "Anglo": 7342,
+ "bearing": 7343,
+ "simultaneously": 7344,
+ "honest": 7345,
+ "##ias": 7346,
+ "Mix": 7347,
+ "Got": 7348,
+ "speaker": 7349,
+ "voiced": 7350,
+ "impressed": 7351,
+ "prices": 7352,
+ "error": 7353,
+ "1869": 7354,
+ "##feld": 7355,
+ "trials": 7356,
+ "Nine": 7357,
+ "Industry": 7358,
+ "substitute": 7359,
+ "Municipal": 7360,
+ "departed": 7361,
+ "slept": 7362,
+ "##ama": 7363,
+ "Junction": 7364,
+ "Socialist": 7365,
+ "flower": 7366,
+ "dropping": 7367,
+ "comment": 7368,
+ "fantasy": 7369,
+ "##ress": 7370,
+ "arrangements": 7371,
+ "travelled": 7372,
+ "furniture": 7373,
+ "fist": 7374,
+ "relieved": 7375,
+ "##tics": 7376,
+ "Leonard": 7377,
+ "linear": 7378,
+ "earn": 7379,
+ "expand": 7380,
+ "Soul": 7381,
+ "Plan": 7382,
+ "Leeds": 7383,
+ "Sierra": 7384,
+ "accessible": 7385,
+ "innocent": 7386,
+ "Winner": 7387,
+ "Fighter": 7388,
+ "Range": 7389,
+ "winds": 7390,
+ "vertical": 7391,
+ "Pictures": 7392,
+ "101": 7393,
+ "charter": 7394,
+ "cooperation": 7395,
+ "prisoner": 7396,
+ "interviews": 7397,
+ "recognised": 7398,
+ "sung": 7399,
+ "manufacturer": 7400,
+ "exposure": 7401,
+ "submitted": 7402,
+ "Mars": 7403,
+ "leaf": 7404,
+ "gauge": 7405,
+ "screaming": 7406,
+ "likes": 7407,
+ "eligible": 7408,
+ "##ac": 7409,
+ "gathering": 7410,
+ "columns": 7411,
+ "##dra": 7412,
+ "belly": 7413,
+ "UN": 7414,
+ "maps": 7415,
+ "messages": 7416,
+ "speakers": 7417,
+ "##ants": 7418,
+ "garage": 7419,
+ "unincorporated": 7420,
+ "Number": 7421,
+ "Watson": 7422,
+ "sixteen": 7423,
+ "lots": 7424,
+ "beaten": 7425,
+ "Could": 7426,
+ "Municipality": 7427,
+ "##ano": 7428,
+ "Horse": 7429,
+ "talks": 7430,
+ "Drake": 7431,
+ "scores": 7432,
+ "Venice": 7433,
+ "genetic": 7434,
+ "##mal": 7435,
+ "##ère": 7436,
+ "Cold": 7437,
+ "Jose": 7438,
+ "nurse": 7439,
+ "traditionally": 7440,
+ "##bus": 7441,
+ "Territory": 7442,
+ "Key": 7443,
+ "Nancy": 7444,
+ "##win": 7445,
+ "thumb": 7446,
+ "São": 7447,
+ "index": 7448,
+ "dependent": 7449,
+ "carries": 7450,
+ "controls": 7451,
+ "Comics": 7452,
+ "coalition": 7453,
+ "physician": 7454,
+ "referring": 7455,
+ "Ruth": 7456,
+ "Based": 7457,
+ "restricted": 7458,
+ "inherited": 7459,
+ "internationally": 7460,
+ "stretch": 7461,
+ "THE": 7462,
+ "plates": 7463,
+ "margin": 7464,
+ "Holland": 7465,
+ "knock": 7466,
+ "significance": 7467,
+ "valuable": 7468,
+ "Kenya": 7469,
+ "carved": 7470,
+ "emotion": 7471,
+ "conservation": 7472,
+ "municipalities": 7473,
+ "overseas": 7474,
+ "resumed": 7475,
+ "Finance": 7476,
+ "graduation": 7477,
+ "blinked": 7478,
+ "temperatures": 7479,
+ "constantly": 7480,
+ "productions": 7481,
+ "scientist": 7482,
+ "ghost": 7483,
+ "cuts": 7484,
+ "permitted": 7485,
+ "##ches": 7486,
+ "firmly": 7487,
+ "##bert": 7488,
+ "patrol": 7489,
+ "##yo": 7490,
+ "Croatian": 7491,
+ "attacking": 7492,
+ "1850": 7493,
+ "portrait": 7494,
+ "promoting": 7495,
+ "sink": 7496,
+ "conversion": 7497,
+ "##kov": 7498,
+ "locomotives": 7499,
+ "Guide": 7500,
+ "##val": 7501,
+ "nephew": 7502,
+ "relevant": 7503,
+ "Marc": 7504,
+ "drum": 7505,
+ "originated": 7506,
+ "Chair": 7507,
+ "visits": 7508,
+ "dragged": 7509,
+ "Price": 7510,
+ "favour": 7511,
+ "corridor": 7512,
+ "properly": 7513,
+ "respective": 7514,
+ "Caroline": 7515,
+ "reporting": 7516,
+ "inaugural": 7517,
+ "1848": 7518,
+ "industries": 7519,
+ "##ching": 7520,
+ "edges": 7521,
+ "Christianity": 7522,
+ "Maurice": 7523,
+ "Trent": 7524,
+ "Economics": 7525,
+ "carrier": 7526,
+ "Reed": 7527,
+ "##gon": 7528,
+ "tribute": 7529,
+ "Pradesh": 7530,
+ "##ale": 7531,
+ "extend": 7532,
+ "attitude": 7533,
+ "Yale": 7534,
+ "##lu": 7535,
+ "settlements": 7536,
+ "glasses": 7537,
+ "taxes": 7538,
+ "targets": 7539,
+ "##ids": 7540,
+ "quarters": 7541,
+ "##ological": 7542,
+ "connect": 7543,
+ "hence": 7544,
+ "metre": 7545,
+ "collapse": 7546,
+ "underneath": 7547,
+ "banned": 7548,
+ "Future": 7549,
+ "clients": 7550,
+ "alternate": 7551,
+ "explosion": 7552,
+ "kinds": 7553,
+ "Commons": 7554,
+ "hungry": 7555,
+ "dragon": 7556,
+ "Chapel": 7557,
+ "Buddhist": 7558,
+ "lover": 7559,
+ "depression": 7560,
+ "pulls": 7561,
+ "##ges": 7562,
+ "##uk": 7563,
+ "origins": 7564,
+ "computers": 7565,
+ "crosses": 7566,
+ "kissing": 7567,
+ "assume": 7568,
+ "emphasis": 7569,
+ "lighting": 7570,
+ "##ites": 7571,
+ "personally": 7572,
+ "crashed": 7573,
+ "beam": 7574,
+ "touchdown": 7575,
+ "lane": 7576,
+ "comparison": 7577,
+ "##mont": 7578,
+ "Hitler": 7579,
+ "##las": 7580,
+ "execution": 7581,
+ "##ene": 7582,
+ "acre": 7583,
+ "sum": 7584,
+ "Pearl": 7585,
+ "ray": 7586,
+ "##point": 7587,
+ "essentially": 7588,
+ "worker": 7589,
+ "convicted": 7590,
+ "tear": 7591,
+ "Clay": 7592,
+ "recovery": 7593,
+ "Literature": 7594,
+ "Unfortunately": 7595,
+ "##row": 7596,
+ "partial": 7597,
+ "Petersburg": 7598,
+ "Bulgaria": 7599,
+ "coaching": 7600,
+ "evolved": 7601,
+ "reception": 7602,
+ "enters": 7603,
+ "narrowed": 7604,
+ "elevator": 7605,
+ "therapy": 7606,
+ "defended": 7607,
+ "pairs": 7608,
+ "##lam": 7609,
+ "breaks": 7610,
+ "Bennett": 7611,
+ "Uncle": 7612,
+ "cylinder": 7613,
+ "##ison": 7614,
+ "passion": 7615,
+ "bases": 7616,
+ "Actor": 7617,
+ "cancelled": 7618,
+ "battles": 7619,
+ "extensively": 7620,
+ "oxygen": 7621,
+ "Ancient": 7622,
+ "specialized": 7623,
+ "negotiations": 7624,
+ "##rat": 7625,
+ "acquisition": 7626,
+ "convince": 7627,
+ "interpretation": 7628,
+ "##00": 7629,
+ "photos": 7630,
+ "aspect": 7631,
+ "colleges": 7632,
+ "Artist": 7633,
+ "keeps": 7634,
+ "##wing": 7635,
+ "Croatia": 7636,
+ "##ona": 7637,
+ "Hughes": 7638,
+ "Otto": 7639,
+ "comments": 7640,
+ "##du": 7641,
+ "Ph": 7642,
+ "Sweet": 7643,
+ "adventure": 7644,
+ "describing": 7645,
+ "Student": 7646,
+ "Shakespeare": 7647,
+ "scattered": 7648,
+ "objective": 7649,
+ "Aviation": 7650,
+ "Phillips": 7651,
+ "Fourth": 7652,
+ "athletes": 7653,
+ "##hal": 7654,
+ "##tered": 7655,
+ "Guitar": 7656,
+ "intensity": 7657,
+ "née": 7658,
+ "dining": 7659,
+ "curve": 7660,
+ "Obama": 7661,
+ "topics": 7662,
+ "legislative": 7663,
+ "Mill": 7664,
+ "Cruz": 7665,
+ "##ars": 7666,
+ "Members": 7667,
+ "recipient": 7668,
+ "Derby": 7669,
+ "inspiration": 7670,
+ "corresponding": 7671,
+ "fed": 7672,
+ "YouTube": 7673,
+ "coins": 7674,
+ "pressing": 7675,
+ "intent": 7676,
+ "Karen": 7677,
+ "cinema": 7678,
+ "Delta": 7679,
+ "destination": 7680,
+ "shorter": 7681,
+ "Christians": 7682,
+ "imagined": 7683,
+ "canal": 7684,
+ "Newcastle": 7685,
+ "Shah": 7686,
+ "Adrian": 7687,
+ "super": 7688,
+ "Males": 7689,
+ "160": 7690,
+ "liberal": 7691,
+ "lord": 7692,
+ "bat": 7693,
+ "supplied": 7694,
+ "Claude": 7695,
+ "meal": 7696,
+ "worship": 7697,
+ "##atic": 7698,
+ "Han": 7699,
+ "wire": 7700,
+ "°F": 7701,
+ "##tha": 7702,
+ "punishment": 7703,
+ "thirteen": 7704,
+ "fighters": 7705,
+ "##ibility": 7706,
+ "1859": 7707,
+ "Ball": 7708,
+ "gardens": 7709,
+ "##ari": 7710,
+ "Ottawa": 7711,
+ "pole": 7712,
+ "indicating": 7713,
+ "Twenty": 7714,
+ "Higher": 7715,
+ "Bass": 7716,
+ "Ivy": 7717,
+ "farming": 7718,
+ "##urs": 7719,
+ "certified": 7720,
+ "Saudi": 7721,
+ "plenty": 7722,
+ "##ces": 7723,
+ "restaurants": 7724,
+ "Representative": 7725,
+ "Miles": 7726,
+ "payment": 7727,
+ "##inger": 7728,
+ "##rit": 7729,
+ "Confederate": 7730,
+ "festivals": 7731,
+ "references": 7732,
+ "##ić": 7733,
+ "Mario": 7734,
+ "PhD": 7735,
+ "playoffs": 7736,
+ "witness": 7737,
+ "rice": 7738,
+ "mask": 7739,
+ "saving": 7740,
+ "opponents": 7741,
+ "enforcement": 7742,
+ "automatically": 7743,
+ "relegated": 7744,
+ "##oe": 7745,
+ "radar": 7746,
+ "whenever": 7747,
+ "Financial": 7748,
+ "imperial": 7749,
+ "uncredited": 7750,
+ "influences": 7751,
+ "Abraham": 7752,
+ "skull": 7753,
+ "Guardian": 7754,
+ "Haven": 7755,
+ "Bengal": 7756,
+ "impressive": 7757,
+ "input": 7758,
+ "mixture": 7759,
+ "Warsaw": 7760,
+ "altitude": 7761,
+ "distinction": 7762,
+ "1857": 7763,
+ "collective": 7764,
+ "Annie": 7765,
+ "##ean": 7766,
+ "##bal": 7767,
+ "directions": 7768,
+ "Flying": 7769,
+ "##nic": 7770,
+ "faded": 7771,
+ "##ella": 7772,
+ "contributing": 7773,
+ "##ó": 7774,
+ "employee": 7775,
+ "##lum": 7776,
+ "##yl": 7777,
+ "ruler": 7778,
+ "oriented": 7779,
+ "conductor": 7780,
+ "focusing": 7781,
+ "##die": 7782,
+ "Giants": 7783,
+ "Mills": 7784,
+ "mines": 7785,
+ "Deep": 7786,
+ "curled": 7787,
+ "Jessica": 7788,
+ "guitars": 7789,
+ "Louise": 7790,
+ "procedure": 7791,
+ "Machine": 7792,
+ "failing": 7793,
+ "attendance": 7794,
+ "Nepal": 7795,
+ "Brad": 7796,
+ "Liam": 7797,
+ "tourist": 7798,
+ "exhibited": 7799,
+ "Sophie": 7800,
+ "depicted": 7801,
+ "Shaw": 7802,
+ "Chuck": 7803,
+ "##can": 7804,
+ "expecting": 7805,
+ "challenges": 7806,
+ "##nda": 7807,
+ "equally": 7808,
+ "resignation": 7809,
+ "##logical": 7810,
+ "Tigers": 7811,
+ "loop": 7812,
+ "pitched": 7813,
+ "outdoor": 7814,
+ "reviewed": 7815,
+ "hopes": 7816,
+ "True": 7817,
+ "temporarily": 7818,
+ "Borough": 7819,
+ "torn": 7820,
+ "jerked": 7821,
+ "collect": 7822,
+ "Berkeley": 7823,
+ "Independence": 7824,
+ "cotton": 7825,
+ "retreat": 7826,
+ "campaigns": 7827,
+ "participating": 7828,
+ "Intelligence": 7829,
+ "Heaven": 7830,
+ "##ked": 7831,
+ "situations": 7832,
+ "borough": 7833,
+ "Democrats": 7834,
+ "Harbor": 7835,
+ "##len": 7836,
+ "Liga": 7837,
+ "serial": 7838,
+ "circles": 7839,
+ "fourteen": 7840,
+ "##lot": 7841,
+ "seized": 7842,
+ "filling": 7843,
+ "departments": 7844,
+ "finance": 7845,
+ "absolute": 7846,
+ "Roland": 7847,
+ "Nate": 7848,
+ "floors": 7849,
+ "raced": 7850,
+ "struggling": 7851,
+ "deliver": 7852,
+ "protests": 7853,
+ "##tel": 7854,
+ "Exchange": 7855,
+ "efficient": 7856,
+ "experiments": 7857,
+ "##dar": 7858,
+ "faint": 7859,
+ "3D": 7860,
+ "binding": 7861,
+ "Lions": 7862,
+ "lightly": 7863,
+ "skill": 7864,
+ "proteins": 7865,
+ "difficulties": 7866,
+ "##cal": 7867,
+ "monthly": 7868,
+ "camps": 7869,
+ "flood": 7870,
+ "loves": 7871,
+ "Amanda": 7872,
+ "Commerce": 7873,
+ "##oid": 7874,
+ "##lies": 7875,
+ "elementary": 7876,
+ "##tre": 7877,
+ "organic": 7878,
+ "##stein": 7879,
+ "##ph": 7880,
+ "receives": 7881,
+ "Tech": 7882,
+ "enormous": 7883,
+ "distinctive": 7884,
+ "Joint": 7885,
+ "experiment": 7886,
+ "Circuit": 7887,
+ "citizen": 7888,
+ "##hy": 7889,
+ "shelter": 7890,
+ "ideal": 7891,
+ "practically": 7892,
+ "formula": 7893,
+ "addressed": 7894,
+ "Foster": 7895,
+ "Productions": 7896,
+ "##ax": 7897,
+ "variable": 7898,
+ "punk": 7899,
+ "Voice": 7900,
+ "fastest": 7901,
+ "concentrated": 7902,
+ "##oma": 7903,
+ "##yer": 7904,
+ "stored": 7905,
+ "surrender": 7906,
+ "vary": 7907,
+ "Sergeant": 7908,
+ "Wells": 7909,
+ "ward": 7910,
+ "Wait": 7911,
+ "##ven": 7912,
+ "playoff": 7913,
+ "reducing": 7914,
+ "cavalry": 7915,
+ "##dle": 7916,
+ "Venezuela": 7917,
+ "tissue": 7918,
+ "amounts": 7919,
+ "sweat": 7920,
+ "##we": 7921,
+ "Non": 7922,
+ "##nik": 7923,
+ "beetle": 7924,
+ "##bu": 7925,
+ "##tu": 7926,
+ "Jared": 7927,
+ "Hunt": 7928,
+ "##₂": 7929,
+ "fat": 7930,
+ "Sultan": 7931,
+ "Living": 7932,
+ "Circle": 7933,
+ "Secondary": 7934,
+ "Suddenly": 7935,
+ "reverse": 7936,
+ "##min": 7937,
+ "Travel": 7938,
+ "##bin": 7939,
+ "Lebanon": 7940,
+ "##mas": 7941,
+ "virus": 7942,
+ "Wind": 7943,
+ "dissolved": 7944,
+ "enrolled": 7945,
+ "holiday": 7946,
+ "Keep": 7947,
+ "helicopter": 7948,
+ "Clarke": 7949,
+ "constitutional": 7950,
+ "technologies": 7951,
+ "doubles": 7952,
+ "instructions": 7953,
+ "##ace": 7954,
+ "Azerbaijan": 7955,
+ "##ill": 7956,
+ "occasional": 7957,
+ "frozen": 7958,
+ "trick": 7959,
+ "wiped": 7960,
+ "writings": 7961,
+ "Shanghai": 7962,
+ "preparing": 7963,
+ "challenged": 7964,
+ "mainstream": 7965,
+ "summit": 7966,
+ "180": 7967,
+ "##arian": 7968,
+ "##rating": 7969,
+ "designation": 7970,
+ "##ada": 7971,
+ "revenge": 7972,
+ "filming": 7973,
+ "tightened": 7974,
+ "Miguel": 7975,
+ "Montana": 7976,
+ "reflect": 7977,
+ "celebration": 7978,
+ "bitch": 7979,
+ "flashed": 7980,
+ "signals": 7981,
+ "rounded": 7982,
+ "peoples": 7983,
+ "##tation": 7984,
+ "renowned": 7985,
+ "Google": 7986,
+ "characteristic": 7987,
+ "Campaign": 7988,
+ "sliding": 7989,
+ "##rman": 7990,
+ "usage": 7991,
+ "Record": 7992,
+ "Using": 7993,
+ "woke": 7994,
+ "solutions": 7995,
+ "holes": 7996,
+ "theories": 7997,
+ "logo": 7998,
+ "Protestant": 7999,
+ "relaxed": 8000,
+ "brow": 8001,
+ "nickname": 8002,
+ "Reading": 8003,
+ "marble": 8004,
+ "##tro": 8005,
+ "symptoms": 8006,
+ "Overall": 8007,
+ "capita": 8008,
+ "##ila": 8009,
+ "outbreak": 8010,
+ "revolution": 8011,
+ "deemed": 8012,
+ "Principal": 8013,
+ "Hannah": 8014,
+ "approaches": 8015,
+ "inducted": 8016,
+ "Wellington": 8017,
+ "vulnerable": 8018,
+ "Environmental": 8019,
+ "Drama": 8020,
+ "incumbent": 8021,
+ "Dame": 8022,
+ "1854": 8023,
+ "travels": 8024,
+ "samples": 8025,
+ "accurate": 8026,
+ "physically": 8027,
+ "Sony": 8028,
+ "Nashville": 8029,
+ "##sville": 8030,
+ "##lic": 8031,
+ "##og": 8032,
+ "Producer": 8033,
+ "Lucky": 8034,
+ "tough": 8035,
+ "Stanford": 8036,
+ "resort": 8037,
+ "repeatedly": 8038,
+ "eyebrows": 8039,
+ "Far": 8040,
+ "choir": 8041,
+ "commenced": 8042,
+ "##ep": 8043,
+ "##ridge": 8044,
+ "rage": 8045,
+ "swing": 8046,
+ "sequel": 8047,
+ "heir": 8048,
+ "buses": 8049,
+ "ad": 8050,
+ "Grove": 8051,
+ "##late": 8052,
+ "##rick": 8053,
+ "updated": 8054,
+ "##SA": 8055,
+ "Delaware": 8056,
+ "##fa": 8057,
+ "Athletics": 8058,
+ "warmth": 8059,
+ "Off": 8060,
+ "excitement": 8061,
+ "verse": 8062,
+ "Protection": 8063,
+ "Villa": 8064,
+ "corruption": 8065,
+ "intellectual": 8066,
+ "Jenny": 8067,
+ "##lyn": 8068,
+ "mystery": 8069,
+ "prayer": 8070,
+ "healthy": 8071,
+ "##ologist": 8072,
+ "Bear": 8073,
+ "lab": 8074,
+ "Ernest": 8075,
+ "Remix": 8076,
+ "register": 8077,
+ "basement": 8078,
+ "Montgomery": 8079,
+ "consistent": 8080,
+ "tier": 8081,
+ "1855": 8082,
+ "Preston": 8083,
+ "Brooks": 8084,
+ "##maker": 8085,
+ "vocalist": 8086,
+ "laboratory": 8087,
+ "delayed": 8088,
+ "wheels": 8089,
+ "rope": 8090,
+ "bachelor": 8091,
+ "pitcher": 8092,
+ "Block": 8093,
+ "Nevertheless": 8094,
+ "suspect": 8095,
+ "efficiency": 8096,
+ "Nebraska": 8097,
+ "siege": 8098,
+ "FBI": 8099,
+ "planted": 8100,
+ "##AC": 8101,
+ "Newton": 8102,
+ "breeding": 8103,
+ "##ain": 8104,
+ "eighteen": 8105,
+ "Argentine": 8106,
+ "encounter": 8107,
+ "servant": 8108,
+ "1858": 8109,
+ "elder": 8110,
+ "Shadow": 8111,
+ "Episode": 8112,
+ "fabric": 8113,
+ "doctors": 8114,
+ "survival": 8115,
+ "removal": 8116,
+ "chemistry": 8117,
+ "volunteers": 8118,
+ "Kane": 8119,
+ "variant": 8120,
+ "arrives": 8121,
+ "Eagle": 8122,
+ "Left": 8123,
+ "##fe": 8124,
+ "Jo": 8125,
+ "divorce": 8126,
+ "##ret": 8127,
+ "yesterday": 8128,
+ "Bryan": 8129,
+ "handling": 8130,
+ "diseases": 8131,
+ "customer": 8132,
+ "Sheriff": 8133,
+ "Tiger": 8134,
+ "Harper": 8135,
+ "##oi": 8136,
+ "resting": 8137,
+ "Linda": 8138,
+ "Sheffield": 8139,
+ "gasped": 8140,
+ "sexy": 8141,
+ "economics": 8142,
+ "alien": 8143,
+ "tale": 8144,
+ "footage": 8145,
+ "Liberty": 8146,
+ "yeah": 8147,
+ "fundamental": 8148,
+ "Ground": 8149,
+ "flames": 8150,
+ "Actress": 8151,
+ "photographer": 8152,
+ "Maggie": 8153,
+ "Additional": 8154,
+ "joke": 8155,
+ "custom": 8156,
+ "Survey": 8157,
+ "Abu": 8158,
+ "silk": 8159,
+ "consumption": 8160,
+ "Ellis": 8161,
+ "bread": 8162,
+ "##uous": 8163,
+ "engagement": 8164,
+ "puts": 8165,
+ "Dog": 8166,
+ "##hr": 8167,
+ "poured": 8168,
+ "guilt": 8169,
+ "CDP": 8170,
+ "boxes": 8171,
+ "hardware": 8172,
+ "clenched": 8173,
+ "##cio": 8174,
+ "stem": 8175,
+ "arena": 8176,
+ "extending": 8177,
+ "##com": 8178,
+ "examination": 8179,
+ "Steel": 8180,
+ "encountered": 8181,
+ "revised": 8182,
+ "140": 8183,
+ "picking": 8184,
+ "Car": 8185,
+ "hasn": 8186,
+ "Minor": 8187,
+ "pride": 8188,
+ "Roosevelt": 8189,
+ "boards": 8190,
+ "##mia": 8191,
+ "blocked": 8192,
+ "curious": 8193,
+ "drag": 8194,
+ "narrative": 8195,
+ "brigade": 8196,
+ "Prefecture": 8197,
+ "mysterious": 8198,
+ "namely": 8199,
+ "connects": 8200,
+ "Devil": 8201,
+ "historians": 8202,
+ "CHAPTER": 8203,
+ "quit": 8204,
+ "installation": 8205,
+ "Golf": 8206,
+ "empire": 8207,
+ "elevated": 8208,
+ "##eo": 8209,
+ "releasing": 8210,
+ "Bond": 8211,
+ "##uri": 8212,
+ "harsh": 8213,
+ "ban": 8214,
+ "##BA": 8215,
+ "contracts": 8216,
+ "cloth": 8217,
+ "presents": 8218,
+ "stake": 8219,
+ "chorus": 8220,
+ "##eau": 8221,
+ "swear": 8222,
+ "##mp": 8223,
+ "allies": 8224,
+ "generations": 8225,
+ "Motor": 8226,
+ "meter": 8227,
+ "pen": 8228,
+ "warrior": 8229,
+ "veteran": 8230,
+ "##EC": 8231,
+ "comprehensive": 8232,
+ "missile": 8233,
+ "interaction": 8234,
+ "instruction": 8235,
+ "Renaissance": 8236,
+ "rested": 8237,
+ "Dale": 8238,
+ "fix": 8239,
+ "fluid": 8240,
+ "les": 8241,
+ "investigate": 8242,
+ "loaded": 8243,
+ "widow": 8244,
+ "exhibit": 8245,
+ "artificial": 8246,
+ "select": 8247,
+ "rushing": 8248,
+ "tasks": 8249,
+ "signature": 8250,
+ "nowhere": 8251,
+ "Engineer": 8252,
+ "feared": 8253,
+ "Prague": 8254,
+ "bother": 8255,
+ "extinct": 8256,
+ "gates": 8257,
+ "Bird": 8258,
+ "climbing": 8259,
+ "heels": 8260,
+ "striking": 8261,
+ "artwork": 8262,
+ "hunt": 8263,
+ "awake": 8264,
+ "##hin": 8265,
+ "Formula": 8266,
+ "thereby": 8267,
+ "commitment": 8268,
+ "imprisoned": 8269,
+ "Beyond": 8270,
+ "##MA": 8271,
+ "transformed": 8272,
+ "Agriculture": 8273,
+ "Low": 8274,
+ "Movie": 8275,
+ "radical": 8276,
+ "complicated": 8277,
+ "Yellow": 8278,
+ "Auckland": 8279,
+ "mansion": 8280,
+ "tenth": 8281,
+ "Trevor": 8282,
+ "predecessor": 8283,
+ "##eer": 8284,
+ "disbanded": 8285,
+ "sucked": 8286,
+ "circular": 8287,
+ "witch": 8288,
+ "gaining": 8289,
+ "lean": 8290,
+ "Behind": 8291,
+ "illustrated": 8292,
+ "rang": 8293,
+ "celebrate": 8294,
+ "bike": 8295,
+ "consist": 8296,
+ "framework": 8297,
+ "##cent": 8298,
+ "Shane": 8299,
+ "owns": 8300,
+ "350": 8301,
+ "comprises": 8302,
+ "collaborated": 8303,
+ "colleagues": 8304,
+ "##cast": 8305,
+ "engage": 8306,
+ "fewer": 8307,
+ "##ave": 8308,
+ "1856": 8309,
+ "observation": 8310,
+ "diplomatic": 8311,
+ "legislature": 8312,
+ "improvements": 8313,
+ "Interstate": 8314,
+ "craft": 8315,
+ "MTV": 8316,
+ "martial": 8317,
+ "administered": 8318,
+ "jet": 8319,
+ "approaching": 8320,
+ "permanently": 8321,
+ "attraction": 8322,
+ "manuscript": 8323,
+ "numbered": 8324,
+ "Happy": 8325,
+ "Andrea": 8326,
+ "shallow": 8327,
+ "Gothic": 8328,
+ "Anti": 8329,
+ "##bad": 8330,
+ "improvement": 8331,
+ "trace": 8332,
+ "preserve": 8333,
+ "regardless": 8334,
+ "rode": 8335,
+ "dies": 8336,
+ "achievement": 8337,
+ "maintaining": 8338,
+ "Hamburg": 8339,
+ "spine": 8340,
+ "##air": 8341,
+ "flowing": 8342,
+ "encourage": 8343,
+ "widened": 8344,
+ "posts": 8345,
+ "##bound": 8346,
+ "125": 8347,
+ "Southeast": 8348,
+ "Santiago": 8349,
+ "##bles": 8350,
+ "impression": 8351,
+ "receiver": 8352,
+ "Single": 8353,
+ "closure": 8354,
+ "##unt": 8355,
+ "communist": 8356,
+ "honors": 8357,
+ "Northwest": 8358,
+ "105": 8359,
+ "##ulated": 8360,
+ "cared": 8361,
+ "un": 8362,
+ "hug": 8363,
+ "magnetic": 8364,
+ "seeds": 8365,
+ "topic": 8366,
+ "perceived": 8367,
+ "prey": 8368,
+ "prevented": 8369,
+ "Marvel": 8370,
+ "Eight": 8371,
+ "Michel": 8372,
+ "Transportation": 8373,
+ "rings": 8374,
+ "Gate": 8375,
+ "##gne": 8376,
+ "Byzantine": 8377,
+ "accommodate": 8378,
+ "floating": 8379,
+ "##dor": 8380,
+ "equation": 8381,
+ "ministry": 8382,
+ "##ito": 8383,
+ "##gled": 8384,
+ "Rules": 8385,
+ "earthquake": 8386,
+ "revealing": 8387,
+ "Brother": 8388,
+ "Celtic": 8389,
+ "blew": 8390,
+ "chairs": 8391,
+ "Panama": 8392,
+ "Leon": 8393,
+ "attractive": 8394,
+ "descendants": 8395,
+ "Care": 8396,
+ "Ambassador": 8397,
+ "tours": 8398,
+ "breathed": 8399,
+ "threatening": 8400,
+ "##cho": 8401,
+ "smiles": 8402,
+ "Lt": 8403,
+ "Beginning": 8404,
+ "##iness": 8405,
+ "fake": 8406,
+ "assists": 8407,
+ "fame": 8408,
+ "strings": 8409,
+ "Mobile": 8410,
+ "Liu": 8411,
+ "parks": 8412,
+ "http": 8413,
+ "1852": 8414,
+ "brush": 8415,
+ "Aunt": 8416,
+ "bullet": 8417,
+ "consciousness": 8418,
+ "##sta": 8419,
+ "##ther": 8420,
+ "consequences": 8421,
+ "gather": 8422,
+ "dug": 8423,
+ "1851": 8424,
+ "bridges": 8425,
+ "Doug": 8426,
+ "##sion": 8427,
+ "Artists": 8428,
+ "ignore": 8429,
+ "Carol": 8430,
+ "brilliant": 8431,
+ "radiation": 8432,
+ "temples": 8433,
+ "basin": 8434,
+ "clouds": 8435,
+ "##cted": 8436,
+ "Stevens": 8437,
+ "spite": 8438,
+ "soap": 8439,
+ "consumer": 8440,
+ "Damn": 8441,
+ "Snow": 8442,
+ "recruited": 8443,
+ "##craft": 8444,
+ "Advanced": 8445,
+ "tournaments": 8446,
+ "Quinn": 8447,
+ "undergraduate": 8448,
+ "questioned": 8449,
+ "Palmer": 8450,
+ "Annual": 8451,
+ "Others": 8452,
+ "feeding": 8453,
+ "Spider": 8454,
+ "printing": 8455,
+ "##orn": 8456,
+ "cameras": 8457,
+ "functional": 8458,
+ "Chester": 8459,
+ "readers": 8460,
+ "Alpha": 8461,
+ "universal": 8462,
+ "Faith": 8463,
+ "Brandon": 8464,
+ "François": 8465,
+ "authored": 8466,
+ "Ring": 8467,
+ "el": 8468,
+ "aims": 8469,
+ "athletic": 8470,
+ "possessed": 8471,
+ "Vermont": 8472,
+ "programmes": 8473,
+ "##uck": 8474,
+ "bore": 8475,
+ "Fisher": 8476,
+ "statements": 8477,
+ "shed": 8478,
+ "saxophone": 8479,
+ "neighboring": 8480,
+ "pronounced": 8481,
+ "barrel": 8482,
+ "bags": 8483,
+ "##dge": 8484,
+ "organisations": 8485,
+ "pilots": 8486,
+ "casualties": 8487,
+ "Kenneth": 8488,
+ "##brook": 8489,
+ "silently": 8490,
+ "Malcolm": 8491,
+ "span": 8492,
+ "Essex": 8493,
+ "anchor": 8494,
+ "##hl": 8495,
+ "virtual": 8496,
+ "lessons": 8497,
+ "Henri": 8498,
+ "Trump": 8499,
+ "Page": 8500,
+ "pile": 8501,
+ "locomotive": 8502,
+ "wounds": 8503,
+ "uncomfortable": 8504,
+ "sustained": 8505,
+ "Diana": 8506,
+ "Eagles": 8507,
+ "##pi": 8508,
+ "2000s": 8509,
+ "documented": 8510,
+ "##bel": 8511,
+ "Cassie": 8512,
+ "delay": 8513,
+ "kisses": 8514,
+ "##ines": 8515,
+ "variation": 8516,
+ "##ag": 8517,
+ "growled": 8518,
+ "##mark": 8519,
+ "##ways": 8520,
+ "Leslie": 8521,
+ "studios": 8522,
+ "Friedrich": 8523,
+ "aunt": 8524,
+ "actively": 8525,
+ "armor": 8526,
+ "eaten": 8527,
+ "historically": 8528,
+ "Better": 8529,
+ "purse": 8530,
+ "honey": 8531,
+ "ratings": 8532,
+ "##ée": 8533,
+ "naturally": 8534,
+ "1840": 8535,
+ "peer": 8536,
+ "Kenny": 8537,
+ "Cardinal": 8538,
+ "database": 8539,
+ "Looking": 8540,
+ "runners": 8541,
+ "handsome": 8542,
+ "Double": 8543,
+ "PA": 8544,
+ "##boat": 8545,
+ "##sted": 8546,
+ "protecting": 8547,
+ "##jan": 8548,
+ "Diamond": 8549,
+ "concepts": 8550,
+ "interface": 8551,
+ "##aki": 8552,
+ "Watch": 8553,
+ "Article": 8554,
+ "Columbus": 8555,
+ "dialogue": 8556,
+ "pause": 8557,
+ "##rio": 8558,
+ "extends": 8559,
+ "blanket": 8560,
+ "pulse": 8561,
+ "1853": 8562,
+ "affiliate": 8563,
+ "ladies": 8564,
+ "Ronald": 8565,
+ "counted": 8566,
+ "kills": 8567,
+ "demons": 8568,
+ "##zation": 8569,
+ "Airlines": 8570,
+ "Marco": 8571,
+ "Cat": 8572,
+ "companion": 8573,
+ "mere": 8574,
+ "Yugoslavia": 8575,
+ "Forum": 8576,
+ "Allan": 8577,
+ "pioneer": 8578,
+ "Competition": 8579,
+ "Methodist": 8580,
+ "patent": 8581,
+ "nobody": 8582,
+ "Stockholm": 8583,
+ "##ien": 8584,
+ "regulation": 8585,
+ "##ois": 8586,
+ "accomplished": 8587,
+ "##itive": 8588,
+ "washed": 8589,
+ "sake": 8590,
+ "Vladimir": 8591,
+ "crops": 8592,
+ "prestigious": 8593,
+ "humor": 8594,
+ "Sally": 8595,
+ "labour": 8596,
+ "tributary": 8597,
+ "trap": 8598,
+ "altered": 8599,
+ "examined": 8600,
+ "Mumbai": 8601,
+ "bombing": 8602,
+ "Ash": 8603,
+ "noble": 8604,
+ "suspension": 8605,
+ "ruins": 8606,
+ "##bank": 8607,
+ "spare": 8608,
+ "displays": 8609,
+ "guided": 8610,
+ "dimensional": 8611,
+ "Iraqi": 8612,
+ "##hon": 8613,
+ "sciences": 8614,
+ "Franz": 8615,
+ "relating": 8616,
+ "fence": 8617,
+ "followers": 8618,
+ "Palestine": 8619,
+ "invented": 8620,
+ "proceeded": 8621,
+ "Batman": 8622,
+ "Bradley": 8623,
+ "##yard": 8624,
+ "##ova": 8625,
+ "crystal": 8626,
+ "Kerala": 8627,
+ "##ima": 8628,
+ "shipping": 8629,
+ "handled": 8630,
+ "Want": 8631,
+ "abolished": 8632,
+ "Drew": 8633,
+ "##tter": 8634,
+ "Powell": 8635,
+ "Half": 8636,
+ "##table": 8637,
+ "##cker": 8638,
+ "exhibitions": 8639,
+ "Were": 8640,
+ "assignment": 8641,
+ "assured": 8642,
+ "##rine": 8643,
+ "Indonesian": 8644,
+ "Grammy": 8645,
+ "acknowledged": 8646,
+ "Kylie": 8647,
+ "coaches": 8648,
+ "structural": 8649,
+ "clearing": 8650,
+ "stationed": 8651,
+ "Say": 8652,
+ "Total": 8653,
+ "Rail": 8654,
+ "besides": 8655,
+ "glow": 8656,
+ "threats": 8657,
+ "afford": 8658,
+ "Tree": 8659,
+ "Musical": 8660,
+ "##pp": 8661,
+ "elite": 8662,
+ "centered": 8663,
+ "explore": 8664,
+ "Engineers": 8665,
+ "Stakes": 8666,
+ "Hello": 8667,
+ "tourism": 8668,
+ "severely": 8669,
+ "assessment": 8670,
+ "##tly": 8671,
+ "crack": 8672,
+ "politicians": 8673,
+ "##rrow": 8674,
+ "sheets": 8675,
+ "volunteer": 8676,
+ "##borough": 8677,
+ "##hold": 8678,
+ "announcement": 8679,
+ "recover": 8680,
+ "contribute": 8681,
+ "lungs": 8682,
+ "##ille": 8683,
+ "mainland": 8684,
+ "presentation": 8685,
+ "Johann": 8686,
+ "Writing": 8687,
+ "1849": 8688,
+ "##bird": 8689,
+ "Study": 8690,
+ "Boulevard": 8691,
+ "coached": 8692,
+ "fail": 8693,
+ "airline": 8694,
+ "Congo": 8695,
+ "Plus": 8696,
+ "Syrian": 8697,
+ "introduce": 8698,
+ "ridge": 8699,
+ "Casey": 8700,
+ "manages": 8701,
+ "##fi": 8702,
+ "searched": 8703,
+ "Support": 8704,
+ "succession": 8705,
+ "progressive": 8706,
+ "coup": 8707,
+ "cultures": 8708,
+ "##lessly": 8709,
+ "sensation": 8710,
+ "Cork": 8711,
+ "Elena": 8712,
+ "Sofia": 8713,
+ "Philosophy": 8714,
+ "mini": 8715,
+ "trunk": 8716,
+ "academy": 8717,
+ "Mass": 8718,
+ "Liz": 8719,
+ "practiced": 8720,
+ "Reid": 8721,
+ "##ule": 8722,
+ "satisfied": 8723,
+ "experts": 8724,
+ "Wilhelm": 8725,
+ "Woods": 8726,
+ "invitation": 8727,
+ "Angels": 8728,
+ "calendar": 8729,
+ "joy": 8730,
+ "Sr": 8731,
+ "Dam": 8732,
+ "packed": 8733,
+ "##uan": 8734,
+ "bastard": 8735,
+ "Workers": 8736,
+ "broadcasts": 8737,
+ "logic": 8738,
+ "cooking": 8739,
+ "backward": 8740,
+ "##ack": 8741,
+ "Chen": 8742,
+ "creates": 8743,
+ "enzyme": 8744,
+ "##xi": 8745,
+ "Davies": 8746,
+ "aviation": 8747,
+ "VII": 8748,
+ "Conservation": 8749,
+ "fucking": 8750,
+ "Knights": 8751,
+ "##kan": 8752,
+ "requiring": 8753,
+ "hectares": 8754,
+ "wars": 8755,
+ "ate": 8756,
+ "##box": 8757,
+ "Mind": 8758,
+ "desired": 8759,
+ "oak": 8760,
+ "absorbed": 8761,
+ "Really": 8762,
+ "Vietnamese": 8763,
+ "Paulo": 8764,
+ "athlete": 8765,
+ "##car": 8766,
+ "##eth": 8767,
+ "Talk": 8768,
+ "Wu": 8769,
+ "##cks": 8770,
+ "survivors": 8771,
+ "Yang": 8772,
+ "Joel": 8773,
+ "Almost": 8774,
+ "Holmes": 8775,
+ "Armed": 8776,
+ "Joshua": 8777,
+ "priests": 8778,
+ "discontinued": 8779,
+ "##sey": 8780,
+ "blond": 8781,
+ "Rolling": 8782,
+ "suggesting": 8783,
+ "CA": 8784,
+ "clay": 8785,
+ "exterior": 8786,
+ "Scientific": 8787,
+ "##sive": 8788,
+ "Giovanni": 8789,
+ "Hi": 8790,
+ "farther": 8791,
+ "contents": 8792,
+ "Winners": 8793,
+ "animation": 8794,
+ "neutral": 8795,
+ "mall": 8796,
+ "Notes": 8797,
+ "layers": 8798,
+ "professionals": 8799,
+ "Armstrong": 8800,
+ "Against": 8801,
+ "Piano": 8802,
+ "involve": 8803,
+ "monitor": 8804,
+ "angel": 8805,
+ "parked": 8806,
+ "bears": 8807,
+ "seated": 8808,
+ "feat": 8809,
+ "beliefs": 8810,
+ "##kers": 8811,
+ "Version": 8812,
+ "suffer": 8813,
+ "##ceae": 8814,
+ "guidance": 8815,
+ "##eur": 8816,
+ "honored": 8817,
+ "raid": 8818,
+ "alarm": 8819,
+ "Glen": 8820,
+ "Ellen": 8821,
+ "Jamaica": 8822,
+ "trio": 8823,
+ "enabled": 8824,
+ "##ils": 8825,
+ "procedures": 8826,
+ "##hus": 8827,
+ "moderate": 8828,
+ "upstairs": 8829,
+ "##ses": 8830,
+ "torture": 8831,
+ "Georgian": 8832,
+ "rebellion": 8833,
+ "Fernando": 8834,
+ "Nice": 8835,
+ "##are": 8836,
+ "Aires": 8837,
+ "Campus": 8838,
+ "beast": 8839,
+ "##hing": 8840,
+ "1847": 8841,
+ "##FA": 8842,
+ "Isle": 8843,
+ "##logist": 8844,
+ "Princeton": 8845,
+ "cathedral": 8846,
+ "Oakland": 8847,
+ "Solomon": 8848,
+ "##tto": 8849,
+ "Milwaukee": 8850,
+ "upcoming": 8851,
+ "midfielder": 8852,
+ "Neither": 8853,
+ "sacred": 8854,
+ "Eyes": 8855,
+ "appreciate": 8856,
+ "Brunswick": 8857,
+ "secrets": 8858,
+ "Rice": 8859,
+ "Somerset": 8860,
+ "Chancellor": 8861,
+ "Curtis": 8862,
+ "##gel": 8863,
+ "Rich": 8864,
+ "separation": 8865,
+ "grid": 8866,
+ "##los": 8867,
+ "##bon": 8868,
+ "urge": 8869,
+ "##ees": 8870,
+ "##ree": 8871,
+ "freight": 8872,
+ "towers": 8873,
+ "psychology": 8874,
+ "requirement": 8875,
+ "dollar": 8876,
+ "##fall": 8877,
+ "##sman": 8878,
+ "exile": 8879,
+ "tomb": 8880,
+ "Salt": 8881,
+ "Stefan": 8882,
+ "Buenos": 8883,
+ "Revival": 8884,
+ "Porter": 8885,
+ "tender": 8886,
+ "diesel": 8887,
+ "chocolate": 8888,
+ "Eugene": 8889,
+ "Legion": 8890,
+ "Laboratory": 8891,
+ "sheep": 8892,
+ "arched": 8893,
+ "hospitals": 8894,
+ "orbit": 8895,
+ "Full": 8896,
+ "##hall": 8897,
+ "drinks": 8898,
+ "ripped": 8899,
+ "##RS": 8900,
+ "tense": 8901,
+ "Hank": 8902,
+ "leagues": 8903,
+ "##nberg": 8904,
+ "PlayStation": 8905,
+ "fool": 8906,
+ "Punjab": 8907,
+ "relatives": 8908,
+ "Comedy": 8909,
+ "sur": 8910,
+ "1846": 8911,
+ "Tonight": 8912,
+ "Sox": 8913,
+ "##if": 8914,
+ "Rabbi": 8915,
+ "org": 8916,
+ "speaks": 8917,
+ "institute": 8918,
+ "defender": 8919,
+ "painful": 8920,
+ "wishes": 8921,
+ "Weekly": 8922,
+ "literacy": 8923,
+ "portions": 8924,
+ "snake": 8925,
+ "item": 8926,
+ "deals": 8927,
+ "##tum": 8928,
+ "autumn": 8929,
+ "sharply": 8930,
+ "reforms": 8931,
+ "thighs": 8932,
+ "prototype": 8933,
+ "##ition": 8934,
+ "argues": 8935,
+ "disorder": 8936,
+ "Physics": 8937,
+ "terror": 8938,
+ "provisions": 8939,
+ "refugees": 8940,
+ "predominantly": 8941,
+ "independently": 8942,
+ "march": 8943,
+ "##graphy": 8944,
+ "Arabia": 8945,
+ "Andrews": 8946,
+ "Bus": 8947,
+ "Money": 8948,
+ "drops": 8949,
+ "##zar": 8950,
+ "pistol": 8951,
+ "matrix": 8952,
+ "revolutionary": 8953,
+ "##ust": 8954,
+ "Starting": 8955,
+ "##ptic": 8956,
+ "Oak": 8957,
+ "Monica": 8958,
+ "##ides": 8959,
+ "servants": 8960,
+ "##hed": 8961,
+ "archaeological": 8962,
+ "divorced": 8963,
+ "rocket": 8964,
+ "enjoying": 8965,
+ "fires": 8966,
+ "##nel": 8967,
+ "assembled": 8968,
+ "qualification": 8969,
+ "retiring": 8970,
+ "##fied": 8971,
+ "Distinguished": 8972,
+ "handful": 8973,
+ "infection": 8974,
+ "Durham": 8975,
+ "##itz": 8976,
+ "fortune": 8977,
+ "renewed": 8978,
+ "Chelsea": 8979,
+ "##sley": 8980,
+ "curved": 8981,
+ "gesture": 8982,
+ "retain": 8983,
+ "exhausted": 8984,
+ "##ifying": 8985,
+ "Perth": 8986,
+ "jumping": 8987,
+ "Palestinian": 8988,
+ "Simpson": 8989,
+ "colonies": 8990,
+ "steal": 8991,
+ "##chy": 8992,
+ "corners": 8993,
+ "Finn": 8994,
+ "arguing": 8995,
+ "Martha": 8996,
+ "##var": 8997,
+ "Betty": 8998,
+ "emerging": 8999,
+ "Heights": 9000,
+ "Hindi": 9001,
+ "Manila": 9002,
+ "pianist": 9003,
+ "founders": 9004,
+ "regret": 9005,
+ "Napoleon": 9006,
+ "elbow": 9007,
+ "overhead": 9008,
+ "bold": 9009,
+ "praise": 9010,
+ "humanity": 9011,
+ "##ori": 9012,
+ "Revolutionary": 9013,
+ "##ere": 9014,
+ "fur": 9015,
+ "##ole": 9016,
+ "Ashley": 9017,
+ "Official": 9018,
+ "##rm": 9019,
+ "lovely": 9020,
+ "Architecture": 9021,
+ "##sch": 9022,
+ "Baronet": 9023,
+ "virtually": 9024,
+ "##OS": 9025,
+ "descended": 9026,
+ "immigration": 9027,
+ "##das": 9028,
+ "##kes": 9029,
+ "Holly": 9030,
+ "Wednesday": 9031,
+ "maintains": 9032,
+ "theatrical": 9033,
+ "Evan": 9034,
+ "Gardens": 9035,
+ "citing": 9036,
+ "##gia": 9037,
+ "segments": 9038,
+ "Bailey": 9039,
+ "Ghost": 9040,
+ "##city": 9041,
+ "governing": 9042,
+ "graphics": 9043,
+ "##ined": 9044,
+ "privately": 9045,
+ "potentially": 9046,
+ "transformation": 9047,
+ "Crystal": 9048,
+ "Cabinet": 9049,
+ "sacrifice": 9050,
+ "hesitated": 9051,
+ "mud": 9052,
+ "Apollo": 9053,
+ "Desert": 9054,
+ "bin": 9055,
+ "victories": 9056,
+ "Editor": 9057,
+ "Railways": 9058,
+ "Web": 9059,
+ "Case": 9060,
+ "tourists": 9061,
+ "Brussels": 9062,
+ "Franco": 9063,
+ "compiled": 9064,
+ "topped": 9065,
+ "Gene": 9066,
+ "engineers": 9067,
+ "commentary": 9068,
+ "egg": 9069,
+ "escort": 9070,
+ "nerve": 9071,
+ "arch": 9072,
+ "necessarily": 9073,
+ "frustration": 9074,
+ "Michelle": 9075,
+ "democracy": 9076,
+ "genes": 9077,
+ "Facebook": 9078,
+ "halfway": 9079,
+ "##ient": 9080,
+ "102": 9081,
+ "flipped": 9082,
+ "Won": 9083,
+ "##mit": 9084,
+ "NASA": 9085,
+ "Lynn": 9086,
+ "Provincial": 9087,
+ "ambassador": 9088,
+ "Inspector": 9089,
+ "glared": 9090,
+ "Change": 9091,
+ "McDonald": 9092,
+ "developments": 9093,
+ "tucked": 9094,
+ "noting": 9095,
+ "Gibson": 9096,
+ "circulation": 9097,
+ "dubbed": 9098,
+ "armies": 9099,
+ "resource": 9100,
+ "Headquarters": 9101,
+ "##iest": 9102,
+ "Mia": 9103,
+ "Albanian": 9104,
+ "Oil": 9105,
+ "Albums": 9106,
+ "excuse": 9107,
+ "intervention": 9108,
+ "Grande": 9109,
+ "Hugo": 9110,
+ "integration": 9111,
+ "civilians": 9112,
+ "depends": 9113,
+ "reserves": 9114,
+ "Dee": 9115,
+ "compositions": 9116,
+ "identification": 9117,
+ "restrictions": 9118,
+ "quarterback": 9119,
+ "Miranda": 9120,
+ "Universe": 9121,
+ "favourite": 9122,
+ "ranges": 9123,
+ "hint": 9124,
+ "loyal": 9125,
+ "Op": 9126,
+ "entity": 9127,
+ "Manual": 9128,
+ "quoted": 9129,
+ "dealt": 9130,
+ "specialist": 9131,
+ "Zhang": 9132,
+ "download": 9133,
+ "Westminster": 9134,
+ "Rebecca": 9135,
+ "streams": 9136,
+ "Anglican": 9137,
+ "variations": 9138,
+ "Mine": 9139,
+ "detective": 9140,
+ "Films": 9141,
+ "reserved": 9142,
+ "##oke": 9143,
+ "##key": 9144,
+ "sailing": 9145,
+ "##gger": 9146,
+ "expanding": 9147,
+ "recall": 9148,
+ "discovers": 9149,
+ "particles": 9150,
+ "behaviour": 9151,
+ "Gavin": 9152,
+ "blank": 9153,
+ "permit": 9154,
+ "Java": 9155,
+ "Fraser": 9156,
+ "Pass": 9157,
+ "##non": 9158,
+ "##TA": 9159,
+ "panels": 9160,
+ "statistics": 9161,
+ "notion": 9162,
+ "courage": 9163,
+ "dare": 9164,
+ "venues": 9165,
+ "##roy": 9166,
+ "Box": 9167,
+ "Newport": 9168,
+ "travelling": 9169,
+ "Thursday": 9170,
+ "warriors": 9171,
+ "Glenn": 9172,
+ "criteria": 9173,
+ "360": 9174,
+ "mutual": 9175,
+ "restore": 9176,
+ "varied": 9177,
+ "bitter": 9178,
+ "Katherine": 9179,
+ "##lant": 9180,
+ "ritual": 9181,
+ "bits": 9182,
+ "##à": 9183,
+ "Henderson": 9184,
+ "trips": 9185,
+ "Richardson": 9186,
+ "Detective": 9187,
+ "curse": 9188,
+ "psychological": 9189,
+ "Il": 9190,
+ "midnight": 9191,
+ "streak": 9192,
+ "facts": 9193,
+ "Dawn": 9194,
+ "Indies": 9195,
+ "Edmund": 9196,
+ "roster": 9197,
+ "Gen": 9198,
+ "##nation": 9199,
+ "1830": 9200,
+ "congregation": 9201,
+ "shaft": 9202,
+ "##ically": 9203,
+ "##mination": 9204,
+ "Indianapolis": 9205,
+ "Sussex": 9206,
+ "loving": 9207,
+ "##bit": 9208,
+ "sounding": 9209,
+ "horrible": 9210,
+ "Continental": 9211,
+ "Griffin": 9212,
+ "advised": 9213,
+ "magical": 9214,
+ "millions": 9215,
+ "##date": 9216,
+ "1845": 9217,
+ "Safety": 9218,
+ "lifting": 9219,
+ "determination": 9220,
+ "valid": 9221,
+ "dialect": 9222,
+ "Penn": 9223,
+ "Know": 9224,
+ "triple": 9225,
+ "avoided": 9226,
+ "dancer": 9227,
+ "judgment": 9228,
+ "sixty": 9229,
+ "farmer": 9230,
+ "lakes": 9231,
+ "blast": 9232,
+ "aggressive": 9233,
+ "Abby": 9234,
+ "tag": 9235,
+ "chains": 9236,
+ "inscription": 9237,
+ "##nn": 9238,
+ "conducting": 9239,
+ "Scout": 9240,
+ "buying": 9241,
+ "##wich": 9242,
+ "spreading": 9243,
+ "##OC": 9244,
+ "array": 9245,
+ "hurried": 9246,
+ "Environment": 9247,
+ "improving": 9248,
+ "prompted": 9249,
+ "fierce": 9250,
+ "Taking": 9251,
+ "Away": 9252,
+ "tune": 9253,
+ "pissed": 9254,
+ "Bull": 9255,
+ "catching": 9256,
+ "##ying": 9257,
+ "eyebrow": 9258,
+ "metropolitan": 9259,
+ "terrain": 9260,
+ "##rel": 9261,
+ "Lodge": 9262,
+ "manufacturers": 9263,
+ "creator": 9264,
+ "##etic": 9265,
+ "happiness": 9266,
+ "ports": 9267,
+ "##ners": 9268,
+ "Relations": 9269,
+ "fortress": 9270,
+ "targeted": 9271,
+ "##ST": 9272,
+ "allegedly": 9273,
+ "blues": 9274,
+ "##osa": 9275,
+ "Bosnia": 9276,
+ "##dom": 9277,
+ "burial": 9278,
+ "similarly": 9279,
+ "stranger": 9280,
+ "pursued": 9281,
+ "symbols": 9282,
+ "rebels": 9283,
+ "reflection": 9284,
+ "routine": 9285,
+ "traced": 9286,
+ "indoor": 9287,
+ "eventual": 9288,
+ "##ska": 9289,
+ "##ão": 9290,
+ "##una": 9291,
+ "MD": 9292,
+ "##phone": 9293,
+ "oh": 9294,
+ "grants": 9295,
+ "Reynolds": 9296,
+ "rid": 9297,
+ "operators": 9298,
+ "##nus": 9299,
+ "Joey": 9300,
+ "vital": 9301,
+ "siblings": 9302,
+ "keyboard": 9303,
+ "br": 9304,
+ "removing": 9305,
+ "societies": 9306,
+ "drives": 9307,
+ "solely": 9308,
+ "princess": 9309,
+ "lighter": 9310,
+ "Various": 9311,
+ "Cavalry": 9312,
+ "believing": 9313,
+ "SC": 9314,
+ "underwent": 9315,
+ "relay": 9316,
+ "smelled": 9317,
+ "syndrome": 9318,
+ "welfare": 9319,
+ "authorized": 9320,
+ "seemingly": 9321,
+ "Hard": 9322,
+ "chicken": 9323,
+ "##rina": 9324,
+ "Ages": 9325,
+ "Bo": 9326,
+ "democratic": 9327,
+ "barn": 9328,
+ "Eye": 9329,
+ "shorts": 9330,
+ "##coming": 9331,
+ "##hand": 9332,
+ "disappointed": 9333,
+ "unexpected": 9334,
+ "centres": 9335,
+ "Exhibition": 9336,
+ "Stories": 9337,
+ "Site": 9338,
+ "banking": 9339,
+ "accidentally": 9340,
+ "Agent": 9341,
+ "conjunction": 9342,
+ "André": 9343,
+ "Chloe": 9344,
+ "resist": 9345,
+ "width": 9346,
+ "Queens": 9347,
+ "provision": 9348,
+ "##art": 9349,
+ "Melissa": 9350,
+ "Honorary": 9351,
+ "Del": 9352,
+ "prefer": 9353,
+ "abruptly": 9354,
+ "duration": 9355,
+ "##vis": 9356,
+ "Glass": 9357,
+ "enlisted": 9358,
+ "##ado": 9359,
+ "discipline": 9360,
+ "Sisters": 9361,
+ "carriage": 9362,
+ "##ctor": 9363,
+ "##sburg": 9364,
+ "Lancashire": 9365,
+ "log": 9366,
+ "fuck": 9367,
+ "##iz": 9368,
+ "closet": 9369,
+ "collecting": 9370,
+ "holy": 9371,
+ "rape": 9372,
+ "trusted": 9373,
+ "cleaning": 9374,
+ "inhabited": 9375,
+ "Rocky": 9376,
+ "104": 9377,
+ "editorial": 9378,
+ "##yu": 9379,
+ "##ju": 9380,
+ "succeed": 9381,
+ "strict": 9382,
+ "Cuban": 9383,
+ "##iya": 9384,
+ "Bronze": 9385,
+ "outcome": 9386,
+ "##ifies": 9387,
+ "##set": 9388,
+ "corps": 9389,
+ "Hero": 9390,
+ "barrier": 9391,
+ "Kumar": 9392,
+ "groaned": 9393,
+ "Nina": 9394,
+ "Burton": 9395,
+ "enable": 9396,
+ "stability": 9397,
+ "Milton": 9398,
+ "knots": 9399,
+ "##ination": 9400,
+ "slavery": 9401,
+ "##borg": 9402,
+ "curriculum": 9403,
+ "trailer": 9404,
+ "warfare": 9405,
+ "Dante": 9406,
+ "Edgar": 9407,
+ "revival": 9408,
+ "Copenhagen": 9409,
+ "define": 9410,
+ "advocate": 9411,
+ "Garrett": 9412,
+ "Luther": 9413,
+ "overcome": 9414,
+ "pipe": 9415,
+ "750": 9416,
+ "construct": 9417,
+ "Scotia": 9418,
+ "kings": 9419,
+ "flooding": 9420,
+ "##hard": 9421,
+ "Ferdinand": 9422,
+ "Felix": 9423,
+ "forgot": 9424,
+ "Fish": 9425,
+ "Kurt": 9426,
+ "elaborate": 9427,
+ "##BC": 9428,
+ "graphic": 9429,
+ "gripped": 9430,
+ "colonel": 9431,
+ "Sophia": 9432,
+ "Advisory": 9433,
+ "Self": 9434,
+ "##uff": 9435,
+ "##lio": 9436,
+ "monitoring": 9437,
+ "seal": 9438,
+ "senses": 9439,
+ "rises": 9440,
+ "peaceful": 9441,
+ "journals": 9442,
+ "1837": 9443,
+ "checking": 9444,
+ "legendary": 9445,
+ "Ghana": 9446,
+ "##power": 9447,
+ "ammunition": 9448,
+ "Rosa": 9449,
+ "Richards": 9450,
+ "nineteenth": 9451,
+ "ferry": 9452,
+ "aggregate": 9453,
+ "Troy": 9454,
+ "inter": 9455,
+ "##wall": 9456,
+ "Triple": 9457,
+ "steep": 9458,
+ "tent": 9459,
+ "Cyprus": 9460,
+ "1844": 9461,
+ "##woman": 9462,
+ "commanding": 9463,
+ "farms": 9464,
+ "doi": 9465,
+ "navy": 9466,
+ "specified": 9467,
+ "na": 9468,
+ "cricketer": 9469,
+ "transported": 9470,
+ "Think": 9471,
+ "comprising": 9472,
+ "grateful": 9473,
+ "solve": 9474,
+ "##core": 9475,
+ "beings": 9476,
+ "clerk": 9477,
+ "grain": 9478,
+ "vector": 9479,
+ "discrimination": 9480,
+ "##TC": 9481,
+ "Katie": 9482,
+ "reasonable": 9483,
+ "drawings": 9484,
+ "veins": 9485,
+ "consideration": 9486,
+ "Monroe": 9487,
+ "repeat": 9488,
+ "breed": 9489,
+ "dried": 9490,
+ "witnessed": 9491,
+ "ordained": 9492,
+ "Current": 9493,
+ "spirits": 9494,
+ "remarkable": 9495,
+ "consultant": 9496,
+ "urged": 9497,
+ "Remember": 9498,
+ "anime": 9499,
+ "singers": 9500,
+ "phenomenon": 9501,
+ "Rhode": 9502,
+ "Carlo": 9503,
+ "demanding": 9504,
+ "findings": 9505,
+ "manual": 9506,
+ "varying": 9507,
+ "Fellowship": 9508,
+ "generate": 9509,
+ "safely": 9510,
+ "heated": 9511,
+ "withdrawn": 9512,
+ "##ao": 9513,
+ "headquartered": 9514,
+ "##zon": 9515,
+ "##lav": 9516,
+ "##ency": 9517,
+ "Col": 9518,
+ "Memphis": 9519,
+ "imposed": 9520,
+ "rivals": 9521,
+ "Planet": 9522,
+ "healing": 9523,
+ "##hs": 9524,
+ "ensemble": 9525,
+ "Warriors": 9526,
+ "##bone": 9527,
+ "cult": 9528,
+ "Frankfurt": 9529,
+ "##HL": 9530,
+ "diversity": 9531,
+ "Gerald": 9532,
+ "intermediate": 9533,
+ "##izes": 9534,
+ "reactions": 9535,
+ "Sister": 9536,
+ "##ously": 9537,
+ "##lica": 9538,
+ "quantum": 9539,
+ "awkward": 9540,
+ "mentions": 9541,
+ "pursuit": 9542,
+ "##ography": 9543,
+ "varies": 9544,
+ "profession": 9545,
+ "molecular": 9546,
+ "consequence": 9547,
+ "lectures": 9548,
+ "cracked": 9549,
+ "103": 9550,
+ "slowed": 9551,
+ "##tsu": 9552,
+ "cheese": 9553,
+ "upgraded": 9554,
+ "suite": 9555,
+ "substance": 9556,
+ "Kingston": 9557,
+ "1800": 9558,
+ "Idaho": 9559,
+ "Theory": 9560,
+ "##een": 9561,
+ "ain": 9562,
+ "Carson": 9563,
+ "Molly": 9564,
+ "##OR": 9565,
+ "configuration": 9566,
+ "Whitney": 9567,
+ "reads": 9568,
+ "audiences": 9569,
+ "##tie": 9570,
+ "Geneva": 9571,
+ "Outside": 9572,
+ "##nen": 9573,
+ "##had": 9574,
+ "transit": 9575,
+ "volleyball": 9576,
+ "Randy": 9577,
+ "Chad": 9578,
+ "rubber": 9579,
+ "motorcycle": 9580,
+ "respected": 9581,
+ "eager": 9582,
+ "Level": 9583,
+ "coin": 9584,
+ "##lets": 9585,
+ "neighbouring": 9586,
+ "##wski": 9587,
+ "confident": 9588,
+ "##cious": 9589,
+ "poll": 9590,
+ "uncertain": 9591,
+ "punch": 9592,
+ "thesis": 9593,
+ "Tucker": 9594,
+ "IATA": 9595,
+ "Alec": 9596,
+ "##ographic": 9597,
+ "##law": 9598,
+ "1841": 9599,
+ "desperately": 9600,
+ "1812": 9601,
+ "Lithuania": 9602,
+ "accent": 9603,
+ "Cox": 9604,
+ "lightning": 9605,
+ "skirt": 9606,
+ "##load": 9607,
+ "Burns": 9608,
+ "Dynasty": 9609,
+ "##ug": 9610,
+ "chapters": 9611,
+ "Working": 9612,
+ "dense": 9613,
+ "Morocco": 9614,
+ "##kins": 9615,
+ "casting": 9616,
+ "Set": 9617,
+ "activated": 9618,
+ "oral": 9619,
+ "Brien": 9620,
+ "horn": 9621,
+ "HIV": 9622,
+ "dawn": 9623,
+ "stumbled": 9624,
+ "altar": 9625,
+ "tore": 9626,
+ "considerably": 9627,
+ "Nicole": 9628,
+ "interchange": 9629,
+ "registration": 9630,
+ "biography": 9631,
+ "Hull": 9632,
+ "Stan": 9633,
+ "bulk": 9634,
+ "consent": 9635,
+ "Pierce": 9636,
+ "##ER": 9637,
+ "Fifth": 9638,
+ "marched": 9639,
+ "terrorist": 9640,
+ "##piece": 9641,
+ "##itt": 9642,
+ "Presidential": 9643,
+ "Heather": 9644,
+ "staged": 9645,
+ "Plant": 9646,
+ "relegation": 9647,
+ "sporting": 9648,
+ "joins": 9649,
+ "##ced": 9650,
+ "Pakistani": 9651,
+ "dynamic": 9652,
+ "Heat": 9653,
+ "##lf": 9654,
+ "ourselves": 9655,
+ "Except": 9656,
+ "Elliott": 9657,
+ "nationally": 9658,
+ "goddess": 9659,
+ "investors": 9660,
+ "Burke": 9661,
+ "Jackie": 9662,
+ "##ā": 9663,
+ "##RA": 9664,
+ "Tristan": 9665,
+ "Associate": 9666,
+ "Tuesday": 9667,
+ "scope": 9668,
+ "Near": 9669,
+ "bunch": 9670,
+ "##abad": 9671,
+ "##ben": 9672,
+ "sunlight": 9673,
+ "##aire": 9674,
+ "manga": 9675,
+ "Willie": 9676,
+ "trucks": 9677,
+ "boarding": 9678,
+ "Lion": 9679,
+ "lawsuit": 9680,
+ "Learning": 9681,
+ "Der": 9682,
+ "pounding": 9683,
+ "awful": 9684,
+ "##mine": 9685,
+ "IT": 9686,
+ "Legend": 9687,
+ "romance": 9688,
+ "Serie": 9689,
+ "AC": 9690,
+ "gut": 9691,
+ "precious": 9692,
+ "Robertson": 9693,
+ "hometown": 9694,
+ "realm": 9695,
+ "Guards": 9696,
+ "Tag": 9697,
+ "batting": 9698,
+ "##vre": 9699,
+ "halt": 9700,
+ "conscious": 9701,
+ "1838": 9702,
+ "acquire": 9703,
+ "collar": 9704,
+ "##gg": 9705,
+ "##ops": 9706,
+ "Herald": 9707,
+ "nationwide": 9708,
+ "citizenship": 9709,
+ "Aircraft": 9710,
+ "decrease": 9711,
+ "em": 9712,
+ "Fiction": 9713,
+ "Female": 9714,
+ "corporation": 9715,
+ "Located": 9716,
+ "##ip": 9717,
+ "fights": 9718,
+ "unconscious": 9719,
+ "Tampa": 9720,
+ "Poetry": 9721,
+ "lobby": 9722,
+ "Malta": 9723,
+ "##sar": 9724,
+ "##bie": 9725,
+ "layout": 9726,
+ "Tate": 9727,
+ "reader": 9728,
+ "stained": 9729,
+ "##bre": 9730,
+ "##rst": 9731,
+ "##ulate": 9732,
+ "loudly": 9733,
+ "Eva": 9734,
+ "Cohen": 9735,
+ "exploded": 9736,
+ "Merit": 9737,
+ "Maya": 9738,
+ "##rable": 9739,
+ "Rovers": 9740,
+ "##IC": 9741,
+ "Morrison": 9742,
+ "Should": 9743,
+ "vinyl": 9744,
+ "##mie": 9745,
+ "onwards": 9746,
+ "##gie": 9747,
+ "vicinity": 9748,
+ "Wildlife": 9749,
+ "probability": 9750,
+ "Mar": 9751,
+ "Barnes": 9752,
+ "##ook": 9753,
+ "spinning": 9754,
+ "Moses": 9755,
+ "##vie": 9756,
+ "Surrey": 9757,
+ "Planning": 9758,
+ "conferences": 9759,
+ "protective": 9760,
+ "Plaza": 9761,
+ "deny": 9762,
+ "Canterbury": 9763,
+ "manor": 9764,
+ "Estate": 9765,
+ "tilted": 9766,
+ "comics": 9767,
+ "IBM": 9768,
+ "destroying": 9769,
+ "server": 9770,
+ "Dorothy": 9771,
+ "##horn": 9772,
+ "Oslo": 9773,
+ "lesser": 9774,
+ "heaven": 9775,
+ "Marshal": 9776,
+ "scales": 9777,
+ "strikes": 9778,
+ "##ath": 9779,
+ "firms": 9780,
+ "attract": 9781,
+ "##BS": 9782,
+ "controlling": 9783,
+ "Bradford": 9784,
+ "southeastern": 9785,
+ "Amazon": 9786,
+ "Travis": 9787,
+ "Janet": 9788,
+ "governed": 9789,
+ "1842": 9790,
+ "Train": 9791,
+ "Holden": 9792,
+ "bleeding": 9793,
+ "gifts": 9794,
+ "rent": 9795,
+ "1839": 9796,
+ "palms": 9797,
+ "##ū": 9798,
+ "judicial": 9799,
+ "Ho": 9800,
+ "Finals": 9801,
+ "conflicts": 9802,
+ "unlikely": 9803,
+ "draws": 9804,
+ "##cies": 9805,
+ "compensation": 9806,
+ "adds": 9807,
+ "elderly": 9808,
+ "Anton": 9809,
+ "lasting": 9810,
+ "Nintendo": 9811,
+ "codes": 9812,
+ "ministers": 9813,
+ "pot": 9814,
+ "associations": 9815,
+ "capabilities": 9816,
+ "##cht": 9817,
+ "libraries": 9818,
+ "##sie": 9819,
+ "chances": 9820,
+ "performers": 9821,
+ "runway": 9822,
+ "##af": 9823,
+ "##nder": 9824,
+ "Mid": 9825,
+ "Vocals": 9826,
+ "##uch": 9827,
+ "##eon": 9828,
+ "interpreted": 9829,
+ "priority": 9830,
+ "Uganda": 9831,
+ "ruined": 9832,
+ "Mathematics": 9833,
+ "cook": 9834,
+ "AFL": 9835,
+ "Lutheran": 9836,
+ "AIDS": 9837,
+ "Capitol": 9838,
+ "chase": 9839,
+ "axis": 9840,
+ "Moreover": 9841,
+ "María": 9842,
+ "Saxon": 9843,
+ "storyline": 9844,
+ "##ffed": 9845,
+ "Tears": 9846,
+ "Kid": 9847,
+ "cent": 9848,
+ "colours": 9849,
+ "Sex": 9850,
+ "##long": 9851,
+ "pm": 9852,
+ "blonde": 9853,
+ "Edwin": 9854,
+ "CE": 9855,
+ "diocese": 9856,
+ "##ents": 9857,
+ "##boy": 9858,
+ "Inn": 9859,
+ "##ller": 9860,
+ "Saskatchewan": 9861,
+ "##kh": 9862,
+ "stepping": 9863,
+ "Windsor": 9864,
+ "##oka": 9865,
+ "##eri": 9866,
+ "Xavier": 9867,
+ "Resources": 9868,
+ "1843": 9869,
+ "##top": 9870,
+ "##rad": 9871,
+ "##lls": 9872,
+ "Testament": 9873,
+ "poorly": 9874,
+ "1836": 9875,
+ "drifted": 9876,
+ "slope": 9877,
+ "CIA": 9878,
+ "remix": 9879,
+ "Lords": 9880,
+ "mature": 9881,
+ "hosting": 9882,
+ "diamond": 9883,
+ "beds": 9884,
+ "##ncies": 9885,
+ "luxury": 9886,
+ "trigger": 9887,
+ "##lier": 9888,
+ "preliminary": 9889,
+ "hybrid": 9890,
+ "journalists": 9891,
+ "Enterprise": 9892,
+ "proven": 9893,
+ "expelled": 9894,
+ "insects": 9895,
+ "Beautiful": 9896,
+ "lifestyle": 9897,
+ "vanished": 9898,
+ "##ake": 9899,
+ "##ander": 9900,
+ "matching": 9901,
+ "surfaces": 9902,
+ "Dominican": 9903,
+ "Kids": 9904,
+ "referendum": 9905,
+ "Orlando": 9906,
+ "Truth": 9907,
+ "Sandy": 9908,
+ "privacy": 9909,
+ "Calgary": 9910,
+ "Speaker": 9911,
+ "sts": 9912,
+ "Nobody": 9913,
+ "shifting": 9914,
+ "##gers": 9915,
+ "Roll": 9916,
+ "Armenia": 9917,
+ "Hand": 9918,
+ "##ES": 9919,
+ "106": 9920,
+ "##ont": 9921,
+ "Guild": 9922,
+ "larvae": 9923,
+ "Stock": 9924,
+ "flame": 9925,
+ "gravity": 9926,
+ "enhanced": 9927,
+ "Marion": 9928,
+ "surely": 9929,
+ "##tering": 9930,
+ "Tales": 9931,
+ "algorithm": 9932,
+ "Emmy": 9933,
+ "darker": 9934,
+ "VIII": 9935,
+ "##lash": 9936,
+ "hamlet": 9937,
+ "deliberately": 9938,
+ "occurring": 9939,
+ "choices": 9940,
+ "Gage": 9941,
+ "fees": 9942,
+ "settling": 9943,
+ "ridiculous": 9944,
+ "##ela": 9945,
+ "Sons": 9946,
+ "cop": 9947,
+ "custody": 9948,
+ "##ID": 9949,
+ "proclaimed": 9950,
+ "Cardinals": 9951,
+ "##pm": 9952,
+ "Metal": 9953,
+ "Ana": 9954,
+ "1835": 9955,
+ "clue": 9956,
+ "Cardiff": 9957,
+ "riders": 9958,
+ "observations": 9959,
+ "MA": 9960,
+ "sometime": 9961,
+ "##och": 9962,
+ "performer": 9963,
+ "intact": 9964,
+ "Points": 9965,
+ "allegations": 9966,
+ "rotation": 9967,
+ "Tennis": 9968,
+ "tenor": 9969,
+ "Directors": 9970,
+ "##ats": 9971,
+ "Transit": 9972,
+ "thigh": 9973,
+ "Complex": 9974,
+ "##works": 9975,
+ "twentieth": 9976,
+ "Factory": 9977,
+ "doctrine": 9978,
+ "Daddy": 9979,
+ "##ished": 9980,
+ "pretend": 9981,
+ "Winston": 9982,
+ "cigarette": 9983,
+ "##IA": 9984,
+ "specimens": 9985,
+ "hydrogen": 9986,
+ "smoking": 9987,
+ "mathematical": 9988,
+ "arguments": 9989,
+ "openly": 9990,
+ "developer": 9991,
+ "##iro": 9992,
+ "fists": 9993,
+ "somebody": 9994,
+ "##san": 9995,
+ "Standing": 9996,
+ "Caleb": 9997,
+ "intelligent": 9998,
+ "Stay": 9999,
+ "Interior": 10000,
+ "echoed": 10001,
+ "Valentine": 10002,
+ "varieties": 10003,
+ "Brady": 10004,
+ "cluster": 10005,
+ "Ever": 10006,
+ "voyage": 10007,
+ "##of": 10008,
+ "deposits": 10009,
+ "ultimate": 10010,
+ "Hayes": 10011,
+ "horizontal": 10012,
+ "proximity": 10013,
+ "##ás": 10014,
+ "estates": 10015,
+ "exploration": 10016,
+ "NATO": 10017,
+ "Classical": 10018,
+ "##most": 10019,
+ "bills": 10020,
+ "condemned": 10021,
+ "1832": 10022,
+ "hunger": 10023,
+ "##ato": 10024,
+ "planes": 10025,
+ "deserve": 10026,
+ "offense": 10027,
+ "sequences": 10028,
+ "rendered": 10029,
+ "acceptance": 10030,
+ "##ony": 10031,
+ "manufacture": 10032,
+ "Plymouth": 10033,
+ "innovative": 10034,
+ "predicted": 10035,
+ "##RC": 10036,
+ "Fantasy": 10037,
+ "##une": 10038,
+ "supporter": 10039,
+ "absent": 10040,
+ "Picture": 10041,
+ "bassist": 10042,
+ "rescued": 10043,
+ "##MC": 10044,
+ "Ahmed": 10045,
+ "Monte": 10046,
+ "##sts": 10047,
+ "##rius": 10048,
+ "insane": 10049,
+ "novelist": 10050,
+ "##és": 10051,
+ "agrees": 10052,
+ "Antarctic": 10053,
+ "Lancaster": 10054,
+ "Hopkins": 10055,
+ "calculated": 10056,
+ "startled": 10057,
+ "##star": 10058,
+ "tribal": 10059,
+ "Amendment": 10060,
+ "##hoe": 10061,
+ "invisible": 10062,
+ "patron": 10063,
+ "deer": 10064,
+ "Walk": 10065,
+ "tracking": 10066,
+ "Lyon": 10067,
+ "tickets": 10068,
+ "##ED": 10069,
+ "philosopher": 10070,
+ "compounds": 10071,
+ "chuckled": 10072,
+ "##wi": 10073,
+ "pound": 10074,
+ "loyalty": 10075,
+ "Academic": 10076,
+ "petition": 10077,
+ "refuses": 10078,
+ "marking": 10079,
+ "Mercury": 10080,
+ "northeastern": 10081,
+ "dimensions": 10082,
+ "scandal": 10083,
+ "Canyon": 10084,
+ "patch": 10085,
+ "publish": 10086,
+ "##oning": 10087,
+ "Peak": 10088,
+ "minds": 10089,
+ "##boro": 10090,
+ "Presbyterian": 10091,
+ "Hardy": 10092,
+ "theoretical": 10093,
+ "magnitude": 10094,
+ "bombs": 10095,
+ "cage": 10096,
+ "##ders": 10097,
+ "##kai": 10098,
+ "measuring": 10099,
+ "explaining": 10100,
+ "avoiding": 10101,
+ "touchdowns": 10102,
+ "Card": 10103,
+ "theology": 10104,
+ "##ured": 10105,
+ "Popular": 10106,
+ "export": 10107,
+ "suspicious": 10108,
+ "Probably": 10109,
+ "photograph": 10110,
+ "Lou": 10111,
+ "Parks": 10112,
+ "Arms": 10113,
+ "compact": 10114,
+ "Apparently": 10115,
+ "excess": 10116,
+ "Banks": 10117,
+ "lied": 10118,
+ "stunned": 10119,
+ "territorial": 10120,
+ "Filipino": 10121,
+ "spectrum": 10122,
+ "learns": 10123,
+ "wash": 10124,
+ "imprisonment": 10125,
+ "ugly": 10126,
+ "##rose": 10127,
+ "Albany": 10128,
+ "Erik": 10129,
+ "sends": 10130,
+ "##hara": 10131,
+ "##rid": 10132,
+ "consumed": 10133,
+ "##gling": 10134,
+ "Belgrade": 10135,
+ "Da": 10136,
+ "opposing": 10137,
+ "Magnus": 10138,
+ "footsteps": 10139,
+ "glowing": 10140,
+ "delicate": 10141,
+ "Alexandria": 10142,
+ "Ludwig": 10143,
+ "gorgeous": 10144,
+ "Bros": 10145,
+ "Index": 10146,
+ "##PA": 10147,
+ "customs": 10148,
+ "preservation": 10149,
+ "bonds": 10150,
+ "##mond": 10151,
+ "environments": 10152,
+ "##nto": 10153,
+ "instructed": 10154,
+ "parted": 10155,
+ "adoption": 10156,
+ "locality": 10157,
+ "workshops": 10158,
+ "goalkeeper": 10159,
+ "##rik": 10160,
+ "##uma": 10161,
+ "Brighton": 10162,
+ "Slovenia": 10163,
+ "##ulating": 10164,
+ "##tical": 10165,
+ "towel": 10166,
+ "hugged": 10167,
+ "stripped": 10168,
+ "Bears": 10169,
+ "upright": 10170,
+ "Wagner": 10171,
+ "##aux": 10172,
+ "secretly": 10173,
+ "Adventures": 10174,
+ "nest": 10175,
+ "Course": 10176,
+ "Lauren": 10177,
+ "Boeing": 10178,
+ "Abdul": 10179,
+ "Lakes": 10180,
+ "450": 10181,
+ "##cu": 10182,
+ "USSR": 10183,
+ "caps": 10184,
+ "Chan": 10185,
+ "##nna": 10186,
+ "conceived": 10187,
+ "Actually": 10188,
+ "Belfast": 10189,
+ "Lithuanian": 10190,
+ "concentrate": 10191,
+ "possess": 10192,
+ "militia": 10193,
+ "pine": 10194,
+ "protagonist": 10195,
+ "Helena": 10196,
+ "##PS": 10197,
+ "##band": 10198,
+ "Belle": 10199,
+ "Clara": 10200,
+ "Reform": 10201,
+ "currency": 10202,
+ "pregnancy": 10203,
+ "1500": 10204,
+ "##rim": 10205,
+ "Isabella": 10206,
+ "hull": 10207,
+ "Name": 10208,
+ "trend": 10209,
+ "journalism": 10210,
+ "diet": 10211,
+ "##mel": 10212,
+ "Recording": 10213,
+ "acclaimed": 10214,
+ "Tang": 10215,
+ "Jace": 10216,
+ "steering": 10217,
+ "vacant": 10218,
+ "suggestion": 10219,
+ "costume": 10220,
+ "laser": 10221,
+ "##š": 10222,
+ "##ink": 10223,
+ "##pan": 10224,
+ "##vić": 10225,
+ "integral": 10226,
+ "achievements": 10227,
+ "wise": 10228,
+ "classroom": 10229,
+ "unions": 10230,
+ "southwestern": 10231,
+ "##uer": 10232,
+ "Garcia": 10233,
+ "toss": 10234,
+ "Tara": 10235,
+ "Large": 10236,
+ "##tate": 10237,
+ "evident": 10238,
+ "responsibilities": 10239,
+ "populated": 10240,
+ "satisfaction": 10241,
+ "##bia": 10242,
+ "casual": 10243,
+ "Ecuador": 10244,
+ "##ght": 10245,
+ "arose": 10246,
+ "##ović": 10247,
+ "Cornwall": 10248,
+ "embrace": 10249,
+ "refuse": 10250,
+ "Heavyweight": 10251,
+ "XI": 10252,
+ "Eden": 10253,
+ "activists": 10254,
+ "##uation": 10255,
+ "biology": 10256,
+ "##shan": 10257,
+ "fraud": 10258,
+ "Fuck": 10259,
+ "matched": 10260,
+ "legacy": 10261,
+ "Rivers": 10262,
+ "missionary": 10263,
+ "extraordinary": 10264,
+ "Didn": 10265,
+ "holder": 10266,
+ "wickets": 10267,
+ "crucial": 10268,
+ "Writers": 10269,
+ "Hurricane": 10270,
+ "Iceland": 10271,
+ "gross": 10272,
+ "trumpet": 10273,
+ "accordance": 10274,
+ "hurry": 10275,
+ "flooded": 10276,
+ "doctorate": 10277,
+ "Albania": 10278,
+ "##yi": 10279,
+ "united": 10280,
+ "deceased": 10281,
+ "jealous": 10282,
+ "grief": 10283,
+ "flute": 10284,
+ "portraits": 10285,
+ "##а": 10286,
+ "pleasant": 10287,
+ "Founded": 10288,
+ "Face": 10289,
+ "crowned": 10290,
+ "Raja": 10291,
+ "advisor": 10292,
+ "Salem": 10293,
+ "##ec": 10294,
+ "Achievement": 10295,
+ "admission": 10296,
+ "freely": 10297,
+ "minimal": 10298,
+ "Sudan": 10299,
+ "developers": 10300,
+ "estimate": 10301,
+ "disabled": 10302,
+ "##lane": 10303,
+ "downstairs": 10304,
+ "Bruno": 10305,
+ "##pus": 10306,
+ "pinyin": 10307,
+ "##ude": 10308,
+ "lecture": 10309,
+ "deadly": 10310,
+ "underlying": 10311,
+ "optical": 10312,
+ "witnesses": 10313,
+ "Combat": 10314,
+ "Julius": 10315,
+ "tapped": 10316,
+ "variants": 10317,
+ "##like": 10318,
+ "Colonial": 10319,
+ "Critics": 10320,
+ "Similarly": 10321,
+ "mouse": 10322,
+ "voltage": 10323,
+ "sculptor": 10324,
+ "Concert": 10325,
+ "salary": 10326,
+ "Frances": 10327,
+ "##ground": 10328,
+ "hook": 10329,
+ "premises": 10330,
+ "Software": 10331,
+ "instructor": 10332,
+ "nominee": 10333,
+ "##ited": 10334,
+ "fog": 10335,
+ "slopes": 10336,
+ "##zu": 10337,
+ "vegetation": 10338,
+ "sail": 10339,
+ "##rch": 10340,
+ "Body": 10341,
+ "Apart": 10342,
+ "atop": 10343,
+ "View": 10344,
+ "utility": 10345,
+ "ribs": 10346,
+ "cab": 10347,
+ "migration": 10348,
+ "##wyn": 10349,
+ "bounded": 10350,
+ "2019": 10351,
+ "pillow": 10352,
+ "trails": 10353,
+ "##ub": 10354,
+ "Halifax": 10355,
+ "shade": 10356,
+ "Rush": 10357,
+ "##lah": 10358,
+ "##dian": 10359,
+ "Notre": 10360,
+ "interviewed": 10361,
+ "Alexandra": 10362,
+ "Springfield": 10363,
+ "Indeed": 10364,
+ "rubbing": 10365,
+ "dozens": 10366,
+ "amusement": 10367,
+ "legally": 10368,
+ "##lers": 10369,
+ "Jill": 10370,
+ "Cinema": 10371,
+ "ignoring": 10372,
+ "Choice": 10373,
+ "##ures": 10374,
+ "pockets": 10375,
+ "##nell": 10376,
+ "laying": 10377,
+ "Blair": 10378,
+ "tackles": 10379,
+ "separately": 10380,
+ "##teen": 10381,
+ "Criminal": 10382,
+ "performs": 10383,
+ "theorem": 10384,
+ "Communication": 10385,
+ "suburbs": 10386,
+ "##iel": 10387,
+ "competitors": 10388,
+ "rows": 10389,
+ "##hai": 10390,
+ "Manitoba": 10391,
+ "Eleanor": 10392,
+ "interactions": 10393,
+ "nominations": 10394,
+ "assassination": 10395,
+ "##dis": 10396,
+ "Edmonton": 10397,
+ "diving": 10398,
+ "##dine": 10399,
+ "essay": 10400,
+ "##tas": 10401,
+ "AFC": 10402,
+ "Edge": 10403,
+ "directing": 10404,
+ "imagination": 10405,
+ "sunk": 10406,
+ "implement": 10407,
+ "Theodore": 10408,
+ "trembling": 10409,
+ "sealed": 10410,
+ "##rock": 10411,
+ "Nobel": 10412,
+ "##ancy": 10413,
+ "##dorf": 10414,
+ "##chen": 10415,
+ "genuine": 10416,
+ "apartments": 10417,
+ "Nicolas": 10418,
+ "AA": 10419,
+ "Bach": 10420,
+ "Globe": 10421,
+ "Store": 10422,
+ "220": 10423,
+ "##10": 10424,
+ "Rochester": 10425,
+ "##ño": 10426,
+ "alert": 10427,
+ "107": 10428,
+ "Beck": 10429,
+ "##nin": 10430,
+ "Naples": 10431,
+ "Basin": 10432,
+ "Crawford": 10433,
+ "fears": 10434,
+ "Tracy": 10435,
+ "##hen": 10436,
+ "disk": 10437,
+ "##pped": 10438,
+ "seventeen": 10439,
+ "Lead": 10440,
+ "backup": 10441,
+ "reconstruction": 10442,
+ "##lines": 10443,
+ "terrified": 10444,
+ "sleeve": 10445,
+ "nicknamed": 10446,
+ "popped": 10447,
+ "##making": 10448,
+ "##ern": 10449,
+ "Holiday": 10450,
+ "Gospel": 10451,
+ "ibn": 10452,
+ "##ime": 10453,
+ "convert": 10454,
+ "divine": 10455,
+ "resolved": 10456,
+ "##quet": 10457,
+ "ski": 10458,
+ "realizing": 10459,
+ "##RT": 10460,
+ "Legislature": 10461,
+ "reservoir": 10462,
+ "Rain": 10463,
+ "sinking": 10464,
+ "rainfall": 10465,
+ "elimination": 10466,
+ "challenging": 10467,
+ "tobacco": 10468,
+ "##outs": 10469,
+ "Given": 10470,
+ "smallest": 10471,
+ "Commercial": 10472,
+ "pin": 10473,
+ "rebel": 10474,
+ "comedian": 10475,
+ "exchanged": 10476,
+ "airing": 10477,
+ "dish": 10478,
+ "Salvador": 10479,
+ "promising": 10480,
+ "##wl": 10481,
+ "relax": 10482,
+ "presenter": 10483,
+ "toll": 10484,
+ "aerial": 10485,
+ "##eh": 10486,
+ "Fletcher": 10487,
+ "brass": 10488,
+ "disappear": 10489,
+ "zones": 10490,
+ "adjusted": 10491,
+ "contacts": 10492,
+ "##lk": 10493,
+ "sensed": 10494,
+ "Walt": 10495,
+ "mild": 10496,
+ "toes": 10497,
+ "flies": 10498,
+ "shame": 10499,
+ "considers": 10500,
+ "wildlife": 10501,
+ "Hanna": 10502,
+ "Arsenal": 10503,
+ "Ladies": 10504,
+ "naming": 10505,
+ "##ishing": 10506,
+ "anxiety": 10507,
+ "discussions": 10508,
+ "cute": 10509,
+ "undertaken": 10510,
+ "Cash": 10511,
+ "strain": 10512,
+ "Wyoming": 10513,
+ "dishes": 10514,
+ "precise": 10515,
+ "Angela": 10516,
+ "##ided": 10517,
+ "hostile": 10518,
+ "twins": 10519,
+ "115": 10520,
+ "Built": 10521,
+ "##pel": 10522,
+ "Online": 10523,
+ "tactics": 10524,
+ "Newman": 10525,
+ "##bourne": 10526,
+ "unclear": 10527,
+ "repairs": 10528,
+ "embarrassed": 10529,
+ "listing": 10530,
+ "tugged": 10531,
+ "Vale": 10532,
+ "##gin": 10533,
+ "Meredith": 10534,
+ "bout": 10535,
+ "##cle": 10536,
+ "velocity": 10537,
+ "tips": 10538,
+ "froze": 10539,
+ "evaluation": 10540,
+ "demonstrate": 10541,
+ "##card": 10542,
+ "criticised": 10543,
+ "Nash": 10544,
+ "lineup": 10545,
+ "Rao": 10546,
+ "monks": 10547,
+ "bacteria": 10548,
+ "lease": 10549,
+ "##lish": 10550,
+ "frightened": 10551,
+ "den": 10552,
+ "revived": 10553,
+ "finale": 10554,
+ "##rance": 10555,
+ "flee": 10556,
+ "Letters": 10557,
+ "decreased": 10558,
+ "##oh": 10559,
+ "Sounds": 10560,
+ "wrap": 10561,
+ "Sharon": 10562,
+ "incidents": 10563,
+ "renovated": 10564,
+ "everybody": 10565,
+ "stole": 10566,
+ "Bath": 10567,
+ "boxing": 10568,
+ "1815": 10569,
+ "withdraw": 10570,
+ "backs": 10571,
+ "interim": 10572,
+ "react": 10573,
+ "murders": 10574,
+ "Rhodes": 10575,
+ "Copa": 10576,
+ "framed": 10577,
+ "flown": 10578,
+ "Estonia": 10579,
+ "Heavy": 10580,
+ "explored": 10581,
+ "##rra": 10582,
+ "##GA": 10583,
+ "##ali": 10584,
+ "Istanbul": 10585,
+ "1834": 10586,
+ "##rite": 10587,
+ "##aging": 10588,
+ "##ues": 10589,
+ "Episcopal": 10590,
+ "arc": 10591,
+ "orientation": 10592,
+ "Maxwell": 10593,
+ "infected": 10594,
+ "##rot": 10595,
+ "BCE": 10596,
+ "Brook": 10597,
+ "grasp": 10598,
+ "Roberto": 10599,
+ "Excellence": 10600,
+ "108": 10601,
+ "withdrawal": 10602,
+ "Marines": 10603,
+ "rider": 10604,
+ "Lo": 10605,
+ "##sin": 10606,
+ "##run": 10607,
+ "Subsequently": 10608,
+ "garrison": 10609,
+ "hurricane": 10610,
+ "facade": 10611,
+ "Prussia": 10612,
+ "crushed": 10613,
+ "enterprise": 10614,
+ "##mber": 10615,
+ "Twitter": 10616,
+ "Generation": 10617,
+ "Physical": 10618,
+ "Sugar": 10619,
+ "editing": 10620,
+ "communicate": 10621,
+ "Ellie": 10622,
+ "##hurst": 10623,
+ "Ernst": 10624,
+ "wagon": 10625,
+ "promotional": 10626,
+ "conquest": 10627,
+ "Parliamentary": 10628,
+ "courtyard": 10629,
+ "lawyers": 10630,
+ "Superman": 10631,
+ "email": 10632,
+ "Prussian": 10633,
+ "lately": 10634,
+ "lecturer": 10635,
+ "Singer": 10636,
+ "Majesty": 10637,
+ "Paradise": 10638,
+ "sooner": 10639,
+ "Heath": 10640,
+ "slot": 10641,
+ "curves": 10642,
+ "convoy": 10643,
+ "##vian": 10644,
+ "induced": 10645,
+ "synonym": 10646,
+ "breeze": 10647,
+ "##plane": 10648,
+ "##ox": 10649,
+ "peered": 10650,
+ "Coalition": 10651,
+ "##hia": 10652,
+ "odds": 10653,
+ "##esh": 10654,
+ "##lina": 10655,
+ "Tomorrow": 10656,
+ "Nadu": 10657,
+ "##ico": 10658,
+ "##rah": 10659,
+ "damp": 10660,
+ "autonomous": 10661,
+ "console": 10662,
+ "Victory": 10663,
+ "counts": 10664,
+ "Luxembourg": 10665,
+ "intimate": 10666,
+ "Archived": 10667,
+ "Carroll": 10668,
+ "spy": 10669,
+ "Zero": 10670,
+ "habit": 10671,
+ "Always": 10672,
+ "faction": 10673,
+ "teenager": 10674,
+ "Johnston": 10675,
+ "chaos": 10676,
+ "ruin": 10677,
+ "commerce": 10678,
+ "blog": 10679,
+ "##shed": 10680,
+ "##the": 10681,
+ "reliable": 10682,
+ "Word": 10683,
+ "Yu": 10684,
+ "Norton": 10685,
+ "parade": 10686,
+ "Catholics": 10687,
+ "damned": 10688,
+ "##iling": 10689,
+ "surgeon": 10690,
+ "##tia": 10691,
+ "Allison": 10692,
+ "Jonas": 10693,
+ "remarked": 10694,
+ "##ès": 10695,
+ "idiot": 10696,
+ "Making": 10697,
+ "proposals": 10698,
+ "Industries": 10699,
+ "strategies": 10700,
+ "artifacts": 10701,
+ "batteries": 10702,
+ "reward": 10703,
+ "##vers": 10704,
+ "Agricultural": 10705,
+ "distinguish": 10706,
+ "lengths": 10707,
+ "Jeffrey": 10708,
+ "Progressive": 10709,
+ "kicking": 10710,
+ "Patricia": 10711,
+ "##gio": 10712,
+ "ballot": 10713,
+ "##ios": 10714,
+ "skilled": 10715,
+ "##gation": 10716,
+ "Colt": 10717,
+ "limestone": 10718,
+ "##AS": 10719,
+ "peninsula": 10720,
+ "##itis": 10721,
+ "LA": 10722,
+ "hotels": 10723,
+ "shapes": 10724,
+ "Crime": 10725,
+ "depicting": 10726,
+ "northwestern": 10727,
+ "HD": 10728,
+ "silly": 10729,
+ "Das": 10730,
+ "##²": 10731,
+ "##ws": 10732,
+ "##ash": 10733,
+ "##matic": 10734,
+ "thermal": 10735,
+ "Has": 10736,
+ "forgive": 10737,
+ "surrendered": 10738,
+ "Palm": 10739,
+ "Nacional": 10740,
+ "drank": 10741,
+ "haired": 10742,
+ "Mercedes": 10743,
+ "##foot": 10744,
+ "loading": 10745,
+ "Timothy": 10746,
+ "##roll": 10747,
+ "mechanisms": 10748,
+ "traces": 10749,
+ "digging": 10750,
+ "discussing": 10751,
+ "Natalie": 10752,
+ "##zhou": 10753,
+ "Forbes": 10754,
+ "landmark": 10755,
+ "Anyway": 10756,
+ "Manor": 10757,
+ "conspiracy": 10758,
+ "gym": 10759,
+ "knocking": 10760,
+ "viewing": 10761,
+ "Formation": 10762,
+ "Pink": 10763,
+ "Beauty": 10764,
+ "limbs": 10765,
+ "Phillip": 10766,
+ "sponsor": 10767,
+ "Joy": 10768,
+ "granite": 10769,
+ "Harbour": 10770,
+ "##ero": 10771,
+ "payments": 10772,
+ "Ballet": 10773,
+ "conviction": 10774,
+ "##dam": 10775,
+ "Hood": 10776,
+ "estimates": 10777,
+ "lacked": 10778,
+ "Mad": 10779,
+ "Jorge": 10780,
+ "##wen": 10781,
+ "refuge": 10782,
+ "##LA": 10783,
+ "invaded": 10784,
+ "Kat": 10785,
+ "suburban": 10786,
+ "##fold": 10787,
+ "investigated": 10788,
+ "Ari": 10789,
+ "complained": 10790,
+ "creek": 10791,
+ "Georges": 10792,
+ "##uts": 10793,
+ "powder": 10794,
+ "accepting": 10795,
+ "deserved": 10796,
+ "carpet": 10797,
+ "Thunder": 10798,
+ "molecules": 10799,
+ "Legal": 10800,
+ "cliff": 10801,
+ "strictly": 10802,
+ "enrollment": 10803,
+ "ranch": 10804,
+ "##rg": 10805,
+ "##mba": 10806,
+ "proportion": 10807,
+ "renovation": 10808,
+ "crop": 10809,
+ "grabbing": 10810,
+ "##liga": 10811,
+ "finest": 10812,
+ "entries": 10813,
+ "receptor": 10814,
+ "helmet": 10815,
+ "blown": 10816,
+ "Listen": 10817,
+ "flagship": 10818,
+ "workshop": 10819,
+ "resolve": 10820,
+ "nails": 10821,
+ "Shannon": 10822,
+ "portal": 10823,
+ "jointly": 10824,
+ "shining": 10825,
+ "Violet": 10826,
+ "overwhelming": 10827,
+ "upward": 10828,
+ "Mick": 10829,
+ "proceedings": 10830,
+ "##dies": 10831,
+ "##aring": 10832,
+ "Laurence": 10833,
+ "Churchill": 10834,
+ "##rice": 10835,
+ "commit": 10836,
+ "170": 10837,
+ "inclusion": 10838,
+ "Examples": 10839,
+ "##verse": 10840,
+ "##rma": 10841,
+ "fury": 10842,
+ "paths": 10843,
+ "##SC": 10844,
+ "ankle": 10845,
+ "nerves": 10846,
+ "Chemistry": 10847,
+ "rectangular": 10848,
+ "sworn": 10849,
+ "screenplay": 10850,
+ "cake": 10851,
+ "Mann": 10852,
+ "Seoul": 10853,
+ "Animal": 10854,
+ "sizes": 10855,
+ "Speed": 10856,
+ "vol": 10857,
+ "Population": 10858,
+ "Southwest": 10859,
+ "Hold": 10860,
+ "continuously": 10861,
+ "Qualified": 10862,
+ "wishing": 10863,
+ "Fighting": 10864,
+ "Made": 10865,
+ "disappointment": 10866,
+ "Portsmouth": 10867,
+ "Thirty": 10868,
+ "##beck": 10869,
+ "Ahmad": 10870,
+ "teammate": 10871,
+ "MLB": 10872,
+ "graph": 10873,
+ "Charleston": 10874,
+ "realizes": 10875,
+ "##dium": 10876,
+ "exhibits": 10877,
+ "preventing": 10878,
+ "##int": 10879,
+ "fever": 10880,
+ "rivalry": 10881,
+ "Male": 10882,
+ "mentally": 10883,
+ "dull": 10884,
+ "##lor": 10885,
+ "##rich": 10886,
+ "consistently": 10887,
+ "##igan": 10888,
+ "Madame": 10889,
+ "certificate": 10890,
+ "suited": 10891,
+ "Krishna": 10892,
+ "accuracy": 10893,
+ "Webb": 10894,
+ "Budapest": 10895,
+ "Rex": 10896,
+ "1831": 10897,
+ "Cornell": 10898,
+ "OK": 10899,
+ "surveillance": 10900,
+ "##gated": 10901,
+ "habitats": 10902,
+ "Adventure": 10903,
+ "Conrad": 10904,
+ "Superior": 10905,
+ "Gay": 10906,
+ "sofa": 10907,
+ "aka": 10908,
+ "boot": 10909,
+ "Statistics": 10910,
+ "Jessie": 10911,
+ "Liberation": 10912,
+ "##lip": 10913,
+ "##rier": 10914,
+ "brands": 10915,
+ "saint": 10916,
+ "Heinrich": 10917,
+ "Christine": 10918,
+ "bath": 10919,
+ "Rhine": 10920,
+ "ballet": 10921,
+ "Jin": 10922,
+ "consensus": 10923,
+ "chess": 10924,
+ "Arctic": 10925,
+ "stack": 10926,
+ "furious": 10927,
+ "cheap": 10928,
+ "toy": 10929,
+ "##yre": 10930,
+ "##face": 10931,
+ "##gging": 10932,
+ "gastropod": 10933,
+ "##nne": 10934,
+ "Romans": 10935,
+ "membrane": 10936,
+ "answering": 10937,
+ "25th": 10938,
+ "architects": 10939,
+ "sustainable": 10940,
+ "##yne": 10941,
+ "Hon": 10942,
+ "1814": 10943,
+ "Baldwin": 10944,
+ "dome": 10945,
+ "##awa": 10946,
+ "##zen": 10947,
+ "celebrity": 10948,
+ "enclosed": 10949,
+ "##uit": 10950,
+ "##mmer": 10951,
+ "Electronic": 10952,
+ "locals": 10953,
+ "##CE": 10954,
+ "supervision": 10955,
+ "mineral": 10956,
+ "Chemical": 10957,
+ "Slovakia": 10958,
+ "alley": 10959,
+ "hub": 10960,
+ "##az": 10961,
+ "heroes": 10962,
+ "Creative": 10963,
+ "##AM": 10964,
+ "incredible": 10965,
+ "politically": 10966,
+ "ESPN": 10967,
+ "yanked": 10968,
+ "halls": 10969,
+ "Aboriginal": 10970,
+ "Greatest": 10971,
+ "yield": 10972,
+ "##20": 10973,
+ "congressional": 10974,
+ "robot": 10975,
+ "Kiss": 10976,
+ "welcomed": 10977,
+ "MS": 10978,
+ "speeds": 10979,
+ "proceed": 10980,
+ "Sherman": 10981,
+ "eased": 10982,
+ "Greene": 10983,
+ "Walsh": 10984,
+ "Geoffrey": 10985,
+ "variables": 10986,
+ "rocky": 10987,
+ "##print": 10988,
+ "acclaim": 10989,
+ "Reverend": 10990,
+ "Wonder": 10991,
+ "tonnes": 10992,
+ "recurring": 10993,
+ "Dawson": 10994,
+ "continent": 10995,
+ "finite": 10996,
+ "AP": 10997,
+ "continental": 10998,
+ "ID": 10999,
+ "facilitate": 11000,
+ "essays": 11001,
+ "Rafael": 11002,
+ "Neal": 11003,
+ "1833": 11004,
+ "ancestors": 11005,
+ "##met": 11006,
+ "##gic": 11007,
+ "Especially": 11008,
+ "teenage": 11009,
+ "frustrated": 11010,
+ "Jules": 11011,
+ "cock": 11012,
+ "expense": 11013,
+ "##oli": 11014,
+ "##old": 11015,
+ "blocking": 11016,
+ "Notable": 11017,
+ "prohibited": 11018,
+ "ca": 11019,
+ "dock": 11020,
+ "organize": 11021,
+ "##wald": 11022,
+ "Burma": 11023,
+ "Gloria": 11024,
+ "dimension": 11025,
+ "aftermath": 11026,
+ "choosing": 11027,
+ "Mickey": 11028,
+ "torpedo": 11029,
+ "pub": 11030,
+ "##used": 11031,
+ "manuscripts": 11032,
+ "laps": 11033,
+ "Ulster": 11034,
+ "staircase": 11035,
+ "sphere": 11036,
+ "Insurance": 11037,
+ "Contest": 11038,
+ "lens": 11039,
+ "risks": 11040,
+ "investigations": 11041,
+ "ERA": 11042,
+ "glare": 11043,
+ "##play": 11044,
+ "Graduate": 11045,
+ "auction": 11046,
+ "Chronicle": 11047,
+ "##tric": 11048,
+ "##50": 11049,
+ "Coming": 11050,
+ "seating": 11051,
+ "Wade": 11052,
+ "seeks": 11053,
+ "inland": 11054,
+ "Thames": 11055,
+ "Rather": 11056,
+ "butterfly": 11057,
+ "contracted": 11058,
+ "positioned": 11059,
+ "consumers": 11060,
+ "contestants": 11061,
+ "fragments": 11062,
+ "Yankees": 11063,
+ "Santos": 11064,
+ "administrator": 11065,
+ "hypothesis": 11066,
+ "retire": 11067,
+ "Denis": 11068,
+ "agreements": 11069,
+ "Winnipeg": 11070,
+ "##rill": 11071,
+ "1820": 11072,
+ "trophy": 11073,
+ "crap": 11074,
+ "shakes": 11075,
+ "Jenkins": 11076,
+ "##rium": 11077,
+ "ya": 11078,
+ "twist": 11079,
+ "labels": 11080,
+ "Maritime": 11081,
+ "##lings": 11082,
+ "##iv": 11083,
+ "111": 11084,
+ "##ensis": 11085,
+ "Cairo": 11086,
+ "Anything": 11087,
+ "##fort": 11088,
+ "opinions": 11089,
+ "crowded": 11090,
+ "##nian": 11091,
+ "abandon": 11092,
+ "##iff": 11093,
+ "drained": 11094,
+ "imported": 11095,
+ "##rr": 11096,
+ "tended": 11097,
+ "##rain": 11098,
+ "Going": 11099,
+ "introducing": 11100,
+ "sculptures": 11101,
+ "bankruptcy": 11102,
+ "danced": 11103,
+ "demonstration": 11104,
+ "stance": 11105,
+ "settings": 11106,
+ "gazed": 11107,
+ "abstract": 11108,
+ "pet": 11109,
+ "Calvin": 11110,
+ "stiff": 11111,
+ "strongest": 11112,
+ "wrestler": 11113,
+ "##dre": 11114,
+ "Republicans": 11115,
+ "grace": 11116,
+ "allocated": 11117,
+ "cursed": 11118,
+ "snail": 11119,
+ "advancing": 11120,
+ "Return": 11121,
+ "errors": 11122,
+ "Mall": 11123,
+ "presenting": 11124,
+ "eliminate": 11125,
+ "Amateur": 11126,
+ "Institution": 11127,
+ "counting": 11128,
+ "##wind": 11129,
+ "warehouse": 11130,
+ "##nde": 11131,
+ "Ethiopia": 11132,
+ "trailed": 11133,
+ "hollow": 11134,
+ "##press": 11135,
+ "Literary": 11136,
+ "capability": 11137,
+ "nursing": 11138,
+ "preceding": 11139,
+ "lamp": 11140,
+ "Thomson": 11141,
+ "Morton": 11142,
+ "##ctic": 11143,
+ "Crew": 11144,
+ "Close": 11145,
+ "composers": 11146,
+ "boom": 11147,
+ "Clare": 11148,
+ "missiles": 11149,
+ "112": 11150,
+ "hunter": 11151,
+ "snap": 11152,
+ "##oni": 11153,
+ "##tail": 11154,
+ "Us": 11155,
+ "declaration": 11156,
+ "##cock": 11157,
+ "rally": 11158,
+ "huh": 11159,
+ "lion": 11160,
+ "straightened": 11161,
+ "Philippe": 11162,
+ "Sutton": 11163,
+ "alpha": 11164,
+ "valued": 11165,
+ "maker": 11166,
+ "navigation": 11167,
+ "detected": 11168,
+ "favorable": 11169,
+ "perception": 11170,
+ "Charter": 11171,
+ "##ña": 11172,
+ "Ricky": 11173,
+ "rebounds": 11174,
+ "tunnels": 11175,
+ "slapped": 11176,
+ "Emergency": 11177,
+ "supposedly": 11178,
+ "##act": 11179,
+ "deployment": 11180,
+ "socialist": 11181,
+ "tubes": 11182,
+ "anybody": 11183,
+ "corn": 11184,
+ "##NA": 11185,
+ "Seminary": 11186,
+ "heating": 11187,
+ "pump": 11188,
+ "##AA": 11189,
+ "achieving": 11190,
+ "souls": 11191,
+ "##ass": 11192,
+ "Link": 11193,
+ "##ele": 11194,
+ "##smith": 11195,
+ "greeted": 11196,
+ "Bates": 11197,
+ "Americas": 11198,
+ "Elder": 11199,
+ "cure": 11200,
+ "contestant": 11201,
+ "240": 11202,
+ "fold": 11203,
+ "Runner": 11204,
+ "Uh": 11205,
+ "licked": 11206,
+ "Politics": 11207,
+ "committees": 11208,
+ "neighbors": 11209,
+ "fairy": 11210,
+ "Silva": 11211,
+ "Leipzig": 11212,
+ "tipped": 11213,
+ "correctly": 11214,
+ "exciting": 11215,
+ "electronics": 11216,
+ "foundations": 11217,
+ "cottage": 11218,
+ "governmental": 11219,
+ "##hat": 11220,
+ "allied": 11221,
+ "claws": 11222,
+ "presidency": 11223,
+ "cruel": 11224,
+ "Agreement": 11225,
+ "slender": 11226,
+ "accompanying": 11227,
+ "precisely": 11228,
+ "##pass": 11229,
+ "driveway": 11230,
+ "swim": 11231,
+ "Stand": 11232,
+ "crews": 11233,
+ "##mission": 11234,
+ "rely": 11235,
+ "everyday": 11236,
+ "Wings": 11237,
+ "demo": 11238,
+ "##hic": 11239,
+ "recreational": 11240,
+ "min": 11241,
+ "nationality": 11242,
+ "##duction": 11243,
+ "Easter": 11244,
+ "##hole": 11245,
+ "canvas": 11246,
+ "Kay": 11247,
+ "Leicester": 11248,
+ "talented": 11249,
+ "Discovery": 11250,
+ "shells": 11251,
+ "##ech": 11252,
+ "Kerry": 11253,
+ "Ferguson": 11254,
+ "Leave": 11255,
+ "##place": 11256,
+ "altogether": 11257,
+ "adopt": 11258,
+ "butt": 11259,
+ "wolves": 11260,
+ "##nsis": 11261,
+ "##ania": 11262,
+ "modest": 11263,
+ "soprano": 11264,
+ "Boris": 11265,
+ "##ught": 11266,
+ "electron": 11267,
+ "depicts": 11268,
+ "hid": 11269,
+ "cruise": 11270,
+ "differ": 11271,
+ "treasure": 11272,
+ "##nch": 11273,
+ "Gun": 11274,
+ "Mama": 11275,
+ "Bengali": 11276,
+ "trainer": 11277,
+ "merchants": 11278,
+ "innovation": 11279,
+ "presumably": 11280,
+ "Shirley": 11281,
+ "bottles": 11282,
+ "proceeds": 11283,
+ "Fear": 11284,
+ "invested": 11285,
+ "Pirates": 11286,
+ "particle": 11287,
+ "Dominic": 11288,
+ "blamed": 11289,
+ "Fight": 11290,
+ "Daisy": 11291,
+ "##pper": 11292,
+ "##graphic": 11293,
+ "nods": 11294,
+ "knight": 11295,
+ "Doyle": 11296,
+ "tales": 11297,
+ "Carnegie": 11298,
+ "Evil": 11299,
+ "Inter": 11300,
+ "Shore": 11301,
+ "Nixon": 11302,
+ "transform": 11303,
+ "Savannah": 11304,
+ "##gas": 11305,
+ "Baltic": 11306,
+ "stretching": 11307,
+ "worlds": 11308,
+ "protocol": 11309,
+ "Percy": 11310,
+ "Toby": 11311,
+ "Heroes": 11312,
+ "brave": 11313,
+ "dancers": 11314,
+ "##aria": 11315,
+ "backwards": 11316,
+ "responses": 11317,
+ "Chi": 11318,
+ "Gaelic": 11319,
+ "Berry": 11320,
+ "crush": 11321,
+ "embarked": 11322,
+ "promises": 11323,
+ "Madonna": 11324,
+ "researcher": 11325,
+ "realised": 11326,
+ "inaugurated": 11327,
+ "Cherry": 11328,
+ "Mikhail": 11329,
+ "Nottingham": 11330,
+ "reinforced": 11331,
+ "subspecies": 11332,
+ "rapper": 11333,
+ "##kie": 11334,
+ "Dreams": 11335,
+ "Re": 11336,
+ "Damon": 11337,
+ "Minneapolis": 11338,
+ "monsters": 11339,
+ "suspicion": 11340,
+ "Tel": 11341,
+ "surroundings": 11342,
+ "afterward": 11343,
+ "complaints": 11344,
+ "OF": 11345,
+ "sectors": 11346,
+ "Algeria": 11347,
+ "lanes": 11348,
+ "Sabha": 11349,
+ "objectives": 11350,
+ "Donna": 11351,
+ "bothered": 11352,
+ "distracted": 11353,
+ "deciding": 11354,
+ "##ives": 11355,
+ "##CA": 11356,
+ "##onia": 11357,
+ "bishops": 11358,
+ "Strange": 11359,
+ "machinery": 11360,
+ "Voiced": 11361,
+ "synthesis": 11362,
+ "reflects": 11363,
+ "interference": 11364,
+ "##TS": 11365,
+ "##ury": 11366,
+ "keen": 11367,
+ "##ign": 11368,
+ "frown": 11369,
+ "freestyle": 11370,
+ "ton": 11371,
+ "Dixon": 11372,
+ "Sacred": 11373,
+ "Ruby": 11374,
+ "Prison": 11375,
+ "##ión": 11376,
+ "1825": 11377,
+ "outfit": 11378,
+ "##tain": 11379,
+ "curiosity": 11380,
+ "##ight": 11381,
+ "frames": 11382,
+ "steadily": 11383,
+ "emigrated": 11384,
+ "horizon": 11385,
+ "##erly": 11386,
+ "Doc": 11387,
+ "philosophical": 11388,
+ "Table": 11389,
+ "UTC": 11390,
+ "Marina": 11391,
+ "##DA": 11392,
+ "secular": 11393,
+ "##eed": 11394,
+ "Zimbabwe": 11395,
+ "cops": 11396,
+ "Mack": 11397,
+ "sheriff": 11398,
+ "Sanskrit": 11399,
+ "Francesco": 11400,
+ "catches": 11401,
+ "questioning": 11402,
+ "streaming": 11403,
+ "Kill": 11404,
+ "testimony": 11405,
+ "hissed": 11406,
+ "tackle": 11407,
+ "countryside": 11408,
+ "copyright": 11409,
+ "##IP": 11410,
+ "Buddhism": 11411,
+ "##rator": 11412,
+ "ladder": 11413,
+ "##ON": 11414,
+ "Past": 11415,
+ "rookie": 11416,
+ "depths": 11417,
+ "##yama": 11418,
+ "##ister": 11419,
+ "##HS": 11420,
+ "Samantha": 11421,
+ "Dana": 11422,
+ "Educational": 11423,
+ "brows": 11424,
+ "Hammond": 11425,
+ "raids": 11426,
+ "envelope": 11427,
+ "##sco": 11428,
+ "##hart": 11429,
+ "##ulus": 11430,
+ "epic": 11431,
+ "detection": 11432,
+ "Streets": 11433,
+ "Potter": 11434,
+ "statistical": 11435,
+ "für": 11436,
+ "ni": 11437,
+ "accounting": 11438,
+ "##pot": 11439,
+ "employer": 11440,
+ "Sidney": 11441,
+ "Depression": 11442,
+ "commands": 11443,
+ "Tracks": 11444,
+ "averaged": 11445,
+ "lets": 11446,
+ "Ram": 11447,
+ "longtime": 11448,
+ "suits": 11449,
+ "branded": 11450,
+ "chip": 11451,
+ "Shield": 11452,
+ "loans": 11453,
+ "ought": 11454,
+ "Said": 11455,
+ "sip": 11456,
+ "##rome": 11457,
+ "requests": 11458,
+ "Vernon": 11459,
+ "bordered": 11460,
+ "veterans": 11461,
+ "##ament": 11462,
+ "Marsh": 11463,
+ "Herzegovina": 11464,
+ "Pine": 11465,
+ "##igo": 11466,
+ "mills": 11467,
+ "anticipation": 11468,
+ "reconnaissance": 11469,
+ "##ef": 11470,
+ "expectations": 11471,
+ "protested": 11472,
+ "arrow": 11473,
+ "guessed": 11474,
+ "depot": 11475,
+ "maternal": 11476,
+ "weakness": 11477,
+ "##ap": 11478,
+ "projected": 11479,
+ "pour": 11480,
+ "Carmen": 11481,
+ "provider": 11482,
+ "newer": 11483,
+ "remind": 11484,
+ "freed": 11485,
+ "##rily": 11486,
+ "##wal": 11487,
+ "##tones": 11488,
+ "intentions": 11489,
+ "Fiji": 11490,
+ "timing": 11491,
+ "Match": 11492,
+ "managers": 11493,
+ "Kosovo": 11494,
+ "Herman": 11495,
+ "Wesley": 11496,
+ "Chang": 11497,
+ "135": 11498,
+ "semifinals": 11499,
+ "shouting": 11500,
+ "Indo": 11501,
+ "Janeiro": 11502,
+ "Chess": 11503,
+ "Macedonia": 11504,
+ "Buck": 11505,
+ "##onies": 11506,
+ "rulers": 11507,
+ "Mail": 11508,
+ "##vas": 11509,
+ "##sel": 11510,
+ "MHz": 11511,
+ "Programme": 11512,
+ "Task": 11513,
+ "commercially": 11514,
+ "subtle": 11515,
+ "propaganda": 11516,
+ "spelled": 11517,
+ "bowling": 11518,
+ "basically": 11519,
+ "Raven": 11520,
+ "1828": 11521,
+ "Colony": 11522,
+ "109": 11523,
+ "##ingham": 11524,
+ "##wara": 11525,
+ "anticipated": 11526,
+ "1829": 11527,
+ "##iers": 11528,
+ "graduates": 11529,
+ "##rton": 11530,
+ "##fication": 11531,
+ "endangered": 11532,
+ "ISO": 11533,
+ "diagnosed": 11534,
+ "##tage": 11535,
+ "exercises": 11536,
+ "Battery": 11537,
+ "bolt": 11538,
+ "poison": 11539,
+ "cartoon": 11540,
+ "##ción": 11541,
+ "hood": 11542,
+ "bowed": 11543,
+ "heal": 11544,
+ "Meyer": 11545,
+ "Reagan": 11546,
+ "##wed": 11547,
+ "subfamily": 11548,
+ "##gent": 11549,
+ "momentum": 11550,
+ "infant": 11551,
+ "detect": 11552,
+ "##sse": 11553,
+ "Chapman": 11554,
+ "Darwin": 11555,
+ "mechanics": 11556,
+ "NSW": 11557,
+ "Cancer": 11558,
+ "Brooke": 11559,
+ "Nuclear": 11560,
+ "comprised": 11561,
+ "hire": 11562,
+ "sanctuary": 11563,
+ "wingspan": 11564,
+ "contrary": 11565,
+ "remembering": 11566,
+ "surprising": 11567,
+ "Basic": 11568,
+ "stealing": 11569,
+ "OS": 11570,
+ "hatred": 11571,
+ "##lled": 11572,
+ "masters": 11573,
+ "violation": 11574,
+ "Rule": 11575,
+ "##nger": 11576,
+ "assuming": 11577,
+ "conquered": 11578,
+ "louder": 11579,
+ "robe": 11580,
+ "Beatles": 11581,
+ "legitimate": 11582,
+ "##vation": 11583,
+ "massacre": 11584,
+ "Rica": 11585,
+ "unsuccessfully": 11586,
+ "poets": 11587,
+ "##enberg": 11588,
+ "careers": 11589,
+ "doubled": 11590,
+ "premier": 11591,
+ "battalions": 11592,
+ "Dubai": 11593,
+ "Paper": 11594,
+ "Louisville": 11595,
+ "gestured": 11596,
+ "dressing": 11597,
+ "successive": 11598,
+ "mumbled": 11599,
+ "Vic": 11600,
+ "referee": 11601,
+ "pupil": 11602,
+ "##cated": 11603,
+ "##rre": 11604,
+ "ceremonies": 11605,
+ "picks": 11606,
+ "##IN": 11607,
+ "diplomat": 11608,
+ "alike": 11609,
+ "geographical": 11610,
+ "rays": 11611,
+ "##HA": 11612,
+ "##read": 11613,
+ "harbour": 11614,
+ "factories": 11615,
+ "pastor": 11616,
+ "playwright": 11617,
+ "Ultimate": 11618,
+ "nationalist": 11619,
+ "uniforms": 11620,
+ "obtaining": 11621,
+ "kit": 11622,
+ "Amber": 11623,
+ "##pling": 11624,
+ "screenwriter": 11625,
+ "ancestry": 11626,
+ "##cott": 11627,
+ "Fields": 11628,
+ "PR": 11629,
+ "Coleman": 11630,
+ "rat": 11631,
+ "Bavaria": 11632,
+ "squeeze": 11633,
+ "highlighted": 11634,
+ "Adult": 11635,
+ "reflecting": 11636,
+ "Mel": 11637,
+ "1824": 11638,
+ "bicycle": 11639,
+ "organizing": 11640,
+ "sided": 11641,
+ "Previously": 11642,
+ "Underground": 11643,
+ "Prof": 11644,
+ "athletics": 11645,
+ "coupled": 11646,
+ "mortal": 11647,
+ "Hampton": 11648,
+ "worthy": 11649,
+ "immune": 11650,
+ "Ava": 11651,
+ "##gun": 11652,
+ "encouraging": 11653,
+ "simplified": 11654,
+ "##ssa": 11655,
+ "##nte": 11656,
+ "##ann": 11657,
+ "Providence": 11658,
+ "entities": 11659,
+ "Pablo": 11660,
+ "Strong": 11661,
+ "Housing": 11662,
+ "##ista": 11663,
+ "##ators": 11664,
+ "kidnapped": 11665,
+ "mosque": 11666,
+ "Kirk": 11667,
+ "whispers": 11668,
+ "fruits": 11669,
+ "shattered": 11670,
+ "fossil": 11671,
+ "Empress": 11672,
+ "Johns": 11673,
+ "Webster": 11674,
+ "Thing": 11675,
+ "refusing": 11676,
+ "differently": 11677,
+ "specimen": 11678,
+ "Ha": 11679,
+ "##EN": 11680,
+ "##tina": 11681,
+ "##elle": 11682,
+ "##night": 11683,
+ "Horn": 11684,
+ "neighbourhood": 11685,
+ "Bolivia": 11686,
+ "##rth": 11687,
+ "genres": 11688,
+ "Pre": 11689,
+ "##vich": 11690,
+ "Amelia": 11691,
+ "swallow": 11692,
+ "Tribune": 11693,
+ "Forever": 11694,
+ "Psychology": 11695,
+ "Use": 11696,
+ "##bers": 11697,
+ "Gazette": 11698,
+ "ash": 11699,
+ "##usa": 11700,
+ "Monster": 11701,
+ "##cular": 11702,
+ "delegation": 11703,
+ "blowing": 11704,
+ "Oblast": 11705,
+ "retreated": 11706,
+ "automobile": 11707,
+ "##ex": 11708,
+ "profits": 11709,
+ "shirts": 11710,
+ "devil": 11711,
+ "Treasury": 11712,
+ "##backs": 11713,
+ "Drums": 11714,
+ "Ronnie": 11715,
+ "gameplay": 11716,
+ "expertise": 11717,
+ "Evening": 11718,
+ "resides": 11719,
+ "Caesar": 11720,
+ "unity": 11721,
+ "Crazy": 11722,
+ "linking": 11723,
+ "Vision": 11724,
+ "donations": 11725,
+ "Isabel": 11726,
+ "valve": 11727,
+ "Sue": 11728,
+ "WWE": 11729,
+ "logical": 11730,
+ "availability": 11731,
+ "fitting": 11732,
+ "revolt": 11733,
+ "##mill": 11734,
+ "Linux": 11735,
+ "taxi": 11736,
+ "Access": 11737,
+ "pollution": 11738,
+ "statues": 11739,
+ "Augustus": 11740,
+ "##pen": 11741,
+ "cello": 11742,
+ "##some": 11743,
+ "lacking": 11744,
+ "##ati": 11745,
+ "Gwen": 11746,
+ "##aka": 11747,
+ "##ovich": 11748,
+ "1821": 11749,
+ "Wow": 11750,
+ "initiatives": 11751,
+ "Uruguay": 11752,
+ "Cain": 11753,
+ "stroked": 11754,
+ "examine": 11755,
+ "##ī": 11756,
+ "mentor": 11757,
+ "moist": 11758,
+ "disorders": 11759,
+ "buttons": 11760,
+ "##tica": 11761,
+ "##anna": 11762,
+ "Species": 11763,
+ "Lynch": 11764,
+ "museums": 11765,
+ "scorer": 11766,
+ "Poor": 11767,
+ "eligibility": 11768,
+ "op": 11769,
+ "unveiled": 11770,
+ "cats": 11771,
+ "Title": 11772,
+ "wheat": 11773,
+ "critically": 11774,
+ "Syracuse": 11775,
+ "##osis": 11776,
+ "marketed": 11777,
+ "enhance": 11778,
+ "Ryder": 11779,
+ "##NG": 11780,
+ "##ull": 11781,
+ "##rna": 11782,
+ "embedded": 11783,
+ "throws": 11784,
+ "foods": 11785,
+ "happily": 11786,
+ "##ami": 11787,
+ "lesson": 11788,
+ "formats": 11789,
+ "punched": 11790,
+ "##rno": 11791,
+ "expressions": 11792,
+ "qualities": 11793,
+ "##sal": 11794,
+ "Gods": 11795,
+ "##lity": 11796,
+ "elect": 11797,
+ "wives": 11798,
+ "##lling": 11799,
+ "jungle": 11800,
+ "Toyota": 11801,
+ "reversed": 11802,
+ "Grammar": 11803,
+ "Cloud": 11804,
+ "Agnes": 11805,
+ "##ules": 11806,
+ "disputed": 11807,
+ "verses": 11808,
+ "Lucien": 11809,
+ "threshold": 11810,
+ "##rea": 11811,
+ "scanned": 11812,
+ "##bled": 11813,
+ "##dley": 11814,
+ "##lice": 11815,
+ "Kazakhstan": 11816,
+ "Gardner": 11817,
+ "Freeman": 11818,
+ "##rz": 11819,
+ "inspection": 11820,
+ "Rita": 11821,
+ "accommodation": 11822,
+ "advances": 11823,
+ "chill": 11824,
+ "Elliot": 11825,
+ "thriller": 11826,
+ "Constantinople": 11827,
+ "##mos": 11828,
+ "debris": 11829,
+ "whoever": 11830,
+ "1810": 11831,
+ "Santo": 11832,
+ "Carey": 11833,
+ "remnants": 11834,
+ "Guatemala": 11835,
+ "##irs": 11836,
+ "carriers": 11837,
+ "equations": 11838,
+ "mandatory": 11839,
+ "##WA": 11840,
+ "anxious": 11841,
+ "measurement": 11842,
+ "Summit": 11843,
+ "Terminal": 11844,
+ "Erin": 11845,
+ "##zes": 11846,
+ "LLC": 11847,
+ "##uo": 11848,
+ "glancing": 11849,
+ "sin": 11850,
+ "##₃": 11851,
+ "Downtown": 11852,
+ "flowering": 11853,
+ "Euro": 11854,
+ "Leigh": 11855,
+ "Lance": 11856,
+ "warn": 11857,
+ "decent": 11858,
+ "recommendations": 11859,
+ "##ote": 11860,
+ "Quartet": 11861,
+ "##rrell": 11862,
+ "Clarence": 11863,
+ "colleague": 11864,
+ "guarantee": 11865,
+ "230": 11866,
+ "Clayton": 11867,
+ "Beast": 11868,
+ "addresses": 11869,
+ "prospect": 11870,
+ "destroyer": 11871,
+ "vegetables": 11872,
+ "Leadership": 11873,
+ "fatal": 11874,
+ "prints": 11875,
+ "190": 11876,
+ "##makers": 11877,
+ "Hyde": 11878,
+ "persuaded": 11879,
+ "illustrations": 11880,
+ "Southampton": 11881,
+ "Joyce": 11882,
+ "beats": 11883,
+ "editors": 11884,
+ "mount": 11885,
+ "##grave": 11886,
+ "Malaysian": 11887,
+ "Bombay": 11888,
+ "endorsed": 11889,
+ "##sian": 11890,
+ "##bee": 11891,
+ "applying": 11892,
+ "Religion": 11893,
+ "nautical": 11894,
+ "bomber": 11895,
+ "Na": 11896,
+ "airfield": 11897,
+ "gravel": 11898,
+ "##rew": 11899,
+ "Cave": 11900,
+ "bye": 11901,
+ "dig": 11902,
+ "decree": 11903,
+ "burden": 11904,
+ "Election": 11905,
+ "Hawk": 11906,
+ "Fe": 11907,
+ "##iled": 11908,
+ "reunited": 11909,
+ "##tland": 11910,
+ "liver": 11911,
+ "Teams": 11912,
+ "Put": 11913,
+ "delegates": 11914,
+ "Ella": 11915,
+ "##fect": 11916,
+ "Cal": 11917,
+ "invention": 11918,
+ "Castro": 11919,
+ "bored": 11920,
+ "##kawa": 11921,
+ "##ail": 11922,
+ "Trinidad": 11923,
+ "NASCAR": 11924,
+ "pond": 11925,
+ "develops": 11926,
+ "##pton": 11927,
+ "expenses": 11928,
+ "Zoe": 11929,
+ "Released": 11930,
+ "##rf": 11931,
+ "organs": 11932,
+ "beta": 11933,
+ "parameters": 11934,
+ "Neill": 11935,
+ "##lene": 11936,
+ "lateral": 11937,
+ "Beat": 11938,
+ "blades": 11939,
+ "Either": 11940,
+ "##hale": 11941,
+ "Mitch": 11942,
+ "##ET": 11943,
+ "##vous": 11944,
+ "Rod": 11945,
+ "burnt": 11946,
+ "phones": 11947,
+ "Rising": 11948,
+ "##front": 11949,
+ "investigating": 11950,
+ "##dent": 11951,
+ "Stephanie": 11952,
+ "##keeper": 11953,
+ "screening": 11954,
+ "##uro": 11955,
+ "Swan": 11956,
+ "Sinclair": 11957,
+ "modes": 11958,
+ "bullets": 11959,
+ "Nigerian": 11960,
+ "melody": 11961,
+ "##ques": 11962,
+ "Rifle": 11963,
+ "##12": 11964,
+ "128": 11965,
+ "##jin": 11966,
+ "charm": 11967,
+ "Venus": 11968,
+ "##tian": 11969,
+ "fusion": 11970,
+ "advocated": 11971,
+ "visitor": 11972,
+ "pinned": 11973,
+ "genera": 11974,
+ "3000": 11975,
+ "Ferry": 11976,
+ "Solo": 11977,
+ "quantity": 11978,
+ "regained": 11979,
+ "platinum": 11980,
+ "shoots": 11981,
+ "narrowly": 11982,
+ "preceded": 11983,
+ "update": 11984,
+ "##ichi": 11985,
+ "equality": 11986,
+ "unaware": 11987,
+ "regiments": 11988,
+ "ally": 11989,
+ "##tos": 11990,
+ "transmitter": 11991,
+ "locks": 11992,
+ "Seeing": 11993,
+ "outlets": 11994,
+ "feast": 11995,
+ "reopened": 11996,
+ "##ows": 11997,
+ "struggles": 11998,
+ "Buddy": 11999,
+ "1826": 12000,
+ "bark": 12001,
+ "elegant": 12002,
+ "amused": 12003,
+ "Pretty": 12004,
+ "themed": 12005,
+ "schemes": 12006,
+ "Lisbon": 12007,
+ "Te": 12008,
+ "patted": 12009,
+ "terrorism": 12010,
+ "Mystery": 12011,
+ "##croft": 12012,
+ "##imo": 12013,
+ "Madagascar": 12014,
+ "Journey": 12015,
+ "dealer": 12016,
+ "contacted": 12017,
+ "##quez": 12018,
+ "ITV": 12019,
+ "vacation": 12020,
+ "Wong": 12021,
+ "Sacramento": 12022,
+ "organisms": 12023,
+ "##pts": 12024,
+ "balcony": 12025,
+ "coloured": 12026,
+ "sheer": 12027,
+ "defines": 12028,
+ "MC": 12029,
+ "abortion": 12030,
+ "forbidden": 12031,
+ "accredited": 12032,
+ "Newfoundland": 12033,
+ "tendency": 12034,
+ "entrepreneur": 12035,
+ "Benny": 12036,
+ "Tanzania": 12037,
+ "needing": 12038,
+ "finalist": 12039,
+ "mythology": 12040,
+ "weakened": 12041,
+ "gown": 12042,
+ "sentences": 12043,
+ "Guest": 12044,
+ "websites": 12045,
+ "Tibetan": 12046,
+ "UFC": 12047,
+ "voluntary": 12048,
+ "annoyed": 12049,
+ "Welcome": 12050,
+ "honestly": 12051,
+ "correspondence": 12052,
+ "geometry": 12053,
+ "Deutsche": 12054,
+ "Biology": 12055,
+ "Help": 12056,
+ "##aya": 12057,
+ "Lines": 12058,
+ "Hector": 12059,
+ "##ael": 12060,
+ "reluctant": 12061,
+ "##ages": 12062,
+ "wears": 12063,
+ "inquiry": 12064,
+ "##dell": 12065,
+ "Holocaust": 12066,
+ "Tourism": 12067,
+ "Wei": 12068,
+ "volcanic": 12069,
+ "##mates": 12070,
+ "Visual": 12071,
+ "sorts": 12072,
+ "neighborhoods": 12073,
+ "Running": 12074,
+ "apple": 12075,
+ "shy": 12076,
+ "Laws": 12077,
+ "bend": 12078,
+ "Northeast": 12079,
+ "feminist": 12080,
+ "Speedway": 12081,
+ "Murder": 12082,
+ "visa": 12083,
+ "stuffed": 12084,
+ "fangs": 12085,
+ "transmitted": 12086,
+ "fiscal": 12087,
+ "Ain": 12088,
+ "enlarged": 12089,
+ "##ndi": 12090,
+ "Cecil": 12091,
+ "Peterson": 12092,
+ "Benson": 12093,
+ "Bedford": 12094,
+ "acceptable": 12095,
+ "##CC": 12096,
+ "##wer": 12097,
+ "purely": 12098,
+ "triangle": 12099,
+ "foster": 12100,
+ "Alberto": 12101,
+ "educator": 12102,
+ "Highland": 12103,
+ "acute": 12104,
+ "LGBT": 12105,
+ "Tina": 12106,
+ "Mi": 12107,
+ "adventures": 12108,
+ "Davidson": 12109,
+ "Honda": 12110,
+ "translator": 12111,
+ "monk": 12112,
+ "enacted": 12113,
+ "summoned": 12114,
+ "##ional": 12115,
+ "collector": 12116,
+ "Genesis": 12117,
+ "Un": 12118,
+ "liner": 12119,
+ "Di": 12120,
+ "Statistical": 12121,
+ "##CS": 12122,
+ "filter": 12123,
+ "Knox": 12124,
+ "Religious": 12125,
+ "Stella": 12126,
+ "Estonian": 12127,
+ "Turn": 12128,
+ "##ots": 12129,
+ "primitive": 12130,
+ "parishes": 12131,
+ "##lles": 12132,
+ "complexity": 12133,
+ "autobiography": 12134,
+ "rigid": 12135,
+ "cannon": 12136,
+ "pursuing": 12137,
+ "exploring": 12138,
+ "##gram": 12139,
+ "##mme": 12140,
+ "freshman": 12141,
+ "caves": 12142,
+ "Expedition": 12143,
+ "Traditional": 12144,
+ "iTunes": 12145,
+ "certification": 12146,
+ "cooling": 12147,
+ "##ort": 12148,
+ "##gna": 12149,
+ "##IT": 12150,
+ "##lman": 12151,
+ "##VA": 12152,
+ "Motion": 12153,
+ "explosive": 12154,
+ "licence": 12155,
+ "boxer": 12156,
+ "shrine": 12157,
+ "loosely": 12158,
+ "Brigadier": 12159,
+ "Savage": 12160,
+ "Brett": 12161,
+ "MVP": 12162,
+ "heavier": 12163,
+ "##elli": 12164,
+ "##gged": 12165,
+ "Buddha": 12166,
+ "Easy": 12167,
+ "spells": 12168,
+ "fails": 12169,
+ "incredibly": 12170,
+ "Georg": 12171,
+ "stern": 12172,
+ "compatible": 12173,
+ "Perfect": 12174,
+ "applies": 12175,
+ "cognitive": 12176,
+ "excessive": 12177,
+ "nightmare": 12178,
+ "neighbor": 12179,
+ "Sicily": 12180,
+ "appealed": 12181,
+ "static": 12182,
+ "##₁": 12183,
+ "Aberdeen": 12184,
+ "##leigh": 12185,
+ "slipping": 12186,
+ "bride": 12187,
+ "##guard": 12188,
+ "Um": 12189,
+ "Clyde": 12190,
+ "1818": 12191,
+ "##gible": 12192,
+ "Hal": 12193,
+ "Frost": 12194,
+ "Sanders": 12195,
+ "interactive": 12196,
+ "Hour": 12197,
+ "##vor": 12198,
+ "hurting": 12199,
+ "bull": 12200,
+ "termed": 12201,
+ "shelf": 12202,
+ "capturing": 12203,
+ "##pace": 12204,
+ "rolls": 12205,
+ "113": 12206,
+ "##bor": 12207,
+ "Chilean": 12208,
+ "teaches": 12209,
+ "##rey": 12210,
+ "exam": 12211,
+ "shipped": 12212,
+ "Twin": 12213,
+ "borrowed": 12214,
+ "##lift": 12215,
+ "Shit": 12216,
+ "##hot": 12217,
+ "Lindsay": 12218,
+ "Below": 12219,
+ "Kiev": 12220,
+ "Lin": 12221,
+ "leased": 12222,
+ "##sto": 12223,
+ "Eli": 12224,
+ "Diane": 12225,
+ "Val": 12226,
+ "subtropical": 12227,
+ "shoe": 12228,
+ "Bolton": 12229,
+ "Dragons": 12230,
+ "##rification": 12231,
+ "Vatican": 12232,
+ "##pathy": 12233,
+ "Crisis": 12234,
+ "dramatically": 12235,
+ "talents": 12236,
+ "babies": 12237,
+ "##ores": 12238,
+ "surname": 12239,
+ "##AP": 12240,
+ "##cology": 12241,
+ "cubic": 12242,
+ "opted": 12243,
+ "Archer": 12244,
+ "sweep": 12245,
+ "tends": 12246,
+ "Karnataka": 12247,
+ "Judy": 12248,
+ "stint": 12249,
+ "Similar": 12250,
+ "##nut": 12251,
+ "explicitly": 12252,
+ "##nga": 12253,
+ "interact": 12254,
+ "Mae": 12255,
+ "portfolio": 12256,
+ "clinic": 12257,
+ "abbreviated": 12258,
+ "Counties": 12259,
+ "##iko": 12260,
+ "hearts": 12261,
+ "##ı": 12262,
+ "providers": 12263,
+ "screams": 12264,
+ "Individual": 12265,
+ "##etti": 12266,
+ "Monument": 12267,
+ "##iana": 12268,
+ "accessed": 12269,
+ "encounters": 12270,
+ "gasp": 12271,
+ "##rge": 12272,
+ "defunct": 12273,
+ "Avery": 12274,
+ "##rne": 12275,
+ "nobility": 12276,
+ "useless": 12277,
+ "Phase": 12278,
+ "Vince": 12279,
+ "senator": 12280,
+ "##FL": 12281,
+ "1813": 12282,
+ "surprisingly": 12283,
+ "##illo": 12284,
+ "##chin": 12285,
+ "Boyd": 12286,
+ "rumors": 12287,
+ "equity": 12288,
+ "Gone": 12289,
+ "Hearts": 12290,
+ "chassis": 12291,
+ "overnight": 12292,
+ "Trek": 12293,
+ "wrists": 12294,
+ "submit": 12295,
+ "civic": 12296,
+ "designers": 12297,
+ "##rity": 12298,
+ "prominence": 12299,
+ "decorative": 12300,
+ "derives": 12301,
+ "starter": 12302,
+ "##AF": 12303,
+ "wisdom": 12304,
+ "Powers": 12305,
+ "reluctantly": 12306,
+ "measurements": 12307,
+ "doctoral": 12308,
+ "Noel": 12309,
+ "Gideon": 12310,
+ "Baden": 12311,
+ "Cologne": 12312,
+ "lawn": 12313,
+ "Hawaiian": 12314,
+ "anthology": 12315,
+ "##rov": 12316,
+ "Raiders": 12317,
+ "embassy": 12318,
+ "Sterling": 12319,
+ "##pal": 12320,
+ "Telugu": 12321,
+ "troubled": 12322,
+ "##FC": 12323,
+ "##bian": 12324,
+ "fountain": 12325,
+ "observe": 12326,
+ "ore": 12327,
+ "##uru": 12328,
+ "##gence": 12329,
+ "spelling": 12330,
+ "Border": 12331,
+ "grinning": 12332,
+ "sketch": 12333,
+ "Benedict": 12334,
+ "Xbox": 12335,
+ "dialects": 12336,
+ "readily": 12337,
+ "immigrant": 12338,
+ "Constitutional": 12339,
+ "aided": 12340,
+ "nevertheless": 12341,
+ "SE": 12342,
+ "tragedy": 12343,
+ "##ager": 12344,
+ "##rden": 12345,
+ "Flash": 12346,
+ "##MP": 12347,
+ "Europa": 12348,
+ "emissions": 12349,
+ "##ield": 12350,
+ "panties": 12351,
+ "Beverly": 12352,
+ "Homer": 12353,
+ "curtain": 12354,
+ "##oto": 12355,
+ "toilet": 12356,
+ "Isn": 12357,
+ "Jerome": 12358,
+ "Chiefs": 12359,
+ "Hermann": 12360,
+ "supernatural": 12361,
+ "juice": 12362,
+ "integrity": 12363,
+ "Scots": 12364,
+ "auto": 12365,
+ "Patriots": 12366,
+ "Strategic": 12367,
+ "engaging": 12368,
+ "prosecution": 12369,
+ "cleaned": 12370,
+ "Byron": 12371,
+ "investments": 12372,
+ "adequate": 12373,
+ "vacuum": 12374,
+ "laughs": 12375,
+ "##inus": 12376,
+ "##nge": 12377,
+ "Usually": 12378,
+ "Roth": 12379,
+ "Cities": 12380,
+ "Brand": 12381,
+ "corpse": 12382,
+ "##ffy": 12383,
+ "Gas": 12384,
+ "rifles": 12385,
+ "Plains": 12386,
+ "sponsorship": 12387,
+ "Levi": 12388,
+ "tray": 12389,
+ "owed": 12390,
+ "della": 12391,
+ "commanders": 12392,
+ "##ead": 12393,
+ "tactical": 12394,
+ "##rion": 12395,
+ "García": 12396,
+ "harbor": 12397,
+ "discharge": 12398,
+ "##hausen": 12399,
+ "gentleman": 12400,
+ "endless": 12401,
+ "highways": 12402,
+ "##itarian": 12403,
+ "pleaded": 12404,
+ "##eta": 12405,
+ "archive": 12406,
+ "Midnight": 12407,
+ "exceptions": 12408,
+ "instances": 12409,
+ "Gibraltar": 12410,
+ "cart": 12411,
+ "##NS": 12412,
+ "Darren": 12413,
+ "Bonnie": 12414,
+ "##yle": 12415,
+ "##iva": 12416,
+ "OCLC": 12417,
+ "bra": 12418,
+ "Jess": 12419,
+ "##EA": 12420,
+ "consulting": 12421,
+ "Archives": 12422,
+ "Chance": 12423,
+ "distances": 12424,
+ "commissioner": 12425,
+ "##AR": 12426,
+ "LL": 12427,
+ "sailors": 12428,
+ "##sters": 12429,
+ "enthusiasm": 12430,
+ "Lang": 12431,
+ "##zia": 12432,
+ "Yugoslav": 12433,
+ "confirm": 12434,
+ "possibilities": 12435,
+ "Suffolk": 12436,
+ "##eman": 12437,
+ "banner": 12438,
+ "1822": 12439,
+ "Supporting": 12440,
+ "fingertips": 12441,
+ "civilization": 12442,
+ "##gos": 12443,
+ "technically": 12444,
+ "1827": 12445,
+ "Hastings": 12446,
+ "sidewalk": 12447,
+ "strained": 12448,
+ "monuments": 12449,
+ "Floyd": 12450,
+ "Chennai": 12451,
+ "Elvis": 12452,
+ "villagers": 12453,
+ "Cumberland": 12454,
+ "strode": 12455,
+ "albeit": 12456,
+ "Believe": 12457,
+ "planets": 12458,
+ "combining": 12459,
+ "Mohammad": 12460,
+ "container": 12461,
+ "##mouth": 12462,
+ "##tures": 12463,
+ "verb": 12464,
+ "BA": 12465,
+ "Tank": 12466,
+ "Midland": 12467,
+ "screened": 12468,
+ "Gang": 12469,
+ "Democracy": 12470,
+ "Helsinki": 12471,
+ "screens": 12472,
+ "thread": 12473,
+ "charitable": 12474,
+ "##version": 12475,
+ "swiftly": 12476,
+ "ma": 12477,
+ "rational": 12478,
+ "combine": 12479,
+ "##SS": 12480,
+ "##antly": 12481,
+ "dragging": 12482,
+ "Cliff": 12483,
+ "Tasmania": 12484,
+ "quest": 12485,
+ "professionally": 12486,
+ "##aj": 12487,
+ "rap": 12488,
+ "##lion": 12489,
+ "livestock": 12490,
+ "##hua": 12491,
+ "informal": 12492,
+ "specially": 12493,
+ "lonely": 12494,
+ "Matthews": 12495,
+ "Dictionary": 12496,
+ "1816": 12497,
+ "Observatory": 12498,
+ "correspondent": 12499,
+ "constitute": 12500,
+ "homeless": 12501,
+ "waving": 12502,
+ "appreciated": 12503,
+ "Analysis": 12504,
+ "Meeting": 12505,
+ "dagger": 12506,
+ "##AL": 12507,
+ "Gandhi": 12508,
+ "flank": 12509,
+ "Giant": 12510,
+ "Choir": 12511,
+ "##not": 12512,
+ "glimpse": 12513,
+ "toe": 12514,
+ "Writer": 12515,
+ "teasing": 12516,
+ "springs": 12517,
+ "##dt": 12518,
+ "Glory": 12519,
+ "healthcare": 12520,
+ "regulated": 12521,
+ "complaint": 12522,
+ "math": 12523,
+ "Publications": 12524,
+ "makers": 12525,
+ "##hips": 12526,
+ "cement": 12527,
+ "Need": 12528,
+ "apologize": 12529,
+ "disputes": 12530,
+ "finishes": 12531,
+ "Partners": 12532,
+ "boring": 12533,
+ "ups": 12534,
+ "gains": 12535,
+ "1793": 12536,
+ "Congressional": 12537,
+ "clergy": 12538,
+ "Folk": 12539,
+ "##made": 12540,
+ "##nza": 12541,
+ "Waters": 12542,
+ "stays": 12543,
+ "encoded": 12544,
+ "spider": 12545,
+ "betrayed": 12546,
+ "Applied": 12547,
+ "inception": 12548,
+ "##urt": 12549,
+ "##zzo": 12550,
+ "wards": 12551,
+ "bells": 12552,
+ "UCLA": 12553,
+ "Worth": 12554,
+ "bombers": 12555,
+ "Mo": 12556,
+ "trademark": 12557,
+ "Piper": 12558,
+ "##vel": 12559,
+ "incorporates": 12560,
+ "1801": 12561,
+ "##cial": 12562,
+ "dim": 12563,
+ "Twelve": 12564,
+ "##word": 12565,
+ "Appeals": 12566,
+ "tighter": 12567,
+ "spacecraft": 12568,
+ "##tine": 12569,
+ "coordinates": 12570,
+ "##iac": 12571,
+ "mistakes": 12572,
+ "Zach": 12573,
+ "laptop": 12574,
+ "Teresa": 12575,
+ "##llar": 12576,
+ "##yr": 12577,
+ "favored": 12578,
+ "Nora": 12579,
+ "sophisticated": 12580,
+ "Irving": 12581,
+ "hammer": 12582,
+ "División": 12583,
+ "corporations": 12584,
+ "niece": 12585,
+ "##rley": 12586,
+ "Patterson": 12587,
+ "UNESCO": 12588,
+ "trafficking": 12589,
+ "Ming": 12590,
+ "balanced": 12591,
+ "plaque": 12592,
+ "Latvia": 12593,
+ "broader": 12594,
+ "##owed": 12595,
+ "Save": 12596,
+ "confined": 12597,
+ "##vable": 12598,
+ "Dalton": 12599,
+ "tide": 12600,
+ "##right": 12601,
+ "##ural": 12602,
+ "##num": 12603,
+ "swords": 12604,
+ "caring": 12605,
+ "##eg": 12606,
+ "IX": 12607,
+ "Acting": 12608,
+ "paved": 12609,
+ "##moto": 12610,
+ "launching": 12611,
+ "Antoine": 12612,
+ "substantially": 12613,
+ "Pride": 12614,
+ "Philharmonic": 12615,
+ "grammar": 12616,
+ "Indoor": 12617,
+ "Ensemble": 12618,
+ "enabling": 12619,
+ "114": 12620,
+ "resided": 12621,
+ "Angelo": 12622,
+ "publicity": 12623,
+ "chaired": 12624,
+ "crawled": 12625,
+ "Maharashtra": 12626,
+ "Telegraph": 12627,
+ "lengthy": 12628,
+ "preference": 12629,
+ "differential": 12630,
+ "anonymous": 12631,
+ "Honey": 12632,
+ "##itation": 12633,
+ "wage": 12634,
+ "##iki": 12635,
+ "consecrated": 12636,
+ "Bryant": 12637,
+ "regulatory": 12638,
+ "Carr": 12639,
+ "##én": 12640,
+ "functioning": 12641,
+ "watches": 12642,
+ "##ú": 12643,
+ "shifts": 12644,
+ "diagnosis": 12645,
+ "Search": 12646,
+ "app": 12647,
+ "Peters": 12648,
+ "##SE": 12649,
+ "##cat": 12650,
+ "Andreas": 12651,
+ "honours": 12652,
+ "temper": 12653,
+ "counsel": 12654,
+ "Urdu": 12655,
+ "Anniversary": 12656,
+ "maritime": 12657,
+ "##uka": 12658,
+ "harmony": 12659,
+ "##unk": 12660,
+ "essence": 12661,
+ "Lorenzo": 12662,
+ "choked": 12663,
+ "Quarter": 12664,
+ "indie": 12665,
+ "##oll": 12666,
+ "loses": 12667,
+ "##prints": 12668,
+ "amendment": 12669,
+ "Adolf": 12670,
+ "scenario": 12671,
+ "similarities": 12672,
+ "##rade": 12673,
+ "##LC": 12674,
+ "technological": 12675,
+ "metric": 12676,
+ "Russians": 12677,
+ "thoroughly": 12678,
+ "##tead": 12679,
+ "cruiser": 12680,
+ "1806": 12681,
+ "##nier": 12682,
+ "1823": 12683,
+ "Teddy": 12684,
+ "##psy": 12685,
+ "au": 12686,
+ "progressed": 12687,
+ "exceptional": 12688,
+ "broadcaster": 12689,
+ "partnered": 12690,
+ "fitness": 12691,
+ "irregular": 12692,
+ "placement": 12693,
+ "mothers": 12694,
+ "unofficial": 12695,
+ "Garion": 12696,
+ "Johannes": 12697,
+ "1817": 12698,
+ "regain": 12699,
+ "Solar": 12700,
+ "publishes": 12701,
+ "Gates": 12702,
+ "Broken": 12703,
+ "thirds": 12704,
+ "conversations": 12705,
+ "dive": 12706,
+ "Raj": 12707,
+ "contributor": 12708,
+ "quantities": 12709,
+ "Worcester": 12710,
+ "governance": 12711,
+ "##flow": 12712,
+ "generating": 12713,
+ "pretending": 12714,
+ "Belarus": 12715,
+ "##voy": 12716,
+ "radius": 12717,
+ "skating": 12718,
+ "Marathon": 12719,
+ "1819": 12720,
+ "affection": 12721,
+ "undertook": 12722,
+ "##wright": 12723,
+ "los": 12724,
+ "##bro": 12725,
+ "locate": 12726,
+ "PS": 12727,
+ "excluded": 12728,
+ "recreation": 12729,
+ "tortured": 12730,
+ "jewelry": 12731,
+ "moaned": 12732,
+ "##logue": 12733,
+ "##cut": 12734,
+ "Complete": 12735,
+ "##rop": 12736,
+ "117": 12737,
+ "##II": 12738,
+ "plantation": 12739,
+ "whipped": 12740,
+ "slower": 12741,
+ "crater": 12742,
+ "##drome": 12743,
+ "Volunteer": 12744,
+ "attributes": 12745,
+ "celebrations": 12746,
+ "regards": 12747,
+ "Publishers": 12748,
+ "oath": 12749,
+ "utilized": 12750,
+ "Robbie": 12751,
+ "Giuseppe": 12752,
+ "fiber": 12753,
+ "indication": 12754,
+ "melted": 12755,
+ "archives": 12756,
+ "Damien": 12757,
+ "storey": 12758,
+ "affecting": 12759,
+ "identifying": 12760,
+ "dances": 12761,
+ "alumni": 12762,
+ "comparable": 12763,
+ "upgrade": 12764,
+ "rented": 12765,
+ "sprint": 12766,
+ "##kle": 12767,
+ "Marty": 12768,
+ "##lous": 12769,
+ "treating": 12770,
+ "railways": 12771,
+ "Lebanese": 12772,
+ "erupted": 12773,
+ "occupy": 12774,
+ "sympathy": 12775,
+ "Jude": 12776,
+ "Darling": 12777,
+ "Qatar": 12778,
+ "drainage": 12779,
+ "McCarthy": 12780,
+ "heel": 12781,
+ "Klein": 12782,
+ "computing": 12783,
+ "wireless": 12784,
+ "flip": 12785,
+ "Du": 12786,
+ "Bella": 12787,
+ "##ast": 12788,
+ "##ssen": 12789,
+ "narrator": 12790,
+ "mist": 12791,
+ "sings": 12792,
+ "alignment": 12793,
+ "121": 12794,
+ "2020": 12795,
+ "securing": 12796,
+ "##rail": 12797,
+ "Progress": 12798,
+ "missionaries": 12799,
+ "brutal": 12800,
+ "mercy": 12801,
+ "##shing": 12802,
+ "Hip": 12803,
+ "##ache": 12804,
+ "##olo": 12805,
+ "switching": 12806,
+ "##here": 12807,
+ "Malay": 12808,
+ "##ob": 12809,
+ "constituted": 12810,
+ "Mohammed": 12811,
+ "Often": 12812,
+ "standings": 12813,
+ "surge": 12814,
+ "teachings": 12815,
+ "ink": 12816,
+ "detached": 12817,
+ "systematic": 12818,
+ "Trial": 12819,
+ "Myanmar": 12820,
+ "##wo": 12821,
+ "offs": 12822,
+ "Reyes": 12823,
+ "decoration": 12824,
+ "translations": 12825,
+ "wherever": 12826,
+ "reviewer": 12827,
+ "speculation": 12828,
+ "Bangkok": 12829,
+ "terminated": 12830,
+ "##ester": 12831,
+ "beard": 12832,
+ "RCA": 12833,
+ "Aidan": 12834,
+ "Associated": 12835,
+ "Emerson": 12836,
+ "Charity": 12837,
+ "1803": 12838,
+ "generous": 12839,
+ "Dudley": 12840,
+ "ATP": 12841,
+ "##haven": 12842,
+ "prizes": 12843,
+ "toxic": 12844,
+ "gloves": 12845,
+ "##iles": 12846,
+ "##dos": 12847,
+ "Turning": 12848,
+ "myth": 12849,
+ "Parade": 12850,
+ "##building": 12851,
+ "Hits": 12852,
+ "##eva": 12853,
+ "teamed": 12854,
+ "Above": 12855,
+ "Duchess": 12856,
+ "Holt": 12857,
+ "##oth": 12858,
+ "Sub": 12859,
+ "Ace": 12860,
+ "atomic": 12861,
+ "inform": 12862,
+ "Ship": 12863,
+ "depend": 12864,
+ "Jun": 12865,
+ "##bes": 12866,
+ "Norwich": 12867,
+ "globe": 12868,
+ "Baroque": 12869,
+ "Christina": 12870,
+ "Cotton": 12871,
+ "Tunnel": 12872,
+ "kidding": 12873,
+ "Concerto": 12874,
+ "Brittany": 12875,
+ "tasted": 12876,
+ "phases": 12877,
+ "stems": 12878,
+ "angles": 12879,
+ "##TE": 12880,
+ "##nam": 12881,
+ "##40": 12882,
+ "charted": 12883,
+ "Alison": 12884,
+ "intensive": 12885,
+ "Willis": 12886,
+ "glory": 12887,
+ "##lit": 12888,
+ "Bergen": 12889,
+ "est": 12890,
+ "taller": 12891,
+ "##dicate": 12892,
+ "labeled": 12893,
+ "##ido": 12894,
+ "commentator": 12895,
+ "Warrior": 12896,
+ "Viscount": 12897,
+ "shortened": 12898,
+ "aisle": 12899,
+ "Aria": 12900,
+ "Spike": 12901,
+ "spectators": 12902,
+ "goodbye": 12903,
+ "overlooking": 12904,
+ "mammals": 12905,
+ "##lude": 12906,
+ "wholly": 12907,
+ "Barrett": 12908,
+ "##gus": 12909,
+ "accompany": 12910,
+ "seventy": 12911,
+ "employ": 12912,
+ "##mb": 12913,
+ "ambitious": 12914,
+ "beloved": 12915,
+ "basket": 12916,
+ "##mma": 12917,
+ "##lding": 12918,
+ "halted": 12919,
+ "descendant": 12920,
+ "pad": 12921,
+ "exclaimed": 12922,
+ "cloak": 12923,
+ "##pet": 12924,
+ "Strait": 12925,
+ "Bang": 12926,
+ "Aviv": 12927,
+ "sadness": 12928,
+ "##ffer": 12929,
+ "Donovan": 12930,
+ "1880s": 12931,
+ "agenda": 12932,
+ "swinging": 12933,
+ "##quin": 12934,
+ "jerk": 12935,
+ "Boat": 12936,
+ "##rist": 12937,
+ "nervously": 12938,
+ "Silence": 12939,
+ "Echo": 12940,
+ "shout": 12941,
+ "implies": 12942,
+ "##iser": 12943,
+ "##cking": 12944,
+ "Shiva": 12945,
+ "Weston": 12946,
+ "damages": 12947,
+ "##tist": 12948,
+ "effectiveness": 12949,
+ "Horace": 12950,
+ "cycling": 12951,
+ "Rey": 12952,
+ "ache": 12953,
+ "Photography": 12954,
+ "PDF": 12955,
+ "Dear": 12956,
+ "leans": 12957,
+ "Lea": 12958,
+ "##vision": 12959,
+ "booth": 12960,
+ "attained": 12961,
+ "disbelief": 12962,
+ "##eus": 12963,
+ "##ution": 12964,
+ "Hop": 12965,
+ "pension": 12966,
+ "toys": 12967,
+ "Eurovision": 12968,
+ "faithful": 12969,
+ "##heads": 12970,
+ "Andre": 12971,
+ "owe": 12972,
+ "default": 12973,
+ "Atlas": 12974,
+ "Megan": 12975,
+ "highlights": 12976,
+ "lovers": 12977,
+ "Constantine": 12978,
+ "Sixth": 12979,
+ "masses": 12980,
+ "##garh": 12981,
+ "emerge": 12982,
+ "Auto": 12983,
+ "Slovak": 12984,
+ "##oa": 12985,
+ "##vert": 12986,
+ "Superintendent": 12987,
+ "flicked": 12988,
+ "inventor": 12989,
+ "Chambers": 12990,
+ "Frankie": 12991,
+ "Romeo": 12992,
+ "pottery": 12993,
+ "companions": 12994,
+ "Rudolf": 12995,
+ "##liers": 12996,
+ "diary": 12997,
+ "Unless": 12998,
+ "tap": 12999,
+ "alter": 13000,
+ "Randall": 13001,
+ "##ddle": 13002,
+ "##eal": 13003,
+ "limitations": 13004,
+ "##boards": 13005,
+ "utterly": 13006,
+ "knelt": 13007,
+ "guaranteed": 13008,
+ "Cowboys": 13009,
+ "Islander": 13010,
+ "horns": 13011,
+ "##ike": 13012,
+ "Wendy": 13013,
+ "sexually": 13014,
+ "Smart": 13015,
+ "breasts": 13016,
+ "##cian": 13017,
+ "compromise": 13018,
+ "Duchy": 13019,
+ "AT": 13020,
+ "Galaxy": 13021,
+ "analog": 13022,
+ "Style": 13023,
+ "##aking": 13024,
+ "weighed": 13025,
+ "Nigel": 13026,
+ "optional": 13027,
+ "Czechoslovakia": 13028,
+ "practicing": 13029,
+ "Ham": 13030,
+ "##0s": 13031,
+ "feedback": 13032,
+ "batted": 13033,
+ "uprising": 13034,
+ "operative": 13035,
+ "applicable": 13036,
+ "criminals": 13037,
+ "classrooms": 13038,
+ "Somehow": 13039,
+ "##ode": 13040,
+ "##OM": 13041,
+ "Naomi": 13042,
+ "Winchester": 13043,
+ "##pping": 13044,
+ "Bart": 13045,
+ "Regina": 13046,
+ "competitor": 13047,
+ "Recorded": 13048,
+ "Yuan": 13049,
+ "Vera": 13050,
+ "lust": 13051,
+ "Confederation": 13052,
+ "##test": 13053,
+ "suck": 13054,
+ "1809": 13055,
+ "Lambert": 13056,
+ "175": 13057,
+ "Friend": 13058,
+ "##ppa": 13059,
+ "Slowly": 13060,
+ "##⁺": 13061,
+ "Wake": 13062,
+ "Dec": 13063,
+ "##aneous": 13064,
+ "chambers": 13065,
+ "Color": 13066,
+ "Gus": 13067,
+ "##site": 13068,
+ "Alternative": 13069,
+ "##world": 13070,
+ "Exeter": 13071,
+ "Omaha": 13072,
+ "celebrities": 13073,
+ "striker": 13074,
+ "210": 13075,
+ "dwarf": 13076,
+ "meals": 13077,
+ "Oriental": 13078,
+ "Pearson": 13079,
+ "financing": 13080,
+ "revenues": 13081,
+ "underwater": 13082,
+ "Steele": 13083,
+ "screw": 13084,
+ "Feeling": 13085,
+ "Mt": 13086,
+ "acids": 13087,
+ "badge": 13088,
+ "swore": 13089,
+ "theaters": 13090,
+ "Moving": 13091,
+ "admired": 13092,
+ "lung": 13093,
+ "knot": 13094,
+ "penalties": 13095,
+ "116": 13096,
+ "fork": 13097,
+ "##cribed": 13098,
+ "Afghan": 13099,
+ "outskirts": 13100,
+ "Cambodia": 13101,
+ "oval": 13102,
+ "wool": 13103,
+ "fossils": 13104,
+ "Ned": 13105,
+ "Countess": 13106,
+ "Darkness": 13107,
+ "delicious": 13108,
+ "##nica": 13109,
+ "Evelyn": 13110,
+ "Recordings": 13111,
+ "guidelines": 13112,
+ "##CP": 13113,
+ "Sandra": 13114,
+ "meantime": 13115,
+ "Antarctica": 13116,
+ "modeling": 13117,
+ "granddaughter": 13118,
+ "##rial": 13119,
+ "Roma": 13120,
+ "Seventh": 13121,
+ "Sunshine": 13122,
+ "Gabe": 13123,
+ "##nton": 13124,
+ "Shop": 13125,
+ "Turks": 13126,
+ "prolific": 13127,
+ "soup": 13128,
+ "parody": 13129,
+ "##nta": 13130,
+ "Judith": 13131,
+ "disciplines": 13132,
+ "resign": 13133,
+ "Companies": 13134,
+ "Libya": 13135,
+ "Jets": 13136,
+ "inserted": 13137,
+ "Mile": 13138,
+ "retrieve": 13139,
+ "filmmaker": 13140,
+ "##rand": 13141,
+ "realistic": 13142,
+ "unhappy": 13143,
+ "##30": 13144,
+ "sandstone": 13145,
+ "##nas": 13146,
+ "##lent": 13147,
+ "##ush": 13148,
+ "##rous": 13149,
+ "Brent": 13150,
+ "trash": 13151,
+ "Rescue": 13152,
+ "##unted": 13153,
+ "Autumn": 13154,
+ "disgust": 13155,
+ "flexible": 13156,
+ "infinite": 13157,
+ "sideways": 13158,
+ "##oss": 13159,
+ "##vik": 13160,
+ "trailing": 13161,
+ "disturbed": 13162,
+ "50th": 13163,
+ "Newark": 13164,
+ "posthumously": 13165,
+ "##rol": 13166,
+ "Schmidt": 13167,
+ "Josef": 13168,
+ "##eous": 13169,
+ "determining": 13170,
+ "menu": 13171,
+ "Pole": 13172,
+ "Anita": 13173,
+ "Luc": 13174,
+ "peaks": 13175,
+ "118": 13176,
+ "Yard": 13177,
+ "warrant": 13178,
+ "generic": 13179,
+ "deserted": 13180,
+ "Walking": 13181,
+ "stamp": 13182,
+ "tracked": 13183,
+ "##berger": 13184,
+ "paired": 13185,
+ "surveyed": 13186,
+ "sued": 13187,
+ "Rainbow": 13188,
+ "##isk": 13189,
+ "Carpenter": 13190,
+ "submarines": 13191,
+ "realization": 13192,
+ "touches": 13193,
+ "sweeping": 13194,
+ "Fritz": 13195,
+ "module": 13196,
+ "Whether": 13197,
+ "resembles": 13198,
+ "##form": 13199,
+ "##lop": 13200,
+ "unsure": 13201,
+ "hunters": 13202,
+ "Zagreb": 13203,
+ "unemployment": 13204,
+ "Senators": 13205,
+ "Georgetown": 13206,
+ "##onic": 13207,
+ "Barker": 13208,
+ "foul": 13209,
+ "commercials": 13210,
+ "Dresden": 13211,
+ "Words": 13212,
+ "collision": 13213,
+ "Carlton": 13214,
+ "Fashion": 13215,
+ "doubted": 13216,
+ "##ril": 13217,
+ "precision": 13218,
+ "MIT": 13219,
+ "Jacobs": 13220,
+ "mob": 13221,
+ "Monk": 13222,
+ "retaining": 13223,
+ "gotta": 13224,
+ "##rod": 13225,
+ "remake": 13226,
+ "Fast": 13227,
+ "chips": 13228,
+ "##pled": 13229,
+ "sufficiently": 13230,
+ "##lights": 13231,
+ "delivering": 13232,
+ "##enburg": 13233,
+ "Dancing": 13234,
+ "Barton": 13235,
+ "Officers": 13236,
+ "metals": 13237,
+ "##lake": 13238,
+ "religions": 13239,
+ "##ré": 13240,
+ "motivated": 13241,
+ "differs": 13242,
+ "dorsal": 13243,
+ "##birds": 13244,
+ "##rts": 13245,
+ "Priest": 13246,
+ "polished": 13247,
+ "##aling": 13248,
+ "Saxony": 13249,
+ "Wyatt": 13250,
+ "knockout": 13251,
+ "##hor": 13252,
+ "Lopez": 13253,
+ "RNA": 13254,
+ "##link": 13255,
+ "metallic": 13256,
+ "##kas": 13257,
+ "daylight": 13258,
+ "Montenegro": 13259,
+ "##lining": 13260,
+ "wrapping": 13261,
+ "resemble": 13262,
+ "Jam": 13263,
+ "Viking": 13264,
+ "uncertainty": 13265,
+ "angels": 13266,
+ "enables": 13267,
+ "##fy": 13268,
+ "Stuttgart": 13269,
+ "tricks": 13270,
+ "tattoo": 13271,
+ "127": 13272,
+ "wicked": 13273,
+ "asset": 13274,
+ "breach": 13275,
+ "##yman": 13276,
+ "MW": 13277,
+ "breaths": 13278,
+ "Jung": 13279,
+ "im": 13280,
+ "1798": 13281,
+ "noon": 13282,
+ "vowel": 13283,
+ "##qua": 13284,
+ "calmly": 13285,
+ "seasonal": 13286,
+ "chat": 13287,
+ "ingredients": 13288,
+ "cooled": 13289,
+ "Randolph": 13290,
+ "ensuring": 13291,
+ "##ib": 13292,
+ "##idal": 13293,
+ "flashing": 13294,
+ "1808": 13295,
+ "Macedonian": 13296,
+ "Cool": 13297,
+ "councils": 13298,
+ "##lick": 13299,
+ "advantages": 13300,
+ "Immediately": 13301,
+ "Madras": 13302,
+ "##cked": 13303,
+ "Pain": 13304,
+ "fancy": 13305,
+ "chronic": 13306,
+ "Malayalam": 13307,
+ "begged": 13308,
+ "##nese": 13309,
+ "Inner": 13310,
+ "feathers": 13311,
+ "##vey": 13312,
+ "Names": 13313,
+ "dedication": 13314,
+ "Sing": 13315,
+ "pan": 13316,
+ "Fischer": 13317,
+ "nurses": 13318,
+ "Sharp": 13319,
+ "inning": 13320,
+ "stamps": 13321,
+ "Meg": 13322,
+ "##ello": 13323,
+ "edged": 13324,
+ "motioned": 13325,
+ "Jacksonville": 13326,
+ "##ffle": 13327,
+ "##dic": 13328,
+ "##US": 13329,
+ "divide": 13330,
+ "garnered": 13331,
+ "Ranking": 13332,
+ "chasing": 13333,
+ "modifications": 13334,
+ "##oc": 13335,
+ "clever": 13336,
+ "midst": 13337,
+ "flushed": 13338,
+ "##DP": 13339,
+ "void": 13340,
+ "##sby": 13341,
+ "ambulance": 13342,
+ "beaches": 13343,
+ "groan": 13344,
+ "isolation": 13345,
+ "strengthen": 13346,
+ "prevention": 13347,
+ "##ffs": 13348,
+ "Scouts": 13349,
+ "reformed": 13350,
+ "geographic": 13351,
+ "squadrons": 13352,
+ "Fiona": 13353,
+ "Kai": 13354,
+ "Consequently": 13355,
+ "##uss": 13356,
+ "overtime": 13357,
+ "##yas": 13358,
+ "Fr": 13359,
+ "##BL": 13360,
+ "Papua": 13361,
+ "Mixed": 13362,
+ "glances": 13363,
+ "Haiti": 13364,
+ "Sporting": 13365,
+ "sandy": 13366,
+ "confronted": 13367,
+ "René": 13368,
+ "Tanner": 13369,
+ "1811": 13370,
+ "##IM": 13371,
+ "advisory": 13372,
+ "trim": 13373,
+ "##ibe": 13374,
+ "González": 13375,
+ "gambling": 13376,
+ "Jupiter": 13377,
+ "##ility": 13378,
+ "##owski": 13379,
+ "##nar": 13380,
+ "122": 13381,
+ "apology": 13382,
+ "teased": 13383,
+ "Pool": 13384,
+ "feminine": 13385,
+ "wicket": 13386,
+ "eagle": 13387,
+ "shiny": 13388,
+ "##lator": 13389,
+ "blend": 13390,
+ "peaking": 13391,
+ "nasty": 13392,
+ "nodding": 13393,
+ "fraction": 13394,
+ "tech": 13395,
+ "Noble": 13396,
+ "Kuwait": 13397,
+ "brushing": 13398,
+ "Italia": 13399,
+ "Canberra": 13400,
+ "duet": 13401,
+ "Johan": 13402,
+ "1805": 13403,
+ "Written": 13404,
+ "cameo": 13405,
+ "Stalin": 13406,
+ "pig": 13407,
+ "cord": 13408,
+ "##zio": 13409,
+ "Surely": 13410,
+ "SA": 13411,
+ "owing": 13412,
+ "holidays": 13413,
+ "123": 13414,
+ "Ranger": 13415,
+ "lighthouse": 13416,
+ "##ige": 13417,
+ "miners": 13418,
+ "1804": 13419,
+ "##ë": 13420,
+ "##gren": 13421,
+ "##ried": 13422,
+ "crashing": 13423,
+ "##atory": 13424,
+ "wartime": 13425,
+ "highlight": 13426,
+ "inclined": 13427,
+ "Torres": 13428,
+ "Tax": 13429,
+ "##zel": 13430,
+ "##oud": 13431,
+ "Own": 13432,
+ "##corn": 13433,
+ "Divine": 13434,
+ "EMI": 13435,
+ "Relief": 13436,
+ "Northwestern": 13437,
+ "ethics": 13438,
+ "BMW": 13439,
+ "click": 13440,
+ "plasma": 13441,
+ "Christie": 13442,
+ "coordinator": 13443,
+ "Shepherd": 13444,
+ "washing": 13445,
+ "cooked": 13446,
+ "##dio": 13447,
+ "##eat": 13448,
+ "Cerambycidae": 13449,
+ "algebra": 13450,
+ "Engine": 13451,
+ "costumes": 13452,
+ "Vampire": 13453,
+ "vault": 13454,
+ "submission": 13455,
+ "virtue": 13456,
+ "assumption": 13457,
+ "##rell": 13458,
+ "Toledo": 13459,
+ "##oting": 13460,
+ "##rva": 13461,
+ "crept": 13462,
+ "emphasized": 13463,
+ "##lton": 13464,
+ "##ood": 13465,
+ "Greeks": 13466,
+ "surgical": 13467,
+ "crest": 13468,
+ "Patrol": 13469,
+ "Beta": 13470,
+ "Tessa": 13471,
+ "##GS": 13472,
+ "pizza": 13473,
+ "traits": 13474,
+ "rats": 13475,
+ "Iris": 13476,
+ "spray": 13477,
+ "##GC": 13478,
+ "Lightning": 13479,
+ "binary": 13480,
+ "escapes": 13481,
+ "##take": 13482,
+ "Clary": 13483,
+ "crowds": 13484,
+ "##zong": 13485,
+ "hauled": 13486,
+ "maid": 13487,
+ "##fen": 13488,
+ "Manning": 13489,
+ "##yang": 13490,
+ "Nielsen": 13491,
+ "aesthetic": 13492,
+ "sympathetic": 13493,
+ "affiliation": 13494,
+ "soaked": 13495,
+ "Mozart": 13496,
+ "personalities": 13497,
+ "begging": 13498,
+ "##iga": 13499,
+ "clip": 13500,
+ "Raphael": 13501,
+ "yearly": 13502,
+ "Lima": 13503,
+ "abundant": 13504,
+ "##lm": 13505,
+ "1794": 13506,
+ "strips": 13507,
+ "Initiative": 13508,
+ "reporters": 13509,
+ "##vsky": 13510,
+ "consolidated": 13511,
+ "##itated": 13512,
+ "Civic": 13513,
+ "rankings": 13514,
+ "mandate": 13515,
+ "symbolic": 13516,
+ "##ively": 13517,
+ "1807": 13518,
+ "rental": 13519,
+ "duck": 13520,
+ "nave": 13521,
+ "complications": 13522,
+ "##nor": 13523,
+ "Irene": 13524,
+ "Nazis": 13525,
+ "haunted": 13526,
+ "scholarly": 13527,
+ "Pratt": 13528,
+ "Gran": 13529,
+ "Embassy": 13530,
+ "Wave": 13531,
+ "pity": 13532,
+ "genius": 13533,
+ "bats": 13534,
+ "canton": 13535,
+ "Tropical": 13536,
+ "marker": 13537,
+ "##cos": 13538,
+ "escorted": 13539,
+ "Climate": 13540,
+ "##posed": 13541,
+ "appreciation": 13542,
+ "freezing": 13543,
+ "puzzle": 13544,
+ "Internal": 13545,
+ "pools": 13546,
+ "Shawn": 13547,
+ "pathway": 13548,
+ "Daniels": 13549,
+ "Fitzgerald": 13550,
+ "extant": 13551,
+ "olive": 13552,
+ "Vanessa": 13553,
+ "marriages": 13554,
+ "cocked": 13555,
+ "##dging": 13556,
+ "prone": 13557,
+ "chemicals": 13558,
+ "doll": 13559,
+ "drawer": 13560,
+ "##HF": 13561,
+ "Stark": 13562,
+ "Property": 13563,
+ "##tai": 13564,
+ "flowed": 13565,
+ "Sheridan": 13566,
+ "##uated": 13567,
+ "Less": 13568,
+ "Omar": 13569,
+ "remarks": 13570,
+ "catalogue": 13571,
+ "Seymour": 13572,
+ "wreck": 13573,
+ "Carrie": 13574,
+ "##bby": 13575,
+ "Mercer": 13576,
+ "displaced": 13577,
+ "sovereignty": 13578,
+ "rip": 13579,
+ "Flynn": 13580,
+ "Archie": 13581,
+ "Quarterfinals": 13582,
+ "Hassan": 13583,
+ "##ards": 13584,
+ "vein": 13585,
+ "Osaka": 13586,
+ "pouring": 13587,
+ "wages": 13588,
+ "Romance": 13589,
+ "##cript": 13590,
+ "##phere": 13591,
+ "550": 13592,
+ "##eil": 13593,
+ "##stown": 13594,
+ "Documentary": 13595,
+ "ancestor": 13596,
+ "CNN": 13597,
+ "Panthers": 13598,
+ "publishers": 13599,
+ "Rise": 13600,
+ "##mu": 13601,
+ "biting": 13602,
+ "Bright": 13603,
+ "String": 13604,
+ "succeeding": 13605,
+ "119": 13606,
+ "loaned": 13607,
+ "Warwick": 13608,
+ "Sheikh": 13609,
+ "Von": 13610,
+ "Afterwards": 13611,
+ "Jax": 13612,
+ "Camden": 13613,
+ "helicopters": 13614,
+ "Hence": 13615,
+ "Laurel": 13616,
+ "##ddy": 13617,
+ "transaction": 13618,
+ "Corp": 13619,
+ "clause": 13620,
+ "##owing": 13621,
+ "##kel": 13622,
+ "Investment": 13623,
+ "cups": 13624,
+ "Lucia": 13625,
+ "Moss": 13626,
+ "Giles": 13627,
+ "chef": 13628,
+ "López": 13629,
+ "decisive": 13630,
+ "30th": 13631,
+ "distress": 13632,
+ "linguistic": 13633,
+ "surveys": 13634,
+ "Ready": 13635,
+ "maiden": 13636,
+ "Touch": 13637,
+ "frontier": 13638,
+ "incorporate": 13639,
+ "exotic": 13640,
+ "mollusk": 13641,
+ "Leopold": 13642,
+ "Ride": 13643,
+ "##wain": 13644,
+ "##ndo": 13645,
+ "teammates": 13646,
+ "tones": 13647,
+ "drift": 13648,
+ "ordering": 13649,
+ "Feb": 13650,
+ "Penny": 13651,
+ "Normandy": 13652,
+ "Present": 13653,
+ "Flag": 13654,
+ "pipes": 13655,
+ "##rro": 13656,
+ "delight": 13657,
+ "motto": 13658,
+ "Tibet": 13659,
+ "leap": 13660,
+ "Eliza": 13661,
+ "Produced": 13662,
+ "teenagers": 13663,
+ "sitcom": 13664,
+ "Try": 13665,
+ "Hansen": 13666,
+ "Cody": 13667,
+ "wandered": 13668,
+ "terrestrial": 13669,
+ "frog": 13670,
+ "scare": 13671,
+ "resisted": 13672,
+ "employers": 13673,
+ "coined": 13674,
+ "##DS": 13675,
+ "resistant": 13676,
+ "Fly": 13677,
+ "captive": 13678,
+ "dissolution": 13679,
+ "judged": 13680,
+ "associates": 13681,
+ "defining": 13682,
+ "##court": 13683,
+ "Hale": 13684,
+ "##mbo": 13685,
+ "raises": 13686,
+ "clusters": 13687,
+ "twelfth": 13688,
+ "##metric": 13689,
+ "Roads": 13690,
+ "##itude": 13691,
+ "satisfy": 13692,
+ "Android": 13693,
+ "Reds": 13694,
+ "Gloucester": 13695,
+ "Category": 13696,
+ "Valencia": 13697,
+ "Daemon": 13698,
+ "stabbed": 13699,
+ "Luna": 13700,
+ "Churches": 13701,
+ "Canton": 13702,
+ "##eller": 13703,
+ "Attack": 13704,
+ "Kashmir": 13705,
+ "annexed": 13706,
+ "grabs": 13707,
+ "asteroid": 13708,
+ "Hartford": 13709,
+ "recommendation": 13710,
+ "Rodriguez": 13711,
+ "handing": 13712,
+ "stressed": 13713,
+ "frequencies": 13714,
+ "delegate": 13715,
+ "Bones": 13716,
+ "Erie": 13717,
+ "Weber": 13718,
+ "Hands": 13719,
+ "Acts": 13720,
+ "millimetres": 13721,
+ "24th": 13722,
+ "Fat": 13723,
+ "Howe": 13724,
+ "casually": 13725,
+ "##SL": 13726,
+ "convent": 13727,
+ "1790": 13728,
+ "IF": 13729,
+ "##sity": 13730,
+ "1795": 13731,
+ "yelling": 13732,
+ "##ises": 13733,
+ "drain": 13734,
+ "addressing": 13735,
+ "amino": 13736,
+ "Marcel": 13737,
+ "Sylvia": 13738,
+ "Paramount": 13739,
+ "Gerard": 13740,
+ "Volleyball": 13741,
+ "butter": 13742,
+ "124": 13743,
+ "Albion": 13744,
+ "##GB": 13745,
+ "triggered": 13746,
+ "1792": 13747,
+ "folding": 13748,
+ "accepts": 13749,
+ "##ße": 13750,
+ "preparations": 13751,
+ "Wimbledon": 13752,
+ "dose": 13753,
+ "##grass": 13754,
+ "escaping": 13755,
+ "##tling": 13756,
+ "import": 13757,
+ "charging": 13758,
+ "##dation": 13759,
+ "280": 13760,
+ "Nolan": 13761,
+ "##fried": 13762,
+ "Calcutta": 13763,
+ "##pool": 13764,
+ "Cove": 13765,
+ "examining": 13766,
+ "minded": 13767,
+ "heartbeat": 13768,
+ "twisting": 13769,
+ "domains": 13770,
+ "bush": 13771,
+ "Tunisia": 13772,
+ "Purple": 13773,
+ "Leone": 13774,
+ "##code": 13775,
+ "evacuated": 13776,
+ "battlefield": 13777,
+ "tiger": 13778,
+ "Electrical": 13779,
+ "##ared": 13780,
+ "chased": 13781,
+ "##cre": 13782,
+ "cultivated": 13783,
+ "Jet": 13784,
+ "solved": 13785,
+ "shrug": 13786,
+ "ringing": 13787,
+ "Impact": 13788,
+ "##iant": 13789,
+ "kilometre": 13790,
+ "##log": 13791,
+ "commemorate": 13792,
+ "migrated": 13793,
+ "singular": 13794,
+ "designing": 13795,
+ "promptly": 13796,
+ "Higgins": 13797,
+ "##own": 13798,
+ "##aves": 13799,
+ "freshwater": 13800,
+ "Marketing": 13801,
+ "Payne": 13802,
+ "beg": 13803,
+ "locker": 13804,
+ "pray": 13805,
+ "implied": 13806,
+ "AAA": 13807,
+ "corrected": 13808,
+ "Trans": 13809,
+ "Europeans": 13810,
+ "Ashe": 13811,
+ "acknowledge": 13812,
+ "Introduction": 13813,
+ "##writer": 13814,
+ "##llen": 13815,
+ "Munster": 13816,
+ "auxiliary": 13817,
+ "growl": 13818,
+ "Hours": 13819,
+ "Poems": 13820,
+ "##AT": 13821,
+ "reduces": 13822,
+ "Plain": 13823,
+ "plague": 13824,
+ "canceled": 13825,
+ "detention": 13826,
+ "polite": 13827,
+ "necklace": 13828,
+ "Gustav": 13829,
+ "##gu": 13830,
+ "##lance": 13831,
+ "En": 13832,
+ "Angola": 13833,
+ "##bb": 13834,
+ "dwelling": 13835,
+ "##hea": 13836,
+ "5000": 13837,
+ "Qing": 13838,
+ "Dodgers": 13839,
+ "rim": 13840,
+ "##ored": 13841,
+ "##haus": 13842,
+ "spilled": 13843,
+ "Elisabeth": 13844,
+ "Viktor": 13845,
+ "backpack": 13846,
+ "1802": 13847,
+ "amended": 13848,
+ "##worthy": 13849,
+ "Phantom": 13850,
+ "##ctive": 13851,
+ "keeper": 13852,
+ "##loom": 13853,
+ "Vikings": 13854,
+ "##gua": 13855,
+ "employs": 13856,
+ "Tehran": 13857,
+ "specialty": 13858,
+ "##bate": 13859,
+ "Marx": 13860,
+ "Mirror": 13861,
+ "Jenna": 13862,
+ "rides": 13863,
+ "needle": 13864,
+ "prayers": 13865,
+ "clarinet": 13866,
+ "forewings": 13867,
+ "##walk": 13868,
+ "Midlands": 13869,
+ "convincing": 13870,
+ "advocacy": 13871,
+ "Cao": 13872,
+ "Birds": 13873,
+ "cycles": 13874,
+ "Clement": 13875,
+ "Gil": 13876,
+ "bubble": 13877,
+ "Maximum": 13878,
+ "humanitarian": 13879,
+ "Tan": 13880,
+ "cries": 13881,
+ "##SI": 13882,
+ "Parsons": 13883,
+ "Trio": 13884,
+ "offshore": 13885,
+ "Innovation": 13886,
+ "clutched": 13887,
+ "260": 13888,
+ "##mund": 13889,
+ "##duct": 13890,
+ "Prairie": 13891,
+ "relied": 13892,
+ "Falcon": 13893,
+ "##ste": 13894,
+ "Kolkata": 13895,
+ "Gill": 13896,
+ "Swift": 13897,
+ "Negro": 13898,
+ "Zoo": 13899,
+ "valleys": 13900,
+ "##OL": 13901,
+ "Opening": 13902,
+ "beams": 13903,
+ "MPs": 13904,
+ "outline": 13905,
+ "Bermuda": 13906,
+ "Personal": 13907,
+ "exceed": 13908,
+ "productive": 13909,
+ "##MT": 13910,
+ "republic": 13911,
+ "forum": 13912,
+ "##sty": 13913,
+ "tornado": 13914,
+ "Known": 13915,
+ "dipped": 13916,
+ "Edith": 13917,
+ "folks": 13918,
+ "mathematician": 13919,
+ "watershed": 13920,
+ "Ricardo": 13921,
+ "synthetic": 13922,
+ "##dication": 13923,
+ "deity": 13924,
+ "##₄": 13925,
+ "gaming": 13926,
+ "subjected": 13927,
+ "suspects": 13928,
+ "Foot": 13929,
+ "swollen": 13930,
+ "Motors": 13931,
+ "##tty": 13932,
+ "##ý": 13933,
+ "aloud": 13934,
+ "ceremonial": 13935,
+ "es": 13936,
+ "nuts": 13937,
+ "intend": 13938,
+ "Carlisle": 13939,
+ "tasked": 13940,
+ "hesitation": 13941,
+ "sponsors": 13942,
+ "unified": 13943,
+ "inmates": 13944,
+ "##ctions": 13945,
+ "##stan": 13946,
+ "tiles": 13947,
+ "jokes": 13948,
+ "whereby": 13949,
+ "outcomes": 13950,
+ "Lights": 13951,
+ "scary": 13952,
+ "Stoke": 13953,
+ "Portrait": 13954,
+ "Blind": 13955,
+ "sergeant": 13956,
+ "violations": 13957,
+ "cultivation": 13958,
+ "fuselage": 13959,
+ "Mister": 13960,
+ "Alfonso": 13961,
+ "candy": 13962,
+ "sticks": 13963,
+ "teen": 13964,
+ "agony": 13965,
+ "Enough": 13966,
+ "invite": 13967,
+ "Perkins": 13968,
+ "Appeal": 13969,
+ "mapping": 13970,
+ "undergo": 13971,
+ "Glacier": 13972,
+ "Melanie": 13973,
+ "affects": 13974,
+ "incomplete": 13975,
+ "##dd": 13976,
+ "Colombian": 13977,
+ "##nate": 13978,
+ "CBC": 13979,
+ "purchasing": 13980,
+ "bypass": 13981,
+ "Drug": 13982,
+ "Electronics": 13983,
+ "Frontier": 13984,
+ "Coventry": 13985,
+ "##aan": 13986,
+ "autonomy": 13987,
+ "scrambled": 13988,
+ "Recent": 13989,
+ "bounced": 13990,
+ "cow": 13991,
+ "experiencing": 13992,
+ "Rouge": 13993,
+ "cuisine": 13994,
+ "Elite": 13995,
+ "disability": 13996,
+ "Ji": 13997,
+ "inheritance": 13998,
+ "wildly": 13999,
+ "Into": 14000,
+ "##wig": 14001,
+ "confrontation": 14002,
+ "Wheeler": 14003,
+ "shiver": 14004,
+ "Performing": 14005,
+ "aligned": 14006,
+ "consequently": 14007,
+ "Alexis": 14008,
+ "Sin": 14009,
+ "woodland": 14010,
+ "executives": 14011,
+ "Stevenson": 14012,
+ "Ferrari": 14013,
+ "inevitable": 14014,
+ "##cist": 14015,
+ "##dha": 14016,
+ "##base": 14017,
+ "Corner": 14018,
+ "comeback": 14019,
+ "León": 14020,
+ "##eck": 14021,
+ "##urus": 14022,
+ "MacDonald": 14023,
+ "pioneering": 14024,
+ "breakdown": 14025,
+ "landscapes": 14026,
+ "Veterans": 14027,
+ "Rican": 14028,
+ "Theological": 14029,
+ "stirred": 14030,
+ "participant": 14031,
+ "Credit": 14032,
+ "Hyderabad": 14033,
+ "snails": 14034,
+ "Claudia": 14035,
+ "##ocene": 14036,
+ "compliance": 14037,
+ "##MI": 14038,
+ "Flags": 14039,
+ "Middlesex": 14040,
+ "storms": 14041,
+ "winding": 14042,
+ "asserted": 14043,
+ "er": 14044,
+ "##ault": 14045,
+ "##kal": 14046,
+ "waking": 14047,
+ "##rates": 14048,
+ "abbey": 14049,
+ "Augusta": 14050,
+ "tooth": 14051,
+ "trustees": 14052,
+ "Commodore": 14053,
+ "##uded": 14054,
+ "Cunningham": 14055,
+ "NC": 14056,
+ "Witch": 14057,
+ "marching": 14058,
+ "Sword": 14059,
+ "Same": 14060,
+ "spiral": 14061,
+ "Harley": 14062,
+ "##ahan": 14063,
+ "Zack": 14064,
+ "Audio": 14065,
+ "1890s": 14066,
+ "##fit": 14067,
+ "Simmons": 14068,
+ "Kara": 14069,
+ "Veronica": 14070,
+ "negotiated": 14071,
+ "Speaking": 14072,
+ "FIBA": 14073,
+ "Conservatory": 14074,
+ "formations": 14075,
+ "constituencies": 14076,
+ "explicit": 14077,
+ "facial": 14078,
+ "eleventh": 14079,
+ "##ilt": 14080,
+ "villain": 14081,
+ "##dog": 14082,
+ "##case": 14083,
+ "##hol": 14084,
+ "armored": 14085,
+ "tin": 14086,
+ "hairs": 14087,
+ "##umi": 14088,
+ "##rai": 14089,
+ "mattress": 14090,
+ "Angus": 14091,
+ "cease": 14092,
+ "verbal": 14093,
+ "Recreation": 14094,
+ "savings": 14095,
+ "Aurora": 14096,
+ "peers": 14097,
+ "Monastery": 14098,
+ "Airways": 14099,
+ "drowned": 14100,
+ "additions": 14101,
+ "downstream": 14102,
+ "sticking": 14103,
+ "Shi": 14104,
+ "mice": 14105,
+ "skiing": 14106,
+ "##CD": 14107,
+ "Raw": 14108,
+ "Riverside": 14109,
+ "warming": 14110,
+ "hooked": 14111,
+ "boost": 14112,
+ "memorable": 14113,
+ "posed": 14114,
+ "treatments": 14115,
+ "320": 14116,
+ "##dai": 14117,
+ "celebrating": 14118,
+ "blink": 14119,
+ "helpless": 14120,
+ "circa": 14121,
+ "Flowers": 14122,
+ "PM": 14123,
+ "uncommon": 14124,
+ "Oct": 14125,
+ "Hawks": 14126,
+ "overwhelmed": 14127,
+ "Sparhawk": 14128,
+ "repaired": 14129,
+ "Mercy": 14130,
+ "pose": 14131,
+ "counterpart": 14132,
+ "compare": 14133,
+ "survives": 14134,
+ "##½": 14135,
+ "##eum": 14136,
+ "coordinate": 14137,
+ "Lil": 14138,
+ "grandchildren": 14139,
+ "notorious": 14140,
+ "Yi": 14141,
+ "Judaism": 14142,
+ "Juliet": 14143,
+ "accusations": 14144,
+ "1789": 14145,
+ "floated": 14146,
+ "marathon": 14147,
+ "roar": 14148,
+ "fortified": 14149,
+ "reunion": 14150,
+ "145": 14151,
+ "Nov": 14152,
+ "Paula": 14153,
+ "##fare": 14154,
+ "##toria": 14155,
+ "tearing": 14156,
+ "Cedar": 14157,
+ "disappearance": 14158,
+ "Si": 14159,
+ "gifted": 14160,
+ "scar": 14161,
+ "270": 14162,
+ "PBS": 14163,
+ "Technologies": 14164,
+ "Marvin": 14165,
+ "650": 14166,
+ "roller": 14167,
+ "cupped": 14168,
+ "negotiate": 14169,
+ "##erman": 14170,
+ "passport": 14171,
+ "tram": 14172,
+ "miracle": 14173,
+ "styled": 14174,
+ "##tier": 14175,
+ "necessity": 14176,
+ "Des": 14177,
+ "rehabilitation": 14178,
+ "Lara": 14179,
+ "USD": 14180,
+ "psychic": 14181,
+ "wipe": 14182,
+ "##lem": 14183,
+ "mistaken": 14184,
+ "##lov": 14185,
+ "charming": 14186,
+ "Rider": 14187,
+ "pageant": 14188,
+ "dynamics": 14189,
+ "Cassidy": 14190,
+ "##icus": 14191,
+ "defenses": 14192,
+ "##tadt": 14193,
+ "##vant": 14194,
+ "aging": 14195,
+ "##inal": 14196,
+ "declare": 14197,
+ "mistress": 14198,
+ "supervised": 14199,
+ "##alis": 14200,
+ "##rest": 14201,
+ "Ashton": 14202,
+ "submerged": 14203,
+ "sack": 14204,
+ "Dodge": 14205,
+ "grocery": 14206,
+ "ramp": 14207,
+ "Teacher": 14208,
+ "lineage": 14209,
+ "imagery": 14210,
+ "arrange": 14211,
+ "inscriptions": 14212,
+ "Organisation": 14213,
+ "Siege": 14214,
+ "combines": 14215,
+ "pounded": 14216,
+ "Fleming": 14217,
+ "legends": 14218,
+ "columnist": 14219,
+ "Apostolic": 14220,
+ "prose": 14221,
+ "insight": 14222,
+ "Arabian": 14223,
+ "expired": 14224,
+ "##uses": 14225,
+ "##nos": 14226,
+ "Alone": 14227,
+ "elbows": 14228,
+ "##asis": 14229,
+ "##adi": 14230,
+ "##combe": 14231,
+ "Step": 14232,
+ "Waterloo": 14233,
+ "Alternate": 14234,
+ "interval": 14235,
+ "Sonny": 14236,
+ "plains": 14237,
+ "Goals": 14238,
+ "incorporating": 14239,
+ "recruit": 14240,
+ "adjoining": 14241,
+ "Cheshire": 14242,
+ "excluding": 14243,
+ "marrying": 14244,
+ "ducked": 14245,
+ "Cherokee": 14246,
+ "par": 14247,
+ "##inate": 14248,
+ "hiking": 14249,
+ "Coal": 14250,
+ "##bow": 14251,
+ "natives": 14252,
+ "ribbon": 14253,
+ "Allies": 14254,
+ "con": 14255,
+ "descriptions": 14256,
+ "positively": 14257,
+ "##lal": 14258,
+ "defendant": 14259,
+ "22nd": 14260,
+ "Vivian": 14261,
+ "##beat": 14262,
+ "Weather": 14263,
+ "possessions": 14264,
+ "Date": 14265,
+ "sweetheart": 14266,
+ "inability": 14267,
+ "Salisbury": 14268,
+ "adviser": 14269,
+ "ideology": 14270,
+ "Nordic": 14271,
+ "##eu": 14272,
+ "Cubs": 14273,
+ "IP": 14274,
+ "Administrative": 14275,
+ "##nick": 14276,
+ "facto": 14277,
+ "liberation": 14278,
+ "Burnett": 14279,
+ "Javier": 14280,
+ "fashioned": 14281,
+ "Electoral": 14282,
+ "Turin": 14283,
+ "theft": 14284,
+ "unanimous": 14285,
+ "Per": 14286,
+ "1799": 14287,
+ "Clan": 14288,
+ "Hawkins": 14289,
+ "Teachers": 14290,
+ "##wes": 14291,
+ "Cameroon": 14292,
+ "Parkway": 14293,
+ "##gment": 14294,
+ "demolition": 14295,
+ "atoms": 14296,
+ "nucleus": 14297,
+ "##thi": 14298,
+ "recovering": 14299,
+ "##yte": 14300,
+ "##vice": 14301,
+ "lifts": 14302,
+ "Must": 14303,
+ "deposit": 14304,
+ "Hancock": 14305,
+ "Semi": 14306,
+ "darkened": 14307,
+ "Declaration": 14308,
+ "moan": 14309,
+ "muscular": 14310,
+ "Myers": 14311,
+ "attractions": 14312,
+ "sauce": 14313,
+ "simulation": 14314,
+ "##weed": 14315,
+ "Alps": 14316,
+ "barriers": 14317,
+ "##baum": 14318,
+ "Barack": 14319,
+ "galleries": 14320,
+ "Min": 14321,
+ "holders": 14322,
+ "Greenwich": 14323,
+ "donation": 14324,
+ "Everybody": 14325,
+ "Wolfgang": 14326,
+ "sandwich": 14327,
+ "Kendra": 14328,
+ "Collegiate": 14329,
+ "casino": 14330,
+ "Slavic": 14331,
+ "ensuing": 14332,
+ "Porto": 14333,
+ "##grapher": 14334,
+ "Jesuit": 14335,
+ "suppressed": 14336,
+ "tires": 14337,
+ "Ibrahim": 14338,
+ "protesters": 14339,
+ "Ibn": 14340,
+ "Amos": 14341,
+ "1796": 14342,
+ "phenomena": 14343,
+ "Hayden": 14344,
+ "Paraguay": 14345,
+ "Squad": 14346,
+ "Reilly": 14347,
+ "complement": 14348,
+ "aluminum": 14349,
+ "##eers": 14350,
+ "doubts": 14351,
+ "decay": 14352,
+ "demise": 14353,
+ "Practice": 14354,
+ "patience": 14355,
+ "fireplace": 14356,
+ "transparent": 14357,
+ "monarchy": 14358,
+ "##person": 14359,
+ "Rodney": 14360,
+ "mattered": 14361,
+ "rotating": 14362,
+ "Clifford": 14363,
+ "disposal": 14364,
+ "Standards": 14365,
+ "paced": 14366,
+ "##llie": 14367,
+ "arise": 14368,
+ "tallest": 14369,
+ "tug": 14370,
+ "documentation": 14371,
+ "node": 14372,
+ "freeway": 14373,
+ "Nikolai": 14374,
+ "##cite": 14375,
+ "clicked": 14376,
+ "imaging": 14377,
+ "Lorraine": 14378,
+ "Tactical": 14379,
+ "Different": 14380,
+ "Regular": 14381,
+ "Holding": 14382,
+ "165": 14383,
+ "Pilot": 14384,
+ "guarded": 14385,
+ "##polis": 14386,
+ "Classics": 14387,
+ "Mongolia": 14388,
+ "Brock": 14389,
+ "monarch": 14390,
+ "cellular": 14391,
+ "receptors": 14392,
+ "Mini": 14393,
+ "Chandler": 14394,
+ "financed": 14395,
+ "financially": 14396,
+ "Lives": 14397,
+ "erection": 14398,
+ "Fuller": 14399,
+ "unnamed": 14400,
+ "Kannada": 14401,
+ "cc": 14402,
+ "passive": 14403,
+ "plateau": 14404,
+ "##arity": 14405,
+ "freak": 14406,
+ "##rde": 14407,
+ "retrieved": 14408,
+ "transactions": 14409,
+ "##sus": 14410,
+ "23rd": 14411,
+ "swimmer": 14412,
+ "beef": 14413,
+ "fulfill": 14414,
+ "Arlington": 14415,
+ "offspring": 14416,
+ "reasoning": 14417,
+ "Rhys": 14418,
+ "saves": 14419,
+ "pseudonym": 14420,
+ "centimetres": 14421,
+ "shivered": 14422,
+ "shuddered": 14423,
+ "##ME": 14424,
+ "Feel": 14425,
+ "##otic": 14426,
+ "professors": 14427,
+ "Blackburn": 14428,
+ "##eng": 14429,
+ "##life": 14430,
+ "##haw": 14431,
+ "interred": 14432,
+ "lodge": 14433,
+ "fragile": 14434,
+ "Della": 14435,
+ "guardian": 14436,
+ "##bbled": 14437,
+ "catalog": 14438,
+ "clad": 14439,
+ "observer": 14440,
+ "tract": 14441,
+ "declaring": 14442,
+ "##headed": 14443,
+ "Lok": 14444,
+ "dean": 14445,
+ "Isabelle": 14446,
+ "1776": 14447,
+ "irrigation": 14448,
+ "spectacular": 14449,
+ "shuttle": 14450,
+ "mastering": 14451,
+ "##aro": 14452,
+ "Nathaniel": 14453,
+ "Retired": 14454,
+ "##lves": 14455,
+ "Brennan": 14456,
+ "##kha": 14457,
+ "dick": 14458,
+ "##dated": 14459,
+ "##hler": 14460,
+ "Rookie": 14461,
+ "leapt": 14462,
+ "televised": 14463,
+ "weekends": 14464,
+ "Baghdad": 14465,
+ "Yemen": 14466,
+ "##fo": 14467,
+ "factions": 14468,
+ "ion": 14469,
+ "Lab": 14470,
+ "mortality": 14471,
+ "passionate": 14472,
+ "Hammer": 14473,
+ "encompasses": 14474,
+ "confluence": 14475,
+ "demonstrations": 14476,
+ "Ki": 14477,
+ "derivative": 14478,
+ "soils": 14479,
+ "##unch": 14480,
+ "Ranch": 14481,
+ "Universities": 14482,
+ "conventions": 14483,
+ "outright": 14484,
+ "aiming": 14485,
+ "hierarchy": 14486,
+ "reside": 14487,
+ "illusion": 14488,
+ "graves": 14489,
+ "rituals": 14490,
+ "126": 14491,
+ "Antwerp": 14492,
+ "Dover": 14493,
+ "##ema": 14494,
+ "campuses": 14495,
+ "Hobart": 14496,
+ "lifelong": 14497,
+ "aliens": 14498,
+ "##vity": 14499,
+ "Memory": 14500,
+ "coordination": 14501,
+ "alphabet": 14502,
+ "##mina": 14503,
+ "Titans": 14504,
+ "pushes": 14505,
+ "Flanders": 14506,
+ "##holder": 14507,
+ "Normal": 14508,
+ "excellence": 14509,
+ "capped": 14510,
+ "profound": 14511,
+ "Taipei": 14512,
+ "portrayal": 14513,
+ "sparked": 14514,
+ "scratch": 14515,
+ "se": 14516,
+ "##eas": 14517,
+ "##hir": 14518,
+ "Mackenzie": 14519,
+ "##cation": 14520,
+ "Neo": 14521,
+ "Shin": 14522,
+ "##lined": 14523,
+ "magnificent": 14524,
+ "poster": 14525,
+ "batsman": 14526,
+ "##rgent": 14527,
+ "persuade": 14528,
+ "##ement": 14529,
+ "Icelandic": 14530,
+ "miserable": 14531,
+ "collegiate": 14532,
+ "Feature": 14533,
+ "geography": 14534,
+ "##mura": 14535,
+ "Comic": 14536,
+ "Circus": 14537,
+ "processor": 14538,
+ "barracks": 14539,
+ "Tale": 14540,
+ "##11": 14541,
+ "Bulls": 14542,
+ "##rap": 14543,
+ "strengthened": 14544,
+ "##bell": 14545,
+ "injection": 14546,
+ "miniature": 14547,
+ "broadly": 14548,
+ "Letter": 14549,
+ "fare": 14550,
+ "hostage": 14551,
+ "traders": 14552,
+ "##nium": 14553,
+ "##mere": 14554,
+ "Fortune": 14555,
+ "Rivera": 14556,
+ "Lu": 14557,
+ "triumph": 14558,
+ "Browns": 14559,
+ "Bangalore": 14560,
+ "cooperative": 14561,
+ "Basel": 14562,
+ "announcing": 14563,
+ "Sawyer": 14564,
+ "##him": 14565,
+ "##cco": 14566,
+ "##kara": 14567,
+ "darted": 14568,
+ "##AD": 14569,
+ "##nova": 14570,
+ "sucking": 14571,
+ "##position": 14572,
+ "perimeter": 14573,
+ "flung": 14574,
+ "Holdings": 14575,
+ "##NP": 14576,
+ "Basque": 14577,
+ "sketches": 14578,
+ "Augustine": 14579,
+ "Silk": 14580,
+ "Elijah": 14581,
+ "analyst": 14582,
+ "armour": 14583,
+ "riots": 14584,
+ "acquiring": 14585,
+ "ghosts": 14586,
+ "##ems": 14587,
+ "132": 14588,
+ "Pioneer": 14589,
+ "Colleges": 14590,
+ "Simone": 14591,
+ "Economy": 14592,
+ "Author": 14593,
+ "semester": 14594,
+ "Soldier": 14595,
+ "il": 14596,
+ "##unting": 14597,
+ "##bid": 14598,
+ "freaking": 14599,
+ "Vista": 14600,
+ "tumor": 14601,
+ "##bat": 14602,
+ "murderer": 14603,
+ "##eda": 14604,
+ "unreleased": 14605,
+ "##grove": 14606,
+ "##sser": 14607,
+ "##té": 14608,
+ "edit": 14609,
+ "statute": 14610,
+ "sovereign": 14611,
+ "##gawa": 14612,
+ "Killer": 14613,
+ "stares": 14614,
+ "Fury": 14615,
+ "comply": 14616,
+ "##lord": 14617,
+ "##nant": 14618,
+ "barrels": 14619,
+ "Andhra": 14620,
+ "Maple": 14621,
+ "generator": 14622,
+ "mascot": 14623,
+ "unusually": 14624,
+ "eds": 14625,
+ "##ante": 14626,
+ "##runner": 14627,
+ "rod": 14628,
+ "##tles": 14629,
+ "Historically": 14630,
+ "Jennings": 14631,
+ "dumped": 14632,
+ "Established": 14633,
+ "resemblance": 14634,
+ "##lium": 14635,
+ "##cise": 14636,
+ "##body": 14637,
+ "##voke": 14638,
+ "Lydia": 14639,
+ "##hou": 14640,
+ "##iring": 14641,
+ "nonetheless": 14642,
+ "1797": 14643,
+ "corrupt": 14644,
+ "patrons": 14645,
+ "physicist": 14646,
+ "sneak": 14647,
+ "Livingston": 14648,
+ "Citizens": 14649,
+ "Architects": 14650,
+ "Werner": 14651,
+ "trends": 14652,
+ "Melody": 14653,
+ "eighty": 14654,
+ "markings": 14655,
+ "brakes": 14656,
+ "##titled": 14657,
+ "oversaw": 14658,
+ "processed": 14659,
+ "mock": 14660,
+ "Midwest": 14661,
+ "intervals": 14662,
+ "##EF": 14663,
+ "stretches": 14664,
+ "werewolf": 14665,
+ "##MG": 14666,
+ "Pack": 14667,
+ "controller": 14668,
+ "##dition": 14669,
+ "Honours": 14670,
+ "cane": 14671,
+ "Griffith": 14672,
+ "vague": 14673,
+ "repertoire": 14674,
+ "Courtney": 14675,
+ "orgasm": 14676,
+ "Abdullah": 14677,
+ "dominance": 14678,
+ "occupies": 14679,
+ "Ya": 14680,
+ "introduces": 14681,
+ "Lester": 14682,
+ "instinct": 14683,
+ "collaborative": 14684,
+ "Indigenous": 14685,
+ "refusal": 14686,
+ "##rank": 14687,
+ "outlet": 14688,
+ "debts": 14689,
+ "spear": 14690,
+ "155": 14691,
+ "##keeping": 14692,
+ "##ulu": 14693,
+ "Catalan": 14694,
+ "##osh": 14695,
+ "tensions": 14696,
+ "##OT": 14697,
+ "bred": 14698,
+ "crude": 14699,
+ "Dunn": 14700,
+ "abdomen": 14701,
+ "accurately": 14702,
+ "##fu": 14703,
+ "##lough": 14704,
+ "accidents": 14705,
+ "Row": 14706,
+ "Audrey": 14707,
+ "rude": 14708,
+ "Getting": 14709,
+ "promotes": 14710,
+ "replies": 14711,
+ "Paolo": 14712,
+ "merge": 14713,
+ "##nock": 14714,
+ "trans": 14715,
+ "Evangelical": 14716,
+ "automated": 14717,
+ "Canon": 14718,
+ "##wear": 14719,
+ "##ggy": 14720,
+ "##gma": 14721,
+ "Broncos": 14722,
+ "foolish": 14723,
+ "icy": 14724,
+ "Voices": 14725,
+ "knives": 14726,
+ "Aside": 14727,
+ "dreamed": 14728,
+ "generals": 14729,
+ "molecule": 14730,
+ "AG": 14731,
+ "rejection": 14732,
+ "insufficient": 14733,
+ "##nagar": 14734,
+ "deposited": 14735,
+ "sacked": 14736,
+ "Landing": 14737,
+ "arches": 14738,
+ "helpful": 14739,
+ "devotion": 14740,
+ "intake": 14741,
+ "Flower": 14742,
+ "PGA": 14743,
+ "dragons": 14744,
+ "evolutionary": 14745,
+ "##mail": 14746,
+ "330": 14747,
+ "GM": 14748,
+ "tissues": 14749,
+ "##tree": 14750,
+ "arcade": 14751,
+ "composite": 14752,
+ "lid": 14753,
+ "Across": 14754,
+ "implications": 14755,
+ "lacks": 14756,
+ "theological": 14757,
+ "assessed": 14758,
+ "concentrations": 14759,
+ "Den": 14760,
+ "##mans": 14761,
+ "##ulous": 14762,
+ "Fu": 14763,
+ "homeland": 14764,
+ "##stream": 14765,
+ "Harriet": 14766,
+ "ecclesiastical": 14767,
+ "troop": 14768,
+ "ecological": 14769,
+ "winked": 14770,
+ "##xed": 14771,
+ "eighteenth": 14772,
+ "Casino": 14773,
+ "specializing": 14774,
+ "##sworth": 14775,
+ "unlocked": 14776,
+ "supreme": 14777,
+ "devastated": 14778,
+ "snatched": 14779,
+ "trauma": 14780,
+ "GDP": 14781,
+ "Nord": 14782,
+ "saddle": 14783,
+ "Wes": 14784,
+ "convenient": 14785,
+ "competes": 14786,
+ "##nu": 14787,
+ "##iss": 14788,
+ "Marian": 14789,
+ "subway": 14790,
+ "##rri": 14791,
+ "successes": 14792,
+ "umbrella": 14793,
+ "##far": 14794,
+ "##ually": 14795,
+ "Dundee": 14796,
+ "##cence": 14797,
+ "spark": 14798,
+ "##rix": 14799,
+ "##я": 14800,
+ "Quality": 14801,
+ "Geological": 14802,
+ "cockpit": 14803,
+ "rpm": 14804,
+ "Cam": 14805,
+ "Bucharest": 14806,
+ "riot": 14807,
+ "##PM": 14808,
+ "Leah": 14809,
+ "##dad": 14810,
+ "##pose": 14811,
+ "Ka": 14812,
+ "m³": 14813,
+ "Bundesliga": 14814,
+ "Wolfe": 14815,
+ "grim": 14816,
+ "textile": 14817,
+ "quartet": 14818,
+ "expressing": 14819,
+ "fantastic": 14820,
+ "destroyers": 14821,
+ "eternal": 14822,
+ "picnic": 14823,
+ "##oro": 14824,
+ "contractor": 14825,
+ "1775": 14826,
+ "spanning": 14827,
+ "declining": 14828,
+ "##cating": 14829,
+ "Lowe": 14830,
+ "Sutherland": 14831,
+ "Emirates": 14832,
+ "downward": 14833,
+ "nineteen": 14834,
+ "violently": 14835,
+ "scout": 14836,
+ "viral": 14837,
+ "melting": 14838,
+ "enterprises": 14839,
+ "##cer": 14840,
+ "Crosby": 14841,
+ "Jubilee": 14842,
+ "antenna": 14843,
+ "urgent": 14844,
+ "Rory": 14845,
+ "##uin": 14846,
+ "##sure": 14847,
+ "wandering": 14848,
+ "##gler": 14849,
+ "##vent": 14850,
+ "Suzuki": 14851,
+ "Lifetime": 14852,
+ "Dirty": 14853,
+ "occupying": 14854,
+ "##quent": 14855,
+ "Disc": 14856,
+ "Guru": 14857,
+ "mound": 14858,
+ "Lennon": 14859,
+ "Humanities": 14860,
+ "listeners": 14861,
+ "Walton": 14862,
+ "uh": 14863,
+ "Braves": 14864,
+ "Bologna": 14865,
+ "##bis": 14866,
+ "##gra": 14867,
+ "Dwight": 14868,
+ "crawl": 14869,
+ "flags": 14870,
+ "memoir": 14871,
+ "Thorne": 14872,
+ "Archdiocese": 14873,
+ "dairy": 14874,
+ "##uz": 14875,
+ "##tery": 14876,
+ "roared": 14877,
+ "adjust": 14878,
+ "patches": 14879,
+ "inn": 14880,
+ "Knowing": 14881,
+ "##bbed": 14882,
+ "##zan": 14883,
+ "scan": 14884,
+ "Papa": 14885,
+ "precipitation": 14886,
+ "angrily": 14887,
+ "passages": 14888,
+ "postal": 14889,
+ "Phi": 14890,
+ "embraced": 14891,
+ "blacks": 14892,
+ "economist": 14893,
+ "triangular": 14894,
+ "Sen": 14895,
+ "shooter": 14896,
+ "punished": 14897,
+ "Millennium": 14898,
+ "Swimming": 14899,
+ "confessed": 14900,
+ "Aston": 14901,
+ "defeats": 14902,
+ "Era": 14903,
+ "cousins": 14904,
+ "Williamson": 14905,
+ "##rer": 14906,
+ "daytime": 14907,
+ "dumb": 14908,
+ "##rek": 14909,
+ "underway": 14910,
+ "specification": 14911,
+ "Buchanan": 14912,
+ "prayed": 14913,
+ "concealed": 14914,
+ "activation": 14915,
+ "##issa": 14916,
+ "canon": 14917,
+ "awesome": 14918,
+ "Starr": 14919,
+ "plural": 14920,
+ "summers": 14921,
+ "##fields": 14922,
+ "Slam": 14923,
+ "unnecessary": 14924,
+ "1791": 14925,
+ "resume": 14926,
+ "trilogy": 14927,
+ "compression": 14928,
+ "##rough": 14929,
+ "selective": 14930,
+ "dignity": 14931,
+ "Yan": 14932,
+ "##xton": 14933,
+ "immense": 14934,
+ "##yun": 14935,
+ "lone": 14936,
+ "seeded": 14937,
+ "hiatus": 14938,
+ "lightweight": 14939,
+ "summary": 14940,
+ "Yo": 14941,
+ "approve": 14942,
+ "Galway": 14943,
+ "rejoined": 14944,
+ "Elise": 14945,
+ "garbage": 14946,
+ "burns": 14947,
+ "speeches": 14948,
+ "129": 14949,
+ "Honduras": 14950,
+ "##liness": 14951,
+ "inventory": 14952,
+ "jersey": 14953,
+ "FK": 14954,
+ "assure": 14955,
+ "slumped": 14956,
+ "Lionel": 14957,
+ "Suite": 14958,
+ "##sbury": 14959,
+ "Lena": 14960,
+ "continuation": 14961,
+ "##AN": 14962,
+ "brightly": 14963,
+ "##nti": 14964,
+ "GT": 14965,
+ "Knowledge": 14966,
+ "##park": 14967,
+ "##lius": 14968,
+ "lethal": 14969,
+ "##tribution": 14970,
+ "##sions": 14971,
+ "Certificate": 14972,
+ "Mara": 14973,
+ "##lby": 14974,
+ "algorithms": 14975,
+ "Jade": 14976,
+ "blows": 14977,
+ "pirates": 14978,
+ "fleeing": 14979,
+ "wheelchair": 14980,
+ "Stein": 14981,
+ "sophomore": 14982,
+ "Alt": 14983,
+ "Territorial": 14984,
+ "diploma": 14985,
+ "snakes": 14986,
+ "##olic": 14987,
+ "##tham": 14988,
+ "Tiffany": 14989,
+ "Pius": 14990,
+ "flush": 14991,
+ "urging": 14992,
+ "Hanover": 14993,
+ "Reich": 14994,
+ "##olate": 14995,
+ "Unity": 14996,
+ "Pike": 14997,
+ "collectively": 14998,
+ "Theme": 14999,
+ "ballad": 15000,
+ "kindergarten": 15001,
+ "rocked": 15002,
+ "zoo": 15003,
+ "##page": 15004,
+ "whip": 15005,
+ "Rodríguez": 15006,
+ "strokes": 15007,
+ "checks": 15008,
+ "Becky": 15009,
+ "Stern": 15010,
+ "upstream": 15011,
+ "##uta": 15012,
+ "Silent": 15013,
+ "volunteered": 15014,
+ "Sigma": 15015,
+ "##ingen": 15016,
+ "##tract": 15017,
+ "##ede": 15018,
+ "Gujarat": 15019,
+ "screwed": 15020,
+ "entertaining": 15021,
+ "##action": 15022,
+ "##ryn": 15023,
+ "defenders": 15024,
+ "innocence": 15025,
+ "lesbian": 15026,
+ "que": 15027,
+ "Richie": 15028,
+ "nodes": 15029,
+ "Lie": 15030,
+ "juvenile": 15031,
+ "Jakarta": 15032,
+ "safer": 15033,
+ "confront": 15034,
+ "Bert": 15035,
+ "breakthrough": 15036,
+ "gospel": 15037,
+ "Cable": 15038,
+ "##zie": 15039,
+ "institutional": 15040,
+ "Archive": 15041,
+ "brake": 15042,
+ "liquor": 15043,
+ "feeds": 15044,
+ "##iate": 15045,
+ "chancellor": 15046,
+ "Encyclopedia": 15047,
+ "Animation": 15048,
+ "scanning": 15049,
+ "teens": 15050,
+ "##mother": 15051,
+ "Core": 15052,
+ "Rear": 15053,
+ "Wine": 15054,
+ "##flower": 15055,
+ "reactor": 15056,
+ "Ave": 15057,
+ "cardinal": 15058,
+ "sodium": 15059,
+ "strands": 15060,
+ "Olivier": 15061,
+ "crouched": 15062,
+ "Vaughan": 15063,
+ "Sammy": 15064,
+ "Image": 15065,
+ "scars": 15066,
+ "Emmanuel": 15067,
+ "flour": 15068,
+ "bias": 15069,
+ "nipple": 15070,
+ "revelation": 15071,
+ "##ucci": 15072,
+ "Denny": 15073,
+ "##ssy": 15074,
+ "Form": 15075,
+ "Runners": 15076,
+ "admits": 15077,
+ "Rama": 15078,
+ "violated": 15079,
+ "Burmese": 15080,
+ "feud": 15081,
+ "underwear": 15082,
+ "Mohamed": 15083,
+ "Named": 15084,
+ "swift": 15085,
+ "statewide": 15086,
+ "Door": 15087,
+ "Recently": 15088,
+ "comparing": 15089,
+ "Hundred": 15090,
+ "##idge": 15091,
+ "##nity": 15092,
+ "##rds": 15093,
+ "Rally": 15094,
+ "Reginald": 15095,
+ "Auburn": 15096,
+ "solving": 15097,
+ "waitress": 15098,
+ "Treasurer": 15099,
+ "##ilization": 15100,
+ "Halloween": 15101,
+ "Ministers": 15102,
+ "Boss": 15103,
+ "Shut": 15104,
+ "##listic": 15105,
+ "Rahman": 15106,
+ "demonstrating": 15107,
+ "##pies": 15108,
+ "Gaza": 15109,
+ "Yuri": 15110,
+ "installations": 15111,
+ "Math": 15112,
+ "schooling": 15113,
+ "##bble": 15114,
+ "Bronx": 15115,
+ "exiled": 15116,
+ "gasoline": 15117,
+ "133": 15118,
+ "bundle": 15119,
+ "humid": 15120,
+ "FCC": 15121,
+ "proportional": 15122,
+ "relate": 15123,
+ "VFL": 15124,
+ "##dez": 15125,
+ "continuity": 15126,
+ "##cene": 15127,
+ "syndicated": 15128,
+ "atmospheric": 15129,
+ "arrows": 15130,
+ "Wanderers": 15131,
+ "reinforcements": 15132,
+ "Willow": 15133,
+ "Lexington": 15134,
+ "Rotten": 15135,
+ "##yon": 15136,
+ "discovering": 15137,
+ "Serena": 15138,
+ "portable": 15139,
+ "##lysis": 15140,
+ "targeting": 15141,
+ "£1": 15142,
+ "Goodman": 15143,
+ "Steam": 15144,
+ "sensors": 15145,
+ "detachment": 15146,
+ "Malik": 15147,
+ "##erie": 15148,
+ "attitudes": 15149,
+ "Goes": 15150,
+ "Kendall": 15151,
+ "Read": 15152,
+ "Sleep": 15153,
+ "beans": 15154,
+ "Nikki": 15155,
+ "modification": 15156,
+ "Jeanne": 15157,
+ "knuckles": 15158,
+ "Eleven": 15159,
+ "##iously": 15160,
+ "Gross": 15161,
+ "Jaime": 15162,
+ "dioxide": 15163,
+ "moisture": 15164,
+ "Stones": 15165,
+ "UCI": 15166,
+ "displacement": 15167,
+ "Metacritic": 15168,
+ "Jury": 15169,
+ "lace": 15170,
+ "rendering": 15171,
+ "elephant": 15172,
+ "Sergei": 15173,
+ "##quire": 15174,
+ "GP": 15175,
+ "Abbott": 15176,
+ "##type": 15177,
+ "projection": 15178,
+ "Mouse": 15179,
+ "Bishops": 15180,
+ "whispering": 15181,
+ "Kathleen": 15182,
+ "Rams": 15183,
+ "##jar": 15184,
+ "whites": 15185,
+ "##oran": 15186,
+ "assess": 15187,
+ "dispatched": 15188,
+ "##hire": 15189,
+ "kin": 15190,
+ "##mir": 15191,
+ "Nursing": 15192,
+ "advocates": 15193,
+ "tremendous": 15194,
+ "sweater": 15195,
+ "assisting": 15196,
+ "##bil": 15197,
+ "Farmer": 15198,
+ "prominently": 15199,
+ "reddish": 15200,
+ "Hague": 15201,
+ "cyclone": 15202,
+ "##SD": 15203,
+ "Sage": 15204,
+ "Lawson": 15205,
+ "Sanctuary": 15206,
+ "discharged": 15207,
+ "retains": 15208,
+ "##ube": 15209,
+ "shotgun": 15210,
+ "wilderness": 15211,
+ "Reformed": 15212,
+ "similarity": 15213,
+ "Entry": 15214,
+ "Watts": 15215,
+ "Bahá": 15216,
+ "Quest": 15217,
+ "Looks": 15218,
+ "visions": 15219,
+ "Reservoir": 15220,
+ "Arabs": 15221,
+ "curls": 15222,
+ "Blu": 15223,
+ "dripping": 15224,
+ "accomplish": 15225,
+ "Verlag": 15226,
+ "drill": 15227,
+ "sensor": 15228,
+ "Dillon": 15229,
+ "physicians": 15230,
+ "smashed": 15231,
+ "##dir": 15232,
+ "painters": 15233,
+ "Renault": 15234,
+ "straw": 15235,
+ "fading": 15236,
+ "Directorate": 15237,
+ "lounge": 15238,
+ "commissions": 15239,
+ "Brain": 15240,
+ "##graph": 15241,
+ "neo": 15242,
+ "##urg": 15243,
+ "plug": 15244,
+ "coordinated": 15245,
+ "##houses": 15246,
+ "Critical": 15247,
+ "lamps": 15248,
+ "illustrator": 15249,
+ "Returning": 15250,
+ "erosion": 15251,
+ "Crow": 15252,
+ "##ciation": 15253,
+ "blessing": 15254,
+ "Thought": 15255,
+ "Wife": 15256,
+ "medalist": 15257,
+ "synthesizer": 15258,
+ "Pam": 15259,
+ "Thornton": 15260,
+ "Esther": 15261,
+ "HBO": 15262,
+ "fond": 15263,
+ "Associates": 15264,
+ "##raz": 15265,
+ "pirate": 15266,
+ "permits": 15267,
+ "Wide": 15268,
+ "tire": 15269,
+ "##PC": 15270,
+ "Ernie": 15271,
+ "Nassau": 15272,
+ "transferring": 15273,
+ "RFC": 15274,
+ "##ntly": 15275,
+ "um": 15276,
+ "spit": 15277,
+ "AS": 15278,
+ "##mps": 15279,
+ "Mining": 15280,
+ "polar": 15281,
+ "villa": 15282,
+ "anchored": 15283,
+ "##zzi": 15284,
+ "embarrassment": 15285,
+ "relates": 15286,
+ "##ă": 15287,
+ "Rupert": 15288,
+ "counterparts": 15289,
+ "131": 15290,
+ "Baxter": 15291,
+ "##18": 15292,
+ "Igor": 15293,
+ "recognizes": 15294,
+ "Clive": 15295,
+ "##hane": 15296,
+ "##eries": 15297,
+ "##ibly": 15298,
+ "occurrence": 15299,
+ "##scope": 15300,
+ "fin": 15301,
+ "colorful": 15302,
+ "Rapids": 15303,
+ "banker": 15304,
+ "tile": 15305,
+ "##rative": 15306,
+ "##dus": 15307,
+ "delays": 15308,
+ "destinations": 15309,
+ "##llis": 15310,
+ "Pond": 15311,
+ "Dane": 15312,
+ "grandparents": 15313,
+ "rewarded": 15314,
+ "socially": 15315,
+ "motorway": 15316,
+ "##hof": 15317,
+ "##lying": 15318,
+ "##human": 15319,
+ "modeled": 15320,
+ "Dayton": 15321,
+ "Forward": 15322,
+ "conscience": 15323,
+ "Sharma": 15324,
+ "whistle": 15325,
+ "Mayer": 15326,
+ "Sasha": 15327,
+ "##pical": 15328,
+ "circuits": 15329,
+ "Zhou": 15330,
+ "##ça": 15331,
+ "Latvian": 15332,
+ "finalists": 15333,
+ "predators": 15334,
+ "Lafayette": 15335,
+ "closes": 15336,
+ "obligations": 15337,
+ "Resolution": 15338,
+ "##vier": 15339,
+ "Trustees": 15340,
+ "reminiscent": 15341,
+ "##hos": 15342,
+ "Highlands": 15343,
+ "Protected": 15344,
+ "asylum": 15345,
+ "evacuation": 15346,
+ "##acy": 15347,
+ "Chevrolet": 15348,
+ "confession": 15349,
+ "Somalia": 15350,
+ "emergence": 15351,
+ "separating": 15352,
+ "##rica": 15353,
+ "alright": 15354,
+ "calcium": 15355,
+ "Laurent": 15356,
+ "Welfare": 15357,
+ "Leonardo": 15358,
+ "ashes": 15359,
+ "dental": 15360,
+ "Deal": 15361,
+ "minerals": 15362,
+ "##lump": 15363,
+ "##mount": 15364,
+ "accounted": 15365,
+ "staggered": 15366,
+ "slogan": 15367,
+ "photographic": 15368,
+ "builder": 15369,
+ "##imes": 15370,
+ "##raft": 15371,
+ "tragic": 15372,
+ "144": 15373,
+ "SEC": 15374,
+ "Hit": 15375,
+ "tailed": 15376,
+ "##ples": 15377,
+ "##rring": 15378,
+ "##rson": 15379,
+ "ethical": 15380,
+ "wrestlers": 15381,
+ "concludes": 15382,
+ "lunar": 15383,
+ "##ept": 15384,
+ "nitrogen": 15385,
+ "Aid": 15386,
+ "cyclist": 15387,
+ "quarterfinals": 15388,
+ "##ه": 15389,
+ "harvest": 15390,
+ "##hem": 15391,
+ "Pasha": 15392,
+ "IL": 15393,
+ "##mis": 15394,
+ "continually": 15395,
+ "##forth": 15396,
+ "Intel": 15397,
+ "bucket": 15398,
+ "##ended": 15399,
+ "witches": 15400,
+ "pretended": 15401,
+ "dresses": 15402,
+ "viewer": 15403,
+ "peculiar": 15404,
+ "lowering": 15405,
+ "volcano": 15406,
+ "Marilyn": 15407,
+ "Qualifier": 15408,
+ "clung": 15409,
+ "##sher": 15410,
+ "Cut": 15411,
+ "modules": 15412,
+ "Bowie": 15413,
+ "##lded": 15414,
+ "onset": 15415,
+ "transcription": 15416,
+ "residences": 15417,
+ "##pie": 15418,
+ "##itor": 15419,
+ "scrapped": 15420,
+ "##bic": 15421,
+ "Monaco": 15422,
+ "Mayo": 15423,
+ "eternity": 15424,
+ "Strike": 15425,
+ "uncovered": 15426,
+ "skeleton": 15427,
+ "##wicz": 15428,
+ "Isles": 15429,
+ "bug": 15430,
+ "Promoted": 15431,
+ "##rush": 15432,
+ "Mechanical": 15433,
+ "XII": 15434,
+ "##ivo": 15435,
+ "gripping": 15436,
+ "stubborn": 15437,
+ "velvet": 15438,
+ "TD": 15439,
+ "decommissioned": 15440,
+ "operas": 15441,
+ "spatial": 15442,
+ "unstable": 15443,
+ "Congressman": 15444,
+ "wasted": 15445,
+ "##aga": 15446,
+ "##ume": 15447,
+ "advertisements": 15448,
+ "##nya": 15449,
+ "obliged": 15450,
+ "Cannes": 15451,
+ "Conway": 15452,
+ "bricks": 15453,
+ "##gnant": 15454,
+ "##mity": 15455,
+ "##uise": 15456,
+ "jumps": 15457,
+ "Clear": 15458,
+ "##cine": 15459,
+ "##sche": 15460,
+ "chord": 15461,
+ "utter": 15462,
+ "Su": 15463,
+ "podium": 15464,
+ "spokesman": 15465,
+ "Royce": 15466,
+ "assassin": 15467,
+ "confirmation": 15468,
+ "licensing": 15469,
+ "liberty": 15470,
+ "##rata": 15471,
+ "Geographic": 15472,
+ "individually": 15473,
+ "detained": 15474,
+ "##ffe": 15475,
+ "Saturn": 15476,
+ "crushing": 15477,
+ "airplane": 15478,
+ "bushes": 15479,
+ "knights": 15480,
+ "##PD": 15481,
+ "Lilly": 15482,
+ "hurts": 15483,
+ "unexpectedly": 15484,
+ "Conservatives": 15485,
+ "pumping": 15486,
+ "Forty": 15487,
+ "candle": 15488,
+ "Pérez": 15489,
+ "peasants": 15490,
+ "supplement": 15491,
+ "Sundays": 15492,
+ "##ggs": 15493,
+ "##rries": 15494,
+ "risen": 15495,
+ "enthusiastic": 15496,
+ "corresponds": 15497,
+ "pending": 15498,
+ "##IF": 15499,
+ "Owens": 15500,
+ "floods": 15501,
+ "Painter": 15502,
+ "inflation": 15503,
+ "presumed": 15504,
+ "inscribed": 15505,
+ "Chamberlain": 15506,
+ "bizarre": 15507,
+ "1200": 15508,
+ "liability": 15509,
+ "reacted": 15510,
+ "tub": 15511,
+ "Legacy": 15512,
+ "##eds": 15513,
+ "##pted": 15514,
+ "shone": 15515,
+ "##litz": 15516,
+ "##NC": 15517,
+ "Tiny": 15518,
+ "genome": 15519,
+ "bays": 15520,
+ "Eduardo": 15521,
+ "robbery": 15522,
+ "stall": 15523,
+ "hatch": 15524,
+ "Depot": 15525,
+ "Variety": 15526,
+ "Flora": 15527,
+ "reprinted": 15528,
+ "trembled": 15529,
+ "outlined": 15530,
+ "CR": 15531,
+ "Theresa": 15532,
+ "spans": 15533,
+ "##plication": 15534,
+ "Jensen": 15535,
+ "##eering": 15536,
+ "posting": 15537,
+ "##rky": 15538,
+ "pays": 15539,
+ "##ost": 15540,
+ "Marcos": 15541,
+ "fortifications": 15542,
+ "inferior": 15543,
+ "##ential": 15544,
+ "Devi": 15545,
+ "despair": 15546,
+ "Talbot": 15547,
+ "##chus": 15548,
+ "updates": 15549,
+ "ego": 15550,
+ "Booth": 15551,
+ "Darius": 15552,
+ "tops": 15553,
+ "##lau": 15554,
+ "Scene": 15555,
+ "##DC": 15556,
+ "Harlem": 15557,
+ "Trey": 15558,
+ "Generally": 15559,
+ "candles": 15560,
+ "##α": 15561,
+ "Neville": 15562,
+ "Admiralty": 15563,
+ "##hong": 15564,
+ "iconic": 15565,
+ "victorious": 15566,
+ "1600": 15567,
+ "Rowan": 15568,
+ "abundance": 15569,
+ "miniseries": 15570,
+ "clutching": 15571,
+ "sanctioned": 15572,
+ "##words": 15573,
+ "obscure": 15574,
+ "##ision": 15575,
+ "##rle": 15576,
+ "##EM": 15577,
+ "disappearing": 15578,
+ "Resort": 15579,
+ "Obviously": 15580,
+ "##eb": 15581,
+ "exceeded": 15582,
+ "1870s": 15583,
+ "Adults": 15584,
+ "##cts": 15585,
+ "Cry": 15586,
+ "Kerr": 15587,
+ "ragged": 15588,
+ "selfish": 15589,
+ "##lson": 15590,
+ "circled": 15591,
+ "pillars": 15592,
+ "galaxy": 15593,
+ "##asco": 15594,
+ "##mental": 15595,
+ "rebuild": 15596,
+ "caution": 15597,
+ "Resistance": 15598,
+ "Start": 15599,
+ "bind": 15600,
+ "splitting": 15601,
+ "Baba": 15602,
+ "Hogan": 15603,
+ "ps": 15604,
+ "partnerships": 15605,
+ "slam": 15606,
+ "Peggy": 15607,
+ "courthouse": 15608,
+ "##OD": 15609,
+ "organizational": 15610,
+ "packages": 15611,
+ "Angie": 15612,
+ "##nds": 15613,
+ "possesses": 15614,
+ "##rp": 15615,
+ "Expressway": 15616,
+ "Gould": 15617,
+ "Terror": 15618,
+ "Him": 15619,
+ "Geoff": 15620,
+ "nobles": 15621,
+ "##ope": 15622,
+ "shark": 15623,
+ "##nh": 15624,
+ "identifies": 15625,
+ "##oor": 15626,
+ "testified": 15627,
+ "Playing": 15628,
+ "##ump": 15629,
+ "##isa": 15630,
+ "stool": 15631,
+ "Idol": 15632,
+ "##pice": 15633,
+ "##tana": 15634,
+ "Byrne": 15635,
+ "Gerry": 15636,
+ "grunted": 15637,
+ "26th": 15638,
+ "observing": 15639,
+ "habits": 15640,
+ "privilege": 15641,
+ "immortal": 15642,
+ "wagons": 15643,
+ "##thy": 15644,
+ "dot": 15645,
+ "Bring": 15646,
+ "##lian": 15647,
+ "##witz": 15648,
+ "newest": 15649,
+ "##uga": 15650,
+ "constraints": 15651,
+ "Screen": 15652,
+ "Issue": 15653,
+ "##RNA": 15654,
+ "##vil": 15655,
+ "reminder": 15656,
+ "##gles": 15657,
+ "addiction": 15658,
+ "piercing": 15659,
+ "stunning": 15660,
+ "var": 15661,
+ "##rita": 15662,
+ "Signal": 15663,
+ "accumulated": 15664,
+ "##wide": 15665,
+ "float": 15666,
+ "devastating": 15667,
+ "viable": 15668,
+ "cartoons": 15669,
+ "Uttar": 15670,
+ "flared": 15671,
+ "##encies": 15672,
+ "Theology": 15673,
+ "patents": 15674,
+ "##bahn": 15675,
+ "privileges": 15676,
+ "##ava": 15677,
+ "##CO": 15678,
+ "137": 15679,
+ "##oped": 15680,
+ "##NT": 15681,
+ "orchestral": 15682,
+ "medication": 15683,
+ "225": 15684,
+ "erect": 15685,
+ "Nadia": 15686,
+ "École": 15687,
+ "fried": 15688,
+ "Sales": 15689,
+ "scripts": 15690,
+ "##rease": 15691,
+ "airs": 15692,
+ "Cage": 15693,
+ "inadequate": 15694,
+ "structured": 15695,
+ "countless": 15696,
+ "Avengers": 15697,
+ "Kathy": 15698,
+ "disguise": 15699,
+ "mirrors": 15700,
+ "Investigation": 15701,
+ "reservation": 15702,
+ "##nson": 15703,
+ "Legends": 15704,
+ "humorous": 15705,
+ "Mona": 15706,
+ "decorations": 15707,
+ "attachment": 15708,
+ "Via": 15709,
+ "motivation": 15710,
+ "Browne": 15711,
+ "strangers": 15712,
+ "##ński": 15713,
+ "Shadows": 15714,
+ "Twins": 15715,
+ "##pressed": 15716,
+ "Alma": 15717,
+ "Nominated": 15718,
+ "##ott": 15719,
+ "Sergio": 15720,
+ "canopy": 15721,
+ "152": 15722,
+ "Semifinals": 15723,
+ "devised": 15724,
+ "##irk": 15725,
+ "upwards": 15726,
+ "Traffic": 15727,
+ "Goddess": 15728,
+ "Move": 15729,
+ "beetles": 15730,
+ "138": 15731,
+ "spat": 15732,
+ "##anne": 15733,
+ "holdings": 15734,
+ "##SP": 15735,
+ "tangled": 15736,
+ "Whilst": 15737,
+ "Fowler": 15738,
+ "anthem": 15739,
+ "##ING": 15740,
+ "##ogy": 15741,
+ "snarled": 15742,
+ "moonlight": 15743,
+ "songwriting": 15744,
+ "tolerance": 15745,
+ "Worlds": 15746,
+ "exams": 15747,
+ "##pia": 15748,
+ "notices": 15749,
+ "sensitivity": 15750,
+ "poetic": 15751,
+ "Stephens": 15752,
+ "Boone": 15753,
+ "insect": 15754,
+ "reconstructed": 15755,
+ "Fresh": 15756,
+ "27th": 15757,
+ "balloon": 15758,
+ "##ables": 15759,
+ "Brendan": 15760,
+ "mug": 15761,
+ "##gee": 15762,
+ "1780": 15763,
+ "apex": 15764,
+ "exports": 15765,
+ "slides": 15766,
+ "Lahore": 15767,
+ "hiring": 15768,
+ "Shell": 15769,
+ "electorate": 15770,
+ "sexuality": 15771,
+ "poker": 15772,
+ "nonprofit": 15773,
+ "##imate": 15774,
+ "cone": 15775,
+ "##uce": 15776,
+ "Okinawa": 15777,
+ "superintendent": 15778,
+ "##HC": 15779,
+ "referenced": 15780,
+ "turret": 15781,
+ "Sprint": 15782,
+ "Citizen": 15783,
+ "equilibrium": 15784,
+ "Stafford": 15785,
+ "curb": 15786,
+ "Driver": 15787,
+ "Valerie": 15788,
+ "##rona": 15789,
+ "aching": 15790,
+ "impacts": 15791,
+ "##bol": 15792,
+ "observers": 15793,
+ "Downs": 15794,
+ "Shri": 15795,
+ "##uth": 15796,
+ "airports": 15797,
+ "##uda": 15798,
+ "assignments": 15799,
+ "curtains": 15800,
+ "solitary": 15801,
+ "icon": 15802,
+ "patrols": 15803,
+ "substances": 15804,
+ "Jasper": 15805,
+ "mountainous": 15806,
+ "Published": 15807,
+ "ached": 15808,
+ "##ingly": 15809,
+ "announce": 15810,
+ "dove": 15811,
+ "damaging": 15812,
+ "##tism": 15813,
+ "Primera": 15814,
+ "Dexter": 15815,
+ "limiting": 15816,
+ "batch": 15817,
+ "##uli": 15818,
+ "undergoing": 15819,
+ "refugee": 15820,
+ "Ye": 15821,
+ "admiral": 15822,
+ "pavement": 15823,
+ "##WR": 15824,
+ "##reed": 15825,
+ "pipeline": 15826,
+ "desires": 15827,
+ "Ramsey": 15828,
+ "Sheila": 15829,
+ "thickness": 15830,
+ "Brotherhood": 15831,
+ "Tea": 15832,
+ "instituted": 15833,
+ "Belt": 15834,
+ "Break": 15835,
+ "plots": 15836,
+ "##ais": 15837,
+ "masculine": 15838,
+ "##where": 15839,
+ "Theo": 15840,
+ "##aged": 15841,
+ "##mined": 15842,
+ "Experience": 15843,
+ "scratched": 15844,
+ "Ethiopian": 15845,
+ "Teaching": 15846,
+ "##nov": 15847,
+ "Aiden": 15848,
+ "Abe": 15849,
+ "Samoa": 15850,
+ "conditioning": 15851,
+ "##mous": 15852,
+ "Otherwise": 15853,
+ "fade": 15854,
+ "Jenks": 15855,
+ "##encing": 15856,
+ "Nat": 15857,
+ "##lain": 15858,
+ "Anyone": 15859,
+ "##kis": 15860,
+ "smirk": 15861,
+ "Riding": 15862,
+ "##nny": 15863,
+ "Bavarian": 15864,
+ "blessed": 15865,
+ "potatoes": 15866,
+ "Hook": 15867,
+ "##wise": 15868,
+ "likewise": 15869,
+ "hardened": 15870,
+ "Merry": 15871,
+ "amid": 15872,
+ "persecution": 15873,
+ "##sten": 15874,
+ "Elections": 15875,
+ "Hoffman": 15876,
+ "Pitt": 15877,
+ "##vering": 15878,
+ "distraction": 15879,
+ "exploitation": 15880,
+ "infamous": 15881,
+ "quote": 15882,
+ "averaging": 15883,
+ "healed": 15884,
+ "Rhythm": 15885,
+ "Germanic": 15886,
+ "Mormon": 15887,
+ "illuminated": 15888,
+ "guides": 15889,
+ "##ische": 15890,
+ "interfere": 15891,
+ "##ilized": 15892,
+ "rector": 15893,
+ "perennial": 15894,
+ "##ival": 15895,
+ "Everett": 15896,
+ "courtesy": 15897,
+ "##nham": 15898,
+ "Kirby": 15899,
+ "Mk": 15900,
+ "##vic": 15901,
+ "Medieval": 15902,
+ "##tale": 15903,
+ "Luigi": 15904,
+ "limp": 15905,
+ "##diction": 15906,
+ "Alive": 15907,
+ "greeting": 15908,
+ "shove": 15909,
+ "##force": 15910,
+ "##fly": 15911,
+ "Jasmine": 15912,
+ "Bend": 15913,
+ "Capt": 15914,
+ "Suzanne": 15915,
+ "ditch": 15916,
+ "134": 15917,
+ "##nning": 15918,
+ "Host": 15919,
+ "fathers": 15920,
+ "rebuilding": 15921,
+ "Vocal": 15922,
+ "wires": 15923,
+ "##manship": 15924,
+ "tan": 15925,
+ "Factor": 15926,
+ "fixture": 15927,
+ "##LS": 15928,
+ "Māori": 15929,
+ "Plate": 15930,
+ "pyramid": 15931,
+ "##umble": 15932,
+ "slap": 15933,
+ "Schneider": 15934,
+ "yell": 15935,
+ "##ulture": 15936,
+ "##tional": 15937,
+ "Goodbye": 15938,
+ "sore": 15939,
+ "##pher": 15940,
+ "depressed": 15941,
+ "##dox": 15942,
+ "pitching": 15943,
+ "Find": 15944,
+ "Lotus": 15945,
+ "##wang": 15946,
+ "strand": 15947,
+ "Teen": 15948,
+ "debates": 15949,
+ "prevalent": 15950,
+ "##bilities": 15951,
+ "exposing": 15952,
+ "hears": 15953,
+ "billed": 15954,
+ "##rse": 15955,
+ "reorganized": 15956,
+ "compelled": 15957,
+ "disturbing": 15958,
+ "displaying": 15959,
+ "##tock": 15960,
+ "Clinical": 15961,
+ "emotionally": 15962,
+ "##iah": 15963,
+ "Derbyshire": 15964,
+ "grouped": 15965,
+ "##quel": 15966,
+ "Bahrain": 15967,
+ "Journalism": 15968,
+ "IN": 15969,
+ "persistent": 15970,
+ "blankets": 15971,
+ "Crane": 15972,
+ "camping": 15973,
+ "Direct": 15974,
+ "proving": 15975,
+ "Lola": 15976,
+ "##dding": 15977,
+ "Corporate": 15978,
+ "birthplace": 15979,
+ "##boats": 15980,
+ "##ender": 15981,
+ "Figure": 15982,
+ "dared": 15983,
+ "Assam": 15984,
+ "precursor": 15985,
+ "##nched": 15986,
+ "Tribe": 15987,
+ "Restoration": 15988,
+ "slate": 15989,
+ "Meyrick": 15990,
+ "hunted": 15991,
+ "stroking": 15992,
+ "Earlier": 15993,
+ "Kind": 15994,
+ "polls": 15995,
+ "appeals": 15996,
+ "monetary": 15997,
+ "##reate": 15998,
+ "Kira": 15999,
+ "Langdon": 16000,
+ "explores": 16001,
+ "GPS": 16002,
+ "extensions": 16003,
+ "squares": 16004,
+ "Results": 16005,
+ "draped": 16006,
+ "announcer": 16007,
+ "merit": 16008,
+ "##ennial": 16009,
+ "##tral": 16010,
+ "##roved": 16011,
+ "##cion": 16012,
+ "robots": 16013,
+ "supervisor": 16014,
+ "snorted": 16015,
+ "##group": 16016,
+ "Cannon": 16017,
+ "procession": 16018,
+ "monkey": 16019,
+ "freeze": 16020,
+ "sleeves": 16021,
+ "Nile": 16022,
+ "verdict": 16023,
+ "ropes": 16024,
+ "firearms": 16025,
+ "extraction": 16026,
+ "tensed": 16027,
+ "EC": 16028,
+ "Saunders": 16029,
+ "##tches": 16030,
+ "diamonds": 16031,
+ "Marriage": 16032,
+ "##amble": 16033,
+ "curling": 16034,
+ "Amazing": 16035,
+ "##haling": 16036,
+ "unrelated": 16037,
+ "##roads": 16038,
+ "Daughter": 16039,
+ "cum": 16040,
+ "discarded": 16041,
+ "kidney": 16042,
+ "cliffs": 16043,
+ "forested": 16044,
+ "Candy": 16045,
+ "##lap": 16046,
+ "authentic": 16047,
+ "tablet": 16048,
+ "notation": 16049,
+ "##nburg": 16050,
+ "Bulldogs": 16051,
+ "Callum": 16052,
+ "Meet": 16053,
+ "mouths": 16054,
+ "coated": 16055,
+ "##xe": 16056,
+ "Truman": 16057,
+ "combinations": 16058,
+ "##mation": 16059,
+ "Steelers": 16060,
+ "Fan": 16061,
+ "Than": 16062,
+ "paternal": 16063,
+ "##father": 16064,
+ "##uti": 16065,
+ "Rebellion": 16066,
+ "inviting": 16067,
+ "Fun": 16068,
+ "theatres": 16069,
+ "##ي": 16070,
+ "##rom": 16071,
+ "curator": 16072,
+ "##cision": 16073,
+ "networking": 16074,
+ "Oz": 16075,
+ "drought": 16076,
+ "##ssel": 16077,
+ "granting": 16078,
+ "MBA": 16079,
+ "Shelby": 16080,
+ "Elaine": 16081,
+ "jealousy": 16082,
+ "Kyoto": 16083,
+ "shores": 16084,
+ "signaling": 16085,
+ "tenants": 16086,
+ "debated": 16087,
+ "Intermediate": 16088,
+ "Wise": 16089,
+ "##hes": 16090,
+ "##pu": 16091,
+ "Havana": 16092,
+ "duke": 16093,
+ "vicious": 16094,
+ "exited": 16095,
+ "servers": 16096,
+ "Nonetheless": 16097,
+ "Reports": 16098,
+ "explode": 16099,
+ "##beth": 16100,
+ "Nationals": 16101,
+ "offerings": 16102,
+ "Oval": 16103,
+ "conferred": 16104,
+ "eponymous": 16105,
+ "folklore": 16106,
+ "##NR": 16107,
+ "Shire": 16108,
+ "planting": 16109,
+ "1783": 16110,
+ "Zeus": 16111,
+ "accelerated": 16112,
+ "Constable": 16113,
+ "consuming": 16114,
+ "troubles": 16115,
+ "McCartney": 16116,
+ "texture": 16117,
+ "bust": 16118,
+ "Immigration": 16119,
+ "excavated": 16120,
+ "hopefully": 16121,
+ "##cession": 16122,
+ "##coe": 16123,
+ "##name": 16124,
+ "##ully": 16125,
+ "lining": 16126,
+ "Einstein": 16127,
+ "Venezuelan": 16128,
+ "reissued": 16129,
+ "minorities": 16130,
+ "Beatrice": 16131,
+ "crystals": 16132,
+ "##nies": 16133,
+ "circus": 16134,
+ "lava": 16135,
+ "Beirut": 16136,
+ "extinction": 16137,
+ "##shu": 16138,
+ "Becker": 16139,
+ "##uke": 16140,
+ "issuing": 16141,
+ "Zurich": 16142,
+ "extract": 16143,
+ "##esta": 16144,
+ "##rred": 16145,
+ "regulate": 16146,
+ "progression": 16147,
+ "hut": 16148,
+ "alcoholic": 16149,
+ "plea": 16150,
+ "AB": 16151,
+ "Norse": 16152,
+ "Hubert": 16153,
+ "Mansfield": 16154,
+ "ashamed": 16155,
+ "##put": 16156,
+ "Bombardment": 16157,
+ "stripes": 16158,
+ "electrons": 16159,
+ "Denise": 16160,
+ "horrified": 16161,
+ "Nor": 16162,
+ "arranger": 16163,
+ "Hay": 16164,
+ "Koch": 16165,
+ "##ddling": 16166,
+ "##iner": 16167,
+ "Birthday": 16168,
+ "Josie": 16169,
+ "deliberate": 16170,
+ "explorer": 16171,
+ "##jiang": 16172,
+ "##signed": 16173,
+ "Arrow": 16174,
+ "wiping": 16175,
+ "satellites": 16176,
+ "baritone": 16177,
+ "mobility": 16178,
+ "##rals": 16179,
+ "Dorset": 16180,
+ "turbine": 16181,
+ "Coffee": 16182,
+ "185": 16183,
+ "##lder": 16184,
+ "Cara": 16185,
+ "Colts": 16186,
+ "pits": 16187,
+ "Crossing": 16188,
+ "coral": 16189,
+ "##birth": 16190,
+ "Tai": 16191,
+ "zombie": 16192,
+ "smoothly": 16193,
+ "##hp": 16194,
+ "mates": 16195,
+ "##ady": 16196,
+ "Marguerite": 16197,
+ "##tary": 16198,
+ "puzzled": 16199,
+ "tapes": 16200,
+ "overly": 16201,
+ "Sonic": 16202,
+ "Prayer": 16203,
+ "Thinking": 16204,
+ "##uf": 16205,
+ "IEEE": 16206,
+ "obligation": 16207,
+ "##cliffe": 16208,
+ "Basil": 16209,
+ "redesignated": 16210,
+ "##mmy": 16211,
+ "nostrils": 16212,
+ "Barney": 16213,
+ "XIII": 16214,
+ "##phones": 16215,
+ "vacated": 16216,
+ "unused": 16217,
+ "Berg": 16218,
+ "##roid": 16219,
+ "Towards": 16220,
+ "viola": 16221,
+ "136": 16222,
+ "Event": 16223,
+ "subdivided": 16224,
+ "rabbit": 16225,
+ "recruiting": 16226,
+ "##nery": 16227,
+ "Namibia": 16228,
+ "##16": 16229,
+ "##ilation": 16230,
+ "recruits": 16231,
+ "Famous": 16232,
+ "Francesca": 16233,
+ "##hari": 16234,
+ "Goa": 16235,
+ "##lat": 16236,
+ "Karachi": 16237,
+ "haul": 16238,
+ "biblical": 16239,
+ "##cible": 16240,
+ "MGM": 16241,
+ "##rta": 16242,
+ "horsepower": 16243,
+ "profitable": 16244,
+ "Grandma": 16245,
+ "importantly": 16246,
+ "Martinez": 16247,
+ "incoming": 16248,
+ "##kill": 16249,
+ "beneficial": 16250,
+ "nominal": 16251,
+ "praying": 16252,
+ "##isch": 16253,
+ "gable": 16254,
+ "nail": 16255,
+ "noises": 16256,
+ "##ttle": 16257,
+ "Polytechnic": 16258,
+ "rub": 16259,
+ "##cope": 16260,
+ "Thor": 16261,
+ "audition": 16262,
+ "erotic": 16263,
+ "##ending": 16264,
+ "##iano": 16265,
+ "Ultimately": 16266,
+ "armoured": 16267,
+ "##mum": 16268,
+ "presently": 16269,
+ "pedestrian": 16270,
+ "##tled": 16271,
+ "Ipswich": 16272,
+ "offence": 16273,
+ "##ffin": 16274,
+ "##borne": 16275,
+ "Flemish": 16276,
+ "##hman": 16277,
+ "echo": 16278,
+ "##cting": 16279,
+ "auditorium": 16280,
+ "gentlemen": 16281,
+ "winged": 16282,
+ "##tched": 16283,
+ "Nicaragua": 16284,
+ "Unknown": 16285,
+ "prosperity": 16286,
+ "exhaust": 16287,
+ "pie": 16288,
+ "Peruvian": 16289,
+ "compartment": 16290,
+ "heights": 16291,
+ "disabilities": 16292,
+ "##pole": 16293,
+ "Harding": 16294,
+ "Humphrey": 16295,
+ "postponed": 16296,
+ "moths": 16297,
+ "Mathematical": 16298,
+ "Mets": 16299,
+ "posters": 16300,
+ "axe": 16301,
+ "##nett": 16302,
+ "Nights": 16303,
+ "Typically": 16304,
+ "chuckle": 16305,
+ "councillors": 16306,
+ "alternating": 16307,
+ "141": 16308,
+ "Norris": 16309,
+ "##ately": 16310,
+ "##etus": 16311,
+ "deficit": 16312,
+ "dreaming": 16313,
+ "cooler": 16314,
+ "oppose": 16315,
+ "Beethoven": 16316,
+ "##esis": 16317,
+ "Marquis": 16318,
+ "flashlight": 16319,
+ "headache": 16320,
+ "investor": 16321,
+ "responding": 16322,
+ "appointments": 16323,
+ "##shore": 16324,
+ "Elias": 16325,
+ "ideals": 16326,
+ "shades": 16327,
+ "torch": 16328,
+ "lingering": 16329,
+ "##real": 16330,
+ "pier": 16331,
+ "fertile": 16332,
+ "Diploma": 16333,
+ "currents": 16334,
+ "Snake": 16335,
+ "##horse": 16336,
+ "##15": 16337,
+ "Briggs": 16338,
+ "##ota": 16339,
+ "##hima": 16340,
+ "##romatic": 16341,
+ "Coastal": 16342,
+ "Kuala": 16343,
+ "ankles": 16344,
+ "Rae": 16345,
+ "slice": 16346,
+ "Hilton": 16347,
+ "locking": 16348,
+ "Approximately": 16349,
+ "Workshop": 16350,
+ "Niagara": 16351,
+ "strangely": 16352,
+ "##scence": 16353,
+ "functionality": 16354,
+ "advertisement": 16355,
+ "Rapid": 16356,
+ "Anders": 16357,
+ "ho": 16358,
+ "Soviets": 16359,
+ "packing": 16360,
+ "basal": 16361,
+ "Sunderland": 16362,
+ "Permanent": 16363,
+ "##fting": 16364,
+ "rack": 16365,
+ "tying": 16366,
+ "Lowell": 16367,
+ "##ncing": 16368,
+ "Wizard": 16369,
+ "mighty": 16370,
+ "tertiary": 16371,
+ "pencil": 16372,
+ "dismissal": 16373,
+ "torso": 16374,
+ "grasped": 16375,
+ "##yev": 16376,
+ "Sand": 16377,
+ "gossip": 16378,
+ "##nae": 16379,
+ "Beer": 16380,
+ "implementing": 16381,
+ "##19": 16382,
+ "##riya": 16383,
+ "Fork": 16384,
+ "Bee": 16385,
+ "##eria": 16386,
+ "Win": 16387,
+ "##cid": 16388,
+ "sailor": 16389,
+ "pressures": 16390,
+ "##oping": 16391,
+ "speculated": 16392,
+ "Freddie": 16393,
+ "originating": 16394,
+ "##DF": 16395,
+ "##SR": 16396,
+ "##outh": 16397,
+ "28th": 16398,
+ "melt": 16399,
+ "Brenda": 16400,
+ "lump": 16401,
+ "Burlington": 16402,
+ "USC": 16403,
+ "marginal": 16404,
+ "##bine": 16405,
+ "Dogs": 16406,
+ "swamp": 16407,
+ "cu": 16408,
+ "Ex": 16409,
+ "uranium": 16410,
+ "metro": 16411,
+ "spill": 16412,
+ "Pietro": 16413,
+ "seize": 16414,
+ "Chorus": 16415,
+ "partition": 16416,
+ "##dock": 16417,
+ "##media": 16418,
+ "engineered": 16419,
+ "##oria": 16420,
+ "conclusions": 16421,
+ "subdivision": 16422,
+ "##uid": 16423,
+ "Illustrated": 16424,
+ "Leading": 16425,
+ "##hora": 16426,
+ "Berkshire": 16427,
+ "definite": 16428,
+ "##books": 16429,
+ "##cin": 16430,
+ "##suke": 16431,
+ "noun": 16432,
+ "winced": 16433,
+ "Doris": 16434,
+ "dissertation": 16435,
+ "Wilderness": 16436,
+ "##quest": 16437,
+ "braced": 16438,
+ "arbitrary": 16439,
+ "kidnapping": 16440,
+ "Kurdish": 16441,
+ "##but": 16442,
+ "clearance": 16443,
+ "excavations": 16444,
+ "wanna": 16445,
+ "Allmusic": 16446,
+ "insult": 16447,
+ "presided": 16448,
+ "yacht": 16449,
+ "##SM": 16450,
+ "Honour": 16451,
+ "Tin": 16452,
+ "attracting": 16453,
+ "explosives": 16454,
+ "Gore": 16455,
+ "Bride": 16456,
+ "##ience": 16457,
+ "Packers": 16458,
+ "Devils": 16459,
+ "Observer": 16460,
+ "##course": 16461,
+ "Loser": 16462,
+ "##erry": 16463,
+ "##hardt": 16464,
+ "##mble": 16465,
+ "Cyrillic": 16466,
+ "undefeated": 16467,
+ "##stra": 16468,
+ "subordinate": 16469,
+ "##ame": 16470,
+ "Wigan": 16471,
+ "compulsory": 16472,
+ "Pauline": 16473,
+ "Cruise": 16474,
+ "Opposition": 16475,
+ "##ods": 16476,
+ "Period": 16477,
+ "dispersed": 16478,
+ "expose": 16479,
+ "##60": 16480,
+ "##has": 16481,
+ "Certain": 16482,
+ "Clerk": 16483,
+ "Wolves": 16484,
+ "##hibition": 16485,
+ "apparatus": 16486,
+ "allegiance": 16487,
+ "orbital": 16488,
+ "justified": 16489,
+ "thanked": 16490,
+ "##ević": 16491,
+ "Biblical": 16492,
+ "Carolyn": 16493,
+ "Graves": 16494,
+ "##tton": 16495,
+ "Hercules": 16496,
+ "backgrounds": 16497,
+ "replica": 16498,
+ "1788": 16499,
+ "aquatic": 16500,
+ "Mega": 16501,
+ "Stirling": 16502,
+ "obstacles": 16503,
+ "filing": 16504,
+ "Founder": 16505,
+ "vowels": 16506,
+ "Deborah": 16507,
+ "Rotterdam": 16508,
+ "surpassed": 16509,
+ "Belarusian": 16510,
+ "##ologists": 16511,
+ "Zambia": 16512,
+ "Ren": 16513,
+ "Olga": 16514,
+ "Alpine": 16515,
+ "bi": 16516,
+ "councillor": 16517,
+ "Oaks": 16518,
+ "Animals": 16519,
+ "eliminating": 16520,
+ "digit": 16521,
+ "Managing": 16522,
+ "##GE": 16523,
+ "laundry": 16524,
+ "##rdo": 16525,
+ "presses": 16526,
+ "slamming": 16527,
+ "Tudor": 16528,
+ "thief": 16529,
+ "posterior": 16530,
+ "##bas": 16531,
+ "Rodgers": 16532,
+ "smells": 16533,
+ "##ining": 16534,
+ "Hole": 16535,
+ "SUV": 16536,
+ "trombone": 16537,
+ "numbering": 16538,
+ "representations": 16539,
+ "Domingo": 16540,
+ "Paralympics": 16541,
+ "cartridge": 16542,
+ "##rash": 16543,
+ "Combined": 16544,
+ "shelves": 16545,
+ "Kraków": 16546,
+ "revision": 16547,
+ "##frame": 16548,
+ "Sánchez": 16549,
+ "##tracted": 16550,
+ "##bler": 16551,
+ "Alain": 16552,
+ "townships": 16553,
+ "sic": 16554,
+ "trousers": 16555,
+ "Gibbs": 16556,
+ "anterior": 16557,
+ "symmetry": 16558,
+ "vaguely": 16559,
+ "Castile": 16560,
+ "IRA": 16561,
+ "resembling": 16562,
+ "Penguin": 16563,
+ "##ulent": 16564,
+ "infections": 16565,
+ "##stant": 16566,
+ "raped": 16567,
+ "##pressive": 16568,
+ "worrying": 16569,
+ "brains": 16570,
+ "bending": 16571,
+ "JR": 16572,
+ "Evidence": 16573,
+ "Venetian": 16574,
+ "complexes": 16575,
+ "Jonah": 16576,
+ "850": 16577,
+ "exported": 16578,
+ "Ambrose": 16579,
+ "Gap": 16580,
+ "philanthropist": 16581,
+ "##atus": 16582,
+ "Marxist": 16583,
+ "weighing": 16584,
+ "##KO": 16585,
+ "##nath": 16586,
+ "Soldiers": 16587,
+ "chiefs": 16588,
+ "reject": 16589,
+ "repeating": 16590,
+ "shaky": 16591,
+ "Zürich": 16592,
+ "preserving": 16593,
+ "##xin": 16594,
+ "cigarettes": 16595,
+ "##break": 16596,
+ "mortar": 16597,
+ "##fin": 16598,
+ "Already": 16599,
+ "reproduction": 16600,
+ "socks": 16601,
+ "Waiting": 16602,
+ "amazed": 16603,
+ "##aca": 16604,
+ "dash": 16605,
+ "##path": 16606,
+ "Airborne": 16607,
+ "##harf": 16608,
+ "##get": 16609,
+ "descending": 16610,
+ "OBE": 16611,
+ "Sant": 16612,
+ "Tess": 16613,
+ "Lucius": 16614,
+ "enjoys": 16615,
+ "##ttered": 16616,
+ "##ivation": 16617,
+ "##ete": 16618,
+ "Leinster": 16619,
+ "Phillies": 16620,
+ "execute": 16621,
+ "geological": 16622,
+ "unfinished": 16623,
+ "Courts": 16624,
+ "SP": 16625,
+ "Beaver": 16626,
+ "Duck": 16627,
+ "motions": 16628,
+ "Platinum": 16629,
+ "friction": 16630,
+ "##aud": 16631,
+ "##bet": 16632,
+ "Parts": 16633,
+ "Stade": 16634,
+ "entirety": 16635,
+ "sprang": 16636,
+ "Smithsonian": 16637,
+ "coffin": 16638,
+ "prolonged": 16639,
+ "Borneo": 16640,
+ "##vise": 16641,
+ "unanimously": 16642,
+ "##uchi": 16643,
+ "Cars": 16644,
+ "Cassandra": 16645,
+ "Australians": 16646,
+ "##CT": 16647,
+ "##rgen": 16648,
+ "Louisa": 16649,
+ "spur": 16650,
+ "Constance": 16651,
+ "##lities": 16652,
+ "Patent": 16653,
+ "racism": 16654,
+ "tempo": 16655,
+ "##ssion": 16656,
+ "##chard": 16657,
+ "##nology": 16658,
+ "##claim": 16659,
+ "Million": 16660,
+ "Nichols": 16661,
+ "##dah": 16662,
+ "Numerous": 16663,
+ "ing": 16664,
+ "Pure": 16665,
+ "plantations": 16666,
+ "donor": 16667,
+ "##EP": 16668,
+ "##rip": 16669,
+ "convenience": 16670,
+ "##plate": 16671,
+ "dots": 16672,
+ "indirect": 16673,
+ "##written": 16674,
+ "Dong": 16675,
+ "failures": 16676,
+ "adapt": 16677,
+ "wizard": 16678,
+ "unfortunately": 16679,
+ "##gion": 16680,
+ "practitioners": 16681,
+ "economically": 16682,
+ "Enrique": 16683,
+ "unchanged": 16684,
+ "kingdoms": 16685,
+ "refined": 16686,
+ "definitions": 16687,
+ "lazy": 16688,
+ "worries": 16689,
+ "railing": 16690,
+ "##nay": 16691,
+ "Kaiser": 16692,
+ "##lug": 16693,
+ "cracks": 16694,
+ "sells": 16695,
+ "ninety": 16696,
+ "##WC": 16697,
+ "Directed": 16698,
+ "denotes": 16699,
+ "developmental": 16700,
+ "papal": 16701,
+ "unfortunate": 16702,
+ "disappointing": 16703,
+ "sixteenth": 16704,
+ "Jen": 16705,
+ "##urier": 16706,
+ "NWA": 16707,
+ "drifting": 16708,
+ "Horror": 16709,
+ "##chemical": 16710,
+ "behaviors": 16711,
+ "bury": 16712,
+ "surfaced": 16713,
+ "foreigners": 16714,
+ "slick": 16715,
+ "AND": 16716,
+ "##rene": 16717,
+ "##ditions": 16718,
+ "##teral": 16719,
+ "scrap": 16720,
+ "kicks": 16721,
+ "comprise": 16722,
+ "buddy": 16723,
+ "##anda": 16724,
+ "Mental": 16725,
+ "##ype": 16726,
+ "Dom": 16727,
+ "wines": 16728,
+ "Limerick": 16729,
+ "Luca": 16730,
+ "Rand": 16731,
+ "##won": 16732,
+ "Tomatoes": 16733,
+ "homage": 16734,
+ "geometric": 16735,
+ "##nted": 16736,
+ "telescope": 16737,
+ "Shelley": 16738,
+ "poles": 16739,
+ "##fan": 16740,
+ "shareholders": 16741,
+ "Autonomous": 16742,
+ "cope": 16743,
+ "intensified": 16744,
+ "Genoa": 16745,
+ "Reformation": 16746,
+ "grazing": 16747,
+ "##tern": 16748,
+ "Zhao": 16749,
+ "provisional": 16750,
+ "##bies": 16751,
+ "Con": 16752,
+ "##riel": 16753,
+ "Cynthia": 16754,
+ "Raleigh": 16755,
+ "vivid": 16756,
+ "threaten": 16757,
+ "Length": 16758,
+ "subscription": 16759,
+ "roses": 16760,
+ "Müller": 16761,
+ "##isms": 16762,
+ "robin": 16763,
+ "##tial": 16764,
+ "Laos": 16765,
+ "Stanton": 16766,
+ "nationalism": 16767,
+ "##clave": 16768,
+ "##ND": 16769,
+ "##17": 16770,
+ "##zz": 16771,
+ "staging": 16772,
+ "Busch": 16773,
+ "Cindy": 16774,
+ "relieve": 16775,
+ "##spective": 16776,
+ "packs": 16777,
+ "neglected": 16778,
+ "CBE": 16779,
+ "alpine": 16780,
+ "Evolution": 16781,
+ "uneasy": 16782,
+ "coastline": 16783,
+ "Destiny": 16784,
+ "Barber": 16785,
+ "Julio": 16786,
+ "##tted": 16787,
+ "informs": 16788,
+ "unprecedented": 16789,
+ "Pavilion": 16790,
+ "##bei": 16791,
+ "##ference": 16792,
+ "betrayal": 16793,
+ "awaiting": 16794,
+ "leaked": 16795,
+ "V8": 16796,
+ "puppet": 16797,
+ "adverse": 16798,
+ "Bourne": 16799,
+ "Sunset": 16800,
+ "collectors": 16801,
+ "##glass": 16802,
+ "##sque": 16803,
+ "copied": 16804,
+ "Demon": 16805,
+ "conceded": 16806,
+ "resembled": 16807,
+ "Rafe": 16808,
+ "Levy": 16809,
+ "prosecutor": 16810,
+ "##ject": 16811,
+ "flora": 16812,
+ "manned": 16813,
+ "deaf": 16814,
+ "Mosque": 16815,
+ "reminds": 16816,
+ "Lizzie": 16817,
+ "Products": 16818,
+ "Funny": 16819,
+ "cassette": 16820,
+ "congress": 16821,
+ "##rong": 16822,
+ "Rover": 16823,
+ "tossing": 16824,
+ "prompting": 16825,
+ "chooses": 16826,
+ "Satellite": 16827,
+ "cautiously": 16828,
+ "Reese": 16829,
+ "##UT": 16830,
+ "Huang": 16831,
+ "Gloucestershire": 16832,
+ "giggled": 16833,
+ "Kitty": 16834,
+ "##å": 16835,
+ "Pleasant": 16836,
+ "Aye": 16837,
+ "##ond": 16838,
+ "judging": 16839,
+ "1860s": 16840,
+ "intentionally": 16841,
+ "Hurling": 16842,
+ "aggression": 16843,
+ "##xy": 16844,
+ "transfers": 16845,
+ "employing": 16846,
+ "##fies": 16847,
+ "##oda": 16848,
+ "Archibald": 16849,
+ "Blessed": 16850,
+ "Ski": 16851,
+ "flavor": 16852,
+ "Rosie": 16853,
+ "##burgh": 16854,
+ "sunset": 16855,
+ "Scholarship": 16856,
+ "WC": 16857,
+ "surround": 16858,
+ "ranged": 16859,
+ "##jay": 16860,
+ "Degree": 16861,
+ "Houses": 16862,
+ "squeezing": 16863,
+ "limb": 16864,
+ "premium": 16865,
+ "Leningrad": 16866,
+ "steals": 16867,
+ "##inated": 16868,
+ "##ssie": 16869,
+ "madness": 16870,
+ "vacancy": 16871,
+ "hydraulic": 16872,
+ "Northampton": 16873,
+ "##prise": 16874,
+ "Marks": 16875,
+ "Boxing": 16876,
+ "##fying": 16877,
+ "academics": 16878,
+ "##lich": 16879,
+ "##TY": 16880,
+ "CDs": 16881,
+ "##lma": 16882,
+ "hardcore": 16883,
+ "monitors": 16884,
+ "paperback": 16885,
+ "cables": 16886,
+ "Dimitri": 16887,
+ "upside": 16888,
+ "advent": 16889,
+ "Ra": 16890,
+ "##clusive": 16891,
+ "Aug": 16892,
+ "Christchurch": 16893,
+ "objected": 16894,
+ "stalked": 16895,
+ "Simple": 16896,
+ "colonists": 16897,
+ "##laid": 16898,
+ "CT": 16899,
+ "discusses": 16900,
+ "fellowship": 16901,
+ "Carnival": 16902,
+ "cares": 16903,
+ "Miracle": 16904,
+ "pastoral": 16905,
+ "rooted": 16906,
+ "shortage": 16907,
+ "borne": 16908,
+ "Quentin": 16909,
+ "meditation": 16910,
+ "tapping": 16911,
+ "Novel": 16912,
+ "##ades": 16913,
+ "Alicia": 16914,
+ "Burn": 16915,
+ "famed": 16916,
+ "residency": 16917,
+ "Fernández": 16918,
+ "Johannesburg": 16919,
+ "Zhu": 16920,
+ "offended": 16921,
+ "Mao": 16922,
+ "outward": 16923,
+ "##inas": 16924,
+ "XV": 16925,
+ "denial": 16926,
+ "noticing": 16927,
+ "##ís": 16928,
+ "quarry": 16929,
+ "##hound": 16930,
+ "##amo": 16931,
+ "Bernie": 16932,
+ "Bentley": 16933,
+ "Joanna": 16934,
+ "mortgage": 16935,
+ "##rdi": 16936,
+ "##sumption": 16937,
+ "lenses": 16938,
+ "extracted": 16939,
+ "depiction": 16940,
+ "##RE": 16941,
+ "Networks": 16942,
+ "Broad": 16943,
+ "Revenue": 16944,
+ "flickered": 16945,
+ "virgin": 16946,
+ "flanked": 16947,
+ "##о": 16948,
+ "Enterprises": 16949,
+ "probable": 16950,
+ "Liberals": 16951,
+ "Falcons": 16952,
+ "drowning": 16953,
+ "phrases": 16954,
+ "loads": 16955,
+ "assumes": 16956,
+ "inhaled": 16957,
+ "awe": 16958,
+ "logs": 16959,
+ "slightest": 16960,
+ "spiders": 16961,
+ "waterfall": 16962,
+ "##pate": 16963,
+ "rocking": 16964,
+ "shrub": 16965,
+ "##uil": 16966,
+ "roofs": 16967,
+ "##gard": 16968,
+ "prehistoric": 16969,
+ "wary": 16970,
+ "##rak": 16971,
+ "TO": 16972,
+ "clips": 16973,
+ "sustain": 16974,
+ "treason": 16975,
+ "microphone": 16976,
+ "voter": 16977,
+ "Lamb": 16978,
+ "psychologist": 16979,
+ "wrinkled": 16980,
+ "##ères": 16981,
+ "mating": 16982,
+ "Carrier": 16983,
+ "340": 16984,
+ "##lbert": 16985,
+ "sensing": 16986,
+ "##rino": 16987,
+ "destiny": 16988,
+ "distract": 16989,
+ "weaker": 16990,
+ "UC": 16991,
+ "Nearly": 16992,
+ "neurons": 16993,
+ "spends": 16994,
+ "Apache": 16995,
+ "##rem": 16996,
+ "genuinely": 16997,
+ "wells": 16998,
+ "##lanted": 16999,
+ "stereo": 17000,
+ "##girl": 17001,
+ "Lois": 17002,
+ "Leaving": 17003,
+ "consul": 17004,
+ "fungi": 17005,
+ "Pier": 17006,
+ "Cyril": 17007,
+ "80s": 17008,
+ "Jungle": 17009,
+ "##tani": 17010,
+ "illustration": 17011,
+ "Split": 17012,
+ "##hana": 17013,
+ "Abigail": 17014,
+ "##patrick": 17015,
+ "1787": 17016,
+ "diminished": 17017,
+ "Selected": 17018,
+ "packaging": 17019,
+ "##EG": 17020,
+ "Martínez": 17021,
+ "communal": 17022,
+ "Manufacturing": 17023,
+ "sentiment": 17024,
+ "143": 17025,
+ "unwilling": 17026,
+ "praising": 17027,
+ "Citation": 17028,
+ "pills": 17029,
+ "##iti": 17030,
+ "##rax": 17031,
+ "muffled": 17032,
+ "neatly": 17033,
+ "workforce": 17034,
+ "Yep": 17035,
+ "leisure": 17036,
+ "Tu": 17037,
+ "##nding": 17038,
+ "Wakefield": 17039,
+ "ancestral": 17040,
+ "##uki": 17041,
+ "destructive": 17042,
+ "seas": 17043,
+ "Passion": 17044,
+ "showcase": 17045,
+ "##ceptive": 17046,
+ "heroic": 17047,
+ "142": 17048,
+ "exhaustion": 17049,
+ "Customs": 17050,
+ "##aker": 17051,
+ "Scholar": 17052,
+ "sliced": 17053,
+ "##inian": 17054,
+ "Direction": 17055,
+ "##OW": 17056,
+ "Swansea": 17057,
+ "aluminium": 17058,
+ "##eep": 17059,
+ "ceramic": 17060,
+ "McCoy": 17061,
+ "Career": 17062,
+ "Sector": 17063,
+ "chartered": 17064,
+ "Damascus": 17065,
+ "pictured": 17066,
+ "Interest": 17067,
+ "stiffened": 17068,
+ "Plateau": 17069,
+ "obsolete": 17070,
+ "##tant": 17071,
+ "irritated": 17072,
+ "inappropriate": 17073,
+ "overs": 17074,
+ "##nko": 17075,
+ "bail": 17076,
+ "Talent": 17077,
+ "Sur": 17078,
+ "ours": 17079,
+ "##nah": 17080,
+ "barred": 17081,
+ "legged": 17082,
+ "sociology": 17083,
+ "Bud": 17084,
+ "dictionary": 17085,
+ "##luk": 17086,
+ "Cover": 17087,
+ "obey": 17088,
+ "##oring": 17089,
+ "annoying": 17090,
+ "##dong": 17091,
+ "apprentice": 17092,
+ "Cyrus": 17093,
+ "Role": 17094,
+ "##GP": 17095,
+ "##uns": 17096,
+ "##bag": 17097,
+ "Greenland": 17098,
+ "Porsche": 17099,
+ "Rocket": 17100,
+ "##32": 17101,
+ "organism": 17102,
+ "##ntary": 17103,
+ "reliability": 17104,
+ "##vocation": 17105,
+ "##й": 17106,
+ "Found": 17107,
+ "##hine": 17108,
+ "motors": 17109,
+ "promoter": 17110,
+ "unfair": 17111,
+ "##oms": 17112,
+ "##note": 17113,
+ "distribute": 17114,
+ "eminent": 17115,
+ "rails": 17116,
+ "appealing": 17117,
+ "chiefly": 17118,
+ "meaningful": 17119,
+ "Stephan": 17120,
+ "##rehension": 17121,
+ "Consumer": 17122,
+ "psychiatric": 17123,
+ "bowler": 17124,
+ "saints": 17125,
+ "##iful": 17126,
+ "##н": 17127,
+ "1777": 17128,
+ "Pol": 17129,
+ "Dorian": 17130,
+ "Townsend": 17131,
+ "hastily": 17132,
+ "##jima": 17133,
+ "Quincy": 17134,
+ "Sol": 17135,
+ "fascinated": 17136,
+ "Scarlet": 17137,
+ "alto": 17138,
+ "Avon": 17139,
+ "certainty": 17140,
+ "##eding": 17141,
+ "Keys": 17142,
+ "##chu": 17143,
+ "Chu": 17144,
+ "##VE": 17145,
+ "ions": 17146,
+ "tributaries": 17147,
+ "Thanksgiving": 17148,
+ "##fusion": 17149,
+ "astronomer": 17150,
+ "oxide": 17151,
+ "pavilion": 17152,
+ "Supply": 17153,
+ "Casa": 17154,
+ "Bollywood": 17155,
+ "sadly": 17156,
+ "mutations": 17157,
+ "Keller": 17158,
+ "##wave": 17159,
+ "nationals": 17160,
+ "##rgo": 17161,
+ "##ym": 17162,
+ "predict": 17163,
+ "Catholicism": 17164,
+ "Vega": 17165,
+ "##eration": 17166,
+ "##ums": 17167,
+ "Mali": 17168,
+ "tuned": 17169,
+ "Lankan": 17170,
+ "Plans": 17171,
+ "radial": 17172,
+ "Bosnian": 17173,
+ "Lexi": 17174,
+ "##14": 17175,
+ "##ü": 17176,
+ "sacks": 17177,
+ "unpleasant": 17178,
+ "Empty": 17179,
+ "handles": 17180,
+ "##taking": 17181,
+ "Bon": 17182,
+ "switches": 17183,
+ "intently": 17184,
+ "tuition": 17185,
+ "antique": 17186,
+ "##jk": 17187,
+ "fraternity": 17188,
+ "notebook": 17189,
+ "Desmond": 17190,
+ "##sei": 17191,
+ "prostitution": 17192,
+ "##how": 17193,
+ "deed": 17194,
+ "##OP": 17195,
+ "501": 17196,
+ "Somewhere": 17197,
+ "Rocks": 17198,
+ "##mons": 17199,
+ "campaigned": 17200,
+ "frigate": 17201,
+ "gases": 17202,
+ "suppress": 17203,
+ "##hang": 17204,
+ "Merlin": 17205,
+ "Northumberland": 17206,
+ "dominate": 17207,
+ "expeditions": 17208,
+ "thunder": 17209,
+ "##ups": 17210,
+ "##rical": 17211,
+ "Cap": 17212,
+ "thorough": 17213,
+ "Ariel": 17214,
+ "##kind": 17215,
+ "renewable": 17216,
+ "constructing": 17217,
+ "pacing": 17218,
+ "terrorists": 17219,
+ "Bowen": 17220,
+ "documentaries": 17221,
+ "westward": 17222,
+ "##lass": 17223,
+ "##nage": 17224,
+ "Merchant": 17225,
+ "##ued": 17226,
+ "Beaumont": 17227,
+ "Din": 17228,
+ "##hian": 17229,
+ "Danube": 17230,
+ "peasant": 17231,
+ "Garrison": 17232,
+ "encourages": 17233,
+ "gratitude": 17234,
+ "reminding": 17235,
+ "stormed": 17236,
+ "##ouse": 17237,
+ "pronunciation": 17238,
+ "##ailed": 17239,
+ "Weekend": 17240,
+ "suggestions": 17241,
+ "##ffing": 17242,
+ "##DI": 17243,
+ "Active": 17244,
+ "Colombo": 17245,
+ "##logists": 17246,
+ "Merrill": 17247,
+ "##cens": 17248,
+ "Archaeological": 17249,
+ "Medina": 17250,
+ "captained": 17251,
+ "##yk": 17252,
+ "duel": 17253,
+ "cracking": 17254,
+ "Wilkinson": 17255,
+ "Guam": 17256,
+ "pickup": 17257,
+ "renovations": 17258,
+ "##ël": 17259,
+ "##izer": 17260,
+ "delighted": 17261,
+ "##iri": 17262,
+ "Weaver": 17263,
+ "##ctional": 17264,
+ "tens": 17265,
+ "##hab": 17266,
+ "Clint": 17267,
+ "##usion": 17268,
+ "##each": 17269,
+ "petals": 17270,
+ "Farrell": 17271,
+ "##sable": 17272,
+ "caste": 17273,
+ "##will": 17274,
+ "Ezra": 17275,
+ "##qi": 17276,
+ "##standing": 17277,
+ "thrilled": 17278,
+ "ambush": 17279,
+ "exhaled": 17280,
+ "##SU": 17281,
+ "Resource": 17282,
+ "blur": 17283,
+ "forearm": 17284,
+ "specifications": 17285,
+ "contingent": 17286,
+ "cafe": 17287,
+ "##iology": 17288,
+ "Antony": 17289,
+ "fundraising": 17290,
+ "grape": 17291,
+ "##rgy": 17292,
+ "turnout": 17293,
+ "##udi": 17294,
+ "Clifton": 17295,
+ "laboratories": 17296,
+ "Irvine": 17297,
+ "##opus": 17298,
+ "##lid": 17299,
+ "Monthly": 17300,
+ "Bihar": 17301,
+ "statutory": 17302,
+ "Roses": 17303,
+ "Emil": 17304,
+ "##rig": 17305,
+ "lumber": 17306,
+ "optimal": 17307,
+ "##DR": 17308,
+ "pumps": 17309,
+ "plaster": 17310,
+ "Mozambique": 17311,
+ "##aco": 17312,
+ "nightclub": 17313,
+ "propelled": 17314,
+ "##hun": 17315,
+ "ked": 17316,
+ "surplus": 17317,
+ "wax": 17318,
+ "##urai": 17319,
+ "pioneered": 17320,
+ "Sunny": 17321,
+ "imprint": 17322,
+ "Forget": 17323,
+ "Eliot": 17324,
+ "approximate": 17325,
+ "patronage": 17326,
+ "##bek": 17327,
+ "##ely": 17328,
+ "##mbe": 17329,
+ "Partnership": 17330,
+ "curl": 17331,
+ "snapping": 17332,
+ "29th": 17333,
+ "Patriarch": 17334,
+ "##jord": 17335,
+ "seldom": 17336,
+ "##ature": 17337,
+ "astronomy": 17338,
+ "Bremen": 17339,
+ "XIV": 17340,
+ "airborne": 17341,
+ "205": 17342,
+ "1778": 17343,
+ "recognizing": 17344,
+ "stranded": 17345,
+ "arrogant": 17346,
+ "bombardment": 17347,
+ "destined": 17348,
+ "ensured": 17349,
+ "146": 17350,
+ "robust": 17351,
+ "Davenport": 17352,
+ "Interactive": 17353,
+ "Offensive": 17354,
+ "Fi": 17355,
+ "prevents": 17356,
+ "probe": 17357,
+ "propeller": 17358,
+ "sorrow": 17359,
+ "Blade": 17360,
+ "mounting": 17361,
+ "automotive": 17362,
+ "##dged": 17363,
+ "wallet": 17364,
+ "201": 17365,
+ "lashes": 17366,
+ "Forrest": 17367,
+ "##ift": 17368,
+ "Cell": 17369,
+ "Younger": 17370,
+ "shouts": 17371,
+ "##cki": 17372,
+ "folds": 17373,
+ "##chet": 17374,
+ "Epic": 17375,
+ "yields": 17376,
+ "homosexual": 17377,
+ "tunes": 17378,
+ "##minate": 17379,
+ "##text": 17380,
+ "Manny": 17381,
+ "chemist": 17382,
+ "hindwings": 17383,
+ "##urn": 17384,
+ "pilgrimage": 17385,
+ "##sfield": 17386,
+ "##riff": 17387,
+ "MLS": 17388,
+ "##rive": 17389,
+ "Huntington": 17390,
+ "translates": 17391,
+ "Path": 17392,
+ "slim": 17393,
+ "##ndra": 17394,
+ "##oz": 17395,
+ "climax": 17396,
+ "commuter": 17397,
+ "desperation": 17398,
+ "##reet": 17399,
+ "denying": 17400,
+ "##rious": 17401,
+ "daring": 17402,
+ "seminary": 17403,
+ "polo": 17404,
+ "##clamation": 17405,
+ "Teatro": 17406,
+ "Torah": 17407,
+ "Cats": 17408,
+ "identities": 17409,
+ "Poles": 17410,
+ "photographed": 17411,
+ "fiery": 17412,
+ "popularly": 17413,
+ "##cross": 17414,
+ "winters": 17415,
+ "Hesse": 17416,
+ "##vio": 17417,
+ "Nurse": 17418,
+ "Senegal": 17419,
+ "Salon": 17420,
+ "prescribed": 17421,
+ "justify": 17422,
+ "##gues": 17423,
+ "##и": 17424,
+ "##orted": 17425,
+ "HQ": 17426,
+ "##hiro": 17427,
+ "evaluated": 17428,
+ "momentarily": 17429,
+ "##unts": 17430,
+ "Debbie": 17431,
+ "##licity": 17432,
+ "##TP": 17433,
+ "Mighty": 17434,
+ "Rabbit": 17435,
+ "##chal": 17436,
+ "Events": 17437,
+ "Savoy": 17438,
+ "##ht": 17439,
+ "Brandenburg": 17440,
+ "Bordeaux": 17441,
+ "##laus": 17442,
+ "Release": 17443,
+ "##IE": 17444,
+ "##kowski": 17445,
+ "1900s": 17446,
+ "SK": 17447,
+ "Strauss": 17448,
+ "##aly": 17449,
+ "Sonia": 17450,
+ "Updated": 17451,
+ "synagogue": 17452,
+ "McKay": 17453,
+ "flattened": 17454,
+ "370": 17455,
+ "clutch": 17456,
+ "contests": 17457,
+ "toast": 17458,
+ "evaluate": 17459,
+ "pope": 17460,
+ "heirs": 17461,
+ "jam": 17462,
+ "tutor": 17463,
+ "reverted": 17464,
+ "##ading": 17465,
+ "nonsense": 17466,
+ "hesitate": 17467,
+ "Lars": 17468,
+ "Ceylon": 17469,
+ "Laurie": 17470,
+ "##guchi": 17471,
+ "accordingly": 17472,
+ "customary": 17473,
+ "148": 17474,
+ "Ethics": 17475,
+ "Multiple": 17476,
+ "instincts": 17477,
+ "IGN": 17478,
+ "##ä": 17479,
+ "bullshit": 17480,
+ "##hit": 17481,
+ "##par": 17482,
+ "desirable": 17483,
+ "##ducing": 17484,
+ "##yam": 17485,
+ "alias": 17486,
+ "ashore": 17487,
+ "licenses": 17488,
+ "##lification": 17489,
+ "misery": 17490,
+ "147": 17491,
+ "Cola": 17492,
+ "assassinated": 17493,
+ "fiercely": 17494,
+ "##aft": 17495,
+ "las": 17496,
+ "goat": 17497,
+ "substrate": 17498,
+ "lords": 17499,
+ "Cass": 17500,
+ "Bridges": 17501,
+ "ICC": 17502,
+ "lasts": 17503,
+ "sights": 17504,
+ "reproductive": 17505,
+ "##asi": 17506,
+ "Ivory": 17507,
+ "Clean": 17508,
+ "fixing": 17509,
+ "##lace": 17510,
+ "seeming": 17511,
+ "aide": 17512,
+ "1850s": 17513,
+ "harassment": 17514,
+ "##FF": 17515,
+ "##LE": 17516,
+ "reasonably": 17517,
+ "##coat": 17518,
+ "##cano": 17519,
+ "NYC": 17520,
+ "1784": 17521,
+ "Fifty": 17522,
+ "immunity": 17523,
+ "Canadians": 17524,
+ "Cheng": 17525,
+ "comforting": 17526,
+ "meanwhile": 17527,
+ "##tera": 17528,
+ "##blin": 17529,
+ "breeds": 17530,
+ "glowed": 17531,
+ "##vour": 17532,
+ "Aden": 17533,
+ "##verted": 17534,
+ "##aded": 17535,
+ "##oral": 17536,
+ "neat": 17537,
+ "enforced": 17538,
+ "poisoning": 17539,
+ "##ews": 17540,
+ "##hone": 17541,
+ "enforce": 17542,
+ "predecessors": 17543,
+ "survivor": 17544,
+ "Month": 17545,
+ "unfamiliar": 17546,
+ "pierced": 17547,
+ "waived": 17548,
+ "dump": 17549,
+ "responds": 17550,
+ "Mai": 17551,
+ "Declan": 17552,
+ "angular": 17553,
+ "Doesn": 17554,
+ "interpretations": 17555,
+ "##yar": 17556,
+ "invest": 17557,
+ "Dhaka": 17558,
+ "policeman": 17559,
+ "Congregation": 17560,
+ "Eighth": 17561,
+ "painfully": 17562,
+ "##este": 17563,
+ "##vior": 17564,
+ "Württemberg": 17565,
+ "##cles": 17566,
+ "blockade": 17567,
+ "encouragement": 17568,
+ "##fie": 17569,
+ "Caucasus": 17570,
+ "Malone": 17571,
+ "Universidad": 17572,
+ "utilize": 17573,
+ "Nissan": 17574,
+ "inherent": 17575,
+ "151": 17576,
+ "agreeing": 17577,
+ "syllable": 17578,
+ "determines": 17579,
+ "Protocol": 17580,
+ "conclude": 17581,
+ "##gara": 17582,
+ "40th": 17583,
+ "Xu": 17584,
+ "Taiwanese": 17585,
+ "##ather": 17586,
+ "boiler": 17587,
+ "printer": 17588,
+ "Lacey": 17589,
+ "titular": 17590,
+ "Klaus": 17591,
+ "Fallon": 17592,
+ "Wembley": 17593,
+ "fox": 17594,
+ "Chandra": 17595,
+ "Governorate": 17596,
+ "obsessed": 17597,
+ "##Ps": 17598,
+ "micro": 17599,
+ "##25": 17600,
+ "Cooke": 17601,
+ "gymnasium": 17602,
+ "weaving": 17603,
+ "Shall": 17604,
+ "Hussein": 17605,
+ "glaring": 17606,
+ "softball": 17607,
+ "Reader": 17608,
+ "Dominion": 17609,
+ "Trouble": 17610,
+ "varsity": 17611,
+ "Cooperation": 17612,
+ "Chaos": 17613,
+ "Kang": 17614,
+ "Kramer": 17615,
+ "Eisenhower": 17616,
+ "proves": 17617,
+ "Connie": 17618,
+ "consortium": 17619,
+ "governors": 17620,
+ "Bethany": 17621,
+ "opener": 17622,
+ "Normally": 17623,
+ "Willy": 17624,
+ "linebacker": 17625,
+ "Regent": 17626,
+ "Used": 17627,
+ "AllMusic": 17628,
+ "Twilight": 17629,
+ "##shaw": 17630,
+ "Companion": 17631,
+ "Tribunal": 17632,
+ "simpler": 17633,
+ "##gam": 17634,
+ "Experimental": 17635,
+ "Slovenian": 17636,
+ "cellar": 17637,
+ "deadline": 17638,
+ "trout": 17639,
+ "Hubbard": 17640,
+ "ads": 17641,
+ "idol": 17642,
+ "##hetto": 17643,
+ "Granada": 17644,
+ "clues": 17645,
+ "salmon": 17646,
+ "1700": 17647,
+ "Omega": 17648,
+ "Caldwell": 17649,
+ "softened": 17650,
+ "Bills": 17651,
+ "Honolulu": 17652,
+ "##gn": 17653,
+ "Terrace": 17654,
+ "suitcase": 17655,
+ "##IL": 17656,
+ "frantic": 17657,
+ "##oons": 17658,
+ "Abbot": 17659,
+ "Sitting": 17660,
+ "Fortress": 17661,
+ "Riders": 17662,
+ "sickness": 17663,
+ "enzymes": 17664,
+ "trustee": 17665,
+ "Bern": 17666,
+ "forged": 17667,
+ "##13": 17668,
+ "##ruff": 17669,
+ "##rl": 17670,
+ "##versity": 17671,
+ "inspector": 17672,
+ "champagne": 17673,
+ "##held": 17674,
+ "##FI": 17675,
+ "hereditary": 17676,
+ "Taliban": 17677,
+ "handball": 17678,
+ "##wine": 17679,
+ "Sioux": 17680,
+ "##dicated": 17681,
+ "honoured": 17682,
+ "139": 17683,
+ "##tude": 17684,
+ "Skye": 17685,
+ "meanings": 17686,
+ "##rkin": 17687,
+ "cardiac": 17688,
+ "analyzed": 17689,
+ "vegetable": 17690,
+ "##FS": 17691,
+ "Royals": 17692,
+ "dial": 17693,
+ "freelance": 17694,
+ "##fest": 17695,
+ "partisan": 17696,
+ "petroleum": 17697,
+ "ridden": 17698,
+ "Lincolnshire": 17699,
+ "panting": 17700,
+ "##comb": 17701,
+ "presidents": 17702,
+ "Haley": 17703,
+ "##chs": 17704,
+ "contributes": 17705,
+ "Jew": 17706,
+ "discoveries": 17707,
+ "panicked": 17708,
+ "Woody": 17709,
+ "eyelids": 17710,
+ "Fate": 17711,
+ "Tulsa": 17712,
+ "mg": 17713,
+ "whiskey": 17714,
+ "zombies": 17715,
+ "Wii": 17716,
+ "##udge": 17717,
+ "investigators": 17718,
+ "##bull": 17719,
+ "centred": 17720,
+ "##screen": 17721,
+ "Bone": 17722,
+ "Lana": 17723,
+ "##oise": 17724,
+ "forts": 17725,
+ "##ske": 17726,
+ "Conan": 17727,
+ "Lyons": 17728,
+ "##writing": 17729,
+ "SH": 17730,
+ "##ride": 17731,
+ "rhythmic": 17732,
+ "154": 17733,
+ "##llah": 17734,
+ "pioneers": 17735,
+ "##bright": 17736,
+ "captivity": 17737,
+ "Sanchez": 17738,
+ "Oman": 17739,
+ "##mith": 17740,
+ "Flint": 17741,
+ "Platform": 17742,
+ "##ioned": 17743,
+ "emission": 17744,
+ "packet": 17745,
+ "Persia": 17746,
+ "##formed": 17747,
+ "takeover": 17748,
+ "tempted": 17749,
+ "Vance": 17750,
+ "Few": 17751,
+ "Toni": 17752,
+ "receptions": 17753,
+ "##ن": 17754,
+ "exchanges": 17755,
+ "Camille": 17756,
+ "whale": 17757,
+ "Chronicles": 17758,
+ "##rent": 17759,
+ "##ushing": 17760,
+ "##rift": 17761,
+ "Alto": 17762,
+ "Genus": 17763,
+ "##asing": 17764,
+ "onward": 17765,
+ "foremost": 17766,
+ "longing": 17767,
+ "Rockefeller": 17768,
+ "containers": 17769,
+ "##cribe": 17770,
+ "intercepted": 17771,
+ "##olt": 17772,
+ "pleading": 17773,
+ "Bye": 17774,
+ "bee": 17775,
+ "##umbling": 17776,
+ "153": 17777,
+ "undertake": 17778,
+ "Izzy": 17779,
+ "cheaper": 17780,
+ "Ultra": 17781,
+ "validity": 17782,
+ "##pse": 17783,
+ "Sa": 17784,
+ "hovering": 17785,
+ "##pert": 17786,
+ "vintage": 17787,
+ "engraved": 17788,
+ "##rise": 17789,
+ "farmland": 17790,
+ "##ever": 17791,
+ "##ifier": 17792,
+ "Atlantis": 17793,
+ "propose": 17794,
+ "Catalonia": 17795,
+ "plunged": 17796,
+ "##edly": 17797,
+ "demonstrates": 17798,
+ "gig": 17799,
+ "##cover": 17800,
+ "156": 17801,
+ "Osborne": 17802,
+ "cowboy": 17803,
+ "herd": 17804,
+ "investigator": 17805,
+ "loops": 17806,
+ "Burning": 17807,
+ "rests": 17808,
+ "Instrumental": 17809,
+ "embarrassing": 17810,
+ "focal": 17811,
+ "install": 17812,
+ "readings": 17813,
+ "swirling": 17814,
+ "Chatham": 17815,
+ "parameter": 17816,
+ "##zin": 17817,
+ "##holders": 17818,
+ "Mandarin": 17819,
+ "Moody": 17820,
+ "converting": 17821,
+ "Escape": 17822,
+ "warnings": 17823,
+ "##chester": 17824,
+ "incarnation": 17825,
+ "##ophone": 17826,
+ "adopting": 17827,
+ "##lins": 17828,
+ "Cromwell": 17829,
+ "##laws": 17830,
+ "Axis": 17831,
+ "Verde": 17832,
+ "Kappa": 17833,
+ "Schwartz": 17834,
+ "Serbs": 17835,
+ "caliber": 17836,
+ "Wanna": 17837,
+ "Chung": 17838,
+ "##ality": 17839,
+ "nursery": 17840,
+ "principally": 17841,
+ "Bulletin": 17842,
+ "likelihood": 17843,
+ "logging": 17844,
+ "##erty": 17845,
+ "Boyle": 17846,
+ "supportive": 17847,
+ "twitched": 17848,
+ "##usive": 17849,
+ "builds": 17850,
+ "Marseille": 17851,
+ "omitted": 17852,
+ "motif": 17853,
+ "Lands": 17854,
+ "##lusion": 17855,
+ "##ssed": 17856,
+ "Barrow": 17857,
+ "Airfield": 17858,
+ "Harmony": 17859,
+ "WWF": 17860,
+ "endured": 17861,
+ "merging": 17862,
+ "convey": 17863,
+ "branding": 17864,
+ "examinations": 17865,
+ "167": 17866,
+ "Italians": 17867,
+ "##dh": 17868,
+ "dude": 17869,
+ "1781": 17870,
+ "##teau": 17871,
+ "crawling": 17872,
+ "thoughtful": 17873,
+ "clasped": 17874,
+ "concluding": 17875,
+ "brewery": 17876,
+ "Moldova": 17877,
+ "Wan": 17878,
+ "Towers": 17879,
+ "Heidelberg": 17880,
+ "202": 17881,
+ "##ict": 17882,
+ "Lagos": 17883,
+ "imposing": 17884,
+ "##eval": 17885,
+ "##serve": 17886,
+ "Bacon": 17887,
+ "frowning": 17888,
+ "thirteenth": 17889,
+ "conception": 17890,
+ "calculations": 17891,
+ "##ович": 17892,
+ "##mile": 17893,
+ "##ivated": 17894,
+ "mutation": 17895,
+ "strap": 17896,
+ "##lund": 17897,
+ "demographic": 17898,
+ "nude": 17899,
+ "perfection": 17900,
+ "stocks": 17901,
+ "##renched": 17902,
+ "##dit": 17903,
+ "Alejandro": 17904,
+ "bites": 17905,
+ "fragment": 17906,
+ "##hack": 17907,
+ "##rchy": 17908,
+ "GB": 17909,
+ "Surgery": 17910,
+ "Berger": 17911,
+ "punish": 17912,
+ "boiling": 17913,
+ "consume": 17914,
+ "Elle": 17915,
+ "Sid": 17916,
+ "Dome": 17917,
+ "relies": 17918,
+ "Crescent": 17919,
+ "treasurer": 17920,
+ "Bloody": 17921,
+ "1758": 17922,
+ "upheld": 17923,
+ "Guess": 17924,
+ "Restaurant": 17925,
+ "signatures": 17926,
+ "font": 17927,
+ "millennium": 17928,
+ "mural": 17929,
+ "stakes": 17930,
+ "Abel": 17931,
+ "hailed": 17932,
+ "insists": 17933,
+ "Alumni": 17934,
+ "Breton": 17935,
+ "##jun": 17936,
+ "digits": 17937,
+ "##FM": 17938,
+ "##thal": 17939,
+ "Talking": 17940,
+ "motive": 17941,
+ "reigning": 17942,
+ "babe": 17943,
+ "masks": 17944,
+ "##ø": 17945,
+ "Shaun": 17946,
+ "potato": 17947,
+ "sour": 17948,
+ "whitish": 17949,
+ "Somali": 17950,
+ "##derman": 17951,
+ "##rab": 17952,
+ "##wy": 17953,
+ "chancel": 17954,
+ "telecommunications": 17955,
+ "Noise": 17956,
+ "messenger": 17957,
+ "tidal": 17958,
+ "grinding": 17959,
+ "##ogenic": 17960,
+ "Rebel": 17961,
+ "constituent": 17962,
+ "peripheral": 17963,
+ "recruitment": 17964,
+ "##ograph": 17965,
+ "##tler": 17966,
+ "pumped": 17967,
+ "Ravi": 17968,
+ "poked": 17969,
+ "##gley": 17970,
+ "Olive": 17971,
+ "diabetes": 17972,
+ "discs": 17973,
+ "liking": 17974,
+ "sting": 17975,
+ "fits": 17976,
+ "stir": 17977,
+ "Mari": 17978,
+ "Sega": 17979,
+ "creativity": 17980,
+ "weights": 17981,
+ "Macau": 17982,
+ "mandated": 17983,
+ "Bohemia": 17984,
+ "disastrous": 17985,
+ "Katrina": 17986,
+ "Baku": 17987,
+ "Rajasthan": 17988,
+ "waiter": 17989,
+ "##psis": 17990,
+ "Siberia": 17991,
+ "verbs": 17992,
+ "##truction": 17993,
+ "patented": 17994,
+ "1782": 17995,
+ "##ndon": 17996,
+ "Relegated": 17997,
+ "Hunters": 17998,
+ "Greenwood": 17999,
+ "Shock": 18000,
+ "accusing": 18001,
+ "skipped": 18002,
+ "Sessions": 18003,
+ "markers": 18004,
+ "subset": 18005,
+ "monumental": 18006,
+ "Viola": 18007,
+ "comparative": 18008,
+ "Alright": 18009,
+ "Barbados": 18010,
+ "setup": 18011,
+ "Session": 18012,
+ "standardized": 18013,
+ "##ík": 18014,
+ "##sket": 18015,
+ "appoint": 18016,
+ "AFB": 18017,
+ "Nationalist": 18018,
+ "##WS": 18019,
+ "Troop": 18020,
+ "leaped": 18021,
+ "Treasure": 18022,
+ "goodness": 18023,
+ "weary": 18024,
+ "originates": 18025,
+ "100th": 18026,
+ "compassion": 18027,
+ "expresses": 18028,
+ "recommend": 18029,
+ "168": 18030,
+ "composing": 18031,
+ "seventeenth": 18032,
+ "Tex": 18033,
+ "Atlético": 18034,
+ "bald": 18035,
+ "Finding": 18036,
+ "Presidency": 18037,
+ "Sharks": 18038,
+ "favoured": 18039,
+ "inactive": 18040,
+ "##lter": 18041,
+ "suffix": 18042,
+ "princes": 18043,
+ "brighter": 18044,
+ "##ctus": 18045,
+ "classics": 18046,
+ "defendants": 18047,
+ "culminated": 18048,
+ "terribly": 18049,
+ "Strategy": 18050,
+ "evenings": 18051,
+ "##ção": 18052,
+ "##iver": 18053,
+ "##urance": 18054,
+ "absorb": 18055,
+ "##rner": 18056,
+ "Territories": 18057,
+ "RBI": 18058,
+ "soothing": 18059,
+ "Martín": 18060,
+ "concurrently": 18061,
+ "##tr": 18062,
+ "Nicholson": 18063,
+ "fibers": 18064,
+ "swam": 18065,
+ "##oney": 18066,
+ "Allie": 18067,
+ "Algerian": 18068,
+ "Dartmouth": 18069,
+ "Mafia": 18070,
+ "##bos": 18071,
+ "##tts": 18072,
+ "Councillor": 18073,
+ "vocabulary": 18074,
+ "##bla": 18075,
+ "##lé": 18076,
+ "intending": 18077,
+ "##dler": 18078,
+ "Guerrero": 18079,
+ "sunshine": 18080,
+ "pedal": 18081,
+ "##TO": 18082,
+ "administrators": 18083,
+ "periodic": 18084,
+ "scholarships": 18085,
+ "Loop": 18086,
+ "Madeline": 18087,
+ "exaggerated": 18088,
+ "##ressed": 18089,
+ "Regan": 18090,
+ "##cellular": 18091,
+ "Explorer": 18092,
+ "##oids": 18093,
+ "Alexandre": 18094,
+ "vows": 18095,
+ "Reporter": 18096,
+ "Unable": 18097,
+ "Average": 18098,
+ "absorption": 18099,
+ "##bedience": 18100,
+ "Fortunately": 18101,
+ "Auxiliary": 18102,
+ "Grandpa": 18103,
+ "##HP": 18104,
+ "##ovo": 18105,
+ "potent": 18106,
+ "temporal": 18107,
+ "adrenaline": 18108,
+ "##udo": 18109,
+ "confusing": 18110,
+ "guiding": 18111,
+ "Dry": 18112,
+ "qualifications": 18113,
+ "joking": 18114,
+ "wherein": 18115,
+ "heavyweight": 18116,
+ "##ices": 18117,
+ "nightmares": 18118,
+ "pharmaceutical": 18119,
+ "Commanding": 18120,
+ "##aled": 18121,
+ "##ove": 18122,
+ "Gregor": 18123,
+ "##UP": 18124,
+ "censorship": 18125,
+ "degradation": 18126,
+ "glorious": 18127,
+ "Austro": 18128,
+ "##rench": 18129,
+ "380": 18130,
+ "Miriam": 18131,
+ "sped": 18132,
+ "##orous": 18133,
+ "offset": 18134,
+ "##KA": 18135,
+ "fined": 18136,
+ "specialists": 18137,
+ "Pune": 18138,
+ "João": 18139,
+ "##dina": 18140,
+ "propped": 18141,
+ "fungus": 18142,
+ "##ς": 18143,
+ "frantically": 18144,
+ "Gabrielle": 18145,
+ "Hare": 18146,
+ "committing": 18147,
+ "##plied": 18148,
+ "Ask": 18149,
+ "Wilmington": 18150,
+ "stunt": 18151,
+ "numb": 18152,
+ "warmer": 18153,
+ "preacher": 18154,
+ "earnings": 18155,
+ "##lating": 18156,
+ "integer": 18157,
+ "##ija": 18158,
+ "federation": 18159,
+ "homosexuality": 18160,
+ "##cademia": 18161,
+ "epidemic": 18162,
+ "grumbled": 18163,
+ "shoving": 18164,
+ "Milk": 18165,
+ "Satan": 18166,
+ "Tobias": 18167,
+ "innovations": 18168,
+ "##dington": 18169,
+ "geology": 18170,
+ "memoirs": 18171,
+ "##IR": 18172,
+ "spared": 18173,
+ "culminating": 18174,
+ "Daphne": 18175,
+ "Focus": 18176,
+ "severed": 18177,
+ "stricken": 18178,
+ "Paige": 18179,
+ "Mans": 18180,
+ "flats": 18181,
+ "Russo": 18182,
+ "communes": 18183,
+ "litigation": 18184,
+ "strengthening": 18185,
+ "##powered": 18186,
+ "Staffordshire": 18187,
+ "Wiltshire": 18188,
+ "Painting": 18189,
+ "Watkins": 18190,
+ "##د": 18191,
+ "specializes": 18192,
+ "Select": 18193,
+ "##rane": 18194,
+ "##aver": 18195,
+ "Fulton": 18196,
+ "playable": 18197,
+ "##VN": 18198,
+ "openings": 18199,
+ "sampling": 18200,
+ "##coon": 18201,
+ "##21": 18202,
+ "Allah": 18203,
+ "travelers": 18204,
+ "allocation": 18205,
+ "##arily": 18206,
+ "Loch": 18207,
+ "##hm": 18208,
+ "commentators": 18209,
+ "fulfilled": 18210,
+ "##troke": 18211,
+ "Emeritus": 18212,
+ "Vanderbilt": 18213,
+ "Vijay": 18214,
+ "pledged": 18215,
+ "##tative": 18216,
+ "diagram": 18217,
+ "drilling": 18218,
+ "##MD": 18219,
+ "##plain": 18220,
+ "Edison": 18221,
+ "productivity": 18222,
+ "31st": 18223,
+ "##rying": 18224,
+ "##ption": 18225,
+ "##gano": 18226,
+ "##oration": 18227,
+ "##bara": 18228,
+ "posture": 18229,
+ "bothering": 18230,
+ "platoon": 18231,
+ "politely": 18232,
+ "##inating": 18233,
+ "redevelopment": 18234,
+ "Job": 18235,
+ "##vale": 18236,
+ "stark": 18237,
+ "incorrect": 18238,
+ "Mansion": 18239,
+ "renewal": 18240,
+ "threatens": 18241,
+ "Bahamas": 18242,
+ "fridge": 18243,
+ "##tata": 18244,
+ "Uzbekistan": 18245,
+ "##edia": 18246,
+ "Sainte": 18247,
+ "##mio": 18248,
+ "gaps": 18249,
+ "neural": 18250,
+ "##storm": 18251,
+ "overturned": 18252,
+ "Preservation": 18253,
+ "shields": 18254,
+ "##ngo": 18255,
+ "##physics": 18256,
+ "ah": 18257,
+ "gradual": 18258,
+ "killings": 18259,
+ "##anza": 18260,
+ "consultation": 18261,
+ "premiership": 18262,
+ "Felipe": 18263,
+ "coincidence": 18264,
+ "##ène": 18265,
+ "##any": 18266,
+ "Handbook": 18267,
+ "##loaded": 18268,
+ "Edit": 18269,
+ "Guns": 18270,
+ "arguably": 18271,
+ "##ş": 18272,
+ "compressed": 18273,
+ "depict": 18274,
+ "seller": 18275,
+ "##qui": 18276,
+ "Kilkenny": 18277,
+ "##kling": 18278,
+ "Olympia": 18279,
+ "librarian": 18280,
+ "##acles": 18281,
+ "dramas": 18282,
+ "JP": 18283,
+ "Kit": 18284,
+ "Maj": 18285,
+ "##lists": 18286,
+ "proprietary": 18287,
+ "##nged": 18288,
+ "##ettes": 18289,
+ "##tok": 18290,
+ "exceeding": 18291,
+ "Lock": 18292,
+ "induction": 18293,
+ "numerical": 18294,
+ "##vist": 18295,
+ "Straight": 18296,
+ "foyer": 18297,
+ "imaginary": 18298,
+ "##pop": 18299,
+ "violinist": 18300,
+ "Carla": 18301,
+ "bouncing": 18302,
+ "##ashi": 18303,
+ "abolition": 18304,
+ "##uction": 18305,
+ "restoring": 18306,
+ "scenic": 18307,
+ "##č": 18308,
+ "Doom": 18309,
+ "overthrow": 18310,
+ "para": 18311,
+ "##vid": 18312,
+ "##ughty": 18313,
+ "Concord": 18314,
+ "HC": 18315,
+ "cocaine": 18316,
+ "deputies": 18317,
+ "##aul": 18318,
+ "visibility": 18319,
+ "##wart": 18320,
+ "Kapoor": 18321,
+ "Hutchinson": 18322,
+ "##agan": 18323,
+ "flashes": 18324,
+ "kn": 18325,
+ "decreasing": 18326,
+ "##ronology": 18327,
+ "quotes": 18328,
+ "vain": 18329,
+ "satisfying": 18330,
+ "##iam": 18331,
+ "##linger": 18332,
+ "310": 18333,
+ "Hanson": 18334,
+ "fauna": 18335,
+ "##zawa": 18336,
+ "##rrel": 18337,
+ "Trenton": 18338,
+ "##VB": 18339,
+ "Employment": 18340,
+ "vocational": 18341,
+ "Exactly": 18342,
+ "bartender": 18343,
+ "butterflies": 18344,
+ "tow": 18345,
+ "##chers": 18346,
+ "##ocks": 18347,
+ "pigs": 18348,
+ "merchandise": 18349,
+ "##game": 18350,
+ "##pine": 18351,
+ "Shea": 18352,
+ "##gration": 18353,
+ "Connell": 18354,
+ "Josephine": 18355,
+ "monopoly": 18356,
+ "##dled": 18357,
+ "Cobb": 18358,
+ "warships": 18359,
+ "cancellation": 18360,
+ "someday": 18361,
+ "stove": 18362,
+ "##Cs": 18363,
+ "candidacy": 18364,
+ "superhero": 18365,
+ "unrest": 18366,
+ "Toulouse": 18367,
+ "admiration": 18368,
+ "undergone": 18369,
+ "whirled": 18370,
+ "Reconnaissance": 18371,
+ "costly": 18372,
+ "##ships": 18373,
+ "290": 18374,
+ "Cafe": 18375,
+ "amber": 18376,
+ "Tory": 18377,
+ "##mpt": 18378,
+ "definitive": 18379,
+ "##dress": 18380,
+ "proposes": 18381,
+ "redesigned": 18382,
+ "acceleration": 18383,
+ "##asa": 18384,
+ "##raphy": 18385,
+ "Presley": 18386,
+ "exits": 18387,
+ "Languages": 18388,
+ "##cel": 18389,
+ "Mode": 18390,
+ "spokesperson": 18391,
+ "##tius": 18392,
+ "Ban": 18393,
+ "forthcoming": 18394,
+ "grounded": 18395,
+ "ACC": 18396,
+ "compelling": 18397,
+ "logistics": 18398,
+ "retailers": 18399,
+ "abused": 18400,
+ "##gating": 18401,
+ "soda": 18402,
+ "##yland": 18403,
+ "##lution": 18404,
+ "Landmark": 18405,
+ "XVI": 18406,
+ "blush": 18407,
+ "##tem": 18408,
+ "hurling": 18409,
+ "dread": 18410,
+ "Tobago": 18411,
+ "Foley": 18412,
+ "##uad": 18413,
+ "scenarios": 18414,
+ "##mentation": 18415,
+ "##rks": 18416,
+ "Score": 18417,
+ "fatigue": 18418,
+ "hairy": 18419,
+ "correspond": 18420,
+ "##iard": 18421,
+ "defences": 18422,
+ "confiscated": 18423,
+ "##rudence": 18424,
+ "1785": 18425,
+ "Formerly": 18426,
+ "Shot": 18427,
+ "advertised": 18428,
+ "460": 18429,
+ "Text": 18430,
+ "ridges": 18431,
+ "Promise": 18432,
+ "Dev": 18433,
+ "exclusion": 18434,
+ "NHS": 18435,
+ "tuberculosis": 18436,
+ "rockets": 18437,
+ "##offs": 18438,
+ "sparkling": 18439,
+ "256": 18440,
+ "disappears": 18441,
+ "mankind": 18442,
+ "##hore": 18443,
+ "HP": 18444,
+ "##omo": 18445,
+ "taxation": 18446,
+ "Multi": 18447,
+ "DS": 18448,
+ "Virgil": 18449,
+ "##ams": 18450,
+ "Dell": 18451,
+ "stacked": 18452,
+ "guessing": 18453,
+ "Jump": 18454,
+ "Nope": 18455,
+ "cheer": 18456,
+ "hates": 18457,
+ "ballots": 18458,
+ "overlooked": 18459,
+ "analyses": 18460,
+ "Prevention": 18461,
+ "maturity": 18462,
+ "dos": 18463,
+ "##cards": 18464,
+ "##lect": 18465,
+ "Mare": 18466,
+ "##yssa": 18467,
+ "Petty": 18468,
+ "##wning": 18469,
+ "differing": 18470,
+ "iOS": 18471,
+ "##ior": 18472,
+ "Joachim": 18473,
+ "Sentinel": 18474,
+ "##nstein": 18475,
+ "90s": 18476,
+ "Pamela": 18477,
+ "480": 18478,
+ "Asher": 18479,
+ "##lary": 18480,
+ "Vicente": 18481,
+ "landings": 18482,
+ "portray": 18483,
+ "##rda": 18484,
+ "##xley": 18485,
+ "Virtual": 18486,
+ "##uary": 18487,
+ "finances": 18488,
+ "Jain": 18489,
+ "Somebody": 18490,
+ "Tri": 18491,
+ "behave": 18492,
+ "Michele": 18493,
+ "##ider": 18494,
+ "dwellings": 18495,
+ "FAA": 18496,
+ "Gallagher": 18497,
+ "##lide": 18498,
+ "Monkey": 18499,
+ "195": 18500,
+ "aforementioned": 18501,
+ "##rism": 18502,
+ "##bey": 18503,
+ "##kim": 18504,
+ "##puted": 18505,
+ "Mesa": 18506,
+ "hopped": 18507,
+ "unopposed": 18508,
+ "recipients": 18509,
+ "Reality": 18510,
+ "Been": 18511,
+ "gritted": 18512,
+ "149": 18513,
+ "playground": 18514,
+ "pillar": 18515,
+ "##rone": 18516,
+ "Guinness": 18517,
+ "##tad": 18518,
+ "Théâtre": 18519,
+ "depended": 18520,
+ "Tipperary": 18521,
+ "Reuben": 18522,
+ "frightening": 18523,
+ "wooded": 18524,
+ "Target": 18525,
+ "globally": 18526,
+ "##uted": 18527,
+ "Morales": 18528,
+ "Baptiste": 18529,
+ "drunken": 18530,
+ "Institut": 18531,
+ "characterised": 18532,
+ "##chemistry": 18533,
+ "Strip": 18534,
+ "discrete": 18535,
+ "Premiership": 18536,
+ "##zzling": 18537,
+ "gazing": 18538,
+ "Outer": 18539,
+ "##quisition": 18540,
+ "Sikh": 18541,
+ "Booker": 18542,
+ "##yal": 18543,
+ "contemporaries": 18544,
+ "Jericho": 18545,
+ "##chan": 18546,
+ "##physical": 18547,
+ "##witch": 18548,
+ "Militia": 18549,
+ "##rez": 18550,
+ "##zard": 18551,
+ "dangers": 18552,
+ "##utter": 18553,
+ "##₀": 18554,
+ "Programs": 18555,
+ "darling": 18556,
+ "participates": 18557,
+ "railroads": 18558,
+ "##ienne": 18559,
+ "behavioral": 18560,
+ "bureau": 18561,
+ "##rook": 18562,
+ "161": 18563,
+ "Hicks": 18564,
+ "##rises": 18565,
+ "Comes": 18566,
+ "inflicted": 18567,
+ "bees": 18568,
+ "kindness": 18569,
+ "norm": 18570,
+ "##ković": 18571,
+ "generators": 18572,
+ "##pard": 18573,
+ "##omy": 18574,
+ "##ili": 18575,
+ "methodology": 18576,
+ "Alvin": 18577,
+ "façade": 18578,
+ "latitude": 18579,
+ "##plified": 18580,
+ "DE": 18581,
+ "Morse": 18582,
+ "##mered": 18583,
+ "educate": 18584,
+ "intersects": 18585,
+ "##MF": 18586,
+ "##cz": 18587,
+ "##vated": 18588,
+ "AL": 18589,
+ "##graded": 18590,
+ "##fill": 18591,
+ "constitutes": 18592,
+ "artery": 18593,
+ "feudal": 18594,
+ "avant": 18595,
+ "cautious": 18596,
+ "##ogue": 18597,
+ "immigrated": 18598,
+ "##chenko": 18599,
+ "Saul": 18600,
+ "Clinic": 18601,
+ "Fang": 18602,
+ "choke": 18603,
+ "Cornelius": 18604,
+ "flexibility": 18605,
+ "temperate": 18606,
+ "pins": 18607,
+ "##erson": 18608,
+ "oddly": 18609,
+ "inequality": 18610,
+ "157": 18611,
+ "Natasha": 18612,
+ "Sal": 18613,
+ "##uter": 18614,
+ "215": 18615,
+ "aft": 18616,
+ "blinking": 18617,
+ "##ntino": 18618,
+ "northward": 18619,
+ "Exposition": 18620,
+ "cookies": 18621,
+ "Wedding": 18622,
+ "impulse": 18623,
+ "Overseas": 18624,
+ "terrifying": 18625,
+ "##ough": 18626,
+ "Mortimer": 18627,
+ "##see": 18628,
+ "440": 18629,
+ "https": 18630,
+ "og": 18631,
+ "imagining": 18632,
+ "##cars": 18633,
+ "Nicola": 18634,
+ "exceptionally": 18635,
+ "threads": 18636,
+ "##cup": 18637,
+ "Oswald": 18638,
+ "Provisional": 18639,
+ "dismantled": 18640,
+ "deserves": 18641,
+ "1786": 18642,
+ "Fairy": 18643,
+ "discourse": 18644,
+ "Counsel": 18645,
+ "departing": 18646,
+ "Arc": 18647,
+ "guarding": 18648,
+ "##orse": 18649,
+ "420": 18650,
+ "alterations": 18651,
+ "vibrant": 18652,
+ "Em": 18653,
+ "squinted": 18654,
+ "terrace": 18655,
+ "rowing": 18656,
+ "Led": 18657,
+ "accessories": 18658,
+ "SF": 18659,
+ "Sgt": 18660,
+ "cheating": 18661,
+ "Atomic": 18662,
+ "##raj": 18663,
+ "Blackpool": 18664,
+ "##iary": 18665,
+ "boarded": 18666,
+ "substituted": 18667,
+ "bestowed": 18668,
+ "lime": 18669,
+ "kernel": 18670,
+ "##jah": 18671,
+ "Belmont": 18672,
+ "shaken": 18673,
+ "sticky": 18674,
+ "retrospective": 18675,
+ "Louie": 18676,
+ "migrants": 18677,
+ "weigh": 18678,
+ "sunglasses": 18679,
+ "thumbs": 18680,
+ "##hoff": 18681,
+ "excavation": 18682,
+ "##nks": 18683,
+ "Extra": 18684,
+ "Polo": 18685,
+ "motives": 18686,
+ "Drum": 18687,
+ "infrared": 18688,
+ "tastes": 18689,
+ "berth": 18690,
+ "verge": 18691,
+ "##stand": 18692,
+ "programmed": 18693,
+ "warmed": 18694,
+ "Shankar": 18695,
+ "Titan": 18696,
+ "chromosome": 18697,
+ "cafeteria": 18698,
+ "dividing": 18699,
+ "pepper": 18700,
+ "CPU": 18701,
+ "Stevie": 18702,
+ "satirical": 18703,
+ "Nagar": 18704,
+ "scowled": 18705,
+ "Died": 18706,
+ "backyard": 18707,
+ "##gata": 18708,
+ "##reath": 18709,
+ "##bir": 18710,
+ "Governors": 18711,
+ "portraying": 18712,
+ "##yah": 18713,
+ "Revenge": 18714,
+ "##acing": 18715,
+ "1772": 18716,
+ "margins": 18717,
+ "Bahn": 18718,
+ "OH": 18719,
+ "lowland": 18720,
+ "##razed": 18721,
+ "catcher": 18722,
+ "replay": 18723,
+ "##yoshi": 18724,
+ "Seriously": 18725,
+ "##licit": 18726,
+ "Aristotle": 18727,
+ "##ald": 18728,
+ "Habsburg": 18729,
+ "weekday": 18730,
+ "Secretariat": 18731,
+ "CO": 18732,
+ "##dly": 18733,
+ "##joy": 18734,
+ "##stad": 18735,
+ "litre": 18736,
+ "ultra": 18737,
+ "##cke": 18738,
+ "Mongol": 18739,
+ "Tucson": 18740,
+ "correlation": 18741,
+ "compose": 18742,
+ "traps": 18743,
+ "Groups": 18744,
+ "Hai": 18745,
+ "Salvatore": 18746,
+ "##dea": 18747,
+ "cents": 18748,
+ "##eese": 18749,
+ "concession": 18750,
+ "clash": 18751,
+ "Trip": 18752,
+ "Panzer": 18753,
+ "Moroccan": 18754,
+ "cruisers": 18755,
+ "torque": 18756,
+ "Ba": 18757,
+ "grossed": 18758,
+ "##arate": 18759,
+ "restriction": 18760,
+ "concentrating": 18761,
+ "FDA": 18762,
+ "##Leod": 18763,
+ "##ones": 18764,
+ "Scholars": 18765,
+ "##esi": 18766,
+ "throbbing": 18767,
+ "specialised": 18768,
+ "##heses": 18769,
+ "Chicken": 18770,
+ "##fia": 18771,
+ "##ificant": 18772,
+ "Erich": 18773,
+ "Residence": 18774,
+ "##trate": 18775,
+ "manipulation": 18776,
+ "namesake": 18777,
+ "##tom": 18778,
+ "Hoover": 18779,
+ "cue": 18780,
+ "Lindsey": 18781,
+ "Lonely": 18782,
+ "275": 18783,
+ "##HT": 18784,
+ "combustion": 18785,
+ "subscribers": 18786,
+ "Punjabi": 18787,
+ "respects": 18788,
+ "Jeremiah": 18789,
+ "penned": 18790,
+ "##gor": 18791,
+ "##rilla": 18792,
+ "suppression": 18793,
+ "##tration": 18794,
+ "Crimson": 18795,
+ "piston": 18796,
+ "Derry": 18797,
+ "crimson": 18798,
+ "lyrical": 18799,
+ "oversee": 18800,
+ "portrays": 18801,
+ "CF": 18802,
+ "Districts": 18803,
+ "Lenin": 18804,
+ "Cora": 18805,
+ "searches": 18806,
+ "clans": 18807,
+ "VHS": 18808,
+ "##hel": 18809,
+ "Jacqueline": 18810,
+ "Redskins": 18811,
+ "Clubs": 18812,
+ "desktop": 18813,
+ "indirectly": 18814,
+ "alternatives": 18815,
+ "marijuana": 18816,
+ "suffrage": 18817,
+ "##smos": 18818,
+ "Irwin": 18819,
+ "##liff": 18820,
+ "Process": 18821,
+ "##hawks": 18822,
+ "Sloane": 18823,
+ "##bson": 18824,
+ "Sonata": 18825,
+ "yielded": 18826,
+ "Flores": 18827,
+ "##ares": 18828,
+ "armament": 18829,
+ "adaptations": 18830,
+ "integrate": 18831,
+ "neighbours": 18832,
+ "shelters": 18833,
+ "##tour": 18834,
+ "Skinner": 18835,
+ "##jet": 18836,
+ "##tations": 18837,
+ "1774": 18838,
+ "Peterborough": 18839,
+ "##elles": 18840,
+ "ripping": 18841,
+ "Liang": 18842,
+ "Dickinson": 18843,
+ "charities": 18844,
+ "Rwanda": 18845,
+ "monasteries": 18846,
+ "crossover": 18847,
+ "racist": 18848,
+ "barked": 18849,
+ "guerrilla": 18850,
+ "##ivate": 18851,
+ "Grayson": 18852,
+ "##iques": 18853,
+ "##vious": 18854,
+ "##got": 18855,
+ "Rolls": 18856,
+ "denominations": 18857,
+ "atom": 18858,
+ "affinity": 18859,
+ "##delity": 18860,
+ "Wish": 18861,
+ "##inted": 18862,
+ "##inae": 18863,
+ "interrogation": 18864,
+ "##cey": 18865,
+ "##erina": 18866,
+ "##lifting": 18867,
+ "192": 18868,
+ "Sands": 18869,
+ "1779": 18870,
+ "mast": 18871,
+ "Likewise": 18872,
+ "##hyl": 18873,
+ "##oft": 18874,
+ "contempt": 18875,
+ "##por": 18876,
+ "assaulted": 18877,
+ "fills": 18878,
+ "establishments": 18879,
+ "Mal": 18880,
+ "consulted": 18881,
+ "##omi": 18882,
+ "##sight": 18883,
+ "greet": 18884,
+ "##roma": 18885,
+ "##egan": 18886,
+ "Pulitzer": 18887,
+ "##rried": 18888,
+ "##dius": 18889,
+ "##ractical": 18890,
+ "##voked": 18891,
+ "Hasan": 18892,
+ "CB": 18893,
+ "##zzy": 18894,
+ "Romanesque": 18895,
+ "Panic": 18896,
+ "wheeled": 18897,
+ "recorder": 18898,
+ "##tters": 18899,
+ "##warm": 18900,
+ "##gly": 18901,
+ "botanist": 18902,
+ "Balkan": 18903,
+ "Lockheed": 18904,
+ "Polly": 18905,
+ "farewell": 18906,
+ "suffers": 18907,
+ "purchases": 18908,
+ "Eaton": 18909,
+ "##80": 18910,
+ "Quick": 18911,
+ "commenting": 18912,
+ "Saga": 18913,
+ "beasts": 18914,
+ "hides": 18915,
+ "motifs": 18916,
+ "##icks": 18917,
+ "Alonso": 18918,
+ "Springer": 18919,
+ "Wikipedia": 18920,
+ "circulated": 18921,
+ "encoding": 18922,
+ "jurisdictions": 18923,
+ "snout": 18924,
+ "UAE": 18925,
+ "Integrated": 18926,
+ "unmarried": 18927,
+ "Heinz": 18928,
+ "##lein": 18929,
+ "##figured": 18930,
+ "deleted": 18931,
+ "##tley": 18932,
+ "Zen": 18933,
+ "Cycling": 18934,
+ "Fuel": 18935,
+ "Scandinavian": 18936,
+ "##rants": 18937,
+ "Conner": 18938,
+ "reef": 18939,
+ "Marino": 18940,
+ "curiously": 18941,
+ "lingered": 18942,
+ "Gina": 18943,
+ "manners": 18944,
+ "activism": 18945,
+ "Mines": 18946,
+ "Expo": 18947,
+ "Micah": 18948,
+ "promotions": 18949,
+ "Server": 18950,
+ "booked": 18951,
+ "derivatives": 18952,
+ "eastward": 18953,
+ "detailing": 18954,
+ "reelection": 18955,
+ "##chase": 18956,
+ "182": 18957,
+ "Campeonato": 18958,
+ "Po": 18959,
+ "158": 18960,
+ "Peel": 18961,
+ "winger": 18962,
+ "##itch": 18963,
+ "canyon": 18964,
+ "##pit": 18965,
+ "LDS": 18966,
+ "A1": 18967,
+ "##shin": 18968,
+ "Giorgio": 18969,
+ "pathetic": 18970,
+ "##rga": 18971,
+ "##mist": 18972,
+ "Aren": 18973,
+ "##lag": 18974,
+ "confronts": 18975,
+ "motel": 18976,
+ "textbook": 18977,
+ "shine": 18978,
+ "turbines": 18979,
+ "1770": 18980,
+ "Darcy": 18981,
+ "##cot": 18982,
+ "Southeastern": 18983,
+ "##lessness": 18984,
+ "Banner": 18985,
+ "recognise": 18986,
+ "stray": 18987,
+ "Kitchen": 18988,
+ "paperwork": 18989,
+ "realism": 18990,
+ "Chrysler": 18991,
+ "filmmakers": 18992,
+ "fishermen": 18993,
+ "##hetic": 18994,
+ "variously": 18995,
+ "Vishnu": 18996,
+ "fiddle": 18997,
+ "Eddy": 18998,
+ "Origin": 18999,
+ "##tec": 19000,
+ "##ulin": 19001,
+ "Flames": 19002,
+ "Rs": 19003,
+ "bankrupt": 19004,
+ "Extreme": 19005,
+ "Pomeranian": 19006,
+ "##emption": 19007,
+ "ratified": 19008,
+ "##iu": 19009,
+ "jockey": 19010,
+ "Stratford": 19011,
+ "##ivating": 19012,
+ "##oire": 19013,
+ "Babylon": 19014,
+ "pardon": 19015,
+ "AI": 19016,
+ "affordable": 19017,
+ "deities": 19018,
+ "disturbance": 19019,
+ "Trying": 19020,
+ "##sai": 19021,
+ "Ida": 19022,
+ "Papers": 19023,
+ "advancement": 19024,
+ "70s": 19025,
+ "archbishop": 19026,
+ "Luftwaffe": 19027,
+ "announces": 19028,
+ "tugging": 19029,
+ "##lphin": 19030,
+ "##sistence": 19031,
+ "##eel": 19032,
+ "##ishes": 19033,
+ "ambition": 19034,
+ "aura": 19035,
+ "##fled": 19036,
+ "##lected": 19037,
+ "##vue": 19038,
+ "Prasad": 19039,
+ "boiled": 19040,
+ "clarity": 19041,
+ "Violin": 19042,
+ "investigative": 19043,
+ "routing": 19044,
+ "Yankee": 19045,
+ "##uckle": 19046,
+ "McMahon": 19047,
+ "bugs": 19048,
+ "eruption": 19049,
+ "##rooms": 19050,
+ "Minutes": 19051,
+ "relics": 19052,
+ "##ckle": 19053,
+ "##nse": 19054,
+ "sipped": 19055,
+ "valves": 19056,
+ "weakly": 19057,
+ "##ital": 19058,
+ "Middleton": 19059,
+ "collided": 19060,
+ "##quer": 19061,
+ "bamboo": 19062,
+ "insignia": 19063,
+ "Tyne": 19064,
+ "exercised": 19065,
+ "Ninth": 19066,
+ "echoing": 19067,
+ "polynomial": 19068,
+ "considerations": 19069,
+ "lunged": 19070,
+ "##bius": 19071,
+ "objections": 19072,
+ "complain": 19073,
+ "disguised": 19074,
+ "plaza": 19075,
+ "##VC": 19076,
+ "institutes": 19077,
+ "Judicial": 19078,
+ "ascent": 19079,
+ "imminent": 19080,
+ "Waterford": 19081,
+ "hello": 19082,
+ "Lumpur": 19083,
+ "Niger": 19084,
+ "Goldman": 19085,
+ "vendors": 19086,
+ "Kensington": 19087,
+ "Wren": 19088,
+ "browser": 19089,
+ "##bner": 19090,
+ "##tri": 19091,
+ "##mize": 19092,
+ "##pis": 19093,
+ "##lea": 19094,
+ "Cheyenne": 19095,
+ "Bold": 19096,
+ "Settlement": 19097,
+ "Hollow": 19098,
+ "Paralympic": 19099,
+ "axle": 19100,
+ "##toire": 19101,
+ "##actic": 19102,
+ "impose": 19103,
+ "perched": 19104,
+ "utilizing": 19105,
+ "slips": 19106,
+ "Benz": 19107,
+ "Michaels": 19108,
+ "manipulate": 19109,
+ "Chiang": 19110,
+ "##mian": 19111,
+ "Dolphins": 19112,
+ "prohibition": 19113,
+ "attacker": 19114,
+ "ecology": 19115,
+ "Estadio": 19116,
+ "##SB": 19117,
+ "##uild": 19118,
+ "attracts": 19119,
+ "recalls": 19120,
+ "glacier": 19121,
+ "lad": 19122,
+ "##rima": 19123,
+ "Barlow": 19124,
+ "kHz": 19125,
+ "melodic": 19126,
+ "##aby": 19127,
+ "##iracy": 19128,
+ "assumptions": 19129,
+ "Cornish": 19130,
+ "##aru": 19131,
+ "DOS": 19132,
+ "Maddie": 19133,
+ "##mers": 19134,
+ "lyric": 19135,
+ "Luton": 19136,
+ "nm": 19137,
+ "##tron": 19138,
+ "Reno": 19139,
+ "Fin": 19140,
+ "YOU": 19141,
+ "Broadcast": 19142,
+ "Finch": 19143,
+ "sensory": 19144,
+ "##bent": 19145,
+ "Jeep": 19146,
+ "##uman": 19147,
+ "additionally": 19148,
+ "Buildings": 19149,
+ "businessmen": 19150,
+ "treaties": 19151,
+ "235": 19152,
+ "Stranger": 19153,
+ "gateway": 19154,
+ "Charlton": 19155,
+ "accomplishments": 19156,
+ "Diary": 19157,
+ "apologized": 19158,
+ "zinc": 19159,
+ "histories": 19160,
+ "supplier": 19161,
+ "##tting": 19162,
+ "162": 19163,
+ "asphalt": 19164,
+ "Treatment": 19165,
+ "Abbas": 19166,
+ "##pating": 19167,
+ "##yres": 19168,
+ "Bloom": 19169,
+ "sedan": 19170,
+ "soloist": 19171,
+ "##cum": 19172,
+ "antagonist": 19173,
+ "denounced": 19174,
+ "Fairfax": 19175,
+ "##aving": 19176,
+ "##enko": 19177,
+ "noticeable": 19178,
+ "Budget": 19179,
+ "Buckingham": 19180,
+ "Snyder": 19181,
+ "retreating": 19182,
+ "Jai": 19183,
+ "spoon": 19184,
+ "invading": 19185,
+ "giggle": 19186,
+ "woven": 19187,
+ "gunfire": 19188,
+ "arrests": 19189,
+ "##vered": 19190,
+ "##come": 19191,
+ "respiratory": 19192,
+ "violet": 19193,
+ "##aws": 19194,
+ "Byrd": 19195,
+ "shocking": 19196,
+ "tenant": 19197,
+ "Jamaican": 19198,
+ "Ottomans": 19199,
+ "Seal": 19200,
+ "theirs": 19201,
+ "##isse": 19202,
+ "##48": 19203,
+ "cooperate": 19204,
+ "peering": 19205,
+ "##nius": 19206,
+ "163": 19207,
+ "Composer": 19208,
+ "organist": 19209,
+ "Mongolian": 19210,
+ "Bauer": 19211,
+ "Spy": 19212,
+ "collects": 19213,
+ "prophecy": 19214,
+ "congregations": 19215,
+ "##moor": 19216,
+ "Brick": 19217,
+ "calculation": 19218,
+ "fixtures": 19219,
+ "exempt": 19220,
+ "##dden": 19221,
+ "Ada": 19222,
+ "Thousand": 19223,
+ "##lue": 19224,
+ "tracing": 19225,
+ "##achi": 19226,
+ "bodyguard": 19227,
+ "vicar": 19228,
+ "supplying": 19229,
+ "Łódź": 19230,
+ "interception": 19231,
+ "monitored": 19232,
+ "##heart": 19233,
+ "Paso": 19234,
+ "overlap": 19235,
+ "annoyance": 19236,
+ "##dice": 19237,
+ "yellowish": 19238,
+ "stables": 19239,
+ "elders": 19240,
+ "illegally": 19241,
+ "honesty": 19242,
+ "##oar": 19243,
+ "skinny": 19244,
+ "spinal": 19245,
+ "##puram": 19246,
+ "Bourbon": 19247,
+ "##cor": 19248,
+ "flourished": 19249,
+ "Medium": 19250,
+ "##stics": 19251,
+ "##aba": 19252,
+ "Follow": 19253,
+ "##ckey": 19254,
+ "stationary": 19255,
+ "##scription": 19256,
+ "dresser": 19257,
+ "scrutiny": 19258,
+ "Buckley": 19259,
+ "Clearly": 19260,
+ "##SF": 19261,
+ "Lyrics": 19262,
+ "##heimer": 19263,
+ "drying": 19264,
+ "Oracle": 19265,
+ "internally": 19266,
+ "rains": 19267,
+ "##last": 19268,
+ "Enemy": 19269,
+ "##oes": 19270,
+ "McLean": 19271,
+ "Ole": 19272,
+ "phosphate": 19273,
+ "Rosario": 19274,
+ "Rifles": 19275,
+ "##mium": 19276,
+ "battered": 19277,
+ "Pepper": 19278,
+ "Presidents": 19279,
+ "conquer": 19280,
+ "Château": 19281,
+ "castles": 19282,
+ "##aldo": 19283,
+ "##ulf": 19284,
+ "Depending": 19285,
+ "Lesser": 19286,
+ "Boom": 19287,
+ "trades": 19288,
+ "Peyton": 19289,
+ "164": 19290,
+ "emphasize": 19291,
+ "accustomed": 19292,
+ "SM": 19293,
+ "Ai": 19294,
+ "Classification": 19295,
+ "##mins": 19296,
+ "##35": 19297,
+ "##rons": 19298,
+ "leak": 19299,
+ "piled": 19300,
+ "deeds": 19301,
+ "lush": 19302,
+ "##self": 19303,
+ "beginnings": 19304,
+ "breathless": 19305,
+ "1660": 19306,
+ "McGill": 19307,
+ "##ago": 19308,
+ "##chaft": 19309,
+ "##gies": 19310,
+ "humour": 19311,
+ "Bomb": 19312,
+ "securities": 19313,
+ "Might": 19314,
+ "##zone": 19315,
+ "##eves": 19316,
+ "Matthias": 19317,
+ "Movies": 19318,
+ "Levine": 19319,
+ "vengeance": 19320,
+ "##ads": 19321,
+ "Challenger": 19322,
+ "Misty": 19323,
+ "Traditionally": 19324,
+ "constellation": 19325,
+ "##rass": 19326,
+ "deepest": 19327,
+ "workplace": 19328,
+ "##oof": 19329,
+ "##vina": 19330,
+ "impatient": 19331,
+ "##ML": 19332,
+ "Mughal": 19333,
+ "Alessandro": 19334,
+ "scenery": 19335,
+ "Slater": 19336,
+ "postseason": 19337,
+ "troupe": 19338,
+ "##ń": 19339,
+ "Volunteers": 19340,
+ "Facility": 19341,
+ "militants": 19342,
+ "Reggie": 19343,
+ "sanctions": 19344,
+ "Expeditionary": 19345,
+ "Nam": 19346,
+ "countered": 19347,
+ "interpret": 19348,
+ "Basilica": 19349,
+ "coding": 19350,
+ "expectation": 19351,
+ "Duffy": 19352,
+ "def": 19353,
+ "Tong": 19354,
+ "wakes": 19355,
+ "Bowling": 19356,
+ "Vehicle": 19357,
+ "Adler": 19358,
+ "salad": 19359,
+ "intricate": 19360,
+ "stronghold": 19361,
+ "medley": 19362,
+ "##uries": 19363,
+ "##bur": 19364,
+ "joints": 19365,
+ "##rac": 19366,
+ "##yx": 19367,
+ "##IO": 19368,
+ "Ordnance": 19369,
+ "Welch": 19370,
+ "distributor": 19371,
+ "Ark": 19372,
+ "cavern": 19373,
+ "trench": 19374,
+ "Weiss": 19375,
+ "Mauritius": 19376,
+ "decreases": 19377,
+ "docks": 19378,
+ "eagerly": 19379,
+ "irritation": 19380,
+ "Matilda": 19381,
+ "biographer": 19382,
+ "Visiting": 19383,
+ "##marked": 19384,
+ "##iter": 19385,
+ "##ear": 19386,
+ "##gong": 19387,
+ "Moreno": 19388,
+ "attendant": 19389,
+ "Bury": 19390,
+ "instrumentation": 19391,
+ "theologian": 19392,
+ "clit": 19393,
+ "nuns": 19394,
+ "symphony": 19395,
+ "translate": 19396,
+ "375": 19397,
+ "loser": 19398,
+ "##user": 19399,
+ "##VR": 19400,
+ "##meter": 19401,
+ "##orious": 19402,
+ "harmful": 19403,
+ "##yuki": 19404,
+ "Commissioners": 19405,
+ "Mendoza": 19406,
+ "sniffed": 19407,
+ "Hulk": 19408,
+ "##dded": 19409,
+ "##ulator": 19410,
+ "##nz": 19411,
+ "Donnell": 19412,
+ "##eka": 19413,
+ "deported": 19414,
+ "Met": 19415,
+ "SD": 19416,
+ "Aerospace": 19417,
+ "##cultural": 19418,
+ "##odes": 19419,
+ "Fantastic": 19420,
+ "cavity": 19421,
+ "remark": 19422,
+ "emblem": 19423,
+ "fearing": 19424,
+ "##iance": 19425,
+ "ICAO": 19426,
+ "Liberia": 19427,
+ "stab": 19428,
+ "##yd": 19429,
+ "Pac": 19430,
+ "Gymnasium": 19431,
+ "IS": 19432,
+ "Everton": 19433,
+ "##vanna": 19434,
+ "mantle": 19435,
+ "##ief": 19436,
+ "Ramon": 19437,
+ "##genic": 19438,
+ "Shooting": 19439,
+ "Smoke": 19440,
+ "Random": 19441,
+ "Africans": 19442,
+ "MB": 19443,
+ "tavern": 19444,
+ "bargain": 19445,
+ "voluntarily": 19446,
+ "Ion": 19447,
+ "Peoples": 19448,
+ "Rusty": 19449,
+ "attackers": 19450,
+ "Patton": 19451,
+ "sins": 19452,
+ "##cake": 19453,
+ "Hat": 19454,
+ "moderately": 19455,
+ "##hala": 19456,
+ "##alia": 19457,
+ "requesting": 19458,
+ "mechanic": 19459,
+ "##eae": 19460,
+ "Seine": 19461,
+ "Robbins": 19462,
+ "##ulum": 19463,
+ "susceptible": 19464,
+ "Bravo": 19465,
+ "Slade": 19466,
+ "Strasbourg": 19467,
+ "rubble": 19468,
+ "entrusted": 19469,
+ "Creation": 19470,
+ "##amp": 19471,
+ "smoothed": 19472,
+ "##uintet": 19473,
+ "evenly": 19474,
+ "reviewers": 19475,
+ "skip": 19476,
+ "Sculpture": 19477,
+ "177": 19478,
+ "Rough": 19479,
+ "##rrie": 19480,
+ "Reeves": 19481,
+ "##cede": 19482,
+ "Administrator": 19483,
+ "garde": 19484,
+ "minus": 19485,
+ "carriages": 19486,
+ "grenade": 19487,
+ "Ninja": 19488,
+ "fuscous": 19489,
+ "##kley": 19490,
+ "Punk": 19491,
+ "contributors": 19492,
+ "Aragon": 19493,
+ "Tottenham": 19494,
+ "##cca": 19495,
+ "##sir": 19496,
+ "VA": 19497,
+ "laced": 19498,
+ "dealers": 19499,
+ "##sonic": 19500,
+ "crisp": 19501,
+ "harmonica": 19502,
+ "Artistic": 19503,
+ "Butch": 19504,
+ "Andes": 19505,
+ "Farmers": 19506,
+ "corridors": 19507,
+ "unseen": 19508,
+ "##tium": 19509,
+ "Countries": 19510,
+ "Lone": 19511,
+ "envisioned": 19512,
+ "Katy": 19513,
+ "##lang": 19514,
+ "##cc": 19515,
+ "Quarterly": 19516,
+ "##neck": 19517,
+ "consort": 19518,
+ "##aceae": 19519,
+ "bidding": 19520,
+ "Corey": 19521,
+ "concurrent": 19522,
+ "##acts": 19523,
+ "##gum": 19524,
+ "Highness": 19525,
+ "##lient": 19526,
+ "##rators": 19527,
+ "arising": 19528,
+ "##unta": 19529,
+ "pathways": 19530,
+ "49ers": 19531,
+ "bolted": 19532,
+ "complaining": 19533,
+ "ecosystem": 19534,
+ "libretto": 19535,
+ "Ser": 19536,
+ "narrated": 19537,
+ "212": 19538,
+ "Soft": 19539,
+ "influx": 19540,
+ "##dder": 19541,
+ "incorporation": 19542,
+ "plagued": 19543,
+ "tents": 19544,
+ "##ddled": 19545,
+ "1750": 19546,
+ "Risk": 19547,
+ "citation": 19548,
+ "Tomas": 19549,
+ "hostilities": 19550,
+ "seals": 19551,
+ "Bruins": 19552,
+ "Dominique": 19553,
+ "attic": 19554,
+ "competent": 19555,
+ "##UR": 19556,
+ "##cci": 19557,
+ "hugging": 19558,
+ "Breuning": 19559,
+ "bacterial": 19560,
+ "Shrewsbury": 19561,
+ "vowed": 19562,
+ "eh": 19563,
+ "elongated": 19564,
+ "hangs": 19565,
+ "render": 19566,
+ "centimeters": 19567,
+ "##ficient": 19568,
+ "Mu": 19569,
+ "turtle": 19570,
+ "besieged": 19571,
+ "##gaard": 19572,
+ "grapes": 19573,
+ "bravery": 19574,
+ "collaborations": 19575,
+ "deprived": 19576,
+ "##amine": 19577,
+ "##using": 19578,
+ "##gins": 19579,
+ "arid": 19580,
+ "##uve": 19581,
+ "coats": 19582,
+ "hanged": 19583,
+ "##sting": 19584,
+ "Pa": 19585,
+ "prefix": 19586,
+ "##ranged": 19587,
+ "Exit": 19588,
+ "Chain": 19589,
+ "Flood": 19590,
+ "Materials": 19591,
+ "suspicions": 19592,
+ "##ö": 19593,
+ "hovered": 19594,
+ "Hidden": 19595,
+ "##state": 19596,
+ "Malawi": 19597,
+ "##24": 19598,
+ "Mandy": 19599,
+ "norms": 19600,
+ "fascinating": 19601,
+ "airlines": 19602,
+ "delivers": 19603,
+ "##rust": 19604,
+ "Cretaceous": 19605,
+ "spanned": 19606,
+ "pillows": 19607,
+ "##onomy": 19608,
+ "jar": 19609,
+ "##kka": 19610,
+ "regent": 19611,
+ "fireworks": 19612,
+ "morality": 19613,
+ "discomfort": 19614,
+ "lure": 19615,
+ "uneven": 19616,
+ "##jack": 19617,
+ "Lucian": 19618,
+ "171": 19619,
+ "archaeology": 19620,
+ "##til": 19621,
+ "mornings": 19622,
+ "Billie": 19623,
+ "Marquess": 19624,
+ "impending": 19625,
+ "spilling": 19626,
+ "tombs": 19627,
+ "##volved": 19628,
+ "Celia": 19629,
+ "Coke": 19630,
+ "underside": 19631,
+ "##bation": 19632,
+ "Vaughn": 19633,
+ "Daytona": 19634,
+ "Godfrey": 19635,
+ "Pascal": 19636,
+ "Alien": 19637,
+ "##sign": 19638,
+ "172": 19639,
+ "##lage": 19640,
+ "iPhone": 19641,
+ "Gonna": 19642,
+ "genocide": 19643,
+ "##rber": 19644,
+ "oven": 19645,
+ "endure": 19646,
+ "dashed": 19647,
+ "simultaneous": 19648,
+ "##phism": 19649,
+ "Wally": 19650,
+ "##rō": 19651,
+ "ants": 19652,
+ "predator": 19653,
+ "reissue": 19654,
+ "##aper": 19655,
+ "Speech": 19656,
+ "funk": 19657,
+ "Rudy": 19658,
+ "claw": 19659,
+ "Hindus": 19660,
+ "Numbers": 19661,
+ "Bing": 19662,
+ "lantern": 19663,
+ "##aurus": 19664,
+ "scattering": 19665,
+ "poisoned": 19666,
+ "##active": 19667,
+ "Andrei": 19668,
+ "algebraic": 19669,
+ "baseman": 19670,
+ "##ritz": 19671,
+ "Gregg": 19672,
+ "##cola": 19673,
+ "selections": 19674,
+ "##putation": 19675,
+ "lick": 19676,
+ "Laguna": 19677,
+ "##IX": 19678,
+ "Sumatra": 19679,
+ "Warning": 19680,
+ "turf": 19681,
+ "buyers": 19682,
+ "Burgess": 19683,
+ "Oldham": 19684,
+ "exploit": 19685,
+ "worm": 19686,
+ "initiate": 19687,
+ "strapped": 19688,
+ "tuning": 19689,
+ "filters": 19690,
+ "haze": 19691,
+ "##е": 19692,
+ "##ledge": 19693,
+ "##ydro": 19694,
+ "##culture": 19695,
+ "amendments": 19696,
+ "Promotion": 19697,
+ "##union": 19698,
+ "Clair": 19699,
+ "##uria": 19700,
+ "petty": 19701,
+ "shutting": 19702,
+ "##eveloped": 19703,
+ "Phoebe": 19704,
+ "Zeke": 19705,
+ "conducts": 19706,
+ "grains": 19707,
+ "clashes": 19708,
+ "##latter": 19709,
+ "illegitimate": 19710,
+ "willingly": 19711,
+ "Deer": 19712,
+ "Lakers": 19713,
+ "Reference": 19714,
+ "chaplain": 19715,
+ "commitments": 19716,
+ "interrupt": 19717,
+ "salvation": 19718,
+ "Panther": 19719,
+ "Qualifying": 19720,
+ "Assessment": 19721,
+ "cancel": 19722,
+ "efficiently": 19723,
+ "attorneys": 19724,
+ "Dynamo": 19725,
+ "impress": 19726,
+ "accession": 19727,
+ "clinging": 19728,
+ "randomly": 19729,
+ "reviewing": 19730,
+ "Romero": 19731,
+ "Cathy": 19732,
+ "charting": 19733,
+ "clapped": 19734,
+ "rebranded": 19735,
+ "Azerbaijani": 19736,
+ "coma": 19737,
+ "indicator": 19738,
+ "punches": 19739,
+ "##tons": 19740,
+ "Sami": 19741,
+ "monastic": 19742,
+ "prospects": 19743,
+ "Pastor": 19744,
+ "##rville": 19745,
+ "electrified": 19746,
+ "##CI": 19747,
+ "##utical": 19748,
+ "tumbled": 19749,
+ "Chef": 19750,
+ "muzzle": 19751,
+ "selecting": 19752,
+ "UP": 19753,
+ "Wheel": 19754,
+ "protocols": 19755,
+ "##tat": 19756,
+ "Extended": 19757,
+ "beautifully": 19758,
+ "nests": 19759,
+ "##stal": 19760,
+ "Andersen": 19761,
+ "##anu": 19762,
+ "##³": 19763,
+ "##rini": 19764,
+ "kneeling": 19765,
+ "##reis": 19766,
+ "##xia": 19767,
+ "anatomy": 19768,
+ "dusty": 19769,
+ "Safe": 19770,
+ "turmoil": 19771,
+ "Bianca": 19772,
+ "##elo": 19773,
+ "analyze": 19774,
+ "##ر": 19775,
+ "##eran": 19776,
+ "podcast": 19777,
+ "Slovene": 19778,
+ "Locke": 19779,
+ "Rue": 19780,
+ "##retta": 19781,
+ "##uni": 19782,
+ "Person": 19783,
+ "Prophet": 19784,
+ "crooked": 19785,
+ "disagreed": 19786,
+ "Versailles": 19787,
+ "Sarajevo": 19788,
+ "Utrecht": 19789,
+ "##ogen": 19790,
+ "chewing": 19791,
+ "##ception": 19792,
+ "##iidae": 19793,
+ "Missile": 19794,
+ "attribute": 19795,
+ "majors": 19796,
+ "Arch": 19797,
+ "intellectuals": 19798,
+ "##andra": 19799,
+ "ideological": 19800,
+ "Cory": 19801,
+ "Salzburg": 19802,
+ "##fair": 19803,
+ "Lot": 19804,
+ "electromagnetic": 19805,
+ "Distribution": 19806,
+ "##oper": 19807,
+ "##pered": 19808,
+ "Russ": 19809,
+ "Terra": 19810,
+ "repeats": 19811,
+ "fluttered": 19812,
+ "Riga": 19813,
+ "##ific": 19814,
+ "##gt": 19815,
+ "cows": 19816,
+ "Hair": 19817,
+ "labelled": 19818,
+ "protects": 19819,
+ "Gale": 19820,
+ "Personnel": 19821,
+ "Düsseldorf": 19822,
+ "Moran": 19823,
+ "rematch": 19824,
+ "##OE": 19825,
+ "Slow": 19826,
+ "forgiveness": 19827,
+ "##ssi": 19828,
+ "proudly": 19829,
+ "Macmillan": 19830,
+ "insist": 19831,
+ "undoubtedly": 19832,
+ "Québec": 19833,
+ "Violence": 19834,
+ "##yuan": 19835,
+ "##aine": 19836,
+ "mourning": 19837,
+ "linen": 19838,
+ "accidental": 19839,
+ "##iol": 19840,
+ "##arium": 19841,
+ "grossing": 19842,
+ "lattice": 19843,
+ "maneuver": 19844,
+ "##marine": 19845,
+ "prestige": 19846,
+ "petrol": 19847,
+ "gradient": 19848,
+ "invasive": 19849,
+ "militant": 19850,
+ "Galerie": 19851,
+ "widening": 19852,
+ "##aman": 19853,
+ "##quist": 19854,
+ "disagreement": 19855,
+ "##ales": 19856,
+ "creepy": 19857,
+ "remembers": 19858,
+ "buzz": 19859,
+ "##erial": 19860,
+ "Exempt": 19861,
+ "Dirk": 19862,
+ "mon": 19863,
+ "Addison": 19864,
+ "##inen": 19865,
+ "deposed": 19866,
+ "##agon": 19867,
+ "fifteenth": 19868,
+ "Hang": 19869,
+ "ornate": 19870,
+ "slab": 19871,
+ "##lades": 19872,
+ "Fountain": 19873,
+ "contractors": 19874,
+ "das": 19875,
+ "Warwickshire": 19876,
+ "1763": 19877,
+ "##rc": 19878,
+ "Carly": 19879,
+ "Essays": 19880,
+ "Indy": 19881,
+ "Ligue": 19882,
+ "greenhouse": 19883,
+ "slit": 19884,
+ "##sea": 19885,
+ "chewed": 19886,
+ "wink": 19887,
+ "##azi": 19888,
+ "Playhouse": 19889,
+ "##kon": 19890,
+ "Gram": 19891,
+ "Ko": 19892,
+ "Samson": 19893,
+ "creators": 19894,
+ "revive": 19895,
+ "##rians": 19896,
+ "spawned": 19897,
+ "seminars": 19898,
+ "Craft": 19899,
+ "Tall": 19900,
+ "diverted": 19901,
+ "assistants": 19902,
+ "computational": 19903,
+ "enclosure": 19904,
+ "##acity": 19905,
+ "Coca": 19906,
+ "##eve": 19907,
+ "databases": 19908,
+ "Drop": 19909,
+ "##loading": 19910,
+ "##hage": 19911,
+ "Greco": 19912,
+ "Privy": 19913,
+ "entrances": 19914,
+ "pork": 19915,
+ "prospective": 19916,
+ "Memories": 19917,
+ "robes": 19918,
+ "##market": 19919,
+ "transporting": 19920,
+ "##lik": 19921,
+ "Rudolph": 19922,
+ "Horton": 19923,
+ "visually": 19924,
+ "##uay": 19925,
+ "##nja": 19926,
+ "Centro": 19927,
+ "Tor": 19928,
+ "Howell": 19929,
+ "##rsey": 19930,
+ "admitting": 19931,
+ "postgraduate": 19932,
+ "herbs": 19933,
+ "##att": 19934,
+ "Chin": 19935,
+ "Rutherford": 19936,
+ "##bot": 19937,
+ "##etta": 19938,
+ "Seasons": 19939,
+ "explanations": 19940,
+ "##bery": 19941,
+ "Friedman": 19942,
+ "heap": 19943,
+ "##ryl": 19944,
+ "##sberg": 19945,
+ "jaws": 19946,
+ "##agh": 19947,
+ "Choi": 19948,
+ "Killing": 19949,
+ "Fanny": 19950,
+ "##suming": 19951,
+ "##hawk": 19952,
+ "hopeful": 19953,
+ "##aid": 19954,
+ "Monty": 19955,
+ "gum": 19956,
+ "remarkably": 19957,
+ "Secrets": 19958,
+ "disco": 19959,
+ "harp": 19960,
+ "advise": 19961,
+ "##avia": 19962,
+ "Marathi": 19963,
+ "##cycle": 19964,
+ "Truck": 19965,
+ "abbot": 19966,
+ "sincere": 19967,
+ "urine": 19968,
+ "##mology": 19969,
+ "masked": 19970,
+ "bathing": 19971,
+ "##tun": 19972,
+ "Fellows": 19973,
+ "##TM": 19974,
+ "##gnetic": 19975,
+ "owl": 19976,
+ "##jon": 19977,
+ "hymn": 19978,
+ "##leton": 19979,
+ "208": 19980,
+ "hostility": 19981,
+ "##cée": 19982,
+ "baked": 19983,
+ "Bottom": 19984,
+ "##AB": 19985,
+ "shudder": 19986,
+ "##ater": 19987,
+ "##von": 19988,
+ "##hee": 19989,
+ "reorganization": 19990,
+ "Cycle": 19991,
+ "##phs": 19992,
+ "Lex": 19993,
+ "##style": 19994,
+ "##rms": 19995,
+ "Translation": 19996,
+ "##erick": 19997,
+ "##imeter": 19998,
+ "##ière": 19999,
+ "attested": 20000,
+ "Hillary": 20001,
+ "##DM": 20002,
+ "gal": 20003,
+ "wander": 20004,
+ "Salle": 20005,
+ "##laming": 20006,
+ "Perez": 20007,
+ "Pit": 20008,
+ "##LP": 20009,
+ "USAF": 20010,
+ "contexts": 20011,
+ "Disease": 20012,
+ "blazing": 20013,
+ "aroused": 20014,
+ "razor": 20015,
+ "walled": 20016,
+ "Danielle": 20017,
+ "Mont": 20018,
+ "Funk": 20019,
+ "royalty": 20020,
+ "thee": 20021,
+ "203": 20022,
+ "donors": 20023,
+ "##erton": 20024,
+ "famously": 20025,
+ "processors": 20026,
+ "reassigned": 20027,
+ "welcoming": 20028,
+ "Goldberg": 20029,
+ "##quities": 20030,
+ "undisclosed": 20031,
+ "Orient": 20032,
+ "Patty": 20033,
+ "vaccine": 20034,
+ "refrigerator": 20035,
+ "Cypriot": 20036,
+ "consonant": 20037,
+ "##waters": 20038,
+ "176": 20039,
+ "sober": 20040,
+ "##lement": 20041,
+ "Racecourse": 20042,
+ "##uate": 20043,
+ "Luckily": 20044,
+ "Selection": 20045,
+ "conceptual": 20046,
+ "vines": 20047,
+ "Breaking": 20048,
+ "wa": 20049,
+ "lions": 20050,
+ "oversight": 20051,
+ "sheltered": 20052,
+ "Dancer": 20053,
+ "ponds": 20054,
+ "borrow": 20055,
+ "##BB": 20056,
+ "##pulsion": 20057,
+ "Daly": 20058,
+ "##eek": 20059,
+ "fertility": 20060,
+ "spontaneous": 20061,
+ "Worldwide": 20062,
+ "gasping": 20063,
+ "##tino": 20064,
+ "169": 20065,
+ "ABS": 20066,
+ "Vickers": 20067,
+ "ambient": 20068,
+ "energetic": 20069,
+ "prisons": 20070,
+ "##eson": 20071,
+ "Stacy": 20072,
+ "##roach": 20073,
+ "GmbH": 20074,
+ "Afro": 20075,
+ "Marin": 20076,
+ "farmhouse": 20077,
+ "pinched": 20078,
+ "##cursion": 20079,
+ "##sp": 20080,
+ "Sabine": 20081,
+ "##pire": 20082,
+ "181": 20083,
+ "nak": 20084,
+ "swelling": 20085,
+ "humble": 20086,
+ "perfume": 20087,
+ "##balls": 20088,
+ "Rai": 20089,
+ "cannons": 20090,
+ "##taker": 20091,
+ "Married": 20092,
+ "Maltese": 20093,
+ "canals": 20094,
+ "interceptions": 20095,
+ "hats": 20096,
+ "lever": 20097,
+ "slowing": 20098,
+ "##ppy": 20099,
+ "Nike": 20100,
+ "Silas": 20101,
+ "Scarborough": 20102,
+ "skirts": 20103,
+ "166": 20104,
+ "inauguration": 20105,
+ "Shuttle": 20106,
+ "alloy": 20107,
+ "beads": 20108,
+ "belts": 20109,
+ "Compton": 20110,
+ "Cause": 20111,
+ "battling": 20112,
+ "critique": 20113,
+ "surf": 20114,
+ "Dock": 20115,
+ "roommate": 20116,
+ "##ulet": 20117,
+ "invade": 20118,
+ "Garland": 20119,
+ "##slow": 20120,
+ "nutrition": 20121,
+ "persona": 20122,
+ "##zam": 20123,
+ "Wichita": 20124,
+ "acquaintance": 20125,
+ "coincided": 20126,
+ "##cate": 20127,
+ "Dracula": 20128,
+ "clamped": 20129,
+ "##gau": 20130,
+ "overhaul": 20131,
+ "##broken": 20132,
+ "##rrier": 20133,
+ "melodies": 20134,
+ "ventures": 20135,
+ "Paz": 20136,
+ "convex": 20137,
+ "Roots": 20138,
+ "##holding": 20139,
+ "Tribute": 20140,
+ "transgender": 20141,
+ "##ò": 20142,
+ "chimney": 20143,
+ "##riad": 20144,
+ "Ajax": 20145,
+ "Thereafter": 20146,
+ "messed": 20147,
+ "nowadays": 20148,
+ "pH": 20149,
+ "##100": 20150,
+ "##alog": 20151,
+ "Pomerania": 20152,
+ "##yra": 20153,
+ "Rossi": 20154,
+ "glove": 20155,
+ "##TL": 20156,
+ "Races": 20157,
+ "##asily": 20158,
+ "tablets": 20159,
+ "Jase": 20160,
+ "##ttes": 20161,
+ "diner": 20162,
+ "##rns": 20163,
+ "Hu": 20164,
+ "Mohan": 20165,
+ "anytime": 20166,
+ "weighted": 20167,
+ "remixes": 20168,
+ "Dove": 20169,
+ "cherry": 20170,
+ "imports": 20171,
+ "##urity": 20172,
+ "GA": 20173,
+ "##TT": 20174,
+ "##iated": 20175,
+ "##sford": 20176,
+ "Clarkson": 20177,
+ "evidently": 20178,
+ "rugged": 20179,
+ "Dust": 20180,
+ "siding": 20181,
+ "##ometer": 20182,
+ "acquitted": 20183,
+ "choral": 20184,
+ "##mite": 20185,
+ "infants": 20186,
+ "Domenico": 20187,
+ "gallons": 20188,
+ "Atkinson": 20189,
+ "gestures": 20190,
+ "slated": 20191,
+ "##xa": 20192,
+ "Archaeology": 20193,
+ "unwanted": 20194,
+ "##ibes": 20195,
+ "##duced": 20196,
+ "premise": 20197,
+ "Colby": 20198,
+ "Geelong": 20199,
+ "disqualified": 20200,
+ "##pf": 20201,
+ "##voking": 20202,
+ "simplicity": 20203,
+ "Walkover": 20204,
+ "Qaeda": 20205,
+ "Warden": 20206,
+ "##bourg": 20207,
+ "##ān": 20208,
+ "Invasion": 20209,
+ "Babe": 20210,
+ "harness": 20211,
+ "183": 20212,
+ "##tated": 20213,
+ "maze": 20214,
+ "Burt": 20215,
+ "bedrooms": 20216,
+ "##nsley": 20217,
+ "Horizon": 20218,
+ "##oast": 20219,
+ "minimize": 20220,
+ "peeked": 20221,
+ "MLA": 20222,
+ "Trains": 20223,
+ "tractor": 20224,
+ "nudged": 20225,
+ "##iform": 20226,
+ "Growth": 20227,
+ "Benton": 20228,
+ "separates": 20229,
+ "##about": 20230,
+ "##kari": 20231,
+ "buffer": 20232,
+ "anthropology": 20233,
+ "brigades": 20234,
+ "foil": 20235,
+ "##wu": 20236,
+ "Domain": 20237,
+ "licking": 20238,
+ "whore": 20239,
+ "##rage": 20240,
+ "##sham": 20241,
+ "Initial": 20242,
+ "Courthouse": 20243,
+ "Rutgers": 20244,
+ "dams": 20245,
+ "villains": 20246,
+ "supermarket": 20247,
+ "##brush": 20248,
+ "Brunei": 20249,
+ "Palermo": 20250,
+ "arises": 20251,
+ "Passenger": 20252,
+ "outreach": 20253,
+ "##gill": 20254,
+ "Labrador": 20255,
+ "McLaren": 20256,
+ "##uy": 20257,
+ "Lori": 20258,
+ "##fires": 20259,
+ "Heads": 20260,
+ "magistrate": 20261,
+ "¹⁄₂": 20262,
+ "Weapons": 20263,
+ "##wai": 20264,
+ "##roke": 20265,
+ "projecting": 20266,
+ "##ulates": 20267,
+ "bordering": 20268,
+ "McKenzie": 20269,
+ "Pavel": 20270,
+ "midway": 20271,
+ "Guangzhou": 20272,
+ "streamed": 20273,
+ "racer": 20274,
+ "##lished": 20275,
+ "eccentric": 20276,
+ "spectral": 20277,
+ "206": 20278,
+ "##mism": 20279,
+ "Wilde": 20280,
+ "Grange": 20281,
+ "preparatory": 20282,
+ "lent": 20283,
+ "##tam": 20284,
+ "starving": 20285,
+ "Gertrude": 20286,
+ "##cea": 20287,
+ "##ricted": 20288,
+ "Breakfast": 20289,
+ "Mira": 20290,
+ "blurted": 20291,
+ "derive": 20292,
+ "##lair": 20293,
+ "blunt": 20294,
+ "sob": 20295,
+ "Cheltenham": 20296,
+ "Henrik": 20297,
+ "reinstated": 20298,
+ "intends": 20299,
+ "##istan": 20300,
+ "unite": 20301,
+ "##ector": 20302,
+ "playful": 20303,
+ "sparks": 20304,
+ "mapped": 20305,
+ "Cadet": 20306,
+ "luggage": 20307,
+ "prosperous": 20308,
+ "##ein": 20309,
+ "salon": 20310,
+ "##utes": 20311,
+ "Biological": 20312,
+ "##rland": 20313,
+ "Tyrone": 20314,
+ "buyer": 20315,
+ "##lose": 20316,
+ "amounted": 20317,
+ "Saw": 20318,
+ "smirked": 20319,
+ "Ronan": 20320,
+ "Reviews": 20321,
+ "Adele": 20322,
+ "trait": 20323,
+ "##proof": 20324,
+ "Bhutan": 20325,
+ "Ginger": 20326,
+ "##junct": 20327,
+ "digitally": 20328,
+ "stirring": 20329,
+ "##isted": 20330,
+ "coconut": 20331,
+ "Hamlet": 20332,
+ "Dinner": 20333,
+ "Scale": 20334,
+ "pledge": 20335,
+ "##RP": 20336,
+ "Wrong": 20337,
+ "Goal": 20338,
+ "Panel": 20339,
+ "therapeutic": 20340,
+ "elevations": 20341,
+ "infectious": 20342,
+ "priesthood": 20343,
+ "##inda": 20344,
+ "Guyana": 20345,
+ "diagnostic": 20346,
+ "##mbre": 20347,
+ "Blackwell": 20348,
+ "sails": 20349,
+ "##arm": 20350,
+ "literal": 20351,
+ "periodically": 20352,
+ "gleaming": 20353,
+ "Robot": 20354,
+ "Rector": 20355,
+ "##abulous": 20356,
+ "##tres": 20357,
+ "Reaching": 20358,
+ "Romantic": 20359,
+ "CP": 20360,
+ "Wonderful": 20361,
+ "##tur": 20362,
+ "ornamental": 20363,
+ "##nges": 20364,
+ "traitor": 20365,
+ "##zilla": 20366,
+ "genetics": 20367,
+ "mentioning": 20368,
+ "##eim": 20369,
+ "resonance": 20370,
+ "Areas": 20371,
+ "Shopping": 20372,
+ "##nard": 20373,
+ "Gail": 20374,
+ "Solid": 20375,
+ "##rito": 20376,
+ "##mara": 20377,
+ "Willem": 20378,
+ "Chip": 20379,
+ "Matches": 20380,
+ "Volkswagen": 20381,
+ "obstacle": 20382,
+ "Organ": 20383,
+ "invites": 20384,
+ "Coral": 20385,
+ "attain": 20386,
+ "##anus": 20387,
+ "##dates": 20388,
+ "Midway": 20389,
+ "shuffled": 20390,
+ "Cecilia": 20391,
+ "dessert": 20392,
+ "Gateway": 20393,
+ "Ch": 20394,
+ "Napoleonic": 20395,
+ "Petroleum": 20396,
+ "jets": 20397,
+ "goose": 20398,
+ "striped": 20399,
+ "bowls": 20400,
+ "vibration": 20401,
+ "Sims": 20402,
+ "nickel": 20403,
+ "Thirteen": 20404,
+ "problematic": 20405,
+ "intervene": 20406,
+ "##grading": 20407,
+ "##unds": 20408,
+ "Mum": 20409,
+ "semifinal": 20410,
+ "Radical": 20411,
+ "##izations": 20412,
+ "refurbished": 20413,
+ "##sation": 20414,
+ "##harine": 20415,
+ "Maximilian": 20416,
+ "cites": 20417,
+ "Advocate": 20418,
+ "Potomac": 20419,
+ "surged": 20420,
+ "preserves": 20421,
+ "Curry": 20422,
+ "angled": 20423,
+ "ordination": 20424,
+ "##pad": 20425,
+ "Cade": 20426,
+ "##DE": 20427,
+ "##sko": 20428,
+ "researched": 20429,
+ "torpedoes": 20430,
+ "Resident": 20431,
+ "wetlands": 20432,
+ "hay": 20433,
+ "applicants": 20434,
+ "depart": 20435,
+ "Bernstein": 20436,
+ "##pic": 20437,
+ "##ario": 20438,
+ "##rae": 20439,
+ "favourable": 20440,
+ "##wari": 20441,
+ "##р": 20442,
+ "metabolism": 20443,
+ "nobleman": 20444,
+ "Defaulted": 20445,
+ "calculate": 20446,
+ "ignition": 20447,
+ "Celebrity": 20448,
+ "Belize": 20449,
+ "sulfur": 20450,
+ "Flat": 20451,
+ "Sc": 20452,
+ "USB": 20453,
+ "flicker": 20454,
+ "Hertfordshire": 20455,
+ "Sept": 20456,
+ "CFL": 20457,
+ "Pasadena": 20458,
+ "Saturdays": 20459,
+ "Titus": 20460,
+ "##nir": 20461,
+ "Canary": 20462,
+ "Computing": 20463,
+ "Isaiah": 20464,
+ "##mler": 20465,
+ "formidable": 20466,
+ "pulp": 20467,
+ "orchid": 20468,
+ "Called": 20469,
+ "Solutions": 20470,
+ "kilograms": 20471,
+ "steamer": 20472,
+ "##hil": 20473,
+ "Doncaster": 20474,
+ "successors": 20475,
+ "Stokes": 20476,
+ "Holstein": 20477,
+ "##sius": 20478,
+ "sperm": 20479,
+ "API": 20480,
+ "Rogue": 20481,
+ "instability": 20482,
+ "Acoustic": 20483,
+ "##rag": 20484,
+ "159": 20485,
+ "undercover": 20486,
+ "Wouldn": 20487,
+ "##pra": 20488,
+ "##medical": 20489,
+ "Eliminated": 20490,
+ "honorable": 20491,
+ "##chel": 20492,
+ "denomination": 20493,
+ "abrupt": 20494,
+ "Buffy": 20495,
+ "blouse": 20496,
+ "fi": 20497,
+ "Regardless": 20498,
+ "Subsequent": 20499,
+ "##rdes": 20500,
+ "Lover": 20501,
+ "##tford": 20502,
+ "bacon": 20503,
+ "##emia": 20504,
+ "carving": 20505,
+ "##cripts": 20506,
+ "Massacre": 20507,
+ "Ramos": 20508,
+ "Latter": 20509,
+ "##ulp": 20510,
+ "ballroom": 20511,
+ "##gement": 20512,
+ "richest": 20513,
+ "bruises": 20514,
+ "Rest": 20515,
+ "Wiley": 20516,
+ "##aster": 20517,
+ "explosions": 20518,
+ "##lastic": 20519,
+ "Edo": 20520,
+ "##LD": 20521,
+ "Mir": 20522,
+ "choking": 20523,
+ "disgusted": 20524,
+ "faintly": 20525,
+ "Barracks": 20526,
+ "blasted": 20527,
+ "headlights": 20528,
+ "Tours": 20529,
+ "ensued": 20530,
+ "presentations": 20531,
+ "##cale": 20532,
+ "wrought": 20533,
+ "##oat": 20534,
+ "##coa": 20535,
+ "Quaker": 20536,
+ "##sdale": 20537,
+ "recipe": 20538,
+ "##gny": 20539,
+ "corpses": 20540,
+ "##liance": 20541,
+ "comfortably": 20542,
+ "##wat": 20543,
+ "Landscape": 20544,
+ "niche": 20545,
+ "catalyst": 20546,
+ "##leader": 20547,
+ "Securities": 20548,
+ "messy": 20549,
+ "##RL": 20550,
+ "Rodrigo": 20551,
+ "backdrop": 20552,
+ "##opping": 20553,
+ "treats": 20554,
+ "Emilio": 20555,
+ "Anand": 20556,
+ "bilateral": 20557,
+ "meadow": 20558,
+ "VC": 20559,
+ "socialism": 20560,
+ "##grad": 20561,
+ "clinics": 20562,
+ "##itating": 20563,
+ "##ppe": 20564,
+ "##ymphonic": 20565,
+ "seniors": 20566,
+ "Advisor": 20567,
+ "Armoured": 20568,
+ "Method": 20569,
+ "Alley": 20570,
+ "##orio": 20571,
+ "Sad": 20572,
+ "fueled": 20573,
+ "raided": 20574,
+ "Axel": 20575,
+ "NH": 20576,
+ "rushes": 20577,
+ "Dixie": 20578,
+ "Otis": 20579,
+ "wrecked": 20580,
+ "##22": 20581,
+ "capitalism": 20582,
+ "café": 20583,
+ "##bbe": 20584,
+ "##pion": 20585,
+ "##forcing": 20586,
+ "Aubrey": 20587,
+ "Lublin": 20588,
+ "Whenever": 20589,
+ "Sears": 20590,
+ "Scheme": 20591,
+ "##lana": 20592,
+ "Meadows": 20593,
+ "treatise": 20594,
+ "##RI": 20595,
+ "##ustic": 20596,
+ "sacrifices": 20597,
+ "sustainability": 20598,
+ "Biography": 20599,
+ "mystical": 20600,
+ "Wanted": 20601,
+ "multiplayer": 20602,
+ "Applications": 20603,
+ "disliked": 20604,
+ "##tisfied": 20605,
+ "impaired": 20606,
+ "empirical": 20607,
+ "forgetting": 20608,
+ "Fairfield": 20609,
+ "Sunni": 20610,
+ "blurred": 20611,
+ "Growing": 20612,
+ "Avalon": 20613,
+ "coil": 20614,
+ "Camera": 20615,
+ "Skin": 20616,
+ "bruised": 20617,
+ "terminals": 20618,
+ "##fted": 20619,
+ "##roving": 20620,
+ "Commando": 20621,
+ "##hya": 20622,
+ "##sper": 20623,
+ "reservations": 20624,
+ "needles": 20625,
+ "dangling": 20626,
+ "##rsch": 20627,
+ "##rsten": 20628,
+ "##spect": 20629,
+ "##mbs": 20630,
+ "yoga": 20631,
+ "regretted": 20632,
+ "Bliss": 20633,
+ "Orion": 20634,
+ "Rufus": 20635,
+ "glucose": 20636,
+ "Olsen": 20637,
+ "autobiographical": 20638,
+ "##dened": 20639,
+ "222": 20640,
+ "humidity": 20641,
+ "Shan": 20642,
+ "##ifiable": 20643,
+ "supper": 20644,
+ "##rou": 20645,
+ "flare": 20646,
+ "##MO": 20647,
+ "campaigning": 20648,
+ "descend": 20649,
+ "socio": 20650,
+ "declares": 20651,
+ "Mounted": 20652,
+ "Gracie": 20653,
+ "Arte": 20654,
+ "endurance": 20655,
+ "##ety": 20656,
+ "Copper": 20657,
+ "costa": 20658,
+ "airplay": 20659,
+ "##MB": 20660,
+ "Proceedings": 20661,
+ "dislike": 20662,
+ "grimaced": 20663,
+ "occupants": 20664,
+ "births": 20665,
+ "glacial": 20666,
+ "oblivious": 20667,
+ "cans": 20668,
+ "installment": 20669,
+ "muddy": 20670,
+ "##ł": 20671,
+ "captains": 20672,
+ "pneumonia": 20673,
+ "Quiet": 20674,
+ "Sloan": 20675,
+ "Excuse": 20676,
+ "##nine": 20677,
+ "Geography": 20678,
+ "gymnastics": 20679,
+ "multimedia": 20680,
+ "drains": 20681,
+ "Anthology": 20682,
+ "Gear": 20683,
+ "cylindrical": 20684,
+ "Fry": 20685,
+ "undertaking": 20686,
+ "##pler": 20687,
+ "##tility": 20688,
+ "Nan": 20689,
+ "##recht": 20690,
+ "Dub": 20691,
+ "philosophers": 20692,
+ "piss": 20693,
+ "Atari": 20694,
+ "##pha": 20695,
+ "Galicia": 20696,
+ "México": 20697,
+ "##nking": 20698,
+ "Continuing": 20699,
+ "bump": 20700,
+ "graveyard": 20701,
+ "persisted": 20702,
+ "Shrine": 20703,
+ "##erapy": 20704,
+ "defects": 20705,
+ "Advance": 20706,
+ "Bomber": 20707,
+ "##oil": 20708,
+ "##ffling": 20709,
+ "cheerful": 20710,
+ "##lix": 20711,
+ "scrub": 20712,
+ "##eto": 20713,
+ "awkwardly": 20714,
+ "collaborator": 20715,
+ "fencing": 20716,
+ "##alo": 20717,
+ "prophet": 20718,
+ "Croix": 20719,
+ "coughed": 20720,
+ "##lication": 20721,
+ "roadway": 20722,
+ "slaughter": 20723,
+ "elephants": 20724,
+ "##erated": 20725,
+ "Simpsons": 20726,
+ "vulnerability": 20727,
+ "ivory": 20728,
+ "Birth": 20729,
+ "lizard": 20730,
+ "scarce": 20731,
+ "cylinders": 20732,
+ "fortunes": 20733,
+ "##NL": 20734,
+ "Hate": 20735,
+ "Priory": 20736,
+ "##lai": 20737,
+ "McBride": 20738,
+ "##copy": 20739,
+ "Lenny": 20740,
+ "liaison": 20741,
+ "Triangle": 20742,
+ "coronation": 20743,
+ "sampled": 20744,
+ "savage": 20745,
+ "amidst": 20746,
+ "Grady": 20747,
+ "whatsoever": 20748,
+ "instinctively": 20749,
+ "Reconstruction": 20750,
+ "insides": 20751,
+ "seizure": 20752,
+ "Drawing": 20753,
+ "##rlin": 20754,
+ "Antioch": 20755,
+ "Gao": 20756,
+ "Díaz": 20757,
+ "1760": 20758,
+ "Sparks": 20759,
+ "##tien": 20760,
+ "##bidae": 20761,
+ "rehearsal": 20762,
+ "##bbs": 20763,
+ "botanical": 20764,
+ "##hers": 20765,
+ "compensate": 20766,
+ "wholesale": 20767,
+ "Seville": 20768,
+ "shareholder": 20769,
+ "prediction": 20770,
+ "astronomical": 20771,
+ "Reddy": 20772,
+ "hardest": 20773,
+ "circling": 20774,
+ "whereabouts": 20775,
+ "termination": 20776,
+ "Rep": 20777,
+ "Assistance": 20778,
+ "Dramatic": 20779,
+ "Herb": 20780,
+ "##ghter": 20781,
+ "climbs": 20782,
+ "188": 20783,
+ "Poole": 20784,
+ "301": 20785,
+ "##pable": 20786,
+ "wit": 20787,
+ "##istice": 20788,
+ "Walters": 20789,
+ "relying": 20790,
+ "Jakob": 20791,
+ "##redo": 20792,
+ "proceeding": 20793,
+ "Langley": 20794,
+ "affiliates": 20795,
+ "ou": 20796,
+ "##allo": 20797,
+ "##holm": 20798,
+ "Samsung": 20799,
+ "##ishi": 20800,
+ "Missing": 20801,
+ "Xi": 20802,
+ "vertices": 20803,
+ "Claus": 20804,
+ "foam": 20805,
+ "restless": 20806,
+ "##uating": 20807,
+ "##sso": 20808,
+ "##ttering": 20809,
+ "Philips": 20810,
+ "delta": 20811,
+ "bombed": 20812,
+ "Catalogue": 20813,
+ "coaster": 20814,
+ "Ling": 20815,
+ "Willard": 20816,
+ "satire": 20817,
+ "410": 20818,
+ "Composition": 20819,
+ "Net": 20820,
+ "Orioles": 20821,
+ "##ldon": 20822,
+ "fins": 20823,
+ "Palatinate": 20824,
+ "Woodward": 20825,
+ "tease": 20826,
+ "tilt": 20827,
+ "brightness": 20828,
+ "##70": 20829,
+ "##bbling": 20830,
+ "##loss": 20831,
+ "##dhi": 20832,
+ "##uilt": 20833,
+ "Whoever": 20834,
+ "##yers": 20835,
+ "hitter": 20836,
+ "Elton": 20837,
+ "Extension": 20838,
+ "ace": 20839,
+ "Affair": 20840,
+ "restructuring": 20841,
+ "##loping": 20842,
+ "Paterson": 20843,
+ "hi": 20844,
+ "##rya": 20845,
+ "spouse": 20846,
+ "Shay": 20847,
+ "Himself": 20848,
+ "piles": 20849,
+ "preaching": 20850,
+ "##gical": 20851,
+ "bikes": 20852,
+ "Brave": 20853,
+ "expulsion": 20854,
+ "Mirza": 20855,
+ "stride": 20856,
+ "Trees": 20857,
+ "commemorated": 20858,
+ "famine": 20859,
+ "masonry": 20860,
+ "Selena": 20861,
+ "Watt": 20862,
+ "Banking": 20863,
+ "Rancho": 20864,
+ "Stockton": 20865,
+ "dip": 20866,
+ "tattoos": 20867,
+ "Vlad": 20868,
+ "acquainted": 20869,
+ "Flyers": 20870,
+ "ruthless": 20871,
+ "fourteenth": 20872,
+ "illustrate": 20873,
+ "##akes": 20874,
+ "EPA": 20875,
+ "##rows": 20876,
+ "##uiz": 20877,
+ "bumped": 20878,
+ "Designed": 20879,
+ "Leaders": 20880,
+ "mastered": 20881,
+ "Manfred": 20882,
+ "swirled": 20883,
+ "McCain": 20884,
+ "##rout": 20885,
+ "Artemis": 20886,
+ "rabbi": 20887,
+ "flinched": 20888,
+ "upgrades": 20889,
+ "penetrate": 20890,
+ "shipyard": 20891,
+ "transforming": 20892,
+ "caretaker": 20893,
+ "##eiro": 20894,
+ "Maureen": 20895,
+ "tightening": 20896,
+ "##founded": 20897,
+ "RAM": 20898,
+ "##icular": 20899,
+ "##mper": 20900,
+ "##rung": 20901,
+ "Fifteen": 20902,
+ "exploited": 20903,
+ "consistency": 20904,
+ "interstate": 20905,
+ "##ynn": 20906,
+ "Bridget": 20907,
+ "contamination": 20908,
+ "Mistress": 20909,
+ "##rup": 20910,
+ "coating": 20911,
+ "##FP": 20912,
+ "##jective": 20913,
+ "Libyan": 20914,
+ "211": 20915,
+ "Gemma": 20916,
+ "dependence": 20917,
+ "shrubs": 20918,
+ "##ggled": 20919,
+ "Germain": 20920,
+ "retaliation": 20921,
+ "traction": 20922,
+ "##PP": 20923,
+ "Dangerous": 20924,
+ "terminology": 20925,
+ "psychiatrist": 20926,
+ "##garten": 20927,
+ "hurdles": 20928,
+ "Natal": 20929,
+ "wasting": 20930,
+ "Weir": 20931,
+ "revolves": 20932,
+ "stripe": 20933,
+ "##reased": 20934,
+ "preferences": 20935,
+ "##entation": 20936,
+ "##lde": 20937,
+ "##áil": 20938,
+ "##otherapy": 20939,
+ "Flame": 20940,
+ "##ologies": 20941,
+ "viruses": 20942,
+ "Label": 20943,
+ "Pandora": 20944,
+ "veil": 20945,
+ "##ogical": 20946,
+ "Coliseum": 20947,
+ "Cottage": 20948,
+ "creeping": 20949,
+ "Jong": 20950,
+ "lectured": 20951,
+ "##çaise": 20952,
+ "shoreline": 20953,
+ "##fference": 20954,
+ "##hra": 20955,
+ "Shade": 20956,
+ "Clock": 20957,
+ "Faye": 20958,
+ "bilingual": 20959,
+ "Humboldt": 20960,
+ "Operating": 20961,
+ "##fter": 20962,
+ "##was": 20963,
+ "algae": 20964,
+ "towed": 20965,
+ "amphibious": 20966,
+ "Parma": 20967,
+ "impacted": 20968,
+ "smacked": 20969,
+ "Piedmont": 20970,
+ "Monsters": 20971,
+ "##omb": 20972,
+ "Moor": 20973,
+ "##lberg": 20974,
+ "sinister": 20975,
+ "Postal": 20976,
+ "178": 20977,
+ "Drummond": 20978,
+ "Sign": 20979,
+ "textbooks": 20980,
+ "hazardous": 20981,
+ "Brass": 20982,
+ "Rosemary": 20983,
+ "Pick": 20984,
+ "Sit": 20985,
+ "Architect": 20986,
+ "transverse": 20987,
+ "Centennial": 20988,
+ "confess": 20989,
+ "polling": 20990,
+ "##aia": 20991,
+ "Julien": 20992,
+ "##mand": 20993,
+ "consolidation": 20994,
+ "Ethel": 20995,
+ "##ulse": 20996,
+ "severity": 20997,
+ "Yorker": 20998,
+ "choreographer": 20999,
+ "1840s": 21000,
+ "##ltry": 21001,
+ "softer": 21002,
+ "versa": 21003,
+ "##geny": 21004,
+ "##quila": 21005,
+ "##jō": 21006,
+ "Caledonia": 21007,
+ "Friendship": 21008,
+ "Visa": 21009,
+ "rogue": 21010,
+ "##zzle": 21011,
+ "bait": 21012,
+ "feather": 21013,
+ "incidence": 21014,
+ "Foods": 21015,
+ "Ships": 21016,
+ "##uto": 21017,
+ "##stead": 21018,
+ "arousal": 21019,
+ "##rote": 21020,
+ "Hazel": 21021,
+ "##bolic": 21022,
+ "Swing": 21023,
+ "##ej": 21024,
+ "##cule": 21025,
+ "##jana": 21026,
+ "##metry": 21027,
+ "##uity": 21028,
+ "Valuable": 21029,
+ "##ₙ": 21030,
+ "Shropshire": 21031,
+ "##nect": 21032,
+ "365": 21033,
+ "Ones": 21034,
+ "realise": 21035,
+ "Café": 21036,
+ "Albuquerque": 21037,
+ "##grown": 21038,
+ "##stadt": 21039,
+ "209": 21040,
+ "##ᵢ": 21041,
+ "prefers": 21042,
+ "withstand": 21043,
+ "Lillian": 21044,
+ "MacArthur": 21045,
+ "Hara": 21046,
+ "##fulness": 21047,
+ "domination": 21048,
+ "##VO": 21049,
+ "##school": 21050,
+ "Freddy": 21051,
+ "ethnicity": 21052,
+ "##while": 21053,
+ "adorned": 21054,
+ "hormone": 21055,
+ "Calder": 21056,
+ "Domestic": 21057,
+ "Freud": 21058,
+ "Shields": 21059,
+ "##phus": 21060,
+ "##rgan": 21061,
+ "BP": 21062,
+ "Segunda": 21063,
+ "Mustang": 21064,
+ "##GI": 21065,
+ "Bonn": 21066,
+ "patiently": 21067,
+ "remarried": 21068,
+ "##umbria": 21069,
+ "Crete": 21070,
+ "Elephant": 21071,
+ "Nuremberg": 21072,
+ "tolerate": 21073,
+ "Tyson": 21074,
+ "##evich": 21075,
+ "Programming": 21076,
+ "##lander": 21077,
+ "Bethlehem": 21078,
+ "segregation": 21079,
+ "Constituency": 21080,
+ "quarterly": 21081,
+ "blushed": 21082,
+ "photographers": 21083,
+ "Sheldon": 21084,
+ "porcelain": 21085,
+ "Blanche": 21086,
+ "goddamn": 21087,
+ "lively": 21088,
+ "##fused": 21089,
+ "bumps": 21090,
+ "##eli": 21091,
+ "curated": 21092,
+ "coherent": 21093,
+ "provoked": 21094,
+ "##vet": 21095,
+ "Madeleine": 21096,
+ "##isco": 21097,
+ "rainy": 21098,
+ "Bethel": 21099,
+ "accusation": 21100,
+ "ponytail": 21101,
+ "gag": 21102,
+ "##lington": 21103,
+ "quicker": 21104,
+ "scroll": 21105,
+ "##vate": 21106,
+ "Bow": 21107,
+ "Gender": 21108,
+ "Ira": 21109,
+ "crashes": 21110,
+ "ACT": 21111,
+ "Maintenance": 21112,
+ "##aton": 21113,
+ "##ieu": 21114,
+ "bitterly": 21115,
+ "strains": 21116,
+ "rattled": 21117,
+ "vectors": 21118,
+ "##arina": 21119,
+ "##ishly": 21120,
+ "173": 21121,
+ "parole": 21122,
+ "##nx": 21123,
+ "amusing": 21124,
+ "Gonzalez": 21125,
+ "##erative": 21126,
+ "Caucus": 21127,
+ "sensual": 21128,
+ "Penelope": 21129,
+ "coefficient": 21130,
+ "Mateo": 21131,
+ "##mani": 21132,
+ "proposition": 21133,
+ "Duty": 21134,
+ "lacrosse": 21135,
+ "proportions": 21136,
+ "Plato": 21137,
+ "profiles": 21138,
+ "Botswana": 21139,
+ "Brandt": 21140,
+ "reins": 21141,
+ "mandolin": 21142,
+ "encompassing": 21143,
+ "##gens": 21144,
+ "Kahn": 21145,
+ "prop": 21146,
+ "summon": 21147,
+ "##MR": 21148,
+ "##yrian": 21149,
+ "##zaki": 21150,
+ "Falling": 21151,
+ "conditional": 21152,
+ "thy": 21153,
+ "##bao": 21154,
+ "##ych": 21155,
+ "radioactive": 21156,
+ "##nics": 21157,
+ "Newspaper": 21158,
+ "##people": 21159,
+ "##nded": 21160,
+ "Gaming": 21161,
+ "sunny": 21162,
+ "##look": 21163,
+ "Sherwood": 21164,
+ "crafted": 21165,
+ "NJ": 21166,
+ "awoke": 21167,
+ "187": 21168,
+ "timeline": 21169,
+ "giants": 21170,
+ "possessing": 21171,
+ "##ycle": 21172,
+ "Cheryl": 21173,
+ "ng": 21174,
+ "Ruiz": 21175,
+ "polymer": 21176,
+ "potassium": 21177,
+ "Ramsay": 21178,
+ "relocation": 21179,
+ "##leen": 21180,
+ "Sociology": 21181,
+ "##bana": 21182,
+ "Franciscan": 21183,
+ "propulsion": 21184,
+ "denote": 21185,
+ "##erjee": 21186,
+ "registers": 21187,
+ "headline": 21188,
+ "Tests": 21189,
+ "emerges": 21190,
+ "Articles": 21191,
+ "Mint": 21192,
+ "livery": 21193,
+ "breakup": 21194,
+ "kits": 21195,
+ "Rap": 21196,
+ "Browning": 21197,
+ "Bunny": 21198,
+ "##mington": 21199,
+ "##watch": 21200,
+ "Anastasia": 21201,
+ "Zachary": 21202,
+ "arranging": 21203,
+ "biographical": 21204,
+ "Erica": 21205,
+ "Nippon": 21206,
+ "##membrance": 21207,
+ "Carmel": 21208,
+ "##sport": 21209,
+ "##xes": 21210,
+ "Paddy": 21211,
+ "##holes": 21212,
+ "Issues": 21213,
+ "Spears": 21214,
+ "compliment": 21215,
+ "##stro": 21216,
+ "##graphs": 21217,
+ "Castillo": 21218,
+ "##MU": 21219,
+ "##space": 21220,
+ "Corporal": 21221,
+ "##nent": 21222,
+ "174": 21223,
+ "Gentlemen": 21224,
+ "##ilize": 21225,
+ "##vage": 21226,
+ "convinces": 21227,
+ "Carmine": 21228,
+ "Crash": 21229,
+ "##hashi": 21230,
+ "Files": 21231,
+ "Doctors": 21232,
+ "brownish": 21233,
+ "sweating": 21234,
+ "goats": 21235,
+ "##conductor": 21236,
+ "rendition": 21237,
+ "##bt": 21238,
+ "NL": 21239,
+ "##spiration": 21240,
+ "generates": 21241,
+ "##cans": 21242,
+ "obsession": 21243,
+ "##noy": 21244,
+ "Danger": 21245,
+ "Diaz": 21246,
+ "heats": 21247,
+ "Realm": 21248,
+ "priorities": 21249,
+ "##phon": 21250,
+ "1300": 21251,
+ "initiation": 21252,
+ "pagan": 21253,
+ "bursts": 21254,
+ "archipelago": 21255,
+ "chloride": 21256,
+ "Screenplay": 21257,
+ "Hewitt": 21258,
+ "Khmer": 21259,
+ "bang": 21260,
+ "judgement": 21261,
+ "negotiating": 21262,
+ "##ait": 21263,
+ "Mabel": 21264,
+ "densely": 21265,
+ "Boulder": 21266,
+ "knob": 21267,
+ "430": 21268,
+ "Alfredo": 21269,
+ "##kt": 21270,
+ "pitches": 21271,
+ "##ées": 21272,
+ "##ان": 21273,
+ "Macdonald": 21274,
+ "##llum": 21275,
+ "imply": 21276,
+ "##mot": 21277,
+ "Smile": 21278,
+ "spherical": 21279,
+ "##tura": 21280,
+ "Derrick": 21281,
+ "Kelley": 21282,
+ "Nico": 21283,
+ "cortex": 21284,
+ "launches": 21285,
+ "differed": 21286,
+ "parallels": 21287,
+ "Navigation": 21288,
+ "##child": 21289,
+ "##rming": 21290,
+ "canoe": 21291,
+ "forestry": 21292,
+ "reinforce": 21293,
+ "##mote": 21294,
+ "confirming": 21295,
+ "tasting": 21296,
+ "scaled": 21297,
+ "##resh": 21298,
+ "##eting": 21299,
+ "Understanding": 21300,
+ "prevailing": 21301,
+ "Pearce": 21302,
+ "CW": 21303,
+ "earnest": 21304,
+ "Gaius": 21305,
+ "asserts": 21306,
+ "denoted": 21307,
+ "landmarks": 21308,
+ "Chargers": 21309,
+ "warns": 21310,
+ "##flies": 21311,
+ "Judges": 21312,
+ "jagged": 21313,
+ "##dain": 21314,
+ "tails": 21315,
+ "Historian": 21316,
+ "Millie": 21317,
+ "##sler": 21318,
+ "221": 21319,
+ "##uard": 21320,
+ "absurd": 21321,
+ "Dion": 21322,
+ "##ially": 21323,
+ "makeshift": 21324,
+ "Specifically": 21325,
+ "ignorance": 21326,
+ "Eat": 21327,
+ "##ieri": 21328,
+ "comparisons": 21329,
+ "forensic": 21330,
+ "186": 21331,
+ "Giro": 21332,
+ "skeptical": 21333,
+ "disciplinary": 21334,
+ "battleship": 21335,
+ "##45": 21336,
+ "Libby": 21337,
+ "520": 21338,
+ "Odyssey": 21339,
+ "ledge": 21340,
+ "##post": 21341,
+ "Eternal": 21342,
+ "Missionary": 21343,
+ "deficiency": 21344,
+ "settler": 21345,
+ "wonders": 21346,
+ "##gai": 21347,
+ "raging": 21348,
+ "##cis": 21349,
+ "Romney": 21350,
+ "Ulrich": 21351,
+ "annexation": 21352,
+ "boxers": 21353,
+ "sect": 21354,
+ "204": 21355,
+ "ARIA": 21356,
+ "dei": 21357,
+ "Hitchcock": 21358,
+ "te": 21359,
+ "Varsity": 21360,
+ "##fic": 21361,
+ "CC": 21362,
+ "lending": 21363,
+ "##nial": 21364,
+ "##tag": 21365,
+ "##rdy": 21366,
+ "##obe": 21367,
+ "Defensive": 21368,
+ "##dson": 21369,
+ "##pore": 21370,
+ "stellar": 21371,
+ "Lam": 21372,
+ "Trials": 21373,
+ "contention": 21374,
+ "Sung": 21375,
+ "##uminous": 21376,
+ "Poe": 21377,
+ "superiority": 21378,
+ "##plicate": 21379,
+ "325": 21380,
+ "bitten": 21381,
+ "conspicuous": 21382,
+ "##olly": 21383,
+ "Lila": 21384,
+ "Pub": 21385,
+ "Petit": 21386,
+ "distorted": 21387,
+ "ISIL": 21388,
+ "distinctly": 21389,
+ "##family": 21390,
+ "Cowboy": 21391,
+ "mutant": 21392,
+ "##cats": 21393,
+ "##week": 21394,
+ "Changes": 21395,
+ "Sinatra": 21396,
+ "epithet": 21397,
+ "neglect": 21398,
+ "Innocent": 21399,
+ "gamma": 21400,
+ "thrill": 21401,
+ "reggae": 21402,
+ "##adia": 21403,
+ "##ational": 21404,
+ "##due": 21405,
+ "landlord": 21406,
+ "##leaf": 21407,
+ "visibly": 21408,
+ "##ì": 21409,
+ "Darlington": 21410,
+ "Gomez": 21411,
+ "##iting": 21412,
+ "scarf": 21413,
+ "##lade": 21414,
+ "Hinduism": 21415,
+ "Fever": 21416,
+ "scouts": 21417,
+ "##roi": 21418,
+ "convened": 21419,
+ "##oki": 21420,
+ "184": 21421,
+ "Lao": 21422,
+ "boycott": 21423,
+ "unemployed": 21424,
+ "##lore": 21425,
+ "##ß": 21426,
+ "##hammer": 21427,
+ "Curran": 21428,
+ "disciples": 21429,
+ "odor": 21430,
+ "##ygiene": 21431,
+ "Lighthouse": 21432,
+ "Played": 21433,
+ "whales": 21434,
+ "discretion": 21435,
+ "Yves": 21436,
+ "##ceived": 21437,
+ "pauses": 21438,
+ "coincide": 21439,
+ "##nji": 21440,
+ "dizzy": 21441,
+ "##scopic": 21442,
+ "routed": 21443,
+ "Guardians": 21444,
+ "Kellan": 21445,
+ "carnival": 21446,
+ "nasal": 21447,
+ "224": 21448,
+ "##awed": 21449,
+ "Mitsubishi": 21450,
+ "640": 21451,
+ "Cast": 21452,
+ "silky": 21453,
+ "Projects": 21454,
+ "joked": 21455,
+ "Huddersfield": 21456,
+ "Rothschild": 21457,
+ "zu": 21458,
+ "##olar": 21459,
+ "Divisions": 21460,
+ "mildly": 21461,
+ "##eni": 21462,
+ "##lge": 21463,
+ "Appalachian": 21464,
+ "Sahara": 21465,
+ "pinch": 21466,
+ "##roon": 21467,
+ "wardrobe": 21468,
+ "##dham": 21469,
+ "##etal": 21470,
+ "Bubba": 21471,
+ "##lini": 21472,
+ "##rumbling": 21473,
+ "Communities": 21474,
+ "Poznań": 21475,
+ "unification": 21476,
+ "Beau": 21477,
+ "Kris": 21478,
+ "SV": 21479,
+ "Rowing": 21480,
+ "Minh": 21481,
+ "reconciliation": 21482,
+ "##saki": 21483,
+ "##sor": 21484,
+ "taped": 21485,
+ "##reck": 21486,
+ "certificates": 21487,
+ "gubernatorial": 21488,
+ "rainbow": 21489,
+ "##uing": 21490,
+ "litter": 21491,
+ "##lique": 21492,
+ "##oted": 21493,
+ "Butterfly": 21494,
+ "benefited": 21495,
+ "Images": 21496,
+ "induce": 21497,
+ "Balkans": 21498,
+ "Velvet": 21499,
+ "##90": 21500,
+ "##xon": 21501,
+ "Bowman": 21502,
+ "##breaker": 21503,
+ "penis": 21504,
+ "##nitz": 21505,
+ "##oint": 21506,
+ "##otive": 21507,
+ "crust": 21508,
+ "##pps": 21509,
+ "organizers": 21510,
+ "Outdoor": 21511,
+ "nominees": 21512,
+ "##rika": 21513,
+ "TX": 21514,
+ "##ucks": 21515,
+ "Protestants": 21516,
+ "##imation": 21517,
+ "appetite": 21518,
+ "Baja": 21519,
+ "awaited": 21520,
+ "##points": 21521,
+ "windshield": 21522,
+ "##igh": 21523,
+ "##zled": 21524,
+ "Brody": 21525,
+ "Buster": 21526,
+ "stylized": 21527,
+ "Bryce": 21528,
+ "##sz": 21529,
+ "Dollar": 21530,
+ "vest": 21531,
+ "mold": 21532,
+ "ounce": 21533,
+ "ok": 21534,
+ "receivers": 21535,
+ "##uza": 21536,
+ "Purdue": 21537,
+ "Harrington": 21538,
+ "Hodges": 21539,
+ "captures": 21540,
+ "##ggio": 21541,
+ "Reservation": 21542,
+ "##ssin": 21543,
+ "##tman": 21544,
+ "cosmic": 21545,
+ "straightforward": 21546,
+ "flipping": 21547,
+ "remixed": 21548,
+ "##athed": 21549,
+ "Gómez": 21550,
+ "Lim": 21551,
+ "motorcycles": 21552,
+ "economies": 21553,
+ "owning": 21554,
+ "Dani": 21555,
+ "##rosis": 21556,
+ "myths": 21557,
+ "sire": 21558,
+ "kindly": 21559,
+ "1768": 21560,
+ "Bean": 21561,
+ "graphs": 21562,
+ "##mee": 21563,
+ "##RO": 21564,
+ "##geon": 21565,
+ "puppy": 21566,
+ "Stephenson": 21567,
+ "notified": 21568,
+ "##jer": 21569,
+ "Watching": 21570,
+ "##rama": 21571,
+ "Sino": 21572,
+ "urgency": 21573,
+ "Islanders": 21574,
+ "##mash": 21575,
+ "Plata": 21576,
+ "fumble": 21577,
+ "##chev": 21578,
+ "##stance": 21579,
+ "##rack": 21580,
+ "##she": 21581,
+ "facilitated": 21582,
+ "swings": 21583,
+ "akin": 21584,
+ "enduring": 21585,
+ "payload": 21586,
+ "##phine": 21587,
+ "Deputies": 21588,
+ "murals": 21589,
+ "##tooth": 21590,
+ "610": 21591,
+ "Jays": 21592,
+ "eyeing": 21593,
+ "##quito": 21594,
+ "transparency": 21595,
+ "##cote": 21596,
+ "Timor": 21597,
+ "negatively": 21598,
+ "##isan": 21599,
+ "battled": 21600,
+ "##fected": 21601,
+ "thankful": 21602,
+ "Rage": 21603,
+ "hospitality": 21604,
+ "incorrectly": 21605,
+ "207": 21606,
+ "entrepreneurs": 21607,
+ "##cula": 21608,
+ "##wley": 21609,
+ "hedge": 21610,
+ "##cratic": 21611,
+ "Corpus": 21612,
+ "Odessa": 21613,
+ "Whereas": 21614,
+ "##ln": 21615,
+ "fetch": 21616,
+ "happier": 21617,
+ "Amherst": 21618,
+ "bullying": 21619,
+ "graceful": 21620,
+ "Height": 21621,
+ "Bartholomew": 21622,
+ "willingness": 21623,
+ "qualifier": 21624,
+ "191": 21625,
+ "Syed": 21626,
+ "Wesleyan": 21627,
+ "Layla": 21628,
+ "##rrence": 21629,
+ "Webber": 21630,
+ "##hum": 21631,
+ "Rat": 21632,
+ "##cket": 21633,
+ "##herence": 21634,
+ "Monterey": 21635,
+ "contaminated": 21636,
+ "Beside": 21637,
+ "Mustafa": 21638,
+ "Nana": 21639,
+ "213": 21640,
+ "##pruce": 21641,
+ "Reason": 21642,
+ "##spense": 21643,
+ "spike": 21644,
+ "##gé": 21645,
+ "AU": 21646,
+ "disciple": 21647,
+ "charcoal": 21648,
+ "##lean": 21649,
+ "formulated": 21650,
+ "Diesel": 21651,
+ "Mariners": 21652,
+ "accreditation": 21653,
+ "glossy": 21654,
+ "1800s": 21655,
+ "##ih": 21656,
+ "Mainz": 21657,
+ "unison": 21658,
+ "Marianne": 21659,
+ "shear": 21660,
+ "overseeing": 21661,
+ "vernacular": 21662,
+ "bowled": 21663,
+ "##lett": 21664,
+ "unpopular": 21665,
+ "##ckoned": 21666,
+ "##monia": 21667,
+ "Gaston": 21668,
+ "##TI": 21669,
+ "##oters": 21670,
+ "Cups": 21671,
+ "##bones": 21672,
+ "##ports": 21673,
+ "Museo": 21674,
+ "minors": 21675,
+ "1773": 21676,
+ "Dickens": 21677,
+ "##EL": 21678,
+ "##NBC": 21679,
+ "Presents": 21680,
+ "ambitions": 21681,
+ "axes": 21682,
+ "Río": 21683,
+ "Yukon": 21684,
+ "bedside": 21685,
+ "Ribbon": 21686,
+ "Units": 21687,
+ "faults": 21688,
+ "conceal": 21689,
+ "##lani": 21690,
+ "prevailed": 21691,
+ "214": 21692,
+ "Goodwin": 21693,
+ "Jaguar": 21694,
+ "crumpled": 21695,
+ "Cullen": 21696,
+ "Wireless": 21697,
+ "ceded": 21698,
+ "remotely": 21699,
+ "Bin": 21700,
+ "mocking": 21701,
+ "straps": 21702,
+ "ceramics": 21703,
+ "##avi": 21704,
+ "##uding": 21705,
+ "##ader": 21706,
+ "Taft": 21707,
+ "twenties": 21708,
+ "##aked": 21709,
+ "Problem": 21710,
+ "quasi": 21711,
+ "Lamar": 21712,
+ "##ntes": 21713,
+ "##avan": 21714,
+ "Barr": 21715,
+ "##eral": 21716,
+ "hooks": 21717,
+ "sa": 21718,
+ "##ône": 21719,
+ "194": 21720,
+ "##ross": 21721,
+ "Nero": 21722,
+ "Caine": 21723,
+ "trance": 21724,
+ "Homeland": 21725,
+ "benches": 21726,
+ "Guthrie": 21727,
+ "dismiss": 21728,
+ "##lex": 21729,
+ "César": 21730,
+ "foliage": 21731,
+ "##oot": 21732,
+ "##alty": 21733,
+ "Assyrian": 21734,
+ "Ahead": 21735,
+ "Murdoch": 21736,
+ "dictatorship": 21737,
+ "wraps": 21738,
+ "##ntal": 21739,
+ "Corridor": 21740,
+ "Mackay": 21741,
+ "respectable": 21742,
+ "jewels": 21743,
+ "understands": 21744,
+ "##pathic": 21745,
+ "Bryn": 21746,
+ "##tep": 21747,
+ "ON": 21748,
+ "capsule": 21749,
+ "intrigued": 21750,
+ "Sleeping": 21751,
+ "communists": 21752,
+ "##chayat": 21753,
+ "##current": 21754,
+ "##vez": 21755,
+ "doubling": 21756,
+ "booklet": 21757,
+ "##uche": 21758,
+ "Creed": 21759,
+ "##NU": 21760,
+ "spies": 21761,
+ "##sef": 21762,
+ "adjusting": 21763,
+ "197": 21764,
+ "Imam": 21765,
+ "heaved": 21766,
+ "Tanya": 21767,
+ "canonical": 21768,
+ "restraint": 21769,
+ "senators": 21770,
+ "stainless": 21771,
+ "##gnate": 21772,
+ "Matter": 21773,
+ "cache": 21774,
+ "restrained": 21775,
+ "conflicting": 21776,
+ "stung": 21777,
+ "##ool": 21778,
+ "Sustainable": 21779,
+ "antiquity": 21780,
+ "193": 21781,
+ "heavens": 21782,
+ "inclusive": 21783,
+ "##ador": 21784,
+ "fluent": 21785,
+ "303": 21786,
+ "911": 21787,
+ "archaeologist": 21788,
+ "superseded": 21789,
+ "##plex": 21790,
+ "Tammy": 21791,
+ "inspire": 21792,
+ "##passing": 21793,
+ "##lub": 21794,
+ "Lama": 21795,
+ "Mixing": 21796,
+ "##activated": 21797,
+ "##yote": 21798,
+ "parlor": 21799,
+ "tactic": 21800,
+ "198": 21801,
+ "Stefano": 21802,
+ "prostitute": 21803,
+ "recycling": 21804,
+ "sorted": 21805,
+ "banana": 21806,
+ "Stacey": 21807,
+ "Musée": 21808,
+ "aristocratic": 21809,
+ "cough": 21810,
+ "##rting": 21811,
+ "authorised": 21812,
+ "gangs": 21813,
+ "runoff": 21814,
+ "thoughtfully": 21815,
+ "##nish": 21816,
+ "Fisheries": 21817,
+ "Provence": 21818,
+ "detector": 21819,
+ "hum": 21820,
+ "##zhen": 21821,
+ "pill": 21822,
+ "##árez": 21823,
+ "Map": 21824,
+ "Leaves": 21825,
+ "Peabody": 21826,
+ "skater": 21827,
+ "vent": 21828,
+ "##color": 21829,
+ "390": 21830,
+ "cerebral": 21831,
+ "hostages": 21832,
+ "mare": 21833,
+ "Jurassic": 21834,
+ "swell": 21835,
+ "##isans": 21836,
+ "Knoxville": 21837,
+ "Naked": 21838,
+ "Malaya": 21839,
+ "scowl": 21840,
+ "Cobra": 21841,
+ "##anga": 21842,
+ "Sexual": 21843,
+ "##dron": 21844,
+ "##iae": 21845,
+ "196": 21846,
+ "##drick": 21847,
+ "Ravens": 21848,
+ "Blaine": 21849,
+ "##throp": 21850,
+ "Ismail": 21851,
+ "symmetric": 21852,
+ "##lossom": 21853,
+ "Leicestershire": 21854,
+ "Sylvester": 21855,
+ "glazed": 21856,
+ "##tended": 21857,
+ "Radar": 21858,
+ "fused": 21859,
+ "Families": 21860,
+ "Blacks": 21861,
+ "Sale": 21862,
+ "Zion": 21863,
+ "foothills": 21864,
+ "microwave": 21865,
+ "slain": 21866,
+ "Collingwood": 21867,
+ "##pants": 21868,
+ "##dling": 21869,
+ "killers": 21870,
+ "routinely": 21871,
+ "Janice": 21872,
+ "hearings": 21873,
+ "##chanted": 21874,
+ "##ltration": 21875,
+ "continents": 21876,
+ "##iving": 21877,
+ "##yster": 21878,
+ "##shot": 21879,
+ "##yna": 21880,
+ "injected": 21881,
+ "Guillaume": 21882,
+ "##ibi": 21883,
+ "kinda": 21884,
+ "Confederacy": 21885,
+ "Barnett": 21886,
+ "disasters": 21887,
+ "incapable": 21888,
+ "##grating": 21889,
+ "rhythms": 21890,
+ "betting": 21891,
+ "draining": 21892,
+ "##hak": 21893,
+ "Callie": 21894,
+ "Glover": 21895,
+ "##iliated": 21896,
+ "Sherlock": 21897,
+ "hearted": 21898,
+ "punching": 21899,
+ "Wolverhampton": 21900,
+ "Leaf": 21901,
+ "Pi": 21902,
+ "builders": 21903,
+ "furnished": 21904,
+ "knighted": 21905,
+ "Photo": 21906,
+ "##zle": 21907,
+ "Touring": 21908,
+ "fumbled": 21909,
+ "pads": 21910,
+ "##ий": 21911,
+ "Bartlett": 21912,
+ "Gunner": 21913,
+ "eerie": 21914,
+ "Marius": 21915,
+ "Bonus": 21916,
+ "pots": 21917,
+ "##hino": 21918,
+ "##pta": 21919,
+ "Bray": 21920,
+ "Frey": 21921,
+ "Ortiz": 21922,
+ "stalls": 21923,
+ "belongings": 21924,
+ "Subway": 21925,
+ "fascination": 21926,
+ "metaphor": 21927,
+ "Bat": 21928,
+ "Boer": 21929,
+ "Colchester": 21930,
+ "sway": 21931,
+ "##gro": 21932,
+ "rhetoric": 21933,
+ "##dheim": 21934,
+ "Fool": 21935,
+ "PMID": 21936,
+ "admire": 21937,
+ "##hsil": 21938,
+ "Strand": 21939,
+ "TNA": 21940,
+ "##roth": 21941,
+ "Nottinghamshire": 21942,
+ "##mat": 21943,
+ "##yler": 21944,
+ "Oxfordshire": 21945,
+ "##nacle": 21946,
+ "##roner": 21947,
+ "BS": 21948,
+ "##nces": 21949,
+ "stimulus": 21950,
+ "transports": 21951,
+ "Sabbath": 21952,
+ "##postle": 21953,
+ "Richter": 21954,
+ "4000": 21955,
+ "##grim": 21956,
+ "##shima": 21957,
+ "##lette": 21958,
+ "deteriorated": 21959,
+ "analogous": 21960,
+ "##ratic": 21961,
+ "UHF": 21962,
+ "energies": 21963,
+ "inspiring": 21964,
+ "Yiddish": 21965,
+ "Activities": 21966,
+ "##quential": 21967,
+ "##boe": 21968,
+ "Melville": 21969,
+ "##ilton": 21970,
+ "Judd": 21971,
+ "consonants": 21972,
+ "labs": 21973,
+ "smuggling": 21974,
+ "##fari": 21975,
+ "avid": 21976,
+ "##uc": 21977,
+ "truce": 21978,
+ "undead": 21979,
+ "##raith": 21980,
+ "Mostly": 21981,
+ "bracelet": 21982,
+ "Connection": 21983,
+ "Hussain": 21984,
+ "awhile": 21985,
+ "##UC": 21986,
+ "##vention": 21987,
+ "liable": 21988,
+ "genetically": 21989,
+ "##phic": 21990,
+ "Important": 21991,
+ "Wildcats": 21992,
+ "daddy": 21993,
+ "transmit": 21994,
+ "##cas": 21995,
+ "conserved": 21996,
+ "Yesterday": 21997,
+ "##lite": 21998,
+ "Nicky": 21999,
+ "Guys": 22000,
+ "Wilder": 22001,
+ "Lay": 22002,
+ "skinned": 22003,
+ "Communists": 22004,
+ "Garfield": 22005,
+ "Nearby": 22006,
+ "organizer": 22007,
+ "Loss": 22008,
+ "crafts": 22009,
+ "walkway": 22010,
+ "Chocolate": 22011,
+ "Sundance": 22012,
+ "Synod": 22013,
+ "##enham": 22014,
+ "modify": 22015,
+ "swayed": 22016,
+ "Surface": 22017,
+ "analysts": 22018,
+ "brackets": 22019,
+ "drone": 22020,
+ "parachute": 22021,
+ "smelling": 22022,
+ "Andrés": 22023,
+ "filthy": 22024,
+ "frogs": 22025,
+ "vertically": 22026,
+ "##OK": 22027,
+ "localities": 22028,
+ "marries": 22029,
+ "AHL": 22030,
+ "35th": 22031,
+ "##pian": 22032,
+ "Palazzo": 22033,
+ "cube": 22034,
+ "dismay": 22035,
+ "relocate": 22036,
+ "##на": 22037,
+ "Hear": 22038,
+ "##digo": 22039,
+ "##oxide": 22040,
+ "prefecture": 22041,
+ "converts": 22042,
+ "hangar": 22043,
+ "##oya": 22044,
+ "##ucking": 22045,
+ "Spectrum": 22046,
+ "deepened": 22047,
+ "spoiled": 22048,
+ "Keeping": 22049,
+ "##phobic": 22050,
+ "Verona": 22051,
+ "outrage": 22052,
+ "Improvement": 22053,
+ "##UI": 22054,
+ "masterpiece": 22055,
+ "slung": 22056,
+ "Calling": 22057,
+ "chant": 22058,
+ "Haute": 22059,
+ "mediated": 22060,
+ "manipulated": 22061,
+ "affirmed": 22062,
+ "##hesis": 22063,
+ "Hangul": 22064,
+ "skies": 22065,
+ "##llan": 22066,
+ "Worcestershire": 22067,
+ "##kos": 22068,
+ "mosaic": 22069,
+ "##bage": 22070,
+ "##wned": 22071,
+ "Putnam": 22072,
+ "folder": 22073,
+ "##LM": 22074,
+ "guts": 22075,
+ "noteworthy": 22076,
+ "##rada": 22077,
+ "AJ": 22078,
+ "sculpted": 22079,
+ "##iselle": 22080,
+ "##rang": 22081,
+ "recognizable": 22082,
+ "##pent": 22083,
+ "dolls": 22084,
+ "lobbying": 22085,
+ "impatiently": 22086,
+ "Se": 22087,
+ "staple": 22088,
+ "Serb": 22089,
+ "tandem": 22090,
+ "Hiroshima": 22091,
+ "thieves": 22092,
+ "##ynx": 22093,
+ "faculties": 22094,
+ "Norte": 22095,
+ "##alle": 22096,
+ "##trusion": 22097,
+ "chords": 22098,
+ "##ylon": 22099,
+ "Gareth": 22100,
+ "##lops": 22101,
+ "##escu": 22102,
+ "FIA": 22103,
+ "Levin": 22104,
+ "auspices": 22105,
+ "groin": 22106,
+ "Hui": 22107,
+ "nun": 22108,
+ "Listed": 22109,
+ "Honourable": 22110,
+ "Larsen": 22111,
+ "rigorous": 22112,
+ "##erer": 22113,
+ "Tonga": 22114,
+ "##pment": 22115,
+ "##rave": 22116,
+ "##track": 22117,
+ "##aa": 22118,
+ "##enary": 22119,
+ "540": 22120,
+ "clone": 22121,
+ "sediment": 22122,
+ "esteem": 22123,
+ "sighted": 22124,
+ "cruelty": 22125,
+ "##boa": 22126,
+ "inverse": 22127,
+ "violating": 22128,
+ "Amtrak": 22129,
+ "Status": 22130,
+ "amalgamated": 22131,
+ "vertex": 22132,
+ "AR": 22133,
+ "harmless": 22134,
+ "Amir": 22135,
+ "mounts": 22136,
+ "Coronation": 22137,
+ "counseling": 22138,
+ "Audi": 22139,
+ "CO₂": 22140,
+ "splits": 22141,
+ "##eyer": 22142,
+ "Humans": 22143,
+ "Salmon": 22144,
+ "##have": 22145,
+ "##rado": 22146,
+ "##čić": 22147,
+ "216": 22148,
+ "takeoff": 22149,
+ "classmates": 22150,
+ "psychedelic": 22151,
+ "##gni": 22152,
+ "Gypsy": 22153,
+ "231": 22154,
+ "Anger": 22155,
+ "GAA": 22156,
+ "ME": 22157,
+ "##nist": 22158,
+ "##tals": 22159,
+ "Lissa": 22160,
+ "Odd": 22161,
+ "baptized": 22162,
+ "Fiat": 22163,
+ "fringe": 22164,
+ "##hren": 22165,
+ "179": 22166,
+ "elevators": 22167,
+ "perspectives": 22168,
+ "##TF": 22169,
+ "##ngle": 22170,
+ "Question": 22171,
+ "frontal": 22172,
+ "950": 22173,
+ "thicker": 22174,
+ "Molecular": 22175,
+ "##nological": 22176,
+ "Sixteen": 22177,
+ "Baton": 22178,
+ "Hearing": 22179,
+ "commemorative": 22180,
+ "dorm": 22181,
+ "Architectural": 22182,
+ "purity": 22183,
+ "##erse": 22184,
+ "risky": 22185,
+ "Georgie": 22186,
+ "relaxing": 22187,
+ "##ugs": 22188,
+ "downed": 22189,
+ "##rar": 22190,
+ "Slim": 22191,
+ "##phy": 22192,
+ "IUCN": 22193,
+ "##thorpe": 22194,
+ "Parkinson": 22195,
+ "217": 22196,
+ "Marley": 22197,
+ "Shipping": 22198,
+ "sweaty": 22199,
+ "Jesuits": 22200,
+ "Sindh": 22201,
+ "Janata": 22202,
+ "implying": 22203,
+ "Armenians": 22204,
+ "intercept": 22205,
+ "Ankara": 22206,
+ "commissioners": 22207,
+ "ascended": 22208,
+ "sniper": 22209,
+ "Grass": 22210,
+ "Walls": 22211,
+ "salvage": 22212,
+ "Dewey": 22213,
+ "generalized": 22214,
+ "learnt": 22215,
+ "PT": 22216,
+ "##fighter": 22217,
+ "##tech": 22218,
+ "DR": 22219,
+ "##itrus": 22220,
+ "##zza": 22221,
+ "mercenaries": 22222,
+ "slots": 22223,
+ "##burst": 22224,
+ "##finger": 22225,
+ "##nsky": 22226,
+ "Princes": 22227,
+ "Rhodesia": 22228,
+ "##munication": 22229,
+ "##strom": 22230,
+ "Fremantle": 22231,
+ "homework": 22232,
+ "ins": 22233,
+ "##Os": 22234,
+ "##hao": 22235,
+ "##uffed": 22236,
+ "Thorpe": 22237,
+ "Xiao": 22238,
+ "exquisite": 22239,
+ "firstly": 22240,
+ "liberated": 22241,
+ "technician": 22242,
+ "Oilers": 22243,
+ "Phyllis": 22244,
+ "herb": 22245,
+ "sharks": 22246,
+ "MBE": 22247,
+ "##stock": 22248,
+ "Product": 22249,
+ "banjo": 22250,
+ "##morandum": 22251,
+ "##than": 22252,
+ "Visitors": 22253,
+ "unavailable": 22254,
+ "unpublished": 22255,
+ "oxidation": 22256,
+ "Vogue": 22257,
+ "##copic": 22258,
+ "##etics": 22259,
+ "Yates": 22260,
+ "##ppard": 22261,
+ "Leiden": 22262,
+ "Trading": 22263,
+ "cottages": 22264,
+ "Principles": 22265,
+ "##Millan": 22266,
+ "##wife": 22267,
+ "##hiva": 22268,
+ "Vicar": 22269,
+ "nouns": 22270,
+ "strolled": 22271,
+ "##eorological": 22272,
+ "##eton": 22273,
+ "##science": 22274,
+ "precedent": 22275,
+ "Armand": 22276,
+ "Guido": 22277,
+ "rewards": 22278,
+ "##ilis": 22279,
+ "##tise": 22280,
+ "clipped": 22281,
+ "chick": 22282,
+ "##endra": 22283,
+ "averages": 22284,
+ "tentatively": 22285,
+ "1830s": 22286,
+ "##vos": 22287,
+ "Certainly": 22288,
+ "305": 22289,
+ "Société": 22290,
+ "Commandant": 22291,
+ "##crats": 22292,
+ "##dified": 22293,
+ "##nka": 22294,
+ "marsh": 22295,
+ "angered": 22296,
+ "ventilation": 22297,
+ "Hutton": 22298,
+ "Ritchie": 22299,
+ "##having": 22300,
+ "Eclipse": 22301,
+ "flick": 22302,
+ "motionless": 22303,
+ "Amor": 22304,
+ "Fest": 22305,
+ "Loire": 22306,
+ "lays": 22307,
+ "##icit": 22308,
+ "##sband": 22309,
+ "Guggenheim": 22310,
+ "Luck": 22311,
+ "disrupted": 22312,
+ "##ncia": 22313,
+ "Disco": 22314,
+ "##vigator": 22315,
+ "criticisms": 22316,
+ "grins": 22317,
+ "##lons": 22318,
+ "##vial": 22319,
+ "##ody": 22320,
+ "salute": 22321,
+ "Coaches": 22322,
+ "junk": 22323,
+ "saxophonist": 22324,
+ "##eology": 22325,
+ "Uprising": 22326,
+ "Diet": 22327,
+ "##marks": 22328,
+ "chronicles": 22329,
+ "robbed": 22330,
+ "##iet": 22331,
+ "##ahi": 22332,
+ "Bohemian": 22333,
+ "magician": 22334,
+ "wavelength": 22335,
+ "Kenyan": 22336,
+ "augmented": 22337,
+ "fashionable": 22338,
+ "##ogies": 22339,
+ "Luce": 22340,
+ "F1": 22341,
+ "Monmouth": 22342,
+ "##jos": 22343,
+ "##loop": 22344,
+ "enjoyment": 22345,
+ "exemption": 22346,
+ "Centers": 22347,
+ "##visor": 22348,
+ "Soundtrack": 22349,
+ "blinding": 22350,
+ "practitioner": 22351,
+ "solidarity": 22352,
+ "sacrificed": 22353,
+ "##oso": 22354,
+ "##cture": 22355,
+ "##riated": 22356,
+ "blended": 22357,
+ "Abd": 22358,
+ "Copyright": 22359,
+ "##nob": 22360,
+ "34th": 22361,
+ "##reak": 22362,
+ "Claudio": 22363,
+ "hectare": 22364,
+ "rotor": 22365,
+ "testify": 22366,
+ "##ends": 22367,
+ "##iably": 22368,
+ "##sume": 22369,
+ "landowner": 22370,
+ "##cess": 22371,
+ "##ckman": 22372,
+ "Eduard": 22373,
+ "Silesian": 22374,
+ "backseat": 22375,
+ "mutually": 22376,
+ "##abe": 22377,
+ "Mallory": 22378,
+ "bounds": 22379,
+ "Collective": 22380,
+ "Poet": 22381,
+ "Winkler": 22382,
+ "pertaining": 22383,
+ "scraped": 22384,
+ "Phelps": 22385,
+ "crane": 22386,
+ "flickering": 22387,
+ "Proto": 22388,
+ "bubbles": 22389,
+ "popularized": 22390,
+ "removes": 22391,
+ "##86": 22392,
+ "Cadillac": 22393,
+ "Warfare": 22394,
+ "audible": 22395,
+ "rites": 22396,
+ "shivering": 22397,
+ "##sist": 22398,
+ "##nst": 22399,
+ "##biotic": 22400,
+ "Mon": 22401,
+ "fascist": 22402,
+ "Bali": 22403,
+ "Kathryn": 22404,
+ "ambiguous": 22405,
+ "furiously": 22406,
+ "morale": 22407,
+ "patio": 22408,
+ "Sang": 22409,
+ "inconsistent": 22410,
+ "topology": 22411,
+ "Greens": 22412,
+ "monkeys": 22413,
+ "Köppen": 22414,
+ "189": 22415,
+ "Toy": 22416,
+ "vow": 22417,
+ "##ías": 22418,
+ "bombings": 22419,
+ "##culus": 22420,
+ "improvised": 22421,
+ "lodged": 22422,
+ "subsidiaries": 22423,
+ "garment": 22424,
+ "startling": 22425,
+ "practised": 22426,
+ "Hume": 22427,
+ "Thorn": 22428,
+ "categorized": 22429,
+ "Till": 22430,
+ "Eileen": 22431,
+ "wedge": 22432,
+ "##64": 22433,
+ "Federico": 22434,
+ "patriotic": 22435,
+ "unlock": 22436,
+ "##oshi": 22437,
+ "badminton": 22438,
+ "Compared": 22439,
+ "Vilnius": 22440,
+ "##KE": 22441,
+ "Crimean": 22442,
+ "Kemp": 22443,
+ "decks": 22444,
+ "spaced": 22445,
+ "resolutions": 22446,
+ "sighs": 22447,
+ "##mind": 22448,
+ "Imagine": 22449,
+ "Cartoon": 22450,
+ "huddled": 22451,
+ "policemen": 22452,
+ "forwards": 22453,
+ "##rouch": 22454,
+ "equals": 22455,
+ "##nter": 22456,
+ "inspected": 22457,
+ "Charley": 22458,
+ "MG": 22459,
+ "##rte": 22460,
+ "pamphlet": 22461,
+ "Arturo": 22462,
+ "dans": 22463,
+ "scarcely": 22464,
+ "##ulton": 22465,
+ "##rvin": 22466,
+ "parental": 22467,
+ "unconstitutional": 22468,
+ "watts": 22469,
+ "Susannah": 22470,
+ "Dare": 22471,
+ "##sitive": 22472,
+ "Rowland": 22473,
+ "Valle": 22474,
+ "invalid": 22475,
+ "##ué": 22476,
+ "Detachment": 22477,
+ "acronym": 22478,
+ "Yokohama": 22479,
+ "verified": 22480,
+ "##lsson": 22481,
+ "groove": 22482,
+ "Liza": 22483,
+ "clarified": 22484,
+ "compromised": 22485,
+ "265": 22486,
+ "##rgon": 22487,
+ "##orf": 22488,
+ "hesitant": 22489,
+ "Fruit": 22490,
+ "Application": 22491,
+ "Mathias": 22492,
+ "icons": 22493,
+ "##cell": 22494,
+ "Qin": 22495,
+ "interventions": 22496,
+ "##uron": 22497,
+ "punt": 22498,
+ "remnant": 22499,
+ "##rien": 22500,
+ "Ames": 22501,
+ "manifold": 22502,
+ "spines": 22503,
+ "floral": 22504,
+ "##zable": 22505,
+ "comrades": 22506,
+ "Fallen": 22507,
+ "orbits": 22508,
+ "Annals": 22509,
+ "hobby": 22510,
+ "Auditorium": 22511,
+ "implicated": 22512,
+ "researching": 22513,
+ "Pueblo": 22514,
+ "Ta": 22515,
+ "terminate": 22516,
+ "##pella": 22517,
+ "Rings": 22518,
+ "approximation": 22519,
+ "fuzzy": 22520,
+ "##ús": 22521,
+ "thriving": 22522,
+ "##ket": 22523,
+ "Conor": 22524,
+ "alarmed": 22525,
+ "etched": 22526,
+ "Cary": 22527,
+ "##rdon": 22528,
+ "Ally": 22529,
+ "##rington": 22530,
+ "Pay": 22531,
+ "mint": 22532,
+ "##hasa": 22533,
+ "##unity": 22534,
+ "##dman": 22535,
+ "##itate": 22536,
+ "Oceania": 22537,
+ "furrowed": 22538,
+ "trams": 22539,
+ "##aq": 22540,
+ "Wentworth": 22541,
+ "ventured": 22542,
+ "choreography": 22543,
+ "prototypes": 22544,
+ "Patel": 22545,
+ "mouthed": 22546,
+ "trenches": 22547,
+ "##licing": 22548,
+ "##yya": 22549,
+ "Lies": 22550,
+ "deception": 22551,
+ "##erve": 22552,
+ "##vations": 22553,
+ "Bertrand": 22554,
+ "earthquakes": 22555,
+ "##tography": 22556,
+ "Southwestern": 22557,
+ "##aja": 22558,
+ "token": 22559,
+ "Gupta": 22560,
+ "##yō": 22561,
+ "Beckett": 22562,
+ "initials": 22563,
+ "ironic": 22564,
+ "Tsar": 22565,
+ "subdued": 22566,
+ "shootout": 22567,
+ "sobbing": 22568,
+ "liar": 22569,
+ "Scandinavia": 22570,
+ "Souls": 22571,
+ "ch": 22572,
+ "therapist": 22573,
+ "trader": 22574,
+ "Regulation": 22575,
+ "Kali": 22576,
+ "busiest": 22577,
+ "##pation": 22578,
+ "32nd": 22579,
+ "Telephone": 22580,
+ "Vargas": 22581,
+ "##moky": 22582,
+ "##nose": 22583,
+ "##uge": 22584,
+ "Favorite": 22585,
+ "abducted": 22586,
+ "bonding": 22587,
+ "219": 22588,
+ "255": 22589,
+ "correction": 22590,
+ "mat": 22591,
+ "drown": 22592,
+ "fl": 22593,
+ "unbeaten": 22594,
+ "Pocket": 22595,
+ "Summers": 22596,
+ "Quite": 22597,
+ "rods": 22598,
+ "Percussion": 22599,
+ "##ndy": 22600,
+ "buzzing": 22601,
+ "cadet": 22602,
+ "Wilkes": 22603,
+ "attire": 22604,
+ "directory": 22605,
+ "utilities": 22606,
+ "naive": 22607,
+ "populous": 22608,
+ "Hendrix": 22609,
+ "##actor": 22610,
+ "disadvantage": 22611,
+ "1400": 22612,
+ "Landon": 22613,
+ "Underworld": 22614,
+ "##ense": 22615,
+ "Occasionally": 22616,
+ "mercury": 22617,
+ "Davey": 22618,
+ "Morley": 22619,
+ "spa": 22620,
+ "wrestled": 22621,
+ "##vender": 22622,
+ "eclipse": 22623,
+ "Sienna": 22624,
+ "supplemented": 22625,
+ "thou": 22626,
+ "Stream": 22627,
+ "liturgical": 22628,
+ "##gall": 22629,
+ "##berries": 22630,
+ "##piration": 22631,
+ "1769": 22632,
+ "Bucks": 22633,
+ "abandoning": 22634,
+ "##jutant": 22635,
+ "##nac": 22636,
+ "232": 22637,
+ "venom": 22638,
+ "##31": 22639,
+ "Roche": 22640,
+ "dotted": 22641,
+ "Currie": 22642,
+ "Córdoba": 22643,
+ "Milo": 22644,
+ "Sharif": 22645,
+ "divides": 22646,
+ "justification": 22647,
+ "prejudice": 22648,
+ "fortunate": 22649,
+ "##vide": 22650,
+ "##ābād": 22651,
+ "Rowe": 22652,
+ "inflammatory": 22653,
+ "##eld": 22654,
+ "avenue": 22655,
+ "Sources": 22656,
+ "##rimal": 22657,
+ "Messenger": 22658,
+ "Blanco": 22659,
+ "advocating": 22660,
+ "formulation": 22661,
+ "##pute": 22662,
+ "emphasizes": 22663,
+ "nut": 22664,
+ "Armored": 22665,
+ "##ented": 22666,
+ "nutrients": 22667,
+ "##tment": 22668,
+ "insistence": 22669,
+ "Martins": 22670,
+ "landowners": 22671,
+ "##RB": 22672,
+ "comparatively": 22673,
+ "headlines": 22674,
+ "snaps": 22675,
+ "##qing": 22676,
+ "Celebration": 22677,
+ "##mad": 22678,
+ "republican": 22679,
+ "##NE": 22680,
+ "Trace": 22681,
+ "##500": 22682,
+ "1771": 22683,
+ "proclamation": 22684,
+ "NRL": 22685,
+ "Rubin": 22686,
+ "Buzz": 22687,
+ "Weimar": 22688,
+ "##AG": 22689,
+ "199": 22690,
+ "posthumous": 22691,
+ "##ental": 22692,
+ "##deacon": 22693,
+ "Distance": 22694,
+ "intensely": 22695,
+ "overheard": 22696,
+ "Arcade": 22697,
+ "diagonal": 22698,
+ "hazard": 22699,
+ "Giving": 22700,
+ "weekdays": 22701,
+ "##ù": 22702,
+ "Verdi": 22703,
+ "actresses": 22704,
+ "##hare": 22705,
+ "Pulling": 22706,
+ "##erries": 22707,
+ "##pores": 22708,
+ "catering": 22709,
+ "shortest": 22710,
+ "##ctors": 22711,
+ "##cure": 22712,
+ "##restle": 22713,
+ "##reta": 22714,
+ "##runch": 22715,
+ "##brecht": 22716,
+ "##uddin": 22717,
+ "Moments": 22718,
+ "senate": 22719,
+ "Feng": 22720,
+ "Prescott": 22721,
+ "##thest": 22722,
+ "218": 22723,
+ "divisional": 22724,
+ "Bertie": 22725,
+ "sparse": 22726,
+ "surrounds": 22727,
+ "coupling": 22728,
+ "gravitational": 22729,
+ "werewolves": 22730,
+ "##lax": 22731,
+ "Rankings": 22732,
+ "##mated": 22733,
+ "##tries": 22734,
+ "Shia": 22735,
+ "##mart": 22736,
+ "##23": 22737,
+ "##vocative": 22738,
+ "interfaces": 22739,
+ "morphology": 22740,
+ "newscast": 22741,
+ "##bide": 22742,
+ "inputs": 22743,
+ "solicitor": 22744,
+ "Olaf": 22745,
+ "cabinets": 22746,
+ "puzzles": 22747,
+ "##tains": 22748,
+ "Unified": 22749,
+ "##firmed": 22750,
+ "WA": 22751,
+ "solemn": 22752,
+ "##opy": 22753,
+ "Tito": 22754,
+ "Jaenelle": 22755,
+ "Neolithic": 22756,
+ "horseback": 22757,
+ "##ires": 22758,
+ "pharmacy": 22759,
+ "prevalence": 22760,
+ "##lint": 22761,
+ "Swami": 22762,
+ "##bush": 22763,
+ "##tudes": 22764,
+ "Philipp": 22765,
+ "mythical": 22766,
+ "divers": 22767,
+ "Scouting": 22768,
+ "aperture": 22769,
+ "progressively": 22770,
+ "##bay": 22771,
+ "##nio": 22772,
+ "bounce": 22773,
+ "Floor": 22774,
+ "##elf": 22775,
+ "Lucan": 22776,
+ "adulthood": 22777,
+ "helm": 22778,
+ "Bluff": 22779,
+ "Passage": 22780,
+ "Salvation": 22781,
+ "lemon": 22782,
+ "napkin": 22783,
+ "scheduling": 22784,
+ "##gets": 22785,
+ "Elements": 22786,
+ "Mina": 22787,
+ "Novak": 22788,
+ "stalled": 22789,
+ "##llister": 22790,
+ "Infrastructure": 22791,
+ "##nky": 22792,
+ "##tania": 22793,
+ "##uished": 22794,
+ "Katz": 22795,
+ "Norma": 22796,
+ "sucks": 22797,
+ "trusting": 22798,
+ "1765": 22799,
+ "boilers": 22800,
+ "Accordingly": 22801,
+ "##hered": 22802,
+ "223": 22803,
+ "Crowley": 22804,
+ "##fight": 22805,
+ "##ulo": 22806,
+ "Henrietta": 22807,
+ "##hani": 22808,
+ "pounder": 22809,
+ "surprises": 22810,
+ "##chor": 22811,
+ "##glia": 22812,
+ "Dukes": 22813,
+ "##cracy": 22814,
+ "##zier": 22815,
+ "##fs": 22816,
+ "Patriot": 22817,
+ "silicon": 22818,
+ "##VP": 22819,
+ "simulcast": 22820,
+ "telegraph": 22821,
+ "Mysore": 22822,
+ "cardboard": 22823,
+ "Len": 22824,
+ "##QL": 22825,
+ "Auguste": 22826,
+ "accordion": 22827,
+ "analytical": 22828,
+ "specify": 22829,
+ "ineffective": 22830,
+ "hunched": 22831,
+ "abnormal": 22832,
+ "Transylvania": 22833,
+ "##dn": 22834,
+ "##tending": 22835,
+ "Emilia": 22836,
+ "glittering": 22837,
+ "Maddy": 22838,
+ "##wana": 22839,
+ "1762": 22840,
+ "External": 22841,
+ "Lecture": 22842,
+ "endorsement": 22843,
+ "Hernández": 22844,
+ "Anaheim": 22845,
+ "Ware": 22846,
+ "offences": 22847,
+ "##phorus": 22848,
+ "Plantation": 22849,
+ "popping": 22850,
+ "Bonaparte": 22851,
+ "disgusting": 22852,
+ "neared": 22853,
+ "##notes": 22854,
+ "Identity": 22855,
+ "heroin": 22856,
+ "nicely": 22857,
+ "##raverse": 22858,
+ "apron": 22859,
+ "congestion": 22860,
+ "##PR": 22861,
+ "padded": 22862,
+ "##fts": 22863,
+ "invaders": 22864,
+ "##came": 22865,
+ "freshly": 22866,
+ "Halle": 22867,
+ "endowed": 22868,
+ "fracture": 22869,
+ "ROM": 22870,
+ "##max": 22871,
+ "sediments": 22872,
+ "diffusion": 22873,
+ "dryly": 22874,
+ "##tara": 22875,
+ "Tam": 22876,
+ "Draw": 22877,
+ "Spin": 22878,
+ "Talon": 22879,
+ "Anthropology": 22880,
+ "##lify": 22881,
+ "nausea": 22882,
+ "##shirt": 22883,
+ "insert": 22884,
+ "Fresno": 22885,
+ "capitalist": 22886,
+ "indefinitely": 22887,
+ "apples": 22888,
+ "Gift": 22889,
+ "scooped": 22890,
+ "60s": 22891,
+ "Cooperative": 22892,
+ "mistakenly": 22893,
+ "##lover": 22894,
+ "murmur": 22895,
+ "##iger": 22896,
+ "Equipment": 22897,
+ "abusive": 22898,
+ "orphanage": 22899,
+ "##9th": 22900,
+ "##lterweight": 22901,
+ "##unda": 22902,
+ "Baird": 22903,
+ "ant": 22904,
+ "saloon": 22905,
+ "33rd": 22906,
+ "Chesapeake": 22907,
+ "##chair": 22908,
+ "##sound": 22909,
+ "##tend": 22910,
+ "chaotic": 22911,
+ "pornography": 22912,
+ "brace": 22913,
+ "##aret": 22914,
+ "heiress": 22915,
+ "SSR": 22916,
+ "resentment": 22917,
+ "Arbor": 22918,
+ "headmaster": 22919,
+ "##uren": 22920,
+ "unlimited": 22921,
+ "##with": 22922,
+ "##jn": 22923,
+ "Bram": 22924,
+ "Ely": 22925,
+ "Pokémon": 22926,
+ "pivotal": 22927,
+ "##guous": 22928,
+ "Database": 22929,
+ "Marta": 22930,
+ "Shine": 22931,
+ "stumbling": 22932,
+ "##ovsky": 22933,
+ "##skin": 22934,
+ "Henley": 22935,
+ "Polk": 22936,
+ "functioned": 22937,
+ "##layer": 22938,
+ "##pas": 22939,
+ "##udd": 22940,
+ "##MX": 22941,
+ "blackness": 22942,
+ "cadets": 22943,
+ "feral": 22944,
+ "Damian": 22945,
+ "##actions": 22946,
+ "2D": 22947,
+ "##yla": 22948,
+ "Apocalypse": 22949,
+ "##aic": 22950,
+ "inactivated": 22951,
+ "##china": 22952,
+ "##kovic": 22953,
+ "##bres": 22954,
+ "destroys": 22955,
+ "nap": 22956,
+ "Macy": 22957,
+ "sums": 22958,
+ "Madhya": 22959,
+ "Wisdom": 22960,
+ "rejects": 22961,
+ "##amel": 22962,
+ "60th": 22963,
+ "Cho": 22964,
+ "bandwidth": 22965,
+ "##sons": 22966,
+ "##obbing": 22967,
+ "##orama": 22968,
+ "Mutual": 22969,
+ "shafts": 22970,
+ "##estone": 22971,
+ "##rsen": 22972,
+ "accord": 22973,
+ "replaces": 22974,
+ "waterfront": 22975,
+ "##gonal": 22976,
+ "##rida": 22977,
+ "convictions": 22978,
+ "##ays": 22979,
+ "calmed": 22980,
+ "suppliers": 22981,
+ "Cummings": 22982,
+ "GMA": 22983,
+ "fearful": 22984,
+ "Scientist": 22985,
+ "Sinai": 22986,
+ "examines": 22987,
+ "experimented": 22988,
+ "Netflix": 22989,
+ "Enforcement": 22990,
+ "Scarlett": 22991,
+ "##lasia": 22992,
+ "Healthcare": 22993,
+ "##onte": 22994,
+ "Dude": 22995,
+ "inverted": 22996,
+ "##36": 22997,
+ "##regation": 22998,
+ "##lidae": 22999,
+ "Munro": 23000,
+ "##angay": 23001,
+ "Airbus": 23002,
+ "overlapping": 23003,
+ "Drivers": 23004,
+ "lawsuits": 23005,
+ "bodily": 23006,
+ "##udder": 23007,
+ "Wanda": 23008,
+ "Effects": 23009,
+ "Fathers": 23010,
+ "##finery": 23011,
+ "##islav": 23012,
+ "Ridley": 23013,
+ "observatory": 23014,
+ "pod": 23015,
+ "##utrition": 23016,
+ "Electricity": 23017,
+ "landslide": 23018,
+ "##mable": 23019,
+ "##zoic": 23020,
+ "##imator": 23021,
+ "##uration": 23022,
+ "Estates": 23023,
+ "sleepy": 23024,
+ "Nickelodeon": 23025,
+ "steaming": 23026,
+ "irony": 23027,
+ "schedules": 23028,
+ "snack": 23029,
+ "spikes": 23030,
+ "Hmm": 23031,
+ "##nesia": 23032,
+ "##bella": 23033,
+ "##hibit": 23034,
+ "Greenville": 23035,
+ "plucked": 23036,
+ "Harald": 23037,
+ "##ono": 23038,
+ "Gamma": 23039,
+ "infringement": 23040,
+ "roaring": 23041,
+ "deposition": 23042,
+ "##pol": 23043,
+ "##orum": 23044,
+ "660": 23045,
+ "seminal": 23046,
+ "passports": 23047,
+ "engagements": 23048,
+ "Akbar": 23049,
+ "rotated": 23050,
+ "##bina": 23051,
+ "##gart": 23052,
+ "Hartley": 23053,
+ "##lown": 23054,
+ "##truct": 23055,
+ "uttered": 23056,
+ "traumatic": 23057,
+ "Dex": 23058,
+ "##ôme": 23059,
+ "Holloway": 23060,
+ "MV": 23061,
+ "apartheid": 23062,
+ "##nee": 23063,
+ "Counter": 23064,
+ "Colton": 23065,
+ "OR": 23066,
+ "245": 23067,
+ "Spaniards": 23068,
+ "Regency": 23069,
+ "Schedule": 23070,
+ "scratching": 23071,
+ "squads": 23072,
+ "verify": 23073,
+ "##alk": 23074,
+ "keyboardist": 23075,
+ "rotten": 23076,
+ "Forestry": 23077,
+ "aids": 23078,
+ "commemorating": 23079,
+ "##yed": 23080,
+ "##érie": 23081,
+ "Sting": 23082,
+ "##elly": 23083,
+ "Dai": 23084,
+ "##fers": 23085,
+ "##berley": 23086,
+ "##ducted": 23087,
+ "Melvin": 23088,
+ "cannabis": 23089,
+ "glider": 23090,
+ "##enbach": 23091,
+ "##rban": 23092,
+ "Costello": 23093,
+ "Skating": 23094,
+ "cartoonist": 23095,
+ "AN": 23096,
+ "audit": 23097,
+ "##pectator": 23098,
+ "distributing": 23099,
+ "226": 23100,
+ "312": 23101,
+ "interpreter": 23102,
+ "header": 23103,
+ "Alternatively": 23104,
+ "##ases": 23105,
+ "smug": 23106,
+ "##kumar": 23107,
+ "cabins": 23108,
+ "remastered": 23109,
+ "Connolly": 23110,
+ "Kelsey": 23111,
+ "LED": 23112,
+ "tentative": 23113,
+ "Check": 23114,
+ "Sichuan": 23115,
+ "shaved": 23116,
+ "##42": 23117,
+ "Gerhard": 23118,
+ "Harvest": 23119,
+ "inward": 23120,
+ "##rque": 23121,
+ "Hopefully": 23122,
+ "hem": 23123,
+ "##34": 23124,
+ "Typical": 23125,
+ "binds": 23126,
+ "wrath": 23127,
+ "Woodstock": 23128,
+ "forcibly": 23129,
+ "Fergus": 23130,
+ "##charged": 23131,
+ "##tured": 23132,
+ "prepares": 23133,
+ "amenities": 23134,
+ "penetration": 23135,
+ "##ghan": 23136,
+ "coarse": 23137,
+ "##oned": 23138,
+ "enthusiasts": 23139,
+ "##av": 23140,
+ "##twined": 23141,
+ "fielded": 23142,
+ "##cky": 23143,
+ "Kiel": 23144,
+ "##obia": 23145,
+ "470": 23146,
+ "beers": 23147,
+ "tremble": 23148,
+ "youths": 23149,
+ "attendees": 23150,
+ "##cademies": 23151,
+ "##sex": 23152,
+ "Macon": 23153,
+ "communism": 23154,
+ "dir": 23155,
+ "##abi": 23156,
+ "Lennox": 23157,
+ "Wen": 23158,
+ "differentiate": 23159,
+ "jewel": 23160,
+ "##SO": 23161,
+ "activate": 23162,
+ "assert": 23163,
+ "laden": 23164,
+ "unto": 23165,
+ "Gillespie": 23166,
+ "Guillermo": 23167,
+ "accumulation": 23168,
+ "##GM": 23169,
+ "NGO": 23170,
+ "Rosenberg": 23171,
+ "calculating": 23172,
+ "drastically": 23173,
+ "##omorphic": 23174,
+ "peeled": 23175,
+ "Liège": 23176,
+ "insurgents": 23177,
+ "outdoors": 23178,
+ "##enia": 23179,
+ "Aspen": 23180,
+ "Sep": 23181,
+ "awakened": 23182,
+ "##eye": 23183,
+ "Consul": 23184,
+ "Maiden": 23185,
+ "insanity": 23186,
+ "##brian": 23187,
+ "furnace": 23188,
+ "Colours": 23189,
+ "distributions": 23190,
+ "longitudinal": 23191,
+ "syllables": 23192,
+ "##scent": 23193,
+ "Martian": 23194,
+ "accountant": 23195,
+ "Atkins": 23196,
+ "husbands": 23197,
+ "sewage": 23198,
+ "zur": 23199,
+ "collaborate": 23200,
+ "highlighting": 23201,
+ "##rites": 23202,
+ "##PI": 23203,
+ "colonization": 23204,
+ "nearer": 23205,
+ "##XT": 23206,
+ "dunes": 23207,
+ "positioning": 23208,
+ "Ku": 23209,
+ "multitude": 23210,
+ "luxurious": 23211,
+ "Volvo": 23212,
+ "linguistics": 23213,
+ "plotting": 23214,
+ "squared": 23215,
+ "##inder": 23216,
+ "outstretched": 23217,
+ "##uds": 23218,
+ "Fuji": 23219,
+ "ji": 23220,
+ "##feit": 23221,
+ "##ahu": 23222,
+ "##loat": 23223,
+ "##gado": 23224,
+ "##luster": 23225,
+ "##oku": 23226,
+ "América": 23227,
+ "##iza": 23228,
+ "Residents": 23229,
+ "vine": 23230,
+ "Pieces": 23231,
+ "DD": 23232,
+ "Vampires": 23233,
+ "##ová": 23234,
+ "smoked": 23235,
+ "harshly": 23236,
+ "spreads": 23237,
+ "##turn": 23238,
+ "##zhi": 23239,
+ "betray": 23240,
+ "electors": 23241,
+ "##settled": 23242,
+ "Considering": 23243,
+ "exploits": 23244,
+ "stamped": 23245,
+ "Dusty": 23246,
+ "enraged": 23247,
+ "Nairobi": 23248,
+ "##38": 23249,
+ "intervened": 23250,
+ "##luck": 23251,
+ "orchestras": 23252,
+ "##lda": 23253,
+ "Hereford": 23254,
+ "Jarvis": 23255,
+ "calf": 23256,
+ "##itzer": 23257,
+ "##CH": 23258,
+ "salesman": 23259,
+ "Lovers": 23260,
+ "cigar": 23261,
+ "Angelica": 23262,
+ "doomed": 23263,
+ "heroine": 23264,
+ "##tible": 23265,
+ "Sanford": 23266,
+ "offenders": 23267,
+ "##ulously": 23268,
+ "articulated": 23269,
+ "##oam": 23270,
+ "Emanuel": 23271,
+ "Gardiner": 23272,
+ "Edna": 23273,
+ "Shu": 23274,
+ "gigantic": 23275,
+ "##stable": 23276,
+ "Tallinn": 23277,
+ "coasts": 23278,
+ "Maker": 23279,
+ "ale": 23280,
+ "stalking": 23281,
+ "##oga": 23282,
+ "##smus": 23283,
+ "lucrative": 23284,
+ "southbound": 23285,
+ "##changing": 23286,
+ "Reg": 23287,
+ "##lants": 23288,
+ "Schleswig": 23289,
+ "discount": 23290,
+ "grouping": 23291,
+ "physiological": 23292,
+ "##OH": 23293,
+ "##sun": 23294,
+ "Galen": 23295,
+ "assurance": 23296,
+ "reconcile": 23297,
+ "rib": 23298,
+ "scarlet": 23299,
+ "Thatcher": 23300,
+ "anarchist": 23301,
+ "##oom": 23302,
+ "Turnpike": 23303,
+ "##ceding": 23304,
+ "cocktail": 23305,
+ "Sweeney": 23306,
+ "Allegheny": 23307,
+ "concessions": 23308,
+ "oppression": 23309,
+ "reassuring": 23310,
+ "##poli": 23311,
+ "##ticus": 23312,
+ "##TR": 23313,
+ "##VI": 23314,
+ "##uca": 23315,
+ "##zione": 23316,
+ "directional": 23317,
+ "strikeouts": 23318,
+ "Beneath": 23319,
+ "Couldn": 23320,
+ "Kabul": 23321,
+ "##national": 23322,
+ "hydroelectric": 23323,
+ "##jit": 23324,
+ "Desire": 23325,
+ "##riot": 23326,
+ "enhancing": 23327,
+ "northbound": 23328,
+ "##PO": 23329,
+ "Ok": 23330,
+ "Routledge": 23331,
+ "volatile": 23332,
+ "Bernardo": 23333,
+ "Python": 23334,
+ "333": 23335,
+ "ample": 23336,
+ "chestnut": 23337,
+ "automobiles": 23338,
+ "##innamon": 23339,
+ "##care": 23340,
+ "##hering": 23341,
+ "BWF": 23342,
+ "salaries": 23343,
+ "Turbo": 23344,
+ "acquisitions": 23345,
+ "##stituting": 23346,
+ "strengths": 23347,
+ "pilgrims": 23348,
+ "Ponce": 23349,
+ "Pig": 23350,
+ "Actors": 23351,
+ "Beard": 23352,
+ "sanitation": 23353,
+ "##RD": 23354,
+ "##mett": 23355,
+ "Telecommunications": 23356,
+ "worms": 23357,
+ "##idas": 23358,
+ "Juno": 23359,
+ "Larson": 23360,
+ "Ventura": 23361,
+ "Northeastern": 23362,
+ "weighs": 23363,
+ "Houghton": 23364,
+ "collaborating": 23365,
+ "lottery": 23366,
+ "##rano": 23367,
+ "Wonderland": 23368,
+ "gigs": 23369,
+ "##lmer": 23370,
+ "##zano": 23371,
+ "##edd": 23372,
+ "##nife": 23373,
+ "mixtape": 23374,
+ "predominant": 23375,
+ "tripped": 23376,
+ "##ruly": 23377,
+ "Alexei": 23378,
+ "investing": 23379,
+ "Belgarath": 23380,
+ "Brasil": 23381,
+ "hiss": 23382,
+ "##crat": 23383,
+ "##xham": 23384,
+ "Côte": 23385,
+ "560": 23386,
+ "kilometer": 23387,
+ "##cological": 23388,
+ "analyzing": 23389,
+ "##As": 23390,
+ "engined": 23391,
+ "listener": 23392,
+ "##cakes": 23393,
+ "negotiation": 23394,
+ "##hisky": 23395,
+ "Santana": 23396,
+ "##lemma": 23397,
+ "IAAF": 23398,
+ "Seneca": 23399,
+ "skeletal": 23400,
+ "Covenant": 23401,
+ "Steiner": 23402,
+ "##lev": 23403,
+ "##uen": 23404,
+ "Neptune": 23405,
+ "retention": 23406,
+ "##upon": 23407,
+ "Closing": 23408,
+ "Czechoslovak": 23409,
+ "chalk": 23410,
+ "Navarre": 23411,
+ "NZ": 23412,
+ "##IG": 23413,
+ "##hop": 23414,
+ "##oly": 23415,
+ "##quatorial": 23416,
+ "##sad": 23417,
+ "Brewery": 23418,
+ "Conflict": 23419,
+ "Them": 23420,
+ "renew": 23421,
+ "turrets": 23422,
+ "disagree": 23423,
+ "Petra": 23424,
+ "Slave": 23425,
+ "##reole": 23426,
+ "adjustment": 23427,
+ "##dela": 23428,
+ "##regard": 23429,
+ "##sner": 23430,
+ "framing": 23431,
+ "stature": 23432,
+ "##rca": 23433,
+ "##sies": 23434,
+ "##46": 23435,
+ "##mata": 23436,
+ "Logic": 23437,
+ "inadvertently": 23438,
+ "naturalist": 23439,
+ "spheres": 23440,
+ "towering": 23441,
+ "heightened": 23442,
+ "Dodd": 23443,
+ "rink": 23444,
+ "##fle": 23445,
+ "Keyboards": 23446,
+ "bulb": 23447,
+ "diver": 23448,
+ "ul": 23449,
+ "##tsk": 23450,
+ "Exodus": 23451,
+ "Deacon": 23452,
+ "España": 23453,
+ "Canadiens": 23454,
+ "oblique": 23455,
+ "thud": 23456,
+ "reigned": 23457,
+ "rug": 23458,
+ "Whitman": 23459,
+ "Dash": 23460,
+ "##iens": 23461,
+ "Haifa": 23462,
+ "pets": 23463,
+ "##arland": 23464,
+ "manually": 23465,
+ "dart": 23466,
+ "##bial": 23467,
+ "Sven": 23468,
+ "textiles": 23469,
+ "subgroup": 23470,
+ "Napier": 23471,
+ "graffiti": 23472,
+ "revolver": 23473,
+ "humming": 23474,
+ "Babu": 23475,
+ "protector": 23476,
+ "typed": 23477,
+ "Provinces": 23478,
+ "Sparta": 23479,
+ "Wills": 23480,
+ "subjective": 23481,
+ "##rella": 23482,
+ "temptation": 23483,
+ "##liest": 23484,
+ "FL": 23485,
+ "Sadie": 23486,
+ "manifest": 23487,
+ "Guangdong": 23488,
+ "Transfer": 23489,
+ "entertain": 23490,
+ "eve": 23491,
+ "recipes": 23492,
+ "##33": 23493,
+ "Benedictine": 23494,
+ "retailer": 23495,
+ "##dence": 23496,
+ "establishes": 23497,
+ "##cluded": 23498,
+ "##rked": 23499,
+ "Ursula": 23500,
+ "##ltz": 23501,
+ "##lars": 23502,
+ "##rena": 23503,
+ "qualifiers": 23504,
+ "##curement": 23505,
+ "colt": 23506,
+ "depictions": 23507,
+ "##oit": 23508,
+ "Spiritual": 23509,
+ "differentiation": 23510,
+ "staffed": 23511,
+ "transitional": 23512,
+ "##lew": 23513,
+ "1761": 23514,
+ "fatalities": 23515,
+ "##oan": 23516,
+ "Bayern": 23517,
+ "Northamptonshire": 23518,
+ "Weeks": 23519,
+ "##CU": 23520,
+ "Fife": 23521,
+ "capacities": 23522,
+ "hoarse": 23523,
+ "##latt": 23524,
+ "##ة": 23525,
+ "evidenced": 23526,
+ "##HD": 23527,
+ "##ographer": 23528,
+ "assessing": 23529,
+ "evolve": 23530,
+ "hints": 23531,
+ "42nd": 23532,
+ "streaked": 23533,
+ "##lve": 23534,
+ "Yahoo": 23535,
+ "##estive": 23536,
+ "##rned": 23537,
+ "##zas": 23538,
+ "baggage": 23539,
+ "Elected": 23540,
+ "secrecy": 23541,
+ "##champ": 23542,
+ "Character": 23543,
+ "Pen": 23544,
+ "Decca": 23545,
+ "cape": 23546,
+ "Bernardino": 23547,
+ "vapor": 23548,
+ "Dolly": 23549,
+ "counselor": 23550,
+ "##isers": 23551,
+ "Benin": 23552,
+ "##khar": 23553,
+ "##CR": 23554,
+ "notch": 23555,
+ "##thus": 23556,
+ "##racy": 23557,
+ "bounty": 23558,
+ "lend": 23559,
+ "grassland": 23560,
+ "##chtenstein": 23561,
+ "##dating": 23562,
+ "pseudo": 23563,
+ "golfer": 23564,
+ "simplest": 23565,
+ "##ceive": 23566,
+ "Lucivar": 23567,
+ "Triumph": 23568,
+ "dinosaur": 23569,
+ "dinosaurs": 23570,
+ "##šić": 23571,
+ "Seahawks": 23572,
+ "##nco": 23573,
+ "resorts": 23574,
+ "reelected": 23575,
+ "1766": 23576,
+ "reproduce": 23577,
+ "universally": 23578,
+ "##OA": 23579,
+ "ER": 23580,
+ "tendencies": 23581,
+ "Consolidated": 23582,
+ "Massey": 23583,
+ "Tasmanian": 23584,
+ "reckless": 23585,
+ "##icz": 23586,
+ "##ricks": 23587,
+ "1755": 23588,
+ "questionable": 23589,
+ "Audience": 23590,
+ "##lates": 23591,
+ "preseason": 23592,
+ "Quran": 23593,
+ "trivial": 23594,
+ "Haitian": 23595,
+ "Freeway": 23596,
+ "dialed": 23597,
+ "Appointed": 23598,
+ "Heard": 23599,
+ "ecosystems": 23600,
+ "##bula": 23601,
+ "hormones": 23602,
+ "Carbon": 23603,
+ "Rd": 23604,
+ "##arney": 23605,
+ "##working": 23606,
+ "Christoph": 23607,
+ "presiding": 23608,
+ "pu": 23609,
+ "##athy": 23610,
+ "Morrow": 23611,
+ "Dar": 23612,
+ "ensures": 23613,
+ "posing": 23614,
+ "remedy": 23615,
+ "EA": 23616,
+ "disclosed": 23617,
+ "##hui": 23618,
+ "##rten": 23619,
+ "rumours": 23620,
+ "surveying": 23621,
+ "##ficiency": 23622,
+ "Aziz": 23623,
+ "Jewel": 23624,
+ "Plays": 23625,
+ "##smatic": 23626,
+ "Bernhard": 23627,
+ "Christi": 23628,
+ "##eanut": 23629,
+ "##friend": 23630,
+ "jailed": 23631,
+ "##dr": 23632,
+ "govern": 23633,
+ "neighbour": 23634,
+ "butler": 23635,
+ "Acheron": 23636,
+ "murdering": 23637,
+ "oils": 23638,
+ "mac": 23639,
+ "Editorial": 23640,
+ "detectives": 23641,
+ "bolts": 23642,
+ "##ulon": 23643,
+ "Guitars": 23644,
+ "malaria": 23645,
+ "36th": 23646,
+ "Pembroke": 23647,
+ "Opened": 23648,
+ "##hium": 23649,
+ "harmonic": 23650,
+ "serum": 23651,
+ "##sio": 23652,
+ "Franks": 23653,
+ "fingernails": 23654,
+ "##gli": 23655,
+ "culturally": 23656,
+ "evolving": 23657,
+ "scalp": 23658,
+ "VP": 23659,
+ "deploy": 23660,
+ "uploaded": 23661,
+ "mater": 23662,
+ "##evo": 23663,
+ "Jammu": 23664,
+ "Spa": 23665,
+ "##icker": 23666,
+ "flirting": 23667,
+ "##cursions": 23668,
+ "Heidi": 23669,
+ "Majority": 23670,
+ "sprawled": 23671,
+ "##alytic": 23672,
+ "Zheng": 23673,
+ "bunker": 23674,
+ "##lena": 23675,
+ "ST": 23676,
+ "##tile": 23677,
+ "Jiang": 23678,
+ "ceilings": 23679,
+ "##ently": 23680,
+ "##ols": 23681,
+ "Recovery": 23682,
+ "dire": 23683,
+ "##good": 23684,
+ "Manson": 23685,
+ "Honestly": 23686,
+ "Montréal": 23687,
+ "1764": 23688,
+ "227": 23689,
+ "quota": 23690,
+ "Lakshmi": 23691,
+ "incentive": 23692,
+ "Accounting": 23693,
+ "##cilla": 23694,
+ "Eureka": 23695,
+ "Reaper": 23696,
+ "buzzed": 23697,
+ "##uh": 23698,
+ "courtroom": 23699,
+ "dub": 23700,
+ "##mberg": 23701,
+ "KC": 23702,
+ "Gong": 23703,
+ "Theodor": 23704,
+ "Académie": 23705,
+ "NPR": 23706,
+ "criticizing": 23707,
+ "protesting": 23708,
+ "##pired": 23709,
+ "##yric": 23710,
+ "abuses": 23711,
+ "fisheries": 23712,
+ "##minated": 23713,
+ "1767": 23714,
+ "yd": 23715,
+ "Gemini": 23716,
+ "Subcommittee": 23717,
+ "##fuse": 23718,
+ "Duff": 23719,
+ "Wasn": 23720,
+ "Wight": 23721,
+ "cleaner": 23722,
+ "##tite": 23723,
+ "planetary": 23724,
+ "Survivor": 23725,
+ "Zionist": 23726,
+ "mounds": 23727,
+ "##rary": 23728,
+ "landfall": 23729,
+ "disruption": 23730,
+ "yielding": 23731,
+ "##yana": 23732,
+ "bids": 23733,
+ "unidentified": 23734,
+ "Garry": 23735,
+ "Ellison": 23736,
+ "Elmer": 23737,
+ "Fishing": 23738,
+ "Hayward": 23739,
+ "demos": 23740,
+ "modelling": 23741,
+ "##anche": 23742,
+ "##stick": 23743,
+ "caressed": 23744,
+ "entertained": 23745,
+ "##hesion": 23746,
+ "piers": 23747,
+ "Crimea": 23748,
+ "##mass": 23749,
+ "WHO": 23750,
+ "boulder": 23751,
+ "trunks": 23752,
+ "1640": 23753,
+ "Biennale": 23754,
+ "Palestinians": 23755,
+ "Pursuit": 23756,
+ "##udes": 23757,
+ "Dora": 23758,
+ "contender": 23759,
+ "##dridge": 23760,
+ "Nanjing": 23761,
+ "##ezer": 23762,
+ "##former": 23763,
+ "##ibel": 23764,
+ "Whole": 23765,
+ "proliferation": 23766,
+ "##tide": 23767,
+ "##weiler": 23768,
+ "fuels": 23769,
+ "predictions": 23770,
+ "##ente": 23771,
+ "##onium": 23772,
+ "Filming": 23773,
+ "absorbing": 23774,
+ "Ramón": 23775,
+ "strangled": 23776,
+ "conveyed": 23777,
+ "inhabit": 23778,
+ "prostitutes": 23779,
+ "recession": 23780,
+ "bonded": 23781,
+ "clinched": 23782,
+ "##eak": 23783,
+ "##iji": 23784,
+ "##edar": 23785,
+ "Pleasure": 23786,
+ "Rite": 23787,
+ "Christy": 23788,
+ "Therapy": 23789,
+ "sarcasm": 23790,
+ "##collegiate": 23791,
+ "hilt": 23792,
+ "probation": 23793,
+ "Sarawak": 23794,
+ "coefficients": 23795,
+ "underworld": 23796,
+ "biodiversity": 23797,
+ "SBS": 23798,
+ "groom": 23799,
+ "brewing": 23800,
+ "dungeon": 23801,
+ "##claiming": 23802,
+ "Hari": 23803,
+ "turnover": 23804,
+ "##ntina": 23805,
+ "##omer": 23806,
+ "##opped": 23807,
+ "orthodox": 23808,
+ "styling": 23809,
+ "##tars": 23810,
+ "##ulata": 23811,
+ "priced": 23812,
+ "Marjorie": 23813,
+ "##eley": 23814,
+ "##abar": 23815,
+ "Yong": 23816,
+ "##tically": 23817,
+ "Crambidae": 23818,
+ "Hernandez": 23819,
+ "##ego": 23820,
+ "##rricular": 23821,
+ "##ark": 23822,
+ "##lamour": 23823,
+ "##llin": 23824,
+ "##augh": 23825,
+ "##tens": 23826,
+ "Advancement": 23827,
+ "Loyola": 23828,
+ "##4th": 23829,
+ "##hh": 23830,
+ "goin": 23831,
+ "marshes": 23832,
+ "Sardinia": 23833,
+ "##ša": 23834,
+ "Ljubljana": 23835,
+ "Singing": 23836,
+ "suspiciously": 23837,
+ "##hesive": 23838,
+ "Félix": 23839,
+ "Regarding": 23840,
+ "flap": 23841,
+ "stimulation": 23842,
+ "##raught": 23843,
+ "Apr": 23844,
+ "Yin": 23845,
+ "gaping": 23846,
+ "tighten": 23847,
+ "skier": 23848,
+ "##itas": 23849,
+ "##lad": 23850,
+ "##rani": 23851,
+ "264": 23852,
+ "Ashes": 23853,
+ "Olson": 23854,
+ "Problems": 23855,
+ "Tabitha": 23856,
+ "##rading": 23857,
+ "balancing": 23858,
+ "sunrise": 23859,
+ "##ease": 23860,
+ "##iture": 23861,
+ "##ritic": 23862,
+ "Fringe": 23863,
+ "##iciency": 23864,
+ "Inspired": 23865,
+ "Linnaeus": 23866,
+ "PBA": 23867,
+ "disapproval": 23868,
+ "##kles": 23869,
+ "##rka": 23870,
+ "##tails": 23871,
+ "##urger": 23872,
+ "Disaster": 23873,
+ "Laboratories": 23874,
+ "apps": 23875,
+ "paradise": 23876,
+ "Aero": 23877,
+ "Came": 23878,
+ "sneaking": 23879,
+ "Gee": 23880,
+ "Beacon": 23881,
+ "ODI": 23882,
+ "commodity": 23883,
+ "Ellington": 23884,
+ "graphical": 23885,
+ "Gretchen": 23886,
+ "spire": 23887,
+ "##skaya": 23888,
+ "##trine": 23889,
+ "RTÉ": 23890,
+ "efficacy": 23891,
+ "plc": 23892,
+ "tribunal": 23893,
+ "##ytic": 23894,
+ "downhill": 23895,
+ "flu": 23896,
+ "medications": 23897,
+ "##kaya": 23898,
+ "widen": 23899,
+ "Sunrise": 23900,
+ "##nous": 23901,
+ "distinguishing": 23902,
+ "pawn": 23903,
+ "##BO": 23904,
+ "##irn": 23905,
+ "##ssing": 23906,
+ "##ν": 23907,
+ "Easton": 23908,
+ "##vila": 23909,
+ "Rhineland": 23910,
+ "##aque": 23911,
+ "defect": 23912,
+ "##saurus": 23913,
+ "Goose": 23914,
+ "Ju": 23915,
+ "##classified": 23916,
+ "Middlesbrough": 23917,
+ "shaping": 23918,
+ "preached": 23919,
+ "1759": 23920,
+ "##erland": 23921,
+ "Ein": 23922,
+ "Hailey": 23923,
+ "musicals": 23924,
+ "##altered": 23925,
+ "Galileo": 23926,
+ "Hilda": 23927,
+ "Fighters": 23928,
+ "Lac": 23929,
+ "##ometric": 23930,
+ "295": 23931,
+ "Leafs": 23932,
+ "Milano": 23933,
+ "##lta": 23934,
+ "##VD": 23935,
+ "##ivist": 23936,
+ "penetrated": 23937,
+ "Mask": 23938,
+ "Orchard": 23939,
+ "plaintiff": 23940,
+ "##icorn": 23941,
+ "Yvonne": 23942,
+ "##fred": 23943,
+ "outfielder": 23944,
+ "peek": 23945,
+ "Collier": 23946,
+ "Caracas": 23947,
+ "repealed": 23948,
+ "Bois": 23949,
+ "dell": 23950,
+ "restrict": 23951,
+ "Dolores": 23952,
+ "Hadley": 23953,
+ "peacefully": 23954,
+ "##LL": 23955,
+ "condom": 23956,
+ "Granny": 23957,
+ "Orders": 23958,
+ "sabotage": 23959,
+ "##toon": 23960,
+ "##rings": 23961,
+ "compass": 23962,
+ "marshal": 23963,
+ "gears": 23964,
+ "brigadier": 23965,
+ "dye": 23966,
+ "Yunnan": 23967,
+ "communicating": 23968,
+ "donate": 23969,
+ "emerald": 23970,
+ "vitamin": 23971,
+ "administer": 23972,
+ "Fulham": 23973,
+ "##classical": 23974,
+ "##llas": 23975,
+ "Buckinghamshire": 23976,
+ "Held": 23977,
+ "layered": 23978,
+ "disclosure": 23979,
+ "Akira": 23980,
+ "programmer": 23981,
+ "shrimp": 23982,
+ "Crusade": 23983,
+ "##ximal": 23984,
+ "Luzon": 23985,
+ "bakery": 23986,
+ "##cute": 23987,
+ "Garth": 23988,
+ "Citadel": 23989,
+ "uniquely": 23990,
+ "Curling": 23991,
+ "info": 23992,
+ "mum": 23993,
+ "Para": 23994,
+ "##ști": 23995,
+ "sleek": 23996,
+ "##ione": 23997,
+ "hey": 23998,
+ "Lantern": 23999,
+ "mesh": 24000,
+ "##lacing": 24001,
+ "##lizzard": 24002,
+ "##gade": 24003,
+ "prosecuted": 24004,
+ "Alba": 24005,
+ "Gilles": 24006,
+ "greedy": 24007,
+ "twists": 24008,
+ "##ogged": 24009,
+ "Viper": 24010,
+ "##kata": 24011,
+ "Appearances": 24012,
+ "Skyla": 24013,
+ "hymns": 24014,
+ "##pelled": 24015,
+ "curving": 24016,
+ "predictable": 24017,
+ "Grave": 24018,
+ "Watford": 24019,
+ "##dford": 24020,
+ "##liptic": 24021,
+ "##vary": 24022,
+ "Westwood": 24023,
+ "fluids": 24024,
+ "Models": 24025,
+ "statutes": 24026,
+ "##ynamite": 24027,
+ "1740": 24028,
+ "##culate": 24029,
+ "Framework": 24030,
+ "Johanna": 24031,
+ "##gression": 24032,
+ "Vuelta": 24033,
+ "imp": 24034,
+ "##otion": 24035,
+ "##raga": 24036,
+ "##thouse": 24037,
+ "Ciudad": 24038,
+ "festivities": 24039,
+ "##love": 24040,
+ "Beyoncé": 24041,
+ "italics": 24042,
+ "##vance": 24043,
+ "DB": 24044,
+ "##haman": 24045,
+ "outs": 24046,
+ "Singers": 24047,
+ "##ueva": 24048,
+ "##urning": 24049,
+ "##51": 24050,
+ "##ntiary": 24051,
+ "##mobile": 24052,
+ "285": 24053,
+ "Mimi": 24054,
+ "emeritus": 24055,
+ "nesting": 24056,
+ "Keeper": 24057,
+ "Ways": 24058,
+ "##onal": 24059,
+ "##oux": 24060,
+ "Edmond": 24061,
+ "MMA": 24062,
+ "##bark": 24063,
+ "##oop": 24064,
+ "Hampson": 24065,
+ "##ñez": 24066,
+ "##rets": 24067,
+ "Gladstone": 24068,
+ "wreckage": 24069,
+ "Pont": 24070,
+ "Playboy": 24071,
+ "reluctance": 24072,
+ "##ná": 24073,
+ "apprenticeship": 24074,
+ "preferring": 24075,
+ "Value": 24076,
+ "originate": 24077,
+ "##wei": 24078,
+ "##olio": 24079,
+ "Alexia": 24080,
+ "##rog": 24081,
+ "Parachute": 24082,
+ "jammed": 24083,
+ "stud": 24084,
+ "Eton": 24085,
+ "vols": 24086,
+ "##ganized": 24087,
+ "1745": 24088,
+ "straining": 24089,
+ "creep": 24090,
+ "indicators": 24091,
+ "##mán": 24092,
+ "humiliation": 24093,
+ "hinted": 24094,
+ "alma": 24095,
+ "tanker": 24096,
+ "##egation": 24097,
+ "Haynes": 24098,
+ "Penang": 24099,
+ "amazement": 24100,
+ "branched": 24101,
+ "rumble": 24102,
+ "##ddington": 24103,
+ "archaeologists": 24104,
+ "paranoid": 24105,
+ "expenditure": 24106,
+ "Absolutely": 24107,
+ "Musicians": 24108,
+ "banished": 24109,
+ "##fining": 24110,
+ "baptism": 24111,
+ "Joker": 24112,
+ "Persons": 24113,
+ "hemisphere": 24114,
+ "##tieth": 24115,
+ "##ück": 24116,
+ "flock": 24117,
+ "##xing": 24118,
+ "lbs": 24119,
+ "Kung": 24120,
+ "crab": 24121,
+ "##dak": 24122,
+ "##tinent": 24123,
+ "Regulations": 24124,
+ "barrage": 24125,
+ "parcel": 24126,
+ "##ós": 24127,
+ "Tanaka": 24128,
+ "##rsa": 24129,
+ "Natalia": 24130,
+ "Voyage": 24131,
+ "flaws": 24132,
+ "stepfather": 24133,
+ "##aven": 24134,
+ "##eological": 24135,
+ "Botanical": 24136,
+ "Minsk": 24137,
+ "##ckers": 24138,
+ "Cinderella": 24139,
+ "Feast": 24140,
+ "Loving": 24141,
+ "Previous": 24142,
+ "Shark": 24143,
+ "##took": 24144,
+ "barrister": 24145,
+ "collaborators": 24146,
+ "##nnes": 24147,
+ "Croydon": 24148,
+ "Graeme": 24149,
+ "Juniors": 24150,
+ "##7th": 24151,
+ "##formation": 24152,
+ "##ulos": 24153,
+ "##ák": 24154,
+ "£2": 24155,
+ "##hwa": 24156,
+ "##rove": 24157,
+ "##ș": 24158,
+ "Whig": 24159,
+ "demeanor": 24160,
+ "Otago": 24161,
+ "##TH": 24162,
+ "##ooster": 24163,
+ "Faber": 24164,
+ "instructors": 24165,
+ "##ahl": 24166,
+ "##bha": 24167,
+ "emptied": 24168,
+ "##schen": 24169,
+ "saga": 24170,
+ "##lora": 24171,
+ "exploding": 24172,
+ "##rges": 24173,
+ "Crusaders": 24174,
+ "##caster": 24175,
+ "##uations": 24176,
+ "streaks": 24177,
+ "CBN": 24178,
+ "bows": 24179,
+ "insights": 24180,
+ "ka": 24181,
+ "1650": 24182,
+ "diversion": 24183,
+ "LSU": 24184,
+ "Wingspan": 24185,
+ "##liva": 24186,
+ "Response": 24187,
+ "sanity": 24188,
+ "Producers": 24189,
+ "imitation": 24190,
+ "##fine": 24191,
+ "Lange": 24192,
+ "Spokane": 24193,
+ "splash": 24194,
+ "weed": 24195,
+ "Siberian": 24196,
+ "magnet": 24197,
+ "##rocodile": 24198,
+ "capitals": 24199,
+ "##rgus": 24200,
+ "swelled": 24201,
+ "Rani": 24202,
+ "Bells": 24203,
+ "Silesia": 24204,
+ "arithmetic": 24205,
+ "rumor": 24206,
+ "##hampton": 24207,
+ "favors": 24208,
+ "Weird": 24209,
+ "marketplace": 24210,
+ "##orm": 24211,
+ "tsunami": 24212,
+ "unpredictable": 24213,
+ "##citation": 24214,
+ "##ferno": 24215,
+ "Tradition": 24216,
+ "postwar": 24217,
+ "stench": 24218,
+ "succeeds": 24219,
+ "##roup": 24220,
+ "Anya": 24221,
+ "Users": 24222,
+ "oversized": 24223,
+ "totaling": 24224,
+ "pouch": 24225,
+ "##nat": 24226,
+ "Tripoli": 24227,
+ "leverage": 24228,
+ "satin": 24229,
+ "##cline": 24230,
+ "Bathurst": 24231,
+ "Lund": 24232,
+ "Niall": 24233,
+ "thereof": 24234,
+ "##quid": 24235,
+ "Bangor": 24236,
+ "barge": 24237,
+ "Animated": 24238,
+ "##53": 24239,
+ "##alan": 24240,
+ "Ballard": 24241,
+ "utilizes": 24242,
+ "Done": 24243,
+ "ballistic": 24244,
+ "NDP": 24245,
+ "gatherings": 24246,
+ "##elin": 24247,
+ "##vening": 24248,
+ "Rockets": 24249,
+ "Sabrina": 24250,
+ "Tamara": 24251,
+ "Tribal": 24252,
+ "WTA": 24253,
+ "##citing": 24254,
+ "blinded": 24255,
+ "flux": 24256,
+ "Khalid": 24257,
+ "Una": 24258,
+ "prescription": 24259,
+ "##jee": 24260,
+ "Parents": 24261,
+ "##otics": 24262,
+ "##food": 24263,
+ "Silicon": 24264,
+ "cured": 24265,
+ "electro": 24266,
+ "perpendicular": 24267,
+ "intimacy": 24268,
+ "##rified": 24269,
+ "Lots": 24270,
+ "##ceiving": 24271,
+ "##powder": 24272,
+ "incentives": 24273,
+ "McKenna": 24274,
+ "##arma": 24275,
+ "##ounced": 24276,
+ "##rinkled": 24277,
+ "Alzheimer": 24278,
+ "##tarian": 24279,
+ "262": 24280,
+ "Seas": 24281,
+ "##cam": 24282,
+ "Novi": 24283,
+ "##hout": 24284,
+ "##morphic": 24285,
+ "##hazar": 24286,
+ "##hul": 24287,
+ "##nington": 24288,
+ "Huron": 24289,
+ "Bahadur": 24290,
+ "Pirate": 24291,
+ "pursed": 24292,
+ "Griffiths": 24293,
+ "indicted": 24294,
+ "swap": 24295,
+ "refrain": 24296,
+ "##mulating": 24297,
+ "Lal": 24298,
+ "stomped": 24299,
+ "##Pad": 24300,
+ "##mamoto": 24301,
+ "Reef": 24302,
+ "disposed": 24303,
+ "plastered": 24304,
+ "weeping": 24305,
+ "##rato": 24306,
+ "Minas": 24307,
+ "hourly": 24308,
+ "tumors": 24309,
+ "##ruising": 24310,
+ "Lyle": 24311,
+ "##yper": 24312,
+ "##sol": 24313,
+ "Odisha": 24314,
+ "credibility": 24315,
+ "##Dowell": 24316,
+ "Braun": 24317,
+ "Graphic": 24318,
+ "lurched": 24319,
+ "muster": 24320,
+ "##nex": 24321,
+ "##ührer": 24322,
+ "##connected": 24323,
+ "##iek": 24324,
+ "##ruba": 24325,
+ "Carthage": 24326,
+ "Peck": 24327,
+ "maple": 24328,
+ "bursting": 24329,
+ "##lava": 24330,
+ "Enrico": 24331,
+ "rite": 24332,
+ "##jak": 24333,
+ "Moment": 24334,
+ "##skar": 24335,
+ "Styx": 24336,
+ "poking": 24337,
+ "Spartan": 24338,
+ "##urney": 24339,
+ "Hepburn": 24340,
+ "Mart": 24341,
+ "Titanic": 24342,
+ "newsletter": 24343,
+ "waits": 24344,
+ "Mecklenburg": 24345,
+ "agitated": 24346,
+ "eats": 24347,
+ "##dious": 24348,
+ "Chow": 24349,
+ "matrices": 24350,
+ "Maud": 24351,
+ "##sexual": 24352,
+ "sermon": 24353,
+ "234": 24354,
+ "##sible": 24355,
+ "##lung": 24356,
+ "Qi": 24357,
+ "cemeteries": 24358,
+ "mined": 24359,
+ "sprinter": 24360,
+ "##ckett": 24361,
+ "coward": 24362,
+ "##gable": 24363,
+ "##hell": 24364,
+ "##thin": 24365,
+ "##FB": 24366,
+ "Contact": 24367,
+ "##hay": 24368,
+ "rainforest": 24369,
+ "238": 24370,
+ "Hemisphere": 24371,
+ "boasts": 24372,
+ "##nders": 24373,
+ "##verance": 24374,
+ "##kat": 24375,
+ "Convent": 24376,
+ "Dunedin": 24377,
+ "Lecturer": 24378,
+ "lyricist": 24379,
+ "##bject": 24380,
+ "Iberian": 24381,
+ "comune": 24382,
+ "##pphire": 24383,
+ "chunk": 24384,
+ "##boo": 24385,
+ "thrusting": 24386,
+ "fore": 24387,
+ "informing": 24388,
+ "pistols": 24389,
+ "echoes": 24390,
+ "Tier": 24391,
+ "battleships": 24392,
+ "substitution": 24393,
+ "##belt": 24394,
+ "moniker": 24395,
+ "##charya": 24396,
+ "##lland": 24397,
+ "Thoroughbred": 24398,
+ "38th": 24399,
+ "##01": 24400,
+ "##tah": 24401,
+ "parting": 24402,
+ "tongues": 24403,
+ "Cale": 24404,
+ "##seau": 24405,
+ "Unionist": 24406,
+ "modular": 24407,
+ "celebrates": 24408,
+ "preview": 24409,
+ "steamed": 24410,
+ "Bismarck": 24411,
+ "302": 24412,
+ "737": 24413,
+ "vamp": 24414,
+ "##finity": 24415,
+ "##nbridge": 24416,
+ "weaknesses": 24417,
+ "husky": 24418,
+ "##berman": 24419,
+ "absently": 24420,
+ "##icide": 24421,
+ "Craven": 24422,
+ "tailored": 24423,
+ "Tokugawa": 24424,
+ "VIP": 24425,
+ "syntax": 24426,
+ "Kazan": 24427,
+ "captives": 24428,
+ "doses": 24429,
+ "filtered": 24430,
+ "overview": 24431,
+ "Cleopatra": 24432,
+ "Conversely": 24433,
+ "stallion": 24434,
+ "Burger": 24435,
+ "Suez": 24436,
+ "Raoul": 24437,
+ "th": 24438,
+ "##reaves": 24439,
+ "Dickson": 24440,
+ "Nell": 24441,
+ "Rate": 24442,
+ "anal": 24443,
+ "colder": 24444,
+ "##sław": 24445,
+ "Arm": 24446,
+ "Semitic": 24447,
+ "##green": 24448,
+ "reflective": 24449,
+ "1100": 24450,
+ "episcopal": 24451,
+ "journeys": 24452,
+ "##ours": 24453,
+ "##pository": 24454,
+ "##dering": 24455,
+ "residue": 24456,
+ "Gunn": 24457,
+ "##27": 24458,
+ "##ntial": 24459,
+ "##crates": 24460,
+ "##zig": 24461,
+ "Astros": 24462,
+ "Renee": 24463,
+ "Emerald": 24464,
+ "##vili": 24465,
+ "connectivity": 24466,
+ "undrafted": 24467,
+ "Sampson": 24468,
+ "treasures": 24469,
+ "##kura": 24470,
+ "##theon": 24471,
+ "##vern": 24472,
+ "Destroyer": 24473,
+ "##iable": 24474,
+ "##ener": 24475,
+ "Frederic": 24476,
+ "briefcase": 24477,
+ "confinement": 24478,
+ "Bree": 24479,
+ "##WD": 24480,
+ "Athena": 24481,
+ "233": 24482,
+ "Padres": 24483,
+ "Thom": 24484,
+ "speeding": 24485,
+ "##hali": 24486,
+ "Dental": 24487,
+ "ducks": 24488,
+ "Putin": 24489,
+ "##rcle": 24490,
+ "##lou": 24491,
+ "Asylum": 24492,
+ "##usk": 24493,
+ "dusk": 24494,
+ "pasture": 24495,
+ "Institutes": 24496,
+ "ONE": 24497,
+ "jack": 24498,
+ "##named": 24499,
+ "diplomacy": 24500,
+ "Intercontinental": 24501,
+ "Leagues": 24502,
+ "Towns": 24503,
+ "comedic": 24504,
+ "premature": 24505,
+ "##edic": 24506,
+ "##mona": 24507,
+ "##ories": 24508,
+ "trimmed": 24509,
+ "Charge": 24510,
+ "Cream": 24511,
+ "guarantees": 24512,
+ "Dmitry": 24513,
+ "splashed": 24514,
+ "Philosophical": 24515,
+ "tramway": 24516,
+ "##cape": 24517,
+ "Maynard": 24518,
+ "predatory": 24519,
+ "redundant": 24520,
+ "##gratory": 24521,
+ "##wry": 24522,
+ "sobs": 24523,
+ "Burgundy": 24524,
+ "edible": 24525,
+ "outfits": 24526,
+ "Handel": 24527,
+ "dazed": 24528,
+ "dangerously": 24529,
+ "idle": 24530,
+ "Operational": 24531,
+ "organizes": 24532,
+ "##sional": 24533,
+ "blackish": 24534,
+ "broker": 24535,
+ "weddings": 24536,
+ "##halt": 24537,
+ "Becca": 24538,
+ "McGee": 24539,
+ "##gman": 24540,
+ "protagonists": 24541,
+ "##pelling": 24542,
+ "Keynes": 24543,
+ "aux": 24544,
+ "stumble": 24545,
+ "##ordination": 24546,
+ "Nokia": 24547,
+ "reel": 24548,
+ "sexes": 24549,
+ "##woods": 24550,
+ "##pheric": 24551,
+ "##quished": 24552,
+ "##voc": 24553,
+ "##oir": 24554,
+ "##pathian": 24555,
+ "##ptus": 24556,
+ "##sma": 24557,
+ "##tating": 24558,
+ "##ê": 24559,
+ "fulfilling": 24560,
+ "sheath": 24561,
+ "##ayne": 24562,
+ "Mei": 24563,
+ "Ordinary": 24564,
+ "Collin": 24565,
+ "Sharpe": 24566,
+ "grasses": 24567,
+ "interdisciplinary": 24568,
+ "##OX": 24569,
+ "Background": 24570,
+ "##ignment": 24571,
+ "Assault": 24572,
+ "transforms": 24573,
+ "Hamas": 24574,
+ "Serge": 24575,
+ "ratios": 24576,
+ "##sik": 24577,
+ "swaying": 24578,
+ "##rcia": 24579,
+ "Rosen": 24580,
+ "##gant": 24581,
+ "##versible": 24582,
+ "cinematographer": 24583,
+ "curly": 24584,
+ "penny": 24585,
+ "Kamal": 24586,
+ "Mellon": 24587,
+ "Sailor": 24588,
+ "Spence": 24589,
+ "phased": 24590,
+ "Brewers": 24591,
+ "amassed": 24592,
+ "Societies": 24593,
+ "##ropriations": 24594,
+ "##buted": 24595,
+ "mythological": 24596,
+ "##SN": 24597,
+ "##byss": 24598,
+ "##ired": 24599,
+ "Sovereign": 24600,
+ "preface": 24601,
+ "Parry": 24602,
+ "##ife": 24603,
+ "altitudes": 24604,
+ "crossings": 24605,
+ "##28": 24606,
+ "Crewe": 24607,
+ "southernmost": 24608,
+ "taut": 24609,
+ "McKinley": 24610,
+ "##owa": 24611,
+ "##tore": 24612,
+ "254": 24613,
+ "##ckney": 24614,
+ "compiling": 24615,
+ "Shelton": 24616,
+ "##hiko": 24617,
+ "228": 24618,
+ "Poll": 24619,
+ "Shepard": 24620,
+ "Labs": 24621,
+ "Pace": 24622,
+ "Carlson": 24623,
+ "grasping": 24624,
+ "##ов": 24625,
+ "Delaney": 24626,
+ "Winning": 24627,
+ "robotic": 24628,
+ "intentional": 24629,
+ "shattering": 24630,
+ "##boarding": 24631,
+ "##git": 24632,
+ "##grade": 24633,
+ "Editions": 24634,
+ "Reserves": 24635,
+ "ignorant": 24636,
+ "proposing": 24637,
+ "##hanna": 24638,
+ "cutter": 24639,
+ "Mongols": 24640,
+ "NW": 24641,
+ "##eux": 24642,
+ "Codex": 24643,
+ "Cristina": 24644,
+ "Daughters": 24645,
+ "Rees": 24646,
+ "forecast": 24647,
+ "##hita": 24648,
+ "NGOs": 24649,
+ "Stations": 24650,
+ "Beaux": 24651,
+ "Erwin": 24652,
+ "##jected": 24653,
+ "##EX": 24654,
+ "##trom": 24655,
+ "Schumacher": 24656,
+ "##hrill": 24657,
+ "##rophe": 24658,
+ "Maharaja": 24659,
+ "Oricon": 24660,
+ "##sul": 24661,
+ "##dynamic": 24662,
+ "##fighting": 24663,
+ "Ce": 24664,
+ "Ingrid": 24665,
+ "rumbled": 24666,
+ "Prospect": 24667,
+ "stairwell": 24668,
+ "Barnard": 24669,
+ "applause": 24670,
+ "complementary": 24671,
+ "##uba": 24672,
+ "grunt": 24673,
+ "##mented": 24674,
+ "Bloc": 24675,
+ "Carleton": 24676,
+ "loft": 24677,
+ "noisy": 24678,
+ "##hey": 24679,
+ "490": 24680,
+ "contrasted": 24681,
+ "##inator": 24682,
+ "##rief": 24683,
+ "##centric": 24684,
+ "##fica": 24685,
+ "Cantonese": 24686,
+ "Blanc": 24687,
+ "Lausanne": 24688,
+ "License": 24689,
+ "artifact": 24690,
+ "##ddin": 24691,
+ "rot": 24692,
+ "Amongst": 24693,
+ "Prakash": 24694,
+ "RF": 24695,
+ "##topia": 24696,
+ "milestone": 24697,
+ "##vard": 24698,
+ "Winters": 24699,
+ "Mead": 24700,
+ "churchyard": 24701,
+ "Lulu": 24702,
+ "estuary": 24703,
+ "##ind": 24704,
+ "Cha": 24705,
+ "Infinity": 24706,
+ "Meadow": 24707,
+ "subsidies": 24708,
+ "##valent": 24709,
+ "CONCACAF": 24710,
+ "Ching": 24711,
+ "medicinal": 24712,
+ "navigate": 24713,
+ "Carver": 24714,
+ "Twice": 24715,
+ "abdominal": 24716,
+ "regulating": 24717,
+ "RB": 24718,
+ "toilets": 24719,
+ "Brewer": 24720,
+ "weakening": 24721,
+ "ambushed": 24722,
+ "##aut": 24723,
+ "##vignon": 24724,
+ "Lansing": 24725,
+ "unacceptable": 24726,
+ "reliance": 24727,
+ "stabbing": 24728,
+ "##mpo": 24729,
+ "##naire": 24730,
+ "Interview": 24731,
+ "##ested": 24732,
+ "##imed": 24733,
+ "bearings": 24734,
+ "##lts": 24735,
+ "Rashid": 24736,
+ "##iation": 24737,
+ "authenticity": 24738,
+ "vigorous": 24739,
+ "##frey": 24740,
+ "##uel": 24741,
+ "biologist": 24742,
+ "NFC": 24743,
+ "##rmaid": 24744,
+ "##wash": 24745,
+ "Makes": 24746,
+ "##aunt": 24747,
+ "##steries": 24748,
+ "withdrawing": 24749,
+ "##qa": 24750,
+ "Buccaneers": 24751,
+ "bleed": 24752,
+ "inclination": 24753,
+ "stain": 24754,
+ "##ilo": 24755,
+ "##ppel": 24756,
+ "Torre": 24757,
+ "privileged": 24758,
+ "cereal": 24759,
+ "trailers": 24760,
+ "alumnus": 24761,
+ "neon": 24762,
+ "Cochrane": 24763,
+ "Mariana": 24764,
+ "caress": 24765,
+ "##47": 24766,
+ "##ients": 24767,
+ "experimentation": 24768,
+ "Window": 24769,
+ "convict": 24770,
+ "signaled": 24771,
+ "##YP": 24772,
+ "rower": 24773,
+ "Pharmacy": 24774,
+ "interacting": 24775,
+ "241": 24776,
+ "Strings": 24777,
+ "dominating": 24778,
+ "kinase": 24779,
+ "Dinamo": 24780,
+ "Wire": 24781,
+ "pains": 24782,
+ "sensations": 24783,
+ "##suse": 24784,
+ "Twenty20": 24785,
+ "##39": 24786,
+ "spotlight": 24787,
+ "##hend": 24788,
+ "elemental": 24789,
+ "##pura": 24790,
+ "Jameson": 24791,
+ "Swindon": 24792,
+ "honoring": 24793,
+ "pained": 24794,
+ "##ediatric": 24795,
+ "##lux": 24796,
+ "Psychological": 24797,
+ "assemblies": 24798,
+ "ingredient": 24799,
+ "Martial": 24800,
+ "Penguins": 24801,
+ "beverage": 24802,
+ "Monitor": 24803,
+ "mysteries": 24804,
+ "##ION": 24805,
+ "emigration": 24806,
+ "mused": 24807,
+ "##sique": 24808,
+ "crore": 24809,
+ "AMC": 24810,
+ "Funding": 24811,
+ "Chinatown": 24812,
+ "Establishment": 24813,
+ "Finalist": 24814,
+ "enjoyable": 24815,
+ "1756": 24816,
+ "##mada": 24817,
+ "##rams": 24818,
+ "NO": 24819,
+ "newborn": 24820,
+ "CS": 24821,
+ "comprehend": 24822,
+ "Invisible": 24823,
+ "Siemens": 24824,
+ "##acon": 24825,
+ "246": 24826,
+ "contraction": 24827,
+ "##volving": 24828,
+ "##moration": 24829,
+ "##rok": 24830,
+ "montane": 24831,
+ "##ntation": 24832,
+ "Galloway": 24833,
+ "##llow": 24834,
+ "Verity": 24835,
+ "directorial": 24836,
+ "pearl": 24837,
+ "Leaning": 24838,
+ "##rase": 24839,
+ "Fernandez": 24840,
+ "swallowing": 24841,
+ "Automatic": 24842,
+ "Madness": 24843,
+ "haunting": 24844,
+ "paddle": 24845,
+ "##UE": 24846,
+ "##rrows": 24847,
+ "##vies": 24848,
+ "##zuki": 24849,
+ "##bolt": 24850,
+ "##iber": 24851,
+ "Fender": 24852,
+ "emails": 24853,
+ "paste": 24854,
+ "##lancing": 24855,
+ "hind": 24856,
+ "homestead": 24857,
+ "hopeless": 24858,
+ "##dles": 24859,
+ "Rockies": 24860,
+ "garlic": 24861,
+ "fatty": 24862,
+ "shrieked": 24863,
+ "##ismic": 24864,
+ "Gillian": 24865,
+ "Inquiry": 24866,
+ "Schultz": 24867,
+ "XML": 24868,
+ "##cius": 24869,
+ "##uld": 24870,
+ "Domesday": 24871,
+ "grenades": 24872,
+ "northernmost": 24873,
+ "##igi": 24874,
+ "Tbilisi": 24875,
+ "optimistic": 24876,
+ "##poon": 24877,
+ "Refuge": 24878,
+ "stacks": 24879,
+ "Bose": 24880,
+ "smash": 24881,
+ "surreal": 24882,
+ "Nah": 24883,
+ "Straits": 24884,
+ "Conquest": 24885,
+ "##roo": 24886,
+ "##weet": 24887,
+ "##kell": 24888,
+ "Gladys": 24889,
+ "CH": 24890,
+ "##lim": 24891,
+ "##vitation": 24892,
+ "Doctorate": 24893,
+ "NRHP": 24894,
+ "knocks": 24895,
+ "Bey": 24896,
+ "Romano": 24897,
+ "##pile": 24898,
+ "242": 24899,
+ "Diamonds": 24900,
+ "strides": 24901,
+ "eclectic": 24902,
+ "Betsy": 24903,
+ "clade": 24904,
+ "##hady": 24905,
+ "##leashed": 24906,
+ "dissolve": 24907,
+ "moss": 24908,
+ "Suburban": 24909,
+ "silvery": 24910,
+ "##bria": 24911,
+ "tally": 24912,
+ "turtles": 24913,
+ "##uctive": 24914,
+ "finely": 24915,
+ "industrialist": 24916,
+ "##nary": 24917,
+ "Ernesto": 24918,
+ "oz": 24919,
+ "pact": 24920,
+ "loneliness": 24921,
+ "##hov": 24922,
+ "Tomb": 24923,
+ "multinational": 24924,
+ "risked": 24925,
+ "Layne": 24926,
+ "USL": 24927,
+ "ne": 24928,
+ "##quiries": 24929,
+ "Ad": 24930,
+ "Message": 24931,
+ "Kamen": 24932,
+ "Kristen": 24933,
+ "reefs": 24934,
+ "implements": 24935,
+ "##itative": 24936,
+ "educators": 24937,
+ "garments": 24938,
+ "gunshot": 24939,
+ "##essed": 24940,
+ "##rve": 24941,
+ "Montevideo": 24942,
+ "vigorously": 24943,
+ "Stamford": 24944,
+ "assemble": 24945,
+ "packaged": 24946,
+ "##same": 24947,
+ "état": 24948,
+ "Viva": 24949,
+ "paragraph": 24950,
+ "##eter": 24951,
+ "##wire": 24952,
+ "Stick": 24953,
+ "Navajo": 24954,
+ "MCA": 24955,
+ "##pressing": 24956,
+ "ensembles": 24957,
+ "ABA": 24958,
+ "##zor": 24959,
+ "##llus": 24960,
+ "Partner": 24961,
+ "raked": 24962,
+ "##BI": 24963,
+ "Iona": 24964,
+ "thump": 24965,
+ "Celeste": 24966,
+ "Kiran": 24967,
+ "##iscovered": 24968,
+ "##rith": 24969,
+ "inflammation": 24970,
+ "##arel": 24971,
+ "Features": 24972,
+ "loosened": 24973,
+ "##yclic": 24974,
+ "Deluxe": 24975,
+ "Speak": 24976,
+ "economical": 24977,
+ "Frankenstein": 24978,
+ "Picasso": 24979,
+ "showcased": 24980,
+ "##zad": 24981,
+ "##eira": 24982,
+ "##planes": 24983,
+ "##linear": 24984,
+ "##overs": 24985,
+ "monsoon": 24986,
+ "prosecutors": 24987,
+ "slack": 24988,
+ "Horses": 24989,
+ "##urers": 24990,
+ "Angry": 24991,
+ "coughing": 24992,
+ "##truder": 24993,
+ "Questions": 24994,
+ "##tō": 24995,
+ "##zak": 24996,
+ "challenger": 24997,
+ "clocks": 24998,
+ "##ieving": 24999,
+ "Newmarket": 25000,
+ "##acle": 25001,
+ "cursing": 25002,
+ "stimuli": 25003,
+ "##mming": 25004,
+ "##qualified": 25005,
+ "slapping": 25006,
+ "##vasive": 25007,
+ "narration": 25008,
+ "##kini": 25009,
+ "Advertising": 25010,
+ "CSI": 25011,
+ "alliances": 25012,
+ "mixes": 25013,
+ "##yes": 25014,
+ "covert": 25015,
+ "amalgamation": 25016,
+ "reproduced": 25017,
+ "##ardt": 25018,
+ "##gis": 25019,
+ "1648": 25020,
+ "id": 25021,
+ "Annette": 25022,
+ "Boots": 25023,
+ "Champagne": 25024,
+ "Brest": 25025,
+ "Daryl": 25026,
+ "##emon": 25027,
+ "##jou": 25028,
+ "##llers": 25029,
+ "Mean": 25030,
+ "adaptive": 25031,
+ "technicians": 25032,
+ "##pair": 25033,
+ "##usal": 25034,
+ "Yoga": 25035,
+ "fronts": 25036,
+ "leaping": 25037,
+ "Jul": 25038,
+ "harvesting": 25039,
+ "keel": 25040,
+ "##44": 25041,
+ "petitioned": 25042,
+ "##lved": 25043,
+ "yells": 25044,
+ "Endowment": 25045,
+ "proponent": 25046,
+ "##spur": 25047,
+ "##tised": 25048,
+ "##zal": 25049,
+ "Homes": 25050,
+ "Includes": 25051,
+ "##ifer": 25052,
+ "##oodoo": 25053,
+ "##rvette": 25054,
+ "awarding": 25055,
+ "mirrored": 25056,
+ "ransom": 25057,
+ "Flute": 25058,
+ "outlook": 25059,
+ "##ganj": 25060,
+ "DVDs": 25061,
+ "Sufi": 25062,
+ "frontman": 25063,
+ "Goddard": 25064,
+ "barren": 25065,
+ "##astic": 25066,
+ "Suicide": 25067,
+ "hillside": 25068,
+ "Harlow": 25069,
+ "Lau": 25070,
+ "notions": 25071,
+ "Amnesty": 25072,
+ "Homestead": 25073,
+ "##irt": 25074,
+ "GE": 25075,
+ "hooded": 25076,
+ "umpire": 25077,
+ "mustered": 25078,
+ "Catch": 25079,
+ "Masonic": 25080,
+ "##erd": 25081,
+ "Dynamics": 25082,
+ "Equity": 25083,
+ "Oro": 25084,
+ "Charts": 25085,
+ "Mussolini": 25086,
+ "populace": 25087,
+ "muted": 25088,
+ "accompaniment": 25089,
+ "##lour": 25090,
+ "##ndes": 25091,
+ "ignited": 25092,
+ "##iferous": 25093,
+ "##laced": 25094,
+ "##atch": 25095,
+ "anguish": 25096,
+ "registry": 25097,
+ "##tub": 25098,
+ "##hards": 25099,
+ "##neer": 25100,
+ "251": 25101,
+ "Hooker": 25102,
+ "uncomfortably": 25103,
+ "##6th": 25104,
+ "##ivers": 25105,
+ "Catalina": 25106,
+ "MiG": 25107,
+ "giggling": 25108,
+ "1754": 25109,
+ "Dietrich": 25110,
+ "Kaladin": 25111,
+ "pricing": 25112,
+ "##quence": 25113,
+ "Sabah": 25114,
+ "##lving": 25115,
+ "##nical": 25116,
+ "Gettysburg": 25117,
+ "Vita": 25118,
+ "Telecom": 25119,
+ "Worst": 25120,
+ "Palais": 25121,
+ "Pentagon": 25122,
+ "##brand": 25123,
+ "##chichte": 25124,
+ "Graf": 25125,
+ "unnatural": 25126,
+ "1715": 25127,
+ "bio": 25128,
+ "##26": 25129,
+ "Radcliffe": 25130,
+ "##utt": 25131,
+ "chatting": 25132,
+ "spices": 25133,
+ "##aus": 25134,
+ "untouched": 25135,
+ "##eper": 25136,
+ "Doll": 25137,
+ "turkey": 25138,
+ "Syndicate": 25139,
+ "##rlene": 25140,
+ "##JP": 25141,
+ "##roots": 25142,
+ "Como": 25143,
+ "clashed": 25144,
+ "modernization": 25145,
+ "1757": 25146,
+ "fantasies": 25147,
+ "##iating": 25148,
+ "dissipated": 25149,
+ "Sicilian": 25150,
+ "inspect": 25151,
+ "sensible": 25152,
+ "reputed": 25153,
+ "##final": 25154,
+ "Milford": 25155,
+ "poised": 25156,
+ "RC": 25157,
+ "metabolic": 25158,
+ "Tobacco": 25159,
+ "Mecca": 25160,
+ "optimization": 25161,
+ "##heat": 25162,
+ "lobe": 25163,
+ "rabbits": 25164,
+ "NAS": 25165,
+ "geologist": 25166,
+ "##liner": 25167,
+ "Kilda": 25168,
+ "carpenter": 25169,
+ "nationalists": 25170,
+ "##brae": 25171,
+ "summarized": 25172,
+ "##venge": 25173,
+ "Designer": 25174,
+ "misleading": 25175,
+ "beamed": 25176,
+ "##meyer": 25177,
+ "Matrix": 25178,
+ "excuses": 25179,
+ "##aines": 25180,
+ "##biology": 25181,
+ "401": 25182,
+ "Moose": 25183,
+ "drafting": 25184,
+ "Sai": 25185,
+ "##ggle": 25186,
+ "Comprehensive": 25187,
+ "dripped": 25188,
+ "skate": 25189,
+ "##WI": 25190,
+ "##enan": 25191,
+ "##ruk": 25192,
+ "narrower": 25193,
+ "outgoing": 25194,
+ "##enter": 25195,
+ "##nounce": 25196,
+ "overseen": 25197,
+ "##structure": 25198,
+ "travellers": 25199,
+ "banging": 25200,
+ "scarred": 25201,
+ "##thing": 25202,
+ "##arra": 25203,
+ "Ebert": 25204,
+ "Sometime": 25205,
+ "##nated": 25206,
+ "BAFTA": 25207,
+ "Hurricanes": 25208,
+ "configurations": 25209,
+ "##MLL": 25210,
+ "immortality": 25211,
+ "##heus": 25212,
+ "gothic": 25213,
+ "##mpest": 25214,
+ "clergyman": 25215,
+ "viewpoint": 25216,
+ "Maxim": 25217,
+ "Instituto": 25218,
+ "emitted": 25219,
+ "quantitative": 25220,
+ "1689": 25221,
+ "Consortium": 25222,
+ "##rsk": 25223,
+ "Meat": 25224,
+ "Tao": 25225,
+ "swimmers": 25226,
+ "Shaking": 25227,
+ "Terence": 25228,
+ "mainline": 25229,
+ "##linity": 25230,
+ "Quantum": 25231,
+ "##rogate": 25232,
+ "Nair": 25233,
+ "banquet": 25234,
+ "39th": 25235,
+ "reprised": 25236,
+ "lagoon": 25237,
+ "subdivisions": 25238,
+ "synonymous": 25239,
+ "incurred": 25240,
+ "password": 25241,
+ "sprung": 25242,
+ "##vere": 25243,
+ "Credits": 25244,
+ "Petersen": 25245,
+ "Faces": 25246,
+ "##vu": 25247,
+ "statesman": 25248,
+ "Zombie": 25249,
+ "gesturing": 25250,
+ "##going": 25251,
+ "Sergey": 25252,
+ "dormant": 25253,
+ "possessive": 25254,
+ "totals": 25255,
+ "southward": 25256,
+ "Ángel": 25257,
+ "##odies": 25258,
+ "HM": 25259,
+ "Mariano": 25260,
+ "Ramirez": 25261,
+ "Wicked": 25262,
+ "impressions": 25263,
+ "##Net": 25264,
+ "##cap": 25265,
+ "##ème": 25266,
+ "Transformers": 25267,
+ "Poker": 25268,
+ "RIAA": 25269,
+ "Redesignated": 25270,
+ "##chuk": 25271,
+ "Harcourt": 25272,
+ "Peña": 25273,
+ "spacious": 25274,
+ "tinged": 25275,
+ "alternatively": 25276,
+ "narrowing": 25277,
+ "Brigham": 25278,
+ "authorization": 25279,
+ "Membership": 25280,
+ "Zeppelin": 25281,
+ "##amed": 25282,
+ "Handball": 25283,
+ "steer": 25284,
+ "##orium": 25285,
+ "##rnal": 25286,
+ "##rops": 25287,
+ "Committees": 25288,
+ "endings": 25289,
+ "##MM": 25290,
+ "##yung": 25291,
+ "ejected": 25292,
+ "grams": 25293,
+ "##relli": 25294,
+ "Birch": 25295,
+ "Hilary": 25296,
+ "Stadion": 25297,
+ "orphan": 25298,
+ "clawed": 25299,
+ "##kner": 25300,
+ "Motown": 25301,
+ "Wilkins": 25302,
+ "ballads": 25303,
+ "outspoken": 25304,
+ "##ancipation": 25305,
+ "##bankment": 25306,
+ "##cheng": 25307,
+ "Advances": 25308,
+ "harvested": 25309,
+ "novelty": 25310,
+ "ineligible": 25311,
+ "oversees": 25312,
+ "##´s": 25313,
+ "obeyed": 25314,
+ "inevitably": 25315,
+ "Kingdoms": 25316,
+ "burying": 25317,
+ "Fabian": 25318,
+ "relevance": 25319,
+ "Tatiana": 25320,
+ "##MCA": 25321,
+ "sarcastic": 25322,
+ "##onda": 25323,
+ "Akron": 25324,
+ "229": 25325,
+ "sandwiches": 25326,
+ "Adobe": 25327,
+ "Maddox": 25328,
+ "##azar": 25329,
+ "Hunting": 25330,
+ "##onized": 25331,
+ "Smiling": 25332,
+ "##tology": 25333,
+ "Juventus": 25334,
+ "Leroy": 25335,
+ "Poets": 25336,
+ "attach": 25337,
+ "lo": 25338,
+ "##rly": 25339,
+ "##film": 25340,
+ "Structure": 25341,
+ "##igate": 25342,
+ "olds": 25343,
+ "projections": 25344,
+ "SMS": 25345,
+ "outnumbered": 25346,
+ "##tase": 25347,
+ "judiciary": 25348,
+ "paramilitary": 25349,
+ "playfully": 25350,
+ "##rsing": 25351,
+ "##tras": 25352,
+ "Chico": 25353,
+ "Vin": 25354,
+ "informally": 25355,
+ "abandonment": 25356,
+ "##russ": 25357,
+ "Baroness": 25358,
+ "injuring": 25359,
+ "octagonal": 25360,
+ "deciduous": 25361,
+ "##nea": 25362,
+ "##olm": 25363,
+ "Hz": 25364,
+ "Norwood": 25365,
+ "poses": 25366,
+ "Marissa": 25367,
+ "alerted": 25368,
+ "willed": 25369,
+ "##KS": 25370,
+ "Dino": 25371,
+ "##ddler": 25372,
+ "##vani": 25373,
+ "Barbie": 25374,
+ "Thankfully": 25375,
+ "625": 25376,
+ "bicycles": 25377,
+ "shimmering": 25378,
+ "##tinuum": 25379,
+ "##wolf": 25380,
+ "Chesterfield": 25381,
+ "##idy": 25382,
+ "##urgency": 25383,
+ "Knowles": 25384,
+ "sweetly": 25385,
+ "Ventures": 25386,
+ "##ponents": 25387,
+ "##valence": 25388,
+ "Darryl": 25389,
+ "Powerplant": 25390,
+ "RAAF": 25391,
+ "##pec": 25392,
+ "Kingsley": 25393,
+ "Parramatta": 25394,
+ "penetrating": 25395,
+ "spectacle": 25396,
+ "##inia": 25397,
+ "Marlborough": 25398,
+ "residual": 25399,
+ "compatibility": 25400,
+ "hike": 25401,
+ "Underwood": 25402,
+ "depleted": 25403,
+ "ministries": 25404,
+ "##odus": 25405,
+ "##ropriation": 25406,
+ "rotting": 25407,
+ "Faso": 25408,
+ "##inn": 25409,
+ "Happiness": 25410,
+ "Lille": 25411,
+ "Suns": 25412,
+ "cookie": 25413,
+ "rift": 25414,
+ "warmly": 25415,
+ "##lvin": 25416,
+ "Bugs": 25417,
+ "Gotham": 25418,
+ "Gothenburg": 25419,
+ "Properties": 25420,
+ "##seller": 25421,
+ "##ubi": 25422,
+ "Created": 25423,
+ "MAC": 25424,
+ "Noelle": 25425,
+ "Requiem": 25426,
+ "Ulysses": 25427,
+ "##ails": 25428,
+ "franchises": 25429,
+ "##icious": 25430,
+ "##rwick": 25431,
+ "celestial": 25432,
+ "kinetic": 25433,
+ "720": 25434,
+ "STS": 25435,
+ "transmissions": 25436,
+ "amplitude": 25437,
+ "forums": 25438,
+ "freeing": 25439,
+ "reptiles": 25440,
+ "tumbling": 25441,
+ "##continent": 25442,
+ "##rising": 25443,
+ "##tropy": 25444,
+ "physiology": 25445,
+ "##uster": 25446,
+ "Loves": 25447,
+ "bodied": 25448,
+ "neutrality": 25449,
+ "Neumann": 25450,
+ "assessments": 25451,
+ "Vicky": 25452,
+ "##hom": 25453,
+ "hampered": 25454,
+ "##uku": 25455,
+ "Custom": 25456,
+ "timed": 25457,
+ "##eville": 25458,
+ "##xious": 25459,
+ "elastic": 25460,
+ "##section": 25461,
+ "rig": 25462,
+ "stilled": 25463,
+ "shipment": 25464,
+ "243": 25465,
+ "artworks": 25466,
+ "boulders": 25467,
+ "Bournemouth": 25468,
+ "##hly": 25469,
+ "##LF": 25470,
+ "##linary": 25471,
+ "rumored": 25472,
+ "##bino": 25473,
+ "##drum": 25474,
+ "Chun": 25475,
+ "Freiburg": 25476,
+ "##dges": 25477,
+ "Equality": 25478,
+ "252": 25479,
+ "Guadalajara": 25480,
+ "##sors": 25481,
+ "##taire": 25482,
+ "Roach": 25483,
+ "cramped": 25484,
+ "##ultural": 25485,
+ "Logistics": 25486,
+ "Punch": 25487,
+ "fines": 25488,
+ "Lai": 25489,
+ "caravan": 25490,
+ "##55": 25491,
+ "lame": 25492,
+ "Collector": 25493,
+ "pausing": 25494,
+ "315": 25495,
+ "migrant": 25496,
+ "hawk": 25497,
+ "signalling": 25498,
+ "##erham": 25499,
+ "##oughs": 25500,
+ "Demons": 25501,
+ "surfing": 25502,
+ "Rana": 25503,
+ "insisting": 25504,
+ "Wien": 25505,
+ "adolescent": 25506,
+ "##jong": 25507,
+ "##rera": 25508,
+ "##umba": 25509,
+ "Regis": 25510,
+ "brushes": 25511,
+ "##iman": 25512,
+ "residues": 25513,
+ "storytelling": 25514,
+ "Consider": 25515,
+ "contrasting": 25516,
+ "regeneration": 25517,
+ "##elling": 25518,
+ "##hlete": 25519,
+ "afforded": 25520,
+ "reactors": 25521,
+ "costing": 25522,
+ "##biotics": 25523,
+ "##gat": 25524,
+ "##евич": 25525,
+ "chanting": 25526,
+ "secondly": 25527,
+ "confesses": 25528,
+ "##ikos": 25529,
+ "##uang": 25530,
+ "##ronological": 25531,
+ "##−": 25532,
+ "Giacomo": 25533,
+ "##eca": 25534,
+ "vaudeville": 25535,
+ "weeds": 25536,
+ "rejecting": 25537,
+ "revoked": 25538,
+ "affluent": 25539,
+ "fullback": 25540,
+ "progresses": 25541,
+ "geologic": 25542,
+ "proprietor": 25543,
+ "replication": 25544,
+ "gliding": 25545,
+ "recounted": 25546,
+ "##bah": 25547,
+ "##igma": 25548,
+ "Flow": 25549,
+ "ii": 25550,
+ "newcomer": 25551,
+ "##lasp": 25552,
+ "##miya": 25553,
+ "Candace": 25554,
+ "fractured": 25555,
+ "interiors": 25556,
+ "confidential": 25557,
+ "Inverness": 25558,
+ "footing": 25559,
+ "##robe": 25560,
+ "Coordinator": 25561,
+ "Westphalia": 25562,
+ "jumper": 25563,
+ "##chism": 25564,
+ "dormitory": 25565,
+ "##gno": 25566,
+ "281": 25567,
+ "acknowledging": 25568,
+ "leveled": 25569,
+ "##éra": 25570,
+ "Algiers": 25571,
+ "migrate": 25572,
+ "Frog": 25573,
+ "Rare": 25574,
+ "##iovascular": 25575,
+ "##urous": 25576,
+ "DSO": 25577,
+ "nomadic": 25578,
+ "##iera": 25579,
+ "woken": 25580,
+ "lifeless": 25581,
+ "##graphical": 25582,
+ "##ifications": 25583,
+ "Dot": 25584,
+ "Sachs": 25585,
+ "crow": 25586,
+ "nmi": 25587,
+ "Tacoma": 25588,
+ "Weight": 25589,
+ "mushroom": 25590,
+ "RS": 25591,
+ "conditioned": 25592,
+ "##zine": 25593,
+ "Tunisian": 25594,
+ "altering": 25595,
+ "##mizing": 25596,
+ "Handicap": 25597,
+ "Patti": 25598,
+ "Monsieur": 25599,
+ "clicking": 25600,
+ "gorge": 25601,
+ "interrupting": 25602,
+ "##powerment": 25603,
+ "drawers": 25604,
+ "Serra": 25605,
+ "##icides": 25606,
+ "Specialist": 25607,
+ "##itte": 25608,
+ "connector": 25609,
+ "worshipped": 25610,
+ "##ask": 25611,
+ "consoles": 25612,
+ "tags": 25613,
+ "##iler": 25614,
+ "glued": 25615,
+ "##zac": 25616,
+ "fences": 25617,
+ "Bratislava": 25618,
+ "honeymoon": 25619,
+ "313": 25620,
+ "A2": 25621,
+ "disposition": 25622,
+ "Gentleman": 25623,
+ "Gilmore": 25624,
+ "glaciers": 25625,
+ "##scribed": 25626,
+ "Calhoun": 25627,
+ "convergence": 25628,
+ "Aleppo": 25629,
+ "shortages": 25630,
+ "##43": 25631,
+ "##orax": 25632,
+ "##worm": 25633,
+ "##codes": 25634,
+ "##rmal": 25635,
+ "neutron": 25636,
+ "##ossa": 25637,
+ "Bloomberg": 25638,
+ "Salford": 25639,
+ "periodicals": 25640,
+ "##ryan": 25641,
+ "Slayer": 25642,
+ "##ynasties": 25643,
+ "credentials": 25644,
+ "##tista": 25645,
+ "surveyor": 25646,
+ "File": 25647,
+ "stinging": 25648,
+ "unnoticed": 25649,
+ "Medici": 25650,
+ "ecstasy": 25651,
+ "espionage": 25652,
+ "Jett": 25653,
+ "Leary": 25654,
+ "circulating": 25655,
+ "bargaining": 25656,
+ "concerto": 25657,
+ "serviced": 25658,
+ "37th": 25659,
+ "HK": 25660,
+ "##fueling": 25661,
+ "Delilah": 25662,
+ "Marcia": 25663,
+ "graded": 25664,
+ "##join": 25665,
+ "Kaplan": 25666,
+ "feasible": 25667,
+ "##nale": 25668,
+ "##yt": 25669,
+ "Burnley": 25670,
+ "dreadful": 25671,
+ "ministerial": 25672,
+ "Brewster": 25673,
+ "Judah": 25674,
+ "##ngled": 25675,
+ "##rrey": 25676,
+ "recycled": 25677,
+ "Iroquois": 25678,
+ "backstage": 25679,
+ "parchment": 25680,
+ "##numbered": 25681,
+ "Kern": 25682,
+ "Motorsports": 25683,
+ "Organizations": 25684,
+ "##mini": 25685,
+ "Seems": 25686,
+ "Warrington": 25687,
+ "Dunbar": 25688,
+ "Ezio": 25689,
+ "##eor": 25690,
+ "paralyzed": 25691,
+ "Ara": 25692,
+ "yeast": 25693,
+ "##olis": 25694,
+ "cheated": 25695,
+ "reappeared": 25696,
+ "banged": 25697,
+ "##ymph": 25698,
+ "##dick": 25699,
+ "Lyndon": 25700,
+ "glide": 25701,
+ "Mat": 25702,
+ "##natch": 25703,
+ "Hotels": 25704,
+ "Household": 25705,
+ "parasite": 25706,
+ "irrelevant": 25707,
+ "youthful": 25708,
+ "##smic": 25709,
+ "##tero": 25710,
+ "##anti": 25711,
+ "2d": 25712,
+ "Ignacio": 25713,
+ "squash": 25714,
+ "##nets": 25715,
+ "shale": 25716,
+ "##اد": 25717,
+ "Abrams": 25718,
+ "##oese": 25719,
+ "assaults": 25720,
+ "##dier": 25721,
+ "##otte": 25722,
+ "Swamp": 25723,
+ "287": 25724,
+ "Spurs": 25725,
+ "##economic": 25726,
+ "Fargo": 25727,
+ "auditioned": 25728,
+ "##mé": 25729,
+ "Haas": 25730,
+ "une": 25731,
+ "abbreviation": 25732,
+ "Turkic": 25733,
+ "##tisfaction": 25734,
+ "favorites": 25735,
+ "specials": 25736,
+ "##lial": 25737,
+ "Enlightenment": 25738,
+ "Burkina": 25739,
+ "##vir": 25740,
+ "Comparative": 25741,
+ "Lacrosse": 25742,
+ "elves": 25743,
+ "##lerical": 25744,
+ "##pear": 25745,
+ "Borders": 25746,
+ "controllers": 25747,
+ "##villa": 25748,
+ "excelled": 25749,
+ "##acher": 25750,
+ "##varo": 25751,
+ "camouflage": 25752,
+ "perpetual": 25753,
+ "##ffles": 25754,
+ "devoid": 25755,
+ "schooner": 25756,
+ "##bered": 25757,
+ "##oris": 25758,
+ "Gibbons": 25759,
+ "Lia": 25760,
+ "discouraged": 25761,
+ "sue": 25762,
+ "##gnition": 25763,
+ "Excellent": 25764,
+ "Layton": 25765,
+ "noir": 25766,
+ "smack": 25767,
+ "##ivable": 25768,
+ "##evity": 25769,
+ "##lone": 25770,
+ "Myra": 25771,
+ "weaken": 25772,
+ "weaponry": 25773,
+ "##azza": 25774,
+ "Shake": 25775,
+ "backbone": 25776,
+ "Certified": 25777,
+ "clown": 25778,
+ "occupational": 25779,
+ "caller": 25780,
+ "enslaved": 25781,
+ "soaking": 25782,
+ "Wexford": 25783,
+ "perceive": 25784,
+ "shortlisted": 25785,
+ "##pid": 25786,
+ "feminism": 25787,
+ "Bari": 25788,
+ "Indie": 25789,
+ "##avelin": 25790,
+ "##ldo": 25791,
+ "Hellenic": 25792,
+ "Hundreds": 25793,
+ "Savings": 25794,
+ "comedies": 25795,
+ "Honors": 25796,
+ "Mohawk": 25797,
+ "Told": 25798,
+ "coded": 25799,
+ "Incorporated": 25800,
+ "hideous": 25801,
+ "trusts": 25802,
+ "hose": 25803,
+ "Calais": 25804,
+ "Forster": 25805,
+ "Gabon": 25806,
+ "Internationale": 25807,
+ "AK": 25808,
+ "Colour": 25809,
+ "##UM": 25810,
+ "##heist": 25811,
+ "McGregor": 25812,
+ "localized": 25813,
+ "##tronomy": 25814,
+ "Darrell": 25815,
+ "##iara": 25816,
+ "squirrel": 25817,
+ "freaked": 25818,
+ "##eking": 25819,
+ "##manned": 25820,
+ "##ungen": 25821,
+ "radiated": 25822,
+ "##dua": 25823,
+ "commence": 25824,
+ "Donaldson": 25825,
+ "##iddle": 25826,
+ "MR": 25827,
+ "SAS": 25828,
+ "Tavern": 25829,
+ "Teenage": 25830,
+ "admissions": 25831,
+ "Instruments": 25832,
+ "##ilizer": 25833,
+ "Konrad": 25834,
+ "contemplated": 25835,
+ "##ductor": 25836,
+ "Jing": 25837,
+ "Reacher": 25838,
+ "recalling": 25839,
+ "Dhabi": 25840,
+ "emphasizing": 25841,
+ "illumination": 25842,
+ "##tony": 25843,
+ "legitimacy": 25844,
+ "Goethe": 25845,
+ "Ritter": 25846,
+ "McDonnell": 25847,
+ "Polar": 25848,
+ "Seconds": 25849,
+ "aspiring": 25850,
+ "derby": 25851,
+ "tunic": 25852,
+ "##rmed": 25853,
+ "outlines": 25854,
+ "Changing": 25855,
+ "distortion": 25856,
+ "##cter": 25857,
+ "Mechanics": 25858,
+ "##urly": 25859,
+ "##vana": 25860,
+ "Egg": 25861,
+ "Wolverine": 25862,
+ "Stupid": 25863,
+ "centralized": 25864,
+ "knit": 25865,
+ "##Ms": 25866,
+ "Saratoga": 25867,
+ "Ogden": 25868,
+ "storylines": 25869,
+ "##vres": 25870,
+ "lavish": 25871,
+ "beverages": 25872,
+ "##grarian": 25873,
+ "Kyrgyzstan": 25874,
+ "forcefully": 25875,
+ "superb": 25876,
+ "Elm": 25877,
+ "Thessaloniki": 25878,
+ "follower": 25879,
+ "Plants": 25880,
+ "slang": 25881,
+ "trajectory": 25882,
+ "Nowadays": 25883,
+ "Bengals": 25884,
+ "Ingram": 25885,
+ "perch": 25886,
+ "coloring": 25887,
+ "carvings": 25888,
+ "doubtful": 25889,
+ "##aph": 25890,
+ "##gratulations": 25891,
+ "##41": 25892,
+ "Curse": 25893,
+ "253": 25894,
+ "nightstand": 25895,
+ "Campo": 25896,
+ "Meiji": 25897,
+ "decomposition": 25898,
+ "##giri": 25899,
+ "McCormick": 25900,
+ "Yours": 25901,
+ "##amon": 25902,
+ "##bang": 25903,
+ "Texans": 25904,
+ "injunction": 25905,
+ "organise": 25906,
+ "periodical": 25907,
+ "##peculative": 25908,
+ "oceans": 25909,
+ "##aley": 25910,
+ "Success": 25911,
+ "Lehigh": 25912,
+ "##guin": 25913,
+ "1730": 25914,
+ "Davy": 25915,
+ "allowance": 25916,
+ "obituary": 25917,
+ "##tov": 25918,
+ "treasury": 25919,
+ "##wayne": 25920,
+ "euros": 25921,
+ "readiness": 25922,
+ "systematically": 25923,
+ "##stered": 25924,
+ "##igor": 25925,
+ "##xen": 25926,
+ "##cliff": 25927,
+ "##lya": 25928,
+ "Send": 25929,
+ "##umatic": 25930,
+ "Celtics": 25931,
+ "Judiciary": 25932,
+ "425": 25933,
+ "propagation": 25934,
+ "rebellious": 25935,
+ "##ims": 25936,
+ "##lut": 25937,
+ "Dal": 25938,
+ "##ayman": 25939,
+ "##cloth": 25940,
+ "Boise": 25941,
+ "pairing": 25942,
+ "Waltz": 25943,
+ "torment": 25944,
+ "Hatch": 25945,
+ "aspirations": 25946,
+ "diaspora": 25947,
+ "##hame": 25948,
+ "Rank": 25949,
+ "237": 25950,
+ "Including": 25951,
+ "Muir": 25952,
+ "chained": 25953,
+ "toxicity": 25954,
+ "Université": 25955,
+ "##aroo": 25956,
+ "Mathews": 25957,
+ "meadows": 25958,
+ "##bio": 25959,
+ "Editing": 25960,
+ "Khorasan": 25961,
+ "##them": 25962,
+ "##ahn": 25963,
+ "##bari": 25964,
+ "##umes": 25965,
+ "evacuate": 25966,
+ "##sium": 25967,
+ "gram": 25968,
+ "kidnap": 25969,
+ "pinning": 25970,
+ "##diation": 25971,
+ "##orms": 25972,
+ "beacon": 25973,
+ "organising": 25974,
+ "McGrath": 25975,
+ "##ogist": 25976,
+ "Qur": 25977,
+ "Tango": 25978,
+ "##ceptor": 25979,
+ "##rud": 25980,
+ "##cend": 25981,
+ "##cie": 25982,
+ "##jas": 25983,
+ "##sided": 25984,
+ "Tuscany": 25985,
+ "Venture": 25986,
+ "creations": 25987,
+ "exhibiting": 25988,
+ "##rcerer": 25989,
+ "##tten": 25990,
+ "Butcher": 25991,
+ "Divinity": 25992,
+ "Pet": 25993,
+ "Whitehead": 25994,
+ "falsely": 25995,
+ "perished": 25996,
+ "handy": 25997,
+ "Moines": 25998,
+ "cyclists": 25999,
+ "synthesizers": 26000,
+ "Mortal": 26001,
+ "notoriety": 26002,
+ "##ronic": 26003,
+ "Dialogue": 26004,
+ "expressive": 26005,
+ "uk": 26006,
+ "Nightingale": 26007,
+ "grimly": 26008,
+ "vineyards": 26009,
+ "Driving": 26010,
+ "relentless": 26011,
+ "compiler": 26012,
+ "##district": 26013,
+ "##tuated": 26014,
+ "Hades": 26015,
+ "medicines": 26016,
+ "objection": 26017,
+ "Answer": 26018,
+ "Soap": 26019,
+ "Chattanooga": 26020,
+ "##gogue": 26021,
+ "Haryana": 26022,
+ "Parties": 26023,
+ "Turtle": 26024,
+ "##ferred": 26025,
+ "explorers": 26026,
+ "stakeholders": 26027,
+ "##aar": 26028,
+ "##rbonne": 26029,
+ "tempered": 26030,
+ "conjecture": 26031,
+ "##tee": 26032,
+ "##hur": 26033,
+ "Reeve": 26034,
+ "bumper": 26035,
+ "stew": 26036,
+ "##church": 26037,
+ "##generate": 26038,
+ "##ilitating": 26039,
+ "##chanized": 26040,
+ "##elier": 26041,
+ "##enne": 26042,
+ "translucent": 26043,
+ "##lows": 26044,
+ "Publisher": 26045,
+ "evangelical": 26046,
+ "inherit": 26047,
+ "##rted": 26048,
+ "247": 26049,
+ "SmackDown": 26050,
+ "bitterness": 26051,
+ "lesions": 26052,
+ "##worked": 26053,
+ "mosques": 26054,
+ "wed": 26055,
+ "##lashes": 26056,
+ "Ng": 26057,
+ "Rebels": 26058,
+ "booking": 26059,
+ "##nail": 26060,
+ "Incident": 26061,
+ "Sailing": 26062,
+ "yo": 26063,
+ "confirms": 26064,
+ "Chaplin": 26065,
+ "baths": 26066,
+ "##kled": 26067,
+ "modernist": 26068,
+ "pulsing": 26069,
+ "Cicero": 26070,
+ "slaughtered": 26071,
+ "boasted": 26072,
+ "##losure": 26073,
+ "zipper": 26074,
+ "##hales": 26075,
+ "aristocracy": 26076,
+ "halftime": 26077,
+ "jolt": 26078,
+ "unlawful": 26079,
+ "Marching": 26080,
+ "sustaining": 26081,
+ "Yerevan": 26082,
+ "bracket": 26083,
+ "ram": 26084,
+ "Markus": 26085,
+ "##zef": 26086,
+ "butcher": 26087,
+ "massage": 26088,
+ "##quisite": 26089,
+ "Leisure": 26090,
+ "Pizza": 26091,
+ "collapsing": 26092,
+ "##lante": 26093,
+ "commentaries": 26094,
+ "scripted": 26095,
+ "##disciplinary": 26096,
+ "##sused": 26097,
+ "eroded": 26098,
+ "alleging": 26099,
+ "vase": 26100,
+ "Chichester": 26101,
+ "Peacock": 26102,
+ "commencement": 26103,
+ "dice": 26104,
+ "hotter": 26105,
+ "poisonous": 26106,
+ "executions": 26107,
+ "##occo": 26108,
+ "frost": 26109,
+ "fielding": 26110,
+ "vendor": 26111,
+ "Counts": 26112,
+ "Troops": 26113,
+ "maize": 26114,
+ "Divisional": 26115,
+ "analogue": 26116,
+ "shadowy": 26117,
+ "Nuevo": 26118,
+ "Ville": 26119,
+ "radiating": 26120,
+ "worthless": 26121,
+ "Adriatic": 26122,
+ "Buy": 26123,
+ "blaze": 26124,
+ "brutally": 26125,
+ "horizontally": 26126,
+ "longed": 26127,
+ "##matical": 26128,
+ "federally": 26129,
+ "Rolf": 26130,
+ "Root": 26131,
+ "exclude": 26132,
+ "rag": 26133,
+ "agitation": 26134,
+ "Lounge": 26135,
+ "astonished": 26136,
+ "##wirl": 26137,
+ "Impossible": 26138,
+ "transformations": 26139,
+ "##IVE": 26140,
+ "##ceded": 26141,
+ "##slav": 26142,
+ "downloaded": 26143,
+ "fucked": 26144,
+ "Egyptians": 26145,
+ "Welles": 26146,
+ "##ffington": 26147,
+ "U2": 26148,
+ "befriended": 26149,
+ "radios": 26150,
+ "##jid": 26151,
+ "archaic": 26152,
+ "compares": 26153,
+ "##ccelerator": 26154,
+ "##imated": 26155,
+ "##tosis": 26156,
+ "Hung": 26157,
+ "Scientists": 26158,
+ "Thousands": 26159,
+ "geographically": 26160,
+ "##LR": 26161,
+ "Macintosh": 26162,
+ "fluorescent": 26163,
+ "##ipur": 26164,
+ "Wehrmacht": 26165,
+ "##BR": 26166,
+ "##firmary": 26167,
+ "Chao": 26168,
+ "##ague": 26169,
+ "Boyer": 26170,
+ "##grounds": 26171,
+ "##hism": 26172,
+ "##mento": 26173,
+ "##taining": 26174,
+ "infancy": 26175,
+ "##cton": 26176,
+ "510": 26177,
+ "Boca": 26178,
+ "##loy": 26179,
+ "1644": 26180,
+ "ben": 26181,
+ "dong": 26182,
+ "stresses": 26183,
+ "Sweat": 26184,
+ "expressway": 26185,
+ "graders": 26186,
+ "ochreous": 26187,
+ "nets": 26188,
+ "Lawn": 26189,
+ "thirst": 26190,
+ "Uruguayan": 26191,
+ "satisfactory": 26192,
+ "##tracts": 26193,
+ "baroque": 26194,
+ "rusty": 26195,
+ "##ław": 26196,
+ "Shen": 26197,
+ "Gdańsk": 26198,
+ "chickens": 26199,
+ "##graving": 26200,
+ "Hodge": 26201,
+ "Papal": 26202,
+ "SAT": 26203,
+ "bearer": 26204,
+ "##ogo": 26205,
+ "##rger": 26206,
+ "merits": 26207,
+ "Calendar": 26208,
+ "Highest": 26209,
+ "Skills": 26210,
+ "##ortex": 26211,
+ "Roberta": 26212,
+ "paradigm": 26213,
+ "recounts": 26214,
+ "frigates": 26215,
+ "swamps": 26216,
+ "unitary": 26217,
+ "##oker": 26218,
+ "balloons": 26219,
+ "Hawthorne": 26220,
+ "Muse": 26221,
+ "spurred": 26222,
+ "advisors": 26223,
+ "reclaimed": 26224,
+ "stimulate": 26225,
+ "fibre": 26226,
+ "pat": 26227,
+ "repeal": 26228,
+ "##dgson": 26229,
+ "##iar": 26230,
+ "##rana": 26231,
+ "anthropologist": 26232,
+ "descends": 26233,
+ "flinch": 26234,
+ "reared": 26235,
+ "##chang": 26236,
+ "##eric": 26237,
+ "##lithic": 26238,
+ "commissioning": 26239,
+ "##cumenical": 26240,
+ "##lume": 26241,
+ "##rchen": 26242,
+ "Wolff": 26243,
+ "##tsky": 26244,
+ "Eurasian": 26245,
+ "Nepali": 26246,
+ "Nightmare": 26247,
+ "ZIP": 26248,
+ "playback": 26249,
+ "##latz": 26250,
+ "##vington": 26251,
+ "Warm": 26252,
+ "##75": 26253,
+ "Martina": 26254,
+ "Rollins": 26255,
+ "Saetan": 26256,
+ "Variations": 26257,
+ "sorting": 26258,
+ "##م": 26259,
+ "530": 26260,
+ "Joaquin": 26261,
+ "Ptolemy": 26262,
+ "thinner": 26263,
+ "##iator": 26264,
+ "##pticism": 26265,
+ "Cebu": 26266,
+ "Highlanders": 26267,
+ "Linden": 26268,
+ "Vanguard": 26269,
+ "##SV": 26270,
+ "##mor": 26271,
+ "##ulge": 26272,
+ "ISSN": 26273,
+ "cartridges": 26274,
+ "repression": 26275,
+ "Étienne": 26276,
+ "311": 26277,
+ "Lauderdale": 26278,
+ "commodities": 26279,
+ "null": 26280,
+ "##rb": 26281,
+ "1720": 26282,
+ "gearbox": 26283,
+ "##reator": 26284,
+ "Ang": 26285,
+ "Forgotten": 26286,
+ "dubious": 26287,
+ "##rls": 26288,
+ "##dicative": 26289,
+ "##phate": 26290,
+ "Groove": 26291,
+ "Herrera": 26292,
+ "##çais": 26293,
+ "Collections": 26294,
+ "Maximus": 26295,
+ "##published": 26296,
+ "Fell": 26297,
+ "Qualification": 26298,
+ "filtering": 26299,
+ "##tized": 26300,
+ "Roe": 26301,
+ "hazards": 26302,
+ "##37": 26303,
+ "##lative": 26304,
+ "##tröm": 26305,
+ "Guadalupe": 26306,
+ "Tajikistan": 26307,
+ "Preliminary": 26308,
+ "fronted": 26309,
+ "glands": 26310,
+ "##paper": 26311,
+ "##iche": 26312,
+ "##iding": 26313,
+ "Cairns": 26314,
+ "rallies": 26315,
+ "Location": 26316,
+ "seduce": 26317,
+ "##mple": 26318,
+ "BYU": 26319,
+ "##itic": 26320,
+ "##FT": 26321,
+ "Carmichael": 26322,
+ "Prentice": 26323,
+ "songwriters": 26324,
+ "forefront": 26325,
+ "Physicians": 26326,
+ "##rille": 26327,
+ "##zee": 26328,
+ "Preparatory": 26329,
+ "##cherous": 26330,
+ "UV": 26331,
+ "##dized": 26332,
+ "Navarro": 26333,
+ "misses": 26334,
+ "##nney": 26335,
+ "Inland": 26336,
+ "resisting": 26337,
+ "##sect": 26338,
+ "Hurt": 26339,
+ "##lino": 26340,
+ "galaxies": 26341,
+ "##raze": 26342,
+ "Institutions": 26343,
+ "devote": 26344,
+ "##lamp": 26345,
+ "##ciating": 26346,
+ "baron": 26347,
+ "##bracing": 26348,
+ "Hess": 26349,
+ "operatic": 26350,
+ "##CL": 26351,
+ "##ος": 26352,
+ "Chevalier": 26353,
+ "Guiana": 26354,
+ "##lattered": 26355,
+ "Fed": 26356,
+ "##cuted": 26357,
+ "##smo": 26358,
+ "Skull": 26359,
+ "denies": 26360,
+ "236": 26361,
+ "Waller": 26362,
+ "##mah": 26363,
+ "Sakura": 26364,
+ "mole": 26365,
+ "nominate": 26366,
+ "sermons": 26367,
+ "##bering": 26368,
+ "widowed": 26369,
+ "##röm": 26370,
+ "Cavendish": 26371,
+ "##struction": 26372,
+ "Nehru": 26373,
+ "Revelation": 26374,
+ "doom": 26375,
+ "Gala": 26376,
+ "baking": 26377,
+ "Nr": 26378,
+ "Yourself": 26379,
+ "banning": 26380,
+ "Individuals": 26381,
+ "Sykes": 26382,
+ "orchestrated": 26383,
+ "630": 26384,
+ "Phone": 26385,
+ "steered": 26386,
+ "620": 26387,
+ "specialising": 26388,
+ "starvation": 26389,
+ "##AV": 26390,
+ "##alet": 26391,
+ "##upation": 26392,
+ "seductive": 26393,
+ "##jects": 26394,
+ "##zure": 26395,
+ "Tolkien": 26396,
+ "Benito": 26397,
+ "Wizards": 26398,
+ "Submarine": 26399,
+ "dictator": 26400,
+ "Duo": 26401,
+ "Caden": 26402,
+ "approx": 26403,
+ "basins": 26404,
+ "##nc": 26405,
+ "shrink": 26406,
+ "##icles": 26407,
+ "##sponsible": 26408,
+ "249": 26409,
+ "mit": 26410,
+ "outpost": 26411,
+ "##bayashi": 26412,
+ "##rouse": 26413,
+ "##tl": 26414,
+ "Jana": 26415,
+ "Lombard": 26416,
+ "RBIs": 26417,
+ "finalized": 26418,
+ "humanities": 26419,
+ "##function": 26420,
+ "Honorable": 26421,
+ "tomato": 26422,
+ "##iot": 26423,
+ "Pie": 26424,
+ "tee": 26425,
+ "##pect": 26426,
+ "Beaufort": 26427,
+ "Ferris": 26428,
+ "bucks": 26429,
+ "##graduate": 26430,
+ "##ocytes": 26431,
+ "Directory": 26432,
+ "anxiously": 26433,
+ "##nating": 26434,
+ "flanks": 26435,
+ "##Ds": 26436,
+ "virtues": 26437,
+ "##believable": 26438,
+ "Grades": 26439,
+ "criterion": 26440,
+ "manufactures": 26441,
+ "sourced": 26442,
+ "##balt": 26443,
+ "##dance": 26444,
+ "##tano": 26445,
+ "Ying": 26446,
+ "##BF": 26447,
+ "##sett": 26448,
+ "adequately": 26449,
+ "blacksmith": 26450,
+ "totaled": 26451,
+ "trapping": 26452,
+ "expanse": 26453,
+ "Historia": 26454,
+ "Worker": 26455,
+ "Sense": 26456,
+ "ascending": 26457,
+ "housekeeper": 26458,
+ "##oos": 26459,
+ "Crafts": 26460,
+ "Resurrection": 26461,
+ "##verty": 26462,
+ "encryption": 26463,
+ "##aris": 26464,
+ "##vat": 26465,
+ "##pox": 26466,
+ "##runk": 26467,
+ "##iability": 26468,
+ "gazes": 26469,
+ "spying": 26470,
+ "##ths": 26471,
+ "helmets": 26472,
+ "wired": 26473,
+ "##zophrenia": 26474,
+ "Cheung": 26475,
+ "WR": 26476,
+ "downloads": 26477,
+ "stereotypes": 26478,
+ "239": 26479,
+ "Lucknow": 26480,
+ "bleak": 26481,
+ "Bragg": 26482,
+ "hauling": 26483,
+ "##haft": 26484,
+ "prohibit": 26485,
+ "##ermined": 26486,
+ "##castle": 26487,
+ "barony": 26488,
+ "##hta": 26489,
+ "Typhoon": 26490,
+ "antibodies": 26491,
+ "##ascism": 26492,
+ "Hawthorn": 26493,
+ "Kurdistan": 26494,
+ "Minority": 26495,
+ "Gorge": 26496,
+ "Herr": 26497,
+ "appliances": 26498,
+ "disrupt": 26499,
+ "Drugs": 26500,
+ "Lazarus": 26501,
+ "##ilia": 26502,
+ "##ryo": 26503,
+ "##tany": 26504,
+ "Gotta": 26505,
+ "Masovian": 26506,
+ "Roxy": 26507,
+ "choreographed": 26508,
+ "##rissa": 26509,
+ "turbulent": 26510,
+ "##listed": 26511,
+ "Anatomy": 26512,
+ "exiting": 26513,
+ "##det": 26514,
+ "##isław": 26515,
+ "580": 26516,
+ "Kaufman": 26517,
+ "sage": 26518,
+ "##apa": 26519,
+ "Symposium": 26520,
+ "##rolls": 26521,
+ "Kaye": 26522,
+ "##ptera": 26523,
+ "##rocław": 26524,
+ "jerking": 26525,
+ "##menclature": 26526,
+ "Guo": 26527,
+ "M1": 26528,
+ "resurrected": 26529,
+ "trophies": 26530,
+ "##lard": 26531,
+ "Gathering": 26532,
+ "nestled": 26533,
+ "serpent": 26534,
+ "Dow": 26535,
+ "reservoirs": 26536,
+ "Claremont": 26537,
+ "arbitration": 26538,
+ "chronicle": 26539,
+ "eki": 26540,
+ "##arded": 26541,
+ "##zers": 26542,
+ "##mmoth": 26543,
+ "Congregational": 26544,
+ "Astronomical": 26545,
+ "NE": 26546,
+ "RA": 26547,
+ "Robson": 26548,
+ "Scotch": 26549,
+ "modelled": 26550,
+ "slashed": 26551,
+ "##imus": 26552,
+ "exceeds": 26553,
+ "##roper": 26554,
+ "##utile": 26555,
+ "Laughing": 26556,
+ "vascular": 26557,
+ "superficial": 26558,
+ "##arians": 26559,
+ "Barclay": 26560,
+ "Caucasian": 26561,
+ "classmate": 26562,
+ "sibling": 26563,
+ "Kimberly": 26564,
+ "Shreveport": 26565,
+ "##ilde": 26566,
+ "##liche": 26567,
+ "Cheney": 26568,
+ "Deportivo": 26569,
+ "Veracruz": 26570,
+ "berries": 26571,
+ "##lase": 26572,
+ "Bed": 26573,
+ "MI": 26574,
+ "Anatolia": 26575,
+ "Mindanao": 26576,
+ "broadband": 26577,
+ "##olia": 26578,
+ "##arte": 26579,
+ "##wab": 26580,
+ "darts": 26581,
+ "##immer": 26582,
+ "##uze": 26583,
+ "believers": 26584,
+ "ordinance": 26585,
+ "violate": 26586,
+ "##wheel": 26587,
+ "##ynth": 26588,
+ "Alongside": 26589,
+ "Coupe": 26590,
+ "Hobbs": 26591,
+ "arrondissement": 26592,
+ "earl": 26593,
+ "townland": 26594,
+ "##dote": 26595,
+ "##lihood": 26596,
+ "##sla": 26597,
+ "Ghosts": 26598,
+ "midfield": 26599,
+ "pulmonary": 26600,
+ "##eno": 26601,
+ "cues": 26602,
+ "##gol": 26603,
+ "##zda": 26604,
+ "322": 26605,
+ "Siena": 26606,
+ "Sultanate": 26607,
+ "Bradshaw": 26608,
+ "Pieter": 26609,
+ "##thical": 26610,
+ "Raceway": 26611,
+ "bared": 26612,
+ "competence": 26613,
+ "##ssent": 26614,
+ "Bet": 26615,
+ "##urer": 26616,
+ "##ła": 26617,
+ "Alistair": 26618,
+ "Göttingen": 26619,
+ "appropriately": 26620,
+ "forge": 26621,
+ "##osterone": 26622,
+ "##ugen": 26623,
+ "DL": 26624,
+ "345": 26625,
+ "convoys": 26626,
+ "inventions": 26627,
+ "##resses": 26628,
+ "##cturnal": 26629,
+ "Fay": 26630,
+ "Integration": 26631,
+ "slash": 26632,
+ "##roats": 26633,
+ "Widow": 26634,
+ "barking": 26635,
+ "##fant": 26636,
+ "1A": 26637,
+ "Hooper": 26638,
+ "##cona": 26639,
+ "##runched": 26640,
+ "unreliable": 26641,
+ "##emont": 26642,
+ "##esign": 26643,
+ "##stabulary": 26644,
+ "##stop": 26645,
+ "Journalists": 26646,
+ "bony": 26647,
+ "##iba": 26648,
+ "##trata": 26649,
+ "##ège": 26650,
+ "horrific": 26651,
+ "##bish": 26652,
+ "Jocelyn": 26653,
+ "##rmon": 26654,
+ "##apon": 26655,
+ "##cier": 26656,
+ "trainers": 26657,
+ "##ulatory": 26658,
+ "1753": 26659,
+ "BR": 26660,
+ "corpus": 26661,
+ "synthesized": 26662,
+ "##bidden": 26663,
+ "##rafford": 26664,
+ "Elgin": 26665,
+ "##entry": 26666,
+ "Doherty": 26667,
+ "clockwise": 26668,
+ "##played": 26669,
+ "spins": 26670,
+ "##ample": 26671,
+ "##bley": 26672,
+ "Cope": 26673,
+ "constructions": 26674,
+ "seater": 26675,
+ "warlord": 26676,
+ "Voyager": 26677,
+ "documenting": 26678,
+ "fairies": 26679,
+ "##viator": 26680,
+ "Lviv": 26681,
+ "jewellery": 26682,
+ "suites": 26683,
+ "##gold": 26684,
+ "Maia": 26685,
+ "NME": 26686,
+ "##eavor": 26687,
+ "##kus": 26688,
+ "Eugène": 26689,
+ "furnishings": 26690,
+ "##risto": 26691,
+ "MCC": 26692,
+ "Metropolis": 26693,
+ "Older": 26694,
+ "Telangana": 26695,
+ "##mpus": 26696,
+ "amplifier": 26697,
+ "supervising": 26698,
+ "1710": 26699,
+ "buffalo": 26700,
+ "cushion": 26701,
+ "terminating": 26702,
+ "##powering": 26703,
+ "steak": 26704,
+ "Quickly": 26705,
+ "contracting": 26706,
+ "dem": 26707,
+ "sarcastically": 26708,
+ "Elsa": 26709,
+ "##hein": 26710,
+ "bastards": 26711,
+ "narratives": 26712,
+ "Takes": 26713,
+ "304": 26714,
+ "composure": 26715,
+ "typing": 26716,
+ "variance": 26717,
+ "##ifice": 26718,
+ "Softball": 26719,
+ "##rations": 26720,
+ "McLaughlin": 26721,
+ "gaped": 26722,
+ "shrines": 26723,
+ "##hogany": 26724,
+ "Glamorgan": 26725,
+ "##icle": 26726,
+ "##nai": 26727,
+ "##ntin": 26728,
+ "Fleetwood": 26729,
+ "Woodland": 26730,
+ "##uxe": 26731,
+ "fictitious": 26732,
+ "shrugs": 26733,
+ "##iper": 26734,
+ "BWV": 26735,
+ "conform": 26736,
+ "##uckled": 26737,
+ "Launch": 26738,
+ "##ductory": 26739,
+ "##mized": 26740,
+ "Tad": 26741,
+ "##stituted": 26742,
+ "##free": 26743,
+ "Bel": 26744,
+ "Chávez": 26745,
+ "messing": 26746,
+ "quartz": 26747,
+ "##iculate": 26748,
+ "##folia": 26749,
+ "##lynn": 26750,
+ "ushered": 26751,
+ "##29": 26752,
+ "##ailing": 26753,
+ "dictated": 26754,
+ "Pony": 26755,
+ "##opsis": 26756,
+ "precinct": 26757,
+ "802": 26758,
+ "Plastic": 26759,
+ "##ughter": 26760,
+ "##uno": 26761,
+ "##porated": 26762,
+ "Denton": 26763,
+ "Matters": 26764,
+ "SPD": 26765,
+ "hating": 26766,
+ "##rogen": 26767,
+ "Essential": 26768,
+ "Deck": 26769,
+ "Dortmund": 26770,
+ "obscured": 26771,
+ "##maging": 26772,
+ "Earle": 26773,
+ "##bred": 26774,
+ "##ittle": 26775,
+ "##ropolis": 26776,
+ "saturated": 26777,
+ "##fiction": 26778,
+ "##ression": 26779,
+ "Pereira": 26780,
+ "Vinci": 26781,
+ "mute": 26782,
+ "warehouses": 26783,
+ "##ún": 26784,
+ "biographies": 26785,
+ "##icking": 26786,
+ "sealing": 26787,
+ "##dered": 26788,
+ "executing": 26789,
+ "pendant": 26790,
+ "##wives": 26791,
+ "murmurs": 26792,
+ "##oko": 26793,
+ "substrates": 26794,
+ "symmetrical": 26795,
+ "Susie": 26796,
+ "##mare": 26797,
+ "Yusuf": 26798,
+ "analogy": 26799,
+ "##urage": 26800,
+ "Lesley": 26801,
+ "limitation": 26802,
+ "##rby": 26803,
+ "##ío": 26804,
+ "disagreements": 26805,
+ "##mise": 26806,
+ "embroidered": 26807,
+ "nape": 26808,
+ "unarmed": 26809,
+ "Sumner": 26810,
+ "Stores": 26811,
+ "dwell": 26812,
+ "Wilcox": 26813,
+ "creditors": 26814,
+ "##rivatization": 26815,
+ "##shes": 26816,
+ "##amia": 26817,
+ "directs": 26818,
+ "recaptured": 26819,
+ "scouting": 26820,
+ "McGuire": 26821,
+ "cradle": 26822,
+ "##onnell": 26823,
+ "Sato": 26824,
+ "insulin": 26825,
+ "mercenary": 26826,
+ "tolerant": 26827,
+ "Macquarie": 26828,
+ "transitions": 26829,
+ "cradled": 26830,
+ "##berto": 26831,
+ "##ivism": 26832,
+ "##yotes": 26833,
+ "FF": 26834,
+ "Ke": 26835,
+ "Reach": 26836,
+ "##dbury": 26837,
+ "680": 26838,
+ "##bill": 26839,
+ "##oja": 26840,
+ "##sui": 26841,
+ "prairie": 26842,
+ "##ogan": 26843,
+ "reactive": 26844,
+ "##icient": 26845,
+ "##rits": 26846,
+ "Cyclone": 26847,
+ "Sirius": 26848,
+ "Survival": 26849,
+ "Pak": 26850,
+ "##coach": 26851,
+ "##trar": 26852,
+ "halves": 26853,
+ "Agatha": 26854,
+ "Opus": 26855,
+ "contrasts": 26856,
+ "##jection": 26857,
+ "ominous": 26858,
+ "##iden": 26859,
+ "Baylor": 26860,
+ "Woodrow": 26861,
+ "duct": 26862,
+ "fortification": 26863,
+ "intercourse": 26864,
+ "##rois": 26865,
+ "Colbert": 26866,
+ "envy": 26867,
+ "##isi": 26868,
+ "Afterward": 26869,
+ "geared": 26870,
+ "##flections": 26871,
+ "accelerate": 26872,
+ "##lenching": 26873,
+ "Witness": 26874,
+ "##rrer": 26875,
+ "Angelina": 26876,
+ "Material": 26877,
+ "assertion": 26878,
+ "misconduct": 26879,
+ "Nix": 26880,
+ "cringed": 26881,
+ "tingling": 26882,
+ "##eti": 26883,
+ "##gned": 26884,
+ "Everest": 26885,
+ "disturb": 26886,
+ "sturdy": 26887,
+ "##keepers": 26888,
+ "##vied": 26889,
+ "Profile": 26890,
+ "heavenly": 26891,
+ "##kova": 26892,
+ "##victed": 26893,
+ "translating": 26894,
+ "##sses": 26895,
+ "316": 26896,
+ "Invitational": 26897,
+ "Mention": 26898,
+ "martyr": 26899,
+ "##uristic": 26900,
+ "Barron": 26901,
+ "hardness": 26902,
+ "Nakamura": 26903,
+ "405": 26904,
+ "Genevieve": 26905,
+ "reflections": 26906,
+ "##falls": 26907,
+ "jurist": 26908,
+ "##LT": 26909,
+ "Pyramid": 26910,
+ "##yme": 26911,
+ "Shoot": 26912,
+ "heck": 26913,
+ "linguist": 26914,
+ "##tower": 26915,
+ "Ives": 26916,
+ "superiors": 26917,
+ "##leo": 26918,
+ "Achilles": 26919,
+ "##phological": 26920,
+ "Christophe": 26921,
+ "Padma": 26922,
+ "precedence": 26923,
+ "grassy": 26924,
+ "Oral": 26925,
+ "resurrection": 26926,
+ "##itting": 26927,
+ "clumsy": 26928,
+ "##lten": 26929,
+ "##rue": 26930,
+ "huts": 26931,
+ "##stars": 26932,
+ "Equal": 26933,
+ "##queduct": 26934,
+ "Devin": 26935,
+ "Gaga": 26936,
+ "diocesan": 26937,
+ "##plating": 26938,
+ "##upe": 26939,
+ "##graphers": 26940,
+ "Patch": 26941,
+ "Scream": 26942,
+ "hail": 26943,
+ "moaning": 26944,
+ "tracts": 26945,
+ "##hdi": 26946,
+ "Examination": 26947,
+ "outsider": 26948,
+ "##ergic": 26949,
+ "##oter": 26950,
+ "Archipelago": 26951,
+ "Havilland": 26952,
+ "greenish": 26953,
+ "tilting": 26954,
+ "Aleksandr": 26955,
+ "Konstantin": 26956,
+ "warship": 26957,
+ "##emann": 26958,
+ "##gelist": 26959,
+ "##ought": 26960,
+ "billionaire": 26961,
+ "##blivion": 26962,
+ "321": 26963,
+ "Hungarians": 26964,
+ "transplant": 26965,
+ "##jured": 26966,
+ "##fters": 26967,
+ "Corbin": 26968,
+ "autism": 26969,
+ "pitchers": 26970,
+ "Garner": 26971,
+ "thence": 26972,
+ "Scientology": 26973,
+ "transitioned": 26974,
+ "integrating": 26975,
+ "repetitive": 26976,
+ "##dant": 26977,
+ "Rene": 26978,
+ "vomit": 26979,
+ "##burne": 26980,
+ "1661": 26981,
+ "Researchers": 26982,
+ "Wallis": 26983,
+ "insulted": 26984,
+ "wavy": 26985,
+ "##wati": 26986,
+ "Ewing": 26987,
+ "excitedly": 26988,
+ "##kor": 26989,
+ "frescoes": 26990,
+ "injustice": 26991,
+ "##achal": 26992,
+ "##lumber": 26993,
+ "##úl": 26994,
+ "novella": 26995,
+ "##sca": 26996,
+ "Liv": 26997,
+ "##enstein": 26998,
+ "##river": 26999,
+ "monstrous": 27000,
+ "topping": 27001,
+ "downfall": 27002,
+ "looming": 27003,
+ "sinks": 27004,
+ "trillion": 27005,
+ "##pont": 27006,
+ "Effect": 27007,
+ "##phi": 27008,
+ "##urley": 27009,
+ "Sites": 27010,
+ "catchment": 27011,
+ "##H1": 27012,
+ "Hopper": 27013,
+ "##raiser": 27014,
+ "1642": 27015,
+ "Maccabi": 27016,
+ "lance": 27017,
+ "##chia": 27018,
+ "##sboro": 27019,
+ "NSA": 27020,
+ "branching": 27021,
+ "retorted": 27022,
+ "tensor": 27023,
+ "Immaculate": 27024,
+ "drumming": 27025,
+ "feeder": 27026,
+ "##mony": 27027,
+ "Dyer": 27028,
+ "homicide": 27029,
+ "Temeraire": 27030,
+ "fishes": 27031,
+ "protruding": 27032,
+ "skins": 27033,
+ "orchards": 27034,
+ "##nso": 27035,
+ "inlet": 27036,
+ "ventral": 27037,
+ "##finder": 27038,
+ "Asiatic": 27039,
+ "Sul": 27040,
+ "1688": 27041,
+ "Melinda": 27042,
+ "assigns": 27043,
+ "paranormal": 27044,
+ "gardening": 27045,
+ "Tau": 27046,
+ "calming": 27047,
+ "##inge": 27048,
+ "##crow": 27049,
+ "regimental": 27050,
+ "Nik": 27051,
+ "fastened": 27052,
+ "correlated": 27053,
+ "##gene": 27054,
+ "##rieve": 27055,
+ "Sick": 27056,
+ "##minster": 27057,
+ "##politan": 27058,
+ "hardwood": 27059,
+ "hurled": 27060,
+ "##ssler": 27061,
+ "Cinematography": 27062,
+ "rhyme": 27063,
+ "Montenegrin": 27064,
+ "Packard": 27065,
+ "debating": 27066,
+ "##itution": 27067,
+ "Helens": 27068,
+ "Trick": 27069,
+ "Museums": 27070,
+ "defiance": 27071,
+ "encompassed": 27072,
+ "##EE": 27073,
+ "##TU": 27074,
+ "##nees": 27075,
+ "##uben": 27076,
+ "##ünster": 27077,
+ "##nosis": 27078,
+ "435": 27079,
+ "Hagen": 27080,
+ "cinemas": 27081,
+ "Corbett": 27082,
+ "commended": 27083,
+ "##fines": 27084,
+ "##oman": 27085,
+ "bosses": 27086,
+ "ripe": 27087,
+ "scraping": 27088,
+ "##loc": 27089,
+ "filly": 27090,
+ "Saddam": 27091,
+ "pointless": 27092,
+ "Faust": 27093,
+ "Orléans": 27094,
+ "Syriac": 27095,
+ "##♭": 27096,
+ "longitude": 27097,
+ "##ropic": 27098,
+ "Alfa": 27099,
+ "bliss": 27100,
+ "gangster": 27101,
+ "##ckling": 27102,
+ "SL": 27103,
+ "blending": 27104,
+ "##eptide": 27105,
+ "##nner": 27106,
+ "bends": 27107,
+ "escorting": 27108,
+ "##bloid": 27109,
+ "##quis": 27110,
+ "burials": 27111,
+ "##sle": 27112,
+ "##è": 27113,
+ "Ambulance": 27114,
+ "insults": 27115,
+ "##gth": 27116,
+ "Antrim": 27117,
+ "unfolded": 27118,
+ "##missible": 27119,
+ "splendid": 27120,
+ "Cure": 27121,
+ "warily": 27122,
+ "Saigon": 27123,
+ "Waste": 27124,
+ "astonishment": 27125,
+ "boroughs": 27126,
+ "##VS": 27127,
+ "##dalgo": 27128,
+ "##reshing": 27129,
+ "##usage": 27130,
+ "rue": 27131,
+ "marital": 27132,
+ "versatile": 27133,
+ "unpaid": 27134,
+ "allotted": 27135,
+ "bacterium": 27136,
+ "##coil": 27137,
+ "##cue": 27138,
+ "Dorothea": 27139,
+ "IDF": 27140,
+ "##location": 27141,
+ "##yke": 27142,
+ "RPG": 27143,
+ "##tropical": 27144,
+ "devotees": 27145,
+ "liter": 27146,
+ "##pree": 27147,
+ "Johnstone": 27148,
+ "astronaut": 27149,
+ "attends": 27150,
+ "pollen": 27151,
+ "periphery": 27152,
+ "doctrines": 27153,
+ "meta": 27154,
+ "showered": 27155,
+ "##tyn": 27156,
+ "GO": 27157,
+ "Huh": 27158,
+ "laude": 27159,
+ "244": 27160,
+ "Amar": 27161,
+ "Christensen": 27162,
+ "Ping": 27163,
+ "Pontifical": 27164,
+ "Austen": 27165,
+ "raiding": 27166,
+ "realities": 27167,
+ "##dric": 27168,
+ "urges": 27169,
+ "##dek": 27170,
+ "Cambridgeshire": 27171,
+ "##otype": 27172,
+ "Cascade": 27173,
+ "Greenberg": 27174,
+ "Pact": 27175,
+ "##cognition": 27176,
+ "##aran": 27177,
+ "##urion": 27178,
+ "Riot": 27179,
+ "mimic": 27180,
+ "Eastwood": 27181,
+ "##imating": 27182,
+ "reversal": 27183,
+ "##blast": 27184,
+ "##henian": 27185,
+ "Pitchfork": 27186,
+ "##sunderstanding": 27187,
+ "Staten": 27188,
+ "WCW": 27189,
+ "lieu": 27190,
+ "##bard": 27191,
+ "##sang": 27192,
+ "experimenting": 27193,
+ "Aquino": 27194,
+ "##lums": 27195,
+ "TNT": 27196,
+ "Hannibal": 27197,
+ "catastrophic": 27198,
+ "##lsive": 27199,
+ "272": 27200,
+ "308": 27201,
+ "##otypic": 27202,
+ "41st": 27203,
+ "Highways": 27204,
+ "aggregator": 27205,
+ "##fluenza": 27206,
+ "Featured": 27207,
+ "Reece": 27208,
+ "dispatch": 27209,
+ "simulated": 27210,
+ "##BE": 27211,
+ "Communion": 27212,
+ "Vinnie": 27213,
+ "hardcover": 27214,
+ "inexpensive": 27215,
+ "til": 27216,
+ "##adores": 27217,
+ "groundwater": 27218,
+ "kicker": 27219,
+ "blogs": 27220,
+ "frenzy": 27221,
+ "##wala": 27222,
+ "dealings": 27223,
+ "erase": 27224,
+ "Anglia": 27225,
+ "##umour": 27226,
+ "Hapoel": 27227,
+ "Marquette": 27228,
+ "##raphic": 27229,
+ "##tives": 27230,
+ "consult": 27231,
+ "atrocities": 27232,
+ "concussion": 27233,
+ "##érard": 27234,
+ "Decree": 27235,
+ "ethanol": 27236,
+ "##aen": 27237,
+ "Rooney": 27238,
+ "##chemist": 27239,
+ "##hoot": 27240,
+ "1620": 27241,
+ "menacing": 27242,
+ "Schuster": 27243,
+ "##bearable": 27244,
+ "laborers": 27245,
+ "sultan": 27246,
+ "Juliana": 27247,
+ "erased": 27248,
+ "onstage": 27249,
+ "##ync": 27250,
+ "Eastman": 27251,
+ "##tick": 27252,
+ "hushed": 27253,
+ "##yrinth": 27254,
+ "Lexie": 27255,
+ "Wharton": 27256,
+ "Lev": 27257,
+ "##PL": 27258,
+ "Testing": 27259,
+ "Bangladeshi": 27260,
+ "##bba": 27261,
+ "##usions": 27262,
+ "communicated": 27263,
+ "integers": 27264,
+ "internship": 27265,
+ "societal": 27266,
+ "##odles": 27267,
+ "Loki": 27268,
+ "ET": 27269,
+ "Ghent": 27270,
+ "broadcasters": 27271,
+ "Unix": 27272,
+ "##auer": 27273,
+ "Kildare": 27274,
+ "Yamaha": 27275,
+ "##quencing": 27276,
+ "##zman": 27277,
+ "chilled": 27278,
+ "##rapped": 27279,
+ "##uant": 27280,
+ "Duval": 27281,
+ "sentiments": 27282,
+ "Oliveira": 27283,
+ "packets": 27284,
+ "Horne": 27285,
+ "##rient": 27286,
+ "Harlan": 27287,
+ "Mirage": 27288,
+ "invariant": 27289,
+ "##anger": 27290,
+ "##tensive": 27291,
+ "flexed": 27292,
+ "sweetness": 27293,
+ "##wson": 27294,
+ "alleviate": 27295,
+ "insulting": 27296,
+ "limo": 27297,
+ "Hahn": 27298,
+ "##llars": 27299,
+ "##hesia": 27300,
+ "##lapping": 27301,
+ "buys": 27302,
+ "##oaming": 27303,
+ "mocked": 27304,
+ "pursuits": 27305,
+ "scooted": 27306,
+ "##conscious": 27307,
+ "##ilian": 27308,
+ "Ballad": 27309,
+ "jackets": 27310,
+ "##kra": 27311,
+ "hilly": 27312,
+ "##cane": 27313,
+ "Scenic": 27314,
+ "McGraw": 27315,
+ "silhouette": 27316,
+ "whipping": 27317,
+ "##roduced": 27318,
+ "##wark": 27319,
+ "##chess": 27320,
+ "##rump": 27321,
+ "Lemon": 27322,
+ "calculus": 27323,
+ "demonic": 27324,
+ "##latine": 27325,
+ "Bharatiya": 27326,
+ "Govt": 27327,
+ "Que": 27328,
+ "Trilogy": 27329,
+ "Ducks": 27330,
+ "Suit": 27331,
+ "stairway": 27332,
+ "##ceipt": 27333,
+ "Isa": 27334,
+ "regulator": 27335,
+ "Automobile": 27336,
+ "flatly": 27337,
+ "##buster": 27338,
+ "##lank": 27339,
+ "Spartans": 27340,
+ "topography": 27341,
+ "Tavi": 27342,
+ "usable": 27343,
+ "Chartered": 27344,
+ "Fairchild": 27345,
+ "##sance": 27346,
+ "##vyn": 27347,
+ "Digest": 27348,
+ "nuclei": 27349,
+ "typhoon": 27350,
+ "##llon": 27351,
+ "Alvarez": 27352,
+ "DJs": 27353,
+ "Grimm": 27354,
+ "authoritative": 27355,
+ "firearm": 27356,
+ "##chschule": 27357,
+ "Origins": 27358,
+ "lair": 27359,
+ "unmistakable": 27360,
+ "##xial": 27361,
+ "##cribing": 27362,
+ "Mouth": 27363,
+ "##genesis": 27364,
+ "##shū": 27365,
+ "##gaon": 27366,
+ "##ulter": 27367,
+ "Jaya": 27368,
+ "Neck": 27369,
+ "##UN": 27370,
+ "##oing": 27371,
+ "##static": 27372,
+ "relativity": 27373,
+ "##mott": 27374,
+ "##utive": 27375,
+ "##esan": 27376,
+ "##uveau": 27377,
+ "BT": 27378,
+ "salts": 27379,
+ "##roa": 27380,
+ "Dustin": 27381,
+ "preoccupied": 27382,
+ "Novgorod": 27383,
+ "##asus": 27384,
+ "Magnum": 27385,
+ "tempting": 27386,
+ "##histling": 27387,
+ "##ilated": 27388,
+ "Musa": 27389,
+ "##ghty": 27390,
+ "Ashland": 27391,
+ "pubs": 27392,
+ "routines": 27393,
+ "##etto": 27394,
+ "Soto": 27395,
+ "257": 27396,
+ "Featuring": 27397,
+ "Augsburg": 27398,
+ "##alaya": 27399,
+ "Bit": 27400,
+ "loomed": 27401,
+ "expects": 27402,
+ "##abby": 27403,
+ "##ooby": 27404,
+ "Auschwitz": 27405,
+ "Pendleton": 27406,
+ "vodka": 27407,
+ "##sent": 27408,
+ "rescuing": 27409,
+ "systemic": 27410,
+ "##inet": 27411,
+ "##leg": 27412,
+ "Yun": 27413,
+ "applicant": 27414,
+ "revered": 27415,
+ "##nacht": 27416,
+ "##ndas": 27417,
+ "Muller": 27418,
+ "characterization": 27419,
+ "##patient": 27420,
+ "##roft": 27421,
+ "Carole": 27422,
+ "##asperated": 27423,
+ "Amiga": 27424,
+ "disconnected": 27425,
+ "gel": 27426,
+ "##cologist": 27427,
+ "Patriotic": 27428,
+ "rallied": 27429,
+ "assign": 27430,
+ "veterinary": 27431,
+ "installing": 27432,
+ "##cedural": 27433,
+ "258": 27434,
+ "Jang": 27435,
+ "Parisian": 27436,
+ "incarcerated": 27437,
+ "stalk": 27438,
+ "##iment": 27439,
+ "Jamal": 27440,
+ "McPherson": 27441,
+ "Palma": 27442,
+ "##oken": 27443,
+ "##viation": 27444,
+ "512": 27445,
+ "Rourke": 27446,
+ "irrational": 27447,
+ "##rippled": 27448,
+ "Devlin": 27449,
+ "erratic": 27450,
+ "##NI": 27451,
+ "##payers": 27452,
+ "Ni": 27453,
+ "engages": 27454,
+ "Portal": 27455,
+ "aesthetics": 27456,
+ "##rrogance": 27457,
+ "Milne": 27458,
+ "assassins": 27459,
+ "##rots": 27460,
+ "335": 27461,
+ "385": 27462,
+ "Cambodian": 27463,
+ "Females": 27464,
+ "fellows": 27465,
+ "si": 27466,
+ "##block": 27467,
+ "##otes": 27468,
+ "Jayne": 27469,
+ "Toro": 27470,
+ "flutter": 27471,
+ "##eera": 27472,
+ "Burr": 27473,
+ "##lanche": 27474,
+ "relaxation": 27475,
+ "##fra": 27476,
+ "Fitzroy": 27477,
+ "##undy": 27478,
+ "1751": 27479,
+ "261": 27480,
+ "comb": 27481,
+ "conglomerate": 27482,
+ "ribbons": 27483,
+ "veto": 27484,
+ "##Es": 27485,
+ "casts": 27486,
+ "##ege": 27487,
+ "1748": 27488,
+ "Ares": 27489,
+ "spears": 27490,
+ "spirituality": 27491,
+ "comet": 27492,
+ "##nado": 27493,
+ "##yeh": 27494,
+ "Veterinary": 27495,
+ "aquarium": 27496,
+ "yer": 27497,
+ "Councils": 27498,
+ "##oked": 27499,
+ "##ynamic": 27500,
+ "Malmö": 27501,
+ "remorse": 27502,
+ "auditions": 27503,
+ "drilled": 27504,
+ "Hoffmann": 27505,
+ "Moe": 27506,
+ "Nagoya": 27507,
+ "Yacht": 27508,
+ "##hakti": 27509,
+ "##race": 27510,
+ "##rrick": 27511,
+ "Talmud": 27512,
+ "coordinating": 27513,
+ "##EI": 27514,
+ "##bul": 27515,
+ "##his": 27516,
+ "##itors": 27517,
+ "##ligent": 27518,
+ "##uerra": 27519,
+ "Narayan": 27520,
+ "goaltender": 27521,
+ "taxa": 27522,
+ "##asures": 27523,
+ "Det": 27524,
+ "##mage": 27525,
+ "Infinite": 27526,
+ "Maid": 27527,
+ "bean": 27528,
+ "intriguing": 27529,
+ "##cription": 27530,
+ "gasps": 27531,
+ "socket": 27532,
+ "##mentary": 27533,
+ "##reus": 27534,
+ "sewing": 27535,
+ "transmitting": 27536,
+ "##different": 27537,
+ "##furbishment": 27538,
+ "##traction": 27539,
+ "Grimsby": 27540,
+ "sprawling": 27541,
+ "Shipyard": 27542,
+ "##destine": 27543,
+ "##hropic": 27544,
+ "##icked": 27545,
+ "trolley": 27546,
+ "##agi": 27547,
+ "##lesh": 27548,
+ "Josiah": 27549,
+ "invasions": 27550,
+ "Content": 27551,
+ "firefighters": 27552,
+ "intro": 27553,
+ "Lucifer": 27554,
+ "subunit": 27555,
+ "Sahib": 27556,
+ "Myrtle": 27557,
+ "inhibitor": 27558,
+ "maneuvers": 27559,
+ "##teca": 27560,
+ "Wrath": 27561,
+ "slippery": 27562,
+ "##versing": 27563,
+ "Shoes": 27564,
+ "##dial": 27565,
+ "##illiers": 27566,
+ "##luded": 27567,
+ "##mmal": 27568,
+ "##pack": 27569,
+ "handkerchief": 27570,
+ "##edestal": 27571,
+ "##stones": 27572,
+ "Fusion": 27573,
+ "cumulative": 27574,
+ "##mell": 27575,
+ "##cacia": 27576,
+ "##rudge": 27577,
+ "##utz": 27578,
+ "foe": 27579,
+ "storing": 27580,
+ "swiped": 27581,
+ "##meister": 27582,
+ "##orra": 27583,
+ "batter": 27584,
+ "strung": 27585,
+ "##venting": 27586,
+ "##kker": 27587,
+ "Doo": 27588,
+ "Taste": 27589,
+ "immensely": 27590,
+ "Fairbanks": 27591,
+ "Jarrett": 27592,
+ "Boogie": 27593,
+ "1746": 27594,
+ "mage": 27595,
+ "Kick": 27596,
+ "legislators": 27597,
+ "medial": 27598,
+ "##ilon": 27599,
+ "##logies": 27600,
+ "##ranton": 27601,
+ "Hybrid": 27602,
+ "##uters": 27603,
+ "Tide": 27604,
+ "deportation": 27605,
+ "Metz": 27606,
+ "##secration": 27607,
+ "##virus": 27608,
+ "UFO": 27609,
+ "##fell": 27610,
+ "##orage": 27611,
+ "##raction": 27612,
+ "##rrigan": 27613,
+ "1747": 27614,
+ "fabricated": 27615,
+ "##BM": 27616,
+ "##GR": 27617,
+ "##rter": 27618,
+ "muttering": 27619,
+ "theorist": 27620,
+ "##tamine": 27621,
+ "BMG": 27622,
+ "Kincaid": 27623,
+ "solvent": 27624,
+ "##azed": 27625,
+ "Thin": 27626,
+ "adorable": 27627,
+ "Wendell": 27628,
+ "ta": 27629,
+ "##viour": 27630,
+ "pulses": 27631,
+ "##pologies": 27632,
+ "counters": 27633,
+ "exposition": 27634,
+ "sewer": 27635,
+ "Luciano": 27636,
+ "Clancy": 27637,
+ "##angelo": 27638,
+ "##riars": 27639,
+ "Showtime": 27640,
+ "observes": 27641,
+ "frankly": 27642,
+ "##oppy": 27643,
+ "Bergman": 27644,
+ "lobes": 27645,
+ "timetable": 27646,
+ "##bri": 27647,
+ "##uest": 27648,
+ "FX": 27649,
+ "##dust": 27650,
+ "##genus": 27651,
+ "Glad": 27652,
+ "Helmut": 27653,
+ "Meridian": 27654,
+ "##besity": 27655,
+ "##ontaine": 27656,
+ "Revue": 27657,
+ "miracles": 27658,
+ "##titis": 27659,
+ "PP": 27660,
+ "bluff": 27661,
+ "syrup": 27662,
+ "307": 27663,
+ "Messiah": 27664,
+ "##erne": 27665,
+ "interfering": 27666,
+ "picturesque": 27667,
+ "unconventional": 27668,
+ "dipping": 27669,
+ "hurriedly": 27670,
+ "Kerman": 27671,
+ "248": 27672,
+ "Ethnic": 27673,
+ "Toward": 27674,
+ "acidic": 27675,
+ "Harrisburg": 27676,
+ "##65": 27677,
+ "intimidating": 27678,
+ "##aal": 27679,
+ "Jed": 27680,
+ "Pontiac": 27681,
+ "munitions": 27682,
+ "##nchen": 27683,
+ "growling": 27684,
+ "mausoleum": 27685,
+ "##ération": 27686,
+ "##wami": 27687,
+ "Cy": 27688,
+ "aerospace": 27689,
+ "caucus": 27690,
+ "Doing": 27691,
+ "##around": 27692,
+ "##miring": 27693,
+ "Cuthbert": 27694,
+ "##poradic": 27695,
+ "##rovisation": 27696,
+ "##wth": 27697,
+ "evaluating": 27698,
+ "##scraper": 27699,
+ "Belinda": 27700,
+ "owes": 27701,
+ "##sitic": 27702,
+ "##thermal": 27703,
+ "##fast": 27704,
+ "economists": 27705,
+ "##lishing": 27706,
+ "##uerre": 27707,
+ "##ân": 27708,
+ "credible": 27709,
+ "##koto": 27710,
+ "Fourteen": 27711,
+ "cones": 27712,
+ "##ebrates": 27713,
+ "bookstore": 27714,
+ "towels": 27715,
+ "##phony": 27716,
+ "Appearance": 27717,
+ "newscasts": 27718,
+ "##olin": 27719,
+ "Karin": 27720,
+ "Bingham": 27721,
+ "##elves": 27722,
+ "1680": 27723,
+ "306": 27724,
+ "disks": 27725,
+ "##lston": 27726,
+ "##secutor": 27727,
+ "Levant": 27728,
+ "##vout": 27729,
+ "Micro": 27730,
+ "snuck": 27731,
+ "##ogel": 27732,
+ "##racker": 27733,
+ "Exploration": 27734,
+ "drastic": 27735,
+ "##kening": 27736,
+ "Elsie": 27737,
+ "endowment": 27738,
+ "##utnant": 27739,
+ "Blaze": 27740,
+ "##rrosion": 27741,
+ "leaking": 27742,
+ "45th": 27743,
+ "##rug": 27744,
+ "##uernsey": 27745,
+ "760": 27746,
+ "Shapiro": 27747,
+ "cakes": 27748,
+ "##ehan": 27749,
+ "##mei": 27750,
+ "##ité": 27751,
+ "##kla": 27752,
+ "repetition": 27753,
+ "successively": 27754,
+ "Friendly": 27755,
+ "Île": 27756,
+ "Koreans": 27757,
+ "Au": 27758,
+ "Tirana": 27759,
+ "flourish": 27760,
+ "Spirits": 27761,
+ "Yao": 27762,
+ "reasoned": 27763,
+ "##leam": 27764,
+ "Consort": 27765,
+ "cater": 27766,
+ "marred": 27767,
+ "ordeal": 27768,
+ "supremacy": 27769,
+ "##ritable": 27770,
+ "Paisley": 27771,
+ "euro": 27772,
+ "healer": 27773,
+ "portico": 27774,
+ "wetland": 27775,
+ "##kman": 27776,
+ "restart": 27777,
+ "##habilitation": 27778,
+ "##zuka": 27779,
+ "##Script": 27780,
+ "emptiness": 27781,
+ "communion": 27782,
+ "##CF": 27783,
+ "##inhabited": 27784,
+ "##wamy": 27785,
+ "Casablanca": 27786,
+ "pulsed": 27787,
+ "##rrible": 27788,
+ "##safe": 27789,
+ "395": 27790,
+ "Dual": 27791,
+ "Terrorism": 27792,
+ "##urge": 27793,
+ "##found": 27794,
+ "##gnolia": 27795,
+ "Courage": 27796,
+ "patriarch": 27797,
+ "segregated": 27798,
+ "intrinsic": 27799,
+ "##liography": 27800,
+ "##phe": 27801,
+ "PD": 27802,
+ "convection": 27803,
+ "##icidal": 27804,
+ "Dharma": 27805,
+ "Jimmie": 27806,
+ "texted": 27807,
+ "constituents": 27808,
+ "twitch": 27809,
+ "##calated": 27810,
+ "##mitage": 27811,
+ "##ringing": 27812,
+ "415": 27813,
+ "milling": 27814,
+ "##geons": 27815,
+ "Armagh": 27816,
+ "Geometridae": 27817,
+ "evergreen": 27818,
+ "needy": 27819,
+ "reflex": 27820,
+ "template": 27821,
+ "##pina": 27822,
+ "Schubert": 27823,
+ "##bruck": 27824,
+ "##icted": 27825,
+ "##scher": 27826,
+ "##wildered": 27827,
+ "1749": 27828,
+ "Joanne": 27829,
+ "clearer": 27830,
+ "##narl": 27831,
+ "278": 27832,
+ "Print": 27833,
+ "automation": 27834,
+ "consciously": 27835,
+ "flashback": 27836,
+ "occupations": 27837,
+ "##ests": 27838,
+ "Casimir": 27839,
+ "differentiated": 27840,
+ "policing": 27841,
+ "repay": 27842,
+ "##aks": 27843,
+ "##gnesium": 27844,
+ "Evaluation": 27845,
+ "commotion": 27846,
+ "##CM": 27847,
+ "##smopolitan": 27848,
+ "Clapton": 27849,
+ "mitochondrial": 27850,
+ "Kobe": 27851,
+ "1752": 27852,
+ "Ignoring": 27853,
+ "Vincenzo": 27854,
+ "Wet": 27855,
+ "bandage": 27856,
+ "##rassed": 27857,
+ "##unate": 27858,
+ "Maris": 27859,
+ "##eted": 27860,
+ "##hetical": 27861,
+ "figuring": 27862,
+ "##eit": 27863,
+ "##nap": 27864,
+ "leopard": 27865,
+ "strategically": 27866,
+ "##reer": 27867,
+ "Fen": 27868,
+ "Iain": 27869,
+ "##ggins": 27870,
+ "##pipe": 27871,
+ "Matteo": 27872,
+ "McIntyre": 27873,
+ "##chord": 27874,
+ "##feng": 27875,
+ "Romani": 27876,
+ "asshole": 27877,
+ "flopped": 27878,
+ "reassure": 27879,
+ "Founding": 27880,
+ "Styles": 27881,
+ "Torino": 27882,
+ "patrolling": 27883,
+ "##erging": 27884,
+ "##ibrating": 27885,
+ "##ructural": 27886,
+ "sincerity": 27887,
+ "##ät": 27888,
+ "##teacher": 27889,
+ "Juliette": 27890,
+ "##cé": 27891,
+ "##hog": 27892,
+ "##idated": 27893,
+ "##span": 27894,
+ "Winfield": 27895,
+ "##fender": 27896,
+ "##nast": 27897,
+ "##pliant": 27898,
+ "1690": 27899,
+ "Bai": 27900,
+ "Je": 27901,
+ "Saharan": 27902,
+ "expands": 27903,
+ "Bolshevik": 27904,
+ "rotate": 27905,
+ "##root": 27906,
+ "Britannia": 27907,
+ "Severn": 27908,
+ "##cini": 27909,
+ "##gering": 27910,
+ "##say": 27911,
+ "sly": 27912,
+ "Steps": 27913,
+ "insertion": 27914,
+ "rooftop": 27915,
+ "Piece": 27916,
+ "cuffs": 27917,
+ "plausible": 27918,
+ "##zai": 27919,
+ "Provost": 27920,
+ "semantic": 27921,
+ "##data": 27922,
+ "##vade": 27923,
+ "##cimal": 27924,
+ "IPA": 27925,
+ "indictment": 27926,
+ "Libraries": 27927,
+ "flaming": 27928,
+ "highlands": 27929,
+ "liberties": 27930,
+ "##pio": 27931,
+ "Elders": 27932,
+ "aggressively": 27933,
+ "##pecific": 27934,
+ "Decision": 27935,
+ "pigeon": 27936,
+ "nominally": 27937,
+ "descriptive": 27938,
+ "adjustments": 27939,
+ "equestrian": 27940,
+ "heaving": 27941,
+ "##mour": 27942,
+ "##dives": 27943,
+ "##fty": 27944,
+ "##yton": 27945,
+ "intermittent": 27946,
+ "##naming": 27947,
+ "##sets": 27948,
+ "Calvert": 27949,
+ "Casper": 27950,
+ "Tarzan": 27951,
+ "##kot": 27952,
+ "Ramírez": 27953,
+ "##IB": 27954,
+ "##erus": 27955,
+ "Gustavo": 27956,
+ "Roller": 27957,
+ "vaulted": 27958,
+ "##solation": 27959,
+ "##formatics": 27960,
+ "##tip": 27961,
+ "Hunger": 27962,
+ "colloquially": 27963,
+ "handwriting": 27964,
+ "hearth": 27965,
+ "launcher": 27966,
+ "##idian": 27967,
+ "##ilities": 27968,
+ "##lind": 27969,
+ "##locating": 27970,
+ "Magdalena": 27971,
+ "Soo": 27972,
+ "clubhouse": 27973,
+ "##kushima": 27974,
+ "##ruit": 27975,
+ "Bogotá": 27976,
+ "Organic": 27977,
+ "Worship": 27978,
+ "##Vs": 27979,
+ "##wold": 27980,
+ "upbringing": 27981,
+ "##kick": 27982,
+ "groundbreaking": 27983,
+ "##urable": 27984,
+ "##ván": 27985,
+ "repulsed": 27986,
+ "##dira": 27987,
+ "##ditional": 27988,
+ "##ici": 27989,
+ "melancholy": 27990,
+ "##bodied": 27991,
+ "##cchi": 27992,
+ "404": 27993,
+ "concurrency": 27994,
+ "H₂O": 27995,
+ "bouts": 27996,
+ "##gami": 27997,
+ "288": 27998,
+ "Leto": 27999,
+ "troll": 28000,
+ "##lak": 28001,
+ "advising": 28002,
+ "bundled": 28003,
+ "##nden": 28004,
+ "lipstick": 28005,
+ "littered": 28006,
+ "##leading": 28007,
+ "##mogeneous": 28008,
+ "Experiment": 28009,
+ "Nikola": 28010,
+ "grove": 28011,
+ "##ogram": 28012,
+ "Mace": 28013,
+ "##jure": 28014,
+ "cheat": 28015,
+ "Annabelle": 28016,
+ "Tori": 28017,
+ "lurking": 28018,
+ "Emery": 28019,
+ "Walden": 28020,
+ "##riz": 28021,
+ "paints": 28022,
+ "Markets": 28023,
+ "brutality": 28024,
+ "overrun": 28025,
+ "##agu": 28026,
+ "##sat": 28027,
+ "din": 28028,
+ "ostensibly": 28029,
+ "Fielding": 28030,
+ "flees": 28031,
+ "##eron": 28032,
+ "Pound": 28033,
+ "ornaments": 28034,
+ "tornadoes": 28035,
+ "##nikov": 28036,
+ "##organisation": 28037,
+ "##reen": 28038,
+ "##Works": 28039,
+ "##ldred": 28040,
+ "##olten": 28041,
+ "##stillery": 28042,
+ "soluble": 28043,
+ "Mata": 28044,
+ "Grimes": 28045,
+ "Léon": 28046,
+ "##NF": 28047,
+ "coldly": 28048,
+ "permitting": 28049,
+ "##inga": 28050,
+ "##reaked": 28051,
+ "Agents": 28052,
+ "hostess": 28053,
+ "##dl": 28054,
+ "Dyke": 28055,
+ "Kota": 28056,
+ "avail": 28057,
+ "orderly": 28058,
+ "##saur": 28059,
+ "##sities": 28060,
+ "Arroyo": 28061,
+ "##ceps": 28062,
+ "##egro": 28063,
+ "Hawke": 28064,
+ "Noctuidae": 28065,
+ "html": 28066,
+ "seminar": 28067,
+ "##ggles": 28068,
+ "##wasaki": 28069,
+ "Clube": 28070,
+ "recited": 28071,
+ "##sace": 28072,
+ "Ascension": 28073,
+ "Fitness": 28074,
+ "dough": 28075,
+ "##ixel": 28076,
+ "Nationale": 28077,
+ "##solidate": 28078,
+ "pulpit": 28079,
+ "vassal": 28080,
+ "570": 28081,
+ "Annapolis": 28082,
+ "bladder": 28083,
+ "phylogenetic": 28084,
+ "##iname": 28085,
+ "convertible": 28086,
+ "##ppan": 28087,
+ "Comet": 28088,
+ "paler": 28089,
+ "##definite": 28090,
+ "Spot": 28091,
+ "##dices": 28092,
+ "frequented": 28093,
+ "Apostles": 28094,
+ "slalom": 28095,
+ "##ivision": 28096,
+ "##mana": 28097,
+ "##runcated": 28098,
+ "Trojan": 28099,
+ "##agger": 28100,
+ "##iq": 28101,
+ "##league": 28102,
+ "Concept": 28103,
+ "Controller": 28104,
+ "##barian": 28105,
+ "##curate": 28106,
+ "##spersed": 28107,
+ "##tring": 28108,
+ "engulfed": 28109,
+ "inquired": 28110,
+ "##hmann": 28111,
+ "286": 28112,
+ "##dict": 28113,
+ "##osy": 28114,
+ "##raw": 28115,
+ "MacKenzie": 28116,
+ "su": 28117,
+ "##ienced": 28118,
+ "##iggs": 28119,
+ "##quitaine": 28120,
+ "bisexual": 28121,
+ "##noon": 28122,
+ "runways": 28123,
+ "subsp": 28124,
+ "##!": 28125,
+ "##\"": 28126,
+ "###": 28127,
+ "##$": 28128,
+ "##%": 28129,
+ "##&": 28130,
+ "##'": 28131,
+ "##(": 28132,
+ "##)": 28133,
+ "##*": 28134,
+ "##+": 28135,
+ "##,": 28136,
+ "##-": 28137,
+ "##.": 28138,
+ "##/": 28139,
+ "##:": 28140,
+ "##;": 28141,
+ "##<": 28142,
+ "##=": 28143,
+ "##>": 28144,
+ "##?": 28145,
+ "##@": 28146,
+ "##[": 28147,
+ "##\\": 28148,
+ "##]": 28149,
+ "##^": 28150,
+ "##_": 28151,
+ "##`": 28152,
+ "##{": 28153,
+ "##|": 28154,
+ "##}": 28155,
+ "##~": 28156,
+ "##¡": 28157,
+ "##¢": 28158,
+ "##£": 28159,
+ "##¥": 28160,
+ "##§": 28161,
+ "##¨": 28162,
+ "##©": 28163,
+ "##ª": 28164,
+ "##«": 28165,
+ "##¬": 28166,
+ "##®": 28167,
+ "##±": 28168,
+ "##´": 28169,
+ "##µ": 28170,
+ "##¶": 28171,
+ "##·": 28172,
+ "##¹": 28173,
+ "##º": 28174,
+ "##»": 28175,
+ "##¼": 28176,
+ "##¾": 28177,
+ "##¿": 28178,
+ "##À": 28179,
+ "##Á": 28180,
+ "##Â": 28181,
+ "##Ä": 28182,
+ "##Å": 28183,
+ "##Æ": 28184,
+ "##Ç": 28185,
+ "##È": 28186,
+ "##É": 28187,
+ "##Í": 28188,
+ "##Î": 28189,
+ "##Ñ": 28190,
+ "##Ó": 28191,
+ "##Ö": 28192,
+ "##×": 28193,
+ "##Ø": 28194,
+ "##Ú": 28195,
+ "##Ü": 28196,
+ "##Þ": 28197,
+ "##â": 28198,
+ "##ã": 28199,
+ "##æ": 28200,
+ "##ç": 28201,
+ "##î": 28202,
+ "##ï": 28203,
+ "##ð": 28204,
+ "##ñ": 28205,
+ "##ô": 28206,
+ "##õ": 28207,
+ "##÷": 28208,
+ "##û": 28209,
+ "##þ": 28210,
+ "##ÿ": 28211,
+ "##Ā": 28212,
+ "##ą": 28213,
+ "##Ć": 28214,
+ "##Č": 28215,
+ "##ď": 28216,
+ "##Đ": 28217,
+ "##đ": 28218,
+ "##ē": 28219,
+ "##ė": 28220,
+ "##ę": 28221,
+ "##ě": 28222,
+ "##ğ": 28223,
+ "##ġ": 28224,
+ "##Ħ": 28225,
+ "##ħ": 28226,
+ "##ĩ": 28227,
+ "##Ī": 28228,
+ "##İ": 28229,
+ "##ļ": 28230,
+ "##Ľ": 28231,
+ "##ľ": 28232,
+ "##Ł": 28233,
+ "##ņ": 28234,
+ "##ň": 28235,
+ "##ŋ": 28236,
+ "##Ō": 28237,
+ "##ŏ": 28238,
+ "##ő": 28239,
+ "##Œ": 28240,
+ "##œ": 28241,
+ "##ř": 28242,
+ "##Ś": 28243,
+ "##ś": 28244,
+ "##Ş": 28245,
+ "##Š": 28246,
+ "##Ţ": 28247,
+ "##ţ": 28248,
+ "##ť": 28249,
+ "##ũ": 28250,
+ "##ŭ": 28251,
+ "##ů": 28252,
+ "##ű": 28253,
+ "##ų": 28254,
+ "##ŵ": 28255,
+ "##ŷ": 28256,
+ "##ź": 28257,
+ "##Ż": 28258,
+ "##ż": 28259,
+ "##Ž": 28260,
+ "##ž": 28261,
+ "##Ə": 28262,
+ "##ƒ": 28263,
+ "##ơ": 28264,
+ "##ư": 28265,
+ "##ǎ": 28266,
+ "##ǐ": 28267,
+ "##ǒ": 28268,
+ "##ǔ": 28269,
+ "##ǫ": 28270,
+ "##Ș": 28271,
+ "##Ț": 28272,
+ "##ț": 28273,
+ "##ɐ": 28274,
+ "##ɑ": 28275,
+ "##ɔ": 28276,
+ "##ɕ": 28277,
+ "##ə": 28278,
+ "##ɛ": 28279,
+ "##ɡ": 28280,
+ "##ɣ": 28281,
+ "##ɨ": 28282,
+ "##ɪ": 28283,
+ "##ɲ": 28284,
+ "##ɾ": 28285,
+ "##ʀ": 28286,
+ "##ʁ": 28287,
+ "##ʂ": 28288,
+ "##ʃ": 28289,
+ "##ʊ": 28290,
+ "##ʋ": 28291,
+ "##ʌ": 28292,
+ "##ʐ": 28293,
+ "##ʑ": 28294,
+ "##ʒ": 28295,
+ "##ʔ": 28296,
+ "##ʰ": 28297,
+ "##ʲ": 28298,
+ "##ʳ": 28299,
+ "##ʷ": 28300,
+ "##ʻ": 28301,
+ "##ʼ": 28302,
+ "##ʾ": 28303,
+ "##ʿ": 28304,
+ "##ˈ": 28305,
+ "##ː": 28306,
+ "##ˡ": 28307,
+ "##ˢ": 28308,
+ "##ˣ": 28309,
+ "##́": 28310,
+ "##̃": 28311,
+ "##̍": 28312,
+ "##̯": 28313,
+ "##͡": 28314,
+ "##Α": 28315,
+ "##Β": 28316,
+ "##Γ": 28317,
+ "##Δ": 28318,
+ "##Ε": 28319,
+ "##Η": 28320,
+ "##Θ": 28321,
+ "##Ι": 28322,
+ "##Κ": 28323,
+ "##Λ": 28324,
+ "##Μ": 28325,
+ "##Ν": 28326,
+ "##Ο": 28327,
+ "##Π": 28328,
+ "##Σ": 28329,
+ "##Τ": 28330,
+ "##Φ": 28331,
+ "##Χ": 28332,
+ "##Ψ": 28333,
+ "##Ω": 28334,
+ "##ά": 28335,
+ "##έ": 28336,
+ "##ή": 28337,
+ "##ί": 28338,
+ "##β": 28339,
+ "##γ": 28340,
+ "##δ": 28341,
+ "##ε": 28342,
+ "##ζ": 28343,
+ "##η": 28344,
+ "##θ": 28345,
+ "##ι": 28346,
+ "##κ": 28347,
+ "##λ": 28348,
+ "##μ": 28349,
+ "##ξ": 28350,
+ "##ο": 28351,
+ "##π": 28352,
+ "##ρ": 28353,
+ "##σ": 28354,
+ "##τ": 28355,
+ "##υ": 28356,
+ "##φ": 28357,
+ "##χ": 28358,
+ "##ψ": 28359,
+ "##ω": 28360,
+ "##ό": 28361,
+ "##ύ": 28362,
+ "##ώ": 28363,
+ "##І": 28364,
+ "##Ј": 28365,
+ "##А": 28366,
+ "##Б": 28367,
+ "##В": 28368,
+ "##Г": 28369,
+ "##Д": 28370,
+ "##Е": 28371,
+ "##Ж": 28372,
+ "##З": 28373,
+ "##И": 28374,
+ "##К": 28375,
+ "##Л": 28376,
+ "##М": 28377,
+ "##Н": 28378,
+ "##О": 28379,
+ "##П": 28380,
+ "##Р": 28381,
+ "##С": 28382,
+ "##Т": 28383,
+ "##У": 28384,
+ "##Ф": 28385,
+ "##Х": 28386,
+ "##Ц": 28387,
+ "##Ч": 28388,
+ "##Ш": 28389,
+ "##Э": 28390,
+ "##Ю": 28391,
+ "##Я": 28392,
+ "##б": 28393,
+ "##в": 28394,
+ "##г": 28395,
+ "##д": 28396,
+ "##ж": 28397,
+ "##з": 28398,
+ "##к": 28399,
+ "##л": 28400,
+ "##м": 28401,
+ "##п": 28402,
+ "##с": 28403,
+ "##т": 28404,
+ "##у": 28405,
+ "##ф": 28406,
+ "##х": 28407,
+ "##ц": 28408,
+ "##ч": 28409,
+ "##ш": 28410,
+ "##щ": 28411,
+ "##ъ": 28412,
+ "##ы": 28413,
+ "##ь": 28414,
+ "##э": 28415,
+ "##ю": 28416,
+ "##ё": 28417,
+ "##і": 28418,
+ "##ї": 28419,
+ "##ј": 28420,
+ "##њ": 28421,
+ "##ћ": 28422,
+ "##Ա": 28423,
+ "##Հ": 28424,
+ "##ա": 28425,
+ "##ե": 28426,
+ "##ի": 28427,
+ "##կ": 28428,
+ "##մ": 28429,
+ "##յ": 28430,
+ "##ն": 28431,
+ "##ո": 28432,
+ "##ս": 28433,
+ "##տ": 28434,
+ "##ր": 28435,
+ "##ւ": 28436,
+ "##ְ": 28437,
+ "##ִ": 28438,
+ "##ֵ": 28439,
+ "##ֶ": 28440,
+ "##ַ": 28441,
+ "##ָ": 28442,
+ "##ֹ": 28443,
+ "##ּ": 28444,
+ "##א": 28445,
+ "##ב": 28446,
+ "##ג": 28447,
+ "##ד": 28448,
+ "##ה": 28449,
+ "##ו": 28450,
+ "##ז": 28451,
+ "##ח": 28452,
+ "##ט": 28453,
+ "##י": 28454,
+ "##כ": 28455,
+ "##ל": 28456,
+ "##ם": 28457,
+ "##מ": 28458,
+ "##ן": 28459,
+ "##נ": 28460,
+ "##ס": 28461,
+ "##ע": 28462,
+ "##פ": 28463,
+ "##צ": 28464,
+ "##ק": 28465,
+ "##ר": 28466,
+ "##ש": 28467,
+ "##ת": 28468,
+ "##،": 28469,
+ "##ء": 28470,
+ "##آ": 28471,
+ "##أ": 28472,
+ "##إ": 28473,
+ "##ئ": 28474,
+ "##ا": 28475,
+ "##ب": 28476,
+ "##ت": 28477,
+ "##ث": 28478,
+ "##ج": 28479,
+ "##ح": 28480,
+ "##خ": 28481,
+ "##ذ": 28482,
+ "##ز": 28483,
+ "##س": 28484,
+ "##ش": 28485,
+ "##ص": 28486,
+ "##ض": 28487,
+ "##ط": 28488,
+ "##ظ": 28489,
+ "##ع": 28490,
+ "##غ": 28491,
+ "##ف": 28492,
+ "##ق": 28493,
+ "##ك": 28494,
+ "##ل": 28495,
+ "##و": 28496,
+ "##ى": 28497,
+ "##َ": 28498,
+ "##ِ": 28499,
+ "##ٹ": 28500,
+ "##پ": 28501,
+ "##چ": 28502,
+ "##ک": 28503,
+ "##گ": 28504,
+ "##ہ": 28505,
+ "##ی": 28506,
+ "##ے": 28507,
+ "##ं": 28508,
+ "##आ": 28509,
+ "##क": 28510,
+ "##ग": 28511,
+ "##च": 28512,
+ "##ज": 28513,
+ "##ण": 28514,
+ "##त": 28515,
+ "##द": 28516,
+ "##ध": 28517,
+ "##न": 28518,
+ "##प": 28519,
+ "##ब": 28520,
+ "##भ": 28521,
+ "##म": 28522,
+ "##य": 28523,
+ "##र": 28524,
+ "##ल": 28525,
+ "##व": 28526,
+ "##श": 28527,
+ "##ष": 28528,
+ "##स": 28529,
+ "##ह": 28530,
+ "##ा": 28531,
+ "##ि": 28532,
+ "##ी": 28533,
+ "##ु": 28534,
+ "##े": 28535,
+ "##ो": 28536,
+ "##्": 28537,
+ "##।": 28538,
+ "##॥": 28539,
+ "##আ": 28540,
+ "##ই": 28541,
+ "##এ": 28542,
+ "##ও": 28543,
+ "##ক": 28544,
+ "##খ": 28545,
+ "##গ": 28546,
+ "##চ": 28547,
+ "##ছ": 28548,
+ "##জ": 28549,
+ "##ট": 28550,
+ "##ত": 28551,
+ "##থ": 28552,
+ "##দ": 28553,
+ "##ধ": 28554,
+ "##ন": 28555,
+ "##প": 28556,
+ "##ব": 28557,
+ "##ম": 28558,
+ "##য": 28559,
+ "##র": 28560,
+ "##ল": 28561,
+ "##শ": 28562,
+ "##স": 28563,
+ "##হ": 28564,
+ "##়": 28565,
+ "##া": 28566,
+ "##ি": 28567,
+ "##ী": 28568,
+ "##ু": 28569,
+ "##ে": 28570,
+ "##ো": 28571,
+ "##্": 28572,
+ "##য়": 28573,
+ "##க": 28574,
+ "##த": 28575,
+ "##ப": 28576,
+ "##ம": 28577,
+ "##ய": 28578,
+ "##ர": 28579,
+ "##ல": 28580,
+ "##வ": 28581,
+ "##ா": 28582,
+ "##ி": 28583,
+ "##ு": 28584,
+ "##்": 28585,
+ "##ร": 28586,
+ "##་": 28587,
+ "##ག": 28588,
+ "##ང": 28589,
+ "##ད": 28590,
+ "##ན": 28591,
+ "##བ": 28592,
+ "##མ": 28593,
+ "##ར": 28594,
+ "##ལ": 28595,
+ "##ས": 28596,
+ "##ི": 28597,
+ "##ུ": 28598,
+ "##ེ": 28599,
+ "##ོ": 28600,
+ "##ა": 28601,
+ "##ე": 28602,
+ "##ი": 28603,
+ "##ლ": 28604,
+ "##ნ": 28605,
+ "##ო": 28606,
+ "##რ": 28607,
+ "##ს": 28608,
+ "##ᴬ": 28609,
+ "##ᴵ": 28610,
+ "##ᵀ": 28611,
+ "##ᵃ": 28612,
+ "##ᵇ": 28613,
+ "##ᵈ": 28614,
+ "##ᵉ": 28615,
+ "##ᵍ": 28616,
+ "##ᵏ": 28617,
+ "##ᵐ": 28618,
+ "##ᵒ": 28619,
+ "##ᵖ": 28620,
+ "##ᵗ": 28621,
+ "##ᵘ": 28622,
+ "##ᵣ": 28623,
+ "##ᵤ": 28624,
+ "##ᵥ": 28625,
+ "##ᶜ": 28626,
+ "##ᶠ": 28627,
+ "##ḍ": 28628,
+ "##Ḥ": 28629,
+ "##ḥ": 28630,
+ "##Ḩ": 28631,
+ "##ḩ": 28632,
+ "##ḳ": 28633,
+ "##ṃ": 28634,
+ "##ṅ": 28635,
+ "##ṇ": 28636,
+ "##ṛ": 28637,
+ "##ṣ": 28638,
+ "##ṭ": 28639,
+ "##ạ": 28640,
+ "##ả": 28641,
+ "##ấ": 28642,
+ "##ầ": 28643,
+ "##ẩ": 28644,
+ "##ậ": 28645,
+ "##ắ": 28646,
+ "##ế": 28647,
+ "##ề": 28648,
+ "##ể": 28649,
+ "##ễ": 28650,
+ "##ệ": 28651,
+ "##ị": 28652,
+ "##ọ": 28653,
+ "##ố": 28654,
+ "##ồ": 28655,
+ "##ổ": 28656,
+ "##ộ": 28657,
+ "##ớ": 28658,
+ "##ờ": 28659,
+ "##ợ": 28660,
+ "##ụ": 28661,
+ "##ủ": 28662,
+ "##ứ": 28663,
+ "##ừ": 28664,
+ "##ử": 28665,
+ "##ữ": 28666,
+ "##ự": 28667,
+ "##ỳ": 28668,
+ "##ỹ": 28669,
+ "##ἀ": 28670,
+ "##ἐ": 28671,
+ "##ὁ": 28672,
+ "##ὐ": 28673,
+ "##ὰ": 28674,
+ "##ὶ": 28675,
+ "##ὸ": 28676,
+ "##ῆ": 28677,
+ "##ῖ": 28678,
+ "##ῦ": 28679,
+ "##ῶ": 28680,
+ "##‐": 28681,
+ "##‑": 28682,
+ "##‒": 28683,
+ "##–": 28684,
+ "##—": 28685,
+ "##―": 28686,
+ "##‖": 28687,
+ "##‘": 28688,
+ "##’": 28689,
+ "##‚": 28690,
+ "##“": 28691,
+ "##”": 28692,
+ "##„": 28693,
+ "##†": 28694,
+ "##‡": 28695,
+ "##•": 28696,
+ "##…": 28697,
+ "##‰": 28698,
+ "##′": 28699,
+ "##″": 28700,
+ "##⁄": 28701,
+ "##⁰": 28702,
+ "##ⁱ": 28703,
+ "##⁴": 28704,
+ "##⁵": 28705,
+ "##⁶": 28706,
+ "##⁷": 28707,
+ "##⁸": 28708,
+ "##⁹": 28709,
+ "##⁻": 28710,
+ "##ⁿ": 28711,
+ "##₅": 28712,
+ "##₆": 28713,
+ "##₇": 28714,
+ "##₈": 28715,
+ "##₉": 28716,
+ "##₊": 28717,
+ "##₍": 28718,
+ "##₎": 28719,
+ "##ₐ": 28720,
+ "##ₑ": 28721,
+ "##ₒ": 28722,
+ "##ₓ": 28723,
+ "##ₕ": 28724,
+ "##ₖ": 28725,
+ "##ₘ": 28726,
+ "##ₚ": 28727,
+ "##ₛ": 28728,
+ "##ₜ": 28729,
+ "##₤": 28730,
+ "##€": 28731,
+ "##₱": 28732,
+ "##₹": 28733,
+ "##ℓ": 28734,
+ "##№": 28735,
+ "##ℝ": 28736,
+ "##⅓": 28737,
+ "##←": 28738,
+ "##↑": 28739,
+ "##→": 28740,
+ "##↔": 28741,
+ "##⇌": 28742,
+ "##⇒": 28743,
+ "##∂": 28744,
+ "##∈": 28745,
+ "##∗": 28746,
+ "##∘": 28747,
+ "##√": 28748,
+ "##∞": 28749,
+ "##∧": 28750,
+ "##∨": 28751,
+ "##∩": 28752,
+ "##∪": 28753,
+ "##≈": 28754,
+ "##≠": 28755,
+ "##≡": 28756,
+ "##≤": 28757,
+ "##≥": 28758,
+ "##⊂": 28759,
+ "##⊆": 28760,
+ "##⊕": 28761,
+ "##⋅": 28762,
+ "##─": 28763,
+ "##│": 28764,
+ "##■": 28765,
+ "##●": 28766,
+ "##★": 28767,
+ "##☆": 28768,
+ "##☉": 28769,
+ "##♠": 28770,
+ "##♣": 28771,
+ "##♥": 28772,
+ "##♦": 28773,
+ "##♯": 28774,
+ "##⟨": 28775,
+ "##⟩": 28776,
+ "##ⱼ": 28777,
+ "##、": 28778,
+ "##。": 28779,
+ "##《": 28780,
+ "##》": 28781,
+ "##「": 28782,
+ "##」": 28783,
+ "##『": 28784,
+ "##』": 28785,
+ "##〜": 28786,
+ "##い": 28787,
+ "##う": 28788,
+ "##え": 28789,
+ "##お": 28790,
+ "##か": 28791,
+ "##き": 28792,
+ "##く": 28793,
+ "##け": 28794,
+ "##こ": 28795,
+ "##さ": 28796,
+ "##し": 28797,
+ "##す": 28798,
+ "##せ": 28799,
+ "##そ": 28800,
+ "##た": 28801,
+ "##ち": 28802,
+ "##つ": 28803,
+ "##て": 28804,
+ "##と": 28805,
+ "##な": 28806,
+ "##に": 28807,
+ "##の": 28808,
+ "##は": 28809,
+ "##ひ": 28810,
+ "##ま": 28811,
+ "##み": 28812,
+ "##む": 28813,
+ "##め": 28814,
+ "##も": 28815,
+ "##や": 28816,
+ "##ゆ": 28817,
+ "##よ": 28818,
+ "##ら": 28819,
+ "##り": 28820,
+ "##る": 28821,
+ "##れ": 28822,
+ "##ん": 28823,
+ "##ア": 28824,
+ "##ィ": 28825,
+ "##イ": 28826,
+ "##ウ": 28827,
+ "##エ": 28828,
+ "##オ": 28829,
+ "##カ": 28830,
+ "##ガ": 28831,
+ "##キ": 28832,
+ "##ク": 28833,
+ "##グ": 28834,
+ "##コ": 28835,
+ "##サ": 28836,
+ "##シ": 28837,
+ "##ジ": 28838,
+ "##ス": 28839,
+ "##ズ": 28840,
+ "##タ": 28841,
+ "##ダ": 28842,
+ "##ッ": 28843,
+ "##テ": 28844,
+ "##デ": 28845,
+ "##ト": 28846,
+ "##ド": 28847,
+ "##ナ": 28848,
+ "##ニ": 28849,
+ "##ハ": 28850,
+ "##バ": 28851,
+ "##パ": 28852,
+ "##フ": 28853,
+ "##ブ": 28854,
+ "##プ": 28855,
+ "##マ": 28856,
+ "##ミ": 28857,
+ "##ム": 28858,
+ "##ャ": 28859,
+ "##ュ": 28860,
+ "##ラ": 28861,
+ "##リ": 28862,
+ "##ル": 28863,
+ "##レ": 28864,
+ "##ロ": 28865,
+ "##ン": 28866,
+ "##・": 28867,
+ "##ー": 28868,
+ "##一": 28869,
+ "##三": 28870,
+ "##上": 28871,
+ "##下": 28872,
+ "##中": 28873,
+ "##事": 28874,
+ "##二": 28875,
+ "##井": 28876,
+ "##京": 28877,
+ "##人": 28878,
+ "##亻": 28879,
+ "##仁": 28880,
+ "##佐": 28881,
+ "##侍": 28882,
+ "##光": 28883,
+ "##公": 28884,
+ "##力": 28885,
+ "##北": 28886,
+ "##十": 28887,
+ "##南": 28888,
+ "##原": 28889,
+ "##口": 28890,
+ "##史": 28891,
+ "##司": 28892,
+ "##吉": 28893,
+ "##同": 28894,
+ "##和": 28895,
+ "##囗": 28896,
+ "##国": 28897,
+ "##國": 28898,
+ "##土": 28899,
+ "##城": 28900,
+ "##士": 28901,
+ "##大": 28902,
+ "##天": 28903,
+ "##太": 28904,
+ "##夫": 28905,
+ "##女": 28906,
+ "##子": 28907,
+ "##宀": 28908,
+ "##安": 28909,
+ "##宮": 28910,
+ "##宿": 28911,
+ "##小": 28912,
+ "##尚": 28913,
+ "##山": 28914,
+ "##島": 28915,
+ "##川": 28916,
+ "##州": 28917,
+ "##平": 28918,
+ "##年": 28919,
+ "##心": 28920,
+ "##愛": 28921,
+ "##戸": 28922,
+ "##文": 28923,
+ "##新": 28924,
+ "##方": 28925,
+ "##日": 28926,
+ "##明": 28927,
+ "##星": 28928,
+ "##書": 28929,
+ "##月": 28930,
+ "##木": 28931,
+ "##本": 28932,
+ "##李": 28933,
+ "##村": 28934,
+ "##東": 28935,
+ "##松": 28936,
+ "##林": 28937,
+ "##正": 28938,
+ "##武": 28939,
+ "##氏": 28940,
+ "##水": 28941,
+ "##氵": 28942,
+ "##江": 28943,
+ "##河": 28944,
+ "##海": 28945,
+ "##版": 28946,
+ "##犬": 28947,
+ "##王": 28948,
+ "##生": 28949,
+ "##田": 28950,
+ "##白": 28951,
+ "##皇": 28952,
+ "##省": 28953,
+ "##真": 28954,
+ "##石": 28955,
+ "##社": 28956,
+ "##神": 28957,
+ "##竹": 28958,
+ "##美": 28959,
+ "##義": 28960,
+ "##花": 28961,
+ "##藤": 28962,
+ "##西": 28963,
+ "##谷": 28964,
+ "##車": 28965,
+ "##辶": 28966,
+ "##道": 28967,
+ "##郎": 28968,
+ "##郡": 28969,
+ "##部": 28970,
+ "##野": 28971,
+ "##金": 28972,
+ "##長": 28973,
+ "##門": 28974,
+ "##陽": 28975,
+ "##青": 28976,
+ "##食": 28977,
+ "##馬": 28978,
+ "##高": 28979,
+ "##龍": 28980,
+ "##龸": 28981,
+ "##사": 28982,
+ "##씨": 28983,
+ "##의": 28984,
+ "##이": 28985,
+ "##한": 28986,
+ "##fi": 28987,
+ "##fl": 28988,
+ "##！": 28989,
+ "##（": 28990,
+ "##）": 28991,
+ "##，": 28992,
+ "##－": 28993,
+ "##／": 28994,
+ "##：": 28995
+ }
+ }
+}
\ No newline at end of file
diff --git a/darkbert-main/darkbert-main/tokenizer_config.json b/darkbert-main/darkbert-main/tokenizer_config.json
new file mode 100644
index 0000000000000000000000000000000000000000..6e26b925c76663726bdaefebe6400e8c570f52c0
--- /dev/null
+++ b/darkbert-main/darkbert-main/tokenizer_config.json
@@ -0,0 +1,14 @@
+{
+ "cls_token": "[CLS]",
+ "do_lower_case": false,
+ "mask_token": "[MASK]",
+ "model_max_length": 512,
+ "name_or_path": "bert-base-cased",
+ "pad_token": "[PAD]",
+ "sep_token": "[SEP]",
+ "special_tokens_map_file": null,
+ "strip_accents": null,
+ "tokenize_chinese_chars": true,
+ "tokenizer_class": "BertTokenizer",
+ "unk_token": "[UNK]"
+}
diff --git a/darkbert-main/darkbert-main/training_args.bin b/darkbert-main/darkbert-main/training_args.bin
new file mode 100644
index 0000000000000000000000000000000000000000..fde86cdfe8b4ecba60b4a6550eaa43092d28f1f5
--- /dev/null
+++ b/darkbert-main/darkbert-main/training_args.bin
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4ea86c3e8509c5fef6d6b91a39c5f8e73fb0cc9bca85d28f63529a95d83a912
+size 129
diff --git a/darkbert-main/darkbert-main/vocab.txt b/darkbert-main/darkbert-main/vocab.txt
new file mode 100644
index 0000000000000000000000000000000000000000..2ea941cc79a6f3d7985ca6991ef4f67dad62af04
--- /dev/null
+++ b/darkbert-main/darkbert-main/vocab.txt
@@ -0,0 +1,28996 @@
+[PAD]
+[unused1]
+[unused2]
+[unused3]
+[unused4]
+[unused5]
+[unused6]
+[unused7]
+[unused8]
+[unused9]
+[unused10]
+[unused11]
+[unused12]
+[unused13]
+[unused14]
+[unused15]
+[unused16]
+[unused17]
+[unused18]
+[unused19]
+[unused20]
+[unused21]
+[unused22]
+[unused23]
+[unused24]
+[unused25]
+[unused26]
+[unused27]
+[unused28]
+[unused29]
+[unused30]
+[unused31]
+[unused32]
+[unused33]
+[unused34]
+[unused35]
+[unused36]
+[unused37]
+[unused38]
+[unused39]
+[unused40]
+[unused41]
+[unused42]
+[unused43]
+[unused44]
+[unused45]
+[unused46]
+[unused47]
+[unused48]
+[unused49]
+[unused50]
+[unused51]
+[unused52]
+[unused53]
+[unused54]
+[unused55]
+[unused56]
+[unused57]
+[unused58]
+[unused59]
+[unused60]
+[unused61]
+[unused62]
+[unused63]
+[unused64]
+[unused65]
+[unused66]
+[unused67]
+[unused68]
+[unused69]
+[unused70]
+[unused71]
+[unused72]
+[unused73]
+[unused74]
+[unused75]
+[unused76]
+[unused77]
+[unused78]
+[unused79]
+[unused80]
+[unused81]
+[unused82]
+[unused83]
+[unused84]
+[unused85]
+[unused86]
+[unused87]
+[unused88]
+[unused89]
+[unused90]
+[unused91]
+[unused92]
+[unused93]
+[unused94]
+[unused95]
+[unused96]
+[unused97]
+[unused98]
+[unused99]
+[UNK]
+[CLS]
+[SEP]
+[MASK]
+[unused100]
+[unused101]
+!
+"
+#
+$
+%
+&
+'
+(
+)
+*
++
+,
+-
+.
+/
+0
+1
+2
+3
+4
+5
+6
+7
+8
+9
+:
+;
+<
+=
+>
+?
+@
+A
+B
+C
+D
+E
+F
+G
+H
+I
+J
+K
+L
+M
+N
+O
+P
+Q
+R
+S
+T
+U
+V
+W
+X
+Y
+Z
+[
+\
+]
+^
+_
+`
+a
+b
+c
+d
+e
+f
+g
+h
+i
+j
+k
+l
+m
+n
+o
+p
+q
+r
+s
+t
+u
+v
+w
+x
+y
+z
+{
+|
+}
+~
+¡
+¢
+£
+¥
+§
+¨
+©
+ª
+«
+¬
+®
+°
+±
+²
+³
+´
+µ
+¶
+·
+¹
+º
+»
+¼
+½
+¾
+¿
+À
+Á
+Â
+Ä
+Å
+Æ
+Ç
+È
+É
+Í
+Î
+Ñ
+Ó
+Ö
+×
+Ø
+Ú
+Ü
+Þ
+ß
+à
+á
+â
+ã
+ä
+å
+æ
+ç
+è
+é
+ê
+ë
+ì
+í
+î
+ï
+ð
+ñ
+ò
+ó
+ô
+õ
+ö
+÷
+ø
+ù
+ú
+û
+ü
+ý
+þ
+ÿ
+Ā
+ā
+ă
+ą
+Ć
+ć
+Č
+č
+ď
+Đ
+đ
+ē
+ė
+ę
+ě
+ğ
+ġ
+Ħ
+ħ
+ĩ
+Ī
+ī
+İ
+ı
+ļ
+Ľ
+ľ
+Ł
+ł
+ń
+ņ
+ň
+ŋ
+Ō
+ō
+ŏ
+ő
+Œ
+œ
+ř
+Ś
+ś
+Ş
+ş
+Š
+š
+Ţ
+ţ
+ť
+ũ
+ū
+ŭ
+ů
+ű
+ų
+ŵ
+ŷ
+ź
+Ż
+ż
+Ž
+ž
+Ə
+ƒ
+ơ
+ư
+ǎ
+ǐ
+ǒ
+ǔ
+ǫ
+Ș
+ș
+Ț
+ț
+ɐ
+ɑ
+ɔ
+ɕ
+ə
+ɛ
+ɡ
+ɣ
+ɨ
+ɪ
+ɲ
+ɾ
+ʀ
+ʁ
+ʂ
+ʃ
+ʊ
+ʋ
+ʌ
+ʐ
+ʑ
+ʒ
+ʔ
+ʰ
+ʲ
+ʳ
+ʷ
+ʻ
+ʼ
+ʾ
+ʿ
+ˈ
+ː
+ˡ
+ˢ
+ˣ
+́
+̃
+̍
+̯
+͡
+Α
+Β
+Γ
+Δ
+Ε
+Η
+Θ
+Ι
+Κ
+Λ
+Μ
+Ν
+Ο
+Π
+Σ
+Τ
+Φ
+Χ
+Ψ
+Ω
+ά
+έ
+ή
+ί
+α
+β
+γ
+δ
+ε
+ζ
+η
+θ
+ι
+κ
+λ
+μ
+ν
+ξ
+ο
+π
+ρ
+ς
+σ
+τ
+υ
+φ
+χ
+ψ
+ω
+ό
+ύ
+ώ
+І
+Ј
+А
+Б
+В
+Г
+Д
+Е
+Ж
+З
+И
+К
+Л
+М
+Н
+О
+П
+Р
+С
+Т
+У
+Ф
+Х
+Ц
+Ч
+Ш
+Э
+Ю
+Я
+а
+б
+в
+г
+д
+е
+ж
+з
+и
+й
+к
+л
+м
+н
+о
+п
+р
+с
+т
+у
+ф
+х
+ц
+ч
+ш
+щ
+ъ
+ы
+ь
+э
+ю
+я
+ё
+і
+ї
+ј
+њ
+ћ
+Ա
+Հ
+ա
+ե
+ի
+կ
+մ
+յ
+ն
+ո
+ս
+տ
+ր
+ւ
+ְ
+ִ
+ֵ
+ֶ
+ַ
+ָ
+ֹ
+ּ
+א
+ב
+ג
+ד
+ה
+ו
+ז
+ח
+ט
+י
+כ
+ל
+ם
+מ
+ן
+נ
+ס
+ע
+פ
+צ
+ק
+ר
+ש
+ת
+،
+ء
+آ
+أ
+إ
+ئ
+ا
+ب
+ة
+ت
+ث
+ج
+ح
+خ
+د
+ذ
+ر
+ز
+س
+ش
+ص
+ض
+ط
+ظ
+ع
+غ
+ف
+ق
+ك
+ل
+م
+ن
+ه
+و
+ى
+ي
+َ
+ِ
+ٹ
+پ
+چ
+ک
+گ
+ہ
+ی
+ے
+ं
+आ
+क
+ग
+च
+ज
+ण
+त
+द
+ध
+न
+प
+ब
+भ
+म
+य
+र
+ल
+व
+श
+ष
+स
+ह
+ा
+ि
+ी
+ु
+े
+ो
+्
+।
+॥
+আ
+ই
+এ
+ও
+ক
+খ
+গ
+চ
+ছ
+জ
+ট
+ত
+থ
+দ
+ধ
+ন
+প
+ব
+ম
+য
+র
+ল
+শ
+স
+হ
+়
+া
+ি
+ী
+ু
+ে
+ো
+্
+য়
+க
+த
+ப
+ம
+ய
+ர
+ல
+வ
+ா
+ி
+ு
+்
+ร
+་
+ག
+ང
+ད
+ན
+བ
+མ
+ར
+ལ
+ས
+ི
+ུ
+ེ
+ོ
+ა
+ე
+ი
+ლ
+ნ
+ო
+რ
+ს
+ᴬ
+ᴵ
+ᵀ
+ᵃ
+ᵇ
+ᵈ
+ᵉ
+ᵍ
+ᵏ
+ᵐ
+ᵒ
+ᵖ
+ᵗ
+ᵘ
+ᵢ
+ᵣ
+ᵤ
+ᵥ
+ᶜ
+ᶠ
+ḍ
+Ḥ
+ḥ
+Ḩ
+ḩ
+ḳ
+ṃ
+ṅ
+ṇ
+ṛ
+ṣ
+ṭ
+ạ
+ả
+ấ
+ầ
+ẩ
+ậ
+ắ
+ế
+ề
+ể
+ễ
+ệ
+ị
+ọ
+ố
+ồ
+ổ
+ộ
+ớ
+ờ
+ợ
+ụ
+ủ
+ứ
+ừ
+ử
+ữ
+ự
+ỳ
+ỹ
+ἀ
+ἐ
+ὁ
+ὐ
+ὰ
+ὶ
+ὸ
+ῆ
+ῖ
+ῦ
+ῶ
+‐
+‑
+‒
+–
+—
+―
+‖
+‘
+’
+‚
+“
+”
+„
+†
+‡
+•
+…
+‰
+′
+″
+⁄
+⁰
+ⁱ
+⁴
+⁵
+⁶
+⁷
+⁸
+⁹
+⁺
+⁻
+ⁿ
+₀
+₁
+₂
+₃
+₄
+₅
+₆
+₇
+₈
+₉
+₊
+₍
+₎
+ₐ
+ₑ
+ₒ
+ₓ
+ₕ
+ₖ
+ₘ
+ₙ
+ₚ
+ₛ
+ₜ
+₤
+€
+₱
+₹
+ℓ
+№
+ℝ
+⅓
+←
+↑
+→
+↔
+⇌
+⇒
+∂
+∈
+−
+∗
+∘
+√
+∞
+∧
+∨
+∩
+∪
+≈
+≠
+≡
+≤
+≥
+⊂
+⊆
+⊕
+⋅
+─
+│
+■
+●
+★
+☆
+☉
+♠
+♣
+♥
+♦
+♭
+♯
+⟨
+⟩
+ⱼ
+、
+。
+《
+》
+「
+」
+『
+』
+〜
+い
+う
+え
+お
+か
+き
+く
+け
+こ
+さ
+し
+す
+せ
+そ
+た
+ち
+つ
+て
+と
+な
+に
+の
+は
+ひ
+ま
+み
+む
+め
+も
+や
+ゆ
+よ
+ら
+り
+る
+れ
+ん
+ア
+ィ
+イ
+ウ
+エ
+オ
+カ
+ガ
+キ
+ク
+グ
+コ
+サ
+シ
+ジ
+ス
+ズ
+タ
+ダ
+ッ
+テ
+デ
+ト
+ド
+ナ
+ニ
+ハ
+バ
+パ
+フ
+ブ
+プ
+マ
+ミ
+ム
+ャ
+ュ
+ラ
+リ
+ル
+レ
+ロ
+ン
+・
+ー
+一
+三
+上
+下
+中
+事
+二
+井
+京
+人
+亻
+仁
+佐
+侍
+光
+公
+力
+北
+十
+南
+原
+口
+史
+司
+吉
+同
+和
+囗
+国
+國
+土
+城
+士
+大
+天
+太
+夫
+女
+子
+宀
+安
+宮
+宿
+小
+尚
+山
+島
+川
+州
+平
+年
+心
+愛
+戸
+文
+新
+方
+日
+明
+星
+書
+月
+木
+本
+李
+村
+東
+松
+林
+正
+武
+氏
+水
+氵
+江
+河
+海
+版
+犬
+王
+生
+田
+白
+皇
+省
+真
+石
+社
+神
+竹
+美
+義
+花
+藤
+西
+谷
+車
+辶
+道
+郎
+郡
+部
+野
+金
+長
+門
+陽
+青
+食
+馬
+高
+龍
+龸
+사
+씨
+의
+이
+한
+fi
+fl
+！
+（
+）
+，
+－
+／
+：
+the
+of
+and
+to
+in
+was
+The
+is
+for
+as
+on
+with
+that
+##s
+his
+by
+he
+at
+from
+it
+her
+He
+had
+an
+were
+you
+be
+In
+she
+are
+but
+which
+It
+not
+or
+have
+my
+him
+one
+this
+me
+has
+also
+up
+their
+first
+out
+who
+been
+they
+She
+into
+all
+would
+its
+##ing
+time
+two
+##a
+##e
+said
+about
+when
+over
+more
+other
+can
+after
+back
+them
+then
+##ed
+there
+like
+so
+only
+##n
+could
+##d
+##i
+##y
+what
+no
+##o
+where
+This
+made
+than
+if
+You
+##ly
+through
+we
+before
+##r
+just
+some
+##er
+years
+do
+New
+##t
+down
+between
+new
+now
+will
+three
+most
+On
+around
+year
+used
+such
+being
+well
+during
+They
+know
+against
+under
+later
+did
+part
+known
+off
+while
+His
+re
+...
+##l
+people
+until
+way
+American
+didn
+University
+your
+both
+many
+get
+United
+became
+head
+There
+second
+As
+work
+any
+But
+still
+again
+born
+even
+eyes
+After
+including
+de
+took
+And
+long
+team
+season
+family
+see
+right
+same
+called
+name
+because
+film
+don
+10
+found
+much
+school
+##es
+going
+won
+place
+away
+We
+day
+left
+John
+000
+hand
+since
+World
+these
+how
+make
+number
+each
+life
+area
+man
+four
+go
+No
+here
+very
+National
+##m
+played
+released
+never
+began
+States
+album
+home
+last
+too
+held
+several
+May
+own
+##on
+take
+end
+School
+##h
+ll
+series
+What
+want
+use
+another
+city
+When
+2010
+side
+At
+may
+That
+came
+face
+June
+think
+game
+those
+high
+March
+early
+September
+##al
+2011
+looked
+July
+state
+small
+thought
+went
+January
+October
+##u
+based
+August
+##us
+world
+good
+April
+York
+us
+12
+2012
+2008
+For
+2009
+group
+along
+few
+South
+little
+##k
+following
+November
+something
+2013
+December
+set
+2007
+old
+2006
+2014
+located
+##an
+music
+County
+City
+former
+##in
+room
+ve
+next
+All
+##man
+got
+father
+house
+##g
+body
+15
+20
+18
+started
+If
+2015
+town
+our
+line
+War
+large
+population
+named
+British
+company
+member
+five
+My
+single
+##en
+age
+State
+moved
+February
+11
+Her
+should
+century
+government
+built
+come
+best
+show
+However
+within
+look
+men
+door
+without
+need
+wasn
+2016
+water
+One
+system
+knew
+every
+died
+League
+turned
+asked
+North
+St
+wanted
+building
+received
+song
+served
+though
+felt
+##ia
+station
+band
+##ers
+local
+public
+himself
+different
+death
+say
+##1
+30
+##2
+2005
+16
+night
+behind
+children
+English
+members
+near
+saw
+together
+son
+14
+voice
+village
+13
+hands
+help
+##3
+due
+French
+London
+top
+told
+open
+published
+third
+2017
+play
+across
+During
+put
+final
+often
+include
+25
+##le
+main
+having
+2004
+once
+ever
+let
+book
+led
+gave
+late
+front
+find
+club
+##4
+German
+included
+species
+College
+form
+opened
+mother
+women
+enough
+West
+must
+2000
+power
+really
+17
+making
+half
+##6
+order
+might
+##is
+given
+million
+times
+days
+point
+full
+service
+With
+km
+major
+##7
+original
+become
+seen
+II
+north
+six
+##te
+love
+##0
+national
+International
+##5
+24
+So
+District
+lost
+run
+couldn
+career
+always
+##9
+2003
+##th
+country
+##z
+House
+air
+tell
+south
+worked
+woman
+player
+##A
+almost
+war
+River
+##ic
+married
+continued
+Then
+James
+close
+black
+short
+##8
+##na
+using
+history
+returned
+light
+car
+##ra
+sure
+William
+things
+General
+##ry
+2002
+better
+support
+100
+among
+From
+feet
+King
+anything
+21
+19
+established
+district
+2001
+feel
+great
+##ton
+level
+Cup
+These
+written
+games
+others
+already
+title
+story
+##p
+law
+thing
+US
+record
+role
+however
+By
+students
+England
+white
+control
+least
+inside
+land
+##C
+22
+give
+community
+hard
+##ie
+non
+##c
+produced
+George
+round
+period
+Park
+business
+various
+##ne
+does
+present
+wife
+far
+taken
+per
+reached
+David
+able
+version
+working
+young
+live
+created
+joined
+East
+living
+appeared
+case
+High
+done
+23
+important
+President
+Award
+France
+position
+office
+looking
+total
+general
+class
+To
+production
+##S
+football
+party
+brother
+keep
+mind
+free
+Street
+hair
+announced
+development
+either
+nothing
+moment
+Church
+followed
+wrote
+why
+India
+San
+election
+1999
+lead
+How
+##ch
+##rs
+words
+European
+course
+considered
+America
+arms
+Army
+political
+##la
+28
+26
+west
+east
+ground
+further
+church
+less
+site
+First
+Not
+Australia
+toward
+California
+##ness
+described
+works
+An
+Council
+heart
+past
+military
+27
+##or
+heard
+field
+human
+soon
+founded
+1998
+playing
+trying
+##x
+##ist
+##ta
+television
+mouth
+although
+taking
+win
+fire
+Division
+##ity
+Party
+Royal
+program
+Some
+Don
+Association
+According
+tried
+TV
+Paul
+outside
+daughter
+Best
+While
+someone
+match
+recorded
+Canada
+closed
+region
+Air
+above
+months
+elected
+##da
+##ian
+road
+##ar
+brought
+move
+1997
+leave
+##um
+Thomas
+1996
+am
+low
+Robert
+formed
+person
+services
+points
+Mr
+miles
+##b
+stop
+rest
+doing
+needed
+international
+release
+floor
+start
+sound
+call
+killed
+real
+dark
+research
+finished
+language
+Michael
+professional
+change
+sent
+50
+upon
+29
+track
+hit
+event
+2018
+term
+example
+Germany
+similar
+return
+##ism
+fact
+pulled
+stood
+says
+ran
+information
+yet
+result
+developed
+girl
+##re
+God
+1995
+areas
+signed
+decided
+##ment
+Company
+seemed
+##el
+co
+turn
+race
+common
+video
+Charles
+Indian
+##ation
+blood
+art
+red
+##able
+added
+rather
+1994
+met
+director
+addition
+design
+average
+minutes
+##ies
+##ted
+available
+bed
+coming
+friend
+idea
+kind
+Union
+Road
+remained
+##ting
+everything
+##ma
+running
+care
+finally
+Chinese
+appointed
+1992
+Australian
+##ley
+popular
+mean
+teams
+probably
+##land
+usually
+project
+social
+Championship
+possible
+word
+Russian
+instead
+mi
+herself
+##T
+Peter
+Hall
+Center
+seat
+style
+money
+1993
+else
+Department
+table
+Music
+current
+31
+features
+special
+events
+character
+Two
+square
+sold
+debut
+##v
+process
+Although
+Since
+##ka
+40
+Central
+currently
+education
+placed
+lot
+China
+quickly
+forward
+seven
+##ling
+Europe
+arm
+performed
+Japanese
+1991
+Henry
+Now
+Dr
+##ion
+week
+Group
+myself
+big
+UK
+Washington
+ten
+deep
+1990
+Club
+Japan
+space
+La
+directed
+smile
+episode
+hours
+whole
+##de
+##less
+Why
+wouldn
+designed
+strong
+training
+changed
+Society
+stage
+involved
+hadn
+towards
+leading
+police
+eight
+kept
+Institute
+study
+largest
+child
+eventually
+private
+modern
+Court
+throughout
+getting
+originally
+attack
+##E
+talk
+Great
+longer
+songs
+alone
+##ine
+wide
+dead
+walked
+shot
+##ri
+Oh
+force
+##st
+Art
+today
+friends
+Island
+Richard
+1989
+center
+construction
+believe
+size
+White
+ship
+completed
+##B
+gone
+Just
+rock
+sat
+##R
+radio
+below
+entire
+families
+league
+includes
+type
+lived
+official
+range
+hold
+featured
+Most
+##ter
+president
+passed
+means
+##f
+forces
+lips
+Mary
+Do
+guitar
+##ce
+food
+wall
+Of
+spent
+Its
+performance
+hear
+##P
+Western
+reported
+sister
+##et
+morning
+##M
+especially
+##ive
+Minister
+itself
+post
+bit
+groups
+1988
+##tion
+Black
+##ng
+Well
+raised
+sometimes
+Canadian
+Paris
+Spanish
+replaced
+schools
+Academy
+leaving
+central
+female
+Christian
+Jack
+whose
+college
+onto
+provided
+##D
+##ville
+players
+actually
+stopped
+##son
+Museum
+doesn
+##ts
+books
+fight
+allowed
+##ur
+beginning
+Records
+awarded
+parents
+coach
+##os
+Red
+saying
+##ck
+Smith
+Yes
+Lake
+##L
+aircraft
+1987
+##ble
+previous
+ft
+action
+Italian
+African
+happened
+vocals
+Act
+future
+court
+##ge
+1986
+degree
+phone
+##ro
+Is
+countries
+winning
+breath
+Love
+river
+matter
+Lord
+Other
+list
+self
+parts
+##ate
+provide
+cut
+shows
+plan
+1st
+interest
+##ized
+Africa
+stated
+Sir
+fell
+owned
+earlier
+ended
+competition
+attention
+1985
+lower
+nearly
+bad
+older
+stay
+Saint
+##se
+certain
+1984
+fingers
+blue
+try
+fourth
+Grand
+##as
+king
+##nt
+makes
+chest
+movement
+states
+moving
+data
+introduced
+model
+date
+section
+Los
+deal
+##I
+skin
+entered
+middle
+success
+Texas
+##w
+summer
+island
+##N
+Republic
+length
+husband
+1980
+##ey
+reason
+anyone
+forced
+via
+base
+500
+job
+covered
+Festival
+Roman
+successful
+rights
+cover
+Man
+writing
+Ireland
+##F
+related
+goal
+takes
+buildings
+true
+weeks
+1983
+Because
+opening
+novel
+ISBN
+meet
+gold
+##ous
+mid
+km²
+standing
+Football
+Chicago
+shook
+whom
+##ki
+1982
+Day
+feeling
+scored
+boy
+higher
+Force
+leader
+heavy
+fall
+question
+sense
+army
+Second
+energy
+meeting
+themselves
+kill
+##am
+board
+census
+##ya
+##ns
+mine
+meant
+market
+required
+battle
+campaign
+attended
+approximately
+Kingdom
+runs
+active
+##ha
+contract
+clear
+previously
+health
+1979
+Arts
+complete
+Catholic
+couple
+units
+##ll
+##ty
+Committee
+shoulder
+sea
+systems
+listed
+##O
+caught
+tournament
+##G
+northern
+author
+Film
+Your
+##men
+holding
+offered
+personal
+1981
+southern
+artist
+traditional
+studio
+200
+capital
+##ful
+regular
+ask
+giving
+organization
+month
+news
+Are
+read
+managed
+helped
+studied
+student
+defeated
+natural
+industry
+Year
+noted
+decision
+Government
+quite
+##id
+smiled
+1972
+Maybe
+tracks
+##ke
+Mark
+al
+media
+engine
+hour
+Their
+relationship
+plays
+property
+structure
+1976
+ago
+Hill
+Martin
+1978
+ready
+Many
+Like
+Bay
+immediately
+generally
+Italy
+Greek
+practice
+caused
+division
+significant
+Joseph
+speed
+Let
+thinking
+completely
+1974
+primary
+mostly
+##field
+##K
+1975
+##to
+Even
+writer
+##led
+dropped
+magazine
+collection
+understand
+route
+highest
+particular
+films
+lines
+network
+Science
+loss
+carried
+direction
+green
+1977
+location
+producer
+according
+Women
+Queen
+neck
+thus
+independent
+view
+1970
+Angeles
+Soviet
+distance
+problem
+Board
+tour
+western
+income
+appearance
+access
+Mexico
+nodded
+street
+surface
+arrived
+believed
+Old
+1968
+1973
+becoming
+whether
+1945
+figure
+singer
+stand
+Following
+issue
+window
+wrong
+pain
+everyone
+lives
+issues
+park
+slowly
+la
+act
+##va
+bring
+Lee
+operations
+key
+comes
+fine
+cold
+famous
+Navy
+1971
+Me
+additional
+individual
+##ner
+Zealand
+goals
+county
+contains
+Service
+minute
+2nd
+reach
+talking
+particularly
+##ham
+movie
+Director
+glass
+paper
+studies
+##co
+railway
+standard
+Education
+45
+represented
+Chief
+Louis
+launched
+Star
+terms
+60
+1969
+experience
+watched
+Another
+Press
+Tom
+staff
+starting
+subject
+break
+Virginia
+nine
+eye
+##age
+evidence
+foot
+##est
+companies
+Prince
+##V
+gun
+create
+Big
+People
+guy
+Green
+simply
+numerous
+##line
+increased
+twenty
+##ga
+##do
+1967
+award
+officer
+stone
+Before
+material
+Northern
+grew
+male
+plant
+Life
+legs
+step
+Al
+unit
+35
+except
+answer
+##U
+report
+response
+Edward
+commercial
+edition
+trade
+science
+##ca
+Irish
+Law
+shown
+rate
+failed
+##ni
+remains
+changes
+mm
+limited
+larger
+Later
+cause
+waiting
+Time
+##wood
+cost
+Bill
+manager
+activities
+likely
+allow
+operated
+retired
+##ping
+65
+directly
+Who
+associated
+effect
+hell
+Florida
+straight
+hot
+Valley
+management
+girls
+expected
+eastern
+Mike
+chance
+cast
+centre
+chair
+hurt
+problems
+##li
+walk
+programs
+Team
+characters
+Battle
+edge
+pay
+maybe
+corner
+majority
+medical
+Joe
+Summer
+##io
+attempt
+Pacific
+command
+Radio
+##by
+names
+municipality
+1964
+train
+economic
+Brown
+feature
+sex
+source
+agreed
+remember
+Three
+1966
+1965
+Pennsylvania
+victory
+senior
+annual
+III
+Southern
+results
+Sam
+serving
+religious
+Jones
+appears
+##der
+despite
+claimed
+Both
+musical
+matches
+fast
+security
+selected
+Young
+double
+complex
+hospital
+chief
+Times
+##ve
+Championships
+filled
+Public
+Despite
+beautiful
+Research
+plans
+Province
+##ally
+Wales
+##ko
+artists
+metal
+nearby
+Spain
+##il
+32
+houses
+supported
+piece
+##no
+stared
+recording
+nature
+legal
+Russia
+##ization
+remaining
+looks
+##sh
+bridge
+closer
+cases
+scene
+marriage
+Little
+##é
+uses
+Earth
+specific
+Frank
+theory
+Good
+discovered
+referred
+bass
+culture
+university
+presented
+Congress
+##go
+metres
+continue
+1960
+isn
+Awards
+meaning
+cell
+composed
+separate
+Series
+forms
+Blue
+cross
+##tor
+increase
+test
+computer
+slightly
+Where
+Jewish
+Town
+tree
+status
+1944
+variety
+responsible
+pretty
+initially
+##way
+realized
+pass
+provides
+Captain
+Alexander
+recent
+score
+broke
+Scott
+drive
+financial
+showed
+Line
+stories
+ordered
+soldiers
+genus
+operation
+gaze
+sitting
+society
+Only
+hope
+actor
+follow
+Empire
+Yeah
+technology
+happy
+focus
+policy
+spread
+situation
+##ford
+##ba
+Mrs
+watch
+Can
+1963
+Commission
+touch
+earned
+troops
+Under
+1962
+individuals
+cannot
+19th
+##lin
+mile
+expression
+exactly
+suddenly
+weight
+dance
+stepped
+places
+appear
+difficult
+Railway
+anti
+numbers
+kilometres
+star
+##ier
+department
+ice
+Britain
+removed
+Once
+##lo
+Boston
+value
+##ant
+mission
+trees
+Order
+sports
+join
+serve
+Major
+poor
+Poland
+mainly
+Theatre
+pushed
+Station
+##it
+Lady
+federal
+silver
+##ler
+foreign
+##ard
+Eastern
+##den
+box
+hall
+subsequently
+lies
+acquired
+1942
+ancient
+CD
+History
+Jean
+beyond
+##ger
+El
+##les
+growing
+championship
+native
+Parliament
+Williams
+watching
+direct
+overall
+offer
+Also
+80
+Secretary
+spoke
+Latin
+ability
+##ated
+safe
+presence
+##ial
+headed
+regional
+planned
+1961
+Johnson
+throat
+consists
+##W
+extended
+Or
+bar
+walls
+Chris
+stations
+politician
+Olympics
+influence
+share
+fighting
+speak
+hundred
+Carolina
+die
+stars
+##tic
+color
+Chapter
+##ish
+fear
+sleep
+goes
+Francisco
+oil
+Bank
+sign
+physical
+##berg
+Dutch
+seasons
+##rd
+Games
+Governor
+sorry
+lack
+Centre
+memory
+baby
+smaller
+charge
+Did
+multiple
+ships
+shirt
+Assembly
+amount
+leaves
+3rd
+Foundation
+conditions
+1943
+Rock
+Democratic
+Daniel
+##at
+winner
+products
+##ina
+store
+latter
+Professor
+civil
+prior
+host
+1956
+soft
+vote
+needs
+Each
+rules
+1958
+pressure
+letter
+normal
+proposed
+levels
+records
+1959
+paid
+intended
+Victoria
+purpose
+okay
+historical
+issued
+1980s
+broadcast
+rule
+simple
+picked
+firm
+Sea
+1941
+Elizabeth
+1940
+serious
+featuring
+highly
+graduated
+mentioned
+choice
+1948
+replied
+percent
+Scotland
+##hi
+females
+constructed
+1957
+settled
+Steve
+recognized
+cities
+crew
+glanced
+kiss
+competed
+flight
+knowledge
+editor
+More
+Conference
+##H
+fifth
+elements
+##ee
+##tes
+function
+newspaper
+recently
+Miss
+cultural
+brown
+twice
+Office
+1939
+truth
+Creek
+1946
+households
+USA
+1950
+quality
+##tt
+border
+seconds
+destroyed
+pre
+wait
+ahead
+build
+image
+90
+cars
+##mi
+33
+promoted
+professor
+et
+bank
+medal
+text
+broken
+Middle
+revealed
+sides
+wing
+seems
+channel
+1970s
+Ben
+loved
+effort
+officers
+Will
+##ff
+70
+Israel
+Jim
+upper
+fully
+label
+Jr
+assistant
+powerful
+pair
+positive
+##ary
+gives
+1955
+20th
+races
+remain
+kitchen
+primarily
+##ti
+Sydney
+easy
+Tour
+whispered
+buried
+300
+News
+Polish
+1952
+Duke
+Columbia
+produce
+accepted
+00
+approach
+minor
+1947
+Special
+44
+Asian
+basis
+visit
+Fort
+Civil
+finish
+formerly
+beside
+leaned
+##ite
+median
+rose
+coast
+effects
+supposed
+Cross
+##hip
+Corps
+residents
+Jackson
+##ir
+Bob
+basketball
+36
+Asia
+seem
+Bishop
+Book
+##ber
+ring
+##ze
+owner
+BBC
+##ja
+transferred
+acting
+De
+appearances
+walking
+Le
+press
+grabbed
+1954
+officially
+1953
+##pe
+risk
+taught
+review
+##X
+lay
+##well
+council
+Avenue
+seeing
+losing
+Ohio
+Super
+province
+ones
+travel
+##sa
+projects
+equipment
+spot
+Berlin
+administrative
+heat
+potential
+shut
+capacity
+elections
+growth
+fought
+Republican
+mixed
+Andrew
+teacher
+turning
+strength
+shoulders
+beat
+wind
+1949
+Health
+follows
+camp
+suggested
+perhaps
+Alex
+mountain
+contact
+divided
+candidate
+fellow
+34
+Show
+necessary
+workers
+ball
+horse
+ways
+questions
+protect
+gas
+activity
+younger
+bottom
+founder
+Scottish
+screen
+treatment
+easily
+com
+##house
+dedicated
+Master
+warm
+Night
+Georgia
+Long
+von
+##me
+perfect
+website
+1960s
+piano
+efforts
+##ide
+Tony
+sort
+offers
+Development
+Simon
+executive
+##nd
+save
+Over
+Senate
+1951
+1990s
+draw
+master
+Police
+##ius
+renamed
+boys
+initial
+prominent
+damage
+Co
+##ov
+##za
+online
+begin
+occurred
+captured
+youth
+Top
+account
+tells
+Justice
+conducted
+forest
+##town
+bought
+teeth
+Jersey
+##di
+purchased
+agreement
+Michigan
+##ure
+campus
+prison
+becomes
+product
+secret
+guess
+Route
+huge
+types
+drums
+64
+split
+defeat
+estate
+housing
+##ot
+brothers
+Coast
+declared
+happen
+titled
+therefore
+sun
+commonly
+alongside
+Stadium
+library
+Home
+article
+steps
+telling
+slow
+assigned
+refused
+laughed
+wants
+Nick
+wearing
+Rome
+Open
+##ah
+Hospital
+pointed
+Taylor
+lifted
+escape
+participated
+##j
+drama
+parish
+Santa
+##per
+organized
+mass
+pick
+Airport
+gets
+Library
+unable
+pull
+Live
+##ging
+surrounding
+##ries
+focused
+Adam
+facilities
+##ning
+##ny
+38
+##ring
+notable
+era
+connected
+gained
+operating
+laid
+Regiment
+branch
+defined
+Christmas
+machine
+Four
+academic
+Iran
+adopted
+concept
+Men
+compared
+search
+traffic
+Max
+Maria
+greater
+##ding
+widely
+##burg
+serves
+1938
+37
+Go
+hotel
+shared
+typically
+scale
+1936
+leg
+suffered
+yards
+pieces
+Ministry
+Wilson
+episodes
+empty
+1918
+safety
+continues
+yellow
+historic
+settlement
+400
+Come
+Corporation
+enemy
+content
+picture
+evening
+territory
+method
+trial
+solo
+driver
+Here
+##ls
+entrance
+Prize
+spring
+whatever
+##ent
+75
+##ji
+reading
+Arthur
+##cy
+Our
+clothes
+Prime
+Illinois
+Kong
+code
+##ria
+sit
+Harry
+Federal
+chosen
+administration
+bodies
+begins
+stomach
+Though
+seats
+Hong
+density
+Sun
+leaders
+Field
+museum
+chart
+platform
+languages
+##ron
+birth
+holds
+Gold
+##un
+fish
+combined
+##ps
+4th
+1937
+largely
+captain
+trust
+Game
+van
+boat
+Oxford
+basic
+beneath
+Islands
+painting
+nice
+Toronto
+path
+males
+sources
+block
+conference
+parties
+murder
+clubs
+crowd
+calling
+About
+Business
+peace
+knows
+lake
+speaking
+stayed
+Brazil
+allowing
+Born
+unique
+thick
+Technology
+##que
+receive
+des
+semi
+alive
+noticed
+format
+##ped
+coffee
+digital
+##ned
+handed
+guard
+tall
+faced
+setting
+plants
+partner
+claim
+reduced
+temple
+animals
+determined
+classes
+##out
+estimated
+##ad
+Olympic
+providing
+Massachusetts
+learned
+Inc
+Philadelphia
+Social
+carry
+42
+possibly
+hosted
+tonight
+respectively
+Today
+shape
+Mount
+roles
+designated
+brain
+etc
+Korea
+thoughts
+Brian
+Highway
+doors
+background
+drew
+models
+footballer
+tone
+turns
+1935
+quiet
+tower
+wood
+bus
+write
+software
+weapons
+flat
+marked
+1920
+newly
+tight
+Eric
+finger
+Journal
+FC
+Van
+rise
+critical
+Atlantic
+granted
+returning
+communities
+humans
+quick
+39
+48
+ranked
+sight
+pop
+Swedish
+Stephen
+card
+analysis
+attacked
+##wa
+Sunday
+identified
+Jason
+champion
+situated
+1930
+expanded
+tears
+##nce
+reaching
+Davis
+protection
+Emperor
+positions
+nominated
+Bridge
+tax
+dress
+allows
+avoid
+leadership
+killing
+actress
+guest
+steel
+knowing
+electric
+cells
+disease
+grade
+unknown
+##ium
+resulted
+Pakistan
+confirmed
+##ged
+tongue
+covers
+##Y
+roof
+entirely
+applied
+votes
+drink
+interview
+exchange
+Township
+reasons
+##ised
+page
+calls
+dog
+agent
+nose
+teaching
+##ds
+##ists
+advanced
+wish
+Golden
+existing
+vehicle
+del
+1919
+develop
+attacks
+pressed
+Sports
+planning
+resulting
+facility
+Sarah
+notes
+1933
+Class
+Historic
+winter
+##mo
+audience
+Community
+household
+Netherlands
+creation
+##ize
+keeping
+1914
+claims
+dry
+guys
+opposite
+##ak
+explained
+Ontario
+secondary
+difference
+Francis
+actions
+organizations
+yard
+animal
+Up
+Lewis
+titles
+Several
+1934
+Ryan
+55
+Supreme
+rolled
+1917
+distribution
+figures
+afraid
+rural
+yourself
+##rt
+sets
+barely
+Instead
+passing
+awards
+41
+silence
+authority
+occupied
+environment
+windows
+engineering
+surprised
+flying
+crime
+reports
+Mountain
+powers
+driving
+succeeded
+reviews
+1929
+Head
+missing
+Song
+Jesus
+opportunity
+inspired
+ends
+albums
+conversation
+impact
+injury
+surprise
+billion
+learning
+heavily
+oldest
+union
+creating
+##ky
+festival
+literature
+letters
+sexual
+##tte
+apartment
+Final
+comedy
+nation
+orders
+##sen
+contemporary
+Power
+drawn
+existence
+connection
+##ating
+Post
+Junior
+remembered
+message
+Medal
+castle
+note
+engineer
+sounds
+Beach
+crossed
+##dy
+ear
+scientific
+sales
+##ai
+theme
+starts
+clearly
+##ut
+trouble
+##gan
+bag
+##han
+BC
+sons
+1928
+silent
+versions
+daily
+Studies
+ending
+Rose
+guns
+1932
+headquarters
+reference
+obtained
+Squadron
+concert
+none
+du
+Among
+##don
+prevent
+Member
+answered
+staring
+Between
+##lla
+portion
+drug
+liked
+association
+performances
+Nations
+formation
+Castle
+lose
+learn
+scoring
+relatively
+quarter
+47
+Premier
+##ors
+Sweden
+baseball
+attempted
+trip
+worth
+perform
+airport
+fields
+enter
+honor
+Medical
+rear
+commander
+officials
+condition
+supply
+materials
+52
+Anna
+volume
+threw
+Persian
+43
+interested
+Gallery
+achieved
+visited
+laws
+relief
+Area
+Matt
+singles
+Lieutenant
+Country
+fans
+Cambridge
+sky
+Miller
+effective
+tradition
+Port
+##ana
+minister
+extra
+entitled
+System
+sites
+authorities
+acres
+committee
+racing
+1931
+desk
+trains
+ass
+weren
+Family
+farm
+##ance
+industrial
+##head
+iron
+49
+abandoned
+Out
+Holy
+chairman
+waited
+frequently
+display
+Light
+transport
+starring
+Patrick
+Engineering
+eat
+FM
+judge
+reaction
+centuries
+price
+##tive
+Korean
+defense
+Get
+arrested
+1927
+send
+urban
+##ss
+pilot
+Okay
+Media
+reality
+arts
+soul
+thirty
+##be
+catch
+generation
+##nes
+apart
+Anne
+drop
+See
+##ving
+sixth
+trained
+Management
+magic
+cm
+height
+Fox
+Ian
+resources
+vampire
+principal
+Was
+haven
+##au
+Walter
+Albert
+rich
+1922
+causing
+entry
+##ell
+shortly
+46
+worry
+doctor
+composer
+rank
+Network
+bright
+showing
+regions
+1924
+wave
+carrying
+kissed
+finding
+missed
+Earl
+lying
+target
+vehicles
+Military
+controlled
+dinner
+##board
+briefly
+lyrics
+motion
+duty
+strange
+attempts
+invited
+kg
+villages
+5th
+Land
+##mer
+Christ
+prepared
+twelve
+check
+thousand
+earth
+copies
+en
+transfer
+citizens
+Americans
+politics
+nor
+theatre
+Project
+##bo
+clean
+rooms
+laugh
+##ran
+application
+contained
+anyway
+containing
+Sciences
+1925
+rare
+speech
+exist
+1950s
+falling
+passenger
+##im
+stands
+51
+##ol
+##ow
+phase
+governor
+kids
+details
+methods
+Vice
+employed
+performing
+counter
+Jane
+heads
+Channel
+wine
+opposition
+aged
+1912
+Every
+1926
+highway
+##ura
+1921
+aired
+978
+permanent
+Forest
+finds
+joint
+approved
+##pur
+brief
+doubt
+acts
+brand
+wild
+closely
+Ford
+Kevin
+chose
+shall
+port
+sweet
+fun
+asking
+Be
+##bury
+sought
+Dave
+Mexican
+mom
+Right
+Howard
+Moscow
+Charlie
+Stone
+##mann
+admitted
+##ver
+wooden
+1923
+Officer
+relations
+Hot
+combat
+publication
+chain
+shop
+inhabitants
+proved
+ideas
+address
+1915
+Memorial
+explain
+increasing
+conflict
+Anthony
+Melbourne
+narrow
+temperature
+slid
+1916
+worse
+selling
+documentary
+Ali
+Ray
+opposed
+vision
+dad
+extensive
+Infantry
+commissioned
+Doctor
+offices
+programming
+core
+respect
+storm
+##pa
+##ay
+##om
+promotion
+der
+struck
+anymore
+shit
+Region
+receiving
+DVD
+alternative
+##ue
+ride
+maximum
+1910
+##ious
+Third
+Affairs
+cancer
+Executive
+##op
+dream
+18th
+Due
+##ker
+##worth
+economy
+IV
+Billboard
+identity
+subsequent
+statement
+skills
+##back
+funding
+##ons
+Round
+Foreign
+truck
+Please
+lights
+wondered
+##ms
+frame
+yes
+Still
+districts
+fiction
+Colonel
+converted
+150
+grown
+accident
+critics
+fit
+Information
+architecture
+Point
+Five
+armed
+Billy
+poet
+functions
+consisted
+suit
+Turkish
+Band
+object
+desire
+##ities
+sounded
+flow
+Norwegian
+articles
+Marie
+pulling
+thin
+singing
+Hunter
+Human
+Battalion
+Federation
+Kim
+origin
+represent
+dangerous
+weather
+fuel
+ex
+##sing
+Last
+bedroom
+aid
+knees
+Alan
+angry
+assumed
+plane
+Something
+founding
+concerned
+global
+Fire
+di
+please
+Portuguese
+touched
+Roger
+nuclear
+Register
+Jeff
+fixed
+royal
+lie
+finals
+NFL
+Manchester
+towns
+handle
+shaped
+Chairman
+Dean
+launch
+understanding
+Children
+violence
+failure
+sector
+Brigade
+wrapped
+fired
+sharp
+tiny
+developing
+expansion
+Free
+institutions
+technical
+Nothing
+otherwise
+Main
+inch
+Saturday
+wore
+Senior
+attached
+cheek
+representing
+Kansas
+##chi
+##kin
+actual
+advantage
+Dan
+Austria
+##dale
+hoped
+multi
+squad
+Norway
+streets
+1913
+Services
+hired
+grow
+pp
+wear
+painted
+Minnesota
+stuff
+Building
+54
+Philippines
+1900
+##ties
+educational
+Khan
+Magazine
+##port
+Cape
+signal
+Gordon
+sword
+Anderson
+cool
+engaged
+Commander
+images
+Upon
+tied
+Security
+cup
+rail
+Vietnam
+successfully
+##red
+Muslim
+gain
+bringing
+Native
+hers
+occurs
+negative
+Philip
+Kelly
+Colorado
+category
+##lan
+600
+Have
+supporting
+wet
+56
+stairs
+Grace
+observed
+##ung
+funds
+restaurant
+1911
+Jews
+##ments
+##che
+Jake
+Back
+53
+asks
+journalist
+accept
+bands
+bronze
+helping
+##ice
+decades
+mayor
+survived
+usual
+influenced
+Douglas
+Hey
+##izing
+surrounded
+retirement
+Temple
+derived
+Pope
+registered
+producing
+##ral
+structures
+Johnny
+contributed
+finishing
+buy
+specifically
+##king
+patients
+Jordan
+internal
+regarding
+Samuel
+Clark
+##q
+afternoon
+Finally
+scenes
+notice
+refers
+quietly
+threat
+Water
+Those
+Hamilton
+promise
+freedom
+Turkey
+breaking
+maintained
+device
+lap
+ultimately
+Champion
+Tim
+Bureau
+expressed
+investigation
+extremely
+capable
+qualified
+recognition
+items
+##up
+Indiana
+adult
+rain
+greatest
+architect
+Morgan
+dressed
+equal
+Antonio
+collected
+drove
+occur
+Grant
+graduate
+anger
+Sri
+worried
+standards
+##ore
+injured
+somewhere
+damn
+Singapore
+Jimmy
+pocket
+homes
+stock
+religion
+aware
+regarded
+Wisconsin
+##tra
+passes
+fresh
+##ea
+argued
+Ltd
+EP
+Diego
+importance
+Census
+incident
+Egypt
+Missouri
+domestic
+leads
+ceremony
+Early
+camera
+Father
+challenge
+Switzerland
+lands
+familiar
+hearing
+spend
+educated
+Tennessee
+Thank
+##ram
+Thus
+concern
+putting
+inches
+map
+classical
+Allen
+crazy
+valley
+Space
+softly
+##my
+pool
+worldwide
+climate
+experienced
+neighborhood
+scheduled
+neither
+fleet
+1908
+Girl
+##J
+Part
+engines
+locations
+darkness
+Revolution
+establishment
+lawyer
+objects
+apparently
+Queensland
+Entertainment
+bill
+mark
+Television
+##ong
+pale
+demand
+Hotel
+selection
+##rn
+##ino
+Labour
+Liberal
+burned
+Mom
+merged
+Arizona
+request
+##lia
+##light
+hole
+employees
+##ical
+incorporated
+95
+independence
+Walker
+covering
+joining
+##ica
+task
+papers
+backing
+sell
+biggest
+6th
+strike
+establish
+##ō
+gently
+59
+Orchestra
+Winter
+protein
+Juan
+locked
+dates
+Boy
+aren
+shooting
+Luke
+solid
+charged
+Prior
+resigned
+interior
+garden
+spoken
+improve
+wonder
+promote
+hidden
+##med
+combination
+Hollywood
+Swiss
+consider
+##ks
+Lincoln
+literary
+drawing
+Marine
+weapon
+Victor
+Trust
+Maryland
+properties
+##ara
+exhibition
+understood
+hung
+Tell
+installed
+loud
+fashion
+affected
+junior
+landing
+flowers
+##he
+Internet
+beach
+Heart
+tries
+Mayor
+programme
+800
+wins
+noise
+##ster
+##ory
+58
+contain
+fair
+delivered
+##ul
+wedding
+Square
+advance
+behavior
+Program
+Oregon
+##rk
+residence
+realize
+certainly
+hill
+Houston
+57
+indicated
+##water
+wounded
+Village
+massive
+Moore
+thousands
+personnel
+dating
+opera
+poetry
+##her
+causes
+feelings
+Frederick
+applications
+push
+approached
+foundation
+pleasure
+sale
+fly
+gotten
+northeast
+costs
+raise
+paintings
+##ney
+views
+horses
+formal
+Arab
+hockey
+typical
+representative
+rising
+##des
+clock
+stadium
+shifted
+Dad
+peak
+Fame
+vice
+disappeared
+users
+Way
+Naval
+prize
+hoping
+values
+evil
+Bell
+consisting
+##ón
+Regional
+##ics
+improved
+circle
+carefully
+broad
+##ini
+Fine
+maintain
+operate
+offering
+mention
+Death
+stupid
+Through
+Princess
+attend
+interests
+ruled
+somewhat
+wings
+roads
+grounds
+##ual
+Greece
+Champions
+facing
+hide
+voted
+require
+Dark
+Matthew
+credit
+sighed
+separated
+manner
+##ile
+Boys
+1905
+committed
+impossible
+lip
+candidates
+7th
+Bruce
+arranged
+Islamic
+courses
+criminal
+##ened
+smell
+##bed
+08
+consecutive
+##ening
+proper
+purchase
+weak
+Prix
+1906
+aside
+introduction
+Look
+##ku
+changing
+budget
+resistance
+factory
+Forces
+agency
+##tone
+northwest
+user
+1907
+stating
+##one
+sport
+Design
+environmental
+cards
+concluded
+Carl
+250
+accused
+##ology
+Girls
+sick
+intelligence
+Margaret
+responsibility
+Guard
+##tus
+17th
+sq
+goods
+1909
+hate
+##ek
+capture
+stores
+Gray
+comic
+Modern
+Silver
+Andy
+electronic
+wheel
+##ied
+Deputy
+##bs
+Czech
+zone
+choose
+constant
+reserve
+##lle
+Tokyo
+spirit
+sub
+degrees
+flew
+pattern
+compete
+Dance
+##ik
+secretary
+Imperial
+99
+reduce
+Hungarian
+confused
+##rin
+Pierre
+describes
+regularly
+Rachel
+85
+landed
+passengers
+##ise
+##sis
+historian
+meters
+Youth
+##ud
+participate
+##cing
+arrival
+tired
+Mother
+##gy
+jumped
+Kentucky
+faces
+feed
+Israeli
+Ocean
+##Q
+##án
+plus
+snow
+techniques
+plate
+sections
+falls
+jazz
+##ris
+tank
+loan
+repeated
+opinion
+##res
+unless
+rugby
+journal
+Lawrence
+moments
+shock
+distributed
+##ded
+adjacent
+Argentina
+crossing
+uncle
+##ric
+Detroit
+communication
+mental
+tomorrow
+session
+Emma
+Without
+##gen
+Miami
+charges
+Administration
+hits
+coat
+protected
+Cole
+invasion
+priest
+09
+Gary
+enjoyed
+plot
+measure
+bound
+friendly
+throw
+musician
+##lon
+##ins
+Age
+knife
+damaged
+birds
+driven
+lit
+ears
+breathing
+Arabic
+Jan
+faster
+Jonathan
+##gate
+Independent
+starred
+Harris
+teachers
+Alice
+sequence
+mph
+file
+translated
+decide
+determine
+Review
+documents
+sudden
+threatened
+##ft
+bear
+distinct
+decade
+burning
+##sky
+1930s
+replace
+begun
+extension
+##time
+1904
+equivalent
+accompanied
+Christopher
+Danish
+##ye
+Besides
+##more
+persons
+fallen
+Rural
+roughly
+saved
+willing
+ensure
+Belgium
+05
+musicians
+##ang
+giant
+Six
+Retrieved
+worst
+purposes
+##bly
+mountains
+seventh
+slipped
+brick
+07
+##py
+somehow
+Carter
+Iraq
+cousin
+favor
+islands
+journey
+FIFA
+contrast
+planet
+vs
+calm
+##ings
+concrete
+branches
+gray
+profit
+Russell
+##ae
+##ux
+##ens
+philosophy
+businesses
+talked
+parking
+##ming
+owners
+Place
+##tle
+agricultural
+Kate
+06
+southeast
+draft
+Eddie
+earliest
+forget
+Dallas
+Commonwealth
+edited
+66
+inner
+ed
+operates
+16th
+Harvard
+assistance
+##si
+designs
+Take
+bathroom
+indicate
+CEO
+Command
+Louisiana
+1902
+Dublin
+Books
+1901
+tropical
+1903
+##tors
+Places
+tie
+progress
+forming
+solution
+62
+letting
+##ery
+studying
+##jo
+duties
+Baseball
+taste
+Reserve
+##ru
+Ann
+##gh
+visible
+##vi
+notably
+link
+NCAA
+southwest
+Never
+storage
+mobile
+writers
+favorite
+Pro
+pages
+truly
+count
+##tta
+string
+kid
+98
+Ross
+row
+##idae
+Kennedy
+##tan
+Hockey
+hip
+waist
+grandfather
+listen
+##ho
+feels
+busy
+72
+stream
+obvious
+cycle
+shaking
+Knight
+##ren
+Carlos
+painter
+trail
+web
+linked
+04
+Palace
+existed
+##ira
+responded
+closing
+End
+examples
+Marshall
+weekend
+jaw
+Denmark
+lady
+township
+medium
+chin
+Story
+option
+fifteen
+Moon
+represents
+makeup
+investment
+jump
+childhood
+Oklahoma
+roll
+normally
+Ten
+Operation
+Graham
+Seattle
+Atlanta
+paused
+promised
+rejected
+treated
+returns
+flag
+##ita
+Hungary
+danger
+glad
+movements
+visual
+subjects
+credited
+soldier
+Norman
+ill
+translation
+José
+Quebec
+medicine
+warning
+theater
+praised
+municipal
+01
+commune
+churches
+acid
+folk
+8th
+testing
+add
+survive
+Sound
+devices
+residential
+severe
+presidential
+Mississippi
+Austin
+Perhaps
+Charlotte
+hanging
+Montreal
+grin
+##ten
+racial
+partnership
+shoot
+shift
+##nie
+Les
+downtown
+Brothers
+Garden
+matters
+restored
+mirror
+forever
+winners
+rapidly
+poverty
+##ible
+Until
+DC
+faith
+hundreds
+Real
+Ukraine
+Nelson
+balance
+Adams
+contest
+relative
+ethnic
+Edinburgh
+composition
+##nts
+emergency
+##van
+marine
+reputation
+Down
+pack
+12th
+Communist
+Mountains
+pro
+stages
+measures
+##ld
+ABC
+Li
+victims
+benefit
+Iowa
+Broadway
+gathered
+rating
+Defense
+classic
+##ily
+ceiling
+##ions
+snapped
+Everything
+constituency
+Franklin
+Thompson
+Stewart
+entering
+Judge
+forth
+##sk
+wanting
+smiling
+moves
+tunnel
+premiered
+grass
+unusual
+Ukrainian
+bird
+Friday
+tail
+Portugal
+coal
+element
+Fred
+guards
+Senator
+collaboration
+beauty
+Wood
+chemical
+beer
+justice
+signs
+##Z
+sees
+##zi
+Puerto
+##zed
+96
+smooth
+Bowl
+gift
+limit
+97
+heading
+Source
+wake
+requires
+Ed
+Constitution
+factor
+Lane
+factors
+adding
+Note
+cleared
+pictures
+pink
+##ola
+Kent
+Local
+Singh
+moth
+Ty
+##ture
+courts
+Seven
+temporary
+involving
+Vienna
+emerged
+fishing
+agree
+defensive
+stuck
+secure
+Tamil
+##ick
+bottle
+03
+Player
+instruments
+Spring
+patient
+flesh
+contributions
+cry
+Malaysia
+120
+Global
+da
+Alabama
+Within
+##work
+debuted
+expect
+Cleveland
+concerns
+retained
+horror
+10th
+spending
+Peace
+Transport
+grand
+Crown
+instance
+institution
+acted
+Hills
+mounted
+Campbell
+shouldn
+1898
+##ably
+chamber
+soil
+88
+Ethan
+sand
+cheeks
+##gi
+marry
+61
+weekly
+classification
+DNA
+Elementary
+Roy
+definitely
+Soon
+Rights
+gate
+suggests
+aspects
+imagine
+golden
+beating
+Studios
+Warren
+differences
+significantly
+glance
+occasionally
+##od
+clothing
+Assistant
+depth
+sending
+possibility
+mode
+prisoners
+requirements
+daughters
+dated
+Representatives
+prove
+guilty
+interesting
+smoke
+cricket
+93
+##ates
+rescue
+Connecticut
+underground
+Opera
+13th
+reign
+##ski
+thanks
+leather
+equipped
+routes
+fan
+##ans
+script
+Wright
+bishop
+Welsh
+jobs
+faculty
+eleven
+Railroad
+appearing
+anniversary
+Upper
+##down
+anywhere
+Rugby
+Metropolitan
+Meanwhile
+Nicholas
+champions
+forehead
+mining
+drinking
+76
+Jerry
+membership
+Brazilian
+Wild
+Rio
+scheme
+Unlike
+strongly
+##bility
+fill
+##rian
+easier
+MP
+Hell
+##sha
+Stanley
+banks
+Baron
+##ique
+Robinson
+67
+Gabriel
+Austrian
+Wayne
+exposed
+##wan
+Alfred
+1899
+manage
+mix
+visitors
+eating
+##rate
+Sean
+commission
+Cemetery
+policies
+Camp
+parallel
+traveled
+guitarist
+02
+supplies
+couples
+poem
+blocks
+Rick
+Training
+Energy
+achieve
+appointment
+Wing
+Jamie
+63
+novels
+##em
+1890
+songwriter
+Base
+Jay
+##gar
+naval
+scared
+miss
+labor
+technique
+crisis
+Additionally
+backed
+destroy
+seriously
+tools
+tennis
+91
+god
+##ington
+continuing
+steam
+obviously
+Bobby
+adapted
+fifty
+enjoy
+Jacob
+publishing
+column
+##ular
+Baltimore
+Donald
+Liverpool
+92
+drugs
+movies
+##ock
+Heritage
+##je
+##istic
+vocal
+strategy
+gene
+advice
+##bi
+Ottoman
+riding
+##side
+Agency
+Indonesia
+11th
+laughing
+sleeping
+und
+muttered
+listening
+deck
+tip
+77
+ownership
+grey
+Claire
+deeply
+provincial
+popularity
+Cooper
+##á
+Emily
+##sed
+designer
+Murray
+describe
+Danny
+Around
+Parker
+##dae
+68
+rates
+suffering
+considerable
+78
+nervous
+powered
+tons
+circumstances
+wished
+belonged
+Pittsburgh
+flows
+9th
+##use
+belt
+81
+useful
+15th
+context
+List
+Dead
+Iron
+seek
+Season
+worn
+frequency
+legislation
+replacement
+memories
+Tournament
+Again
+Barry
+organisation
+copy
+Gulf
+waters
+meets
+struggle
+Oliver
+1895
+Susan
+protest
+kick
+Alliance
+components
+1896
+Tower
+Windows
+demanded
+regiment
+sentence
+Woman
+Logan
+Referee
+hosts
+debate
+knee
+Blood
+##oo
+universities
+practices
+Ward
+ranking
+correct
+happening
+Vincent
+attracted
+classified
+##stic
+processes
+immediate
+waste
+increasingly
+Helen
+##po
+Lucas
+Phil
+organ
+1897
+tea
+suicide
+actors
+lb
+crash
+approval
+waves
+##ered
+hated
+grip
+700
+amongst
+69
+74
+hunting
+dying
+lasted
+illegal
+##rum
+stare
+defeating
+##gs
+shrugged
+°C
+Jon
+Count
+Orleans
+94
+affairs
+formally
+##and
+##ves
+criticized
+Disney
+Vol
+successor
+tests
+scholars
+palace
+Would
+celebrated
+rounds
+grant
+Schools
+Such
+commanded
+demon
+Romania
+##all
+Karl
+71
+##yn
+84
+Daily
+totally
+Medicine
+fruit
+Die
+upset
+Lower
+Conservative
+14th
+Mitchell
+escaped
+shoes
+Morris
+##tz
+queen
+harder
+prime
+Thanks
+indeed
+Sky
+authors
+rocks
+definition
+Nazi
+accounts
+printed
+experiences
+##ters
+divisions
+Cathedral
+denied
+depending
+Express
+##let
+73
+appeal
+loose
+colors
+filed
+##isation
+gender
+##ew
+throne
+forests
+Finland
+domain
+boats
+Baker
+squadron
+shore
+remove
+##ification
+careful
+wound
+railroad
+82
+seeking
+agents
+##ved
+Blues
+##off
+customers
+ignored
+net
+##ction
+hiding
+Originally
+declined
+##ess
+franchise
+eliminated
+NBA
+merely
+pure
+appropriate
+visiting
+forty
+markets
+offensive
+coverage
+cave
+##nia
+spell
+##lar
+Benjamin
+##ire
+Convention
+filmed
+Trade
+##sy
+##ct
+Having
+palm
+1889
+Evans
+intense
+plastic
+Julia
+document
+jeans
+vessel
+SR
+##fully
+proposal
+Birmingham
+le
+##ative
+assembly
+89
+fund
+lock
+1893
+AD
+meetings
+occupation
+modified
+Years
+odd
+aimed
+reform
+Mission
+Works
+shake
+cat
+exception
+convinced
+executed
+pushing
+dollars
+replacing
+soccer
+manufacturing
+##ros
+expensive
+kicked
+minimum
+Josh
+coastal
+Chase
+ha
+Thailand
+publications
+deputy
+Sometimes
+Angel
+effectively
+##illa
+criticism
+conduct
+Serbian
+landscape
+NY
+absence
+passage
+##ula
+Blake
+Indians
+1892
+admit
+Trophy
+##ball
+Next
+##rated
+##ians
+charts
+kW
+orchestra
+79
+heritage
+1894
+rough
+exists
+boundary
+Bible
+Legislative
+moon
+medieval
+##over
+cutting
+print
+##ett
+birthday
+##hood
+destruction
+Julian
+injuries
+influential
+sisters
+raising
+statue
+colour
+dancing
+characteristics
+orange
+##ok
+##aries
+Ken
+colonial
+twin
+Larry
+surviving
+##shi
+Barbara
+personality
+entertainment
+assault
+##ering
+talent
+happens
+license
+86
+couch
+Century
+soundtrack
+shower
+swimming
+cash
+Staff
+bent
+1885
+bay
+lunch
+##lus
+dozen
+vessels
+CBS
+greatly
+critic
+Test
+symbol
+panel
+shell
+output
+reaches
+87
+Front
+motor
+ocean
+##era
+##ala
+maintenance
+violent
+scent
+Limited
+Las
+Hope
+Theater
+Which
+survey
+Robin
+recordings
+compilation
+##ward
+bomb
+insurance
+Authority
+sponsored
+satellite
+Jazz
+refer
+stronger
+blow
+whilst
+Wrestling
+suggest
+##rie
+climbed
+##els
+voices
+shopping
+1891
+Neil
+discovery
+##vo
+##ations
+burst
+Baby
+peaked
+Brooklyn
+knocked
+lift
+##try
+false
+nations
+Hugh
+Catherine
+preserved
+distinguished
+terminal
+resolution
+ratio
+pants
+cited
+competitions
+completion
+DJ
+bone
+uniform
+schedule
+shouted
+83
+1920s
+rarely
+Basketball
+Taiwan
+artistic
+bare
+vampires
+arrest
+Utah
+Marcus
+assist
+gradually
+qualifying
+Victorian
+vast
+rival
+Warner
+Terry
+Economic
+##cia
+losses
+boss
+versus
+audio
+runner
+apply
+surgery
+Play
+twisted
+comfortable
+##cs
+Everyone
+guests
+##lt
+Harrison
+UEFA
+lowered
+occasions
+##lly
+##cher
+chapter
+youngest
+eighth
+Culture
+##room
+##stone
+1888
+Songs
+Seth
+Digital
+involvement
+expedition
+relationships
+signing
+1000
+fault
+annually
+circuit
+afterwards
+meat
+creature
+##ou
+cable
+Bush
+##net
+Hispanic
+rapid
+gonna
+figured
+extent
+considering
+cried
+##tin
+sigh
+dynasty
+##ration
+cabinet
+Richmond
+stable
+##zo
+1864
+Admiral
+Unit
+occasion
+shares
+badly
+longest
+##ify
+Connor
+extreme
+wondering
+girlfriend
+Studio
+##tions
+1865
+tribe
+exact
+muscles
+hat
+Luis
+Orthodox
+decisions
+amateur
+description
+##lis
+hips
+kingdom
+##ute
+Portland
+whereas
+Bachelor
+outer
+discussion
+partly
+Arkansas
+1880
+dreams
+perfectly
+Lloyd
+##bridge
+asleep
+##tti
+Greg
+permission
+trading
+pitch
+mill
+Stage
+liquid
+Keith
+##tal
+wolf
+processing
+stick
+Jerusalem
+profile
+rushed
+spiritual
+argument
+Ice
+Guy
+till
+Delhi
+roots
+Section
+missions
+Glasgow
+penalty
+NBC
+encouraged
+identify
+keyboards
+##zing
+##ston
+disc
+plain
+informed
+Bernard
+thinks
+fled
+Justin
+##day
+newspapers
+##wick
+Ralph
+##zer
+unlike
+Stars
+artillery
+##ified
+recovered
+arrangement
+searching
+##pers
+##tory
+##rus
+deaths
+Egyptian
+diameter
+##í
+marketing
+corporate
+teach
+marks
+Turner
+staying
+hallway
+Sebastian
+chapel
+naked
+mistake
+possession
+1887
+dominated
+jacket
+creative
+Fellow
+Falls
+Defence
+suspended
+employment
+##rry
+Hebrew
+Hudson
+Week
+Wars
+recognize
+Natural
+controversial
+Tommy
+thank
+Athletic
+benefits
+decline
+intention
+##ets
+Lost
+Wall
+participation
+elevation
+supports
+parliament
+1861
+concentration
+Movement
+##IS
+competing
+stops
+behalf
+##mm
+limits
+funded
+discuss
+Collins
+departure
+obtain
+woods
+latest
+universe
+alcohol
+Laura
+rush
+blade
+funny
+Dennis
+forgotten
+Amy
+Symphony
+apparent
+graduating
+1862
+Rob
+Grey
+collections
+Mason
+emotions
+##ugh
+literally
+Any
+counties
+1863
+nomination
+fighter
+habitat
+respond
+external
+Capital
+exit
+Video
+carbon
+sharing
+Bad
+opportunities
+Perry
+photo
+##mus
+Orange
+posted
+remainder
+transportation
+portrayed
+Labor
+recommended
+percussion
+rated
+Grade
+rivers
+partially
+suspected
+strip
+adults
+button
+struggled
+intersection
+Canal
+##ability
+poems
+claiming
+Madrid
+1886
+Together
+##our
+Much
+Vancouver
+instrument
+instrumental
+1870
+mad
+angle
+Control
+Phoenix
+Leo
+Communications
+mail
+##ette
+##ev
+preferred
+adaptation
+alleged
+discussed
+deeper
+##ane
+Yet
+Monday
+volumes
+thrown
+Zane
+##logy
+displayed
+rolling
+dogs
+Along
+Todd
+##ivity
+withdrew
+representation
+belief
+##sia
+crown
+Late
+Short
+hardly
+grinned
+romantic
+Pete
+##ken
+networks
+enemies
+Colin
+Eventually
+Side
+donated
+##su
+steady
+grab
+guide
+Finnish
+Milan
+pregnant
+controversy
+reminded
+1884
+Stuart
+##bach
+##ade
+Race
+Belgian
+LP
+Production
+Zone
+lieutenant
+infantry
+Child
+confusion
+sang
+resident
+##ez
+victim
+1881
+channels
+Ron
+businessman
+##gle
+Dick
+colony
+pace
+producers
+##ese
+agencies
+Craig
+Lucy
+Very
+centers
+Yorkshire
+photography
+##ched
+Album
+championships
+Metro
+substantial
+Standard
+terrible
+directors
+contribution
+advertising
+emotional
+##its
+layer
+segment
+sir
+folded
+Roberts
+ceased
+Hampshire
+##ray
+detailed
+partners
+m²
+##pt
+Beth
+genre
+commented
+generated
+remote
+aim
+Hans
+credits
+concerts
+periods
+breakfast
+gay
+shadow
+defence
+Too
+Had
+transition
+Afghanistan
+##book
+eggs
+defend
+##lli
+writes
+Systems
+bones
+mess
+seed
+scientists
+Shortly
+Romanian
+##zy
+Freedom
+muscle
+hero
+parent
+agriculture
+checked
+Islam
+Bristol
+Freyja
+Arena
+cabin
+Germans
+electricity
+ranks
+viewed
+medals
+Wolf
+associate
+Madison
+Sorry
+fort
+Chile
+detail
+widespread
+attorney
+boyfriend
+##nan
+Students
+Spencer
+##ig
+bite
+Maine
+demolished
+Lisa
+erected
+Someone
+operational
+Commissioner
+NHL
+Coach
+Bar
+forcing
+Dream
+Rico
+cargo
+Murphy
+##fish
+##ase
+distant
+##master
+##ora
+Organization
+doorway
+Steven
+traded
+electrical
+frequent
+##wn
+Branch
+Sure
+1882
+placing
+Manhattan
+attending
+attributed
+excellent
+pounds
+ruling
+principles
+component
+Mediterranean
+Vegas
+machines
+percentage
+infrastructure
+throwing
+affiliated
+Kings
+secured
+Caribbean
+Track
+Ted
+honour
+opponent
+Virgin
+Construction
+grave
+produces
+Challenge
+stretched
+paying
+murmured
+##ata
+integrated
+waved
+Nathan
+##ator
+transmission
+videos
+##yan
+##hu
+Nova
+descent
+AM
+Harold
+conservative
+Therefore
+venue
+competitive
+##ui
+conclusion
+funeral
+confidence
+releases
+scholar
+##sson
+Treaty
+stress
+mood
+##sm
+Mac
+residing
+Action
+Fund
+##ship
+animated
+fitted
+##kar
+defending
+voting
+tend
+##berry
+answers
+believes
+##ci
+helps
+Aaron
+##tis
+themes
+##lay
+populations
+Players
+stroke
+Trinity
+electoral
+paint
+abroad
+charity
+keys
+Fair
+##pes
+interrupted
+participants
+murdered
+Days
+supporters
+##ab
+expert
+borders
+mate
+##llo
+solar
+architectural
+tension
+##bling
+Parish
+tape
+operator
+Cultural
+Clinton
+indicates
+publisher
+ordinary
+sugar
+arrive
+rifle
+acoustic
+##uring
+assets
+##shire
+SS
+sufficient
+options
+HMS
+Classic
+bars
+rebuilt
+governments
+Beijing
+reporter
+screamed
+Abbey
+crying
+mechanical
+instantly
+communications
+Political
+cemetery
+Cameron
+Stop
+representatives
+USS
+texts
+mathematics
+innings
+civilian
+Serbia
+##hill
+practical
+patterns
+dust
+Faculty
+debt
+##end
+##cus
+junction
+suppose
+experimental
+Computer
+Food
+wrist
+abuse
+dealing
+bigger
+cap
+principle
+##pin
+Muhammad
+Fleet
+Collection
+attempting
+dismissed
+##burn
+regime
+Herbert
+##ua
+shadows
+1883
+Eve
+Lanka
+1878
+Performance
+fictional
+##lock
+Noah
+Run
+Voivodeship
+exercise
+broadcasting
+##fer
+RAF
+Magic
+Bangladesh
+suitable
+##low
+##del
+styles
+toured
+Code
+identical
+links
+insisted
+110
+flash
+Model
+slave
+Derek
+Rev
+fairly
+Greater
+sole
+##lands
+connecting
+zero
+bench
+##ome
+switched
+Fall
+Owen
+yours
+Electric
+shocked
+convention
+##bra
+climb
+memorial
+swept
+Racing
+decides
+belong
+##nk
+parliamentary
+##und
+ages
+proof
+##dan
+delivery
+1860
+##ów
+sad
+publicly
+leaning
+Archbishop
+dirt
+##ose
+categories
+1876
+burn
+##bing
+requested
+Guinea
+Historical
+rhythm
+relation
+##heim
+ye
+pursue
+merchant
+##mes
+lists
+continuous
+frowned
+colored
+tool
+gods
+involves
+Duncan
+photographs
+Cricket
+slight
+Gregory
+atmosphere
+wider
+Cook
+##tar
+essential
+Being
+FA
+emperor
+wealthy
+nights
+##bar
+licensed
+Hawaii
+viewers
+Language
+load
+nearest
+milk
+kilometers
+platforms
+##ys
+territories
+Rogers
+sheet
+Rangers
+contested
+##lation
+isolated
+assisted
+swallowed
+Small
+Contemporary
+Technical
+Edwards
+express
+Volume
+endemic
+##ei
+tightly
+Whatever
+indigenous
+Colombia
+##ulation
+hp
+characterized
+##ida
+Nigeria
+Professional
+duo
+Soccer
+slaves
+Farm
+smart
+Attorney
+Attendance
+Common
+salt
+##vin
+tribes
+nod
+sentenced
+bid
+sample
+Drive
+switch
+instant
+21st
+Cuba
+drunk
+Alaska
+proud
+awareness
+hitting
+sessions
+Thai
+locally
+elsewhere
+Dragon
+gentle
+touching
+##lee
+Springs
+Universal
+Latino
+spin
+1871
+Chart
+recalled
+Type
+pointing
+##ii
+lowest
+##ser
+grandmother
+Adelaide
+Jacques
+spotted
+Buffalo
+restoration
+Son
+Joan
+farmers
+Lily
+1879
+lucky
+##dal
+luck
+eldest
+##rant
+Market
+drummer
+deployed
+warned
+prince
+sing
+amazing
+sailed
+##oon
+1875
+Primary
+traveling
+Masters
+Sara
+cattle
+Trail
+gang
+Further
+desert
+relocated
+##tch
+##ord
+Flight
+illness
+Munich
+ninth
+repair
+Singles
+##lated
+Tyler
+tossed
+boots
+Work
+sized
+earning
+shoved
+magazines
+housed
+dam
+researchers
+Former
+spun
+premiere
+spaces
+organised
+wealth
+crimes
+devoted
+stones
+Urban
+automatic
+hop
+affect
+outstanding
+tanks
+mechanism
+Muslims
+Ms
+shots
+argue
+Jeremy
+connections
+Armenian
+increases
+rubbed
+1867
+retail
+gear
+Pan
+bonus
+jurisdiction
+weird
+concerning
+whisper
+##gal
+Microsoft
+tenure
+hills
+www
+Gmina
+porch
+files
+reportedly
+venture
+Storm
+##ence
+Nature
+killer
+panic
+fate
+Secret
+Wang
+scream
+drivers
+belongs
+Chamber
+clan
+monument
+mixing
+Peru
+bet
+Riley
+Friends
+Isaac
+submarine
+1877
+130
+judges
+harm
+ranging
+affair
+prepare
+pupils
+householder
+Policy
+decorated
+Nation
+slammed
+activist
+implemented
+Room
+qualify
+Publishing
+establishing
+Baptist
+touring
+subsidiary
+##nal
+legend
+1872
+laughter
+PC
+Athens
+settlers
+ties
+dual
+dear
+Draft
+strategic
+Ivan
+reveal
+closest
+dominant
+Ah
+##ult
+Denver
+bond
+boundaries
+drafted
+tables
+##TV
+eyed
+Edition
+##ena
+1868
+belonging
+1874
+Industrial
+cream
+Ridge
+Hindu
+scholarship
+Ma
+opens
+initiated
+##ith
+yelled
+compound
+random
+Throughout
+grades
+physics
+sank
+grows
+exclusively
+settle
+Saints
+brings
+Amsterdam
+Make
+Hart
+walks
+battery
+violin
+##born
+explanation
+##ware
+1873
+##har
+provinces
+thrust
+exclusive
+sculpture
+shops
+##fire
+VI
+constitution
+Barcelona
+monster
+Devon
+Jefferson
+Sullivan
+bow
+##din
+desperate
+##ć
+Julie
+##mon
+##ising
+terminus
+Jesse
+abilities
+golf
+##ple
+##via
+##away
+Raymond
+measured
+jury
+firing
+revenue
+suburb
+Bulgarian
+1866
+##cha
+timber
+Things
+##weight
+Morning
+spots
+Alberta
+Data
+explains
+Kyle
+friendship
+raw
+tube
+demonstrated
+aboard
+immigrants
+reply
+breathe
+Manager
+ease
+##ban
+##dia
+Diocese
+##vy
+##ía
+pit
+ongoing
+##lie
+Gilbert
+Costa
+1940s
+Report
+voters
+cloud
+traditions
+##MS
+gallery
+Jennifer
+swung
+Broadcasting
+Does
+diverse
+reveals
+arriving
+initiative
+##ani
+Give
+Allied
+Pat
+Outstanding
+monastery
+blind
+Currently
+##war
+bloody
+stopping
+focuses
+managing
+Florence
+Harvey
+creatures
+900
+breast
+internet
+Artillery
+purple
+##mate
+alliance
+excited
+fee
+Brisbane
+lifetime
+Private
+##aw
+##nis
+##gue
+##ika
+phrase
+regulations
+reflected
+manufactured
+conventional
+pleased
+client
+##ix
+##ncy
+Pedro
+reduction
+##con
+welcome
+jail
+comfort
+Iranian
+Norfolk
+Dakota
+##tein
+evolution
+everywhere
+Initially
+sensitive
+Olivia
+Oscar
+implementation
+sits
+stolen
+demands
+slide
+grandson
+##ich
+merger
+##mic
+Spirit
+##°
+ticket
+root
+difficulty
+Nevada
+##als
+lined
+Dylan
+Original
+Call
+biological
+EU
+dramatic
+##hn
+Operations
+treaty
+gap
+##list
+Am
+Romanized
+moral
+Butler
+perspective
+Furthermore
+Manuel
+absolutely
+unsuccessful
+disaster
+dispute
+preparation
+tested
+discover
+##ach
+shield
+squeezed
+brushed
+battalion
+Arnold
+##ras
+superior
+treat
+clinical
+##so
+Apple
+Syria
+Cincinnati
+package
+flights
+editions
+Leader
+minority
+wonderful
+hang
+Pop
+Philippine
+telephone
+bell
+honorary
+##mar
+balls
+Democrat
+dirty
+thereafter
+collapsed
+Inside
+slip
+wrestling
+##ín
+listened
+regard
+bowl
+None
+Sport
+completing
+trapped
+##view
+copper
+Wallace
+Honor
+blame
+Peninsula
+##ert
+##oy
+Anglo
+bearing
+simultaneously
+honest
+##ias
+Mix
+Got
+speaker
+voiced
+impressed
+prices
+error
+1869
+##feld
+trials
+Nine
+Industry
+substitute
+Municipal
+departed
+slept
+##ama
+Junction
+Socialist
+flower
+dropping
+comment
+fantasy
+##ress
+arrangements
+travelled
+furniture
+fist
+relieved
+##tics
+Leonard
+linear
+earn
+expand
+Soul
+Plan
+Leeds
+Sierra
+accessible
+innocent
+Winner
+Fighter
+Range
+winds
+vertical
+Pictures
+101
+charter
+cooperation
+prisoner
+interviews
+recognised
+sung
+manufacturer
+exposure
+submitted
+Mars
+leaf
+gauge
+screaming
+likes
+eligible
+##ac
+gathering
+columns
+##dra
+belly
+UN
+maps
+messages
+speakers
+##ants
+garage
+unincorporated
+Number
+Watson
+sixteen
+lots
+beaten
+Could
+Municipality
+##ano
+Horse
+talks
+Drake
+scores
+Venice
+genetic
+##mal
+##ère
+Cold
+Jose
+nurse
+traditionally
+##bus
+Territory
+Key
+Nancy
+##win
+thumb
+São
+index
+dependent
+carries
+controls
+Comics
+coalition
+physician
+referring
+Ruth
+Based
+restricted
+inherited
+internationally
+stretch
+THE
+plates
+margin
+Holland
+knock
+significance
+valuable
+Kenya
+carved
+emotion
+conservation
+municipalities
+overseas
+resumed
+Finance
+graduation
+blinked
+temperatures
+constantly
+productions
+scientist
+ghost
+cuts
+permitted
+##ches
+firmly
+##bert
+patrol
+##yo
+Croatian
+attacking
+1850
+portrait
+promoting
+sink
+conversion
+##kov
+locomotives
+Guide
+##val
+nephew
+relevant
+Marc
+drum
+originated
+Chair
+visits
+dragged
+Price
+favour
+corridor
+properly
+respective
+Caroline
+reporting
+inaugural
+1848
+industries
+##ching
+edges
+Christianity
+Maurice
+Trent
+Economics
+carrier
+Reed
+##gon
+tribute
+Pradesh
+##ale
+extend
+attitude
+Yale
+##lu
+settlements
+glasses
+taxes
+targets
+##ids
+quarters
+##ological
+connect
+hence
+metre
+collapse
+underneath
+banned
+Future
+clients
+alternate
+explosion
+kinds
+Commons
+hungry
+dragon
+Chapel
+Buddhist
+lover
+depression
+pulls
+##ges
+##uk
+origins
+computers
+crosses
+kissing
+assume
+emphasis
+lighting
+##ites
+personally
+crashed
+beam
+touchdown
+lane
+comparison
+##mont
+Hitler
+##las
+execution
+##ene
+acre
+sum
+Pearl
+ray
+##point
+essentially
+worker
+convicted
+tear
+Clay
+recovery
+Literature
+Unfortunately
+##row
+partial
+Petersburg
+Bulgaria
+coaching
+evolved
+reception
+enters
+narrowed
+elevator
+therapy
+defended
+pairs
+##lam
+breaks
+Bennett
+Uncle
+cylinder
+##ison
+passion
+bases
+Actor
+cancelled
+battles
+extensively
+oxygen
+Ancient
+specialized
+negotiations
+##rat
+acquisition
+convince
+interpretation
+##00
+photos
+aspect
+colleges
+Artist
+keeps
+##wing
+Croatia
+##ona
+Hughes
+Otto
+comments
+##du
+Ph
+Sweet
+adventure
+describing
+Student
+Shakespeare
+scattered
+objective
+Aviation
+Phillips
+Fourth
+athletes
+##hal
+##tered
+Guitar
+intensity
+née
+dining
+curve
+Obama
+topics
+legislative
+Mill
+Cruz
+##ars
+Members
+recipient
+Derby
+inspiration
+corresponding
+fed
+YouTube
+coins
+pressing
+intent
+Karen
+cinema
+Delta
+destination
+shorter
+Christians
+imagined
+canal
+Newcastle
+Shah
+Adrian
+super
+Males
+160
+liberal
+lord
+bat
+supplied
+Claude
+meal
+worship
+##atic
+Han
+wire
+°F
+##tha
+punishment
+thirteen
+fighters
+##ibility
+1859
+Ball
+gardens
+##ari
+Ottawa
+pole
+indicating
+Twenty
+Higher
+Bass
+Ivy
+farming
+##urs
+certified
+Saudi
+plenty
+##ces
+restaurants
+Representative
+Miles
+payment
+##inger
+##rit
+Confederate
+festivals
+references
+##ić
+Mario
+PhD
+playoffs
+witness
+rice
+mask
+saving
+opponents
+enforcement
+automatically
+relegated
+##oe
+radar
+whenever
+Financial
+imperial
+uncredited
+influences
+Abraham
+skull
+Guardian
+Haven
+Bengal
+impressive
+input
+mixture
+Warsaw
+altitude
+distinction
+1857
+collective
+Annie
+##ean
+##bal
+directions
+Flying
+##nic
+faded
+##ella
+contributing
+##ó
+employee
+##lum
+##yl
+ruler
+oriented
+conductor
+focusing
+##die
+Giants
+Mills
+mines
+Deep
+curled
+Jessica
+guitars
+Louise
+procedure
+Machine
+failing
+attendance
+Nepal
+Brad
+Liam
+tourist
+exhibited
+Sophie
+depicted
+Shaw
+Chuck
+##can
+expecting
+challenges
+##nda
+equally
+resignation
+##logical
+Tigers
+loop
+pitched
+outdoor
+reviewed
+hopes
+True
+temporarily
+Borough
+torn
+jerked
+collect
+Berkeley
+Independence
+cotton
+retreat
+campaigns
+participating
+Intelligence
+Heaven
+##ked
+situations
+borough
+Democrats
+Harbor
+##len
+Liga
+serial
+circles
+fourteen
+##lot
+seized
+filling
+departments
+finance
+absolute
+Roland
+Nate
+floors
+raced
+struggling
+deliver
+protests
+##tel
+Exchange
+efficient
+experiments
+##dar
+faint
+3D
+binding
+Lions
+lightly
+skill
+proteins
+difficulties
+##cal
+monthly
+camps
+flood
+loves
+Amanda
+Commerce
+##oid
+##lies
+elementary
+##tre
+organic
+##stein
+##ph
+receives
+Tech
+enormous
+distinctive
+Joint
+experiment
+Circuit
+citizen
+##hy
+shelter
+ideal
+practically
+formula
+addressed
+Foster
+Productions
+##ax
+variable
+punk
+Voice
+fastest
+concentrated
+##oma
+##yer
+stored
+surrender
+vary
+Sergeant
+Wells
+ward
+Wait
+##ven
+playoff
+reducing
+cavalry
+##dle
+Venezuela
+tissue
+amounts
+sweat
+##we
+Non
+##nik
+beetle
+##bu
+##tu
+Jared
+Hunt
+##₂
+fat
+Sultan
+Living
+Circle
+Secondary
+Suddenly
+reverse
+##min
+Travel
+##bin
+Lebanon
+##mas
+virus
+Wind
+dissolved
+enrolled
+holiday
+Keep
+helicopter
+Clarke
+constitutional
+technologies
+doubles
+instructions
+##ace
+Azerbaijan
+##ill
+occasional
+frozen
+trick
+wiped
+writings
+Shanghai
+preparing
+challenged
+mainstream
+summit
+180
+##arian
+##rating
+designation
+##ada
+revenge
+filming
+tightened
+Miguel
+Montana
+reflect
+celebration
+bitch
+flashed
+signals
+rounded
+peoples
+##tation
+renowned
+Google
+characteristic
+Campaign
+sliding
+##rman
+usage
+Record
+Using
+woke
+solutions
+holes
+theories
+logo
+Protestant
+relaxed
+brow
+nickname
+Reading
+marble
+##tro
+symptoms
+Overall
+capita
+##ila
+outbreak
+revolution
+deemed
+Principal
+Hannah
+approaches
+inducted
+Wellington
+vulnerable
+Environmental
+Drama
+incumbent
+Dame
+1854
+travels
+samples
+accurate
+physically
+Sony
+Nashville
+##sville
+##lic
+##og
+Producer
+Lucky
+tough
+Stanford
+resort
+repeatedly
+eyebrows
+Far
+choir
+commenced
+##ep
+##ridge
+rage
+swing
+sequel
+heir
+buses
+ad
+Grove
+##late
+##rick
+updated
+##SA
+Delaware
+##fa
+Athletics
+warmth
+Off
+excitement
+verse
+Protection
+Villa
+corruption
+intellectual
+Jenny
+##lyn
+mystery
+prayer
+healthy
+##ologist
+Bear
+lab
+Ernest
+Remix
+register
+basement
+Montgomery
+consistent
+tier
+1855
+Preston
+Brooks
+##maker
+vocalist
+laboratory
+delayed
+wheels
+rope
+bachelor
+pitcher
+Block
+Nevertheless
+suspect
+efficiency
+Nebraska
+siege
+FBI
+planted
+##AC
+Newton
+breeding
+##ain
+eighteen
+Argentine
+encounter
+servant
+1858
+elder
+Shadow
+Episode
+fabric
+doctors
+survival
+removal
+chemistry
+volunteers
+Kane
+variant
+arrives
+Eagle
+Left
+##fe
+Jo
+divorce
+##ret
+yesterday
+Bryan
+handling
+diseases
+customer
+Sheriff
+Tiger
+Harper
+##oi
+resting
+Linda
+Sheffield
+gasped
+sexy
+economics
+alien
+tale
+footage
+Liberty
+yeah
+fundamental
+Ground
+flames
+Actress
+photographer
+Maggie
+Additional
+joke
+custom
+Survey
+Abu
+silk
+consumption
+Ellis
+bread
+##uous
+engagement
+puts
+Dog
+##hr
+poured
+guilt
+CDP
+boxes
+hardware
+clenched
+##cio
+stem
+arena
+extending
+##com
+examination
+Steel
+encountered
+revised
+140
+picking
+Car
+hasn
+Minor
+pride
+Roosevelt
+boards
+##mia
+blocked
+curious
+drag
+narrative
+brigade
+Prefecture
+mysterious
+namely
+connects
+Devil
+historians
+CHAPTER
+quit
+installation
+Golf
+empire
+elevated
+##eo
+releasing
+Bond
+##uri
+harsh
+ban
+##BA
+contracts
+cloth
+presents
+stake
+chorus
+##eau
+swear
+##mp
+allies
+generations
+Motor
+meter
+pen
+warrior
+veteran
+##EC
+comprehensive
+missile
+interaction
+instruction
+Renaissance
+rested
+Dale
+fix
+fluid
+les
+investigate
+loaded
+widow
+exhibit
+artificial
+select
+rushing
+tasks
+signature
+nowhere
+Engineer
+feared
+Prague
+bother
+extinct
+gates
+Bird
+climbing
+heels
+striking
+artwork
+hunt
+awake
+##hin
+Formula
+thereby
+commitment
+imprisoned
+Beyond
+##MA
+transformed
+Agriculture
+Low
+Movie
+radical
+complicated
+Yellow
+Auckland
+mansion
+tenth
+Trevor
+predecessor
+##eer
+disbanded
+sucked
+circular
+witch
+gaining
+lean
+Behind
+illustrated
+rang
+celebrate
+bike
+consist
+framework
+##cent
+Shane
+owns
+350
+comprises
+collaborated
+colleagues
+##cast
+engage
+fewer
+##ave
+1856
+observation
+diplomatic
+legislature
+improvements
+Interstate
+craft
+MTV
+martial
+administered
+jet
+approaching
+permanently
+attraction
+manuscript
+numbered
+Happy
+Andrea
+shallow
+Gothic
+Anti
+##bad
+improvement
+trace
+preserve
+regardless
+rode
+dies
+achievement
+maintaining
+Hamburg
+spine
+##air
+flowing
+encourage
+widened
+posts
+##bound
+125
+Southeast
+Santiago
+##bles
+impression
+receiver
+Single
+closure
+##unt
+communist
+honors
+Northwest
+105
+##ulated
+cared
+un
+hug
+magnetic
+seeds
+topic
+perceived
+prey
+prevented
+Marvel
+Eight
+Michel
+Transportation
+rings
+Gate
+##gne
+Byzantine
+accommodate
+floating
+##dor
+equation
+ministry
+##ito
+##gled
+Rules
+earthquake
+revealing
+Brother
+Celtic
+blew
+chairs
+Panama
+Leon
+attractive
+descendants
+Care
+Ambassador
+tours
+breathed
+threatening
+##cho
+smiles
+Lt
+Beginning
+##iness
+fake
+assists
+fame
+strings
+Mobile
+Liu
+parks
+http
+1852
+brush
+Aunt
+bullet
+consciousness
+##sta
+##ther
+consequences
+gather
+dug
+1851
+bridges
+Doug
+##sion
+Artists
+ignore
+Carol
+brilliant
+radiation
+temples
+basin
+clouds
+##cted
+Stevens
+spite
+soap
+consumer
+Damn
+Snow
+recruited
+##craft
+Advanced
+tournaments
+Quinn
+undergraduate
+questioned
+Palmer
+Annual
+Others
+feeding
+Spider
+printing
+##orn
+cameras
+functional
+Chester
+readers
+Alpha
+universal
+Faith
+Brandon
+François
+authored
+Ring
+el
+aims
+athletic
+possessed
+Vermont
+programmes
+##uck
+bore
+Fisher
+statements
+shed
+saxophone
+neighboring
+pronounced
+barrel
+bags
+##dge
+organisations
+pilots
+casualties
+Kenneth
+##brook
+silently
+Malcolm
+span
+Essex
+anchor
+##hl
+virtual
+lessons
+Henri
+Trump
+Page
+pile
+locomotive
+wounds
+uncomfortable
+sustained
+Diana
+Eagles
+##pi
+2000s
+documented
+##bel
+Cassie
+delay
+kisses
+##ines
+variation
+##ag
+growled
+##mark
+##ways
+Leslie
+studios
+Friedrich
+aunt
+actively
+armor
+eaten
+historically
+Better
+purse
+honey
+ratings
+##ée
+naturally
+1840
+peer
+Kenny
+Cardinal
+database
+Looking
+runners
+handsome
+Double
+PA
+##boat
+##sted
+protecting
+##jan
+Diamond
+concepts
+interface
+##aki
+Watch
+Article
+Columbus
+dialogue
+pause
+##rio
+extends
+blanket
+pulse
+1853
+affiliate
+ladies
+Ronald
+counted
+kills
+demons
+##zation
+Airlines
+Marco
+Cat
+companion
+mere
+Yugoslavia
+Forum
+Allan
+pioneer
+Competition
+Methodist
+patent
+nobody
+Stockholm
+##ien
+regulation
+##ois
+accomplished
+##itive
+washed
+sake
+Vladimir
+crops
+prestigious
+humor
+Sally
+labour
+tributary
+trap
+altered
+examined
+Mumbai
+bombing
+Ash
+noble
+suspension
+ruins
+##bank
+spare
+displays
+guided
+dimensional
+Iraqi
+##hon
+sciences
+Franz
+relating
+fence
+followers
+Palestine
+invented
+proceeded
+Batman
+Bradley
+##yard
+##ova
+crystal
+Kerala
+##ima
+shipping
+handled
+Want
+abolished
+Drew
+##tter
+Powell
+Half
+##table
+##cker
+exhibitions
+Were
+assignment
+assured
+##rine
+Indonesian
+Grammy
+acknowledged
+Kylie
+coaches
+structural
+clearing
+stationed
+Say
+Total
+Rail
+besides
+glow
+threats
+afford
+Tree
+Musical
+##pp
+elite
+centered
+explore
+Engineers
+Stakes
+Hello
+tourism
+severely
+assessment
+##tly
+crack
+politicians
+##rrow
+sheets
+volunteer
+##borough
+##hold
+announcement
+recover
+contribute
+lungs
+##ille
+mainland
+presentation
+Johann
+Writing
+1849
+##bird
+Study
+Boulevard
+coached
+fail
+airline
+Congo
+Plus
+Syrian
+introduce
+ridge
+Casey
+manages
+##fi
+searched
+Support
+succession
+progressive
+coup
+cultures
+##lessly
+sensation
+Cork
+Elena
+Sofia
+Philosophy
+mini
+trunk
+academy
+Mass
+Liz
+practiced
+Reid
+##ule
+satisfied
+experts
+Wilhelm
+Woods
+invitation
+Angels
+calendar
+joy
+Sr
+Dam
+packed
+##uan
+bastard
+Workers
+broadcasts
+logic
+cooking
+backward
+##ack
+Chen
+creates
+enzyme
+##xi
+Davies
+aviation
+VII
+Conservation
+fucking
+Knights
+##kan
+requiring
+hectares
+wars
+ate
+##box
+Mind
+desired
+oak
+absorbed
+Really
+Vietnamese
+Paulo
+athlete
+##car
+##eth
+Talk
+Wu
+##cks
+survivors
+Yang
+Joel
+Almost
+Holmes
+Armed
+Joshua
+priests
+discontinued
+##sey
+blond
+Rolling
+suggesting
+CA
+clay
+exterior
+Scientific
+##sive
+Giovanni
+Hi
+farther
+contents
+Winners
+animation
+neutral
+mall
+Notes
+layers
+professionals
+Armstrong
+Against
+Piano
+involve
+monitor
+angel
+parked
+bears
+seated
+feat
+beliefs
+##kers
+Version
+suffer
+##ceae
+guidance
+##eur
+honored
+raid
+alarm
+Glen
+Ellen
+Jamaica
+trio
+enabled
+##ils
+procedures
+##hus
+moderate
+upstairs
+##ses
+torture
+Georgian
+rebellion
+Fernando
+Nice
+##are
+Aires
+Campus
+beast
+##hing
+1847
+##FA
+Isle
+##logist
+Princeton
+cathedral
+Oakland
+Solomon
+##tto
+Milwaukee
+upcoming
+midfielder
+Neither
+sacred
+Eyes
+appreciate
+Brunswick
+secrets
+Rice
+Somerset
+Chancellor
+Curtis
+##gel
+Rich
+separation
+grid
+##los
+##bon
+urge
+##ees
+##ree
+freight
+towers
+psychology
+requirement
+dollar
+##fall
+##sman
+exile
+tomb
+Salt
+Stefan
+Buenos
+Revival
+Porter
+tender
+diesel
+chocolate
+Eugene
+Legion
+Laboratory
+sheep
+arched
+hospitals
+orbit
+Full
+##hall
+drinks
+ripped
+##RS
+tense
+Hank
+leagues
+##nberg
+PlayStation
+fool
+Punjab
+relatives
+Comedy
+sur
+1846
+Tonight
+Sox
+##if
+Rabbi
+org
+speaks
+institute
+defender
+painful
+wishes
+Weekly
+literacy
+portions
+snake
+item
+deals
+##tum
+autumn
+sharply
+reforms
+thighs
+prototype
+##ition
+argues
+disorder
+Physics
+terror
+provisions
+refugees
+predominantly
+independently
+march
+##graphy
+Arabia
+Andrews
+Bus
+Money
+drops
+##zar
+pistol
+matrix
+revolutionary
+##ust
+Starting
+##ptic
+Oak
+Monica
+##ides
+servants
+##hed
+archaeological
+divorced
+rocket
+enjoying
+fires
+##nel
+assembled
+qualification
+retiring
+##fied
+Distinguished
+handful
+infection
+Durham
+##itz
+fortune
+renewed
+Chelsea
+##sley
+curved
+gesture
+retain
+exhausted
+##ifying
+Perth
+jumping
+Palestinian
+Simpson
+colonies
+steal
+##chy
+corners
+Finn
+arguing
+Martha
+##var
+Betty
+emerging
+Heights
+Hindi
+Manila
+pianist
+founders
+regret
+Napoleon
+elbow
+overhead
+bold
+praise
+humanity
+##ori
+Revolutionary
+##ere
+fur
+##ole
+Ashley
+Official
+##rm
+lovely
+Architecture
+##sch
+Baronet
+virtually
+##OS
+descended
+immigration
+##das
+##kes
+Holly
+Wednesday
+maintains
+theatrical
+Evan
+Gardens
+citing
+##gia
+segments
+Bailey
+Ghost
+##city
+governing
+graphics
+##ined
+privately
+potentially
+transformation
+Crystal
+Cabinet
+sacrifice
+hesitated
+mud
+Apollo
+Desert
+bin
+victories
+Editor
+Railways
+Web
+Case
+tourists
+Brussels
+Franco
+compiled
+topped
+Gene
+engineers
+commentary
+egg
+escort
+nerve
+arch
+necessarily
+frustration
+Michelle
+democracy
+genes
+Facebook
+halfway
+##ient
+102
+flipped
+Won
+##mit
+NASA
+Lynn
+Provincial
+ambassador
+Inspector
+glared
+Change
+McDonald
+developments
+tucked
+noting
+Gibson
+circulation
+dubbed
+armies
+resource
+Headquarters
+##iest
+Mia
+Albanian
+Oil
+Albums
+excuse
+intervention
+Grande
+Hugo
+integration
+civilians
+depends
+reserves
+Dee
+compositions
+identification
+restrictions
+quarterback
+Miranda
+Universe
+favourite
+ranges
+hint
+loyal
+Op
+entity
+Manual
+quoted
+dealt
+specialist
+Zhang
+download
+Westminster
+Rebecca
+streams
+Anglican
+variations
+Mine
+detective
+Films
+reserved
+##oke
+##key
+sailing
+##gger
+expanding
+recall
+discovers
+particles
+behaviour
+Gavin
+blank
+permit
+Java
+Fraser
+Pass
+##non
+##TA
+panels
+statistics
+notion
+courage
+dare
+venues
+##roy
+Box
+Newport
+travelling
+Thursday
+warriors
+Glenn
+criteria
+360
+mutual
+restore
+varied
+bitter
+Katherine
+##lant
+ritual
+bits
+##à
+Henderson
+trips
+Richardson
+Detective
+curse
+psychological
+Il
+midnight
+streak
+facts
+Dawn
+Indies
+Edmund
+roster
+Gen
+##nation
+1830
+congregation
+shaft
+##ically
+##mination
+Indianapolis
+Sussex
+loving
+##bit
+sounding
+horrible
+Continental
+Griffin
+advised
+magical
+millions
+##date
+1845
+Safety
+lifting
+determination
+valid
+dialect
+Penn
+Know
+triple
+avoided
+dancer
+judgment
+sixty
+farmer
+lakes
+blast
+aggressive
+Abby
+tag
+chains
+inscription
+##nn
+conducting
+Scout
+buying
+##wich
+spreading
+##OC
+array
+hurried
+Environment
+improving
+prompted
+fierce
+Taking
+Away
+tune
+pissed
+Bull
+catching
+##ying
+eyebrow
+metropolitan
+terrain
+##rel
+Lodge
+manufacturers
+creator
+##etic
+happiness
+ports
+##ners
+Relations
+fortress
+targeted
+##ST
+allegedly
+blues
+##osa
+Bosnia
+##dom
+burial
+similarly
+stranger
+pursued
+symbols
+rebels
+reflection
+routine
+traced
+indoor
+eventual
+##ska
+##ão
+##una
+MD
+##phone
+oh
+grants
+Reynolds
+rid
+operators
+##nus
+Joey
+vital
+siblings
+keyboard
+br
+removing
+societies
+drives
+solely
+princess
+lighter
+Various
+Cavalry
+believing
+SC
+underwent
+relay
+smelled
+syndrome
+welfare
+authorized
+seemingly
+Hard
+chicken
+##rina
+Ages
+Bo
+democratic
+barn
+Eye
+shorts
+##coming
+##hand
+disappointed
+unexpected
+centres
+Exhibition
+Stories
+Site
+banking
+accidentally
+Agent
+conjunction
+André
+Chloe
+resist
+width
+Queens
+provision
+##art
+Melissa
+Honorary
+Del
+prefer
+abruptly
+duration
+##vis
+Glass
+enlisted
+##ado
+discipline
+Sisters
+carriage
+##ctor
+##sburg
+Lancashire
+log
+fuck
+##iz
+closet
+collecting
+holy
+rape
+trusted
+cleaning
+inhabited
+Rocky
+104
+editorial
+##yu
+##ju
+succeed
+strict
+Cuban
+##iya
+Bronze
+outcome
+##ifies
+##set
+corps
+Hero
+barrier
+Kumar
+groaned
+Nina
+Burton
+enable
+stability
+Milton
+knots
+##ination
+slavery
+##borg
+curriculum
+trailer
+warfare
+Dante
+Edgar
+revival
+Copenhagen
+define
+advocate
+Garrett
+Luther
+overcome
+pipe
+750
+construct
+Scotia
+kings
+flooding
+##hard
+Ferdinand
+Felix
+forgot
+Fish
+Kurt
+elaborate
+##BC
+graphic
+gripped
+colonel
+Sophia
+Advisory
+Self
+##uff
+##lio
+monitoring
+seal
+senses
+rises
+peaceful
+journals
+1837
+checking
+legendary
+Ghana
+##power
+ammunition
+Rosa
+Richards
+nineteenth
+ferry
+aggregate
+Troy
+inter
+##wall
+Triple
+steep
+tent
+Cyprus
+1844
+##woman
+commanding
+farms
+doi
+navy
+specified
+na
+cricketer
+transported
+Think
+comprising
+grateful
+solve
+##core
+beings
+clerk
+grain
+vector
+discrimination
+##TC
+Katie
+reasonable
+drawings
+veins
+consideration
+Monroe
+repeat
+breed
+dried
+witnessed
+ordained
+Current
+spirits
+remarkable
+consultant
+urged
+Remember
+anime
+singers
+phenomenon
+Rhode
+Carlo
+demanding
+findings
+manual
+varying
+Fellowship
+generate
+safely
+heated
+withdrawn
+##ao
+headquartered
+##zon
+##lav
+##ency
+Col
+Memphis
+imposed
+rivals
+Planet
+healing
+##hs
+ensemble
+Warriors
+##bone
+cult
+Frankfurt
+##HL
+diversity
+Gerald
+intermediate
+##izes
+reactions
+Sister
+##ously
+##lica
+quantum
+awkward
+mentions
+pursuit
+##ography
+varies
+profession
+molecular
+consequence
+lectures
+cracked
+103
+slowed
+##tsu
+cheese
+upgraded
+suite
+substance
+Kingston
+1800
+Idaho
+Theory
+##een
+ain
+Carson
+Molly
+##OR
+configuration
+Whitney
+reads
+audiences
+##tie
+Geneva
+Outside
+##nen
+##had
+transit
+volleyball
+Randy
+Chad
+rubber
+motorcycle
+respected
+eager
+Level
+coin
+##lets
+neighbouring
+##wski
+confident
+##cious
+poll
+uncertain
+punch
+thesis
+Tucker
+IATA
+Alec
+##ographic
+##law
+1841
+desperately
+1812
+Lithuania
+accent
+Cox
+lightning
+skirt
+##load
+Burns
+Dynasty
+##ug
+chapters
+Working
+dense
+Morocco
+##kins
+casting
+Set
+activated
+oral
+Brien
+horn
+HIV
+dawn
+stumbled
+altar
+tore
+considerably
+Nicole
+interchange
+registration
+biography
+Hull
+Stan
+bulk
+consent
+Pierce
+##ER
+Fifth
+marched
+terrorist
+##piece
+##itt
+Presidential
+Heather
+staged
+Plant
+relegation
+sporting
+joins
+##ced
+Pakistani
+dynamic
+Heat
+##lf
+ourselves
+Except
+Elliott
+nationally
+goddess
+investors
+Burke
+Jackie
+##ā
+##RA
+Tristan
+Associate
+Tuesday
+scope
+Near
+bunch
+##abad
+##ben
+sunlight
+##aire
+manga
+Willie
+trucks
+boarding
+Lion
+lawsuit
+Learning
+Der
+pounding
+awful
+##mine
+IT
+Legend
+romance
+Serie
+AC
+gut
+precious
+Robertson
+hometown
+realm
+Guards
+Tag
+batting
+##vre
+halt
+conscious
+1838
+acquire
+collar
+##gg
+##ops
+Herald
+nationwide
+citizenship
+Aircraft
+decrease
+em
+Fiction
+Female
+corporation
+Located
+##ip
+fights
+unconscious
+Tampa
+Poetry
+lobby
+Malta
+##sar
+##bie
+layout
+Tate
+reader
+stained
+##bre
+##rst
+##ulate
+loudly
+Eva
+Cohen
+exploded
+Merit
+Maya
+##rable
+Rovers
+##IC
+Morrison
+Should
+vinyl
+##mie
+onwards
+##gie
+vicinity
+Wildlife
+probability
+Mar
+Barnes
+##ook
+spinning
+Moses
+##vie
+Surrey
+Planning
+conferences
+protective
+Plaza
+deny
+Canterbury
+manor
+Estate
+tilted
+comics
+IBM
+destroying
+server
+Dorothy
+##horn
+Oslo
+lesser
+heaven
+Marshal
+scales
+strikes
+##ath
+firms
+attract
+##BS
+controlling
+Bradford
+southeastern
+Amazon
+Travis
+Janet
+governed
+1842
+Train
+Holden
+bleeding
+gifts
+rent
+1839
+palms
+##ū
+judicial
+Ho
+Finals
+conflicts
+unlikely
+draws
+##cies
+compensation
+adds
+elderly
+Anton
+lasting
+Nintendo
+codes
+ministers
+pot
+associations
+capabilities
+##cht
+libraries
+##sie
+chances
+performers
+runway
+##af
+##nder
+Mid
+Vocals
+##uch
+##eon
+interpreted
+priority
+Uganda
+ruined
+Mathematics
+cook
+AFL
+Lutheran
+AIDS
+Capitol
+chase
+axis
+Moreover
+María
+Saxon
+storyline
+##ffed
+Tears
+Kid
+cent
+colours
+Sex
+##long
+pm
+blonde
+Edwin
+CE
+diocese
+##ents
+##boy
+Inn
+##ller
+Saskatchewan
+##kh
+stepping
+Windsor
+##oka
+##eri
+Xavier
+Resources
+1843
+##top
+##rad
+##lls
+Testament
+poorly
+1836
+drifted
+slope
+CIA
+remix
+Lords
+mature
+hosting
+diamond
+beds
+##ncies
+luxury
+trigger
+##lier
+preliminary
+hybrid
+journalists
+Enterprise
+proven
+expelled
+insects
+Beautiful
+lifestyle
+vanished
+##ake
+##ander
+matching
+surfaces
+Dominican
+Kids
+referendum
+Orlando
+Truth
+Sandy
+privacy
+Calgary
+Speaker
+sts
+Nobody
+shifting
+##gers
+Roll
+Armenia
+Hand
+##ES
+106
+##ont
+Guild
+larvae
+Stock
+flame
+gravity
+enhanced
+Marion
+surely
+##tering
+Tales
+algorithm
+Emmy
+darker
+VIII
+##lash
+hamlet
+deliberately
+occurring
+choices
+Gage
+fees
+settling
+ridiculous
+##ela
+Sons
+cop
+custody
+##ID
+proclaimed
+Cardinals
+##pm
+Metal
+Ana
+1835
+clue
+Cardiff
+riders
+observations
+MA
+sometime
+##och
+performer
+intact
+Points
+allegations
+rotation
+Tennis
+tenor
+Directors
+##ats
+Transit
+thigh
+Complex
+##works
+twentieth
+Factory
+doctrine
+Daddy
+##ished
+pretend
+Winston
+cigarette
+##IA
+specimens
+hydrogen
+smoking
+mathematical
+arguments
+openly
+developer
+##iro
+fists
+somebody
+##san
+Standing
+Caleb
+intelligent
+Stay
+Interior
+echoed
+Valentine
+varieties
+Brady
+cluster
+Ever
+voyage
+##of
+deposits
+ultimate
+Hayes
+horizontal
+proximity
+##ás
+estates
+exploration
+NATO
+Classical
+##most
+bills
+condemned
+1832
+hunger
+##ato
+planes
+deserve
+offense
+sequences
+rendered
+acceptance
+##ony
+manufacture
+Plymouth
+innovative
+predicted
+##RC
+Fantasy
+##une
+supporter
+absent
+Picture
+bassist
+rescued
+##MC
+Ahmed
+Monte
+##sts
+##rius
+insane
+novelist
+##és
+agrees
+Antarctic
+Lancaster
+Hopkins
+calculated
+startled
+##star
+tribal
+Amendment
+##hoe
+invisible
+patron
+deer
+Walk
+tracking
+Lyon
+tickets
+##ED
+philosopher
+compounds
+chuckled
+##wi
+pound
+loyalty
+Academic
+petition
+refuses
+marking
+Mercury
+northeastern
+dimensions
+scandal
+Canyon
+patch
+publish
+##oning
+Peak
+minds
+##boro
+Presbyterian
+Hardy
+theoretical
+magnitude
+bombs
+cage
+##ders
+##kai
+measuring
+explaining
+avoiding
+touchdowns
+Card
+theology
+##ured
+Popular
+export
+suspicious
+Probably
+photograph
+Lou
+Parks
+Arms
+compact
+Apparently
+excess
+Banks
+lied
+stunned
+territorial
+Filipino
+spectrum
+learns
+wash
+imprisonment
+ugly
+##rose
+Albany
+Erik
+sends
+##hara
+##rid
+consumed
+##gling
+Belgrade
+Da
+opposing
+Magnus
+footsteps
+glowing
+delicate
+Alexandria
+Ludwig
+gorgeous
+Bros
+Index
+##PA
+customs
+preservation
+bonds
+##mond
+environments
+##nto
+instructed
+parted
+adoption
+locality
+workshops
+goalkeeper
+##rik
+##uma
+Brighton
+Slovenia
+##ulating
+##tical
+towel
+hugged
+stripped
+Bears
+upright
+Wagner
+##aux
+secretly
+Adventures
+nest
+Course
+Lauren
+Boeing
+Abdul
+Lakes
+450
+##cu
+USSR
+caps
+Chan
+##nna
+conceived
+Actually
+Belfast
+Lithuanian
+concentrate
+possess
+militia
+pine
+protagonist
+Helena
+##PS
+##band
+Belle
+Clara
+Reform
+currency
+pregnancy
+1500
+##rim
+Isabella
+hull
+Name
+trend
+journalism
+diet
+##mel
+Recording
+acclaimed
+Tang
+Jace
+steering
+vacant
+suggestion
+costume
+laser
+##š
+##ink
+##pan
+##vić
+integral
+achievements
+wise
+classroom
+unions
+southwestern
+##uer
+Garcia
+toss
+Tara
+Large
+##tate
+evident
+responsibilities
+populated
+satisfaction
+##bia
+casual
+Ecuador
+##ght
+arose
+##ović
+Cornwall
+embrace
+refuse
+Heavyweight
+XI
+Eden
+activists
+##uation
+biology
+##shan
+fraud
+Fuck
+matched
+legacy
+Rivers
+missionary
+extraordinary
+Didn
+holder
+wickets
+crucial
+Writers
+Hurricane
+Iceland
+gross
+trumpet
+accordance
+hurry
+flooded
+doctorate
+Albania
+##yi
+united
+deceased
+jealous
+grief
+flute
+portraits
+##а
+pleasant
+Founded
+Face
+crowned
+Raja
+advisor
+Salem
+##ec
+Achievement
+admission
+freely
+minimal
+Sudan
+developers
+estimate
+disabled
+##lane
+downstairs
+Bruno
+##pus
+pinyin
+##ude
+lecture
+deadly
+underlying
+optical
+witnesses
+Combat
+Julius
+tapped
+variants
+##like
+Colonial
+Critics
+Similarly
+mouse
+voltage
+sculptor
+Concert
+salary
+Frances
+##ground
+hook
+premises
+Software
+instructor
+nominee
+##ited
+fog
+slopes
+##zu
+vegetation
+sail
+##rch
+Body
+Apart
+atop
+View
+utility
+ribs
+cab
+migration
+##wyn
+bounded
+2019
+pillow
+trails
+##ub
+Halifax
+shade
+Rush
+##lah
+##dian
+Notre
+interviewed
+Alexandra
+Springfield
+Indeed
+rubbing
+dozens
+amusement
+legally
+##lers
+Jill
+Cinema
+ignoring
+Choice
+##ures
+pockets
+##nell
+laying
+Blair
+tackles
+separately
+##teen
+Criminal
+performs
+theorem
+Communication
+suburbs
+##iel
+competitors
+rows
+##hai
+Manitoba
+Eleanor
+interactions
+nominations
+assassination
+##dis
+Edmonton
+diving
+##dine
+essay
+##tas
+AFC
+Edge
+directing
+imagination
+sunk
+implement
+Theodore
+trembling
+sealed
+##rock
+Nobel
+##ancy
+##dorf
+##chen
+genuine
+apartments
+Nicolas
+AA
+Bach
+Globe
+Store
+220
+##10
+Rochester
+##ño
+alert
+107
+Beck
+##nin
+Naples
+Basin
+Crawford
+fears
+Tracy
+##hen
+disk
+##pped
+seventeen
+Lead
+backup
+reconstruction
+##lines
+terrified
+sleeve
+nicknamed
+popped
+##making
+##ern
+Holiday
+Gospel
+ibn
+##ime
+convert
+divine
+resolved
+##quet
+ski
+realizing
+##RT
+Legislature
+reservoir
+Rain
+sinking
+rainfall
+elimination
+challenging
+tobacco
+##outs
+Given
+smallest
+Commercial
+pin
+rebel
+comedian
+exchanged
+airing
+dish
+Salvador
+promising
+##wl
+relax
+presenter
+toll
+aerial
+##eh
+Fletcher
+brass
+disappear
+zones
+adjusted
+contacts
+##lk
+sensed
+Walt
+mild
+toes
+flies
+shame
+considers
+wildlife
+Hanna
+Arsenal
+Ladies
+naming
+##ishing
+anxiety
+discussions
+cute
+undertaken
+Cash
+strain
+Wyoming
+dishes
+precise
+Angela
+##ided
+hostile
+twins
+115
+Built
+##pel
+Online
+tactics
+Newman
+##bourne
+unclear
+repairs
+embarrassed
+listing
+tugged
+Vale
+##gin
+Meredith
+bout
+##cle
+velocity
+tips
+froze
+evaluation
+demonstrate
+##card
+criticised
+Nash
+lineup
+Rao
+monks
+bacteria
+lease
+##lish
+frightened
+den
+revived
+finale
+##rance
+flee
+Letters
+decreased
+##oh
+Sounds
+wrap
+Sharon
+incidents
+renovated
+everybody
+stole
+Bath
+boxing
+1815
+withdraw
+backs
+interim
+react
+murders
+Rhodes
+Copa
+framed
+flown
+Estonia
+Heavy
+explored
+##rra
+##GA
+##ali
+Istanbul
+1834
+##rite
+##aging
+##ues
+Episcopal
+arc
+orientation
+Maxwell
+infected
+##rot
+BCE
+Brook
+grasp
+Roberto
+Excellence
+108
+withdrawal
+Marines
+rider
+Lo
+##sin
+##run
+Subsequently
+garrison
+hurricane
+facade
+Prussia
+crushed
+enterprise
+##mber
+Twitter
+Generation
+Physical
+Sugar
+editing
+communicate
+Ellie
+##hurst
+Ernst
+wagon
+promotional
+conquest
+Parliamentary
+courtyard
+lawyers
+Superman
+email
+Prussian
+lately
+lecturer
+Singer
+Majesty
+Paradise
+sooner
+Heath
+slot
+curves
+convoy
+##vian
+induced
+synonym
+breeze
+##plane
+##ox
+peered
+Coalition
+##hia
+odds
+##esh
+##lina
+Tomorrow
+Nadu
+##ico
+##rah
+damp
+autonomous
+console
+Victory
+counts
+Luxembourg
+intimate
+Archived
+Carroll
+spy
+Zero
+habit
+Always
+faction
+teenager
+Johnston
+chaos
+ruin
+commerce
+blog
+##shed
+##the
+reliable
+Word
+Yu
+Norton
+parade
+Catholics
+damned
+##iling
+surgeon
+##tia
+Allison
+Jonas
+remarked
+##ès
+idiot
+Making
+proposals
+Industries
+strategies
+artifacts
+batteries
+reward
+##vers
+Agricultural
+distinguish
+lengths
+Jeffrey
+Progressive
+kicking
+Patricia
+##gio
+ballot
+##ios
+skilled
+##gation
+Colt
+limestone
+##AS
+peninsula
+##itis
+LA
+hotels
+shapes
+Crime
+depicting
+northwestern
+HD
+silly
+Das
+##²
+##ws
+##ash
+##matic
+thermal
+Has
+forgive
+surrendered
+Palm
+Nacional
+drank
+haired
+Mercedes
+##foot
+loading
+Timothy
+##roll
+mechanisms
+traces
+digging
+discussing
+Natalie
+##zhou
+Forbes
+landmark
+Anyway
+Manor
+conspiracy
+gym
+knocking
+viewing
+Formation
+Pink
+Beauty
+limbs
+Phillip
+sponsor
+Joy
+granite
+Harbour
+##ero
+payments
+Ballet
+conviction
+##dam
+Hood
+estimates
+lacked
+Mad
+Jorge
+##wen
+refuge
+##LA
+invaded
+Kat
+suburban
+##fold
+investigated
+Ari
+complained
+creek
+Georges
+##uts
+powder
+accepting
+deserved
+carpet
+Thunder
+molecules
+Legal
+cliff
+strictly
+enrollment
+ranch
+##rg
+##mba
+proportion
+renovation
+crop
+grabbing
+##liga
+finest
+entries
+receptor
+helmet
+blown
+Listen
+flagship
+workshop
+resolve
+nails
+Shannon
+portal
+jointly
+shining
+Violet
+overwhelming
+upward
+Mick
+proceedings
+##dies
+##aring
+Laurence
+Churchill
+##rice
+commit
+170
+inclusion
+Examples
+##verse
+##rma
+fury
+paths
+##SC
+ankle
+nerves
+Chemistry
+rectangular
+sworn
+screenplay
+cake
+Mann
+Seoul
+Animal
+sizes
+Speed
+vol
+Population
+Southwest
+Hold
+continuously
+Qualified
+wishing
+Fighting
+Made
+disappointment
+Portsmouth
+Thirty
+##beck
+Ahmad
+teammate
+MLB
+graph
+Charleston
+realizes
+##dium
+exhibits
+preventing
+##int
+fever
+rivalry
+Male
+mentally
+dull
+##lor
+##rich
+consistently
+##igan
+Madame
+certificate
+suited
+Krishna
+accuracy
+Webb
+Budapest
+Rex
+1831
+Cornell
+OK
+surveillance
+##gated
+habitats
+Adventure
+Conrad
+Superior
+Gay
+sofa
+aka
+boot
+Statistics
+Jessie
+Liberation
+##lip
+##rier
+brands
+saint
+Heinrich
+Christine
+bath
+Rhine
+ballet
+Jin
+consensus
+chess
+Arctic
+stack
+furious
+cheap
+toy
+##yre
+##face
+##gging
+gastropod
+##nne
+Romans
+membrane
+answering
+25th
+architects
+sustainable
+##yne
+Hon
+1814
+Baldwin
+dome
+##awa
+##zen
+celebrity
+enclosed
+##uit
+##mmer
+Electronic
+locals
+##CE
+supervision
+mineral
+Chemical
+Slovakia
+alley
+hub
+##az
+heroes
+Creative
+##AM
+incredible
+politically
+ESPN
+yanked
+halls
+Aboriginal
+Greatest
+yield
+##20
+congressional
+robot
+Kiss
+welcomed
+MS
+speeds
+proceed
+Sherman
+eased
+Greene
+Walsh
+Geoffrey
+variables
+rocky
+##print
+acclaim
+Reverend
+Wonder
+tonnes
+recurring
+Dawson
+continent
+finite
+AP
+continental
+ID
+facilitate
+essays
+Rafael
+Neal
+1833
+ancestors
+##met
+##gic
+Especially
+teenage
+frustrated
+Jules
+cock
+expense
+##oli
+##old
+blocking
+Notable
+prohibited
+ca
+dock
+organize
+##wald
+Burma
+Gloria
+dimension
+aftermath
+choosing
+Mickey
+torpedo
+pub
+##used
+manuscripts
+laps
+Ulster
+staircase
+sphere
+Insurance
+Contest
+lens
+risks
+investigations
+ERA
+glare
+##play
+Graduate
+auction
+Chronicle
+##tric
+##50
+Coming
+seating
+Wade
+seeks
+inland
+Thames
+Rather
+butterfly
+contracted
+positioned
+consumers
+contestants
+fragments
+Yankees
+Santos
+administrator
+hypothesis
+retire
+Denis
+agreements
+Winnipeg
+##rill
+1820
+trophy
+crap
+shakes
+Jenkins
+##rium
+ya
+twist
+labels
+Maritime
+##lings
+##iv
+111
+##ensis
+Cairo
+Anything
+##fort
+opinions
+crowded
+##nian
+abandon
+##iff
+drained
+imported
+##rr
+tended
+##rain
+Going
+introducing
+sculptures
+bankruptcy
+danced
+demonstration
+stance
+settings
+gazed
+abstract
+pet
+Calvin
+stiff
+strongest
+wrestler
+##dre
+Republicans
+grace
+allocated
+cursed
+snail
+advancing
+Return
+errors
+Mall
+presenting
+eliminate
+Amateur
+Institution
+counting
+##wind
+warehouse
+##nde
+Ethiopia
+trailed
+hollow
+##press
+Literary
+capability
+nursing
+preceding
+lamp
+Thomson
+Morton
+##ctic
+Crew
+Close
+composers
+boom
+Clare
+missiles
+112
+hunter
+snap
+##oni
+##tail
+Us
+declaration
+##cock
+rally
+huh
+lion
+straightened
+Philippe
+Sutton
+alpha
+valued
+maker
+navigation
+detected
+favorable
+perception
+Charter
+##ña
+Ricky
+rebounds
+tunnels
+slapped
+Emergency
+supposedly
+##act
+deployment
+socialist
+tubes
+anybody
+corn
+##NA
+Seminary
+heating
+pump
+##AA
+achieving
+souls
+##ass
+Link
+##ele
+##smith
+greeted
+Bates
+Americas
+Elder
+cure
+contestant
+240
+fold
+Runner
+Uh
+licked
+Politics
+committees
+neighbors
+fairy
+Silva
+Leipzig
+tipped
+correctly
+exciting
+electronics
+foundations
+cottage
+governmental
+##hat
+allied
+claws
+presidency
+cruel
+Agreement
+slender
+accompanying
+precisely
+##pass
+driveway
+swim
+Stand
+crews
+##mission
+rely
+everyday
+Wings
+demo
+##hic
+recreational
+min
+nationality
+##duction
+Easter
+##hole
+canvas
+Kay
+Leicester
+talented
+Discovery
+shells
+##ech
+Kerry
+Ferguson
+Leave
+##place
+altogether
+adopt
+butt
+wolves
+##nsis
+##ania
+modest
+soprano
+Boris
+##ught
+electron
+depicts
+hid
+cruise
+differ
+treasure
+##nch
+Gun
+Mama
+Bengali
+trainer
+merchants
+innovation
+presumably
+Shirley
+bottles
+proceeds
+Fear
+invested
+Pirates
+particle
+Dominic
+blamed
+Fight
+Daisy
+##pper
+##graphic
+nods
+knight
+Doyle
+tales
+Carnegie
+Evil
+Inter
+Shore
+Nixon
+transform
+Savannah
+##gas
+Baltic
+stretching
+worlds
+protocol
+Percy
+Toby
+Heroes
+brave
+dancers
+##aria
+backwards
+responses
+Chi
+Gaelic
+Berry
+crush
+embarked
+promises
+Madonna
+researcher
+realised
+inaugurated
+Cherry
+Mikhail
+Nottingham
+reinforced
+subspecies
+rapper
+##kie
+Dreams
+Re
+Damon
+Minneapolis
+monsters
+suspicion
+Tel
+surroundings
+afterward
+complaints
+OF
+sectors
+Algeria
+lanes
+Sabha
+objectives
+Donna
+bothered
+distracted
+deciding
+##ives
+##CA
+##onia
+bishops
+Strange
+machinery
+Voiced
+synthesis
+reflects
+interference
+##TS
+##ury
+keen
+##ign
+frown
+freestyle
+ton
+Dixon
+Sacred
+Ruby
+Prison
+##ión
+1825
+outfit
+##tain
+curiosity
+##ight
+frames
+steadily
+emigrated
+horizon
+##erly
+Doc
+philosophical
+Table
+UTC
+Marina
+##DA
+secular
+##eed
+Zimbabwe
+cops
+Mack
+sheriff
+Sanskrit
+Francesco
+catches
+questioning
+streaming
+Kill
+testimony
+hissed
+tackle
+countryside
+copyright
+##IP
+Buddhism
+##rator
+ladder
+##ON
+Past
+rookie
+depths
+##yama
+##ister
+##HS
+Samantha
+Dana
+Educational
+brows
+Hammond
+raids
+envelope
+##sco
+##hart
+##ulus
+epic
+detection
+Streets
+Potter
+statistical
+für
+ni
+accounting
+##pot
+employer
+Sidney
+Depression
+commands
+Tracks
+averaged
+lets
+Ram
+longtime
+suits
+branded
+chip
+Shield
+loans
+ought
+Said
+sip
+##rome
+requests
+Vernon
+bordered
+veterans
+##ament
+Marsh
+Herzegovina
+Pine
+##igo
+mills
+anticipation
+reconnaissance
+##ef
+expectations
+protested
+arrow
+guessed
+depot
+maternal
+weakness
+##ap
+projected
+pour
+Carmen
+provider
+newer
+remind
+freed
+##rily
+##wal
+##tones
+intentions
+Fiji
+timing
+Match
+managers
+Kosovo
+Herman
+Wesley
+Chang
+135
+semifinals
+shouting
+Indo
+Janeiro
+Chess
+Macedonia
+Buck
+##onies
+rulers
+Mail
+##vas
+##sel
+MHz
+Programme
+Task
+commercially
+subtle
+propaganda
+spelled
+bowling
+basically
+Raven
+1828
+Colony
+109
+##ingham
+##wara
+anticipated
+1829
+##iers
+graduates
+##rton
+##fication
+endangered
+ISO
+diagnosed
+##tage
+exercises
+Battery
+bolt
+poison
+cartoon
+##ción
+hood
+bowed
+heal
+Meyer
+Reagan
+##wed
+subfamily
+##gent
+momentum
+infant
+detect
+##sse
+Chapman
+Darwin
+mechanics
+NSW
+Cancer
+Brooke
+Nuclear
+comprised
+hire
+sanctuary
+wingspan
+contrary
+remembering
+surprising
+Basic
+stealing
+OS
+hatred
+##lled
+masters
+violation
+Rule
+##nger
+assuming
+conquered
+louder
+robe
+Beatles
+legitimate
+##vation
+massacre
+Rica
+unsuccessfully
+poets
+##enberg
+careers
+doubled
+premier
+battalions
+Dubai
+Paper
+Louisville
+gestured
+dressing
+successive
+mumbled
+Vic
+referee
+pupil
+##cated
+##rre
+ceremonies
+picks
+##IN
+diplomat
+alike
+geographical
+rays
+##HA
+##read
+harbour
+factories
+pastor
+playwright
+Ultimate
+nationalist
+uniforms
+obtaining
+kit
+Amber
+##pling
+screenwriter
+ancestry
+##cott
+Fields
+PR
+Coleman
+rat
+Bavaria
+squeeze
+highlighted
+Adult
+reflecting
+Mel
+1824
+bicycle
+organizing
+sided
+Previously
+Underground
+Prof
+athletics
+coupled
+mortal
+Hampton
+worthy
+immune
+Ava
+##gun
+encouraging
+simplified
+##ssa
+##nte
+##ann
+Providence
+entities
+Pablo
+Strong
+Housing
+##ista
+##ators
+kidnapped
+mosque
+Kirk
+whispers
+fruits
+shattered
+fossil
+Empress
+Johns
+Webster
+Thing
+refusing
+differently
+specimen
+Ha
+##EN
+##tina
+##elle
+##night
+Horn
+neighbourhood
+Bolivia
+##rth
+genres
+Pre
+##vich
+Amelia
+swallow
+Tribune
+Forever
+Psychology
+Use
+##bers
+Gazette
+ash
+##usa
+Monster
+##cular
+delegation
+blowing
+Oblast
+retreated
+automobile
+##ex
+profits
+shirts
+devil
+Treasury
+##backs
+Drums
+Ronnie
+gameplay
+expertise
+Evening
+resides
+Caesar
+unity
+Crazy
+linking
+Vision
+donations
+Isabel
+valve
+Sue
+WWE
+logical
+availability
+fitting
+revolt
+##mill
+Linux
+taxi
+Access
+pollution
+statues
+Augustus
+##pen
+cello
+##some
+lacking
+##ati
+Gwen
+##aka
+##ovich
+1821
+Wow
+initiatives
+Uruguay
+Cain
+stroked
+examine
+##ī
+mentor
+moist
+disorders
+buttons
+##tica
+##anna
+Species
+Lynch
+museums
+scorer
+Poor
+eligibility
+op
+unveiled
+cats
+Title
+wheat
+critically
+Syracuse
+##osis
+marketed
+enhance
+Ryder
+##NG
+##ull
+##rna
+embedded
+throws
+foods
+happily
+##ami
+lesson
+formats
+punched
+##rno
+expressions
+qualities
+##sal
+Gods
+##lity
+elect
+wives
+##lling
+jungle
+Toyota
+reversed
+Grammar
+Cloud
+Agnes
+##ules
+disputed
+verses
+Lucien
+threshold
+##rea
+scanned
+##bled
+##dley
+##lice
+Kazakhstan
+Gardner
+Freeman
+##rz
+inspection
+Rita
+accommodation
+advances
+chill
+Elliot
+thriller
+Constantinople
+##mos
+debris
+whoever
+1810
+Santo
+Carey
+remnants
+Guatemala
+##irs
+carriers
+equations
+mandatory
+##WA
+anxious
+measurement
+Summit
+Terminal
+Erin
+##zes
+LLC
+##uo
+glancing
+sin
+##₃
+Downtown
+flowering
+Euro
+Leigh
+Lance
+warn
+decent
+recommendations
+##ote
+Quartet
+##rrell
+Clarence
+colleague
+guarantee
+230
+Clayton
+Beast
+addresses
+prospect
+destroyer
+vegetables
+Leadership
+fatal
+prints
+190
+##makers
+Hyde
+persuaded
+illustrations
+Southampton
+Joyce
+beats
+editors
+mount
+##grave
+Malaysian
+Bombay
+endorsed
+##sian
+##bee
+applying
+Religion
+nautical
+bomber
+Na
+airfield
+gravel
+##rew
+Cave
+bye
+dig
+decree
+burden
+Election
+Hawk
+Fe
+##iled
+reunited
+##tland
+liver
+Teams
+Put
+delegates
+Ella
+##fect
+Cal
+invention
+Castro
+bored
+##kawa
+##ail
+Trinidad
+NASCAR
+pond
+develops
+##pton
+expenses
+Zoe
+Released
+##rf
+organs
+beta
+parameters
+Neill
+##lene
+lateral
+Beat
+blades
+Either
+##hale
+Mitch
+##ET
+##vous
+Rod
+burnt
+phones
+Rising
+##front
+investigating
+##dent
+Stephanie
+##keeper
+screening
+##uro
+Swan
+Sinclair
+modes
+bullets
+Nigerian
+melody
+##ques
+Rifle
+##12
+128
+##jin
+charm
+Venus
+##tian
+fusion
+advocated
+visitor
+pinned
+genera
+3000
+Ferry
+Solo
+quantity
+regained
+platinum
+shoots
+narrowly
+preceded
+update
+##ichi
+equality
+unaware
+regiments
+ally
+##tos
+transmitter
+locks
+Seeing
+outlets
+feast
+reopened
+##ows
+struggles
+Buddy
+1826
+bark
+elegant
+amused
+Pretty
+themed
+schemes
+Lisbon
+Te
+patted
+terrorism
+Mystery
+##croft
+##imo
+Madagascar
+Journey
+dealer
+contacted
+##quez
+ITV
+vacation
+Wong
+Sacramento
+organisms
+##pts
+balcony
+coloured
+sheer
+defines
+MC
+abortion
+forbidden
+accredited
+Newfoundland
+tendency
+entrepreneur
+Benny
+Tanzania
+needing
+finalist
+mythology
+weakened
+gown
+sentences
+Guest
+websites
+Tibetan
+UFC
+voluntary
+annoyed
+Welcome
+honestly
+correspondence
+geometry
+Deutsche
+Biology
+Help
+##aya
+Lines
+Hector
+##ael
+reluctant
+##ages
+wears
+inquiry
+##dell
+Holocaust
+Tourism
+Wei
+volcanic
+##mates
+Visual
+sorts
+neighborhoods
+Running
+apple
+shy
+Laws
+bend
+Northeast
+feminist
+Speedway
+Murder
+visa
+stuffed
+fangs
+transmitted
+fiscal
+Ain
+enlarged
+##ndi
+Cecil
+Peterson
+Benson
+Bedford
+acceptable
+##CC
+##wer
+purely
+triangle
+foster
+Alberto
+educator
+Highland
+acute
+LGBT
+Tina
+Mi
+adventures
+Davidson
+Honda
+translator
+monk
+enacted
+summoned
+##ional
+collector
+Genesis
+Un
+liner
+Di
+Statistical
+##CS
+filter
+Knox
+Religious
+Stella
+Estonian
+Turn
+##ots
+primitive
+parishes
+##lles
+complexity
+autobiography
+rigid
+cannon
+pursuing
+exploring
+##gram
+##mme
+freshman
+caves
+Expedition
+Traditional
+iTunes
+certification
+cooling
+##ort
+##gna
+##IT
+##lman
+##VA
+Motion
+explosive
+licence
+boxer
+shrine
+loosely
+Brigadier
+Savage
+Brett
+MVP
+heavier
+##elli
+##gged
+Buddha
+Easy
+spells
+fails
+incredibly
+Georg
+stern
+compatible
+Perfect
+applies
+cognitive
+excessive
+nightmare
+neighbor
+Sicily
+appealed
+static
+##₁
+Aberdeen
+##leigh
+slipping
+bride
+##guard
+Um
+Clyde
+1818
+##gible
+Hal
+Frost
+Sanders
+interactive
+Hour
+##vor
+hurting
+bull
+termed
+shelf
+capturing
+##pace
+rolls
+113
+##bor
+Chilean
+teaches
+##rey
+exam
+shipped
+Twin
+borrowed
+##lift
+Shit
+##hot
+Lindsay
+Below
+Kiev
+Lin
+leased
+##sto
+Eli
+Diane
+Val
+subtropical
+shoe
+Bolton
+Dragons
+##rification
+Vatican
+##pathy
+Crisis
+dramatically
+talents
+babies
+##ores
+surname
+##AP
+##cology
+cubic
+opted
+Archer
+sweep
+tends
+Karnataka
+Judy
+stint
+Similar
+##nut
+explicitly
+##nga
+interact
+Mae
+portfolio
+clinic
+abbreviated
+Counties
+##iko
+hearts
+##ı
+providers
+screams
+Individual
+##etti
+Monument
+##iana
+accessed
+encounters
+gasp
+##rge
+defunct
+Avery
+##rne
+nobility
+useless
+Phase
+Vince
+senator
+##FL
+1813
+surprisingly
+##illo
+##chin
+Boyd
+rumors
+equity
+Gone
+Hearts
+chassis
+overnight
+Trek
+wrists
+submit
+civic
+designers
+##rity
+prominence
+decorative
+derives
+starter
+##AF
+wisdom
+Powers
+reluctantly
+measurements
+doctoral
+Noel
+Gideon
+Baden
+Cologne
+lawn
+Hawaiian
+anthology
+##rov
+Raiders
+embassy
+Sterling
+##pal
+Telugu
+troubled
+##FC
+##bian
+fountain
+observe
+ore
+##uru
+##gence
+spelling
+Border
+grinning
+sketch
+Benedict
+Xbox
+dialects
+readily
+immigrant
+Constitutional
+aided
+nevertheless
+SE
+tragedy
+##ager
+##rden
+Flash
+##MP
+Europa
+emissions
+##ield
+panties
+Beverly
+Homer
+curtain
+##oto
+toilet
+Isn
+Jerome
+Chiefs
+Hermann
+supernatural
+juice
+integrity
+Scots
+auto
+Patriots
+Strategic
+engaging
+prosecution
+cleaned
+Byron
+investments
+adequate
+vacuum
+laughs
+##inus
+##nge
+Usually
+Roth
+Cities
+Brand
+corpse
+##ffy
+Gas
+rifles
+Plains
+sponsorship
+Levi
+tray
+owed
+della
+commanders
+##ead
+tactical
+##rion
+García
+harbor
+discharge
+##hausen
+gentleman
+endless
+highways
+##itarian
+pleaded
+##eta
+archive
+Midnight
+exceptions
+instances
+Gibraltar
+cart
+##NS
+Darren
+Bonnie
+##yle
+##iva
+OCLC
+bra
+Jess
+##EA
+consulting
+Archives
+Chance
+distances
+commissioner
+##AR
+LL
+sailors
+##sters
+enthusiasm
+Lang
+##zia
+Yugoslav
+confirm
+possibilities
+Suffolk
+##eman
+banner
+1822
+Supporting
+fingertips
+civilization
+##gos
+technically
+1827
+Hastings
+sidewalk
+strained
+monuments
+Floyd
+Chennai
+Elvis
+villagers
+Cumberland
+strode
+albeit
+Believe
+planets
+combining
+Mohammad
+container
+##mouth
+##tures
+verb
+BA
+Tank
+Midland
+screened
+Gang
+Democracy
+Helsinki
+screens
+thread
+charitable
+##version
+swiftly
+ma
+rational
+combine
+##SS
+##antly
+dragging
+Cliff
+Tasmania
+quest
+professionally
+##aj
+rap
+##lion
+livestock
+##hua
+informal
+specially
+lonely
+Matthews
+Dictionary
+1816
+Observatory
+correspondent
+constitute
+homeless
+waving
+appreciated
+Analysis
+Meeting
+dagger
+##AL
+Gandhi
+flank
+Giant
+Choir
+##not
+glimpse
+toe
+Writer
+teasing
+springs
+##dt
+Glory
+healthcare
+regulated
+complaint
+math
+Publications
+makers
+##hips
+cement
+Need
+apologize
+disputes
+finishes
+Partners
+boring
+ups
+gains
+1793
+Congressional
+clergy
+Folk
+##made
+##nza
+Waters
+stays
+encoded
+spider
+betrayed
+Applied
+inception
+##urt
+##zzo
+wards
+bells
+UCLA
+Worth
+bombers
+Mo
+trademark
+Piper
+##vel
+incorporates
+1801
+##cial
+dim
+Twelve
+##word
+Appeals
+tighter
+spacecraft
+##tine
+coordinates
+##iac
+mistakes
+Zach
+laptop
+Teresa
+##llar
+##yr
+favored
+Nora
+sophisticated
+Irving
+hammer
+División
+corporations
+niece
+##rley
+Patterson
+UNESCO
+trafficking
+Ming
+balanced
+plaque
+Latvia
+broader
+##owed
+Save
+confined
+##vable
+Dalton
+tide
+##right
+##ural
+##num
+swords
+caring
+##eg
+IX
+Acting
+paved
+##moto
+launching
+Antoine
+substantially
+Pride
+Philharmonic
+grammar
+Indoor
+Ensemble
+enabling
+114
+resided
+Angelo
+publicity
+chaired
+crawled
+Maharashtra
+Telegraph
+lengthy
+preference
+differential
+anonymous
+Honey
+##itation
+wage
+##iki
+consecrated
+Bryant
+regulatory
+Carr
+##én
+functioning
+watches
+##ú
+shifts
+diagnosis
+Search
+app
+Peters
+##SE
+##cat
+Andreas
+honours
+temper
+counsel
+Urdu
+Anniversary
+maritime
+##uka
+harmony
+##unk
+essence
+Lorenzo
+choked
+Quarter
+indie
+##oll
+loses
+##prints
+amendment
+Adolf
+scenario
+similarities
+##rade
+##LC
+technological
+metric
+Russians
+thoroughly
+##tead
+cruiser
+1806
+##nier
+1823
+Teddy
+##psy
+au
+progressed
+exceptional
+broadcaster
+partnered
+fitness
+irregular
+placement
+mothers
+unofficial
+Garion
+Johannes
+1817
+regain
+Solar
+publishes
+Gates
+Broken
+thirds
+conversations
+dive
+Raj
+contributor
+quantities
+Worcester
+governance
+##flow
+generating
+pretending
+Belarus
+##voy
+radius
+skating
+Marathon
+1819
+affection
+undertook
+##wright
+los
+##bro
+locate
+PS
+excluded
+recreation
+tortured
+jewelry
+moaned
+##logue
+##cut
+Complete
+##rop
+117
+##II
+plantation
+whipped
+slower
+crater
+##drome
+Volunteer
+attributes
+celebrations
+regards
+Publishers
+oath
+utilized
+Robbie
+Giuseppe
+fiber
+indication
+melted
+archives
+Damien
+storey
+affecting
+identifying
+dances
+alumni
+comparable
+upgrade
+rented
+sprint
+##kle
+Marty
+##lous
+treating
+railways
+Lebanese
+erupted
+occupy
+sympathy
+Jude
+Darling
+Qatar
+drainage
+McCarthy
+heel
+Klein
+computing
+wireless
+flip
+Du
+Bella
+##ast
+##ssen
+narrator
+mist
+sings
+alignment
+121
+2020
+securing
+##rail
+Progress
+missionaries
+brutal
+mercy
+##shing
+Hip
+##ache
+##olo
+switching
+##here
+Malay
+##ob
+constituted
+Mohammed
+Often
+standings
+surge
+teachings
+ink
+detached
+systematic
+Trial
+Myanmar
+##wo
+offs
+Reyes
+decoration
+translations
+wherever
+reviewer
+speculation
+Bangkok
+terminated
+##ester
+beard
+RCA
+Aidan
+Associated
+Emerson
+Charity
+1803
+generous
+Dudley
+ATP
+##haven
+prizes
+toxic
+gloves
+##iles
+##dos
+Turning
+myth
+Parade
+##building
+Hits
+##eva
+teamed
+Above
+Duchess
+Holt
+##oth
+Sub
+Ace
+atomic
+inform
+Ship
+depend
+Jun
+##bes
+Norwich
+globe
+Baroque
+Christina
+Cotton
+Tunnel
+kidding
+Concerto
+Brittany
+tasted
+phases
+stems
+angles
+##TE
+##nam
+##40
+charted
+Alison
+intensive
+Willis
+glory
+##lit
+Bergen
+est
+taller
+##dicate
+labeled
+##ido
+commentator
+Warrior
+Viscount
+shortened
+aisle
+Aria
+Spike
+spectators
+goodbye
+overlooking
+mammals
+##lude
+wholly
+Barrett
+##gus
+accompany
+seventy
+employ
+##mb
+ambitious
+beloved
+basket
+##mma
+##lding
+halted
+descendant
+pad
+exclaimed
+cloak
+##pet
+Strait
+Bang
+Aviv
+sadness
+##ffer
+Donovan
+1880s
+agenda
+swinging
+##quin
+jerk
+Boat
+##rist
+nervously
+Silence
+Echo
+shout
+implies
+##iser
+##cking
+Shiva
+Weston
+damages
+##tist
+effectiveness
+Horace
+cycling
+Rey
+ache
+Photography
+PDF
+Dear
+leans
+Lea
+##vision
+booth
+attained
+disbelief
+##eus
+##ution
+Hop
+pension
+toys
+Eurovision
+faithful
+##heads
+Andre
+owe
+default
+Atlas
+Megan
+highlights
+lovers
+Constantine
+Sixth
+masses
+##garh
+emerge
+Auto
+Slovak
+##oa
+##vert
+Superintendent
+flicked
+inventor
+Chambers
+Frankie
+Romeo
+pottery
+companions
+Rudolf
+##liers
+diary
+Unless
+tap
+alter
+Randall
+##ddle
+##eal
+limitations
+##boards
+utterly
+knelt
+guaranteed
+Cowboys
+Islander
+horns
+##ike
+Wendy
+sexually
+Smart
+breasts
+##cian
+compromise
+Duchy
+AT
+Galaxy
+analog
+Style
+##aking
+weighed
+Nigel
+optional
+Czechoslovakia
+practicing
+Ham
+##0s
+feedback
+batted
+uprising
+operative
+applicable
+criminals
+classrooms
+Somehow
+##ode
+##OM
+Naomi
+Winchester
+##pping
+Bart
+Regina
+competitor
+Recorded
+Yuan
+Vera
+lust
+Confederation
+##test
+suck
+1809
+Lambert
+175
+Friend
+##ppa
+Slowly
+##⁺
+Wake
+Dec
+##aneous
+chambers
+Color
+Gus
+##site
+Alternative
+##world
+Exeter
+Omaha
+celebrities
+striker
+210
+dwarf
+meals
+Oriental
+Pearson
+financing
+revenues
+underwater
+Steele
+screw
+Feeling
+Mt
+acids
+badge
+swore
+theaters
+Moving
+admired
+lung
+knot
+penalties
+116
+fork
+##cribed
+Afghan
+outskirts
+Cambodia
+oval
+wool
+fossils
+Ned
+Countess
+Darkness
+delicious
+##nica
+Evelyn
+Recordings
+guidelines
+##CP
+Sandra
+meantime
+Antarctica
+modeling
+granddaughter
+##rial
+Roma
+Seventh
+Sunshine
+Gabe
+##nton
+Shop
+Turks
+prolific
+soup
+parody
+##nta
+Judith
+disciplines
+resign
+Companies
+Libya
+Jets
+inserted
+Mile
+retrieve
+filmmaker
+##rand
+realistic
+unhappy
+##30
+sandstone
+##nas
+##lent
+##ush
+##rous
+Brent
+trash
+Rescue
+##unted
+Autumn
+disgust
+flexible
+infinite
+sideways
+##oss
+##vik
+trailing
+disturbed
+50th
+Newark
+posthumously
+##rol
+Schmidt
+Josef
+##eous
+determining
+menu
+Pole
+Anita
+Luc
+peaks
+118
+Yard
+warrant
+generic
+deserted
+Walking
+stamp
+tracked
+##berger
+paired
+surveyed
+sued
+Rainbow
+##isk
+Carpenter
+submarines
+realization
+touches
+sweeping
+Fritz
+module
+Whether
+resembles
+##form
+##lop
+unsure
+hunters
+Zagreb
+unemployment
+Senators
+Georgetown
+##onic
+Barker
+foul
+commercials
+Dresden
+Words
+collision
+Carlton
+Fashion
+doubted
+##ril
+precision
+MIT
+Jacobs
+mob
+Monk
+retaining
+gotta
+##rod
+remake
+Fast
+chips
+##pled
+sufficiently
+##lights
+delivering
+##enburg
+Dancing
+Barton
+Officers
+metals
+##lake
+religions
+##ré
+motivated
+differs
+dorsal
+##birds
+##rts
+Priest
+polished
+##aling
+Saxony
+Wyatt
+knockout
+##hor
+Lopez
+RNA
+##link
+metallic
+##kas
+daylight
+Montenegro
+##lining
+wrapping
+resemble
+Jam
+Viking
+uncertainty
+angels
+enables
+##fy
+Stuttgart
+tricks
+tattoo
+127
+wicked
+asset
+breach
+##yman
+MW
+breaths
+Jung
+im
+1798
+noon
+vowel
+##qua
+calmly
+seasonal
+chat
+ingredients
+cooled
+Randolph
+ensuring
+##ib
+##idal
+flashing
+1808
+Macedonian
+Cool
+councils
+##lick
+advantages
+Immediately
+Madras
+##cked
+Pain
+fancy
+chronic
+Malayalam
+begged
+##nese
+Inner
+feathers
+##vey
+Names
+dedication
+Sing
+pan
+Fischer
+nurses
+Sharp
+inning
+stamps
+Meg
+##ello
+edged
+motioned
+Jacksonville
+##ffle
+##dic
+##US
+divide
+garnered
+Ranking
+chasing
+modifications
+##oc
+clever
+midst
+flushed
+##DP
+void
+##sby
+ambulance
+beaches
+groan
+isolation
+strengthen
+prevention
+##ffs
+Scouts
+reformed
+geographic
+squadrons
+Fiona
+Kai
+Consequently
+##uss
+overtime
+##yas
+Fr
+##BL
+Papua
+Mixed
+glances
+Haiti
+Sporting
+sandy
+confronted
+René
+Tanner
+1811
+##IM
+advisory
+trim
+##ibe
+González
+gambling
+Jupiter
+##ility
+##owski
+##nar
+122
+apology
+teased
+Pool
+feminine
+wicket
+eagle
+shiny
+##lator
+blend
+peaking
+nasty
+nodding
+fraction
+tech
+Noble
+Kuwait
+brushing
+Italia
+Canberra
+duet
+Johan
+1805
+Written
+cameo
+Stalin
+pig
+cord
+##zio
+Surely
+SA
+owing
+holidays
+123
+Ranger
+lighthouse
+##ige
+miners
+1804
+##ë
+##gren
+##ried
+crashing
+##atory
+wartime
+highlight
+inclined
+Torres
+Tax
+##zel
+##oud
+Own
+##corn
+Divine
+EMI
+Relief
+Northwestern
+ethics
+BMW
+click
+plasma
+Christie
+coordinator
+Shepherd
+washing
+cooked
+##dio
+##eat
+Cerambycidae
+algebra
+Engine
+costumes
+Vampire
+vault
+submission
+virtue
+assumption
+##rell
+Toledo
+##oting
+##rva
+crept
+emphasized
+##lton
+##ood
+Greeks
+surgical
+crest
+Patrol
+Beta
+Tessa
+##GS
+pizza
+traits
+rats
+Iris
+spray
+##GC
+Lightning
+binary
+escapes
+##take
+Clary
+crowds
+##zong
+hauled
+maid
+##fen
+Manning
+##yang
+Nielsen
+aesthetic
+sympathetic
+affiliation
+soaked
+Mozart
+personalities
+begging
+##iga
+clip
+Raphael
+yearly
+Lima
+abundant
+##lm
+1794
+strips
+Initiative
+reporters
+##vsky
+consolidated
+##itated
+Civic
+rankings
+mandate
+symbolic
+##ively
+1807
+rental
+duck
+nave
+complications
+##nor
+Irene
+Nazis
+haunted
+scholarly
+Pratt
+Gran
+Embassy
+Wave
+pity
+genius
+bats
+canton
+Tropical
+marker
+##cos
+escorted
+Climate
+##posed
+appreciation
+freezing
+puzzle
+Internal
+pools
+Shawn
+pathway
+Daniels
+Fitzgerald
+extant
+olive
+Vanessa
+marriages
+cocked
+##dging
+prone
+chemicals
+doll
+drawer
+##HF
+Stark
+Property
+##tai
+flowed
+Sheridan
+##uated
+Less
+Omar
+remarks
+catalogue
+Seymour
+wreck
+Carrie
+##bby
+Mercer
+displaced
+sovereignty
+rip
+Flynn
+Archie
+Quarterfinals
+Hassan
+##ards
+vein
+Osaka
+pouring
+wages
+Romance
+##cript
+##phere
+550
+##eil
+##stown
+Documentary
+ancestor
+CNN
+Panthers
+publishers
+Rise
+##mu
+biting
+Bright
+String
+succeeding
+119
+loaned
+Warwick
+Sheikh
+Von
+Afterwards
+Jax
+Camden
+helicopters
+Hence
+Laurel
+##ddy
+transaction
+Corp
+clause
+##owing
+##kel
+Investment
+cups
+Lucia
+Moss
+Giles
+chef
+López
+decisive
+30th
+distress
+linguistic
+surveys
+Ready
+maiden
+Touch
+frontier
+incorporate
+exotic
+mollusk
+Leopold
+Ride
+##wain
+##ndo
+teammates
+tones
+drift
+ordering
+Feb
+Penny
+Normandy
+Present
+Flag
+pipes
+##rro
+delight
+motto
+Tibet
+leap
+Eliza
+Produced
+teenagers
+sitcom
+Try
+Hansen
+Cody
+wandered
+terrestrial
+frog
+scare
+resisted
+employers
+coined
+##DS
+resistant
+Fly
+captive
+dissolution
+judged
+associates
+defining
+##court
+Hale
+##mbo
+raises
+clusters
+twelfth
+##metric
+Roads
+##itude
+satisfy
+Android
+Reds
+Gloucester
+Category
+Valencia
+Daemon
+stabbed
+Luna
+Churches
+Canton
+##eller
+Attack
+Kashmir
+annexed
+grabs
+asteroid
+Hartford
+recommendation
+Rodriguez
+handing
+stressed
+frequencies
+delegate
+Bones
+Erie
+Weber
+Hands
+Acts
+millimetres
+24th
+Fat
+Howe
+casually
+##SL
+convent
+1790
+IF
+##sity
+1795
+yelling
+##ises
+drain
+addressing
+amino
+Marcel
+Sylvia
+Paramount
+Gerard
+Volleyball
+butter
+124
+Albion
+##GB
+triggered
+1792
+folding
+accepts
+##ße
+preparations
+Wimbledon
+dose
+##grass
+escaping
+##tling
+import
+charging
+##dation
+280
+Nolan
+##fried
+Calcutta
+##pool
+Cove
+examining
+minded
+heartbeat
+twisting
+domains
+bush
+Tunisia
+Purple
+Leone
+##code
+evacuated
+battlefield
+tiger
+Electrical
+##ared
+chased
+##cre
+cultivated
+Jet
+solved
+shrug
+ringing
+Impact
+##iant
+kilometre
+##log
+commemorate
+migrated
+singular
+designing
+promptly
+Higgins
+##own
+##aves
+freshwater
+Marketing
+Payne
+beg
+locker
+pray
+implied
+AAA
+corrected
+Trans
+Europeans
+Ashe
+acknowledge
+Introduction
+##writer
+##llen
+Munster
+auxiliary
+growl
+Hours
+Poems
+##AT
+reduces
+Plain
+plague
+canceled
+detention
+polite
+necklace
+Gustav
+##gu
+##lance
+En
+Angola
+##bb
+dwelling
+##hea
+5000
+Qing
+Dodgers
+rim
+##ored
+##haus
+spilled
+Elisabeth
+Viktor
+backpack
+1802
+amended
+##worthy
+Phantom
+##ctive
+keeper
+##loom
+Vikings
+##gua
+employs
+Tehran
+specialty
+##bate
+Marx
+Mirror
+Jenna
+rides
+needle
+prayers
+clarinet
+forewings
+##walk
+Midlands
+convincing
+advocacy
+Cao
+Birds
+cycles
+Clement
+Gil
+bubble
+Maximum
+humanitarian
+Tan
+cries
+##SI
+Parsons
+Trio
+offshore
+Innovation
+clutched
+260
+##mund
+##duct
+Prairie
+relied
+Falcon
+##ste
+Kolkata
+Gill
+Swift
+Negro
+Zoo
+valleys
+##OL
+Opening
+beams
+MPs
+outline
+Bermuda
+Personal
+exceed
+productive
+##MT
+republic
+forum
+##sty
+tornado
+Known
+dipped
+Edith
+folks
+mathematician
+watershed
+Ricardo
+synthetic
+##dication
+deity
+##₄
+gaming
+subjected
+suspects
+Foot
+swollen
+Motors
+##tty
+##ý
+aloud
+ceremonial
+es
+nuts
+intend
+Carlisle
+tasked
+hesitation
+sponsors
+unified
+inmates
+##ctions
+##stan
+tiles
+jokes
+whereby
+outcomes
+Lights
+scary
+Stoke
+Portrait
+Blind
+sergeant
+violations
+cultivation
+fuselage
+Mister
+Alfonso
+candy
+sticks
+teen
+agony
+Enough
+invite
+Perkins
+Appeal
+mapping
+undergo
+Glacier
+Melanie
+affects
+incomplete
+##dd
+Colombian
+##nate
+CBC
+purchasing
+bypass
+Drug
+Electronics
+Frontier
+Coventry
+##aan
+autonomy
+scrambled
+Recent
+bounced
+cow
+experiencing
+Rouge
+cuisine
+Elite
+disability
+Ji
+inheritance
+wildly
+Into
+##wig
+confrontation
+Wheeler
+shiver
+Performing
+aligned
+consequently
+Alexis
+Sin
+woodland
+executives
+Stevenson
+Ferrari
+inevitable
+##cist
+##dha
+##base
+Corner
+comeback
+León
+##eck
+##urus
+MacDonald
+pioneering
+breakdown
+landscapes
+Veterans
+Rican
+Theological
+stirred
+participant
+Credit
+Hyderabad
+snails
+Claudia
+##ocene
+compliance
+##MI
+Flags
+Middlesex
+storms
+winding
+asserted
+er
+##ault
+##kal
+waking
+##rates
+abbey
+Augusta
+tooth
+trustees
+Commodore
+##uded
+Cunningham
+NC
+Witch
+marching
+Sword
+Same
+spiral
+Harley
+##ahan
+Zack
+Audio
+1890s
+##fit
+Simmons
+Kara
+Veronica
+negotiated
+Speaking
+FIBA
+Conservatory
+formations
+constituencies
+explicit
+facial
+eleventh
+##ilt
+villain
+##dog
+##case
+##hol
+armored
+tin
+hairs
+##umi
+##rai
+mattress
+Angus
+cease
+verbal
+Recreation
+savings
+Aurora
+peers
+Monastery
+Airways
+drowned
+additions
+downstream
+sticking
+Shi
+mice
+skiing
+##CD
+Raw
+Riverside
+warming
+hooked
+boost
+memorable
+posed
+treatments
+320
+##dai
+celebrating
+blink
+helpless
+circa
+Flowers
+PM
+uncommon
+Oct
+Hawks
+overwhelmed
+Sparhawk
+repaired
+Mercy
+pose
+counterpart
+compare
+survives
+##½
+##eum
+coordinate
+Lil
+grandchildren
+notorious
+Yi
+Judaism
+Juliet
+accusations
+1789
+floated
+marathon
+roar
+fortified
+reunion
+145
+Nov
+Paula
+##fare
+##toria
+tearing
+Cedar
+disappearance
+Si
+gifted
+scar
+270
+PBS
+Technologies
+Marvin
+650
+roller
+cupped
+negotiate
+##erman
+passport
+tram
+miracle
+styled
+##tier
+necessity
+Des
+rehabilitation
+Lara
+USD
+psychic
+wipe
+##lem
+mistaken
+##lov
+charming
+Rider
+pageant
+dynamics
+Cassidy
+##icus
+defenses
+##tadt
+##vant
+aging
+##inal
+declare
+mistress
+supervised
+##alis
+##rest
+Ashton
+submerged
+sack
+Dodge
+grocery
+ramp
+Teacher
+lineage
+imagery
+arrange
+inscriptions
+Organisation
+Siege
+combines
+pounded
+Fleming
+legends
+columnist
+Apostolic
+prose
+insight
+Arabian
+expired
+##uses
+##nos
+Alone
+elbows
+##asis
+##adi
+##combe
+Step
+Waterloo
+Alternate
+interval
+Sonny
+plains
+Goals
+incorporating
+recruit
+adjoining
+Cheshire
+excluding
+marrying
+ducked
+Cherokee
+par
+##inate
+hiking
+Coal
+##bow
+natives
+ribbon
+Allies
+con
+descriptions
+positively
+##lal
+defendant
+22nd
+Vivian
+##beat
+Weather
+possessions
+Date
+sweetheart
+inability
+Salisbury
+adviser
+ideology
+Nordic
+##eu
+Cubs
+IP
+Administrative
+##nick
+facto
+liberation
+Burnett
+Javier
+fashioned
+Electoral
+Turin
+theft
+unanimous
+Per
+1799
+Clan
+Hawkins
+Teachers
+##wes
+Cameroon
+Parkway
+##gment
+demolition
+atoms
+nucleus
+##thi
+recovering
+##yte
+##vice
+lifts
+Must
+deposit
+Hancock
+Semi
+darkened
+Declaration
+moan
+muscular
+Myers
+attractions
+sauce
+simulation
+##weed
+Alps
+barriers
+##baum
+Barack
+galleries
+Min
+holders
+Greenwich
+donation
+Everybody
+Wolfgang
+sandwich
+Kendra
+Collegiate
+casino
+Slavic
+ensuing
+Porto
+##grapher
+Jesuit
+suppressed
+tires
+Ibrahim
+protesters
+Ibn
+Amos
+1796
+phenomena
+Hayden
+Paraguay
+Squad
+Reilly
+complement
+aluminum
+##eers
+doubts
+decay
+demise
+Practice
+patience
+fireplace
+transparent
+monarchy
+##person
+Rodney
+mattered
+rotating
+Clifford
+disposal
+Standards
+paced
+##llie
+arise
+tallest
+tug
+documentation
+node
+freeway
+Nikolai
+##cite
+clicked
+imaging
+Lorraine
+Tactical
+Different
+Regular
+Holding
+165
+Pilot
+guarded
+##polis
+Classics
+Mongolia
+Brock
+monarch
+cellular
+receptors
+Mini
+Chandler
+financed
+financially
+Lives
+erection
+Fuller
+unnamed
+Kannada
+cc
+passive
+plateau
+##arity
+freak
+##rde
+retrieved
+transactions
+##sus
+23rd
+swimmer
+beef
+fulfill
+Arlington
+offspring
+reasoning
+Rhys
+saves
+pseudonym
+centimetres
+shivered
+shuddered
+##ME
+Feel
+##otic
+professors
+Blackburn
+##eng
+##life
+##haw
+interred
+lodge
+fragile
+Della
+guardian
+##bbled
+catalog
+clad
+observer
+tract
+declaring
+##headed
+Lok
+dean
+Isabelle
+1776
+irrigation
+spectacular
+shuttle
+mastering
+##aro
+Nathaniel
+Retired
+##lves
+Brennan
+##kha
+dick
+##dated
+##hler
+Rookie
+leapt
+televised
+weekends
+Baghdad
+Yemen
+##fo
+factions
+ion
+Lab
+mortality
+passionate
+Hammer
+encompasses
+confluence
+demonstrations
+Ki
+derivative
+soils
+##unch
+Ranch
+Universities
+conventions
+outright
+aiming
+hierarchy
+reside
+illusion
+graves
+rituals
+126
+Antwerp
+Dover
+##ema
+campuses
+Hobart
+lifelong
+aliens
+##vity
+Memory
+coordination
+alphabet
+##mina
+Titans
+pushes
+Flanders
+##holder
+Normal
+excellence
+capped
+profound
+Taipei
+portrayal
+sparked
+scratch
+se
+##eas
+##hir
+Mackenzie
+##cation
+Neo
+Shin
+##lined
+magnificent
+poster
+batsman
+##rgent
+persuade
+##ement
+Icelandic
+miserable
+collegiate
+Feature
+geography
+##mura
+Comic
+Circus
+processor
+barracks
+Tale
+##11
+Bulls
+##rap
+strengthened
+##bell
+injection
+miniature
+broadly
+Letter
+fare
+hostage
+traders
+##nium
+##mere
+Fortune
+Rivera
+Lu
+triumph
+Browns
+Bangalore
+cooperative
+Basel
+announcing
+Sawyer
+##him
+##cco
+##kara
+darted
+##AD
+##nova
+sucking
+##position
+perimeter
+flung
+Holdings
+##NP
+Basque
+sketches
+Augustine
+Silk
+Elijah
+analyst
+armour
+riots
+acquiring
+ghosts
+##ems
+132
+Pioneer
+Colleges
+Simone
+Economy
+Author
+semester
+Soldier
+il
+##unting
+##bid
+freaking
+Vista
+tumor
+##bat
+murderer
+##eda
+unreleased
+##grove
+##sser
+##té
+edit
+statute
+sovereign
+##gawa
+Killer
+stares
+Fury
+comply
+##lord
+##nant
+barrels
+Andhra
+Maple
+generator
+mascot
+unusually
+eds
+##ante
+##runner
+rod
+##tles
+Historically
+Jennings
+dumped
+Established
+resemblance
+##lium
+##cise
+##body
+##voke
+Lydia
+##hou
+##iring
+nonetheless
+1797
+corrupt
+patrons
+physicist
+sneak
+Livingston
+Citizens
+Architects
+Werner
+trends
+Melody
+eighty
+markings
+brakes
+##titled
+oversaw
+processed
+mock
+Midwest
+intervals
+##EF
+stretches
+werewolf
+##MG
+Pack
+controller
+##dition
+Honours
+cane
+Griffith
+vague
+repertoire
+Courtney
+orgasm
+Abdullah
+dominance
+occupies
+Ya
+introduces
+Lester
+instinct
+collaborative
+Indigenous
+refusal
+##rank
+outlet
+debts
+spear
+155
+##keeping
+##ulu
+Catalan
+##osh
+tensions
+##OT
+bred
+crude
+Dunn
+abdomen
+accurately
+##fu
+##lough
+accidents
+Row
+Audrey
+rude
+Getting
+promotes
+replies
+Paolo
+merge
+##nock
+trans
+Evangelical
+automated
+Canon
+##wear
+##ggy
+##gma
+Broncos
+foolish
+icy
+Voices
+knives
+Aside
+dreamed
+generals
+molecule
+AG
+rejection
+insufficient
+##nagar
+deposited
+sacked
+Landing
+arches
+helpful
+devotion
+intake
+Flower
+PGA
+dragons
+evolutionary
+##mail
+330
+GM
+tissues
+##tree
+arcade
+composite
+lid
+Across
+implications
+lacks
+theological
+assessed
+concentrations
+Den
+##mans
+##ulous
+Fu
+homeland
+##stream
+Harriet
+ecclesiastical
+troop
+ecological
+winked
+##xed
+eighteenth
+Casino
+specializing
+##sworth
+unlocked
+supreme
+devastated
+snatched
+trauma
+GDP
+Nord
+saddle
+Wes
+convenient
+competes
+##nu
+##iss
+Marian
+subway
+##rri
+successes
+umbrella
+##far
+##ually
+Dundee
+##cence
+spark
+##rix
+##я
+Quality
+Geological
+cockpit
+rpm
+Cam
+Bucharest
+riot
+##PM
+Leah
+##dad
+##pose
+Ka
+m³
+Bundesliga
+Wolfe
+grim
+textile
+quartet
+expressing
+fantastic
+destroyers
+eternal
+picnic
+##oro
+contractor
+1775
+spanning
+declining
+##cating
+Lowe
+Sutherland
+Emirates
+downward
+nineteen
+violently
+scout
+viral
+melting
+enterprises
+##cer
+Crosby
+Jubilee
+antenna
+urgent
+Rory
+##uin
+##sure
+wandering
+##gler
+##vent
+Suzuki
+Lifetime
+Dirty
+occupying
+##quent
+Disc
+Guru
+mound
+Lennon
+Humanities
+listeners
+Walton
+uh
+Braves
+Bologna
+##bis
+##gra
+Dwight
+crawl
+flags
+memoir
+Thorne
+Archdiocese
+dairy
+##uz
+##tery
+roared
+adjust
+patches
+inn
+Knowing
+##bbed
+##zan
+scan
+Papa
+precipitation
+angrily
+passages
+postal
+Phi
+embraced
+blacks
+economist
+triangular
+Sen
+shooter
+punished
+Millennium
+Swimming
+confessed
+Aston
+defeats
+Era
+cousins
+Williamson
+##rer
+daytime
+dumb
+##rek
+underway
+specification
+Buchanan
+prayed
+concealed
+activation
+##issa
+canon
+awesome
+Starr
+plural
+summers
+##fields
+Slam
+unnecessary
+1791
+resume
+trilogy
+compression
+##rough
+selective
+dignity
+Yan
+##xton
+immense
+##yun
+lone
+seeded
+hiatus
+lightweight
+summary
+Yo
+approve
+Galway
+rejoined
+Elise
+garbage
+burns
+speeches
+129
+Honduras
+##liness
+inventory
+jersey
+FK
+assure
+slumped
+Lionel
+Suite
+##sbury
+Lena
+continuation
+##AN
+brightly
+##nti
+GT
+Knowledge
+##park
+##lius
+lethal
+##tribution
+##sions
+Certificate
+Mara
+##lby
+algorithms
+Jade
+blows
+pirates
+fleeing
+wheelchair
+Stein
+sophomore
+Alt
+Territorial
+diploma
+snakes
+##olic
+##tham
+Tiffany
+Pius
+flush
+urging
+Hanover
+Reich
+##olate
+Unity
+Pike
+collectively
+Theme
+ballad
+kindergarten
+rocked
+zoo
+##page
+whip
+Rodríguez
+strokes
+checks
+Becky
+Stern
+upstream
+##uta
+Silent
+volunteered
+Sigma
+##ingen
+##tract
+##ede
+Gujarat
+screwed
+entertaining
+##action
+##ryn
+defenders
+innocence
+lesbian
+que
+Richie
+nodes
+Lie
+juvenile
+Jakarta
+safer
+confront
+Bert
+breakthrough
+gospel
+Cable
+##zie
+institutional
+Archive
+brake
+liquor
+feeds
+##iate
+chancellor
+Encyclopedia
+Animation
+scanning
+teens
+##mother
+Core
+Rear
+Wine
+##flower
+reactor
+Ave
+cardinal
+sodium
+strands
+Olivier
+crouched
+Vaughan
+Sammy
+Image
+scars
+Emmanuel
+flour
+bias
+nipple
+revelation
+##ucci
+Denny
+##ssy
+Form
+Runners
+admits
+Rama
+violated
+Burmese
+feud
+underwear
+Mohamed
+Named
+swift
+statewide
+Door
+Recently
+comparing
+Hundred
+##idge
+##nity
+##rds
+Rally
+Reginald
+Auburn
+solving
+waitress
+Treasurer
+##ilization
+Halloween
+Ministers
+Boss
+Shut
+##listic
+Rahman
+demonstrating
+##pies
+Gaza
+Yuri
+installations
+Math
+schooling
+##bble
+Bronx
+exiled
+gasoline
+133
+bundle
+humid
+FCC
+proportional
+relate
+VFL
+##dez
+continuity
+##cene
+syndicated
+atmospheric
+arrows
+Wanderers
+reinforcements
+Willow
+Lexington
+Rotten
+##yon
+discovering
+Serena
+portable
+##lysis
+targeting
+£1
+Goodman
+Steam
+sensors
+detachment
+Malik
+##erie
+attitudes
+Goes
+Kendall
+Read
+Sleep
+beans
+Nikki
+modification
+Jeanne
+knuckles
+Eleven
+##iously
+Gross
+Jaime
+dioxide
+moisture
+Stones
+UCI
+displacement
+Metacritic
+Jury
+lace
+rendering
+elephant
+Sergei
+##quire
+GP
+Abbott
+##type
+projection
+Mouse
+Bishops
+whispering
+Kathleen
+Rams
+##jar
+whites
+##oran
+assess
+dispatched
+##hire
+kin
+##mir
+Nursing
+advocates
+tremendous
+sweater
+assisting
+##bil
+Farmer
+prominently
+reddish
+Hague
+cyclone
+##SD
+Sage
+Lawson
+Sanctuary
+discharged
+retains
+##ube
+shotgun
+wilderness
+Reformed
+similarity
+Entry
+Watts
+Bahá
+Quest
+Looks
+visions
+Reservoir
+Arabs
+curls
+Blu
+dripping
+accomplish
+Verlag
+drill
+sensor
+Dillon
+physicians
+smashed
+##dir
+painters
+Renault
+straw
+fading
+Directorate
+lounge
+commissions
+Brain
+##graph
+neo
+##urg
+plug
+coordinated
+##houses
+Critical
+lamps
+illustrator
+Returning
+erosion
+Crow
+##ciation
+blessing
+Thought
+Wife
+medalist
+synthesizer
+Pam
+Thornton
+Esther
+HBO
+fond
+Associates
+##raz
+pirate
+permits
+Wide
+tire
+##PC
+Ernie
+Nassau
+transferring
+RFC
+##ntly
+um
+spit
+AS
+##mps
+Mining
+polar
+villa
+anchored
+##zzi
+embarrassment
+relates
+##ă
+Rupert
+counterparts
+131
+Baxter
+##18
+Igor
+recognizes
+Clive
+##hane
+##eries
+##ibly
+occurrence
+##scope
+fin
+colorful
+Rapids
+banker
+tile
+##rative
+##dus
+delays
+destinations
+##llis
+Pond
+Dane
+grandparents
+rewarded
+socially
+motorway
+##hof
+##lying
+##human
+modeled
+Dayton
+Forward
+conscience
+Sharma
+whistle
+Mayer
+Sasha
+##pical
+circuits
+Zhou
+##ça
+Latvian
+finalists
+predators
+Lafayette
+closes
+obligations
+Resolution
+##vier
+Trustees
+reminiscent
+##hos
+Highlands
+Protected
+asylum
+evacuation
+##acy
+Chevrolet
+confession
+Somalia
+emergence
+separating
+##rica
+alright
+calcium
+Laurent
+Welfare
+Leonardo
+ashes
+dental
+Deal
+minerals
+##lump
+##mount
+accounted
+staggered
+slogan
+photographic
+builder
+##imes
+##raft
+tragic
+144
+SEC
+Hit
+tailed
+##ples
+##rring
+##rson
+ethical
+wrestlers
+concludes
+lunar
+##ept
+nitrogen
+Aid
+cyclist
+quarterfinals
+##ه
+harvest
+##hem
+Pasha
+IL
+##mis
+continually
+##forth
+Intel
+bucket
+##ended
+witches
+pretended
+dresses
+viewer
+peculiar
+lowering
+volcano
+Marilyn
+Qualifier
+clung
+##sher
+Cut
+modules
+Bowie
+##lded
+onset
+transcription
+residences
+##pie
+##itor
+scrapped
+##bic
+Monaco
+Mayo
+eternity
+Strike
+uncovered
+skeleton
+##wicz
+Isles
+bug
+Promoted
+##rush
+Mechanical
+XII
+##ivo
+gripping
+stubborn
+velvet
+TD
+decommissioned
+operas
+spatial
+unstable
+Congressman
+wasted
+##aga
+##ume
+advertisements
+##nya
+obliged
+Cannes
+Conway
+bricks
+##gnant
+##mity
+##uise
+jumps
+Clear
+##cine
+##sche
+chord
+utter
+Su
+podium
+spokesman
+Royce
+assassin
+confirmation
+licensing
+liberty
+##rata
+Geographic
+individually
+detained
+##ffe
+Saturn
+crushing
+airplane
+bushes
+knights
+##PD
+Lilly
+hurts
+unexpectedly
+Conservatives
+pumping
+Forty
+candle
+Pérez
+peasants
+supplement
+Sundays
+##ggs
+##rries
+risen
+enthusiastic
+corresponds
+pending
+##IF
+Owens
+floods
+Painter
+inflation
+presumed
+inscribed
+Chamberlain
+bizarre
+1200
+liability
+reacted
+tub
+Legacy
+##eds
+##pted
+shone
+##litz
+##NC
+Tiny
+genome
+bays
+Eduardo
+robbery
+stall
+hatch
+Depot
+Variety
+Flora
+reprinted
+trembled
+outlined
+CR
+Theresa
+spans
+##plication
+Jensen
+##eering
+posting
+##rky
+pays
+##ost
+Marcos
+fortifications
+inferior
+##ential
+Devi
+despair
+Talbot
+##chus
+updates
+ego
+Booth
+Darius
+tops
+##lau
+Scene
+##DC
+Harlem
+Trey
+Generally
+candles
+##α
+Neville
+Admiralty
+##hong
+iconic
+victorious
+1600
+Rowan
+abundance
+miniseries
+clutching
+sanctioned
+##words
+obscure
+##ision
+##rle
+##EM
+disappearing
+Resort
+Obviously
+##eb
+exceeded
+1870s
+Adults
+##cts
+Cry
+Kerr
+ragged
+selfish
+##lson
+circled
+pillars
+galaxy
+##asco
+##mental
+rebuild
+caution
+Resistance
+Start
+bind
+splitting
+Baba
+Hogan
+ps
+partnerships
+slam
+Peggy
+courthouse
+##OD
+organizational
+packages
+Angie
+##nds
+possesses
+##rp
+Expressway
+Gould
+Terror
+Him
+Geoff
+nobles
+##ope
+shark
+##nh
+identifies
+##oor
+testified
+Playing
+##ump
+##isa
+stool
+Idol
+##pice
+##tana
+Byrne
+Gerry
+grunted
+26th
+observing
+habits
+privilege
+immortal
+wagons
+##thy
+dot
+Bring
+##lian
+##witz
+newest
+##uga
+constraints
+Screen
+Issue
+##RNA
+##vil
+reminder
+##gles
+addiction
+piercing
+stunning
+var
+##rita
+Signal
+accumulated
+##wide
+float
+devastating
+viable
+cartoons
+Uttar
+flared
+##encies
+Theology
+patents
+##bahn
+privileges
+##ava
+##CO
+137
+##oped
+##NT
+orchestral
+medication
+225
+erect
+Nadia
+École
+fried
+Sales
+scripts
+##rease
+airs
+Cage
+inadequate
+structured
+countless
+Avengers
+Kathy
+disguise
+mirrors
+Investigation
+reservation
+##nson
+Legends
+humorous
+Mona
+decorations
+attachment
+Via
+motivation
+Browne
+strangers
+##ński
+Shadows
+Twins
+##pressed
+Alma
+Nominated
+##ott
+Sergio
+canopy
+152
+Semifinals
+devised
+##irk
+upwards
+Traffic
+Goddess
+Move
+beetles
+138
+spat
+##anne
+holdings
+##SP
+tangled
+Whilst
+Fowler
+anthem
+##ING
+##ogy
+snarled
+moonlight
+songwriting
+tolerance
+Worlds
+exams
+##pia
+notices
+sensitivity
+poetic
+Stephens
+Boone
+insect
+reconstructed
+Fresh
+27th
+balloon
+##ables
+Brendan
+mug
+##gee
+1780
+apex
+exports
+slides
+Lahore
+hiring
+Shell
+electorate
+sexuality
+poker
+nonprofit
+##imate
+cone
+##uce
+Okinawa
+superintendent
+##HC
+referenced
+turret
+Sprint
+Citizen
+equilibrium
+Stafford
+curb
+Driver
+Valerie
+##rona
+aching
+impacts
+##bol
+observers
+Downs
+Shri
+##uth
+airports
+##uda
+assignments
+curtains
+solitary
+icon
+patrols
+substances
+Jasper
+mountainous
+Published
+ached
+##ingly
+announce
+dove
+damaging
+##tism
+Primera
+Dexter
+limiting
+batch
+##uli
+undergoing
+refugee
+Ye
+admiral
+pavement
+##WR
+##reed
+pipeline
+desires
+Ramsey
+Sheila
+thickness
+Brotherhood
+Tea
+instituted
+Belt
+Break
+plots
+##ais
+masculine
+##where
+Theo
+##aged
+##mined
+Experience
+scratched
+Ethiopian
+Teaching
+##nov
+Aiden
+Abe
+Samoa
+conditioning
+##mous
+Otherwise
+fade
+Jenks
+##encing
+Nat
+##lain
+Anyone
+##kis
+smirk
+Riding
+##nny
+Bavarian
+blessed
+potatoes
+Hook
+##wise
+likewise
+hardened
+Merry
+amid
+persecution
+##sten
+Elections
+Hoffman
+Pitt
+##vering
+distraction
+exploitation
+infamous
+quote
+averaging
+healed
+Rhythm
+Germanic
+Mormon
+illuminated
+guides
+##ische
+interfere
+##ilized
+rector
+perennial
+##ival
+Everett
+courtesy
+##nham
+Kirby
+Mk
+##vic
+Medieval
+##tale
+Luigi
+limp
+##diction
+Alive
+greeting
+shove
+##force
+##fly
+Jasmine
+Bend
+Capt
+Suzanne
+ditch
+134
+##nning
+Host
+fathers
+rebuilding
+Vocal
+wires
+##manship
+tan
+Factor
+fixture
+##LS
+Māori
+Plate
+pyramid
+##umble
+slap
+Schneider
+yell
+##ulture
+##tional
+Goodbye
+sore
+##pher
+depressed
+##dox
+pitching
+Find
+Lotus
+##wang
+strand
+Teen
+debates
+prevalent
+##bilities
+exposing
+hears
+billed
+##rse
+reorganized
+compelled
+disturbing
+displaying
+##tock
+Clinical
+emotionally
+##iah
+Derbyshire
+grouped
+##quel
+Bahrain
+Journalism
+IN
+persistent
+blankets
+Crane
+camping
+Direct
+proving
+Lola
+##dding
+Corporate
+birthplace
+##boats
+##ender
+Figure
+dared
+Assam
+precursor
+##nched
+Tribe
+Restoration
+slate
+Meyrick
+hunted
+stroking
+Earlier
+Kind
+polls
+appeals
+monetary
+##reate
+Kira
+Langdon
+explores
+GPS
+extensions
+squares
+Results
+draped
+announcer
+merit
+##ennial
+##tral
+##roved
+##cion
+robots
+supervisor
+snorted
+##group
+Cannon
+procession
+monkey
+freeze
+sleeves
+Nile
+verdict
+ropes
+firearms
+extraction
+tensed
+EC
+Saunders
+##tches
+diamonds
+Marriage
+##amble
+curling
+Amazing
+##haling
+unrelated
+##roads
+Daughter
+cum
+discarded
+kidney
+cliffs
+forested
+Candy
+##lap
+authentic
+tablet
+notation
+##nburg
+Bulldogs
+Callum
+Meet
+mouths
+coated
+##xe
+Truman
+combinations
+##mation
+Steelers
+Fan
+Than
+paternal
+##father
+##uti
+Rebellion
+inviting
+Fun
+theatres
+##ي
+##rom
+curator
+##cision
+networking
+Oz
+drought
+##ssel
+granting
+MBA
+Shelby
+Elaine
+jealousy
+Kyoto
+shores
+signaling
+tenants
+debated
+Intermediate
+Wise
+##hes
+##pu
+Havana
+duke
+vicious
+exited
+servers
+Nonetheless
+Reports
+explode
+##beth
+Nationals
+offerings
+Oval
+conferred
+eponymous
+folklore
+##NR
+Shire
+planting
+1783
+Zeus
+accelerated
+Constable
+consuming
+troubles
+McCartney
+texture
+bust
+Immigration
+excavated
+hopefully
+##cession
+##coe
+##name
+##ully
+lining
+Einstein
+Venezuelan
+reissued
+minorities
+Beatrice
+crystals
+##nies
+circus
+lava
+Beirut
+extinction
+##shu
+Becker
+##uke
+issuing
+Zurich
+extract
+##esta
+##rred
+regulate
+progression
+hut
+alcoholic
+plea
+AB
+Norse
+Hubert
+Mansfield
+ashamed
+##put
+Bombardment
+stripes
+electrons
+Denise
+horrified
+Nor
+arranger
+Hay
+Koch
+##ddling
+##iner
+Birthday
+Josie
+deliberate
+explorer
+##jiang
+##signed
+Arrow
+wiping
+satellites
+baritone
+mobility
+##rals
+Dorset
+turbine
+Coffee
+185
+##lder
+Cara
+Colts
+pits
+Crossing
+coral
+##birth
+Tai
+zombie
+smoothly
+##hp
+mates
+##ady
+Marguerite
+##tary
+puzzled
+tapes
+overly
+Sonic
+Prayer
+Thinking
+##uf
+IEEE
+obligation
+##cliffe
+Basil
+redesignated
+##mmy
+nostrils
+Barney
+XIII
+##phones
+vacated
+unused
+Berg
+##roid
+Towards
+viola
+136
+Event
+subdivided
+rabbit
+recruiting
+##nery
+Namibia
+##16
+##ilation
+recruits
+Famous
+Francesca
+##hari
+Goa
+##lat
+Karachi
+haul
+biblical
+##cible
+MGM
+##rta
+horsepower
+profitable
+Grandma
+importantly
+Martinez
+incoming
+##kill
+beneficial
+nominal
+praying
+##isch
+gable
+nail
+noises
+##ttle
+Polytechnic
+rub
+##cope
+Thor
+audition
+erotic
+##ending
+##iano
+Ultimately
+armoured
+##mum
+presently
+pedestrian
+##tled
+Ipswich
+offence
+##ffin
+##borne
+Flemish
+##hman
+echo
+##cting
+auditorium
+gentlemen
+winged
+##tched
+Nicaragua
+Unknown
+prosperity
+exhaust
+pie
+Peruvian
+compartment
+heights
+disabilities
+##pole
+Harding
+Humphrey
+postponed
+moths
+Mathematical
+Mets
+posters
+axe
+##nett
+Nights
+Typically
+chuckle
+councillors
+alternating
+141
+Norris
+##ately
+##etus
+deficit
+dreaming
+cooler
+oppose
+Beethoven
+##esis
+Marquis
+flashlight
+headache
+investor
+responding
+appointments
+##shore
+Elias
+ideals
+shades
+torch
+lingering
+##real
+pier
+fertile
+Diploma
+currents
+Snake
+##horse
+##15
+Briggs
+##ota
+##hima
+##romatic
+Coastal
+Kuala
+ankles
+Rae
+slice
+Hilton
+locking
+Approximately
+Workshop
+Niagara
+strangely
+##scence
+functionality
+advertisement
+Rapid
+Anders
+ho
+Soviets
+packing
+basal
+Sunderland
+Permanent
+##fting
+rack
+tying
+Lowell
+##ncing
+Wizard
+mighty
+tertiary
+pencil
+dismissal
+torso
+grasped
+##yev
+Sand
+gossip
+##nae
+Beer
+implementing
+##19
+##riya
+Fork
+Bee
+##eria
+Win
+##cid
+sailor
+pressures
+##oping
+speculated
+Freddie
+originating
+##DF
+##SR
+##outh
+28th
+melt
+Brenda
+lump
+Burlington
+USC
+marginal
+##bine
+Dogs
+swamp
+cu
+Ex
+uranium
+metro
+spill
+Pietro
+seize
+Chorus
+partition
+##dock
+##media
+engineered
+##oria
+conclusions
+subdivision
+##uid
+Illustrated
+Leading
+##hora
+Berkshire
+definite
+##books
+##cin
+##suke
+noun
+winced
+Doris
+dissertation
+Wilderness
+##quest
+braced
+arbitrary
+kidnapping
+Kurdish
+##but
+clearance
+excavations
+wanna
+Allmusic
+insult
+presided
+yacht
+##SM
+Honour
+Tin
+attracting
+explosives
+Gore
+Bride
+##ience
+Packers
+Devils
+Observer
+##course
+Loser
+##erry
+##hardt
+##mble
+Cyrillic
+undefeated
+##stra
+subordinate
+##ame
+Wigan
+compulsory
+Pauline
+Cruise
+Opposition
+##ods
+Period
+dispersed
+expose
+##60
+##has
+Certain
+Clerk
+Wolves
+##hibition
+apparatus
+allegiance
+orbital
+justified
+thanked
+##ević
+Biblical
+Carolyn
+Graves
+##tton
+Hercules
+backgrounds
+replica
+1788
+aquatic
+Mega
+Stirling
+obstacles
+filing
+Founder
+vowels
+Deborah
+Rotterdam
+surpassed
+Belarusian
+##ologists
+Zambia
+Ren
+Olga
+Alpine
+bi
+councillor
+Oaks
+Animals
+eliminating
+digit
+Managing
+##GE
+laundry
+##rdo
+presses
+slamming
+Tudor
+thief
+posterior
+##bas
+Rodgers
+smells
+##ining
+Hole
+SUV
+trombone
+numbering
+representations
+Domingo
+Paralympics
+cartridge
+##rash
+Combined
+shelves
+Kraków
+revision
+##frame
+Sánchez
+##tracted
+##bler
+Alain
+townships
+sic
+trousers
+Gibbs
+anterior
+symmetry
+vaguely
+Castile
+IRA
+resembling
+Penguin
+##ulent
+infections
+##stant
+raped
+##pressive
+worrying
+brains
+bending
+JR
+Evidence
+Venetian
+complexes
+Jonah
+850
+exported
+Ambrose
+Gap
+philanthropist
+##atus
+Marxist
+weighing
+##KO
+##nath
+Soldiers
+chiefs
+reject
+repeating
+shaky
+Zürich
+preserving
+##xin
+cigarettes
+##break
+mortar
+##fin
+Already
+reproduction
+socks
+Waiting
+amazed
+##aca
+dash
+##path
+Airborne
+##harf
+##get
+descending
+OBE
+Sant
+Tess
+Lucius
+enjoys
+##ttered
+##ivation
+##ete
+Leinster
+Phillies
+execute
+geological
+unfinished
+Courts
+SP
+Beaver
+Duck
+motions
+Platinum
+friction
+##aud
+##bet
+Parts
+Stade
+entirety
+sprang
+Smithsonian
+coffin
+prolonged
+Borneo
+##vise
+unanimously
+##uchi
+Cars
+Cassandra
+Australians
+##CT
+##rgen
+Louisa
+spur
+Constance
+##lities
+Patent
+racism
+tempo
+##ssion
+##chard
+##nology
+##claim
+Million
+Nichols
+##dah
+Numerous
+ing
+Pure
+plantations
+donor
+##EP
+##rip
+convenience
+##plate
+dots
+indirect
+##written
+Dong
+failures
+adapt
+wizard
+unfortunately
+##gion
+practitioners
+economically
+Enrique
+unchanged
+kingdoms
+refined
+definitions
+lazy
+worries
+railing
+##nay
+Kaiser
+##lug
+cracks
+sells
+ninety
+##WC
+Directed
+denotes
+developmental
+papal
+unfortunate
+disappointing
+sixteenth
+Jen
+##urier
+NWA
+drifting
+Horror
+##chemical
+behaviors
+bury
+surfaced
+foreigners
+slick
+AND
+##rene
+##ditions
+##teral
+scrap
+kicks
+comprise
+buddy
+##anda
+Mental
+##ype
+Dom
+wines
+Limerick
+Luca
+Rand
+##won
+Tomatoes
+homage
+geometric
+##nted
+telescope
+Shelley
+poles
+##fan
+shareholders
+Autonomous
+cope
+intensified
+Genoa
+Reformation
+grazing
+##tern
+Zhao
+provisional
+##bies
+Con
+##riel
+Cynthia
+Raleigh
+vivid
+threaten
+Length
+subscription
+roses
+Müller
+##isms
+robin
+##tial
+Laos
+Stanton
+nationalism
+##clave
+##ND
+##17
+##zz
+staging
+Busch
+Cindy
+relieve
+##spective
+packs
+neglected
+CBE
+alpine
+Evolution
+uneasy
+coastline
+Destiny
+Barber
+Julio
+##tted
+informs
+unprecedented
+Pavilion
+##bei
+##ference
+betrayal
+awaiting
+leaked
+V8
+puppet
+adverse
+Bourne
+Sunset
+collectors
+##glass
+##sque
+copied
+Demon
+conceded
+resembled
+Rafe
+Levy
+prosecutor
+##ject
+flora
+manned
+deaf
+Mosque
+reminds
+Lizzie
+Products
+Funny
+cassette
+congress
+##rong
+Rover
+tossing
+prompting
+chooses
+Satellite
+cautiously
+Reese
+##UT
+Huang
+Gloucestershire
+giggled
+Kitty
+##å
+Pleasant
+Aye
+##ond
+judging
+1860s
+intentionally
+Hurling
+aggression
+##xy
+transfers
+employing
+##fies
+##oda
+Archibald
+Blessed
+Ski
+flavor
+Rosie
+##burgh
+sunset
+Scholarship
+WC
+surround
+ranged
+##jay
+Degree
+Houses
+squeezing
+limb
+premium
+Leningrad
+steals
+##inated
+##ssie
+madness
+vacancy
+hydraulic
+Northampton
+##prise
+Marks
+Boxing
+##fying
+academics
+##lich
+##TY
+CDs
+##lma
+hardcore
+monitors
+paperback
+cables
+Dimitri
+upside
+advent
+Ra
+##clusive
+Aug
+Christchurch
+objected
+stalked
+Simple
+colonists
+##laid
+CT
+discusses
+fellowship
+Carnival
+cares
+Miracle
+pastoral
+rooted
+shortage
+borne
+Quentin
+meditation
+tapping
+Novel
+##ades
+Alicia
+Burn
+famed
+residency
+Fernández
+Johannesburg
+Zhu
+offended
+Mao
+outward
+##inas
+XV
+denial
+noticing
+##ís
+quarry
+##hound
+##amo
+Bernie
+Bentley
+Joanna
+mortgage
+##rdi
+##sumption
+lenses
+extracted
+depiction
+##RE
+Networks
+Broad
+Revenue
+flickered
+virgin
+flanked
+##о
+Enterprises
+probable
+Liberals
+Falcons
+drowning
+phrases
+loads
+assumes
+inhaled
+awe
+logs
+slightest
+spiders
+waterfall
+##pate
+rocking
+shrub
+##uil
+roofs
+##gard
+prehistoric
+wary
+##rak
+TO
+clips
+sustain
+treason
+microphone
+voter
+Lamb
+psychologist
+wrinkled
+##ères
+mating
+Carrier
+340
+##lbert
+sensing
+##rino
+destiny
+distract
+weaker
+UC
+Nearly
+neurons
+spends
+Apache
+##rem
+genuinely
+wells
+##lanted
+stereo
+##girl
+Lois
+Leaving
+consul
+fungi
+Pier
+Cyril
+80s
+Jungle
+##tani
+illustration
+Split
+##hana
+Abigail
+##patrick
+1787
+diminished
+Selected
+packaging
+##EG
+Martínez
+communal
+Manufacturing
+sentiment
+143
+unwilling
+praising
+Citation
+pills
+##iti
+##rax
+muffled
+neatly
+workforce
+Yep
+leisure
+Tu
+##nding
+Wakefield
+ancestral
+##uki
+destructive
+seas
+Passion
+showcase
+##ceptive
+heroic
+142
+exhaustion
+Customs
+##aker
+Scholar
+sliced
+##inian
+Direction
+##OW
+Swansea
+aluminium
+##eep
+ceramic
+McCoy
+Career
+Sector
+chartered
+Damascus
+pictured
+Interest
+stiffened
+Plateau
+obsolete
+##tant
+irritated
+inappropriate
+overs
+##nko
+bail
+Talent
+Sur
+ours
+##nah
+barred
+legged
+sociology
+Bud
+dictionary
+##luk
+Cover
+obey
+##oring
+annoying
+##dong
+apprentice
+Cyrus
+Role
+##GP
+##uns
+##bag
+Greenland
+Porsche
+Rocket
+##32
+organism
+##ntary
+reliability
+##vocation
+##й
+Found
+##hine
+motors
+promoter
+unfair
+##oms
+##note
+distribute
+eminent
+rails
+appealing
+chiefly
+meaningful
+Stephan
+##rehension
+Consumer
+psychiatric
+bowler
+saints
+##iful
+##н
+1777
+Pol
+Dorian
+Townsend
+hastily
+##jima
+Quincy
+Sol
+fascinated
+Scarlet
+alto
+Avon
+certainty
+##eding
+Keys
+##chu
+Chu
+##VE
+ions
+tributaries
+Thanksgiving
+##fusion
+astronomer
+oxide
+pavilion
+Supply
+Casa
+Bollywood
+sadly
+mutations
+Keller
+##wave
+nationals
+##rgo
+##ym
+predict
+Catholicism
+Vega
+##eration
+##ums
+Mali
+tuned
+Lankan
+Plans
+radial
+Bosnian
+Lexi
+##14
+##ü
+sacks
+unpleasant
+Empty
+handles
+##taking
+Bon
+switches
+intently
+tuition
+antique
+##jk
+fraternity
+notebook
+Desmond
+##sei
+prostitution
+##how
+deed
+##OP
+501
+Somewhere
+Rocks
+##mons
+campaigned
+frigate
+gases
+suppress
+##hang
+Merlin
+Northumberland
+dominate
+expeditions
+thunder
+##ups
+##rical
+Cap
+thorough
+Ariel
+##kind
+renewable
+constructing
+pacing
+terrorists
+Bowen
+documentaries
+westward
+##lass
+##nage
+Merchant
+##ued
+Beaumont
+Din
+##hian
+Danube
+peasant
+Garrison
+encourages
+gratitude
+reminding
+stormed
+##ouse
+pronunciation
+##ailed
+Weekend
+suggestions
+##ffing
+##DI
+Active
+Colombo
+##logists
+Merrill
+##cens
+Archaeological
+Medina
+captained
+##yk
+duel
+cracking
+Wilkinson
+Guam
+pickup
+renovations
+##ël
+##izer
+delighted
+##iri
+Weaver
+##ctional
+tens
+##hab
+Clint
+##usion
+##each
+petals
+Farrell
+##sable
+caste
+##will
+Ezra
+##qi
+##standing
+thrilled
+ambush
+exhaled
+##SU
+Resource
+blur
+forearm
+specifications
+contingent
+cafe
+##iology
+Antony
+fundraising
+grape
+##rgy
+turnout
+##udi
+Clifton
+laboratories
+Irvine
+##opus
+##lid
+Monthly
+Bihar
+statutory
+Roses
+Emil
+##rig
+lumber
+optimal
+##DR
+pumps
+plaster
+Mozambique
+##aco
+nightclub
+propelled
+##hun
+ked
+surplus
+wax
+##urai
+pioneered
+Sunny
+imprint
+Forget
+Eliot
+approximate
+patronage
+##bek
+##ely
+##mbe
+Partnership
+curl
+snapping
+29th
+Patriarch
+##jord
+seldom
+##ature
+astronomy
+Bremen
+XIV
+airborne
+205
+1778
+recognizing
+stranded
+arrogant
+bombardment
+destined
+ensured
+146
+robust
+Davenport
+Interactive
+Offensive
+Fi
+prevents
+probe
+propeller
+sorrow
+Blade
+mounting
+automotive
+##dged
+wallet
+201
+lashes
+Forrest
+##ift
+Cell
+Younger
+shouts
+##cki
+folds
+##chet
+Epic
+yields
+homosexual
+tunes
+##minate
+##text
+Manny
+chemist
+hindwings
+##urn
+pilgrimage
+##sfield
+##riff
+MLS
+##rive
+Huntington
+translates
+Path
+slim
+##ndra
+##oz
+climax
+commuter
+desperation
+##reet
+denying
+##rious
+daring
+seminary
+polo
+##clamation
+Teatro
+Torah
+Cats
+identities
+Poles
+photographed
+fiery
+popularly
+##cross
+winters
+Hesse
+##vio
+Nurse
+Senegal
+Salon
+prescribed
+justify
+##gues
+##и
+##orted
+HQ
+##hiro
+evaluated
+momentarily
+##unts
+Debbie
+##licity
+##TP
+Mighty
+Rabbit
+##chal
+Events
+Savoy
+##ht
+Brandenburg
+Bordeaux
+##laus
+Release
+##IE
+##kowski
+1900s
+SK
+Strauss
+##aly
+Sonia
+Updated
+synagogue
+McKay
+flattened
+370
+clutch
+contests
+toast
+evaluate
+pope
+heirs
+jam
+tutor
+reverted
+##ading
+nonsense
+hesitate
+Lars
+Ceylon
+Laurie
+##guchi
+accordingly
+customary
+148
+Ethics
+Multiple
+instincts
+IGN
+##ä
+bullshit
+##hit
+##par
+desirable
+##ducing
+##yam
+alias
+ashore
+licenses
+##lification
+misery
+147
+Cola
+assassinated
+fiercely
+##aft
+las
+goat
+substrate
+lords
+Cass
+Bridges
+ICC
+lasts
+sights
+reproductive
+##asi
+Ivory
+Clean
+fixing
+##lace
+seeming
+aide
+1850s
+harassment
+##FF
+##LE
+reasonably
+##coat
+##cano
+NYC
+1784
+Fifty
+immunity
+Canadians
+Cheng
+comforting
+meanwhile
+##tera
+##blin
+breeds
+glowed
+##vour
+Aden
+##verted
+##aded
+##oral
+neat
+enforced
+poisoning
+##ews
+##hone
+enforce
+predecessors
+survivor
+Month
+unfamiliar
+pierced
+waived
+dump
+responds
+Mai
+Declan
+angular
+Doesn
+interpretations
+##yar
+invest
+Dhaka
+policeman
+Congregation
+Eighth
+painfully
+##este
+##vior
+Württemberg
+##cles
+blockade
+encouragement
+##fie
+Caucasus
+Malone
+Universidad
+utilize
+Nissan
+inherent
+151
+agreeing
+syllable
+determines
+Protocol
+conclude
+##gara
+40th
+Xu
+Taiwanese
+##ather
+boiler
+printer
+Lacey
+titular
+Klaus
+Fallon
+Wembley
+fox
+Chandra
+Governorate
+obsessed
+##Ps
+micro
+##25
+Cooke
+gymnasium
+weaving
+Shall
+Hussein
+glaring
+softball
+Reader
+Dominion
+Trouble
+varsity
+Cooperation
+Chaos
+Kang
+Kramer
+Eisenhower
+proves
+Connie
+consortium
+governors
+Bethany
+opener
+Normally
+Willy
+linebacker
+Regent
+Used
+AllMusic
+Twilight
+##shaw
+Companion
+Tribunal
+simpler
+##gam
+Experimental
+Slovenian
+cellar
+deadline
+trout
+Hubbard
+ads
+idol
+##hetto
+Granada
+clues
+salmon
+1700
+Omega
+Caldwell
+softened
+Bills
+Honolulu
+##gn
+Terrace
+suitcase
+##IL
+frantic
+##oons
+Abbot
+Sitting
+Fortress
+Riders
+sickness
+enzymes
+trustee
+Bern
+forged
+##13
+##ruff
+##rl
+##versity
+inspector
+champagne
+##held
+##FI
+hereditary
+Taliban
+handball
+##wine
+Sioux
+##dicated
+honoured
+139
+##tude
+Skye
+meanings
+##rkin
+cardiac
+analyzed
+vegetable
+##FS
+Royals
+dial
+freelance
+##fest
+partisan
+petroleum
+ridden
+Lincolnshire
+panting
+##comb
+presidents
+Haley
+##chs
+contributes
+Jew
+discoveries
+panicked
+Woody
+eyelids
+Fate
+Tulsa
+mg
+whiskey
+zombies
+Wii
+##udge
+investigators
+##bull
+centred
+##screen
+Bone
+Lana
+##oise
+forts
+##ske
+Conan
+Lyons
+##writing
+SH
+##ride
+rhythmic
+154
+##llah
+pioneers
+##bright
+captivity
+Sanchez
+Oman
+##mith
+Flint
+Platform
+##ioned
+emission
+packet
+Persia
+##formed
+takeover
+tempted
+Vance
+Few
+Toni
+receptions
+##ن
+exchanges
+Camille
+whale
+Chronicles
+##rent
+##ushing
+##rift
+Alto
+Genus
+##asing
+onward
+foremost
+longing
+Rockefeller
+containers
+##cribe
+intercepted
+##olt
+pleading
+Bye
+bee
+##umbling
+153
+undertake
+Izzy
+cheaper
+Ultra
+validity
+##pse
+Sa
+hovering
+##pert
+vintage
+engraved
+##rise
+farmland
+##ever
+##ifier
+Atlantis
+propose
+Catalonia
+plunged
+##edly
+demonstrates
+gig
+##cover
+156
+Osborne
+cowboy
+herd
+investigator
+loops
+Burning
+rests
+Instrumental
+embarrassing
+focal
+install
+readings
+swirling
+Chatham
+parameter
+##zin
+##holders
+Mandarin
+Moody
+converting
+Escape
+warnings
+##chester
+incarnation
+##ophone
+adopting
+##lins
+Cromwell
+##laws
+Axis
+Verde
+Kappa
+Schwartz
+Serbs
+caliber
+Wanna
+Chung
+##ality
+nursery
+principally
+Bulletin
+likelihood
+logging
+##erty
+Boyle
+supportive
+twitched
+##usive
+builds
+Marseille
+omitted
+motif
+Lands
+##lusion
+##ssed
+Barrow
+Airfield
+Harmony
+WWF
+endured
+merging
+convey
+branding
+examinations
+167
+Italians
+##dh
+dude
+1781
+##teau
+crawling
+thoughtful
+clasped
+concluding
+brewery
+Moldova
+Wan
+Towers
+Heidelberg
+202
+##ict
+Lagos
+imposing
+##eval
+##serve
+Bacon
+frowning
+thirteenth
+conception
+calculations
+##ович
+##mile
+##ivated
+mutation
+strap
+##lund
+demographic
+nude
+perfection
+stocks
+##renched
+##dit
+Alejandro
+bites
+fragment
+##hack
+##rchy
+GB
+Surgery
+Berger
+punish
+boiling
+consume
+Elle
+Sid
+Dome
+relies
+Crescent
+treasurer
+Bloody
+1758
+upheld
+Guess
+Restaurant
+signatures
+font
+millennium
+mural
+stakes
+Abel
+hailed
+insists
+Alumni
+Breton
+##jun
+digits
+##FM
+##thal
+Talking
+motive
+reigning
+babe
+masks
+##ø
+Shaun
+potato
+sour
+whitish
+Somali
+##derman
+##rab
+##wy
+chancel
+telecommunications
+Noise
+messenger
+tidal
+grinding
+##ogenic
+Rebel
+constituent
+peripheral
+recruitment
+##ograph
+##tler
+pumped
+Ravi
+poked
+##gley
+Olive
+diabetes
+discs
+liking
+sting
+fits
+stir
+Mari
+Sega
+creativity
+weights
+Macau
+mandated
+Bohemia
+disastrous
+Katrina
+Baku
+Rajasthan
+waiter
+##psis
+Siberia
+verbs
+##truction
+patented
+1782
+##ndon
+Relegated
+Hunters
+Greenwood
+Shock
+accusing
+skipped
+Sessions
+markers
+subset
+monumental
+Viola
+comparative
+Alright
+Barbados
+setup
+Session
+standardized
+##ík
+##sket
+appoint
+AFB
+Nationalist
+##WS
+Troop
+leaped
+Treasure
+goodness
+weary
+originates
+100th
+compassion
+expresses
+recommend
+168
+composing
+seventeenth
+Tex
+Atlético
+bald
+Finding
+Presidency
+Sharks
+favoured
+inactive
+##lter
+suffix
+princes
+brighter
+##ctus
+classics
+defendants
+culminated
+terribly
+Strategy
+evenings
+##ção
+##iver
+##urance
+absorb
+##rner
+Territories
+RBI
+soothing
+Martín
+concurrently
+##tr
+Nicholson
+fibers
+swam
+##oney
+Allie
+Algerian
+Dartmouth
+Mafia
+##bos
+##tts
+Councillor
+vocabulary
+##bla
+##lé
+intending
+##dler
+Guerrero
+sunshine
+pedal
+##TO
+administrators
+periodic
+scholarships
+Loop
+Madeline
+exaggerated
+##ressed
+Regan
+##cellular
+Explorer
+##oids
+Alexandre
+vows
+Reporter
+Unable
+Average
+absorption
+##bedience
+Fortunately
+Auxiliary
+Grandpa
+##HP
+##ovo
+potent
+temporal
+adrenaline
+##udo
+confusing
+guiding
+Dry
+qualifications
+joking
+wherein
+heavyweight
+##ices
+nightmares
+pharmaceutical
+Commanding
+##aled
+##ove
+Gregor
+##UP
+censorship
+degradation
+glorious
+Austro
+##rench
+380
+Miriam
+sped
+##orous
+offset
+##KA
+fined
+specialists
+Pune
+João
+##dina
+propped
+fungus
+##ς
+frantically
+Gabrielle
+Hare
+committing
+##plied
+Ask
+Wilmington
+stunt
+numb
+warmer
+preacher
+earnings
+##lating
+integer
+##ija
+federation
+homosexuality
+##cademia
+epidemic
+grumbled
+shoving
+Milk
+Satan
+Tobias
+innovations
+##dington
+geology
+memoirs
+##IR
+spared
+culminating
+Daphne
+Focus
+severed
+stricken
+Paige
+Mans
+flats
+Russo
+communes
+litigation
+strengthening
+##powered
+Staffordshire
+Wiltshire
+Painting
+Watkins
+##د
+specializes
+Select
+##rane
+##aver
+Fulton
+playable
+##VN
+openings
+sampling
+##coon
+##21
+Allah
+travelers
+allocation
+##arily
+Loch
+##hm
+commentators
+fulfilled
+##troke
+Emeritus
+Vanderbilt
+Vijay
+pledged
+##tative
+diagram
+drilling
+##MD
+##plain
+Edison
+productivity
+31st
+##rying
+##ption
+##gano
+##oration
+##bara
+posture
+bothering
+platoon
+politely
+##inating
+redevelopment
+Job
+##vale
+stark
+incorrect
+Mansion
+renewal
+threatens
+Bahamas
+fridge
+##tata
+Uzbekistan
+##edia
+Sainte
+##mio
+gaps
+neural
+##storm
+overturned
+Preservation
+shields
+##ngo
+##physics
+ah
+gradual
+killings
+##anza
+consultation
+premiership
+Felipe
+coincidence
+##ène
+##any
+Handbook
+##loaded
+Edit
+Guns
+arguably
+##ş
+compressed
+depict
+seller
+##qui
+Kilkenny
+##kling
+Olympia
+librarian
+##acles
+dramas
+JP
+Kit
+Maj
+##lists
+proprietary
+##nged
+##ettes
+##tok
+exceeding
+Lock
+induction
+numerical
+##vist
+Straight
+foyer
+imaginary
+##pop
+violinist
+Carla
+bouncing
+##ashi
+abolition
+##uction
+restoring
+scenic
+##č
+Doom
+overthrow
+para
+##vid
+##ughty
+Concord
+HC
+cocaine
+deputies
+##aul
+visibility
+##wart
+Kapoor
+Hutchinson
+##agan
+flashes
+kn
+decreasing
+##ronology
+quotes
+vain
+satisfying
+##iam
+##linger
+310
+Hanson
+fauna
+##zawa
+##rrel
+Trenton
+##VB
+Employment
+vocational
+Exactly
+bartender
+butterflies
+tow
+##chers
+##ocks
+pigs
+merchandise
+##game
+##pine
+Shea
+##gration
+Connell
+Josephine
+monopoly
+##dled
+Cobb
+warships
+cancellation
+someday
+stove
+##Cs
+candidacy
+superhero
+unrest
+Toulouse
+admiration
+undergone
+whirled
+Reconnaissance
+costly
+##ships
+290
+Cafe
+amber
+Tory
+##mpt
+definitive
+##dress
+proposes
+redesigned
+acceleration
+##asa
+##raphy
+Presley
+exits
+Languages
+##cel
+Mode
+spokesperson
+##tius
+Ban
+forthcoming
+grounded
+ACC
+compelling
+logistics
+retailers
+abused
+##gating
+soda
+##yland
+##lution
+Landmark
+XVI
+blush
+##tem
+hurling
+dread
+Tobago
+Foley
+##uad
+scenarios
+##mentation
+##rks
+Score
+fatigue
+hairy
+correspond
+##iard
+defences
+confiscated
+##rudence
+1785
+Formerly
+Shot
+advertised
+460
+Text
+ridges
+Promise
+Dev
+exclusion
+NHS
+tuberculosis
+rockets
+##offs
+sparkling
+256
+disappears
+mankind
+##hore
+HP
+##omo
+taxation
+Multi
+DS
+Virgil
+##ams
+Dell
+stacked
+guessing
+Jump
+Nope
+cheer
+hates
+ballots
+overlooked
+analyses
+Prevention
+maturity
+dos
+##cards
+##lect
+Mare
+##yssa
+Petty
+##wning
+differing
+iOS
+##ior
+Joachim
+Sentinel
+##nstein
+90s
+Pamela
+480
+Asher
+##lary
+Vicente
+landings
+portray
+##rda
+##xley
+Virtual
+##uary
+finances
+Jain
+Somebody
+Tri
+behave
+Michele
+##ider
+dwellings
+FAA
+Gallagher
+##lide
+Monkey
+195
+aforementioned
+##rism
+##bey
+##kim
+##puted
+Mesa
+hopped
+unopposed
+recipients
+Reality
+Been
+gritted
+149
+playground
+pillar
+##rone
+Guinness
+##tad
+Théâtre
+depended
+Tipperary
+Reuben
+frightening
+wooded
+Target
+globally
+##uted
+Morales
+Baptiste
+drunken
+Institut
+characterised
+##chemistry
+Strip
+discrete
+Premiership
+##zzling
+gazing
+Outer
+##quisition
+Sikh
+Booker
+##yal
+contemporaries
+Jericho
+##chan
+##physical
+##witch
+Militia
+##rez
+##zard
+dangers
+##utter
+##₀
+Programs
+darling
+participates
+railroads
+##ienne
+behavioral
+bureau
+##rook
+161
+Hicks
+##rises
+Comes
+inflicted
+bees
+kindness
+norm
+##ković
+generators
+##pard
+##omy
+##ili
+methodology
+Alvin
+façade
+latitude
+##plified
+DE
+Morse
+##mered
+educate
+intersects
+##MF
+##cz
+##vated
+AL
+##graded
+##fill
+constitutes
+artery
+feudal
+avant
+cautious
+##ogue
+immigrated
+##chenko
+Saul
+Clinic
+Fang
+choke
+Cornelius
+flexibility
+temperate
+pins
+##erson
+oddly
+inequality
+157
+Natasha
+Sal
+##uter
+215
+aft
+blinking
+##ntino
+northward
+Exposition
+cookies
+Wedding
+impulse
+Overseas
+terrifying
+##ough
+Mortimer
+##see
+440
+https
+og
+imagining
+##cars
+Nicola
+exceptionally
+threads
+##cup
+Oswald
+Provisional
+dismantled
+deserves
+1786
+Fairy
+discourse
+Counsel
+departing
+Arc
+guarding
+##orse
+420
+alterations
+vibrant
+Em
+squinted
+terrace
+rowing
+Led
+accessories
+SF
+Sgt
+cheating
+Atomic
+##raj
+Blackpool
+##iary
+boarded
+substituted
+bestowed
+lime
+kernel
+##jah
+Belmont
+shaken
+sticky
+retrospective
+Louie
+migrants
+weigh
+sunglasses
+thumbs
+##hoff
+excavation
+##nks
+Extra
+Polo
+motives
+Drum
+infrared
+tastes
+berth
+verge
+##stand
+programmed
+warmed
+Shankar
+Titan
+chromosome
+cafeteria
+dividing
+pepper
+CPU
+Stevie
+satirical
+Nagar
+scowled
+Died
+backyard
+##gata
+##reath
+##bir
+Governors
+portraying
+##yah
+Revenge
+##acing
+1772
+margins
+Bahn
+OH
+lowland
+##razed
+catcher
+replay
+##yoshi
+Seriously
+##licit
+Aristotle
+##ald
+Habsburg
+weekday
+Secretariat
+CO
+##dly
+##joy
+##stad
+litre
+ultra
+##cke
+Mongol
+Tucson
+correlation
+compose
+traps
+Groups
+Hai
+Salvatore
+##dea
+cents
+##eese
+concession
+clash
+Trip
+Panzer
+Moroccan
+cruisers
+torque
+Ba
+grossed
+##arate
+restriction
+concentrating
+FDA
+##Leod
+##ones
+Scholars
+##esi
+throbbing
+specialised
+##heses
+Chicken
+##fia
+##ificant
+Erich
+Residence
+##trate
+manipulation
+namesake
+##tom
+Hoover
+cue
+Lindsey
+Lonely
+275
+##HT
+combustion
+subscribers
+Punjabi
+respects
+Jeremiah
+penned
+##gor
+##rilla
+suppression
+##tration
+Crimson
+piston
+Derry
+crimson
+lyrical
+oversee
+portrays
+CF
+Districts
+Lenin
+Cora
+searches
+clans
+VHS
+##hel
+Jacqueline
+Redskins
+Clubs
+desktop
+indirectly
+alternatives
+marijuana
+suffrage
+##smos
+Irwin
+##liff
+Process
+##hawks
+Sloane
+##bson
+Sonata
+yielded
+Flores
+##ares
+armament
+adaptations
+integrate
+neighbours
+shelters
+##tour
+Skinner
+##jet
+##tations
+1774
+Peterborough
+##elles
+ripping
+Liang
+Dickinson
+charities
+Rwanda
+monasteries
+crossover
+racist
+barked
+guerrilla
+##ivate
+Grayson
+##iques
+##vious
+##got
+Rolls
+denominations
+atom
+affinity
+##delity
+Wish
+##inted
+##inae
+interrogation
+##cey
+##erina
+##lifting
+192
+Sands
+1779
+mast
+Likewise
+##hyl
+##oft
+contempt
+##por
+assaulted
+fills
+establishments
+Mal
+consulted
+##omi
+##sight
+greet
+##roma
+##egan
+Pulitzer
+##rried
+##dius
+##ractical
+##voked
+Hasan
+CB
+##zzy
+Romanesque
+Panic
+wheeled
+recorder
+##tters
+##warm
+##gly
+botanist
+Balkan
+Lockheed
+Polly
+farewell
+suffers
+purchases
+Eaton
+##80
+Quick
+commenting
+Saga
+beasts
+hides
+motifs
+##icks
+Alonso
+Springer
+Wikipedia
+circulated
+encoding
+jurisdictions
+snout
+UAE
+Integrated
+unmarried
+Heinz
+##lein
+##figured
+deleted
+##tley
+Zen
+Cycling
+Fuel
+Scandinavian
+##rants
+Conner
+reef
+Marino
+curiously
+lingered
+Gina
+manners
+activism
+Mines
+Expo
+Micah
+promotions
+Server
+booked
+derivatives
+eastward
+detailing
+reelection
+##chase
+182
+Campeonato
+Po
+158
+Peel
+winger
+##itch
+canyon
+##pit
+LDS
+A1
+##shin
+Giorgio
+pathetic
+##rga
+##mist
+Aren
+##lag
+confronts
+motel
+textbook
+shine
+turbines
+1770
+Darcy
+##cot
+Southeastern
+##lessness
+Banner
+recognise
+stray
+Kitchen
+paperwork
+realism
+Chrysler
+filmmakers
+fishermen
+##hetic
+variously
+Vishnu
+fiddle
+Eddy
+Origin
+##tec
+##ulin
+Flames
+Rs
+bankrupt
+Extreme
+Pomeranian
+##emption
+ratified
+##iu
+jockey
+Stratford
+##ivating
+##oire
+Babylon
+pardon
+AI
+affordable
+deities
+disturbance
+Trying
+##sai
+Ida
+Papers
+advancement
+70s
+archbishop
+Luftwaffe
+announces
+tugging
+##lphin
+##sistence
+##eel
+##ishes
+ambition
+aura
+##fled
+##lected
+##vue
+Prasad
+boiled
+clarity
+Violin
+investigative
+routing
+Yankee
+##uckle
+McMahon
+bugs
+eruption
+##rooms
+Minutes
+relics
+##ckle
+##nse
+sipped
+valves
+weakly
+##ital
+Middleton
+collided
+##quer
+bamboo
+insignia
+Tyne
+exercised
+Ninth
+echoing
+polynomial
+considerations
+lunged
+##bius
+objections
+complain
+disguised
+plaza
+##VC
+institutes
+Judicial
+ascent
+imminent
+Waterford
+hello
+Lumpur
+Niger
+Goldman
+vendors
+Kensington
+Wren
+browser
+##bner
+##tri
+##mize
+##pis
+##lea
+Cheyenne
+Bold
+Settlement
+Hollow
+Paralympic
+axle
+##toire
+##actic
+impose
+perched
+utilizing
+slips
+Benz
+Michaels
+manipulate
+Chiang
+##mian
+Dolphins
+prohibition
+attacker
+ecology
+Estadio
+##SB
+##uild
+attracts
+recalls
+glacier
+lad
+##rima
+Barlow
+kHz
+melodic
+##aby
+##iracy
+assumptions
+Cornish
+##aru
+DOS
+Maddie
+##mers
+lyric
+Luton
+nm
+##tron
+Reno
+Fin
+YOU
+Broadcast
+Finch
+sensory
+##bent
+Jeep
+##uman
+additionally
+Buildings
+businessmen
+treaties
+235
+Stranger
+gateway
+Charlton
+accomplishments
+Diary
+apologized
+zinc
+histories
+supplier
+##tting
+162
+asphalt
+Treatment
+Abbas
+##pating
+##yres
+Bloom
+sedan
+soloist
+##cum
+antagonist
+denounced
+Fairfax
+##aving
+##enko
+noticeable
+Budget
+Buckingham
+Snyder
+retreating
+Jai
+spoon
+invading
+giggle
+woven
+gunfire
+arrests
+##vered
+##come
+respiratory
+violet
+##aws
+Byrd
+shocking
+tenant
+Jamaican
+Ottomans
+Seal
+theirs
+##isse
+##48
+cooperate
+peering
+##nius
+163
+Composer
+organist
+Mongolian
+Bauer
+Spy
+collects
+prophecy
+congregations
+##moor
+Brick
+calculation
+fixtures
+exempt
+##dden
+Ada
+Thousand
+##lue
+tracing
+##achi
+bodyguard
+vicar
+supplying
+Łódź
+interception
+monitored
+##heart
+Paso
+overlap
+annoyance
+##dice
+yellowish
+stables
+elders
+illegally
+honesty
+##oar
+skinny
+spinal
+##puram
+Bourbon
+##cor
+flourished
+Medium
+##stics
+##aba
+Follow
+##ckey
+stationary
+##scription
+dresser
+scrutiny
+Buckley
+Clearly
+##SF
+Lyrics
+##heimer
+drying
+Oracle
+internally
+rains
+##last
+Enemy
+##oes
+McLean
+Ole
+phosphate
+Rosario
+Rifles
+##mium
+battered
+Pepper
+Presidents
+conquer
+Château
+castles
+##aldo
+##ulf
+Depending
+Lesser
+Boom
+trades
+Peyton
+164
+emphasize
+accustomed
+SM
+Ai
+Classification
+##mins
+##35
+##rons
+leak
+piled
+deeds
+lush
+##self
+beginnings
+breathless
+1660
+McGill
+##ago
+##chaft
+##gies
+humour
+Bomb
+securities
+Might
+##zone
+##eves
+Matthias
+Movies
+Levine
+vengeance
+##ads
+Challenger
+Misty
+Traditionally
+constellation
+##rass
+deepest
+workplace
+##oof
+##vina
+impatient
+##ML
+Mughal
+Alessandro
+scenery
+Slater
+postseason
+troupe
+##ń
+Volunteers
+Facility
+militants
+Reggie
+sanctions
+Expeditionary
+Nam
+countered
+interpret
+Basilica
+coding
+expectation
+Duffy
+def
+Tong
+wakes
+Bowling
+Vehicle
+Adler
+salad
+intricate
+stronghold
+medley
+##uries
+##bur
+joints
+##rac
+##yx
+##IO
+Ordnance
+Welch
+distributor
+Ark
+cavern
+trench
+Weiss
+Mauritius
+decreases
+docks
+eagerly
+irritation
+Matilda
+biographer
+Visiting
+##marked
+##iter
+##ear
+##gong
+Moreno
+attendant
+Bury
+instrumentation
+theologian
+clit
+nuns
+symphony
+translate
+375
+loser
+##user
+##VR
+##meter
+##orious
+harmful
+##yuki
+Commissioners
+Mendoza
+sniffed
+Hulk
+##dded
+##ulator
+##nz
+Donnell
+##eka
+deported
+Met
+SD
+Aerospace
+##cultural
+##odes
+Fantastic
+cavity
+remark
+emblem
+fearing
+##iance
+ICAO
+Liberia
+stab
+##yd
+Pac
+Gymnasium
+IS
+Everton
+##vanna
+mantle
+##ief
+Ramon
+##genic
+Shooting
+Smoke
+Random
+Africans
+MB
+tavern
+bargain
+voluntarily
+Ion
+Peoples
+Rusty
+attackers
+Patton
+sins
+##cake
+Hat
+moderately
+##hala
+##alia
+requesting
+mechanic
+##eae
+Seine
+Robbins
+##ulum
+susceptible
+Bravo
+Slade
+Strasbourg
+rubble
+entrusted
+Creation
+##amp
+smoothed
+##uintet
+evenly
+reviewers
+skip
+Sculpture
+177
+Rough
+##rrie
+Reeves
+##cede
+Administrator
+garde
+minus
+carriages
+grenade
+Ninja
+fuscous
+##kley
+Punk
+contributors
+Aragon
+Tottenham
+##cca
+##sir
+VA
+laced
+dealers
+##sonic
+crisp
+harmonica
+Artistic
+Butch
+Andes
+Farmers
+corridors
+unseen
+##tium
+Countries
+Lone
+envisioned
+Katy
+##lang
+##cc
+Quarterly
+##neck
+consort
+##aceae
+bidding
+Corey
+concurrent
+##acts
+##gum
+Highness
+##lient
+##rators
+arising
+##unta
+pathways
+49ers
+bolted
+complaining
+ecosystem
+libretto
+Ser
+narrated
+212
+Soft
+influx
+##dder
+incorporation
+plagued
+tents
+##ddled
+1750
+Risk
+citation
+Tomas
+hostilities
+seals
+Bruins
+Dominique
+attic
+competent
+##UR
+##cci
+hugging
+Breuning
+bacterial
+Shrewsbury
+vowed
+eh
+elongated
+hangs
+render
+centimeters
+##ficient
+Mu
+turtle
+besieged
+##gaard
+grapes
+bravery
+collaborations
+deprived
+##amine
+##using
+##gins
+arid
+##uve
+coats
+hanged
+##sting
+Pa
+prefix
+##ranged
+Exit
+Chain
+Flood
+Materials
+suspicions
+##ö
+hovered
+Hidden
+##state
+Malawi
+##24
+Mandy
+norms
+fascinating
+airlines
+delivers
+##rust
+Cretaceous
+spanned
+pillows
+##onomy
+jar
+##kka
+regent
+fireworks
+morality
+discomfort
+lure
+uneven
+##jack
+Lucian
+171
+archaeology
+##til
+mornings
+Billie
+Marquess
+impending
+spilling
+tombs
+##volved
+Celia
+Coke
+underside
+##bation
+Vaughn
+Daytona
+Godfrey
+Pascal
+Alien
+##sign
+172
+##lage
+iPhone
+Gonna
+genocide
+##rber
+oven
+endure
+dashed
+simultaneous
+##phism
+Wally
+##rō
+ants
+predator
+reissue
+##aper
+Speech
+funk
+Rudy
+claw
+Hindus
+Numbers
+Bing
+lantern
+##aurus
+scattering
+poisoned
+##active
+Andrei
+algebraic
+baseman
+##ritz
+Gregg
+##cola
+selections
+##putation
+lick
+Laguna
+##IX
+Sumatra
+Warning
+turf
+buyers
+Burgess
+Oldham
+exploit
+worm
+initiate
+strapped
+tuning
+filters
+haze
+##е
+##ledge
+##ydro
+##culture
+amendments
+Promotion
+##union
+Clair
+##uria
+petty
+shutting
+##eveloped
+Phoebe
+Zeke
+conducts
+grains
+clashes
+##latter
+illegitimate
+willingly
+Deer
+Lakers
+Reference
+chaplain
+commitments
+interrupt
+salvation
+Panther
+Qualifying
+Assessment
+cancel
+efficiently
+attorneys
+Dynamo
+impress
+accession
+clinging
+randomly
+reviewing
+Romero
+Cathy
+charting
+clapped
+rebranded
+Azerbaijani
+coma
+indicator
+punches
+##tons
+Sami
+monastic
+prospects
+Pastor
+##rville
+electrified
+##CI
+##utical
+tumbled
+Chef
+muzzle
+selecting
+UP
+Wheel
+protocols
+##tat
+Extended
+beautifully
+nests
+##stal
+Andersen
+##anu
+##³
+##rini
+kneeling
+##reis
+##xia
+anatomy
+dusty
+Safe
+turmoil
+Bianca
+##elo
+analyze
+##ر
+##eran
+podcast
+Slovene
+Locke
+Rue
+##retta
+##uni
+Person
+Prophet
+crooked
+disagreed
+Versailles
+Sarajevo
+Utrecht
+##ogen
+chewing
+##ception
+##iidae
+Missile
+attribute
+majors
+Arch
+intellectuals
+##andra
+ideological
+Cory
+Salzburg
+##fair
+Lot
+electromagnetic
+Distribution
+##oper
+##pered
+Russ
+Terra
+repeats
+fluttered
+Riga
+##ific
+##gt
+cows
+Hair
+labelled
+protects
+Gale
+Personnel
+Düsseldorf
+Moran
+rematch
+##OE
+Slow
+forgiveness
+##ssi
+proudly
+Macmillan
+insist
+undoubtedly
+Québec
+Violence
+##yuan
+##aine
+mourning
+linen
+accidental
+##iol
+##arium
+grossing
+lattice
+maneuver
+##marine
+prestige
+petrol
+gradient
+invasive
+militant
+Galerie
+widening
+##aman
+##quist
+disagreement
+##ales
+creepy
+remembers
+buzz
+##erial
+Exempt
+Dirk
+mon
+Addison
+##inen
+deposed
+##agon
+fifteenth
+Hang
+ornate
+slab
+##lades
+Fountain
+contractors
+das
+Warwickshire
+1763
+##rc
+Carly
+Essays
+Indy
+Ligue
+greenhouse
+slit
+##sea
+chewed
+wink
+##azi
+Playhouse
+##kon
+Gram
+Ko
+Samson
+creators
+revive
+##rians
+spawned
+seminars
+Craft
+Tall
+diverted
+assistants
+computational
+enclosure
+##acity
+Coca
+##eve
+databases
+Drop
+##loading
+##hage
+Greco
+Privy
+entrances
+pork
+prospective
+Memories
+robes
+##market
+transporting
+##lik
+Rudolph
+Horton
+visually
+##uay
+##nja
+Centro
+Tor
+Howell
+##rsey
+admitting
+postgraduate
+herbs
+##att
+Chin
+Rutherford
+##bot
+##etta
+Seasons
+explanations
+##bery
+Friedman
+heap
+##ryl
+##sberg
+jaws
+##agh
+Choi
+Killing
+Fanny
+##suming
+##hawk
+hopeful
+##aid
+Monty
+gum
+remarkably
+Secrets
+disco
+harp
+advise
+##avia
+Marathi
+##cycle
+Truck
+abbot
+sincere
+urine
+##mology
+masked
+bathing
+##tun
+Fellows
+##TM
+##gnetic
+owl
+##jon
+hymn
+##leton
+208
+hostility
+##cée
+baked
+Bottom
+##AB
+shudder
+##ater
+##von
+##hee
+reorganization
+Cycle
+##phs
+Lex
+##style
+##rms
+Translation
+##erick
+##imeter
+##ière
+attested
+Hillary
+##DM
+gal
+wander
+Salle
+##laming
+Perez
+Pit
+##LP
+USAF
+contexts
+Disease
+blazing
+aroused
+razor
+walled
+Danielle
+Mont
+Funk
+royalty
+thee
+203
+donors
+##erton
+famously
+processors
+reassigned
+welcoming
+Goldberg
+##quities
+undisclosed
+Orient
+Patty
+vaccine
+refrigerator
+Cypriot
+consonant
+##waters
+176
+sober
+##lement
+Racecourse
+##uate
+Luckily
+Selection
+conceptual
+vines
+Breaking
+wa
+lions
+oversight
+sheltered
+Dancer
+ponds
+borrow
+##BB
+##pulsion
+Daly
+##eek
+fertility
+spontaneous
+Worldwide
+gasping
+##tino
+169
+ABS
+Vickers
+ambient
+energetic
+prisons
+##eson
+Stacy
+##roach
+GmbH
+Afro
+Marin
+farmhouse
+pinched
+##cursion
+##sp
+Sabine
+##pire
+181
+nak
+swelling
+humble
+perfume
+##balls
+Rai
+cannons
+##taker
+Married
+Maltese
+canals
+interceptions
+hats
+lever
+slowing
+##ppy
+Nike
+Silas
+Scarborough
+skirts
+166
+inauguration
+Shuttle
+alloy
+beads
+belts
+Compton
+Cause
+battling
+critique
+surf
+Dock
+roommate
+##ulet
+invade
+Garland
+##slow
+nutrition
+persona
+##zam
+Wichita
+acquaintance
+coincided
+##cate
+Dracula
+clamped
+##gau
+overhaul
+##broken
+##rrier
+melodies
+ventures
+Paz
+convex
+Roots
+##holding
+Tribute
+transgender
+##ò
+chimney
+##riad
+Ajax
+Thereafter
+messed
+nowadays
+pH
+##100
+##alog
+Pomerania
+##yra
+Rossi
+glove
+##TL
+Races
+##asily
+tablets
+Jase
+##ttes
+diner
+##rns
+Hu
+Mohan
+anytime
+weighted
+remixes
+Dove
+cherry
+imports
+##urity
+GA
+##TT
+##iated
+##sford
+Clarkson
+evidently
+rugged
+Dust
+siding
+##ometer
+acquitted
+choral
+##mite
+infants
+Domenico
+gallons
+Atkinson
+gestures
+slated
+##xa
+Archaeology
+unwanted
+##ibes
+##duced
+premise
+Colby
+Geelong
+disqualified
+##pf
+##voking
+simplicity
+Walkover
+Qaeda
+Warden
+##bourg
+##ān
+Invasion
+Babe
+harness
+183
+##tated
+maze
+Burt
+bedrooms
+##nsley
+Horizon
+##oast
+minimize
+peeked
+MLA
+Trains
+tractor
+nudged
+##iform
+Growth
+Benton
+separates
+##about
+##kari
+buffer
+anthropology
+brigades
+foil
+##wu
+Domain
+licking
+whore
+##rage
+##sham
+Initial
+Courthouse
+Rutgers
+dams
+villains
+supermarket
+##brush
+Brunei
+Palermo
+arises
+Passenger
+outreach
+##gill
+Labrador
+McLaren
+##uy
+Lori
+##fires
+Heads
+magistrate
+¹⁄₂
+Weapons
+##wai
+##roke
+projecting
+##ulates
+bordering
+McKenzie
+Pavel
+midway
+Guangzhou
+streamed
+racer
+##lished
+eccentric
+spectral
+206
+##mism
+Wilde
+Grange
+preparatory
+lent
+##tam
+starving
+Gertrude
+##cea
+##ricted
+Breakfast
+Mira
+blurted
+derive
+##lair
+blunt
+sob
+Cheltenham
+Henrik
+reinstated
+intends
+##istan
+unite
+##ector
+playful
+sparks
+mapped
+Cadet
+luggage
+prosperous
+##ein
+salon
+##utes
+Biological
+##rland
+Tyrone
+buyer
+##lose
+amounted
+Saw
+smirked
+Ronan
+Reviews
+Adele
+trait
+##proof
+Bhutan
+Ginger
+##junct
+digitally
+stirring
+##isted
+coconut
+Hamlet
+Dinner
+Scale
+pledge
+##RP
+Wrong
+Goal
+Panel
+therapeutic
+elevations
+infectious
+priesthood
+##inda
+Guyana
+diagnostic
+##mbre
+Blackwell
+sails
+##arm
+literal
+periodically
+gleaming
+Robot
+Rector
+##abulous
+##tres
+Reaching
+Romantic
+CP
+Wonderful
+##tur
+ornamental
+##nges
+traitor
+##zilla
+genetics
+mentioning
+##eim
+resonance
+Areas
+Shopping
+##nard
+Gail
+Solid
+##rito
+##mara
+Willem
+Chip
+Matches
+Volkswagen
+obstacle
+Organ
+invites
+Coral
+attain
+##anus
+##dates
+Midway
+shuffled
+Cecilia
+dessert
+Gateway
+Ch
+Napoleonic
+Petroleum
+jets
+goose
+striped
+bowls
+vibration
+Sims
+nickel
+Thirteen
+problematic
+intervene
+##grading
+##unds
+Mum
+semifinal
+Radical
+##izations
+refurbished
+##sation
+##harine
+Maximilian
+cites
+Advocate
+Potomac
+surged
+preserves
+Curry
+angled
+ordination
+##pad
+Cade
+##DE
+##sko
+researched
+torpedoes
+Resident
+wetlands
+hay
+applicants
+depart
+Bernstein
+##pic
+##ario
+##rae
+favourable
+##wari
+##р
+metabolism
+nobleman
+Defaulted
+calculate
+ignition
+Celebrity
+Belize
+sulfur
+Flat
+Sc
+USB
+flicker
+Hertfordshire
+Sept
+CFL
+Pasadena
+Saturdays
+Titus
+##nir
+Canary
+Computing
+Isaiah
+##mler
+formidable
+pulp
+orchid
+Called
+Solutions
+kilograms
+steamer
+##hil
+Doncaster
+successors
+Stokes
+Holstein
+##sius
+sperm
+API
+Rogue
+instability
+Acoustic
+##rag
+159
+undercover
+Wouldn
+##pra
+##medical
+Eliminated
+honorable
+##chel
+denomination
+abrupt
+Buffy
+blouse
+fi
+Regardless
+Subsequent
+##rdes
+Lover
+##tford
+bacon
+##emia
+carving
+##cripts
+Massacre
+Ramos
+Latter
+##ulp
+ballroom
+##gement
+richest
+bruises
+Rest
+Wiley
+##aster
+explosions
+##lastic
+Edo
+##LD
+Mir
+choking
+disgusted
+faintly
+Barracks
+blasted
+headlights
+Tours
+ensued
+presentations
+##cale
+wrought
+##oat
+##coa
+Quaker
+##sdale
+recipe
+##gny
+corpses
+##liance
+comfortably
+##wat
+Landscape
+niche
+catalyst
+##leader
+Securities
+messy
+##RL
+Rodrigo
+backdrop
+##opping
+treats
+Emilio
+Anand
+bilateral
+meadow
+VC
+socialism
+##grad
+clinics
+##itating
+##ppe
+##ymphonic
+seniors
+Advisor
+Armoured
+Method
+Alley
+##orio
+Sad
+fueled
+raided
+Axel
+NH
+rushes
+Dixie
+Otis
+wrecked
+##22
+capitalism
+café
+##bbe
+##pion
+##forcing
+Aubrey
+Lublin
+Whenever
+Sears
+Scheme
+##lana
+Meadows
+treatise
+##RI
+##ustic
+sacrifices
+sustainability
+Biography
+mystical
+Wanted
+multiplayer
+Applications
+disliked
+##tisfied
+impaired
+empirical
+forgetting
+Fairfield
+Sunni
+blurred
+Growing
+Avalon
+coil
+Camera
+Skin
+bruised
+terminals
+##fted
+##roving
+Commando
+##hya
+##sper
+reservations
+needles
+dangling
+##rsch
+##rsten
+##spect
+##mbs
+yoga
+regretted
+Bliss
+Orion
+Rufus
+glucose
+Olsen
+autobiographical
+##dened
+222
+humidity
+Shan
+##ifiable
+supper
+##rou
+flare
+##MO
+campaigning
+descend
+socio
+declares
+Mounted
+Gracie
+Arte
+endurance
+##ety
+Copper
+costa
+airplay
+##MB
+Proceedings
+dislike
+grimaced
+occupants
+births
+glacial
+oblivious
+cans
+installment
+muddy
+##ł
+captains
+pneumonia
+Quiet
+Sloan
+Excuse
+##nine
+Geography
+gymnastics
+multimedia
+drains
+Anthology
+Gear
+cylindrical
+Fry
+undertaking
+##pler
+##tility
+Nan
+##recht
+Dub
+philosophers
+piss
+Atari
+##pha
+Galicia
+México
+##nking
+Continuing
+bump
+graveyard
+persisted
+Shrine
+##erapy
+defects
+Advance
+Bomber
+##oil
+##ffling
+cheerful
+##lix
+scrub
+##eto
+awkwardly
+collaborator
+fencing
+##alo
+prophet
+Croix
+coughed
+##lication
+roadway
+slaughter
+elephants
+##erated
+Simpsons
+vulnerability
+ivory
+Birth
+lizard
+scarce
+cylinders
+fortunes
+##NL
+Hate
+Priory
+##lai
+McBride
+##copy
+Lenny
+liaison
+Triangle
+coronation
+sampled
+savage
+amidst
+Grady
+whatsoever
+instinctively
+Reconstruction
+insides
+seizure
+Drawing
+##rlin
+Antioch
+Gao
+Díaz
+1760
+Sparks
+##tien
+##bidae
+rehearsal
+##bbs
+botanical
+##hers
+compensate
+wholesale
+Seville
+shareholder
+prediction
+astronomical
+Reddy
+hardest
+circling
+whereabouts
+termination
+Rep
+Assistance
+Dramatic
+Herb
+##ghter
+climbs
+188
+Poole
+301
+##pable
+wit
+##istice
+Walters
+relying
+Jakob
+##redo
+proceeding
+Langley
+affiliates
+ou
+##allo
+##holm
+Samsung
+##ishi
+Missing
+Xi
+vertices
+Claus
+foam
+restless
+##uating
+##sso
+##ttering
+Philips
+delta
+bombed
+Catalogue
+coaster
+Ling
+Willard
+satire
+410
+Composition
+Net
+Orioles
+##ldon
+fins
+Palatinate
+Woodward
+tease
+tilt
+brightness
+##70
+##bbling
+##loss
+##dhi
+##uilt
+Whoever
+##yers
+hitter
+Elton
+Extension
+ace
+Affair
+restructuring
+##loping
+Paterson
+hi
+##rya
+spouse
+Shay
+Himself
+piles
+preaching
+##gical
+bikes
+Brave
+expulsion
+Mirza
+stride
+Trees
+commemorated
+famine
+masonry
+Selena
+Watt
+Banking
+Rancho
+Stockton
+dip
+tattoos
+Vlad
+acquainted
+Flyers
+ruthless
+fourteenth
+illustrate
+##akes
+EPA
+##rows
+##uiz
+bumped
+Designed
+Leaders
+mastered
+Manfred
+swirled
+McCain
+##rout
+Artemis
+rabbi
+flinched
+upgrades
+penetrate
+shipyard
+transforming
+caretaker
+##eiro
+Maureen
+tightening
+##founded
+RAM
+##icular
+##mper
+##rung
+Fifteen
+exploited
+consistency
+interstate
+##ynn
+Bridget
+contamination
+Mistress
+##rup
+coating
+##FP
+##jective
+Libyan
+211
+Gemma
+dependence
+shrubs
+##ggled
+Germain
+retaliation
+traction
+##PP
+Dangerous
+terminology
+psychiatrist
+##garten
+hurdles
+Natal
+wasting
+Weir
+revolves
+stripe
+##reased
+preferences
+##entation
+##lde
+##áil
+##otherapy
+Flame
+##ologies
+viruses
+Label
+Pandora
+veil
+##ogical
+Coliseum
+Cottage
+creeping
+Jong
+lectured
+##çaise
+shoreline
+##fference
+##hra
+Shade
+Clock
+Faye
+bilingual
+Humboldt
+Operating
+##fter
+##was
+algae
+towed
+amphibious
+Parma
+impacted
+smacked
+Piedmont
+Monsters
+##omb
+Moor
+##lberg
+sinister
+Postal
+178
+Drummond
+Sign
+textbooks
+hazardous
+Brass
+Rosemary
+Pick
+Sit
+Architect
+transverse
+Centennial
+confess
+polling
+##aia
+Julien
+##mand
+consolidation
+Ethel
+##ulse
+severity
+Yorker
+choreographer
+1840s
+##ltry
+softer
+versa
+##geny
+##quila
+##jō
+Caledonia
+Friendship
+Visa
+rogue
+##zzle
+bait
+feather
+incidence
+Foods
+Ships
+##uto
+##stead
+arousal
+##rote
+Hazel
+##bolic
+Swing
+##ej
+##cule
+##jana
+##metry
+##uity
+Valuable
+##ₙ
+Shropshire
+##nect
+365
+Ones
+realise
+Café
+Albuquerque
+##grown
+##stadt
+209
+##ᵢ
+prefers
+withstand
+Lillian
+MacArthur
+Hara
+##fulness
+domination
+##VO
+##school
+Freddy
+ethnicity
+##while
+adorned
+hormone
+Calder
+Domestic
+Freud
+Shields
+##phus
+##rgan
+BP
+Segunda
+Mustang
+##GI
+Bonn
+patiently
+remarried
+##umbria
+Crete
+Elephant
+Nuremberg
+tolerate
+Tyson
+##evich
+Programming
+##lander
+Bethlehem
+segregation
+Constituency
+quarterly
+blushed
+photographers
+Sheldon
+porcelain
+Blanche
+goddamn
+lively
+##fused
+bumps
+##eli
+curated
+coherent
+provoked
+##vet
+Madeleine
+##isco
+rainy
+Bethel
+accusation
+ponytail
+gag
+##lington
+quicker
+scroll
+##vate
+Bow
+Gender
+Ira
+crashes
+ACT
+Maintenance
+##aton
+##ieu
+bitterly
+strains
+rattled
+vectors
+##arina
+##ishly
+173
+parole
+##nx
+amusing
+Gonzalez
+##erative
+Caucus
+sensual
+Penelope
+coefficient
+Mateo
+##mani
+proposition
+Duty
+lacrosse
+proportions
+Plato
+profiles
+Botswana
+Brandt
+reins
+mandolin
+encompassing
+##gens
+Kahn
+prop
+summon
+##MR
+##yrian
+##zaki
+Falling
+conditional
+thy
+##bao
+##ych
+radioactive
+##nics
+Newspaper
+##people
+##nded
+Gaming
+sunny
+##look
+Sherwood
+crafted
+NJ
+awoke
+187
+timeline
+giants
+possessing
+##ycle
+Cheryl
+ng
+Ruiz
+polymer
+potassium
+Ramsay
+relocation
+##leen
+Sociology
+##bana
+Franciscan
+propulsion
+denote
+##erjee
+registers
+headline
+Tests
+emerges
+Articles
+Mint
+livery
+breakup
+kits
+Rap
+Browning
+Bunny
+##mington
+##watch
+Anastasia
+Zachary
+arranging
+biographical
+Erica
+Nippon
+##membrance
+Carmel
+##sport
+##xes
+Paddy
+##holes
+Issues
+Spears
+compliment
+##stro
+##graphs
+Castillo
+##MU
+##space
+Corporal
+##nent
+174
+Gentlemen
+##ilize
+##vage
+convinces
+Carmine
+Crash
+##hashi
+Files
+Doctors
+brownish
+sweating
+goats
+##conductor
+rendition
+##bt
+NL
+##spiration
+generates
+##cans
+obsession
+##noy
+Danger
+Diaz
+heats
+Realm
+priorities
+##phon
+1300
+initiation
+pagan
+bursts
+archipelago
+chloride
+Screenplay
+Hewitt
+Khmer
+bang
+judgement
+negotiating
+##ait
+Mabel
+densely
+Boulder
+knob
+430
+Alfredo
+##kt
+pitches
+##ées
+##ان
+Macdonald
+##llum
+imply
+##mot
+Smile
+spherical
+##tura
+Derrick
+Kelley
+Nico
+cortex
+launches
+differed
+parallels
+Navigation
+##child
+##rming
+canoe
+forestry
+reinforce
+##mote
+confirming
+tasting
+scaled
+##resh
+##eting
+Understanding
+prevailing
+Pearce
+CW
+earnest
+Gaius
+asserts
+denoted
+landmarks
+Chargers
+warns
+##flies
+Judges
+jagged
+##dain
+tails
+Historian
+Millie
+##sler
+221
+##uard
+absurd
+Dion
+##ially
+makeshift
+Specifically
+ignorance
+Eat
+##ieri
+comparisons
+forensic
+186
+Giro
+skeptical
+disciplinary
+battleship
+##45
+Libby
+520
+Odyssey
+ledge
+##post
+Eternal
+Missionary
+deficiency
+settler
+wonders
+##gai
+raging
+##cis
+Romney
+Ulrich
+annexation
+boxers
+sect
+204
+ARIA
+dei
+Hitchcock
+te
+Varsity
+##fic
+CC
+lending
+##nial
+##tag
+##rdy
+##obe
+Defensive
+##dson
+##pore
+stellar
+Lam
+Trials
+contention
+Sung
+##uminous
+Poe
+superiority
+##plicate
+325
+bitten
+conspicuous
+##olly
+Lila
+Pub
+Petit
+distorted
+ISIL
+distinctly
+##family
+Cowboy
+mutant
+##cats
+##week
+Changes
+Sinatra
+epithet
+neglect
+Innocent
+gamma
+thrill
+reggae
+##adia
+##ational
+##due
+landlord
+##leaf
+visibly
+##ì
+Darlington
+Gomez
+##iting
+scarf
+##lade
+Hinduism
+Fever
+scouts
+##roi
+convened
+##oki
+184
+Lao
+boycott
+unemployed
+##lore
+##ß
+##hammer
+Curran
+disciples
+odor
+##ygiene
+Lighthouse
+Played
+whales
+discretion
+Yves
+##ceived
+pauses
+coincide
+##nji
+dizzy
+##scopic
+routed
+Guardians
+Kellan
+carnival
+nasal
+224
+##awed
+Mitsubishi
+640
+Cast
+silky
+Projects
+joked
+Huddersfield
+Rothschild
+zu
+##olar
+Divisions
+mildly
+##eni
+##lge
+Appalachian
+Sahara
+pinch
+##roon
+wardrobe
+##dham
+##etal
+Bubba
+##lini
+##rumbling
+Communities
+Poznań
+unification
+Beau
+Kris
+SV
+Rowing
+Minh
+reconciliation
+##saki
+##sor
+taped
+##reck
+certificates
+gubernatorial
+rainbow
+##uing
+litter
+##lique
+##oted
+Butterfly
+benefited
+Images
+induce
+Balkans
+Velvet
+##90
+##xon
+Bowman
+##breaker
+penis
+##nitz
+##oint
+##otive
+crust
+##pps
+organizers
+Outdoor
+nominees
+##rika
+TX
+##ucks
+Protestants
+##imation
+appetite
+Baja
+awaited
+##points
+windshield
+##igh
+##zled
+Brody
+Buster
+stylized
+Bryce
+##sz
+Dollar
+vest
+mold
+ounce
+ok
+receivers
+##uza
+Purdue
+Harrington
+Hodges
+captures
+##ggio
+Reservation
+##ssin
+##tman
+cosmic
+straightforward
+flipping
+remixed
+##athed
+Gómez
+Lim
+motorcycles
+economies
+owning
+Dani
+##rosis
+myths
+sire
+kindly
+1768
+Bean
+graphs
+##mee
+##RO
+##geon
+puppy
+Stephenson
+notified
+##jer
+Watching
+##rama
+Sino
+urgency
+Islanders
+##mash
+Plata
+fumble
+##chev
+##stance
+##rack
+##she
+facilitated
+swings
+akin
+enduring
+payload
+##phine
+Deputies
+murals
+##tooth
+610
+Jays
+eyeing
+##quito
+transparency
+##cote
+Timor
+negatively
+##isan
+battled
+##fected
+thankful
+Rage
+hospitality
+incorrectly
+207
+entrepreneurs
+##cula
+##wley
+hedge
+##cratic
+Corpus
+Odessa
+Whereas
+##ln
+fetch
+happier
+Amherst
+bullying
+graceful
+Height
+Bartholomew
+willingness
+qualifier
+191
+Syed
+Wesleyan
+Layla
+##rrence
+Webber
+##hum
+Rat
+##cket
+##herence
+Monterey
+contaminated
+Beside
+Mustafa
+Nana
+213
+##pruce
+Reason
+##spense
+spike
+##gé
+AU
+disciple
+charcoal
+##lean
+formulated
+Diesel
+Mariners
+accreditation
+glossy
+1800s
+##ih
+Mainz
+unison
+Marianne
+shear
+overseeing
+vernacular
+bowled
+##lett
+unpopular
+##ckoned
+##monia
+Gaston
+##TI
+##oters
+Cups
+##bones
+##ports
+Museo
+minors
+1773
+Dickens
+##EL
+##NBC
+Presents
+ambitions
+axes
+Río
+Yukon
+bedside
+Ribbon
+Units
+faults
+conceal
+##lani
+prevailed
+214
+Goodwin
+Jaguar
+crumpled
+Cullen
+Wireless
+ceded
+remotely
+Bin
+mocking
+straps
+ceramics
+##avi
+##uding
+##ader
+Taft
+twenties
+##aked
+Problem
+quasi
+Lamar
+##ntes
+##avan
+Barr
+##eral
+hooks
+sa
+##ône
+194
+##ross
+Nero
+Caine
+trance
+Homeland
+benches
+Guthrie
+dismiss
+##lex
+César
+foliage
+##oot
+##alty
+Assyrian
+Ahead
+Murdoch
+dictatorship
+wraps
+##ntal
+Corridor
+Mackay
+respectable
+jewels
+understands
+##pathic
+Bryn
+##tep
+ON
+capsule
+intrigued
+Sleeping
+communists
+##chayat
+##current
+##vez
+doubling
+booklet
+##uche
+Creed
+##NU
+spies
+##sef
+adjusting
+197
+Imam
+heaved
+Tanya
+canonical
+restraint
+senators
+stainless
+##gnate
+Matter
+cache
+restrained
+conflicting
+stung
+##ool
+Sustainable
+antiquity
+193
+heavens
+inclusive
+##ador
+fluent
+303
+911
+archaeologist
+superseded
+##plex
+Tammy
+inspire
+##passing
+##lub
+Lama
+Mixing
+##activated
+##yote
+parlor
+tactic
+198
+Stefano
+prostitute
+recycling
+sorted
+banana
+Stacey
+Musée
+aristocratic
+cough
+##rting
+authorised
+gangs
+runoff
+thoughtfully
+##nish
+Fisheries
+Provence
+detector
+hum
+##zhen
+pill
+##árez
+Map
+Leaves
+Peabody
+skater
+vent
+##color
+390
+cerebral
+hostages
+mare
+Jurassic
+swell
+##isans
+Knoxville
+Naked
+Malaya
+scowl
+Cobra
+##anga
+Sexual
+##dron
+##iae
+196
+##drick
+Ravens
+Blaine
+##throp
+Ismail
+symmetric
+##lossom
+Leicestershire
+Sylvester
+glazed
+##tended
+Radar
+fused
+Families
+Blacks
+Sale
+Zion
+foothills
+microwave
+slain
+Collingwood
+##pants
+##dling
+killers
+routinely
+Janice
+hearings
+##chanted
+##ltration
+continents
+##iving
+##yster
+##shot
+##yna
+injected
+Guillaume
+##ibi
+kinda
+Confederacy
+Barnett
+disasters
+incapable
+##grating
+rhythms
+betting
+draining
+##hak
+Callie
+Glover
+##iliated
+Sherlock
+hearted
+punching
+Wolverhampton
+Leaf
+Pi
+builders
+furnished
+knighted
+Photo
+##zle
+Touring
+fumbled
+pads
+##ий
+Bartlett
+Gunner
+eerie
+Marius
+Bonus
+pots
+##hino
+##pta
+Bray
+Frey
+Ortiz
+stalls
+belongings
+Subway
+fascination
+metaphor
+Bat
+Boer
+Colchester
+sway
+##gro
+rhetoric
+##dheim
+Fool
+PMID
+admire
+##hsil
+Strand
+TNA
+##roth
+Nottinghamshire
+##mat
+##yler
+Oxfordshire
+##nacle
+##roner
+BS
+##nces
+stimulus
+transports
+Sabbath
+##postle
+Richter
+4000
+##grim
+##shima
+##lette
+deteriorated
+analogous
+##ratic
+UHF
+energies
+inspiring
+Yiddish
+Activities
+##quential
+##boe
+Melville
+##ilton
+Judd
+consonants
+labs
+smuggling
+##fari
+avid
+##uc
+truce
+undead
+##raith
+Mostly
+bracelet
+Connection
+Hussain
+awhile
+##UC
+##vention
+liable
+genetically
+##phic
+Important
+Wildcats
+daddy
+transmit
+##cas
+conserved
+Yesterday
+##lite
+Nicky
+Guys
+Wilder
+Lay
+skinned
+Communists
+Garfield
+Nearby
+organizer
+Loss
+crafts
+walkway
+Chocolate
+Sundance
+Synod
+##enham
+modify
+swayed
+Surface
+analysts
+brackets
+drone
+parachute
+smelling
+Andrés
+filthy
+frogs
+vertically
+##OK
+localities
+marries
+AHL
+35th
+##pian
+Palazzo
+cube
+dismay
+relocate
+##на
+Hear
+##digo
+##oxide
+prefecture
+converts
+hangar
+##oya
+##ucking
+Spectrum
+deepened
+spoiled
+Keeping
+##phobic
+Verona
+outrage
+Improvement
+##UI
+masterpiece
+slung
+Calling
+chant
+Haute
+mediated
+manipulated
+affirmed
+##hesis
+Hangul
+skies
+##llan
+Worcestershire
+##kos
+mosaic
+##bage
+##wned
+Putnam
+folder
+##LM
+guts
+noteworthy
+##rada
+AJ
+sculpted
+##iselle
+##rang
+recognizable
+##pent
+dolls
+lobbying
+impatiently
+Se
+staple
+Serb
+tandem
+Hiroshima
+thieves
+##ynx
+faculties
+Norte
+##alle
+##trusion
+chords
+##ylon
+Gareth
+##lops
+##escu
+FIA
+Levin
+auspices
+groin
+Hui
+nun
+Listed
+Honourable
+Larsen
+rigorous
+##erer
+Tonga
+##pment
+##rave
+##track
+##aa
+##enary
+540
+clone
+sediment
+esteem
+sighted
+cruelty
+##boa
+inverse
+violating
+Amtrak
+Status
+amalgamated
+vertex
+AR
+harmless
+Amir
+mounts
+Coronation
+counseling
+Audi
+CO₂
+splits
+##eyer
+Humans
+Salmon
+##have
+##rado
+##čić
+216
+takeoff
+classmates
+psychedelic
+##gni
+Gypsy
+231
+Anger
+GAA
+ME
+##nist
+##tals
+Lissa
+Odd
+baptized
+Fiat
+fringe
+##hren
+179
+elevators
+perspectives
+##TF
+##ngle
+Question
+frontal
+950
+thicker
+Molecular
+##nological
+Sixteen
+Baton
+Hearing
+commemorative
+dorm
+Architectural
+purity
+##erse
+risky
+Georgie
+relaxing
+##ugs
+downed
+##rar
+Slim
+##phy
+IUCN
+##thorpe
+Parkinson
+217
+Marley
+Shipping
+sweaty
+Jesuits
+Sindh
+Janata
+implying
+Armenians
+intercept
+Ankara
+commissioners
+ascended
+sniper
+Grass
+Walls
+salvage
+Dewey
+generalized
+learnt
+PT
+##fighter
+##tech
+DR
+##itrus
+##zza
+mercenaries
+slots
+##burst
+##finger
+##nsky
+Princes
+Rhodesia
+##munication
+##strom
+Fremantle
+homework
+ins
+##Os
+##hao
+##uffed
+Thorpe
+Xiao
+exquisite
+firstly
+liberated
+technician
+Oilers
+Phyllis
+herb
+sharks
+MBE
+##stock
+Product
+banjo
+##morandum
+##than
+Visitors
+unavailable
+unpublished
+oxidation
+Vogue
+##copic
+##etics
+Yates
+##ppard
+Leiden
+Trading
+cottages
+Principles
+##Millan
+##wife
+##hiva
+Vicar
+nouns
+strolled
+##eorological
+##eton
+##science
+precedent
+Armand
+Guido
+rewards
+##ilis
+##tise
+clipped
+chick
+##endra
+averages
+tentatively
+1830s
+##vos
+Certainly
+305
+Société
+Commandant
+##crats
+##dified
+##nka
+marsh
+angered
+ventilation
+Hutton
+Ritchie
+##having
+Eclipse
+flick
+motionless
+Amor
+Fest
+Loire
+lays
+##icit
+##sband
+Guggenheim
+Luck
+disrupted
+##ncia
+Disco
+##vigator
+criticisms
+grins
+##lons
+##vial
+##ody
+salute
+Coaches
+junk
+saxophonist
+##eology
+Uprising
+Diet
+##marks
+chronicles
+robbed
+##iet
+##ahi
+Bohemian
+magician
+wavelength
+Kenyan
+augmented
+fashionable
+##ogies
+Luce
+F1
+Monmouth
+##jos
+##loop
+enjoyment
+exemption
+Centers
+##visor
+Soundtrack
+blinding
+practitioner
+solidarity
+sacrificed
+##oso
+##cture
+##riated
+blended
+Abd
+Copyright
+##nob
+34th
+##reak
+Claudio
+hectare
+rotor
+testify
+##ends
+##iably
+##sume
+landowner
+##cess
+##ckman
+Eduard
+Silesian
+backseat
+mutually
+##abe
+Mallory
+bounds
+Collective
+Poet
+Winkler
+pertaining
+scraped
+Phelps
+crane
+flickering
+Proto
+bubbles
+popularized
+removes
+##86
+Cadillac
+Warfare
+audible
+rites
+shivering
+##sist
+##nst
+##biotic
+Mon
+fascist
+Bali
+Kathryn
+ambiguous
+furiously
+morale
+patio
+Sang
+inconsistent
+topology
+Greens
+monkeys
+Köppen
+189
+Toy
+vow
+##ías
+bombings
+##culus
+improvised
+lodged
+subsidiaries
+garment
+startling
+practised
+Hume
+Thorn
+categorized
+Till
+Eileen
+wedge
+##64
+Federico
+patriotic
+unlock
+##oshi
+badminton
+Compared
+Vilnius
+##KE
+Crimean
+Kemp
+decks
+spaced
+resolutions
+sighs
+##mind
+Imagine
+Cartoon
+huddled
+policemen
+forwards
+##rouch
+equals
+##nter
+inspected
+Charley
+MG
+##rte
+pamphlet
+Arturo
+dans
+scarcely
+##ulton
+##rvin
+parental
+unconstitutional
+watts
+Susannah
+Dare
+##sitive
+Rowland
+Valle
+invalid
+##ué
+Detachment
+acronym
+Yokohama
+verified
+##lsson
+groove
+Liza
+clarified
+compromised
+265
+##rgon
+##orf
+hesitant
+Fruit
+Application
+Mathias
+icons
+##cell
+Qin
+interventions
+##uron
+punt
+remnant
+##rien
+Ames
+manifold
+spines
+floral
+##zable
+comrades
+Fallen
+orbits
+Annals
+hobby
+Auditorium
+implicated
+researching
+Pueblo
+Ta
+terminate
+##pella
+Rings
+approximation
+fuzzy
+##ús
+thriving
+##ket
+Conor
+alarmed
+etched
+Cary
+##rdon
+Ally
+##rington
+Pay
+mint
+##hasa
+##unity
+##dman
+##itate
+Oceania
+furrowed
+trams
+##aq
+Wentworth
+ventured
+choreography
+prototypes
+Patel
+mouthed
+trenches
+##licing
+##yya
+Lies
+deception
+##erve
+##vations
+Bertrand
+earthquakes
+##tography
+Southwestern
+##aja
+token
+Gupta
+##yō
+Beckett
+initials
+ironic
+Tsar
+subdued
+shootout
+sobbing
+liar
+Scandinavia
+Souls
+ch
+therapist
+trader
+Regulation
+Kali
+busiest
+##pation
+32nd
+Telephone
+Vargas
+##moky
+##nose
+##uge
+Favorite
+abducted
+bonding
+219
+255
+correction
+mat
+drown
+fl
+unbeaten
+Pocket
+Summers
+Quite
+rods
+Percussion
+##ndy
+buzzing
+cadet
+Wilkes
+attire
+directory
+utilities
+naive
+populous
+Hendrix
+##actor
+disadvantage
+1400
+Landon
+Underworld
+##ense
+Occasionally
+mercury
+Davey
+Morley
+spa
+wrestled
+##vender
+eclipse
+Sienna
+supplemented
+thou
+Stream
+liturgical
+##gall
+##berries
+##piration
+1769
+Bucks
+abandoning
+##jutant
+##nac
+232
+venom
+##31
+Roche
+dotted
+Currie
+Córdoba
+Milo
+Sharif
+divides
+justification
+prejudice
+fortunate
+##vide
+##ābād
+Rowe
+inflammatory
+##eld
+avenue
+Sources
+##rimal
+Messenger
+Blanco
+advocating
+formulation
+##pute
+emphasizes
+nut
+Armored
+##ented
+nutrients
+##tment
+insistence
+Martins
+landowners
+##RB
+comparatively
+headlines
+snaps
+##qing
+Celebration
+##mad
+republican
+##NE
+Trace
+##500
+1771
+proclamation
+NRL
+Rubin
+Buzz
+Weimar
+##AG
+199
+posthumous
+##ental
+##deacon
+Distance
+intensely
+overheard
+Arcade
+diagonal
+hazard
+Giving
+weekdays
+##ù
+Verdi
+actresses
+##hare
+Pulling
+##erries
+##pores
+catering
+shortest
+##ctors
+##cure
+##restle
+##reta
+##runch
+##brecht
+##uddin
+Moments
+senate
+Feng
+Prescott
+##thest
+218
+divisional
+Bertie
+sparse
+surrounds
+coupling
+gravitational
+werewolves
+##lax
+Rankings
+##mated
+##tries
+Shia
+##mart
+##23
+##vocative
+interfaces
+morphology
+newscast
+##bide
+inputs
+solicitor
+Olaf
+cabinets
+puzzles
+##tains
+Unified
+##firmed
+WA
+solemn
+##opy
+Tito
+Jaenelle
+Neolithic
+horseback
+##ires
+pharmacy
+prevalence
+##lint
+Swami
+##bush
+##tudes
+Philipp
+mythical
+divers
+Scouting
+aperture
+progressively
+##bay
+##nio
+bounce
+Floor
+##elf
+Lucan
+adulthood
+helm
+Bluff
+Passage
+Salvation
+lemon
+napkin
+scheduling
+##gets
+Elements
+Mina
+Novak
+stalled
+##llister
+Infrastructure
+##nky
+##tania
+##uished
+Katz
+Norma
+sucks
+trusting
+1765
+boilers
+Accordingly
+##hered
+223
+Crowley
+##fight
+##ulo
+Henrietta
+##hani
+pounder
+surprises
+##chor
+##glia
+Dukes
+##cracy
+##zier
+##fs
+Patriot
+silicon
+##VP
+simulcast
+telegraph
+Mysore
+cardboard
+Len
+##QL
+Auguste
+accordion
+analytical
+specify
+ineffective
+hunched
+abnormal
+Transylvania
+##dn
+##tending
+Emilia
+glittering
+Maddy
+##wana
+1762
+External
+Lecture
+endorsement
+Hernández
+Anaheim
+Ware
+offences
+##phorus
+Plantation
+popping
+Bonaparte
+disgusting
+neared
+##notes
+Identity
+heroin
+nicely
+##raverse
+apron
+congestion
+##PR
+padded
+##fts
+invaders
+##came
+freshly
+Halle
+endowed
+fracture
+ROM
+##max
+sediments
+diffusion
+dryly
+##tara
+Tam
+Draw
+Spin
+Talon
+Anthropology
+##lify
+nausea
+##shirt
+insert
+Fresno
+capitalist
+indefinitely
+apples
+Gift
+scooped
+60s
+Cooperative
+mistakenly
+##lover
+murmur
+##iger
+Equipment
+abusive
+orphanage
+##9th
+##lterweight
+##unda
+Baird
+ant
+saloon
+33rd
+Chesapeake
+##chair
+##sound
+##tend
+chaotic
+pornography
+brace
+##aret
+heiress
+SSR
+resentment
+Arbor
+headmaster
+##uren
+unlimited
+##with
+##jn
+Bram
+Ely
+Pokémon
+pivotal
+##guous
+Database
+Marta
+Shine
+stumbling
+##ovsky
+##skin
+Henley
+Polk
+functioned
+##layer
+##pas
+##udd
+##MX
+blackness
+cadets
+feral
+Damian
+##actions
+2D
+##yla
+Apocalypse
+##aic
+inactivated
+##china
+##kovic
+##bres
+destroys
+nap
+Macy
+sums
+Madhya
+Wisdom
+rejects
+##amel
+60th
+Cho
+bandwidth
+##sons
+##obbing
+##orama
+Mutual
+shafts
+##estone
+##rsen
+accord
+replaces
+waterfront
+##gonal
+##rida
+convictions
+##ays
+calmed
+suppliers
+Cummings
+GMA
+fearful
+Scientist
+Sinai
+examines
+experimented
+Netflix
+Enforcement
+Scarlett
+##lasia
+Healthcare
+##onte
+Dude
+inverted
+##36
+##regation
+##lidae
+Munro
+##angay
+Airbus
+overlapping
+Drivers
+lawsuits
+bodily
+##udder
+Wanda
+Effects
+Fathers
+##finery
+##islav
+Ridley
+observatory
+pod
+##utrition
+Electricity
+landslide
+##mable
+##zoic
+##imator
+##uration
+Estates
+sleepy
+Nickelodeon
+steaming
+irony
+schedules
+snack
+spikes
+Hmm
+##nesia
+##bella
+##hibit
+Greenville
+plucked
+Harald
+##ono
+Gamma
+infringement
+roaring
+deposition
+##pol
+##orum
+660
+seminal
+passports
+engagements
+Akbar
+rotated
+##bina
+##gart
+Hartley
+##lown
+##truct
+uttered
+traumatic
+Dex
+##ôme
+Holloway
+MV
+apartheid
+##nee
+Counter
+Colton
+OR
+245
+Spaniards
+Regency
+Schedule
+scratching
+squads
+verify
+##alk
+keyboardist
+rotten
+Forestry
+aids
+commemorating
+##yed
+##érie
+Sting
+##elly
+Dai
+##fers
+##berley
+##ducted
+Melvin
+cannabis
+glider
+##enbach
+##rban
+Costello
+Skating
+cartoonist
+AN
+audit
+##pectator
+distributing
+226
+312
+interpreter
+header
+Alternatively
+##ases
+smug
+##kumar
+cabins
+remastered
+Connolly
+Kelsey
+LED
+tentative
+Check
+Sichuan
+shaved
+##42
+Gerhard
+Harvest
+inward
+##rque
+Hopefully
+hem
+##34
+Typical
+binds
+wrath
+Woodstock
+forcibly
+Fergus
+##charged
+##tured
+prepares
+amenities
+penetration
+##ghan
+coarse
+##oned
+enthusiasts
+##av
+##twined
+fielded
+##cky
+Kiel
+##obia
+470
+beers
+tremble
+youths
+attendees
+##cademies
+##sex
+Macon
+communism
+dir
+##abi
+Lennox
+Wen
+differentiate
+jewel
+##SO
+activate
+assert
+laden
+unto
+Gillespie
+Guillermo
+accumulation
+##GM
+NGO
+Rosenberg
+calculating
+drastically
+##omorphic
+peeled
+Liège
+insurgents
+outdoors
+##enia
+Aspen
+Sep
+awakened
+##eye
+Consul
+Maiden
+insanity
+##brian
+furnace
+Colours
+distributions
+longitudinal
+syllables
+##scent
+Martian
+accountant
+Atkins
+husbands
+sewage
+zur
+collaborate
+highlighting
+##rites
+##PI
+colonization
+nearer
+##XT
+dunes
+positioning
+Ku
+multitude
+luxurious
+Volvo
+linguistics
+plotting
+squared
+##inder
+outstretched
+##uds
+Fuji
+ji
+##feit
+##ahu
+##loat
+##gado
+##luster
+##oku
+América
+##iza
+Residents
+vine
+Pieces
+DD
+Vampires
+##ová
+smoked
+harshly
+spreads
+##turn
+##zhi
+betray
+electors
+##settled
+Considering
+exploits
+stamped
+Dusty
+enraged
+Nairobi
+##38
+intervened
+##luck
+orchestras
+##lda
+Hereford
+Jarvis
+calf
+##itzer
+##CH
+salesman
+Lovers
+cigar
+Angelica
+doomed
+heroine
+##tible
+Sanford
+offenders
+##ulously
+articulated
+##oam
+Emanuel
+Gardiner
+Edna
+Shu
+gigantic
+##stable
+Tallinn
+coasts
+Maker
+ale
+stalking
+##oga
+##smus
+lucrative
+southbound
+##changing
+Reg
+##lants
+Schleswig
+discount
+grouping
+physiological
+##OH
+##sun
+Galen
+assurance
+reconcile
+rib
+scarlet
+Thatcher
+anarchist
+##oom
+Turnpike
+##ceding
+cocktail
+Sweeney
+Allegheny
+concessions
+oppression
+reassuring
+##poli
+##ticus
+##TR
+##VI
+##uca
+##zione
+directional
+strikeouts
+Beneath
+Couldn
+Kabul
+##national
+hydroelectric
+##jit
+Desire
+##riot
+enhancing
+northbound
+##PO
+Ok
+Routledge
+volatile
+Bernardo
+Python
+333
+ample
+chestnut
+automobiles
+##innamon
+##care
+##hering
+BWF
+salaries
+Turbo
+acquisitions
+##stituting
+strengths
+pilgrims
+Ponce
+Pig
+Actors
+Beard
+sanitation
+##RD
+##mett
+Telecommunications
+worms
+##idas
+Juno
+Larson
+Ventura
+Northeastern
+weighs
+Houghton
+collaborating
+lottery
+##rano
+Wonderland
+gigs
+##lmer
+##zano
+##edd
+##nife
+mixtape
+predominant
+tripped
+##ruly
+Alexei
+investing
+Belgarath
+Brasil
+hiss
+##crat
+##xham
+Côte
+560
+kilometer
+##cological
+analyzing
+##As
+engined
+listener
+##cakes
+negotiation
+##hisky
+Santana
+##lemma
+IAAF
+Seneca
+skeletal
+Covenant
+Steiner
+##lev
+##uen
+Neptune
+retention
+##upon
+Closing
+Czechoslovak
+chalk
+Navarre
+NZ
+##IG
+##hop
+##oly
+##quatorial
+##sad
+Brewery
+Conflict
+Them
+renew
+turrets
+disagree
+Petra
+Slave
+##reole
+adjustment
+##dela
+##regard
+##sner
+framing
+stature
+##rca
+##sies
+##46
+##mata
+Logic
+inadvertently
+naturalist
+spheres
+towering
+heightened
+Dodd
+rink
+##fle
+Keyboards
+bulb
+diver
+ul
+##tsk
+Exodus
+Deacon
+España
+Canadiens
+oblique
+thud
+reigned
+rug
+Whitman
+Dash
+##iens
+Haifa
+pets
+##arland
+manually
+dart
+##bial
+Sven
+textiles
+subgroup
+Napier
+graffiti
+revolver
+humming
+Babu
+protector
+typed
+Provinces
+Sparta
+Wills
+subjective
+##rella
+temptation
+##liest
+FL
+Sadie
+manifest
+Guangdong
+Transfer
+entertain
+eve
+recipes
+##33
+Benedictine
+retailer
+##dence
+establishes
+##cluded
+##rked
+Ursula
+##ltz
+##lars
+##rena
+qualifiers
+##curement
+colt
+depictions
+##oit
+Spiritual
+differentiation
+staffed
+transitional
+##lew
+1761
+fatalities
+##oan
+Bayern
+Northamptonshire
+Weeks
+##CU
+Fife
+capacities
+hoarse
+##latt
+##ة
+evidenced
+##HD
+##ographer
+assessing
+evolve
+hints
+42nd
+streaked
+##lve
+Yahoo
+##estive
+##rned
+##zas
+baggage
+Elected
+secrecy
+##champ
+Character
+Pen
+Decca
+cape
+Bernardino
+vapor
+Dolly
+counselor
+##isers
+Benin
+##khar
+##CR
+notch
+##thus
+##racy
+bounty
+lend
+grassland
+##chtenstein
+##dating
+pseudo
+golfer
+simplest
+##ceive
+Lucivar
+Triumph
+dinosaur
+dinosaurs
+##šić
+Seahawks
+##nco
+resorts
+reelected
+1766
+reproduce
+universally
+##OA
+ER
+tendencies
+Consolidated
+Massey
+Tasmanian
+reckless
+##icz
+##ricks
+1755
+questionable
+Audience
+##lates
+preseason
+Quran
+trivial
+Haitian
+Freeway
+dialed
+Appointed
+Heard
+ecosystems
+##bula
+hormones
+Carbon
+Rd
+##arney
+##working
+Christoph
+presiding
+pu
+##athy
+Morrow
+Dar
+ensures
+posing
+remedy
+EA
+disclosed
+##hui
+##rten
+rumours
+surveying
+##ficiency
+Aziz
+Jewel
+Plays
+##smatic
+Bernhard
+Christi
+##eanut
+##friend
+jailed
+##dr
+govern
+neighbour
+butler
+Acheron
+murdering
+oils
+mac
+Editorial
+detectives
+bolts
+##ulon
+Guitars
+malaria
+36th
+Pembroke
+Opened
+##hium
+harmonic
+serum
+##sio
+Franks
+fingernails
+##gli
+culturally
+evolving
+scalp
+VP
+deploy
+uploaded
+mater
+##evo
+Jammu
+Spa
+##icker
+flirting
+##cursions
+Heidi
+Majority
+sprawled
+##alytic
+Zheng
+bunker
+##lena
+ST
+##tile
+Jiang
+ceilings
+##ently
+##ols
+Recovery
+dire
+##good
+Manson
+Honestly
+Montréal
+1764
+227
+quota
+Lakshmi
+incentive
+Accounting
+##cilla
+Eureka
+Reaper
+buzzed
+##uh
+courtroom
+dub
+##mberg
+KC
+Gong
+Theodor
+Académie
+NPR
+criticizing
+protesting
+##pired
+##yric
+abuses
+fisheries
+##minated
+1767
+yd
+Gemini
+Subcommittee
+##fuse
+Duff
+Wasn
+Wight
+cleaner
+##tite
+planetary
+Survivor
+Zionist
+mounds
+##rary
+landfall
+disruption
+yielding
+##yana
+bids
+unidentified
+Garry
+Ellison
+Elmer
+Fishing
+Hayward
+demos
+modelling
+##anche
+##stick
+caressed
+entertained
+##hesion
+piers
+Crimea
+##mass
+WHO
+boulder
+trunks
+1640
+Biennale
+Palestinians
+Pursuit
+##udes
+Dora
+contender
+##dridge
+Nanjing
+##ezer
+##former
+##ibel
+Whole
+proliferation
+##tide
+##weiler
+fuels
+predictions
+##ente
+##onium
+Filming
+absorbing
+Ramón
+strangled
+conveyed
+inhabit
+prostitutes
+recession
+bonded
+clinched
+##eak
+##iji
+##edar
+Pleasure
+Rite
+Christy
+Therapy
+sarcasm
+##collegiate
+hilt
+probation
+Sarawak
+coefficients
+underworld
+biodiversity
+SBS
+groom
+brewing
+dungeon
+##claiming
+Hari
+turnover
+##ntina
+##omer
+##opped
+orthodox
+styling
+##tars
+##ulata
+priced
+Marjorie
+##eley
+##abar
+Yong
+##tically
+Crambidae
+Hernandez
+##ego
+##rricular
+##ark
+##lamour
+##llin
+##augh
+##tens
+Advancement
+Loyola
+##4th
+##hh
+goin
+marshes
+Sardinia
+##ša
+Ljubljana
+Singing
+suspiciously
+##hesive
+Félix
+Regarding
+flap
+stimulation
+##raught
+Apr
+Yin
+gaping
+tighten
+skier
+##itas
+##lad
+##rani
+264
+Ashes
+Olson
+Problems
+Tabitha
+##rading
+balancing
+sunrise
+##ease
+##iture
+##ritic
+Fringe
+##iciency
+Inspired
+Linnaeus
+PBA
+disapproval
+##kles
+##rka
+##tails
+##urger
+Disaster
+Laboratories
+apps
+paradise
+Aero
+Came
+sneaking
+Gee
+Beacon
+ODI
+commodity
+Ellington
+graphical
+Gretchen
+spire
+##skaya
+##trine
+RTÉ
+efficacy
+plc
+tribunal
+##ytic
+downhill
+flu
+medications
+##kaya
+widen
+Sunrise
+##nous
+distinguishing
+pawn
+##BO
+##irn
+##ssing
+##ν
+Easton
+##vila
+Rhineland
+##aque
+defect
+##saurus
+Goose
+Ju
+##classified
+Middlesbrough
+shaping
+preached
+1759
+##erland
+Ein
+Hailey
+musicals
+##altered
+Galileo
+Hilda
+Fighters
+Lac
+##ometric
+295
+Leafs
+Milano
+##lta
+##VD
+##ivist
+penetrated
+Mask
+Orchard
+plaintiff
+##icorn
+Yvonne
+##fred
+outfielder
+peek
+Collier
+Caracas
+repealed
+Bois
+dell
+restrict
+Dolores
+Hadley
+peacefully
+##LL
+condom
+Granny
+Orders
+sabotage
+##toon
+##rings
+compass
+marshal
+gears
+brigadier
+dye
+Yunnan
+communicating
+donate
+emerald
+vitamin
+administer
+Fulham
+##classical
+##llas
+Buckinghamshire
+Held
+layered
+disclosure
+Akira
+programmer
+shrimp
+Crusade
+##ximal
+Luzon
+bakery
+##cute
+Garth
+Citadel
+uniquely
+Curling
+info
+mum
+Para
+##ști
+sleek
+##ione
+hey
+Lantern
+mesh
+##lacing
+##lizzard
+##gade
+prosecuted
+Alba
+Gilles
+greedy
+twists
+##ogged
+Viper
+##kata
+Appearances
+Skyla
+hymns
+##pelled
+curving
+predictable
+Grave
+Watford
+##dford
+##liptic
+##vary
+Westwood
+fluids
+Models
+statutes
+##ynamite
+1740
+##culate
+Framework
+Johanna
+##gression
+Vuelta
+imp
+##otion
+##raga
+##thouse
+Ciudad
+festivities
+##love
+Beyoncé
+italics
+##vance
+DB
+##haman
+outs
+Singers
+##ueva
+##urning
+##51
+##ntiary
+##mobile
+285
+Mimi
+emeritus
+nesting
+Keeper
+Ways
+##onal
+##oux
+Edmond
+MMA
+##bark
+##oop
+Hampson
+##ñez
+##rets
+Gladstone
+wreckage
+Pont
+Playboy
+reluctance
+##ná
+apprenticeship
+preferring
+Value
+originate
+##wei
+##olio
+Alexia
+##rog
+Parachute
+jammed
+stud
+Eton
+vols
+##ganized
+1745
+straining
+creep
+indicators
+##mán
+humiliation
+hinted
+alma
+tanker
+##egation
+Haynes
+Penang
+amazement
+branched
+rumble
+##ddington
+archaeologists
+paranoid
+expenditure
+Absolutely
+Musicians
+banished
+##fining
+baptism
+Joker
+Persons
+hemisphere
+##tieth
+##ück
+flock
+##xing
+lbs
+Kung
+crab
+##dak
+##tinent
+Regulations
+barrage
+parcel
+##ós
+Tanaka
+##rsa
+Natalia
+Voyage
+flaws
+stepfather
+##aven
+##eological
+Botanical
+Minsk
+##ckers
+Cinderella
+Feast
+Loving
+Previous
+Shark
+##took
+barrister
+collaborators
+##nnes
+Croydon
+Graeme
+Juniors
+##7th
+##formation
+##ulos
+##ák
+£2
+##hwa
+##rove
+##ș
+Whig
+demeanor
+Otago
+##TH
+##ooster
+Faber
+instructors
+##ahl
+##bha
+emptied
+##schen
+saga
+##lora
+exploding
+##rges
+Crusaders
+##caster
+##uations
+streaks
+CBN
+bows
+insights
+ka
+1650
+diversion
+LSU
+Wingspan
+##liva
+Response
+sanity
+Producers
+imitation
+##fine
+Lange
+Spokane
+splash
+weed
+Siberian
+magnet
+##rocodile
+capitals
+##rgus
+swelled
+Rani
+Bells
+Silesia
+arithmetic
+rumor
+##hampton
+favors
+Weird
+marketplace
+##orm
+tsunami
+unpredictable
+##citation
+##ferno
+Tradition
+postwar
+stench
+succeeds
+##roup
+Anya
+Users
+oversized
+totaling
+pouch
+##nat
+Tripoli
+leverage
+satin
+##cline
+Bathurst
+Lund
+Niall
+thereof
+##quid
+Bangor
+barge
+Animated
+##53
+##alan
+Ballard
+utilizes
+Done
+ballistic
+NDP
+gatherings
+##elin
+##vening
+Rockets
+Sabrina
+Tamara
+Tribal
+WTA
+##citing
+blinded
+flux
+Khalid
+Una
+prescription
+##jee
+Parents
+##otics
+##food
+Silicon
+cured
+electro
+perpendicular
+intimacy
+##rified
+Lots
+##ceiving
+##powder
+incentives
+McKenna
+##arma
+##ounced
+##rinkled
+Alzheimer
+##tarian
+262
+Seas
+##cam
+Novi
+##hout
+##morphic
+##hazar
+##hul
+##nington
+Huron
+Bahadur
+Pirate
+pursed
+Griffiths
+indicted
+swap
+refrain
+##mulating
+Lal
+stomped
+##Pad
+##mamoto
+Reef
+disposed
+plastered
+weeping
+##rato
+Minas
+hourly
+tumors
+##ruising
+Lyle
+##yper
+##sol
+Odisha
+credibility
+##Dowell
+Braun
+Graphic
+lurched
+muster
+##nex
+##ührer
+##connected
+##iek
+##ruba
+Carthage
+Peck
+maple
+bursting
+##lava
+Enrico
+rite
+##jak
+Moment
+##skar
+Styx
+poking
+Spartan
+##urney
+Hepburn
+Mart
+Titanic
+newsletter
+waits
+Mecklenburg
+agitated
+eats
+##dious
+Chow
+matrices
+Maud
+##sexual
+sermon
+234
+##sible
+##lung
+Qi
+cemeteries
+mined
+sprinter
+##ckett
+coward
+##gable
+##hell
+##thin
+##FB
+Contact
+##hay
+rainforest
+238
+Hemisphere
+boasts
+##nders
+##verance
+##kat
+Convent
+Dunedin
+Lecturer
+lyricist
+##bject
+Iberian
+comune
+##pphire
+chunk
+##boo
+thrusting
+fore
+informing
+pistols
+echoes
+Tier
+battleships
+substitution
+##belt
+moniker
+##charya
+##lland
+Thoroughbred
+38th
+##01
+##tah
+parting
+tongues
+Cale
+##seau
+Unionist
+modular
+celebrates
+preview
+steamed
+Bismarck
+302
+737
+vamp
+##finity
+##nbridge
+weaknesses
+husky
+##berman
+absently
+##icide
+Craven
+tailored
+Tokugawa
+VIP
+syntax
+Kazan
+captives
+doses
+filtered
+overview
+Cleopatra
+Conversely
+stallion
+Burger
+Suez
+Raoul
+th
+##reaves
+Dickson
+Nell
+Rate
+anal
+colder
+##sław
+Arm
+Semitic
+##green
+reflective
+1100
+episcopal
+journeys
+##ours
+##pository
+##dering
+residue
+Gunn
+##27
+##ntial
+##crates
+##zig
+Astros
+Renee
+Emerald
+##vili
+connectivity
+undrafted
+Sampson
+treasures
+##kura
+##theon
+##vern
+Destroyer
+##iable
+##ener
+Frederic
+briefcase
+confinement
+Bree
+##WD
+Athena
+233
+Padres
+Thom
+speeding
+##hali
+Dental
+ducks
+Putin
+##rcle
+##lou
+Asylum
+##usk
+dusk
+pasture
+Institutes
+ONE
+jack
+##named
+diplomacy
+Intercontinental
+Leagues
+Towns
+comedic
+premature
+##edic
+##mona
+##ories
+trimmed
+Charge
+Cream
+guarantees
+Dmitry
+splashed
+Philosophical
+tramway
+##cape
+Maynard
+predatory
+redundant
+##gratory
+##wry
+sobs
+Burgundy
+edible
+outfits
+Handel
+dazed
+dangerously
+idle
+Operational
+organizes
+##sional
+blackish
+broker
+weddings
+##halt
+Becca
+McGee
+##gman
+protagonists
+##pelling
+Keynes
+aux
+stumble
+##ordination
+Nokia
+reel
+sexes
+##woods
+##pheric
+##quished
+##voc
+##oir
+##pathian
+##ptus
+##sma
+##tating
+##ê
+fulfilling
+sheath
+##ayne
+Mei
+Ordinary
+Collin
+Sharpe
+grasses
+interdisciplinary
+##OX
+Background
+##ignment
+Assault
+transforms
+Hamas
+Serge
+ratios
+##sik
+swaying
+##rcia
+Rosen
+##gant
+##versible
+cinematographer
+curly
+penny
+Kamal
+Mellon
+Sailor
+Spence
+phased
+Brewers
+amassed
+Societies
+##ropriations
+##buted
+mythological
+##SN
+##byss
+##ired
+Sovereign
+preface
+Parry
+##ife
+altitudes
+crossings
+##28
+Crewe
+southernmost
+taut
+McKinley
+##owa
+##tore
+254
+##ckney
+compiling
+Shelton
+##hiko
+228
+Poll
+Shepard
+Labs
+Pace
+Carlson
+grasping
+##ов
+Delaney
+Winning
+robotic
+intentional
+shattering
+##boarding
+##git
+##grade
+Editions
+Reserves
+ignorant
+proposing
+##hanna
+cutter
+Mongols
+NW
+##eux
+Codex
+Cristina
+Daughters
+Rees
+forecast
+##hita
+NGOs
+Stations
+Beaux
+Erwin
+##jected
+##EX
+##trom
+Schumacher
+##hrill
+##rophe
+Maharaja
+Oricon
+##sul
+##dynamic
+##fighting
+Ce
+Ingrid
+rumbled
+Prospect
+stairwell
+Barnard
+applause
+complementary
+##uba
+grunt
+##mented
+Bloc
+Carleton
+loft
+noisy
+##hey
+490
+contrasted
+##inator
+##rief
+##centric
+##fica
+Cantonese
+Blanc
+Lausanne
+License
+artifact
+##ddin
+rot
+Amongst
+Prakash
+RF
+##topia
+milestone
+##vard
+Winters
+Mead
+churchyard
+Lulu
+estuary
+##ind
+Cha
+Infinity
+Meadow
+subsidies
+##valent
+CONCACAF
+Ching
+medicinal
+navigate
+Carver
+Twice
+abdominal
+regulating
+RB
+toilets
+Brewer
+weakening
+ambushed
+##aut
+##vignon
+Lansing
+unacceptable
+reliance
+stabbing
+##mpo
+##naire
+Interview
+##ested
+##imed
+bearings
+##lts
+Rashid
+##iation
+authenticity
+vigorous
+##frey
+##uel
+biologist
+NFC
+##rmaid
+##wash
+Makes
+##aunt
+##steries
+withdrawing
+##qa
+Buccaneers
+bleed
+inclination
+stain
+##ilo
+##ppel
+Torre
+privileged
+cereal
+trailers
+alumnus
+neon
+Cochrane
+Mariana
+caress
+##47
+##ients
+experimentation
+Window
+convict
+signaled
+##YP
+rower
+Pharmacy
+interacting
+241
+Strings
+dominating
+kinase
+Dinamo
+Wire
+pains
+sensations
+##suse
+Twenty20
+##39
+spotlight
+##hend
+elemental
+##pura
+Jameson
+Swindon
+honoring
+pained
+##ediatric
+##lux
+Psychological
+assemblies
+ingredient
+Martial
+Penguins
+beverage
+Monitor
+mysteries
+##ION
+emigration
+mused
+##sique
+crore
+AMC
+Funding
+Chinatown
+Establishment
+Finalist
+enjoyable
+1756
+##mada
+##rams
+NO
+newborn
+CS
+comprehend
+Invisible
+Siemens
+##acon
+246
+contraction
+##volving
+##moration
+##rok
+montane
+##ntation
+Galloway
+##llow
+Verity
+directorial
+pearl
+Leaning
+##rase
+Fernandez
+swallowing
+Automatic
+Madness
+haunting
+paddle
+##UE
+##rrows
+##vies
+##zuki
+##bolt
+##iber
+Fender
+emails
+paste
+##lancing
+hind
+homestead
+hopeless
+##dles
+Rockies
+garlic
+fatty
+shrieked
+##ismic
+Gillian
+Inquiry
+Schultz
+XML
+##cius
+##uld
+Domesday
+grenades
+northernmost
+##igi
+Tbilisi
+optimistic
+##poon
+Refuge
+stacks
+Bose
+smash
+surreal
+Nah
+Straits
+Conquest
+##roo
+##weet
+##kell
+Gladys
+CH
+##lim
+##vitation
+Doctorate
+NRHP
+knocks
+Bey
+Romano
+##pile
+242
+Diamonds
+strides
+eclectic
+Betsy
+clade
+##hady
+##leashed
+dissolve
+moss
+Suburban
+silvery
+##bria
+tally
+turtles
+##uctive
+finely
+industrialist
+##nary
+Ernesto
+oz
+pact
+loneliness
+##hov
+Tomb
+multinational
+risked
+Layne
+USL
+ne
+##quiries
+Ad
+Message
+Kamen
+Kristen
+reefs
+implements
+##itative
+educators
+garments
+gunshot
+##essed
+##rve
+Montevideo
+vigorously
+Stamford
+assemble
+packaged
+##same
+état
+Viva
+paragraph
+##eter
+##wire
+Stick
+Navajo
+MCA
+##pressing
+ensembles
+ABA
+##zor
+##llus
+Partner
+raked
+##BI
+Iona
+thump
+Celeste
+Kiran
+##iscovered
+##rith
+inflammation
+##arel
+Features
+loosened
+##yclic
+Deluxe
+Speak
+economical
+Frankenstein
+Picasso
+showcased
+##zad
+##eira
+##planes
+##linear
+##overs
+monsoon
+prosecutors
+slack
+Horses
+##urers
+Angry
+coughing
+##truder
+Questions
+##tō
+##zak
+challenger
+clocks
+##ieving
+Newmarket
+##acle
+cursing
+stimuli
+##mming
+##qualified
+slapping
+##vasive
+narration
+##kini
+Advertising
+CSI
+alliances
+mixes
+##yes
+covert
+amalgamation
+reproduced
+##ardt
+##gis
+1648
+id
+Annette
+Boots
+Champagne
+Brest
+Daryl
+##emon
+##jou
+##llers
+Mean
+adaptive
+technicians
+##pair
+##usal
+Yoga
+fronts
+leaping
+Jul
+harvesting
+keel
+##44
+petitioned
+##lved
+yells
+Endowment
+proponent
+##spur
+##tised
+##zal
+Homes
+Includes
+##ifer
+##oodoo
+##rvette
+awarding
+mirrored
+ransom
+Flute
+outlook
+##ganj
+DVDs
+Sufi
+frontman
+Goddard
+barren
+##astic
+Suicide
+hillside
+Harlow
+Lau
+notions
+Amnesty
+Homestead
+##irt
+GE
+hooded
+umpire
+mustered
+Catch
+Masonic
+##erd
+Dynamics
+Equity
+Oro
+Charts
+Mussolini
+populace
+muted
+accompaniment
+##lour
+##ndes
+ignited
+##iferous
+##laced
+##atch
+anguish
+registry
+##tub
+##hards
+##neer
+251
+Hooker
+uncomfortably
+##6th
+##ivers
+Catalina
+MiG
+giggling
+1754
+Dietrich
+Kaladin
+pricing
+##quence
+Sabah
+##lving
+##nical
+Gettysburg
+Vita
+Telecom
+Worst
+Palais
+Pentagon
+##brand
+##chichte
+Graf
+unnatural
+1715
+bio
+##26
+Radcliffe
+##utt
+chatting
+spices
+##aus
+untouched
+##eper
+Doll
+turkey
+Syndicate
+##rlene
+##JP
+##roots
+Como
+clashed
+modernization
+1757
+fantasies
+##iating
+dissipated
+Sicilian
+inspect
+sensible
+reputed
+##final
+Milford
+poised
+RC
+metabolic
+Tobacco
+Mecca
+optimization
+##heat
+lobe
+rabbits
+NAS
+geologist
+##liner
+Kilda
+carpenter
+nationalists
+##brae
+summarized
+##venge
+Designer
+misleading
+beamed
+##meyer
+Matrix
+excuses
+##aines
+##biology
+401
+Moose
+drafting
+Sai
+##ggle
+Comprehensive
+dripped
+skate
+##WI
+##enan
+##ruk
+narrower
+outgoing
+##enter
+##nounce
+overseen
+##structure
+travellers
+banging
+scarred
+##thing
+##arra
+Ebert
+Sometime
+##nated
+BAFTA
+Hurricanes
+configurations
+##MLL
+immortality
+##heus
+gothic
+##mpest
+clergyman
+viewpoint
+Maxim
+Instituto
+emitted
+quantitative
+1689
+Consortium
+##rsk
+Meat
+Tao
+swimmers
+Shaking
+Terence
+mainline
+##linity
+Quantum
+##rogate
+Nair
+banquet
+39th
+reprised
+lagoon
+subdivisions
+synonymous
+incurred
+password
+sprung
+##vere
+Credits
+Petersen
+Faces
+##vu
+statesman
+Zombie
+gesturing
+##going
+Sergey
+dormant
+possessive
+totals
+southward
+Ángel
+##odies
+HM
+Mariano
+Ramirez
+Wicked
+impressions
+##Net
+##cap
+##ème
+Transformers
+Poker
+RIAA
+Redesignated
+##chuk
+Harcourt
+Peña
+spacious
+tinged
+alternatively
+narrowing
+Brigham
+authorization
+Membership
+Zeppelin
+##amed
+Handball
+steer
+##orium
+##rnal
+##rops
+Committees
+endings
+##MM
+##yung
+ejected
+grams
+##relli
+Birch
+Hilary
+Stadion
+orphan
+clawed
+##kner
+Motown
+Wilkins
+ballads
+outspoken
+##ancipation
+##bankment
+##cheng
+Advances
+harvested
+novelty
+ineligible
+oversees
+##´s
+obeyed
+inevitably
+Kingdoms
+burying
+Fabian
+relevance
+Tatiana
+##MCA
+sarcastic
+##onda
+Akron
+229
+sandwiches
+Adobe
+Maddox
+##azar
+Hunting
+##onized
+Smiling
+##tology
+Juventus
+Leroy
+Poets
+attach
+lo
+##rly
+##film
+Structure
+##igate
+olds
+projections
+SMS
+outnumbered
+##tase
+judiciary
+paramilitary
+playfully
+##rsing
+##tras
+Chico
+Vin
+informally
+abandonment
+##russ
+Baroness
+injuring
+octagonal
+deciduous
+##nea
+##olm
+Hz
+Norwood
+poses
+Marissa
+alerted
+willed
+##KS
+Dino
+##ddler
+##vani
+Barbie
+Thankfully
+625
+bicycles
+shimmering
+##tinuum
+##wolf
+Chesterfield
+##idy
+##urgency
+Knowles
+sweetly
+Ventures
+##ponents
+##valence
+Darryl
+Powerplant
+RAAF
+##pec
+Kingsley
+Parramatta
+penetrating
+spectacle
+##inia
+Marlborough
+residual
+compatibility
+hike
+Underwood
+depleted
+ministries
+##odus
+##ropriation
+rotting
+Faso
+##inn
+Happiness
+Lille
+Suns
+cookie
+rift
+warmly
+##lvin
+Bugs
+Gotham
+Gothenburg
+Properties
+##seller
+##ubi
+Created
+MAC
+Noelle
+Requiem
+Ulysses
+##ails
+franchises
+##icious
+##rwick
+celestial
+kinetic
+720
+STS
+transmissions
+amplitude
+forums
+freeing
+reptiles
+tumbling
+##continent
+##rising
+##tropy
+physiology
+##uster
+Loves
+bodied
+neutrality
+Neumann
+assessments
+Vicky
+##hom
+hampered
+##uku
+Custom
+timed
+##eville
+##xious
+elastic
+##section
+rig
+stilled
+shipment
+243
+artworks
+boulders
+Bournemouth
+##hly
+##LF
+##linary
+rumored
+##bino
+##drum
+Chun
+Freiburg
+##dges
+Equality
+252
+Guadalajara
+##sors
+##taire
+Roach
+cramped
+##ultural
+Logistics
+Punch
+fines
+Lai
+caravan
+##55
+lame
+Collector
+pausing
+315
+migrant
+hawk
+signalling
+##erham
+##oughs
+Demons
+surfing
+Rana
+insisting
+Wien
+adolescent
+##jong
+##rera
+##umba
+Regis
+brushes
+##iman
+residues
+storytelling
+Consider
+contrasting
+regeneration
+##elling
+##hlete
+afforded
+reactors
+costing
+##biotics
+##gat
+##евич
+chanting
+secondly
+confesses
+##ikos
+##uang
+##ronological
+##−
+Giacomo
+##eca
+vaudeville
+weeds
+rejecting
+revoked
+affluent
+fullback
+progresses
+geologic
+proprietor
+replication
+gliding
+recounted
+##bah
+##igma
+Flow
+ii
+newcomer
+##lasp
+##miya
+Candace
+fractured
+interiors
+confidential
+Inverness
+footing
+##robe
+Coordinator
+Westphalia
+jumper
+##chism
+dormitory
+##gno
+281
+acknowledging
+leveled
+##éra
+Algiers
+migrate
+Frog
+Rare
+##iovascular
+##urous
+DSO
+nomadic
+##iera
+woken
+lifeless
+##graphical
+##ifications
+Dot
+Sachs
+crow
+nmi
+Tacoma
+Weight
+mushroom
+RS
+conditioned
+##zine
+Tunisian
+altering
+##mizing
+Handicap
+Patti
+Monsieur
+clicking
+gorge
+interrupting
+##powerment
+drawers
+Serra
+##icides
+Specialist
+##itte
+connector
+worshipped
+##ask
+consoles
+tags
+##iler
+glued
+##zac
+fences
+Bratislava
+honeymoon
+313
+A2
+disposition
+Gentleman
+Gilmore
+glaciers
+##scribed
+Calhoun
+convergence
+Aleppo
+shortages
+##43
+##orax
+##worm
+##codes
+##rmal
+neutron
+##ossa
+Bloomberg
+Salford
+periodicals
+##ryan
+Slayer
+##ynasties
+credentials
+##tista
+surveyor
+File
+stinging
+unnoticed
+Medici
+ecstasy
+espionage
+Jett
+Leary
+circulating
+bargaining
+concerto
+serviced
+37th
+HK
+##fueling
+Delilah
+Marcia
+graded
+##join
+Kaplan
+feasible
+##nale
+##yt
+Burnley
+dreadful
+ministerial
+Brewster
+Judah
+##ngled
+##rrey
+recycled
+Iroquois
+backstage
+parchment
+##numbered
+Kern
+Motorsports
+Organizations
+##mini
+Seems
+Warrington
+Dunbar
+Ezio
+##eor
+paralyzed
+Ara
+yeast
+##olis
+cheated
+reappeared
+banged
+##ymph
+##dick
+Lyndon
+glide
+Mat
+##natch
+Hotels
+Household
+parasite
+irrelevant
+youthful
+##smic
+##tero
+##anti
+2d
+Ignacio
+squash
+##nets
+shale
+##اد
+Abrams
+##oese
+assaults
+##dier
+##otte
+Swamp
+287
+Spurs
+##economic
+Fargo
+auditioned
+##mé
+Haas
+une
+abbreviation
+Turkic
+##tisfaction
+favorites
+specials
+##lial
+Enlightenment
+Burkina
+##vir
+Comparative
+Lacrosse
+elves
+##lerical
+##pear
+Borders
+controllers
+##villa
+excelled
+##acher
+##varo
+camouflage
+perpetual
+##ffles
+devoid
+schooner
+##bered
+##oris
+Gibbons
+Lia
+discouraged
+sue
+##gnition
+Excellent
+Layton
+noir
+smack
+##ivable
+##evity
+##lone
+Myra
+weaken
+weaponry
+##azza
+Shake
+backbone
+Certified
+clown
+occupational
+caller
+enslaved
+soaking
+Wexford
+perceive
+shortlisted
+##pid
+feminism
+Bari
+Indie
+##avelin
+##ldo
+Hellenic
+Hundreds
+Savings
+comedies
+Honors
+Mohawk
+Told
+coded
+Incorporated
+hideous
+trusts
+hose
+Calais
+Forster
+Gabon
+Internationale
+AK
+Colour
+##UM
+##heist
+McGregor
+localized
+##tronomy
+Darrell
+##iara
+squirrel
+freaked
+##eking
+##manned
+##ungen
+radiated
+##dua
+commence
+Donaldson
+##iddle
+MR
+SAS
+Tavern
+Teenage
+admissions
+Instruments
+##ilizer
+Konrad
+contemplated
+##ductor
+Jing
+Reacher
+recalling
+Dhabi
+emphasizing
+illumination
+##tony
+legitimacy
+Goethe
+Ritter
+McDonnell
+Polar
+Seconds
+aspiring
+derby
+tunic
+##rmed
+outlines
+Changing
+distortion
+##cter
+Mechanics
+##urly
+##vana
+Egg
+Wolverine
+Stupid
+centralized
+knit
+##Ms
+Saratoga
+Ogden
+storylines
+##vres
+lavish
+beverages
+##grarian
+Kyrgyzstan
+forcefully
+superb
+Elm
+Thessaloniki
+follower
+Plants
+slang
+trajectory
+Nowadays
+Bengals
+Ingram
+perch
+coloring
+carvings
+doubtful
+##aph
+##gratulations
+##41
+Curse
+253
+nightstand
+Campo
+Meiji
+decomposition
+##giri
+McCormick
+Yours
+##amon
+##bang
+Texans
+injunction
+organise
+periodical
+##peculative
+oceans
+##aley
+Success
+Lehigh
+##guin
+1730
+Davy
+allowance
+obituary
+##tov
+treasury
+##wayne
+euros
+readiness
+systematically
+##stered
+##igor
+##xen
+##cliff
+##lya
+Send
+##umatic
+Celtics
+Judiciary
+425
+propagation
+rebellious
+##ims
+##lut
+Dal
+##ayman
+##cloth
+Boise
+pairing
+Waltz
+torment
+Hatch
+aspirations
+diaspora
+##hame
+Rank
+237
+Including
+Muir
+chained
+toxicity
+Université
+##aroo
+Mathews
+meadows
+##bio
+Editing
+Khorasan
+##them
+##ahn
+##bari
+##umes
+evacuate
+##sium
+gram
+kidnap
+pinning
+##diation
+##orms
+beacon
+organising
+McGrath
+##ogist
+Qur
+Tango
+##ceptor
+##rud
+##cend
+##cie
+##jas
+##sided
+Tuscany
+Venture
+creations
+exhibiting
+##rcerer
+##tten
+Butcher
+Divinity
+Pet
+Whitehead
+falsely
+perished
+handy
+Moines
+cyclists
+synthesizers
+Mortal
+notoriety
+##ronic
+Dialogue
+expressive
+uk
+Nightingale
+grimly
+vineyards
+Driving
+relentless
+compiler
+##district
+##tuated
+Hades
+medicines
+objection
+Answer
+Soap
+Chattanooga
+##gogue
+Haryana
+Parties
+Turtle
+##ferred
+explorers
+stakeholders
+##aar
+##rbonne
+tempered
+conjecture
+##tee
+##hur
+Reeve
+bumper
+stew
+##church
+##generate
+##ilitating
+##chanized
+##elier
+##enne
+translucent
+##lows
+Publisher
+evangelical
+inherit
+##rted
+247
+SmackDown
+bitterness
+lesions
+##worked
+mosques
+wed
+##lashes
+Ng
+Rebels
+booking
+##nail
+Incident
+Sailing
+yo
+confirms
+Chaplin
+baths
+##kled
+modernist
+pulsing
+Cicero
+slaughtered
+boasted
+##losure
+zipper
+##hales
+aristocracy
+halftime
+jolt
+unlawful
+Marching
+sustaining
+Yerevan
+bracket
+ram
+Markus
+##zef
+butcher
+massage
+##quisite
+Leisure
+Pizza
+collapsing
+##lante
+commentaries
+scripted
+##disciplinary
+##sused
+eroded
+alleging
+vase
+Chichester
+Peacock
+commencement
+dice
+hotter
+poisonous
+executions
+##occo
+frost
+fielding
+vendor
+Counts
+Troops
+maize
+Divisional
+analogue
+shadowy
+Nuevo
+Ville
+radiating
+worthless
+Adriatic
+Buy
+blaze
+brutally
+horizontally
+longed
+##matical
+federally
+Rolf
+Root
+exclude
+rag
+agitation
+Lounge
+astonished
+##wirl
+Impossible
+transformations
+##IVE
+##ceded
+##slav
+downloaded
+fucked
+Egyptians
+Welles
+##ffington
+U2
+befriended
+radios
+##jid
+archaic
+compares
+##ccelerator
+##imated
+##tosis
+Hung
+Scientists
+Thousands
+geographically
+##LR
+Macintosh
+fluorescent
+##ipur
+Wehrmacht
+##BR
+##firmary
+Chao
+##ague
+Boyer
+##grounds
+##hism
+##mento
+##taining
+infancy
+##cton
+510
+Boca
+##loy
+1644
+ben
+dong
+stresses
+Sweat
+expressway
+graders
+ochreous
+nets
+Lawn
+thirst
+Uruguayan
+satisfactory
+##tracts
+baroque
+rusty
+##ław
+Shen
+Gdańsk
+chickens
+##graving
+Hodge
+Papal
+SAT
+bearer
+##ogo
+##rger
+merits
+Calendar
+Highest
+Skills
+##ortex
+Roberta
+paradigm
+recounts
+frigates
+swamps
+unitary
+##oker
+balloons
+Hawthorne
+Muse
+spurred
+advisors
+reclaimed
+stimulate
+fibre
+pat
+repeal
+##dgson
+##iar
+##rana
+anthropologist
+descends
+flinch
+reared
+##chang
+##eric
+##lithic
+commissioning
+##cumenical
+##lume
+##rchen
+Wolff
+##tsky
+Eurasian
+Nepali
+Nightmare
+ZIP
+playback
+##latz
+##vington
+Warm
+##75
+Martina
+Rollins
+Saetan
+Variations
+sorting
+##م
+530
+Joaquin
+Ptolemy
+thinner
+##iator
+##pticism
+Cebu
+Highlanders
+Linden
+Vanguard
+##SV
+##mor
+##ulge
+ISSN
+cartridges
+repression
+Étienne
+311
+Lauderdale
+commodities
+null
+##rb
+1720
+gearbox
+##reator
+Ang
+Forgotten
+dubious
+##rls
+##dicative
+##phate
+Groove
+Herrera
+##çais
+Collections
+Maximus
+##published
+Fell
+Qualification
+filtering
+##tized
+Roe
+hazards
+##37
+##lative
+##tröm
+Guadalupe
+Tajikistan
+Preliminary
+fronted
+glands
+##paper
+##iche
+##iding
+Cairns
+rallies
+Location
+seduce
+##mple
+BYU
+##itic
+##FT
+Carmichael
+Prentice
+songwriters
+forefront
+Physicians
+##rille
+##zee
+Preparatory
+##cherous
+UV
+##dized
+Navarro
+misses
+##nney
+Inland
+resisting
+##sect
+Hurt
+##lino
+galaxies
+##raze
+Institutions
+devote
+##lamp
+##ciating
+baron
+##bracing
+Hess
+operatic
+##CL
+##ος
+Chevalier
+Guiana
+##lattered
+Fed
+##cuted
+##smo
+Skull
+denies
+236
+Waller
+##mah
+Sakura
+mole
+nominate
+sermons
+##bering
+widowed
+##röm
+Cavendish
+##struction
+Nehru
+Revelation
+doom
+Gala
+baking
+Nr
+Yourself
+banning
+Individuals
+Sykes
+orchestrated
+630
+Phone
+steered
+620
+specialising
+starvation
+##AV
+##alet
+##upation
+seductive
+##jects
+##zure
+Tolkien
+Benito
+Wizards
+Submarine
+dictator
+Duo
+Caden
+approx
+basins
+##nc
+shrink
+##icles
+##sponsible
+249
+mit
+outpost
+##bayashi
+##rouse
+##tl
+Jana
+Lombard
+RBIs
+finalized
+humanities
+##function
+Honorable
+tomato
+##iot
+Pie
+tee
+##pect
+Beaufort
+Ferris
+bucks
+##graduate
+##ocytes
+Directory
+anxiously
+##nating
+flanks
+##Ds
+virtues
+##believable
+Grades
+criterion
+manufactures
+sourced
+##balt
+##dance
+##tano
+Ying
+##BF
+##sett
+adequately
+blacksmith
+totaled
+trapping
+expanse
+Historia
+Worker
+Sense
+ascending
+housekeeper
+##oos
+Crafts
+Resurrection
+##verty
+encryption
+##aris
+##vat
+##pox
+##runk
+##iability
+gazes
+spying
+##ths
+helmets
+wired
+##zophrenia
+Cheung
+WR
+downloads
+stereotypes
+239
+Lucknow
+bleak
+Bragg
+hauling
+##haft
+prohibit
+##ermined
+##castle
+barony
+##hta
+Typhoon
+antibodies
+##ascism
+Hawthorn
+Kurdistan
+Minority
+Gorge
+Herr
+appliances
+disrupt
+Drugs
+Lazarus
+##ilia
+##ryo
+##tany
+Gotta
+Masovian
+Roxy
+choreographed
+##rissa
+turbulent
+##listed
+Anatomy
+exiting
+##det
+##isław
+580
+Kaufman
+sage
+##apa
+Symposium
+##rolls
+Kaye
+##ptera
+##rocław
+jerking
+##menclature
+Guo
+M1
+resurrected
+trophies
+##lard
+Gathering
+nestled
+serpent
+Dow
+reservoirs
+Claremont
+arbitration
+chronicle
+eki
+##arded
+##zers
+##mmoth
+Congregational
+Astronomical
+NE
+RA
+Robson
+Scotch
+modelled
+slashed
+##imus
+exceeds
+##roper
+##utile
+Laughing
+vascular
+superficial
+##arians
+Barclay
+Caucasian
+classmate
+sibling
+Kimberly
+Shreveport
+##ilde
+##liche
+Cheney
+Deportivo
+Veracruz
+berries
+##lase
+Bed
+MI
+Anatolia
+Mindanao
+broadband
+##olia
+##arte
+##wab
+darts
+##immer
+##uze
+believers
+ordinance
+violate
+##wheel
+##ynth
+Alongside
+Coupe
+Hobbs
+arrondissement
+earl
+townland
+##dote
+##lihood
+##sla
+Ghosts
+midfield
+pulmonary
+##eno
+cues
+##gol
+##zda
+322
+Siena
+Sultanate
+Bradshaw
+Pieter
+##thical
+Raceway
+bared
+competence
+##ssent
+Bet
+##urer
+##ła
+Alistair
+Göttingen
+appropriately
+forge
+##osterone
+##ugen
+DL
+345
+convoys
+inventions
+##resses
+##cturnal
+Fay
+Integration
+slash
+##roats
+Widow
+barking
+##fant
+1A
+Hooper
+##cona
+##runched
+unreliable
+##emont
+##esign
+##stabulary
+##stop
+Journalists
+bony
+##iba
+##trata
+##ège
+horrific
+##bish
+Jocelyn
+##rmon
+##apon
+##cier
+trainers
+##ulatory
+1753
+BR
+corpus
+synthesized
+##bidden
+##rafford
+Elgin
+##entry
+Doherty
+clockwise
+##played
+spins
+##ample
+##bley
+Cope
+constructions
+seater
+warlord
+Voyager
+documenting
+fairies
+##viator
+Lviv
+jewellery
+suites
+##gold
+Maia
+NME
+##eavor
+##kus
+Eugène
+furnishings
+##risto
+MCC
+Metropolis
+Older
+Telangana
+##mpus
+amplifier
+supervising
+1710
+buffalo
+cushion
+terminating
+##powering
+steak
+Quickly
+contracting
+dem
+sarcastically
+Elsa
+##hein
+bastards
+narratives
+Takes
+304
+composure
+typing
+variance
+##ifice
+Softball
+##rations
+McLaughlin
+gaped
+shrines
+##hogany
+Glamorgan
+##icle
+##nai
+##ntin
+Fleetwood
+Woodland
+##uxe
+fictitious
+shrugs
+##iper
+BWV
+conform
+##uckled
+Launch
+##ductory
+##mized
+Tad
+##stituted
+##free
+Bel
+Chávez
+messing
+quartz
+##iculate
+##folia
+##lynn
+ushered
+##29
+##ailing
+dictated
+Pony
+##opsis
+precinct
+802
+Plastic
+##ughter
+##uno
+##porated
+Denton
+Matters
+SPD
+hating
+##rogen
+Essential
+Deck
+Dortmund
+obscured
+##maging
+Earle
+##bred
+##ittle
+##ropolis
+saturated
+##fiction
+##ression
+Pereira
+Vinci
+mute
+warehouses
+##ún
+biographies
+##icking
+sealing
+##dered
+executing
+pendant
+##wives
+murmurs
+##oko
+substrates
+symmetrical
+Susie
+##mare
+Yusuf
+analogy
+##urage
+Lesley
+limitation
+##rby
+##ío
+disagreements
+##mise
+embroidered
+nape
+unarmed
+Sumner
+Stores
+dwell
+Wilcox
+creditors
+##rivatization
+##shes
+##amia
+directs
+recaptured
+scouting
+McGuire
+cradle
+##onnell
+Sato
+insulin
+mercenary
+tolerant
+Macquarie
+transitions
+cradled
+##berto
+##ivism
+##yotes
+FF
+Ke
+Reach
+##dbury
+680
+##bill
+##oja
+##sui
+prairie
+##ogan
+reactive
+##icient
+##rits
+Cyclone
+Sirius
+Survival
+Pak
+##coach
+##trar
+halves
+Agatha
+Opus
+contrasts
+##jection
+ominous
+##iden
+Baylor
+Woodrow
+duct
+fortification
+intercourse
+##rois
+Colbert
+envy
+##isi
+Afterward
+geared
+##flections
+accelerate
+##lenching
+Witness
+##rrer
+Angelina
+Material
+assertion
+misconduct
+Nix
+cringed
+tingling
+##eti
+##gned
+Everest
+disturb
+sturdy
+##keepers
+##vied
+Profile
+heavenly
+##kova
+##victed
+translating
+##sses
+316
+Invitational
+Mention
+martyr
+##uristic
+Barron
+hardness
+Nakamura
+405
+Genevieve
+reflections
+##falls
+jurist
+##LT
+Pyramid
+##yme
+Shoot
+heck
+linguist
+##tower
+Ives
+superiors
+##leo
+Achilles
+##phological
+Christophe
+Padma
+precedence
+grassy
+Oral
+resurrection
+##itting
+clumsy
+##lten
+##rue
+huts
+##stars
+Equal
+##queduct
+Devin
+Gaga
+diocesan
+##plating
+##upe
+##graphers
+Patch
+Scream
+hail
+moaning
+tracts
+##hdi
+Examination
+outsider
+##ergic
+##oter
+Archipelago
+Havilland
+greenish
+tilting
+Aleksandr
+Konstantin
+warship
+##emann
+##gelist
+##ought
+billionaire
+##blivion
+321
+Hungarians
+transplant
+##jured
+##fters
+Corbin
+autism
+pitchers
+Garner
+thence
+Scientology
+transitioned
+integrating
+repetitive
+##dant
+Rene
+vomit
+##burne
+1661
+Researchers
+Wallis
+insulted
+wavy
+##wati
+Ewing
+excitedly
+##kor
+frescoes
+injustice
+##achal
+##lumber
+##úl
+novella
+##sca
+Liv
+##enstein
+##river
+monstrous
+topping
+downfall
+looming
+sinks
+trillion
+##pont
+Effect
+##phi
+##urley
+Sites
+catchment
+##H1
+Hopper
+##raiser
+1642
+Maccabi
+lance
+##chia
+##sboro
+NSA
+branching
+retorted
+tensor
+Immaculate
+drumming
+feeder
+##mony
+Dyer
+homicide
+Temeraire
+fishes
+protruding
+skins
+orchards
+##nso
+inlet
+ventral
+##finder
+Asiatic
+Sul
+1688
+Melinda
+assigns
+paranormal
+gardening
+Tau
+calming
+##inge
+##crow
+regimental
+Nik
+fastened
+correlated
+##gene
+##rieve
+Sick
+##minster
+##politan
+hardwood
+hurled
+##ssler
+Cinematography
+rhyme
+Montenegrin
+Packard
+debating
+##itution
+Helens
+Trick
+Museums
+defiance
+encompassed
+##EE
+##TU
+##nees
+##uben
+##ünster
+##nosis
+435
+Hagen
+cinemas
+Corbett
+commended
+##fines
+##oman
+bosses
+ripe
+scraping
+##loc
+filly
+Saddam
+pointless
+Faust
+Orléans
+Syriac
+##♭
+longitude
+##ropic
+Alfa
+bliss
+gangster
+##ckling
+SL
+blending
+##eptide
+##nner
+bends
+escorting
+##bloid
+##quis
+burials
+##sle
+##è
+Ambulance
+insults
+##gth
+Antrim
+unfolded
+##missible
+splendid
+Cure
+warily
+Saigon
+Waste
+astonishment
+boroughs
+##VS
+##dalgo
+##reshing
+##usage
+rue
+marital
+versatile
+unpaid
+allotted
+bacterium
+##coil
+##cue
+Dorothea
+IDF
+##location
+##yke
+RPG
+##tropical
+devotees
+liter
+##pree
+Johnstone
+astronaut
+attends
+pollen
+periphery
+doctrines
+meta
+showered
+##tyn
+GO
+Huh
+laude
+244
+Amar
+Christensen
+Ping
+Pontifical
+Austen
+raiding
+realities
+##dric
+urges
+##dek
+Cambridgeshire
+##otype
+Cascade
+Greenberg
+Pact
+##cognition
+##aran
+##urion
+Riot
+mimic
+Eastwood
+##imating
+reversal
+##blast
+##henian
+Pitchfork
+##sunderstanding
+Staten
+WCW
+lieu
+##bard
+##sang
+experimenting
+Aquino
+##lums
+TNT
+Hannibal
+catastrophic
+##lsive
+272
+308
+##otypic
+41st
+Highways
+aggregator
+##fluenza
+Featured
+Reece
+dispatch
+simulated
+##BE
+Communion
+Vinnie
+hardcover
+inexpensive
+til
+##adores
+groundwater
+kicker
+blogs
+frenzy
+##wala
+dealings
+erase
+Anglia
+##umour
+Hapoel
+Marquette
+##raphic
+##tives
+consult
+atrocities
+concussion
+##érard
+Decree
+ethanol
+##aen
+Rooney
+##chemist
+##hoot
+1620
+menacing
+Schuster
+##bearable
+laborers
+sultan
+Juliana
+erased
+onstage
+##ync
+Eastman
+##tick
+hushed
+##yrinth
+Lexie
+Wharton
+Lev
+##PL
+Testing
+Bangladeshi
+##bba
+##usions
+communicated
+integers
+internship
+societal
+##odles
+Loki
+ET
+Ghent
+broadcasters
+Unix
+##auer
+Kildare
+Yamaha
+##quencing
+##zman
+chilled
+##rapped
+##uant
+Duval
+sentiments
+Oliveira
+packets
+Horne
+##rient
+Harlan
+Mirage
+invariant
+##anger
+##tensive
+flexed
+sweetness
+##wson
+alleviate
+insulting
+limo
+Hahn
+##llars
+##hesia
+##lapping
+buys
+##oaming
+mocked
+pursuits
+scooted
+##conscious
+##ilian
+Ballad
+jackets
+##kra
+hilly
+##cane
+Scenic
+McGraw
+silhouette
+whipping
+##roduced
+##wark
+##chess
+##rump
+Lemon
+calculus
+demonic
+##latine
+Bharatiya
+Govt
+Que
+Trilogy
+Ducks
+Suit
+stairway
+##ceipt
+Isa
+regulator
+Automobile
+flatly
+##buster
+##lank
+Spartans
+topography
+Tavi
+usable
+Chartered
+Fairchild
+##sance
+##vyn
+Digest
+nuclei
+typhoon
+##llon
+Alvarez
+DJs
+Grimm
+authoritative
+firearm
+##chschule
+Origins
+lair
+unmistakable
+##xial
+##cribing
+Mouth
+##genesis
+##shū
+##gaon
+##ulter
+Jaya
+Neck
+##UN
+##oing
+##static
+relativity
+##mott
+##utive
+##esan
+##uveau
+BT
+salts
+##roa
+Dustin
+preoccupied
+Novgorod
+##asus
+Magnum
+tempting
+##histling
+##ilated
+Musa
+##ghty
+Ashland
+pubs
+routines
+##etto
+Soto
+257
+Featuring
+Augsburg
+##alaya
+Bit
+loomed
+expects
+##abby
+##ooby
+Auschwitz
+Pendleton
+vodka
+##sent
+rescuing
+systemic
+##inet
+##leg
+Yun
+applicant
+revered
+##nacht
+##ndas
+Muller
+characterization
+##patient
+##roft
+Carole
+##asperated
+Amiga
+disconnected
+gel
+##cologist
+Patriotic
+rallied
+assign
+veterinary
+installing
+##cedural
+258
+Jang
+Parisian
+incarcerated
+stalk
+##iment
+Jamal
+McPherson
+Palma
+##oken
+##viation
+512
+Rourke
+irrational
+##rippled
+Devlin
+erratic
+##NI
+##payers
+Ni
+engages
+Portal
+aesthetics
+##rrogance
+Milne
+assassins
+##rots
+335
+385
+Cambodian
+Females
+fellows
+si
+##block
+##otes
+Jayne
+Toro
+flutter
+##eera
+Burr
+##lanche
+relaxation
+##fra
+Fitzroy
+##undy
+1751
+261
+comb
+conglomerate
+ribbons
+veto
+##Es
+casts
+##ege
+1748
+Ares
+spears
+spirituality
+comet
+##nado
+##yeh
+Veterinary
+aquarium
+yer
+Councils
+##oked
+##ynamic
+Malmö
+remorse
+auditions
+drilled
+Hoffmann
+Moe
+Nagoya
+Yacht
+##hakti
+##race
+##rrick
+Talmud
+coordinating
+##EI
+##bul
+##his
+##itors
+##ligent
+##uerra
+Narayan
+goaltender
+taxa
+##asures
+Det
+##mage
+Infinite
+Maid
+bean
+intriguing
+##cription
+gasps
+socket
+##mentary
+##reus
+sewing
+transmitting
+##different
+##furbishment
+##traction
+Grimsby
+sprawling
+Shipyard
+##destine
+##hropic
+##icked
+trolley
+##agi
+##lesh
+Josiah
+invasions
+Content
+firefighters
+intro
+Lucifer
+subunit
+Sahib
+Myrtle
+inhibitor
+maneuvers
+##teca
+Wrath
+slippery
+##versing
+Shoes
+##dial
+##illiers
+##luded
+##mmal
+##pack
+handkerchief
+##edestal
+##stones
+Fusion
+cumulative
+##mell
+##cacia
+##rudge
+##utz
+foe
+storing
+swiped
+##meister
+##orra
+batter
+strung
+##venting
+##kker
+Doo
+Taste
+immensely
+Fairbanks
+Jarrett
+Boogie
+1746
+mage
+Kick
+legislators
+medial
+##ilon
+##logies
+##ranton
+Hybrid
+##uters
+Tide
+deportation
+Metz
+##secration
+##virus
+UFO
+##fell
+##orage
+##raction
+##rrigan
+1747
+fabricated
+##BM
+##GR
+##rter
+muttering
+theorist
+##tamine
+BMG
+Kincaid
+solvent
+##azed
+Thin
+adorable
+Wendell
+ta
+##viour
+pulses
+##pologies
+counters
+exposition
+sewer
+Luciano
+Clancy
+##angelo
+##riars
+Showtime
+observes
+frankly
+##oppy
+Bergman
+lobes
+timetable
+##bri
+##uest
+FX
+##dust
+##genus
+Glad
+Helmut
+Meridian
+##besity
+##ontaine
+Revue
+miracles
+##titis
+PP
+bluff
+syrup
+307
+Messiah
+##erne
+interfering
+picturesque
+unconventional
+dipping
+hurriedly
+Kerman
+248
+Ethnic
+Toward
+acidic
+Harrisburg
+##65
+intimidating
+##aal
+Jed
+Pontiac
+munitions
+##nchen
+growling
+mausoleum
+##ération
+##wami
+Cy
+aerospace
+caucus
+Doing
+##around
+##miring
+Cuthbert
+##poradic
+##rovisation
+##wth
+evaluating
+##scraper
+Belinda
+owes
+##sitic
+##thermal
+##fast
+economists
+##lishing
+##uerre
+##ân
+credible
+##koto
+Fourteen
+cones
+##ebrates
+bookstore
+towels
+##phony
+Appearance
+newscasts
+##olin
+Karin
+Bingham
+##elves
+1680
+306
+disks
+##lston
+##secutor
+Levant
+##vout
+Micro
+snuck
+##ogel
+##racker
+Exploration
+drastic
+##kening
+Elsie
+endowment
+##utnant
+Blaze
+##rrosion
+leaking
+45th
+##rug
+##uernsey
+760
+Shapiro
+cakes
+##ehan
+##mei
+##ité
+##kla
+repetition
+successively
+Friendly
+Île
+Koreans
+Au
+Tirana
+flourish
+Spirits
+Yao
+reasoned
+##leam
+Consort
+cater
+marred
+ordeal
+supremacy
+##ritable
+Paisley
+euro
+healer
+portico
+wetland
+##kman
+restart
+##habilitation
+##zuka
+##Script
+emptiness
+communion
+##CF
+##inhabited
+##wamy
+Casablanca
+pulsed
+##rrible
+##safe
+395
+Dual
+Terrorism
+##urge
+##found
+##gnolia
+Courage
+patriarch
+segregated
+intrinsic
+##liography
+##phe
+PD
+convection
+##icidal
+Dharma
+Jimmie
+texted
+constituents
+twitch
+##calated
+##mitage
+##ringing
+415
+milling
+##geons
+Armagh
+Geometridae
+evergreen
+needy
+reflex
+template
+##pina
+Schubert
+##bruck
+##icted
+##scher
+##wildered
+1749
+Joanne
+clearer
+##narl
+278
+Print
+automation
+consciously
+flashback
+occupations
+##ests
+Casimir
+differentiated
+policing
+repay
+##aks
+##gnesium
+Evaluation
+commotion
+##CM
+##smopolitan
+Clapton
+mitochondrial
+Kobe
+1752
+Ignoring
+Vincenzo
+Wet
+bandage
+##rassed
+##unate
+Maris
+##eted
+##hetical
+figuring
+##eit
+##nap
+leopard
+strategically
+##reer
+Fen
+Iain
+##ggins
+##pipe
+Matteo
+McIntyre
+##chord
+##feng
+Romani
+asshole
+flopped
+reassure
+Founding
+Styles
+Torino
+patrolling
+##erging
+##ibrating
+##ructural
+sincerity
+##ät
+##teacher
+Juliette
+##cé
+##hog
+##idated
+##span
+Winfield
+##fender
+##nast
+##pliant
+1690
+Bai
+Je
+Saharan
+expands
+Bolshevik
+rotate
+##root
+Britannia
+Severn
+##cini
+##gering
+##say
+sly
+Steps
+insertion
+rooftop
+Piece
+cuffs
+plausible
+##zai
+Provost
+semantic
+##data
+##vade
+##cimal
+IPA
+indictment
+Libraries
+flaming
+highlands
+liberties
+##pio
+Elders
+aggressively
+##pecific
+Decision
+pigeon
+nominally
+descriptive
+adjustments
+equestrian
+heaving
+##mour
+##dives
+##fty
+##yton
+intermittent
+##naming
+##sets
+Calvert
+Casper
+Tarzan
+##kot
+Ramírez
+##IB
+##erus
+Gustavo
+Roller
+vaulted
+##solation
+##formatics
+##tip
+Hunger
+colloquially
+handwriting
+hearth
+launcher
+##idian
+##ilities
+##lind
+##locating
+Magdalena
+Soo
+clubhouse
+##kushima
+##ruit
+Bogotá
+Organic
+Worship
+##Vs
+##wold
+upbringing
+##kick
+groundbreaking
+##urable
+##ván
+repulsed
+##dira
+##ditional
+##ici
+melancholy
+##bodied
+##cchi
+404
+concurrency
+H₂O
+bouts
+##gami
+288
+Leto
+troll
+##lak
+advising
+bundled
+##nden
+lipstick
+littered
+##leading
+##mogeneous
+Experiment
+Nikola
+grove
+##ogram
+Mace
+##jure
+cheat
+Annabelle
+Tori
+lurking
+Emery
+Walden
+##riz
+paints
+Markets
+brutality
+overrun
+##agu
+##sat
+din
+ostensibly
+Fielding
+flees
+##eron
+Pound
+ornaments
+tornadoes
+##nikov
+##organisation
+##reen
+##Works
+##ldred
+##olten
+##stillery
+soluble
+Mata
+Grimes
+Léon
+##NF
+coldly
+permitting
+##inga
+##reaked
+Agents
+hostess
+##dl
+Dyke
+Kota
+avail
+orderly
+##saur
+##sities
+Arroyo
+##ceps
+##egro
+Hawke
+Noctuidae
+html
+seminar
+##ggles
+##wasaki
+Clube
+recited
+##sace
+Ascension
+Fitness
+dough
+##ixel
+Nationale
+##solidate
+pulpit
+vassal
+570
+Annapolis
+bladder
+phylogenetic
+##iname
+convertible
+##ppan
+Comet
+paler
+##definite
+Spot
+##dices
+frequented
+Apostles
+slalom
+##ivision
+##mana
+##runcated
+Trojan
+##agger
+##iq
+##league
+Concept
+Controller
+##barian
+##curate
+##spersed
+##tring
+engulfed
+inquired
+##hmann
+286
+##dict
+##osy
+##raw
+MacKenzie
+su
+##ienced
+##iggs
+##quitaine
+bisexual
+##noon
+runways
+subsp
+##!
+##"
+###
+##$
+##%
+##&
+##'
+##(
+##)
+##*
+##+
+##,
+##-
+##.
+##/
+##:
+##;
+##<
+##=
+##>
+##?
+##@
+##[
+##\
+##]
+##^
+##_
+##`
+##{
+##|
+##}
+##~
+##¡
+##¢
+##£
+##¥
+##§
+##¨
+##©
+##ª
+##«
+##¬
+##®
+##±
+##´
+##µ
+##¶
+##·
+##¹
+##º
+##»
+##¼
+##¾
+##¿
+##À
+##Á
+##Â
+##Ä
+##Å
+##Æ
+##Ç
+##È
+##É
+##Í
+##Î
+##Ñ
+##Ó
+##Ö
+##×
+##Ø
+##Ú
+##Ü
+##Þ
+##â
+##ã
+##æ
+##ç
+##î
+##ï
+##ð
+##ñ
+##ô
+##õ
+##÷
+##û
+##þ
+##ÿ
+##Ā
+##ą
+##Ć
+##Č
+##ď
+##Đ
+##đ
+##ē
+##ė
+##ę
+##ě
+##ğ
+##ġ
+##Ħ
+##ħ
+##ĩ
+##Ī
+##İ
+##ļ
+##Ľ
+##ľ
+##Ł
+##ņ
+##ň
+##ŋ
+##Ō
+##ŏ
+##ő
+##Œ
+##œ
+##ř
+##Ś
+##ś
+##Ş
+##Š
+##Ţ
+##ţ
+##ť
+##ũ
+##ŭ
+##ů
+##ű
+##ų
+##ŵ
+##ŷ
+##ź
+##Ż
+##ż
+##Ž
+##ž
+##Ə
+##ƒ
+##ơ
+##ư
+##ǎ
+##ǐ
+##ǒ
+##ǔ
+##ǫ
+##Ș
+##Ț
+##ț
+##ɐ
+##ɑ
+##ɔ
+##ɕ
+##ə
+##ɛ
+##ɡ
+##ɣ
+##ɨ
+##ɪ
+##ɲ
+##ɾ
+##ʀ
+##ʁ
+##ʂ
+##ʃ
+##ʊ
+##ʋ
+##ʌ
+##ʐ
+##ʑ
+##ʒ
+##ʔ
+##ʰ
+##ʲ
+##ʳ
+##ʷ
+##ʻ
+##ʼ
+##ʾ
+##ʿ
+##ˈ
+##ː
+##ˡ
+##ˢ
+##ˣ
+##́
+##̃
+##̍
+##̯
+##͡
+##Α
+##Β
+##Γ
+##Δ
+##Ε
+##Η
+##Θ
+##Ι
+##Κ
+##Λ
+##Μ
+##Ν
+##Ο
+##Π
+##Σ
+##Τ
+##Φ
+##Χ
+##Ψ
+##Ω
+##ά
+##έ
+##ή
+##ί
+##β
+##γ
+##δ
+##ε
+##ζ
+##η
+##θ
+##ι
+##κ
+##λ
+##μ
+##ξ
+##ο
+##π
+##ρ
+##σ
+##τ
+##υ
+##φ
+##χ
+##ψ
+##ω
+##ό
+##ύ
+##ώ
+##І
+##Ј
+##А
+##Б
+##В
+##Г
+##Д
+##Е
+##Ж
+##З
+##И
+##К
+##Л
+##М
+##Н
+##О
+##П
+##Р
+##С
+##Т
+##У
+##Ф
+##Х
+##Ц
+##Ч
+##Ш
+##Э
+##Ю
+##Я
+##б
+##в
+##г
+##д
+##ж
+##з
+##к
+##л
+##м
+##п
+##с
+##т
+##у
+##ф
+##х
+##ц
+##ч
+##ш
+##щ
+##ъ
+##ы
+##ь
+##э
+##ю
+##ё
+##і
+##ї
+##ј
+##њ
+##ћ
+##Ա
+##Հ
+##ա
+##ե
+##ի
+##կ
+##մ
+##յ
+##ն
+##ո
+##ս
+##տ
+##ր
+##ւ
+##ְ
+##ִ
+##ֵ
+##ֶ
+##ַ
+##ָ
+##ֹ
+##ּ
+##א
+##ב
+##ג
+##ד
+##ה
+##ו
+##ז
+##ח
+##ט
+##י
+##כ
+##ל
+##ם
+##מ
+##ן
+##נ
+##ס
+##ע
+##פ
+##צ
+##ק
+##ר
+##ש
+##ת
+##،
+##ء
+##آ
+##أ
+##إ
+##ئ
+##ا
+##ب
+##ت
+##ث
+##ج
+##ح
+##خ
+##ذ
+##ز
+##س
+##ش
+##ص
+##ض
+##ط
+##ظ
+##ع
+##غ
+##ف
+##ق
+##ك
+##ل
+##و
+##ى
+##َ
+##ِ
+##ٹ
+##پ
+##چ
+##ک
+##گ
+##ہ
+##ی
+##ے
+##ं
+##आ
+##क
+##ग
+##च
+##ज
+##ण
+##त
+##द
+##ध
+##न
+##प
+##ब
+##भ
+##म
+##य
+##र
+##ल
+##व
+##श
+##ष
+##स
+##ह
+##ा
+##ि
+##ी
+##ु
+##े
+##ो
+##्
+##।
+##॥
+##আ
+##ই
+##এ
+##ও
+##ক
+##খ
+##গ
+##চ
+##ছ
+##জ
+##ট
+##ত
+##থ
+##দ
+##ধ
+##ন
+##প
+##ব
+##ম
+##য
+##র
+##ল
+##শ
+##স
+##হ
+##়
+##া
+##ি
+##ী
+##ু
+##ে
+##ো
+##্
+##য়
+##க
+##த
+##ப
+##ம
+##ய
+##ர
+##ல
+##வ
+##ா
+##ி
+##ு
+##்
+##ร
+##་
+##ག
+##ང
+##ད
+##ན
+##བ
+##མ
+##ར
+##ལ
+##ས
+##ི
+##ུ
+##ེ
+##ོ
+##ა
+##ე
+##ი
+##ლ
+##ნ
+##ო
+##რ
+##ს
+##ᴬ
+##ᴵ
+##ᵀ
+##ᵃ
+##ᵇ
+##ᵈ
+##ᵉ
+##ᵍ
+##ᵏ
+##ᵐ
+##ᵒ
+##ᵖ
+##ᵗ
+##ᵘ
+##ᵣ
+##ᵤ
+##ᵥ
+##ᶜ
+##ᶠ
+##ḍ
+##Ḥ
+##ḥ
+##Ḩ
+##ḩ
+##ḳ
+##ṃ
+##ṅ
+##ṇ
+##ṛ
+##ṣ
+##ṭ
+##ạ
+##ả
+##ấ
+##ầ
+##ẩ
+##ậ
+##ắ
+##ế
+##ề
+##ể
+##ễ
+##ệ
+##ị
+##ọ
+##ố
+##ồ
+##ổ
+##ộ
+##ớ
+##ờ
+##ợ
+##ụ
+##ủ
+##ứ
+##ừ
+##ử
+##ữ
+##ự
+##ỳ
+##ỹ
+##ἀ
+##ἐ
+##ὁ
+##ὐ
+##ὰ
+##ὶ
+##ὸ
+##ῆ
+##ῖ
+##ῦ
+##ῶ
+##‐
+##‑
+##‒
+##–
+##—
+##―
+##‖
+##‘
+##’
+##‚
+##“
+##”
+##„
+##†
+##‡
+##•
+##…
+##‰
+##′
+##″
+##⁄
+##⁰
+##ⁱ
+##⁴
+##⁵
+##⁶
+##⁷
+##⁸
+##⁹
+##⁻
+##ⁿ
+##₅
+##₆
+##₇
+##₈
+##₉
+##₊
+##₍
+##₎
+##ₐ
+##ₑ
+##ₒ
+##ₓ
+##ₕ
+##ₖ
+##ₘ
+##ₚ
+##ₛ
+##ₜ
+##₤
+##€
+##₱
+##₹
+##ℓ
+##№
+##ℝ
+##⅓
+##←
+##↑
+##→
+##↔
+##⇌
+##⇒
+##∂
+##∈
+##∗
+##∘
+##√
+##∞
+##∧
+##∨
+##∩
+##∪
+##≈
+##≠
+##≡
+##≤
+##≥
+##⊂
+##⊆
+##⊕
+##⋅
+##─
+##│
+##■
+##●
+##★
+##☆
+##☉
+##♠
+##♣
+##♥
+##♦
+##♯
+##⟨
+##⟩
+##ⱼ
+##、
+##。
+##《
+##》
+##「
+##」
+##『
+##』
+##〜
+##い
+##う
+##え
+##お
+##か
+##き
+##く
+##け
+##こ
+##さ
+##し
+##す
+##せ
+##そ
+##た
+##ち
+##つ
+##て
+##と
+##な
+##に
+##の
+##は
+##ひ
+##ま
+##み
+##む
+##め
+##も
+##や
+##ゆ
+##よ
+##ら
+##り
+##る
+##れ
+##ん
+##ア
+##ィ
+##イ
+##ウ
+##エ
+##オ
+##カ
+##ガ
+##キ
+##ク
+##グ
+##コ
+##サ
+##シ
+##ジ
+##ス
+##ズ
+##タ
+##ダ
+##ッ
+##テ
+##デ
+##ト
+##ド
+##ナ
+##ニ
+##ハ
+##バ
+##パ
+##フ
+##ブ
+##プ
+##マ
+##ミ
+##ム
+##ャ
+##ュ
+##ラ
+##リ
+##ル
+##レ
+##ロ
+##ン
+##・
+##ー
+##一
+##三
+##上
+##下
+##中
+##事
+##二
+##井
+##京
+##人
+##亻
+##仁
+##佐
+##侍
+##光
+##公
+##力
+##北
+##十
+##南
+##原
+##口
+##史
+##司
+##吉
+##同
+##和
+##囗
+##国
+##國
+##土
+##城
+##士
+##大
+##天
+##太
+##夫
+##女
+##子
+##宀
+##安
+##宮
+##宿
+##小
+##尚
+##山
+##島
+##川
+##州
+##平
+##年
+##心
+##愛
+##戸
+##文
+##新
+##方
+##日
+##明
+##星
+##書
+##月
+##木
+##本
+##李
+##村
+##東
+##松
+##林
+##正
+##武
+##氏
+##水
+##氵
+##江
+##河
+##海
+##版
+##犬
+##王
+##生
+##田
+##白
+##皇
+##省
+##真
+##石
+##社
+##神
+##竹
+##美
+##義
+##花
+##藤
+##西
+##谷
+##車
+##辶
+##道
+##郎
+##郡
+##部
+##野
+##金
+##長
+##門
+##陽
+##青
+##食
+##馬
+##高
+##龍
+##龸
+##사
+##씨
+##의
+##이
+##한
+##fi
+##fl
+##！
+##（
+##）
+##，
+##－
+##／
+##：