BGG Blog MD.md

*Auto-BG: The Board Game Concept Generator*

1.  [Introduction]{.underline}

    a.  [Goals Statement - design aid/improved user control over
        > generation]{.underline}

**A Gentle Introduction to Auto-BG & Board Game Data**

**What is Auto-BG?**

How does a board game transform from an idea to physically sitting on
your table?

This application attempts to augment one step, early in that journey,
when the seeds of an idea combine and sprout into a holistic concept. By
interpreting disparate mechanical and descriptive tags to identify a
game concept, Auto-BG uses a custom pipeline of GPT3 and T5 models to
create a new description and proposed titles for a game that doesn't
exist today. These descriptions support designers-to-be as alternatives
to current concepts, seeds for future concepts, or any user as,
hopefully, an entertaining thought experiment.

While, overall, ChatGPT solves this application case by generating
coherent descriptions from sentence-formed prompts, we believe this
design niche would be better served by a dedicated application
leveraging domain-specific transfer learning and granular control
through an extensive tag-prompt framework.

Before digging into the process of generating a board game concept,
let's ground some key terms:

-   *Mechanical* - The gameplay mechanics which, together, create a
    > distinct ruleset for a given game. Represented here by a class of
    > tags each capturing individual mechanics such as "Worker
    > Placement" or "Set Collection".

    -   *Cooperative* - Our application, and the dataset, singles out
        > this mechanic as being important enough to have its own tag
        > class. As an alternative to the domain-default of
        > *Competitive* gaming, *Cooperative* represents both an
        > individual mechanic and an entire design paradigm shift.

-   *Descriptive* - In this context, the narrative, production, or
    > family-connective elements of a game. This can include genre,
    > categorical niches, or any relational tag not captured by the
    > class of mechanical tags described above.

    a.  [Orientation to the data - understanding how assembled tags
        > define a game item]{.underline}

**Understanding the Data**

To train our models, we utilized a processed dataset of 20,769 ranked
board games scraped from the board game database & forum
[[BoardGameGeek.com]{.underline}](https://boardgamegeek.com/)^1^. Each
game includes its name, description, and 3,800 feature tags separated
into five classes including Cooperative, Game Type, Category, Mechanic,
and Family; when you select tags within Auto-BG, you choose and assign
them within these distinct classes. To be ranked, these games must have
received a lifetime minimum of thirty user ratings; non-standalone
expansion and compilation game items have also been removed to reduce
replication in the training data.

As previous studies^2^ identified a disproportionate bias toward english
language descriptions, our approach used a soft pass to remove games
identified as having non-english descriptions using langdetect^3^. We
also implemented a check on all remaining titles to remove any that used
non-english characters. In total, this purged 1,423 or \~6.4% of the
data.

The transformed data powering AutoBG was originally scraped via the BGG
XML API by \@mshepard on GitLab^4^ and customized for The Impact of
Crowdfunding on Board Games (2022)^5^ by N. Canu, A. Oldenkamp, and J.
Ellis & Deconstructing Game Design (2022)^2^ by N. Canu, K. Chen, and R.
Shetty. That work, and Auto-BG, are derivatively licensed under
[[Creative Commons
Attribution-NonCommercial-ShareAlike]{.underline}](http://creativecommons.org/licenses/by-nc-sa/3.0/).
Our GitHub repository includes a python script collating the processing
steps required to turn a raw scraper output file into the final data
constructs needed to run an instance of Auto-BG.

b.  [Ethical considerations in domain - inherited biases from both
    > generative model and BGG framework, role of deep learning as
    > supplement/replacement]{.underline}

**Ethical Considerations**

As a chaotician once said "your scientists were so preoccupied with
whether or not they could, they didn\'t stop to think if they
should"^6^ - but that's not what we're going to do. So before we
continue, a few notes regarding potential biases and concerns when
training or using Auto-BG.

First and foremost, Auto-BG inherits critical biases from the base GPT3
Curie model. Extensive work exists documenting the model family's biases
related to race^7^, gender^7,8^, religion^9^, and abstracted moral
frameworks on right and wrong^10^. A key vector for producing biased
output from these models comes from the conditioning context (i.e. user
input) which prompts a response^11^; while these inherited biases can't
be eliminated, AutoBG attempts to reduce their impact on output text
through strictly controlling the conditioning context by relating
unknown input to pre-approved keys.

Additionally, Auto-BG has inherited significant bias in generating
gender markers within descriptive text from the BoardGameGeek training
data. As seen in figure 1, game descriptions include a disproportionate
volume of male-associated gender markets in the source data. This
transfers through Auto-BG's pipeline, resulting in output that, as in
figure 2, retains a similar distribution. While outside the feasible
scope of this project, future iterations of Auto-BG should include
coreference cleaning on the training input to minimize inherited bias;
this approach would require a robust review of the source data through
the framework of coreference resolution to avoid further harms by making
inferences that fail to acknowledge the complexities of gender within
this data^12^.

Beyond these substantial issues, with any large language model (LLM)
project, we have to ask - are we harming human actors by deploying this
tool? In 2023, discussion of the prospective gains and potential danger
of LLMs has broken out of the data science domain and represents a
growing global conversation. Academics wrestle with the scope and
limitations of GPT models^13^ while crossover publications debate topics
such as the merits of ChatGPT as an author^14^ and which professions AI
models might replace in the near future^15^.

With these concerns in mind, we designed Auto-BG as a strictly
augmentative tool - a board game is more than its conceptual description
and while Auto-BG may suggest rule frameworks based on your input, it
can't generate any of the production elements such as full rulesets or
art needed to take the game from concept to table. In addition, we
strongly encourage the use of Auto-BG as a starting point, not a ground
truth generation. The model makes mistakes that a human actor may not;
while we have implemented catches to discourage it from replicating
existing board games, its GPT3 trained core has an extensive knowledge
of gaming in general. We recommend reviewing output text thoroughly
before committing to your new board games \[maybe insert pictures of it
generating these?\] *Age of Empires II: Age of Kings* or *Everquest^©^*.

c.  [Overall generator framework - birds eye graphic/discussion of
    > pipeline]{.underline}

**What's in a pipeline?**

Auto-BG relies on multiple LLMs to generate the full game concept from
your initial input. The entire process from turning your tags into a
prompt to presenting your finished concept is embedded in a pipeline, or
a framework of related code running sequentially, that enables these
distinct steps to work in unison. It all starts with an
idea...![](media/image1.png){width="0.2760422134733158in"
height="0.44166666666666665in"}

1.  The most complex participant in Auto-BG's pipeline (that's you)
    > translates different elements of a vague concept for a game into
    > defined tags through the Auto-BG interface.

2.  An input parser collates the tags, turning them into a game vector
    > (a binary, or one-hot, representation of a game where all selected
    > tags are valued at 1). This vector activates the internal keys
    > representing each tag selected in step 1 and passes them as a
    > prompt to Auto-BG's GPT3 model.

3.  A generator function calls the model through the OpenAI API,
    > returning a response using selected parameters for the input
    > prompt. This response is decoded and lightly cleaned to remove
    > trailing sentences for readability.

4.  The cleaned output is passed downstream to a local t5 trained
    > sequence-to-sequence model, it uses the output as a prompt to
    > translate to potential candidate titles.

5.  Because the GPT3 model has learned that games should include a
    > title, it may include an artificial title already. The title
    > generator runs once and if any candidate titles are identified in
    > the text, it strips them out for a placeholder before running
    > again.

6.  A title selector rejoins the initial and secondary generations,
    > removes duplicates, validates against existing game titles, then
    > scores the cosine similarity of each title with its reference
    > input. It attaches the highest scoring title to the output and
    > fills all placeholders, storing backups of the unchosen titles.

7.  The Auto-BG user interface returns the generated prompt with its
    > title for you to give feedback on, review alternate titles, or
    > save for future use.

At the end of the pipeline, you have a brand-new game concept customized
by your selected tags! And if you're not happy with the output, running
the generator again will produce a new description given the same tags.

To understand Auto-BG better, let's go through each step in detail,
looking at how an example prompt is transformed into its final output.

2.  [Pipeline Ride Along - How does "your" input become a
    > description/title pair?]{.underline}

**The Inner Workings of Auto-BG w/Examples**

a.  *The Input Step*

***Interpreting User Input***

i.  [User input as a "game" & input design - Building off 1b, cover
    > design choices like dividing feature classes, uncaptured features
    > in training, bridge to input selection.]{.underline}

*How do individual tags add up to a game profile?*

This key question guided every aspect of Auto-BGs design principles. The
complex series of transformations from input to generation required
understanding both how a human reader would interact and interpret these
tags but also how the GPT3 LLM at the heart of Auto-BG could best learn
them.

Auto-BG inherits tag classes from BoardGameGeek, if you go to any game
page, you'll see a collection of type, category, mechanic, and family
tags on the sidebar. We decided Auto-BG 1.0 would focus explicitly on
these four primary classes as they translated coherently to formatting
for a GPT3 prompt training structure, could be universally applied to an
unknown game, and don't rely on post-design information like user
recommendations or publisher data.

*What happens when you choose tags in Auto-BG?*

Inside Auto-BG, each tag has a hidden class prefix that denotes it as a
member of that class and tracks its affiliation throughout the pipeline.
This means when you select a new tag, even if it\'s semantically similar
to another overlapping tag in a different class, Auto-BG remembers its
associations; in this approach, Auto-BG effectively handles unknown
inputs by scanning the associated class, matching the unknown tag to its
closest existing sibling.

By grouping all of your selected tags, Auto-BG creates an approximate
profile of a game to generate. When generating your new concept, Auto-BG
tries to infer additional features based on the tags provided. Some
features like player count, age rating, and playtime may still show up
in your concept if Auto-BG thinks they should be included in the text.
By changing even a single tag out, Auto-BG will try to accommodate that
new design feature and generate a different concept!

With this sensitivity to the prompt selection, users must understand how
their choices impact downstream generation. Auto-BG needed an intuitive
user experience with coherent tutorials throughout to streamline this
process, alongside some targeted limits on profile creation.

ii. [UI design considerations - Making tags easily accessible w/volume
    > of options, input tutorial design and approach to creating min/max
    > tag caps.]{.underline}

##\# UI Input Discussion

iii. [Vectorizing user inputs - text key to vector transformation and
     > syncing with the ground truth key list. Approach to handling
     > unknown user input in tag selection.]{.underline}

     1.  Converting to final input - Need for a consistent prompt
         > structure and design considerations to improve GPT3 prompt
         > interpretation, and discuss autonomous generation from tags
         > vs free prompt RE user's end goal. \<- joining these together
         > for formatting.

Once you've created your new game profile, Auto-BG translates it into a
format that its LLM can understand for text generation. Ultimately,
Auto-BG tries to convert your profile tags following OpenAI's
recommended best practices for conditional generation prompts^16,17^.
This results in a text prompt version of your tag list with each tag
period stopped like so:\
\
##\# example here\
\
Fine-tune training has taught Auto-BGs model that this specific format
of text prompt should translate to a descriptive paragraph of the
associated tags. To create it, Auto-BG passes your tags through an input
manager that performs several steps in sequence before sending a
polished prompt to the generator.

First, Auto-BG establishes a ground truth key dictionary of all existing
tags. This format allows for iterative updating from future data, as new
tags appear on BoardGameGeek, Auto-BG can update to include them. This
python dictionary includes a key for every tag with the equivalent value
set to 0 - tracking that it does not appear. Eventually, it will change
the values for your selected tags to 1, telling Auto-BG that they appear
and should be added to the prompt, but first, it tries to correct for
unknown inputs.

Each tag not recognized as being present in the key dictionary moves to
a bucket matching its attached prefix. With these marked as unknown, the
input manager implements a within-class comparison between the unknown
tag and all known tags that share its prefix. Auto-BG estimates semantic
relevance between tags through token cosine similarity^18^, comparing
the unknown tag to each candidate with SpaCy's token similarity
method^19^. Once it evaluates this for all candidates, it adds the
highest scoring option for each unknown tag to the remaining input tags.

Auto-BGs underlying model inherits the characteristic sensitivity to
prompt design of LLMs; prompt design often requires significant human
effort, and many approaches have been suggested to address this
challenge. Our design philosophy focused on improving the quality and
control of prompts with goals of reliability, creativity, and coherence
to input, specifically inspired by concepts of metaprompting^20^ and
internal prompt generation within the LLM^21^. Auto-BG leverages
existing human tagging work done within the source data to provide the
generator model with a controllable abstracted prompt instead of a
potentially complex natural language sentence-structured prompt.

To pass the now cleaned input to the generator, Auto-BGs' input manager
converts the pooled tags into a text string as above, with each tag key
turned into a period-stopped sentence. The generator will send this
prompt to a remote model through the OpenAI API and return a matching
description!

b.  *The Generation Step*

***Generating a Description***

i.  Approach to model selection - comparative performance of GPT3 models
    > vs cost, tradeoff with controllable output and quality.

ii. Training methodology - literature discussion on optimizing fine tune
    > LLM training for generation relative to other tasks
    > (classification in particular).

iii. Initial output generation - limited key parameters in API
     > generation, tuning strategy, refining with validation set.

     1.  Challenges w/GPT3 - Discuss sensitivity to prompt construction
         > and changes in keys & difficulty of low volume references for
         > keys w/source data. Outside knowledge polluting output (i.e.
         > using video games as names in text). Testing for prompt
         > memorization (increasing chaos in output for learned
         > keysets).

iv. Output processing - Increased text cleanup for user output tasks,
    > approaches to mitigating potentially subpar generation from GPT 3
    > including embedded titles and excessive referencing from BGG
    > (incorrect publishers/designers included in text), challenge of
    > removing references again.

```{=html}
<!-- -->
```
c.  *The Title Step*

***A Fitting Title***

i.  Text to title relationship - Domain considerations w/title as a
    > product, distinct relationship to train on. Role of title in text
    > as a key point when running models in sequence.

To complete the game concept, Auto-BG provides a selection of titles
chosen to fit the newly generated description. But what defines a good
title? It depends on the domain and it depends on the product! A sense
of tension may fit a thriller novel^22^ and sticking to positive
associations enhances commercial products^23^, but where do board games
fit in? To create titles that both fit their description and matched
domain trends as a whole, we needed a model that could learn the unique
relationship between title and description in board gaming.

ii. Transformation approach - seq2seq advantages for this task & model
    > selection, advantages of extending transfer learning from domain
    > to domain (news headlines to game titles)

While our generator uses the prompt-based GPT paradigm, we decided to
take a different approach for title generation inside Auto-BG. T5 models
fit within a unified framework for transfer learning known as
sequence-to-sequence or text-to-text^24^; at their core, they're an
encoder-decoder model operating by treating all NLP operations as a
translation task. This means that given text as input, the model will
attempt to output a predicted target that best fits the training data.

Transfer learning, in machine learning, applies a model trained on one
task or domain and performs additional training to apply the model to a
different task. The total corpus of descriptions for BGG, approximately
20,000 descriptions, is inadequate for fresh training of a generative
NLP model; instead Auto-BG leverages two-stages of upstream transfer
learning through HuggingFace's Transformer library. The final
implementation extends a model^25^ already fine-tuned on an additional
500,000 news articles from t5-base.

1.  Relative model performance - approach to model selection based on
    > above including comparative metrics. Discuss aligning metrics
    > w/goal of tool + need for human review.

To scope the best generator for Auto-BG, we trained a sequence of models
on both t5-base^26^ and the headline-trained model. Beyond training
baseline models, we looked toward work by Tay, et al.^27^ on scaling and
fine-tune training for transformers to guide selecting training
parameters; the final round of testing utilized a higher learning rate
of .001 while introducing weight decay in the optimizer.

With several models trained

iii. Low-cost iterative generation - two-pass competitive scoring on
     > multi-output, using the title generator to select and replace
     > low-quality titles then scoring the maximum pool of two
     > generations.

     1.  Scoring approach - title output cleanup and scoring final title
         > to body text semantic similarity.

```{=html}
<!-- -->
```
d.  *Final Output*

    i.  Rerunning in real-time - approaches to adjusting output
        > including updating text references, finding text generator
        > default "profiles" through hyperparameter testing, and
        > implementation of update processes in the UI.

    ii. The value of user feedback - considerations for future
        > performance w/tuning generation settings, content reporting
        > for title cleanup or problematic text content.

```{=html}
<!-- -->
```
3.  [Discussion & Conclusion]{.underline}

    a.  What's the point? - Practical design tool / Interpreting how
        > "mechanical" changes interact with language descriptions /
        > Extending user interactivity beyond prompts

    b.  Discuss methodology - user interactivity through prompt control,
        > did it work & other potential approaches to same application
        > (reinforcement, different prompt structures)

    c.  Alternative applications - fine-tune human-written desc
        > (training on extended prompt + more deterministic output),
        > rules generation?

    d.  Future extensions/limitations - Expand on language scope, live
        > data + training, discarded feature classes / discussion on
        > approach not including these currently

    e.  What next? - Encourage readers to experiment with tool + give
        > feedback in app

4.  [Extras]{.underline}

    a.  Work Statement + Appendices + Citations

**References**

1.  "BoardGameGeek." Accessed March 19, 2023.
    > [[https://boardgamegeek.com/]{.underline}](https://boardgamegeek.com/).

2.  Canu, Nicholas, Kuan Chen, and Rhea Shetty. "Deconstructing Game
    > Design." Jupyter Notebook, October 20, 2022.
    > [[https://github.com/canunj/deconstructing_games]{.underline}](https://github.com/canunj/deconstructing_games).

3.  Danilák, Michal. "Langdetect." Python, March 15, 2023.
    > [[https://github.com/Mimino666/langdetect]{.underline}](https://github.com/Mimino666/langdetect).

4.  GitLab. "Recommend.Games / Board Game Scraper · GitLab," March
    > 10, 2023.
    > [[https://gitlab.com/recommend.games/board-game-scraper]{.underline}](https://gitlab.com/recommend.games/board-game-scraper).

5.  Canu, Nicholas, Jonathan Ellis, and Oldenkamp, Adam. "The Impact of
    > Crowdfunding on Board Games." Jupyter Notebook, May 20, 2022.
    > [[https://github.com/canunj/BGG_KS_Analysis]{.underline}](https://github.com/canunj/BGG_KS_Analysis/blob/92452d7e7b174bf45763e469b1ed4ce61a84b7ba/The%20Impact%20of%20Crowdfunding%20on%20Board%20Games.pdf).

6.  Spielberg, Steven dir. *Jurassic Park*. 1993; Universal City, CA:
    > Universal Studios, 2022. UHD Blu Ray.

7.  Chiu, Ke-Li, Annie Collins, and Rohan Alexander. "Detecting Hate
    > Speech with GPT-3." arXiv, March 24, 2022.
    > [[https://doi.org/10.48550/arXiv.2103.12407]{.underline}](https://doi.org/10.48550/arXiv.2103.12407).

8.  Lucy, Li, and David Bamman. "Gender and Representation Bias in GPT-3
    > Generated Stories." In *Proceedings of the Third Workshop on
    > Narrative Understanding*, 48--55. Virtual: Association for
    > Computational Linguistics, 2021.
    > [[https://doi.org/10.18653/v1/2021.nuse-1.5]{.underline}](https://doi.org/10.18653/v1/2021.nuse-1.5).

9.  Abid, Abubakar, Maheen Farooqi, and James Zou. "Persistent
    > Anti-Muslim Bias in Large Language Models." In *Proceedings of the
    > 2021 AAAI/ACM Conference on AI, Ethics, and Society*, 298--306.
    > AIES '21. New York, NY, USA: Association for Computing
    > Machinery, 2021.
    > [[https://doi.org/10.1145/3461702.3462624]{.underline}](https://doi.org/10.1145/3461702.3462624).

10. Schramowski, Patrick, Cigdem Turan, Nico Andersen, Constantin A.
    > Rothkopf, and Kristian Kersting. "Large Pre-Trained Language
    > Models Contain Human-like Biases of What Is Right and Wrong to
    > Do." *Nature Machine Intelligence* 4, no. 3 (March 2022): 258--68.
    > [[https://doi.org/10.1038/s42256-022-00458-8]{.underline}](https://doi.org/10.1038/s42256-022-00458-8).

11. Huang, Po-Sen, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes
    > Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli.
    > "Reducing Sentiment Bias in Language Models via Counterfactual
    > Evaluation." arXiv, October 8, 2020.
    > [[https://doi.org/10.48550/arXiv.1911.03064]{.underline}](https://doi.org/10.48550/arXiv.1911.03064).

12. Cao, Yang Trista, and Hal Daumé III. "Toward Gender-Inclusive
    > Coreference Resolution." In *Proceedings of the 58th Annual
    > Meeting of the Association for Computational Linguistics*,
    > 4568--95, 2020.
    > [[https://doi.org/10.18653/v1/2020.acl-main.418]{.underline}](https://doi.org/10.18653/v1/2020.acl-main.418).

13. Floridi, Luciano, and Massimo Chiriatti. "GPT-3: Its Nature, Scope,
    > Limits, and Consequences." *Minds and Machines* 30, no. 4
    > (December 1, 2020): 681--94.
    > [[https://doi.org/10.1007/s11023-020-09548-1]{.underline}](https://doi.org/10.1007/s11023-020-09548-1).

14. Thorp, H. Holden. "ChatGPT Is Fun, but Not an Author." *Science*
    > 379, no. 6630 (January 27, 2023): 313--313.
    > [[https://doi.org/10.1126/science.adg7879]{.underline}](https://doi.org/10.1126/science.adg7879).

15. Zinkula, Aaron Mok, Jacob. "ChatGPT May Be Coming for Our Jobs. Here
    > Are the 10 Roles That AI Is Most Likely to Replace." Business
    > Insider. Accessed March 19, 2023.
    > [[https://www.businessinsider.com/chatgpt-jobs-at-risk-replacement-artificial-intelligence-ai-labor-trends-2023-02]{.underline}](https://www.businessinsider.com/chatgpt-jobs-at-risk-replacement-artificial-intelligence-ai-labor-trends-2023-02).

16. "Fine-Tuning." Accessed March 23, 2023.
    > [[https://platform.openai.com/docs/guides/fine-tuning/conditional-generation]{.underline}](https://platform.openai.com/docs/guides/fine-tuning/conditional-generation).

17. "Best Practices for Prompt Engineering with OpenAI API \| OpenAI
    > Help Center." Accessed March 26, 2023.
    > [[https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api]{.underline}](https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api).

18. Gunawan, D., C. A. Sembiring, and M. A. Budiman. "The Implementation
    > of Cosine Similarity to Calculate Text Relevance between Two
    > Documents." *Journal of Physics: Conference Series* 978, no. 1
    > (March 2018): 012120.
    > [[https://doi.org/10.1088/1742-6596/978/1/012120]{.underline}](https://doi.org/10.1088/1742-6596/978/1/012120).

19. Token. "Token · SpaCy API Documentation." Accessed March 26, 2023.
    > [[https://spacy.io/api/token#similarity]{.underline}](https://spacy.io/api/token#similarity).

20. Reynolds, Laria, and Kyle McDonell. "Prompt Programming for Large
    > Language Models: Beyond the Few-Shot Paradigm." arXiv, February
    > 15, 2021.
    > [[https://doi.org/10.48550/arXiv.2102.07350]{.underline}](https://doi.org/10.48550/arXiv.2102.07350).

21. Zhou, Yongchao, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster,
    > Silviu Pitis, Harris Chan, and Jimmy Ba. "Large Language Models
    > Are Human-Level Prompt Engineers." arXiv, March 10, 2023.
    > [[https://doi.org/10.48550/arXiv.2211.01910]{.underline}](https://doi.org/10.48550/arXiv.2211.01910).

22. MasterClass. "How to Write a Book Title in 7 Tips: Create the Best
    > Book Title - 2023." Accessed March 30, 2023.
    > [[https://www.masterclass.com/articles/how-to-write-a-book-title-in-7-tips-create-the-best-book-title]{.underline}](https://www.masterclass.com/articles/how-to-write-a-book-title-in-7-tips-create-the-best-book-title).

23. Qualtrics. "How to Find the Perfect Product Name in 2023." Accessed
    > March 30, 2023.
    > [[https://www.qualtrics.com/experience-management/product/product-naming/]{.underline}](https://www.qualtrics.com/experience-management/product/product-naming/).

24. Raffel, Colin, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan
    > Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.
    > "Exploring the Limits of Transfer Learning with a Unified
    > Text-to-Text Transformer." arXiv, July 28, 2020.
    > [[http://arxiv.org/abs/1910.10683]{.underline}](http://arxiv.org/abs/1910.10683).

25. "Michau/T5-Base-En-Generate-Headline · Hugging Face." Accessed March
    > 30, 2023.
    > [[https://huggingface.co/Michau/t5-base-en-generate-headline]{.underline}](https://huggingface.co/Michau/t5-base-en-generate-headline).

26. "T5-Base · Hugging Face," November 3, 2022.
    > [[https://huggingface.co/t5-base]{.underline}](https://huggingface.co/t5-base).

27. Tay, Yi, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar,
    > Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and
    > Donald Metzler. "Scale Efficiently: Insights from Pre-Training and
    > Fine-Tuning Transformers." arXiv, January 30, 2022.
    > [[https://doi.org/10.48550/arXiv.2109.10686]{.underline}](https://doi.org/10.48550/arXiv.2109.10686).

28.