content: text: plexity when compared to MoE. This shift in ap-
proach offers a unique and more straightforward
way of working, moving away from conventional
MoE challenges. Not only that, DUS also under-
goes continued pretraining to quickly recover per-
formance of the scaled model.

content: text: B.3 Prompt Engineering

content: text: A key research area to harness the emergent abil-
ities of LLMs is prompt engineering. Prompt en-
gineering is the study of how to design inputs
(prompts) that enable LLMs to better perform spe-
cific tasks. A prime example of this research
is Chain-of-Thought (CoT) (Wei et al., 2022b),
which proposes CoT prompting that decomposes
multi-step problems into a series of intermedi-
ate reasoning steps. Moreover, efforts are under-
way to replace even such prompt engineering with
LLMs (Yang et al., 2023).

content: text: B.4 Instruction Tuning

content: text: To enhance the steerability of LLMs, instruction
tuning (Wei et al., 2021) has emerged as a learning
technique. This involves fine-tuning LLMs using
data formatted as (instruction, input, output) for
various tasks (Wang et al., 2022). Instruction tuning
allows for targeted adjustments, providing a more
controlled and task-oriented improvement to the
model's capabilities.

content: text: Before instruction tuning, existing methods
faced challenges in effectively guiding and control-
ling the behavior of large language models (Zhang
et al., 2023b). The sheer complexity of these mod-
els made it difficult to ensure precise and task-
oriented responses. The need for a more targeted
approach arose from the limitations of existing
methods, leading to the development of instruc-
tion tuning. This targeted approach enables better
control over the model's behavior, making it more
suitable for specific tasks and improving its overall
performance in alignment with user-defined objec-
tives. Therefore, instruction tuning is computation-
ally efficient and facilitates the rapid adaptation
of LLMs to a specific domain without requiring
extensive retraining or architectural changes.

content: text: B.5 Alignment Tuning

content: text: LLM has been observed to generate sentences that
may be perceived as linguistically incongruent by
human readers since they learned not human inten-
tion, but only vast knowledge across various do-
mains in the pretraining step (Ziegler et al., 2019).

content: text: To overcome this limitation and align with human
intentions, previous research (Ziegler et al., 2019)
have proposed Reinforcement Learning with Hu-
man Feedback (RLHF). RLHF operates by learning
a reward model based on human preferences, em-
ploying reinforcement learning to guide the LLM
towards prioritizing answers with the highest re-
ward scores. This process enhances the safety,
propriety, and overall quality of the generated re-
sponses. Despite demonstrating satisfactory per-
formance, RLHF encounters challenges such as
managing numerous hyperparameters and necessi-
tating the incorporation of multiple models (policy,
value, reward, and reference models).

content: text: In response to these challenges, the supervised
fine-tuning based approaches have proposed, such
as Rank Responses to align Human Feedback
(RRHF) (Yuan et al., 2023), Reward rAnked Fine-
Tuning (RAFT) (Dong et al., 2023), and Direct
Policy Optimization (DPO) (Intel, 2023). They
avoid the complexities associated with reinforce-
ment learning while achieving empirical perfor-
mance comparable to RLHF. Among them, DPO
that we used directly guides the LLM to increase
the probability of positive responses and decrease
the probability of negative responses through a "di-
rect" approach. Interestingly, DPO demonstrates
more stable learning results compared to RLHF,
despite its simple training approach.

content: text: B.6 Data Contamination

content: text: Recent researches (Zhou et al., 2023; Sainz et al.,
2023; Golchin and Surdeanu, 2023; Deng et al.,
2023) emphasize the need to measure whether a
specific benchmark was used to train the large lan-
guage models. There are three types of the data
contamination: guideline, raw text and annota-
tion (Sainz et al., 2023). Guideline contamination
occurs when a model accesses detailed annotation
guidelines for a dataset, providing advantages in
specific tasks, and its impact should be considered,
especially in zero and few-shot evaluations. Raw
text contamination occurs when a model has ac-
cess to the original text. Wikipedia is widely used
as a pretraining data, but also as a source for cre-
ating new datasets. The caution is advised in the
development of automatically annotated datasets
sourced from the web. Annotation contamina-
tion occurs when the annotations of the specific
benchmark are exposed during model training.