Update the output preview.

README.md

@@ -10,19 +10,16 @@ python_version: 3.10.10
 
 # Auto-Draft: 文献整理辅助工具
 
-这个项目旨在辅助AI领域学术论文的文献整理。提供如下功能：
-* 主题归纳： 自动搜索相关文献，总结相关工作。
-* 真实引用：所有引用的文献都有对应原文，避免AI胡说八道。
-* 生成LaTeX模板： 输出结果一键编译。  
+这个项目旨在轻松快捷的生成学术论文! 帮助你解决下面的问题:
+* 自动搜索相关文献, 提供真实有出处的引用.
+* 自动生成LaTeX模板, 为图表和算法预留出位置. 只需要在对应位置填入内容就能得到完整论文. 
 
-运行过程需要能使用GPT-4的API Key. 生成一篇论文需要15000 Tokens(大约0.5到0.8美元). 总共过程需要大约十分钟. 
-
-# 体验地址  
-以下链接提供简单功能的免费体验. 如果需要更定制化的功能, 请参照*使用方法*进行本地部署和自行修改.
+# Huggingface Space  
+项目对硬件要求低. 在Huggingface Space上即可流畅运行：
 
 https://huggingface.co/spaces/auto-academic/auto-draft
 
-# 使用方法 
+# 部署方法 
 1. 克隆此仓库：
 ```angular2html
 git clone https://github.com/CCCBora/auto-draft
@@ -31,14 +28,17 @@ git clone https://github.com/CCCBora/auto-draft
 ```angular2html
 pip install -r requirements.txt
 ```
-3. 在环境变量中设定OPENAI_API_KEY。
-4. 编辑`auto_backgrounds.py`以自定义想要探索的主题和描述，然后运行
+3. 在环境变量中设定OPENAI_API_KEY. 
+4. 编辑`auto_backgrounds.py`以自定义论文标题, 然后运行
 ```angular2html
 python auto_backgrounds.py
 ```
 
+# 修改Prompts
+如果希望对生成内容有更多的控制, 可以修改`prompts/instructions.json`中对每个章节的指导. 
+
 # 示例
-`outputs` 文件夹中提供了部分输入的原始输出. 经由Overleaf直接编译得到. 也可以查看本目录下的sample-output.pdf.
+`outputs` 文件夹中提供了部分输入的原始输出. 经由Overleaf直接编译得到. 也可以查看本目录下的`Playing_Atari_with_Deep_Reinforcement_Learning.pdf`.
 
 Page 1            |  Page 2
 :-------------------------:|:-------------------------:

auto_backgrounds.py

@@ -2,11 +2,10 @@ import os.path
 from utils.references import References
 from utils.file_operations import hash_name, make_archive, copy_templates
 from utils.tex_processing import create_copies
-from section_generator import keywords_generation, section_generation # figures_generation, section_generation_bg,
+from section_generator import keywords_generation, section_generation  # figures_generation, section_generation_bg,
 import logging
 import time
 
-
 TOTAL_TOKENS = 0
 TOTAL_PROMPTS_TOKENS = 0
 TOTAL_COMPLETION_TOKENS = 0
@@ -49,9 +48,7 @@ def _generation_setup(title, description="", template="ICLR2022", tldr=False,
                                for the collected papers. Defaults to False.
         max_kw_refs (int, optional): The maximum number of references that can be associated with each keyword.
                                      Defaults to 10.
-        max_num_refs (int, optional): The maximum number of references that can be included in the paper.
-                                      Defaults to 50.
-        bib_refs (list, optional): A list of pre-existing references in BibTeX format. Defaults to None.
+        bib_refs (path to a bibtex file, optional).
 
     Returns:
     tuple: A tuple containing the following elements:
@@ -65,7 +62,7 @@ def _generation_setup(title, description="", template="ICLR2022", tldr=False,
 
     # Create a copy in the outputs folder.
     bibtex_path, destination_folder = copy_templates(template, title)
-    logging.basicConfig(level=logging.INFO, filename=os.path.join(destination_folder, "generation.log") )
+    logging.basicConfig(level=logging.INFO, filename=os.path.join(destination_folder, "generation.log"))
 
     # Generate keywords and references
     # print("Initialize the paper information ...")
@@ -74,7 +71,7 @@ def _generation_setup(title, description="", template="ICLR2022", tldr=False,
     log_usage(usage, "keywords")
 
     # generate keywords dictionary # todo: in some rare situations, collected papers will be an empty list.
-    keywords = {keyword:max_kw_refs for keyword in keywords}
+    keywords = {keyword: max_kw_refs for keyword in keywords}
 
     ref = References(title, bib_refs)
     ref.collect_papers(keywords, tldr=tldr)
@@ -87,8 +84,8 @@ def _generation_setup(title, description="", template="ICLR2022", tldr=False,
     paper["references"] = ref.to_prompts(max_tokens=max_tokens)
     paper["body"] = paper_body
     paper["bibtex"] = bibtex_path
-    return paper, destination_folder, all_paper_ids #todo: use `all_paper_ids` to check if all citations are in this list
-
+    return paper, destination_folder, all_paper_ids
+    # todo: use `all_paper_ids` to check if all citations are in this list
 
 
 def generate_backgrounds(title, description="", template="ICLR2022", model="gpt-4"):
@@ -110,21 +107,47 @@ def generate_backgrounds(title, description="", template="ICLR2022", model="gpt-
     return make_archive(destination_folder, filename)
 
 
-
 def generate_draft(title, description="", template="ICLR2022",
                    tldr=True, max_kw_refs=10, sections=None, bib_refs=None, model="gpt-4"):
+    """
+    This function generates a draft paper using the provided information; it contains three steps: 1. Pre-processing:
+    Initializes the setup for paper generation and filters the sections to be included in the paper. 2. Processing:
+    Generates each section of the paper. 3. Post-processing: Creates backup copies of the paper and returns the paper
+    in a zipped format.
+
+    Parameters:
+        title (str): The title of the paper.
+        description (str, optional): A short description or abstract for the paper. Defaults to an empty string.
+        template (str, optional): The template to be used for paper generation. Defaults to "ICLR2022".
+        tldr (bool, optional): A flag indicating whether a TL;DR (Too Long; Didn't Read) summary should be used
+                               for the collected papers. Defaults to True.
+        max_kw_refs (int, optional): The maximum number of references that can be associated with each keyword.
+                                     Defaults to 10.
+        sections (list, optional): The sections to be included in the paper. If not provided, all the standard
+                                   sections are included.
+        bib_refs (path to a bibtex file, optional).
+        model (str, optional): The language model to be used for paper generation. Defaults to "gpt-4".
+
+    Returns:
+    str: The path to the zipped file containing the generated paper and associated files.
+
+    Note: The function also handles errors that occur during section generation and retries a maximum of 4 times
+    before proceeding.
+    """
 
     def _filter_sections(sections):
         ordered_sections = ["introduction", "related works", "backgrounds", "methodology", "experiments", "conclusion",
                             "abstract"]
         return [section for section in ordered_sections if section in sections]
+
     # pre-processing `sections` parameter;
     print("================START================")
     print(f"Generating the paper '{title}'.")
-    print("\n") # todo: use a configuration file to define parameters
+    print("\n")  # todo: use a configuration file to define parameters
     print("================PRE-PROCESSING================")
     if sections is None:
-        sections = ["introduction", "related works", "backgrounds", "methodology", "experiments", "conclusion", "abstract"]
+        sections = ["introduction", "related works", "backgrounds", "methodology", "experiments", "conclusion",
+                    "abstract"]
     else:
         sections = _filter_sections(sections)
 
@@ -132,7 +155,8 @@ def generate_draft(title, description="", template="ICLR2022",
         max_tokens = 4096
     else:
         max_tokens = 2048
-    paper, destination_folder, _ = _generation_setup(title, description, template, tldr, max_kw_refs, bib_refs, max_tokens=max_tokens)
+    paper, destination_folder, _ = _generation_setup(title, description, template, tldr, max_kw_refs, bib_refs,
+                                                     max_tokens=max_tokens)
 
     # main components
     print(f"================PROCESSING================")
@@ -162,16 +186,10 @@ def generate_draft(title, description="", template="ICLR2022",
     return make_archive(destination_folder, filename)
 
 
-
-
-
-
-
 if __name__ == "__main__":
     import openai
     openai.api_key = os.getenv("OPENAI_API_KEY")
 
-    target_title = "Using interpretable boosting algorithms for modeling environmental and agricultural data"
-    target_description = ""
-    output = generate_draft(target_title, target_description, tldr=True, max_kw_refs=10)
-    print(output)
+    target_title = "Playing Atari with Decentralized Reinforcement Learning"
+    output = generate_draft(target_title)
+    print(output)

outputs/outputs_20230608_115759/abstract.tex

@@ -0,0 +1,3 @@
+\begin{abstract}
+In this paper, we present a novel Decentralized Atari Learning (DAL) algorithm for playing Atari games using decentralized reinforcement learning. Our proposed method combines the strengths of both value-based and policy-based decentralized RL techniques and introduces a unique communication mechanism that enables agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. Through a comprehensive experimental evaluation, we demonstrate the effectiveness of our algorithm in addressing the challenges of high-dimensional sensory input and complex decision-making processes in Atari games. Our experimental results show that the DAL algorithm achieves competitive performance in terms of cumulative reward, outperforming the decentralized Dec-PG method and maintaining comparable performance with the centralized DQN and A3C methods. In terms of training time and communication overhead, the DAL algorithm exhibits significant improvements over the centralized methods, highlighting its scalability and privacy-preserving capabilities. Our work contributes to the growing body of research in decentralized reinforcement learning, offering valuable insights into the trade-offs between scalability, privacy, and performance in this domain.
+\end{abstract}

outputs/outputs_20230608_115759/backgrounds.tex

@@ -0,0 +1,25 @@
+\section{Backgrounds}
+
+The central problem in the field of decentralized reinforcement learning (RL) is to develop efficient algorithms that can learn optimal policies in multi-agent environments while addressing the challenges of scalability, privacy, and convergence. This problem is of great importance in various industrial applications, such as autonomous vehicles \citep{duan2022autonomous}, traffic signal control \citep{yang2021an}, and edge-computing-empowered Internet of Things (IoT) networks \citep{lei2022adaptive}. Theoretical challenges in this field include the design of algorithms that can handle high-dimensional state and action spaces, non-stationarity, and the exponential growth of state-action space \citep{adams2020resolving}.
+
+\subsection{Foundational Concepts and Notations}
+
+Reinforcement learning is a framework for learning optimal policies through interaction with an environment \citep{sutton2005reinforcement}. In this framework, an agent takes actions in an environment to achieve a goal, and the environment provides feedback in the form of rewards. The objective of the agent is to learn a policy that maximizes the expected cumulative reward over time.
+
+A standard RL problem is modeled as a Markov Decision Process (MDP), defined by a tuple $(\mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma)$, where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space, $\mathcal{P}: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow [0, 1]$ is the state transition probability function, $\mathcal{R}: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}$ is the reward function, and $\gamma \in [0, 1)$ is the discount factor. The agent's goal is to learn a policy $\pi: \mathcal{S} \rightarrow \mathcal{A}$ that maximizes the expected cumulative reward, defined as $V^\pi(s) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t R_t | S_0 = s, \pi\right]$.
+
+In decentralized RL, multiple agents interact with the environment and each other to learn optimal policies. The problem can be modeled as a Decentralized Markov Decision Process (D-MDP) \citep{lu2021decentralized}, which extends the MDP framework to include multiple agents and their local observations, actions, and policies. The D-MDP is defined by a tuple $(\mathcal{S}, \mathcal{A}_1, \dots, \mathcal{A}_n, \mathcal{P}, \mathcal{R}_1, \dots, \mathcal{R}_n, \gamma)$, where $n$ is the number of agents, $\mathcal{A}_i$ is the action space of agent $i$, and $\mathcal{R}_i$ is the reward function of agent $i$. Each agent aims to learn a local policy $\pi_i: \mathcal{S} \rightarrow \mathcal{A}_i$ that maximizes its expected cumulative reward.
+
+\subsection{Decentralized Reinforcement Learning Algorithms}
+
+Decentralized RL algorithms can be broadly categorized into two classes: value-based and policy-based methods. Value-based methods, such as decentralized Q-learning \citep{hasselt2015deep}, aim to learn an action-value function $Q^\pi(s, a)$, which represents the expected cumulative reward of taking action $a$ in state $s$ and following policy $\pi$ thereafter. The optimal policy can be derived from the optimal action-value function, $Q^*(s, a) = \max_\pi Q^\pi(s, a)$, as $\pi^*(s) = \arg\max_a Q^*(s, a)$. Deep Q-Networks (DQNs) \citep{mnih2013playing} extend Q-learning to high-dimensional state spaces by using deep neural networks to approximate the action-value function.
+
+Policy-based methods, such as decentralized policy gradient (Dec-PG) \citep{lu2021decentralized}, directly optimize the policy by following the gradient of the expected cumulative reward with respect to the policy parameters. Actor-critic algorithms \citep{lillicrap2015continuous} combine the advantages of both value-based and policy-based methods by using a critic to estimate the action-value function and an actor to update the policy based on the critic's estimates. Decentralized actor-critic algorithms have been proposed for continuous control tasks \citep{mnih2016asynchronous} and multi-agent collision avoidance \citep{thumiger2022a}.
+
+In this paper, we focus on the application of decentralized RL algorithms to the problem of playing Atari games. We build upon the foundational concepts and algorithms introduced above and develop a novel decentralized RL algorithm that addresses the challenges of scalability, privacy, and convergence in multi-agent Atari environments.
+
+\subsection{Decentralized Learning in Atari Environments}
+
+Atari games provide a challenging testbed for RL algorithms due to their high-dimensional state spaces, diverse game dynamics, and complex scoring systems \citep{mnih2013playing}. Recent advances in deep RL have led to the development of algorithms that can learn to play Atari games directly from raw pixel inputs, outperforming human experts in some cases \citep{mnih2013playing}. However, most of these algorithms are centralized and do not scale well to large multi-agent environments.
+
+In this paper, we propose a novel decentralized RL algorithm for playing Atari games that leverages the advantages of both value-based and policy-based methods. Our algorithm builds upon the decentralized Q-learning and Dec-PG frameworks and incorporates techniques from deep RL, such as experience replay \citep{mnih2013playing} and target networks \citep{hasselt2015deep}, to improve stability and convergence. We also introduce a novel communication mechanism that allows agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. Our experimental results demonstrate that our algorithm achieves competitive performance compared to centralized methods and outperforms existing decentralized RL algorithms in the Atari domain.

outputs/outputs_20230608_115759/conclusion.tex

@@ -0,0 +1,7 @@
+\section{Conclusion}
+
+In this paper, we presented a novel Decentralized Atari Learning (DAL) algorithm for playing Atari games using decentralized reinforcement learning. Our proposed method combines the strengths of both value-based and policy-based decentralized RL techniques and introduces a unique communication mechanism that enables agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. Through a comprehensive experimental evaluation, we demonstrated the effectiveness of our algorithm in addressing the challenges of high-dimensional sensory input and complex decision-making processes in Atari games.
+
+Our experimental results showed that the DAL algorithm achieves competitive performance in terms of cumulative reward, outperforming the decentralized Dec-PG method and maintaining comparable performance with the centralized DQN and A3C methods. In terms of training time and communication overhead, the DAL algorithm exhibits significant improvements over the centralized methods, highlighting its scalability and privacy-preserving capabilities.
+
+In conclusion, our proposed Decentralized Atari Learning (DAL) algorithm contributes to the growing body of research in decentralized reinforcement learning, offering valuable insights into the trade-offs between scalability, privacy, and performance in this domain. By building upon recent advancements in decentralized RL and addressing the unique challenges associated with playing Atari games, our work paves the way for future research in large-scale, privacy-preserving multi-agent systems and their applications in various domains.

outputs/outputs_20230608_115759/experiments.tex

@@ -0,0 +1,91 @@
+\section{experiments}
+
+In this section, we present the experimental setup and results of our proposed Decentralized Atari Learning (DAL) algorithm. We begin with a high-level overview of the experimental design, followed by a detailed description of the evaluation metrics, baselines, and the Atari games used for evaluation. Finally, we present the results of our experiments, including comparisons with state-of-the-art centralized and decentralized RL methods, and discuss the insights gained from our analysis.
+
+\subsection{Experimental Design}
+
+Our experiments are designed to evaluate the performance of the DAL algorithm in terms of scalability, privacy, and convergence in multi-agent Atari environments. We compare our method with state-of-the-art centralized and decentralized RL approaches to demonstrate its effectiveness in addressing the challenges of high-dimensional sensory input and complex decision-making processes. The experimental setup consists of the following main components:
+
+\begin{itemize}
+    \item Evaluation Metrics: We use the following metrics to evaluate the performance of our algorithm: cumulative reward, training time, and communication overhead.
+    \item Baselines: We compare our method with state-of-the-art centralized and decentralized RL approaches, including DQN \citep{mnih2013playing}, A3C \citep{mnih2016asynchronous}, and Dec-PG \citep{lu2021decentralized}.
+    \item Atari Games: We evaluate our algorithm on a diverse set of Atari games, including Breakout, Pong, Space Invaders, and Ms. Pac-Man, to demonstrate its generalizability and robustness.
+\end{itemize}
+
+\subsection{Evaluation Metrics}
+
+We use the following evaluation metrics to assess the performance of our proposed DAL algorithm:
+
+\begin{itemize}
+    \item \textbf{Cumulative Reward:} The total reward accumulated by the agents during an episode, which serves as a measure of the agents' performance in the Atari games.
+    \item \textbf{Training Time:} The time taken by the agents to learn their policies, which serves as a measure of the algorithm's scalability and efficiency.
+    \item \textbf{Communication Overhead:} The amount of information exchanged between the agents during the learning process, which serves as a measure of the algorithm's privacy and communication efficiency.
+\end{itemize}
+
+\subsection{Baselines}
+
+We compare the performance of our proposed DAL algorithm with the following state-of-the-art centralized and decentralized RL methods:
+
+\begin{itemize}
+    \item \textbf{DQN} \citep{mnih2013playing}: A centralized deep Q-learning algorithm that learns to play Atari games directly from raw pixel inputs.
+    \item \textbf{A3C} \citep{mnih2016asynchronous}: A centralized actor-critic algorithm that combines the advantages of both value-based and policy-based methods for continuous control tasks and Atari games.
+    \item \textbf{Dec-PG} \citep{lu2021decentralized}: A decentralized policy gradient algorithm that accounts for coupled safety constraints in multi-agent reinforcement learning.
+\end{itemize}
+
+\subsection{Atari Games}
+
+We evaluate our algorithm on a diverse set of Atari games, including the following:
+
+\begin{itemize}
+    \item \textbf{Breakout:} A single-player game in which the agent controls a paddle to bounce a ball and break bricks.
+    \item \textbf{Pong:} A two-player game in which the agents control paddles to bounce a ball and score points by passing the ball past the opponent's paddle.
+    \item \textbf{Space Invaders:} A single-player game in which the agent controls a spaceship to shoot down invading aliens while avoiding their projectiles.
+    \item \textbf{Ms. Pac-Man:} A single-player game in which the agent controls Ms. Pac-Man to eat pellets and avoid ghosts in a maze.
+\end{itemize}
+
+\subsection{Results and Discussion}
+
+We present the results of our experiments in Table \ref{tab:results} and Figures \ref{exp1}, \ref{exp2}, and \ref{exp3}. Our proposed DAL algorithm demonstrates competitive performance compared to the centralized and decentralized baselines in terms of cumulative reward, training time, and communication overhead.
+
+\begin{table}[h]
+  \centering
+  \caption{Comparison of the performance of DAL and baseline methods on Atari games.}
+  \label{tab:results}
+  \begin{tabular}{lccc}
+    \toprule
+    Method & Cumulative Reward & Training Time & Communication Overhead \\
+    \midrule
+    \textbf{DAL (Ours)} & \textbf{X1} & \textbf{Y1} & \textbf{Z1} \\
+    DQN & X2 & Y2 & Z2 \\
+    A3C & X3 & Y3 & Z3 \\
+    Dec-PG & X4 & Y4 & Z4 \\
+    \bottomrule
+  \end{tabular}
+\end{table}
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{exp1.png}
+  \caption{Comparison of the cumulative reward achieved by DAL and baseline methods on Atari games.}
+  \label{exp1}
+\end{figure}
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{exp2.png}
+  \caption{Comparison of the training time required by DAL and baseline methods on Atari games.}
+  \label{exp2}
+\end{figure}
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{exp3.png}
+  \caption{Comparison of the communication overhead incurred by DAL and baseline methods on Atari games.}
+  \label{exp3}
+\end{figure}
+
+Our analysis reveals that the DAL algorithm achieves competitive performance in terms of cumulative reward, outperforming the decentralized Dec-PG method and maintaining comparable performance with the centralized DQN and A3C methods. This demonstrates the effectiveness of our algorithm in addressing the challenges of high-dimensional sensory input and complex decision-making processes in Atari games.
+
+In terms of training time and communication overhead, the DAL algorithm shows significant improvements over the centralized methods, highlighting its scalability and privacy-preserving capabilities. The algorithm also outperforms the Dec-PG method in these aspects, demonstrating the benefits of our novel communication mechanism.
+
+In summary, our experiments demonstrate the effectiveness of our proposed Decentralized Atari Learning (DAL) algorithm in playing Atari games using decentralized reinforcement learning. The algorithm achieves competitive performance compared to state-of-the-art centralized and decentralized RL methods while maintaining scalability, privacy, and convergence in multi-agent Atari environments.

outputs/outputs_20230608_115759/fancyhdr.sty

@@ -0,0 +1,485 @@
+% fancyhdr.sty version 3.2
+% Fancy headers and footers for LaTeX.
+% Piet van Oostrum, 
+% Dept of Computer and Information Sciences, University of Utrecht,
+% Padualaan 14, P.O. Box 80.089, 3508 TB Utrecht, The Netherlands
+% Telephone: +31 30 2532180. Email: piet@cs.uu.nl
+% ========================================================================
+% LICENCE:
+% This file may be distributed under the terms of the LaTeX Project Public
+% License, as described in lppl.txt in the base LaTeX distribution.
+% Either version 1 or, at your option, any later version.
+% ========================================================================
+% MODIFICATION HISTORY:
+% Sep 16, 1994
+% version 1.4: Correction for use with \reversemargin
+% Sep 29, 1994:
+% version 1.5: Added the \iftopfloat, \ifbotfloat and \iffloatpage commands
+% Oct 4, 1994:
+% version 1.6: Reset single spacing in headers/footers for use with
+% setspace.sty or doublespace.sty
+% Oct 4, 1994:
+% version 1.7: changed \let\@mkboth\markboth to
+% \def\@mkboth{\protect\markboth} to make it more robust
+% Dec 5, 1994:
+% version 1.8: corrections for amsbook/amsart: define \@chapapp and (more
+% importantly) use the \chapter/sectionmark definitions from ps@headings if
+% they exist (which should be true for all standard classes).
+% May 31, 1995:
+% version 1.9: The proposed \renewcommand{\headrulewidth}{\iffloatpage...
+% construction in the doc did not work properly with the fancyplain style. 
+% June 1, 1995:
+% version 1.91: The definition of \@mkboth wasn't restored on subsequent
+% \pagestyle{fancy}'s.
+% June 1, 1995:
+% version 1.92: The sequence \pagestyle{fancyplain} \pagestyle{plain}
+% \pagestyle{fancy} would erroneously select the plain version.
+% June 1, 1995:
+% version 1.93: \fancypagestyle command added.
+% Dec 11, 1995:
+% version 1.94: suggested by Conrad Hughes <chughes@maths.tcd.ie>
+% CJCH, Dec 11, 1995: added \footruleskip to allow control over footrule
+% position (old hardcoded value of .3\normalbaselineskip is far too high
+% when used with very small footer fonts).
+% Jan 31, 1996:
+% version 1.95: call \@normalsize in the reset code if that is defined,
+% otherwise \normalsize.
+% this is to solve a problem with ucthesis.cls, as this doesn't
+% define \@currsize. Unfortunately for latex209 calling \normalsize doesn't
+% work as this is optimized to do very little, so there \@normalsize should
+% be called. Hopefully this code works for all versions of LaTeX known to
+% mankind.  
+% April 25, 1996:
+% version 1.96: initialize \headwidth to a magic (negative) value to catch
+% most common cases that people change it before calling \pagestyle{fancy}.
+% Note it can't be initialized when reading in this file, because
+% \textwidth could be changed afterwards. This is quite probable.
+% We also switch to \MakeUppercase rather than \uppercase and introduce a
+% \nouppercase command for use in headers. and footers.
+% May 3, 1996:
+% version 1.97: Two changes:
+% 1. Undo the change in version 1.8 (using the pagestyle{headings} defaults
+% for the chapter and section marks. The current version of amsbook and
+% amsart classes don't seem to need them anymore. Moreover the standard
+% latex classes don't use \markboth if twoside isn't selected, and this is
+% confusing as \leftmark doesn't work as expected.
+% 2. include a call to \ps@empty in ps@@fancy. This is to solve a problem
+% in the amsbook and amsart classes, that make global changes to \topskip,
+% which are reset in \ps@empty. Hopefully this doesn't break other things.
+% May 7, 1996:
+% version 1.98:
+% Added % after the line  \def\nouppercase
+% May 7, 1996:
+% version 1.99: This is the alpha version of fancyhdr 2.0
+% Introduced the new commands \fancyhead, \fancyfoot, and \fancyhf.
+% Changed \headrulewidth, \footrulewidth, \footruleskip to
+% macros rather than length parameters, In this way they can be
+% conditionalized and they don't consume length registers. There is no need
+% to have them as length registers unless you want to do calculations with
+% them, which is unlikely. Note that this may make some uses of them
+% incompatible (i.e. if you have a file that uses \setlength or \xxxx=)
+% May 10, 1996:
+% version 1.99a:
+% Added a few more % signs
+% May 10, 1996:
+% version 1.99b:
+% Changed the syntax of \f@nfor to be resistent to catcode changes of :=
+% Removed the [1] from the defs of \lhead etc. because the parameter is
+% consumed by the \@[xy]lhead etc. macros.
+% June 24, 1997:
+% version 1.99c:
+% corrected \nouppercase to also include the protected form of \MakeUppercase
+% \global added to manipulation of \headwidth.
+% \iffootnote command added.
+% Some comments added about \@fancyhead and \@fancyfoot.
+% Aug 24, 1998
+% version 1.99d
+% Changed the default \ps@empty to \ps@@empty in order to allow
+% \fancypagestyle{empty} redefinition.
+% Oct 11, 2000
+% version 2.0
+% Added LPPL license clause.
+%
+% A check for \headheight is added. An errormessage is given (once) if the
+% header is too large. Empty headers don't generate the error even if
+% \headheight is very small or even 0pt. 
+% Warning added for the use of 'E' option when twoside option is not used.
+% In this case the 'E' fields will never be used.
+%
+% Mar 10, 2002
+% version 2.1beta
+% New command: \fancyhfoffset[place]{length}
+% defines offsets to be applied to the header/footer to let it stick into
+% the margins (if length > 0).
+% place is like in fancyhead, except that only E,O,L,R can be used.
+% This replaces the old calculation based on \headwidth and the marginpar
+% area.
+% \headwidth will be dynamically calculated in the headers/footers when
+% this is used.
+%
+% Mar 26, 2002
+% version 2.1beta2
+% \fancyhfoffset now also takes h,f as possible letters in the argument to
+% allow the header and footer widths to be different.
+% New commands \fancyheadoffset and \fancyfootoffset added comparable to
+% \fancyhead and \fancyfoot.
+% Errormessages and warnings have been made more informative.
+%
+% Dec 9, 2002
+% version 2.1
+% The defaults for \footrulewidth, \plainheadrulewidth and
+% \plainfootrulewidth are changed from \z@skip to 0pt. In this way when
+% someone inadvertantly uses \setlength to change any of these, the value
+% of \z@skip will not be changed, rather an errormessage will be given.
+
+% March 3, 2004
+% Release of version 3.0
+
+% Oct 7, 2004
+% version 3.1
+% Added '\endlinechar=13' to \fancy@reset to prevent problems with
+% includegraphics in header when verbatiminput is active.
+
+% March 22, 2005
+% version 3.2
+% reset \everypar (the real one) in \fancy@reset because spanish.ldf does
+% strange things with \everypar between << and >>.
+
+\def\ifancy@mpty#1{\def\temp@a{#1}\ifx\temp@a\@empty}
+
+\def\fancy@def#1#2{\ifancy@mpty{#2}\fancy@gbl\def#1{\leavevmode}\else
+                                   \fancy@gbl\def#1{#2\strut}\fi}
+
+\let\fancy@gbl\global
+
+\def\@fancyerrmsg#1{%
+        \ifx\PackageError\undefined
+        \errmessage{#1}\else
+        \PackageError{Fancyhdr}{#1}{}\fi}
+\def\@fancywarning#1{%
+        \ifx\PackageWarning\undefined
+        \errmessage{#1}\else
+        \PackageWarning{Fancyhdr}{#1}{}\fi}
+
+% Usage: \@forc \var{charstring}{command to be executed for each char}
+% This is similar to LaTeX's \@tfor, but expands the charstring.
+
+\def\@forc#1#2#3{\expandafter\f@rc\expandafter#1\expandafter{#2}{#3}}
+\def\f@rc#1#2#3{\def\temp@ty{#2}\ifx\@empty\temp@ty\else
+                                    \f@@rc#1#2\f@@rc{#3}\fi}
+\def\f@@rc#1#2#3\f@@rc#4{\def#1{#2}#4\f@rc#1{#3}{#4}}
+
+% Usage: \f@nfor\name:=list\do{body}
+% Like LaTeX's \@for but an empty list is treated as a list with an empty
+% element
+
+\newcommand{\f@nfor}[3]{\edef\@fortmp{#2}%
+    \expandafter\@forloop#2,\@nil,\@nil\@@#1{#3}}
+
+% Usage: \def@ult \cs{defaults}{argument}
+% sets \cs to the characters from defaults appearing in argument
+% or defaults if it would be empty. All characters are lowercased.
+
+\newcommand\def@ult[3]{%
+    \edef\temp@a{\lowercase{\edef\noexpand\temp@a{#3}}}\temp@a
+    \def#1{}%
+    \@forc\tmpf@ra{#2}%
+        {\expandafter\if@in\tmpf@ra\temp@a{\edef#1{#1\tmpf@ra}}{}}%
+    \ifx\@empty#1\def#1{#2}\fi}
+% 
+% \if@in <char><set><truecase><falsecase>
+%
+\newcommand{\if@in}[4]{%
+    \edef\temp@a{#2}\def\temp@b##1#1##2\temp@b{\def\temp@b{##1}}%
+    \expandafter\temp@b#2#1\temp@b\ifx\temp@a\temp@b #4\else #3\fi}
+
+\newcommand{\fancyhead}{\@ifnextchar[{\f@ncyhf\fancyhead h}%
+                                     {\f@ncyhf\fancyhead h[]}}
+\newcommand{\fancyfoot}{\@ifnextchar[{\f@ncyhf\fancyfoot f}%
+                                     {\f@ncyhf\fancyfoot f[]}}
+\newcommand{\fancyhf}{\@ifnextchar[{\f@ncyhf\fancyhf{}}%
+                                   {\f@ncyhf\fancyhf{}[]}}
+
+% New commands for offsets added
+
+\newcommand{\fancyheadoffset}{\@ifnextchar[{\f@ncyhfoffs\fancyheadoffset h}%
+                                           {\f@ncyhfoffs\fancyheadoffset h[]}}
+\newcommand{\fancyfootoffset}{\@ifnextchar[{\f@ncyhfoffs\fancyfootoffset f}%
+                                           {\f@ncyhfoffs\fancyfootoffset f[]}}
+\newcommand{\fancyhfoffset}{\@ifnextchar[{\f@ncyhfoffs\fancyhfoffset{}}%
+                                         {\f@ncyhfoffs\fancyhfoffset{}[]}}
+
+% The header and footer fields are stored in command sequences with
+% names of the form: \f@ncy<x><y><z> with <x> for [eo], <y> from [lcr]
+% and <z> from [hf].
+
+\def\f@ncyhf#1#2[#3]#4{%
+    \def\temp@c{}%
+    \@forc\tmpf@ra{#3}%
+        {\expandafter\if@in\tmpf@ra{eolcrhf,EOLCRHF}%
+            {}{\edef\temp@c{\temp@c\tmpf@ra}}}%
+    \ifx\@empty\temp@c\else
+        \@fancyerrmsg{Illegal char `\temp@c' in \string#1 argument:
+          [#3]}%
+    \fi
+    \f@nfor\temp@c{#3}%
+        {\def@ult\f@@@eo{eo}\temp@c
+         \if@twoside\else
+           \if\f@@@eo e\@fancywarning
+             {\string#1's `E' option without twoside option is useless}\fi\fi
+         \def@ult\f@@@lcr{lcr}\temp@c
+         \def@ult\f@@@hf{hf}{#2\temp@c}%
+         \@forc\f@@eo\f@@@eo
+             {\@forc\f@@lcr\f@@@lcr
+                 {\@forc\f@@hf\f@@@hf
+                     {\expandafter\fancy@def\csname
+                      f@ncy\f@@eo\f@@lcr\f@@hf\endcsname
+                      {#4}}}}}}
+
+\def\f@ncyhfoffs#1#2[#3]#4{%
+    \def\temp@c{}%
+    \@forc\tmpf@ra{#3}%
+        {\expandafter\if@in\tmpf@ra{eolrhf,EOLRHF}%
+            {}{\edef\temp@c{\temp@c\tmpf@ra}}}%
+    \ifx\@empty\temp@c\else
+        \@fancyerrmsg{Illegal char `\temp@c' in \string#1 argument:
+          [#3]}%
+    \fi
+    \f@nfor\temp@c{#3}%
+        {\def@ult\f@@@eo{eo}\temp@c
+         \if@twoside\else
+           \if\f@@@eo e\@fancywarning
+             {\string#1's `E' option without twoside option is useless}\fi\fi
+         \def@ult\f@@@lcr{lr}\temp@c
+         \def@ult\f@@@hf{hf}{#2\temp@c}%
+         \@forc\f@@eo\f@@@eo
+             {\@forc\f@@lcr\f@@@lcr
+                 {\@forc\f@@hf\f@@@hf
+                     {\expandafter\setlength\csname
+                      f@ncyO@\f@@eo\f@@lcr\f@@hf\endcsname
+                      {#4}}}}}%
+     \fancy@setoffs}
+
+% Fancyheadings version 1 commands. These are more or less deprecated,
+% but they continue to work.
+
+\newcommand{\lhead}{\@ifnextchar[{\@xlhead}{\@ylhead}}
+\def\@xlhead[#1]#2{\fancy@def\f@ncyelh{#1}\fancy@def\f@ncyolh{#2}}
+\def\@ylhead#1{\fancy@def\f@ncyelh{#1}\fancy@def\f@ncyolh{#1}}
+
+\newcommand{\chead}{\@ifnextchar[{\@xchead}{\@ychead}}
+\def\@xchead[#1]#2{\fancy@def\f@ncyech{#1}\fancy@def\f@ncyoch{#2}}
+\def\@ychead#1{\fancy@def\f@ncyech{#1}\fancy@def\f@ncyoch{#1}}
+
+\newcommand{\rhead}{\@ifnextchar[{\@xrhead}{\@yrhead}}
+\def\@xrhead[#1]#2{\fancy@def\f@ncyerh{#1}\fancy@def\f@ncyorh{#2}}
+\def\@yrhead#1{\fancy@def\f@ncyerh{#1}\fancy@def\f@ncyorh{#1}}
+
+\newcommand{\lfoot}{\@ifnextchar[{\@xlfoot}{\@ylfoot}}
+\def\@xlfoot[#1]#2{\fancy@def\f@ncyelf{#1}\fancy@def\f@ncyolf{#2}}
+\def\@ylfoot#1{\fancy@def\f@ncyelf{#1}\fancy@def\f@ncyolf{#1}}
+
+\newcommand{\cfoot}{\@ifnextchar[{\@xcfoot}{\@ycfoot}}
+\def\@xcfoot[#1]#2{\fancy@def\f@ncyecf{#1}\fancy@def\f@ncyocf{#2}}
+\def\@ycfoot#1{\fancy@def\f@ncyecf{#1}\fancy@def\f@ncyocf{#1}}
+
+\newcommand{\rfoot}{\@ifnextchar[{\@xrfoot}{\@yrfoot}}
+\def\@xrfoot[#1]#2{\fancy@def\f@ncyerf{#1}\fancy@def\f@ncyorf{#2}}
+\def\@yrfoot#1{\fancy@def\f@ncyerf{#1}\fancy@def\f@ncyorf{#1}}
+
+\newlength{\fancy@headwidth}
+\let\headwidth\fancy@headwidth
+\newlength{\f@ncyO@elh}
+\newlength{\f@ncyO@erh}
+\newlength{\f@ncyO@olh}
+\newlength{\f@ncyO@orh}
+\newlength{\f@ncyO@elf}
+\newlength{\f@ncyO@erf}
+\newlength{\f@ncyO@olf}
+\newlength{\f@ncyO@orf}
+\newcommand{\headrulewidth}{0.4pt}
+\newcommand{\footrulewidth}{0pt}
+\newcommand{\footruleskip}{.3\normalbaselineskip}
+
+% Fancyplain stuff shouldn't be used anymore (rather
+% \fancypagestyle{plain} should be used), but it must be present for
+% compatibility reasons.
+
+\newcommand{\plainheadrulewidth}{0pt}
+\newcommand{\plainfootrulewidth}{0pt}
+\newif\if@fancyplain \@fancyplainfalse
+\def\fancyplain#1#2{\if@fancyplain#1\else#2\fi}
+
+\headwidth=-123456789sp %magic constant
+
+% Command to reset various things in the headers:
+% a.o.  single spacing (taken from setspace.sty)
+% and the catcode of ^^M (so that epsf files in the header work if a
+% verbatim crosses a page boundary)
+% It also defines a \nouppercase command that disables \uppercase and
+% \Makeuppercase. It can only be used in the headers and footers.
+\let\fnch@everypar\everypar% save real \everypar because of spanish.ldf
+\def\fancy@reset{\fnch@everypar{}\restorecr\endlinechar=13
+ \def\baselinestretch{1}%
+ \def\nouppercase##1{{\let\uppercase\relax\let\MakeUppercase\relax
+     \expandafter\let\csname MakeUppercase \endcsname\relax##1}}%
+ \ifx\undefined\@newbaseline% NFSS not present; 2.09 or 2e
+   \ifx\@normalsize\undefined \normalsize % for ucthesis.cls
+   \else \@normalsize \fi
+ \else% NFSS (2.09) present
+  \@newbaseline%
+ \fi}
+
+% Initialization of the head and foot text.
+
+% The default values still contain \fancyplain for compatibility.
+\fancyhf{} % clear all
+% lefthead empty on ``plain'' pages, \rightmark on even, \leftmark on odd pages
+% evenhead empty on ``plain'' pages, \leftmark on even, \rightmark on odd pages
+\if@twoside
+  \fancyhead[el,or]{\fancyplain{}{\sl\rightmark}}
+  \fancyhead[er,ol]{\fancyplain{}{\sl\leftmark}}
+\else
+  \fancyhead[l]{\fancyplain{}{\sl\rightmark}}
+  \fancyhead[r]{\fancyplain{}{\sl\leftmark}}
+\fi
+\fancyfoot[c]{\rm\thepage} % page number
+
+% Use box 0 as a temp box and dimen 0 as temp dimen. 
+% This can be done, because this code will always
+% be used inside another box, and therefore the changes are local.
+
+\def\@fancyvbox#1#2{\setbox0\vbox{#2}\ifdim\ht0>#1\@fancywarning
+  {\string#1 is too small (\the#1): ^^J Make it at least \the\ht0.^^J
+    We now make it that large for the rest of the document.^^J
+    This may cause the page layout to be inconsistent, however\@gobble}%
+  \dimen0=#1\global\setlength{#1}{\ht0}\ht0=\dimen0\fi
+  \box0}
+
+% Put together a header or footer given the left, center and
+% right text, fillers at left and right and a rule.
+% The \lap commands put the text into an hbox of zero size,
+% so overlapping text does not generate an errormessage.
+% These macros have 5 parameters:
+% 1. LEFTSIDE BEARING % This determines at which side the header will stick
+%    out. When \fancyhfoffset is used this calculates \headwidth, otherwise
+%    it is \hss or \relax (after expansion).
+% 2. \f@ncyolh, \f@ncyelh, \f@ncyolf or \f@ncyelf. This is the left component.
+% 3. \f@ncyoch, \f@ncyech, \f@ncyocf or \f@ncyecf. This is the middle comp.
+% 4. \f@ncyorh, \f@ncyerh, \f@ncyorf or \f@ncyerf. This is the right component.
+% 5. RIGHTSIDE BEARING. This is always \relax or \hss (after expansion).
+
+\def\@fancyhead#1#2#3#4#5{#1\hbox to\headwidth{\fancy@reset
+  \@fancyvbox\headheight{\hbox
+    {\rlap{\parbox[b]{\headwidth}{\raggedright#2}}\hfill
+      \parbox[b]{\headwidth}{\centering#3}\hfill
+      \llap{\parbox[b]{\headwidth}{\raggedleft#4}}}\headrule}}#5}
+
+\def\@fancyfoot#1#2#3#4#5{#1\hbox to\headwidth{\fancy@reset
+    \@fancyvbox\footskip{\footrule
+      \hbox{\rlap{\parbox[t]{\headwidth}{\raggedright#2}}\hfill
+        \parbox[t]{\headwidth}{\centering#3}\hfill
+        \llap{\parbox[t]{\headwidth}{\raggedleft#4}}}}}#5}
+
+\def\headrule{{\if@fancyplain\let\headrulewidth\plainheadrulewidth\fi
+    \hrule\@height\headrulewidth\@width\headwidth \vskip-\headrulewidth}}
+
+\def\footrule{{\if@fancyplain\let\footrulewidth\plainfootrulewidth\fi
+    \vskip-\footruleskip\vskip-\footrulewidth
+    \hrule\@width\headwidth\@height\footrulewidth\vskip\footruleskip}}
+
+\def\ps@fancy{%
+\@ifundefined{@chapapp}{\let\@chapapp\chaptername}{}%for amsbook
+%
+% Define \MakeUppercase for old LaTeXen.
+% Note: we used \def rather than \let, so that \let\uppercase\relax (from
+% the version 1 documentation) will still work.
+%
+\@ifundefined{MakeUppercase}{\def\MakeUppercase{\uppercase}}{}%
+\@ifundefined{chapter}{\def\sectionmark##1{\markboth
+{\MakeUppercase{\ifnum \c@secnumdepth>\z@
+ \thesection\hskip 1em\relax \fi ##1}}{}}%
+\def\subsectionmark##1{\markright {\ifnum \c@secnumdepth >\@ne
+ \thesubsection\hskip 1em\relax \fi ##1}}}%
+{\def\chaptermark##1{\markboth {\MakeUppercase{\ifnum \c@secnumdepth>\m@ne
+ \@chapapp\ \thechapter. \ \fi ##1}}{}}%
+\def\sectionmark##1{\markright{\MakeUppercase{\ifnum \c@secnumdepth >\z@
+ \thesection. \ \fi ##1}}}}%
+%\csname ps@headings\endcsname % use \ps@headings defaults if they exist
+\ps@@fancy
+\gdef\ps@fancy{\@fancyplainfalse\ps@@fancy}%
+% Initialize \headwidth if the user didn't
+%
+\ifdim\headwidth<0sp
+%
+% This catches the case that \headwidth hasn't been initialized and the
+% case that the user added something to \headwidth in the expectation that
+% it was initialized to \textwidth. We compensate this now. This loses if
+% the user intended to multiply it by a factor. But that case is more
+% likely done by saying something like \headwidth=1.2\textwidth. 
+% The doc says you have to change \headwidth after the first call to
+% \pagestyle{fancy}. This code is just to catch the most common cases were
+% that requirement is violated.
+%
+    \global\advance\headwidth123456789sp\global\advance\headwidth\textwidth
+\fi}
+\def\ps@fancyplain{\ps@fancy \let\ps@plain\ps@plain@fancy}
+\def\ps@plain@fancy{\@fancyplaintrue\ps@@fancy}
+\let\ps@@empty\ps@empty
+\def\ps@@fancy{%
+\ps@@empty % This is for amsbook/amsart, which do strange things with \topskip
+\def\@mkboth{\protect\markboth}%
+\def\@oddhead{\@fancyhead\fancy@Oolh\f@ncyolh\f@ncyoch\f@ncyorh\fancy@Oorh}%
+\def\@oddfoot{\@fancyfoot\fancy@Oolf\f@ncyolf\f@ncyocf\f@ncyorf\fancy@Oorf}%
+\def\@evenhead{\@fancyhead\fancy@Oelh\f@ncyelh\f@ncyech\f@ncyerh\fancy@Oerh}%
+\def\@evenfoot{\@fancyfoot\fancy@Oelf\f@ncyelf\f@ncyecf\f@ncyerf\fancy@Oerf}%
+}
+% Default definitions for compatibility mode:
+% These cause the header/footer to take the defined \headwidth as width
+% And to shift in the direction of the marginpar area
+
+\def\fancy@Oolh{\if@reversemargin\hss\else\relax\fi}
+\def\fancy@Oorh{\if@reversemargin\relax\else\hss\fi}
+\let\fancy@Oelh\fancy@Oorh
+\let\fancy@Oerh\fancy@Oolh
+
+\let\fancy@Oolf\fancy@Oolh
+\let\fancy@Oorf\fancy@Oorh
+\let\fancy@Oelf\fancy@Oelh
+\let\fancy@Oerf\fancy@Oerh
+
+% New definitions for the use of \fancyhfoffset
+% These calculate the \headwidth from \textwidth and the specified offsets.
+
+\def\fancy@offsolh{\headwidth=\textwidth\advance\headwidth\f@ncyO@olh
+                   \advance\headwidth\f@ncyO@orh\hskip-\f@ncyO@olh}
+\def\fancy@offselh{\headwidth=\textwidth\advance\headwidth\f@ncyO@elh
+                   \advance\headwidth\f@ncyO@erh\hskip-\f@ncyO@elh}
+
+\def\fancy@offsolf{\headwidth=\textwidth\advance\headwidth\f@ncyO@olf
+                   \advance\headwidth\f@ncyO@orf\hskip-\f@ncyO@olf}
+\def\fancy@offself{\headwidth=\textwidth\advance\headwidth\f@ncyO@elf
+                   \advance\headwidth\f@ncyO@erf\hskip-\f@ncyO@elf}
+
+\def\fancy@setoffs{%
+% Just in case \let\headwidth\textwidth was used
+  \fancy@gbl\let\headwidth\fancy@headwidth
+  \fancy@gbl\let\fancy@Oolh\fancy@offsolh
+  \fancy@gbl\let\fancy@Oelh\fancy@offselh
+  \fancy@gbl\let\fancy@Oorh\hss
+  \fancy@gbl\let\fancy@Oerh\hss
+  \fancy@gbl\let\fancy@Oolf\fancy@offsolf
+  \fancy@gbl\let\fancy@Oelf\fancy@offself
+  \fancy@gbl\let\fancy@Oorf\hss
+  \fancy@gbl\let\fancy@Oerf\hss}
+
+\newif\iffootnote
+\let\latex@makecol\@makecol
+\def\@makecol{\ifvoid\footins\footnotetrue\else\footnotefalse\fi
+\let\topfloat\@toplist\let\botfloat\@botlist\latex@makecol}
+\def\iftopfloat#1#2{\ifx\topfloat\empty #2\else #1\fi}
+\def\ifbotfloat#1#2{\ifx\botfloat\empty #2\else #1\fi}
+\def\iffloatpage#1#2{\if@fcolmade #1\else #2\fi}
+
+\newcommand{\fancypagestyle}[2]{%
+  \@namedef{ps@#1}{\let\fancy@gbl\relax#2\relax\ps@fancy}}

outputs/outputs_20230608_115759/iclr2022_conference.bst

@@ -0,0 +1,1440 @@
+%% File: `iclr2017.bst'
+%% A copy of iclm2010.bst, which is a modification of `plainnl.bst' for use with natbib package 
+%%
+%% Copyright 2010 Hal Daum\'e III
+%% Modified by J. F�rnkranz
+%% - Changed labels from (X and Y, 2000) to (X & Y, 2000)
+%%
+%% Copyright 1993-2007 Patrick W Daly
+%% Max-Planck-Institut f\"ur Sonnensystemforschung
+%% Max-Planck-Str. 2
+%% D-37191 Katlenburg-Lindau
+%% Germany
+%% E-mail: daly@mps.mpg.de
+%%
+%% This program can be redistributed and/or modified under the terms
+%% of the LaTeX Project Public License Distributed from CTAN
+%% archives in directory macros/latex/base/lppl.txt; either
+%% version 1 of the License, or any later version.
+%%
+ % Version and source file information:
+ % \ProvidesFile{icml2010.mbs}[2007/11/26 1.93 (PWD)]
+ %
+ % BibTeX `plainnat' family
+ %   version 0.99b for BibTeX versions 0.99a or later,
+ %   for LaTeX versions 2.09 and 2e.
+ %
+ % For use with the `natbib.sty' package; emulates the corresponding
+ %   member of the `plain' family, but with author-year citations.
+ %
+ % With version 6.0 of `natbib.sty', it may also be used for numerical
+ %   citations, while retaining the commands \citeauthor, \citefullauthor,
+ %   and \citeyear to print the corresponding information.
+ %
+ % For version 7.0 of `natbib.sty', the KEY field replaces missing
+ %   authors/editors, and the date is left blank in \bibitem.
+ %
+ % Includes field EID for the sequence/citation number of electronic journals
+ %  which is used instead of page numbers.
+ %
+ % Includes fields ISBN and ISSN.
+ %
+ % Includes field URL for Internet addresses.
+ %
+ % Includes field DOI for Digital Object Idenfifiers.
+ %
+ % Works best with the url.sty package of Donald Arseneau.
+ %
+ % Works with identical authors and year are further sorted by
+ %   citation key, to preserve any natural sequence.
+ %
+ENTRY
+  { address
+    author
+    booktitle
+    chapter
+    doi
+    eid
+    edition
+    editor
+    howpublished
+    institution
+    isbn
+    issn
+    journal
+    key
+    month
+    note
+    number
+    organization
+    pages
+    publisher
+    school
+    series
+    title
+    type
+    url
+    volume
+    year
+  }
+  {}
+  { label extra.label sort.label short.list }
+
+INTEGERS { output.state before.all mid.sentence after.sentence after.block }
+
+FUNCTION {init.state.consts}
+{ #0 'before.all :=
+  #1 'mid.sentence :=
+  #2 'after.sentence :=
+  #3 'after.block :=
+}
+
+STRINGS { s t }
+
+FUNCTION {output.nonnull}
+{ 's :=
+  output.state mid.sentence =
+    { ", " * write$ }
+    { output.state after.block =
+        { add.period$ write$
+          newline$
+          "\newblock " write$
+        }
+        { output.state before.all =
+            'write$
+            { add.period$ " " * write$ }
+          if$
+        }
+      if$
+      mid.sentence 'output.state :=
+    }
+  if$
+  s
+}
+
+FUNCTION {output}
+{ duplicate$ empty$
+    'pop$
+    'output.nonnull
+  if$
+}
+
+FUNCTION {output.check}
+{ 't :=
+  duplicate$ empty$
+    { pop$ "empty " t * " in " * cite$ * warning$ }
+    'output.nonnull
+  if$
+}
+
+FUNCTION {fin.entry}
+{ add.period$
+  write$
+  newline$
+}
+
+FUNCTION {new.block}
+{ output.state before.all =
+    'skip$
+    { after.block 'output.state := }
+  if$
+}
+
+FUNCTION {new.sentence}
+{ output.state after.block =
+    'skip$
+    { output.state before.all =
+        'skip$
+        { after.sentence 'output.state := }
+      if$
+    }
+  if$
+}
+
+FUNCTION {not}
+{   { #0 }
+    { #1 }
+  if$
+}
+
+FUNCTION {and}
+{   'skip$
+    { pop$ #0 }
+  if$
+}
+
+FUNCTION {or}
+{   { pop$ #1 }
+    'skip$
+  if$
+}
+
+FUNCTION {new.block.checka}
+{ empty$
+    'skip$
+    'new.block
+  if$
+}
+
+FUNCTION {new.block.checkb}
+{ empty$
+  swap$ empty$
+  and
+    'skip$
+    'new.block
+  if$
+}
+
+FUNCTION {new.sentence.checka}
+{ empty$
+    'skip$
+    'new.sentence
+  if$
+}
+
+FUNCTION {new.sentence.checkb}
+{ empty$
+  swap$ empty$
+  and
+    'skip$
+    'new.sentence
+  if$
+}
+
+FUNCTION {field.or.null}
+{ duplicate$ empty$
+    { pop$ "" }
+    'skip$
+  if$
+}
+
+FUNCTION {emphasize}
+{ duplicate$ empty$
+    { pop$ "" }
+    { "\emph{" swap$ * "}" * }
+  if$
+}
+
+INTEGERS { nameptr namesleft numnames }
+
+FUNCTION {format.names}
+{ 's :=
+  #1 'nameptr :=
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    { s nameptr "{ff~}{vv~}{ll}{, jj}" format.name$ 't :=
+      nameptr #1 >
+        { namesleft #1 >
+            { ", " * t * }
+            { numnames #2 >
+                { "," * }
+                'skip$
+              if$
+              t "others" =
+                { " et~al." * }
+                { " and " * t * }
+              if$
+            }
+          if$
+        }
+        't
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {format.key}
+{ empty$
+    { key field.or.null }
+    { "" }
+  if$
+}
+
+FUNCTION {format.authors}
+{ author empty$
+    { "" }
+    { author format.names }
+  if$
+}
+
+FUNCTION {format.editors}
+{ editor empty$
+    { "" }
+    { editor format.names
+      editor num.names$ #1 >
+        { " (eds.)" * }
+        { " (ed.)" * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.isbn}
+{ isbn empty$
+    { "" }
+    { new.block "ISBN " isbn * }
+  if$
+}
+
+FUNCTION {format.issn}
+{ issn empty$
+    { "" }
+    { new.block "ISSN " issn * }
+  if$
+}
+
+FUNCTION {format.url}
+{ url empty$
+    { "" }
+    { new.block "URL \url{" url * "}" * }
+  if$
+}
+
+FUNCTION {format.doi}
+{ doi empty$
+    { "" }
+    { new.block "\doi{" doi * "}" * }
+  if$
+}
+
+FUNCTION {format.title}
+{ title empty$
+    { "" }
+    { title "t" change.case$ }
+  if$
+}
+
+FUNCTION {format.full.names}
+{'s :=
+  #1 'nameptr :=
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    { s nameptr
+      "{vv~}{ll}" format.name$ 't :=
+      nameptr #1 >
+        {
+          namesleft #1 >
+            { ", " * t * }
+            {
+              numnames #2 >
+                { "," * }
+                'skip$
+              if$
+              t "others" =
+                { " et~al." * }
+                { " and " * t * }
+              if$
+            }
+          if$
+        }
+        't
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {author.editor.full}
+{ author empty$
+    { editor empty$
+        { "" }
+        { editor format.full.names }
+      if$
+    }
+    { author format.full.names }
+  if$
+}
+
+FUNCTION {author.full}
+{ author empty$
+    { "" }
+    { author format.full.names }
+  if$
+}
+
+FUNCTION {editor.full}
+{ editor empty$
+    { "" }
+    { editor format.full.names }
+  if$
+}
+
+FUNCTION {make.full.names}
+{ type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.full
+    { type$ "proceedings" =
+        'editor.full
+        'author.full
+      if$
+    }
+  if$
+}
+
+FUNCTION {output.bibitem}
+{ newline$
+  "\bibitem[" write$
+  label write$
+  ")" make.full.names duplicate$ short.list =
+     { pop$ }
+     { * }
+   if$
+  "]{" * write$
+  cite$ write$
+  "}" write$
+  newline$
+  ""
+  before.all 'output.state :=
+}
+
+FUNCTION {n.dashify}
+{ 't :=
+  ""
+    { t empty$ not }
+    { t #1 #1 substring$ "-" =
+        { t #1 #2 substring$ "--" = not
+            { "--" *
+              t #2 global.max$ substring$ 't :=
+            }
+            {   { t #1 #1 substring$ "-" = }
+                { "-" *
+                  t #2 global.max$ substring$ 't :=
+                }
+              while$
+            }
+          if$
+        }
+        { t #1 #1 substring$ *
+          t #2 global.max$ substring$ 't :=
+        }
+      if$
+    }
+  while$
+}
+
+FUNCTION {format.date}
+{ year duplicate$ empty$
+    { "empty year in " cite$ * warning$
+       pop$ "" }
+    'skip$
+  if$
+  month empty$
+    'skip$
+    { month
+      " " * swap$ *
+    }
+  if$
+  extra.label *
+}
+
+FUNCTION {format.btitle}
+{ title emphasize
+}
+
+FUNCTION {tie.or.space.connect}
+{ duplicate$ text.length$ #3 <
+    { "~" }
+    { " " }
+  if$
+  swap$ * *
+}
+
+FUNCTION {either.or.check}
+{ empty$
+    'pop$
+    { "can't use both " swap$ * " fields in " * cite$ * warning$ }
+  if$
+}
+
+FUNCTION {format.bvolume}
+{ volume empty$
+    { "" }
+    { "volume" volume tie.or.space.connect
+      series empty$
+        'skip$
+        { " of " * series emphasize * }
+      if$
+      "volume and number" number either.or.check
+    }
+  if$
+}
+
+FUNCTION {format.number.series}
+{ volume empty$
+    { number empty$
+        { series field.or.null }
+        { output.state mid.sentence =
+            { "number" }
+            { "Number" }
+          if$
+          number tie.or.space.connect
+          series empty$
+            { "there's a number but no series in " cite$ * warning$ }
+            { " in " * series * }
+          if$
+        }
+      if$
+    }
+    { "" }
+  if$
+}
+
+FUNCTION {format.edition}
+{ edition empty$
+    { "" }
+    { output.state mid.sentence =
+        { edition "l" change.case$ " edition" * }
+        { edition "t" change.case$ " edition" * }
+      if$
+    }
+  if$
+}
+
+INTEGERS { multiresult }
+
+FUNCTION {multi.page.check}
+{ 't :=
+  #0 'multiresult :=
+    { multiresult not
+      t empty$ not
+      and
+    }
+    { t #1 #1 substring$
+      duplicate$ "-" =
+      swap$ duplicate$ "," =
+      swap$ "+" =
+      or or
+        { #1 'multiresult := }
+        { t #2 global.max$ substring$ 't := }
+      if$
+    }
+  while$
+  multiresult
+}
+
+FUNCTION {format.pages}
+{ pages empty$
+    { "" }
+    { pages multi.page.check
+        { "pp.\ " pages n.dashify tie.or.space.connect }
+        { "pp.\ " pages tie.or.space.connect }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.eid}
+{ eid empty$
+    { "" }
+    { "art." eid tie.or.space.connect }
+  if$
+}
+
+FUNCTION {format.vol.num.pages}
+{ volume field.or.null
+  number empty$
+    'skip$
+    { "\penalty0 (" number * ")" * *
+      volume empty$
+        { "there's a number but no volume in " cite$ * warning$ }
+        'skip$
+      if$
+    }
+  if$
+  pages empty$
+    'skip$
+    { duplicate$ empty$
+        { pop$ format.pages }
+        { ":\penalty0 " * pages n.dashify * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.vol.num.eid}
+{ volume field.or.null
+  number empty$
+    'skip$
+    { "\penalty0 (" number * ")" * *
+      volume empty$
+        { "there's a number but no volume in " cite$ * warning$ }
+        'skip$
+      if$
+    }
+  if$
+  eid empty$
+    'skip$
+    { duplicate$ empty$
+        { pop$ format.eid }
+        { ":\penalty0 " * eid * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.chapter.pages}
+{ chapter empty$
+    'format.pages
+    { type empty$
+        { "chapter" }
+        { type "l" change.case$ }
+      if$
+      chapter tie.or.space.connect
+      pages empty$
+        'skip$
+        { ", " * format.pages * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {format.in.ed.booktitle}
+{ booktitle empty$
+    { "" }
+    { editor empty$
+        { "In " booktitle emphasize * }
+        { "In " format.editors * ", " * booktitle emphasize * }
+      if$
+    }
+  if$
+}
+
+FUNCTION {empty.misc.check}
+{ author empty$ title empty$ howpublished empty$
+  month empty$ year empty$ note empty$
+  and and and and and
+  key empty$ not and
+    { "all relevant fields are empty in " cite$ * warning$ }
+    'skip$
+  if$
+}
+
+FUNCTION {format.thesis.type}
+{ type empty$
+    'skip$
+    { pop$
+      type "t" change.case$
+    }
+  if$
+}
+
+FUNCTION {format.tr.number}
+{ type empty$
+    { "Technical Report" }
+    'type
+  if$
+  number empty$
+    { "t" change.case$ }
+    { number tie.or.space.connect }
+  if$
+}
+
+FUNCTION {format.article.crossref}
+{ key empty$
+    { journal empty$
+        { "need key or journal for " cite$ * " to crossref " * crossref *
+          warning$
+          ""
+        }
+        { "In \emph{" journal * "}" * }
+      if$
+    }
+    { "In " }
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {format.book.crossref}
+{ volume empty$
+    { "empty volume in " cite$ * "'s crossref of " * crossref * warning$
+      "In "
+    }
+    { "Volume" volume tie.or.space.connect
+      " of " *
+    }
+  if$
+  editor empty$
+  editor field.or.null author field.or.null =
+  or
+    { key empty$
+        { series empty$
+            { "need editor, key, or series for " cite$ * " to crossref " *
+              crossref * warning$
+              "" *
+            }
+            { "\emph{" * series * "}" * }
+          if$
+        }
+        'skip$
+      if$
+    }
+    'skip$
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {format.incoll.inproc.crossref}
+{ editor empty$
+  editor field.or.null author field.or.null =
+  or
+    { key empty$
+        { booktitle empty$
+            { "need editor, key, or booktitle for " cite$ * " to crossref " *
+              crossref * warning$
+              ""
+            }
+            { "In \emph{" booktitle * "}" * }
+          if$
+        }
+        { "In " }
+      if$
+    }
+    { "In " }
+  if$
+  " \citet{" * crossref * "}" *
+}
+
+FUNCTION {article}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { journal emphasize "journal" output.check
+      eid empty$
+        { format.vol.num.pages output }
+        { format.vol.num.eid output }
+      if$
+      format.date "year" output.check
+    }
+    { format.article.crossref output.nonnull
+      eid empty$
+        { format.pages output }
+        { format.eid output }
+      if$
+    }
+  if$
+  format.issn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {book}
+{ output.bibitem
+  author empty$
+    { format.editors "author and editor" output.check
+      editor format.key output
+    }
+    { format.authors output.nonnull
+      crossref missing$
+        { "author and editor" editor either.or.check }
+        'skip$
+      if$
+    }
+  if$
+  new.block
+  format.btitle "title" output.check
+  crossref missing$
+    { format.bvolume output
+      new.block
+      format.number.series output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+    }
+    { new.block
+      format.book.crossref output.nonnull
+    }
+  if$
+  format.edition output
+  format.date "year" output.check
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {booklet}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  new.block
+  format.title "title" output.check
+  howpublished address new.block.checkb
+  howpublished output
+  address output
+  format.date output
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {inbook}
+{ output.bibitem
+  author empty$
+    { format.editors "author and editor" output.check
+      editor format.key output
+    }
+    { format.authors output.nonnull
+      crossref missing$
+        { "author and editor" editor either.or.check }
+        'skip$
+      if$
+    }
+  if$
+  new.block
+  format.btitle "title" output.check
+  crossref missing$
+    { format.bvolume output
+      format.chapter.pages "chapter and pages" output.check
+      new.block
+      format.number.series output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+    }
+    { format.chapter.pages "chapter and pages" output.check
+      new.block
+      format.book.crossref output.nonnull
+    }
+  if$
+  format.edition output
+  format.date "year" output.check
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {incollection}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { format.in.ed.booktitle "booktitle" output.check
+      format.bvolume output
+      format.number.series output
+      format.chapter.pages output
+      new.sentence
+      publisher "publisher" output.check
+      address output
+      format.edition output
+      format.date "year" output.check
+    }
+    { format.incoll.inproc.crossref output.nonnull
+      format.chapter.pages output
+    }
+  if$
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {inproceedings}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  crossref missing$
+    { format.in.ed.booktitle "booktitle" output.check
+      format.bvolume output
+      format.number.series output
+      format.pages output
+      address empty$
+        { organization publisher new.sentence.checkb
+          organization output
+          publisher output
+          format.date "year" output.check
+        }
+        { address output.nonnull
+          format.date "year" output.check
+          new.sentence
+          organization output
+          publisher output
+        }
+      if$
+    }
+    { format.incoll.inproc.crossref output.nonnull
+      format.pages output
+    }
+  if$
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {conference} { inproceedings }
+
+FUNCTION {manual}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  new.block
+  format.btitle "title" output.check
+  organization address new.block.checkb
+  organization output
+  address output
+  format.edition output
+  format.date output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {mastersthesis}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  "Master's thesis" format.thesis.type output.nonnull
+  school "school" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {misc}
+{ output.bibitem
+  format.authors output
+  author format.key output
+  title howpublished new.block.checkb
+  format.title output
+  howpublished new.block.checka
+  howpublished output
+  format.date output
+  format.issn output
+  format.url output
+  new.block
+  note output
+  fin.entry
+  empty.misc.check
+}
+
+FUNCTION {phdthesis}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.btitle "title" output.check
+  new.block
+  "PhD thesis" format.thesis.type output.nonnull
+  school "school" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {proceedings}
+{ output.bibitem
+  format.editors output
+  editor format.key output
+  new.block
+  format.btitle "title" output.check
+  format.bvolume output
+  format.number.series output
+  address output
+  format.date "year" output.check
+  new.sentence
+  organization output
+  publisher output
+  format.isbn output
+  format.doi output
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {techreport}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  format.tr.number output.nonnull
+  institution "institution" output.check
+  address output
+  format.date "year" output.check
+  format.url output
+  new.block
+  note output
+  fin.entry
+}
+
+FUNCTION {unpublished}
+{ output.bibitem
+  format.authors "author" output.check
+  author format.key output
+  new.block
+  format.title "title" output.check
+  new.block
+  note "note" output.check
+  format.date output
+  format.url output
+  fin.entry
+}
+
+FUNCTION {default.type} { misc }
+
+
+MACRO {jan} {"January"}
+
+MACRO {feb} {"February"}
+
+MACRO {mar} {"March"}
+
+MACRO {apr} {"April"}
+
+MACRO {may} {"May"}
+
+MACRO {jun} {"June"}
+
+MACRO {jul} {"July"}
+
+MACRO {aug} {"August"}
+
+MACRO {sep} {"September"}
+
+MACRO {oct} {"October"}
+
+MACRO {nov} {"November"}
+
+MACRO {dec} {"December"}
+
+
+
+MACRO {acmcs} {"ACM Computing Surveys"}
+
+MACRO {acta} {"Acta Informatica"}
+
+MACRO {cacm} {"Communications of the ACM"}
+
+MACRO {ibmjrd} {"IBM Journal of Research and Development"}
+
+MACRO {ibmsj} {"IBM Systems Journal"}
+
+MACRO {ieeese} {"IEEE Transactions on Software Engineering"}
+
+MACRO {ieeetc} {"IEEE Transactions on Computers"}
+
+MACRO {ieeetcad}
+ {"IEEE Transactions on Computer-Aided Design of Integrated Circuits"}
+
+MACRO {ipl} {"Information Processing Letters"}
+
+MACRO {jacm} {"Journal of the ACM"}
+
+MACRO {jcss} {"Journal of Computer and System Sciences"}
+
+MACRO {scp} {"Science of Computer Programming"}
+
+MACRO {sicomp} {"SIAM Journal on Computing"}
+
+MACRO {tocs} {"ACM Transactions on Computer Systems"}
+
+MACRO {tods} {"ACM Transactions on Database Systems"}
+
+MACRO {tog} {"ACM Transactions on Graphics"}
+
+MACRO {toms} {"ACM Transactions on Mathematical Software"}
+
+MACRO {toois} {"ACM Transactions on Office Information Systems"}
+
+MACRO {toplas} {"ACM Transactions on Programming Languages and Systems"}
+
+MACRO {tcs} {"Theoretical Computer Science"}
+
+
+READ
+
+FUNCTION {sortify}
+{ purify$
+  "l" change.case$
+}
+
+INTEGERS { len }
+
+FUNCTION {chop.word}
+{ 's :=
+  'len :=
+  s #1 len substring$ =
+    { s len #1 + global.max$ substring$ }
+    's
+  if$
+}
+
+FUNCTION {format.lab.names}
+{ 's :=
+  s #1 "{vv~}{ll}" format.name$
+  s num.names$ duplicate$
+  #2 >
+    { pop$ " et~al." * }
+    { #2 <
+        'skip$
+        { s #2 "{ff }{vv }{ll}{ jj}" format.name$ "others" =
+            { " et~al." * }
+            { " \& " * s #2 "{vv~}{ll}" format.name$ * }
+          if$
+        }
+      if$
+    }
+  if$
+}
+
+FUNCTION {author.key.label}
+{ author empty$
+    { key empty$
+        { cite$ #1 #3 substring$ }
+        'key
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {author.editor.key.label}
+{ author empty$
+    { editor empty$
+        { key empty$
+            { cite$ #1 #3 substring$ }
+            'key
+          if$
+        }
+        { editor format.lab.names }
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {author.key.organization.label}
+{ author empty$
+    { key empty$
+        { organization empty$
+            { cite$ #1 #3 substring$ }
+            { "The " #4 organization chop.word #3 text.prefix$ }
+          if$
+        }
+        'key
+      if$
+    }
+    { author format.lab.names }
+  if$
+}
+
+FUNCTION {editor.key.organization.label}
+{ editor empty$
+    { key empty$
+        { organization empty$
+            { cite$ #1 #3 substring$ }
+            { "The " #4 organization chop.word #3 text.prefix$ }
+          if$
+        }
+        'key
+      if$
+    }
+    { editor format.lab.names }
+  if$
+}
+
+FUNCTION {calc.short.authors}
+{ type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.key.label
+    { type$ "proceedings" =
+        'editor.key.organization.label
+        { type$ "manual" =
+            'author.key.organization.label
+            'author.key.label
+          if$
+        }
+      if$
+    }
+  if$
+  'short.list :=
+}
+
+FUNCTION {calc.label}
+{ calc.short.authors
+  short.list
+  "("
+  *
+  year duplicate$ empty$
+  short.list key field.or.null = or
+     { pop$ "" }
+     'skip$
+  if$
+  *
+  'label :=
+}
+
+FUNCTION {sort.format.names}
+{ 's :=
+  #1 'nameptr :=
+  ""
+  s num.names$ 'numnames :=
+  numnames 'namesleft :=
+    { namesleft #0 > }
+    {
+      s nameptr "{vv{ } }{ll{ }}{  ff{ }}{  jj{ }}" format.name$ 't :=
+      nameptr #1 >
+        {
+          "   "  *
+          namesleft #1 = t "others" = and
+            { "zzzzz" * }
+            { numnames #2 > nameptr #2 = and
+                { "zz" * year field.or.null * "   " * }
+                'skip$
+              if$
+              t sortify *
+            }
+          if$
+        }
+        { t sortify * }
+      if$
+      nameptr #1 + 'nameptr :=
+      namesleft #1 - 'namesleft :=
+    }
+  while$
+}
+
+FUNCTION {sort.format.title}
+{ 't :=
+  "A " #2
+    "An " #3
+      "The " #4 t chop.word
+    chop.word
+  chop.word
+  sortify
+  #1 global.max$ substring$
+}
+
+FUNCTION {author.sort}
+{ author empty$
+    { key empty$
+        { "to sort, need author or key in " cite$ * warning$
+          ""
+        }
+        { key sortify }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {author.editor.sort}
+{ author empty$
+    { editor empty$
+        { key empty$
+            { "to sort, need author, editor, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { editor sort.format.names }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {author.organization.sort}
+{ author empty$
+    { organization empty$
+        { key empty$
+            { "to sort, need author, organization, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { "The " #4 organization chop.word sortify }
+      if$
+    }
+    { author sort.format.names }
+  if$
+}
+
+FUNCTION {editor.organization.sort}
+{ editor empty$
+    { organization empty$
+        { key empty$
+            { "to sort, need editor, organization, or key in " cite$ * warning$
+              ""
+            }
+            { key sortify }
+          if$
+        }
+        { "The " #4 organization chop.word sortify }
+      if$
+    }
+    { editor sort.format.names }
+  if$
+}
+
+
+FUNCTION {presort}
+{ calc.label
+  label sortify
+  "    "
+  *
+  type$ "book" =
+  type$ "inbook" =
+  or
+    'author.editor.sort
+    { type$ "proceedings" =
+        'editor.organization.sort
+        { type$ "manual" =
+            'author.organization.sort
+            'author.sort
+          if$
+        }
+      if$
+    }
+  if$
+  "    "
+  *
+  year field.or.null sortify
+  *
+  "    "
+  *
+  cite$
+  *
+  #1 entry.max$ substring$
+  'sort.label :=
+  sort.label *
+  #1 entry.max$ substring$
+  'sort.key$ :=
+}
+
+ITERATE {presort}
+
+SORT
+
+STRINGS { longest.label last.label next.extra }
+
+INTEGERS { longest.label.width last.extra.num number.label }
+
+FUNCTION {initialize.longest.label}
+{ "" 'longest.label :=
+  #0 int.to.chr$ 'last.label :=
+  "" 'next.extra :=
+  #0 'longest.label.width :=
+  #0 'last.extra.num :=
+  #0 'number.label :=
+}
+
+FUNCTION {forward.pass}
+{ last.label label =
+    { last.extra.num #1 + 'last.extra.num :=
+      last.extra.num int.to.chr$ 'extra.label :=
+    }
+    { "a" chr.to.int$ 'last.extra.num :=
+      "" 'extra.label :=
+      label 'last.label :=
+    }
+  if$
+  number.label #1 + 'number.label :=
+}
+
+FUNCTION {reverse.pass}
+{ next.extra "b" =
+    { "a" 'extra.label := }
+    'skip$
+  if$
+  extra.label 'next.extra :=
+  extra.label
+  duplicate$ empty$
+    'skip$
+    { "{\natexlab{" swap$ * "}}" * }
+  if$
+  'extra.label :=
+  label extra.label * 'label :=
+}
+
+EXECUTE {initialize.longest.label}
+
+ITERATE {forward.pass}
+
+REVERSE {reverse.pass}
+
+FUNCTION {bib.sort.order}
+{ sort.label  'sort.key$ :=
+}
+
+ITERATE {bib.sort.order}
+
+SORT
+
+FUNCTION {begin.bib}
+{   preamble$ empty$
+    'skip$
+    { preamble$ write$ newline$ }
+  if$
+  "\begin{thebibliography}{" number.label int.to.str$ * "}" *
+  write$ newline$
+  "\providecommand{\natexlab}[1]{#1}"
+  write$ newline$
+  "\providecommand{\url}[1]{\texttt{#1}}"
+  write$ newline$
+  "\expandafter\ifx\csname urlstyle\endcsname\relax"
+  write$ newline$
+  "  \providecommand{\doi}[1]{doi: #1}\else"
+  write$ newline$
+  "  \providecommand{\doi}{doi: \begingroup \urlstyle{rm}\Url}\fi"
+  write$ newline$
+}
+
+EXECUTE {begin.bib}
+
+EXECUTE {init.state.consts}
+
+ITERATE {call.type$}
+
+FUNCTION {end.bib}
+{ newline$
+  "\end{thebibliography}" write$ newline$
+}
+
+EXECUTE {end.bib}

outputs/outputs_20230608_115759/iclr2022_conference.sty

@@ -0,0 +1,245 @@
+%%%% ICLR Macros (LaTex)
+%%%% Adapted by Hugo Larochelle from the NIPS stylefile Macros
+%%%% Style File
+%%%% Dec 12, 1990   Rev Aug 14, 1991; Sept, 1995; April, 1997; April, 1999; October 2014
+
+% This file can be used with Latex2e whether running in main mode, or
+% 2.09 compatibility mode.
+%
+% If using main mode, you need to include the commands
+%             \documentclass{article}
+%             \usepackage{iclr14submit_e,times}
+%
+
+% Change the overall width of the page.  If these parameters are
+%       changed, they will require corresponding changes in the
+%       maketitle section.
+%
+\usepackage{eso-pic} % used by \AddToShipoutPicture
+\RequirePackage{fancyhdr}
+\RequirePackage{natbib}
+
+% modification to natbib citations
+\setcitestyle{authoryear,round,citesep={;},aysep={,},yysep={;}}
+
+\renewcommand{\topfraction}{0.95}   % let figure take up nearly whole page
+\renewcommand{\textfraction}{0.05}  % let figure take up nearly whole page
+
+% Define iclrfinal, set to true if iclrfinalcopy is defined
+\newif\ificlrfinal
+\iclrfinalfalse
+\def\iclrfinalcopy{\iclrfinaltrue}
+\font\iclrtenhv  = phvb at 8pt
+
+% Specify the dimensions of each page
+
+\setlength{\paperheight}{11in}
+\setlength{\paperwidth}{8.5in}
+
+
+\oddsidemargin .5in    %   Note \oddsidemargin = \evensidemargin
+\evensidemargin .5in
+\marginparwidth 0.07 true in
+%\marginparwidth 0.75 true in
+%\topmargin 0 true pt           % Nominal distance from top of page to top of
+%\topmargin 0.125in
+\topmargin -0.625in
+\addtolength{\headsep}{0.25in}
+\textheight 9.0 true in       % Height of text (including footnotes & figures)
+\textwidth 5.5 true in        % Width of text line.
+\widowpenalty=10000
+\clubpenalty=10000
+
+% \thispagestyle{empty}        \pagestyle{empty}
+\flushbottom \sloppy
+
+% We're never going to need a table of contents, so just flush it to
+% save space --- suggested by drstrip@sandia-2
+\def\addcontentsline#1#2#3{}
+
+% Title stuff, taken from deproc.
+\def\maketitle{\par
+\begingroup
+   \def\thefootnote{\fnsymbol{footnote}}
+   \def\@makefnmark{\hbox to 0pt{$^{\@thefnmark}$\hss}} % for perfect author
+                                                        % name centering
+%   The footnote-mark was overlapping the footnote-text,
+%   added the following to fix this problem               (MK)
+   \long\def\@makefntext##1{\parindent 1em\noindent
+                            \hbox to1.8em{\hss $\m@th ^{\@thefnmark}$}##1}
+   \@maketitle \@thanks
+\endgroup
+\setcounter{footnote}{0}
+\let\maketitle\relax \let\@maketitle\relax
+\gdef\@thanks{}\gdef\@author{}\gdef\@title{}\let\thanks\relax}
+
+% The toptitlebar has been raised to top-justify the first page
+
+\usepackage{fancyhdr}
+\pagestyle{fancy}
+\fancyhead{}
+
+% Title (includes both anonimized and non-anonimized versions)
+\def\@maketitle{\vbox{\hsize\textwidth
+%\linewidth\hsize \vskip 0.1in \toptitlebar \centering
+{\LARGE\sc \@title\par}
+%\bottomtitlebar % \vskip 0.1in %  minus
+\ificlrfinal
+    \lhead{Published as a conference paper at ICLR 2022}
+    \def\And{\end{tabular}\hfil\linebreak[0]\hfil
+            \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
+  \def\AND{\end{tabular}\hfil\linebreak[4]\hfil
+            \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
+    \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\@author\end{tabular}%
+\else
+       \lhead{Under review as a conference paper at ICLR 2022}
+   \def\And{\end{tabular}\hfil\linebreak[0]\hfil
+            \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
+  \def\AND{\end{tabular}\hfil\linebreak[4]\hfil
+            \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}\ignorespaces}%
+    \begin{tabular}[t]{l}\bf\rule{\z@}{24pt}Anonymous authors\\Paper under double-blind review\end{tabular}%
+\fi
+\vskip 0.3in minus 0.1in}}
+
+\renewenvironment{abstract}{\vskip.075in\centerline{\large\sc
+Abstract}\vspace{0.5ex}\begin{quote}}{\par\end{quote}\vskip 1ex}
+
+% sections with less space
+\def\section{\@startsection {section}{1}{\z@}{-2.0ex plus
+    -0.5ex minus -.2ex}{1.5ex plus 0.3ex
+minus0.2ex}{\large\sc\raggedright}}
+
+\def\subsection{\@startsection{subsection}{2}{\z@}{-1.8ex plus
+-0.5ex minus -.2ex}{0.8ex plus .2ex}{\normalsize\sc\raggedright}}
+\def\subsubsection{\@startsection{subsubsection}{3}{\z@}{-1.5ex
+plus      -0.5ex minus -.2ex}{0.5ex plus
+.2ex}{\normalsize\sc\raggedright}}
+\def\paragraph{\@startsection{paragraph}{4}{\z@}{1.5ex plus
+0.5ex minus .2ex}{-1em}{\normalsize\bf}}
+\def\subparagraph{\@startsection{subparagraph}{5}{\z@}{1.5ex plus
+  0.5ex minus .2ex}{-1em}{\normalsize\sc}}
+\def\subsubsubsection{\vskip
+5pt{\noindent\normalsize\rm\raggedright}}
+
+
+% Footnotes
+\footnotesep 6.65pt %
+\skip\footins 9pt plus 4pt minus 2pt
+\def\footnoterule{\kern-3pt \hrule width 12pc \kern 2.6pt }
+\setcounter{footnote}{0}
+
+% Lists and paragraphs
+\parindent 0pt
+\topsep 4pt plus 1pt minus 2pt
+\partopsep 1pt plus 0.5pt minus 0.5pt
+\itemsep 2pt plus 1pt minus 0.5pt
+\parsep 2pt plus 1pt minus 0.5pt
+\parskip .5pc
+
+
+%\leftmargin2em
+\leftmargin3pc
+\leftmargini\leftmargin \leftmarginii 2em
+\leftmarginiii 1.5em \leftmarginiv 1.0em \leftmarginv .5em
+
+%\labelsep \labelsep 5pt
+
+\def\@listi{\leftmargin\leftmargini}
+\def\@listii{\leftmargin\leftmarginii
+   \labelwidth\leftmarginii\advance\labelwidth-\labelsep
+   \topsep 2pt plus 1pt minus 0.5pt
+   \parsep 1pt plus 0.5pt minus 0.5pt
+   \itemsep \parsep}
+\def\@listiii{\leftmargin\leftmarginiii
+    \labelwidth\leftmarginiii\advance\labelwidth-\labelsep
+    \topsep 1pt plus 0.5pt minus 0.5pt
+    \parsep \z@ \partopsep 0.5pt plus 0pt minus 0.5pt
+    \itemsep \topsep}
+\def\@listiv{\leftmargin\leftmarginiv
+     \labelwidth\leftmarginiv\advance\labelwidth-\labelsep}
+\def\@listv{\leftmargin\leftmarginv
+     \labelwidth\leftmarginv\advance\labelwidth-\labelsep}
+\def\@listvi{\leftmargin\leftmarginvi
+     \labelwidth\leftmarginvi\advance\labelwidth-\labelsep}
+
+\abovedisplayskip 7pt plus2pt minus5pt%
+\belowdisplayskip \abovedisplayskip
+\abovedisplayshortskip  0pt plus3pt%
+\belowdisplayshortskip  4pt plus3pt minus3pt%
+
+% Less leading in most fonts (due to the narrow columns)
+% The choices were between 1-pt and 1.5-pt leading
+%\def\@normalsize{\@setsize\normalsize{11pt}\xpt\@xpt} % got rid of @ (MK)
+\def\normalsize{\@setsize\normalsize{11pt}\xpt\@xpt}
+\def\small{\@setsize\small{10pt}\ixpt\@ixpt}
+\def\footnotesize{\@setsize\footnotesize{10pt}\ixpt\@ixpt}
+\def\scriptsize{\@setsize\scriptsize{8pt}\viipt\@viipt}
+\def\tiny{\@setsize\tiny{7pt}\vipt\@vipt}
+\def\large{\@setsize\large{14pt}\xiipt\@xiipt}
+\def\Large{\@setsize\Large{16pt}\xivpt\@xivpt}
+\def\LARGE{\@setsize\LARGE{20pt}\xviipt\@xviipt}
+\def\huge{\@setsize\huge{23pt}\xxpt\@xxpt}
+\def\Huge{\@setsize\Huge{28pt}\xxvpt\@xxvpt}
+
+\def\toptitlebar{\hrule height4pt\vskip .25in\vskip-\parskip}
+
+\def\bottomtitlebar{\vskip .29in\vskip-\parskip\hrule height1pt\vskip
+.09in} %
+%Reduced second vskip to compensate for adding the strut in \@author
+
+
+%% % Vertical Ruler
+%% % This code is, largely, from the CVPR 2010 conference style file
+%% % ----- define vruler
+%% \makeatletter
+%% \newbox\iclrrulerbox
+%% \newcount\iclrrulercount
+%% \newdimen\iclrruleroffset
+%% \newdimen\cv@lineheight
+%% \newdimen\cv@boxheight
+%% \newbox\cv@tmpbox
+%% \newcount\cv@refno
+%% \newcount\cv@tot
+%% % NUMBER with left flushed zeros  \fillzeros[<WIDTH>]<NUMBER>
+%% \newcount\cv@tmpc@ \newcount\cv@tmpc
+%% \def\fillzeros[#1]#2{\cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi
+%% \cv@tmpc=1 %
+%% \loop\ifnum\cv@tmpc@<10 \else \divide\cv@tmpc@ by 10 \advance\cv@tmpc by 1 \fi
+%%    \ifnum\cv@tmpc@=10\relax\cv@tmpc@=11\relax\fi \ifnum\cv@tmpc@>10 \repeat
+%% \ifnum#2<0\advance\cv@tmpc1\relax-\fi
+%% \loop\ifnum\cv@tmpc<#1\relax0\advance\cv@tmpc1\relax\fi \ifnum\cv@tmpc<#1 \repeat
+%% \cv@tmpc@=#2\relax\ifnum\cv@tmpc@<0\cv@tmpc@=-\cv@tmpc@\fi \relax\the\cv@tmpc@}%
+%% % \makevruler[<SCALE>][<INITIAL_COUNT>][<STEP>][<DIGITS>][<HEIGHT>]
+%% \def\makevruler[#1][#2][#3][#4][#5]{\begingroup\offinterlineskip
+%% \textheight=#5\vbadness=10000\vfuzz=120ex\overfullrule=0pt%
+%% \global\setbox\iclrrulerbox=\vbox to \textheight{%
+%% {\parskip=0pt\hfuzz=150em\cv@boxheight=\textheight
+%% \cv@lineheight=#1\global\iclrrulercount=#2%
+%% \cv@tot\cv@boxheight\divide\cv@tot\cv@lineheight\advance\cv@tot2%
+%% \cv@refno1\vskip-\cv@lineheight\vskip1ex%
+%% \loop\setbox\cv@tmpbox=\hbox to0cm{{\iclrtenhv\hfil\fillzeros[#4]\iclrrulercount}}%
+%% \ht\cv@tmpbox\cv@lineheight\dp\cv@tmpbox0pt\box\cv@tmpbox\break
+%% \advance\cv@refno1\global\advance\iclrrulercount#3\relax
+%% \ifnum\cv@refno<\cv@tot\repeat}}\endgroup}%
+%% \makeatother
+%% % ----- end of vruler
+
+%% % \makevruler[<SCALE>][<INITIAL_COUNT>][<STEP>][<DIGITS>][<HEIGHT>]
+%% \def\iclrruler#1{\makevruler[12pt][#1][1][3][0.993\textheight]\usebox{\iclrrulerbox}}
+%% \AddToShipoutPicture{%
+%% \ificlrfinal\else
+%% \iclrruleroffset=\textheight
+%% \advance\iclrruleroffset by -3.7pt
+%%   \color[rgb]{.7,.7,.7}
+%%   \AtTextUpperLeft{%
+%%     \put(\LenToUnit{-35pt},\LenToUnit{-\iclrruleroffset}){%left ruler
+%%       \iclrruler{\iclrrulercount}}
+%%   }
+%% \fi
+%% }
+%%% To add a vertical bar on the side
+%\AddToShipoutPicture{
+%\AtTextLowerLeft{
+%\hspace*{-1.8cm}
+%\colorbox[rgb]{0.7,0.7,0.7}{\small \parbox[b][\textheight]{0.1cm}{}}}
+%}

outputs/outputs_20230608_115759/introduction.tex

@@ -0,0 +1,11 @@
+\section{Introduction}
+
+The rapid development of artificial intelligence and machine learning has led to significant advancements in various domains, including reinforcement learning (RL) and multi-agent systems. One particularly notable application of RL is in the domain of Atari games, where deep learning models have been successfully employed to learn control policies directly from high-dimensional sensory input \citep{mnih2013playing}. However, the centralized nature of traditional RL algorithms poses challenges in terms of scalability and privacy, motivating the exploration of decentralized RL approaches \citep{liu2022federated}. In this paper, we address the problem of playing Atari games using decentralized reinforcement learning, aiming to develop a scalable and privacy-preserving solution that maintains high performance.
+
+Our proposed solution builds upon recent advancements in decentralized RL, which have demonstrated promising results in various scenarios, such as collision avoidance \citep{thumiger2022a}, cooperative multi-agent reinforcement learning \citep{su2022ma2ql}, and edge-computing-empowered Internet of Things (IoT) networks \citep{lei2022adaptive}. While these works provide valuable insights, our approach specifically targets the unique challenges associated with playing Atari games, such as high-dimensional sensory input and complex decision-making processes. By leveraging the strengths of decentralized RL algorithms, we aim to outperform centralized approaches in terms of scalability and privacy while maintaining competitive performance.
+
+This paper makes three novel contributions to the field of decentralized reinforcement learning. First, we present a new decentralized RL algorithm specifically tailored for playing Atari games, addressing the challenges of high-dimensional sensory input and complex decision-making. Second, we provide a comprehensive analysis of the algorithm's performance, comparing it to state-of-the-art centralized and decentralized RL approaches on a diverse set of Atari games. Finally, we offer insights into the trade-offs between scalability, privacy, and performance in decentralized RL, highlighting the benefits and limitations of our proposed approach.
+
+To contextualize our work, we briefly discuss key related works in the field of decentralized RL. The Safe Dec-PG algorithm, proposed by \citet{lu2021decentralized}, is the first decentralized policy gradient method that accounts for coupled safety constraints in multi-agent reinforcement learning. Another relevant work is the decentralized collision avoidance approach by \citet{thumiger2022a}, which employs a unique architecture incorporating long-short term memory cells and a gradient-based reward function. While these works demonstrate the potential of decentralized RL, our approach specifically targets the challenges associated with playing Atari games, offering a novel solution in this domain.
+
+In summary, this paper presents a novel decentralized RL algorithm for playing Atari games, aiming to achieve high performance while maintaining scalability and privacy. By building upon recent advancements in decentralized RL, we contribute to the growing body of research in this area, offering valuable insights into the trade-offs between scalability, privacy, and performance in decentralized reinforcement learning.

outputs/outputs_20230608_115759/main.tex

@@ -0,0 +1,35 @@
+\documentclass{article} % For LaTeX2e
+\UseRawInputEncoding
+\usepackage{graphicx}
+\usepackage{booktabs}
+\usepackage{iclr2022_conference, times} 
+\input{math_commands.tex} 
+\usepackage{hyperref}
+\usepackage{url} 
+\usepackage{algorithm}
+\usepackage{algpseudocode}
+
+\title{Playing Atari with Decentralized Reinforcement Learning} 
+\author{GPT-4}
+
+\newcommand{\fix}{\marginpar{FIX}}
+\newcommand{\new}{\marginpar{NEW}} 
+ 
+\begin{document} 
+\maketitle
+\input{abstract.tex}
+\input{introduction.tex}
+\input{related works.tex}
+\input{backgrounds.tex}
+\input{methodology.tex}
+\input{experiments.tex}
+\input{conclusion.tex}
+
+\bibliography{ref}
+\bibliographystyle{iclr2022_conference}
+
+%\appendix
+%\section{Appendix}
+%You may include other additional sections here.
+
+\end{document}

outputs/outputs_20230608_115759/math_commands.tex

@@ -0,0 +1,508 @@
+%%%%% NEW MATH DEFINITIONS %%%%%
+
+\usepackage{amsmath,amsfonts,bm}
+
+% Mark sections of captions for referring to divisions of figures
+\newcommand{\figleft}{{\em (Left)}}
+\newcommand{\figcenter}{{\em (Center)}}
+\newcommand{\figright}{{\em (Right)}}
+\newcommand{\figtop}{{\em (Top)}}
+\newcommand{\figbottom}{{\em (Bottom)}}
+\newcommand{\captiona}{{\em (a)}}
+\newcommand{\captionb}{{\em (b)}}
+\newcommand{\captionc}{{\em (c)}}
+\newcommand{\captiond}{{\em (d)}}
+
+% Highlight a newly defined term
+\newcommand{\newterm}[1]{{\bf #1}}
+
+
+% Figure reference, lower-case.
+\def\figref#1{figure~\ref{#1}}
+% Figure reference, capital. For start of sentence
+\def\Figref#1{Figure~\ref{#1}}
+\def\twofigref#1#2{figures \ref{#1} and \ref{#2}}
+\def\quadfigref#1#2#3#4{figures \ref{#1}, \ref{#2}, \ref{#3} and \ref{#4}}
+% Section reference, lower-case.
+\def\secref#1{section~\ref{#1}}
+% Section reference, capital.
+\def\Secref#1{Section~\ref{#1}}
+% Reference to two sections.
+\def\twosecrefs#1#2{sections \ref{#1} and \ref{#2}}
+% Reference to three sections.
+\def\secrefs#1#2#3{sections \ref{#1}, \ref{#2} and \ref{#3}}
+% Reference to an equation, lower-case.
+\def\eqref#1{equation~\ref{#1}}
+% Reference to an equation, upper case
+\def\Eqref#1{Equation~\ref{#1}}
+% A raw reference to an equation---avoid using if possible
+\def\plaineqref#1{\ref{#1}}
+% Reference to a chapter, lower-case.
+\def\chapref#1{chapter~\ref{#1}}
+% Reference to an equation, upper case.
+\def\Chapref#1{Chapter~\ref{#1}}
+% Reference to a range of chapters
+\def\rangechapref#1#2{chapters\ref{#1}--\ref{#2}}
+% Reference to an algorithm, lower-case.
+\def\algref#1{algorithm~\ref{#1}}
+% Reference to an algorithm, upper case.
+\def\Algref#1{Algorithm~\ref{#1}}
+\def\twoalgref#1#2{algorithms \ref{#1} and \ref{#2}}
+\def\Twoalgref#1#2{Algorithms \ref{#1} and \ref{#2}}
+% Reference to a part, lower case
+\def\partref#1{part~\ref{#1}}
+% Reference to a part, upper case
+\def\Partref#1{Part~\ref{#1}}
+\def\twopartref#1#2{parts \ref{#1} and \ref{#2}}
+
+\def\ceil#1{\lceil #1 \rceil}
+\def\floor#1{\lfloor #1 \rfloor}
+\def\1{\bm{1}}
+\newcommand{\train}{\mathcal{D}}
+\newcommand{\valid}{\mathcal{D_{\mathrm{valid}}}}
+\newcommand{\test}{\mathcal{D_{\mathrm{test}}}}
+
+\def\eps{{\epsilon}}
+
+
+% Random variables
+\def\reta{{\textnormal{$\eta$}}}
+\def\ra{{\textnormal{a}}}
+\def\rb{{\textnormal{b}}}
+\def\rc{{\textnormal{c}}}
+\def\rd{{\textnormal{d}}}
+\def\re{{\textnormal{e}}}
+\def\rf{{\textnormal{f}}}
+\def\rg{{\textnormal{g}}}
+\def\rh{{\textnormal{h}}}
+\def\ri{{\textnormal{i}}}
+\def\rj{{\textnormal{j}}}
+\def\rk{{\textnormal{k}}}
+\def\rl{{\textnormal{l}}}
+% rm is already a command, just don't name any random variables m
+\def\rn{{\textnormal{n}}}
+\def\ro{{\textnormal{o}}}
+\def\rp{{\textnormal{p}}}
+\def\rq{{\textnormal{q}}}
+\def\rr{{\textnormal{r}}}
+\def\rs{{\textnormal{s}}}
+\def\rt{{\textnormal{t}}}
+\def\ru{{\textnormal{u}}}
+\def\rv{{\textnormal{v}}}
+\def\rw{{\textnormal{w}}}
+\def\rx{{\textnormal{x}}}
+\def\ry{{\textnormal{y}}}
+\def\rz{{\textnormal{z}}}
+
+% Random vectors
+\def\rvepsilon{{\mathbf{\epsilon}}}
+\def\rvtheta{{\mathbf{\theta}}}
+\def\rva{{\mathbf{a}}}
+\def\rvb{{\mathbf{b}}}
+\def\rvc{{\mathbf{c}}}
+\def\rvd{{\mathbf{d}}}
+\def\rve{{\mathbf{e}}}
+\def\rvf{{\mathbf{f}}}
+\def\rvg{{\mathbf{g}}}
+\def\rvh{{\mathbf{h}}}
+\def\rvu{{\mathbf{i}}}
+\def\rvj{{\mathbf{j}}}
+\def\rvk{{\mathbf{k}}}
+\def\rvl{{\mathbf{l}}}
+\def\rvm{{\mathbf{m}}}
+\def\rvn{{\mathbf{n}}}
+\def\rvo{{\mathbf{o}}}
+\def\rvp{{\mathbf{p}}}
+\def\rvq{{\mathbf{q}}}
+\def\rvr{{\mathbf{r}}}
+\def\rvs{{\mathbf{s}}}
+\def\rvt{{\mathbf{t}}}
+\def\rvu{{\mathbf{u}}}
+\def\rvv{{\mathbf{v}}}
+\def\rvw{{\mathbf{w}}}
+\def\rvx{{\mathbf{x}}}
+\def\rvy{{\mathbf{y}}}
+\def\rvz{{\mathbf{z}}}
+
+% Elements of random vectors
+\def\erva{{\textnormal{a}}}
+\def\ervb{{\textnormal{b}}}
+\def\ervc{{\textnormal{c}}}
+\def\ervd{{\textnormal{d}}}
+\def\erve{{\textnormal{e}}}
+\def\ervf{{\textnormal{f}}}
+\def\ervg{{\textnormal{g}}}
+\def\ervh{{\textnormal{h}}}
+\def\ervi{{\textnormal{i}}}
+\def\ervj{{\textnormal{j}}}
+\def\ervk{{\textnormal{k}}}
+\def\ervl{{\textnormal{l}}}
+\def\ervm{{\textnormal{m}}}
+\def\ervn{{\textnormal{n}}}
+\def\ervo{{\textnormal{o}}}
+\def\ervp{{\textnormal{p}}}
+\def\ervq{{\textnormal{q}}}
+\def\ervr{{\textnormal{r}}}
+\def\ervs{{\textnormal{s}}}
+\def\ervt{{\textnormal{t}}}
+\def\ervu{{\textnormal{u}}}
+\def\ervv{{\textnormal{v}}}
+\def\ervw{{\textnormal{w}}}
+\def\ervx{{\textnormal{x}}}
+\def\ervy{{\textnormal{y}}}
+\def\ervz{{\textnormal{z}}}
+
+% Random matrices
+\def\rmA{{\mathbf{A}}}
+\def\rmB{{\mathbf{B}}}
+\def\rmC{{\mathbf{C}}}
+\def\rmD{{\mathbf{D}}}
+\def\rmE{{\mathbf{E}}}
+\def\rmF{{\mathbf{F}}}
+\def\rmG{{\mathbf{G}}}
+\def\rmH{{\mathbf{H}}}
+\def\rmI{{\mathbf{I}}}
+\def\rmJ{{\mathbf{J}}}
+\def\rmK{{\mathbf{K}}}
+\def\rmL{{\mathbf{L}}}
+\def\rmM{{\mathbf{M}}}
+\def\rmN{{\mathbf{N}}}
+\def\rmO{{\mathbf{O}}}
+\def\rmP{{\mathbf{P}}}
+\def\rmQ{{\mathbf{Q}}}
+\def\rmR{{\mathbf{R}}}
+\def\rmS{{\mathbf{S}}}
+\def\rmT{{\mathbf{T}}}
+\def\rmU{{\mathbf{U}}}
+\def\rmV{{\mathbf{V}}}
+\def\rmW{{\mathbf{W}}}
+\def\rmX{{\mathbf{X}}}
+\def\rmY{{\mathbf{Y}}}
+\def\rmZ{{\mathbf{Z}}}
+
+% Elements of random matrices
+\def\ermA{{\textnormal{A}}}
+\def\ermB{{\textnormal{B}}}
+\def\ermC{{\textnormal{C}}}
+\def\ermD{{\textnormal{D}}}
+\def\ermE{{\textnormal{E}}}
+\def\ermF{{\textnormal{F}}}
+\def\ermG{{\textnormal{G}}}
+\def\ermH{{\textnormal{H}}}
+\def\ermI{{\textnormal{I}}}
+\def\ermJ{{\textnormal{J}}}
+\def\ermK{{\textnormal{K}}}
+\def\ermL{{\textnormal{L}}}
+\def\ermM{{\textnormal{M}}}
+\def\ermN{{\textnormal{N}}}
+\def\ermO{{\textnormal{O}}}
+\def\ermP{{\textnormal{P}}}
+\def\ermQ{{\textnormal{Q}}}
+\def\ermR{{\textnormal{R}}}
+\def\ermS{{\textnormal{S}}}
+\def\ermT{{\textnormal{T}}}
+\def\ermU{{\textnormal{U}}}
+\def\ermV{{\textnormal{V}}}
+\def\ermW{{\textnormal{W}}}
+\def\ermX{{\textnormal{X}}}
+\def\ermY{{\textnormal{Y}}}
+\def\ermZ{{\textnormal{Z}}}
+
+% Vectors
+\def\vzero{{\bm{0}}}
+\def\vone{{\bm{1}}}
+\def\vmu{{\bm{\mu}}}
+\def\vtheta{{\bm{\theta}}}
+\def\va{{\bm{a}}}
+\def\vb{{\bm{b}}}
+\def\vc{{\bm{c}}}
+\def\vd{{\bm{d}}}
+\def\ve{{\bm{e}}}
+\def\vf{{\bm{f}}}
+\def\vg{{\bm{g}}}
+\def\vh{{\bm{h}}}
+\def\vi{{\bm{i}}}
+\def\vj{{\bm{j}}}
+\def\vk{{\bm{k}}}
+\def\vl{{\bm{l}}}
+\def\vm{{\bm{m}}}
+\def\vn{{\bm{n}}}
+\def\vo{{\bm{o}}}
+\def\vp{{\bm{p}}}
+\def\vq{{\bm{q}}}
+\def\vr{{\bm{r}}}
+\def\vs{{\bm{s}}}
+\def\vt{{\bm{t}}}
+\def\vu{{\bm{u}}}
+\def\vv{{\bm{v}}}
+\def\vw{{\bm{w}}}
+\def\vx{{\bm{x}}}
+\def\vy{{\bm{y}}}
+\def\vz{{\bm{z}}}
+
+% Elements of vectors
+\def\evalpha{{\alpha}}
+\def\evbeta{{\beta}}
+\def\evepsilon{{\epsilon}}
+\def\evlambda{{\lambda}}
+\def\evomega{{\omega}}
+\def\evmu{{\mu}}
+\def\evpsi{{\psi}}
+\def\evsigma{{\sigma}}
+\def\evtheta{{\theta}}
+\def\eva{{a}}
+\def\evb{{b}}
+\def\evc{{c}}
+\def\evd{{d}}
+\def\eve{{e}}
+\def\evf{{f}}
+\def\evg{{g}}
+\def\evh{{h}}
+\def\evi{{i}}
+\def\evj{{j}}
+\def\evk{{k}}
+\def\evl{{l}}
+\def\evm{{m}}
+\def\evn{{n}}
+\def\evo{{o}}
+\def\evp{{p}}
+\def\evq{{q}}
+\def\evr{{r}}
+\def\evs{{s}}
+\def\evt{{t}}
+\def\evu{{u}}
+\def\evv{{v}}
+\def\evw{{w}}
+\def\evx{{x}}
+\def\evy{{y}}
+\def\evz{{z}}
+
+% Matrix
+\def\mA{{\bm{A}}}
+\def\mB{{\bm{B}}}
+\def\mC{{\bm{C}}}
+\def\mD{{\bm{D}}}
+\def\mE{{\bm{E}}}
+\def\mF{{\bm{F}}}
+\def\mG{{\bm{G}}}
+\def\mH{{\bm{H}}}
+\def\mI{{\bm{I}}}
+\def\mJ{{\bm{J}}}
+\def\mK{{\bm{K}}}
+\def\mL{{\bm{L}}}
+\def\mM{{\bm{M}}}
+\def\mN{{\bm{N}}}
+\def\mO{{\bm{O}}}
+\def\mP{{\bm{P}}}
+\def\mQ{{\bm{Q}}}
+\def\mR{{\bm{R}}}
+\def\mS{{\bm{S}}}
+\def\mT{{\bm{T}}}
+\def\mU{{\bm{U}}}
+\def\mV{{\bm{V}}}
+\def\mW{{\bm{W}}}
+\def\mX{{\bm{X}}}
+\def\mY{{\bm{Y}}}
+\def\mZ{{\bm{Z}}}
+\def\mBeta{{\bm{\beta}}}
+\def\mPhi{{\bm{\Phi}}}
+\def\mLambda{{\bm{\Lambda}}}
+\def\mSigma{{\bm{\Sigma}}}
+
+% Tensor
+\DeclareMathAlphabet{\mathsfit}{\encodingdefault}{\sfdefault}{m}{sl}
+\SetMathAlphabet{\mathsfit}{bold}{\encodingdefault}{\sfdefault}{bx}{n}
+\newcommand{\tens}[1]{\bm{\mathsfit{#1}}}
+\def\tA{{\tens{A}}}
+\def\tB{{\tens{B}}}
+\def\tC{{\tens{C}}}
+\def\tD{{\tens{D}}}
+\def\tE{{\tens{E}}}
+\def\tF{{\tens{F}}}
+\def\tG{{\tens{G}}}
+\def\tH{{\tens{H}}}
+\def\tI{{\tens{I}}}
+\def\tJ{{\tens{J}}}
+\def\tK{{\tens{K}}}
+\def\tL{{\tens{L}}}
+\def\tM{{\tens{M}}}
+\def\tN{{\tens{N}}}
+\def\tO{{\tens{O}}}
+\def\tP{{\tens{P}}}
+\def\tQ{{\tens{Q}}}
+\def\tR{{\tens{R}}}
+\def\tS{{\tens{S}}}
+\def\tT{{\tens{T}}}
+\def\tU{{\tens{U}}}
+\def\tV{{\tens{V}}}
+\def\tW{{\tens{W}}}
+\def\tX{{\tens{X}}}
+\def\tY{{\tens{Y}}}
+\def\tZ{{\tens{Z}}}
+
+
+% Graph
+\def\gA{{\mathcal{A}}}
+\def\gB{{\mathcal{B}}}
+\def\gC{{\mathcal{C}}}
+\def\gD{{\mathcal{D}}}
+\def\gE{{\mathcal{E}}}
+\def\gF{{\mathcal{F}}}
+\def\gG{{\mathcal{G}}}
+\def\gH{{\mathcal{H}}}
+\def\gI{{\mathcal{I}}}
+\def\gJ{{\mathcal{J}}}
+\def\gK{{\mathcal{K}}}
+\def\gL{{\mathcal{L}}}
+\def\gM{{\mathcal{M}}}
+\def\gN{{\mathcal{N}}}
+\def\gO{{\mathcal{O}}}
+\def\gP{{\mathcal{P}}}
+\def\gQ{{\mathcal{Q}}}
+\def\gR{{\mathcal{R}}}
+\def\gS{{\mathcal{S}}}
+\def\gT{{\mathcal{T}}}
+\def\gU{{\mathcal{U}}}
+\def\gV{{\mathcal{V}}}
+\def\gW{{\mathcal{W}}}
+\def\gX{{\mathcal{X}}}
+\def\gY{{\mathcal{Y}}}
+\def\gZ{{\mathcal{Z}}}
+
+% Sets
+\def\sA{{\mathbb{A}}}
+\def\sB{{\mathbb{B}}}
+\def\sC{{\mathbb{C}}}
+\def\sD{{\mathbb{D}}}
+% Don't use a set called E, because this would be the same as our symbol
+% for expectation.
+\def\sF{{\mathbb{F}}}
+\def\sG{{\mathbb{G}}}
+\def\sH{{\mathbb{H}}}
+\def\sI{{\mathbb{I}}}
+\def\sJ{{\mathbb{J}}}
+\def\sK{{\mathbb{K}}}
+\def\sL{{\mathbb{L}}}
+\def\sM{{\mathbb{M}}}
+\def\sN{{\mathbb{N}}}
+\def\sO{{\mathbb{O}}}
+\def\sP{{\mathbb{P}}}
+\def\sQ{{\mathbb{Q}}}
+\def\sR{{\mathbb{R}}}
+\def\sS{{\mathbb{S}}}
+\def\sT{{\mathbb{T}}}
+\def\sU{{\mathbb{U}}}
+\def\sV{{\mathbb{V}}}
+\def\sW{{\mathbb{W}}}
+\def\sX{{\mathbb{X}}}
+\def\sY{{\mathbb{Y}}}
+\def\sZ{{\mathbb{Z}}}
+
+% Entries of a matrix
+\def\emLambda{{\Lambda}}
+\def\emA{{A}}
+\def\emB{{B}}
+\def\emC{{C}}
+\def\emD{{D}}
+\def\emE{{E}}
+\def\emF{{F}}
+\def\emG{{G}}
+\def\emH{{H}}
+\def\emI{{I}}
+\def\emJ{{J}}
+\def\emK{{K}}
+\def\emL{{L}}
+\def\emM{{M}}
+\def\emN{{N}}
+\def\emO{{O}}
+\def\emP{{P}}
+\def\emQ{{Q}}
+\def\emR{{R}}
+\def\emS{{S}}
+\def\emT{{T}}
+\def\emU{{U}}
+\def\emV{{V}}
+\def\emW{{W}}
+\def\emX{{X}}
+\def\emY{{Y}}
+\def\emZ{{Z}}
+\def\emSigma{{\Sigma}}
+
+% entries of a tensor
+% Same font as tensor, without \bm wrapper
+\newcommand{\etens}[1]{\mathsfit{#1}}
+\def\etLambda{{\etens{\Lambda}}}
+\def\etA{{\etens{A}}}
+\def\etB{{\etens{B}}}
+\def\etC{{\etens{C}}}
+\def\etD{{\etens{D}}}
+\def\etE{{\etens{E}}}
+\def\etF{{\etens{F}}}
+\def\etG{{\etens{G}}}
+\def\etH{{\etens{H}}}
+\def\etI{{\etens{I}}}
+\def\etJ{{\etens{J}}}
+\def\etK{{\etens{K}}}
+\def\etL{{\etens{L}}}
+\def\etM{{\etens{M}}}
+\def\etN{{\etens{N}}}
+\def\etO{{\etens{O}}}
+\def\etP{{\etens{P}}}
+\def\etQ{{\etens{Q}}}
+\def\etR{{\etens{R}}}
+\def\etS{{\etens{S}}}
+\def\etT{{\etens{T}}}
+\def\etU{{\etens{U}}}
+\def\etV{{\etens{V}}}
+\def\etW{{\etens{W}}}
+\def\etX{{\etens{X}}}
+\def\etY{{\etens{Y}}}
+\def\etZ{{\etens{Z}}}
+
+% The true underlying data generating distribution
+\newcommand{\pdata}{p_{\rm{data}}}
+% The empirical distribution defined by the training set
+\newcommand{\ptrain}{\hat{p}_{\rm{data}}}
+\newcommand{\Ptrain}{\hat{P}_{\rm{data}}}
+% The model distribution
+\newcommand{\pmodel}{p_{\rm{model}}}
+\newcommand{\Pmodel}{P_{\rm{model}}}
+\newcommand{\ptildemodel}{\tilde{p}_{\rm{model}}}
+% Stochastic autoencoder distributions
+\newcommand{\pencode}{p_{\rm{encoder}}}
+\newcommand{\pdecode}{p_{\rm{decoder}}}
+\newcommand{\precons}{p_{\rm{reconstruct}}}
+
+\newcommand{\laplace}{\mathrm{Laplace}} % Laplace distribution
+
+\newcommand{\E}{\mathbb{E}}
+\newcommand{\Ls}{\mathcal{L}}
+\newcommand{\R}{\mathbb{R}}
+\newcommand{\emp}{\tilde{p}}
+\newcommand{\lr}{\alpha}
+\newcommand{\reg}{\lambda}
+\newcommand{\rect}{\mathrm{rectifier}}
+\newcommand{\softmax}{\mathrm{softmax}}
+\newcommand{\sigmoid}{\sigma}
+\newcommand{\softplus}{\zeta}
+\newcommand{\KL}{D_{\mathrm{KL}}}
+\newcommand{\Var}{\mathrm{Var}}
+\newcommand{\standarderror}{\mathrm{SE}}
+\newcommand{\Cov}{\mathrm{Cov}}
+% Wolfram Mathworld says $L^2$ is for function spaces and $\ell^2$ is for vectors
+% But then they seem to use $L^2$ for vectors throughout the site, and so does
+% wikipedia.
+\newcommand{\normlzero}{L^0}
+\newcommand{\normlone}{L^1}
+\newcommand{\normltwo}{L^2}
+\newcommand{\normlp}{L^p}
+\newcommand{\normmax}{L^\infty}
+
+\newcommand{\parents}{Pa} % See usage in notation.tex. Chosen to match Daphne's book.
+
+\DeclareMathOperator*{\argmax}{arg\,max}
+\DeclareMathOperator*{\argmin}{arg\,min}
+
+\DeclareMathOperator{\sign}{sign}
+\DeclareMathOperator{\Tr}{Tr}
+\let\ab\allowbreak

outputs/outputs_20230608_115759/methodology.tex

@@ -0,0 +1,68 @@
+\section{methodology}
+
+In this section, we present the methodology of our proposed decentralized reinforcement learning (RL) algorithm for playing Atari games. We begin with a high-level overview of the method, followed by a detailed formulation of the algorithm and an explanation of how it overcomes the weaknesses of existing methods. Finally, we highlight the key concepts in our approach and elaborate on their novelty using formulas and figures.
+
+\subsection{Overview of the Proposed Method}
+
+Our proposed method, Decentralized Atari Learning (DAL), combines the strengths of both value-based and policy-based decentralized RL algorithms to address the challenges of high-dimensional sensory input and complex decision-making processes in Atari games. The key components of DAL include a decentralized Q-learning framework, a policy gradient-based optimization technique, and a novel communication mechanism that enables agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. Figure \ref{fig1} provides a high-level illustration of the DAL architecture.
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{fig1.png}
+  \caption{High-level architecture of the Decentralized Atari Learning (DAL) algorithm.}
+  \label{fig1}
+\end{figure}
+
+\subsection{Formulation of the Decentralized Atari Learning Algorithm}
+
+The DAL algorithm is designed to overcome the weaknesses of existing decentralized RL methods by incorporating techniques from deep RL, such as experience replay and target networks, to improve stability and convergence. The algorithm consists of the following main steps:
+
+\begin{algorithm}[h]
+\caption{Decentralized Atari Learning (DAL)}
+\begin{algorithmic}[1]
+\STATE Initialize the decentralized Q-network $Q(s, a; \theta)$ and the target network $Q(s, a; \theta^-)$ with random weights $\theta$ and $\theta^-$.
+\FOR{each agent $i$}
+    \STATE Initialize the experience replay buffer $D_i$.
+    \FOR{each episode}
+        \STATE Initialize the state $s$.
+        \FOR{each time step $t$}
+            \STATE Agent $i$ selects an action $a$ according to its local policy $\pi_i$ and the decentralized Q-network $Q(s, a; \theta)$.
+            \STATE Agent $i$ takes action $a$, observes the next state $s'$ and reward $r$, and stores the transition $(s, a, r, s')$ in its experience replay buffer $D_i$.
+            \STATE Agent $i$ samples a mini-batch of transitions from $D_i$ and computes the target values $y = r + \gamma \max_{a'} Q(s', a'; \theta^-)$.
+            \STATE Agent $i$ updates the decentralized Q-network $Q(s, a; \theta)$ using the policy gradient-based optimization technique.
+            \STATE Agent $i$ updates the target network $Q(s, a; \theta^-)$ with the weights of the decentralized Q-network $Q(s, a; \theta)$.
+            \STATE Agent $i$ communicates with neighboring agents to share information and coordinate actions while preserving privacy and reducing communication overhead.
+            \STATE Update the state $s \leftarrow s'$.
+        \ENDFOR
+    \ENDFOR
+\ENDFOR
+\end{algorithmic}
+\end{algorithm}
+
+\subsection{Key Concepts and Novelty of the Decentralized Atari Learning Algorithm}
+
+The novelty of the DAL algorithm lies in its combination of value-based and policy-based decentralized RL techniques, as well as its unique communication mechanism that enables agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. In this subsection, we elaborate on these key concepts using formulas and figures.
+
+\paragraph{Decentralized Q-learning and Policy Gradient Optimization}
+
+The DAL algorithm builds upon the decentralized Q-learning framework and incorporates a policy gradient-based optimization technique to balance the trade-offs between exploration and exploitation. The decentralized Q-network $Q(s, a; \theta)$ is used to estimate the action-value function, while the policy gradient-based optimization technique is employed to update the network weights $\theta$. This combination allows the algorithm to learn more efficiently in high-dimensional state spaces and complex decision-making processes, as illustrated in Figure \ref{fig2}.
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{fig2.png}
+  \caption{Illustration of the decentralized Q-learning and policy gradient optimization in the DAL algorithm.}
+  \label{fig2}
+\end{figure}
+
+\paragraph{Novel Communication Mechanism}
+
+The communication mechanism in DAL enables agents to share information and coordinate their actions while preserving privacy and reducing communication overhead. This is achieved through a secure and efficient communication protocol that allows agents to exchange only the necessary information for coordination, without revealing their entire state or action history. Figure \ref{fig3} provides an illustration of the communication mechanism in the DAL algorithm.
+
+\begin{figure}[h]
+  \centering
+  \includegraphics[width=0.8\textwidth]{fig3.png}
+  \caption{Illustration of the novel communication mechanism in the DAL algorithm.}
+  \label{fig3}
+\end{figure}
+
+In summary, our proposed Decentralized Atari Learning (DAL) algorithm combines the strengths of both value-based and policy-based decentralized RL techniques and introduces a novel communication mechanism to address the challenges of high-dimensional sensory input and complex decision-making processes in Atari games. The algorithm demonstrates competitive performance compared to centralized methods and outperforms existing decentralized RL algorithms in the Atari domain.

outputs/outputs_20230608_115759/natbib.sty

@@ -0,0 +1,1246 @@
+%%
+%% This is file `natbib.sty',
+%% generated with the docstrip utility.
+%%
+%% The original source files were:
+%%
+%% natbib.dtx  (with options: `package,all')
+%% =============================================
+%% IMPORTANT NOTICE:
+%% 
+%% This program can be redistributed and/or modified under the terms
+%% of the LaTeX Project Public License Distributed from CTAN
+%% archives in directory macros/latex/base/lppl.txt; either
+%% version 1 of the License, or any later version.
+%% 
+%% This is a generated file.
+%% It may not be distributed without the original source file natbib.dtx.
+%% 
+%% Full documentation can be obtained by LaTeXing that original file.
+%% Only a few abbreviated comments remain here to describe the usage.
+%% =============================================
+%% Copyright 1993-2009 Patrick W Daly
+%% Max-Planck-Institut f\"ur Sonnensystemforschung
+%% Max-Planck-Str. 2
+%% D-37191 Katlenburg-Lindau
+%% Germany
+%% E-mail: daly@mps.mpg.de
+\NeedsTeXFormat{LaTeX2e}[1995/06/01]
+\ProvidesPackage{natbib}
+        [2009/07/16 8.31 (PWD, AO)]
+
+ % This package reimplements the LaTeX \cite command to be used for various
+ % citation styles, both author-year and numerical. It accepts BibTeX
+ % output intended for many other packages, and therefore acts as a
+ % general, all-purpose citation-style interface.
+ %
+ % With standard numerical .bst files, only numerical citations are
+ % possible. With an author-year .bst file, both numerical and
+ % author-year citations are possible.
+ %
+ % If author-year citations are selected, \bibitem must have one of the
+ %   following forms:
+ %   \bibitem[Jones et al.(1990)]{key}...
+ %   \bibitem[Jones et al.(1990)Jones, Baker, and Williams]{key}...
+ %   \bibitem[Jones et al., 1990]{key}...
+ %   \bibitem[\protect\citeauthoryear{Jones, Baker, and Williams}{Jones
+ %       et al.}{1990}]{key}...
+ %   \bibitem[\protect\citeauthoryear{Jones et al.}{1990}]{key}...
+ %   \bibitem[\protect\astroncite{Jones et al.}{1990}]{key}...
+ %   \bibitem[\protect\citename{Jones et al., }1990]{key}...
+ %   \harvarditem[Jones et al.]{Jones, Baker, and Williams}{1990}{key}...
+ %
+ % This is either to be made up manually, or to be generated by an
+ % appropriate .bst file with BibTeX.
+ %                            Author-year mode     ||   Numerical mode
+ % Then, \citet{key}  ==>>  Jones et al. (1990)    ||   Jones et al. [21]
+ %       \citep{key}  ==>> (Jones et al., 1990)    ||   [21]
+ % Multiple citations as normal:
+ % \citep{key1,key2}  ==>> (Jones et al., 1990; Smith, 1989) || [21,24]
+ %                           or  (Jones et al., 1990, 1991)  || [21,24]
+ %                           or  (Jones et al., 1990a,b)     || [21,24]
+ % \cite{key} is the equivalent of \citet{key} in author-year mode
+ %                         and  of \citep{key} in numerical mode
+ % Full author lists may be forced with \citet* or \citep*, e.g.
+ %       \citep*{key}      ==>> (Jones, Baker, and Williams, 1990)
+ % Optional notes as:
+ %   \citep[chap. 2]{key}    ==>> (Jones et al., 1990, chap. 2)
+ %   \citep[e.g.,][]{key}    ==>> (e.g., Jones et al., 1990)
+ %   \citep[see][pg. 34]{key}==>> (see Jones et al., 1990, pg. 34)
+ %  (Note: in standard LaTeX, only one note is allowed, after the ref.
+ %   Here, one note is like the standard, two make pre- and post-notes.)
+ %   \citealt{key}          ==>> Jones et al. 1990
+ %   \citealt*{key}         ==>> Jones, Baker, and Williams 1990
+ %   \citealp{key}          ==>> Jones et al., 1990
+ %   \citealp*{key}         ==>> Jones, Baker, and Williams, 1990
+ % Additional citation possibilities (both author-year and numerical modes)
+ %   \citeauthor{key}       ==>> Jones et al.
+ %   \citeauthor*{key}      ==>> Jones, Baker, and Williams
+ %   \citeyear{key}         ==>> 1990
+ %   \citeyearpar{key}      ==>> (1990)
+ %   \citetext{priv. comm.} ==>> (priv. comm.)
+ %   \citenum{key}          ==>> 11 [non-superscripted]
+ % Note: full author lists depends on whether the bib style supports them;
+ %       if not, the abbreviated list is printed even when full requested.
+ %
+ % For names like della Robbia at the start of a sentence, use
+ %   \Citet{dRob98}         ==>> Della Robbia (1998)
+ %   \Citep{dRob98}         ==>> (Della Robbia, 1998)
+ %   \Citeauthor{dRob98}    ==>> Della Robbia
+ %
+ %
+ % Citation aliasing is achieved with
+ %   \defcitealias{key}{text}
+ %   \citetalias{key}  ==>> text
+ %   \citepalias{key}  ==>> (text)
+ %
+ % Defining the citation mode and punctual (citation style)
+ %   \setcitestyle{<comma-separated list of keywords, same
+ %     as the package options>}
+ % Example: \setcitestyle{square,semicolon}
+ % Alternatively:
+ % Use \bibpunct with 6 mandatory arguments:
+ %    1. opening bracket for citation
+ %    2. closing bracket
+ %    3. citation separator (for multiple citations in one \cite)
+ %    4. the letter n for numerical styles, s for superscripts
+ %        else anything for author-year
+ %    5. punctuation between authors and date
+ %    6. punctuation between years (or numbers) when common authors missing
+ % One optional argument is the character coming before post-notes. It
+ %   appears in square braces before all other arguments. May be left off.
+ % Example (and default) \bibpunct[, ]{(}{)}{;}{a}{,}{,}
+ %
+ % To make this automatic for a given bib style, named newbib, say, make
+ % a local configuration file, natbib.cfg, with the definition
+ %   \newcommand{\bibstyle@newbib}{\bibpunct...}
+ % Then the \bibliographystyle{newbib} will cause \bibstyle@newbib to
+ % be called on THE NEXT LATEX RUN (via the aux file).
+ %
+ % Such preprogrammed definitions may be invoked anywhere in the text
+ %  by calling \citestyle{newbib}. This is only useful if the style specified
+ %  differs from that in \bibliographystyle.
+ %
+ % With \citeindextrue and \citeindexfalse, one can control whether the
+ % \cite commands make an automatic entry of the citation in the .idx
+ % indexing file. For this, \makeindex must also be given in the preamble.
+ %
+ % Package Options: (for selecting punctuation)
+ %   round  -  round parentheses are used (default)
+ %   square -  square brackets are used   [option]
+ %   curly  -  curly braces are used      {option}
+ %   angle  -  angle brackets are used    <option>
+ %   semicolon  -  multiple citations separated by semi-colon (default)
+ %   colon  - same as semicolon, an earlier confusion
+ %   comma  -  separated by comma
+ %   authoryear - selects author-year citations (default)
+ %   numbers-  selects numerical citations
+ %   super  -  numerical citations as superscripts
+ %   sort   -  sorts multiple citations according to order in ref. list
+ %   sort&compress   -  like sort, but also compresses numerical citations
+ %   compress - compresses without sorting
+ %   longnamesfirst  -  makes first citation full author list
+ %   sectionbib - puts bibliography in a \section* instead of \chapter*
+ %   merge - allows the citation key to have a * prefix,
+ %           signifying to merge its reference with that of the previous citation.
+ %   elide - if references are merged, repeated portions of later ones may be removed.
+ %   mcite - recognizes and ignores the * prefix for merging.
+ % Punctuation so selected dominates over any predefined ones.
+ % Package options are called as, e.g.
+ %        \usepackage[square,comma]{natbib}
+ % LaTeX the source file natbib.dtx to obtain more details
+ % or the file natnotes.tex for a brief reference sheet.
+ %-----------------------------------------------------------
+\providecommand\@ifxundefined[1]{%
+ \ifx#1\@undefined\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
+}%
+\providecommand\@ifnum[1]{%
+ \ifnum#1\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
+}%
+\providecommand\@ifx[1]{%
+ \ifx#1\expandafter\@firstoftwo\else\expandafter\@secondoftwo\fi
+}%
+\providecommand\appdef[2]{%
+ \toks@\expandafter{#1}\@temptokena{#2}%
+ \edef#1{\the\toks@\the\@temptokena}%
+}%
+\@ifclassloaded{agu2001}{\PackageError{natbib}
+  {The agu2001 class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+\@ifclassloaded{agutex}{\PackageError{natbib}
+  {The AGUTeX class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+\@ifclassloaded{aguplus}{\PackageError{natbib}
+  {The aguplus class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+\@ifclassloaded{nlinproc}{\PackageError{natbib}
+  {The nlinproc class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+\@ifclassloaded{egs}{\PackageError{natbib}
+  {The egs class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+\@ifclassloaded{egu}{\PackageError{natbib}
+  {The egu class already includes natbib coding,\MessageBreak
+   so you should not add it explicitly}
+  {Type <Return> for now, but then later remove\MessageBreak
+   the command \protect\usepackage{natbib} from the document}
+  \endinput}{}
+ % Define citation punctuation for some author-year styles
+ % One may add and delete at this point
+ % Or put additions into local configuration file natbib.cfg
+\newcommand\bibstyle@chicago{\bibpunct{(}{)}{;}{a}{,}{,}}
+\newcommand\bibstyle@named{\bibpunct{[}{]}{;}{a}{,}{,}}
+\newcommand\bibstyle@agu{\bibpunct{[}{]}{;}{a}{,}{,~}}%Amer. Geophys. Union
+\newcommand\bibstyle@copernicus{\bibpunct{(}{)}{;}{a}{,}{,}}%Copernicus Publications
+\let\bibstyle@egu=\bibstyle@copernicus
+\let\bibstyle@egs=\bibstyle@copernicus
+\newcommand\bibstyle@agsm{\bibpunct{(}{)}{,}{a}{}{,}\gdef\harvardand{\&}}
+\newcommand\bibstyle@kluwer{\bibpunct{(}{)}{,}{a}{}{,}\gdef\harvardand{\&}}
+\newcommand\bibstyle@dcu{\bibpunct{(}{)}{;}{a}{;}{,}\gdef\harvardand{and}}
+\newcommand\bibstyle@aa{\bibpunct{(}{)}{;}{a}{}{,}} %Astronomy & Astrophysics
+\newcommand\bibstyle@pass{\bibpunct{(}{)}{;}{a}{,}{,}}%Planet. & Space Sci
+\newcommand\bibstyle@anngeo{\bibpunct{(}{)}{;}{a}{,}{,}}%Annales Geophysicae
+\newcommand\bibstyle@nlinproc{\bibpunct{(}{)}{;}{a}{,}{,}}%Nonlin.Proc.Geophys.
+ % Define citation punctuation for some numerical styles
+\newcommand\bibstyle@cospar{\bibpunct{/}{/}{,}{n}{}{}%
+     \gdef\bibnumfmt##1{##1.}}
+\newcommand\bibstyle@esa{\bibpunct{(Ref.~}{)}{,}{n}{}{}%
+     \gdef\bibnumfmt##1{##1.\hspace{1em}}}
+\newcommand\bibstyle@nature{\bibpunct{}{}{,}{s}{}{\textsuperscript{,}}%
+     \gdef\bibnumfmt##1{##1.}}
+ % The standard LaTeX styles
+\newcommand\bibstyle@plain{\bibpunct{[}{]}{,}{n}{}{,}}
+\let\bibstyle@alpha=\bibstyle@plain
+\let\bibstyle@abbrv=\bibstyle@plain
+\let\bibstyle@unsrt=\bibstyle@plain
+ % The author-year modifications of the standard styles
+\newcommand\bibstyle@plainnat{\bibpunct{[}{]}{,}{a}{,}{,}}
+\let\bibstyle@abbrvnat=\bibstyle@plainnat
+\let\bibstyle@unsrtnat=\bibstyle@plainnat
+\newif\ifNAT@numbers \NAT@numbersfalse
+\newif\ifNAT@super \NAT@superfalse
+\let\NAT@merge\z@
+\DeclareOption{numbers}{\NAT@numberstrue
+   \ExecuteOptions{square,comma,nobibstyle}}
+\DeclareOption{super}{\NAT@supertrue\NAT@numberstrue
+   \renewcommand\NAT@open{}\renewcommand\NAT@close{}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{authoryear}{\NAT@numbersfalse
+   \ExecuteOptions{round,semicolon,bibstyle}}
+\DeclareOption{round}{%
+      \renewcommand\NAT@open{(} \renewcommand\NAT@close{)}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{square}{%
+      \renewcommand\NAT@open{[} \renewcommand\NAT@close{]}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{angle}{%
+      \renewcommand\NAT@open{$<$} \renewcommand\NAT@close{$>$}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{curly}{%
+      \renewcommand\NAT@open{\{} \renewcommand\NAT@close{\}}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{comma}{\renewcommand\NAT@sep{,}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{semicolon}{\renewcommand\NAT@sep{;}
+   \ExecuteOptions{nobibstyle}}
+\DeclareOption{colon}{\ExecuteOptions{semicolon}}
+\DeclareOption{nobibstyle}{\let\bibstyle=\@gobble}
+\DeclareOption{bibstyle}{\let\bibstyle=\@citestyle}
+\newif\ifNAT@openbib \NAT@openbibfalse
+\DeclareOption{openbib}{\NAT@openbibtrue}
+\DeclareOption{sectionbib}{\def\NAT@sectionbib{on}}
+\def\NAT@sort{\z@}
+\def\NAT@cmprs{\z@}
+\DeclareOption{sort}{\def\NAT@sort{\@ne}}
+\DeclareOption{compress}{\def\NAT@cmprs{\@ne}}
+\DeclareOption{sort&compress}{\def\NAT@sort{\@ne}\def\NAT@cmprs{\@ne}}
+\DeclareOption{mcite}{\let\NAT@merge\@ne}
+\DeclareOption{merge}{\@ifnum{\NAT@merge<\tw@}{\let\NAT@merge\tw@}{}}
+\DeclareOption{elide}{\@ifnum{\NAT@merge<\thr@@}{\let\NAT@merge\thr@@}{}}
+\@ifpackageloaded{cite}{\PackageWarningNoLine{natbib}
+  {The `cite' package should not be used\MessageBreak
+   with natbib. Use option `sort' instead}\ExecuteOptions{sort}}{}
+\@ifpackageloaded{mcite}{\PackageWarningNoLine{natbib}
+  {The `mcite' package should not be used\MessageBreak
+   with natbib. Use option `merge' instead}\ExecuteOptions{merge}}{}
+\@ifpackageloaded{citeref}{\PackageError{natbib}
+  {The `citeref' package must be loaded after natbib}%
+  {Move \protect\usepackage{citeref} to after \string\usepackage{natbib}}}{}
+\newif\ifNAT@longnames\NAT@longnamesfalse
+\DeclareOption{longnamesfirst}{\NAT@longnamestrue}
+\DeclareOption{nonamebreak}{\def\NAT@nmfmt#1{\mbox{\NAT@up#1}}}
+\def\NAT@nmfmt#1{{\NAT@up#1}}
+\renewcommand\bibstyle[1]{\csname bibstyle@#1\endcsname}
+\AtBeginDocument{\global\let\bibstyle=\@gobble}
+\let\@citestyle\bibstyle
+\newcommand\citestyle[1]{\@citestyle{#1}\let\bibstyle\@gobble}
+\newcommand\bibpunct[7][, ]%
+  {\gdef\NAT@open{#2}\gdef\NAT@close{#3}\gdef
+   \NAT@sep{#4}\global\NAT@numbersfalse
+     \ifx #5n\global\NAT@numberstrue\global\NAT@superfalse
+   \else
+     \ifx #5s\global\NAT@numberstrue\global\NAT@supertrue
+   \fi\fi
+   \gdef\NAT@aysep{#6}\gdef\NAT@yrsep{#7}%
+   \gdef\NAT@cmt{#1}%
+   \NAT@@setcites
+  }
+\newcommand\setcitestyle[1]{
+ \@for\@tempa:=#1\do
+ {\def\@tempb{round}\ifx\@tempa\@tempb
+    \renewcommand\NAT@open{(}\renewcommand\NAT@close{)}\fi
+  \def\@tempb{square}\ifx\@tempa\@tempb
+    \renewcommand\NAT@open{[}\renewcommand\NAT@close{]}\fi
+  \def\@tempb{angle}\ifx\@tempa\@tempb
+    \renewcommand\NAT@open{$<$}\renewcommand\NAT@close{$>$}\fi
+  \def\@tempb{curly}\ifx\@tempa\@tempb
+    \renewcommand\NAT@open{\{}\renewcommand\NAT@close{\}}\fi
+  \def\@tempb{semicolon}\ifx\@tempa\@tempb
+    \renewcommand\NAT@sep{;}\fi
+  \def\@tempb{colon}\ifx\@tempa\@tempb
+    \renewcommand\NAT@sep{;}\fi
+  \def\@tempb{comma}\ifx\@tempa\@tempb
+    \renewcommand\NAT@sep{,}\fi
+  \def\@tempb{authoryear}\ifx\@tempa\@tempb
+    \NAT@numbersfalse\fi
+  \def\@tempb{numbers}\ifx\@tempa\@tempb
+    \NAT@numberstrue\NAT@superfalse\fi
+  \def\@tempb{super}\ifx\@tempa\@tempb
+    \NAT@numberstrue\NAT@supertrue\fi
+  \expandafter\NAT@find@eq\@tempa=\relax\@nil
+  \if\@tempc\relax\else
+    \expandafter\NAT@rem@eq\@tempc
+    \def\@tempb{open}\ifx\@tempa\@tempb
+     \xdef\NAT@open{\@tempc}\fi
+    \def\@tempb{close}\ifx\@tempa\@tempb
+     \xdef\NAT@close{\@tempc}\fi
+    \def\@tempb{aysep}\ifx\@tempa\@tempb
+     \xdef\NAT@aysep{\@tempc}\fi
+    \def\@tempb{yysep}\ifx\@tempa\@tempb
+     \xdef\NAT@yrsep{\@tempc}\fi
+    \def\@tempb{notesep}\ifx\@tempa\@tempb
+     \xdef\NAT@cmt{\@tempc}\fi
+    \def\@tempb{citesep}\ifx\@tempa\@tempb
+     \xdef\NAT@sep{\@tempc}\fi
+  \fi
+ }%
+ \NAT@@setcites
+}
+ \def\NAT@find@eq#1=#2\@nil{\def\@tempa{#1}\def\@tempc{#2}}
+ \def\NAT@rem@eq#1={\def\@tempc{#1}}
+ \def\NAT@@setcites{\global\let\bibstyle\@gobble}
+\AtBeginDocument{\let\NAT@@setcites\NAT@set@cites}
+\newcommand\NAT@open{(} \newcommand\NAT@close{)}
+\newcommand\NAT@sep{;}
+\ProcessOptions
+\newcommand\NAT@aysep{,} \newcommand\NAT@yrsep{,}
+\newcommand\NAT@cmt{, }
+\newcommand\NAT@cite%
+    [3]{\ifNAT@swa\NAT@@open\if*#2*\else#2\NAT@spacechar\fi
+        #1\if*#3*\else\NAT@cmt#3\fi\NAT@@close\else#1\fi\endgroup}
+\newcommand\NAT@citenum%
+    [3]{\ifNAT@swa\NAT@@open\if*#2*\else#2\NAT@spacechar\fi
+        #1\if*#3*\else\NAT@cmt#3\fi\NAT@@close\else#1\fi\endgroup}
+\newcommand\NAT@citesuper[3]{\ifNAT@swa
+\if*#2*\else#2\NAT@spacechar\fi
+\unskip\kern\p@\textsuperscript{\NAT@@open#1\NAT@@close}%
+   \if*#3*\else\NAT@spacechar#3\fi\else #1\fi\endgroup}
+\providecommand\textsuperscript[1]{\mbox{$^{\mbox{\scriptsize#1}}$}}
+\begingroup \catcode`\_=8
+\gdef\NAT@ifcat@num#1{%
+ \ifcat_\ifnum\z@<0#1_\else A\fi
+  \expandafter\@firstoftwo
+ \else
+  \expandafter\@secondoftwo
+ \fi
+}%
+\endgroup
+\providecommand\@firstofone[1]{#1}
+\newcommand\NAT@citexnum{}
+\def\NAT@citexnum[#1][#2]#3{%
+  \NAT@reset@parser
+  \NAT@sort@cites{#3}%
+  \NAT@reset@citea
+  \@cite{\def\NAT@num{-1}\let\NAT@last@yr\relax\let\NAT@nm\@empty
+    \@for\@citeb:=\NAT@cite@list\do
+    {\@safe@activestrue
+     \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}%
+     \@safe@activesfalse
+     \@ifundefined{b@\@citeb\@extra@b@citeb}{%
+       {\reset@font\bfseries?}
+        \NAT@citeundefined\PackageWarning{natbib}%
+       {Citation `\@citeb' on page \thepage \space undefined}}%
+     {\let\NAT@last@num\NAT@num\let\NAT@last@nm\NAT@nm
+      \NAT@parse{\@citeb}%
+      \ifNAT@longnames\@ifundefined{bv@\@citeb\@extra@b@citeb}{%
+        \let\NAT@name=\NAT@all@names
+        \global\@namedef{bv@\@citeb\@extra@b@citeb}{}}{}%
+      \fi
+      \ifNAT@full\let\NAT@nm\NAT@all@names\else
+        \let\NAT@nm\NAT@name\fi
+      \ifNAT@swa
+       \@ifnum{\NAT@ctype>\@ne}{%
+        \@citea
+        \NAT@hyper@{\@ifnum{\NAT@ctype=\tw@}{\NAT@test{\NAT@ctype}}{\NAT@alias}}%
+       }{%
+        \@ifnum{\NAT@cmprs>\z@}{%
+         \NAT@ifcat@num\NAT@num
+          {\let\NAT@nm=\NAT@num}%
+          {\def\NAT@nm{-2}}%
+         \NAT@ifcat@num\NAT@last@num
+          {\@tempcnta=\NAT@last@num\relax}%
+          {\@tempcnta\m@ne}%
+         \@ifnum{\NAT@nm=\@tempcnta}{%
+          \@ifnum{\NAT@merge>\@ne}{}{\NAT@last@yr@mbox}%
+         }{%
+           \advance\@tempcnta by\@ne
+           \@ifnum{\NAT@nm=\@tempcnta}{%
+             \ifx\NAT@last@yr\relax
+               \def@NAT@last@yr{\@citea}%
+             \else
+               \def@NAT@last@yr{--\NAT@penalty}%
+             \fi
+           }{%
+             \NAT@last@yr@mbox
+           }%
+         }%
+        }{%
+         \@tempswatrue
+         \@ifnum{\NAT@merge>\@ne}{\@ifnum{\NAT@last@num=\NAT@num\relax}{\@tempswafalse}{}}{}%
+         \if@tempswa\NAT@citea@mbox\fi
+        }%
+       }%
+       \NAT@def@citea
+      \else
+        \ifcase\NAT@ctype
+          \ifx\NAT@last@nm\NAT@nm \NAT@yrsep\NAT@penalty\NAT@space\else
+            \@citea \NAT@test{\@ne}\NAT@spacechar\NAT@mbox{\NAT@super@kern\NAT@@open}%
+          \fi
+          \if*#1*\else#1\NAT@spacechar\fi
+          \NAT@mbox{\NAT@hyper@{{\citenumfont{\NAT@num}}}}%
+          \NAT@def@citea@box
+        \or
+          \NAT@hyper@citea@space{\NAT@test{\NAT@ctype}}%
+        \or
+          \NAT@hyper@citea@space{\NAT@test{\NAT@ctype}}%
+        \or
+          \NAT@hyper@citea@space\NAT@alias
+        \fi
+      \fi
+     }%
+    }%
+      \@ifnum{\NAT@cmprs>\z@}{\NAT@last@yr}{}%
+      \ifNAT@swa\else
+        \@ifnum{\NAT@ctype=\z@}{%
+          \if*#2*\else\NAT@cmt#2\fi
+        }{}%
+        \NAT@mbox{\NAT@@close}%
+      \fi
+  }{#1}{#2}%
+}%
+\def\NAT@citea@mbox{%
+ \@citea\mbox{\NAT@hyper@{{\citenumfont{\NAT@num}}}}%
+}%
+\def\NAT@hyper@#1{%
+ \hyper@natlinkstart{\@citeb\@extra@b@citeb}#1\hyper@natlinkend
+}%
+\def\NAT@hyper@citea#1{%
+ \@citea
+ \NAT@hyper@{#1}%
+ \NAT@def@citea
+}%
+\def\NAT@hyper@citea@space#1{%
+ \@citea
+ \NAT@hyper@{#1}%
+ \NAT@def@citea@space
+}%
+\def\def@NAT@last@yr#1{%
+ \protected@edef\NAT@last@yr{%
+  #1%
+  \noexpand\mbox{%
+   \noexpand\hyper@natlinkstart{\@citeb\@extra@b@citeb}%
+   {\noexpand\citenumfont{\NAT@num}}%
+   \noexpand\hyper@natlinkend
+  }%
+ }%
+}%
+\def\NAT@last@yr@mbox{%
+ \NAT@last@yr\let\NAT@last@yr\relax
+ \NAT@citea@mbox
+}%
+\newcommand\NAT@test[1]{%
+ \@ifnum{#1=\@ne}{%
+  \ifx\NAT@nm\NAT@noname
+   \begingroup\reset@font\bfseries(author?)\endgroup
+   \PackageWarning{natbib}{%
+    Author undefined for citation`\@citeb' \MessageBreak on page \thepage%
+   }%
+  \else \NAT@nm
+  \fi
+ }{%
+  \if\relax\NAT@date\relax
+   \begingroup\reset@font\bfseries(year?)\endgroup
+   \PackageWarning{natbib}{%
+    Year undefined for citation`\@citeb' \MessageBreak on page \thepage%
+   }%
+  \else \NAT@date
+  \fi
+ }%
+}%
+\let\citenumfont=\@empty
+\newcommand\NAT@citex{}
+\def\NAT@citex%
+  [#1][#2]#3{%
+  \NAT@reset@parser
+  \NAT@sort@cites{#3}%
+  \NAT@reset@citea
+  \@cite{\let\NAT@nm\@empty\let\NAT@year\@empty
+    \@for\@citeb:=\NAT@cite@list\do
+    {\@safe@activestrue
+     \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}%
+     \@safe@activesfalse
+     \@ifundefined{b@\@citeb\@extra@b@citeb}{\@citea%
+       {\reset@font\bfseries ?}\NAT@citeundefined
+                 \PackageWarning{natbib}%
+       {Citation `\@citeb' on page \thepage \space undefined}\def\NAT@date{}}%
+     {\let\NAT@last@nm=\NAT@nm\let\NAT@last@yr=\NAT@year
+      \NAT@parse{\@citeb}%
+      \ifNAT@longnames\@ifundefined{bv@\@citeb\@extra@b@citeb}{%
+        \let\NAT@name=\NAT@all@names
+        \global\@namedef{bv@\@citeb\@extra@b@citeb}{}}{}%
+      \fi
+     \ifNAT@full\let\NAT@nm\NAT@all@names\else
+       \let\NAT@nm\NAT@name\fi
+     \ifNAT@swa\ifcase\NAT@ctype
+       \if\relax\NAT@date\relax
+         \@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}\NAT@date}%
+       \else
+         \ifx\NAT@last@nm\NAT@nm\NAT@yrsep
+            \ifx\NAT@last@yr\NAT@year
+              \def\NAT@temp{{?}}%
+              \ifx\NAT@temp\NAT@exlab\PackageWarningNoLine{natbib}%
+               {Multiple citation on page \thepage: same authors and
+               year\MessageBreak without distinguishing extra
+               letter,\MessageBreak appears as question mark}\fi
+              \NAT@hyper@{\NAT@exlab}%
+            \else\unskip\NAT@spacechar
+              \NAT@hyper@{\NAT@date}%
+            \fi
+         \else
+           \@citea\NAT@hyper@{%
+             \NAT@nmfmt{\NAT@nm}%
+             \hyper@natlinkbreak{%
+               \NAT@aysep\NAT@spacechar}{\@citeb\@extra@b@citeb
+             }%
+             \NAT@date
+           }%
+         \fi
+       \fi
+     \or\@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}}%
+     \or\@citea\NAT@hyper@{\NAT@date}%
+     \or\@citea\NAT@hyper@{\NAT@alias}%
+     \fi \NAT@def@citea
+     \else
+       \ifcase\NAT@ctype
+        \if\relax\NAT@date\relax
+          \@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}}%
+        \else
+         \ifx\NAT@last@nm\NAT@nm\NAT@yrsep
+            \ifx\NAT@last@yr\NAT@year
+              \def\NAT@temp{{?}}%
+              \ifx\NAT@temp\NAT@exlab\PackageWarningNoLine{natbib}%
+               {Multiple citation on page \thepage: same authors and
+               year\MessageBreak without distinguishing extra
+               letter,\MessageBreak appears as question mark}\fi
+              \NAT@hyper@{\NAT@exlab}%
+            \else
+              \unskip\NAT@spacechar
+              \NAT@hyper@{\NAT@date}%
+            \fi
+         \else
+           \@citea\NAT@hyper@{%
+             \NAT@nmfmt{\NAT@nm}%
+             \hyper@natlinkbreak{\NAT@spacechar\NAT@@open\if*#1*\else#1\NAT@spacechar\fi}%
+               {\@citeb\@extra@b@citeb}%
+             \NAT@date
+           }%
+         \fi
+        \fi
+       \or\@citea\NAT@hyper@{\NAT@nmfmt{\NAT@nm}}%
+       \or\@citea\NAT@hyper@{\NAT@date}%
+       \or\@citea\NAT@hyper@{\NAT@alias}%
+       \fi
+       \if\relax\NAT@date\relax
+         \NAT@def@citea
+       \else
+         \NAT@def@citea@close
+       \fi
+     \fi
+     }}\ifNAT@swa\else\if*#2*\else\NAT@cmt#2\fi
+     \if\relax\NAT@date\relax\else\NAT@@close\fi\fi}{#1}{#2}}
+\def\NAT@spacechar{\ }%
+\def\NAT@separator{\NAT@sep\NAT@penalty}%
+\def\NAT@reset@citea{\c@NAT@ctr\@ne\let\@citea\@empty}%
+\def\NAT@def@citea{\def\@citea{\NAT@separator\NAT@space}}%
+\def\NAT@def@citea@space{\def\@citea{\NAT@separator\NAT@spacechar}}%
+\def\NAT@def@citea@close{\def\@citea{\NAT@@close\NAT@separator\NAT@space}}%
+\def\NAT@def@citea@box{\def\@citea{\NAT@mbox{\NAT@@close}\NAT@separator\NAT@spacechar}}%
+\newif\ifNAT@par \NAT@partrue
+\newcommand\NAT@@open{\ifNAT@par\NAT@open\fi}
+\newcommand\NAT@@close{\ifNAT@par\NAT@close\fi}
+\newcommand\NAT@alias{\@ifundefined{al@\@citeb\@extra@b@citeb}{%
+  {\reset@font\bfseries(alias?)}\PackageWarning{natbib}
+  {Alias undefined for citation `\@citeb'
+  \MessageBreak on page \thepage}}{\@nameuse{al@\@citeb\@extra@b@citeb}}}
+\let\NAT@up\relax
+\newcommand\NAT@Up[1]{{\let\protect\@unexpandable@protect\let~\relax
+  \expandafter\NAT@deftemp#1}\expandafter\NAT@UP\NAT@temp}
+\newcommand\NAT@deftemp[1]{\xdef\NAT@temp{#1}}
+\newcommand\NAT@UP[1]{\let\@tempa\NAT@UP\ifcat a#1\MakeUppercase{#1}%
+  \let\@tempa\relax\else#1\fi\@tempa}
+\newcommand\shortcites[1]{%
+  \@bsphack\@for\@citeb:=#1\do
+  {\@safe@activestrue
+   \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}%
+   \@safe@activesfalse
+   \global\@namedef{bv@\@citeb\@extra@b@citeb}{}}\@esphack}
+\newcommand\NAT@biblabel[1]{\hfill}
+\newcommand\NAT@biblabelnum[1]{\bibnumfmt{#1}}
+\let\bibnumfmt\@empty
+\providecommand\@biblabel[1]{[#1]}
+\AtBeginDocument{\ifx\bibnumfmt\@empty\let\bibnumfmt\@biblabel\fi}
+\newcommand\NAT@bibsetnum[1]{\settowidth\labelwidth{\@biblabel{#1}}%
+   \setlength{\leftmargin}{\labelwidth}\addtolength{\leftmargin}{\labelsep}%
+   \setlength{\itemsep}{\bibsep}\setlength{\parsep}{\z@}%
+   \ifNAT@openbib
+     \addtolength{\leftmargin}{\bibindent}%
+     \setlength{\itemindent}{-\bibindent}%
+     \setlength{\listparindent}{\itemindent}%
+     \setlength{\parsep}{0pt}%
+   \fi
+}
+\newlength{\bibhang}
+\setlength{\bibhang}{1em}
+\newlength{\bibsep}
+ {\@listi \global\bibsep\itemsep \global\advance\bibsep by\parsep}
+
+\newcommand\NAT@bibsetup%
+   [1]{\setlength{\leftmargin}{\bibhang}\setlength{\itemindent}{-\leftmargin}%
+       \setlength{\itemsep}{\bibsep}\setlength{\parsep}{\z@}}
+\newcommand\NAT@set@cites{%
+  \ifNAT@numbers
+    \ifNAT@super \let\@cite\NAT@citesuper
+       \def\NAT@mbox##1{\unskip\nobreak\textsuperscript{##1}}%
+       \let\citeyearpar=\citeyear
+       \let\NAT@space\relax
+       \def\NAT@super@kern{\kern\p@}%
+    \else
+       \let\NAT@mbox=\mbox
+       \let\@cite\NAT@citenum
+       \let\NAT@space\NAT@spacechar
+       \let\NAT@super@kern\relax
+    \fi
+    \let\@citex\NAT@citexnum
+    \let\@biblabel\NAT@biblabelnum
+    \let\@bibsetup\NAT@bibsetnum
+    \renewcommand\NAT@idxtxt{\NAT@name\NAT@spacechar\NAT@open\NAT@num\NAT@close}%
+    \def\natexlab##1{}%
+    \def\NAT@penalty{\penalty\@m}%
+  \else
+    \let\@cite\NAT@cite
+    \let\@citex\NAT@citex
+    \let\@biblabel\NAT@biblabel
+    \let\@bibsetup\NAT@bibsetup
+    \let\NAT@space\NAT@spacechar
+    \let\NAT@penalty\@empty
+    \renewcommand\NAT@idxtxt{\NAT@name\NAT@spacechar\NAT@open\NAT@date\NAT@close}%
+    \def\natexlab##1{##1}%
+  \fi}
+\AtBeginDocument{\NAT@set@cites}
+\AtBeginDocument{\ifx\SK@def\@undefined\else
+\ifx\SK@cite\@empty\else
+  \SK@def\@citex[#1][#2]#3{\SK@\SK@@ref{#3}\SK@@citex[#1][#2]{#3}}\fi
+\ifx\SK@citeauthor\@undefined\def\HAR@checkdef{}\else
+  \let\citeauthor\SK@citeauthor
+  \let\citefullauthor\SK@citefullauthor
+  \let\citeyear\SK@citeyear\fi
+\fi}
+\newif\ifNAT@full\NAT@fullfalse
+\newif\ifNAT@swa
+\DeclareRobustCommand\citet
+   {\begingroup\NAT@swafalse\let\NAT@ctype\z@\NAT@partrue
+     \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\newcommand\NAT@citetp{\@ifnextchar[{\NAT@@citetp}{\NAT@@citetp[]}}
+\newcommand\NAT@@citetp{}
+\def\NAT@@citetp[#1]{\@ifnextchar[{\@citex[#1]}{\@citex[][#1]}}
+\DeclareRobustCommand\citep
+   {\begingroup\NAT@swatrue\let\NAT@ctype\z@\NAT@partrue
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\cite
+    {\begingroup\let\NAT@ctype\z@\NAT@partrue\NAT@swatrue
+      \@ifstar{\NAT@fulltrue\NAT@cites}{\NAT@fullfalse\NAT@cites}}
+\newcommand\NAT@cites{\@ifnextchar [{\NAT@@citetp}{%
+     \ifNAT@numbers\else
+     \NAT@swafalse
+     \fi
+    \NAT@@citetp[]}}
+\DeclareRobustCommand\citealt
+   {\begingroup\NAT@swafalse\let\NAT@ctype\z@\NAT@parfalse
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\citealp
+   {\begingroup\NAT@swatrue\let\NAT@ctype\z@\NAT@parfalse
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\citenum
+   {\begingroup
+     \NAT@swatrue\let\NAT@ctype\z@\NAT@parfalse\let\textsuperscript\NAT@spacechar
+     \NAT@citexnum[][]}
+\DeclareRobustCommand\citeauthor
+   {\begingroup\NAT@swafalse\let\NAT@ctype\@ne\NAT@parfalse
+    \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\Citet
+   {\begingroup\NAT@swafalse\let\NAT@ctype\z@\NAT@partrue
+     \let\NAT@up\NAT@Up
+     \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\Citep
+   {\begingroup\NAT@swatrue\let\NAT@ctype\z@\NAT@partrue
+     \let\NAT@up\NAT@Up
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\Citealt
+   {\begingroup\NAT@swafalse\let\NAT@ctype\z@\NAT@parfalse
+     \let\NAT@up\NAT@Up
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\Citealp
+   {\begingroup\NAT@swatrue\let\NAT@ctype\z@\NAT@parfalse
+     \let\NAT@up\NAT@Up
+         \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\Citeauthor
+   {\begingroup\NAT@swafalse\let\NAT@ctype\@ne\NAT@parfalse
+     \let\NAT@up\NAT@Up
+    \@ifstar{\NAT@fulltrue\NAT@citetp}{\NAT@fullfalse\NAT@citetp}}
+\DeclareRobustCommand\citeyear
+   {\begingroup\NAT@swafalse\let\NAT@ctype\tw@\NAT@parfalse\NAT@citetp}
+\DeclareRobustCommand\citeyearpar
+   {\begingroup\NAT@swatrue\let\NAT@ctype\tw@\NAT@partrue\NAT@citetp}
+\newcommand\citetext[1]{\NAT@open#1\NAT@close}
+\DeclareRobustCommand\citefullauthor
+   {\citeauthor*}
+\newcommand\defcitealias[2]{%
+   \@ifundefined{al@#1\@extra@b@citeb}{}
+   {\PackageWarning{natbib}{Overwriting existing alias for citation #1}}
+   \@namedef{al@#1\@extra@b@citeb}{#2}}
+\DeclareRobustCommand\citetalias{\begingroup
+   \NAT@swafalse\let\NAT@ctype\thr@@\NAT@parfalse\NAT@citetp}
+\DeclareRobustCommand\citepalias{\begingroup
+   \NAT@swatrue\let\NAT@ctype\thr@@\NAT@partrue\NAT@citetp}
+\renewcommand\nocite[1]{\@bsphack
+  \@for\@citeb:=#1\do{%
+    \@safe@activestrue
+    \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}%
+    \@safe@activesfalse
+    \if@filesw\immediate\write\@auxout{\string\citation{\@citeb}}\fi
+    \if*\@citeb\else
+    \@ifundefined{b@\@citeb\@extra@b@citeb}{%
+       \NAT@citeundefined \PackageWarning{natbib}%
+       {Citation `\@citeb' undefined}}{}\fi}%
+  \@esphack}
+\newcommand\NAT@parse[1]{%
+  \begingroup
+   \let\protect=\@unexpandable@protect
+   \let~\relax
+   \let\active@prefix=\@gobble
+   \edef\NAT@temp{\csname b@#1\@extra@b@citeb\endcsname}%
+   \aftergroup\NAT@split
+   \expandafter
+  \endgroup
+  \NAT@temp{}{}{}{}{}@@%
+  \expandafter\NAT@parse@date\NAT@date??????@@%
+  \ifciteindex\NAT@index\fi
+}%
+\def\NAT@split#1#2#3#4#5@@{%
+  \gdef\NAT@num{#1}\gdef\NAT@name{#3}\gdef\NAT@date{#2}%
+  \gdef\NAT@all@names{#4}%
+  \ifx\NAT@num\@empty\gdef\NAT@num{0}\fi
+  \ifx\NAT@noname\NAT@all@names \gdef\NAT@all@names{#3}\fi
+}%
+\def\NAT@reset@parser{%
+  \global\let\NAT@num\@empty
+  \global\let\NAT@name\@empty
+  \global\let\NAT@date\@empty
+  \global\let\NAT@all@names\@empty
+}%
+\newcommand\NAT@parse@date{}
+\def\NAT@parse@date#1#2#3#4#5#6@@{%
+  \ifnum\the\catcode`#1=11\def\NAT@year{}\def\NAT@exlab{#1}\else
+  \ifnum\the\catcode`#2=11\def\NAT@year{#1}\def\NAT@exlab{#2}\else
+  \ifnum\the\catcode`#3=11\def\NAT@year{#1#2}\def\NAT@exlab{#3}\else
+  \ifnum\the\catcode`#4=11\def\NAT@year{#1#2#3}\def\NAT@exlab{#4}\else
+    \def\NAT@year{#1#2#3#4}\def\NAT@exlab{{#5}}\fi\fi\fi\fi}
+\newcommand\NAT@index{}
+\let\NAT@makeindex=\makeindex
+\renewcommand\makeindex{\NAT@makeindex
+  \renewcommand\NAT@index{\@bsphack\begingroup
+     \def~{\string~}\@wrindex{\NAT@idxtxt}}}
+\newcommand\NAT@idxtxt{\NAT@name\NAT@spacechar\NAT@open\NAT@date\NAT@close}
+\@ifxundefined\@indexfile{}{\let\NAT@makeindex\relax\makeindex}
+\newif\ifciteindex \citeindexfalse
+\newcommand\citeindextype{default}
+\newcommand\NAT@index@alt{{\let\protect=\noexpand\let~\relax
+  \xdef\NAT@temp{\NAT@idxtxt}}\expandafter\NAT@exp\NAT@temp\@nil}
+\newcommand\NAT@exp{}
+\def\NAT@exp#1\@nil{\index[\citeindextype]{#1}}
+
+\AtBeginDocument{%
+\@ifpackageloaded{index}{\let\NAT@index=\NAT@index@alt}{}}
+\newcommand\NAT@ifcmd{\futurelet\NAT@temp\NAT@ifxcmd}
+\newcommand\NAT@ifxcmd{\ifx\NAT@temp\relax\else\expandafter\NAT@bare\fi}
+\def\NAT@bare#1(#2)#3(@)#4\@nil#5{%
+  \if @#2
+    \expandafter\NAT@apalk#1, , \@nil{#5}%
+  \else
+  \NAT@wrout{\the\c@NAT@ctr}{#2}{#1}{#3}{#5}%
+\fi
+}
+\newcommand\NAT@wrout[5]{%
+\if@filesw
+      {\let\protect\noexpand\let~\relax
+       \immediate
+       \write\@auxout{\string\bibcite{#5}{{#1}{#2}{{#3}}{{#4}}}}}\fi
+\ignorespaces}
+\def\NAT@noname{{}}
+\renewcommand\bibitem{\@ifnextchar[{\@lbibitem}{\@lbibitem[]}}%
+\let\NAT@bibitem@first@sw\@secondoftwo
+\def\@lbibitem[#1]#2{%
+  \if\relax\@extra@b@citeb\relax\else
+    \@ifundefined{br@#2\@extra@b@citeb}{}{%
+     \@namedef{br@#2}{\@nameuse{br@#2\@extra@b@citeb}}%
+    }%
+  \fi
+  \@ifundefined{b@#2\@extra@b@citeb}{%
+   \def\NAT@num{}%
+  }{%
+   \NAT@parse{#2}%
+  }%
+  \def\NAT@tmp{#1}%
+  \expandafter\let\expandafter\bibitemOpen\csname NAT@b@open@#2\endcsname
+  \expandafter\let\expandafter\bibitemShut\csname NAT@b@shut@#2\endcsname
+  \@ifnum{\NAT@merge>\@ne}{%
+   \NAT@bibitem@first@sw{%
+    \@firstoftwo
+   }{%
+    \@ifundefined{NAT@b*@#2}{%
+     \@firstoftwo
+    }{%
+     \expandafter\def\expandafter\NAT@num\expandafter{\the\c@NAT@ctr}%
+     \@secondoftwo
+    }%
+   }%
+  }{%
+   \@firstoftwo
+  }%
+  {%
+   \global\advance\c@NAT@ctr\@ne
+   \@ifx{\NAT@tmp\@empty}{\@firstoftwo}{%
+    \@secondoftwo
+   }%
+   {%
+    \expandafter\def\expandafter\NAT@num\expandafter{\the\c@NAT@ctr}%
+    \global\NAT@stdbsttrue
+   }{}%
+   \bibitem@fin
+   \item[\hfil\NAT@anchor{#2}{\NAT@num}]%
+   \global\let\NAT@bibitem@first@sw\@secondoftwo
+   \NAT@bibitem@init
+  }%
+  {%
+   \NAT@anchor{#2}{}%
+   \NAT@bibitem@cont
+   \bibitem@fin
+  }%
+  \@ifx{\NAT@tmp\@empty}{%
+    \NAT@wrout{\the\c@NAT@ctr}{}{}{}{#2}%
+  }{%
+    \expandafter\NAT@ifcmd\NAT@tmp(@)(@)\@nil{#2}%
+  }%
+}%
+\def\bibitem@fin{%
+ \@ifxundefined\@bibstop{}{\csname bibitem@\@bibstop\endcsname}%
+}%
+\def\NAT@bibitem@init{%
+ \let\@bibstop\@undefined
+}%
+\def\NAT@bibitem@cont{%
+ \let\bibitem@Stop\bibitemStop
+ \let\bibitem@NoStop\bibitemContinue
+}%
+\def\BibitemOpen{%
+ \bibitemOpen
+}%
+\def\BibitemShut#1{%
+ \bibitemShut
+ \def\@bibstop{#1}%
+ \let\bibitem@Stop\bibitemStop
+ \let\bibitem@NoStop\bibitemNoStop
+}%
+\def\bibitemStop{}%
+\def\bibitemNoStop{.\spacefactor\@mmm\space}%
+\def\bibitemContinue{\spacefactor\@mmm\space}%
+\mathchardef\@mmm=3000 %
+\providecommand{\bibAnnote}[3]{%
+  \BibitemShut{#1}%
+  \def\@tempa{#3}\@ifx{\@tempa\@empty}{}{%
+   \begin{quotation}\noindent
+    \textsc{Key:}\ #2\\\textsc{Annotation:}\ \@tempa
+   \end{quotation}%
+  }%
+}%
+\providecommand{\bibAnnoteFile}[2]{%
+  \IfFileExists{#2}{%
+    \bibAnnote{#1}{#2}{\input{#2}}%
+  }{%
+    \bibAnnote{#1}{#2}{}%
+  }%
+}%
+\let\bibitemOpen\relax
+\let\bibitemShut\relax
+\def\bibfield{\@ifnum{\NAT@merge>\tw@}{\@bibfield}{\@secondoftwo}}%
+\def\@bibfield#1#2{%
+ \begingroup
+  \let\Doi\@gobble
+  \let\bibinfo\relax
+  \let\restore@protect\@empty
+  \protected@edef\@tempa{#2}%
+  \aftergroup\def\aftergroup\@tempa
+ \expandafter\endgroup\expandafter{\@tempa}%
+ \expandafter\@ifx\expandafter{\csname @bib#1\endcsname\@tempa}{%
+  \expandafter\let\expandafter\@tempa\csname @bib@X#1\endcsname
+ }{%
+  \expandafter\let\csname @bib#1\endcsname\@tempa
+  \expandafter\let\expandafter\@tempa\csname @bib@Y#1\endcsname
+ }%
+ \@ifx{\@tempa\relax}{\let\@tempa\@firstofone}{}%
+ \@tempa{#2}%
+}%
+\def\bibinfo#1{%
+ \expandafter\let\expandafter\@tempa\csname bibinfo@X@#1\endcsname
+ \@ifx{\@tempa\relax}{\@firstofone}{\@tempa}%
+}%
+\def\@bib@Xauthor#1{\let\@bib@Xjournal\@gobble}%
+\def\@bib@Xjournal#1{\begingroup\let\bibinfo@X@journal\@bib@Z@journal#1\endgroup}%
+\def\@bibibid@#1{\textit{ibid}.}%
+\appdef\NAT@bibitem@init{%
+ \let\@bibauthor  \@empty
+ \let\@bibjournal \@empty
+ \let\@bib@Z@journal\@bibibid@
+}%
+\ifx\SK@lbibitem\@undefined\else
+   \let\SK@lbibitem\@lbibitem
+   \def\@lbibitem[#1]#2{%
+     \SK@lbibitem[#1]{#2}\SK@\SK@@label{#2}\ignorespaces}\fi
+\newif\ifNAT@stdbst \NAT@stdbstfalse
+
+\AtEndDocument{%
+  \ifNAT@stdbst\if@filesw
+   \immediate\write\@auxout{%
+    \string\providecommand\string\NAT@force@numbers{}%
+    \string\NAT@force@numbers
+   }%
+  \fi\fi
+ }
+\newcommand\NAT@force@numbers{%
+  \ifNAT@numbers\else
+  \PackageError{natbib}{Bibliography not compatible with author-year
+  citations.\MessageBreak
+  Press <return> to continue in numerical citation style}
+  {Check the bibliography entries for non-compliant syntax,\MessageBreak
+   or select author-year BibTeX style, e.g. plainnat}%
+  \global\NAT@numberstrue\fi}
+
+\providecommand\bibcite{}
+\renewcommand\bibcite[2]{%
+ \@ifundefined{b@#1\@extra@binfo}{\relax}{%
+   \NAT@citemultiple
+   \PackageWarningNoLine{natbib}{Citation `#1' multiply defined}%
+ }%
+ \global\@namedef{b@#1\@extra@binfo}{#2}%
+}%
+\AtEndDocument{\NAT@swatrue\let\bibcite\NAT@testdef}
+\newcommand\NAT@testdef[2]{%
+  \def\NAT@temp{#2}%
+  \expandafter \ifx \csname b@#1\@extra@binfo\endcsname\NAT@temp
+  \else
+    \ifNAT@swa \NAT@swafalse
+      \PackageWarningNoLine{natbib}{%
+        Citation(s) may have changed.\MessageBreak
+        Rerun to get citations correct%
+      }%
+    \fi
+  \fi
+}%
+\newcommand\NAT@apalk{}
+\def\NAT@apalk#1, #2, #3\@nil#4{%
+  \if\relax#2\relax
+    \global\NAT@stdbsttrue
+    \NAT@wrout{#1}{}{}{}{#4}%
+  \else
+    \NAT@wrout{\the\c@NAT@ctr}{#2}{#1}{}{#4}%
+  \fi
+}%
+\newcommand\citeauthoryear{}
+\def\citeauthoryear#1#2#3(@)(@)\@nil#4{%
+  \if\relax#3\relax
+    \NAT@wrout{\the\c@NAT@ctr}{#2}{#1}{}{#4}%
+  \else
+    \NAT@wrout{\the\c@NAT@ctr}{#3}{#2}{#1}{#4}%
+  \fi
+}%
+\newcommand\citestarts{\NAT@open}%
+\newcommand\citeends{\NAT@close}%
+\newcommand\betweenauthors{and}%
+\newcommand\astroncite{}
+\def\astroncite#1#2(@)(@)\@nil#3{%
+ \NAT@wrout{\the\c@NAT@ctr}{#2}{#1}{}{#3}%
+}%
+\newcommand\citename{}
+\def\citename#1#2(@)(@)\@nil#3{\expandafter\NAT@apalk#1#2, \@nil{#3}}
+\newcommand\harvarditem[4][]{%
+ \if\relax#1\relax
+   \bibitem[#2(#3)]{#4}%
+ \else
+   \bibitem[#1(#3)#2]{#4}%
+ \fi
+}%
+\newcommand\harvardleft{\NAT@open}
+\newcommand\harvardright{\NAT@close}
+\newcommand\harvardyearleft{\NAT@open}
+\newcommand\harvardyearright{\NAT@close}
+\AtBeginDocument{\providecommand{\harvardand}{and}}
+\newcommand\harvardurl[1]{\textbf{URL:} \textit{#1}}
+\providecommand\bibsection{}
+\@ifundefined{chapter}{%
+  \renewcommand\bibsection{%
+   \section*{\refname\@mkboth{\MakeUppercase{\refname}}{\MakeUppercase{\refname}}}%
+  }%
+}{%
+  \@ifxundefined\NAT@sectionbib{%
+    \renewcommand\bibsection{%
+      \chapter*{\bibname\@mkboth{\MakeUppercase{\bibname}}{\MakeUppercase{\bibname}}}%
+    }%
+  }{%
+    \renewcommand\bibsection{%
+      \section*{\bibname\ifx\@mkboth\@gobbletwo\else\markright{\MakeUppercase{\bibname}}\fi}%
+    }%
+  }%
+}%
+\@ifclassloaded{amsart}{\renewcommand\bibsection{\section*{\refname}}}{}%
+\@ifclassloaded{amsbook}{\renewcommand\bibsection{\chapter*{\bibname}}}{}%
+\@ifxundefined\bib@heading{}{\let\bibsection\bib@heading}%
+\newcounter{NAT@ctr}
+\renewenvironment{thebibliography}[1]{%
+ \bibsection
+ \parindent\z@
+ \bibpreamble
+ \bibfont
+ \list{\@biblabel{\the\c@NAT@ctr}}{\@bibsetup{#1}\global\c@NAT@ctr\z@}%
+ \ifNAT@openbib
+   \renewcommand\newblock{\par}%
+ \else
+   \renewcommand\newblock{\hskip .11em \@plus.33em \@minus.07em}%
+ \fi
+ \sloppy\clubpenalty4000\widowpenalty4000
+ \sfcode`\.\@m
+ \let\NAT@bibitem@first@sw\@firstoftwo
+    \let\citeN\cite \let\shortcite\cite
+    \let\citeasnoun\cite
+}{%
+ \bibitem@fin
+ \bibpostamble
+ \def\@noitemerr{%
+  \PackageWarning{natbib}{Empty `thebibliography' environment}%
+ }%
+ \endlist
+ \bibcleanup
+}%
+\let\bibfont\@empty
+\let\bibpreamble\@empty
+\let\bibpostamble\@empty
+\def\bibcleanup{\vskip-\lastskip}%
+\providecommand\reset@font{\relax}
+\providecommand\bibname{Bibliography}
+\providecommand\refname{References}
+\newcommand\NAT@citeundefined{\gdef \NAT@undefined {%
+    \PackageWarningNoLine{natbib}{There were undefined citations}}}
+\let \NAT@undefined \relax
+\newcommand\NAT@citemultiple{\gdef \NAT@multiple {%
+    \PackageWarningNoLine{natbib}{There were multiply defined citations}}}
+\let \NAT@multiple \relax
+\AtEndDocument{\NAT@undefined\NAT@multiple}
+\providecommand\@mkboth[2]{}
+\providecommand\MakeUppercase{\uppercase}
+\providecommand{\@extra@b@citeb}{}
+\gdef\@extra@binfo{}
+\def\NAT@anchor#1#2{%
+ \hyper@natanchorstart{#1\@extra@b@citeb}%
+  \def\@tempa{#2}\@ifx{\@tempa\@empty}{}{\@biblabel{#2}}%
+ \hyper@natanchorend
+}%
+\providecommand\hyper@natanchorstart[1]{}%
+\providecommand\hyper@natanchorend{}%
+\providecommand\hyper@natlinkstart[1]{}%
+\providecommand\hyper@natlinkend{}%
+\providecommand\hyper@natlinkbreak[2]{#1}%
+\AtBeginDocument{%
+  \@ifpackageloaded{babel}{%
+     \let\org@@citex\@citex}{}}
+\providecommand\@safe@activestrue{}%
+\providecommand\@safe@activesfalse{}%
+
+\newcommand\NAT@sort@cites[1]{%
+  \let\NAT@cite@list\@empty
+  \@for\@citeb:=#1\do{\expandafter\NAT@star@cite\@citeb\@@}%
+  \if@filesw
+    \expandafter\immediate\expandafter\write\expandafter\@auxout
+      \expandafter{\expandafter\string\expandafter\citation\expandafter{\NAT@cite@list}}%
+  \fi
+  \@ifnum{\NAT@sort>\z@}{%
+    \expandafter\NAT@sort@cites@\expandafter{\NAT@cite@list}%
+  }{}%
+}%
+\def\NAT@star@cite{%
+  \let\NAT@star@sw\@secondoftwo
+  \@ifnum{\NAT@merge>\z@}{%
+   \@ifnextchar*{%
+    \let\NAT@star@sw\@firstoftwo
+    \NAT@star@cite@star
+   }{%
+    \NAT@star@cite@nostar
+   }%
+  }{%
+   \NAT@star@cite@noextension
+  }%
+}%
+\def\NAT@star@cite@star*{%
+ \NAT@star@cite@nostar
+}%
+\def\NAT@star@cite@nostar{%
+ \let\nat@keyopt@open\@empty
+ \let\nat@keyopt@shut\@empty
+ \@ifnextchar[{\NAT@star@cite@pre}{\NAT@star@cite@pre[]}%
+}%
+\def\NAT@star@cite@pre[#1]{%
+ \def\nat@keyopt@open{#1}%
+ \@ifnextchar[{\NAT@star@cite@post}{\NAT@star@cite@post[]}%
+}%
+\def\NAT@star@cite@post[#1]#2\@@{%
+ \def\nat@keyopt@shut{#1}%
+ \NAT@star@sw{\expandafter\global\expandafter\let\csname NAT@b*@#2\endcsname\@empty}{}%
+ \NAT@cite@list@append{#2}%
+}%
+\def\NAT@star@cite@noextension#1\@@{%
+  \let\nat@keyopt@open\@empty
+  \let\nat@keyopt@shut\@empty
+  \NAT@cite@list@append{#1}%
+}%
+\def\NAT@cite@list@append#1{%
+  \edef\@citeb{\@firstofone#1\@empty}%
+  \if@filesw\@ifxundefined\@cprwrite{}{\expandafter\@cprwrite\@citeb=}\fi
+  \if\relax\nat@keyopt@open\relax\else
+   \global\expandafter\let\csname NAT@b@open@\@citeb\endcsname\nat@keyopt@open
+  \fi
+  \if\relax\nat@keyopt@shut\relax\else
+   \global\expandafter\let\csname NAT@b@shut@\@citeb\endcsname\nat@keyopt@shut
+  \fi
+  \toks@\expandafter{\NAT@cite@list}%
+  \ifx\NAT@cite@list\@empty
+    \@temptokena\expandafter{\@citeb}%
+  \else
+    \@temptokena\expandafter{\expandafter,\@citeb}%
+  \fi
+  \edef\NAT@cite@list{\the\toks@\the\@temptokena}%
+}%
+\newcommand\NAT@sort@cites@[1]{%
+  \count@\z@
+  \@tempcntb\m@ne
+  \let\@celt\delimiter
+  \def\NAT@num@list{}%
+  \let\NAT@cite@list\@empty
+  \let\NAT@nonsort@list\@empty
+  \@for \@citeb:=#1\do{\NAT@make@cite@list}%
+  \ifx\NAT@nonsort@list\@empty\else
+   \protected@edef\NAT@cite@list{\NAT@cite@list\NAT@nonsort@list}%
+  \fi
+  \ifx\NAT@cite@list\@empty\else
+   \protected@edef\NAT@cite@list{\expandafter\NAT@xcom\NAT@cite@list @@}%
+  \fi
+}%
+\def\NAT@make@cite@list{%
+  \advance\count@\@ne
+  \@safe@activestrue
+  \edef\@citeb{\expandafter\@firstofone\@citeb\@empty}%
+  \@safe@activesfalse
+  \@ifundefined{b@\@citeb\@extra@b@citeb}%
+   {\def\NAT@num{A}}%
+   {\NAT@parse{\@citeb}}%
+  \NAT@ifcat@num\NAT@num
+   {\@tempcnta\NAT@num \relax
+    \@ifnum{\@tempcnta<\@tempcntb}{%
+      \let\NAT@@cite@list=\NAT@cite@list
+      \let\NAT@cite@list\@empty
+      \begingroup\let\@celt=\NAT@celt\NAT@num@list\endgroup
+      \protected@edef\NAT@num@list{%
+       \expandafter\NAT@num@celt \NAT@num@list \@gobble @%
+      }%
+    }{%
+      \protected@edef\NAT@num@list{\NAT@num@list \@celt{\NAT@num}}%
+      \protected@edef\NAT@cite@list{\NAT@cite@list\@citeb,}%
+      \@tempcntb\@tempcnta
+    }%
+   }%
+   {\protected@edef\NAT@nonsort@list{\NAT@nonsort@list\@citeb,}}%
+}%
+\def\NAT@celt#1{%
+  \@ifnum{#1>\@tempcnta}{%
+    \xdef\NAT@cite@list{\NAT@cite@list\@citeb,\NAT@@cite@list}%
+    \let\@celt\@gobble
+  }{%
+    \expandafter\def@NAT@cite@lists\NAT@@cite@list\@@
+  }%
+}%
+\def\NAT@num@celt#1#2{%
+ \ifx#1\@celt
+  \@ifnum{#2>\@tempcnta}{%
+    \@celt{\number\@tempcnta}%
+    \@celt{#2}%
+  }{%
+    \@celt{#2}%
+    \expandafter\NAT@num@celt
+  }%
+ \fi
+}%
+\def\def@NAT@cite@lists#1,#2\@@{%
+  \xdef\NAT@cite@list{\NAT@cite@list#1,}%
+  \xdef\NAT@@cite@list{#2}%
+}%
+\def\NAT@nextc#1,#2@@{#1,}
+\def\NAT@restc#1,#2{#2}
+\def\NAT@xcom#1,@@{#1}
+\InputIfFileExists{natbib.cfg}
+       {\typeout{Local config file natbib.cfg used}}{}
+%% 
+%% <<<<< End of generated file <<<<<<
+%%
+%% End of file `natbib.sty'.

outputs/outputs_20230608_115759/ref.bib

@@ -0,0 +1,784 @@
+@article{haarnoja2018soft,
+          title = {Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor},
+          author = {Tuomas Haarnoja and Aurick Zhou and P. Abbeel and S. Levine}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2018}, 
+          url = {dblp.org/rec/conf/icml/HaarnojaZAL18}
+        }
+
+@article{mnih2016asynchronous,
+          title = {Asynchronous Methods for Deep Reinforcement Learning},
+          author = {Volodymyr Mnih and Adrià Puigdomènech Badia and Mehdi Mirza and A. Graves and T. Lillicrap and Tim Harley and David Silver and K. Kavukcuoglu}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2016}, 
+          url = {dblp.org/rec/journals/corr/MnihBMGLHSK16}
+        }
+
+@article{zoph2016neural,
+          title = {Neural Architecture Search with Reinforcement Learning},
+          author = {Barret Zoph and Quoc V. Le}, 
+          journal={International Conference on Learning Representations}, 
+          year = {2016}, 
+          url = {dblp.org/rec/conf/iclr/ZophL17}
+        }
+
+@article{mnih2013playing,
+          title = {Playing Atari with Deep Reinforcement Learning},
+          author = {Volodymyr Mnih and K. Kavukcuoglu and David Silver and A. Graves and Ioannis Antonoglou and Daan Wierstra and Martin A. Riedmiller}, 
+          journal={arXiv.org}, 
+          year = {2013}, 
+          url = {dblp.org/rec/journals/corr/MnihKSGAWR13}
+        }
+
+@article{lillicrap2015continuous,
+          title = {Continuous control with deep reinforcement learning},
+          author = {T. Lillicrap and Jonathan J. Hunt and A. Pritzel and N. Heess and T. Erez and Yuval Tassa and David Silver and Daan Wierstra}, 
+          journal={International Conference on Learning Representations}, 
+          year = {2015}, 
+          url = {dblp.org/rec/journals/corr/LillicrapHPHETS15}
+        }
+
+@article{hasselt2015deep,
+          title = {Deep Reinforcement Learning with Double Q-Learning},
+          author = {H. V. Hasselt and A. Guez and David Silver}, 
+          journal={AAAI Conference on Artificial Intelligence}, 
+          year = {2015}, 
+          url = {dblp.org/rec/journals/corr/HasseltGS15}
+        }
+
+@article{sutton2005reinforcement,
+          title = {Reinforcement Learning: An Introduction},
+          author = {R. Sutton and A. Barto}, 
+          journal={IEEE Transactions on Neural Networks}, 
+          year = {2005}, 
+          url = {dblp.org/rec/journals/tnn/SuttonB98}
+        }
+
+@article{silver2018a,
+          title = {A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play},
+          author = {David Silver and T. Hubert and Julian Schrittwieser and Ioannis Antonoglou and Matthew Lai and A. Guez and Marc Lanctot and L. Sifre and D. Kumaran and T. Graepel and T. Lillicrap and K. Simonyan and D. Hassabis}, 
+          journal={Science}, 
+          year = {2018}, 
+          url = {}
+        }
+
+@article{mcmahan2016communication,
+          title = {Communication-Efficient Learning of Deep Networks from Decentralized Data},
+          author = {H. B. McMahan and Eider Moore and D. Ramage and S. Hampson and B. A. Y. Arcas}, 
+          journal={International Conference on Artificial Intelligence and Statistics}, 
+          year = {2016}, 
+          url = {dblp.org/rec/conf/aistats/McMahanMRHA17}
+        }
+
+@article{he2022byzantine,
+          title = {Byzantine-Robust Decentralized Learning via Self-Centered Clipping},
+          author = {Lie He and Sai Praneeth Karimireddy and Martin Jaggi}, 
+          journal={arXiv.org}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/corr/abs-2202-01545}
+        }
+
+@article{fu2022lightweight,
+          title = {Lightweight Automatic Modulation Classification Based on Decentralized Learning},
+          author = {Xue Fu and Guan Gui and Yu Wang and T. Ohtsuki and B. Adebisi and H. Gačanin and F. Adachi}, 
+          journal={IEEE Transactions on Cognitive Communications and Networking}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tccn/FuGWOAGA22}
+        }
+
+@article{fu2022automatic,
+          title = {Automatic Modulation Classification Based on Decentralized Learning and Ensemble Learning},
+          author = {Xue Fu and Guan Gui and Yu Wang and H. Gačanin and F. Adachi}, 
+          journal={IEEE Transactions on Vehicular Technology}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tvt/FuG0GA22}
+        }
+
+@article{dandi2022data,
+          title = {Data-heterogeneity-aware Mixing for Decentralized Learning},
+          author = {Yatin Dandi and Anastasia Koloskova and Martin Jaggi and S. Stich}, 
+          journal={arXiv.org}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/corr/abs-2204-06477}
+        }
+
+@article{jeong2022asynchronous,
+          title = {Asynchronous Decentralized Learning over Unreliable Wireless Networks},
+          author = {Eunjeong Jeong and Matteo Zecchin and M. Kountouris}, 
+          journal={ICC 2022 - IEEE International Conference on Communications}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/icc/JeongZK22}
+        }
+
+@article{li2022learning,
+          title = {Learning to Collaborate in Decentralized Learning of Personalized Models},
+          author = {Shuang-Yang Li and Tianyi Zhou and Xinmei Tian and Dacheng Tao}, 
+          journal={Computer Vision and Pattern Recognition}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/cvpr/LiZ0T22}
+        }
+
+@article{esfandiari2021cross,
+          title = {Cross-Gradient Aggregation for Decentralized Learning from Non-IID data},
+          author = {Yasaman Esfandiari and Sin Yong Tan and Zhanhong Jiang and Aditya Balu and Ethan Herron and C. Hegde and S. Sarkar}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2021}, 
+          url = {dblp.org/rec/journals/corr/abs-2103-02051}
+        }
+
+@article{taheri2022on,
+          title = {On Generalization of Decentralized Learning with Separable Data},
+          author = {Hossein Taheri and Christos Thrampoulidis}, 
+          journal={arXiv preprint}, 
+          year = {2022}, 
+          url = {arxiv.org/abs/2209.07116}
+        }
+
+@article{paszke2019pytorch,
+          title = {PyTorch: An Imperative Style, High-Performance Deep Learning Library},
+          author = {Adam Paszke and Sam Gross and Francisco Massa and Adam Lerer and James Bradbury and Gregory Chanan and Trevor Killeen and Zeming Lin and N. Gimelshein and L. Antiga and Alban Desmaison and Andreas Köpf and E. Yang and Zach DeVito and Martin Raison and A. Tejani and Sasank Chilamkurthy and Benoit Steiner and Lu Fang and Junjie Bai and Soumith Chintala}, 
+          journal={Neural Information Processing Systems}, 
+          year = {2019}, 
+          url = {dblp.org/rec/journals/corr/abs-1912-01703}
+        }
+
+@article{madry2017towards,
+          title = {Towards Deep Learning Models Resistant to Adversarial Attacks},
+          author = {A. Madry and Aleksandar Makelov and Ludwig Schmidt and Dimitris Tsipras and Adrian Vladu}, 
+          journal={International Conference on Learning Representations}, 
+          year = {2017}, 
+          url = {dblp.org/rec/conf/iclr/MadryMSTV18}
+        }
+
+@article{gal2015dropout,
+          title = {Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning},
+          author = {Y. Gal and Zoubin Ghahramani}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2015}, 
+          url = {dblp.org/rec/journals/corr/GalG15}
+        }
+
+@article{chollet2016xception,
+          title = {Xception: Deep Learning with Depthwise Separable Convolutions},
+          author = {François Chollet}, 
+          journal={Computer Vision and Pattern Recognition}, 
+          year = {2016}, 
+          url = {dblp.org/rec/journals/corr/Chollet16a}
+        }
+
+@article{liu2014deep,
+          title = {Deep Learning Face Attributes in the Wild},
+          author = {Ziwei Liu and Ping Luo and Xiaogang Wang and Xiaoou Tang}, 
+          journal={IEEE International Conference on Computer Vision}, 
+          year = {2014}, 
+          url = {dblp.org/rec/journals/corr/LiuLWT14}
+        }
+
+@article{qi2016pointnet,
+          title = {PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation},
+          author = {C. Qi and Hao Su and Kaichun Mo and L. Guibas}, 
+          journal={Computer Vision and Pattern Recognition}, 
+          year = {2016}, 
+          url = {dblp.org/rec/journals/corr/QiSMG16}
+        }
+
+@article{gulshan2016development,
+          title = {Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.},
+          author = {Varun Gulshan and L. Peng and Marc Coram and Martin C. Stumpe and Derek J. Wu and Arunachalam Narayanaswamy and Subhashini Venugopalan and Kasumi Widner and T. Madams and Jorge A Cuadros and R. Kim and R. Raman and Philip Nelson and J. Mega and D. Webster}, 
+          journal={Journal of the American Medical Association (JAMA)}, 
+          year = {2016}, 
+          url = {}
+        }
+
+@article{zhang2016understanding,
+          title = {Understanding deep learning requires rethinking generalization},
+          author = {Chiyuan Zhang and Samy Bengio and Moritz Hardt and B. Recht and Oriol Vinyals}, 
+          journal={International Conference on Learning Representations}, 
+          year = {2016}, 
+          url = {dblp.org/rec/conf/iclr/ZhangBHRV17}
+        }
+
+@article{adadi2018peeking,
+          title = {Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI)},
+          author = {Amina Adadi and M. Berrada}, 
+          journal={IEEE Access}, 
+          year = {2018}, 
+          url = {dblp.org/rec/journals/access/AdadiB18}
+        }
+
+@article{ridley2022explainable,
+          title = {Explainable Artificial Intelligence (XAI)},
+          author = {M. Ridley}, 
+          journal={Information Technology and Libraries}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{russell1995artificial,
+          title = {Artificial Intelligence: A Modern Approach},
+          author = {Stuart J. Russell and Peter Norvig}, 
+          journal={arXiv preprint}, 
+          year = {1995}, 
+          url = {}
+        }
+
+@article{ma2022artificial,
+          title = {Artificial Intelligence A Modern Approach Global Edition},
+          author = {}, 
+          journal={arXiv preprint}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{holland1992adaptation,
+          title = {Adaptation in Natural and Artificial Systems: An Introductory Analysis with Applications to Biology, Control, and Artificial Intelligence},
+          author = {J. Holland}, 
+          journal={arXiv preprint}, 
+          year = {1992}, 
+          url = {dblp.org/rec/books/mit/H1992}
+        }
+
+@article{dufwenberg2011game,
+          title = {Game theory.},
+          author = {M. Dufwenberg}, 
+          journal={Wiley Interdisciplinary Reviews: Cognitive Science}, 
+          year = {2011}, 
+          url = {}
+        }
+
+@article{9–jan2022combinatorial,
+          title = {Combinatorial Game Theory},
+          author = {Jan. 9–Jan}, 
+          journal={arXiv preprint}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{osborne1995a,
+          title = {A Course in Game Theory},
+          author = {M. Osborne and A. Rubinstein}, 
+          journal={arXiv preprint}, 
+          year = {1995}, 
+          url = {}
+        }
+
+@article{camerer2003behavioral,
+          title = {Behavioral Game Theory: Experiments in Strategic Interaction},
+          author = {Colin Camerer}, 
+          journal={arXiv preprint}, 
+          year = {2003}, 
+          url = {}
+        }
+
+@article{myerson1991game,
+          title = {Game theory - Analysis of Conflict},
+          author = {R. Myerson}, 
+          journal={arXiv preprint}, 
+          year = {1991}, 
+          url = {dblp.org/rec/books/daglib/0023252}
+        }
+
+@article{rabin1993incorporating,
+          title = {Incorporating Fairness into Game Theory and Economics},
+          author = {M. Rabin}, 
+          journal={arXiv preprint}, 
+          year = {1993}, 
+          url = {}
+        }
+
+@article{roughgarden2010algorithmic,
+          title = {Algorithmic game theory},
+          author = {T. Roughgarden}, 
+          journal={Communications of the ACM}, 
+          year = {2010}, 
+          url = {dblp.org/rec/journals/cacm/Roughgarden10}
+        }
+
+@article{hesteren2017evolutionary,
+          title = {Evolutionary Game Theory},
+          author = {D. M. V. Hesteren}, 
+          journal={arXiv preprint}, 
+          year = {2017}, 
+          url = {}
+        }
+
+@article{chen2022multi,
+          title = {Multi-Agent Reinforcement Learning for Decentralized Resilient Secondary Control of Energy Storage Systems Against DoS Attacks},
+          author = {Pengcheng Chen and Shichao Liu and Bo Chen and Li Yu}, 
+          journal={IEEE Transactions on Smart Grid}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tsg/ChenLCY22}
+        }
+
+@article{lyu2021contrasting,
+          title = {Contrasting Centralized and Decentralized Critics in Multi-Agent Reinforcement Learning},
+          author = {Xueguang Lyu and Yuchen Xiao and Brett Daley and Chris Amato}, 
+          journal={Adaptive Agents and Multi-Agent Systems}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/atal/LyuXDA21}
+        }
+
+@article{su2022ma2ql,
+          title = {MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning},
+          author = {Kefan Su and Siyuan Zhou and Chuang Gan and Xiangjun Wang and Zongqing Lu}, 
+          journal={arXiv.org}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/corr/abs-2209-08244}
+        }
+
+@article{liu2022federated,
+          title = {Federated Reinforcement Learning for Decentralized Voltage Control in Distribution Networks},
+          author = {Haotian Liu and Wenchuan Wu}, 
+          journal={IEEE Transactions on Smart Grid}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tsg/LiuW22a}
+        }
+
+@article{lei2022adaptive,
+          title = {Adaptive Stochastic ADMM for Decentralized Reinforcement Learning in Edge IoT},
+          author = {Wanlu Lei and Yu Ye and M. Xiao and M. Skoglund and Zhu Han}, 
+          journal={IEEE Internet of Things Journal}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/iotj/LeiYXSH22}
+        }
+
+@article{lu2021decentralized,
+          title = {Decentralized Policy Gradient Descent Ascent for Safe Multi-Agent Reinforcement Learning},
+          author = {Songtao Lu and K. Zhang and Tianyi Chen and T. Başar and L. Horesh}, 
+          journal={AAAI Conference on Artificial Intelligence}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/aaai/LuZCBH21}
+        }
+
+@article{thumiger2022a,
+          title = {A Multi-Agent Deep Reinforcement Learning Approach for Practical Decentralized UAV Collision Avoidance},
+          author = {Nicholas Thumiger and M. Deghat}, 
+          journal={IEEE Control Systems Letters}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/csysl/ThumigerD22}
+        }
+
+@article{rakkini2022comprehensive,
+          title = {Comprehensive overview on the deployment of machine learning, deep learning, reinforcement learning algorithms in Selfish mining attack in blockchain},
+          author = {M. J. Jeyasheela Rakkini and K. Geetha}, 
+          journal={2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon)}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{sahu2023an,
+          title = {An Overview of Machine Learning, Deep Learning, and Reinforcement Learning-Based Techniques in Quantitative Finance: Recent Progress and Challenges},
+          author = {S. Sahu and A. Mokhade and N. Bokde}, 
+          journal={Applied Sciences}, 
+          year = {2023}, 
+          url = {}
+        }
+
+@article{zhao2022alphaholdem,
+          title = {AlphaHoldem: High-Performance Artificial Intelligence for Heads-Up No-Limit Poker via End-to-End Reinforcement Learning},
+          author = {Enmin Zhao and Renye Yan and Jinqiu Li and Kai Li and Junliang Xing}, 
+          journal={AAAI Conference on Artificial Intelligence}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/aaai/ZhaoYLLX22}
+        }
+
+@article{ma2021algorithms,
+          title = {Algorithms For Reinforcement Learning Synthesis Lectures On Artificial Intelligence And Machine Learning Epdf Read},
+          author = {}, 
+          journal={arXiv preprint}, 
+          year = {2021}, 
+          url = {}
+        }
+
+@article{yang2021an,
+          title = {An Information Fusion Approach to Intelligent Traffic Signal Control Using the Joint Methods of Multiagent Reinforcement Learning and Artificial Intelligence of Things},
+          author = {Xiaoxian Yang and Yueshen Xu and Li Kuang and Zhiying Wang and Honghao Gao and Xuejie Wang}, 
+          journal={IEEE transactions on intelligent transportation systems (Print)}, 
+          year = {2021}, 
+          url = {dblp.org/rec/journals/tits/YangXKWGW22}
+        }
+
+@article{ribba2020model,
+          title = {Model‐Informed Artificial Intelligence: Reinforcement Learning for Precision Dosing},
+          author = {B. Ribba and S. Dudal and T. Lavé and R. Peck}, 
+          journal={Clinical pharmacology and therapy}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{hrinivich2020artificial,
+          title = {Artificial intelligence-based radiotherapy machine parameter optimization using reinforcement learning.},
+          author = {W. Hrinivich and Junghoon Lee}, 
+          journal={Medical Physics (Lancaster)}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{liu2022joint,
+          title = {Joint Beamforming, Power Allocation, and Splitting Control for SWIPT-Enabled IoT Networks with Deep Reinforcement Learning and Game Theory},
+          author = {Jain-Shing Liu and C. Lin and Yu‐Chen Hu and Praveen Kumar Donta}, 
+          journal={Italian National Conference on Sensors}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/sensors/LiuLHD22}
+        }
+
+@article{duan2022autonomous,
+          title = {Autonomous driving planning and decision making based on game theory and reinforcement learning},
+          author = {Weiping Duan and Zhongyi Tang and Wei Liu and Hongbiao Zhou}, 
+          journal={Expert Syst. J. Knowl. Eng.}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/es/DuanTLZ23}
+        }
+
+@article{liang2022gadqn,
+          title = {GaDQN-IDS: A Novel Self-Adaptive IDS for VANETs Based on Bayesian Game Theory and Deep Reinforcement Learning},
+          author = {Junwei Liang and M. Ma and Xu Tan}, 
+          journal={IEEE transactions on intelligent transportation systems (Print)}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tits/LiangMT22}
+        }
+
+@article{jin2022security,
+          title = {Security State Estimation for Cyber-Physical Systems against DoS Attacks via Reinforcement Learning and Game Theory},
+          author = {Zengwang Jin and Shuting Zhang and Yanyan Hu and Yanning Zhang and Changyin Sun}, 
+          journal={Actuators}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{yin2022air,
+          title = {Air Combat Maneuver Decision Based on Deep Reinforcement Learning and Game Theory},
+          author = {Shuhui Yin and Yu Kang and Yunbo Zhao and Jian Xue}, 
+          journal={Cybersecurity and Cyberforensics Conference}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{zwillinger2022distributing,
+          title = {Distributing data throughout a MANET in a communications denied environment: reinforcement learning and game theory approaches},
+          author = {D. Zwillinger and J. Sierchio and M. Gerken and Emily Clark}, 
+          journal={Defense + Commercial Sensing}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{purfatideh2014efficent,
+          title = {Efficent Congestion Control Scheme in Computer Networks using Fuzzy Reinforcement Learning & Game Theory},
+          author = {Mohamad Bagher Safari Purfatideh and S. Jamali and Morteza Analoei}, 
+          journal={arXiv preprint}, 
+          year = {2014}, 
+          url = {}
+        }
+
+@article{teymoori2022dynamic,
+          title = {Dynamic Multi-user Computation Offloading for Mobile Edge Computing using Game Theory and Deep Reinforcement Learning},
+          author = {P. Teymoori and A. Boukerche}, 
+          journal={ICC 2022 - IEEE International Conference on Communications}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/icc/TeymooriB22}
+        }
+
+@article{huang2022toward,
+          title = {Toward Decentralized and Collaborative Deep Learning Inference for Intelligent IoT Devices},
+          author = {Yakun Huang and Xiuquan Qiao and S. Dustdar and Jianwei Zhang and Jiulin Li}, 
+          journal={IEEE Network}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/network/HuangQDZL22}
+        }
+
+@article{lin2021quasi,
+          title = {Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data},
+          author = {Tao Lin and Sai Praneeth Karimireddy and S. Stich and Martin Jaggi}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2021}, 
+          url = {dblp.org/rec/journals/corr/abs-2102-04761}
+        }
+
+@article{takezawa2022momentum,
+          title = {Momentum Tracking: Momentum Acceleration for Decentralized Deep Learning on Heterogeneous Data},
+          author = {Yuki Takezawa and Hang Bao and K. Niwa and R. Sato and Makoto Yamada}, 
+          journal={arXiv.org}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/corr/abs-2209-15505}
+        }
+
+@article{kong2021consensus,
+          title = {Consensus Control for Decentralized Deep Learning},
+          author = {Lingjing Kong and Tao Lin and Anastasia Koloskova and Martin Jaggi and S. Stich}, 
+          journal={International Conference on Machine Learning}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/icml/00010KJS21}
+        }
+
+@article{vogels2021relaysum,
+          title = {RelaySum for Decentralized Deep Learning on Heterogeneous Data},
+          author = {Thijs Vogels and Lie He and Anastasia Koloskova and Tao Lin and Sai Praneeth Karimireddy and S. Stich and Martin Jaggi}, 
+          journal={Neural Information Processing Systems}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/nips/VogelsHKKLSJ21}
+        }
+
+@article{shiri2022decentralized,
+          title = {Decentralized Distributed Multi-institutional PET Image Segmentation Using a Federated Deep Learning Framework},
+          author = {I. Shiri and A. Vafaei Sadr and Mehdi Amini and Y. Salimi and Amirhossein Sanaat and A. Akhavanallaf and Behrooz Razeghi and Sohrab Ferdowsi and A. Saberi and Hossein ARABI and M. Becker and S. Voloshynovskiy and Deniz Gündüz and A. Rahmim and H. Zaidi}, 
+          journal={Clinical Nuclear Medicine}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{jayakody2022fake,
+          title = {Fake News Detection using a Decentralized Deep Learning Model and Federated Learning},
+          author = {Nirosh Jayakody and Azeem Mohammad and M. Halgamuge}, 
+          journal={Annual Conference of the IEEE Industrial Electronics Society}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/iecon/JayakodyMH22}
+        }
+
+@article{sun2021decentralized,
+          title = {Decentralized Deep Learning for Multi-Access Edge Computing: A Survey on Communication Efficiency and Trustworthiness},
+          author = {Yuwei Sun and H. Ochiai and H. Esaki}, 
+          journal={IEEE Transactions on Artificial Intelligence}, 
+          year = {2021}, 
+          url = {dblp.org/rec/journals/tai/SunOE22}
+        }
+
+@article{lim2022decentralized,
+          title = {Decentralized Edge Intelligence: A Dynamic Resource Allocation Framework for Hierarchical Federated Learning},
+          author = {Wei Yang Bryan Lim and Jer Shyuan Ng and Zehui Xiong and Jiangming Jin and Yang Zhang and D. Niyato and C. Leung and C. Miao}, 
+          journal={IEEE Transactions on Parallel and Distributed Systems}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tpds/LimNXJZNLM22}
+        }
+
+@article{kang2022communication,
+          title = {Communication-Efficient and Cross-Chain Empowered Federated Learning for Artificial Intelligence of Things},
+          author = {Jiawen Kang and Xuandi Li and Jiangtian Nie and Yi Liu and Minrui Xu and Zehui Xiong and D. Niyato and Qiang Yan}, 
+          journal={IEEE Transactions on Network Science and Engineering}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/tnse/KangLNLXXNY22}
+        }
+
+@article{s2022converging,
+          title = {Converging Blockchain and Artificial-Intelligence Towards Healthcare: A Decentralized-Private and Intelligence Health Record System},
+          author = {Arpith S and G. Mufeed and Anusha K R and Gahana}, 
+          journal={2022 2nd International Conference on Intelligent Technologies (CONIT)}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{facchini2022decentralized,
+          title = {Decentralized Autonomous Organizations and Multi-agent Systems for Artificial Intelligence Applications and Data Analysis},
+          author = {Sante Dino Facchini}, 
+          journal={International Joint Conference on Artificial Intelligence}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/ijcai/Facchini22}
+        }
+
+@article{nouruzi2022toward,
+          title = {Toward a Smart Resource Allocation Policy via Artificial Intelligence in 6G Networks: Centralized or Decentralized?},
+          author = {A. Nouruzi and A. Rezaei and Ata Khalili and N. Mokari and M. Javan and Eduard Axel Jorswieck and H. Yanikomeroglu}, 
+          journal={arXiv.org}, 
+          year = {2022}, 
+          url = {dblp.org/rec/journals/corr/abs-2202-09093}
+        }
+
+@article{jose2022application,
+          title = {Application of artificial intelligence in secure decentralized computation enabled by TOTEM},
+          author = {Dhanya Therese Jose and Chunming Rong and Antorweep Chakravorty}, 
+          journal={2022 IEEE Asia-Pacific Conference on Computer Science and Data Engineering (CSDE)}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{clough2020artificial,
+          title = {Artificial Intelligence, Data-Driven Learning, and the Decentralized Structure of Platform Ecosystems},
+          author = {David R. Clough and Andy Wu}, 
+          journal={arXiv preprint}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{hu2020energy,
+          title = {Energy Management for isolated renewable-powered microgrids using reinforcement learning and game theory},
+          author = {R. Hu and A. Kwasinski}, 
+          journal={EPE}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{adams2020resolving,
+          title = {Resolving Implicit Coordination in Multi-Agent Deep Reinforcement Learning with Deep Q-Networks & Game Theory},
+          author = {Griffin Adams and Sarguna Padmanabhan and S. Shekhar}, 
+          journal={arXiv.org}, 
+          year = {2020}, 
+          url = {dblp.org/rec/journals/corr/abs-2012-09136}
+        }
+
+@article{zhou2019intelligent,
+          title = {Intelligent Decentralized Dynamic Power Allocation in MANET at Tactical Edge based on Mean-Field Game Theory},
+          author = {Zejian Zhou and Lijun Qian and Hao Xu}, 
+          journal={IEEE Military Communications Conference}, 
+          year = {2019}, 
+          url = {dblp.org/rec/conf/milcom/ZhouQX19}
+        }
+
+@article{zhou2021decentralized,
+          title = {Decentralized Optimal Tracking Control for Large-scale Multi-Agent Systems under Complex Environment: A Constrained Mean Field Game with Reinforcement Learning Approach},
+          author = {Zejian Zhou and Hao Xu}, 
+          journal={Conference on Control Technology and Applications}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/ccta/ZhouX21a}
+        }
+
+@article{blum2006machine,
+          title = {Machine Learning , Game Theory , and Mechanism Design for a Networked World},
+          author = {A. Blum}, 
+          journal={arXiv preprint}, 
+          year = {2006}, 
+          url = {}
+        }
+
+@article{goktas2022an,
+          title = {An Algorithmic Theory of Markets and Their Application to Decentralized Markets},
+          author = {Denizalp Goktas}, 
+          journal={AAAI Conference on Artificial Intelligence}, 
+          year = {2022}, 
+          url = {dblp.org/rec/conf/aaai/Goktas22}
+        }
+
+@article{celli2021decentralized,
+          title = {Decentralized No-regret Learning Algorithms for Extensive-form Correlated Equilibria (Extended Abstract)},
+          author = {A. Celli and A. Marchesi and Gabriele Farina and N. Gatti}, 
+          journal={International Joint Conference on Artificial Intelligence}, 
+          year = {2021}, 
+          url = {dblp.org/rec/conf/ijcai/CelliMF021}
+        }
+
+@article{kim2018a,
+          title = {A better-performing Q-learning game-theoretic distributed routing for underwater wireless sensor networks},
+          author = {Sungwook Kim}, 
+          journal={Int. J. Distributed Sens. Networks}, 
+          year = {2018}, 
+          url = {dblp.org/rec/journals/ijdsn/Kim18}
+        }
+
+@article{zhu2020deep,
+          title = {Deep‐learning artificial intelligence analysis of clinical variables predicts mortality in COVID‐19 patients},
+          author = {Jocelyn Zhu and Peilin Ge and Chun-guo Jiang and Yong Zhang and Xiaoran Li and Zirun Zhao and Liming Zhang and T. Duong}, 
+          journal={Journal of the American College of Emergency Physicians Open}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{hiraiwa2019a,
+          title = {A deep-learning artificial intelligence system for assessment of root morphology of the mandibular first molar on panoramic radiography.},
+          author = {T. Hiraiwa and Y. Ariji and M. Fukuda and Yoshitaka Kise and K. Nakata and A. Katsumata and H. Fujita and E. Ariji}, 
+          journal={Dento maxillo facial radiology}, 
+          year = {2019}, 
+          url = {}
+        }
+
+@article{lemley2017deep,
+          title = {Deep Learning for Consumer Devices and Services: Pushing the limits for machine learning, artificial intelligence, and computer vision.},
+          author = {Joseph Lemley and S. Bazrafkan and P. Corcoran}, 
+          journal={IEEE Consumer Electronics Magazine}, 
+          year = {2017}, 
+          url = {dblp.org/rec/journals/cem/LemleyBC17}
+        }
+
+@article{joonmyun2020application,
+          title = {Application Trends of Deep Learning Artificial Intelligence in Autonomous Things},
+          author = {Cho Joonmyun}, 
+          journal={arXiv preprint}, 
+          year = {2020}, 
+          url = {}
+        }
+
+@article{xie2022intelligent,
+          title = {INTELLIGENT ACQUISITION METHOD OF HERBACEOUS FLOWERS IMAGE BASED ON THEME CRAWLER, DEEP LEARNING AND GAME THEORY},
+          author = {Zhouyi Xie and Yanrong Hu and Weijun Hu}, 
+          journal={Chronos}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{anishfathima2022secure,
+          title = {Secure Wireless Sensor Network Energy Optimization Model with Game Theory and Deep Learning Algorithm},
+          author = {B. Anishfathima and M. Mahaboob and S.Gokul Kumar and A. Jabakumar}, 
+          journal={2022 8th International Conference on Advanced Computing and Communication Systems (ICACCS)}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{ardekani2022combining,
+          title = {Combining Deep Learning and Game Theory for Path Planning in Autonomous Racing Cars},
+          author = {Amirhossein Afkhami Ardekani and Amirhosein Chahe and M. R. Hairi Yazdi}, 
+          journal={International Conference on Robotics and Mechatronics}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{cunningham2023a,
+          title = {A Deep Learning Game Theoretic Model for Defending Against Large Scale Smart Grid Attacks},
+          author = {James Cunningham and Alexander J. Aved and David Ferris and Philip Morrone and Conrad S. Tucker}, 
+          journal={IEEE Transactions on Smart Grid}, 
+          year = {2023}, 
+          url = {dblp.org/rec/journals/tsg/CunninghamAFMT23}
+        }
+
+@article{jia2021lane,
+          title = {Lane-Changing Behavior Prediction Based on Game Theory and Deep Learning},
+          author = {Shuo Jia and F. Hui and Cheng Wei and Xiangmo Zhao and Jianbei Liu}, 
+          journal={Journal of Advanced Transportation}, 
+          year = {2021}, 
+          url = {}
+        }
+
+@article{sahin2022artificial,
+          title = {Artificial Intelligence, Game Theory, Programming Used Languages and Platforms, Game Types and Training Methods},
+          author = {Fatih Sahin}, 
+          journal={2022 International Conference on Artificial Intelligence of Things (ICAIoT)}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{hanley2021games,
+          title = {GAMES, game theory and artificial intelligence Game theory and artificial intelligence},
+          author = {J. Hanley}, 
+          journal={arXiv preprint}, 
+          year = {2021}, 
+          url = {}
+        }
+
+@article{wang2022deepholdem,
+          title = {DeepHoldem: An Efficient End-to-End Texas Hold'em Artificial Intelligence Fusion of Algorithmic Game Theory and Game Information},
+          author = {Ke Wang and Dongdong Bai and Qibin Zhou}, 
+          journal={International Conference on Innovative Computing and Cloud Computing}, 
+          year = {2022}, 
+          url = {}
+        }
+
+@article{shen2021interactive,
+          title = {Interactive Artificial Intelligence Meets Game Theory in Next-Generation Communication Networks},
+          author = {Jingyu Shen and Chungang Yang and Tong Li and Xinwei Wang and Yanbo Song and M. Guizani}, 
+          journal={IEEE wireless communications}, 
+          year = {2021}, 
+          url = {dblp.org/rec/journals/wc/ShenYLWSG21}
+        }
+
+@article{bai2021smart,
+          title = {Smart financial policy adjustment system based on multiple game theory and artificial intelligence},
+          author = {Yunru Bai and Guang Zhang and San Sun}, 
+          journal={2021 5th International Conference on Trends in Electronics and Informatics (ICOEI)}, 
+          year = {2021}, 
+          url = {}
+        }
+
+@article{dowe2020game,
+          title = {Game theory and Artificial Intelligence in just preservation},
+          author = {D. Dowe and N. Chmait}, 
+          journal={arXiv preprint}, 
+          year = {2020}, 
+          url = {}
+        }
+

outputs/outputs_20230608_115759/related works.tex

@@ -0,0 +1,16 @@
+\section{Related Works}
+
+\paragraph{Deep Reinforcement Learning for Atari Games}
+The seminal work by \citet{mnih2013playing} introduced the first deep learning model to successfully learn control policies directly from high-dimensional sensory input using reinforcement learning. This model outperformed all previous approaches on six of the games and surpassed a human expert on three of them. The authors later extended their work with asynchronous gradient descent for optimization of deep neural network controllers, showing success on a wide variety of continuous motor control problems and a new task of navigating random 3D mazes using a visual input \citep{mnih2016asynchronous}. However, these approaches suffer from overestimations in value function approximations, which were addressed by \citet{hasselt2015deep} through a specific adaptation to the DQN algorithm, leading to much better performance on several games.
+
+\paragraph{Decentralized Reinforcement Learning}
+Decentralized reinforcement learning has been studied in various contexts. \citet{lu2021decentralized} proposed a decentralized policy gradient (PG) method, Safe Dec-PG, to perform policy optimization based on the D-CMDP model over a network. This was the first decentralized PG algorithm that accounted for coupled safety constraints with a quantifiable convergence rate in multi-agent reinforcement learning. \citet{lei2022adaptive} introduced an adaptive stochastic incremental ADMM (asI-ADMM) algorithm for decentralized RL with edge-computing-empowered IoT networks, showing better performance in terms of communication costs and scalability compared to the state of the art. However, the work by \citet{lyu2021contrasting} highlighted misconceptions regarding centralized critics in the literature, emphasizing that both centralized and decentralized critics have different pros and cons that should be considered by algorithm designers.
+
+\paragraph{Game Theory and Multi-Agent Reinforcement Learning}
+Game theory has been widely used in combination with reinforcement learning to tackle multi-agent problems. \citet{yin2022air} proposed an algorithm based on deep reinforcement learning and game theory to solve Nash equilibrium strategy in highly competitive environments, demonstrating good convergence through simulation tests. \citet{adams2020resolving} addressed the challenges of implicit coordination in multi-agent deep reinforcement learning by combining Deep-Q Networks for policy learning with Nash equilibrium for action selection. In the context of autonomous driving, \citet{duan2022autonomous} proposed an automatic drive model based on game theory and reinforcement learning, enabling multi-agent cooperative driving with strategic reasoning and negotiation in traffic scenarios. However, these approaches often require complex computations and may not scale well to large-scale problems.
+
+\paragraph{Decentralized Learning with Communication Constraints}
+One of the challenges in decentralized learning is to handle communication constraints. \citet{kong2021consensus} showed that decentralized training converges as fast as the centralized counterpart when the training consensus distance is lower than a critical quantity, providing insights for designing better decentralized training schemes. \citet{fu2022automatic} proposed a decentralized ensemble learning framework for automatic modulation classification, reducing communication overhead while maintaining similar classification performance. In the context of multi-agent systems, \citet{su2022ma2ql} introduced MA2QL, a minimalist approach to fully decentralized cooperative MARL with theoretical guarantees on convergence to a Nash equilibrium when each agent achieves $\varepsilon$-convergence at each turn. However, these methods may still suffer from limitations in highly dynamic and complex environments.
+
+\paragraph{Decentralized Collision Avoidance}
+Decentralized collision avoidance has been an important application of reinforcement learning. \citet{thumiger2022a} proposed an improved deep reinforcement learning controller for decentralized collision avoidance using a unique architecture incorporating long-short term memory cells and a reward function inspired by gradient-based approaches. This controller outperformed existing techniques in environments with variable numbers of agents. In the context of autonomous vehicles, \citet{ardekani2022combining} suggested a novel algorithm based on Nash equilibrium and memory neural networks for path selection in highly dynamic and complex environments, showing that the obtained response matched with Nash equilibrium in 90.2 percent of the situations during simulation experiments. However, these approaches may require extensive training and computational resources, which could be a concern in real-world applications.

outputs/outputs_20230608_115759/template.tex

@@ -0,0 +1,35 @@
+\documentclass{article} % For LaTeX2e
+\UseRawInputEncoding
+\usepackage{graphicx}
+\usepackage{booktabs}
+\usepackage{iclr2022_conference, times} 
+\input{math_commands.tex} 
+\usepackage{hyperref}
+\usepackage{url} 
+\usepackage{algorithm}
+\usepackage{algpseudocode}
+
+\title{TITLE} 
+\author{GPT-4}
+
+\newcommand{\fix}{\marginpar{FIX}}
+\newcommand{\new}{\marginpar{NEW}} 
+ 
+\begin{document} 
+\maketitle
+\input{abstract.tex}
+\input{introduction.tex}
+\input{related works.tex}
+\input{backgrounds.tex}
+\input{methodology.tex}
+\input{experiments.tex}
+\input{conclusion.tex}
+
+\bibliography{ref}
+\bibliographystyle{iclr2022_conference}
+
+%\appendix
+%\section{Appendix}
+%You may include other additional sections here.
+
+\end{document}

prompts/instructions.json

@@ -0,0 +1 @@
+{"introduction": "- Include five paragraph: Establishing the motivation for the research. Explaining its importance and relevance to the AI community. Clearly state the problem you're addressing, your proposed solution, and the specific research questions or objectives. Briefly mention key related works for context and explain the main differences from this work. List three novel contributions of this paper.", "results": "Write the theoretical results section using LaTeX. Include theorem and corollary to support this paper (with formulas). Explain what assumptions are used and why they are standard and necessary. Do not include \\section{...}. ", "conclusion": "- Read the existing parts of paper and write the conclusion section.", "abstract": "- Read the existing parts of paper and write the abstract.", "backgrounds": "- Start from one high-level paragraph to state the central problem in this field with detailed examples in industrial applications and theoretical challenges. \n- Followed by two to three subsections:  Explain the foundational concepts and notations that underpin your research using as many as mathematical formulas (written in LaTeX). Introduce more necessary mathematical notations, equations, or algorithms that are connected to this work. Present detailed discussions on how these concepts are applied in this paper.", "related works": "- Discuss three to five main related fields to this paper. For each field, select five to ten key publications from references. For each reference, analyze its strengths and weaknesses in one or two sentences. Present the related works in a logical manner, often chronologically. Consider using a taxonomy or categorization to structure the discussion. Do not use \\section{...} or \\subsection{...}; use \\paragraph{...} to list related fields. ", "methodology": "- Provide a high-level overview of the proposed method at the beginning of this section. \n - Assume you have some figures ('fig1.png', 'fig2.png', ...); they can be any figures you need (e.g. flow chart, model architecture, sample output, simulation result, or others you need). Insert figures you need with informative caption. \n- Use one subsection to give a detailed formulation of the proposed method and explain how it overcomes the weakness of existing methods mentioned in this paper.  If necessary, write pseudo codes wrapped by \\begin{{algorithm}} ... \\end{{algorithm}} to explain the detailed steps instead of simply listing them. \n- Use one follow-up subsection to highlight the key concepts in the proposed method.   Elaborate the novelty of these key concepts using formulas and inserting appropriate figures. \n- Ensure the name of each subsection to be specific. \n", "experiments": "- Provide a high-level overview at the beginning of this section.\n - If necessary, include a table to compare with other methods and bold our method.\n- Assume you have some figures ('exp1.png', 'exp2.png', ...); they can be any figures you need (e.g. loss curves, comparison with other methods, visualization, or others you need). Insert figures you need with informative caption. \n- If necessary, use different subsections to distinguish different experimental setup."}

utils/prompts.py

@@ -1,34 +1,11 @@
 import logging
 from langchain import PromptTemplate
+import os, json
 
 
 log = logging.getLogger(__name__)
 
 # todo: load prompts from configurations
-
-######################################################################################################################
-# Some basic functions
-######################################################################################################################
-# def generate_keywords_prompts(title, description="", num_refs=5):
-#     prompts = f"I am writing a machine learning paper with the title '{title}'. {description}\n" \
-#                 f"Generate three to five keywords. For each keyword, rate it from 1 to {num_refs}; the larger number means more important." \
-#                 r"Your response must be in JSON format like {\"keyword1\":1, \"keyword2\":3}."
-#     return prompts
-#
-# def generate_rename_prompts(paper_info, section):
-#     prompts = f"Please read the {section} section of the paper {paper_info['title']}: {paper_info['body'][section]}. \n" \
-#               f"You need to rename this section to make it more specific to the context. " \
-#               r"Response in a dictionary format like {\"option_1\": \"new_section_name_1\", \"option_2\": \"new_section_name_2\", ...}."
-#     return prompts
-#
-# def generate_experiments_prompts(paper_info):
-#     prompts = f"I am writing a machine learning paper with the title {paper_info['title']}\n" \
-#               f"Please list two to four methods that I should compare my methods with and assign them with scores (5 means most related, 1 means least related). " \
-#               r"Response in a dictionary format like {\"method_name_1\": 2, \"method_name_2\": 5, ...}. Use abbreviation to make their names have 5 characters or less."
-#     return prompts
-
-
-
 ######################################################################################################################
 # System Message
 ######################################################################################################################
@@ -65,47 +42,53 @@ SECTION_GENERATION_SYSTEM = PromptTemplate(input_variables=["research_field"],
 
 
 ######################################################################################################################
-# Academic Paper
+# Prompts for Generating Academic Paper
 ######################################################################################################################
 
+cur_path = os.path.dirname(__file__)
+prompts_path = os.path.join(cur_path, '..\\prompts\\instructions.json')
+print(prompts_path)
+with open(prompts_path, "r") as f:
+    INSTRUCTIONS = json.load(f)
+# f = open(file_path)
 # When generating Academic Paper. Load instructions.
 # with open("../prompts/instructions.json", "r") as f:
 #     INSTRUCTIONS = json.load(f)
 #
-INSTRUCTIONS = {"introduction":
-                    "- Include five paragraph: Establishing the motivation for the research. Explaining its importance and relevance to the AI community. Clearly state the problem you're addressing, your proposed solution, and the specific research questions or objectives. Briefly mention key related works for context and explain the main differences from this work. List three novel contributions of this paper.",
-               "results":
-                    "Write the theoretical results section using LaTeX. Include theorem and corollary to support this paper (with formulas). Explain what assumptions are used and why they are standard and necessary. Do not include \section{...}. ",
-                "conclusion":
-                    "- Read the existing parts of paper and write the conclusion section.",
-                "abstract":
-                    "- Read the existing parts of paper and write the abstract."}
-
-
-INSTRUCTIONS["backgrounds"] = "- Start from one high-level paragraph to state the central problem in this field with detailed examples in industrial applications and theoretical challenges. \n" \
-                              "- Followed by two to three subsections:  Explain the foundational concepts and notations that underpin your research using as many as mathematical formulas (written in LaTeX). " \
-                              "Introduce more necessary mathematical notations, equations, or algorithms that are connected to this work. Present detailed discussions on how these concepts are applied in this paper."
-
-
-INSTRUCTIONS["related works"] = r"- Discuss three to five main related fields to this paper. " \
-                                r"For each field, select five to ten key publications from references. " \
-                                r"For each reference, analyze its strengths and weaknesses in one or two sentences. " \
-                                r"Present the related works in a logical manner, often chronologically. " \
-                                r"Consider using a taxonomy or categorization to structure the discussion. " \
-                                r"Do not use \section{...} or \subsection{...}; use \paragraph{...} to list related fields. "
-
-INSTRUCTIONS["methodology"] =  "- Provide a high-level overview of the proposed method at the beginning of this section. \n " \
-                               "- Assume you have some figures ('fig1.png', 'fig2.png', ...); they can be any figures you need (e.g. flow chart, model architecture, sample output, simulation result, or others you need). Insert figures you need with informative caption. \n" \
-                               "- Use one subsection to give a detailed formulation of the proposed method and explain how it overcomes the weakness of existing methods mentioned in this paper. " \
-                                 " If necessary, write pseudo codes wrapped by \\begin{{algorithm}} ... \\end{{algorithm}} to explain the detailed steps instead of simply listing them. \n" \
-                                "- Use one follow-up subsection to highlight the key concepts in the proposed method. " \
-                                "  Elaborate the novelty of these key concepts using formulas and inserting appropriate figures. \n" \
-                                "- Ensure the name of each subsection to be specific. \n"
-
-INSTRUCTIONS["experiments"] =  "- Provide a high-level overview at the beginning of this section.\n " \
-                               "- If necessary, include a table to compare with other methods and bold our method.\n" \
-                               "- Assume you have some figures ('exp1.png', 'exp2.png', ...); they can be any figures you need (e.g. loss curves, comparison with other methods, visualization, or others you need). Insert figures you need with informative caption. \n" \
-                               "- If necessary, use different subsections to distinguish different experimental setup."
+# INSTRUCTIONS = {"introduction":
+#                     "- Include five paragraph: Establishing the motivation for the research. Explaining its importance and relevance to the AI community. Clearly state the problem you're addressing, your proposed solution, and the specific research questions or objectives. Briefly mention key related works for context and explain the main differences from this work. List three novel contributions of this paper.",
+#                "results":
+#                     "Write the theoretical results section using LaTeX. Include theorem and corollary to support this paper (with formulas). Explain what assumptions are used and why they are standard and necessary. Do not include \section{...}. ",
+#                 "conclusion":
+#                     "- Read the existing parts of paper and write the conclusion section.",
+#                 "abstract":
+#                     "- Read the existing parts of paper and write the abstract."}
+#
+#
+# INSTRUCTIONS["backgrounds"] = "- Start from one high-level paragraph to state the central problem in this field with detailed examples in industrial applications and theoretical challenges. \n" \
+#                               "- Followed by two to three subsections:  Explain the foundational concepts and notations that underpin your research using as many as mathematical formulas (written in LaTeX). " \
+#                               "Introduce more necessary mathematical notations, equations, or algorithms that are connected to this work. Present detailed discussions on how these concepts are applied in this paper."
+#
+#
+# INSTRUCTIONS["related works"] = r"- Discuss three to five main related fields to this paper. " \
+#                                 r"For each field, select five to ten key publications from references. " \
+#                                 r"For each reference, analyze its strengths and weaknesses in one or two sentences. " \
+#                                 r"Present the related works in a logical manner, often chronologically. " \
+#                                 r"Consider using a taxonomy or categorization to structure the discussion. " \
+#                                 r"Do not use \section{...} or \subsection{...}; use \paragraph{...} to list related fields. "
+#
+# INSTRUCTIONS["methodology"] =  "- Provide a high-level overview of the proposed method at the beginning of this section. \n " \
+#                                "- Assume you have some figures ('fig1.png', 'fig2.png', ...); they can be any figures you need (e.g. flow chart, model architecture, sample output, simulation result, or others you need). Insert figures you need with informative caption. \n" \
+#                                "- Use one subsection to give a detailed formulation of the proposed method and explain how it overcomes the weakness of existing methods mentioned in this paper. " \
+#                                  " If necessary, write pseudo codes wrapped by \\begin{{algorithm}} ... \\end{{algorithm}} to explain the detailed steps instead of simply listing them. \n" \
+#                                 "- Use one follow-up subsection to highlight the key concepts in the proposed method. " \
+#                                 "  Elaborate the novelty of these key concepts using formulas and inserting appropriate figures. \n" \
+#                                 "- Ensure the name of each subsection to be specific. \n"
+#
+# INSTRUCTIONS["experiments"] =  "- Provide a high-level overview at the beginning of this section.\n " \
+#                                "- If necessary, include a table to compare with other methods and bold our method.\n" \
+#                                "- Assume you have some figures ('exp1.png', 'exp2.png', ...); they can be any figures you need (e.g. loss curves, comparison with other methods, visualization, or others you need). Insert figures you need with informative caption. \n" \
+#                                "- If necessary, use different subsections to distinguish different experimental setup."
 
 
 def generate_paper_prompts(paper_info, section):
@@ -220,9 +203,9 @@ def generate_bg_summary_prompts(paper_info, section):
     return prompts
 
 if __name__ == "__main__":
-    # import json
-    # with open("../prompts/instructions.json", "w") as f:
-    #     json.dump(INSTRUCTIONS, f)
+    import json
+    with open("../prompts/instructions.json", "w") as f:
+        json.dump(INSTRUCTIONS, f)
     import json
     with open("../prompts/instructions.json", "r") as f:
         ins = json.load(f)