CORL: Research-oriented Deep Offline Reinforcement Learning Library
ai
Alexander Nikulin
Tinkoff a.p.nikulin@tinkoff.ai
Dmitry Akimov
Tinkoff d.akimov@tinkoff.ai
Vladislav Kurenkov
Tinkoff v kurenkov ai
Sergey Kolesnikov
Tinkoff s.s.kolesnikov@tinkoff.ai
Abstract
CORL 1 is an open-source library that provides single-file implementations of Deep
Offline Reinforcement Learning algorithms.
It emphasizes a simple developing experience with a straightforward codebase and a modern analysis tracking tool.
In CORL, we isolate methods implementation into distinct single files, making performance-relevant details easier to recognise.
Additionally, an experiment tracking feature is available to help log metrics, hyperparameters, dependencies, and more to the cloud.
Finally, we have ensured the reliability of the implementations by benchmarking a commonly employed D4RL benchmark.
Introduction
Deep Offline Reinforcement Learning (Deep ORL)
[24] has been showing significant advancements in numerous domains such as robotics [30, 21], autonomous driving [6] and recommender systems [4].
Due to such rapid development, many open-source ORL solutions 2 emerged to help RL practitioners understand and improve well-known ORL techniques in different fields.
On the one hand, they introduce ORL algorithms standard interfaces and user-friendly APIs, simplifying ORL methods incorporation into existing projects.
On the other hand, introduced abstractions may hinder the learning curve for newcomers and the ease of adoption for researchers interested in developing new algorithms.
One needs to understand the modularity design (several files on average), which (1) can be comprised of thousands of lines of code or (2) can hardly fit for a novel method 3 .
In this technical report, we take a different perspective on an ORL library.
We propose CORL (Clean Offline Reinforcement Learning) – minimalistic and isolated single-file implementations of deep ORL algorithms, that are backed up by open-sourced D4RL benchmark results.
The unadorned design allows practitioners to read and understand the implementations of the algorithms straightforwardly.
Moreover, CORL supports optional integration with experiments tracking tool such as Weighs&Biases 4 .
This provides practitioners with a convenient way to analyze the results and behavior of all algorithms, not merely relying on a final performance commonly reported in papers.
We hope that CORL library will help ORL newcomers to study implemented algorithms and aid the researchers in quickly modifying existing methods without a need to fight through different levels of abstraction.
Finally, the obtained results may serve as a point of reference for D4RL benchmarks without a need to re-implement existing algorithms and tune hyperparameters
[Table]
=(cid:79)(cid:82)g(cid:86)/d(cid:87)- h(cid:82)(cid:83)(cid:83)e(cid:85) -- (cid:81)(cid:88)(cid:80)- e(cid:83)(cid:82)ch(cid:86)=
Wa(cid:81)db (cid:79)(cid:82)g(cid:86)
Wa(cid:81)db (cid:85)e(cid:83)(cid:82)(cid:85)(cid:87)(cid:86)
Related Work
Since the Atari breakthrough [28], numerous open-source RL frameworks and libraries have been developed over the last years: [5, 15, 2, 13, 17, 12, 7, 19, 11, 25, 11, 26, 16, 33, 31], focusing on different perspectives of the RL.
For example, stable-baselines ([15]) provides many deep RL implementations that carefully reproduce results to back up RL practitioners with reliable baselines during methods comparison.
On the other hand, Ray ([25]) is focusing on implementations scalability and production-friendly usage.
Finally, more nuanced solutions exist, such as Dopamine ([2]), which emphasizes different DQN variants, or ReAgent ([13]), which applies RL for the RecSys domain.
At the same time, the ORL branch, which we are interested in this paper, is not yet covered as much: the only library that precisely focus on offline RL setting is d3rlpy [32].
While CORL do also focus on ORL methods, similar to d3rlpy, it takes a different perspective on library design and provides non-modular independent algorithms implementations.
More precisely, CORL does not introduce additional abstractions to make ORL more general but instead gives an "easy-to-hack" starter kit for research needs.
Although CORL does not represent a first non-modular RL library, which is more likely the CleanRL [16] case, it has two significant differences with its predecessor.
First, CORL is focused on offline RL, while CleanRL implements online RL algorithms.
Second, CORL intent to minimize the complexity of the requirements and external dependencies.
To be more concrete, CORL does not have additional requirements with useful abstractions such as stable - baselines or envpool but instead implements everything from scratch in the codebase.
[Table]
CORL Design Single-File Implementations
It is known that implementation subtleties significantly impact agent performance in deep RL [14, 8, 10].
Unfortunately, user-friendly abstractions and general interfaces, the core idea behind modular libraries, encapsulate and often hide these important nuances from the practitioners.
For such a reason, CORL unwraps these details by adopting single-file implementations.
To be more concrete
[Table]
we put environment details, algorithms hyperparameters
[Table]
and evaluation parameters into a single file 5 .
For example, we have a
• any _ percent _ bc.py (399 LOC 6 ) as a baseline algorithm for ORL methods comparison, • td 3 _ bc.py (507 LOC) as a competitive minimalistic ORL algorithm [10], • dt.py
(542 LOC) as an example of the recently proposed trajectory optimization approach [3]
Figure 1 depicts an overall library design.
While such design produces code duplications among implementations, it has several essential benefits from the both educational and research perspective:
• Smooth learning curve .
Having the entire code in one place makes understanding all its aspects more straightforward.
In other words, one may find it easier to dive into 512 LOC of single-file Decision Transformer implementation rather than 10+ files of the original implementation.
• Simple prototyping .
As we are not interested in code general applicability, we could make it implementation-specific.
Such a design also removes the need for inheritance from general primitives or their refactoring, reducing abstraction overhead to zero.
At the same time, this idea gives us complete freedom during code modification.
• Faster debugging .
Without additional abstractions, implementation simplifies to a single for-loop with a global python name scope.
Furthermore, such flat architecture makes it easier to access and inspect any created variable during the training process, which is crucial during modification and debugging.
Configuration files
Although it is a typical pattern to use a command line interface (CLI) for single-file experiments in the research community, CORL slightly improves it with predefined configuration files.
Utilizing YAML parsing through CLI, for each experiment, we gather all environment and algorithm hyperparameters
Lines Of Code
[Table]
into such files so that you could use them as an initial setup.
We found that such setup (1) simplifies experiments, eliminating the need to keep all algorithm-environment-specific parameters in mind, and (2) keeps it convenient with the familiar CLI approach.
Experiment Tracking
ORL evaluation is another challenging aspect of the current ORL state [23].
To face this uncertainty,
CORL supports integration with Wandb, a modern experiment tracking tool.
With each experiment,
CORL automatically saves: (1) source code, (2) dependencies (requirements.txt), (3) hardware setup, (4) OS environment variables, (5) hyperparameters, (6) training and system metrics, (7) logs (stdout, stderr).
See Appendix B for an example.
Although, Wandb is a proprietary solution, other alternatives such as Tensorboard or Aim could be used within a few lines of code change.
It is also important to note that with Wandb tracking, one could straightforwardly use CORL with Wandb hyperparameter tuning or public reports.
We found full metrics tracking during the training process necessary for two reasons.
First, it removes the possible bias of final or best performance commonly reported in papers.
For example, one could evaluate ORL performance as max archived score, while another uses the average performance over
N (last) evaluations [32].
Second, it opens an opportunity for advanced performance analysis such as
EOP [23].
In short, provided with all metrics logs, one can utilize any performance statistics, not merely relying on commonly used alternatives.
Benchmarks
In our library we implemented the following algorithms: N % 7 Behavioral Cloning (BC), TD3 +
BC [10], CQL [22], IQL [20], AWAC [29], SAC-N, EDAC [1], and Decision Transformer (DT) [3].
We evaluated every algorithm on the D4RL benchmark [9], focusing on Locomotion, Maze2D, and
AntMaze tasks.
Each algorithm was run for 1 million gradient steps 8 and evaluated every 5000steps using 10 and 100 episodes for locomotion and maze tasks, respectively.
For our experiments, we used hyperparameters proposed in the original works (see Appendix C for details).
The final performance results are reported in Tables 1 and 2.
The scores are normalized to the range between 0 and 100 [9].
Following recent works [32] we report both last (Table 1) and best (Table 2) obtained scores to illustrate potential performance and overfitting properties of each algorithm.
See
Appendix A for full training performance graphs.
According to the observed results, AWAC, SAC-N, and EDAC show the most competitive performance in both the last and best evaluation settings.
At the same time, TD3 + BC performs well across all tasks without any hyperparameters tuning while other methods rely on it.
Finally, simple yet effective
BC- 10% performs well on locomotion tasks when high-quality data is available and converges there almost instantly (see Appendix A).
Conclusion
In this paper, we introduced CORL – a single-file implementation library for Offline Reinforcement
Learning with configuration files and advanced metrics tracking support.
All implemented algorithms were benchmarked on D4RL datasets, closely matching (sometimes overperforming) the original results.
Focusing on implementation clarity and reproducibility, we hope that CORL will help RL practitioners in their research and applications.
A Additional Benchmark Information
[Figure]
c h ee t a h
[Figure]
ee
[Figure]
li
[Figure]
S c
o r
[Figure]
h a l f c h ee t a hm h a l f c h ee t a hm - e h a l f c h ee t a hm - r e h
o pp e r - m h
o pp e r - m - e h o pp e r - m - r e w a l k e r
[Figure]
d - m w a l k e r 2 d - m
[Figure]
e w a l k e
[Figure]
d
[Figure]
m
[Figure]
o r m a li
z
e
d
S c
o r e
[Figure]
a li
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
r m a li
z
[Figure]
d
[Figure]
c o r
[Figure] [Figure]
[Figure]
B Wandb Tracking
[Figure]
C Experimental Details
We modify reward on AntMaze task by substructing 1 from reward as it is done in previous works.
We used original implementation of TD3 + BC 9 , SAC- N 10 and EDAC 10 and custom implementations of IQL 11 and CQL 12 as the basis for ours.
For most of the algorithms and datasets we use default hyperparameters if available.
Configuration files for every algorithm and environment are presented in our GitHub repository.
Hyperparameters are also provided in subsection C.2.
C.1 Number of update steps and evaluation rate
Following original work SAC- N and EDAC are trained for 3 million steps (except AntMaze which is trained for 1 million steps) in order to obtain state-of-the-art performance and tested everysteps.
Decision Transformer (DT) training is splitted into datasets pass epochs, we train DT forepochs on each dataset and evaluate every 5 epochs.
All other algorithms are trained for 1 million steps and evaluated every 5000 steps.
We evaluate every policy for 10 episodes on locomotion tasks and for 100 for Maze2d and AntMaze tasks.
C.2 Hyperparameters
[Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]