40 Resources to Completely Master Markov Decision Processes
Tags: machinelearning, ai, math
Published: 2019-07-01 12:09:43

<!-- wp:paragraph -->
<p>The Markov Decision Process is the formal description of the Reinforcement Learning problem. It includes concepts like states, actions, rewards, and how an agent makes decisions based on a given policy. So, what Reinforcement Learning algorithms do is to find optimal solutions to Markov Decision Processes.</p>
<!-- /wp:paragraph -->

<!-- wp:more -->
<!--more-->
<!-- /wp:more -->

<!-- wp:image {"align":"center","id":228,"width":374,"height":299} -->
<div class="wp-block-image"><figure class="aligncenter is-resized"><img src="https://launchyourintelligentapphome.files.wordpress.com/2019/06/1_quboz2yq5fy6ynzyvspxzw.png?w=800" alt="" class="wp-image-228" width="374" height="299"/><figcaption>Markov Decision Process</figcaption></figure></div>
<!-- /wp:image -->

<!-- wp:paragraph -->
<p>Because it is a fundamental concept in the Reinforcement Learning domain, we selected more than 40 resources about Markov Decision Process, including blog posts, books, and videos. Check the links below.</p>
<!-- /wp:paragraph -->

<!-- wp:heading -->
<h2>Blog Posts</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><a rel="noreferrer noopener" aria-label="Reinforcement Learning Demystified: Markov Decision Processes (Part&nbsp;1) (opens in a new tab)" href="https://towardsdatascience.com/reinforcement-learning-demystified-markov-decision-processes-part-1-bf00dda41690" target="_blank">Reinforcement Learning Demystified: Markov Decision Processes (Part&nbsp;1)</a></li><li><a rel="noreferrer noopener" aria-label="Reinforcement Learning Demystified: Markov Decision Processes (Part&nbsp;2) (opens in a new tab)" href="https://towardsdatascience.com/reinforcement-learning-demystified-markov-decision-processes-part-2-b209e8617c5a" target="_blank">Reinforcement Learning Demystified: Markov Decision Processes (Part&nbsp;2)</a></li><li><a rel="noreferrer noopener" aria-label="RL part 3. Markov Decision Process, policy, Bellman Optimality&nbsp;Equation. (opens in a new tab)" href="https://programmingbee.net/2019/01/01/rl-part-3-markov-decision-process-policy-bellman-optimality-equation/" target="_blank">RL part 3. Markov Decision Process, policy, Bellman Optimality&nbsp;Equation.</a></li><li><a rel="noreferrer noopener" aria-label=" AN INTRODUCTION TO REINFORCEMENT LEARNING – I :: MARKOV DECISION&nbsp;PROCESSES (opens in a new tab)" href="https://insufficientinformation.wordpress.com/2019/04/20/an-introduction-to-reinforcement-learning-i-markov-decision-processes/" target="_blank">AN INTRODUCTION TO REINFORCEMENT LEARNING – I :: MARKOV DECISION&nbsp;PROCESSES</a></li><li><a href="https://appliedprobability.blog/2019/01/26/markov-decision-processes-3/" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes, by Applied Probability Notes (opens in a new tab)">Markov Decision Processes, by Applied Probability Notes</a></li><li><a href="https://ambareeshr.wordpress.com/2019/01/07/markov-decision-process-for-tic-tac-toe/">Markov Decision Process for Tic Tac&nbsp;Toe</a></li><li><a href="https://towardsdatascience.com/self-learning-ai-agents-part-i-markov-decision-processes-baf6b8fc4c5f" target="_blank" rel="noreferrer noopener" aria-label="Self Learning AI-Agents Part I: Markov Decision Processes (opens in a new tab)">Self Learning AI-Agents Part I: Markov Decision Processes</a></li><li><a href="https://towardsdatascience.com/getting-started-with-markov-decision-processes-reinforcement-learning-ada7b4572ffb" target="_blank" rel="noreferrer noopener" aria-label="Getting Started with Markov Decision Processes: Reinforcement Learning (opens in a new tab)">Getting Started with Markov Decision Processes: Reinforcement Learning</a></li><li><a href="https://towardsdatascience.com/understanding-markov-decision-processes-b5862c192ddb" target="_blank" rel="noreferrer noopener" aria-label="Understanding Markov Decision Processes (opens in a new tab)">Understanding Markov Decision Processes</a></li><li><a href="https://medium.com/@sanchittanwar75/markov-chains-and-markov-decision-process-e91cda7fa8f2" target="_blank" rel="noreferrer noopener" aria-label="Markov chains and Markov Decision&nbsp;process (opens in a new tab)">Markov chains and Markov Decision&nbsp;process</a></li><li><a href="https://towardsdatascience.com/not-so-deep-reinforcement-learning-for-dummies-part-2-854216d1fe0d" target="_blank" rel="noreferrer noopener" aria-label="Not-So-Deep Reinforcement Learning for dummies — Part&nbsp;2 (opens in a new tab)">Not-So-Deep Reinforcement Learning for dummies — Part&nbsp;2</a></li><li><a href="https://leonardoaraujosantos.gitbooks.io/artificial-inteligence/content/markov_decision_process.html" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision process (opens in a new tab)">Markov Decision process</a></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Books</h2>
<!-- /wp:heading -->

<!-- wp:list -->
<ul><li><a href="https://onlinelibrary.wiley.com/doi/book/10.1002/9780470316887" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes: Discrete Stochastic Dynamic Programming (opens in a new tab)">Markov Decision Processes: Discrete Stochastic Dynamic Programming</a></li><li><a href="https://www.springer.com/gp/book/9783642025464" target="_blank" rel="noreferrer noopener" aria-label="Continuous-Time Markov Decision Processes (opens in a new tab)">Continuous-Time Markov Decision Processes</a></li><li><a href="https://www.morganclaypool.com/doi/abs/10.2200/S00426ED1V01Y201206AIM017" target="_blank" rel="noreferrer noopener" aria-label="Planning with Markov Decision Processes: An AI Perspective (opens in a new tab)">Planning with Markov Decision Processes: An AI Perspective</a></li><li><a href="https://www.springer.com/gp/book/9783319477640" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes in Practice (opens in a new tab)">Markov Decision Processes in Practice</a></li><li><a href="https://www.crcpress.com/Constrained-Markov-Decision-Processes/Altman/p/book/9780849303821" target="_blank" rel="noreferrer noopener" aria-label="Constrained Markov Decision Processes (opens in a new tab)">Constrained Markov Decision Processes</a></li><li><a href="http://incompleteideas.net/book/RLbook2018.pdf" target="_blank" rel="noreferrer noopener" aria-label="Reinforcement Learning, an Introduction (opens in a new tab)">Reinforcement Learning, an Introduction</a></li></ul>
<!-- /wp:list -->

<!-- wp:heading -->
<h2>Videos</h2>
<!-- /wp:heading -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=my207WNoeyA","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=my207WNoeyA
</div><figcaption><a href="https://www.youtube.com/watch?v=my207WNoeyA" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes (MDPs) - Structuring a Reinforcement Learning Problem (opens in a new tab)">Markov Decision Processes (MDPs) - Structuring a Reinforcement Learning Problem</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=Jk2V9yA82YU","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=Jk2V9yA82YU
</div><figcaption><a href="https://www.youtube.com/watch?v=Jk2V9yA82YU" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes - Georgia Tech - Machine Learning (opens in a new tab)">Markov Decision Processes - Georgia Tech - Machine Learning</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=BxIG76-C37k","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=BxIG76-C37k
</div><figcaption><a href="https://www.youtube.com/watch?v=BxIG76-C37k" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes Two - Georgia Tech - Machine Learning (opens in a new tab)">Markov Decision Processes Two - Georgia Tech - Machine Learning</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=Jw3ZnWFjDfM","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=Jw3ZnWFjDfM
</div><figcaption><a href="https://www.youtube.com/watch?v=Jw3ZnWFjDfM" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes Three - Georgia Tech - Machine Learning (opens in a new tab)">Markov Decision Processes Three - Georgia Tech - Machine Learning</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=dkBZ9YKuOVA","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=dkBZ9YKuOVA
</div><figcaption><a href="https://www.youtube.com/watch?v=dkBZ9YKuOVA">Markov Decision Processes Four - Georgia Tech - Machine Learning</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=lfHX2hHRMVQ","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=lfHX2hHRMVQ
</div><figcaption><a href="https://www.youtube.com/watch?v=lfHX2hHRMVQ" target="_blank" rel="noreferrer noopener" aria-label="RL Course by David Silver - Lecture 2: Markov Decision Process (opens in a new tab)">RL Course by David Silver - Lecture 2: Markov Decision Process</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=hMbxmRyDw5M","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=hMbxmRyDw5M
</div><figcaption><a href="https://www.youtube.com/watch?v=hMbxmRyDw5M" target="_blank" rel="noreferrer noopener" aria-label="Reinforcement Learning 3: Markov Decision Processes and Dynamic Programming (opens in a new tab)">Reinforcement Learning 3: Markov Decision Processes and Dynamic Programming</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=Oxqwwnm_x0s","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=Oxqwwnm_x0s
</div><figcaption><a href="https://www.youtube.com/watch?v=Oxqwwnm_x0s" target="_blank" rel="noreferrer noopener" aria-label="Lecture 8 MDPs (opens in a new tab)">Lecture 8 MDPs</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=Csiiv6WGzKM","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=Csiiv6WGzKM
</div><figcaption><a href="https://www.youtube.com/watch?v=Csiiv6WGzKM" target="_blank" rel="noreferrer noopener" aria-label="Lecture 9: Markov Decision Process II (opens in a new tab)">Lecture 9: Markov Decision Process II</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=mNGVkKeMUtc","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=mNGVkKeMUtc
</div><figcaption><a href="https://www.youtube.com/watch?v=mNGVkKeMUtc" target="_blank" rel="noreferrer noopener" aria-label="Markov Rewards and Dynamic Programming (opens in a new tab)">Markov Rewards and Dynamic Programming</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=IkbkEtOOC1Y","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=IkbkEtOOC1Y
</div><figcaption><a href="https://youtu.be/IkbkEtOOC1Y" target="_blank" rel="noreferrer noopener" aria-label="Markov Chains I (opens in a new tab)">Markov Chains I</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=ZulMqrvP-Pk","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=ZulMqrvP-Pk
</div><figcaption><a href="https://www.youtube.com/watch?v=ZulMqrvP-Pk" target="_blank" rel="noreferrer noopener" aria-label="Markov Chains II (opens in a new tab)">Markov Chains II</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=HIMxdWDLEK8","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=HIMxdWDLEK8
</div><figcaption><a href="https://www.youtube.com/watch?v=HIMxdWDLEK8" target="_blank" rel="noreferrer noopener" aria-label="Markov Chains III (opens in a new tab)">Markov Chains III</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=1GoxbgSQwfM","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=1GoxbgSQwfM
</div><figcaption><a href="https://www.youtube.com/watch?v=1GoxbgSQwfM" target="_blank" rel="noreferrer noopener" aria-label="Reinforcement Learning and Markov Decision Processes I (opens in a new tab)">Reinforcement Learning and Markov Decision Processes I</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=g2GnL8MVHHc","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=g2GnL8MVHHc
</div><figcaption><a href="https://www.youtube.com/watch?v=g2GnL8MVHHc" target="_blank" rel="noreferrer noopener" aria-label="Reinforcement Learning and Markov Decision Processes II (opens in a new tab)">Reinforcement Learning and Markov Decision Processes II</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=KovN7WKI9Y0","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=KovN7WKI9Y0
</div><figcaption><a href="https://youtu.be/KovN7WKI9Y0" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Processes (opens in a new tab)">Markov Decision Processes</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=DgRqxKt2DYE","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=DgRqxKt2DYE
</div><figcaption><a href="https://youtu.be/DgRqxKt2DYE" target="_blank" rel="noreferrer noopener" aria-label="Markov Decision Process (MDP) Tutorial (opens in a new tab)">Markov Decision Process (MDP) Tutorial</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=HmcyDQZbEFs","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=HmcyDQZbEFs
</div><figcaption><a rel="noreferrer noopener" target="_blank" href="https://www.youtube.com/watch?v=HmcyDQZbEFs">introduction to Markov Decision Processes (MFD)</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=jpmZp3eX-wI","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=jpmZp3eX-wI
</div><figcaption><a rel="noreferrer noopener" target="_blank" href="https://www.youtube.com/watch?v=jpmZp3eX-wI">An Introduction to Markov Decision Processes and Reinforcement Learning</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=SWYceQpEVKM","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=SWYceQpEVKM
</div><figcaption><a rel="noreferrer noopener" target="_blank" href="https://www.youtube.com/watch?v=SWYceQpEVKM">CS885 Lecture 2a: Markov Decision Processes</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=qkdB37Wq6tw","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=qkdB37Wq6tw
</div><figcaption><a rel="noreferrer noopener" target="_blank" href="https://www.youtube.com/watch?v=qkdB37Wq6tw">Grad Course in AI (#11): Markov Decision Processes</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->

<!-- wp:core-embed/youtube {"url":"https://www.youtube.com/watch?v=7-HMYuMroak","type":"rich","providerNameSlug":"","className":"wp-embed-aspect-16-9 wp-has-aspect-ratio"} -->
<figure class="wp-block-embed-youtube wp-block-embed is-type-rich wp-embed-aspect-16-9 wp-has-aspect-ratio"><div class="wp-block-embed__wrapper">
https://www.youtube.com/watch?v=7-HMYuMroak
</div><figcaption><a rel="noreferrer noopener" target="_blank" href="https://www.youtube.com/watch?v=7-HMYuMroak">Marcus Hutter - Markov Decision Processes</a></figcaption></figure>
<!-- /wp:core-embed/youtube -->