Hi, Tino. Thanks for the question. I think RL will continue to have an impact on several industries in the next few years. I’m not too familiar with finance, but I know there are lots of R&D investments that involve RL (and AI in general). I’m more familiar with the military industry, and my guess is other industries see AI/ML/RL as critical technologies of the future, along with 5G and Quantum.

Hi, Wendy Mak. Thank you for the questions. I think, in general, supervised learning is commonly used to make better decisions. That’s true for classification and regression. Are you working on models that classify traffic signs? Are you working on models that predict stock prices? Usually, the real need behind those is better decision making–self-driving cars, automatic trading, etc.
The question is whether RL is ready to take over the decisions of those models. And, the answer to this is, it depends. It depends on whether you can create a simulation for the AIs to learn, it depends on the number of people working on the project, and whether you’ll be able to work on techniques such as transfer learning, it depends.
The way I see it, many ML problems can be transformed into RL problems to let the decision model learn features used for decision making directly, as opposed to having the model learn features for prediction of a proxy objective, which will be the classification/regression objective in a decision-making problem, which may bias the model.

Kind of follow up:
Are there tools for “RL production ready” at this moment? to enable this trasformation from ML to RL?

Hi, Grzegorz Sajko. Thanks for the question. I think RLlib is a pretty good starting point. But, I don’t think there are any tools to convert an ML problem into RL smoothly.

Yeah, I think using Grid search first to find approximately good hyperparameters and then Random search to fine-tune the best set is an excellent practical strategy for tuning RL algorithms. Also, there are more advanced methods and can be helpful: population-based training is a good one, for instance.

Hi, Saulius Lukauskas. Thanks for the question. Funny one! Yes, unfortunately, RL still looks like this if you think of training from scratch, but there are several strategies for reducing sample complexity, too. It all begins with the type of RL methods that you select. For instance, a technique called “deep neuroevolution,” related to genetic algorithms, has very high sample complexity. While those techniques are okay to use, if you have a simulation and lots of computing, I wouldn’t train it in the real world. On the other side of the spectrum of sample complexity, we have the family of model-based RL methods. These methods are commonly used to train robots in the real world. These methods need fewer samples to come up with a solution. But of course, everything has pros and cons, and with these methods, wall-clock time may be affected.
But, then, you get into techniques such as transfer learning, multi-task learning, and even a more recent, causal RL. I think researchers are aware of the problem the xkcd illustrates and have been prioritizing it.

Hi, David Cox. Thanks for the question! This question is interesting and related to behavioral cloning, inverse RL, and imitation learning. The critical distinction is that many of these methods assume optimal behavior, which may be a limitation. However, that’s not the case for other algorithms, such as DAgger (Dataset Aggregation), in which the goal is the clone the behavior of another agent, whether it is optimal or not. Overall, behavioral cloning is related to RL in some contexts, but more so to supervised learning. You are trying to predict the action given a data set of behavior. Yet, it is not supervised learning directly because the data is not i.i.d. (independent and identically distributed), so many of the assumptions of the optimization algorithms no longer hold, and we need different methods to solve the problem. The field you are looking for, though, is imitation learning. I would start there!

Thanks, Miguel Morales! This is extremely helpful!! Any offhand recommendations on researchers or authors who are particularly good in this area?

Yeah, I like the work of Sergey Levine, Chelsea Finn, Pieter Abbeel, and Andrew Ng. Of course, they do a lot more than just imitation learning. But they have also done lots of good work in imitation learning and inverse RL.

Great! Thank you!!

Hi, Matthew Emerick. Thank you for the questions. I think a background in supervised learning methods is sufficient. Have you trained a couple of NN just following the TensorFlow, or PyTorch tutorial? Then, that’s sufficient. If not, it should take an hour or so to review. After that, I’m not assuming any RL background. So everything moves from the perspective of RL, then to DeepRL.

Hi, Vladimir Finkelshtein. Thanks for the question. Hard to answer because you use the word “intuitively,” and that can make it subjective. If you understand what RL does, it becomes intuitive to use it and what to expect. Surprisingly, some folks don’t think so. For instance, when RL methods mine rewards in a loop, instead of playing a game the way a human plays. Why would the human expect the agent to learn to play like a human if the objective is not that?! But, interestingly, many folks argue that’s a problem with RL.
Now, to answer something related to your question (removing the word “intuitively”), RL should work better at learning a hierarchy of policies. So, that one policy learns to fly an airplane, another learns the tactics of a dogfight, another learns to command multiple platforms, another one an entire campaign. RL should be better at understanding the causal relationships in data; although that is a problem with machine learning overall, RL methods would benefit from it. Also, the fact that an agent trained to perform at a superhuman level in Breakout (Atari) doesn’t do well at all in Pong (Atari) is a clear issue. You would have to train a new model from scratch with Pong. So, multi-task learning is another area where RL methods should be better.

Oh, I have too many, but to focus, I think Berkeley’s course is great. http://rail.eecs.berkeley.edu/deeprlcourse/

I think Grokking Deep RL is the best first book, best I’ve found. It does a good job at introducing the notation common in reinforcement learning. http://incompleteideas.net/book/the-book.html is a famous book in the RL field, and it’s much less intimidating after having read Miguel’s book, so perhaps that is a good second book on the subject, or at least skim through it.

I think Causal RL is something to keep an eye on. Also, the use of RL with other methods, such as Unsupervised Learning methods and auxiliary tasks. Meta-learning is another area with a lot of potential, though I see that more long-term, but we’ll see!

Hi, Bayram Kapti. Thanks for the question. I think the main different is the sequential nature of RL. Thompson Sampling is commonly use for selecting actions under an environment with a single state (often called multi-armed bandits). That’s a single state and multiple actions (arms). RL and DRL is about learning in environments with multiple, even infinite states. However, some of the same action-selection techniques of Thompson Sampling can be ported and use in a RL context. So, I would say it is not this or that, it could be this and that.

Thank you for the answer Miguel Morales !
Does your book cover both or does it focuses on DRL?

Of course! It covers Thompson Sampling on a small section of the chapter related to multi-armed bandits and the exploration vs. exploitation tradeoff. But the book covers many other things related to DeepRL, and it doesn’t go too deep on Thompson Sampling.

Got it thanks!

Hi, Glenn. Thanks! And, thanks for the question. Yes. It wasn’t that difficult. I’ve been teaching a reinforcement learning course at Georgia Tech for a few years now, and some of these are ideas that I think would help folks ‘grok’ deep RL. Lots of the students I’ve had over the years have had related questions, so as I was reviewing a chapter, I would write the exercises. Some of the exercises are related to “gotchas” in the practical sense (when implementing an algorithm). I think they are not as important, given algorithms come and go, and frameworks change almost daily, but exciting things may help folks get a complete picture.

I’ve worked with that LM team, and I’ve also worked with the winning team (Heron Systems). However, I did not participate in that release of the project. My involvement began after those results were published.

That’s so cool. I used to fly with USAF. That was so interesting for me. A little bit sad though. I wanted the human beat the AI :) a little biased :)

Oh cool! Haha, well have in mind the AI had perfect knowledge of the environment. I think without that the results would have been different. The human only has a “perspective,” and it needs to look around and find the bandit, so it was a bit unfair. But, there is work being done to improve those results.

Haha, That’s absolutely true :) Overall, so much fun for me, both aviation and AI fan!

Hey, DevJac. Thanks for the question–a fascinating observation you make!
I’m not sure 100%, but I think it has to do with on-policy vs. off-policy learning, and of course, the implementation. In on-policy learning, the critic evaluates the policy, assuming that policy will continue to run in the environment. The policy generating the data and the policy being trained are the same. In off-policy learning, the critic is evaluating the policy, assuming optimal behavior after that. And, because we don’t have an optimal policy, those are different policies. So, you can train off-policy actor-critic methods the other way (first the critic, then the policy).
Papers, in general, follow this pattern. If you see A3C, PPO, PPG, they all have policy phases first, then critic phases. If you check DDPG, TD3, SAC, they all train the critic first.
Again, I’m not 100% certain, but that is my first guess. Excellent observation, BTW! Let me know if you find more info.

Hi, Rona Ainslie. Thank you for the question. There is a field called Explainable Reinforcement Learning that is focused on making Reinforcement Learning more explainable. What that means is we (all humans: users, developers, researchers, and stakeholders) need to be able to understand the decision made by an algorithm. We also need to trust the decision-making system, and in RL (and ML/AI in general), currently, a slight change of a training seed may yield significant discrepancies in training results. We’re far, and there’s lots of work to be done here.
To answer your question more directly, RL also needs explainability. The main difference when you hear “Explainable AI” vs. “Explainable RL” is the former subsumes the latter. In “Explainable AI,” as introduced by DARPA, there are two tracks, a “Data Analytics” (Supervised Learning for the most part) and an “Autonomy” (Reinforcement Learning for the most part). Folks who focus on that second track, the Autonomy track, are studying “Explainable RL.”
At the moment, there are ways to understand how decisions are made, but I would argue that those are not yet user-friendly. I think lots of researchers continue to work on their narrow RL problems, hoping that researchers working in Explainable RL can come up with solutions to those problems. Later in the future, we can merge paths to deploy explainable solutions. I think having explainable models is a blocking issue before we can release AI to several industries, including mine, of course.

Thank you Miguel Morales Lots of interesting points (and I’m off to google DARPA explainable AI)! Do you think that explainable RL is a field that is being taken seriously? That is, in your experience is there enough money going in to the research yet?

It obviously depends on the field, but if you think about it, DARPA, being a government institution, cares about explainability. Why is explainability important to the government and the military? Are those reasons likely to continue there?
I think money is going to continue to flow into it for sure.

Thank you!

Hey, John Savage. This is an excellent question. I think the same idea applies, but a bit differently. In RL, there is an environment and an agent. Usually, the environment is given, and researchers concentrate on the agent. However, in an engineering sense, you likely don’t care about Atari games. You care about your problem. So, one of the main challenges in RL is finding an environment to train your agent, or better yet, creating one yourself. What simulation engine are you using? Will you use the real world for training? Will the agent be able to collect enough data? Will you use a sample efficient algorithm for training? Etc.
My recommendation here is to spend time implementing an environment, use off-the-shelf algorithms first, and make your environment look like the environment those algorithms are tune for solving. Once you iterate a few times over your environment, then start changing the agents, models, and training schemes, then go back to the environment and polish once again. Lots of folks want to solve everything at once and trust me, RL is challenging. So, isolating the problem is the way to go.

That’s a great insight, thanks a lot, makes a bunch of sense.

Hi, Tim Becker. Thanks for the questions!

• Yeah. The LunarLander environment is a bit more complex (8 variables for observation, 4 for action). You also have a Lunar Lander continuous with continuous variables for the action space. I recommend staying away from image-based environments until you have solved a couple of other environments with the same implementation. The problem with image-based environments is the feature extraction process can take quite a bit of time without the proper equipment (and even with the right equipment, it takes much longer than non-image-based environments.)
  
• Unfortunately, this is a challenging problem, usually left out because it is not as “researchy” but more of an engineering problem. However, for most folks, this is really where the money goes. You may not be able to invent a new RL algorithm, and maybe you’re not even interested in that. Instead, you may want to use one of the available algorithms and train them to solve your problem. Modeling problems (as you say: “assigning useful rewards”) is essential, and someone should spend some time creating a book or some content that explains how to do that. Sadly, I don’t have any recommendations at the moment, only to study how MDPs work (chapter 2) and try to replicate, add complexity. Feel free to explore how others do it. Look for gym <search term> environment and dig into codebases (e.g. atari-py, mujuco-py, hfo-py, etc.)
  
• To compare policies in practice, you would evaluate them in environments under similar conditions with several random seeds. Capture the mean total return (sum of rewards from the initial state to the end of the episode, averaged x times, over n seeds), and go from there. You can monitor the accuracy of a value function by comparing the estimates to the actual returns. But on a policy-to-policy comparison, I commonly use the sum of rewards.
  
• This is sooo challenging. What I always try to do is to simplify the problem. As opposed to debugging a complex system, simplify the solution. Train against a simpler environment. Use standard hyperparameters for the type of environment that you’re using. RL is very challenging to debug because a single typo can break things. And, what is probably worse in my opining, an implementation with a bug can “work” under certain conditions. That’s cruel, let me tell ya! :)
  
• I somewhat agree. Many folks attribute their incompetence to RL and use the phrase: “RL is hard” to excuse themselves. Yes, I believe RL is challenging, but more often than not, it is not working due to user error. When you look at implementations available online many out there are flat-out wrong (including in books, BTW–sadly). But, if you start small and build up, then it is not that bad. You need tenacity and focus, but you’ll have so much fun! Hyperparameter tuning is not that bad once you learn what parameters work for certain kinds of environments. I recommend using a hyperparameter tuning framework and trying a random search. RLlib and tune are excellent starting points.
  Thanks for the great questions. I didn’t do justice to them, but it hopefully helps!

Thank you Miguel! This helps a lot! As you expected, I was looking at image-based environments that gave me headaches. The LunarLander sound like fun 🙂 I will give it a try.