$$ \newcommand{\bx}{\boldsymbol{x}} \newcommand{\bt}{\boldsymbol{\theta}} \newcommand{\dkl}{\mathrm{d}_{\mathrm{KL}}} \newcommand{\dtv}{\mathrm{d}_{\mathrm{TV}}} \newcommand{\emv}{\hat{\theta}_{\mathrm{emv}}} \newcommand{\ent}{\mathrm{Ent}} $$

4  Lab work n°2

You can download Lab Work n°2: RL for stochastic control problems as a Jupyter Notebook here.¹

All the necessary informations are already included in the notebook². Below is a brief summary of the lab content and some expected results.

4.1 Summary of the lab

Content :

The Lab work is divided into 2 parts :

• The first part is devoted to the implementation of some RL algorithms for solving a market impact problem. We give an expected plot for the evolution of the sum of the rewards for the market impact problem

• The second part is devoted to retrieve the results of the course on the optimal policy for the linear quadratic control problems in continuous time, i.e. to recover the associated Ricatti equations and the associated optimal policy.

4.2 Towards the open ended mini-project

The project will be fairly open-ended, allowing you to explore on your own the use of Reinforcement Learning algorithms to solve stochastic control problems. The expected workload corresponds to approximately one full weekend of work.

You may be asked to reproduce results from selected research papers. For more ambitious students, we may also propose some topics which will rely on the methods studied in lectures and tutorials to address more challenging control problems.

If you already have other project ideas in mind that make use of the methods covered during the course, please feel free to inform us in advance (between today and second week of February) so that the project can potentially be validated.

In all cases, the final mini-project will require the submission of a PDF report of at most 8 pages. The report should include the references to the papers used, a clear formulation of the problem addressed in these papers, a description of the numerical methods employed, as well as your numerical results. You will also need to submit the associated code (either in .py or .ipynb format).

The mini-project³ will be announced at the end of the third tutorial session on generative models, which will take place on February 19.⁴ The submission deadline for the mini-project and the answers to the lab session⁵ will be announced later.

---

1. If you end-up with a .txt file, download it and rename it as a .ipynb file.

2. There will be coding and math questions.

3. i.e., the list of proposed papers along with the expected work, as well as alternative, more challenging project options.

4. A follow-up email will be sent to recall you to complete the Excel sheet available here to choose your lab.

5. No PDF submission is required for the lab session answers. The answers should be written directly in the Jupyter notebook and submitted along with the documents associated with the mini-project.