rl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Reinforcement Learning\n",
    "\n",
    "This IPy notebook acts as supporting material for **Chapter 21 Reinforcement Learning** of the book* Artificial Intelligence: A Modern Approach*. This notebook makes use of the implementations in rl.py module. We also make use of implementation of MDPs in the mdp.py module to test our agents. It might be helpful if you have already gone through the IPy notebook dealing with Markov decision process. Let us import everything from the rl module. It might be helpful to view the source of some of our implementations. Please refer to the Introductory IPy file for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from rl import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Review\n",
    "Before we start playing with the actual implementations let us review a couple of things about RL.\n",
    "\n",
    "1. Reinforcement Learning is concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. \n",
    "\n",
    "2. Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).\n",
    "\n",
    "-- Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n",
    "\n",
    "In summary we have a sequence of state action transitions with rewards associated with some states. Our goal is to find the optimal policy (pi) which tells us what action to take in each state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passive Reinforcement Learning\n",
    "\n",
    "In passive Reinforcement Learning the agent follows a fixed policy and tries to learn the Reward function and the Transition model (if it is not aware of that).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passive Temporal Difference Agent\n",
    "\n",
    "The PassiveTDAgent class in the rl module implements the Agent Program (notice the usage of word Program) described in **Fig 21.4** of the AIMA Book. PassiveTDAgent uses temporal differences to learn utility estimates. In simple terms we learn the difference between the states and backup the values to previous states while following a fixed policy.  Let us look into the source before we see some usage examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%psource PassiveTDAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a policy(pi) and a mdp whose utility of states will be estimated. Let us import a GridMDP object from the mdp module. **Figure 17.1 (sequential_decision_environment)** is similar to **Figure 21.1** but has some discounting as **gamma = 0.9**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from mdp import sequential_decision_environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mdp.GridMDP at 0x7f688c0702e8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequential_decision_environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 17.1 (sequential_decision_environment)** is a GridMDP object and is similar to the grid shown in **Figure 21.1**. The rewards in the terminal states are **+1** and **-1** and **-0.04** in rest of the states. <img src=\"files/images/mdp.png\"> Now we define a policy similar to **Fig 21.1** in the book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Action Directions\n",
    "north = (0, 1)\n",
    "south = (0,-1)\n",
    "west = (-1, 0)\n",
    "east = (1, 0)\n",
    "\n",
    "policy = {\n",
    "    (0, 2): east,  (1, 2): east,  (2, 2): east,   (3, 2): None,\n",
    "    (0, 1): north,                (2, 1): north,  (3, 1): None,\n",
    "    (0, 0): north, (1, 0): west,  (2, 0): west,   (3, 0): west, \n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "our_agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rl module also has a simple implementation to simulate iterations. The function is called **run_single_trial**. Now we can try our implementation. We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import value_iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values calculated by value iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n"
     ]
    }
   ],
   "source": [
    "print(value_iteration(sequential_decision_environment))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the values estimated by our agent after **200 trials**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.4496668011879283, (1, 2): 0.619085803445832, (3, 2): 1, (0, 0): 0.32062531035042224, (2, 0): 0.0, (3, 0): 0.0, (1, 0): 0.235638474671875, (3, 1): -1, (2, 2): 0.7597530664991547, (2, 1): 0.4275522091676434, (0, 2): 0.5333144285450669}\n"
     ]
    }
   ],
   "source": [
    "for i in range(200):\n",
    "    run_single_trial(our_agent,sequential_decision_environment)\n",
    "print(our_agent.U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also explore how these estimates vary with time by using plots similar to **Fig 21.5a**. To do so we define a function to help us with the same. We will first enable matplotlib using the inline backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def graph_utility_estimates(agent_program, mdp, no_of_iterations, states_to_graph):\n",
    "    graphs = {state:[] for state in states_to_graph}\n",
    "    for iteration in range(1,no_of_iterations+1):\n",
    "        run_single_trial(agent_program, mdp)\n",
    "        for state in states_to_graph:\n",
    "            graphs[state].append((iteration, agent_program.U[state]))\n",
    "    for state, value in graphs.items():\n",
    "        state_x, state_y = zip(*value)\n",
    "        plt.plot(state_x, state_y, label=str(state))\n",
    "    plt.ylim([0,1.2])\n",
    "    plt.legend(loc='lower right')\n",
    "    plt.xlabel('Iterations')\n",
    "    plt.ylabel('U')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a plot of state (2,2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYVMXZt+/ZIMIgAygiu8ENxQWDShR13FAxLjEuwS0a\nEzX5fF3eJKLRKEaNJppoVGI04hKjYtRoJC4R8zrghojIIouyB1AEBWSHGaa+P54u+/Q63T3dMz1n\nfvd19dV9tjp1qs+pXz3PU1UHhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQIJQ8DnwMzUmw/B5gG\nTAfeBvZtonwJIYRoRg4DBpJaHL4NdIz8Ph6Y2BSZEkII0fz0JbU4BOkELC1sVoQQQmRCaXNnIMBF\nwMvNnQkhhBBNQ18athyOBGZh1oMQQohmpry5M4AFof+CxRxWJ9thv/32c9OmTWvSTAkhRAiYBuyf\ny4HN7VbqDfwDOBeYl2qnadOm4ZzTxzluvPHGZs9DsXxUFioLlUX6D7BfrpVzoS2Hp4AjgB2AJcCN\nQEVk2wPADZgr6f7IulrgoALnSQghRAMUWhyGN7D9R5GPEEKIIqK53UoiS6qrq5s7C0WDyiKKyiKK\nyiI/lDR3BjLERfxnQgghMqSkpARyrOdlOQghhEhA4iCEECIBiYMQQogEJA5CCCESkDgIIYRIQOIg\nhBAiAYmDEEKIBCQOQgghEpA4CCGESEDiIIQQIgGJgxBCiAQkDkIIIRKQOAghhEhA4iCEECIBiYMQ\nQogEJA5CCCESkDgIIYRIQOIghBAiAYmDEEKIBCQOQgghEpA4CCGESEDiIIQQIgGJgxBCiAQkDkII\nIRKQOAghhEhA4iCEECKBQovDw8DnwIw0+9wDzAWmAQMLnB8hhBAZUGhxeAQ4Ps32YcCuwG7AxcD9\nBc6PEEKIDCi0OLwJrE6z/WTgscjv94AqYKcC50kIIUQDNHfMoQewJLC8FOjZTHkRQggRobnFAaAk\nbtk1Sy6EEEJ8TXkzn38Z0Cuw3DOyLoGRI0d+/bu6uprq6upC5ksIIVocNTU11NTU5CWt+FZ7IegL\njAX2SbJtGHBZ5HswcHfkOx7nnAwKIYTIhpKSEsixni+05fAUcASwAxZbuBGoiGx7AHgZE4Z5wAbg\nwgLnRwghRAY0heWQD2Q5CCFEljTGciiGgLQQQogiQ+IghBAiAYmDEEKIBCQOQgghEpA4CCGESEDi\nIIQQIgGJgxBCiAQkDkIIIRKQOAghhEhA4iCEECIBiYMQQogEJA5CCCESkDgIIYRIQOIghBAiAYmD\nEEKIBCQOQgghEpA4CCGESEDiIIQQIgGJgxBCiAQkDkIIIRKQOAghhEigvLkzkClXXglffQX33Qf/\n+Ae89hpcdBGsXQsvvQR33gkdOiQ/9uWXbf+1a2HjRigrg7vvhh13tO2bN8Mzz8DkydC5M9x4Y9Nd\nV5C5c+Gjj+C7381/2s5BSUn6fbZtg6lT4eOP4dNP4dJLobIyur2uDubMgVmzYPFi6NgRLr44dVoz\nZ8K8efDf/1rZr1sH69dDba1tr6+Pfn/nO3D22fm73mT5WbDAru3gg6P/vUjOxo32H3/xBaxcCV9+\naf/dhg2w++5w5pnNncPiYts2K5vtt2/unOSPBqqLosEdcohj4UJ44AH44Q/h5JPhgw/g889h+XKr\nVPfeO/HA+++HP/zBKrHOnaFdO/jVr+CJJ6yS2LIFhgyBLl3s+D/8wSrSpmTFCjjvPJgxw65nzZrU\nQhfPE0/AuefCqFHw058mbl++HK6/Hp56yirGnj0T91m3Dn7/exPenXaCffaBiRPhL3+BY4+1iv3m\nm+GvfzVBGDAAdtkF/vQnO7Y80MSYOxfuuMPEdqedYM89oXdvqKoyoamshDZtoLTURLq01ER52TJ4\n9tncyi8VmzfDk0/C00/DW29B166waJH9x1ddld9zZcOKFdbA2WMPOPLI5suHxzn7v//zHyunWbNM\nEHr1sjLbcUd7djp0gIoKePhh214aAr+DczB/Prz3Hrz/PvTvD5dckn7/BQvg3Xdh0iS73+fPNyHd\nutWe365dmy7/DVFiLcKWUs/nhLv3XucGDnRu2DDnfvEL5yZPdg6cu/xy5/bbz7kpU1wCtbXO9elj\n+wYZPNi5t9+233fd5dx3vuNcfb1zy5c7t+OOienkSn29c+PGpd9n82a7rmuucW7rVuc6d3Zu5crM\n0r/jDud22825c85x7ic/Sdw+b55zPXs6d/XVtl+yMlqyxLndd7c05s6Nrr/kEufuu8+5Dz+0NC66\nKHa7c1a2wXV/+5tzO+zg3I03OrdsWWbX4JxzY8c6d8IJme+fCa++avk+4QTnnnnGudWrbf311zv3\n61/n91yZsnKllWunTs7tu69zF16Yn3S3bXPuo4+yP27dOuduv925fv2c69/fuZ/9zLkXXnBu/nzn\n6upSH7fbbs4991zq7W+95dwFF0TLvBiZP9+5ESPs3t95Z+e+9z3nLr3U6oZkvP++c1dcYfdUz57O\nnXGGPX9jxzo3a5ZzGzdaPfTBB5nnYfny7PbPBaCJm7pNj5s2zbkhQ5wbMMC5UaOsUm3Txm7SQYOc\ne++9xIL54AO76eM59FDnJkyw39XV9gc759yKFc516ZK/P+att0zA0nH33SZ49fW23K1bbMX64IP2\n8MczZYpzO+1klftLLzk3dKitnz7dxGDrVucOOMC5P/7R1h96qHNvvhmbRm2tc9/6lnO/+U1i+r//\nvXNnnmkPztNPJ8/7ccc5969/2e+//tXEYvr09NebjP/8x/6HxrBmjXNbttjv++5zrlcvSzee22+3\n8mlqxo2z//bKK+0+e+YZ57773canu3Spc4cfbvfZxo2ZH/fPfzrXvbtzw4fbs+Pvv0z44Q/tfPGN\nhcWLnTvxROf69rXt48dnnmZTMWeOlXuXLs79/OdW6ftrX7rU/qMgr71m5du7t3M33eTczJmp0z7u\nOHsWG2LePOfOO8+5ykp7ZgoJrUEctm2zwt9hB+cee8wu/M47nfvqK+cOOcQq4nieftpaBPEcfrhz\nb7zh3Nq19getX2/rv/jCWu754vzz7SFJ9eDV19uDNHFidF2fPs4tXGi/vXWUzJI4+WTn7r3Xfs+Z\nY60/55z79rftmIcftgrXn/u445x75RX7ff/9Vjb33OPcMcckz9/YsZbONdekvr7LLzcRWbzYHrYZ\nM1Lvm4533nHu4INzO9Y5a6GCcyNHWqXbo0e0DOO5917nfvrT7NKfONG5//mf3PP34otmkb7xRnTd\nuHHOHXVU7mk6Z+W9887O3XKLieGCBQ0fU1/v3HXX2X2W7JnJhK1brbERFN+XX7Zn85ZbTKSHD3fu\n8cdjj1u6NDuLMleWLbPWvHMmfF99ZZbQzTdbHu+4I/rMB9m2zbm2bU1kv/jCudNPd+6b33TuySet\nIdUQP/iBc6NHp96+caNzV11lz8rIkXaOtm2d27Ahp8uMYeLE5J4BGiEOLSYgXVoK7dtbYMwHSX/2\nM/uuqDB/Xzzz5sGuuyauLyuzANLcudCvn6Xrz1Ffn3meVq82X2yyGIVzMG6c/d62LdYv75k2zfJ+\n8MHRdW3aWBwE4JFH7Lu2Nva45cthwgQYM8aW+/aFJUvMnz5nDrRtCyNHwujR0SB0ZaUFFOvq4Cc/\nsXLZssV838kC1fvsY9/XXZf6+vfay3y1M2damgMGpN43He3aWQA0V0aMsDRee81iL6++amWSjMpK\nCxxmyurVcNZZsN12ueVt9myLkb38Mhx4YHR9x44WW2qIH/3I/v/HHotdP28eHHecxYqGD7dOGZ9+\narGgdPzyl1Y+kyfDDjtkfz1g92yfPrB0qS2PGWMdRl58Eb79bVvXu7d1RPDMmmUxvWOOgRdeiD5z\n+WbqVDj+eOjeHc4/3+qIq6+2Z23jRtveo0fyY0tLLc7y4ot2T33vexZny/S/79bNYg5gZbL77nDA\nAbY8dy6ccgrsv789o77s+/Wzbfvtl9v11tXZs37rrXDiifCvf+WWTjJaVEipfXurdIM9aMBu1vgK\ndNUqCxqlE4fFi+0m92QrDjNmpN42f76dp7w8mubWrVYR++V//9tu5CBt2kSF7rXX7NuLhef55+1G\n8Ddt27YWaL7lFjjtNBg0yMrp6KOjx3hx+M9/rLwWLbKHZNCg5Pnv08fyGV/WQQ480ATw+eetcsiV\n7baDTZtyO3bWLKts3noL3nkHLrgg+kAmo317K4dMufpqE8H4+ysTtmyB738fbrstVhjAAvRffZX+\n+DffNIGfMyd2fW2tCcLVV9s3wM47w2efpU/vySfhuefg9ddzFwZPjx7WiWDCBLj8ckvTCwPEisPS\npXDCCXDXXbZfunvKM3q0CTNY8HvgQBPYk06y/zwZU6aYYN5zj/Wy+tOfrDPC7bdb54jXX08tDJ6+\nfe0e+t3vTHizaRR062YNt5tugnPOsc4iYIHrIUPgiivsPwiW/Z57WkeRdCxaBH/8Y+L6DRtMcCZO\nNEH79NPM85oJLU4cILEnT5s29qesWhVdd801pqL9+iWmExSH3r2j67MVh3R/6rvvwqGH2rl8mn7/\nujr7fvNNqK6OPa5tWxOHJUusZdmvX6JV9NprJg5BBg2y1uXJJ1sL/rTTYi0CLw7//CfccIOtO+OM\n9NfXUNfXffaxB/fkk623V65st13mlsOoUbGi/NvfmjDtv789KCNGpD8+G8th/nwTvltvzU0cRo+2\nSvuiixK3VVWltxy2bLEedpdfbvd3kN/8xiqYyy+Pruve3SqhAQOSW9ELF1rl9PTTjfuvPD17wvTp\nZlX97W+JVmOfPtaz8LbbTCAvucT+p/vvt+3p/u+HHzaL6fXX7Tk9+2xr7Jx2mllI8WIJ9vyfeqrd\nH2eeaSL41lvW+n/tNfsvklnv8fzoR1bR5tJVt1s3K4u//x0efNAs6mnTTNBGj07eC6p/f9svFYsX\nW482/8x6Nm60hmXXrmYJHnGE1S++bgGzDouZ44E5wFwg2WO7A/AqMBX4CLggRTrOOetNAYlBz1NO\nsfW77hpdd9VVzh1xhAWu4xk2zHzqV11l/kfP+vXOtWuXuZ/vqqvsvMkCxtdfb712ttsu6t98/HHb\n3y/H9/ZxLtqT6u9/d+7UU53be+9YX359vQWiFy2KPe6OO5wrKTH/+8qVFqANMmKEc7fdZuecNcu5\niy+2YHZjOeGExgcev/jCevA0RG2tld/Pf27Lq1Y517Fj5r27nLO8Dhliv597Lr0v+YILzDe8dKn5\n9rNh0yaLfUyalHz7li3OlZenjkfddZcFd6dNs3vAs3y5xcX++9/Y/W+91dID5z75JDG9M84wn3u+\nePZZO9eVVybfPm+ebS8ttdhX8BnZay+LiR16qHMPPWSxP8+CBRYXOOEE6yhxyy0WI1y92v73c86x\nThpBtm2zZ33kyPxdXy68/749X//9r/WG6trVlseMSX3MCy84d/zx0eXPP4/+h2vX2n//+9871759\ntPdXXZ1zJ53k3LnnxparSajdUx9+aOVII2IOhbQcyoD7MIHYCxgO9I/b5zLgQ2B/oBr4PWniIN5y\nSOZWAjOrH3rIPuvXm8ndtm2SjGXgVlq40MZRgLVA1q1LTMe3YLZtS9zm4x1By2H6dPuuq7OxAytX\nJvqIvVtp5kxrjfnliRPh9NMtz6WlsRYPwOGHm6VSVWWtyo4dY7dXVsInn1i57LmnteqSjXnIlpdf\ntnM3hkwshyVLov7U7t3t+7nnbBxGNi4SbzksW2atymStUDC3xPPPW2s7VUxr0ybbJxmPPGKukHh3\nkqdNG0s32XXX1poL5sYbLabl3Stg7o5zzjHfeJDu3W3fgw+2fvhBJk2Ct9/O79iOPfc0d9vNNyff\n3q+fVVW//rVZtMExEbvvDpddZu6hH/3Ini+w/S++2Nxlp51mLeK77jL3TFWVjZ/p0cPijkEefNAs\nreuvz9/15cKgQVb2vXqZe2r9erNAzjor9TEHH2xxO1+1n3++rX/rLbjwQjjkEPvf+vSxZx/Mfbxu\nnVkjwXK96y77njzZrPlRoxp3PYUUh4OAecAioBYYA5wSt89ngB9TuD3wJVBHClKJgze727SBH//Y\nbrD161P7Nr04LFsWW0EGxeGYY6L++MMOswFinlmzrGLwPt5kLgcvDqWlUfHwbqXaWkujf3/LSxDv\nVpo50x4+H6AePdoqw6lTzace7/I56CBzU6WistLiA4MHN+wuamq+8Q275nQuvXPPtUqxoiIaM3j+\neavgs8HHHJ56ypZ9ADGeMWPMdVdVlTymBRbrOP302MCr57HHkg9KDJIqKP3ss1a5HHggdOoUdZeu\nWmUul2uuSTzm9NMtnrTfflYhB0Xv17+2gZ/5DALvvbcNPG0ofnDddYkNmT32MJdkTY09A/45GjfO\nGgFXXWUCMmGCBZSDz2jnzrHisHKlXdtDDyU+S82Br6xLS+GNN8wFmI5u3WxU9dy55nJbtcpcpbfe\nao25e++159WLw5tvwp//bIIZ72688koThVNPNVdeY0exF1IcegBLAstLI+uC/AXYG/gUmAZckS7B\nVDEHbzn47/LyzMRh7drYFnYwWBz/0AYDU3vvba0YX7HUxcmZc/Znx1sOvndHXZ2Jx267JebNWwq+\nd4cXC3+uTz6xhytbKivt/AcdlP2xhaa01K5z8+bk29ets9bvxo3WClu/3sR5wgQLdGZDMPbSqVPq\nIN6zz9oDBqnF4bnnLND55JOx6+fNM8vz2GPT5yVVUHrUqGgrv107u382bzZf9tChUcsp/roGDLBt\n77xjlQpYhfLuu9EWaT7JtZFx9dUwdizsu6+J+5Il9sz88pdmiZSX27bTTku0drp0iY0t3nGH3RPJ\nZkdobg46KLM4x+DB1tC44QbrHTV4sAWhH3kk6vno08fuq4svtkB7snsAolPDpLLosqGQXVkz8XX9\nEos3VAP9gHHAfkCCE2fkyJFMnWo35LvvVnPkkdVfb/Oi4JU0U3FYty52n9LSaLfUtWtjj4kXpNpa\na7W0b58oDn7+oC5dYq2RpUttubY20WrxtGljFcH8+SYeXiyWL7ftn3yS2lWRDn+dxfgQQbQ7a7t2\nidsmTLCeMNdea9c/c6a52QYMSHSfNUT79tZSXb/eKswnnrBW7P77R/f58kv48EOzHiG5OGzcaC3d\nSy9NdHM89ZS12hqqGJIFpT/7zK7Pi15JiYnYtGlWcaTrWgzwzW/at+/9NXq0WVzJyrW56Nw5+rtn\nT3ODjBtnFrK3BKuqTHzj6dIlWt4rVtj1TZtW+DwXksGDTTCvuMLcdbvvblN5DBwY3We//Uw8Bw82\nyyAVgwbVsG5dDbfd1vh8FVIclgFBz2gvzHoIcghwa+T3fGAhsAeQEGcfOXIkzzxjpmj8fDReFOIt\nh1TzE3lxiN8nWJHX1Zlqe7Ho0MFuYO9q2rzZ1pWXJ4rDypWm3iUlUbfSli3mO+7Rw/Zftiz6IAdp\n29a2dehg1koycTjnnOTXlQ4vDnvumf2xTUG67qzvvmtdAY89Nlqx19Qk9vTKBG99Hnqo+cVHjbKy\nnTrVWm4jRph75rDDotZiRYX9Z8HJC1991VqG3bsn+viffdZadw3RrVti91PfTTkYK1u1yiqFnXay\nrprpOO88i2P9/Od2340ebV2mi5Veveya//xn+H//r+H5moLiMGqUWQ35iJ01J4cdZoLphb+0NFYY\nwNzl8+fbeKJ0FtvQodUMHVr99fJNN92Uc74K6VaaDOwG9AXaAGcBL8btMweItM/YCROGuEctSvv2\nySv8fFkOJSXRwBCYL9AHoktKzHc7caItr11rD2s6cfDnqq+3SqBbN3vovTgk63Pdpo1VNn6bjzms\nWGHLs2cnd0c1hC+bZOM+ioF0A+EmTYq6wyor7T/xXYWzxd8rBx0UnSBtt90szZtvNl/9G2/ENkD8\nJIHB//mVV6yLYnDQIljFtWhR7MDGVPTta/t6/vY3y0N8HMVbLbffnpmbon9/u08mTTKrI9fBiU1B\nr14mzDU1mTV6fMyhrs6E79JLC57FgvOtb9l9UFWVep+yMnOhJWtQFopCikMd1hvp38As4GlgNnBJ\n5APwG2AQFm94HbgaWJWQUoT27ZNX+Kksh3TisGGD7eePARMALxBg4uD9m3V1VoF4N8CKFSYOyVwO\nX3wR7UHjLYelS62FU14edSulEoeFC6Pb2ra1eEPbtnbzrFplfeezxV9Tst5bxUAqy8E5M7HjxeGD\nD+yhypWBA6MPY319VPTXrUtuncb/z+PHm+XiY0KeN980F1gmlXjfvvZfg13neeeZFRNvHRx5pMUb\nLrggs2vr0sUaUaNG2VToxcxee1k5XnhhZjMR77ijPUujRtkzsu++Bc9ik/CNbzR3DhIp9PQZr0Q+\nQR4I/P4COCnTxHr3Tt4iy8Vy+Oqr5NuDrqV4cVi7NuoGWLzYWpxLlmRmOXz6qbkg/BQWqcShbVuz\nHA45JHpNixfbsZs2WSWaS6+MoUOjAfFiJJXlsGSJXbNv5XfoYD75tm1zE0mwAPKJJ9p98q9/WYvc\nd6dcsMDOGV/pBMXhs8+s9TpggHVPDorD+PE2ICkTdtklOsXKRx+Zm2v27NgGC8D//V/21zhokMU+\nxo/P/timpKzMeoZlOk3+TjvZ//W//9v4rpoiPS1qhPQuu8Cjjyauj++tVFGR6DIKUlaW+p0J6cRh\n3bqoOHz2mZm4DbmVfHp+Hqby8mgMIVnl5i0H70dt08ZMzp13tvzkWiGWlDQ8dUBzkspymDkzNohe\nWWllF++TzYbhw+0eKSmxCnnFCrNEunQx//zAgYkt/6A4TJpkjRTfyyroVspGHIJupfHjzUKIF4Zc\nGTTIgvWDB+cnvUKTTc+nn/7UnpELLyxcfkQLmngvHfFupfp6u9ni+wF7MrUcOnZMbTl88YUdn4k4\nbNtm4tCpk+Vx9WozI5Plr00ba0F7EfDi0K+fVU7Z9s5pKaSyHOJf4pTvXlddu5o4rF9vvZNeeSX5\ndBdBcZg8OdoxITgX1po11oU51XxV8Xzzm1bJ1daa6+ywwxp/PZ4TT7R7LxP3Vksk10aSyJwWZTmk\nIt6ttHlz+sE56SyHkpLo3DsVFVFx2LjRKgEvDnV1do6GYg7erbRmjfm4y8tNPFIFn3xMwB/ftq25\nlbzlkKp/c0sn3nfvmTXL/NIe/5/lq9eV7066erUNLly7NrZbqyedOHjLYdo06xabqlEST2WluUpn\nzTJxyFRUMmHgQOv6KESuhEoc/Hcm4pDOcvBB5/r6aGvWT2Hgu5RCasvhyy+jk5sFLQc/2nblSrMi\nkuErFr/dTyq4885mNYS1xeQD9fHMm2f9vj357pLru06edVZUkJP17gmKw9SpUQEJipqf8iQbBg2y\n3lGLFxfvGBTROgmF0RmcPgPsYU036MdbDskq2tLSqO9727bo7+AUBn6UbapxDuvWRV80HrQcOnVq\n2HLw1+AHCvnlrl2tBd0/fnaqkJBqFPKCBbHd97xllUt33lRs2mRuPj/oKpnw+PytWmWWpZ/bKGg5\nfPRRbuLwwANmceQr3iBEPgil5RD/O56GLAdvLfhpCyBqOaxZE624U1kOGzbEvkAoF8shXhyqqmxi\nsWznEmop+IFmQTZtMhddMJBeUmKuPR/TyQfx3QiTNSy8OPgAuQ+gBi2HXMThqKOsh1I+XUpC5INQ\niUPQ15uuu6e3HBoSB285lJVFLYfNm6Muo1Qxh2A3Wh/gztRy8C1jLx7xy2El6FaqrTUxXbTI5pSJ\n/y+7dStMHk49NfWLZOLFweMtB+cSt2XCgAF2PRIHUWyEQhzi3UolJel7aZSVmesnWQsx6Faqr7ff\nHTrETpvsW/Wp3EpBcfBupUwtB99TyuctaDmEmaDI/uIXVsYLFzb82st8UlaW2m3n8/fxx7FuJ285\nLF9u944fj5EpJSU25fRJGY/2EaJpCIU4JHMrNSQOmzcndz0lcytVVsbOBBnvVho6NPpKQEi0HLZt\ny9xy8Of2bov4AHVYCYrs1Kn2vWRJ4nTPzYUXh/j3knvLwbuUcpmp9KST8vN2NiHySajEISgIDbmV\namuTdzn04lBSEnUrpbIcvDiAvbDDs2FDrOXgxaFjx4Yth/hXWLYWcQhaDn4aaz+qvBjw+Zs/P/bV\ns8H3b6i3kQgToRAHX4H6IfiZuJUgueVQUmKC4OfR9+Kwfn20B1J8zCGYVm2tfXyswLup/Gja8nKz\nQlINZvMvsoknmxedt0SCAWnflTjVFCPNQUWFWZELF8b2nvKD4FK9n0OIlkooxCFYMYO11BuyHILH\nBfGVebt2lo53K23cGB2AFR9zgKhAeavBuxfKyqzCD07/DFGhiec734l9gY23JIrt7W35JhiQ9uJQ\nbJbDokVmwQXfqObdSk0dHxGi0IRqnIPvdlpbm7vlEBQHbzn4+EC8OLRvHz2PTyt+wr/SUlvnA8x+\n/1TjMI44InZunoberRwWkrmVis1ymDfP5kMK4t1K8RaFEC2dUFkOwYnbGiMO/o1kwZgDRFv7nTtb\npVBRkWg5NCQO/pyZvpmrNYlDXZ2VuXM29qCYLIc2bSzeEB8gD1oO8cIhREsmtOLQGLeSFwffWyle\nHHr0sPn1g2n472Aw2p8rG8shnu9+F849N7N9WzLerTR/vpXv1q3mXsrnYLfGUFFho7X9yGiPn37j\nG99IP2WLEC2NUIiDH+GaL8shWUAaot/t2sHJJ8eeJ2g5BH3SpaU2piI+5pCpOBxwADz+eGb7tmS8\nW+mjj+yay8qi7+AuBrw4pOpaW6iBeUI0F0Xy6DWOAQPgV7+KxhygsG6lYM+hhmIOjbUcWgverTRj\nhv2f7dtHJ8IrBioqYudUiqdYLBwh8kUoxKGkxLoRBqd8zqdbyVf26cTBfyezHBoTc2gteLfS7Nk2\nwWD79sVV4fr/TeIgWguhEAcwgQhOY5GPcQ6pLIdgxR58wRBEj/Vk21upteLdSitW2Gy5lZXFVeF6\n99YeeyTfnu20GUIUO6ESh23bosv5iDnEi4P/TmY5+HNv3Zo4AWBwnIPfP+yD2rLFu5VWrbLeYMVm\nOfjXeaaohB79AAATFElEQVQKOhdTXoXIB6ERBz+HkW/h5WMQ3JYtJjo+4L399la5J5vDyYvDli3R\n0dE+veAkf36/dPlrjXi3UrGKQ6rZWj3FlFch8kEoBsFB1HIoL7fWez4C0hs2WAvfp9WpU+IEaQ2J\nQ3xAOtmrMEXUreTfoldsbqWjjko95uTBB+GMM5o2P0IUmtCIg7ccKiqsAs5HQHrTpuh8SGCV1Zw5\nsfv7NNJZDhs2RMUh2dvOhJXx+vVWPu3b21QUxTRX0ejRqbf9+MdNlw8hmorQiIMPSMf3HkpGNm6l\ndu1ixzLET7XdUMwh3q0kyyE5FRUWjO7c2f7LUaOaO0dCtG5CF3NINn13PJlaDlu2WDo+raBF4MnU\nreQD0LIcklNRAZ9/Hp23SgjRvIRGHHzMwVf4jXUrBUc0N0Yc4ruyynJITnm5iYNeeiNEcRA6cciH\nW2nLltiup+nEoaGYQ1lZ7NgHiUNyKiqsDGU5CFEchEYc8uVW8iLj4wZBcUj25rhMYg4Q7Q6bLl+t\nGf9fhP1d2UK0FAotDscDc4C5wIgU+1QDHwIfATW5nsgHpPPhVoLk4pDMcvDr0lkOwf1+9zv48MP0\n19Ia8WWc6g15QoimpZDt2DLgPuAYYBnwPvAiMDuwTxUwCjgOWArkPNVaPgPSkLk4fO97MGUKTJgA\nd92VPOYQTK+qCvbfP7Nrak009IY8IUTTkk4cfha37ICVwFvAwgzSPgiYByyKLI8BTiFWHM4GnsOE\nAeCLDNJNSnxAOl/ikOyFPkEqK20a57/+FSZPTu1WSnYuEUXiIERxkc6t1AGoDHw6AAcCrwLDM0i7\nB7AksLw0si7IbkBn4A1gMnBeRrlOQnxAurFupaDIpLMcfHqbN9v5U7mVkgmLiOLLWOIgRHGQznIY\nmWJ9Z+A/wFMNpO0yOH8FcABwNNAOeBeYiMUoYjMzMpqd6upqqqurY7Y3hVspVQWfThzi0xPJ8f+F\nYg5C5E5NTQ01NTV5SSuXmMOqDPdbBgRnv+9F1H3kWYK5kjZFPhOA/WhAHJJRUmLf+ejKCrHiUFFh\ny/4cqdKT5ZA7cisJ0XjiG8433XRTzmnl0lvpSGB1BvtNxtxGfYE2wFlYQDrIP4EhWPC6HXAw0MD8\nl8mJdwflu7dSuso9KA6pYg4Sh/TIrSREcZHOcpiRZF0n4DPg/AzSrgMuA/6NVf6jsWD0JZHtD2Dd\nXF8FpgP1wF/IURx8qz5Tt1JJSXIB8ekExaGqCq68Mn16kN6tpIB0euRWEqK4SCcOJ8UtO+BLYH0W\n6b8S+QR5IG75zsinUfhKONOAdKrKOllvpYoKuPnm9OmB3EqNQZaDEMVFOnFY1FSZyAe+xe+tgoYs\nh4bEIRMLJJgeRN1KCkhnj2IOQhQXoZk+w4tDaalV1ukq9dLShsWhvNx+ZyMO9fVmOcS/JhQkDg0h\ny0GI4iI04uArdS8OjXUrlZVlLw7qypo7bdrAk09q7ikhioXQiIO3HHygubFupVwsBwWkc6ekBIZn\nMrRSCNEkhEYc4i2HdJX6DjvA0KHp0/HWRy4xB7mVhBAtndCIQ3zMIZ1bqWNHeOSR5Nvi3UqZtPj9\nuerq7E1vQSFwLnYfIYRoCYRGHLKxHNIR7PWUreWwaZOJSXAktZ/KWwghWhKhEYeg5ZBprCAZpaWW\nVjbpeHHYsiXR0qiryy0fQgjRnIROHHxAOlc3TlAQsrUctmxJ3F/iIIRoiYRGHPLlVvLH+9/ZiMPW\nrYn7y60khGiJhEYcshkEl46gIOQiDvEWi8RBCNESCY04ZDMIrqF0/LHpxkMESedWkjgIIVoioRGH\nYrAcFHMQQoSF0IlDPgLSQcshG3GorZXlIIQIB6ERh6BbaYcd7B0MuRCc0TVbyyH+N8hyEEK0TEIz\nzVnQrfTWW7mn0xjLAWQ5CCHCQSgth8amk2tXVpA4CCHCQWjEIWg5NIb4gHQ2vZVAAWkhRDgIjTh4\nUQjOa5RrOnIrCSFaO6ERh0JZDgpICyFaIxKHOPJtOdx9N7zySuPyJIQQTU1oeivlMyDdGMshfv/e\nve0jhBAtCVkOSdIJ9lZqbEBaCCFaIqERh3wGpHOdsjv+txBCtFRCIw6FiDnkw60khBAtkdCIQyFi\nDvkISAshREskNOJQCMthxx2hc+eGj8m2d5MQQhQ7oanKCiEO//hHZsd4UWjMu6uFEKKYKLTlcDww\nB5gLjEiz34FAHXBaricqREA6U7bbDk4+uXFThQshRDFRSHEoA+7DBGIvYDjQP8V+vwVeBXKu2gth\nOWRKeTk895wsByFEeCikOBwEzAMWAbXAGOCUJPv9D/AssLIxJytEQDpbFHMQQoSFQopDD2BJYHlp\nZF38PqcA90eWXa4nK8QguGyROAghwkIhq7JMKvq7gWsi+5aQxq00cuTIr39XV1dTXV0ds70QE+9l\ni2IOQojmpKamhpqamrykVUhxWAb0Ciz3wqyHIN/C3E0AOwAnYC6oF+MTC4pDMgoxZXe2yHIQQjQn\n8Q3nm266Kee0ClmVTQZ2A/oCnwJnYUHpIN8M/H4EGEsSYciEYrEcJA5CiDBQyKqsDrgM+DfWI2k0\nMBu4JLL9gXyerBCvCc0WiYMQIiwUuip7JfIJkkoULmzMiZqzK6tHMQchRFgIzfQZ6soqhBD5IzRV\nmbccGhuQ7tULamtzO1aD4IQQYSE0VVm+3Eqn5TyBhywHIUR4kFspj0gchBBhITTikC/LoTEoIC2E\nCAuhEQdZDkIIkT9CIw75Ckg3BomDECIshEYcZDkIIUT+CI04KOYghBD5Q+KQR2Q5CCHCQmjEAUwg\nmlMcNAhOCBEWQicOCkgLIUTjCZU4lJbKrSSEEPkgVOLQ3G4lBaSFEGFB4pBHKirsI4QQLZ1QOUFK\nS5s35nDnndCzZ/OdXwgh8kWoxKG5LYfdd2++cwshRD4JlVupuQPSQggRFkJVlTa35SCEEGEhVFWp\nLAchhMgPoapKm3sQnBBChIXQiYMsByGEaDyhqkrlVhJCiPwQqqpUloMQQuSHUFWlshyEECI/hKoq\nVUBaCCHyQ6jEQZaDEELkh1BVpYo5CCFEfmiKqvR4YA4wFxiRZPs5wDRgOvA2sG+uJ5I4CCFEfij0\nxHtlwH3AMcAy4H3gRWB2YJ8FwOHAV5iQPAgMzuVkcisJIUR+KHRVehAwD1gE1AJjgFPi9nkXEwaA\n94CcJ71WQFoIIfJDocWhB7AksLw0si4VFwEv53oyWQ5CCJEfCu1WclnseyTwQ+DQXE+mmIMQQuSH\nQovDMqBXYLkXZj3Esy/wFyzmsDpZQiNHjvz6d3V1NdXV1Qn7SByEEK2Zmpoaampq8pJWoT305cDH\nwNHAp8AkYDixAenewP8B5wITU6TjnGvYCNl1V3j2Wdh//8ZkWQghwkGJBWFzqucLbTnUAZcB/8Z6\nLo3GhOGSyPYHgBuATsD9kXW1WCA7axSQFkKI/NBSqtKMLIc99jDLYZ99miBHQghR5DTGcgiVh14x\nByGEyA+hqkrPOAO6d2/uXAghRMsnVG4lIYQQUeRWEkIIkVcK3VtJCCFyonPnzqxenXTYk4ijU6dO\nrFq1Kq9pyq0khChKSkpK0HOfGanKSm4lIYQQeUXiIIQQIgGJgxBCiAQkDkIIIRKQOAghRI5ce+21\n/PGPfyz4ecaOHcv3v//9gp8niMRBCCFyYOXKlTz++ONceumlAEycOJFjjz2WLl260LVrV84880yW\nL1+ecVrDhw+nR48eVFVVMWTIECZNmvT19pNOOomZM2cyY8aMglxLMiQOQgiRA48++ignnngibdu2\nBWDNmjVceumlLF68mMWLF9OhQwcuvPDCjNJav349Bx98MFOmTGH16tX84Ac/4MQTT2TDhg1f7zN8\n+HAefPDBglxLMjTOQQhRlBT7OIejjz6aiy66iLPPPjvp9ilTplBdXc3atWtzSr9jx47U1NQwcOBA\nAN555x3OPfdcFixYkLCvxjkIIUSRMGPGDPbYY4+U2ydMmMCAAQNySnvq1Kls3bqVXXfd9et1e+65\nJ4sWLWL9+vU5pZktmj5DCNFiydfLvXIxUNasWUOHDh2Sbps+fTo333wzL774Ytbprl27lvPOO4+R\nI0fGpO9/r1mzhsrKyuwznCUSByFEi6U5vU6dOnVi3bp1CevnzZvHsGHDuOeeezj00EOzSnPTpk2c\ndNJJHHLIIYwYMSJmmz9XVVVV7pnOArmVhBAiB/bdd18+/vjjmHWLFy/m2GOP5YYbbuCcc87JKr0t\nW7Zw6qmn0rt3bx544IGE7bNnz6Zv375NYjWAxEEIIXJi2LBhjB8//uvlZcuWcdRRR3HZZZdx8cUX\nJ+z/6KOPsssuuyRNq7a2ltNPP5127drx6KOPJt1n/PjxDBs2LC95zwSJgxBC5MD555/Pyy+/zObN\nmwF46KGHWLhw4dexgg4dOrD99tt/vf+SJUsYMmRI0rTeeecdXnrpJcaNG0dVVdXXx7/99ttf7zNm\nzBguueSSwl5UAHVlFUIUJcXelRXguuuuo2vXrlxxxRUN7nvcccdxzz33pO3hlIqxY8fyxBNPMGbM\nmKTbC9GVVeIghChKWoI4FAsa5yCEEKJJkDgIIYRIQOIghBAiAYmDEEKIBCQOQgghEtD0GUKIoqRT\np06+t41ogE6dOuU9zUKX/PHA3UAZ8BDw2yT73AOcAGwELgA+TLKPurIKIUSWFGtX1jLgPkwg9gKG\nA/3j9hkG7ArsBlwM3F/A/ISCmpqa5s5C0aCyiKKyiKKyyA+FFIeDgHnAIqAWGAOcErfPycBjkd/v\nAVXATgXMU4tHN34UlUUUlUUUlUV+KKQ49ACWBJaXRtY1tE/PAuZJCCFEBhRSHDINEsT7wxRcEEKI\nZqaQAenBwEgs5gBwLVBPbFD6z0AN5nICmAMcAXwel9Y8oF+B8imEEGFlPhbXLSrKsYz1BdoAU0ke\nkH458nswMLGpMieEEKL5OAH4GGv5XxtZd0nk47kvsn0acECT5k4IIYQQQggRDo7H4hBzgREN7BsG\nHsbiLTMC6zoD44BPgNew7r6ea7GymQMMbaI8NhW9gDeAmcBHwOWR9a2xPL6BdfWeCswCbousb41l\n4SnDBsyOjSy31rJYBEzHymJSZF3oy6IMczf1BSpIHrMIG4cBA4kVh98BV0d+jwBuj/zeCyuTCqyM\n5hGuubK6AftHfldi7sn+tN7yaBf5Lsdic0NovWUB8L/AE8CLkeXWWhYLMTEIEvqy+DbwamD5msgn\n7PQlVhzmEB0Y2C2yDNYCCFpTr2JB/bDyAnAMKo92wPvA3rTesugJvA4cSdRyaK1lsRDoErcuL2VR\nzKqRySC61sBORLv2fk70T++OlYknzOXTF7Oo3qP1lkcp1ur7nKi7rbWWxV3AL7Cu8Z7WWhYOE8rJ\nwI8j6/JSFsU8K6sGwyXiSF8uYSyzSuA54ApgXdy21lQe9ZibrSPwb6zVHKS1lMV3gBWYj706xT6t\npSwADgU+A3bE4gxz4rbnXBbFbDksw4KSnl7Eql5r4XPMNATYGXswILF8ekbWhYkKTBgex9xK0LrL\nA+Ar4CXgW7TOsjgEm5NtIfAUcBR2f7TGsgATBoCVwPPYnHahL4tMBtGFkb4kBqS9n/AaEoNLbYBd\nsLIK0+T3JcBfMRdCkNZYHjsQ7XGyHTABOJrWWRZBjiAac2iNZdEO6BD53R54G+uB1CrKItkgujDz\nFPApsBWLt1yI9UR4neTd0n6Jlc0c4LgmzWnhGYK5UqZiLoQPsa7NrbE89gGmYGUxHfO3Q+ssiyBH\nEO2t1BrLYhfsnpiKdff2dWRrLAshhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYRoSayPfPcBhuc5\n7V/GLb+d5/SFEEIUCD8nUzXREbWZ0tD8Y/HzPQkhhGgh+Ap8IrAGG219BTa32B3YS1KmARdH9qsG\n3gT+SXQisxewmS8/Ijr75e1AXSS9xyPrvJVSEkl7Bjaq+cxA2jXAM8Bs4G+BfN6OzbY6LXKsEEKI\nAuLFITgXD5gYXBf53RZ7T0JfrAJfj7mhPJ0i39thFb5fjrcc/PL3sKkLSoCuwGJsMrRqTKC6R7a9\ng82s2YXYGTW3z/TihCgExTwrqxD5Jn6SsaHA+VjLfyI2J82ukW2TsArdcwU2h8272MyWuzVwriHA\nk9iUyCuA8cCBkeVJ2BxaLpJmH0wwNgOjge8Cm7K9OCHyicRBtHYuw14kNBDoh01YBrAhsE81Ngvq\nYOydCh9i73VOhyNRjPzc+VsC67ZhU5Nvw6ZbfhZ7Z8GrCNGMSBxEa2Id0SmOwV6a81OiQefdib6r\nOcj2wGqsZb8nsa9WrCV50PpN4CzsGdsROByzGFJNkdwemz3zFez9yPs1eDVCFJBifhOcEPnCt9in\nYS30qcAjwD1YjGEKVmmvwFw68W/PehW4FJiFTSH/bmDbg1jA+QPgvMBxz2PvQZ8WWfeLSPr9SXz7\nlsNE65+YRVICXJXz1QohhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQgghhBBChJn/D14FxN7T\nQhWsAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f688c024198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))\n",
    "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to plot multiple states on the same plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XlcFuX+//EXIqaIO2LuWJZaxzp+TbOyE9lmlprntJkt\nxzYr7VtZHVt+X8XK3CrNLJfMTHNp0bTSY2aGuea+5EKY+66IgiDr/fn9MYDAAN4gt6C8n48HD++Z\nue65r7mEed8z18w1ICIiIiIiIiIiIiIiIiIiIiIiIiIickEaDxwCNuaxvBuwHtgALAGuOkf1EhGR\nYnQj0IK8w+E6oEr66/bA8nNRKRERKX6h5B0OWVUD9vq2KiIi4o0yxV2BLJ4A5hR3JURE5NwI5cxH\nDjcDm3GOHkREpJiVLe4K4HRCf4rT5xCTW4Grr77a1q9ff04rJSJyAVgP/L0wbyzu00oNgBnAw8C2\nvAqtX78eM9OPGf369Sv2OpSUH7WF2kJtkf8PcHVhd86+PnKYCtwEBAN7gH5AQPqyMUBfnFNJo9Ln\npQCtfVwnERE5A1+HQ9czLH8y/UdEREqQ4j6tJAUUFhZW3FUoMdQWp6ktTlNbFA2/4q6Alyz9/JmI\niHjJz88PCrmf15GDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKicBAREReFg4iI\nuCgcRETEReEgIiIuCgcREXFROIiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgoH\nERFxUTiIiIiLwkFERFwUDiIi4qJwEBERF4WDiIi4KBxERMTF1+EwHjgEbMynzAggClgPtPBxfURE\nxAu+DofPgfb5LO8ANAYuA54GRvm4PiIi4gVfh8MiICaf5Z2AL9Jf/w5UBWr5uE4iInIGxd3nUBfY\nk2V6L1CvmOoiIiLpijscAPxyTFux1EJERDKVLebP3wfUzzJdL32ei59feJapsPQfERE5LSL95+zl\n/NbuC6HAD0DzXJZ1AHql/9sGGJ7+b05mpgMKEZGC8PPzg0Lu53195DAVuAkIxulb6AcEpC8bA8zB\nCYZtQDzQ3cf1ERERL5yLI4eioCMHEZECOpsjh5LQIS0iIiWMwkFERFwUDiIi4qJwEBERF4WDiIi4\nKBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKicBAREReFg4iIuCgcRETEReEgIiIuCgcR\nEXFROIiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgqHc+xI/BHGrRnHg98+yDu/\nvVOg9+6L3cf0zdMxMx/VTkTEUWrDYdPhTbw09yX2xe7z+WclpyXz5YYvafdFOxp/1Jh5f82jbqW6\nzN8+/4zvjU2KZfSq0Vw77lqaj2rOYzMfY+vRrT6vs8iFxMw4kXiiuKtxXjkvw8HMiEuKK9R7d8Ts\noNuMbrSb2I6V+1fywtwXuHPynURFRxVxLSElLYURv48gdHgoE9ZNoFfrXhx8+SBf3/c1j/39MY4m\nHM3zvdEJ0bzxyxuEDg9l/vb59A/rz6FXDnH7pbez8fDGIq9rQURFR/HU90/RZlybbPPTPGmkelKL\nqVYi2ZkZS3YvoefsnjQc3pCQ90JYvHsxAPHJ8UzeMJkHvn2A5XuXF3NNi1ZyWjJ7Tuw56/Wcd+Fw\nOP4wnad15sbPbyzQ+8yM0atG0+rTVjSp0YRtz29jUpdJbI/ZzoG4AyzavShb+ajoKDYd3lToei7c\nuZDmo5ozO2o2c7rNYf6j8/lns39SIaACAMGBwUSfina9z2Mexq4eS9OPm3I04Shre6zl2/u/pX3j\n9gT4B3BVratYc2ANaZ60QtetsA6dPMSzPz7L9eOvp17lekQdi+JA3AHMjDlRc2g+qjmvzHvlnNdL\nSq79cfuJ2BlxTj/zaMJRBvw2gMs+uoynfniKupXr8tPDPzH9/uk8PONhnvr+KeoNq8fkjZPx9/Nn\nzOoxbDi0gZPJJ89pPYuSmbF0z1Ke+fEZ6rxfh2YfN+Oh6Q8Vd7XOCTMz23hoozUc1tB6z+1tF719\nkaWmpZo34pPjrcu0LtZqbCvbfHiza/ngxYPtpbkvZU5P3zzdgt4Nsi7TupiZWUpayhk/w+PxWGpa\nqqWmpVq/X/vZxe9dbLO2zjKPx5Nr+cSURAt4KyDb8kMnD1m7L9pZq7GtbMPBDbm+79tN3xrh2Ee/\nf5RvfVLTUu3thW/b9M3Tc/3sYcuG2cmkk2fcroxtm7pxqoUMDbHec3vb0fijZmZ228TbbMLaCdZl\nWhdr8lET6zm7p3We2tmrdUrJ5vF47MfIH63t+LYWmxhb4PfvPbHXnp/zvAW9G2Shw0N9UEOzpbuX\nWofJHWztgbVmZrbr+C7rObunVRtUzR6f+bit3LfS9ff33pL3bOCigbYvdp+Zme05scf8+/sb4dj4\nNeO9/uxTKae83v/4UmJKok1YO8FajG5hjUc0tnd/e9d2xuy0FXtX2NAlQw244DsobdL6SRYyNMS+\nXP+lmZk1GNbAtkVvO2PjHT913Fp/2toe++4xS0pNyrXMnD/nGOHY1I1TbdCiQdZgWAObsmGKNRjW\nwDYe2mh13q9jE9dNzPc/6IFvHrCbJ9xsnaZ2spsn3Gz7Y/efsW5B7wbZicQTZma2/uB6azisof2/\nX/5fvr90iSmJ9q+v/mWvzns12/z3lrxn90y7x8zMjsYftdsn3W6tP21ttYbWyhYCe0/stTbj2ph/\nf39bsH1Bnp/j8Xhsf+x+S0xJtH/P/Lc1G9nMVuxdka1Mn5/7WJn+ZezVea9aYkqiLd612NqMa3PG\n7c5pzf41Nv+v+Xlu79d/fJ1nyIr3foz80W754hZLTEnMt1xUdJTd+eWd1uSjJkY4tnr/aq8/IzEl\n0d5Z+I5VH1zdes/tbZFHIy3o3aCzrXo2fx37y+77+j6r+35du/yjy637zO72xvw3rPrg6vbaz695\n9beX1R+H/rBXfnrFBvw2IM8yaZ40W7xrsSWnJtvolaOt5pCadt/X9xXb72VKWoqNXzPeGgxrYLdN\nvM1m/znb0jxprnKU4HBoD2wFooA+uSwPBuYC64A/gH/nsR7z7++f7VvwrRNvte4zu+e5wzczO5l0\n0q4bd509P+f5fP8TD8YdNMKxigMq2uUfXW77YvdZmifNqgysYrWG1rL/zPuPNRzWMNf3JqcmW4fJ\nHeyeaffY1aOutoemP5RvnbJqOKyhbT+23ZbuXmo1h9S0KRumePW+L9d/aV2/7Wpmzi9t77m9rdnI\nZlZxQEXbfmy7XfHxFdZ7bm9LSUux1p+2tsW7FpuZWeTRSKv3QT0b8NsAe2LWE/bJik9c656yYYpd\nPepq6z6zu5V/p7y1GdfG7pl2T65HGbuP77Zle5ZlTkdFR1mj4Y282oYM0zZOs4oDKmaGysp9K+34\nqeNmZnb45GG7cfyNRjiZRytScPHJ8fbErCfskg8vsSoDq9iOmB25lvN4PDZq5SgLHhJsQ5cMtaTU\nJOs4paN9t+U7rz5n9f7V1nRkU+s4paNtP7Y9c50BbwXYqZRTZ70dKWkpNnDRQKsxuIa9FfGWxSfH\n259H/zT//v72yIxHbM+JPYVe9/Blw63X7F65LttwcIO1GdfGKg6oaCFDQyxsQljm3+zu47u9/oz8\nfoejoqMyj2aySkhOsITkhGzzFmxfYM1GNrMbx9+Y+bedF0poOPgD24BQIAAnAJrlKBMODEx/HQxE\nA2VzWZetO7Au20bfPeVuIxxbtW9V5ryYUzGZrz0ej9339X32yIxHvE73WVtnZfvW8eSsJ23C2gnm\n8Xis0ruVLDohOlt5j8djD8942DpO6WgpaSmWlJpUoG8SLce0tLGrxlrNITXtv1H/9fp9ETsi7Mbx\nN5rH47H/nfO/dt246yw6IdquG3ed1Rhcw8J/Dc8s+9h3j9m41eMs8mik1X2/rn225jMzM3t/6fv2\n/Jzns6134rqJVvu92vbANw/YnV/eaZ+s+MT6Luib6zeS3MQmxlrFARXzXJ6UmpS54zcz+2zNZ1b3\n/bq2ZPcSCxwQaFM2TLGyb5W1YcuG2c6YnXbJh5fYG/PfsOafNC/Qt9eTSSet09RO+X4TLC0ij0ba\nFR9fYd2md7PYxFhrOaalLd+z3FUuLinOukzrYi3HtLStR7Zmzu85u6d9uPzDfD/D4/HYB0s/sOAh\nwTZ5w2TX8trv1T6rHbeZ2c6YndZmXBtr90U72xmzM9uyrH/3hTVt4zS79+t7s81LTUu1QYsGWfCQ\nYBuzaoxFJ0TbL9t/yfwbb/1pa1u6e2m+6117YK1tPrzZ3pj/hvn398/WtmanA6/c2+Ws5+ye2ZbN\n/2u+EY49OetJM3O288lZT1q9D+rZzC0zvdrXUELD4Tqco4IMr6X/ZNUD+Dj99SXAn3msy7XRh08e\ntlZjW9mPkT+amdmfR/80wrH1B9ebmbPzazW2VZF8YzFzduQZ35KTU5Nt6sapNnjxYGv9aWtXsnvr\n9km3W8BbAfb1H18X6H1R0VEWOjzUhiweYs0/aZ75xzFq5SgbvHhwtrIDFw20btO7Wf0P6mcGg5lz\nKu22ibdlTs/YPMPqvF8n1z4Zb3k8HqvwToVcjzKSU5Ptjkl32H1f32dmZmNXjbX6H9S3yKORZmbW\n/JPmFjwk2F77+TW74bMb7NIPL7Xhy4abmVmnqZ1sxuYZXtXhVMopu+WLW+zBbx+0ygMr25H4I4Xe\nnvNNbGKsDVk8JDPMl+5earWG1rIxq8Zk7kg6TO5g32/9Ptv7dh/fbVePutoen/m466h38OLB9vJP\nL1vEjgjbcmSLHTp5KNvypNQke3zm49ZidIvMo4Wcrhp1VWa/gLcy+vAGLx5sn67+1C5+72IbumSo\n119UCmrhzoV2w2c3mJlzaiw6Idpun3S7/ePzf+R5pPXPr/7p+tvdEbPDohOizePx2NAlQy14SLAF\nvBVgHad0tJs+v8m++uOrzLL7Y/db2/Ftrd0X7WziuonW+tPWZua06avzXrU679extxe+bc0/aW5L\ndi+x+h/Ut2d/fDbzVLQ3OItwyO1belGpC2S9nmovcG2OMp8CC4D9QCXgfm9XXrNiTa4MuZIDJw/g\nMQ+Pf/84AJFHIynnX453F73LyqdWUr5s+bPaiAxNgpsQeTSSNvXaMGDRAN7+7W2qla/Gmh5rMq9A\nKqjmIc35R4N/cN+V9xXofXUr1WXPiT18sPwDVj61kqrlqwLwzDXPuMo2DW7K67+8zlthb/F4i8cz\n57eo3YJV+1eRkpbCH4f/oMePPfhvt//SrGbOgzvv+fn5EVIxhEPxh7ik3CUAxCXFMWz5MHYc38Hx\nxOP8cfgP5v01j74RfVnUfRGNqzcGYNCtgwitGkq18tUYtGQQA28ZyAttXgCgYZWG7Dy+M9/P3h+3\nn24zulHGrwzBgcF82eVLWo5tyZ4TewgODC70Np1Le2P3UjuoNv5l/Av0vphTMRhGx6kdWbZnGR2b\ndGT3id10m9GNifdM5M7L7swsW6tiLQ7FH8qc3h6znXZftKNX6168fN3L+Pn5ZVt3gyoNGLR4EB+v\n/JjE1ERevu5l3rv9PQASUhLoPK0zFcpW4LfuvxFULijX+gUHBud72XZWUzdO5f9+/T885iEwIJDy\nZcsTfSqaL7t8yS2X3FKgdimIi4Mu5uDJg0zZOIUnvn+C6hWq0/VvXRl06yDKlsl9N1m3Ul32xTn3\nSSWlJvHrzl+5/5v76dy0MwBbjmxh1VOrCPAPoHZQbfr+2pdNhzcxLHYYY1aPIS45jmdaPsOb/3iT\nhJQEevzYgwNxB+g6vSsVy1Vk/TPrqVq+KkOWDKHLV134rNNn3H353T5rg5x8GQ7eJNYbOKebwoBL\ngZ+BqwHXTQzh4eGZr8PCwggLC6N2UG0OnjzI+LXjSUlLoXeb3kQdi2L478PpH9afRtUaFcV2ANCk\nRhMioyNZtX8Vo1aNYvXTqylbpiwNqjQo9Doz/sgKqkJABa4MuZJhdwyjXuV6+Za9scGNvNvuXV5r\nm/2g7eKgi2lcvTFzoubw8ryX+ejOj2hZp2Wh6pNzvQfiDnBJtUswM7rP6s7v+36ndlBtFjy2gNDh\noTw0/SFmPDAjMxgAOlzWIfN1ZK9ILq9xeeZ0wyoN2XViV56fmZCSQKepnbisxmUEBQTx8V0f41/G\nnzqV6rAvbh8tarc46+3ytQ2HNtB2fFu+uverbDvz/ByJP8LxxOOEfRHG8cTjPHb1Y9SvXJ8Pln3A\nd1u/Y+YDM7mhwQ3Z3lOrYi0OnTzE8r3LmbpxKrMiZ/GfG/7Dc62ey/UzWlzcgtZ1W/N5588ZvWp0\n5v9DQkoCHad2pF7leozvND7fQAsODOZI/JEzbs9naz6jX0Q/+t3Uj7JlyhKXHMez1zxLgH+AV+1x\nNmoH1WbH8R28Nv81Zj4wE8No37h9vu+pV7keu0/s5oX/vsDEDRMpX7Y8E7tM5IFvH6Bzk8781v03\nAgMCM8tfGXIlPX7sQf3K9Xm97evUq1wvM/CCygXRum5rmn7clOeueY532r2T2aZj7h5Dm3ptvNqf\nRUREEBERUfiGOEfakP200uu4O6XnAFl/e38BrsllXbkeMn24/EN7ZMYjFjI0xNYeWGtjVo2xeh/U\nsxajWxT54efsP2fbDZ/dYK3GtrIJaycU6boLoyiukhixfISVf6e8PTHriSKokeOluS/Z7ZNuty/W\nfWEf/f6RXTP2mmyX/fX4oUeBLhk0cy4tvmvyXXkuf/r7p63rt11dbfLkrCdt9MrRBd+Ic2xf7D6r\n/0F9azyisY1ZNcar9+w6vssufu9i8+/vb8OWDbOf//rZPB6PjVg+wgLeCrCftv2U6/uGLRtmHSZ3\nsFpDa1mXaV28/jwzpyP0xvE3Zp4ifGj6Q15dztlzdk8bsXyEmVmep3mnbZxmdd6vY38e/dPr+hQl\nj8djr/z0SoE6mCdvmGwV3qlg1427ziJ2RGT2hURFR+W6/9kXu8+enPVktn63rOKT423J7iWF24A8\nUEL7HMoCf+F0SJcj9w7pD4B+6a9r4Zx6qp7LunLd8K//+Nr8+/vbwzMeNrPTHTi/bP+lSBvYzLlq\nIHBAoF376bU+O+95rnk8Hvtuy3de3+/gjdX7V5tfuJ+FDA2xGoNrZPYpnI2dMTut1tBauQbid1u+\ns0bDG+V6Hrbvgr7Wd0Hfs/78onAq5VSuO4Wk1CRrNbaVDfhtgPVd0Nf6/drPVWbW1ll2MO5g5nRi\nSqK1GtvKhiwe4rof5vip465LjrOasmGKEU6BQiHDjpgdVu+DetZzdk9r/2V7r+7/MTN7e+Hb9spP\nr9ifR/+0aoOq2cp9K7Mtn7dtnoUMDcnsLzxfRB6NtIemP2RxSXHFXZU8UULDAeBOIBLnqqXX0+f1\nSP8B5wqlH4D1wEYgr1v6ct3wRbsWmX9/f4uKjjIzp0PuTFdWnI1BiwYVuGOtNNp7Yq+1GtuqyP4v\nPB6Plelfxggn2w7pQNwBqzW0Vp7ftsasGmOEc8YrSnKK2BFhHSZ3KLJr2D0ej5V9q6zdOvFW17I+\nP/exu6fcbR6Px0avHG1Pff9UtuWLdi2yMv3LWP+I/pnzes7uaV2mdSlU/Q7EHch2YUJBpKSlWNm3\nylrTkU3z/Pabm8W7Fluzkc2s2chm1nBYw2w3cP517C8LGRpiC3cuLFSdJH+U4HAoKrlueFJqks3b\nNu8cN7d4Izk1uUjXV/f9ukY4diDuQOa8btO7WZ+f++T5nhmbZ2S7FNAbp1JO2eUfXW6VB1a2X3f8\nWuB6pqalur5Rj1s9zggn80qtDAu2L7Da79XOvALoh8gf7K7Jd1l8crwNXzbcTiadtEs/vNT6/NzH\n/vbJ38zMuUP+kg8vKZLLNwvjxf++WOBTPylpKVZtUDV79sdnbfTK0VZlYBX7dtO3tv7gert61NWZ\np5yk6FFaw0FKj7ikOGs2slnmaZSFOxda/Q/q53tKLDUt1d785U179LtHvf6cfr/2sy7TutjUjVOt\nxuAa9vT3Txeonj1n97TgIcG26/guM3OGRKk5pKYNWjTI7ph0R2a5hOQECx0eanP+nJM5b/X+1UY4\n1vyT5kY4dueXd9rDMx62NE+aVR9c3bYc2WK136t9xhufSqLNhzdbcmqyrdm/xgjHAt4KsCoDq9iz\nPz6ru999iBJ6KatIkQkqF0RIxRCOJBzBzHh53ssMuW0IFctVzPM9/mX8uTn0Zt7+7W2vPiMqOoqR\nK0ay7pl11Ktcj9suuY3QD0MZdfcoyvideYzKDYc28M3mb7i8xuWsPbCWBlUa8NJPL/Hvv/+bsNAw\npm+Znll28JLBtKrTKtuVSXUr1QXgiRZPsPHwRmZHzWbTc5so41eG6+tfT6epnbin6T2uK5DOBxmX\nSLeo3YKjrx7l83Wfc+8V9xJaNbR4KyZ5UjjIeaNmxZocTTjKzK0zSfWkcv+VZ74tpmHVhqzYt4Lp\nm6fzryv+lW/Z//v1/3ipzUuZlwfXCKxBlYuqsPXoVq6oeQUpaSl5XlZpZrw490X63dSPTYc3sevE\nLhbvXszi3YvZ/Nxm9sft59ipY4AzbPzIFSNZ02NNtnXUCqrF7hd3U79KfY6dOsbL171M9QrO9Rlt\n67dlzYE1DLxloOuzzzc1AmvwyvUavbekO++G7JbSK7hCMIfjD9M3oi/v3PyOV9/m61euz6nUU9z7\nzb35llt7YC0Ldy3kxTYvZptfr3I9rvzkShbvXswlIy5h0+FN7D6xO9vT+E4knuDJ75/kSMIRnm75\nNA2rNmTX8V28ueBN+of1p2K5ilSvUJ2YxBgAXv35VV5s82Ku98jUr1IfgOoVqme7IfG5Vs+x8N8L\nqVK+yhm3WaQoKBzkvFGzYk0mrp9IQJmAbDfN5eeishfRsEpDQiqG5Fvurd/e4vW2r7tOU9WuVBuA\n7rO6szd2L/O3z6f5qOZsOLQBgMTURPpF9GP/yf3MuH9G5o2RX2/+mkMnD/HwVQ8DULV8VU4knmDd\nwXUs2bOE3tf1LtC2V7qoUrabBkV8TeEg542agTVZuX9lrsM85GfeI/OofFHlPJdvPbqVJbuX8OT/\nPOla9tW9XzHln1PYdmwb3Zp3I3xhOLFJseyP2w/A83OeZ+GuhYy6axSX1bgMcO7o3hu7l/5h/TOH\nXvAv409QuSBe/flVXr7u5Wx3zoqUROpzkPNGzYo1qVe5nld9DVkFBgSSkJKQ5/L3lr5Hz1Y9c91h\nl/Mvx92X383nnT/n4qCLmbxxMq3qtOJQ/CEOxB3g2y3fEvV8VLbxm66oeQW9WvVyjZlVvUJ11h5Y\ny3cPfFeg+osUB++/fhUvy3qOV0qnuKQ4dh7fSfNazQv0vmOnjnHpiEuJ6RPjWhadEM2lIy5l2/9u\nO+MAffHJ8czYMoONhzdSo0INYpNiOZF0gpEdRnpVj2vGXsM/m/2TN258o0D1Fyms9CPsQu3ndeQg\n541KF1UqcDAAVAyomOeRwxfrv6BTk05ejdxasVxFHrn6Ed5f+j5/xfzFN5u/YdkTy7yux6i7RvG3\nkL95XV6kOKnPQS545fzLkepJJdWTmjnPzDAzxqweQ4+WPfJ5t1vG6aUWF7coUCdxq7qtCj28u8i5\npnCQC56fn1+2foeInRHcOulWFu1eRECZAK6vf32B1lcrqBaxSbE83fJpX1RXpERQOEipkDUcJqyb\nwO97f2fS+kk8evWjBbryCZwH4NQOqk2nJp18UVWREkEd0lIqNPqwEb88+gu1g2pT54M6+OHHqdRT\nRPaKLNQDm+KT4/MdukOkJFCHtMgZZHRKz46azTV1riGgTAAnk08W+kl+Cga50CkcpFTIOK309aav\nefDKBzEsc9wiEXFTOEipEBgQSMypGOb9NY+RHUaecTgNkdJOHdJSKgQGBDI7ajZXhlypYBDxgsJB\nSoXAgECm/jGVzk06F3dVRM4LCgcpNY4mHNXlpyJeUjhIqbA3di/VylejaXDT4q6KyHlB9zlIqRBz\nKoYKARUoX7Z8cVdF5Jw5m/scFA4iIheoswkHnVYSEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiI\niIiLr8OhPbAViAL65FEmDFgL/AFE+Lg+IiLiBV/e5+APRAK3AvuAlUBXYEuWMlWBJcAdwF4gGDia\ny7p0n4OISAH56mE/L+eYNuAIsBjY4cW6WwPbgJ3p09OAzmQPh4eA6TjBALkHg4iInGP5nVaqBARl\n+akEtALm4hwBnEldYE+W6b3p87K6DKgO/AqsAh7xqtYiIuJT+R05hOcxvzrwCzD1DOv25jxQAPA/\nwC1AILAMWI7TR5G9MuGnqxMWFkZYWJgXqxcRKT0iIiKIiIgoknUVts9hLdDiDGXa4ARM+/Tp1wEP\nMDhLmT5ABU4H0TicI5Nvc6xLfQ4iIgV0rsdWuhmI8aLcKpzTRqFAOeAB4PscZWYBbXE6rwOBa4HN\nhaiTiIgUofxOK23MZV414ADwqBfrTgV6AT/h7Pw/w+mM7pG+fAzOZa5zgQ04RxWfonAQESl2+R1u\nhOaYNiAaOOmz2uRNp5VERApIz3MQEREXPc9BRESKlMJBRERcFA4iIuKicBAREReFg4iIuCgcRETE\nReEgIiIuCgcREXFROIiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiI\niIiLwkFERFwUDiIi4qJwEBERF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuLi\n63BoD2wFooA++ZRrBaQC//RxfURExAu+DAd/YCROQFwBdAWa5VFuMDAX8PNhfURExEu+DIfWwDZg\nJ5ACTAM651LueeBb4IgP6yIiIgXgy3CoC+zJMr03fV7OMp2BUenT5sP6iIiIl8r6cN3e7OiHA6+l\nl/Ujn9NK4eHhma/DwsIICws7u9qJiFxgIiIiiIiIKJJ1+fIcfxsgHKfPAeB1wIPTv5Bhe5Y6BAMJ\nwFPA9znWZWY6qBARKQg/Pz8o5H7el+FQFogEbgH2AytwOqW35FH+c+AHYEYuyxQOIiIFdDbh4MvT\nSqlAL+BhZPYoAAAMJklEQVQnnCuSPsMJhh7py8f48LNFROQsnC+XjurIQUSkgM7myEF3SIuIiIvC\nQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiIiIiLwkFERFwUDiIi4qJwEBER\nF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKicBAREReFg4iIuCgcRETEpWxx\nV0BEJDfVq1cnJiamuKtxXqhWrRrHjh0r0nX6FenafMfMrLjrICLnkJ+fH/q7905ebeXn5weF3M/r\ntJKIiLgoHERExEXhICIiLgoHERFxORfh0B7YCkQBfXJZ3g1YD2wAlgBXnYM6iYictddff50PP/zQ\n55/zww8/8OCDD/r8c7LydTj4AyNxAuIKoCvQLEeZ7cA/cELhbWCsj+skInLWjhw5wqRJk3jmmWcA\n2Lx5M9dccw3Vq1enatWq3HDDDSxevNjrdXXt2pW6detStWpV2rZty4oVKzKXd+zYkU2bNrFx40af\nbEtufB0OrYFtwE4gBZgGdM5RZhlwIv3170A9H9dJROSsTZgwgbvuuouLLroIgLp16/LNN98QHR1N\nTEwMDz74IPfee69X6zp58iTXXnsta9asISYmhscee4y77rqL+Pj4zDJdu3Zl7Nhz993Z1+FQF9iT\nZXpv+ry8PAHM8WmNRESKwNy5c7npppsyp6tUqUKjRo3w8/MjLS2NMmXKULt2ba/W1ahRI1588UVq\n1aqFn58fTz31FMnJyfz555+ZZcLCwpg9e3aRb0defH2HdEHuYLkZeBy4wUd1EREpMhs3bqRJkyau\n+VWrViU+Pp46deqwYMGCQq173bp1JCcn07hx48x5TZs2ZefOnZw8eZKgoKBC19tbvg6HfUD9LNP1\ncY4ecroK+BSnbyLX++XDw8MzX4eFhREWFlZUdRSR85RfEY3xUJgbsY8fP06lSpVynZ+QkED//v25\n7777WL16dcadyl6JjY3lkUceITw8PNv6M14fP348z3CIiIggIiKiYBuSB18Pn1EWiARuAfYDK3A6\npbdkKdMAWAA8DCzPYz0aPkOklCnpw2fUqlWLOXPm0LJly1yXmxmVKlVi6dKlXHWVdxdhnjp1ivbt\n29O0aVPGjBmTbdmxY8cIDg4mNjbWFQ7n4/AZqUAv4CdgM/AVTjD0SP8B6AtUA0YBa3ECRESkRLvq\nqquIjIzMc3laWhoej4fAwECv1peUlMQ999xDgwYNXMEAsGXLFkJDQ8/JKSU4N/c5/BdoAjQGBqbP\nG5P+A/AkUANokf7T+hzUSUTkrHTo0IGFCxdmTs+fP59169aRlpZGbGwsvXv3pkmTJpn9BhMmTKBR\no0a5rislJYV7772XwMBAJkyYkGuZhQsX0qFDhyLfjrzoDmkRkUJ49NFHmTNnDomJiYDTF9C1a1eq\nVq1KkyZNOHLkCN9//31m+T179tC2bdtc17V06VJmz57Nzz//TNWqValUqRKVKlViyZIlmWWmTZtG\njx49cn2/L2jIbhEpkUp6nwPAm2++SUhICC+88MIZy95xxx2MGDEi1yuczuSHH35g8uTJTJs2Ldfl\nvuhzUDiISIl0PoRDSXE+dkiLiMh5SOEgIiIuCgcREXFROIiIiIvCQUREXBQOIiLionAQEREXhYOI\nSCHpMaEiIpJNzseELl++nNtuu40aNWoQEhLC/fffz8GDB71eV2l7TKiIyAUp52NCjx8/zjPPPMOu\nXbvYtWsXlSpVonv37l6tqyQ+JlTDZ4hIiVTSh8+45ZZbeOKJJ3jooYdyXb5mzRrCwsKIjY0t1Pqr\nVKlCREQELVq0AJzB+R5++GG2b9/uKqvhM0RESoi8HhOa4bfffuNvf/tbodZ9pseEngu+fkyoiIjP\n+PUvmpMf1q/gRyh5PSYUYMOGDbz99tvZhuz21tk8JrQoKRxE5LxVmJ16UalWrRpxcXGu+du2baND\nhw6MGDGCG264oUDrPHXqFB07duT666+nT58+2ZZlfFbVqlULX+kC0GklEZFCyO0xobt27eK2226j\nb9++dOvWrUDrK42PCRURueDkfEzovn37aNeuHb169eLpp592lddjQkVESoGcjwkdN24cO3bsyOwr\nqFSpEpUrV84sr8eE+oYuZRUpZUr6paygx4SWBAoHkVLmfAiHkkL3OYiIyDmhcBAREReFg4iIuCgc\nRETEReEgIiIuGj5DREqkatWqZVxtI2dQrVq1Il+nr1u+PTAc8AfGAYNzKTMCuBNIAP4NrM2ljC5l\nFREpoJJ6Kas/MBInIK4AugLNcpTpADQGLgOeBkb5sD4XhIiIiOKuQomhtjhNbXGa2qJo+DIcWgPb\ngJ1ACjAN6JyjTCfgi/TXvwNVgVo+rNN5T7/4p6ktTlNbnKa2KBq+DIe6wJ4s03vT552pTD0f1klE\nRLzgy3DwtpMg5/kwdS6IiBQzX3ZItwHCcfocAF4HPGTvlB4NROCccgLYCtwEHMqxrm3ApT6qp4jI\nheovnH7dEqUsTsVCgXLAOnLvkJ6T/roNsPxcVU5ERIrPnUAkzjf/19Pn9Uj/yTAyffl64H/Oae1E\nREREROTC0B6nHyIK6HOGsheC8Tj9LRuzzKsO/Az8CczDudw3w+s4bbMVuP0c1fFcqQ/8CmwC/gD+\nN31+aWyP8jiXeq8DNgMD0+eXxrbI4I9zw+wP6dOltS12Ahtw2mJF+rwLvi38cU43hQIB5N5ncaG5\nEWhB9nAYAvwn/XUfYFD66ytw2iQAp422cWGNlXUx8Pf010E4pyebUXrbIzD937I4fXNtKb1tAdAb\nmAx8nz5dWttiB04YZHXBt8V1wNws06+l/1zoQskeDls5fWPgxenT4HwDyHo0NRenU/9CNRO4FbVH\nILASuJLS2xb1gPnAzZw+ciitbbEDqJFjXpG0RUlODW9uoisNanH60t5DnP5Pr4PTJhku5PYJxTmi\n+p3S2x5lcL71HeL06bbS2hbDgFdxLo3PUFrbwnCCchXwVPq8ImmLkjwqq26GczPyb5cLsc2CgOnA\nC0BcjmWlqT08OKfZqgA/4Xxrzqq0tMXdwGGcc+xheZQpLW0BcANwAKiJ08+wNcfyQrdFST5y2IfT\nKZmhPtlTr7Q4hHNoCFAb5w8D3O1TL33ehSQAJxgm4ZxWgtLdHgAngNlAS0pnW1yPMybbDmAq0A7n\n96M0tgU4wQBwBPgOZ0y7C74tvLmJ7kIUirtDOuM84Wu4O5fKAY1w2upCGvzeD5iIcwohq9LYHsGc\nvuKkAvAbcAulsy2yuonTfQ6lsS0CgUrprysCS3CuQCoVbZHbTXQXsqnAfiAZp7+lO86VCPPJ/bK0\nN3DaZitwxzmtqe+1xTmVsg7nFMJanEubS2N7NAfW4LTFBpzz7VA62yKrmzh9tVJpbItGOL8T63Au\n987YR5bGthARERERERERERERERERERERERERERE5n5xM/7ch0LWI1/1GjuklRbx+ERHxkYwxmcI4\nfUett840/ljO8Z5EROQ8kbEDXw4cx7nb+gWcscWG4jwkZT3wdHq5MGARMIvTA5nNxBn58g9Oj345\nCEhNX9+k9HkZRyl+6eveiHNX8/1Z1h0BfANsAb7MUs9BOKOtrk9/r4iI+FBGOGQdiwecMHgz/fVF\nOM9JCMXZgZ/EOQ2VoVr6vxVwdvgZ0zmPHDKm/4UzdIEfEALswhkMLQwnoOqkL1uKM7JmDbKPqFnZ\n240T8YWSPCqrSFHLOcjY7cCjON/8l+OMSdM4fdkKnB16hhdwxrBZhjOy5WVn+Ky2wBScIZEPAwuB\nVunTK3DG0LL0dTbECYxE4DOgC3CqoBsnUpQUDlLa9cJ5kFAL4FKcAcsA4rOUCcMZBbUNzjMV1uI8\n1zk/hjuMMsbOT8oyLw1naPI0nOGWv8V5ZsFcRIqRwkFKkzhOD3EMzkNznuN0p/PlnH5Wc1aVgRic\nb/ZNyf5oxRRy77ReBDyA8zdWE/gHzhFDXkMkV8QZPfO/OM9HvvqMWyPiQyX5SXAiRSXjG/t6nG/o\n64DPgRE4fQxrcHbah3FO6eR8etZc4BlgM84Q8suyLBuL0+G8Gngky/u+w3kO+vr0ea+mr78Z7qdv\nGU5ozcI5IvEDXir01oqIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIhcyP4/S815E0SFY3EAAAAA\nSUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f688c070be0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2), (3,2)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Active Reinforcement Learning\n",
    "\n",
    "Unlike Passive Reinforcement Learning in Active Reinforcement Learning we are not bound by a policy pi and we need to select our actions. In other words the agent needs to learn an optimal policy. The fundamental tradeoff the agent needs to face is that of exploration vs. exploitation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### QLearning Agent\n",
    "\n",
    "The QLearningAgent class in the rl module implements the Agent Program described in **Fig 21.8** of the AIMA Book. In Q-Learning the agent learns an action-value function Q which gives the utility of taking a given action in a particular state. Q-Learning does not required a transition model and hence is a model free method. Let us look into the source before we see some usage examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%psource QLearningAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a mdp similar to the PassiveTDAgent.\n",
    "\n",
    " Let us use the same GridMDP object we used above. **Figure 17.1 (sequential_decision_environment)** is similar to **Figure 21.1** but has some discounting as **gamma = 0.9**. The class also implements an exploration function **f** which returns fixed **Rplus** untill agent has visited state, action **Ne** number of times. This is the same as the one defined on page **842** of the book. The method **actions_in_state** returns actions possible in given state. It is useful when applying max and argmax operations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**. We use **Rplus = 2** and **Ne = 5** as defined on page 843. **Fig 21.7**  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "q_agent = QLearningAgent(sequential_decision_environment, Ne=5, Rplus=2, \n",
    "                         alpha=lambda n: 60./(59+n))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to try out the q_agent we make use of the **run_single_trial** function in rl.py (which was also used above). Let us use **200** iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for i in range(200):\n",
    "    run_single_trial(q_agent,sequential_decision_environment)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let us see the Q Values. The keys are state-action pairs. Where differnt actions correspond according to:\n",
    "\n",
    "north = (0, 1)\n",
    "south = (0,-1)\n",
    "west = (-1, 0)\n",
    "east = (1, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(float,\n",
       "            {((0, 0), (-1, 0)): -0.07323076923076924,\n",
       "             ((0, 0), (0, -1)): -0.0759999433406361,\n",
       "             ((0, 0), (0, 1)): 0.2244371077466747,\n",
       "             ((0, 0), (1, 0)): -0.07085714285714287,\n",
       "             ((0, 1), (-1, 0)): -0.04883916667786259,\n",
       "             ((0, 1), (0, -1)): -0.05252175603090532,\n",
       "             ((0, 1), (0, 1)): 0.3396752416362625,\n",
       "             ((0, 1), (1, 0)): -0.07323076923076924,\n",
       "             ((0, 2), (-1, 0)): -0.05158410382845185,\n",
       "             ((0, 2), (0, -1)): -0.04733337973118637,\n",
       "             ((0, 2), (0, 1)): -0.048398095611170026,\n",
       "             ((0, 2), (1, 0)): 0.4729172313717893,\n",
       "             ((1, 0), (-1, 0)): 0.14857758363326573,\n",
       "             ((1, 0), (0, -1)): -0.0759999433406361,\n",
       "             ((1, 0), (0, 1)): -0.07695450531425811,\n",
       "             ((1, 0), (1, 0)): -0.09719395035017139,\n",
       "             ((1, 2), (-1, 0)): 0.21593724199115555,\n",
       "             ((1, 2), (0, -1)): 0.26570820298073916,\n",
       "             ((1, 2), (0, 1)): 0.19612684250448048,\n",
       "             ((1, 2), (1, 0)): 0.6105607273543103,\n",
       "             ((2, 0), (-1, 0)): 0.06795076480003,\n",
       "             ((2, 0), (0, -1)): -0.11306695825372484,\n",
       "             ((2, 0), (0, 1)): -0.105596446586541,\n",
       "             ((2, 0), (1, 0)): -0.10409381636745853,\n",
       "             ((2, 1), (-1, 0)): -0.0383184014263534,\n",
       "             ((2, 1), (0, -1)): -0.7913059177862865,\n",
       "             ((2, 1), (0, 1)): -0.7672970392961057,\n",
       "             ((2, 1), (1, 0)): -0.8402721538112866,\n",
       "             ((2, 2), (-1, 0)): 0.2351847866756862,\n",
       "             ((2, 2), (0, -1)): 0.24909509983624728,\n",
       "             ((2, 2), (0, 1)): 0.25112211666264095,\n",
       "             ((2, 2), (1, 0)): 0.7743960998734626,\n",
       "             ((3, 0), (-1, 0)): -0.1037923159515085,\n",
       "             ((3, 0), (0, -1)): -0.07807333741195537,\n",
       "             ((3, 0), (0, 1)): -0.9374064176172849,\n",
       "             ((3, 0), (1, 0)): -0.07323076923076924,\n",
       "             ((3, 1), None): -1,\n",
       "             ((3, 2), None): 1})"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "q_agent.Q"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Utility **U** of each state is related to **Q** by the following equation.\n",
    "\n",
    "**U (s) = max <sub>a</sub> Q(s, a)**\n",
    "\n",
    "Let us convert the Q Values above into U estimates.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "U = defaultdict(lambda: -1000.) # Very Large Negative Value for Comparison see below.\n",
    "for state_action, value in q_agent.Q.items():\n",
    "    state, action = state_action\n",
    "    if U[state] < value:\n",
    "                U[state] = value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "defaultdict(<function __main__.<lambda>>,\n",
       "            {(0, 0): 0.2244371077466747,\n",
       "             (0, 1): 0.3396752416362625,\n",
       "             (0, 2): 0.4729172313717893,\n",
       "             (1, 0): 0.14857758363326573,\n",
       "             (1, 2): 0.6105607273543103,\n",
       "             (2, 0): 0.06795076480003,\n",
       "             (2, 1): -0.0383184014263534,\n",
       "             (2, 2): 0.7743960998734626,\n",
       "             (3, 0): -0.07323076923076924,\n",
       "             (3, 1): -1,\n",
       "             (3, 2): 1})"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "U"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us finally compare these estimates to value_iteration results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n"
     ]
    }
   ],
   "source": [
    "print(value_iteration(sequential_decision_environment))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}