rl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Reinforcement Learning\n",
    "\n",
    "This IPy notebook acts as supporting material for **Chapter 21 Reinforcement Learning** of the book* Artificial Intelligence: A Modern Approach*. This notebook makes use of the implementations in rl.py module. We also make use of implementation of MDPs in the mdp.py module to test our agents. It might be helpful if you have already gone through the IPy notebook dealing with Markov decision process. Let us import everything from the rl module. It might be helpful to view the source of some of our implementations. Please refer to the Introductory IPy file for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from rl import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Review\n",
    "Before we start playing with the actual implementations let us review a couple of things about RL.\n",
    "\n",
    "1. Reinforcement Learning is concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. \n",
    "\n",
    "2. Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).\n",
    "\n",
    "-- Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n",
    "\n",
    "In summary we have a sequence of state action transitions with rewards associated with some states. Our goal is to find the optimal policy (pi) which tells us what action to take in each state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passive Reinforcement Learning\n",
    "\n",
    "In passive Reinforcement Learning the agent follows a fixed policy and tries to learn the Reward function and the Transition model (if it is not aware of that).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passive Temporal Difference Agent\n",
    "\n",
    "The PassiveTDAgent class in the rl module implements the Agent Program (notice the usage of word Program) described in **Fig 21.4** of the AIMA Book. PassiveTDAgent uses temporal differences to learn utility estimates. In simple terms we learn the difference between the states and backup the values to previous states while following a fixed policy.  Let us look into the source before we see some usage examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%psource PassiveTDAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a policy(pi) and a mdp whose utility of states will be estimated. Let us import a GridMDP object from the mdp module. **Figure 17.1 (sequential_decision_environment)** is similar to **Figure 21.1** but has some discounting as **gamma = 0.9**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from mdp import sequential_decision_environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mdp.GridMDP at 0x103bbbb70>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequential_decision_environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Figure 17.1 (sequential_decision_environment)** is a GridMDP object and is similar to the grid shown in **Figure 21.1**. The rewards in the terminal states are **+1** and **-1** and **-0.04** in rest of the states. <img src=\"files/images/mdp.png\"> Now we define a policy similar to **Fig 21.1** in the book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Action Directions\n",
    "north = (0, 1)\n",
    "south = (0,-1)\n",
    "west = (-1, 0)\n",
    "east = (1, 0)\n",
    "\n",
    "policy = {\n",
    "    (0, 2): east,  (1, 2): east,  (2, 2): east,   (3, 2): None,\n",
    "    (0, 1): north,                (2, 1): north,  (3, 1): None,\n",
    "    (0, 0): north, (1, 0): west,  (2, 0): west,   (3, 0): west, \n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "our_agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rl module also has a simple implementation to simulate iterations. The function is called **run_single_trial**. Now we can try our implementation. We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import value_iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values calculated by value iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n"
     ]
    }
   ],
   "source": [
    "print(value_iteration(sequential_decision_environment))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the values estimated by our agent after **200 trials**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.40645681855595944, (1, 2): 0.7159329142704773, (3, 2): 1, (0, 0): 0.2886341019228155, (2, 0): 0.0, (3, 0): 0.0, (1, 0): 0.20553303981983, (3, 1): -1, (2, 2): 0.8560486321875528, (2, 1): 0.606857283945162, (0, 2): 0.5612793239398001}\n"
     ]
    }
   ],
   "source": [
    "for i in range(200):\n",
    "    run_single_trial(our_agent,sequential_decision_environment)\n",
    "print(our_agent.U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also explore how these estimates vary with time by using plots similar to **Fig 21.5a**. To do so we define a function to help us with the same. We will first enable matplotlib using the inline backend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def graph_utility_estimates(agent_program, mdp, no_of_iterations, states_to_graph):\n",
    "    graphs = {state:[] for state in states_to_graph}\n",
    "    for iteration in range(1,no_of_iterations+1):\n",
    "        run_single_trial(agent_program, mdp)\n",
    "        for state in states_to_graph:\n",
    "            graphs[state].append((iteration, agent_program.U[state]))\n",
    "    for state, value in graphs.items():\n",
    "        state_x, state_y = zip(*value)\n",
    "        plt.plot(state_x, state_y, label=str(state))\n",
    "    plt.ylim([0,1.2])\n",
    "    plt.legend(loc='lower right')\n",
    "    plt.xlabel('Iterations')\n",
    "    plt.ylabel('U')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a plot of state (2,2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFOW5NvD7YTYYdlBAEUUFEdzABCSKMq4g7sSogOIW\nxYXoZ1RQj8ugOTloTqIQjwpuJEThxBAiKLtxlMMqsiM7DILsy8giy8A83x9PF13T1T3T3dM13TN9\n/65rrpnuqql+u7q67nqXqhJVBRERkVuNZBeAiIhSD8OBiIg8GA5EROTBcCAiIg+GAxEReTAciIjI\nw9dwEJEPRGSbiCyJML2PiCwSkcUiMkNEzvezPEREFB2/aw4fAuhexvR1AC5T1fMBvAJguM/lISKi\nKPgaDqo6HcCeMqbPUtUfAw/nADjFz/IQEVF0UqnP4X4AE5JdCCIiAjKTXQAAEJHLAdwH4JJkl4WI\niFIgHAKd0O8C6K6qYZugRIQXgCIiioOqSjz/l9RmJRE5FcA/AdypqmvKmldV+aOKl156KellSJUf\nrguuC66Lsn8qwteag4iMAtAVwAkishHASwCyAEBVhwF4EUBDAG+LCAAUq2onP8tERETl8zUcVLVX\nOdN/DeDXfpaBiIhil0qjlSgKeXl5yS5CyuC6COK6COK6SAypaLtUZRARrQrlJCJKJSICrYod0kRE\nlJoYDkRE5MFwICIiD4YDERF5MByIiMiD4UBERB4MByIi8mA4EBGRB8OBiIg8GA5EROTBcCAiIg+G\nAxEReTAciIjIg+FAREQeDAciIvJgOBARkQfDgYiIPBgORETkwXAgIiIPhgMREXkwHIiIyIPhQERE\nHgwHIiLyYDgQEZEHw4GIiDwYDkRE5OFrOIjIByKyTUSWlDHPUBFZLSKLRKSDn+UhIqLo+F1z+BBA\n90gTRaQHgFaq2hrAgwDe9rk8REQUBV/DQVWnA9hTxiw3AvhLYN45ABqISFM/y0REROVLdp9DcwAb\nXY83ATglSWUhIqKAZIcDAEjIY01KKYiI6LjMJL/+DwBauB6fEnjOIz8///jfeXl5yMvL87NcRERV\nTkFBAQoKChKyLFH190BdRFoCGK+q54WZ1gNAf1XtISKdAbyhqp3DzKd+l5OIqLoREahqaOtMVHyt\nOYjIKABdAZwgIhsBvAQgCwBUdZiqThCRHiKyBsABAPf6WR4iIoqO7zWHRGDNgYgodhWpOaRChzQR\nEaUYhgMREXkwHIiIyIPhQEREHgwHIiLyYDgQEZEHw4GIiDwYDkRE5MFwICIiD4YDERF5MByIiMiD\n4UBERB4MByIi8mA4EBGRB8OBiIg8GA5EROTBcCAiIg+GAxEReTAciIjIg+FAREQeDAciIvLITHYB\nonX77dHPm5UFDB0KLFgAjBkD9OwJXHwx8MEHwLJlwM03A926Rb+8w4eBceOAb78F9uwBDh0CjhwB\nVG2681sE+N3vgFatol92eY4cAWbMABYtAk47DbjllviWsWABsGQJsGULsHu3vafiYptWXAxkZgJD\nhgANG8a+/GPHgO++A1atAr7/HigqAg4csJ/Dh4GjR22+QYOA008v/b9FRcDy5cD69UCPHkCDBrG/\nPpXvyBH7bE49FcjOTnZpqCqoMuHQs2f08w4eDLz+OvDuu0DLlraDGjbMdlYNGwKjR0cOh6NHgXPP\nBT79FGjTxnakl18OnHACkJdnO7eaNe0LViOk3vWnP9lOPJpw2L0baNSo7Hneew949lngjDPs5y9/\nsXD46Sfg3nstlP7+98j/f+QI8NprwJtvAs2aAe3bAyefDJxySvA9ZGdbMLzwArB2LfDzn0de3qFD\nFgS1a9vjTZuAP/wB+NvfbP20a2c7n0aNgKZNbb6cHCAjA/jrX4Fp04AHHrDP429/A4YPt2Bo29aW\nlZUF/OpX5a+7cFSBNWuA1q3j+//qZvNmYOJEW+eLFwPr1tnn9/77wH33xb68deuAzz6zA4whQ4Dc\n3PDzqdrrTZoEzJkD9O4N3Hprxd5LLDZvtoOpuXPtYC4vD3jxxeD0HTtsO/nFLyqvTFWWqqb8jxUz\neg8/rJqdrfrGG6pTpqiedJJq8+aqBw+qTp2qev75qvfdF/5/33pLFbD/U1V97DHVBx5QLSkp/3V7\n9VL96KPy55s61V7j8OHI8wwdqtqmjerSpfZ41y7VevVUDx1SvfRS1TPOUG3Vyvt/TjkPHlS98krV\n665TXbas/DJ16aJaUBB5+tq1VuY+fezxxImqTZuqPvWUamFh+csfPFj1iSdUV69W7dBB9aqrVCdM\nsPejqnrnnap/+Uv5yzl4UPXYsdLPbdpkywNUv//e1sHw4aoLF9r03btVv/kmOP/Ikar/+Z/lv1ZZ\ndu5Uvflm1VNPrdhyEm3WLPvMGzZUvf121Q8+UF2wwNbb88+r5udHv6wjR1Q//FD1Zz9TbdJE9de/\nVj355OB6ddu1yz7j005TPfNM1f79bdu45hr7jMeOteX5Ye9e+65fdJG97xtvVP3d7+xzbtbMpr/5\npm3j9evbdlJc7E9ZUk1g3xnffjfef6zMn1jD4fXX7Z3Nnh3cqd1xh01bvdoeuxe5ZUtwh3PZZTbt\n889V9++3jW3jxuhe9557VN9/v+x5SkpUzz3XXmPXrvDzfP+9ve7ataX/r1491d/+VrVHD9vZtW9f\n+v9277blbtyo+uSTqjfdpHr0aHRl79bNdtbhFBXZF/6WW2z9TJ9uO4vp06NbtqrquHGqZ51lQT10\nqDdsH3hA9Z13yl7Gli1Wjg8+CD63cqUt8+WX7T2MGaN6992qNWvaOigsVD37bNXWre01X37Z1lHN\nmtGXXVV1zx7VH3+0v+fOtdf85S/tM/FTUZG9nwULbBudPz/8fAcO2AFP8+aqw4bZ41Bvv23rORrT\np9vByRVXqE6aFNyZXnWVHRg4jh61HXPjxqp9+6rOm1e67HXq2IEMYO8hkQ4etM/zhBNUb7tNdfJk\nbwC1bWuf0S9/ad/pQ4dU69a1sqUDhkOICRNUMzNVf/rJNpaMDNXXXrNphw55wwFQ/fvfbedat67t\nZMaOVf3qK9XOnaN/3X79rObhmDbNdmhuX36pes45dgT2/ffhl/P446pPP+19/swzrazr1lmNom3b\n0tMHDrTpb72l2qiR6o4d0Ze9Z0/VTz4JP+3hh1UffNDeS26ufdnHjYt+2arBUH799fDTH3ss8jRV\n2xF06KB6wQV2BKuqun27leXdd+3x88+rtmhhNat//Uu1Y0dbR6++ajupl15SPe881X//23Z80dq1\nK3iAMXeu6okn2vvfvt3Wc0WsWlV6h+q2ebMF2xlnqA4YYLU2wLsD3L7d1kufPqr79kV+rU8/tVpF\neYYOtSPuMWO80+66KxjOe/aoXnutHZEvXx5+Wb//vdXCO3a0g7VojR9vtY5INfZFi+xgo2dP27Yi\nmT9fdc2a0s+ddJLVNlPB4cNWo77sMn+Wz3AIsX27VWsdZ5xhTTmOrKxgOJSU2N///KdVyTt2VP3V\nr1RHj1b9wx9Uf/Ob6F/38cdL7+AA22G7PfWUVe3POiv8F+rYMftirlzpneYOtdWr7X05Dh2yndb1\n16s2aGA7k1jcdVf4Zp316+2ocOfO4LqKJTAdx47ZOo30ZR840HYkkbz4ojXjzJ5tO0JVa8Z74ong\nPJ9/bkfOW7cGa1F33WWv2bWrHUGuXau6YYPqKadEV+6SEtseOna0z+y004I7zf37VWvVim454ezc\nqdqypWpenreJce9eC8OXX7aDlMxMew+hNdmiItULL1R97rnymz7nzbPa5syZqj/8EH6ewYMtkNat\nCz99wABrkisqsvI98kh0zUWXXKL69dflz6dqte9mzezzGzDAW8P+/HOrLYwcGd3yQrVurbpiRfhp\nx47ZdnXzzfEtOxbr1tk6vPxye6+hzaUV9fnnDIdyLVhQuo3x4YeDO9lt2+zvyZNtY+vVy47A/vpX\n2ynEsgEOGKD6X/9lf+/fb8sdPrz0PB06WJW9Q4fwR4wzZ1rNIpyxY4NfsO+/tx2hY9w4O/p45x17\n3Wj6GdxCaz2OZ58tvQO+9VbVGTNiW3Y08vNVX3gh/LRt22ynuGGD1SByc60G1rx56eaTkhKrLTre\neit4JD16tOpnn9nfO3fa8qIxapRqu3a2s65d22pQjmPHVEWi648KVVJiR/GPPmo129BN/L77VO+9\n1+Y7fNiaE4uKVH/+c9U5c4Lz9eplNaloyrB5s72H2rWDNWn3UfXo0daHUtZR9euvqz70kAXVo49G\n/96vuKL0AVoko0ZZ7W/FCus3AKxcjilT7CBo1qzoXjecCy8M/907dszeG+Btsk00p2lyyBBbh6ec\nYgdibl98UXp7C8fps3P+7tPHvssvv2yfZUXCocqMVqqI9u1LP371VWDkSGDvXuDMM+25I0dsFEOr\nVsAPPwQfn3129K+Tk2MjcQBg3jz7rRqc/tNPwIoVQKdONpLnwAHvMmbMsNFR4dx8c/jXAoApU2wo\naIcOwHnn2cihWIQrj6qtp4kTg8998klsy41WrVo2kiSct98GbrvNRkIBNqKqTx/gmWdKj5oRseU4\nHn44+Ld7KHSkde/2+utA/fo28m34cKBuXWDsWBsS7ahRw0ZYHT5so79iMXGibV9jx9p6fustG1Kc\nlQV88QUwdaoNuxaxEWV//KP938kn2/YJAP/4h43IWbDA5itPkya2XXfsCKxeDRQU2La2aZNN79/f\nXrd588jLOOkkG0F36aU2XDya1wW822s4334L/OY3NsKqTRv7fPfts+8MYKPp+vSx4emdO0f3uuHU\nqQPs3+99Pj8fWLjQPptBg+Jffqhly+z76KyrRYuA66+30ZQ33mjPnX02MGKEjRrMyLDP4fbbbRsb\nNiz8ct9800Yjbthgn+uNN9r2/9lnNkpzzhz7vOLl60lwItJdRFaIyGoRGRhmen0RGS8iC0VkqYjc\n42d5HJmZNmR148bgRuIOh+xs25C3bIlt5dasGfwCLFtmvw8dCk5fudLCKDs78g7qm2/sy1ue0C/b\n1KnA1Vdb8Hz7bfRldtSubeHltmiRvc4558S+vFjVqgUcPOh9vqTEhl+6d/RXXWXBfs898b1WTo59\n/s75F6H27bOdw+OP2xDdrl3t+auvDg7jdeTmetdbeY4eBX77W9vhZ2UB//M/9jq7d1tQ/L//Zzve\nunW9/9u8uQ0VPXgQePJJWzeRhpWGysiw81Hy821bfPRRe37lSuChhywcQg+kQp1+uq2DESO8Q7nL\nUl44HD4M3HWXve8LLgiW97zzrMxHjtgBwgsvWDBVRLhw+PRTe0+ffmrDvmP9TLdsAS68ENi5s/Tz\nkybZe1i40B5v3Qpcdx3w5z8Hg8Ep06BBwPTpwNKlNgR47Fjgxx/Db6cff2zBsG+fHSzceadtL2PG\n2L7gq6/sfVSEb+EgIhkA3gTQHUA7AL1EpG3IbI8CWKqq7QHkAfijiPhem3HCoajIHtepY0dt69db\n4ubk2Maxc6eN149WTk4wDAoL7bc7HFassDH9QGLC4cgR+3vvXgs650uVlRV9mR25ud7yTJxotZFo\njw4rwh0Os2YFn581y06Mc94bAPTta0fa4Xae0RApu/YwerTtgLKz7ei1rPcfKdQc27bZOQJun31m\n54L06BF8rnFjYNcu25lkZAA33RR+ec2a2c69Sxc7Gu3SJfJrh9OqlZ0L8vXX9pr3328B8913ViMo\nT6dO9j05+eTYXtc54Ipk8GAr2x13lH6+XTsr25//DJx4ogVYRdWpYztVx65dQL9+wP/+r9WuYg38\nkhILtiVL7MexcSNw99227c6bZ+cI9e5t6/y220ovY8gQC42vv7bP/k9/soOSRo28gbN4sR24fP65\n1aL79rXQ+egj23Y6drRab0X5uSPuBGCNqhYCgIiMBnATgOWueUoA1Av8XQ/ALlWNcDyXOBkZFg57\n9gDXXmsfwJEj9rhxY9uQN22y5zNjWEPuo6PCQvsShoaD00xVp45353TwoJ3Ec9ZZ5b9WdnbwLO3F\ni+3EvYyM6MsaqnbtYPOCY9Ys2+grQ61a9oUsLLSmm59+sufGjvWeFX7aaRUvlxMO4b5EI0ZYKHz4\noR3Rl1dudzg884z9HjzYfg8YYDu3jz+2o9UOHYB33rGakDt0nHB47TXg6acjB5IT/PPnlw7RWLRo\nAdSrZ82r06cDAwfaCZfRnjkdTyi7D2ZCbdtmO8eFC73vu107O4h75RVrJknEgUpozeGpp6wJxzkx\nLlwtuizDhtk20LevfdZ5efb8I49YmDVoYOFw4IAFifukPMeppwK//CXw4IO2HGf7btIE2L49WAs4\ndMiCZcgQq5G0b28nxy5caOs4kfwMh+YANroebwJwUcg8bwIYLyKbAdQFEJKn/qhRw35277YPztnR\n/vijPc7JsTbZWI+O3M1K69dbELjDYdUqOzoAwh+5Fhbaji+anXyNGhZcxcW2YZTXHFCe0C+EKjB7\ntjV5VAZnJ/vFF/Z42zarxU2dam3+iRap5rB5s5213a1bdDvL0HB49VVrOhw82HZAn35qO/6zzrIj\n0qVLbUcxdmzp5TRuDMycaU08oUeVbk8+aU1AhYUWNPHIyLADgbp17TuQiLAtT1nNSv/930CvXsE+\npdD/GzrUvk9t2iSmLO5wWLgQmDzZ1rsjlprDrl3ASy9ZP8mXX1oY7NplTbHr1lkzz5IlFvpjxlif\nYqTv96WX2vf49deDzzVpYn0Kq1ZZkO/YYaHQu7dN79/fPrtY91XR8DMctPxZ0B3AfFW9XETOBDBV\nRC5Q1X2hM+bn5x//Oy8vD3lOPMcpM9Oqaw0aWJoXF9sRSv36tlMoLIy9Mye0WemKK0qHw+bNdukK\nwHZOoe2e69bZZTKi5VTVly61DaYiQpuVCgttHbVoUbHlRsvZyf773/Z4yxb7EhcWAj/7WeJfL9z6\nB4Dx44Hu3aM/inbvSNautd9O0+G0aVbFv/NO6x8580yr+vfuXbrjHLBwGD7cLjVRVrOgc8mTeK6B\n5eYc/ffoYR3afl9vKVI47NplzVqLFkX+33iuJ1YWp1lp6lQ7+BkwoHRtKJZweOUVu+TL+edb0w5g\nB1UjR1oNMTvbtt+ePa2/qKyAa9XKmpXdTjwReP55ax1o0sRaPJYuDU4/7bTS8xcUFKCgoCC6wpfD\nz3D4AYB719ICVntwuwfAfwGAqq4VkfUA2gCYF7owdzgkgjsc9u0LXiSudm37QDdsiP3IzPkCHDpk\n/QCnnhrsmAZs43Gqh+GOXGMNB+f11q8v3bkVj9Caw7JltsFXFiccZs+25ritW632cPHFsTXtRStS\nzWHKlNh2Ru6aw9Sp9vnt3m2PJ060ZsvevW17eOstYMKE8CNhGje2z78yr0MEWDNNRYMmGpH6HD76\nyMK4sg5CAAuHKVOseadZM2DUqNLTs7ODAxbK2vZ27LBrhjnf8Wuusea+n/3M+guuvDI475Ah8ZW1\nSROrcRQUWBCcdFLZzUehB86DKjDsys/RSvMAtBaRliKSDeB2AONC5vkewFUAICJNYcEQ0n3nj8xM\n+3AbNrSNYccOqzWI2Mrfsyf2L43TrLRrl/VX1KpVuubgZziEXu00VqHlcS6IV1lyc23UxY4d9sXa\nssWOoi4KbYhMkNq1rYrvHmpcUmKjPCINJQ7HHQ7TplmH6s6dttwJE+zIPCvL2pHXr7ejvssu8y6n\ncWMb/HDJJRV7X6kqtObw3XfBCwHef3/llqVuXdvZPvOMdeqG1uJEbHsMHWjw8celm2+HDrUmQHcL\nwwUX2A49UUNhL7jAytm1a3CwTGXxLRwCHcv9AUwG8B2A/1XV5SLST0T6BWZ7BcDFIrIYwDQAA1R1\nt19lcnPXHJxwcC4X7VSx69SJbZlOs9Lu3fZlr1nTdhiDBtlR+eHDwQ7QcOGwaVOw2Sna1zt40C7F\n3LJlbGUNFdqstHx57OdKVEStWhaOF15o1e+tW+0o7MIL/Xm9nBz70rlHEi1dagcEZY3zD+WEw7Fj\n1uZ8++0WbIMGWb+Q04xQr56FRNeu4b/gF1xgQ0srMqgglbk7pIuLrX29Y0fr54sljBOhTh373J5+\nOvL2Fa5p6fe/t+avXbvse/7229aZ7Vajhh0EhDsAiMf991vTVTL4OmxUVScCmBjy3DDX31sAxHBn\nhcRxh0NWVulwcL688YTD4cPBy3HXrGnD2fLz7cixWbPgaItw4bBjR+xDZwsLbYcWevQTq9q17TLH\nM2bY0evy5fFd2jleTvnbt7cjsXnz7HwNP/obAGs2DDVnTuyXcnZ2IpMm2Wd3zjnB8yTuuy/4eYtY\n08m114ZfTo8epYe2Vjc5ORYEgA06aN3a1sXJJ8d2vkQinH++fSfLumR+aDhs2GAHLPXr2/Y5aJA1\nO4e7PH9Fv4upIm3vBJeZaUcA7pqDc1Tv1BxCT3gqT82awZqDEw6OrVtLVz/DhcP27VYljVZOjo16\nieUs7kjatLGjnY8+ssfr18fWxFVRzheqbVvbic6da22+fozCAILtxMXFweeiPcfErVYtC5Xrr7c2\nZufIv3dvOwPW7YUXyh6JVJ25+xzGjbNhmy+9ZPf3qGwXXmid0GUJ7YMbNcr6gzp3tm3m5ZftnirV\nWVqHw+7dVt1PZLOS0+fgNCs5tm4tXSvIyiq9YwIsHE48MbbXe+stO2GmorKz7azdjRst4PbsqfgZ\nlrFwh0O7djY6o21b/07Ae+wx++3+DObNiy8cCgrssg7uIYht2niPiHv1Kv+8ierK3ecwebJ1Qqcy\nd81B1W5O1aePnRPy2GP2fhI9iirVpMW1lcJxwiEnJ9is5NQcEtms5CgqKt3BnZFh7dSOo0et2l3e\n3eFC3wOQuFFFLVpYOGzaZO3ulVndd8KhXTtbB7m5iakRRfLGG9b57ITD0aN2kmKs67JWLeur6Ncv\n+HmMHx88EYqM0+ewdq310Zx7brJLVLbcXDvvpH59O1jat8+aW2vUsP6hbt2qT/NRJGkbDhkZdmSQ\nk2NHzXv3epuVYg2HspqV9u61Wor79d3h4IxwiqVD0mmWSlQnZosW1rm9YYN3/LTfata0e3w7R9Zn\nn524k54icdfe1qyxJqxor1PkcALU3Vdx/fWJKV914hw4TZ5sQz4r45IsFfHTT3aNqzFjrLy33BL8\nrEMvfVJdpXWzUklJ8KQiIHgijFNziLXPwd2sFGs4xNqkBAQ7+BLlhBPsqG758vBnq/pJpHQb7p13\nlh4n7gd3OCxbFt8FBu+7z85v8KvjvLpw+hymTLGdbapzTkZbuTJ4jbF0k9bhANhG65yR6tQUKtrn\nEE3NoUYNCydHKoSDiA2lnTkztiG1fnjiCf93uM7lR4D4w+Gss+wqsVQ2Z5j39OlVo8ntoouA556z\ng6XFixM3NLUqSdtmJXc4hIZBRZuV9u+3Wog7HPbtKz3sLbTm4Fz0LxZ798Y2fzROPtnGcsc6pLMq\nctccVq2yS3KTP3JyrEaam+vfCLREmj3bfk+ebAMzYr1fR3XAmkMZzUqxhoNzRvSBA/YlcDaojIzy\nm5VCp0ejpCTxnWLNmllVOpYhtVWVOxzWrg3e+IkSLyfH+nU6dUp2SWLTvn3ky6dXd6w5lNGsFGuf\nQ0aGLWv3bvtfZ8edmxtdOMRzKeRYA6U8TZtauWI5Ga+qysoK3kjFudET+cM54Kpq4TB8eOp3nvsl\n7cPBGa0EBMPB2ZBjHbkCWCjs2GH/m5NjbZVLl/pTc7juuopfjTWUc25DOtUcfvzR2pbTIRCTxfmO\nVbVwqOyzt1NJWoeDiO2kw/U51K4d34bh3DQnN9f+f/x4O2egvHDYty/2HfJnn8VevvI4O8h0Cgfn\ngofpeoRYGXJybP1yVFfVkba5mJlpISDibVaqX99uvxePOnXsjEqn1uHcSKi80Urx1Bz80KyZBVcs\nJ+NVVU44JOO8jnTTrJldMiMVtnGKTtqHA+CtOdSrZ5dEiIfTT+H8dt9lLtHNSn5o2tSG1KZDddoJ\nh40bK/d+AumoaVPgk0+SXQqKRVo3K0UKh4pwQsEZqeTczrOoqHSHsxMOx47Z2Zjxdkgn2rnnAn/8\nY7JLUTkYDkSRpcHxYXiZmcGOZ6dZKRE7Z2eUkvvI2xku674VoxMOU6bYLSRTpeZQs2bw/rTVHcOB\nKDLWHBDckSfiRJc6dbxDYEODAQiGw549FgypEg7phOFAFFla1xycHbZzIlQiRqvUru0dApud7d3x\nO+Hg3CGO4VD5nHCI9Q58ROmA4YDEjswJFw45OeHDoaTExtcfPmxDWRkOlcsJh9AbMRERwwGAXYHU\nfaP5ioi25lCjRrDm4FwvPhEd4hQ95z4e2dnxnfBIVJ2ldTiEu9F7RUXqcwjt7HaalQ4etJ/iYn/K\nQ5FlZVmTUmXe8Y6oqkjrcAjtJE6EWJqVnJpDUZF1hvMM3cqVlWWd0bxsBpEXwyHBYu2QPnjQwqG6\n33IwFTnhwJoDkReHsiZYu3Y2PNWtvNFKxcUMh2TIyrLOaIYDkRfDIcG6dLEft0jh4IxWAtLzZiLJ\n5pz8yGYlIi82K1WCcH0O7tFKAGsOyeCEQ6y3ZyVKB2kdDpU1Oqi8PgeA4ZAMTjjEentWonSQ1uFQ\nWTWHnj29Nzlx9zkADIdkcMLhhBOSWw6iVJS2fQ7um/z4rW/f8K/vrjmwz6HyseZAFJmvNQcR6S4i\nK0RktYgMjDBPnogsEJGlIlLgZ3ncKrPmEA5rDsnHmgNRZL7VHEQkA8CbAK4C8AOAb0RknKoud83T\nAMD/AOimqptEpNK+prfcAhw4UFmv5uVcCdYpA8Oh8jn3EWfNgcgrYjiIyJMhTymAHQD+T1XXR7Hs\nTgDWqGphYHmjAdwEYLlrnt4AxqjqJgBQ1Z3RF71izjqrsl4psho1gP377bakDIfKd+iQ/U5mDZIo\nVZXVrFQXQB3XT10AHQFMEpFeUSy7OYCNrsebAs+5tQbQSES+FJF5InJX1CWvBjIyLBwaNmSfQzLs\n3ZvsEhClrog1B1XND/e8iDQC8AWAUeUsO5rrnGYBuBDAlQByAcwSkdmqujp0xvz8YHHy8vKQl5cX\nxeJTW0ZRPVIzAAAPPklEQVSGHb02aMCaQzIwkKm6KSgoQEFBQUKWJRrHtapFZIGqdihnns4A8lW1\ne+DxswBKVPVV1zwDAdRygkhE3gMwSVX/EbIsjaecqa52beuQ7tIF6NoV+N3vkl2i9FJSAmzbxns5\nUPUlIlDVuC7pGfNoJRG5HMCecmcE5gFoLSItRSQbwO0AxoXM8ymALiKSISK5AC4C8F2sZaqqjh2z\n3zVrsuaQDDVqMBiIIimrQ3pJmKcbAtgCIMzI/dJU9aiI9AcwGUAGgPdVdbmI9AtMH6aqK0RkEoDF\nAEoAvKuqaRcOOTls4iCi1BKxWUlEWoY8pQB2qep+n8sUrizVslnJuX9Dz57AlVcCjzyS3PIQUfVS\nkWalsjqkC+MuEcWkVi3eppKIUktcHdKVrTrXHDIygFWr7ESs+vWTXSIiqk4qUnNgOCSRiJ2Adfhw\nsktCRNVRpY5WosTKyEh2CYiIvBgOScZwIKJUxHBIssy0vWg6EaUyhkOSseZARKmI4ZBkDAciSkUM\nhyRjOBBRKmI4JBnDgYhSEcMhyRgORJSKGA5JxnAgolTEcEgyhgMRpSKGQ5IxHIgoFTEckozhQESp\niOGQZAwHIkpFDIckYzgQUSpiOCQZr61ERKmI4ZBkrDkQUSpiOCQZw4GIUhHDIckYDkSUihgOScZw\nIKJUxHBIMoYDEaUihkOScbQSEaUi7pqSaNo04Mwzk10KIiIvUdVkl6FcIqJVoZxERKlERKCqEs//\nslmJiIg8GA5EROThaziISHcRWSEiq0VkYBnzdRSRoyLS08/yEBFRdHwLBxHJAPAmgO4A2gHoJSJt\nI8z3KoBJAOJqGyMiosTys+bQCcAaVS1U1WIAowHcFGa+3wD4B4AdPpaFiIhi4Gc4NAew0fV4U+C5\n40SkOSww3g48xSFJREQpwM/zHKLZ0b8B4BlVVRERlNGslJ+ff/zvvLw85OXlVbR8RETVSkFBAQoK\nChKyLN/OcxCRzgDyVbV74PGzAEpU9VXXPOsQDIQTAPwE4AFVHReyLJ7nQEQUo4qc5+BnOGQCWAng\nSgCbAcwF0EtVl0eY/0MA41X1n2GmMRyIiGJUkXDwrVlJVY+KSH8AkwFkAHhfVZeLSL/A9GF+vTYR\nEVUML59BRFRN8fIZRESUUAwHIiLyYDgQEZEHw4GIiDwYDkRE5MFwICIiD4YDERF5MByIiMiD4UBE\nRB4MByIi8mA4EBGRB8OBiIg8GA5EROTBcCAiIg+GAxEReTAciIjIg+FAREQeDAciIvJgOBARkQfD\ngYiIPBgORETkwXAgIiIPhgMREXkwHIiIyIPhQEREHgwHIiLyYDgQEZEHw4GIiDx8DwcR6S4iK0Rk\ntYgMDDO9j4gsEpHFIjJDRM73u0xERFQ2UVX/Fi6SAWAlgKsA/ADgGwC9VHW5a55fAPhOVX8Uke4A\n8lW1c8hy1M9yEhFVRyICVZV4/tfvmkMnAGtUtVBViwGMBnCTewZVnaWqPwYezgFwis9lIiKicvgd\nDs0BbHQ93hR4LpL7AUzwtURERFSuTJ+XH3VbkIhcDuA+AJf4VxwiIoqG3+HwA4AWrsctYLWHUgKd\n0O8C6K6qe8ItKD8///jfeXl5yMvLS2Q5iYiqvIKCAhQUFCRkWX53SGfCOqSvBLAZwFx4O6RPBfBv\nAHeq6uwIy2GHNBFRjCrSIe1rzUFVj4pIfwCTAWQAeF9Vl4tIv8D0YQBeBNAQwNsiAgDFqtrJz3IR\nEVHZfK05JAprDkREsUvloaxERFQFMRyIiMiD4UBERB4MByIi8vD7PAciorgERi9SlBI9aIfhQEQp\ni6MUo+NHkLJZiYiIPBgORETkwXAgIiIPhgMREXkwHIiI4vTss89iyJAhvr/O+PHjcccdd/j+Om4M\nByKiOOzYsQMjR47EQw89BACYPXs2rr76ajRu3BhNmjTBbbfdhq1bt0a9rF69eqF58+Zo0KABunTp\ngrlz5x6ffsMNN2DZsmVYsmSJL+8lHIYDEVEcRowYgeuuuw45OTkAgKKiIjz00EPYsGEDNmzYgLp1\n6+Lee++Naln79+/HRRddhPnz52PPnj24++67cd111+HAgQPH5+nVqxeGDx/uy3sJh1dlJaKUFLii\naLKLEdGVV16J+++/H7179w47ff78+cjLy8PevXvjWn79+vVRUFCADh06AABmzpyJO++8E+vWrfPM\nG2ld8aqsRESVbMmSJWjTpk3E6V9//TXOPffcuJa9cOFCHDlyBK1atTr+3Nlnn43CwkLs378/rmXG\nimdIE1GVlagTg+OpoBQVFaFu3bphpy1evBivvPIKxo0bF/Ny9+7di7vuugv5+fmllu/8XVRUhDp1\n6sRe4BgxHIioykpmq1PDhg2xb98+z/Nr1qxBjx49MHToUFxyySUxLfPgwYO44YYbcPHFF2PgwIGl\npjmv1aBBg/gLHQM2KxERxeH888/HypUrSz23YcMGXH311XjxxRfRp0+fmJZ3+PBh3HzzzTj11FMx\nbNgwz/Tly5ejZcuWlVJrABgORERx6dGjB7766qvjj3/44QdcccUV6N+/Px588EHP/CNGjMDpp58e\ndlnFxcW49dZbkZubixEjRoSd56uvvkKPHj0SUvZoMByIiOLQt29fTJgwAYcOHQIAvPfee1i/fv3x\nvoK6deuiXr16x+ffuHEjunTpEnZZM2fOxOeff46pU6eiQYMGx/9/xowZx+cZPXo0+vXr5++bcuFQ\nViJKSak+lBUA/uM//gNNmjTB448/Xu683bp1w9ChQ8sc4RTJ+PHj8dFHH2H06NFhp/sxlJXhQEQp\nqSqEQ6rgeQ5ERFQpGA5EROTBcCAiIg+GAxEReTAciIjIg5fPIKKUJYm6eBLFzNdwEJHuAN4AkAHg\nPVV9Ncw8QwFcC+AnAPeo6gI/y0REVQOHsSaXb81KIpIB4E0A3QG0A9BLRNqGzNMDQCtVbQ3gQQBv\n+1We6qKgoCDZRUgZXBdBXBdBXBeJ4WefQycAa1S1UFWLAYwGcFPIPDcC+AsAqOocAA1EpKmPZary\nuOEHcV0EcV0EcV0khp/h0BzARtfjTYHnypvnFB/LREREUfAzHKJtMAztcWJDIxFRkvl2bSUR6Qwg\nX1W7Bx4/C6DE3SktIu8AKFDV0YHHKwB0VdVtIctiYBARxSHeayv5OVppHoDWItISwGYAtwPoFTLP\nOAD9AYwOhElRaDAA8b85IiKKj2/hoKpHRaQ/gMmwoazvq+pyEekXmD5MVSeISA8RWQPgAIB7/SoP\nERFFr0pcspuIiCpXSl8+Q0S6i8gKEVktIgPL/4+qTUQ+EJFtIrLE9VwjEZkqIqtEZIqINHBNezaw\nblaIyDXJKbU/RKSFiHwpIstEZKmIPBZ4Pu3Wh4jUFJE5IrIwsC7yA8+n3bpwiEiGiCwQkfGBx2m5\nLkSkUEQWB9bF3MBziVkXqpqSP7CmqDUAWgLIArAQQNtkl8vn93wpgA4Alrieew3AgMDfAwEMDvzd\nLrBOsgLraA2AGsl+DwlcF80AtA/8XQfASgBt03h95AZ+ZwKYDeCidF0Xgff4WwAfARgXeJyW6wLA\negCNQp5LyLpI5ZpDNCfRVSuqOh3AnpCnj58oGPh9c+DvmwCMUtViVS2EfdCdKqOclUFVt6rqwsDf\n+wEsh50Xk67r46fAn9mwL7ciTdeFiJwCoAeA9xAcCp+W6yIgdMBOQtZFKodDNCfRpYOmGhzBtQ2A\ncwb5ybB14qi26ycw4q0DgDlI0/UhIjVEZCHsPU9R1blI03UB4HUATwMocT2XrutCAUwTkXki8kDg\nuYSsi1S+Kit7ykOoqpZzzke1W2ciUgfAGACPq+o+91U602l9qGoJgPYiUh/AWBE5N2R6WqwLEbke\nwHZVXSAieeHmSZd1EXCJqm4RkRMBTA2cK3ZcRdZFKtccfgDQwvW4BUqnXrrYJiLNAEBETgKwPfB8\n6Po5JfBctSEiWbBgGKmq/wo8nbbrAwBU9UcAXwLohvRcFxcDuFFE1gMYBeAKERmJ9FwXUNUtgd87\nAIyFNRMlZF2kcjgcP4lORLJhJ9GNS3KZkmEcgLsDf98N4F+u5+8QkWwROR1AawBzk1A+X4hVEd4H\n8J2qvuGalHbrQ0ROcEaciEgtAFfD+mDSbl2o6nOq2kJVTwdwB4B/q+pdSMN1ISK5IlI38HdtANcA\nWIJErYtk97aX0xN/LWyUyhoAzya7PJXwfkfBziY/AutvuRdAIwDTAKwCMAVAA9f8zwXWzQoA3ZJd\n/gSviy6wNuWFABYEfrqn4/oAcB6A+QAWBb78zweeT7t1EbJeuiI4Wint1gWA0wPfj4UAljr7yESt\nC54ER0REHqncrEREREnCcCAiIg+GAxEReTAciIjIg+FAREQeDAciIvJgOFC1JyL7A79PE5HQuxFW\ndNnPhTyekcjlEyULw4HSgXMyz+kAesfyjyJS3vXHni31QqqXxLJ8olTFcKB0MhjApYEbozweuNLp\nH0RkrogsEpEHAUBE8kRkuoh8CjvzFCLyr8CVL5c6V78UkcEAagWWNzLwnFNLkcCylwRuxnKba9kF\nIvKJiCwXkb85hRORwWI3N1okIn+o1DVDFCKVr8pKlGgDATylqjcAQCAMilS1k4jkAPg/EZkSmLcD\ngHNUdUPg8b2quidwbaO5IvIPVX1GRB5V1Q6u13BqKT0BXADgfAAnAvhGRL4OTGsPu/HKFgAzROQS\n2OUMblbVswNlq+fD+yeKGmsOlE5Cb4pyDYC+IrIAdne1RgBaBabNdQUDADweuJ/CLNiVLVuX81pd\nAHysZjuArwB0hIXHXFXdrHbtmoUATgNQBOCQiLwvIrcAOBj3uyRKAIYDpbv+qtoh8HOmqk4LPH/A\nmSFw34ArAXRW1fawiwDWLGe5Cm8YObWKw67njgHIUtVjsMst/wPA9QAmxfNmiBKF4UDpZB+Auq7H\nkwE84nQ6i8hZIpIb5v/qAdijqodE5GwAnV3TiiN0Wk8HcHugX+NEAJfBLo8cGhgIvHZt2NUzJ8Lu\nj3xBjO+NKKHY50DpwDliXwTgWKB56EMAQ2E3Wp8fuH/EdgC3BOZ3X654EoCHROQ72CXkZ7mmDQew\nWES+VbuvgAKAqo4VkV8EXlMBPK2q20WkLbx331JYaH0qIjVhAfJEQt45UZx4yW4iIvJgsxIREXkw\nHIiIyIPhQEREHgwHIiLyYDgQEZEHw4GIiDwYDkRE5MFwICIij/8Pvox5c+Ssb50AAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x103cea9e8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "agent = PassiveTDAgent(policy, sequential_decision_environment, alpha=lambda n: 60./(59+n))\n",
    "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to plot multiple states on the same plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX+BvD3SxOREiGhS1MpigVRYAUlYkMUd1UsoOKy\nFlBBXV0X2d2fRAGRpvQmTTooRZpREEKv0qX3FjohCaTP+/sjw5jkpkzKkEDez/PkYe69Z84995DM\nO/fcZiQhIiKSVIHcboCIiOQ9CgcREXFQOIiIiIPCQUREHBQOIiLioHAQEREHn4aDmY0xs1Nmti2N\n5a+a2RYz22pmK83sbl+2R0REvOPrPYexAJqns/wAgIdJ3g2gG4CRPm6PiIh4wafhQHI5gAvpLF9N\n8qJ7ci2Ayr5sj4iIeCcvHXN4E8CC3G6EiIgAhXK7AQBgZo8A+AeAxrndFhERyQPh4D4I/R2A5iRT\nHYIyM90ASkQkC0haVt6Xq8NKZlYFwEwAr5Hcl15Zkvoh0bVr11xvQ175UV+oL9QX6f9kh0/3HMxs\nCoCmAPzN7CiArgAKAwDJEQA+B3AzgGFmBgBxJBv4sk0iIpIxn4YDydYZLH8LwFu+bIOIiGReXjpb\nSbwQGBiY203IM9QXf1Jf/El9kTMsu+NSV4OZ8Vpop4hIXmJm4LV4QFpERPImhYOIiDgoHERExEHh\nICIiDgoHERFxUDiIiIiDwkFERBwUDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcVA4iIiI\ng8JBREQcFA4iIuKgcBAREQeFg4iIOCgcRETEQeEgIiIOCgcREXFQOIiIiIPCQUREHBQOIiLioHAQ\nEREHhYOIiDj4NBzMbIyZnTKzbemUGWhme81si5nV82V7RETEO77ecxgLoHlaC82sBYDbSN4O4B0A\nw3zcHhER8YJPw4HkcgAX0inyLIDv3WXXAvAzs3K+bJOIiGQst485VAJwNMn0MQCVc6ktIiLiltvh\nAACWYpq50goREfEolMvrPw7gliTTld3zHMyCkkwFun9ERORPIe6f7DPSt1/UzawagLkk70plWQsA\nHUm2MLNGAPqTbJRKOfq6nSIi1xszA8mUozNe8emeg5lNAdAUgL+ZHQXQFUBhACA5guQCM2thZvsA\nXALQzpftERER7/h8zyEnaM9BRCTzsrPnkBcOSIuISB6jcBAREQeFg4iIOCgcRETEQeEgIiIOCgcR\nEXFQOIiIiIPCQUREHBQOIiLioHAQEREHhYOIiDgoHERExEHhICIiDgoHERFxUDiIiIiDwkFERBwU\nDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcbjuwyHeFY8EV4JP10ESBy4cQGhEqE/XIyJy\ntRTK7Qb4wrHwY5i/Zz7m752PxQcX4+/3/h2DWwxO9z0kYWZer+PIxSMI3heM4H3BWHZ4GcwMt5e+\nHaveXJXd5ouI5LrrZs+BJObtmYenJj2Fe4bfgxVHV6B13dZY/eZqTN42GRExEQCA0IhQnL50GgDg\nogtzds/BY+MfQ+VvKyMqLgrrjq/DlpNbAABRcVEYu2ksHh77MNYeW4uY+BiM2zwOD419CPVH1sey\nw8vwfJ3nsaXDFoR+Eoqj4Uex9dTWXOsDEZGcYiRzuw0ZMjOmbOeBCwdQrHAxlC9eHksPLcW/F/0b\nMfEx+PgvH+PFO17EjYVv9JR9YfoLaFCxAc5FncPgdYPxSPVH8P4D7+OTXz9B8SLF8VHDjzB+63gc\nuXgEZy+fRaUSldDh/g74YukXuK/CfahSsgpWH1uN81HncUfAHXj3/nfR4vYWKFywcLI2tZ/bHnXL\n1kWnhp180g/Hw4+jVNFSKF6kuE/qB4B95/ehxs01UMCum+8NIvmWmYGk90MiSZHM8z8AGBsfy7iE\nOLpcLg5eO5jFehRjy8kt+cGCD1j5m8qcvHUyE1wJTM2i/YtoQcZ2s9vxwPkD9O/tz5qDanLe7nl0\nuVwkyS0nt3DsprGMjotm3aF12ez7ZtxwfANJMjImku/Pf59rj61Ntf4rhq4byjd/ejPdMllxOfYy\nOy/szCLdirDXil6e+aciT/GVH19h/RH1ufzwco7ZOCbL6zh44SCfm/ocLcj42cLPuOfsnpxoeo64\nHHuZJBkWFcb95/fncmtErh2JH/FZ/NzN6huv5g8A1hlch+3ntmer6a14/8j7ufHERpbpVYYv/fAS\nz18+n2EnhUaEel4funCIMfExaZaNjY/1hEZmrDyykvePvD/T70vP5tDNvHPInWw1vRV7Lu/JNjPa\nkCRn7JjB8n3L89NfP2Xj0Y1Z/KvirDWoVrp1Tdk2hauOrEo2L8GVwMFrB7NMrzLsvrQ7lx5aSgTB\nsx5fWnpoKesMruMJtSNhR9hqeivO3zOfJHkx+iLfn/8+i3Qrwq+WfcWK/SqyQt8K7LywM+MT4nOs\nHVtObuHkrZNzrD5fWH54Oe8aeheXH15OkoyOi/b6937bqW050obImEj2WNaDHyz4IEfqyysuRF3I\n7SZkWnh0OMOjwzMsly/C4YuQL1i0e1G2/rE1o+OiSSZ+eGTlQ9xXImIiWKxHMcbExzAyJpKtf2zN\nSVsnZflb+IwdM+jf25/jN4+ny+XixhMbeceQO/j54s9ZrX81zwf9mUtneOjCIRb/qniq9cTEx/Af\ns//Bm3rcxNdmvsa4hDj2W9WPR8KO8IVpL7DBdw2488xOT/lF+xfx4bEPZ6nNaVl3bB1f+fEVulwu\nxsbH8n+//Y/l+5bngDUD6N/bn8PXD2fZPmXZfm57VuhbgcF7g3nrgFv55k9vcsi6IXxg5ANceWQl\nuy/tTgSBe8/tTVb/hagLjImP4ebQzXxq4lM8dOFQhm2KjY9l0JIg3tTjJt455M4c3d6MhBwM4YrD\nKzIsd+bSGb4x6w1W/qYy7xp6Fz/8+UMuP7yctw+8nTUG1Ejzgy3BlcD+q/vT72u/bG+by+XilG1T\nWLFfRT496WmW7VM2W/XltARXAsdsHMMWk1qkOXqQmv3n9/OFaS+w8JeFvfqgzQsuxV5ij2U9iCDw\n5R9ezrB8ng0HAM0B7AKwF0DnVJaXAjAXwGYA2wH8PY16SCb+Z2bmPz833DPsHi7av4hNxjThI+Me\nIYLAMr3KMDY+li//8LLjm3tahq4byor9KvL3E7975kXFRbHgFwXZ8LuGPBlxMll5l8vFm3rcxIvR\nF5PNj4yJZPOJzdlycktuOL6BVb6twpaTWzKgdwCLdi/KN2a94QnbK45dPOb1B8CpyFMZBvTsnbPp\n39ufCAJXHF7BJyc8yScmPOHZhm5Lu7F6/+pcc3QNSfLDnz9kqZ6l+MMfP6RaX/OJzTln1xyGRYVx\n68mtXH10Ncv1KceG3zWkf29/Pj/teb4w7QVHP/Rc3tOzracjT7Pp2KZ8YsITXHN0DSv2q+jV9mZX\nfEI8//vbf4kgsOnYpp75u87s4sYTG5OVXXJwCSt/U5kf/fwRw6PDuf74epb4qgTL9y3PWTtn8YVp\nL3DkhpGOdRy6cIhNxzZl49GNueP0DpbtU5YHzh/wqn0p90ZCI0L5zORneOeQO7nqyCrP71lYVFjm\nNz4HhUeHc9r2adx5ZicfHP0gG41qxCrfVuGG4xsy3FOKiotil0VdWKZXGfZY1oN3DLnDM4ScVFhU\nWJ4ZXnW5XBy7aSwr9avEF6e/yMUHFhNB4Jxdc9J9X54MBwAFAewDUA1AYXcA1ElR5j8Aerpf+wM4\nB6BQKnVlvVevsvfmvUe/r/3YbnY7JrgSuPbYWhbrUYxtZ7UlgsD35r2XYR2jfh/Fav2rpTq+Pn/P\nfM8YfEo1B9XkjtM7SNKz99J4dGO+MesNz/GagN4BbDOjDcOiwjh75+xUP9hdLheLf1Wc5y6f80yn\nZtWRVSz+VXHO3T03zW0Zvn44K/StwPXH1/O9ee+xWI9ifHvO24xLiEu2vqTDRHEJcZ51p+afwf/k\nJ798wnuG3cMSX5Wgf29//vDHD3x33rv84/Qf3HVmF28beJun/NlLZ/nAyAeIIHDpoaXccXoHq35b\nlV0WdWF8QjwjYyJ5Y/cb01xfTrkQdYGPjX+Mj4x7hL8d+I21B9cmSc7aOYslvirBwHGBJBP7o8/K\nPizftzyD9wZ73h+fEM/eK3rzdORpkuSANQPYfm57konDTKERofztwG8s26csv17+tadPP1jwAat+\nW5V/m/q3VNt1KvIUI2Mi2WtFLxb6spBnr2vJwSWs2K8i//fb/5INw947/F6uP74+R/okK3v+G45v\nYI0BNVixX0WW7FmSg9YOYoIrgR3nd2RA7wBakDE0IjTZ3vAVv5/4nXcOuZPPT3veM9T88g8vc+KW\nicnK/XbgN97yzS2sOahmjo5OxCXEMWhJEFtNb+X1ew6HHeYTE57gfSPu4+qjqz3z7x52NxGEZH9L\nKeXVcPgLgOAk058B+CxFmc8ADHG/rgFgTxp1ed2RuW3h/oWeoZsrHhj5AOsNr8fgvcGsN7xeuu8P\n3hvM8n3Lc/fZ3Zle9yPjHuHC/Qu59eRWlupZincNvYtvzHoj2d7W0YtHvdr7ajSqERcfWMyJWyYS\nQUj2ni6LuvC5qc+xbJ+yfGLCE/z0109TrWPiloms/E1lT8htCt3EgWsGZvuPbeSGkUQQ+Pniz7n4\nwGJuP7U92fKouCje0O0GxifEMzQilHWH1uWnv37KDxZ8wDdmvcEKfStw3KZxnvIul4uFvyzs2IPK\nSScjTvKeYfew04JOjEuIY3RcNIt2L8oh64awQt8KDDkY4hkqeHfeu6w7tC6PXjyabp0rj6xk/RH1\neSHqAh8a8xD9vvZjuT7luPjA4mTlouKi+J9F/yGCwE4LOiX7oF97bC0DegewdK/SrD+iPpuMacIp\n26Zw+PrhLNenXLJwuqLp2KZEEHgy4iQvx17mg6Mf5OiNozPdJ+uOreMt39zCrSe3ev2eYeuHMaB3\nAH/44wdGxETwRPgJz7ItJ7fw29Xf8sXpL7L24Nos8EUBDl472LN89MbRDOgdwIlbJib7Hey6pCsf\nG/8Y7x52N89cOsP/W/x/rNivIhfsWcAbut3Aot2LZnrY6XTkaX7484eMjIn0zDt04RAbfteQjUc3\nZkDvAK/qmbBlAv17+7P70u6MjY91LK81qBa3nNySbD1J5dVwaAXguyTTrwEYlKJMcQBLAJwAEAHg\nqTTq8qoj86pNoZt4KvIUY+JjWK5POf6679dky1ccXsET4Sd4OOwwy/Upx6WHlmZpPa/NfI39V/dn\njQE1+Pniz9lxfsdUf6G80WVRF7ac3NIzHHTlm+Lw9cNZe3Bt/nXKXzl752wuPrCYDb5rwElbJ/HM\npTOe9wfvDWbZPmUdH9w54WTESU7dNjXdMpX6VeK2U9t419C7GLQkiC6Xi9O2T2OBLwpw0tZJjvJl\n+5RNdtLCqN9HceH+hTnS3tCIUN4+8HZPO66oNagWq3xbxTN08fy05/nw2If51MSnvBq2uRR7iaV6\nlmKdwXXYaUEn/nbgt3SHj16c/iIRBE7fPp1k4v+Rf29/zt45mzN3zOTl2MvsubwnawyowRoDanDf\nuX2p1jNjxwwiCBywZgAfHvswEQS+P//9zHQJZ+2cxYDeAbxr6F1eBUuCK4H//vXfrDWoluN4U0oh\nB0P4/vz3uXD/QtYbXo8JrgR2WtCJtQbV8uxZJzVzx0yW71ueTcY0YdVvq7LJmCaeIc9p26d5+szb\nkyD+OP0Hq/evTgSByw4tI0kuO7SM5fuWZ5+VfRifEM8i3Yp4RgBi42Mdw1ex8bHsOL8jbxt4GzeH\nbk5zXW1mtGG94fVY+ZvKqS7Pq+Hwghfh0ApAP/frWwEcAFAilbrYtWtXz8+SJUvS7Ky8bvr26Xxk\n3COe6f3n93uGmx4a81CyU1Uza+SGkSz0ZSF2WtAp2+1ctH8REQTO2DGD//rlX/x88edcdWQVA3oH\nJNuriYqL4h1D7iCCwD4r+5Ak953bx4DeAV4dcPWVh8Y8xGr9q/Hdee96PpBj4mOSHcNJKukHx8wd\nM2lBxo9+/ihT65yza45nyOeK8Ohw3jfiPn4R8oWjfPDeYB4JO5KpdaS04vAKDl472Ou9sSnbprDZ\n98245OASBvQO4MojK5Mt33hiIwPHBSYLytSM3jiaFmTsMLcDx24ay1dnvOp1m6dvn87yfctzw/EN\n/Hr51/xn8D85a+csXoq9lGr5+IR4tp3Vlg+OfpBnL531ej3RcdG8qcdNbDW9FR8e+3CagetyuRgd\nF82tJ7eyy6IujjMZ+67sSwSB3/3+XbrrG75+OO8feT8Degdw/ObxfGfOOxy0dhAnb53Msn3K8pd9\nv3jKVu9fnXvO7uHF6It8fPzjrNa/mmdZREwEHxv/GFtMapHhmVTjN4/nC9NeYJFuRRgVF8UlS5Yk\n+6zMq+HQKMWwUpeUB6UBzAPQOMn0bwDuT6WudDvoWnI8/Dj9e/vT5XLR5XKx+cTmfHLCkyzavSgf\nHP1gtg64u1wuzts9L93TdL0VnxDv+eBYemgp7xhyB2sMqMFZO2c5yp6/fJ6jN45mw+8aMjoumvWG\n1+PANQOz3YbseOunt9h8YvN0x2OTajSqEVceWckNxzfQv7c///3rvx0HtcnEP9zdZ3dzxIYRyebP\n3jmbJXuW9AwbRsVF8dkpz7Le8Hp866e38sxZddFx0Szbpyz9e/s7hp8yIywqjKN+H0WXy8XgvcF8\nfPzjXr1v5o6ZLNennOfb8Lzd8+j3tR8RBMceNZn4O/3WT28xcFxgmuGRnmbfN+Mzk59J8zidN85d\nPsdag2p5vvykZuCagazWvxqHrR/m+VI0eO1g1hxUkxX7VXQcJG86tiln7pjJBt814Ntz3mbR7kV5\nKfYSz146ywbfNeCbP72ZqdO1q/WvluqeXl4Nh0IA9rsPSBdJ44D0UABd3a/LATgGoHQqdXndSXmd\ny+Vi6V6lGRoRyunbp7Pu0LqMjIlk3aF1Uz2AlhfEJcTx5q9vZrvZ7dIsExsfy9K9SrPd7HZsObll\nrn8YRsREZGpI7amJT3HqtqmsMaAGp2+fzpVHVrLBdw2SlYlPiPcMoyAIjImP4enI01x8YLHnW3ix\nHsUYHh3ODnM78JnJzzBoSZDXAXW1fL/5e/689+ccq2/jiY28e9jdyeZd2UNJavXR1QzoHZBs7y00\nIpSNRzfmExOeSPWb+Se/fMJGoxpl+VTTi9EXc+SamC9CvuB/f/tvqsuGrBvCqt9W5cELB5PN33hi\nI2sOqpnqGU+vzXyNxb8qzo7zO9LlcrHu0LpccnAJ6w2vx3/98q9M//08NOYhLjm4hAfOH0g2dJYn\nwyGxXXgKwG73WUtd3PPaA2jvfl0BwC8AtgLYBqBNGvVkqqPyuqZjm3Lu7rms1r8aQw6G5HZzvLLi\n8ApGxESkW+bvs//OUj1L8Xj48avUqpzTZkYbVv6msicAj4QdYcV+FTlyw0jO3jmbJPllyJds9n0z\n7jm7h3WH1uWqI6vYaFQjFvqykGdP4v6R9/OzhZ+xWv9quX6659VyPPw4y/ctz6i4KG4O3cxXZ7xK\n/97+LNq9qGdP+HDYYVbsV5Hzds9LtY6uS7ry/xb/X7J5ozeOZs1BNb262M/XBqwZwI7zOzrmz9gx\ng5X6VfL6VOErRmwYwU9//dQTAq2mt2LpXqU9YZFZbWa0YY9lPVj5m8ps/WNrkolfRPNsOOTUz/UW\nDl+EfMHK31TmUxOfyu2m5KjdZ3dzycElud2MLHl//vus1r+a5zqRuIQ4FvqyEIt0K8LXZ77O7ae2\n07+3P49dPEaS7DC3A+sMrsNm3zdLNmTxj9n/YIEvCuTq8ZarLSY+hoW+LMRm3zcjgsC3fnqLv+77\nleX6lOOxi8cYEx/D+0bcl+6wzJiNY9h2VlvP9JW9jLyyNz1u0zi+PvP1ZPPWH19P/97+qV4jkVmD\n1g7i23PezvKwcueFnWlBxraz2vKeYfcwNj6WT054MlvhcF3esjuva3dvO3Rf1h0/vfJTbjclR9Us\nUxM1y9TM7WZkyTv138F7D7yHkjeUBAAUKlAIFUtUxGPVH8P6E+vx3oL3ENQ0CJVKVgIAdG7SGYUK\nFMKnjT9NdpPH5+o8h3vL34vGVRrnynbkhiIFi6BEkRKIS4jD2U/PokyxMgCAGjfXwP4L+zFo3SBU\nLlkZn/zlkzTrqFKqCsZvGY9GlRqh9V2t8dIPL2H0s6NR27/21dqMdJUqWgoXYy56ps9cOoPnpj2H\nkc+MRP2K9bNdf8cGHbP1/sdqPAa/on7o1KATAvoEoNPPnVCwQMFs1XnN3pX1Wnc+6jxK31g6t5sh\n6TgRcQJ+Rf1QomcJ1CtfD2vfWpvtP7jrVc/lPfHq3a+iSqkqnnmvz3odhQsURvC+YGzusBllbyqb\n5vsPhx1GtQHVUPamsnjy1idRokgJDHl6yNVouldCDoWga0hXLP37UpDEM1OewV1l78LXj32d201z\nqD6gOooWKoo1b66B341+YBbvyqo9h1yiYMj7KpaoCAB4vs7z6Ny4s4IhHV0e6uKY16hSI3Re1BnT\nWk1LNxgAoKpfVVzofAEV+lXA6mOrsbn9Zl81NUtK3VAKF6MT9xwGrh2Is5fPotsj3XK5Van7MvBL\n/OWWv6BU0VLZqkd7DiLiM2TmnrD4z+B/4uW6L6NR5UY+bFXmHbhwAI+OfxS/vPYLHhz9INa+tRa3\nlr41t5uVoew8z0HhICKSgfNR53HrwFtRv0J9tLi9BT7+y8e53SSvZCcc9LgvEZEMlLyhJMKiw3Au\n6hw+aPhBbjfnqlA4iIhkoFCBQqjmVw0jnhmBQgXyx6FaDSuJiHghwZVwzZ2UoGElEREfu9aCIbsU\nDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcVA4iIiIg8JBREQcFA4iIuKgcBAREQeFg4iI\nOCgcRETEQeEgIiIOCgcREXFQOIiIiIPCQUREHBQOIiLioHAQEREHhYOIiDj4NBzMrLmZ7TKzvWbW\nOY0ygWa2ycy2m1mIL9sjIiLeMZK+qdisIIDdAB4DcBzAegCtSe5MUsYPwEoAT5I8Zmb+JM+mUhd9\n1U4RkeuVmYGkZeW9hdKp9JMUswjgDIAVJA96UXcDAPtIHnLXNxXAXwHsTFKmDYAZJI8BQGrBICIi\nV196w0olABRP8lMCwAMAgs2stRd1VwJwNMn0Mfe8pG4HUNrMlpjZBjN73euWi4iIz6S550AyKLX5\nZlYawG8ApmRQtzfjQIUB3AfgUQDFAKw2szUk96YsGBT0Z3MCAwMRGBjoRfUiIvlHSEgIQkJCcqSu\nLB1zMLNNJOtlUKYRgCCSzd3TXQC4SPZKUqYzgBuvBJGZjQIQTPLHFHXpmIOISCZl55hDps9WMrNH\nAFzwougGALebWTUzKwLgZQBzUpT5CUATMytoZsUANASwI7NtEhGRnJXeAeltqcy+GUAogLYZVUwy\n3sw6AvgFQEEAo0nuNLP27uUjSO4ys2AAWwG4AHxHUuEgIpLL0hxWMrNqKWYRwDmSkT5uU2pt0bCS\niEgmZWdYyWfXOeQkhYOISOZd1WMOIiJy/VM4iIiIg8JBREQcFA4iIuKgcBAREQeFg4iIOCgcRETE\nQeEgIiIOCgcREXFQOIiIiIPCQUREHBQOIiLioHAQEREHhYOIiDgoHERExEHhICIiDgoHERFxUDiI\niIiDwkFERBwUDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcVA4iIiIg8JBREQcfBoOZtbc\nzHaZ2V4z65xOuQfMLN7Mnvdle0RExDs+CwczKwhgMIDmAO4A0NrM6qRRrheAYADmq/aIiIj3fLnn\n0ADAPpKHSMYBmArgr6mU6wTgRwBnfNgWERHJBF+GQyUAR5NMH3PP8zCzSkgMjGHuWfRhe0RExEuF\nfFi3Nx/0/QF8RpJmZkhnWCkoKMjzOjAwEIGBgdltn4jIdSUkJAQhISE5UpeRvvmybmaNAASRbO6e\n7gLARbJXkjIH8Gcg+AO4DOBtknNS1EVftVNE5HplZiCZpWO5vgyHQgB2A3gUwAkA6wC0JrkzjfJj\nAcwlOTOVZQoHEZFMyk44+GxYiWS8mXUE8AuAggBGk9xpZu3dy0f4at0iIpI9PttzyEnacxARybzs\n7DnoCmkREXFQOIiIiIPCQUREHBQOIiLioHAQEREHhYOIiDgoHERExEHhICIiDgoHERFxUDiIiIiD\nwkFERBwUDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcVA4iIiIg8JBREQcFA4iIuKgcBAR\nEQeFg4iIOBTK7QaIiKTGzHK7CdcUkjlan8JBRPKsnP7Au175Ikg1rCQiIg4KBxERcVA4iIiIg8JB\nREQcfB4OZtbczHaZ2V4z65zK8lfNbIuZbTWzlWZ2t6/bJCKSE7p06YIBAwb4fD1z587FK6+84vP1\nJOXTcDCzggAGA2gO4A4Arc2sTopiBwA8TPJuAN0AjPRlm0REcsKZM2cwYcIEdOjQAQCwY8cO3H//\n/ShdujRKly6Nxx9/HDt37vS6rtatW6NSpUrw8/NDkyZNsG7dOs/yli1b4o8//sC2bdt8si2p8fWe\nQwMA+0geIhkHYCqAvyYtQHI1yYvuybUAKvu4TSIi2TZu3Dg8/fTTuOGGGwAAlSpVwg8//IBz587h\n3LlzePbZZ73+th8ZGYmGDRti48aNuHDhAt544w08/fTTuHTpkqdM69atMXLk1fvu7OtwqATgaJLp\nY+55aXkTwAKftkhEJAcEBwejadOmnulSpUqhevXqMDMkJCSgQIEC2L9/v1d1Va9eHR999BHKlSsH\nM8Pbb7+N2NhY7Nmzx1MmMDAQ8+fPz/HtSIuvL4Lz+goWM3sEwD8ANPZdc0REcsa2bdtQq1Ytx3w/\nPz9cunQJLpcL3bp1y1LdmzdvRmxsLG677TbPvNq1a+PQoUOIjIxE8eLFs9xub/k6HI4DuCXJ9C1I\n3HtIxn0Q+jsAzUleSK2ioKAgz+vAwEAEBgbmZDtF5BqUUxcGZ+VC7LCwMJQoUSLV+ZcvX8b333+P\nqlWrZrr5yicnAAAK30lEQVTe8PBwvP766wgKCkpW/5XXYWFhaYZDSEgIQkJCMr3O1JgvL083s0IA\ndgN4FMAJAOsAtCa5M0mZKgAWA3iN5Jo06qEuoxfJX8wsT98+o1y5cliwYAHq16+f6nKSCAgIwK5d\nu+Dv7+9VnVFRUWjevDlq166NESNGJFt2/vx5+Pv7Izw83BEOafWVe36WItSnxxxIxgPoCOAXADsA\nTCO508zam1l7d7HPAdwMYJiZbTKzdWlUJyKSZ9x9993YvXt3mssTEhJw+fJlHD9+3Kv6YmJi8Le/\n/Q1VqlRxBAMA7Ny5E9WqVbsqQ0rAVbjOgeTPJGuRvI1kT/e8ESRHuF+/RbIMyXrunwa+bpOISHa1\naNECS5cu9UwvWrQImzdvRkJCAsLDw/Hxxx+jdOnSqFMn8ez9cePGoXr16qnWFRcXh1atWqFYsWIY\nN25cqmWWLl2KFi1a5Ph2pEV3ZRURyYK2bdvi3nvvRXR0NIoWLYqwsDB06tQJx44dw4033oiGDRsi\nODgYRYoUAQAcPXoUTZo0SbWuVatWYf78+ShWrBj8/Pw884ODg9G4ceI5OlOnTsWkSZN8v2FuPj3m\nkFN0zEEk/8nrxxwA4L///S/Kli2LDz/8MMOyTz75JAYOHJjqGU4ZmTt3LiZNmoSpU6emutwXxxwU\nDiKSJ10L4ZBXXHMHpEVE5NqkcBAREQeFg4iIOCgcRETEQeEgIiIOCgcREXFQOIiIiIPCQUQki/SY\nUBERSSblY0LXrFmDxx9/HGXKlEHZsmXx0ksv4eTJk17Xld8eEyoicl1K+ZjQsLAwdOjQAYcPH8bh\nw4dRokQJtGvXzqu68uJjQnX7DBHJk/L67TMeffRRvPnmm2jTpk2qyzdu3IjAwECEh4dnqf5SpUoh\nJCQE9erVA5B4c77XXnsNBw4ccJTV7TNERPKItB4TesWyZctQt27dLNWd0WNCrwbdsltErln2Rc48\nJ5RdM7+HktZjQgFg69at6NatG+bMmZPperPzmNCcpHAQkWtWVj7Uc8rNN9+MiIgIx/x9+/ahRYsW\nGDhwoOdZDN6KiopCy5Yt8eCDD6Jz587Jll1ZV9LnPfiShpVERLIgtceEHj58GI8//jg+//xzvPrq\nq5mqL989JlRE5HqU8jGhx48fR7NmzdCxY0e88847jvLX2mNCFQ4iIlnQtm1bLFiwANHR0QCAUaNG\n4eDBg55jBSVKlEDJkiU95b15TOjChQvh5+fnef/KlSs9ZaZOnYr27dv7dqOS0KmsIpIn5fVTWQE9\nJjTXKRxE8p9rIRzyCl3nICIiV4XCQUREHBQOIiLioHAQEREHhYOIiDjo9hkikmeZ5cy9kyTzfBoO\nZtYcQH8ABQGMItkrlTIDATwF4DKAv5Pc5Ms2ici1Qaex5i6fDSuZWUEAgwE0B3AHgNZmVidFmRYA\nbiN5O4B3AAzzVXuuFyEhIbndhDxDffEn9cWf1Bc5w5fHHBoA2EfyEMk4AFMB/DVFmWcBfA8AJNcC\n8DOzcj5s0zVPv/h/Ul/8SX3xJ/VFzvBlOFQCcDTJ9DH3vIzKVPZhm0RExAu+DAdvBwxTHnHSQKOI\nSC7z2b2VzKwRgCCSzd3TXQC4kh6UNrPhAEJITnVP7wLQlOSpFHUpMEREsiCr91by5dlKGwDcbmbV\nAJwA8DKA1inKzAHQEcBUd5iEpQwGIOsbJyIiWeOzcCAZb2YdAfyCxFNZR5PcaWbt3ctHkFxgZi3M\nbB+ASwDa+ao9IiLivWvilt0iInJ15enbZ5hZczPbZWZ7zaxzxu+4tpnZGDM7ZWbbkswrbWYLzWyP\nmf1qZn5JlnVx980uM3sid1rtG2Z2i5ktMbM/zGy7mX3gnp/v+sPMiprZWjPb7O6LIPf8fNcXV5hZ\nQTPbZGZz3dP5si/M7JCZbXX3xTr3vJzpC5J58geJQ1H7AFQDUBjAZgB1crtdPt7mhwDUA7Atybze\nAP7tft0ZwNfu13e4+6Swu4/2ASiQ29uQg31RHsC97tfFAewGUCcf90cx97+FAKwB0DC/9oV7Gz8G\nMAnAHPd0vuwLAAcBlE4xL0f6Ii/vOXhzEd11heRyABdSzPZcKOj+92/u138FMIVkHMlDSPyPbnA1\n2nk1kDxJcrP7dSSAnUi8Lia/9sdl98siSPzjJvJpX5hZZQAtAIzCn6fC58u+cEt5wk6O9EVeDgdv\nLqLLD8rxzzO4TgG4cgV5RST2yRXXbf+4z3irB2At8ml/mFkBM9uMxG3+leQ65NO+APAtgE8BuJLM\ny699QQCLzGyDmb3tnpcjfZGX78qqI+UpkGQG13xcd31mZsUBzADwIcmIpHfpzE/9QdIF4F4zKwVg\nlpnVTbE8X/SFmT0D4DTJTWYWmFqZ/NIXbo1JhppZAICF7mvFPLLTF3l5z+E4gFuSTN+C5KmXX5wy\ns/IAYGYVAJx2z0/ZP5Xd864bZlYYicEwgeRs9+x82x8AQPIigCUAnkT+7IsHATxrZgcBTAHQzMwm\nIH/2BUiGuv89A2AWEoeJcqQv8nI4eC6iM7MiSLyIbk4utyk3zAHwhvv1GwBmJ5n/ipkVMbPqAG4H\nsC4X2ucTlriLMBrADpL9kyzKd/1hZv5XzjgxsxsBPI7EYzD5ri9I/ofkLSSrA3gFwGKSryMf9oWZ\nFTOzEu7XNwF4AsA25FRf5PbR9gyOxD+FxLNU9gHoktvtuQrbOwWJV5PHIvF4SzsApQEsArAHwK8A\n/JKU/4+7b3YBeDK325/DfdEEiWPKmwFscv80z4/9AeAuABsBbHH/8f/PPT/f9UWKfmmKP89Wynd9\nAaC6++9jM4DtVz4jc6ovdBGciIg45OVhJRERySUKBxERcVA4iIiIg8JBREQcFA4iIuKgcBAREQeF\ng1z3zCzS/W9VM0v5NMLs1v2fFNMrc7J+kdyicJD84MrFPNUBtMnMG80so/uPdUm2IrJxZuoXyasU\nDpKffA3gIfeDUT503+m0j5mtM7MtZvYOAJhZoJktN7OfkHjlKcxstvvOl9uv3P3SzL4GcKO7vgnu\neVf2Usxd9zb3w1heSlJ3iJn9YGY7zWzilcaZ2deW+HCjLWbW56r2jEgKefmurCI5rTOAf5FsCQDu\nMAgj2cDMbgCwwsx+dZetB+BOkofd0+1IXnDf22idmf1I8jMze59kvSTruLKX8jyAewDcDSAAwHoz\nW+Zedi8SH7wSCmClmTVG4u0M/kaytrttJX2w/SJe056D5CcpH4ryBIC2ZrYJiU9XKw3gNveydUmC\nAQA+dD9PYTUS72x5ewbragJgMhOdBrAUwANIDI91JE8w8d41mwFUBRAGINrMRpvZcwCisryVIjlA\n4SD5XUeS9dw/t5Jc5J5/6UoB93MDHgXQiOS9SLwJYNEM6iWcYXRlryImybwEAIVJJiDxdss/AngG\nQHBWNkYkpygcJD+JAFAiyfQvAN67ctDZzGqaWbFU3lcSwAWS0WZWG0CjJMvi0jhovRzAy+7jGgEA\nHkbi7ZFTBgbc674JiXfP/BmJz0e+J5PbJpKjdMxB8oMr39i3AEhwDw+NBTAQiQ9a3+h+fsRpAM+5\nyye9XXEwgA5mtgOJt5BfnWTZSABbzex3Jj5XgABAcpaZ/cW9TgL4lORpM6sD59O3iMTQ+snMiiIx\nQP6ZI1sukkW6ZbeIiDhoWElERBwUDiIi4qBwEBERB4WDiIg4KBxERMRB4SAiIg4KBxERcVA4iIiI\nw/8DGqwOkNBaudkAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x103cea7b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_utility_estimates(agent, sequential_decision_environment, 500, [(2,2), (3,2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}