rl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Reinforcement Learning\n",
    "\n",
    "This IPy notebook acts as supporting material for **Chapter 21 Reinforcement Learning** of the book* Artificial Intelligence: A Modern Approach*. This notebook makes use of the implementations in rl.py module. We also make use of implementation of MDPs in the mdp.py module to test our agents. It might be helpful if you have already gone through the IPy notebook dealing with Markov decision process. Let us import everything from the rl module. It might be helpful to view the source of some of our implementations. Please refer to the Introductory IPy file for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from rl import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Review\n",
    "Before we start playing with the actual implementations let us review a couple of things about RL.\n",
    "\n",
    "1. Reinforcement Learning is concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. \n",
    "\n",
    "2. Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).\n",
    "\n",
    "-- Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n",
    "\n",
    "In summary we have a sequence of state action transitions with rewards associated with some states. Our goal is to find the optimal policy (pi) which tells us what action to take in each state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passive Reinforcement Learning\n",
    "\n",
    "In passive Reinforcement Learning the agent follows a fixed policy and tries to learn the Reward function and the Transition model (if it is not aware of that).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passive Temporal Difference Agent\n",
    "\n",
    "The PassiveTDAgent class in the rl module implements the Agent Program (notice the usage of word Program) described in **Fig 21.4** of the AIMA Book. PassiveTDAgent uses temporal differences to learn utility estimates. In simple terms we learn the difference between the states and backup the values to previous states while following a fixed policy.  Let us look into the source before we see some usage examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%psource PassiveTDAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a policy(pi) and a mdp whose utility of states will be estimated. Let us import a GridMDP object from the mdp module. **Fig[17, 1]** is similar to **Fig[21, 1]** but has some discounting as **gamma = 0.9**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import Fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mdp.GridMDP at 0x7fbeb805cac8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Fig[17,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Fig[17,1]** is a GridMDP object and is similar to the grid shown in **Fig 21.1**. The rewards in the terminal states are **+1** and **-1** and **-0.04** in rest of the states. <img src=\"files/images/mdp.png\"> Now we define a policy similar to **Fig 21.1** in the book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "policy = {(0, 0): (0, 1),\n",
    "          (0, 1): (0, 1),\n",
    "          (0, 2): (1, 0),\n",
    "          (1, 0): (-1, 0),\n",
    "          (1, 2): (1, 0),\n",
    "          (2, 0): (-1, 0),\n",
    "          (2, 1): (0, 1),\n",
    "          (2, 2): (1, 0),\n",
    "          (3, 0): (-1, 0),\n",
    "          (3, 1): None,\n",
    "          (3, 2): None,\n",
    "         }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "our_agent = PassiveTDAgent(policy, Fig[17,1], alpha=lambda n: 60./(59+n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rl module also has a simple implementation to simulate iterations. The function is called **run_single_trial**. Now we can try our implementation. We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import value_iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values calculated by value iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n"
     ]
    }
   ],
   "source": [
    "print(value_iteration(Fig[17,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the values estimated by our agent after **200 trials**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.45317974152528395, (1, 2): 0.6904748397040797, (3, 2): 1, (0, 0): 0.32814758326189863, (2, 0): 0.0, (3, 0): 0.0, (1, 0): 0.2173891183100952, (3, 1): -1, (2, 2): 0.8413461352306864, (2, 1): 0.5466536540971639, (0, 2): 0.5623108119928993}\n"
     ]
    }
   ],
   "source": [
    "for i in range(200):\n",
    "    run_single_trial(our_agent,Fig[17,1])\n",
    "print(our_agent.U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also explore how these estimates vary with time by using plots similar to **Fig 21.5a**. To do so we define a function to help us with the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "def graph_utility_estimates(agent_program, mdp, no_of_iterations, states_to_graph):\n",
    "    graphs = {state:[] for state in states_to_graph}\n",
    "    for iteration in range(1,no_of_iterations+1):\n",
    "        run_single_trial(agent_program, mdp)\n",
    "        for state in states_to_graph:\n",
    "            graphs[state].append((iteration, agent_program.U[state]))\n",
    "    for state, value in graphs.items():\n",
    "        state_x, state_y = zip(*value)\n",
    "        pylab.plot(state_x, state_y, label=str(state))\n",
    "    pylab.ylim([0,1.2])\n",
    "    pylab.legend(loc='lower right')\n",
    "    pylab.xlabel('Iterations')\n",
    "    pylab.ylabel('U')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a plot of state (2,2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFNXZ9/HvMAMDwzoiIKAsCq6ggnEnOIqKgShxiQYX\njHFBjXF7XuNCEjFmMdFHI2p8MFFJiJGoMeIWjTGOK6iILMqiREF22UZAdue8f9x97Jqu6m2Y7umZ\n+X2ua67ppbr6dHX1uc99zqkqEBERERERERERERERERERERERERERaZQeAlYCs5M8fw4wE5gFvAkc\nmKdyiYhIPfomMIDkweFIoH3s9knA1HwUSkRE6l8vkgeHoHJgSW6LIiIimWhW3wUIuBB4vr4LISIi\n+dGL9JnDscAcLHsQEZF6VlLfBcAGof+AjTmsi1rgoIMOcjNnzsxroUREGoGZwMG1eWF9dyv1AJ4E\nzgUWJFto5syZOOf05xw333xzvZehUP60LbQttC1S/wEH1bZyznXm8ChwDLArsBi4GWgee2488DOs\nK+n+2GPbgcNyXCYREUkj18FhZJrnL4r9iYhIAanvbiXJUkVFRX0XoWBoW8RpW8RpW9SNovouQIZc\nrP9MREQyVFRUBLWs55U5iIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4iIhIiIKDiIiE\nKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4iIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4\niIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4iIhIiIKDiIiE5Do4PASsBGanWGYc8DEw\nExiQ4/KIiEgGch0cHgZOSvH8MKAP0Be4BLg/x+UREZEM5Do4vA6sS/H8KcCfYrffBjoAXXJcJhER\nSaO+xxy6A4sD95cAu9dTWUREJKa+gwNAUcJ9Vy+lEBGRr5XU8/svBfYI3N899ljI2LFjv75dUVFB\nRUVFLsslItLgVFZWUllZWSfrSmy150Iv4Bmgf8Rzw4ArYv+PAH4X+5/IOaeEQkQkG0VFRVDLej7X\nmcOjwDHArtjYws1A89hz44HnscCwAPgSuCDH5RERkQzkI3OoC8ocRESytDOZQyEMSIuISIFRcBAR\nkRAFBxERCVFwEBGREAUHEREJUXAQEZEQBQcREQlRcBARkRAFBxERCVFwEBGREAUHEREJUXAQEZEQ\nBQcREQlRcBARkRAFBxERCVFwEBGREAUHEREJUXAQEZEQBQcREQlRcBARkRAFBxERCSmp7wJk6qyz\noHVrOOccePRR+PJLKCmB3/8etm2D3/wGTjkFBg2CDz6AP/0JVq6055yDffeFW25Jvv41a2y9H30E\nVVWwYweceSZ85zvhZZcsgRdegD59oKIiZx85a8uWwezZcOKJUFSU2/fasAGmT4devaBnz/TL79gB\n//0vzJ0Ly5fD2rW2nbdtg+3b4/9PPNG+YxGpXw0mczjtNKuUTzoJunWzQPDKK7B4MZxxBtx+O7z4\nolU4xx5rgWTIEBgxAoYNg7vusiARZcUKGDAApk6Fvfay17VsaQEg0WOPwcEHw0MPwbhxuf3MmZo1\nyz5jv362XRYvTr38Rx/B0qW1e6/nnoMTToCuXeH44+EnP4k/N20aXHABfPJJ/LHp062y79TJyvjH\nP8LMmRbcO3WCPfe0ch9+uAWZ++6rXbkK1YIFtt82NNXV8Pbb8Otfw8knQ+/e8O1v109ZnLMG3333\nwauv1k8ZpHA555x7/XXnDj3Uua++cs455w46yLm33nKuZUvnbr7ZuRtucO5//9e5885zIe3bO7d6\ndfhx55y79FLnrr225mN/+INzP/hBzceWLnWuY0fnZsxw7u23nTvkkOj1Zau6uub9HTuc++1v7X86\nlZXO7bqrc/fe69yWLc717evc/PnJl5882Tlw7uKLsyvj5s3OnX++rf+vf7X7b75p38eOHc6NGeNc\n167O9etn227HDueuucYeu+su55YvT/8eq1c717atc1u3OrduXXblKwTB73H2bOe+9S3nSkudO/ts\ne2zz5vTr2L49+XNbt9o2r025nn3WuRNPdO6551Ivu3q1cz//uXM9ezp3wAHOXX21c3//u3P/+Y9z\ne+yR/XtHleM3v4k/tmVL8uWXLLHf9J57WnmGD3fu6KN3rgy5tGJFbtabWD9kA0jSJG48IjfUYYc5\nd889zh14oHN33GEV/AUXOPfAA+GNdPDBzk2bFr3hO3Vy7rPPaj7+8MPhIDN2rHOXXWa3ly1zrnPn\n2n1hiTp2dO73v7fPMHWqfSZIX0GuXevcbrs599JL8cf69XNu1qzo5T/4wALJT39qFVemqqudO/10\n5047zbmNG2u+f9u2FmgGDXJu5Urnbr/duSuvdO7MM5077rjsK/muXZ076ijnvve97F5Xn9assYr3\nJz+xbXX33badx41zbuZMq9gef9y+03feiV7H9u3O3XST7Qtbt4afnz3buf32s3WkCiCJFi1ybuhQ\n5/bf37lRo5wbOTL+3Pr11qjYvt3Kfd99Vu4LL3TuvfdqrmfrVueaN8+swRLllVec69/fGnTt2jm3\ncKFV/C1bOjd9es1lP//cuYsucq683LmrrnLu/fetfFu22Oe/8MLw+mfPdu7WW2tXtp1VWenc4MFW\ntqVLM3tNdbV9Lt/QjbJggXM//KFzrVsn/02nQ1MKDkHf/KZz555rO/24cc5dcYVzxx5bs7L0vvMd\n+4Fu3OjcF1/EH58/37kePcLL/+Uv4QrKZyrO2ZfaokV0a/DNN5NnKYk+/NB2qtGj7f9551kLDZxb\ntSr6NevWObdpk/24EjOAQw5x7t13o183ZIhVAPPn2w/vz3/OrIwPPeTcwIHRlRY416aNVTTOOff8\n8/bYkUembhUmc9JJ9vmHDs3+tfVh7VqreIcPt210440WoD/5xJ6vrrYKt3t3+5s8ObyObdts/zz+\neFvXxIk1A8C//23rmDDBGjKZZGHO2X7Ytatzv/ylvcfixc7tsout+7PPrJzg3AsvWOA/5BDn5sxJ\nvr7Ona1RlIkNG+z/jh3WaNt9d+eefDLe0Gjb1gJVRYVzkybFX/f88/Y+11xj2zbRHXdYmf3vbscO\na7R16uRccXHtg1dtLFni3Le/bZnNhAnODRhgjbt0Pv7Yvmtw7qmnwmXesMG5666z7/ymm+w38ac/\n1a6MNNXgcPzxtkP/4hfO3X+/c5dc4lzv3rbxE11zjbVqDz3UuV694o8//HDN1pT3t7/ZTuxt3Ohc\nq1Y1K7zge919t32JzlklMX58Zl/e2LFWGZaV2c6y226WEXXunLwSOO00+8ydOjn30Uc1nzvySOfe\neCP8mqlTrbzbttmf9eQ6V1WVunybNlml9vbb0c9ffrm1Cr0lS2y9ieXK1KpVFtwHDcrudStW2Psu\nWuTc3ntn1oWzs7Zvt33wqqtsvygrc65Pn3DD4PHHLVicd55VIkHV1fb48OEWfC+7zD7H44/b82+9\nZd/zq6/a/X79LBtJ5733rHJJ7EY66CDbt3v0sIr2pz+1cp99dvpgPmBA8oZH0Esv2Tr/8x/nTj7Z\nGiXBin7aNAsUzlnguO02u33PPc5162bdx6n07WtBbNMm226DB1vQ6tAh80ZZok8+ce7LL+NZVDr/\n+pf9Rm+5Jd5oGjHCuuBSmTDBvpe77rI6AiwAeO+8Y/XTqFHxbqpbb3Xu+uutIXn77dHrra62OvAb\n36j5PdJUg8Pw4c7ttZdzd97p3IMP2o+sRYvonfy22ywa+0rR+8lP7AtO9OSTzp1ySvy+H+8IGjzY\nuZdftts9e9p6v/zSuZIS5370I3v8mWcsRU7msMNsRwHr0wdr6XXrZhVtog0bLBXv1Cm6Aq2oiJcp\naPRoW6+3//7xbTF7dvLy/fnP2bfig11PtTFtmlVE2bjoIvssAwfa/8TAetFFzj322M6VK9G4cc4d\nc0y8lX/XXanHe6680pYJmjjR+vY3bbL7ixfbfnbHHZYh9ujh3NNPx5c/7jirmFJZudL2n6iK6qab\nrIXtK+QFC+w3kEmLe9gw6/MPBt4pU2qO173/vu2bZ5xh3UcjRlhjJJlx4ywgjh9vv6GFC9OX41vf\nsuA5ZIg17Pz279PHuXnz0r8+0RtvWPZ78cX2u7j9dvse1q+PzqQmTrRGnA/Y3uWX2+eJUl1t43J7\n7WWVvHMWVEaPjjdCH3vMtp0PnN4TTzi3zz5WRghn8Nu2Off971v3+hFHWF3o0VSDw2mnxfvrJ060\nyrJ79+gv5847bXANLKX1LrwwupX/zDP2Y/Duvdcyk6ARI5z7xz/8l2B/r79uweHYY+3xG26wnT6K\nz0aqqpwrKrJWme+X7tEj+ocyebKl6ODcr34Vfn7oUEvNg776yrZT4vpOPdXWk6oleswx6VtDdW3u\nXGv9J0rWP7twoXWXXH+9VXwQ79ZxzvqBITzpYGd8/rm1AD/4IPPX/Oxn9uetWGGVwfvv11zu7rut\nr3nUKPsfNHKkdXkmU11tv4sf/zj6+fnzbcC5NoOcQ4bU3F9Wr7ast0sXu++D2WOPWcPmkkviQS+Z\nyZMto+3SxQJVJq64wlrtp51WM6gdfnj2A/Yffmjr8uN8Z51l//fc037DXbvafuMD3OOP22NRQeMX\nv7Dfe5QxY2zcM7GreNYs5/bd17lHHrH1zpgRfu1//2tlfPVVC17PPRdvZGzbZtnZsGFWn/z1r/a7\nds5/ptoHh1xPZT0JmAd8DFwf8fyuwAvADOAD4PvZrLy0FL74Alq1ghYt7LiGjh2jl23RwubSA3To\nEH98xQqblpmoeXObd791q83pX7wYevSouUxZGWzaFL/fp49NuRs61I43APu/aBGsWhV+j9mzYb/9\noH17mDIFDjkEHnwQBg60Yzh27Ai/5q234Pzz7bOfcEL0NvGf05sxA3bdNXw8wogR9r+6OrwegNWr\n4f33bQpqPrVubVNdg6qqoLjYvpNEDzwAo0bB5ZfDPffYMS2bN8ef/7//s/9t29ZdGX/7W/je9+CA\nAzJ/TXm5fY4lS2x66G232ToOPrjmcr172zTqF1+0ZYI6d7b93HOu5j749NMwZ07yY3r23ht++tPa\nHQfz+OM25XjFCrt/5ZU2xbyqyrb3tdfavvLd70L37jB+vP02U+nVCz791KaG77VXZuXYf3/o0sWO\nZSoujj/esaMdr5SJSZNs3x4xwr7LK66AefPsWKcnnoDddoN27eCgg+DOO23696xZcNll8Pzz9rtN\n1L073H+//d6CHngA/v53+Ne/7HcYtPfe9vmvvhpeesneL9Gee9o2HzzYptyfcQZcdx1s3Ag/+IHt\nA089Zb+bQw+F996z4792dqp9LoNDMXAvFiD2B0YCiZv0CuB94GCgAvhfsjgwr7TUKtCWLa3yX7/e\nbkcJBof27eOPL19uO0IiHxwefhhuusmOC+jeveYyPjhUVdn98nJYuBCOPNJ20upqCwC9etl8/0Qz\nZ8Z3hsMPtx/sD35gO3xxcXRwmDIFjjkG3njDgknUNtm6FT7/3I7tAHj5ZTt2I9H559vOliw4vPAC\nHHdc8m2aK1HB4ckn7X9i4KuuhgkT4KKLLHhfdpmVd8sWe/6rr+zYilGjwuusrS1b7D2vuSa713Xo\nAOvWwR13wLPPWuV5fUSTqXdvO2Dw+uuhTZuaz3XubN+td/75tn+BbYubb7YDQnPxnZWXWwNoxQpr\npLz2Gvzyl7D77vCXv8B//mMVbTb697fjKbJpgFx8Mbz7bnjb7LqrNWjSBYhp02y7HXUUfPObdhtg\nn33sN3j66bbvP/EETJ5sB8K++649Pm5cOJh73bpZYzW4DaZNgzFjrPLu1Cn8mtJSayg8+mjqhoYP\n5qecYkF5wADbpz/6CP72N6uvwALs+vXWOIg6TisbuQwOhwELgIXAdmASMCJhmeVAu9jtdsAaIKJK\njFZaav995pAuOPhWZzBzWL48OnMoKbHlly+3yn/pUvsRBJWVWYtp2TK7v3mzBYfeva1sa9daK++w\nw6J32FmzolsK/v0Tg4NzFmQOPRS+8Y3o1p8PDq+8Avfea4+9+679EKI0a2YVaJRXX7UD3fItKjg8\n8YT9T9wmU6daizH4w2rVKp45vP66Bf/DDouv8x//SJ5hJvPPf1qL0ZflkEOsRZeN8nLbnyZOtMA9\nZky4wQH2Az/7bAt0ibp0iWcOW7bYutautfuTJ9v3efLJ2ZUrG1272me49lo7QK51a8tIr7sOfv7z\n7LOzoiL7brJRUhL/7Qd17GiZU48e0Qe8rl1r2cIPf2gZ5mWXJW9dt21r79Oihe1b/+//WTlHjkxe\nruOPt2zms8/s/rZtVoHfc48FnmQefzy68RblzDMtmzz8cKistNeWlcWfLyqyz/W3v2W/fybKZXDo\nDgSP1V0SeyzoD8ABwDJgJnBVNm/gdxCfOWzeHL3TQPx5iO/AX31l3T2dO4eX95nDmjWWHSxZEv4h\nt2plzy1bBnvsEQ8OvXrZcwsWWGuiZctwixesdbj33tHlLSkJV9pLl1prKRjcoj7n1q2WlSxbZjvP\nm29aSyNKcXHyzGHKFMuC8q1FCyuTD+Zbtlim1KxZuFvpqafCpzgJBocnnrBujrKyeHC4/fZ4hRrF\nuXA34G9/C1ddZd/JAw/A6NHZf67ycgvaAwdaN8KYMdHLtWoFjzwS3dDx3TBgweCII2x7OWcB56ab\ncnvqlK5dbZ9at84CGFhw2G23+j/tSceOtj8Es/nNm+HGG237XH21VfBFRZZp3nlnOPuI0q+f7Y8+\nE0+mWTNr2c+YYfvJ7bdboD/rrJ3/bImuvNK6qRK7ugF+9au6Oa1PLs+tlMlAyE3YeEMFsBfwEnAQ\nsCFxwbFjx359u6KigoqKihqZg28ppMocli+32199ZZX+kiXWxdSiRXh5HxxWr7ZKJVW30rJllm7P\nm2fBoWdPe27BAnuNb80n+uwzCypRojKH+fNTt0Ag/l6zZlmleuaZ9niy1yXLHKqq7LP075/6/XKh\nqCiePXToYIGhf38LponB4aWXrH81KNit9OKL1t87b148OEyZUrOvOtGECfG+XLCK8L33rNX+8sv2\n46/NOEyHDratR43K/rVe377x83899JBVEldfbd08c+da5ZRLu+1m2+K++2zfAfj+962yTbVN82G3\n3Ww/+fJLy67Ky61b+Lbb7JQ6L7xgXXpDhsTLnonTT7dMMaoRmahDB+theOklCz7Tp+cmWPftG/14\nZWUllZWVdfIeuQwOS4Fg1bcHlj0EHQX8Mnb7v8CnwD7AtMSVBYODF8wcfAWXLDg0bx7vq9261QZ+\nX389eSs8GBxWr7YvuF27msuUldk6fXCYMsXK0bVrPHPo3j0+3nH//XZiwP79reL57LPoyA/h4LBl\ni6WsyTKN4DbZutUGxn0FO3Ro8h9ussxh1iwrp+/LzLfE4DB4sG2v4DZZu9YCxje+UfO1PnP49FOb\nTNC/v00o2LQp3ured9/k7/3MM/Hb1dVWqRxzjLUyx42z28ky1FQ6dbJ96NRTs3+tt/vu1sgpL4dd\ndrGW8j33wNixNrgd1dCpS9262XcSDHCDB+f2PTN13nm2bU85xX6Xe+1lGV/PnpbpXXaZZX/Zat48\neWUcZfBgC5ijRmV2Usq65BvO3i2pzjaaRi67laYBfYFeQAvgLODphGXmAb5XuwsWGD4hQ4ljDsHH\nErVoYRUFWOX55ZfWdRDsrwtq3twqojVrrPLfZZfwMj5zWL7cgsOWLbZcs2Y1g4OvsC+/PD5YVVVl\nASAx4Hh+QHrpUvjzn62CmjgxfT9iaanNYli+3Fo7gwenHphKljnMmZPdTJy6Fhx38N1bfhzIe+MN\nezwxgPng8Nprll4HM5FXXrEutqhZT96UKfHbo0db98nJJ9sMleees5M/1kbXrhacMunKSMa3eEtK\nbNCxVSubtfbvf+9cRpKpo4+2said+Qy50qKF/f78uMwTT9j437nn2iD6D3+Yn3Icc4yNf95wQ37e\nL1dymTnswGYjvYjNXHoQmAv43trxwK+Ah7HxhmbAj4EUvcE1BTMHL1W30vr1dnvbNqs8Vq8OTy3z\nfOawcaMtFzV1zQ9Ib9hgA77NmsWDiA8OAwfa6323kh8ITdWlBPExh1//2lL4++6zlmG6GTKlpfGx\njp49089aSZY5fPhhYQQHf3bQiRPj34k3bVr0YKbvVpo+PT5m4tf3+utWmT72WPT7rl1r3UhFRZbd\nvfuuffennmqBBWofHCC6kZGt4cNths13v2v3f/pTm96ZmEHlQvPmcOCBuX+fndGli2UOTz1l2UKf\nPvY7z6RbqC6cdpqNDXXpkp/3y5VcX8/hn7G/oPGB26uBWs+tCGYOvoJLFRw2brSW+tatVqmvWpW8\nWyfYreRcdAvfD0ivWWM7X1lZ/MdfVmZdO926WVDyA9L++SVL0geHHTvi2c7SpVZJpevOKC21sYle\nvWywMjhtN0qqzCGXs17S8ZX5okU2gaBz53g2502bFj0w7DOHqVNt7CC4vnfesT7oRx4Jv845m2I6\naJBlJWvWWKBdvdr2qwEDLBvLdD5+rjz7bM37JSW5GfRsqDp3tn1j+nTrYmrZ0hpp+dKqlWVYDV2D\nuZ5DlMTZSsHHEvnn27ePZw7r1yfvViopsTnLfmAzKjj4bqX16+35YHBo1coqlY4daw5I+8wh2Syp\n4PsnBoeoaY9Rn3PePEunL788/QySZJnDggXZ9bPWtdatLZjPnh0fFA92KzlnFUDUsR6tWlm33fz5\n8anCrVtb18LChTZPPapb6c03rUU+eLB9ny+/bK1k3+Do29feUwpbly6WaZ51Vv6P0WlMGkVwaNUq\n3u+cKnOAmpkDpB5z+OILq1Qgev52MDi0bx8ODmCDd6Wl1lURLN+aNam7GGobHEpLrTz+wKh0mjUL\nB4ft260iTTyuI598Sz8YHILdSqtWWcYTtU1atrRWY58+8X2kdWsLGP372/cUFRzefdcOsBozxvaT\nV17JT1eN1K0zzrDupCuvrO+SNGyNIjgEM4dUs5Ugu+AA8ZZ+tpmDX2+HDlY2f+U1XxGvWZP6QCw/\nIO3HSbIJDpDZshDdrbRkiU0LrK+ZSmDbc8MG65rr188eC3YrzZlj3WxR0wRbtbJximDfuP8+/Ays\nqODw/vs2hlFcbI2BV16JzkyksHXqFD+NitRegw8OzZvbjznTbqV27eLdSpA+OLRpY+tMFhw2b7YM\no107q5SSZQ7+UpE+OKxdmzo4+AFpnzksX27jF+n4z5/pYFhUt5I/kK8+tW9v2/Xjj+PHaAS7lebM\nsUHYKK1a2WyV4DEa/vvYb7/o4LBmjQ1W+4MF27Wz4wny2VctUkgafHDwmUK6zCFZt5LvNkrkg0NZ\nmf1FdSv5vu3t2+12VLeSP8guKnPIpFvJn3Zj/Xqb255OtsEhKnPwB/LVp3btLDgsWBAfAA5W6nPn\nRs8gg/g+EHzeZxi9e0cHh1tvtfPs+GDgt2O6gw5FGqsGHxx8JZxNcMgmc/CVfrLMYeVKe66oKBwc\nysrsfUtLbXAa4hVxJpmDP31H69aWgWRyVGddZA6pDs7Ll/bt4ZNPbLv6bRo8MHDBguQHBPp9Iur5\ngQPtMztXMyh++KGdN8cHEX/m0VwfVCZSqHI9lTWngplDcbH9sNN1K7Vta5WCP8AqWXDwFXFZmVXO\nqbqVfEV8zTXx2TFlZfGjr4NlyiZzqKqyz1denv7Ux54PapnO6Q5mDq+9ZpXvihXxfv760r59fFDZ\nV9jBFv8nn1gWEMVvg8QDBoMnY/Pr8keOJ56axHcDijRVDT44+EqzqMgCQLrMwQ9ef/GF3U8WHIqK\nrALxGUBUt5IPGL7yHzo0/lyrVvFjDIKtT18RZzIgvXq1BRifOWTCT73NdApfMHN4+mn73CtW1M/Z\nWIPat7dxhdNPjz/mB6Srqy27STYu4s9amuqYEB8cWraMHy0f7Err1av+j2cQqU8NOjjsvrsdqu41\nb55+tlKmwcG/xo85RGUOfp1+PCGoVavUmcPatekzhzVrLGsoLc38yFp/NspMBTMHfzqRFSuir3GR\nT+3bW7mCBwoGT6PeoUPy7+7gg5OPR3jBLOTjjy0QBM8/9cYbuT27qUiha9BjDt262QVTPN+/HyUY\nHEpLbRaQHydIxmcOp52WvJulbVs7WCtRVHAoKrIKb+NG6+JINhgOVhGuWmXraNMm8+sPHHecnfQr\nU8HM4fPP42e0LITgADWDg6/QU3UpgZ1Pac6c1OsPBodPPw1nCWVlmXfliTRGDTo4JErVreSvrlZa\nGu/mKS/PLHP4n/9J3oWRbDbLnnvGp0UGj86uro5XvqlapiUl1q1UXm5BJNPMYc897TTFmQoeBOeD\nw4oV9X9eGJ+pBQ/E891Kb7+d/GpcmQoGh0KYuitSaBp0t1KiVMEh+LyvrDt2zCxzSOWpp+LXiQg6\n9lj7g3jm4LtKMum28ZlDv3429bYuTtgWJdit9PnnVtbi4vo/62ZU5uC7lf75T7uGwc5QcBBJrdEF\nh1SDkInBo23bnQ8O3bunPxo5MXPIpGXuB6TLy601n+1lLTPlu5X81c+2b6//rAGSdytt3WrnQJo8\neefWH7ym+MKFdXPlLJHGpFEFh1NPTX2mUx88fDfKPfekPj1CSUnq4JGpYObgg0MmmcOaNTbmcO21\nya/7sLN85rBhg1W8K1fW/wFwYJnL6afXnJLbvLmVr3Xrnc9slDmIpNaogoO/kE4yfjbTHnvYVMij\njkq/fF0MSta2W2nTJgsOuaysfeawalX8Gga56sLKRlGRXawlqKTEruhWF4PlweCQ7toaIk1RoxqQ\nTsd3K02alD6QQHxAui7eF8ID0qmUlMRfk0s+c1i7Nt5Kz+Q0HfWheXMLDl271s26tm+3ALx5c+66\n7UQaqiYXHEpLbQbMddelXz5XmcPatekrYB8cUk13rQs+c6iqio+dFELmEMUHh7rMHJYvt2CjYxpE\nampywSGbi3/kKnPYvDl90PEHZOU6OPjMoSEEh1x0Ky1blvnpzUWakiYVHA48MLPTXnt1mTm0b2+D\nyl99lVmRzV5bAAAQmUlEQVRwqI/MoXNne99C7lbatKnuupXmzbPgkM0+IdJUNKng8Mgj2c1KOess\nOOCAnX/fZs2sxduihVXEW7YUTnDwB8FVVVlQaNOmsDMHqJupti1a2NXC3npLmYNIlCYVHLL1ox/V\n3aUy27aNd+EUUuYQ7FbyJ/kr1ODgT4Gy6647vy5/1t2ZM+smExFpbBQc8sh34RTSmEOwW8kHh0Lu\nVoK6mVnkT7w4b15hHPQnUmgUHPKokDOHdessOFx4YfLLb9Y3v03qInNYt87+r1yZ+bUvRJqSRnUQ\nXKHz/fubN6efNVUfA9IdOsA55+T2/XZGXWYOPjiAXZBeRGpS5pBH2XQr+Xn3uT5tdOKYQyHzJwiM\nuvBSttaujd9W5iASpuCQR74izmS2kj+1Q64PzvIB64svcn809s5av97+18U28VfMA2UOIlEUHPKo\nuNjOBFpdHe8iSWbr1vyUyQesjRvr/zTd6fhB5LrUpo0u6iMSRcEhj5o1s9Nvt2qVvvUbbNnmks8c\nmlpwmD0bxo1T1iCSTK4HpE8CfgcUA38EfhOxTAVwF9AcWB273ygVF1slnElLNZ+ZQ0MJDmefbVeC\nqwv9+tkZaJ99tm7WJ9LY5DI4FAP3AscDS4F3gaeBuYFlOgD3AUOBJUAdTFIsXMHMIZ18BoctW6xr\nKdWFkgrB0UfbX13p3x9efLHu1ifSmKQKDv+TcN8Bq4A3gE8zWPdhwAJgYez+JGAENYPD2cDfscAA\nljk0Wj5zyOTkf/6CRLlWXGwDvW3a6MykIhKXasyhLdAm8NcWOBR4ARiZwbq7A4sD95fEHgvqC+wC\nvAJMA87LqNQNVDaZww03wBtv5KdMPjiIiHipMoexSR7fBXgZeDTNul0G798cGAgMAcqAKcBU4ONQ\nYcbGi1NRUUFFA7zob3GxBYdMjvBt375uu1BSlUnBQaRxqKyspLKysk7WVZsxh7XpFwFsnCF48cU9\niHcfeYuxrqTNsb/XgINIExwaqmwyh3xp1sxmASk4iDR8iQ3nW265pdbrqs1U1mOBdWmXsm6ivkAv\noAVwFjYgHTQZGIQNXpcBhwNzalGmBqEQg4PPHHJ9mg4RaVhSZQ6zIx4rB5YDozJY9w7gCuBFrPJ/\nEBuMHh17fjwwDxvDmAVUA3+gEQcHf0xBNlejyzWfOfTpU98lEZFCkio4nJxw3wFrgI1ZrP+fsb+g\n8Qn374j9NXr+GgL+sqGFQGMOIhIlVXBYmK9CNBX+Gg2FFByaNbMTASo4iEiQTp+RRz5zSHdepXzy\nZVJwEJEgBYc8KsTMIV9XnBORhkXBIY8KOXMoK6vfcohIYVFwyKNCzhwKaXqtiNQ/BYc8KuTMQcFB\nRIIUHPKoUKeygoKDiNSk4JBHviIuxMyhkA7ME5H6p+CQR8ocRKShUHDIo0LOHBQcRCRIwSGPCjFz\nUHAQkSgKDnlUiJmDupVEJIqCQx4pcxCRhkLBIY8KOXPQbCURCVJwyCNlDiLSUCg45FEhZw4KDiIS\npOCQR8ocRKShUHDIo0I8t1JRkf3XmIOIBCk45FEhnpV12zb774OEiAgoOORVIWYOPjiIiAQpOORR\nIWcOIiJBCg55pMxBRBoKBYc8KsTMYcAAGDiwvkshIoVGwSGPCjFz6NkT3nuvvkshIoVGwSGPCjFz\nEBGJouCQR4WYOYiIRFFwyKNCPEJaRCSKgkMeFeK5lUREouQ6OJwEzAM+Bq5PsdyhwA7gtByXp14p\ncxCRhiKXwaEYuBcLEPsDI4H9kiz3G+AFoFGfxMEHh5KS+i2HiEg6uQwOhwELgIXAdmASMCJiuR8B\nTwCrcliWgrF6tYKDiBS+XAaH7sDiwP0lsccSlxkB3B+773JYnoLQsWN9l0BEJL1ctmEzqeh/B9wQ\nW7aIFN1KY8eO/fp2RUUFFRUVO1c6EZFGprKyksrKyjpZVy77+I8AxmJjDgA3AtXY+IL3SaAMuwKb\ngIuBpxPW5Zxr9EmFiEidKrJz8deqns9lcCgB5gNDgGXAO9ig9Nwkyz8MPAM8GfGcgoOISJZ2Jjjk\nsltpB3AF8CI2I+lBLDCMjj0/PofvLSIiO6GhTB1V5iAikqWdyRx0hLSIiIQoOIiISIiCg4iIhCg4\niIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4iIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiI\nSIiCg4iIhCg4iIhIiIKDiIiEKDiIiEiIgoOIiIQoOIiISIiCg4iIhCg4iIhIiIKDiIiEKDiIiEiI\ngoOIiIQoOIiISIiCg4iIhCg4iIhISD6Cw0nAPOBj4PqI588BZgKzgDeBA/NQJhERSaEox+svBuYD\nxwNLgXeBkcDcwDJHAnOAL7BAMhY4ImE9zjmX46KKiDQuRUVFUMt6PteZw2HAAmAhsB2YBIxIWGYK\nFhgA3gZ2z3GZREQkjVwHh+7A4sD9JbHHkrkQeD6nJRIRkbRKcrz+bPqCjgV+ABydo7KIiEiGch0c\nlgJ7BO7vgWUPiQ4E/oCNOayLWtHYsWO/vl1RUUFFRUVdlVFEpFGorKyksrKyTtaV6wHpEmxAegiw\nDHiH8IB0D+A/wLnA1CTr0YC0iEiWdmZAOteZww7gCuBFbObSg1hgGB17fjzwM6AcuD/22HZsIFtE\nROpJrjOHuqLMQUQkS4U8lVVERBogBQcREQlRcBARkRAFBxERCcn1bCURkVrZZZddWLcu8rAnSVBe\nXs7atWvrdJ2arSQiBamoqAj97jOTbFtptpKIiNQpBQcREQlRcBARkRAFBxERCVFwEBGppRtvvJG7\n77475+/zzDPP8L3vfS/n7xOk4CAiUgurVq1i4sSJXHrppQBMnTqVE044gY4dO9K5c2fOPPNMVqxY\nkfG6Ro4cSffu3enQoQODBg3inXfe+fr5k08+mQ8//JDZs2fn5LNEUXAQEamFCRMmMHz4cEpLSwGo\nqqri0ksvZdGiRSxatIi2bdtywQUXZLSujRs3cvjhhzN9+nTWrVvH+eefz/Dhw/nyyy+/XmbkyJE8\n8MADOfksUXScg4gUpEI/zmHIkCFceOGFnH322ZHPT58+nYqKCtavX1+r9bdv357KykoGDBgAwFtv\nvcW5557LJ598ElpWxzmIiBSI2bNns88++yR9/rXXXqNfv361WveMGTPYtm0bffr0+fqxfffdl4UL\nF7Jx48ZarTNbOn2GiDRYRXXU91GbBKWqqoq2bdtGPjdr1ixuvfVWnn766azXu379es477zzGjh1b\nY/3+dlVVFW3atMm+wFlScBCRBqs+e53Ky8vZsGFD6PEFCxYwbNgwxo0bx9FHH53VOjdv3szJJ5/M\nUUcdxfXXX1/jOf9eHTp0qH2hs6BuJRGRWjjwwAOZP39+jccWLVrECSecwM9+9jPOOeecrNa3detW\nvvOd79CjRw/Gjx8fen7u3Ln06tUrL1kDKDiIiNTKsGHDePXVV7++v3TpUo477jiuuOIKLrnkktDy\nEyZMoHfv3pHr2r59O2eccQZlZWVMmDAhcplXX32VYcOG1UnZM6HgICJSC6NGjeL5559ny5YtAPzx\nj3/k008//XqsoG3btrRr1+7r5RcvXsygQYMi1/XWW2/x3HPP8dJLL9GhQ4evX//mm29+vcykSZMY\nPXp0bj9UgKayikhBKvSprABjxoyhc+fOXHXVVWmXHTp0KOPGjUs5wymZZ555hkceeYRJkyZFPp+L\nqawKDiJSkBpCcCgUOs5BRETyQsFBRERCFBxERCREwUFEREIUHEREJESnzxCRglReXu5n20ga5eXl\ndb7OXG/5k4DfAcXAH4HfRCwzDvgWsAn4PvB+xDKayioikqVCncpaDNyLBYj9gZHAfgnLDAP6AH2B\nS4D7c1ieRqGysrK+i1AwtC3itC3itC3qRi6Dw2HAAmAhsB2YBIxIWOYU4E+x228DHYAuOSxTg6cd\nP07bIk7bIk7bom7kMjh0BxYH7i+JPZZumd1zWCYREclALoNDpoMEif1hGlwQEalnuRyQPgIYi405\nANwIVFNzUPr/gEqsywlgHnAMsDJhXQuAvXJUThGRxuq/2LhuQSnBCtYLaAHMIHpA+vnY7SOAqfkq\nnIiI1J9vAfOxlv+NscdGx/68e2PPzwQG5rV0IiIiIiLSOJyEjUN8DFyfZtnG4CFsvGV24LFdgJeA\nj4B/YdN9vRuxbTMPODFPZcyXPYBXgA+BD4ArY483xe3REpvqPQOYA/w69nhT3BZeMXbA7DOx+011\nWywEZmHb4p3YY41+WxRj3U29gOZEj1k0Nt8EBlAzOPwW+HHs9vXAbbHb+2PbpDm2jRbQuM6VtRtw\ncOx2G6x7cj+a7vYoi/0vwcbmBtF0twXAtcAjwNOx+011W3yKBYOgRr8tjgReCNy/IfbX2PWiZnCY\nR/zAwN1i98FaAMFs6gVsUL+xego4Hm2PMuBd4ACa7rbYHfg3cCzxzKGpbotPgY4Jj9XJtijkqJHJ\nQXRNQRfiU3tXEv/Su2HbxGvM26cXllG9TdPdHs2wVt9K4t1tTXVb3AVch02N95rqtnBYoJwGXBx7\nrE62RSGflVUHw4U5Um+XxrjN2gB/B64CNiQ815S2RzXWzdYeeBFrNQc1lW3xbeBzrI+9IskyTWVb\nABwNLAc6YeMM8xKer/W2KOTMYSk2KOntQc2o11SsxFJDgK7YDwPC22f32GONSXMsMEzEupWgaW8P\ngC+A54BDaJrb4ijsnGyfAo8Cx2H7R1PcFmCBAWAV8A/snHaNfltkchBdY9SL8IC07ye8gfDgUgug\nN7atGtPJ74uAP2NdCEFNcXvsSnzGSSvgNWAITXNbBB1DfMyhKW6LMqBt7HZr4E1sBlKT2BZRB9E1\nZo8Cy4Bt2HjLBdhMhH8TPS3tJmzbzAOG5rWkuTcI60qZgXUhvI9NbW6K26M/MB3bFrOw/nZomtsi\n6Bjis5Wa4rboje0TM7Dp3r6ObIrbQkRERERERERERERERERERERERERERKQh2Rj73xMYWcfrvinh\n/pt1vH4REckRf06mCuJH1GYq3fnHEs/3JCIiDYSvwKcCVdjR1ldh5xa7HbtIykzgkthyFcDrwGTi\nJzJ7Cjvz5QfEz355G7Ajtr6Jscd8llIUW/ds7KjmMwPrrgQeB+YCfwmU8zbsbKszY68VEZEc8sEh\neC4esGAwJna7FLtOQi+sAt+IdUN55bH/rbAK399PzBz8/dOxUxcUAZ2BRdjJ0CqwANUt9txb2Jk1\nO1LzjJrtMv1wIrlQyGdlFalriScZOxEYhbX8p2LnpOkTe+4drEL3rsLOYTMFO7Nl3zTvNQj4K3ZK\n5M+BV4FDY/ffwc6h5WLr7IkFjC3Ag8CpwOZsP5xIXVJwkKbuCuxCQgOAvbATlgF8GVimAjsL6hHY\nNRXex67rnIojHIz8ufO3Bh77Cjs1+VfY6ZafwK5Z8AIi9UjBQZqSDcRPcQx20ZzLiQ867038Ws1B\n7YB1WMt+X2peWnE70YPWrwNnYb+xTsBgLGNIdork1tjZM/+JXR/5oLSfRiSHCvlKcCJ1xbfYZ2It\n9BnAw8A4bIxhOlZpf4516SRePesF4FJgDnYK+SmB5x7ABpzfA84LvO4f2HXQZ8Yeuy62/v0IX33L\nYUFrMpaRFAHX1PrTioiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiIiDRm/x996qatrF5yPAAAAABJ\nRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbeb802a828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "agent = PassiveTDAgent(policy, Fig[17,1], alpha=lambda n: 60./(59+n))\n",
    "graph_utility_estimates(agent, Fig[17,1], 500, [(2,2)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to plot multiple states on the same plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VFX+x/F3pEkgIYEEUEJTFFREEBcLqBEsoIKyFooV\nQWHVXVx1F1n9aRQXARURBUQRpSi4AlKkiWKQ3glBeu+9JISSMt/fHzcZktyE1CEh+byeJw9z7z1z\n5sxJmM/cc24BERERERERERERERERERERERERERGRImk4cACIzmT7E0AUsBqYDzS4QO0SEZECdDvQ\niMzD4VagQvLjlsCiC9EoEREpeLXIPBxSCwZ2+7YpIiKSHZcUdANS6QxMK+hGiIjIhVGLrPcc7gLW\n4uw9iIhIAStZ0A3AmYT+CmfO4VhGBW644QaLioq6oI0SESkCooCGuXliQQ8r1QAmAE8CmzMrFBUV\nhZnpx4x33nmnwNtQWH7UF+oL9cX5f4Abcvvh7Os9hzHAnUAIsAt4ByiVvG0o8DbOUNKQ5HUJQBMf\nt0lERLLg63DokMX2Lsk/IiJSiBT0sJLkUHh4eEE3odBQX5yjvjhHfZE//Aq6AdlkyeNnIiKSTX5+\nfpDLz3ntOYiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiIiIiLwkFE\nRFwUDiIi4qJwEBERF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKicBAREReF\ng4iIuCgcRETEReEgIiIuCgcREXFROIiIiIvCQUREXHwdDsOBA0D0ecoMBDYBUUAjH7dHRESywdfh\n8A3Q8jzb7wfqAFcBLwBDfNweERHJBl+Hw1zg2Hm2twFGJD9eDAQBVXzcJhERyUJBzzlUA3alWt4N\nhBVQW0REJFlBhwOAX7plK5BWiIiIV8kCfv09QPVUy2HJ61z8/CJSLYUn/4iIyDmRyT95l/5buy/U\nAqYA12ew7X7g5eR/bwEGJP+bnplph0JEJCf8/Pwgl5/zvt5zGAPcCYTgzC28A5RK3jYUmIYTDJuB\nOKCTj9sjIiLZcCH2HPKD9hxERHIoL3sOhWFCWkREChmFg4iIuCgcRETEReEgIiIuCgcREXFROIiI\niIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiIiIiLwkFERFwUDiIi4qJw\nEBERF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcikQ4mBlx8XEF3QwRkSLjog6H0wmn\nGbRkENcNvo56g+qRkJSQZvuCXQsYsnQIAGcTzzJi1Qjemv1Wnl4z9mwsXy3/ikNxhzItcybxDGaW\np9eJ2h/FvJ3z8lSHZN+fB//kYNzBgm6GSK4kehJZuGshkzdMzrc6L8pw8JiHocuGcuXAK5m5ZSZf\nPPgFNSvUpMmwJmw+upmV+1Zy14i7eGLCE7z1+1t8vOBj6g2qx+jo0fRf2J/TCacB5wN44OKB2XrN\nUwmniIiMoPantXnr97eYuH4i4PxSUuo7evoob81+i9APQ5mwbsJ564s9G8ve2L2u9Qt2LaDFyBY0\nH9mcF6e+mJNukXT2n9xP9IHoTLefOHOCzxZ/RuMvG9N0eFOenPBklqG+6cgm3v79bU6cOZHfzZVi\n5PCpwwxfOZwHv38Qv3f9WHdonXdbQlICs7bM4j+//YfdMbuJ2h/FtE3TXHUcijvE0GVDaT2mNSH9\nQvjb1L/x9+l/5+6Rd7P12NYL+XYKlKXYfGSz3fHNHXbLsFts+d7l3vXL9iyzR354xK4ddK2F9gu1\nL5d9afGJ8fb+nPft5q9utt+3/W5mZo2+aGR/bP/DXp/5ulX+sLKVfb+seTweO59J6ydZjU9qWPtx\n7W3zkc32+eLPrcukLrZszzK7btB1Fv5tuH23+jur8mEV6zyps/3t57/Z6zNfz7S+n9b9ZERg9T6v\n5123N2avPTz2YavxSQ0btnyYxZ6NtbLvl7VT8ads9tbZtv3Y9vO2saAkeZJsVNQom7l5ZkE3xWt/\n7H578ecXLaB3gNX9rK53/bZj28zj8diRU0fsjVlvWMW+Fa39uPY2a8ssO5NwxuoPrm8//vljhnUu\n3LXQ2oxpY6H9Qq32gNo2JnrMhXo7cpFbumepTdkwxc4knLEx0WPs7pF3W+AHgfbo/x6171d/b0+M\nf8L6zO1jK/ausG5TulmlvpXs5q9utnY/trOy75e1mp/UtIDeARZzJsZOnj1pw5YPsxYjWliFDypY\nh3EdbGz0WDt48qCZOX/jtQfUts8Xf25mZkDehjAuAubxeGzm5plW+cPK1n9Bf0tMSnT9Eg6cPGCv\nz3zd9sfuz/QX1XlSZ6vwQQV78PsH7VDcIQvtF2p7Y/ZmWDY+Md5enfGq1RpQy2Zvne1dv3j3Ygvq\nE2Sh/UJtdNRoqzOwjtUfXN+W7F5iZmbTN023m7+62fbF7ktTX+zZWOs4vqNdNfAqm7B2glX4oIIl\neZJsbPRYC+0Xam/99padSTjjLd/wi4bWanQrK9+7vNUfXN8+mv9RNv8c09p6dKsdOHkgV889ny1H\nt1iz4c2swgcVrMukLvlef04lJCVY7z96W6W+leyfM/5p+2P3W1CfIFu5b6U98sMjRgTWZVIXC+0X\nai9MfsG2Ht2a5vkzN8+0Rl80SrNu27Ft9tj/HrOw/mE2ZOkQi4uPs08XfWoNhjSwdYfWXci3JxeR\nJE+SjftznP3ly79YzU9qWrWPq1lIvxBrPqK5jY0ea6fiT3nLTt041Ur3Km01Pqlhveb0sh3Hd5iZ\nmcfj8X6haTm6pbUc3dIq9q1obca0sfFrx6epI7Vhy4dZx/EdbfL6ycUjHP45459W+cPK9sf2P/L0\nS5u7Y64NWTrEu7fQ5KsmNn/nfO/2hKQE6/lrT1u+d7k1H9HcWo1uZUdOHUlTx9nEs9Z9enfvt/lD\ncYfsbOJZ7/YDJw8YEdh1g67zrtt1Ypc1/KKhdZrYyeLi48zMLKx/mHWZ1MVqDaiVZi8oRc9fe9rT\nPz1th+IOWfMRzY0ILMmTlK33+dvW3+ylqS/ZT+t+svK9y9t7ke9ls4ey57vV31lIvxD7eMHHNmLV\nCOs4vmOO6zidcNrVt+cTnxifafm1B9faTV/eZPeMvMe2HN3iXf/Adw/Ype9fam/99pb9sf0P++sP\nf7XV+1dnWMep+FN26fuXWnxivHk8HhuydIiF9Aux9yLf8/7OzMw2Ht5oRGAPjXnIVcfWo1sL7V5e\nXsTFx9nbs9+2katG2huz3rChy4baol2L7MDJA9Z5Umcbv3Z8QTexUPB4PDZ+7Xi7fvD11nhoY/tp\n3U+WmJRo+2L32aYjmzJ8TmJSoi3YuSDDL7wpZm2ZZW/+9ma2/rbWHlxrRGANv2hYqMOhJbAe2AT0\nyGB7CDADWAWsAZ7NpB6r1LeSrdi7Ite/tMy0+7GdjY4abWZOMLQf197C+odZyfdKWpdJXc77Czuf\n8WvH29WfXW1mZpuObLLq/atb33l90wxhtR3b1m4ddmu2v9WH9Q+zl6a+ZPGJ8WnWxyfGW0JSgnd5\nTPQYC+0XanU/q2vVPq5m7ce1P+8wV054PB6L+D3Cag+obav2rTIzswlrJ2T4QZmRJE+Szd853/bF\n7rPGQxtb6+9bZ+t5h+IO2e3Db7c7vrnDtW3iuokW0i/Evlj6hWuIMPpAtP158M9svYaZ2dWfXW0L\ndi6wNmPa2E1f3pTpc7cf226h/ULtwe8ftLZj21qPWT1sdNRoK9+7vHWd0jXL10lISrCP5n/kHQ4o\nCB6P57xDqqcTTltCUoIt3r3Y6gysYw+PfdgCPwi09uPaW6W+lazCBxUstF+o1R9c3/79y78vYMvz\nxuPx2NcrvrZKfSul+X+TV+sPrbcWI1pYgyEN7OcNP2c5XO1Law6sMY/HU2jDoQSwGagFlMIJgGvS\nlYkAPkh+HAIcAUpmUFea4Zb89MasN6zXnF5mZtZ9endrPqK5xZyJsf+t+V+2v6Vn5HTCaSvTq4xt\nO7bNan5S075c9qWrzPHTx9PscWSl1ehWRgQWuS3SzMwmr59sMzfPtBuG3OAdcvpm5TcW1j/MVu9f\nbftj99uhuEM2eMngbH1gZcXj8VjXKV2t8dDGaYbMZmyaYfeMvCfL5ycmJdpzE58zvwg/C+kXYq9M\nf8WC+wRn2c+bjmyyKz+90l6d8aqV+285+2DuB/bS1JfMzOzD+R9aWP8w75BeXj3ywyMW0DvAXpj8\nQpa/mxuG3GD/nPFPazOmjTX6opHVGVjH+sztY3ePvDvT54yOGu0tQwQ2fdP0fGl3Tq07tM6u/uxq\nGxU1KsPtK/ausNoDaluDIQ2s8oeVbdyf48zMvP8PE5MSbfX+1Ra1P8qGLR9mnSZ2yvI1c/tFK69m\nb53t3bPZF7vPHvjuAWv0RSMjAu8QTl4kJCXYe5HvWaW+leyThZ/ka+DkFXkIh4w+iPNLE5xw2J68\nPBZ4CFiXqsw+oEHy40CccEjMqLIyJcv4pJHXV7mecWvHMXzlcGZsnsHCzgsJKBPAY9c9lqd6Ly15\nKZcFXEbzEc3pdlM3nm/8vKtMhUsr5KjOSe0n8fbvbxM+Ipznb3yeCesmcPT0URpUacDC3Qv5eePP\nvPHrG0Q+G0m9kHre5wWUCSA2PjZP7weg5289Wbl/Jb8/8zsBZQK868uVLkdcwvnPM/GYh+cmP8fO\nEztZ/sJydsfspnXd1kzeOJn1h9dzbei13nK/b/udFle0AGD78e20GNmCns160u2mbizbt4xhK4Zx\nMO4gVcpVYdTqUSzqvIhqgdXy/P4Anr7hae698l6ev/F5/Pz8zlt2ZdeV3jLxSfF4zMPumN18sfyL\nDMv3m9+PwUsHU6ZkGR686kHKlSrHyfiTuWrn/pP7eWjsQ/S9uy/htcJz9NwZm2fw9E9Pc2XFK1l/\neL1r+88bf6bTpE4MuG8Au2J20fH6jtSoUAM49/+wxCUluL7K9QBsO7aNw6cOZ/p6HvPw/OTnGb5q\nOPbOhfkiuy92H/0X9se/lD+fL/2cCmUqULlcZdqNa0enhp2Y0G4C4d+Gs/PETu97y43dMbvpOL4j\npUqUYlW3VYQFhuXjuyhYvgyHasCuVMu7gZvTlfkKmA3sBQKAx33Yngw1r92cbj93Y+7Oucx5dg7B\nZYPzre5rQ6+lQpkK9Gia0YhazpUqUYqH6z3MqNWjWLBrARPaTSD40mBKlyhNs2+a8ceOP5jcYXKa\nYAAIKB1A7Nm8hcPAxQOZuH4i856blyYYAPxL+XMq4dR5n/9/s/+Prce2MvPJmfiX8qfRZY0ACK8Z\nzqwts7zh0GNWDz5a+BEHXj9AkieJFiNb8Pqtr9Ptpm4AfNX6K4IvDebhHx5m5OqRRD4TmW/BANCm\nbptsl00dHqVLlAagZoWa7I3dS+dJnalRoQbvhL8DQERkBOPWjmP+c/O5POBy/Pz8eHbis7kKh50n\ndnL3yLs5evooO0/szNFzf1r3E92mduOndj+x6egmZm+bDTiHZL849UVizsYwZ8ccpnacSpNqTbJV\nZ4h/SKbhEJ8Uz7MTn2X5vuWU8Ctx3nrWHloL4P1byK3VB1bTekxrrgm5hkRPIqu7reb2b27n4bEP\nM6rtKFpd1QqAGhVq5Lj/Ulu8ezFtf2jLy01epkfTHpS45Pzv72Ljy3DIzleE/+AMN4UDVwKzgBsA\n1ydZRESE93F4eDjh4eH50ESoWr4qV1W6iucaPpfnP8r0Rj48ksAygVl+A82Jm8NuZveru9Os85gH\nj3n4oMUH3BJ2i+s5AWUCiDkbk+vXnLtjLr3n9mZRl0WE+Ie4tpcrVe68Z6h/H/0936/5niVdluBf\nyj/NtrbXtKXf/H50v6U7X6/4mikbp9CoaiOW7llK73m9ebrB0/z95r97y19d6WoAhrUeRnDZYKqW\nr5rr9+ULpUqUIiwwjMgdkQSUDuClJi/xw5of+C76O+Y/N5/K5Sp7y5YvXT7HoX0w7iDNRzTnpb+8\nxMYjG3N0ZYBxa8fx8rSXmf7EdG687EbOJJ5hV8wukjxJPDnhSY6dOUaNwBrMeXaOt5+zI304JHoS\nMTM85qHtD20pVaIUS7osIfTDUMwsw/8PS/cspcmwJjx49YNM6TAlzbYjp45wNukslwdcnmVb5u+c\nT9sf2jKw1UDa12/vXT/8oeGEBYZRp2Id77qaFWqy4/gOVx3TNk0j+kA0PZpl/qVuwroJdP25K8Pb\nDKd13dZZtutCiYyMJDIysqCbkaVbcCabU/TEPSk9DWiaavk34KYM6vLpuFxOxv0LqyOnjmQ6AbZ0\nz1K7ceiNua43rH+YTds4LdMyO4/vtGofV8tw25oDayykX4hF7Y/KcPuZhDMW1CfIftv6m4X0C7G1\nB9faP6b9w6p9XM1af986T/M+BWXujrl2KO6QEYERgV3+8eWuw2bNnPmu3n/0zna9J8+etCZfNbG3\nfnvLzMxem/ma9ZvXL9ttCu0Xaiv3rfSu23B4g13x6RXWfXp3C/823E4nnM52W1I7HHfYgvoEmZkz\n19ZseDN78ecX7bH/PWZtx7b1jsH7/9ffYs/GpnluXHycRe2PssofVrYuk7rYA989kGZ7ypF/t319\nW5btmLdjnoX2C832OTf9F/Q3IrDxa8fb0GVD7c+Df9rk9ZOt8oeVLbhPcKaHuI+NHmtVP6qa4RGG\nhQ2FdM5hGXAVzoT0XqAd0CFdmfXA3cB8oApQF7jgp/alDAlczCqWrZjptrwMK70681X+Wu+v3l3x\njGQ055DoSWTCugn0nd+X3s1706BKgwyfW6ZkGVrVacX9393PkAeGcE3oNTSp1oSpm6Yysu1ILvG7\n+E7ib1ajGeAMgdULqUdYYBi1gmq5ypUvXT7bw0pmxjMTn6Fupbq8d9d73udnNdcDsPnoZh7936OM\najuKhlUbeteHBYax9dhWZm2dxfzn5nNpyUuz1Zb0gssGE3s2ll0ndvHaL68RWCaQL1d8yR0172Bq\nx6mUvMT5mAm6NIjjZ45TvnR5b7tu/fpW4uLj+Oahb7iy4pW8MOUFb72nEk7RZkwb2tRtk+HcCMCB\nkwcILRdK9IFo2v7QllFtR3Hvlfdmq92trmrFrK2z6PpzV07Gn+Ta0GvZdWIXUztOZUTUCAYtHUSn\nhp14ZuIzjHt8HFXLV+XHP3/klZmv8MuTv3jnXCR3WgEbcCameyav65r8A84RSlOAKCAa6JhJPQUd\nwBe1PTF7rOpHVXP0nGV7llmdgXWs5ic1Xd/20judcNpK9yrtXZ60fpLd8c0dVrFvRfvrD3/N8pC+\nyG2R9trM17zlkjxJac4rKKo+XfSp/X3a37NV9vPFn1ujLxqlOWqv37x+9trM1877vFPxp6z+4Po2\neMngDLe/+POL+XJeRspe0u3Db7fTCadt6sapFnMmJk2Z6wZd5z3HJPZsrF076Fr7bPFntubAGjNz\nrhJQ+cPK5vF4rM/cPtZ8RHN7asJTFp8Yb/7/9bcTZ06kqW/D4Q0W1CfIIn6PsBqf1LAf1vyQ43af\nTTxrT4x/wlbvX231Pq/nPY9q4+GNVqlvJbvi0yvshiE32GszX7M52+dYaL/QTPeCCyMK6aGs+amg\n+/iiFnMmxsr9t1yadXtj9trLU1+2ncd3usp7PB677evbrMesHt5zGc7H4/HYJe9eYglJCRZzJsYu\n++gy6z69e4ZDKXLO1yu+ztYhoKv2rbKQfiG28fDGNOsHLxls3aZ0O+9zX576srX7sZ3Pj7n/ZOEn\ntvbg2kzP2jUza/p1U/tj+x/m8Xis3Y/t7LmJz6VpV2JSopV6r5QNWTrE6n1ez9787U3vkO+tw25N\nc5WCuPg4qz+4vrUf196IIN9P8jQzm7Zxmn23+jvbfWK3BfcJtsofVrZZW2bl++v4EoV0WEkKiXKl\ny3E68TQe83iHaf4797+MXTOWUiVK0f++/mnKT9k4hdizsfRu0Ttbwzp+fn7eI5Y+WvARd19xNwNa\nDvDJeylKypcun+UhxglJCTz101N8fO/HXFXpKtfzTyZkPiw1deNUpmycwqpuq/L1oIiMvHLLK1mW\nCS4bzLEzxxi8dDCbjm5iXqd5adpV4pISVC5XmTd+fYOFnRdyTei506Ka1XCOxmtaoykl/Erw0rSX\naFi1Id8+9C2PX/s4D9d7ON/fU+qh1H83/TfVAqpx9xV35/vrFFYKh2LgEr9L8C/lz8n4kwSWCWTn\niZ2MWTOGPi36MHmjc4nfjxd8TPUK1Xnkmkd4c/ab9G6evWBIUa5UObYc3cKgpYNY8cIKX72VIiWg\ndECWcw6fLfmMquWr8lSDp1zbypVOe57EnO1zuCb0GiqXq8zJ+JN0m9qN0W1HE3RpUL63PTeCLg1i\n+d7lDFo6iAWdF1C2VFlXmcaXN6bdde3SBANAeK1w3p3zLqNWj+Jg3EGqV6jOki5LKHFJCdpe09bn\nbX+j2Rs+f43CRuFQTASWCSTmbAyBZQLpN78fz9/4PLdVv42PF37Mzxt/pu/8vpxJPIMfzl7Ag1c/\nmKP6/Uv5807kOzzd4GlqBtX00bsoWrKakN4Ts4fec3uzoPOCDL/5ly9d3nso6/6T+2k9pjW97upF\n91u602tOL+6qdRd31rrTZ+3PqeBLg+k9rzd9WvTJ9FDZSe0nZbj+jpp3EFA6gBa1W/DKLa9QtmRZ\nypUu58vmFnsKh2KiYtmKHDl1hHKlyvF99PeseXENFctWZMORDTwz8RkGthxIxJwIXv3lVQbcNyDH\nwxDlSpdj5paZbP1H8bmOfF5ldp7D4VOHefR/j7Lt+Db+dtPfMv0gTX2Gdc/fehJcNpjog9GsO7SO\n4auGE/23zO9lURBC/ENofFnjbA1BpVe+dHl+ffpXH7RKMqNwKCbCAsPYFbOLX7b8wgNXP5DmhKL3\n73qfJxo8wfTN01m4e2Guxm/9S/nzVIOn8vVs5aIuoEzGw0p95vWhZlBNPrr3IxpVbZTp81MOZY0+\nEM30TdP55qFvuP/7+xm1ehQD7htQ6E4QfOWWV+h+c/cidyZxUaVwKCaqB1Znx/EdDFo6iHGPj/Ou\n3/KPLdQOqg3AC41f4KkGT+XqP+8zNzxDqzqZnwshbgGlAzhxNu0d5fbE7GH4yuGseXFNlmcEpwxL\nvTvnXf5127+4rfptAPz61K/cXvN2n7U7twLLBBZ0EyQHfHsIQ/5JPipLcqvXnF5M3TSVs0lnWdl1\nZUE3R3BObAv9MJTxj4/nQNwBHr/ucV6b+RqGuY4gy8j+k/upOaAmFctWZPPfN1OudDm2HtvKFcFX\nXIDWy8UgeXg4V5/z2nMoJqpXqM7iPYsZcJ8OMS0s/Pz8uLX6rYSPCAfgzpp38m3Ut9kO7/KlyxOf\nFM+/bvuXd3JWwSD5ReFQTFQPrE7JS0rS8frMTkKXgtCsejNOJZzi8KnDvPbLa9x75b3ZvoS0fyl/\nOtTv4L1irUh+0rBSMRFzNobxa8fTqVGngm6KpBKfFM+ZxDO8NO0lRq8ezdLnl3LT5Rlde1Ik5zSs\nJFkKLBOoYCiESpcoTekSpWlYpSE7auxQMEihoT0HkULgbOJZTiWcytebTYnkZc9B4SAiUkTlJRwu\nvovli4iIzykcRETEReEgIiIuCgcREXFROIiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERExEXh\nICIiLgoHERFxUTiIiIiLwkFERFwUDiIi4uLrcGgJrAc2AT0yKRMOrATWAJE+bo+IiGSDL2/2UwLY\nANwN7AGWAh2AdanKBAHzgfuA3UAIcDiDunSzHxGRHPLVPaRfS7dswCFgHrAtG3U3ATYD25OXxwIP\nkTYcOgLjcYIBMg4GERG5wM43rBQAlE/1EwD8BZiBsweQlWrArlTLu5PXpXYVUBH4HVgGPJWtVouI\niE+db88hIpP1FYHfgDFZ1J2dcaBSwI1AC8AfWAgswpmjSNuYiHPNCQ8PJzw8PBvVi4gUH5GRkURG\nRuZLXbmdc1gJNMqizC04AdMyebkn4AH6pirTAyjLuSAahrNnMi5dXZpzEBHJobzMOeTmaKW7gGPZ\nKLcMZ9ioFlAaaAdMTldmEtAMZ/LaH7gZWJuLNomISD4637BSdAbrgoF9wNPZqDsReBmYifPh/zXO\nZHTX5O1DcQ5znQGsxtmr+AqFg4hIgTvf7katdMsGHAFO+qw1mdOwkohIDuVlWMmX5znkJ4WDiEgO\nXeg5BxERKeIUDiIi4qJwEBERF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKi\ncBAREReFg4iIuCgcRETEReEgIiIuCgcREXFROIiIiIvCQUREXBQOIiLionAQEREXhYOIiLgoHERE\nxEXhICIiLgoHERFxUTiIiIiLwkFERFwUDiIi4qJwEBERF1+HQ0tgPbAJ6HGecn8BEoG/+rg9IiKS\nDb4MhxLA5zgBcS3QAbgmk3J9gRmAnw/bIyIi2eTLcGgCbAa2AwnAWOChDMr9HRgHHPJhW0REJAd8\nGQ7VgF2plncnr0tf5iFgSPKy+bA9IiKSTSV9WHd2PugHAG8kl/XjPMNKERER3sfh4eGEh4fnrXUi\nIkVMZGQkkZGR+VKXL8f4bwEicOYcAHoCHpz5hRRbU7UhBDgFPA9MTleXmWmnQkQkJ/z8/CCXn/O+\nDIeSwAagBbAXWIIzKb0uk/LfAFOACRlsUziIiORQXsLBl8NKicDLwEycI5K+xgmGrsnbh/rwtUVE\nJA8ulkNHtecgIpJDedlz0BnSIiLionAQEREXhYOIiLgoHERExEXhICIiLgoHERFxUTiIiIiLwkFE\nRFwUDiIi4qJwEBERF4WDiIi4KBxERMRF4SAiIi4KBxERcVE4iIiIi8JBRERcFA4iIuKicBAREReF\ng4iIuCgcRETEReEgIiIuCgcREXEpWdANEBHJSMWKFTl27FhBN+OiEBwczNGjR/O1Tr98rc13zMwK\nug0icgH5+fmh//fZk1lf+fn5QS4/5zWsJCIiLgoHERFxUTiIiIiLwkFERFwuRDi0BNYDm4AeGWx/\nAogCVgMI5EM+AAALJ0lEQVTzgQYXoE0iInnWs2dPPv30U5+/zpQpU2jfvr3PXyc1X4dDCeBznIC4\nFugAXJOuzFbgDpxQ6AV86eM2iYjk2aFDhxg1ahTdunUDYO3atdx0001UrFiRoKAgmjZtyrx587Jd\nV4cOHahWrRpBQUE0a9aMJUuWeLe3bt2aP//8k+joaJ+8l4z4OhyaAJuB7UACMBZ4KF2ZhcCJ5MeL\ngTAft0lEJM++/fZbHnjgAcqUKQNAtWrV+PHHHzly5AjHjh2jffv2PProo9mq6+TJk9x8882sWLGC\nY8eO8cwzz/DAAw8QFxfnLdOhQwe+/PLCfXf2dThUA3alWt6dvC4znYFpPm2RiEg+mDFjBnfeead3\nuUKFCtSuXRs/Pz+SkpK45JJLuOyyy7JVV+3atXnllVeoUqUKfn5+PP/888THx7Nx40ZvmfDwcKZO\nnZrv7yMzvj5DOidnsNwFPAc09VFbRETyTXR0NHXr1nWtDwoKIi4ujssvv5zZs2fnqu5Vq1YRHx9P\nnTp1vOvq1avH9u3bOXnyJOXLl891u7PL1+GwB6ieark6zt5Deg2Ar3DmJjI8Xz4iIsL7ODw8nPDw\n8Pxqo4hcpPzy6RoPuTkR+/jx4wQEBGS4/tSpU7z77rs89thjLF++POVM5WyJiYnhqaeeIiIiIk39\nKY+PHz+eaThERkYSGRmZszeSCV9fPqMksAFoAewFluBMSq9LVaYGMBt4EliUST26fIZIMVPYL59R\npUoVpk2bRuPGjTPcbmYEBASwYMECGjTI3kGYp0+fpmXLltSrV4+hQ4em2Xb06FFCQkKIiYlxhcPF\nePmMROBlYCawFvgBJxi6Jv8AvA0EA0OAlTgBIiJSqDVo0IANGzZkuj0pKQmPx4O/v3+26jt79iwP\nP/wwNWrUcAUDwLp166hVq9YFGVKCC3Oew3SgLlAH+CB53dDkH4AuQCWgUfJPkwvQJhGRPLn//vuZ\nM2eOd/nXX39l1apVJCUlERMTw6uvvkrdunW98wbffvsttWvXzrCuhIQEHn30Ufz9/fn2228zLDNn\nzhzuv//+fH8fmdEZ0iIiufD0008zbdo0zpw5AzhzAR06dCAoKIi6dety6NAhJk+e7C2/a9cumjVr\nlmFdCxYsYOrUqcyaNYugoCACAgIICAhg/vz53jJjx46la9euGT7fF3TJbhEplAr7nAPAm2++SeXK\nlenevXuWZe+77z4GDhyY4RFOWZkyZQrfffcdY8eOzXC7L+YcFA4iUihdDOFQWFyME9IiInIRUjiI\niIiLwkFERFwUDiIi4qJwEBERF4WDiIi4KBxERMRF4SAikku6TaiIiKSR/jahixYt4p577qFSpUpU\nrlyZxx9/nP3792e7ruJ2m1ARkSIp/W1Cjx8/Trdu3dixYwc7duwgICCATp06ZauuwnibUF0+Q0QK\npcJ++YwWLVrQuXNnOnbsmOH2FStWEB4eTkxMTK7qr1ChApGRkTRq1AhwLs735JNPsnXrVldZXT5D\nRKSQyOw2oSn++OMP6tevn6u6s7pN6IXg69uEioj4jN+7+TP4Ye/kfA8ls9uEAqxevZpevXqluWR3\nduXlNqH5SeEgIhet3Hyo55fg4GBiY2Nd6zdv3sz999/PwIEDadq0aY7qPH36NK1bt+a2226jR48e\nabalvFZQUFDuG50DGlYSEcmFjG4TumPHDu655x7efvttnnjiiRzVVxxvEyoiUuSkv03onj17aN68\nOS+//DIvvPCCq7xuEyoiUgykv03osGHD2LZtm3euICAggMDAQG953SbUN3Qoq0gxU9gPZQXdJrQw\nUDiIFDMXQzgUFjrPQURELgiFg4iIuCgcRETEReEgIiIuCgcREXHR5TNEpFAKDg5OOdpGshAcHJzv\ndfq651sCA4ASwDCgbwZlBgKtgFPAs8DKDMroUFYRkRwqrIeylgA+xwmIa4EOwDXpytwP1AGuAl4A\nhviwPUVCZGRkQTeh0FBfnKO+OEd9kT98GQ5NgM3AdiABGAs8lK5MG2BE8uPFQBBQxYdtuujpD/8c\n9cU56otz1Bf5w5fhUA3YlWp5d/K6rMqE+bBNIiKSDb4Mh+xOEqQfD9PkgohIAfPlhPQtQATOnANA\nT8BD2knpL4BInCEngPXAncCBdHVtBq70UTtFRIqqLTjzuoVKSZyG1QJKA6vIeEJ6WvLjW4BFF6px\nIiJScFoBG3C++fdMXtc1+SfF58nbo4AbL2jrRERERESkaGiJMw+xCeiRRdmiYDjOfEt0qnUVgVnA\nRuAXnMN9U/TE6Zv1wL0XqI0XSnXgd+BPYA3wj+T1xbE/LsU51HsVsBb4IHl9ceyLFCVwTpidkrxc\nXPtiO7Aapy+WJK8r8n1RAme4qRZQioznLIqa24FGpA2HfsC/kx/3APokP74Wp09K4fTRZorWtbKq\nAg2TH5fHGZ68huLbH/7J/5bEmZtrRvHtC4BXge+AycnLxbUvtuGEQWpFvi9uBWakWn4j+aeoq0Xa\ncFjPuRMDqyYvg/MNIPXe1AycSf2iaiJwN+oPf2ApcB3Fty/CgF+Buzi351Bc+2IbUCndunzpi8Kc\nGtk5ia44qMK5Q3sPcO6XfjlOn6Qoyv1TC2ePajHFtz8uwfnWd4Bzw23FtS8+Af6Fc2h8iuLaF4YT\nlMuA55PX5UtfFOarsupkODfj/P1SFPusPDAe6A7EpttWnPrDgzPMVgGYifOtObXi0hcPAgdxxtjD\nMylTXPoCoCmwDwjFmWdYn257rvuiMO857MGZlExRnbSpV1wcwNk1BLgM5z8GuPsnLHldUVIKJxhG\n4QwrQfHuD4ATwFSgMcWzL27DuSbbNmAM0Bzn76M49gU4wQBwCPgJ55p2Rb4vsnMSXVFUC/eEdMo4\n4Ru4J5dKA7Vx+qooXfzeDxiJM4SQWnHsjxDOHXFSFvgDaEHx7IvU7uTcnENx7At/ICD5cTlgPs4R\nSMWiLzI6ia4oGwPsBeJx5ls64RyJ8CsZH5b2H5y+WQ/cd0Fb6nvNcIZSVuEMIazEObS5OPbH9cAK\nnL5YjTPeDsWzL1K7k3NHKxXHvqiN8zexCudw75TPyOLYFyIiIiIiIiIiIiIiIiIiIiIiIiIiIiIX\nk5PJ/9YEOuRz3f9Jtzw/n+sXEREfSbkmUzjnzqjNrqyuP5b+ek8iInKRSPkAXwQcxznbujvOtcU+\nxLlJShTwQnK5cGAuMIlzFzKbiHPlyzWcu/plHyAxub5RyetS9lL8kuuOxjmr+fFUdUcCPwLrgNGp\n2tkH52qrUcnPFRERH0oJh9TX4gEnDN5MflwG5z4JtXA+wE/iDEOlCE7+tyzOB37Kcvo9h5TlR3Au\nXeAHVAZ24FwMLRwnoC5P3rYA58qalUh7Rc3A7L45EV8ozFdlFclv6S8ydi/wNM43/0U416Spk7xt\nCc4HeoruONewWYhzZcursnitZsD3OJdEPgjMAf6SvLwE5xpallxnTZzAOAN8DbQFTuf0zYnkJ4WD\nFHcv49xIqBFwJc4FywDiUpUJx7kK6i0491RYiXNf5/Mx3GGUcu38s6nWJeFcmjwJ53LL43DuWTAD\nkQKkcJDiJJZzlzgG56Y5L3Ju0vlqzt2rObVA4BjON/t6pL21YgIZT1rPBdrh/B8LBe7A2WPI7BLJ\n5XCunjkd5/7IN2T5bkR8qDDfCU4kv6R8Y4/C+Ya+CvgGGIgzx7AC50P7IM6QTvq7Z80AugFrcS4h\nvzDVti9xJpyXA0+let5POPdBj0pe96/k+q/BffctwwmtSTh7JH7AP3P9bkVERERERERERERERERE\nRERERERERERERERERESKsv8HNGsHZ8bQ4KMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fbeb802a128>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_utility_estimates(agent, Fig[17,1], 500, [(2,2), (3,2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}