rl.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# Reinforcement Learning\n",
    "\n",
    "This IPy notebook acts as supporting material for **Chapter 21 Reinforcement Learning** of the book* Artificial Intelligence: A Modern Approach*. This notebook makes use of the implementations in rl.py module. We also make use of implementation of MDPs in the mdp.py module to test our agents. It might be helpful if you have already gone through the IPy notebook dealing with Markov decision process. Let us import everything from the rl module. It might be helpful to view the source of some of our implementations. Please refer to the Introductory IPy file for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from rl import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Review\n",
    "Before we start playing with the actual implementations let us review a couple of things about RL.\n",
    "\n",
    "1. Reinforcement Learning is concerned with how software agents ought to take actions in an environment so as to maximize some notion of cumulative reward. \n",
    "\n",
    "2. Reinforcement learning differs from standard supervised learning in that correct input/output pairs are never presented, nor sub-optimal actions explicitly corrected. Further, there is a focus on on-line performance, which involves finding a balance between exploration (of uncharted territory) and exploitation (of current knowledge).\n",
    "\n",
    "-- Source: [Wikipedia](https://en.wikipedia.org/wiki/Reinforcement_learning)\n",
    "\n",
    "In summary we have a sequence of state action transitions with rewards associated with some states. Our goal is to find the optimal policy (pi) which tells us what action to take in each state."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passive Reinforcement Learning\n",
    "\n",
    "In passive Reinforcement Learning the agent follows a fixed policy and tries to learn the Reward function and the Transition model (if it is not aware of that).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Passive Temporal Difference Agent\n",
    "\n",
    "The PassiveTDAgent class in the rl module implements the Agent Program (notice the usage of word Program) described in **Fig 21.4** of the AIMA Book. PassiveTDAgent uses temporal differences to learn utility estimates. In simple terms we learn the difference between the states and backup the values to previous states while following a fixed policy.  Let us look into the source before we see some usage examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%psource PassiveTDAgent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Agent Program can be obtained by creating the instance of the class by passing the appropriate parameters. Because of the __ call __ method the object that is created behaves like a callable and returns an appropriate action as most Agent Programs do. To instantiate the object we need a policy(pi) and a mdp whose utility of states will be estimated. Let us import a GridMDP object from the mdp module. **Fig[17, 1]** is similar to **Fig[21, 1]** but has some discounting as **gamma = 0.9**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import Fig"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mdp.GridMDP at 0x7fb5b0492ac8>"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Fig[17,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Fig[17,1]** is a GridMDP object and is similar to the grid shown in **Fig 21.1**. The rewards in the terminal states are **+1** and **-1** and **-0.04** in rest of the states. <img src=\"files/images/mdp.png\"> Now we define a policy similar to **Fig 21.1** in the book."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Action Directions\n",
    "north = (0, 1)\n",
    "south = (0,-1)\n",
    "west = (-1, 0)\n",
    "east = (1, 0)\n",
    "\n",
    "policy = {\n",
    "    (0, 2): east,  (1, 2): east,  (2, 2): east,   (3, 2): None,\n",
    "    (0, 1): north,                (2, 1): north,  (3, 1): None,\n",
    "    (0, 0): north, (1, 0): west,  (2, 0): west,   (3, 0): west, \n",
    "}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us create our object now. We also use the **same alpha** as given in the footnote of the book on **page 837**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "our_agent = PassiveTDAgent(policy, Fig[17,1], alpha=lambda n: 60./(59+n))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rl module also has a simple implementation to simulate iterations. The function is called **run_single_trial**. Now we can try our implementation. We can also compare the utility estimates learned by our agent to those obtained via **value iteration**.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from mdp import value_iteration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values calculated by value iteration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.3984432178350045, (1, 2): 0.649585681261095, (3, 2): 1.0, (0, 0): 0.2962883154554812, (3, 0): 0.12987274656746342, (3, 1): -1.0, (2, 1): 0.48644001739269643, (2, 0): 0.3447542300124158, (2, 2): 0.7953620878466678, (1, 0): 0.25386699846479516, (0, 2): 0.5093943765842497}\n"
     ]
    }
   ],
   "source": [
    "print(value_iteration(Fig[17,1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the values estimated by our agent after **200 trials**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{(0, 1): 0.45349055962264184, (1, 2): 0.6127179535526855, (3, 2): 1, (0, 0): 0.3686271642983948, (2, 0): 0.0, (3, 0): 0.0, (1, 0): 0.2327528230992705, (3, 1): -1, (2, 2): 0.7269851866778488, (2, 1): 0.5227571939134159, (0, 2): 0.5160077721580049}\n"
     ]
    }
   ],
   "source": [
    "for i in range(200):\n",
    "    run_single_trial(our_agent,Fig[17,1])\n",
    "print(our_agent.U)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also explore how these estimates vary with time by using plots similar to **Fig 21.5a**. To do so we define a function to help us with the same."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "%pylab inline\n",
    "def graph_utility_estimates(agent_program, mdp, no_of_iterations, states_to_graph):\n",
    "    graphs = {state:[] for state in states_to_graph}\n",
    "    for iteration in range(1,no_of_iterations+1):\n",
    "        run_single_trial(agent_program, mdp)\n",
    "        for state in states_to_graph:\n",
    "            graphs[state].append((iteration, agent_program.U[state]))\n",
    "    for state, value in graphs.items():\n",
    "        state_x, state_y = zip(*value)\n",
    "        pylab.plot(state_x, state_y, label=str(state))\n",
    "    pylab.ylim([0,1.2])\n",
    "    pylab.legend(loc='lower right')\n",
    "    pylab.xlabel('Iterations')\n",
    "    pylab.ylabel('U')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a plot of state (2,2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmYFdW1t9+muxFauptJUQGFOBsUhyjOthriFGNu4lVx\n9jqRRONNYjRqoiRm0PhpIjESNSpeo6JooqIokph2AMcggwgqIMggIEMjIGN3fX+ss6k6p+qMnOrp\n/N7n6ed0DafOrl1V67fXWnvvAiGEEEIIIYQQQgghhBBCCCGEEEIIIYQQQggh2iUPAEuAaWm2nwNM\nAaYCE4D9mqlcQgghWpCjgANILw6HAbWJ/08E3myOQgkhhGh5+pFeHIJ0AxbEWxQhhBC50KGlCxDg\nYmBsSxdCCCFE89CP7J7DscAHmPcghBCihalo6QJgSej7sJzDyqgdBg4c6E2ZMqVZCyWEEO2AKcD+\nhXyxpcNKOwN/B84FZqXbacqUKXiepz/P46abbmrxMrSWP9WF6kJ1kfkPGFiocY7bc3gMOAboCcwH\nbgIqE9vuAW7EQkkjEus2AYfEXCYhhBBZiFschmTZfkniTwghRCuipcNKIk/q6upaugitBtWFj+rC\nR3VRHMpaugA54iXiZ0IIIXKkrKwMCrTz8hyEEEKEkDgIIYQIIXEQQggRQuIghBAihMRBCCFECImD\nEEKIEBIHIYQQISQOQgghQkgchBBChJA4CCGECCFxEEIIEULiIIQQIoTEQQghRAiJgxBCiBASByGE\nECEkDkIIIUJIHIQQQoSQOAghhAghcRBCCBFC4iCEECKExEEIIUQIiYMQQogQEgchhBAhJA5CCCFC\nSByEEEKEkDgIIYQIEbc4PAAsAaZl2Gc48DEwBTgg5vIIIYTIgbjF4UHgxAzbTwZ2A3YHLgNGxFwe\nIYQQORC3OLwGrMyw/VvAQ4n/3wK6Ar1iLpMQQogstHTOoTcwP7C8AOjTQmURQgiRoKXFAaAsZdlr\nkVIIIYTYQkUL//5CoG9guU9iXYhhw4Zt+b+uro66uro4yyWEEG2O+vp66uvri3Ks1FZ7HPQDxgD7\nRmw7Gbgi8Xko8MfEZyqe58mhEEKIfCgrK4MC7XzcnsNjwDFATyy3cBNQmdh2DzAWE4ZZwFrgopjL\nI4QQIgeaw3MoBvIchBAiT7bGc2gNCWkhhBCtDImDEEKIEBIHIYQQISQOQgghQkgchBBChJA4CCGE\nCCFxEEIIEULiIIQQIoTEQQghRAiJgxBCiBASByGEECEkDkIIIUJIHIQQQoSQOAghhAghcRBCCBFC\n4iCEECKExEEIIUQIiYMQQogQEgchhBAhJA5CCCFCSByEEEKEqGjpAuTKmWem39axI/z5z9ClC4wc\nCW++Cf36wfXX2/Z582DqVDj11Px+0/NgwgR47TW45BLYbrvo/T77DN5/HwYP9tetXQv/+he89x4s\nWwY/+hF85Svpf2v9evjyS+jePb8yZmL5cqiqgs6d/XWNjVYX06fD/Pm2z9q1cPbZcNRRmY+3cqXV\n9bbbFq+MQojWSZvxHL7znfR///63GbqRI+EPf4CddoIHH7TvLV9uQvGtb2U+/oYNZiQdnmeC8D//\nA/fdB6+8Ev29hx6CffaBb3zDvgMmCrvuCsOHw8aNJi7vvJP+t0ePNgOerYy5Mnu2GfuePeH22/3z\n+/3voXdv2zZ2LDQ0QK9eJm5jxqQ/3uuvw8knm3D94hfFKWOpsWwZ3H8/fPvbdj9Om5b7d1euhKam\n8LrHHoMLLoB997V7TohSxMvEfvt53qRJnjdggOfV13ve7Nmet8sutu2Xv/S8k07yvN69/f0/+sjz\nunRJPsaBB3rewQf7y0895Xlf/arnffml5115pefdcYetnzTJ88aMsf+nTvW87bbzvBkzPK+qyvNW\nr/a8OXM8r2dPz3v5Zf9YQ4Z43t/+Fl32Z5+1sv3+95533HEZT3ML77zjeZ9+Gr3t3Xc9b/vtPe/m\nmz1v+HDPO/dcK9fhh3veN7/pedOmhb9z112eN3RoeH1Tk+f94heet+OOnvfAA3a8Cy/MrYztkaYm\nq4f6+vD6v//d8yZODH9n/nzPu/hiz6ut9bzTT/e8Rx7xvCOO8Lznnsv+e9One95ZZ3keeN6oUbbu\no48874ILPK+mxq7niBF2f/7wh563apWVIy42bPC8TZviO74oPoDX0sY7bjJWwNe+5nlvvul5nTp5\n3po1Zjh32sm2nXSS5/3lL2YwHbfeag+co7HRlo891l/3zW/6Bv222zxv8GDPGzvWfsN994wzPO/2\n2+3/HXc0QzB0qOf9/OfJ5TvvPM976KFwudevN2F45RXPmzDB8w47LPvF/vxz+/2rrw5vW7PGRPHJ\nJ235tdc8b9AgzzvnHDPqTU3Rx3zoIdsnldtu87wDDvC8JUtsefRoz/vOd7KXsa2xalX2fVavtntp\nu+3suju+/NLqFjzvhBOSv/PUU7b/z37mecuX++vPOMPzHnss/W81Nnrer39t3731VhP6Sy7xvN/+\n1vN69LDl4PEmTvS8Pn3sXgLP++KL3M47Hx5/3I5/883FP7aID7ZCHNpMziETlZWwYAFUV1s8fPVq\n2LzZwjzvvAO33AKbNvn7v/FG8vdnzrTPffaxzzVrLIz0t7/Zct++MH68xevXr4ftt7cwzYsvWq4D\noGtXWLIERo2CDz5IPn5FhZUnlaeegr33hqOPhkmT7NjZGD7cPmtrw9v+8Ac47DD47ndteffd4a23\nrFzTp0NZWfQxq6utzoJ8/DHceiu8+66dr/vNVauyl7Et8dZbUFdnobiddoreZ80aOOEEu1avvw7H\nHmv31ubN8F//ZfX3wQew3352z5x7roUbr78enn8eDj44+XhR9e3YtAkuugjmzrV7ok8fmDIFDjgA\nDj0UJk+2dUG+9jXb/r//C5deate7utrf/uWXcPfd8OMfQ4c8A8kbNsAVV9h5X3UV3HknDBqUnF9r\nzSxaZLnCysriH3v16uR6bm+0mZxDJiorzZjtsou/vGmTCUZ5ucV4g+KQGu914tDYaJ8ffGCG1Rng\nvn3ts74e9twTvvgCXn3VxKRnT9tWWwsvvQR77AE77ph8/PLyaHF49FHLaQB06pRdHBob4Z574Pzz\nzWAF2bwZRoyAG27w1zmjPnSoJabTEWWsbr7ZjIGrU3eOuYiD50FNTfo8TWth6VIT0rKy9MYarB6+\n8hW49167L5qa4JNPzGhWVFjsf++97Rqcdx48/TT87GeWB0gVBsgsDj/4geXJxo/3RWDffeGvf4WX\nXw4LA9j9/uyzcNxxlkNavNjftn695Tl++lPLLeXDxo1WP59/Dm+/bZ0qFi60/FoUnmd5lVwaOVG/\ndfbZcPXV+X83HU8/bdfrxReLd0yw87ztNrvHly/PvO/atfYsFVInLU27EIeOHU0c+vWzZScOS5ZY\nArZjx2RxSL2gM2ea4XNJv5kzYa+9/O277OK3PI480vZ95hn731Fba0ndo48Ol6+iwhcex+bNlqh2\nLbBcxOGNN0x4DjooOXkOZoj69oUBA/x1ZWXwxBPwwx9mPm51dbLYLF9uxuaKK5L3q6nJTRxef92M\n37x52feN4oYbrDUcNzfdBKefbgYkXd0//bQ1BEaMsFZ3WRl8/etm/F96yTzFioT/vXgxHH+89awb\nOTL5HgpSUxMtDiNH2j3xxBPJPcw6dLBGRKdO2c+pVy+77x0/+pH14jvoIPj0UzPCCxdmP47nwcUX\n27mNHm33SEWFnWO61vJNN1knjsmTsx8/SFMTnHWWlXvkSPNWtpbHH4fvf9+uQSbhz5emJrjySvMQ\nq6qsgZGOzz+HY46BG2+0hmpz0NRknVCC90ChtAtxSOc5rFwJ3brZ8saNtm3zZt8QBsVgn33Si0Pv\n3tb98ytfMfd9t93gueeS9+na1Yz34YeHyxcVVpo0ycrrPI/OnbOLw/PPwze/aQ97qufw4ou2LZX/\n/u9kQxNFakv2iSfgpJPsnILU1prXdPfd8JvfpD/effeZQcv04KTD8+C3v4U5c/L/biauvNJatWAG\nfepUePJJe3DTCXNjI1x7rQlDly7++hNPNIP5pz8lr+/VCy680ITjhBPSl6W62uoxyLJl1rp/4omt\nC1UExeGZZ2DcOOu517+/icNJJ0V7H6k8+KB1w3700eSQzPbb23Piyv/ppxbq/PvfzWAee6x5Vflw\n0012/i+8YOV8911/m+flfh+tWWNhpH/+0xpE48bBwIGwbl1+5cnENdfAf/5jDYaBA9N7DitXmof1\njW/AV7+avQyPPGINsq1h82aLKlxzjdmKrSVucTgRmAl8DFwbsb0n8CIwGXgfuLCQH+nY0W4uN0bA\niUNDgxm48nJb39hoN1qPHrDNNn4LZdYsCwelEwew5UsusZb+oYdaqzi4jwtB7blnuHxRYaVJk5JD\nDrl6DkcfbXmVVM9h3LjMBikTXboki8PYsRZLT8WFlX7wA3OVU7v/gsW3n33WWqz5hjEAJk60z2xj\nLnJl/Hj7u+ceePhhE4UhQ6zb8NVX2z2Tru5Hj7Z75fjjk9d/61smkFFifO658MtfZi5TVFjpV78y\nj2PfffM7v1R22MHuhUWLzECOHGnXrW9fuy4vv2zPSyYWLTID8/jj4XBkWZmJy8KFZrgvushyGUOH\nmjgcfLDlS3Ll5ZdNiEaPtnLtuKO1uB2PPWbGNZVVqyyX6PA8OOccez7PO88aAPvuaw2jrRWHhgbL\nXd57rwnY889bnfboES0OmzebR3rMMdaIqqqy5yIdzz1n+aLvfc88jKOPzs27C+J5dr0XL7bGVaau\n87kSpziUA3dhArEPMATYO2WfK4D3gP2BOuB2CkiSV1Zaq2GbbRI/XG6GfsUK8xzADy0tXmwP0Dbb\n+N7EihXWInKhn/nzk2PtjmuvNRFxoaCgOLjWXv/+4e9FhZVmzrQ4taNTJ7uJZ882QzZ5srUkx461\nRGdTk7XkDjoo2XPwPCvvvHnm1RRC0Fht3GitolSDCMnezYEHWis0NfzzyivWoho4MDn2nSuPPGJJ\n9dT6KgT3wJx2moUt3nvPRK1/f2tdX3qp7ZdOHG65xcZ1pCbyq6vtQS6UVHGYP9/O+6abCj+mo7bW\nRGDQINh/fz/MufPO5gX8+c92769cmf4YP/+5NYSijDKYJ33jjVbmpUutjq680rzmfv1yF4f1601U\n7r7bPB4wT9oZ3CVLrJHR0ODf74sXm/G/+moLP7r75IEH7Bno2NGuzbHH2nr3XGVi8+ZoIz9vnnly\nl15qocQbbjAPyTVC04nDTTfZPXP77fZZVZW+DHPmmMA+95yJyf77W2gxH4EF82InTLDyHXOMdbTY\nWuLsrXQIMAuYm1geBZwGzAjs8xmwX+L/GmA5EJG6zUxlpT1sThzKyswgL1vmh0acN+HEYeFC33No\naLAL7ozZ8uV24dNx1FEWAw7u4x72qBBOVFhpxozkHh/Okxk2zFpg558P//d/1vqeNs3CZt272292\n6eK32J95xlr5gwb5se98cWGOPn0s1LLrrtHnHzSSGzeaJ5M6ovull8yV3nHH3DyHdevs3Dt0MGM+\nZoy1RF96KbxvQ4OJ50EH5XZekyaZUdm0CS67zNY9/LCJxLx5ftmjxGH6dLsPCvXGMpGac7j/fhOv\ndCPw82GvvcxLWLnSWpCO3Xazhs2ll5oh/fBD84BTef99a5B8+GH635g921r8//iHtaKDddS/v63P\nhTvusBxZcPBn0OD+/Of2HDz9tD2ve+5pQvTkkyaCNTUmTp06wXXXWThpzz19OwDpw7ULFlgC/4UX\nrDE2e7b1HnS4QbD//rfV21VXWVmDkYEocXj7bavfKVP8iEXnztGew+bN5mlef709v7/7nTVmP/44\ns3in4ho9b79tdbLHHsUJy8YpDr2B+YHlBcCglH3uA14GFgHVwBmF/FDHjvawBd3lykq7cVyLxInD\nZ5+Z4QqGlRoa7EK7sFI2caiq8uPXjky9FqLCSqmhqw4drIyLFtmyc62dezl9uh9y2HZbvyX16af2\neeCB6X8/G05UFi60eG+2ZPD++/vlGjDAhKKx0R6C+no/Rp+L51BVZfsPHWq9xCoqLP/zwgvhfc8+\n29a7kejZeOYZ+87FF1vS2d0D++9vf44ocRg92vI1+Xb9zAUnxkuXWiNj8mQzyMXgpJPsnli3Lrmh\ncsop1mW3stI81rfeihaHP/7RvICortKOO+80A7ZsWVg8c/Uc1qyx33r11eT1zuB+/LGJwkcf2T3p\nGnPPPmtGcNMmq7vPPrMQ0re/bR52Kp07h0OfYCJTUWGhvKlT4Ygjkrc/8YRdn6eeMmGN8qJSxWHz\nZrj8cuvJ5HoKujJEeQ4jRpgduuoqW95lF/Pszj03d3HYtMn2/+Mf/ahFTY3Vb67PSTriFIdcinY9\nlm+oA3YFxgMDgVD/gmHDhm35v66ujrq6ui3LlZVmnIIthspKM7BO6Ssrzd3r2NFaaE4c1q+3Sqyq\nMnHYtMluppqa/E72rLPSdxdNDStt3Gg3tetd5ejUyUIMYA8e+EZ49my7ScEM76RJdlPU1MAhh5gh\nKwYur5GJwYPtAQCru9NPtxbn++/bw3zAAb7xy4TrQeZE+eWX7djpxoXk2yXxuefMkO2xhy0PGmQx\n7FRSxcHzzDg88EB+v5cr1dUWOhgwwK7zoEHRhm1rSPVgy8r85PmPfmRi8f3v+8nm5cutvp56KrPX\nABamS8cuu5g4NTVlFtYRI6zlnprb69HDhOFXvzKj2a2bebQLFlhHh2uu8RtJO+5oreb777d7L4rO\nnf1nyTF1qjUyZswwMT399ORz3rzZQkh//asJajp69LA6c9GIBx+0SMU55yTvFxVWWrbMWvsvvxyu\np27drMGaC3/6k3mKZ59ty/X19dTX11NWltytvRDiFIeFQN/Acl/MewhyOOD6vcwGPgH2BN5N2S9J\nHFJxN3iUOLiwUseOFqY55BBrJXTsaEY6mLRuavJ7OOXbYjz9dPuLoqLCz2+AxVK32853Ox1BcXCe\ng/Mk5szxWy9u4rv//Mduyl//2o+xbi0vvWTJsXS8+qp5FrfdZuK2apUZOrAHdZ997DoEczrpeO89\n+3TXbeJEa4mWl4dzDo2NZrRzDSktX24dDQ47LPu+qeIwY4a1vAal+rlFwuWnystNrKLGQsTJgQda\nXS5b5o/JuftuyyOcd15yqzdfOne252nx4vSDChsbTbSfey68rWdP8yBnzoS77rJ1ffrYvfHSSxYC\ncuy4o937Z54ZHlsULE+qYf797+EnP7HnfMIEe7ZOPtnf/thjllfJJAzg91D8wx8sB/LrX5sXk5qj\nigor/eY3cMYZyV3Pg8fNxXNYutRChxMm+L/pGs733GMe4O9+l6V3RAbiTEi/C+wO9AM6AmcCqZ21\nZgJfT/zfCxOGvKNlLpwUJQ4uIe3yEm6mUuc5OHHo0MFu2mwhpUJIDSu50FYqQSPlWt1r11rZ5syx\nXAD4LcBly6zFE9VDKl9+/GMLwS1blr5/Pli+xbVK997bH/fQp09ySCpd6z+IS5q5h/eNN8yYRyXw\nZyQyVe56ZmLtWnv43LGykSoO//yniVS6EeVbi7u/Pv3UDFum2XrjomtX/9p5nuVi7rvPRsVvLf37\nm3FNl3MaP96EIxjac/ToYbMqf/e7fmhrwABrxV9wQbJHv9NOFsK68sr0ZUkVh4ULLYTnOiOUl9sx\nXdfcpiYzuLlMMOk6rdx5pzXS9tknujGS6jksW+aPoI+iW7fcxOGOO+z+iXr+g+dUKHF6Dpux3kjj\nsJ5L92PJ6MsT2+8Bfgs8CEzBhOoaYEW+PxTlOVRUJHsOlZX2EKxYYTeME4cNG3xxaGoycSjmtNmu\nLEFjt2hRdKvKGSMndl272ndXrLBwjTMiznNYvty29+699WW8/XbLa0ydmns/+8GDzS0GK8OMGX5L\nKBdxmDLFzmXdOntgGhosBLRiRfi777xjD2AuvZhGj7bW3HXX5XYeqeJQX+9PQRIH3btvfTx4a+na\n1Q9dzJhhz8HFFxdHEF2Mf/Hi6EbQyJE2HiQKJ5xDh/rrzj7bhCy1e/XOO/vTmqQjVRzuvtvCPsEx\nPEFDWl9vtiKqt14qhx5q59qtm+/BpCtD0HMYPtyiDOk8q27d7DnMxPLlJubO+06ltYsDwAuJvyAB\nx5BlQJ5vWQiTznNYtMhvaTgBSRWHdeuSw0pxeA6phjKd5+B6sNTWmoH2PPvcuNFaSDvvbNvd+Xqe\ntfZTw1OF0quXH5/PxqpV1iL68Y9ted06MzIu95GrOBx6qH132jQTlrKy6LDSO+9YmCeXFtV775mb\n//Of53YuQXHwPAuTuTms2iu1tb44PPecjdkolqe0++52PaNGOjc0WO5oxIjo7/bvHx4hX1Zm+ZFU\nLrrIj7WnIygOTU3WYh83LnmfTp3sXt240QzupZfmXhdVVZbv2G239N8JlmHDBjt3N54niqBwp+Pu\nu00snU1IpRji0G5GSENYHDZs8EMgzqCuXm3rUnMOznNYsSK+sNIHH5hRTScOzs2vrbVWcm2ttS6q\nq+1iR3WTzWW0a67kIw41NX7IZpdd7EacMcNvxbmcTep7CByNjVYfhxzii4NLNEaFlaZOtXxDNs9h\n9Wp78E45JfN8UkGC4jB3rt0bxazX1kgwrOTEoVg8+aR1aojqofPCCxaaTBcerKqyVnguVFYmj1CP\nInhtX3vNchqpPY/Kyux+njvXQk6pCeVs7L57ZjEJhpWeecbGAO2+e/r9s4WVNm+2AXmZwmkShwTp\nxAH8+WiCUwAEPYfUnENDQ+ZufIXgjN3119uDmE4cGhutvDU1Jg5du5o4dOnid8l1vPqqze1UTCN2\n9tn+eIB82HVXe7A2bEg+r1Tv4emnbeqJadNsnEHPnpaYTxWH1ByN5/mznmYThyuvNIHPNXENyQbk\nnXdMsNo7rnW6fLl5cMXq0ACZB36NHWvC3VwEW+2PPpre06ipsV5PJ51U/LByMKz0wAP+ZJvpyCYO\nL7xgYdyBA9PvI3FI4LyC1HEO4Le2g+IQTEivWpXsOaT2Dy8Gzkg2NNhNMm1a+iRyTY3vOXTtasa2\nujosDkcdZb1KiikO++9f2IR3LlHev39yCyooDkuX2oCmhQutq+LHH1vryU0t8OGHvteRGlZavNjW\n7bBDek/EMW2a9TbJZ36ioDjkMs6jPeDCSv/+t91LuUzqlw9Rg88aGy2kFOwZFDdOHG65xcThrLOi\n96upscGnceSanFD+5CfW+IiamiZIz55+b8X77rNelkHuvdfGU2SitjZ5jqpCaBfiUCzPoanJbuhi\ni4NrCa9aZS21adPSt05ra31x2G47iylGeQ5gXkXqWImWwJ1LahfIoDjMmWOCOHCgiYETB/fwzprl\ni0yqOHzwgfUEcd5dOpqawtOS5EKnTlYmz7Nrk6lF1l5wYaWJE8MDwIpBVBfSt9+2xk66OHkcuFb7\nHXdYuCvdb1dXW6eIE0+MpwwzZpiR/+c/s9uX7be3sjQ2micfHOm+YoVFDdJ1m3dUV/tdgQul3bzs\nB/ITh2DOYeed/YT0unXFmcYgiAsrNTTYhf3qV9PHw11IqV8/mxeoUyebUiJKHH7723heYpIPK1ea\nmF16aXgW16A4fPKJeRbOU/j4Y0vide5sgrlype8FpeYc3MSIUYnqIPPmWb4o31lNO3WyB/eggyy5\nGNX3vL3RtasNLJs0qTjdV1OJEodx4+IxvtnK8dFH1hDJNBVKTY3NnxTHy3uqqqx77oUX5jb/WWWl\nPVNu6vPgTM/PPms9qbKVc9asgou7hXYhDlG9lSoqbDm1eyhk9hziDCutWmUt26hpC8DioYMHW0Ks\nvNxP6kaFldz6liYoCKlJxqA4zJ1r4rBhg4nD7Nk2Qra83LrQ9uvnn29qzmHOHOvGm00cnnuusGlE\nXAPi3Xf9xkJ7p7bWwnXTpsUTRoua8G7CBH+qiOaic2d7rrPNkbXTTtlnBtiaMoBN8ZErvXpZDqSs\nLHkw6ejRuSXML7nErus11+RX1iDtQhzSeQ5BI58u55CakF63rvjx1/Jymypi1SoLW6Uz6o88Er2+\nf//iDHSLm2yew4EH2sCvL7+0z513Njf500+Tk5SpIjBnjsWCM4mD59lU2fX1+ZfbXe9//ctCUnHM\np9Ta6N7d6mqvvfxxM8UkNefQ2Jh+Pqc4cTYgmzjcd1+8ZejcOb9Xq/bqZTmSM8/051FraLAeV6NG\nZf/+4MH2tzXi0C4eg3TiEDTyUTmHqK6sceQcKipMGJxnkm+L/ze/ST/AprVw993hN84FxWH69OQE\n9Pz5NieMq+tg7iQ1rJSL57BihdVvISEhNyBt0aLcu/K2dY46yuor6uVUxSA1rPT++9Y6dy+3ai6q\nqux3jzkm835lZfGNiB8wwCbUy7VrNZg4rF1ryWs3qPBf/7Lr1VwRg3YhDukGwaXzHFzOoTnDSsHJ\nv1pDOKjYfO974ZHaThyWLLGk8pFH+q9W3LDB8gPugQn2uooKK/Xvn1kc5s4tPDkfHKzlJjds73Tp\nYj14sg0iK5RUcZg4MT4hykTHjpZbicM7ypVu3WzAXj706mUC3ru3Lw7jx6d/f3cctAtxcIY/OI9O\nqueQLecQTEjH0VspOLVvexSHKJw4jBtnyb5ttvEThH36WEvN1XVQWIIisHq1XaeePeMTh9NOs1wQ\nZB6c1N4YOjS3iQkLweUcrrrKumW2lDhAfB5BnJx0kk2AGXyxl3tXSnPRLsShY8fk5DNEew7uBfFu\n1tB0nkOxcw4VFcnikO904G0VJw7z5/st8qoqG9PQNzFfr7tGQc8hGFZatMiEI920Go6tEYcOHSyn\n07VraYlDnHTubML+l79YY6Al8g1tmRNOsJcguVcCz55tIe90b+eLg3YhDpWVySElMAOTmnPo2tVu\n2rIy/wVBTU22n0tIx5VzWL3aD6GUmuewZInf26qqynrJODGIEodgWGnhQt+ryCYOUa92zYc77ij+\nexVKlc6dbcqTjRuti/GCBW2jU0Vrw4nD+PGWYG5OL6jdikOU59C9u7/OteZra63C4845gG8gS10c\nwMZyQPbeaV9ZAAATlElEQVSwUq7isGCB740UykUXFd9rLFU6d7ZpOcDG9uyxR8uPyWmLuLDS669n\nT6oXm3YhDi6sFCQq59CtW7I4BFvzcYqDmzXV9dQoFXFwHkCUOLjR1Ntua9NdBHtyBEXAhZVS16ey\naFH6F76I5qdTJ+u+XV5uU3SUwsDCOHC9+yZMiC8/lI52IQ65eg777OO/3rKiwkJIrlUfZ0La/Yab\n7bVUxCGT5+CWy8rC890Ecw4LF/rz3rtrFMVnn6WfH180P+4ZOuQQyzm4SRVFfpSXm21raGj+sFy7\nEIdttgmHA6LGOdTU+IaoosKEwBnuYM4hjoQ0SBwgLA5ROA/B85KNfrq5lZqa/Pf4itaBE4fjjrNP\niUPhdOliyfzmHpzZLsRhwIDwqMFUz6FTp+TQRarnoLBS8amoMJd49Wp/GuTUsFIUbkBSU5ONiXD7\npgsrLVtmwp/qPYqWo3Nnu9buPdwSh8LZdtvmDylBO5k+o0OHcEwz1XO49NLkgVVOHNw+wekz4gor\nlaI4LFliHpNr9bi6zeQ5uO82Nlof+WzikO61q6Ll6N/fpqju1cs6fbT3lyfFSXW1xKGoVFQkG/nU\nt7u5sJJ7k1SHDjbuoUOH3F5Kn29ZwC7ymDHFF5/WinuPd/DlKa51n20aBZfMXrrUnyU3nTike3mS\naDm6d4df/co8x7vuapsD0VoLTzzRMt2A2604HHpo5ncYRyWkv/wynq6MLqzUuXNxX8fY2qmoMOMe\nFIfu3eHxx7MLcHm59ZFftcr/fjpxCHoXonVRVQXnntvSpWjb5Pt+kmLRbsUh29umonIOa9fmNzlW\nrrjfKBWPwRElDmVlcMYZuX13yRJ/ahPILA7NPaGbEO2ddpGQLgQnDm5gTocOthycg6mYvwXxCE9r\nJiqslCvl5dYDKfjipQ4drAeTm0XVsWxZ8V/QJESpU9LisGFDuCtrsfMNkBxWKiWc55D6EqBcKC+3\nXEIwXORGsqd6D59/LnEQotiUtDgEP11vmjjEQWGl/L9bXm69kFKNflRoadkyhZWEKDYSh0BCOrgc\nx2+VYlipUHGoqLA3xKW+IyJKHOQ5CFF8Sl4cgjmH4PpiUsphpfXrC/ccPv003D9e4iBE81Dy4qCw\nUny48y405xDlObiR7EEUVhKi+EgcmlEcSjGsBDZCtpDvzpuXPazU1ARffGFdXoUQxSNucTgRmAl8\nDFybZp864D3gfaA+5vJsoTnFoZTDSlDYm+/Ky80jyCYOq1bZKHdXx0KI4hDnILhy4C7g68BC4B3g\nWWBGYJ+uwJ+BE4AFQLMFB1JzDkpIF5+tFQcIz5mUKg7uNa9CiOKSyRT+JGXZAz4HXgc+yeHYhwCz\ngLmJ5VHAaSSLw9nAU5gwACzL4bhFoSU8h1KbNTQ4p1S+bNhgn6neVqo4rFxZWE5DCJGZTGGlaqBL\n4K8aOBh4ERiSw7F7A/MDywsS64LsDnQH/g28C5yXU6mLQHOKQ1mZjeottcnHXJ0WIg5r10avlzgI\n0TxkMoXD0qzvDvwLeCzLsb0s2wEqgQOB44Eq4A3gTSxHkVyYYX5x6urqqKury+Hw6WlOcShVXOu/\nkHxAruKgsJIQPvX19dTX1xflWIWYwhU57rcQCL7yvS9++MgxHwslrUv8vQoMJIs4FIPmHOdQqqxb\nV/h35TkIkT+pDedf/vKXBR+rkN5KxwIrc9jvXSxs1A/oCJyJJaSDPAMciSWvq4BBwAcFlClvmnOE\ndKny5ZeFf3fz5uhrESUO8hyEKD6ZTOG0iHXdgM+A83M49mbgCmAcZvzvx5LRlye234N1c30RmAo0\nAffRQuIgz6H4bI04gP8ipiBRYSV5DkIUn0ym8NSUZQ9YDqzJ4/gvJP6C3JOy/P8Sf81Kqji4ZLHE\noXhsrThsu214XZTnoFeEClF8MpnCuc1ViJYgNefgXmovcSgecYlDcPqMVasKG4EthMiMps8IiEEc\n748uZdav37rvR4lDhw4wfry//MUXhXWVFUJkRuIQEIPycolDMXn0UXjzzcK/H5Vz6NABfvpTWL0a\nBgywGVkLGYEthMhMyZrCDh3CYSR5DsVll13sr1CiPIeNG+1zwwaYPh122EGegxBxULKeA5gQSBxa\nJ507w9FHh9e7PMbKRGfqxYvlOQgRByVtCisq/IQ0SBxaE+mS2W5g3YrAUEx5DkIUH3kO8hzaFE40\nli/310kchCg+EgclpNsUqZ5DWVl0bkIIsXVIHOQ5tClcQtqJQ3V16c12K0RzUPLioJxD28SJg5LR\nQsRDyYuDPIe2SdBzEEIUH4mDxKFNsny5eX0SByHiQeKghHSbZMUK6NNH4iBEXJS0KVTOoe2yYgWc\ndhqccUZLl0SI9ok8B4WV2iTOczjssJYuiRDtE4mDxKFNsmKFQkpCxElJi0P//rDddv6yxKH1s3gx\nnHqqvQFOg9+EiI+SNoVPPZW8rIR066dXLxvb0NQkcRAiTkrac0hFnkPboGNH+6yqatlyCNGekTgE\nkDi0DSQOQsSPxCGAxKFt4LofSxyEiA+JQwCJQ9tAnoMQ8SNxCKCEdNvAeQ5KSAsRHxKHAPIc2gby\nHISIH4lDAIlD20A5ByHiR+IQQOLQNnCeQ6dOLVsOIdozEocAO+0EPXu2dClENiorzWvQG+CEiA+1\nkwOkjpgWrZOOHRVSEiJu4vYcTgRmAh8D12bY72BgM/CdmMsj2gHOcxBCxEec4lAO3IUJxD7AEGDv\nNPvdCrwIKFAgstKxo7qxChE3cYrDIcAsYC6wCRgFnBax35XAk8DnMZZFtCPkOQgRP3GKQ29gfmB5\nQWJd6j6nASMSy16M5RHtBOUchIifOBPSuRj6PwI/S+xbRoaw0rBhw7b8X1dXR11d3daVTrRZ5DkI\nEU19fT319fVFOVacMf5DgWFYzgHgOqAJyy845gTK0BP4ErgUeDblWJ7nyakQxtKlMHUqfP3rLV0S\nIVo3ZdbfuyA7H6c4VAAfAscDi4C3saT0jDT7PwiMAf4esU3iIIQQebI14hBnWGkzcAUwDuuRdD8m\nDJcntt8T428LIYTYCtpK11F5DkIIkSdb4zlo+gwhhBAhJA5CCCFCSByEEEKEkDgIIYQIIXEQQggR\nQuIghBAihMRBCCFECImDEEKIEBIHIYQQISQOQgghQkgchBBChJA4CCGECCFxEEIIEULiIIQQIoTE\nQQghRAiJgxBCiBASByGEECEkDkIIIUJIHIQQQoSQOAghhAghcRBCCBFC4iCEECKExEEIIUQIiYMQ\nQogQEgchhBAhJA5CCCFCSByEEEKEkDgIIYQI0RzicCIwE/gYuDZi+znAFGAqMAHYrxnKJIQQIgNl\nMR+/HPgQ+DqwEHgHGALMCOxzGPABsAoTkmHAoSnH8TzPi7moQgjRvigrK4MC7XzcnsMhwCxgLrAJ\nGAWclrLPG5gwALwF9Im5TEIIIbIQtzj0BuYHlhck1qXjYmBsrCUSQgiRlYqYj59PLOhY4H+AI2Iq\nixBCiByJWxwWAn0Dy30x7yGV/YD7sJzDyqgDDRs2bMv/dXV11NXVFauMQgjRLqivr6e+vr4ox4o7\nIV2BJaSPBxYBbxNOSO8MvAycC7yZ5jhKSAshRJ5sTUI6bs9hM3AFMA7ruXQ/JgyXJ7bfA9wIdANG\nJNZtwhLZQgghWoi4PYdiIc9BCCHypDV3ZRVCCNEGkTgIIYQIIXEQQggRQuIghBAiRNy9lYQQoiC6\nd+/OypWRw55ECt26dWPFihVFPaZ6KwkhWiVlZWXouc+NdHWl3kpCCCGKisRBCCFECImDEEKIEBIH\nIYQQISQOQghRINdddx133nln7L8zZswYzjrrrNh/J4jEQQghCuDzzz/n4YcfZujQoQC8+eabDB48\nmB49erD99ttzxhlnsHjx4pyPNWTIEHr37k3Xrl058sgjefvtt7dsP/XUU5k+fTrTpk2L5VyikDgI\nIUQBjBw5klNOOYVtttkGgIaGBoYOHcq8efOYN28e1dXVXHTRRTkda82aNQwaNIhJkyaxcuVKLrjg\nAk455RTWrl27ZZ8hQ4Zw7733xnIuUWicgxCiVdLaxzkcf/zxXHzxxZx99tmR2ydNmkRdXR1ffPFF\nQcevra2lvr6eAw44AICJEydy7rnnMmfOnNC+GucghBCthGnTprHnnnum3f7qq68yYMCAgo49efJk\nNm7cyG677bZl3V577cXcuXNZs2ZNQcfMF02fIYRos5QVKfZRiIPS0NBAdXV15LapU6dy88038+yz\nz+Z93C+++ILzzjuPYcOGJR3f/d/Q0ECXLl3yL3CeSByEEG2Wlow6devWjdWrV4fWz5o1i5NPPpnh\nw4dzxBFH5HXMdevWceqpp3L44Ydz7bXXJm1zv9W1a9fCC50HCisJIUQB7Lfffnz44YdJ6+bNm8fg\nwYO58cYbOeecc/I63oYNG/j2t7/NzjvvzD333BPaPmPGDPr169csXgNIHIQQoiBOPvlkXnnllS3L\nCxcu5LjjjuOKK67gsssuC+0/cuRI+vfvH3msTZs2cfrpp1NVVcXIkSMj93nllVc4+eSTi1L2XJA4\nCCFEAZx//vmMHTuW9evXA/DXv/6VTz75ZEuuoLq6mpqami37z58/nyOPPDLyWBMnTuT5559n/Pjx\ndO3adcv3J0yYsGWfUaNGcfnll8d7UgHUlVUI0Spp7V1ZAW644Qa23357rrrqqqz7nnDCCQwfPjxj\nD6d0jBkzhkceeYRRo0ZFbo+jK6vEQQjRKmkL4tBa0DgHIYQQzYLEQQghRAiJgxBCiBASByGEECEk\nDkIIIUJo+gwhRKukW7durreNyEK3bt2Kfsy4a/5E4I9AOfBX4NaIfYYDJwFfAhcC70Xso66sQgiR\nJ621K2s5cBcmEPsAQ4C9U/Y5GdgN2B24DBgRY3naBfX19S1dhFaD6sJHdeGjuigOcYrDIcAsYC6w\nCRgFnJayz7eAhxL/vwV0BXrFWKY2j258H9WFj+rCR3VRHOIUh97A/MDygsS6bPv0ibFMQgghciBO\nccg1SZAaD1NyQQghWpg4E9KHAsOwnAPAdUATyUnpvwD1WMgJYCZwDLAk5VizgF1jKqcQQrRXZmN5\n3VZFBVawfkBHYDLRCemxif8PBd5srsIJIYRoOU4CPsRa/tcl1l2e+HPcldg+BTiwWUsnhBBCCCGE\naB+ciOUhPgauzbJve+ABLN8yLbCuOzAe+Ah4Cevu67gOq5uZwDeaqYzNRV/g38B04H3gh4n1pVgf\nnbCu3pOBD4DfJdaXYl04yrEBs2MSy6VaF3OBqVhdvJ1Y1+7rohwLN/UDKonOWbQ3jgIOIFkcfg9c\nk/j/WuCWxP/7YHVSidXRLNrXXFk7APsn/u+ChSf3pnTroyrxWYHl5o6kdOsC4MfAI8CzieVSrYtP\nMDEI0u7r4jDgxcDyzxJ/7Z1+JIvDTPyBgTsklsFaAEFv6kUsqd9eeRr4OqqPKuAd4KuUbl30Af4J\nHIvvOZRqXXwC9EhZV5S6aM2qkcsgulKgF37X3iX4F30nrE4c7bl++mEe1VuUbn10wFp9S/DDbaVa\nF38Afop1jXeUal14mFC+C1yaWFeUumjNs7JqMFwYj8z10h7rrAvwFHAVsDplWynVRxMWZqsFxmGt\n5iClUhffBJZiMfa6NPuUSl0AHAF8BmyH5RlmpmwvuC5as+ewEEtKOvqSrHqlwhLMNQTYEXswIFw/\nfRLr2hOVmDA8jIWVoLTrA2AV8DxwEKVZF4djc7J9AjwGHIfdH6VYF2DCAPA58A9sTrt2Xxe5DKJr\nj/QjnJB2ccKfEU4udQT6Y3XVnia/LwP+DwshBCnF+uiJ3+OkM/AqcDylWRdBjsHPOZRiXVQB1Yn/\ntwUmYD2QSqIuogbRtWceAxYBG7F8y0VYT4R/Et0t7XqsbmYCJzRrSePnSCyUMhkLIbyHdW0uxfrY\nF5iE1cVULN4OpVkXQY7B761UinXRH7snJmPdvZ2NLMW6EEIIIYQQQgghhBBCCCGEEEIIIYQQQggh\nhBBCiLbEmsTnLsCQIh/7+pTlCUU+vhBCiJhwczLV4Y+ozZVs84+lzvckhBCijeAM+JtAAzba+ips\nbrHbsJekTAEuS+xXB7wGPIM/kdnT2MyX7+PPfnkLsDlxvIcT65yXUpY49jRsVPMZgWPXA6OBGcDf\nAuW8BZttdUriu0IIIWLEiUNwLh4wMbgh8f822HsS+mEGfA0WhnJ0S3x2xgy+W071HNzyd7GpC8qA\n7YF52GRodZhA7ZTYNhGbWbMHyTNq1uR6ckLEQWuelVWIYpM6ydg3gPOxlv+b2Jw0uyW2vY0ZdMdV\n2Bw2b2AzW+6e5beOBB7FpkReCrwCHJxYfhubQ8tLHHMXTDDWA/cD/wWsy/fkhCgmEgdR6lyBvUjo\nAGBXbMIygLWBfeqwWVAPxd6p8B72XudMeITFyM2dvyGwrhGbmrwRm275SeydBS8iRAsicRClxGr8\nKY7BXprzffyk8x7472oOUgOsxFr2e5H8asVNRCetXwPOxJ6x7YCjMY8h3RTJ22KzZ76AvR95YNaz\nESJGWvOb4IQoFq7FPgVroU8GHgSGYzmGSZjRXoqFdFLfnvUiMBT4AJtC/o3AtnuxhPN/gPMC3/sH\n9h70KYl1P00cf2/Cb9/yMNF6BvNIyoAfFXy2QgghhBBCCCGEEEIIIYQQQgghhBBCCCGEEEIIIYQQ\nQgghhBDtmf8PygQjKixXA8cAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb5b045f278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "agent = PassiveTDAgent(policy, Fig[17,1], alpha=lambda n: 60./(59+n))\n",
    "graph_utility_estimates(agent, Fig[17,1], 500, [(2,2)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to plot multiple states on the same plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYcAAAEPCAYAAACp/QjLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8FVX+//FXIEFaKAmEErpIE1EsgJWAjQVdlLXBUta1\ngK4F3Z9fRF2NYAMVBJGuIEVRVBSkKKgBpBfpIE1CNQYSSiCQcj+/Pya5JJkEkpCbQt7PxyMP7syc\ne+65h2Q+c8qcARERERERERERERERERERERERERERkYvSJ0AUsDGL4/8E1gMbgCVAi3wql4iIFKCb\ngZZkHRyuByqmvO4ALM+PQomISMGrR9bBIa3KwH7fFkVERLKjREEXII1HgDkFXQgREckf9Th/y6Ed\nsAWn9SAiIgXMv6ALgDMIPQ5nzCE2swRXXnmlrV+/Pl8LJSJyEVgPXJWbNxZ0t1Id4BugO7Azq0Tr\n16/HzPRjxmuvvVbgZSgsP6oL1YXq4tw/wJW5PTn7uuXwOdAWqALsA14DAlKOjQFexelKGpWyLxFo\n5eMyiYjIefg6OHQ9z/FHU35ERKQQKehuJcmhsLCwgi5CoaG6OEt1cZbqIm/4FXQBsslS+s9ERCSb\n/Pz8IJfnebUcRETERcFBRERcFBxERMRFwUFERFwUHERExEXBQUREXBQcRETERcFBRERcFBxERMRF\nwUFERFwUHERExEXBQUREXBQcRETERcFBRERcFBxERMRFwUFERFwUHERExEXBQUREXBQcRETERcFB\nRERcFBxERMRFwUFERFwUHERExEXBQUREXBQcRETERcFBRERcfB0cPgGigI3nSDMc2AGsB1r6uDwi\nIpINvg4OE4AO5zjeEWgIXAY8DozycXlERCQbfB0cFgOx5zj+d+DTlNcrgEpANR+XSUREzqOgxxxC\ngX1ptvcDtQqoLCIikqKggwOAX4ZtK5BSiIiIl38Bf/4BoHaa7Vop+1z8/MLTbIWl/IiIyFkRKT8X\nLuNVuy/UA2YBV2RyrCPwVMq/bYAPUv7NyMzUoBARyQk/Pz/I5Xne1y2Hz4G2QBWcsYXXgICUY2OA\nOTiBYSdwEnjYx+UREZFsyI+WQ15Qy0FEJIcupOVQGAakRUSkkFFwEBERFwUHERFxUXAQEREXBQcR\nEXFRcBARERcFBxERcVFwEBERFwUHERFxUXAQEREXBQcREXFRcBARERcFBxERcVFwEBERFwUHERFx\nUXAQEREXBQcREXFRcBARERcFBxERcVFwEBERFwUHERFxUXAQEREXBQcREXFRcBARERcFBxERcVFw\nKCLiE+PZf3x/jt6zbN8yen3bi9j4WO++2PhYlu1bBkD0yeh0x0Tk4rD32F4OnTh0QXkoOBQB32z9\nhiYfNeGeafd49x08cZBf/vgFM2PqhqlsiNrgPbY7djedPutE16+7snz/ciL2RJCYnMiQZUNoPKIx\nHaZ2oP+C/jQY3oAeM3oUxFfKE1FxUQVdhIuWmZGYnAhATHwM8YnxBVyis06cOZFu28zwmKeASlN4\nHDl1hFGrRnHzhJtpPKIxvb/vfUH5KTjko41RG12/2Ody7PQxes7oyYsLXmTMXWPYEr2FM0lnmLZp\nGleNvoouX3bhjil38OScJxm9ejRmxgfLP6DVuFa0rduW7U9vp9eVvZiycQptPm7Dj7t+ZOG/FvK/\nW/7HnmN7WPXYKtYeWsv2I9tz/Z2iT0bT7etuvB7x+jnTLdu3jJMJJ3P9OWlFxUXxwPQHqP5+dQ6e\nOAjA6oOr6fZ1N04nnc7yfWeSzjDo10Es2L2AU4mn8qQsF6Nf9/5K24ltaTSiES/8+AK1htTijUVv\nFEhZPOZh77G9ACyOXEzYxDBC3gvhie+foMWoFoxZPYYrRl3B03OeLpDyFQYbozbyyHeP0PDDhiza\nu4j/u+H/2Nt3LxF7Igq6aPnCirpJ6yZZwIAAG7lyZLbS747ZbY0+bGS9Z/W2uDNxZmbWfGRz6/JF\nF2swrIGtPbjWRqwYYW8uetOW7VtmjT5sZA9Mf8CuG3ud7Tiyw5vP0r1Lrdyb5WzM6jHm8Xhcn3P7\npNtt7o655y3PqYRTrn1L9y610PdD7YHpD1iTEU0yfV9CUoI9M+cZIxwbu3pspmmSPcnn/fxU0zdP\nt5B3Q6zf/H52x+Q7bOqGqfbqz69atXerWdXBVW3NwTVmZhYbH2vhv4Rb5NFIi0+Mt41RG+3yjy63\ndhPbGeFY0KCgbH3e4sjFdu3Ya23v0b3ZLmNRtf/Yfuv6VVerPaS2fbruU3vtl9fsye+ftBlbZxjh\n2DdbvvHZZy+OXGwDIgak27dk7xK7bux1VmpgKQubGGZ1h9a1Cb9NsG+3fmt95/a18F/Crd3EdtZv\nfj+7c/KdPitbYbXhzw1212d3WY33atgbC9+wv+L+Sne814xeBlhBn7yz0gHYBuwA+mVyvAowD1gH\nbAL+lUU+BVT9eePdJe9a3aF1rfes3vaf2f/JNE3aE/eag2usxns1bMSKEenS9JnVx9p/2t4Onzyc\nbn9icqJVfLui9ZzR0+IT4115Z3ZiT/Xwtw/b+DXjz1n+RXsWWYW3K9jCPQu9+z5Z+4lVHVzVZv0+\ny5I9yRY0KMj2H9uf7n1/xf1lbSe0tY5TO9q7S961OyffaX/E/pEuzafrPrXK71S2PrP62KaoTVmW\nISk5yV6c/6LV+6Cerdi/wsycei37ZllrN7GdHTpxyB766iGbtG6SbfhzgzUY1sCajmhqfuF+ds2Y\na6zK4Co28beJ5vF4bNGeRRYwIMASkxPtaPxRO5lw0vV5yZ5kC/8l3Kq9W82ajmhqX2768px1VNR9\ntuEzqzq4qr3y0yvei5FUHo/Hqr9X3QjP+7/DE2dO2OMzH7fQ90Ot7Jtl7djpY3Y0/qj1mdXHar5f\n06asn2JL9y61UatG2enE05nmsfbgWmsxqoVr/6aoTRY2Mcz6zu1ro1eNzvOyF5RDJw5Zzxk9LeTd\nEBu6bGiW9WJmhTY4lAR2AvWAAJwA0DRDmnDg7ZTXVYAjgH8meeVj1eetUatGWYNhDWz/sf02f9d8\naz2utYVNDLN/fPEPe2PhG2ZmtmL/Cqs9pLbFnIqxTVGbrNq71ezrLV+78kpKTsr06t/M+YXJ6ti5\n/O/n/1n4L+Hp9kUejbQ9sXvMzGz29tlWdXBVu33S7fbs3Gct5lSMjVgxwuoOrWvbord53/OPL/5h\nn6771Lv954k/rdlHzez/fvw/S0pOsl0xu4xw7LLhl5mZc8J5a9FbVndoXXtz0ZtGOPbyTy/b1uit\nduz0sXTlOZN0xu6ddq+FTQyz6JPR3v37ju2zYcuHWVJykpmZDVw40G6bdJtVHVzVpqyfYvGJ8bb2\n4FrrO7evbf5rc7o8Lx12qX295WurNaSWvbnozXTH4hPj7cHpD9oNH99gB48ftFd/ftVeWvCSTV4/\nOdNAUpTFJ8Zbrxm9rMmIJt5WV2YSkhLskoGXnPNElFPL9y23BsMa2MPfPmzHTh+zWybcYm8tesvq\nf1DfHpv5mMXGx2Yrn4PHD1rVwVW928meZBv06yCrMriKvbP4Hes0tZPV+6CeLdi1IM/KnhMej8f7\nO+3xeGzujrm2K2aX60o/O/lM+G2CVR1c1frN7+f6O8kMhTQ4XI/TKkj1YspPWr2Bj1JeNwCy6vzO\n1X9KQfth5w9W/b3qtvPITjNzfokJx+767C4r8XoJIxzbHbPbqr9X3WoNqWUfr/3Yag+pbVPWT8m3\nMo5ZPcYe/e5Rm7J+isWdibO9R/darSG1rOtXXW3J3iVWdXBVW7ZvmS3YtcAIx6q/V93qDq1ru2N2\np8tn5MqR1nNGTzMzi4qLsmYfNbPXfnktXZrVB1Zb+bfK25mkMzbo10HW7KNmduD4ATMzm7N9jl06\n7FIr/1Z5a/9pe/N4PLY1eqtNXj/ZOn/e2Tp/3tnOJJ0553eZuW2mVXi7gv20+6fzfu8uX3SxUgNL\n2d2f3W09vunh3X/izAm7ZcItdv+X93tbYV9t/srKvFHG/ML9bNyacedsiRUV32z5xh797lG7fvz1\ndv+X92cr6DX6sJFt/muzeTwem7djXpbdgbtjdlv7T9vbvmP7sswr9SSX9iKo/4L+VvHtivbV5q9y\n9F2SkpPMf4C/JSQlWGx8rP3987/b9eOv917gmJk9O/dZe2/JeznKNy/Exsda5887W/0P6lvMqRjr\n/HlnqzK4igUMCLA6Q+tYzKmYdOlPJZzK9Pcr+mS0/W3K3+yq0VfZ2oNrs/35FNLgcB8wLs12d+DD\nDGlKABHAQeAE8Lcs8rrg/6T8Fnk00qq9W81++eMX7z6Px2NjV4+104mnLdmTbJXfqWzNPmpmQ5YO\nsf/9/D8LGBBgr/78ar6Wc/b22Vbx7YpGOPbx2o/tipFX2LNzn7XK71S26u9V945HJCQl2LSN02zU\nqlGuwGBmtv3wdqv5fk2LT4y368dfby8teCnTz7tu7HXWa0Yvqzu0brpuqKPxRy3k3RCbu2Ou1fug\nnv2480erO7SuVRlcJVuBwcyp34xdbln5effPNn/XfFscudjajG9jsfGxdvjkYbtt0m328LcPpzvx\nRcVF2ZPfP2kjV440wrFeM3pl6zOyY9SqUTZ5/eQ8yy87Jq2bZDXeq2EPTn/QXo94PdtjPjd8fIOV\nfbOs/fPrfxrh2Maoja40qw+stprv17RaQ2rZ5xs/dx33eDzWf0F/azi8oW35a0u6Y0fjj9qhE4dy\n9Z2qv1fdlu5dak1GNLGnZj/l+n0Zvny4PfH9EznONyk5ybp93c3ajG/jLf+AiAHWcnTLdMEn1enE\n0zZ02VDvOFeDYQ3s6TlPW/ORzS3k3RB7avZTdjrxtB06ccjaTWxn83bM8753U9Qmu3TYpfbcvOfS\n5bl833KrM7SO9ZvfzxKSEnJUfi4gOGTWhZNXslOol3C6m8KAS4H5wJU4gSKd8PBw7+uwsDDCwsLy\noIi+YWY8MvMRnmn9DGH1wrz7/fz8eOyax7zbbWq1obR/afq26ctvf/7GgeMHeC3stXwta8OghgSU\nDOC5Ns/x5OwnebD5gwy9cyi/7v2VR69+lA4NOwAQUDKAB5s/mGU+lwVfRsVLKnLH5DuoXr46A9sP\nzDTdBx0+4MnZTzKv+zxCK4R691csXZFD/z1ECb8SdLqsE3d/fjev3PIKz7Z+lrIBZSlZouR5v4uf\nnx/BZYOz9b3b1W8HODOfth3exs0TbmZnzE7ubnQ34+4eRwm/sxP5QsqF8FGnjziVeIqNf21kzaE1\neMyTLk1uzPx9Jk/MfoLWoa3p3qL7BeWVXZPWT6L/T/35qedPNK2asZf33O5reh8VLqmAxzy0Cm3F\noROHaB7S3Ht81YFVdPqsE2PuGsPWw1tZfXA1DzV/yHvczOg7ry+L9y5m2SPLqFK2Srr8K5auSEUq\n5up71Shfgzun3MmAdgPo26av6/ilQZfy/Y7vc5RnQnICPWb0YEv0Fjb9tYkkTxK9Z/Vmw18baBTc\niGmbptHvprNDqcfPHOfeL+5lyd4l7IrZxRebv2DInUPo3qI7c3bM4cSZE96/oerlq3NltSvZELWB\nOxveScSeCB6Y/gA9r+zJ3J1zeemnl6hTsQ4h5ULo830fxt09js5NOp+3zBEREUREROToexaENqTv\nVuqPe1B6DnBjmu2fgGszySvHET8/bI3emq4P9mTCSfN4PDZ+zXi7Zsw1lpiceM73R8VFZTqAnN8S\nkhJs37F9dt+X93m7F3IygyjV6xGv2+UfXW7HTx+/oPKsPrDaXv7p5VyNoeSUx+OxCm9XsN6zettP\nu386bwvlZMJJI5wLHpxdc3CNVR1c1cJ/CbcOUzq4ypTxqjovzN4+26q/V922Rm+94Lx6fNPDJv42\n0bu94c8NVu3dajZz20wzc7pUwyaGpXtP37l9rfW41tkeS8iJ13557ZzdsduitxnheCdfeDweW3Vg\nlSvdjzt/tH3H9ll8Yrx1mNLB7pl2j8Unxlu9D+rZrZ/eau0/bW9xZ+Js5MqR9vjMx73vO3LqiLUc\n3dKe+P4J+3LTlxb4VuB5xzg+WfuJlXi9hD0/73mrOriq/bz7Z0tKTrJK71Syq8dcbYFvBVrN92vm\nqBspIwppt5I/sAtnQLoUmQ9IDwFSL5WrAfuBoEzyynXl+MrR+KMWPCjYJq2bZGbOCbbpiKY2YsUI\nC3k3xH479FsBlzD/JSQlFMkB25X7V+aouf7Id49cUHD488SfVmtILft6y9e2bN8yazWuVbrjg34d\nZISTqwCdmeOnj3uD0ZK9S/Ikz37z+9lbi94yM7MDxw9Y6PuhNm3jNO/xwycPW4W3K3i/w9BlQ63Z\nR818Ehiy43TiaSMcCxgQYGZmr/z0ihFOuguZ1Cm7fWb1sS5fdLH7vrzPe4HX/ZvudsfkO7zjAfN2\nzLPbJt1mZk79thrXyp6f97x5PJ50A9DnsuWvLUY49uh3j6abDLDqwCo7lXDKRq0aZbtidl3Q96aQ\nBgdwxhB+x5m11D9lX++UH3BmKM0C1gMbgW5Z5HNBFZQbY1ePdQ1gpb2ie2nBSxb4VqD1ntXbDp04\nZCNXjrRK71SygAEB9sh3j+R7eSX/eDweCxgQkKuZOx6PxzpO7Wj9F/Q3M7PfD/9uDYc39B5fdWCV\nVR1c1Qgn1/3vaZ1MOGnNPmpmfuF+eTrRYdjyYd7+89bjWnsDRVp1h9a1UatGWeMPG1vo+6EWeTQy\nzz4/N04lnLIyb5SxsavHWoNhDSx4ULB3+vSK/SusyuAqNn7NePML97PbJt2W7v/3VMIp76w4M2eM\nre7QunbPtHus6uCq1mdWn3xp6eYUhTg45BWfVZ7H47GGwxumG1xLSEowwrFq71ZLl3bejnnmP8Df\nIo9GWtCgIPtk7SdGOHbDxzdY7SG1be6Oudbow0Z28PhBn5VXCoea79f0dj/c8PEN3pP9+YxYMcKu\nHXutt6USfTLaggcFm5lZ3Jk4u2z4ZfbFpi+s5eiWtvrAatf7zySdyfagu5lZ71m9rdvX3byzwvLK\n9M3TjXDsurHX2X1f3pfpibHLF13Mf4C/jV412n4//Huefn5uXf7R5VbpnUq25a8tdsfkO2z29tm2\nJ3aP1Xivhn237TvzeDw2dNlQO3HmxDnzOZN0xvzC/eyOyXfY0r1L86yVl9e4gOBQZJfPMDMuHX7p\nBa2vc+z0MXp+25OdMTv5I/YP7/4Z22ZQwq8E9SvXT5d+0JJBmBmPz3qc+5vd7x0g2hC1geYhzenQ\nsAPb/rONGoE1cl0mKRpCyoUQFRfF24vfplTJUoxZM+a879kZs5PXIl5japepBJQMAKBS6UocPX0U\nj3n4fz/+P66vfT0PXP4AoRVC2Xd8H3N2zEm3OOIT3z/B03Ozt1TEjK0zmL97PqM6jaJmYM3cfdEs\n1K1YF4B/t/w3EzpPwM/Pz5WmW/NufNTxI3pf25tGwY3y9PNz6+lWTzPjwRk0rdqUuhXrsitmFw9+\n9SDPtXmOvzf+O35+fvRt05fypcqfM59SJUvxUceP+PK+L7m+9vUXPDlBcs8VERfuWWiEYxv+3JDr\nqDp8+XBrNa6VVXu3WrrphDd8fIMNWTrEag2pZSNXjrR+8/vZyv0rrc7QOtbjmx7mF+5n2w9vNzNn\n4PbDFR9mOq1PLl53TL7DRq8abUGDgizyaKSVGljqvJML7v7sbnt78duu/RXfrmg/7vzRqr9X3Y7G\nHzUz5274mz+52QIGBFj3b7qbmdmCXQssYECAtZ3Q9rzlOxp/1Gq8V8N+jfw1518um4ri+FJaAxcO\ntOBBwdZxasdCe+V/oSikU1l9avza8QDnXGjtXMyM0WtGM7LjSL7e+jVHTh0h2ZPM6NWj2X98P09c\n9wT9FvRj6PKhzlXiySiebvU01ctXJ9mSuSz4MgBK+JXgqVZP5dn3kqKhWrlq9P+pP8+0foY6FetQ\no3wNDp44SIPKDTJNP3fHXLYe3sr0+6e7jgWVCeLx7x/nrfZvUbG0M5WzVoVajF4zmg/u/IBJGyaR\nmJzIf+b8h/CwcKZunHre8oVHhNPxso7cWOfG86bNrbIBZX2Wd364tPKlXOJ/CRM7T9SVfyaKZI2c\nSjzFzN9n0jCoIfFJ519K2MzSdRuBs/KkxzzcUvcWgsoEERMfw7yd83hv2XuM7jSa0v6lCS4bzJnk\nM2yI2sC3276l55U96d6iO1PuneKrryZFREi5EMoGlOWFG14AILRCqHeF2Ix+3PUjz8x7hg/u/IBL\n/C9xHQ8qE0RIuRB6XdXLu++mOjfxys2v0OPKHuyM2cn4teOpVaEW/275b6JPRp+zbBuiNjB141Te\nvvXtc6Yr7u6//H5+6/0bVctVLeiiFEpFMjjM/H0mrWu1pl6letlqOUTsieC2ybel2/fp+k95pOUj\nzo1TZYI5En+ECesm8OKNL/K3y5wbtWtVqMV/r/8vpf1Lc1uD2wgpFwKQaf+qFC9dm3dl2n3TKFeq\nHAA1A2ty4PgBV7qouCge+uohnm71NJ0adcoyrzF3jUl39dq2XlsGth9IUJkg/Ev48/LPLzP49sFU\nKVuFmPgYkj3J3rQe8zB1w1ScXgR4/ofneT3sdZ30zsO/hL/3b1rcimS30pQNU+h+RXemb5merYeQ\nfLn5S46fOe7dTkhOYMa2GfzW+zfAuXKLPBbJwj0LGXv3WG+6j//+MY2CG7HywEoeu/oxV75SfF1T\n85p02zXL18y05TBk2RC6XdGNZ1o/k2Ve/73hv+f8rIZBDWkU3Iira1wNOHcSD1sxjH3H9jG0w1C+\n3Pwl3Wd0p139ds7kiqN/8OjVj+biW4mcVeSCQ1xCHAsjF/LZPz7j+x3fn7dbKdmTzDfbviEuIc67\n76fdP9E4uDF1KtYBILhsMPN2zuO2BrcRVObsPXgtqrUAYEoXdSPJudUMdAeHI6eOMG7tONb1WXdB\neb9/x/s0Dm7s3Q4pF8L/fvkfzUOa4zEPAxc5S5VsjNrIO0ve4ZWbX/HOhhLJrSIXHBbsXkCbWm2o\ncEkFyviXOW/LYVHkIkIDQzl86jBJniT8S/jzxeYvePDys+sEBZUJIsmTxP3N7vd18eUiVaVsFX4/\n8nu6fcNXDKdL0y7ei5DcuqnOTem2q5atSqmSpdgdu5u5O+ZSqmQpnm39LB+u/JB9x/bR48qi++hX\nKTyK3JjD99u/p9NlTt9taf/S5x1z+Hbbt/yj6T8oF1COkwknSfYk8/327+nStIs3TXCZYPxL+HNP\nk3vOkZNI1qqUrcLhU4e92/GJ8YxcPZJ+N2b2jKsLc2+Texl39zhOJpwkfGE4L9zwAldWu5LZO2bz\nRvs38C9R5K75pBAqUsHBzJizYw53NboLwGk5ZNKtNGz5MI6dPgbAD7t+4G+X/Y3ypcoTlxDH6oOr\nqRFYg9oVa3vTN6jcgIheEem6lERyIrisM6kh1fQt07m25rXeKc956bnrn6NVaCvqVapH9Mlo7m92\nP92u6EZk38h0q6CKXIgidYmxPmo95UuVp2FQQwDKBLi7lU4lnuKF+S/QKrQVNQNrEns6lquqX+UN\nDvN2zqPDpR3SvcfPz8+n88Hl4pex5TBy1Uheuvkln35m4yqNaV+vvXd84UK7r0TSKlIth4g9Edxa\n/1bvdmrLYcX+FZR7y5lSuGTvEhI9iRyJP8IPu37g9ga3U8KvBIGXBBKXEMfcnXO9zygQySupwcHM\nGLhwIDHxMd7uT1+Z2mWqbsAUnylSLYeFkQt5oNkD3u3S/qU5duYY07dM51TiKQB+2fML4MwU+WHX\nD9zT2BlHKF+qPHuP7WVL9BbXAJ/IhapcujKx8bF0n9Gd7Ue2s/BfC7P1gKILUdTvUJbCrci0HDzm\nYXHkYm6pe4t3X2q30vqo9d59P//xM1dWu5LDpw47LY0GTkujfKnyzNs5j+trX5/pXaoiF6JkiZIY\nxqLIRSz810ItvihFXpEJDluit1CpdKV0j5ZM7VZa96czjzz6ZDSbozdzV6O7WLp/KRUuqeBdjbJ8\nqfLM3TmXm+vcXCDll+Lh3ib36opeLgpFJjgsilxE27pt0+0rE1CGzdGbKe1fmpByIczdOZera1xN\nzcCa/LjrR26sfXaQuXxAefYd36cuJfGZL+/7koHtMn92tkhRU2SCw9J9S10zikr7l2b1wdW0qdWG\noDJBzN05lzahbQguE0xcQlz64FCqPAElAmgV2iq/iy7FxP2X3+9dVVWkqCsywWHlgZW0Dm2dbl8Z\n/zJ4zEOb0DZULl2ZH3f9SKvQVgSXDQZIF0zKlyrPNTWvUZNfRCQbikxw+DPuT5pUaZJuX5mAMgDe\nlkNMfAyta7UmuEwwFS6pwOVVL/emrVOxjuv+BhERyVyRmcraskZL19TA0v6l8S/hz9U1riaoTBA1\nA2tSq0ItqpevzsyHZqZL3/va3vldZBGRIqvItByurn61a1/tCrXp3qI7ZQLKULl0ZW+3k38Jf9rW\na+tKLyIi2VNkWg6pa9mnFVohlAmdJwDO+ILHPPldLBGRi1JReaSZbYzaSPOQ5gVdDhGRIiPlqZW5\nOs8XmeCQmJyopYhFRHKgWASH1OfjiohI9lxIcCgyA9IiIpJ/FBxERMRFwUFERFx8HRw6ANuAHUBW\nD9MNA34DNgERPi6PiIhkgy8HpEsCvwO3AQeAVUBXYGuaNJWAJcCdwH6gCnAYNw1Ii4jk0IUMSJ9r\nbuh/M2wbEA38CvyRjbxbATuBPSnb04DOpA8O3YCvcQIDZB4YREQkn52rWykQKJ/mJxC4DpiH0wI4\nn1BgX5rt/Sn70roMCAJ+AVYDPbJVahER8alztRzCs9gfBPwEfH6evLPTDxQAXA3cCpQFlgHLccYo\n0hcm/GxxwsLCCAsLy0b2IiLFR0REBBEREXmSV27HHH4DWp4nTRucAJO6TnZ/wAMMSpOmH1CGs4Fo\nPE7L5KsMeWnMQUQkh/L7Jrh2QGw20q3G6TaqB5QCHgRmZkjzHXATzuB1WaA1sCUXZRIRkTx0rm6l\njZnsqwwcAnpmI+8k4CngB5yT/8c4g9GpD1YYgzPNdR6wAadVMQ4FBxGRAneu5ka9DNsGHAHifFaa\nrKlbSUSGWd2+AAANp0lEQVQkh7TwnoiIuGjhPRERyVMKDiIi4qLgICIiLgoOIiLiouAgIiIuCg4i\nIuKi4CAiIi4KDiIi4qLgICIiLgoOIiLiouAgIiIuCg4iIuKi4CAiIi4KDiIi4qLgICIiLgoOIiLi\nouAgIiIuCg4iIuKi4CAiIi4KDiIi4qLgICIiLgoOIiLiouAgIiIuCg4iIuKi4CAiIi4KDiIi4qLg\nICIiLr4ODh2AbcAOoN850l0HJAFdfFweERHJBl8Gh5LACJwA0QzoCjTNIt0gYB7g58PyiIhINvky\nOLQCdgJ7gERgGtA5k3RPA18B0T4si4iI5IAvg0MosC/N9v6UfRnTdAZGpWybD8sjIiLZ5O/DvLNz\nov8AeDElrR/n6FYKDw/3vg4LCyMsLOzCSicicpGJiIggIiIiT/LyZR9/GyAcZ8wBoD/gwRlfSLU7\nTRmqAKeAx4CZGfIyMzUqRERyws/PD3J5nvdlcPAHfgduBQ4CK3EGpbdmkX4CMAv4JpNjCg4iIjl0\nIcHBl91KScBTwA84M5I+xgkMvVOOj/HhZ4uIyAUoKlNH1XIQEcmhC2k56A5pERFxUXAQEREXBQcR\nEXFRcBARERcFBxERcVFwEBERFwUHERFxUXAQEREXBQcREXFRcBARERcFBxERcVFwEBERFwUHERFx\nUXAQEREXBQcREXFRcBARERcFBxERcVFwEBERFwUHERFxUXAQEREXBQcREXFRcBARERf/gi6AiEhm\ngoKCiI2NLehiFAmVK1cmJiYmT/P0y9PcfMfMrKDLICL5yM/PD/3dZ09WdeXn5we5PM+rW0lERFwU\nHERExEXBQUREXBQcRETEJT+CQwdgG7AD6JfJ8X8C64ENwBKgRT6USUTkgvXv359hw4b5/HNmzZrF\nQw895PPPScvXwaEkMAInQDQDugJNM6TZDdyCExQGAmN9XCYRkQsWHR3N5MmT6dOnDwBbtmzh2muv\nJSgoiEqVKnHjjTfy66+/Zjuvrl27EhoaSqVKlbjppptYuXKl9/jdd9/N5s2b2bhxo0++S2Z8HRxa\nATuBPUAiMA3onCHNMuBYyusVQC0fl0lE5IJNnDiRTp06cckllwAQGhrK9OnTOXLkCLGxsTz00EPc\nd9992corLi6O1q1bs3btWmJjY+nVqxedOnXi5MmT3jRdu3Zl7Nj8u3b2dXAIBfal2d6fsi8rjwBz\nfFoiEZE8MG/ePNq2bevdrlixIvXr18fPz4/k5GRKlChBjRo1spVX/fr16du3L9WqVcPPz4/HHnuM\nhIQEtm/f7k0TFhbG7Nmz8/x7ZMXXd0jn5A6WdsC/gRt9VBYRkTyzceNGGjdu7NpfqVIlTp48Sc2a\nNfn5559zlfe6detISEigYcOG3n1NmjRhz549xMXFUb58+VyXO7t8HRwOALXTbNfGaT1k1AIYhzM2\nken98uHh4d7XYWFhhIWF5VUZRaSI8sujNR5ycyP20aNHCQwMzHT/qVOneP3117n//vtZs2ZN6p3K\n2XL8+HF69OhBeHh4uvxTXx89ejTL4BAREUFERETOvkgWfL18hj/wO3ArcBBYiTMovTVNmjrAz0B3\nYHkW+Wj5DJFiprAvn1GtWjXmzJnDNddck+lxMyMwMJClS5fSokX2JmHGx8fToUMHmjRpwpgxY9Id\ni4mJoUqVKhw/ftwVHIri8hlJwFPAD8AW4AucwNA75QfgVaAyMAr4DSeAiIgUai1atOD333/P8nhy\ncjIej4eyZctmK78zZ85wzz33UKdOHVdgANi6dSv16tXLly4lyJ/7HOYCjYGGwNsp+8ak/AA8CgQD\nLVN+WuVDmURELkjHjh1ZuHChd3vBggWsW7eO5ORkjh8/zvPPP0/jxo294wYTJ06kfv36meaVmJjI\nfffdR9myZZk4cWKmaRYuXEjHjh3z/HtkRXdIi4jkQs+ePZkzZw6nT58GnLGArl27UqlSJRo3bkx0\ndDQzZ870pt+3bx833XRTpnktXbqU2bNnM3/+fCpVqkRgYCCBgYEsWbLEm2batGn07t070/f7gpbs\nFpFCqbCPOQC8/PLLhISE8Oyzz5437Z133snw4cMzneF0PrNmzWLq1KlMmzYt0+O+GHNQcBCRQqko\nBIfCoigOSIuISBGk4CAiIi4KDiIi4qLgICIiLgoOIiLiouAgIiIuCg4iIuKi4CAikkt6TKiIiKST\n8TGhy5cv5/bbbyc4OJiQkBAeeOAB/vzzz2znVdweEyoiclHK+JjQo0eP0qdPHyIjI4mMjCQwMJCH\nH344W3kVxseEavkMESmUCvvyGbfeeiuPPPII3bp1y/T42rVrCQsL4/jx47nKv2LFikRERNCyZUvA\nWZyve/fu7N6925VWy2eIiBQSWT0mNNWiRYto3rx5rvI+32NC84OvHxMqIuIzfq/nTeeHvZbzFkpW\njwkF2LBhAwMHDky3ZHd2XchjQvOSgoOIFFm5OannlcqVK3PixAnX/p07d9KxY0eGDx/OjTfemKM8\n4+Pjufvuu7nhhhvo169fumOpn1WpUqXcFzoH1K0kIpILmT0mNDIykttvv51XX32Vf/7znznKrzg+\nJlRE5KKT8TGhBw4coH379jz11FM8/vjjrvR6TKiISDGQ8TGh48eP548//vCOFQQGBlKhQgVvej0m\n1Dc0lVWkmCnsU1lBjwktDBQcRIqZohAcCgvd5yAiIvlCwUFERFwUHERExEXBQUREXBQcRETERctn\niEihVLly5dTZNnIelStXzvM8fV3zHYAPgJLAeGBQJmmGA38DTgH/An7LJI2msoqI5FBhncpaEhiB\nEyCaAV2BphnSdAQaApcBjwOjfFiei0JERERBF6HQUF2cpbo4S3WRN3wZHFoBO4E9QCIwDeicIc3f\ngU9TXq8AKgHVfFimIk+/+GepLs5SXZylusgbvgwOocC+NNv7U/adL00tH5ZJRESywZfBIbuDBBn7\nwzS4ICJSwHw5IN0GCMcZcwDoD3hIPyg9GojA6XIC2Aa0BaIy5LUTuNRH5RQRuVjtwhnXLVT8cQpW\nDygFrCPzAek5Ka/bAMvzq3AiIlJw/gb8jnPl3z9lX++Un1QjUo6vB67O19KJiIiIiMjFoQPOOMQO\noN950l4MPsEZb9mYZl8QMB/YDvyIM903VX+cutkG3JFPZcwvtYFfgM3AJuCZlP3FsT5K40z1Xgds\nAd5O2V8c6yJVSZwbZmelbBfXutgDbMCpi5Up+y76uiiJ091UDwgg8zGLi83NQEvSB4fBwP+lvO4H\nvJPyuhlOnQTg1NFOLq61sqoDV6W8Lo/TPdmU4lsfZVP+9ccZm7uJ4lsXAM8DU4GZKdvFtS7+wAkG\naV30dXE9MC/N9ospPxe7eqQPDts4e2Ng9ZRtcK4A0ram5uEM6l+svgVuQ/VRFlgFXE7xrYtawAKg\nHWdbDsW1Lv4AgjPsy5O6KMxRIzs30RUH1Tg7tTeKs//pNXHqJNXFXD/1cFpUKyi+9VEC56ovirPd\nbcW1LoYCL+BMjU9VXOvCcALlauCxlH15UheFeVVW3QznZpy7Xi7GOisPfA08C5zIcKw41YcHp5ut\nIvADzlVzWsWlLu4C/sLpYw/LIk1xqQuAG4FDQFWccYZtGY7nui4Kc8vhAM6gZKrapI96xUUUTtMQ\noAbOHwa466dWyr6LSQBOYJiM060Exbs+AI4Bs4FrKJ51cQPOmmx/AJ8D7XF+P4pjXYATGACigRk4\na9pd9HWRnZvoLkb1cA9Ip/YTvoh7cKkUUB+nri6mxe/9gEk4XQhpFcf6qMLZGSdlgEXArRTPukir\nLWfHHIpjXZQFAlNelwOW4MxAKhZ1kdlNdBezz4GDQALOeMvDODMRFpD5tLSXcOpmG3BnvpbU927C\n6UpZh9OF8BvO1ObiWB9XAGtx6mIDTn87FM+6SKstZ2crFce6qI/zO7EOZ7p36jmyONaFiIiIiIiI\niIiIiIiIiIiIiIiIiIiISFESl/JvXaBrHuf9UobtJXmcv4iI+EjqmkxhnL2jNrvOt/5YxvWeRESk\niEg9gS8HjuLcbf0sztpi7+I8JGU98HhKujBgMfAdZxcy+xZn5ctNnF398h0gKSW/ySn7Ulspfil5\nb8S5q/mBNHlHANOBrcCUNOV8B2e11fUp7xURER9KDQ5p1+IBJxi8nPL6EpznJNTDOYHH4XRDpaqc\n8m8ZnBN+6nbGlkPq9j9wli7wA0KASJzF0MJwAlTNlGNLcVbWDCb9ipoVsvvlRHyhMK/KKpLXMi4y\ndgfQE+fKfznOmjQNU46txDmhp3oWZw2bZTgrW152ns+6CfgMZ0nkv4CFwHUp2ytx1tCylDzr4gSM\n08DHwL1AfE6/nEheUnCQ4u4pnAcJtQQuxVmwDOBkmjRhOKugtsF5psJvOM91PhfDHYxS184/k2Zf\nMs7S5Mk4yy1/hfPMgnmIFCAFBylOTnB2iWNwHprzJGcHnRtx9lnNaVUAYnGu7JuQ/tGKiWQ+aL0Y\neBDnb6wqcAtOiyGrJZLL4ayeORfn+chXnvfbiPhQYX4SnEheSb1iX49zhb4OmAAMxxljWItz0v4L\np0sn49Oz5gF9gC04S8gvS3NsLM6A8xqgR5r3zcB5Dvr6lH0vpOTfFPfTtwwnaH2H0yLxA57L9bcV\nERERERERERERERERERERERERERERERERERERuZj9fy4/lYWOgFpPAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fb5874906a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "graph_utility_estimates(agent, Fig[17,1], 500, [(2,2), (3,2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}