{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learners\n",
    "\n",
    "In this section, we will introduce several pre-defined learners to learning the datasets by updating their weights to minimize the loss function. when using a learner to deal with machine learning problems, there are several standard steps:\n",
    "\n",
    "- **Learner initialization**: Before training the network, it usually should be initialized first. There are several choices when initializing the weights: random initialization, initializing weights are zeros or use Gaussian distribution to init the weights.\n",
    "\n",
    "- **Optimizer specification**: Which means specifying the updating rules of learnable parameters of the network. Usually, we can choose Adam optimizer as default.\n",
    "\n",
    "- **Applying back-propagation**: In neural networks, we commonly use back-propagation to pass and calculate gradient information of each layer. Back-propagation needs to be integrated with the chosen optimizer in order to update the weights of NN properly in each epoch.\n",
    "\n",
    "- **Iterations**: Iterating over the forward and back-propagation process of given epochs. Sometimes the iterating process will have to be stopped by triggering early access in case of overfitting.\n",
    "\n",
    "We will introduce several learners with different structures. We will import all necessary packages before that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import os, sys\n",
    "sys.path = [os.path.abspath(\"../../\")] + sys.path\n",
    "from deep_learning4e import *\n",
    "from notebook4e import *\n",
    "from learning4e import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perceptron Learner\n",
    "\n",
    "### Overview\n",
    "\n",
    "The Perceptron is a linear classifier. It works the same way as a neural network with no hidden layers (just input and output). First, it trains its weights given a dataset and then it can classify a new item by running it through the network.\n",
    "\n",
    "Its input layer consists of the item features, while the output layer consists of nodes (also called neurons). Each node in the output layer has *n* synapses (for every item feature), each with its own weight. Then, the nodes find the dot product of the item features and the synapse weights. These values then pass through an activation function (usually a sigmoid). Finally, we pick the largest of the values and we return its index.\n",
    "\n",
    "Note that in classification problems each node represents a class. The final classification is the class/node with the max output value.\n",
    "\n",
    "Below you can see a single node/neuron in the outer layer. With *f* we denote the item features, with *w* the synapse weights, then inside the node we have the dot product and the activation function, *g*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![perceptron](images/perceptron.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Implementation\n",
    "\n",
    "Perceptron learner is actually a neural network learner with only one hidden layer which is pre-defined in the algorithm of `perceptron_learner`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_net = [InputLayer(input_size), DenseLayer(input_size, output_size)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where `input_size` and `output_size` are calculated from dataset examples. In the perceptron learner, the gradient descent optimizer is used to update the weights of the network. we return a function `predict` which we will use in the future to classify a new item. The function computes the (algebraic) dot product of the item with the calculated weights for each node in the outer layer. Then it picks the greatest value and classifies the item in the corresponding class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Example\n",
    "\n",
    "Let's try the perceptron learner with the `iris` dataset examples, first let's regulate the dataset classes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = DataSet(name=\"iris\")\n",
    "classes = [\"setosa\", \"versicolor\", \"virginica\"]\n",
    "iris.classes_to_numbers(classes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:50, total_loss:14.089098023560856\n",
      "epoch:100, total_loss:12.439240091345326\n",
      "epoch:150, total_loss:11.848151059704785\n",
      "epoch:200, total_loss:11.283665595671044\n",
      "epoch:250, total_loss:11.153290841913241\n",
      "epoch:300, total_loss:11.00747536734494\n",
      "epoch:350, total_loss:10.871093050365419\n",
      "epoch:400, total_loss:10.838400319844233\n",
      "epoch:450, total_loss:10.687417928867456\n",
      "epoch:500, total_loss:10.650371951865573\n"
     ]
    }
   ],
   "source": [
    "pl = perceptron_learner(iris, epochs=500, learning_rate=0.01, verbose=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the printed lines that the final total loss is converged to around 10.50. If we check the error ratio of perceptron learner on the dataset after training, we will see it is much higher than randomly guess:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.046666666666666634\n"
     ]
    }
   ],
   "source": [
    "print(err_ratio(pl, iris))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we test the trained learner with some test cases:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n"
     ]
    }
   ],
   "source": [
    "tests = [([5.0, 3.1, 0.9, 0.1], 0),\n",
    "        ([5.1, 3.5, 1.0, 0.0], 0),\n",
    "        ([4.9, 3.3, 1.1, 0.1], 0),\n",
    "        ([6.0, 3.0, 4.0, 1.1], 1),\n",
    "        ([6.1, 2.2, 3.5, 1.0], 1),\n",
    "        ([5.9, 2.5, 3.3, 1.1], 1),\n",
    "        ([7.5, 4.1, 6.2, 2.3], 2),\n",
    "        ([7.3, 4.0, 6.1, 2.4], 2),\n",
    "        ([7.0, 3.3, 6.1, 2.5], 2)]\n",
    "print(grade_learner(pl, tests))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems the learner is correct on all the test examples.\n",
    "\n",
    "Now let's try perceptron learner on a more complicated dataset: the MNIST dataset, to see what the result will be. First, we import the dataset to make the examples a `Dataset` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of training dataset: 60000\n",
      "length of test dataset: 10000\n"
     ]
    }
   ],
   "source": [
    "train_img, train_lbl, test_img, test_lbl = load_MNIST(path=\"../../aima-data/MNIST/Digits\")\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "train_examples = [np.append(train_img[i], train_lbl[i]) for i in range(len(train_img))]\n",
    "test_examples = [np.append(test_img[i], test_lbl[i]) for i in range(len(test_img))]\n",
    "print(\"length of training dataset:\", len(train_examples))\n",
    "print(\"length of test dataset:\", len(test_examples))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's train the perceptron learner on the first 1000 examples of the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:1, total_loss:423.8627535296463\n",
      "epoch:2, total_loss:341.31697581698995\n",
      "epoch:3, total_loss:328.98647291325443\n",
      "epoch:4, total_loss:327.8999700915627\n",
      "epoch:5, total_loss:310.081065570072\n",
      "epoch:6, total_loss:268.5474616202945\n",
      "epoch:7, total_loss:259.0999998773958\n",
      "epoch:8, total_loss:259.09999987481393\n",
      "epoch:9, total_loss:259.09999987211944\n",
      "epoch:10, total_loss:259.0999998693056\n"
     ]
    }
   ],
   "source": [
    "mnist = DataSet(examples=train_examples[:1000])\n",
    "pl = perceptron_learner(mnist, epochs=10, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.893\n"
     ]
    }
   ],
   "source": [
    "print(err_ratio(pl, mnist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like we have a near 90% error ratio on training data after the network is trained on it. Then we can investigate the model's performance on the test dataset which it never has seen before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.92\n"
     ]
    }
   ],
   "source": [
    "test_mnist = DataSet(examples=test_examples[:100])\n",
    "print(err_ratio(pl, test_mnist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems a single layer perceptron learner cannot simulate the structure of the MNIST dataset. To improve accuracy, we may not only increase training epochs but also consider changing to a more complicated network structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Neural Network Learner\n",
    "\n",
    "Although there are many different types of neural networks, the dense neural network we implemented can be treated as a stacked perceptron learner. Adding more layers to the perceptron network could add to the non-linearity to the network thus model will be more flexible when fitting complex data-target relations. Whereas it also adds to the risk of overfitting as the side effect of flexibility.\n",
    "\n",
    "By default we use dense networks with two hidden layers, which has the architecture as the following:\n",
    "\n",
    "<img src=\"images/nn.png\" width=\"500\"/>\n",
    "\n",
    "In our code, we implemented it as:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialize the network\n",
    "raw_net = [InputLayer(input_size)]\n",
    "# add hidden layers\n",
    "hidden_input_size = input_size\n",
    "for h_size in hidden_layer_sizes:\n",
    "    raw_net.append(DenseLayer(hidden_input_size, h_size))\n",
    "    hidden_input_size = h_size\n",
    "raw_net.append(DenseLayer(hidden_input_size, output_size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Where hidden_layer_sizes are the sizes of each hidden layer in a list which can be specified by user. Neural network learner uses gradient descent as default optimizer but user can specify any optimizer when calling `neural_net_learner`. The other special attribute that can be changed in `neural_net_learner` is `batch_size` which controls the number of examples used in each round of update. `neural_net_learner` also returns a `predict` function which calculates prediction by multiplying weight to inputs and applying activation functions.\n",
    "\n",
    "### Example\n",
    "\n",
    "Let's also try `neural_net_learner` on the `iris` dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:10, total_loss:15.931817841643683\n",
      "epoch:20, total_loss:8.248422285412149\n",
      "epoch:30, total_loss:6.102968668275\n",
      "epoch:40, total_loss:5.463915043272969\n",
      "epoch:50, total_loss:5.298986288420822\n",
      "epoch:60, total_loss:4.032928400456889\n",
      "epoch:70, total_loss:3.2628899927346855\n",
      "epoch:80, total_loss:6.01336701367312\n",
      "epoch:90, total_loss:5.412020420311795\n",
      "epoch:100, total_loss:3.1044027319850773\n"
     ]
    }
   ],
   "source": [
    "nn = neural_net_learner(iris, epochs=100, learning_rate=0.15, optimizer=gradient_descent, verbose=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly we check the model's accuracy on both training and test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "error ration on training set: 0.033333333333333326\n"
     ]
    }
   ],
   "source": [
    "print(\"error ration on training set:\",err_ratio(nn, iris))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy on test set: 1\n"
     ]
    }
   ],
   "source": [
    "tests = [([5.0, 3.1, 0.9, 0.1], 0),\n",
    "        ([5.1, 3.5, 1.0, 0.0], 0),\n",
    "        ([4.9, 3.3, 1.1, 0.1], 0),\n",
    "        ([6.0, 3.0, 4.0, 1.1], 1),\n",
    "        ([6.1, 2.2, 3.5, 1.0], 1),\n",
    "        ([5.9, 2.5, 3.3, 1.1], 1),\n",
    "        ([7.5, 4.1, 6.2, 2.3], 2),\n",
    "        ([7.3, 4.0, 6.1, 2.4], 2),\n",
    "        ([7.0, 3.3, 6.1, 2.5], 2)]\n",
    "print(\"accuracy on test set:\",grade_learner(nn, tests))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the error ratio on the training set is smaller than the perceptron learner. As the error ratio is relatively small, let's try the model on the MNIST dataset to see whether there will be a larger difference. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:10, total_loss:89.0002153455983\n",
      "epoch:20, total_loss:87.29675663038348\n",
      "epoch:30, total_loss:86.29591779319225\n",
      "epoch:40, total_loss:83.78091780128402\n",
      "epoch:50, total_loss:82.17091581738829\n",
      "epoch:60, total_loss:83.8434277386084\n",
      "epoch:70, total_loss:83.55209905561495\n",
      "epoch:80, total_loss:83.106898191118\n",
      "epoch:90, total_loss:83.37041170165992\n",
      "epoch:100, total_loss:82.57013813500876\n"
     ]
    }
   ],
   "source": [
    "nn = neural_net_learner(mnist, epochs=100, verbose=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.784\n"
     ]
    }
   ],
   "source": [
    "print(err_ratio(nn, mnist))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After the model converging, the model's error ratio on the training set is still high. We will introduce the convolutional network in the following chapters to see how it helps improve accuracy on learning this dataset."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
