{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning and PyTorch\n",
    "## 11/28/2023\n",
    "\n",
    "<a href=\"?print-pdf\">print view</a>\n",
    "\n",
    "<a href=\"deeplearning.ipynb\">notebook</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<script src=\"https://bits.csb.pitt.edu/preamble.js\"></script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<script src=\"https://bits.csb.pitt.edu/preamble.js\"></script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# OMET Teaching Survey\n",
    "\n",
    "Please fill out."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Perceptron\n",
    "\n",
    "<center><img src=\"../files/perceptron.png\" width=\"400\"></center>\n",
    "\n",
    "$$output = \\begin{cases} 0 \\text{ if } w\\cdot x + b \\le 0 \\\\ 1 \\text{ if } w\\cdot x + b > 0 \\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Perceptron\n",
    "\n",
    "Consider the following perceptron:\n",
    "\n",
    "<center><img src=\"imgs/tikz2.png\"></center>\n",
    "\n",
    "If $x$ takes on only binary values, what are the possible outputs?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"inand\" style=\"width: 500px\"></div>\n",
       "<script>\n",
       "\n",
       "    var divid = '#inand';\n",
       "\tjQuery(divid).asker({\n",
       "\t    id: divid,\n",
       "\t    question: \"What are the corresponding outputs for x = [0,0],[0,1],[1,0], and [1,1]?\",\n",
       "\t\tanswers: [\"0,0,0,0\",\"0,1,1,0\",\"0,0,0,1\",\"0,1,1,1\",\"1,1,1,0\"],\n",
       "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
       "\t\tcharter: chartmaker})\n",
       "    \n",
       "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
       "\n",
       "\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div id=\"inand\" style=\"width: 500px\"></div>\n",
    "<script>\n",
    "\n",
    "    var divid = '#inand';\n",
    "\tjQuery(divid).asker({\n",
    "\t    id: divid,\n",
    "\t    question: \"What are the corresponding outputs for x = [0,0],[0,1],[1,0], and [1,1]?\",\n",
    "\t\tanswers: [\"0,0,0,0\",\"0,1,1,0\",\"0,0,0,1\",\"0,1,1,1\",\"1,1,1,0\"],\n",
    "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
    "\t\tcharter: chartmaker})\n",
    "    \n",
    "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
    "\n",
    "\n",
    "</script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Neurons\n",
    "\n",
    "<img src=\"imgs/tikz9.png\">\n",
    "\n",
    "Instead of a *binary* output, we set the output to the result of an **activation function** $\\sigma$\n",
    "\n",
    "$$output = \\sigma(w\\cdot x + b)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "%matplotlib inline\n",
    "x = np.linspace(-10,10,500)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Activation Functions: Step (Perceptron)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAny0lEQVR4nO3de3RU1eH28ScBMoFCAhiYkDQSvIHKJRRkGqzV1imppVR6sRFZQiOi2GCR2P4gVhLRaqhFpLXYeAFxvWpFXdVeoLAwFX0pKUiAWilgUSAgzADykolRCGT2+0d+GTrmQiYm7Jmc72etWcuc2fvMPhyHPOzbiTPGGAEAAFgSb7sBAADA2QgjAADAKsIIAACwijACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKzqarsBrREMBnXw4EH16tVLcXFxtpsDAABawRij6upqpaWlKT6++f6PmAgjBw8eVEZGhu1mAACANti/f7+++MUvNvt+TISRXr16Saq/mKSkJMutAQAArREIBJSRkRH6Pd6cmAgjDUMzSUlJhBEAAGLM2aZYMIEVAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFWEEQAAYBVhBAAAWBVxGHnrrbc0YcIEpaWlKS4uTq+99tpZ66xbt05f+tKX5HK5dNFFF2n58uVtaCoAAOiMIt4OvqamRiNGjNAtt9yi733ve2ctv2fPHo0fP14zZszQ888/r7KyMt16660aMGCAcnJy2tRoAJ3LkSO2WwA4W79+dj8/zhhj2lw5Lk6vvvqqJk6c2GyZOXPmaOXKlXr33XdDx2688UYdP35cq1evbtXnBAIBJScnq6qqimfTAJ3QWR5bAaCDtT0JtKy1v787fM5IeXm5vF5v2LGcnByVl5c3W+fkyZMKBAJhLwCd08nTdbabADje9oNVVj+/w8OIz+eT2+0OO+Z2uxUIBPTpp582WaekpETJycmhV0ZGRkc3E4AlT//fPbabADje7f+nwurnR+VqmsLCQlVVVYVe+/fvt90kAB3k45OnbTcBcLxnbxlj9fMjnsAaqdTUVPn9/rBjfr9fSUlJ6t69e5N1XC6XXC5XRzcNQJTwzFuvP9/5FdvNAByrX7+eVj+/w8NIdna2Vq1aFXZs7dq1ys7O7uiPBhADjJFcvU5Zn80PwJ6Ih2k+/vhjbdu2Tdu2bZNUv3R327ZtqqyslFQ/xDJlypRQ+RkzZuiDDz7Q//zP/2jnzp16/PHH9dJLL2n27NntcwUAYppRB03jBxAzIg4jmzdv1siRIzVy5EhJUkFBgUaOHKmioiJJ0qFDh0LBRJIGDRqklStXau3atRoxYoQeeeQRPf300+wxAiCElb2As0U8THPNNdeopa1Jmtpd9ZprrtHWrVsj/SgATmDq9ywC4FxRuZoGgHMY0TMCOB1hBAAAWEUYAWCVMYauEcDhCCMArCKLACCMAAAAqwgjAKwyYjUN4HSEEQBWMUwDgDACwCp2YAVAGAFgHaM0gLMRRgBYVT9MQxoBnIwwAgAArCKMALCOYRrA2QgjAKxq6cGbAJyBMALAKqIIAMIIAOvY9AxwNsIIAKvY9AwAYQSAVUaGCayAwxFGAACAVYQRAFYZw9JewOkIIwCsMmIHVsDpCCMAAMAqwggAqximAUAYAWCZYZAGcDjCCACr2A0eAGEEgH2M0wCORhgBYBU7sAIgjACwyvCoPMDxCCMArGOUBnA2wggAqximAUAYAWAVgzQACCMArItjnAZwNMIIAKsYpgFAGAFglZFhAivgcIQRAABgFWEEgF1GimOgBnA0wggAq4zEpBHA4QgjAADAKsIIAKuMMXSMAA5HGAFglRHbwQNORxgBYJVhC1bA8QgjAKxjNQ3gbIQRAFYxTAOAMALAKsM4DeB4hBEA1tEzAjgbYQSAVUbMGQGcjjACwC5DzwjgdIQRAABgFWEEgFVGTGAFnI4wAsAqY6Q4xmkARyOMAAAAqwgjAKwyRqylARyOMALAKiPDahrA4QgjAKxiA1YAhBEA1tExAjgbYQSAVfUPyiOOAE5GGAFgFcM0AAgjAKyjXwRwtjaFkSVLligzM1OJiYnyeDzatGlTi+UXL16swYMHq3v37srIyNDs2bN14sSJNjUYQGfDahrA6SIOIytWrFBBQYGKi4u1ZcsWjRgxQjk5OTp8+HCT5V944QXNnTtXxcXF2rFjh5YuXaoVK1bonnvu+dyNBxD7GKYBEHEYWbRokaZPn668vDxddtllKi0tVY8ePbRs2bImy2/YsEFXXnmlbrrpJmVmZmrcuHGaNGnSWXtTADgJXSOAk0UURmpra1VRUSGv13vmBPHx8nq9Ki8vb7LO2LFjVVFREQofH3zwgVatWqVvfetbzX7OyZMnFQgEwl4AOqf61TS2WwHApq6RFD569Kjq6urkdrvDjrvdbu3cubPJOjfddJOOHj2qr3zlKzLG6PTp05oxY0aLwzQlJSWaP39+JE0DEKOMMYqjZwRwtA5fTbNu3To99NBDevzxx7Vlyxb94Q9/0MqVK/XAAw80W6ewsFBVVVWh1/79+zu6mQAAwJKIekZSUlLUpUsX+f3+sON+v1+pqalN1pk3b55uvvlm3XrrrZKkYcOGqaamRrfddpt+/vOfKz6+cR5yuVxyuVyRNA1AjGKYBkBEPSMJCQkaNWqUysrKQseCwaDKysqUnZ3dZJ1PPvmkUeDo0qWLpPruWQDOVv/UXtII4GQR9YxIUkFBgaZOnarRo0drzJgxWrx4sWpqapSXlydJmjJlitLT01VSUiJJmjBhghYtWqSRI0fK4/Fo9+7dmjdvniZMmBAKJQAAwLkiDiO5ubk6cuSIioqK5PP5lJWVpdWrV4cmtVZWVob1hNx7772Ki4vTvffeqw8//FD9+vXThAkT9OCDD7bfVQCIWQzTAIgzMTBWEggElJycrKqqKiUlJdluDoB2dPPSjeqV2FWPTx5luykA2llrf3/zbBoAAGAVYQSAdUxgBZyNMALAKmPEbvCAwxFGAFhlFPXT1gB0MMIIAOvoGAGcjTACwCpjpDjW9gKORhgBYFX0by4AoKMRRgBYR78I4GyEEQBWGRl2YAUcjjACwKr6B+UBcDLCCAAAsIowAsCq+gfl0TcCOBlhBIBdDNMAjkcYAQAAVhFGAFhl6BoBHI8wAsCq+tU0pBHAyQgjAKxiA1YAhBEA1rGYBnA2wggAq4wxDNIADkcYAWAVwzQACCMArGOYBnA2wggAq1hNA4AwAsCq+u3gbbcCgE2EEQAAYBVhBIBdxtAzAjgcYQSAVfWraUgjgJMRRgAAgFWEEQBWGcMEVsDpCCMArDJiB1bA6QgjAKwybMEKOB5hBIB1DNMAzkYYAWAVO7ACIIwAsIpRGgCEEQDWMUwDOBthBIBVxrCaBnA6wggAALCKMALAujjGaQBHI4wAsIp9RgAQRgBYZcRTewGnI4wAAACrCCMArGLTMwCEEQBWGbHPCOB0hBEAAGAVYQSAVWx6BoAwAsAqhmkAEEYA2MU+I4DjEUYAWMcOrICzEUYAWGUk5owADkcYAWCVYT94wPEIIwDso2sEcDTCCACr6odpSCOAkxFGAFjFKA0AwggA61hMAzgbYQSAVUbswAo4HWEEgFXG0DMCOB1hBAAAWNWmMLJkyRJlZmYqMTFRHo9HmzZtarH88ePHlZ+frwEDBsjlcumSSy7RqlWr2tRgAJ2LMaymAZyua6QVVqxYoYKCApWWlsrj8Wjx4sXKycnRrl271L9//0bla2tr9Y1vfEP9+/fXK6+8ovT0dO3bt0+9e/duj/YD6AQYpgGcLeIwsmjRIk2fPl15eXmSpNLSUq1cuVLLli3T3LlzG5VftmyZjh07pg0bNqhbt26SpMzMzM/XagAA0GlENExTW1uriooKeb3eMyeIj5fX61V5eXmTdf70pz8pOztb+fn5crvdGjp0qB566CHV1dU1+zknT55UIBAIewHonIxhNQ3gdBGFkaNHj6qurk5utzvsuNvtls/na7LOBx98oFdeeUV1dXVatWqV5s2bp0ceeUS/+MUvmv2ckpISJScnh14ZGRmRNBNADDES4zSAw3X4appgMKj+/fvrySef1KhRo5Sbm6uf//znKi0tbbZOYWGhqqqqQq/9+/d3dDMBWMIOrAAimjOSkpKiLl26yO/3hx33+/1KTU1tss6AAQPUrVs3denSJXTs0ksvlc/nU21trRISEhrVcblccrlckTQNQAyjXwRwtoh6RhISEjRq1CiVlZWFjgWDQZWVlSk7O7vJOldeeaV2796tYDAYOvbee+9pwIABTQYRAM5iZBilARwu4mGagoICPfXUU3r22We1Y8cO3XHHHaqpqQmtrpkyZYoKCwtD5e+44w4dO3ZMs2bN0nvvvaeVK1fqoYceUn5+fvtdBYCYxTANgIiX9ubm5urIkSMqKiqSz+dTVlaWVq9eHZrUWllZqfj4MxknIyNDa9as0ezZszV8+HClp6dr1qxZmjNnTvtdBYCYxqZngLPFGRP9/y4JBAJKTk5WVVWVkpKSbDcHQDu64sHXdfOXB+on115suykA2llrf3/zbBoAVtVvBw/AyQgjAADAKsIIAMtYTQM4HWEEgFXGSHGkEcDRCCMAAMAqwggAq6J+OR+ADkcYAWCVMcwZAZyOMALAKnpGABBGAFjHDqyAsxFGAFhVv5rGdisA2EQYAWBVDDyRAkAHI4wAsI6OEcDZCCMArDJimAZwOsIIALsYpQEcjzACwDpW0wDORhgBYBXDNAAIIwCsYjUNAMIIAACwijACwKr6YRrGaQAnI4wAsMoY9hkBnI4wAgAArCKMALDKyLCaBnA4wggAqximAUAYAWAVC3sBEEYAWMdqGsDZCCMA7DLswAo4HWEEgFWGgRrA8QgjAKyjYwRwNsIIAKsMT8oDHI8wAsAqBmkAEEYAWEe/COBshBEAVhnDDqyA0xFGAFhlJMXRNwI4GmEEAABYRRgBYJVh0zPA8QgjAKwjiwDORhgBAABWEUYAWGNM/S4jDNMAzkYYAWDN/2YRVtMADkcYAWANu68CkAgjAKIBHSOAoxFGAFgTmjNiuR0A7CKMALCGYRoAEmEEQBSIYzkN4GiEEQDWnFlNA8DJCCMArDFinxEAhBEAAGAZYQSANaFhGnpGAEcjjACwjh1YAWcjjAAAAKsIIwCsYZgGgEQYAWCRYdszACKMALDIkEUAiDACIAqwAyvgbIQRANY0dIwQRQBnI4wAsMYwTgNAbQwjS5YsUWZmphITE+XxeLRp06ZW1XvxxRcVFxeniRMntuVjAXRSjNIAzhZxGFmxYoUKCgpUXFysLVu2aMSIEcrJydHhw4dbrLd371799Kc/1VVXXdXmxgLoXM4M05BGACeLOIwsWrRI06dPV15eni677DKVlpaqR48eWrZsWbN16urqNHnyZM2fP18XXHDB52owgM6DURoAUoRhpLa2VhUVFfJ6vWdOEB8vr9er8vLyZuvdf//96t+/v6ZNm9aqzzl58qQCgUDYC0DnxTAN4GwRhZGjR4+qrq5Obrc77Ljb7ZbP52uyzvr167V06VI99dRTrf6ckpISJScnh14ZGRmRNBNArGjYgdVuKwBY1qGraaqrq3XzzTfrqaeeUkpKSqvrFRYWqqqqKvTav39/B7YSgC0NO7DSMwI4W9dICqekpKhLly7y+/1hx/1+v1JTUxuVf//997V3715NmDAhdCwYDNZ/cNeu2rVrly688MJG9Vwul1wuVyRNAwAAMSqinpGEhASNGjVKZWVloWPBYFBlZWXKzs5uVH7IkCH617/+pW3btoVe3/nOd/S1r31N27ZtY/gFcLgzE1jpGgGcLKKeEUkqKCjQ1KlTNXr0aI0ZM0aLFy9WTU2N8vLyJElTpkxRenq6SkpKlJiYqKFDh4bV7927tyQ1Og7AeUJLe8kigKNFHEZyc3N15MgRFRUVyefzKSsrS6tXrw5Naq2srFR8PBu7AgCA1okzMbAfcyAQUHJysqqqqpSUlGS7OQDayUcfn9SoX7yuJ28epXGXN553BiC2tfb3N10YAKw5M0zDOA3gZIQRANZEf78sgHOBMALAOvpFAGcjjACwhk3PAEiEEQA2MUwDQIQRAFGAnhHA2QgjAKwJraZh1gjgaIQRANawmgaARBgBEA3oGAEcjTACwJrQahrL7QBgF2EEgDUNwzTswAo4G2EEAABYRRgBYM2Z1TQAnIwwAsCahoeGM0oDOBthBAAAWEUYAWBNaAIrAzWAoxFGAFjHMA3gbIQRANawAysAiTACIArQMQI4G2EEgDUNO7CSRgBnI4wAsIZhGgASYQRAFGA1DeBshBEA1oR2YCWLAI5GGAFgTWgHVsvtAGAXYQQAAFhFGAFgzZlhGvpGACcjjACwJrQdPFkEcDTCCAAAsIowAsAiJrACIIwAsIhhGgASYQSARWzACkAijACICnSNAE5GGAFgDcM0ACTCCACLDAM1AEQYARAF6BgBnI0wAsCaM8M0xBHAyQgjAKwxjNIAEGEEQBSgXwRwNsIIAGsaJrAySgM4G2EEgDWhOSP0jQCORhgBAABWEUYAWMcwDeBshBEA1rCaBoBEGAEAAJYRRgBYw2oaABJhBIBFrKYBIBFGAFjElBEAEmEEQBRgmAZwNsIIAGuMYc4IAMIIAIsYpgEgEUYARAEmsALORhgBYE1oNQ1ZBHA0wggAi/53zojlVgCwizACAACsIowAsIZhGgASYQSARWdW05BGACdrUxhZsmSJMjMzlZiYKI/Ho02bNjVb9qmnntJVV12lPn36qE+fPvJ6vS2WBwAAzhJxGFmxYoUKCgpUXFysLVu2aMSIEcrJydHhw4ebLL9u3TpNmjRJb7zxhsrLy5WRkaFx48bpww8//NyNBxDbGKYBILUhjCxatEjTp09XXl6eLrvsMpWWlqpHjx5atmxZk+Wff/55/fjHP1ZWVpaGDBmip59+WsFgUGVlZZ+78QBiW2gHVsvtAGBXRGGktrZWFRUV8nq9Z04QHy+v16vy8vJWneOTTz7RqVOn1Ldv32bLnDx5UoFAIOwFAAA6p4jCyNGjR1VXVye32x123O12y+fzteocc+bMUVpaWlig+aySkhIlJyeHXhkZGZE0E0CMaJjAGsc4DeBo53Q1zYIFC/Tiiy/q1VdfVWJiYrPlCgsLVVVVFXrt37//HLYSwLkSmjNitxkALOsaSeGUlBR16dJFfr8/7Ljf71dqamqLdRcuXKgFCxbo9ddf1/Dhw1ss63K55HK5ImkagBhkeFQeAEXYM5KQkKBRo0aFTT5tmIyanZ3dbL2HH35YDzzwgFavXq3Ro0e3vbUAOiVGaQBni6hnRJIKCgo0depUjR49WmPGjNHixYtVU1OjvLw8SdKUKVOUnp6ukpISSdIvf/lLFRUV6YUXXlBmZmZobknPnj3Vs2fPdrwUADEnNExDGgGcLOIwkpubqyNHjqioqEg+n09ZWVlavXp1aFJrZWWl4uPPdLj87ne/U21trX7wgx+Enae4uFj33Xff52s9gJjGIA0AqQ1hRJJmzpypmTNnNvneunXrwn7eu3dvWz4CgIMwTAM4G8+mAWCNoWsEgAgjACxqWE1DzwjgbIQRAABgFWEEgDVnHpRH1wjgZIQRANaEtoO32goAthFGAACAVYQRANYYwwRWAIQRABadGaYhjQBORhgBYA/7jAAQYQRAFGCYBnA2wggAa0KbnlluBwC7CCMArGE7eAASYQRANKBrBHA0wggAa0I7sJJGAEcjjACwhlEaABJhBEAUYDUN4GyEEQDWhHZgtdwOAHYRRgBYE9qBla4RwNEIIwAAwCrCCABrzqymAeBkhBEAFvHUXgCEEQAAYBlhBIA1bHoGQCKMALAotOkZWQRwNMIIAGt4UB4AiTACIAowgRVwNsIIAGuM2IEVAGEEgEUM0wCQCCMAogDbwQPORhgBYE3o2TRWWwHANsIIAGtCT+0ljQCORhgBAABWEUYAWMcOrICzEUYAWBPaDp4sAjgaYQQAAFhFGAFgjREbjQAgjACwiGEaABJhBAAAWEYYAWBNqGeE1TSAoxFGAFgT2oGVLAI4GmEEgDWGJ+UBEGEEQBSgYwRwNsIIAGvODNMQRwAnI4wAsIdRGgAijACIAvSLAM5GGAFgTcMOrIzSAM5GGAFgzZkdWEkjgJMRRgAAgFWEEQDWMH8VgEQYAWCRMcwXAUAYAQAAlhFGAFhjZFjWC4AwAsCe+mEa4gjgdIQRANYwgRWARBgBYBn9IgAIIwDsMYbVNADaFkaWLFmizMxMJSYmyuPxaNOmTS2Wf/nllzVkyBAlJiZq2LBhWrVqVZsaC6BzYZgGgCR1jbTCihUrVFBQoNLSUnk8Hi1evFg5OTnatWuX+vfv36j8hg0bNGnSJJWUlOjb3/62XnjhBU2cOFFbtmzR0KFD2+UiPo8jR2y3AHCuwP+LV12Ny3YzAFgWZ4yJ6B8nHo9HV1xxhX77299KkoLBoDIyMnTnnXdq7ty5jcrn5uaqpqZGf/nLX0LHvvzlLysrK0ulpaWt+sxAIKDk5GRVVVUpKSkpkuaeFV3EgH2R/S0EIFa09vd3RMM0tbW1qqiokNfrPXOC+Hh5vV6Vl5c3Wae8vDysvCTl5OQ0W16STp48qUAgEPbqCIerT3TIeQEAQOtFFEaOHj2quro6ud3usONut1s+n6/JOj6fL6LyklRSUqLk5OTQKyMjI5Jmttqqdw51yHkBAEDrReVqmsLCQlVVVYVe+/fv75DPuT4rvUPOCwAAWi+iCawpKSnq0qWL/H5/2HG/36/U1NQm66SmpkZUXpJcLpdcro6f1NbnCwk6fLjDPwYAALQgojCSkJCgUaNGqaysTBMnTpRUP4G1rKxMM2fObLJOdna2ysrKdNddd4WOrV27VtnZ2W1udHvq1892CwAAcLaIl/YWFBRo6tSpGj16tMaMGaPFixerpqZGeXl5kqQpU6YoPT1dJSUlkqRZs2bp6quv1iOPPKLx48frxRdf1ObNm/Xkk0+275UAAICYFHEYyc3N1ZEjR1RUVCSfz6esrCytXr06NEm1srJS8fFnpqKMHTtWL7zwgu69917dc889uvjii/Xaa69FxR4jAADAvoj3GbGhI/cZAQAAHaND9hkBAABob4QRAABgFWEEAABYRRgBAABWEUYAAIBVhBEAAGAVYQQAAFhFGAEAAFYRRgAAgFURbwdvQ8MmsYFAwHJLAABAazX83j7bZu8xEUaqq6slSRkZGZZbAgAAIlVdXa3k5ORm34+JZ9MEg0EdPHhQvXr1UlxcXLudNxAIKCMjQ/v37++0z7zp7NfI9cW+zn6NXF/s6+zX2JHXZ4xRdXW10tLSwh6i+1kx0TMSHx+vL37xix12/qSkpE75P9h/6+zXyPXFvs5+jVxf7Ovs19hR19dSj0gDJrACAACrCCMAAMAqR4cRl8ul4uJiuVwu203pMJ39Grm+2NfZr5Hri32d/Rqj4fpiYgIrAADovBzdMwIAAOwjjAAAAKsIIwAAwCrCCAAAsKrTh5EHH3xQY8eOVY8ePdS7d+8my1RWVmr8+PHq0aOH+vfvr5/97Gc6ffp0i+c9duyYJk+erKSkJPXu3VvTpk3Txx9/3AFX0Hrr1q1TXFxck6+333672XrXXHNNo/IzZsw4hy2PTGZmZqP2LliwoMU6J06cUH5+vs477zz17NlT3//+9+X3+89Ri1tv7969mjZtmgYNGqTu3bvrwgsvVHFxsWpra1usF+33cMmSJcrMzFRiYqI8Ho82bdrUYvmXX35ZQ4YMUWJiooYNG6ZVq1ado5ZGpqSkRFdccYV69eql/v37a+LEidq1a1eLdZYvX97oXiUmJp6jFkfuvvvua9TeIUOGtFgnVu6f1PTfJ3FxccrPz2+yfLTfv7feeksTJkxQWlqa4uLi9Nprr4W9b4xRUVGRBgwYoO7du8vr9eo///nPWc8b6Xc4Up0+jNTW1uqGG27QHXfc0eT7dXV1Gj9+vGpra7VhwwY9++yzWr58uYqKilo87+TJk7V9+3atXbtWf/nLX/TWW2/ptttu64hLaLWxY8fq0KFDYa9bb71VgwYN0ujRo1usO3369LB6Dz/88Dlqddvcf//9Ye298847Wyw/e/Zs/fnPf9bLL7+sN998UwcPHtT3vve9c9Ta1tu5c6eCwaCeeOIJbd++XY8++qhKS0t1zz33nLVutN7DFStWqKCgQMXFxdqyZYtGjBihnJwcHT58uMnyGzZs0KRJkzRt2jRt3bpVEydO1MSJE/Xuu++e45af3Ztvvqn8/Hz94x//0Nq1a3Xq1CmNGzdONTU1LdZLSkoKu1f79u07Ry1um8svvzysvevXr2+2bCzdP0l6++23w65t7dq1kqQbbrih2TrRfP9qamo0YsQILVmypMn3H374Yf3mN79RaWmpNm7cqC984QvKycnRiRMnmj1npN/hNjEO8cwzz5jk5ORGx1etWmXi4+ONz+cLHfvd735nkpKSzMmTJ5s817///W8jybz99tuhY3/9619NXFyc+fDDD9u97W1VW1tr+vXrZ+6///4Wy1199dVm1qxZ56ZR7WDgwIHm0UcfbXX548ePm27dupmXX345dGzHjh1GkikvL++AFravhx9+2AwaNKjFMtF8D8eMGWPy8/NDP9fV1Zm0tDRTUlLSZPkf/vCHZvz48WHHPB6Puf322zu0ne3h8OHDRpJ58803my3T3N9F0aq4uNiMGDGi1eVj+f4ZY8ysWbPMhRdeaILBYJPvx9L9k2ReffXV0M/BYNCkpqaaX/3qV6Fjx48fNy6Xy/z+979v9jyRfofbotP3jJxNeXm5hg0bJrfbHTqWk5OjQCCg7du3N1und+/eYb0NXq9X8fHx2rhxY4e3ubX+9Kc/6aOPPlJeXt5Zyz7//PNKSUnR0KFDVVhYqE8++eQctLDtFixYoPPOO08jR47Ur371qxaH1SoqKnTq1Cl5vd7QsSFDhuj8889XeXn5uWju51JVVaW+ffuetVw03sPa2lpVVFSE/dnHx8fL6/U2+2dfXl4eVl6q/07Gyr2SdNb79fHHH2vgwIHKyMjQ9ddf3+zfNdHiP//5j9LS0nTBBRdo8uTJqqysbLZsLN+/2tpaPffcc7rllltafChrrN2/Bnv27JHP5wu7P8nJyfJ4PM3en7Z8h9siJh6U15F8Pl9YEJEU+tnn8zVbp3///mHHunbtqr59+zZbx4alS5cqJyfnrA8ZvOmmmzRw4EClpaXpnXfe0Zw5c7Rr1y794Q9/OEctjcxPfvITfelLX1Lfvn21YcMGFRYW6tChQ1q0aFGT5X0+nxISEhrNGXK73VF1v5qye/duPfbYY1q4cGGL5aL1Hh49elR1dXVNfsd27tzZZJ3mvpPRfq+CwaDuuusuXXnllRo6dGiz5QYPHqxly5Zp+PDhqqqq0sKFCzV27Fht3769Qx8I2lYej0fLly/X4MGDdejQIc2fP19XXXWV3n33XfXq1atR+Vi9f5L02muv6fjx4/rRj37UbJlYu3//reEeRHJ/2vIdbouYDCNz587VL3/5yxbL7Nix46yTrGJFW673wIEDWrNmjV566aWznv+/57oMGzZMAwYM0LXXXqv3339fF154YdsbHoFIrrGgoCB0bPjw4UpISNDtt9+ukpKSqN2uuS338MMPP9Q3v/lN3XDDDZo+fXqLdaPhHjpdfn6+3n333RbnU0hSdna2srOzQz+PHTtWl156qZ544gk98MADHd3MiF133XWh/x4+fLg8Ho8GDhyol156SdOmTbPYsva3dOlSXXfddUpLS2u2TKzdv1gRk2Hk7rvvbjG5StIFF1zQqnOlpqY2mhXcsMoiNTW12Tqfnbhz+vRpHTt2rNk6n0dbrveZZ57Reeedp+985zsRf57H45FU/6/yc/WL7PPcU4/Ho9OnT2vv3r0aPHhwo/dTU1NVW1ur48ePh/WO+P3+DrlfTYn0+g4ePKivfe1rGjt2rJ588smIP8/GPWxKSkqKunTp0mjlUkt/9qmpqRGVjwYzZ84MTWSP9F/H3bp108iRI7V79+4Oal376t27ty655JJm2xuL90+S9u3bp9dffz3i3sRYun8N98Dv92vAgAGh436/X1lZWU3Wact3uE3abfZJlDvbBFa/3x869sQTT5ikpCRz4sSJJs/VMIF18+bNoWNr1qyJmgmswWDQDBo0yNx9991tqr9+/Xojyfzzn/9s55Z1jOeee87Ex8ebY8eONfl+wwTWV155JXRs586dUTuB9cCBA+biiy82N954ozl9+nSbzhFN93DMmDFm5syZoZ/r6upMenp6ixNYv/3tb4cdy87OjsoJkMFg0OTn55u0tDTz3nvvtekcp0+fNoMHDzazZ89u59Z1jOrqatOnTx/z61//usn3Y+n+/bfi4mKTmppqTp06FVG9aL5/amYC68KFC0PHqqqqWjWBNZLvcJva2m5nilL79u0zW7duNfPnzzc9e/Y0W7duNVu3bjXV1dXGmPr/kYYOHWrGjRtntm3bZlavXm369etnCgsLQ+fYuHGjGTx4sDlw4EDo2De/+U0zcuRIs3HjRrN+/Xpz8cUXm0mTJp3z62vK66+/biSZHTt2NHrvwIEDZvDgwWbjxo3GGGN2795t7r//frN582azZ88e88c//tFccMEF5qtf/eq5bnarbNiwwTz66KNm27Zt5v333zfPPfec6devn5kyZUqozGev0RhjZsyYYc4//3zzt7/9zWzevNlkZ2eb7OxsG5fQogMHDpiLLrrIXHvttebAgQPm0KFDodd/l4mle/jiiy8al8tlli9fbv7973+b2267zfTu3Tu0gu3mm282c+fODZX/+9//brp27WoWLlxoduzYYYqLi023bt3Mv/71L1uX0Kw77rjDJCcnm3Xr1oXdq08++SRU5rPXN3/+fLNmzRrz/vvvm4qKCnPjjTeaxMREs337dhuXcFZ33323WbdundmzZ4/5+9//brxer0lJSTGHDx82xsT2/WtQV1dnzj//fDNnzpxG78Xa/auurg79npNkFi1aZLZu3Wr27dtnjDFmwYIFpnfv3uaPf/yjeeedd8z1119vBg0aZD799NPQOb7+9a+bxx57LPTz2b7D7aHTh5GpU6caSY1eb7zxRqjM3r17zXXXXWe6d+9uUlJSzN133x2Wjt944w0jyezZsyd07KOPPjKTJk0yPXv2NElJSSYvLy8UcGybNGmSGTt2bJPv7dmzJ+z6KysrzVe/+lXTt29f43K5zEUXXWR+9rOfmaqqqnPY4tarqKgwHo/HJCcnm8TERHPppZeahx56KKwX67PXaIwxn376qfnxj39s+vTpY3r06GG++93vhv2CjxbPPPNMk/+//ncnZizew8cee8ycf/75JiEhwYwZM8b84x//CL139dVXm6lTp4aVf+mll8wll1xiEhISzOWXX25Wrlx5jlvcOs3dq2eeeSZU5rPXd9ddd4X+LNxut/nWt75ltmzZcu4b30q5ublmwIABJiEhwaSnp5vc3Fyze/fu0PuxfP8arFmzxkgyu3btavRerN2/ht9Xn301XEMwGDTz5s0zbrfbuFwuc+211za67oEDB5ri4uKwYy19h9tDnDHGtN+gDwAAQGQcv88IAACwizACAACsIowAAACrCCMAAMAqwggAALCKMAIAAKwijAAAAKsIIwAAwCrCCAAAsIowAgAArCKMAAAAqwgjAADAqv8P6ENj0OHBE6UAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, x > 0,linewidth=1,clip_on=False);\n",
    "plt.hlines(xmin=-10,xmax=0,y=0,linewidth=3,color='b')\n",
    "plt.hlines(xmin=0,xmax=10,y=1,linewidth=3,color='b');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Activation Functions: Sigmoid (Logistic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAABpVElEQVR4nO3dd3yT5doH8F92996LMsoehQKlyNQqIqIcPR5eHCAOHKAC4kAFjhNUBBwo6nGdox4HR3GDiOy9EQqFUtpSSveeSZP7/SNt0pJ0pE2apP19P+bT+7mfdYXY9uq9HokQQoCIiIjITqT2DoCIiIi6NiYjREREZFdMRoiIiMiumIwQERGRXTEZISIiIrtiMkJERER2xWSEiIiI7IrJCBEREdmV3N4BtIZOp0NWVhY8PT0hkUjsHQ4RERG1ghACZWVlCAsLg1TadPuHUyQjWVlZiIyMtHcYRERE1AYXL15EREREk/udIhnx9PQEoH8zXl5edo6GiIiIWqO0tBSRkZGG3+NNcYpkpL5rxsvLi8kIERGRk2lpiAUHsBIREZFdMRkhIiIiu2IyQkRERHbFZISIiIjsiskIERER2RWTESIiIrIrJiNERERkV0xGiIiIyK6YjBAREZFdWZyM7NixA1OnTkVYWBgkEgk2bNjQ4jnbtm3DsGHDoFKp0KtXL3z66adtCJWIiIg6I4uXg6+oqMCQIUNwzz334JZbbmnx+AsXLmDKlCl48MEH8cUXX2DLli247777EBoaikmTJrUpaCIiImsQQkAnAK1OQCcEtDoBrRDQavVfdfXbOgEhAJ3QfxV15+q/AkDDekDUb9edgyvrWzq/Li4BfaVoFPMV76HB3ob7rjgMosHORvsabEilEozvHWjBv6B1WJyMTJ48GZMnT2718evWrUP37t3xxhtvAAD69euHXbt2YfXq1UxGiIi6qGqNFqVVGlSqtahUa1GlqUWVWodKdS2qNHV1am1dWb9Po9VBXVv3tUFZoxVQ1+rrNFfUa7Q6Y4KhMyYXOh1Qq9NBd+Vv7C7OTSlD0gvXd/h9bf6gvL179yIxMbFR3aRJkzB//vwmz6mpqUFNTY1hu7S01FbhERE5Dp0WqC4BKguAykJAXQaoKwFNJaAub1CuAGprAJ0G0Na96su6WkCrblDWAEJb9+e2Dvo/s3V1f4I33G64v7ljTX9766D/5a4TxtYE/V/2dX/dm/lafxVXCLi28p+n+UetNTyumQxDUvcyDFJgNlLvK+3VeA8z7HJvmycj2dnZCA4OblQXHByM0tJSVFVVwdXV9H/D5cuX4/nnn7d1aEREHUcIoDwXyDsDFGcAJReBkkx9uSxbn4BUFcEZfzlKYeEAxNZmFdShPFBlt3vbPBlpi8WLF2PhwoWG7dLSUkRGRtoxIiIiC5VmARn7gIv7gcvH9UlIVZEdA5IAEgkgkdaVpcZtiRQCgA5SaAX0Lx2gERJ91wb0+wUk0FmQSYhWHtva4ywhhCVxdj46AFVSoEwqQYUUqJRKUNnE1yopUCmRQFZeDVTYJ16bJyMhISHIyclpVJeTkwMvLy+zrSIAoFKpoFKpbB0aEZH1aKqBtJ1A8m9Ayh9AcXrrz1V6AO4BgKsf4OYPuPnpyy5egMINULobv9aX5S6ATKF/SRWATF73VQFI5Q3qFYBU1jhUrQ5nc8rwV2YJTlwqwV+ZJTiTXQqNtjP+Wu4EJLWQyMohkZfXfa2ARFYJiawKEmmVsSzTlyGtK0ss+zzVNeGQtXyYTdg8GUlISMCvv/7aqG7z5s1ISEiw9a2JiGxLCCDzEHD0P8DJ7/RjPJriGQoE9gWC+gF+PQDvSMAnEvCOAFy8bRqmRqvDicwS7EstwL7UAhxKK0KVRmvTe9qCUiaFQiaBQi6tK0uhrC/LJfrtujpF3bFymRQyiQRyqQRSqQQySd1XKSCTSCCTSiGTwrBPJpVA2vD4BudIJYBUItE3KAGARKIfgiIBJDDW12/X/Wc8p8FxQghU6UpRqs5HaW0+SjUFKK8tQoWmCOW1xfqXphhltUWo1nZMc8WEfl6Y1Xt4h9zrShYnI+Xl5UhJSTFsX7hwAceOHYOfnx+ioqKwePFiXLp0Cf/+978BAA8++CDeeecdPPnkk7jnnnvw559/4ptvvsEvv/xivXdBRNSRdFogaQOwcxWQc9J0v0wJRIwAIuOBqAQgYri+taMD5ZXVYMvpHPxxOgd7zhegUt3xyYdEAni5KODtqn95ucr1X13qtxVwV8rgppTDVSmDm1JW91UOV0XDbRlc5DJIpc4z2KRSU4mLZReRWZ6JnIoc5FTmIKciB9kV2cipyEFuZS7UOrVV7ymBBB5KD3grveGl8oK30hveKm94KD3gqfCEu8IdHkoP/VdFg69K/VcvpRdc5C5Wjam1LE5GDh06hIkTJxq268d2zJo1C59++ikuX76MjIwMw/7u3bvjl19+wYIFC/Dmm28iIiIC//rXvzitl4icjxD6JOTPl4GCc433KT2AflOBvlOAHhMBlUeHh3exsBI/ncjC5qQcHLtYbG7ii1Uo5VJE+Lgi3NcVQZ4uCPRUIdBThQAPJQI9VQjyVCHAQwUvF4VTJRCWEEIgrypPn3CUZRoSj/rtwurCdt/DQ+EBf1d/+Ln4wd+l7qurP3xdfOGj8oGX0gveKm9D8uGh8IBMaq+OlvaRCGGr/12tp7S0FN7e3igpKYGXl5e9wyGirqjgPPDL40Dq1sb14XHAiPuA/jfrx3N0sJJKDX756zK+P5qJg2nWGyDrppShZ6AHYoI80CPQHZF+bojwdUOkrysCPFSdNsm4klanRVZ5FlJLUnG+5DxSi1NxoeQCUktSUa4pb9M1PZWeCHEPQbBbsP7lrv8a4BpgSDr8XP2gkjn/2MnW/v52yNk0REQOQ6cD9r0LbHlev35HvW5jgPFPAN3H6/sjOtiprBL8e086Nhy7hJpaXbuuFeHrisER3hgU7oN+oZ6ICfZEqJdLl0k46pXUlCC5MBlnCs/gTOEZnC06i7TSNNRoa1o+uYEg1yBEeEYYXmHuYYaEI9gtGG4KNxu9A+fFZISIqClVxcB3c4Bzm4x13pHA5FeBPjd0eBKi0wn8npSDj3altrkVRCmTIjbKB6N6+COumy8GhXvDz11p5UgdX3F1MU7kn8CpglM4U6BPPrIqslp9frhHOLp7d0c3r26I9IxEhEcEIj0jEeYRZrdxF86MyQgRkTlFacAX/wDyk411CfOAic90eHeMVifw61+X8c6fKUjOaWbGThP6h3rh6r5BGN3LH8OifOGicM5xBW1Vq6tFSnEKTuSdwPG84ziRdwJppWktnieTyBDlFYWe3j3R3bs7evj0QE/vnoj2joarvLXrxlJrMBkhIrpSThLw75uAijz9tqsfcMuHQExi8+dZmRACm07l4PVNZ3A+r/XTO2VSCUb39Me1/YNxdd8gRPh2rW4BjVaDkwUnceDyARzMPogT+SdQVdv86qJucjf08euDPr590M+/H/r69UUvn15Qyrpeq5E9MBkhImooJwn4bCpQma/f9o8B7vhGvzZIBzp5qQQv/pyE/RdaPytjSIQ3/jY0HDcOCUOAh/MPfmwtrU6LpIIk7M/ej4PZB3E092izyYdCqkA//34YHDAYgwMHo59fP0R5RUEqsWhRe7IiJiNERPWK0oB/32xMRMLjgDv/B7j6dlgIxZVqLP/1DL45fLFVU3PdlTLcGheBu0Z1Q0ywp+0DdBDF1cXYnbUbOy/txO5Lu1FcU9zkscFuwYgNisXggMEYEjQE/fz6scXDwTAZISIC9INVv/gHUJGr3w6PA+78DnD16bAQNp3KxnMbTiKvrOXZGxG+rrhvTHfcGhcBTxdFB0RnX0IInCs+h20Xt2Fn5k6cyD8BnTA/iyjINQgjQ0diZMhIjAgZgQjPiI4NlizGZISISKcD1s82Dlb1jwHuWN9hiUhBeQ2W/XgKP5+43OKx0f5umDuxF6YNDYdC1rm7FYQQSC5Kxu9pv2Nz+uYmB526K9wxOmw0RoWOwsiQkejm1Q0SO0y3prZjMkJEtOsN4Pyf+rJbAHDHtx22fPuelHw89vWxFltDgjxVWDSpD24ZGg55J05ChBA4XXgam9I2YXP6Zlwsu2j2uJ7ePTE2YizGho/F0KChUMg6f+tQZ8ZkhIi6tvQ9wNZX9GWJFLjtE8Cvu81vq9UJvLnlHN7+81yzY0NUcikeGNcDD4zvCXdV5/2RnVuZi59Tf8ZP539CSnGKyX4JJIgLjkNit0RMiJyAcI9wO0RJttJ5/88mImqJuhLY8BBQP/Zg/FNA93E2v21JpQbz/nsEO8/lN3vchD6BePlvgxDu0znXtKiurcaWjC348fyP2Hd5n8kYEKlEihEhI3Bdt+twddTVCHANsFOkZGtMRoio69r6sn4GDaB/uu64J2x+y9S8ctz32SGk5je9boiXixzLpg7ALcPCO+XYh4zSDHyT/A02nN+AkpoSk/3Dgobhxp434pqoa+Dn0rFPOyb7YDJCRF3TpSP6Z84AgEwF3PQOYOMnnu5PLcD9/z6E0uraJo+Z2CcQr946GEFenWtJ8VpdLbZnbsc3yd9gT9Yek/3hHuGY2nMqbupxEyK9Iu0QIdkTkxEi6nqEADY+beyembgYCOhl01tuTsrBvC+PNPlQO7lUgiev74P7xvToVA+oq9RU4n/n/of/JP0HlysazxZSSBWYFD0Jt8bcimHBw7joWBfGZISIup6kDcDF/fpyQG/9M2ds6H+HM/Hk/05AqzM/UjXIU4X37oxDXLeOW1zN1vKr8vHF6S/wdfLXKFM3fp5OuEc4/tHnH5jWaxq7YQgAkxEi6mpqa4DNy4zb174I2HBa6FcHMvD0d381uX9whDc+uGs4Qrw7R7fMxdKL+OjkR/jx/I/Q6DSN9o0NH4vb+92O0WGj2QpCjTAZIaKu5ci/geJ0fbn7eKD3JJvdav3hTCz+vulEZMqgUKy8bQhclc7/FN3Mskx8cOID/Hj+R2iF1lAvl8oxpfsU3D3gbvTytW1XGDkvJiNE1HXUqoFda4zb1z4P2Gi2yg/HLuGJ9cebXEPkzlFReP6mgZA5+fiQy+WX8cFfH2DDuQ2oFcaBuR4KD9zW5zbc0fcOBLsH2zFCcgZMRoio6zj+X6A0U1+OuQ4IG2qT22xLzsXCb5pORB69JgYLEmOcetpuUXUR1h1fh2/OfoNanTEJ8VR4YuaAmbij3x3wVHadB/dR+zAZIaKuQVsL7Fpl3B73pE1uc/JSCR7+4kiTg1WfmNQHcyc6b3eFWqvGf8/8F+8ffx9lGuPAVHeFO+7qfxfu6n8XvJRedoyQnBGTESLqGs7+ZlzgrMcEIHKE1W9xsbASd39yEJVqrdn98xNjnDYREUJgc/pmrD68GpnlmYZ6V7kr7ux3J2YNmAVvlbcdIyRnxmSEiLqG/e8by6Mfsfrly2tqce9nB5Ffbv6Bdw9N6InHromx+n07QnJhMl7Z/wqO5B4x1EkgwbRe0zBv6DwEuQXZMTrqDJiMEFHnl3MKSNupL/v3AnpcbdXLCyGw6JvjOJtTbnb/bXEReHJSH6cbI1KhqcDaY2vx5ekvG82QiQ+Jx6IRi9DXr68do6POhMkIEXV+Bz4wlkfOAaTWXePi3W3nsfFUttl943oH4pVbBjlVIiKEwB8Zf2DFgRXIrcw11Ed7RWPR8EUYFzHOqd4POT4mI0TUudWUAye+1ZeVnsCQGVa9/LbkXKz8Pdnsvv6hXnj3jmFQyJxnga/cyly8sPcFbM/cbqhTyVSYM3gO7h5wN5QypR2jo86KyQgRdW6nfwQ0dU/IHXQr4GK9mR65pdV4vIkpvH7uSnwwMw4eKuf4MSuEwI/nf8SrB19ttHz7mPAxeCb+GUR68uF1ZDvO8V1CRNRWx740lmPvtNpldTqBhd8cR0GF2mSfTCrBO7cPRYSvm9XuZ0vmWkP8XfzxTPwzuLbbteySIZtjMkJEnVdRWoOBqzFAxHCrXfqDnanYlZJvdt/iyX0xumeA1e5lSxvTNuKFvS80ag2Z0mMKFo9czKm61GGYjBBR53X8K2M59narLf3+V2YJVm4yP05k0oBg3Dumu1XuY0uVmkqsOLAC36d8b6jzd/HH0oSluDrKurONiFrCZISIOichgL/qBq5CAgz5P6tcVl2rwxPrj6PWzAqrod4uePXWwQ7frZFUkISndjyFtNI0Q93k6Ml4dtSzbA0hu2AyQkSdU24SUJCiL3e7CvAKs8pl39magjPZZSb1UgmwZnosfNwcd7aJEAKfn/4cqw6vMjxPxlXuimfjn8VNPW9y+CSKOi8mI0TUOZ3aYCwPmGadS2aV4N2tKWb3zZvYC/E9/K1yH1uo0FRg6e6l+D39d0PdAP8BeHXcq+jm1c2OkRExGSGizkgIIGlD3YYE6De13Zes1erwxLcnzHbP9A3xxLyrHXep99TiVCzYtgCpJamGutkDZuORoY9AIVPYMTIiPSYjRNT55J0B8s/qy1EJgGdIuy/5773pSLpcalIvk0qw8rYhUModc2Gz39N+x5LdS1BZWwkA8FB44JUxr2Bi1EQ7R0ZkxGSEiDqfpB+N5f43t/tyuaXVWLX5rNl9D43viYHhjjfoUyd0eOfoO/jwrw8NdTG+MVg9YTW7ZcjhMBkhos7n7EZjud+N7b7cK7+eRnlNrUl9ryAPPHJNr3Zf39qqaqvw7K5nsTl9s6FuSo8pWDpqKdwUzrEQG3UtTEaIqHMpzwWy6h51HzwQ8I5o1+X2pRZgw7Ess/tevHkgVHJZu65vbTkVOXh066NIKkgCAEglUiwavgh39ruTs2XIYTEZIaLO5ZyxNQAx17XrUlqdwD9/PGV2382xYUjo6VizZ5IKkvDIlkeQW6V/0q67wh2vj3sdYyPG2jkyouYxGSGizuXcJmO596R2Xep/hzPNrinioZLj2Rv6teva1rb70m4s2LYAVbVVAIBwj3C8ffXbiPF13Fk+RPUcc/g3EVFbaDXA+a36sqsvEDGizZeqVNdi5e/ml3xfcG1vBHm5tPna1vbT+Z8wb8s8QyISGxiLL274gokIOQ22jBBR55GxD6ipm37bKxGQtn08x792XkBuWY1JfY9Ad8xMcJzZKJ+d+gwrD600bCdGJWLFuBVQyVR2jIrIMkxGiKjzSN1qLPe6ts2XyS2rxrrt583ue/r6vlDI7N+orBM6rD68Gp+e+tRQN73PdCweuRiydiRhRPbAZISIOo8LO4zlHuPbfJm3t6SgUq01qR/Z3Q/X9g9u83WtRavT4oV9L+C7c98Z6ubGzsUDgx/gjBlySkxGiKhzqC4FLtVN6Q3s2+ZVVy8VV+Grgxlm9z17Qz+7/7Kv1dViye4l+Dn1ZwD6qbvPjXoOt/W+za5xEbUHkxEi6hzSdwOirjWje9tbRdZuTYFGa/r8malDwjAk0qfN17UGjU6DxTsXY1OafsaQXCLHinErMCm6fbOGiOyNyQgRdQ6p243l7uPadImLhZX45uBFk3qZVIKF1/Zua2RWodaqsWj7Imy9qB8XI5fK8cb4N3B11NV2jYvIGpiMEFHncKEuGZFIgegxbbrEO3+mmH0q79+GhqN7gHt7omsXtVaNBdsWYEemfkyMUqrEmolruJgZdRpMRojI+ZXnAbn65c8RGgu4+lh8iYyCSqw/kmlSL5NK8OjV9luvQ6PT4MkdTxoSEReZC96+5m2MCh1lt5iIrI3JCBE5v4y9xnIbW0XW7TgPrZlWkdviIhDlb5+Hy2l1Wjy761lsydgCQJ+IvJv4LkaEtH0xNyJHZP/J8kRE7ZWxz1juNtri03NLq7H+kGmriEImwdyJ9nkqr07o8M+9/8RvF34DoO+aeevqt5iIUKfEZISInF/DlpHIeItP/3h3GtRanUn9LUMjEOnX8a0iQgi8sv8VbEjZAEA/a2bVhFVICEvo8FiIOgKTESJybuoK4PJxfTmwL+DmZ9HppdUafLEv3aReIgEeGN/DGhFa7K2jb+Hr5K8B6NcReW38axgf2fbpykSOjskIETm3zEPG9UWiLB/U+fm+dJTV1JrUTx4Ygh6BHu2NzmJfnP4C//rrXwAACSR46aqXcG23ti9tT+QMmIwQkXNrOF4kyrJujJpaLT7elWZ234Pje7YjqLbZeGEjXj3wqmH72fhnMbXn1A6Pg6ijMRkhIufWcLyIhS0jPx2/jPxy0yfzjukVgMERPu0MzDL7Lu/D4l2LIaCf0fPA4Acwve/0Do2ByF6YjBCR89JpgcyD+rJHCODTrdWnCiHwye4LZvd1dKvI6YLTmL91Pmp1+u6iW2NuxdzYuR0aA5E9tSkZWbt2LaKjo+Hi4oL4+HgcOHCg2ePXrFmDPn36wNXVFZGRkViwYAGqq6vbFDARkUH+WUBdri9HjtCPOm2lg2lFOJVValLfN8QTV/Xyt1aELcoqz8JDfzyECk0FAGBC5AQ8N+o5uz+Qj6gjWZyMfP3111i4cCGWLVuGI0eOYMiQIZg0aRJyc3PNHv/ll1/i6aefxrJly3D69Gl89NFH+Prrr/HMM8+0O3gi6uIuHTaWw+MsOrWpVpHZV0V3WCJQri7HvD/noaC6AAAQGxiL18a9BrmU61FS12JxMrJq1Srcf//9mD17Nvr3749169bBzc0NH3/8sdnj9+zZg6uuugq33347oqOjcd1112HGjBkttqYQEbWoUTIyvNWnZRZVYtOpbJN6XzcFbo4Nt0ZkLarV1eLJHU/iXNE5AECUZxTevvptuMpdO+T+RI7EomRErVbj8OHDSExMNF5AKkViYiL27t1r9pzRo0fj8OHDhuQjNTUVv/76K2644YZ2hE1EBP20XgCABAiLbfVp/9mbDjMrv+P2+Ci4KGRWCa0lbxx6Azsv7QQAeCm9sPaatfBx8emQexM5GovaAvPz86HVahEcHNyoPjg4GGfOnDF7zu233478/HyMGTMGQgjU1tbiwQcfbLabpqamBjU1xhHupaWm/bpE1MVpqoCcU/pyYF9A5dmq06o1Wnx96KJJvVwqwV2joq0YYNO+PvM1Pj/9uf6+EjlWT1iNaO+OuTeRI7L5bJpt27bhlVdewbvvvosjR47gu+++wy+//IIXX3yxyXOWL18Ob29vwysyMtLWYRKRs7l8wrjYWUTrx4tsPJmN4kqNSf3kQaEI8XaxVnRN2pO1B8sPLDdsL0lYgpGhI21+XyJHZlHLSEBAAGQyGXJychrV5+TkICQkxOw5S5YswV133YX77rsPADBo0CBUVFRgzpw5ePbZZyGVmuZDixcvxsKFCw3bpaWlTEiIqLE2Dl79Yr/p0u8AMDOh9dOC2+pi2UU8sf0JaOuSqNkDZuOWmFtsfl8iR2dRy4hSqURcXBy2bNliqNPpdNiyZQsSEsyvfFhZWWmScMhk+j5ZIcx02gJQqVTw8vJq9CIiaqQNycjZnDIcTCsyqY8J8sDwbr7WisysSk0lHtv6GErV+m7nCZETMD9uvk3vSeQsLJ4/tnDhQsyaNQvDhw/HyJEjsWbNGlRUVGD27NkAgJkzZyI8PBzLl+ubIadOnYpVq1Zh6NChiI+PR0pKCpYsWYKpU6cakhIiIotlHdV/lamAoP6tOuXL/Rlm6++Ij7LpdF4hBJbtWWaYORPtFY3lY5ZDKuG6k0RAG5KR6dOnIy8vD0uXLkV2djZiY2OxceNGw6DWjIyMRi0hzz2nX7znueeew6VLlxAYGIipU6fi5Zdftt67IKKupboUKDyvLwcPAGSKFk+pUmvxvyOZJvUuCin+NizC2hE28umpT7ExbSMAwF3hjjevfhMeyo5/CB+Ro5KIpvpKHEhpaSm8vb1RUlLCLhsiAtL3AJ9M1pfjZgNT17R4yvrDmVj07XGT+tviIvD6bUOsHKDRnqw9eOiPh6ATOgDAWxPfwsSoiTa7H5Ejae3vb7YREpHzuXzCWA4d3KpTvjUznRfQry1iK1nlWXhyx5OGROShIQ8xESEyg8kIETmfyw1aOEJbbtXIKKjE/guFJvV9QzwRG+ljxcCMNFoNntj+BEpqSgAA4yPG48EhD9rkXkTOjskIETmf7LqWEYkMCBrQ4uHmxooAwD+GR9ps4OrqI6txIl8fZ7hHOF4Z+woHrBI1gd8ZRORcNNVAXt2Kz4F9AEXzC5XpdALfHTVNRuRSCW6ODbNFhNiSvgX/SfoPAEAhVeCNCW/AS8nxbkRNYTJCRM4lNwnQ1erLreiiOZBWiIuFVSb1E/sGwd9DZe3ocLHsIpbsXmLYfnLEkxjg33LrDVFXxmSEiJxLdoPBqyEtD15df9h8F83f46w/nVetVWPR9kUo05QBAK6Pvh7T+0y3+n2IOhsmI0TkXCyYSVNRU4tf/7psUu/rpsDEPkHWjgyrDq9CUkESAKCbVzcsS1hm08XUiDoLJiNE5FxyTxvLwc13f2w8mY1Ktdak/ubYcCjl1v3xtyNzB744/QUAQClV4o3xb3BhM6JWYjJCRM5DCCD3lL7sGQa4Nv88mY7qosmvym80TuSJEU+gj18fq96DqDNjMkJEzqPsMlCtX7cDwc0/j+ZScRX2phaY1PcN8cSAMOvNbBFCYOnupSis1q9jMj5iPMeJEFmIyQgROY+cJGM5qF+zh/58PMts/d/jIqw6juPLM19i56WdAAB/F388P/p5jhMhshCTESJyHvVdNECLi539dMI0GZFKgJusuLbIuaJzWHVolWH7pTEvwd/V32rXJ+oqmIwQkfNoOHi1mZaR1LxynLxUalKf0NMfQZ7NL5LWWjXaGjy540modWoAwJ397sSY8DFWuTZRV8NkhIicR05dy4hEql99tQk/nzCdzgsAUwdbr1Vk9eHVSClOAQD09u2N+XHzrXZtoq6GyQgROQedFshL1pf9egAKV7OHCSHwo5nxIgqZBNcPDLFKKPsu7zNM41XJVHh17KtQyay/mitRV8FkhIicQ2EqoK3Rl4OanklzJrsMKbnlJvXjYgLh46Zsdxjl6nIs3b3UsL0gbgF6+fZq93WJujImI0TkHHIbzKRpZrGzn5qYRTN1iHW6aFYeWonLFfpuoJEhIzGj7wyrXJeoK2MyQkTOoRXTeoUQZmfRqORSJPYPbncIuy7twv/O/Q8A4CZ3wwtXvQCphD9GidqL30VE5BxaMa332MVis0/oTewXDA+VvF23L1WXYtmeZYbtRSMWIdwjvF3XJCI9JiNE5Bzqp/XKXQC/7mYP+el4E7NohoS2+/avHngVuZW5AIDRYaPx95i/t/uaRKTHZISIHJ+mSj+AFdBP6ZXKTA7R6QR+O2majHio5JjQzif0bs3Yih/P/6i/nsKDq6wSWRmTESJyfHnJgNDpy03MpDlxqQSXS6pN6q/rHwwXhWny0lrF1cV4Yd8Lhu0nRzyJEHfrTBEmIj0mI0Tk+OrXFwGaHLy68WS22frJg9rXRfPawdeQX5UPABgXMQ7Tek1r1/WIyBSTESJyfPlnjeUA05VXhRDYaKaLxk0pw9iYgDbfdvel3fgp9ScAgKfCE8sSlrF7hsgGmIwQkeNrlIzEmOxOzilDWkGlSf3EPkFt7qKp1FTixX0vGrYXjViEILf2jT0hIvOYjBCR48s/p/8qUwI+3Ux2N9VF057l39ceW4tL5ZcA6Bc3+1uvv7X5WkTUPCYjROTYtLVA4Xl92a8nIDNdL8RcMqKUSTGxb9taMk7mn8Tnpz8HoH/2zNKEpeyeIbIhJiNE5NiK0wGtWl8200WTll+BM9llJvVjYwLatNCZRqfBsj3LoKubvfPQkIfQzcu0NYaIrIfJCBE5tobjRQJNB69uPGW+i2ZSG7toPjv1Gc4W6e/Z168vZg6Y2abrEFHrMRkhIsfWaPBqb5Pd5rpoZFIJru1n+bNo0krS8N6x9wAAUokU/xz9TyikCouvQ0SWYTJCRI6tmZk0l0uqcOxisckpo3r4wdddadFthBB4fu/zUOv0XUJ39bsLA/ybfjowEVkPkxEicmz1M2kAwL9xMrI5KcfsKdcPsLyL5ufUn3Eo5xAAINwjHA/HPmzxNYiobZiMEJHjEsK4+qpXOKDyaLT7j9O5Zk+7zsJkpKSmBCsPrTRsPzfqObgp3CyLlYjajMkIETmuinygulhfvmK8SHlNLfadLzA5ZUikD4K9XCy6zdtH30ZhdSEA4Npu12JM+Jg2hUtEbcNkhIgcVzODV3edy4NaqzM5JdHCtUVO5p/EN8nfAABc5a54csSTlsdJRO3CZISIHFczg1eb6qK5xoJZNFqdFi/uexECAgAwN3Yun8hLZAdMRojIcTUcvNqgZUSrE9h6xjQZCfN2Qb9Qz1Zf/tuz3yKpIAkA0MunF27vd3vbYyWiNmMyQkSOq4lummMXi1FQoTY5/Jp+wa1etj2/Kh9vHXnLsL1k1BKuKUJkJ0xGiMhx1ScjKi/A09h9suW0+Sm91/Rr/XiRVYdWoUyjX0b+5p43Y1jwsLbHSUTtwmSEiByTphooztCX/XsBDVo8tpgZL+KmlGFUD/9WXfpg9kH8lPoTAMBL6YWFwxe2P14iajMmI0TkmIouAHUDS+Hfy1B9sbASyTnmH4znopC1eFmNVoOX971s2J4fNx9+Ln7tDpeI2o7JCBE5poLzxrJ/T0Ox6S6a1s2i+fLMlzhfor/2oIBBuDXm1rbHSERWwWSEiBxTYYNkxK9BMmJmFo1EAlzdivVF8qvy8d5x/YPwJJDg2VHPQirhj0Eie+N3IRE5poIUY9m/BwCgrFqDfammq67GRvogwEPV4iXfPPImKjQVAIBbYm7hg/CIHASTESJyTAWpxnJdy8jOc/nQaIXJoYmt6KL5K+8vbEjZAADwVHji0WGPWiVMImo/JiNE5Jjqu2ncAgBXHwDAH22c0qsTOqw4sMKw/XDswxy0SuRAmIwQkeNRVwBll/XlusGrOp3AtuQ8k0PDfVzRJ7j5VVd/Tv0ZJ/JPAAB6evfE9L7TrRsvEbULkxEicjyFpl00Jy6VoNDMqquJ/YKaXXW1QlOB1YdXG7afHPkkV1olcjBMRojI8ZgZvLot2fyD8Sa0MIvm/RPvI78qHwBwdeTVGB022joxEpHVMBkhIsfTaI0R/YJn5rpoVHIpEppZdTW9NB3/SfoPAEApVWLRiEXWjZOIrILJCBE5niu6aYoq1DieWWxy2Kge/s2uuvrawddQq6sFAMwaMAuRnpHWjpSIrIDJCBE5noYtI349sONcHoTpjF5M6BPY5CV2Zu7EjswdAIAgtyDcN+g+a0dJRFbCZISIHE/9tF6PEEDlge1mumgAYEIf8+NFNDoNXj/0umH78bjH4aZws3qYRGQdTEaIyLFUlwAVdcmHf0/odALbz5omI1F+boj2N59g/O/s/3Ch5AIAYEjgEEzuPtlm4RJR+zEZISLHckUXzcmsEhSYmdI7oU+g2Sm9ZeoyvHvsXcP2EyOeaHbqLxHZH5MRInIsDQev+vcyO4sGaHq8yId/fYiimiIAwOTukzEkcIjVQyQi62IyQkSOpdG03p5m1xdRyqVI6BFgUp9ZlonPkz7XHyNVYv6w+baKkoisqE3JyNq1axEdHQ0XFxfEx8fjwIEDzR5fXFyMuXPnIjQ0FCqVCr1798avv/7apoCJqJMrNCYjpW5ROHax2OSQ+O5+cFWaTuldc2QNNDoNAOCu/nchzCPMZmESkfXILT3h66+/xsKFC7Fu3TrEx8djzZo1mDRpEpKTkxEUZDqyXa1W49prr0VQUBDWr1+P8PBwpKenw8fHxxrxE1Fn06CbZmeBJ3RmpvSO723aRXMs9xg2pW0CAPi5+HEqL5ETsTgZWbVqFe6//37Mnj0bALBu3Tr88ssv+Pjjj/H000+bHP/xxx+jsLAQe/bsgUKhfx5EdHR0+6Imos6rUD8LBp6h+DOlzOwhV07pFULg9YPGqbxzY+fCQ+lhsxCJyLos6qZRq9U4fPgwEhMTjReQSpGYmIi9e/eaPefHH39EQkIC5s6di+DgYAwcOBCvvPIKtFptk/epqalBaWlpoxcRdQE1ZUCl/jkywjfa7JTeCF9X9Ax0b1S3MW1jo6fy3hJzi+1jJSKrsSgZyc/Ph1arRXBwcKP64OBgZGdnmz0nNTUV69evh1arxa+//oolS5bgjTfewEsvvdTkfZYvXw5vb2/DKzKSSzgTdQn1rSIAil0ikF9eY3LIlVN6a7Q1WHN4jWF70YhFkEstbvQlIjuy+WwanU6HoKAgfPDBB4iLi8P06dPx7LPPYt26dU2es3jxYpSUlBheFy9etHWYROQIitIMxWS16WwZAJjQu3EXzedJnyOrIgsAMDpsNMaEj7FZeERkGxb9+RAQEACZTIacnJxG9Tk5OQgJCTF7TmhoKBQKBWQy48j3fv36ITs7G2q1Gkql0uQclUoFlUplSWhE1BkUGVtGDhR7mexWyCRI6Gl8Sm9BVQE+/OtDAIBUIsXjwx+3fYxEZHUWtYwolUrExcVhy5YthjqdToctW7YgISHB7DlXXXUVUlJSoNPpDHVnz55FaGio2USEiLqwBt002/NNB6DGdfOFu8r4N9S7x95FhaYCAHBLzC3o7dvb9jESkdVZ3E2zcOFCfPjhh/jss89w+vRpPPTQQ6ioqDDMrpk5cyYWL15sOP6hhx5CYWEhHnvsMZw9exa//PILXnnlFcydO9d674KIOocGLSPna02XChgbY5zSm1KUgvXn1gMA3ORumBvLnylEzsriUV7Tp09HXl4eli5diuzsbMTGxmLjxo2GQa0ZGRmQSo05TmRkJDZt2oQFCxZg8ODBCA8Px2OPPYannnrKeu+CiDqHupaRaqk7imHaMjI2xjiOZNXhVdAJfYvrfYPuQ4Cr+TEmROT4JEIIM0sKOZbS0lJ4e3ujpKQEXl6m/chE1AloNcBLwYDQ4py0B66tbDzjztdNgUPPXQuZVIKD2Qdxz6Z7AAAh7iH4adpPcJG72CNqImpGa39/89k0ROQYijMAoV9/6KzGtJVjdK8AyKQSCCGw+vBqQ/282HlMRIicHJMRInIMDab1Zohgk91je+kTlM3pm/FX/l8AgBjfGNzY48YOCY+IbIfJCBE5hgaDV9PNJCNjYgKg0Wnw1tG3DHXzh82HTGr6wDwici5MRojIMRQ2nYz0CHBHhK8bvjv7HdJL0wEAw4OHY2z42A4NkYhsg8kIETmGBt00F0Xjab1jYgJQqanEe8ffM9QtiFvQaFl4InJeTEaIyDHUtYyohQxZwr/RrrExgfgs6TMUVBcAAK7tdi0GBw7u8BCJyDaYjBCR/QlhaBnJFIHQNfjRJJNK0DsM+PTkp/ptiQyPDn3UDkESka0wGSEi+yvPBeqWdb9yJs2wKB98kfwRKmsrAQB/7/13RHtHd3SERGRDTEaIyP4ajBdJv2K8yOBoLb5N/hYA4Cp3xYNDHuzIyIioAzAZISL7azCt98qWkTTdetSKWgDAzP4zuew7USfEZISI7K+Jab2eXpdxIO9PAICfix/uHnB3R0dGRB2AyQgR2V11boqhnGHophHwDv/dUD9n8Bx4KE0fnkdEzo/JCBHZXWWOaTIicz+HEpwGAER4ROAfvf9hl9iIyPaYjBCR3SlK9Kuq5ggfVEMFQAdV0G+G/Y8OexQKmcJO0RGRrTEZISK7EtWl8NQWATCOF5F7HYfM5TIAoL9/f0yKnmS3+IjI9piMEJFdZaSeNpZFMCCphSpwk6FuQdwCSCX8UUXUmfE7nIjs6tyZk4Zyui4ICp99kCqLAQCjw0ZjVOgoO0VGRB2FyQgR2VVBxhlDOVXiC2XAn4btBXEL7BESEXUwJiNEZDc1tVqIBmuM/OWbDalcv+z7lB5T0Nevr71CI6IOxGSEiOzmSHoxwkQ2ACBHJkOR3ykAgBRyzIudZ8/QiKgDMRkhIrvZlZKHbpIcAMBbPn6AVL/s+9Vh0xDhGWHP0IioAzEZISK72Xs2G+GSfKQq5PjJ0xUAIHQqPJ3AVhGiroTJCBHZRVGFGvlZqZBLdHjT1wdCoq+PlNyAYA9/+wZHRB2KyQgR2cWe8wWIkuTimEqJP93dAAA6jSdu6TnDzpERUUdjMkJEdrHzXB6iJNlY5edjqFPnJ2Jin0j7BUVEdsFkhIg6nBACO8/lQ+eRgqMuLgAAVY0X/MUY9Ax0t3N0RNTRmIwQUYdLK6jEpeJyHAzINNTJ8sZjXEwwJBKJHSMjIntgMkJEHW7nuTwovI8gV6kFAAyurkF+WTzGxATaOTIisgcmI0TU4badvQRl4GbD9owCCbSQ46qenEVD1BUxGSGiDqXR6nCg4EdIFaUAgAkVlfCsCsLAcC/4e6jsHB0R2QOTESLqULtT0wEf/cPwpELgsaISZIggjOnFLhqirorJCBF1qHXHP4REVg0AuLm8Ar00GqSLYIyLCbBzZERkL0xGiKjDZJVnIan8NwCATCfBw0UlAIDL0hDERfvaMzQisiMmI0TUYVYfegtCon8Y3qgST4Ro9bNpvMP7QCWX2TM0IrIjJiNE1CGSC5OxKf1XAIDQumJGcY1hX+++A+0VFhE5ACYjRNQhVh9eDQEBAKjJn4j+IhcAkCt8MKovl4An6sqYjBCRze27vA+7s3YDAHQaHyiKhiJQoh8vkiUNQZ9gT3uGR0R2xmSEiGxKJ3RYfXi1Ybsm9zpEociwrfHqxiXgibo4JiNEZFOb0jYhqSAJAKCtDkVtaSy6SXIM+z1CY+wVGhE5CCYjRGQzGq0Gbx15y7BdkzsZgBRRDZKR8B797RAZETkSJiNEZDPfnP0GmeV1T+atioG2Qt8K0rBlxCu0tz1CIyIHwmSEiGyiXF2O94+/b9iuyJ4EQD82JEqSazzQr3sHR0ZEjobJCBHZxMcnP0ZRjX6gag/Xq6CrjjDsq28ZqZV7AG58Ui9RV8dkhIisLrcyF/9J+g8AQC6VQ1c42bBPjlqES/IBAFL/7gBn0hB1eUxGiMjq3j32Lqq1+ofh/a3nbUjKUBj2hUkKIJfoAABSdtEQEZiMEJGVpRan4vuU7wEA7gp3DHC7BVqdMOxvOHiV40WICGAyQkRWtubIGuiEvuXjnoH34GiaptH+RsmIL5MRImIyQkRWdDT3KLZe3AoACHQNxJ397sTOc/mNjuFMGiK6EpMRIrIKIQTeOPSGYfvh2IeRVyqQXlDZ6Di2jBDRlZiMEJFV/JnxJ47nHQcAdPfujmm9ppm0igAwrr4qVQDeESb7iajrYTJCRO1Wq6vFmiNrDNvzh82HXCrHLpNkRBi7aXyiAKmsw2IkIsfFZISI2u37lO+RVpoGABgaNBQTIyeiVqvD7vONk5EAlMJdUqPf4HgRIqrDZISI2qVCU4G1R9cathfGLYREIsHxzBKUVdc2OjaK40WIyAwmI0TULh+f/BgF1QUAgGu7XYvYoFgAwM5zeSbHco0RIjKHyQgRtVl2RTb+ferfAPTLvi8YtsCwz9zg1R7yBtN62TJCRHWYjBBRm7199G3Dsu8z+s5ApFckAKC0WoNjF4tNjh/m0aCOLSNEVKdNycjatWsRHR0NFxcXxMfH48CBA60676uvvoJEIsG0adPaclsiciBJBUn48fyPAAAvpRceGPyAYd/e8wWNloCv10vRoOvGN9rWIRKRk7A4Gfn666+xcOFCLFu2DEeOHMGQIUMwadIk5ObmNnteWloaFi1ahLFjx7Y5WCJyDEIIrDy00rD9wOAH4K3yNmzvOGs6XgQA/NVZ+oJnKKBwtWmMROQ8LE5GVq1ahfvvvx+zZ89G//79sW7dOri5ueHjjz9u8hytVos77rgDzz//PHr06NGugInI/rZnbsfB7IMAgEjPSMzoO8OwTwiBbcmmyUi0hw7yqrpxJBwvQkQNWJSMqNVqHD58GImJicYLSKVITEzE3r17mzzvhRdeQFBQEO69995W3aempgalpaWNXkTkGDQ6TaNl3xfELYBCpjBsn8+rwKXiKpPzpnZTGzc4XoSIGrAoGcnPz4dWq0VwcHCj+uDgYGRnZ5s9Z9euXfjoo4/w4Ycftvo+y5cvh7e3t+EVGRlpSZhEZEPrz65vtMBZYlRio/3bks132Y4PKDNucLwIETVg09k0ZWVluOuuu/Dhhx8iICCg1ectXrwYJSUlhtfFixdtGCURtVaZugzvHXvPsL1o+CJIJJJGx2w3M15EKgEGuBQaK9hNQ0QNyC05OCAgADKZDDk5OY3qc3JyEBISYnL8+fPnkZaWhqlTpxrqdDqd/sZyOZKTk9GzZ0+T81QqFVQqlSWhEVEH+PCvD1FUUwQAmBw9GYMDBzfaX6muxf7UQpPzhkX5wrU8w1jBbhoiasCilhGlUom4uDhs2bLFUKfT6bBlyxYkJCSYHN+3b1/89ddfOHbsmOF10003YeLEiTh27Bi7X4icyKXyS/g86XMAgFKqxGNxj5kcs/d8AdRanUn9+N6BQOF5Y4UfB7ITkZFFLSMAsHDhQsyaNQvDhw/HyJEjsWbNGlRUVGD27NkAgJkzZyI8PBzLly+Hi4sLBg4c2Oh8Hx8fADCpJyLH9ubhN6HRaQAAd/S/A+Ee4SbHmOuiAYAJfYKA46n6DVdfwM3PZnESkfOxOBmZPn068vLysHTpUmRnZyM2NhYbN240DGrNyMiAVMqFXYk6k6O5R/Fb2m8AAB+VD+4fdL/JMU1N6Q3wUGJAoAIozdRX+Jl2zRJR12ZxMgIA8+bNw7x588zu27ZtW7Pnfvrpp225JRHZiU7osOLACsP23Ni58FR6mhx3Ib8CGYWVJvXjegdCWpxmrPBnMkJEjbEJg4ia9UPKD0gqSAIAxPjG4O+9/272OHOtIkBdF02j8SJMRoioMSYjRNSkcnU53jzypmH7qRFPQS4136C6rYkpvWN7BQAFKcZKtowQ0RWYjBBRkz7860MUVBcAAK6JugbxofFmj6vWaLE/tcCkfkikD3zdlUABZ9IQUdOYjBCRWRmlGfhP0n8AAAqpAo8Pf7zJY/emFqCm1nRK74TeQfpCYaqxki0jRHQFJiNEZNbKQysNU3lnDZiFSM+m1wXa3uR4kUB9ob5lxD0QcPE2eywRdV1MRojIxN6svdh6cSsAINA1EPcNuq/Z4809j8bfXYlB4d5ATTlQXvfsKg5eJSIzmIwQUSO1ulq8dvA1w/b8uPlwV7g3efyF/AqkFTQxpVcqaTyThl00RGQGkxEiauTbs98ipVg/+2VQwCDc2OPGZo/fcjrHbP343ld00QAcvEpEZjEZISKD4upirD221rD95IgnIZU0/2Niy2nTLhqppEEy0qhlpJdV4iSizoXJCBEZvHX0LZTUlAAApvSYgtig2GaPL6nS4GCa6VN6h3fz00/pBYACzqQhouYxGSEiAMDJ/JNYf3Y9AMBN7oaFcQtbPGfH2TzU6oRJ/dX9gowbfFovEbWAyQgRQavT4qV9L0FAn1g8HPswgtyCWjgL+POMaRcNACQ2TEbqV1/1DAWUTQ+EJaKui8kIEeG7lO9wquAUAKCXTy/c3u/2Fs+p1eqw1cyU3ig/N/QM9NBvVBUDlXUrs3JaLxE1gckIURdXVF3U6Pkzz8Q/A4VU0eJ5Ry8Wo7hSY1J/Tb8gSCQS/Qan9RJRKzAZIeri3jzyZqNBqyNCRrTqvD+amNJ7Td9g4wYHrxJRKzAZIerCjucdx//O/Q8A4K5wx+NxTT9/5kp/mpnS66GSY2R3P2NF/llj2T+mzXESUefGZISoi9LqtHh538uG7bmxcxHoFtiqczMKKnEut9ykflzvACjlDX6s5Ccby4F92hwrEXVuTEaIuqj1Z9fjdOFpAECMbwxm9J3R6nO3nGlFFw0A5J/Tf5UqAJ9ubYqTiDo/JiNEXVBhdSHePGoctPps/LOQS+WtPt/ceBGJpMFTegFApzVO6/XvCchaf30i6lqYjBB1QW8cegNl6jIAwNQeUxEXHNfqc4sr1diXarrq6tBIH/h7qBocmA5o1fpyAMeLEFHTmIwQdTH7Lu/Dj+d/BAB4KjyxcHjLK602tOV0LrRmVl29bkBI44r6LhoACOhtcZxE1HUwGSHqQqprq/Hi3hcN2/Pj5iPANcCia2w6lW22fpJJMtJgJk0AB68SUdOYjBB1IR+c+AAZZRkAgKFBQ/H33n+36PwqtRY7zuWZ1PcO9kD3gCuWes9rMJOG3TRE1AwmI0RdxLmic/jk5CcAALlUjmUJyyCVWPYjYPvZPFRrdCb1Jq0iwBXdNExGiKhpTEaIugCd0OGFvS+gVtQCAO4ZeA96+li+Iurvre2iAYzdNJ5hgMrT4nsRUdfBZISoC1h/dj2O5R0DAHTz6oY5g+dYfA2NVmd2Sm+4jysGhHk1rqwoAKrqZtwEcvAqETWPyQhRJ5dbmYvVh1cbtpeOWgqVTNXMGebtTy1EaXWtSf21/YOND8ar12jwKpMRImoekxGiTm7FgRUo1+iXbr+5580YGTqyTddp9SwaoPEy8ExGiKgFTEaIOrHf037H5vTNAABflS8WDV/UpuvodAK/J5kmI75uCoyI9jU9gYNXicgCTEaIOqmi6iK8vN/4ILynRz4NHxefNl3rWGYxckprTOoT+wVDLjPzY4TdNERkASYjRJ3U8gPLUVitH0Q6MXIiJnef3OZr/Xristl6s100gDEZUXoCnqFtvi8RdQ1MRog6oS0ZW/Dbhd8AAF5KLywZtcR0kGkr6XQCv/xlmoy4K2UYE2Nm9VZNFVCUri8HxOifoEdE1AwmI0SdTElNSaMl358e+TQC3QKbOaN5RzKKcLmk2qT+2v7BcFHITE/IPwug7tk1gVwGnohaxmSEqJNZcWAFCqoLAADjI8bjxh43tut6PzfRRXPj4DDzJ+QkGctB/dt1byLqGpiMEHUi2y5uw8+pPwMAPJWeWJqwtM3dMwCg1Qn8aqaLxtNFjrG9m3jAXu4pYzmYyQgRtYzJCFEnUVJTghf2vmDYfnLEkwhyC2rXNQ+mFSK3zHQWzXX9Q6CSm+miAYDc08YyW0aIqBWYjBB1Ei/vfxl5Vfon6o4JH4Obe97c7mv+0lQXzZBmZsjUd9O4+HAmDRG1CpMRok7g19RfDbNnPJWeWJawrF3dMwBQq9Xht5OmyYi3qwJjejXRRVNVBJRl6cvBAziThohahckIkZPLrsjGS/teMmwvGbUEIe5NrP9hgf0XCpFfrjapv35ACBTmFjoDruii6dfuGIioa2AyQuTEdEKHZ3c9izJNGQDghu43tGtxs4aanEXTbBdNg8GrHC9CRK3EZITIif0n6T84kH0AABDiHoJnRz1rleuqa3XYaKaLxt9diYQe/k2fyMGrRNQGTEaInNTZorN488ibAAAJJHj5qpfhpfSyyrW3JeeiqFJjUn/9wBDzz6Kpl9twjRF20xBR6zAZIXJCaq0aT+98GhqdPmGY2X8mRoaOtNr1vztyyWz9TUOaWOgMAIQwJiNe4YCrj9XiIaLOjckIkRN668hbOFd0DgAQ4xuDR4c9arVrF1eqseVMjkl9hK8rRkT7NX1iaRZQXaIvs4uGiCzAZITIyezI3IHPkj4DACikCqwYuwJKmdJq1//5xGVotMKk/pah4ZBKm5mqy5k0RNRGTEaInEhORQ6e3WUcpLogbgF6+/a26j2+O5Jptv5vwyKaP7HRMvADrBgREXV2TEaInEStrhZP7XwKxTXFAIAJkRNwZ787rXqPtPwKHMkoNqkfGuWD7gHuzZ/MB+QRURsxGSFyEu+feB+Hcw4DAILdgvHi6Bfbvcrqlb47an7g6i0ttYoAxjVGJDIgwLqtNUTUuTEZIXIC+y/vx/vH3wcAyCQyvDbuNfi4+Fj1HkIIfH/UtItGIZPgxkEtPGOmtgbIqxszEtgHULhYNTYi6tyYjBA5uPyqfDy982kI6AeVzo2di2HBw6x+n0PpRbhYWGVSf3XfIPi6tzBANjcJ0NXqyyGDrR4bEXVuTEaIHFitrhZP7ngS+VX5AIBRoaNw76B7bXKvbw5eNFvfqi6ayyeM5dAhVoqIiLoKJiNEDuytI2/hYPZBAECgayCWj10OqcT637Zl1Rqzz6LxcVNgYp+gli9w+bixHMqWESKyDJMRIge1OX0zPjn1CQBALpHjjQlvIMA1wCb3+uFYFqo0WpP6abHhUMpb8WMiu0HLSMggK0ZGRF0BkxEiB5Rakoolu5cYtheNWIShQUNtdr+vDmaYrf+/kZEtn6zTAtkn9WXf7oCLtxUjI6KugMkIkYOp1FRiwdYFqNBUAABu6H4Dbu97u83ud/JSCU5eKjWpj430Qd+QVjx4L/8cUFs38JVdNETUBkxGiByIEALP7X4OqSWpAIBePr2wLGGZ1dcTaei/B8y3isxoTasIcMV4EQ5eJSLLtSkZWbt2LaKjo+Hi4oL4+HgcOHCgyWM//PBDjB07Fr6+vvD19UViYmKzxxN1ZetOrMPm9M0AAA+FB1ZPWA03hZvN7leprsUPx7JM6t2VMtw4uJkn9DbUMBkJYTJCRJazOBn5+uuvsXDhQixbtgxHjhzBkCFDMGnSJOTm5po9ftu2bZgxYwa2bt2KvXv3IjIyEtdddx0uXTK/0iNRV7U5fTPePfYuAEACCVaMXYFo72ib3nPD0SyU19Sa1N8UGw53lbx1F8k6YiyHxVonMCLqUiRCCNPHczYjPj4eI0aMwDvvvAMA0Ol0iIyMxCOPPIKnn366xfO1Wi18fX3xzjvvYObMma26Z2lpKby9vVFSUgIvr1b0YRM5mdMFpzFr4yxU1Y29WBi3ELMHzrbpPYUQuH7NTiTnlJns+3HeVRgc4dPyRbS1wPII/ZgRn27A/BMtn0NEXUZrf39b1DKiVqtx+PBhJCYmGi8glSIxMRF79+5t1TUqKyuh0Wjg5+fX5DE1NTUoLS1t9CLqrPKr8vHo1kcNicjUHlNx94C7bX7f/RcKzSYiQyK8W5eIAPol4OsHr4bHWS84IupSLEpG8vPzodVqERwc3Kg+ODgY2dnZrbrGU089hbCwsEYJzZWWL18Ob29vwysyspUD6YicTI22BvO3zkd2hf77Z3DgYCwbbdsBq/U+25Nmtn5mQnTrL3LpsLEcMbxd8RBR19Whs2lWrFiBr776Ct9//z1cXJp+kNbixYtRUlJieF28aH6ZaiJnphM6PLvrWRzP0w8ADXYLxpsT34RKprL5vbOKq/B7Uo5Jvb+7ElMGt/BQvIYyDxnLbBkhojZq5Qg1vYCAAMhkMuTkNP4hlpOTg5CQkGbPXblyJVasWIE//vgDgwc3vxaBSqWCSmX7H8hE9rT68GpsStsEAHCVu+Ktq9+y2QqrV/pifzq0OtPhYv83MhIuClnrL3SpbvCqRMYH5BFRm1nUMqJUKhEXF4ctW7YY6nQ6HbZs2YKEhIQmz3vttdfw4osvYuPGjRg+nE25RF+c/gKfnvoUACCVSLFy/Er09+/fIfeuUmvx3wOmrY0yqQR3xHdr/YVqyvVjRgAguD+gtN0UZCLq3CxqGQGAhQsXYtasWRg+fDhGjhyJNWvWoKKiArNn60f+z5w5E+Hh4Vi+fDkA4NVXX8XSpUvx5ZdfIjo62jC2xMPDAx4eHlZ8K0TOYUv6Frx64FXD9nOjnsO4iHEddv/1RzJRWKE2qb+ufzDCfFxbf6HLxwGh05fZRUNE7WBxMjJ9+nTk5eVh6dKlyM7ORmxsLDZu3GgY1JqRkQGp1Njg8t5770GtVuPvf/97o+ssW7YM//znP9sXPZGTOZZ7DE/tfAoC+i6S+wfdj9t639Zh99fqBP61M9XsvrtHR1t2scwGixeGs8WTiNrO4mQEAObNm4d58+aZ3bdt27ZG22lpaW25BVGnc67oHOZumYsabQ0A4MYeN+KRoY90aAybk7KRXlBpUj8kwhsjuzc93d6sjH3GctSodkZGRF0Zn01D1AEull7EnM1zUKrWr5kTHxKPF0a/0CFTeOsJIfD+DvOtInPG9bQsFp3OmIy4+QP+vawQIRF1VUxGiGwspyIH92++H/lV+QCAgf4D8ebVb0IhU3RoHIfSi3A0o9ikPtLPFdcPbH42nIn8ZKC67lpRCUAHJlVE1PkwGSGyoaLqIjyw+QFcKtc/i6mXTy+8l/ge3BXuHR7Lu1tTzNbfN6YHZFILk4mMBisuR8a3IyoiIiYjRDZTri7HQ388hPMl5wEAER4ReP/a9+Hj4tPhsRy/WIytyXkm9T5uCtw2PMLyC2bsN5ajmp7WT0TUGkxGiGygTF2GB/54AKcKTgEAAl0D8eF1HyLILcgu8bz95zmz9TMTouGmbMM49vqWEbkLEDqkHZEREbVxNg0RNa1MXYYHNz+IE/n6J9j6qHzwwbUfIMKzDS0QVnDyUgn+OJ1rUu+hkuOeq6Itv2BpFlCcri+HDwfkyvYFSERdHltGiKzoykTEV+WLf133L/Tytd9sk7e2mG8VuXt0NHzc2pBIXNhpLHdjFw0RtR9bRoisxGwiMulf6O3b224xnbxUYvaBeO5KGe4d071tF72w3Vju3nErxxJR58VkhMgKiqqL8NAfDxnGiDhCIgIAr21KNls/c3Q0fN3b0CoiBJBal4zIXYCIke2IjohIj8kIUTtlV2Tjgc0PILVEv6CYoyQie1LyseOs6QwaV4UM97W1VaQwFSjN1JejRgEKl3ZESESkx2SEqB3SS9Mx5/c5yKrIAqCfNfPBtR/YdYwIoF9t9dWNZ8zuu2dMNPw9VG27cOo2Y7n7+LZdg4joCkxGiNoouTAZD2x+AAXVBQCASM9Iu86aaei3k9k4nlliUu/jpsAD43u2/cINx4v0YDJCRNbB2TREbXAw+yBmb5xtSERifGPw2fWfOUQioq7V4fUmxorMm9gLXi5tXIZepzXOpFF5A6GxbbsOEdEVmIwQWein8z9hzuY5KNOUAQCGBA7BJ5M+QaBboJ0j0/t0zwVcyK8wqQ/3ccWdo7q1/cKXjgBVhfpy97GAVNb2axERNcBuGqJWEkLg/RPvY+2xtYa6MeFj8Mb4N+CmcLNjZEa5pdV48w/z64osuLY3XBTtSCDObTKWe09q+3WIiK7AZISoFTRaDZ7f+zx+OP+Doe4fvf+BxfGLIZc6zrfRqxuTUaHWmtQPCPPC34aGt+/iZxskIzHXte9aREQNOM5PUSIHVVRdhEXbF+FA9gFD3cK4hbh7wN2QSCx82q0NHckowv+OZJrd98+bBlj+ZN6GSi8D2frF3BA6BPAMafu1iIiuwGSEqBlnCs/gsT8fM0zdVUqVeGXsK5gU7VjdFBqtDs99f9LsvptjwzAi2q99N0jZbCzHONZ7JyLnx2SEqAm/XfgNS3cvRbW2GgAQ4BqA1RNWIzYo1r6BmfGvnReQdLnUpN5NKcPiyf3af4OzHC9CRLbDZIToCrW6Wrx19C18cvITQ93ggMFYNWEVgt2D7RiZeRfyK7Dmj7Nm982d2Ash3u1cJVVdAaRs0ZfdAoCwoe27HhHRFZiMEDWQU5GDp3Y+hcM5hw11f+v1Nzw36jkoZW14louNCSGw+LsTqKnVmezrHeyB+8f2aP9Nzv0O1Fbpy/2mckovEVkdkxGiOjszd+LZXc+iqKYIACCXyPHkyCfxf33+z6EGqjb0+b507EstNKmXSIDltwyGUm6FpYRObTCW+9/c/usREV2ByQh1eRqdBm8ffbtRt0yIewheH/e6Q44PqZeSW4aXfjltdt+shGjEdfNt/03UlfqWEQBw8weix7b/mkREV2AyQl1aWkkant31LE7knzDUTYiYgJfGvARvlbcdI2ueulaHx746ZrZ7JszbBYsm9bHOjVI2A5pKfbnvjYCMPzKIyPr4k4W6JJ3Q4evkr7Hq0CrDbBm5RI4FcQtwV/+7HLZbpt6qzWdxKst09gwArLh1MDxUVvrW/mu9scwuGiKyESYj1OVkV2Rj6e6l2Ht5r6EuyjMKy8cux+DAwXaMrHW2nsnFuu3nze6bfVU0xvW20jNyKguB5N/0ZfcgoPs461yXiOgKTEaoyxBCYEPKBrx+6HWUqcsM9dP7TMfCuIUO83yZ5lwsrMT8r4+Z3dc72ANPXd/Xejc7+T9Ap9GXB/8DkLXxab9ERC1gMkJdQlpJGl7Y9wIOZh801AW5BuGFq17AVeFX2TGy1qvWaPHwF0dQUqUx2aeUSbFm+tD2PQjvSse+MJZjb7fedYmIrsBkhDo1jVaDT059gvePvw+1Tm2ov7HHjXh65NMOPUi1ISEEnttwEn9dKjG7f/ENfdE/zMt6N8xJArKO6suhsUDwAOtdm4joCkxGqNPam7UXrx54FedLjOMrwj3CsWTUEqdpDan37rbzWH/Y/EPwpg4Jw92jo617w8PGac5sFSEiW2MyQp1OZlkmVh5aiS0ZWwx1UokUd/W7Cw/HPuwUY0Ma+ul4Fl7flGx2X68gD6y4ZZB1Z/9UlwLHvtSXFW768SJERDbEZIQ6jUpNJT4++TE+OflJoy6ZwQGD8cyoZzDA3/m6GvanFuDxb4+b3eeulGHdnXFwt9Y03nrHvgTU5fry4OmAqxUWTyMiagaTEXJ6Gq0G3579Fu+feB+F1cal0f1d/LEgbgGm9pwKqcQKy6J3sOMXi3HvZ4egNrOwmVQCvHPHMPQK8rDuTXU64MAHxu2Rc6x7fSIiM5iMkNPSCR1+vfAr3jn6Di6VXzLUy6Vy3NXvLswZPAceSiv/su4gydllmPXJAZTX1Jrd/8+bBmBinyDr3/jsb0Bh3Rib6LFAcH/r34OI6ApMRsjpCCGwPXM73jn6DpKLGo+lmBQ9CY8MfQTdvLrZKbr2O5Ndijv/tR/FlaZTeAH9wmYzE6Ktf2MhgO2vGbcT5ln/HkREZjAZIaeh1WmxOX0zPvzrQ5wtOttoX0JoAh6Le8wpx4U0dCKzGDM/PtBkInLj4FA8N8VGrRUpfwCXj+nLIYOB3pNscx8ioiswGSGHp9Fp8EvqL/jor4+QVprWaN8A/wGYHzcfo0JH2Sc4K9qfWoD7PjuEsia6Zq7pG4TV02Mhk9rguTlXtoqMewJw8OfzEFHnwWSEHFZJTQm+O/cd/nvmv7hccbnRvkEBg3D/oPsxPnK8Uw5OvdKGo5fw5PoTUGtNB6sCQEIPf6y9YxgUMhu919M/AZkH9OWg/von9BIRdRAmI+RwUotT8cXpL/BT6k+oqq1qtG948HDMGTwHo0JHOfyTdVtDCIG3tqRg9R9nmzzmql7++HDmcOsu9d5QrRrYvNS4ffVzgNT5Ezwich5MRsghaHQa7MzciW+Sv8HurN0m+8eGj8X9g+/H0KChdojONkqqNFj07XFsTspp8pir+wbh3TuG2S4RAfRTeYsu6MvRY4E+N9juXkREZjAZIbvKKM3Ad+e+ww/nf0B+VX6jfa5yV0zrNQ23970d0d7R9gnQRpKySvHQF4eRXlDZ5DFTBodi9T9ioZTbsJWiJBPYtrxuQwJc9xLHihBRh2MyQh2uQlOBPzP+xPcp3zd6im69cI9w3N73dvwt5m/wVHraIULb0eoEPtqVipWbzjY5PgQAHhjfA09N6gupLQar1hMC+HmhcbXVYXcBYbG2ux8RUROYjFCHUGvV2HlpJ35N/RXbM7ejRlvTaL9cIsfEqIm4JeYWJIQmQCa1YbeEnaQXVODJ9Sew/0Jhk8fIpBK8ePNA3B4fZfuATnwDnNukL3uEANe+aPt7EhGZwWSEbKZGW4P9l/fjj/Q/8Ef6HyjTlJkcE+0VjVtjbsWNPW9EgGuAHaK0vWqNFu9vT8XabSlml3avF+ChxFszhmJ0zw74dyg4D/yy0Lg9ZSXg6mP7+xIRmcFkhKyqpKYEOy/txJ8Zf2LXpV0ms2EAwFfli+uir8OUHlMQGxjbKWbFmCOEwI5z+fjnj6dwIb+i2WPjuvli7e3DEOLtYvvANNXAt7MaPwyv31Tb35eIqAlMRqhdhBA4X3wee7L2YMelHTicfRi1wnTRLje5G66JugY39LgB8aHxUEgVdoi24xzNKMJrG5OxN7Wg2eMkEmDO2B54/Lo+th2oWk8I4Md5QPZf+m3/GGDKKtvfl4ioGUxGyGKF1YXYl7UPe7L2YG/WXuRW5Zo9zkflgwmREzAxciISwhLgKnft4Eg73slLJXhryzn83sx03XoRvq5447YhiO/h3wGR1dn6MvDXt/qy3BW47VNA5ZwPEySizoPJCLUovyofh3MO40jOERzJPYLkwmQICLPHRnhE4OqoqzExciJig2Ihl3b+/8WEENh2Ng8fbE9tsSUEAKQS4K5R3bBoUh94unRgC9Get4Edr9dtSIC/fwSEDOy4+xMRNaHz/6YgiwghkFGWYUg8juQcQUZZRpPHu8hcMDxkOEaHjUZCaAJ6+vTstGNArlRQXoPvj17Cfw9k4Hxe82NC6g2J8MZL0wZhUIS3jaO7wu43G6+yOukVoO+Ujo2BiKgJTEa6MCEEsiuycbLgJE7ln8KpAv2rTG0666WeBBL08euDhLAEjA4bjaFBQ6GSqTowavuq1mix/Wwefjh2CZuTcqDRmm8hupK/uxILru2NGSOjbPOgu6Zoa4FNi/WrrNab+ByQ8HDHxUBE1AImI12ERqtBakkqzhWfQ0pRCpKLkpFUkITC6qbXvAAAhVSBgQEDMSxoGIYFD0NsUCy8lF4dFLVjqFJrsS05F7+ezMafp3NQoda2+lxPlRxzxvXAPWO6w13Vwd9u5bnAd3OA1K3GuquXAOMWdWwcREQtYDLSyWi0GmSWZyKtJA3nis/hXJH+lV6abnaWy5X8XfwxMGAgBgUMQlxwHAYGDISLvAOmmzoQnU4g6XIpdqXkY+e5PBxMK2p2fRBzPFRy3DEqCg+O6wlfd6WNIm2CEEDyr8BP84GKusHFUjkw9S1g6B0dGwsRUSswGXFCtbpaXC6/jPSydKSX6l8ZpRlIL01HVkUWdKJ1vzi9Vd4Y6D8Q/f37Y0DAAAz0H4ggt6AuM+ajXqW6FscvluDoxSIczSjGkfQiFFSo23StUG8X3HNVd0wfGQmvjhycWq8oDfjtKeDsRmOdRzDw94+B6DEdHw8RUSswGXFANdoaZFdkI6s8C5crLuNyxWVjufwysiuzUatruZWjnlwqRw/vHojxjUEvn17o7dsbvXx6IdQ9tEslHkII5JXXIDm7DMnZZTibU4aTl0pxJrsUutYN/WjSVb38MX1EFCYPDIFC1gHrhVypOEM/SPXIf4CGS+33uhaY9h7gEdjxMRERtRKTkQ5Uo61BXmUe8qvykV+Vj7yqPORV5qGgusBQn1uZi4LqlqeHmuOucEeUZxS6eXVDlFcUYnxiEOMbgyivqE6/yFg9IQQKK9TIKKxERmEl0gv0XzMKKnEutwxFlRqr3SvcxxXThoZh+vAoRPm7We26rabTAem79AnIqe+AhgmqZyhw/XKg/zQ+hZeIHB6TkTbS6rQoU5ehqKYIJTUlKKouQnFNsb58RV1RTRHyq/KbnaXSWp4KT4R6hBqSjvrEo5tXN/i7+Hfalo6aWi1KKjUortKgsEKN3LIa5JZWI7esBjml1cgprUZuqb5syQBTS0X6ueKGgaG4YVAoBkd4d/y/t04LZB7UjwlJ+kHfLdOQwh0YeT8w9nHApWsNNCYi59WmZGTt2rV4/fXXkZ2djSFDhuDtt9/GyJEjmzz+22+/xZIlS5CWloaYmBi8+uqruOGGG9octDVodBrkVeahTF2Gck05KjQV+rK6HGWaMuO2phzl6vJGX8vUZShTlzW58FdbySQy+Lv6I9Q9FGHuYQjxCEGYexjCPMIQ4h6CUPdQeCo9rXpPWxBCQKMVUGt10NTqoNbqUKnWoqKmFlUa/ddKtbbuVYuKGi2q1LWoqDumpEqD4rrEo7hSjeJKDao0tkswmiOXShDXzRdjYwIwoU8QBoR5dWwCoq4ALh8HMvYBF/frX1VFpse5+AAj5wCjHgLc/DouPiIiK7A4Gfn666+xcOFCrFu3DvHx8VizZg0mTZqE5ORkBAUFmRy/Z88ezJgxA8uXL8eNN96IL7/8EtOmTcORI0cwcKB9Vn88l1OG3Rmn8EbS/R1yP4VEBXe5LzzkfvCQ+8K97qubTF/nLvOFu8wXbjIvQCKFqMtxRLVAVTVwvgBIEYBAPoTIM1zXcBxEg7K+XicEtDr9y1AWAjqdQK1O/1UrBLQ6GMrGOtHofP25gFang7pWB7VW6L/WavVJR13C0fCrs1LKpBgQ7oWhkb64qpc/4nv4w8PWU3KrS4GSzLrXRaA4HchLBnJP68tNkUiB7uOAoXcBfW8EFF1r1hMRdR4SIYRFf97Hx8djxIgReOeddwAAOp0OkZGReOSRR/D000+bHD99+nRUVFTg559/NtSNGjUKsbGxWLduXavuWVpaCm9vb5SUlMDLq/1Nz69vOoN3dx6GR8wKi84TOhmEzgXQuUBo3Yyv2vqye+P6um2IrjFeo2OZ/m8rMVtnqv44qQTo5ueOfmFeGBrpjdgoHwwI9YJKLtNndEIH6DT6sRjaWv3X+m2dFtBqGtTVbWsq9a0ZmipAUwGoKxvUVepbNSoLgMpC/auqUF/fWq5+QPRVQJ8bgJjrAPeA1p9LRNTBWvv726I/+dRqNQ4fPozFixcb6qRSKRITE7F3716z5+zduxcLFy5sVDdp0iRs2LChyfvU1NSgpsY4I6C0tNSSMFslWFuF4eVqeOgE3HUCbjphKDes89Tp4F63Xb9aRHO/4BrVyQQgu/K4K1n2C7Tlurada77jwbb3NHeuVGLdrq8WVQA4V/dyNEoPILAPENQfiBwJRI4CAmI4IJWIOh2LkpH8/HxotVoEBwc3qg8ODsaZM2fMnpOdnW32+Ozs7Cbvs3z5cjz//POWhGYRCSSQCTneyWs6BjMnEVmPVK5v5XDz17dueEcC3hHGV0CMvo6JBxF1AQ45m2bx4sWNWlNKS0sRGRlptetLJEAtZEjXmY5xEa1quzB/XHP1rbtP2+/d2nPNt820577WjU8ikUAmlUIulUAmlUAulUB6xbZMKoVcJoVCJoVCLoHM8AvbzL+92V/mrTxOIgOkMkCm0CcP9a/mthVugNKt7qs7oHDVz3Cpr3P11Q8wVXkx0SAiqmNRMhIQEACZTIacnJxG9Tk5OQgJCTF7TkhIiEXHA4BKpYJKZbuHr0kA5MAP49VrbHYPRyeVAHKpFFIpIJNIIK37ZW8oS/Tb9ftldfuVcimUMimUcn0yoJJLDXWKunplg7r6Y12VMrgp5XBTyuCmlMFdJYerQv/VXSkz7O/Qh8gREZFDsCgZUSqViIuLw5YtWzBt2jQA+gGsW7Zswbx588yek5CQgC1btmD+/PmGus2bNyMhIaHNQbfXbcMjcVWvAMMUTYnE+LeyvspcvQQSGP+YlUDS6A/bK+vNHdfwfJjUSxrFIGkQQ8N7NHdcg0s3SCYaJBaGr+i065EQEZHzsbibZuHChZg1axaGDx+OkSNHYs2aNaioqMDs2bMBADNnzkR4eDiWL18OAHjssccwfvx4vPHGG5gyZQq++uorHDp0CB988EFzt7GpSD83RPrZYcVMIiIiMmFxMjJ9+nTk5eVh6dKlyM7ORmxsLDZu3GgYpJqRkQGp1PhsjtGjR+PLL7/Ec889h2eeeQYxMTHYsGGD3dYYISIiIsdi8Toj9mDtdUaIiIjI9lr7+9sOjxclIiIiMmIyQkRERHbFZISIiIjsiskIERER2RWTESIiIrIrJiNERERkV0xGiIiIyK6YjBAREZFdMRkhIiIiu7J4OXh7qF8ktrS01M6REBERUWvV/95uabF3p0hGysrKAACRkZF2joSIiIgsVVZWBm9v7yb3O8WzaXQ6HbKysuDp6QmJRGK165aWliIyMhIXL17stM+86ezvke/P+XX298j35/w6+3u05fsTQqCsrAxhYWGNHqJ7JadoGZFKpYiIiLDZ9b28vDrl/2ANdfb3yPfn/Dr7e+T7c36d/T3a6v011yJSjwNYiYiIyK6YjBAREZFddelkRKVSYdmyZVCpVPYOxWY6+3vk+3N+nf098v05v87+Hh3h/TnFAFYiIiLqvLp0ywgRERHZH5MRIiIisismI0RERGRXTEaIiIjIrjp9MvLyyy9j9OjRcHNzg4+Pj9ljMjIyMGXKFLi5uSEoKAhPPPEEamtrm71uYWEh7rjjDnh5ecHHxwf33nsvysvLbfAOWm/btm2QSCRmXwcPHmzyvAkTJpgc/+CDD3Zg5JaJjo42iXfFihXNnlNdXY25c+fC398fHh4euPXWW5GTk9NBEbdeWloa7r33XnTv3h2urq7o2bMnli1bBrVa3ex5jv4Zrl27FtHR0XBxcUF8fDwOHDjQ7PHffvst+vbtCxcXFwwaNAi//vprB0VqmeXLl2PEiBHw9PREUFAQpk2bhuTk5GbP+fTTT00+KxcXlw6K2HL//Oc/TeLt27dvs+c4y+cHmP95IpFIMHfuXLPHO/rnt2PHDkydOhVhYWGQSCTYsGFDo/1CCCxduhShoaFwdXVFYmIizp071+J1Lf0etlSnT0bUajVuu+02PPTQQ2b3a7VaTJkyBWq1Gnv27MFnn32GTz/9FEuXLm32unfccQdOnTqFzZs34+eff8aOHTswZ84cW7yFVhs9ejQuX77c6HXfffehe/fuGD58eLPn3n///Y3Oe+211zoo6rZ54YUXGsX7yCOPNHv8ggUL8NNPP+Hbb7/F9u3bkZWVhVtuuaWDom29M2fOQKfT4f3338epU6ewevVqrFu3Ds8880yL5zrqZ/j1119j4cKFWLZsGY4cOYIhQ4Zg0qRJyM3NNXv8nj17MGPGDNx77704evQopk2bhmnTpuHkyZMdHHnLtm/fjrlz52Lfvn3YvHkzNBoNrrvuOlRUVDR7npeXV6PPKj09vYMibpsBAwY0infXrl1NHutMnx8AHDx4sNF727x5MwDgtttua/IcR/78KioqMGTIEKxdu9bs/tdeew1vvfUW1q1bh/3798Pd3R2TJk1CdXV1k9e09Hu4TUQX8cknnwhvb2+T+l9//VVIpVKRnZ1tqHvvvfeEl5eXqKmpMXutpKQkAUAcPHjQUPfbb78JiUQiLl26ZPXY20qtVovAwEDxwgsvNHvc+PHjxWOPPdYxQVlBt27dxOrVq1t9fHFxsVAoFOLbb7811J0+fVoAEHv37rVBhNb12muvie7duzd7jCN/hiNHjhRz5841bGu1WhEWFiaWL19u9vh//OMfYsqUKY3q4uPjxQMPPGDTOK0hNzdXABDbt29v8pimfhY5qmXLlokhQ4a0+nhn/vyEEOKxxx4TPXv2FDqdzux+Z/r8AIjvv//esK3T6URISIh4/fXXDXXFxcVCpVKJ//73v01ex9Lv4bbo9C0jLdm7dy8GDRqE4OBgQ92kSZNQWlqKU6dONXmOj49Po9aGxMRESKVS7N+/3+Yxt9aPP/6IgoICzJ49u8Vjv/jiCwQEBGDgwIFYvHgxKisrOyDCtluxYgX8/f0xdOhQvP766812qx0+fBgajQaJiYmGur59+yIqKgp79+7tiHDbpaSkBH5+fi0e54ifoVqtxuHDhxv920ulUiQmJjb5b793795GxwP670ln+awAtPh5lZeXo1u3boiMjMTNN9/c5M8aR3Hu3DmEhYWhR48euOOOO5CRkdHksc78+anVanz++ee45557mn0oq7N9fvUuXLiA7OzsRp+Pt7c34uPjm/x82vI93BZO8aA8W8rOzm6UiAAwbGdnZzd5TlBQUKM6uVwOPz+/Js+xh48++giTJk1q8SGDt99+O7p164awsDCcOHECTz31FJKTk/Hdd991UKSWefTRRzFs2DD4+flhz549WLx4MS5fvoxVq1aZPT47OxtKpdJkzFBwcLBDfV7mpKSk4O2338bKlSubPc5RP8P8/HxotVqz32Nnzpwxe05T35OO/lnpdDrMnz8fV111FQYOHNjkcX369MHHH3+MwYMHo6SkBCtXrsTo0aNx6tQpmz4QtK3i4+Px6aefok+fPrh8+TKef/55jB07FidPnoSnp6fJ8c76+QHAhg0bUFxcjLvvvrvJY5zt82uo/jOw5PNpy/dwWzhlMvL000/j1VdfbfaY06dPtzjIylm05f1mZmZi06ZN+Oabb1q8fsOxLoMGDUJoaCiuueYanD9/Hj179mx74Baw5D0uXLjQUDd48GAolUo88MADWL58ucMu19yWz/DSpUu4/vrrcdttt+H+++9v9lxH+Ay7urlz5+LkyZPNjqcAgISEBCQkJBi2R48ejX79+uH999/Hiy++aOswLTZ58mRDefDgwYiPj0e3bt3wzTff4N5777VjZNb30UcfYfLkyQgLC2vyGGf7/JyFUyYjjz/+eLOZKwD06NGjVdcKCQkxGRVcP8siJCSkyXOuHLhTW1uLwsLCJs9pj7a8308++QT+/v646aabLL5ffHw8AP1f5R31i6w9n2l8fDxqa2uRlpaGPn36mOwPCQmBWq1GcXFxo9aRnJwcm3xe5lj6/rKysjBx4kSMHj0aH3zwgcX3s8dnaE5AQABkMpnJzKXm/u1DQkIsOt4RzJs3zzCQ3dK/jhUKBYYOHYqUlBQbRWddPj4+6N27d5PxOuPnBwDp6en4448/LG5NdKbPr/4zyMnJQWhoqKE+JycHsbGxZs9py/dwm1ht9ImDa2kAa05OjqHu/fffF15eXqK6utrsteoHsB46dMhQt2nTJocZwKrT6UT37t3F448/3qbzd+3aJQCI48ePWzky2/j888+FVCoVhYWFZvfXD2Bdv369oe7MmTMOO4A1MzNTxMTEiP/7v/8TtbW1bbqGI32GI0eOFPPmzTNsa7VaER4e3uwA1htvvLFRXUJCgkMOgNTpdGLu3LkiLCxMnD17tk3XqK2tFX369BELFiywcnS2UVZWJnx9fcWbb75pdr8zfX4NLVu2TISEhAiNRmPReY78+aGJAawrV6401JWUlLRqAKsl38NtitVqV3JQ6enp4ujRo+L5558XHh4e4ujRo+Lo0aOirKxMCKH/H2ngwIHiuuuuE8eOHRMbN24UgYGBYvHixYZr7N+/X/Tp00dkZmYa6q6//noxdOhQsX//frFr1y4RExMjZsyY0eHvz5w//vhDABCnT5822ZeZmSn69Okj9u/fL4QQIiUlRbzwwgvi0KFD4sKFC+KHH34QPXr0EOPGjevosFtlz549YvXq1eLYsWPi/Pnz4vPPPxeBgYFi5syZhmOufI9CCPHggw+KqKgo8eeff4pDhw6JhIQEkZCQYI+30KzMzEzRq1cvcc0114jMzExx+fJlw6vhMc70GX711VdCpVKJTz/9VCQlJYk5c+YIHx8fwwy2u+66Szz99NOG43fv3i3kcrlYuXKlOH36tFi2bJlQKBTir7/+stdbaNJDDz0kvL29xbZt2xp9VpWVlYZjrnx/zz//vNi0aZM4f/68OHz4sPi///s/4eLiIk6dOmWPt9Cixx9/XGzbtk1cuHBB7N69WyQmJoqAgACRm5srhHDuz6+eVqsVUVFR4qmnnjLZ52yfX1lZmeH3HACxatUqcfToUZGeni6EEGLFihXCx8dH/PDDD+LEiRPi5ptvFt27dxdVVVWGa1x99dXi7bffNmy39D1sDZ0+GZk1a5YAYPLaunWr4Zi0tDQxefJk4erqKgICAsTjjz/eKDveunWrACAuXLhgqCsoKBAzZswQHh4ewsvLS8yePduQ4NjbjBkzxOjRo83uu3DhQqP3n5GRIcaNGyf8/PyESqUSvXr1Ek888YQoKSnpwIhb7/DhwyI+Pl54e3sLFxcX0a9fP/HKK680asW68j0KIURVVZV4+OGHha+vr3BzcxN/+9vfGv2CdxSffPKJ2f9fGzZiOuNn+Pbbb4uoqCihVCrFyJEjxb59+wz7xo8fL2bNmtXo+G+++Ub07t1bKJVKMWDAAPHLL790cMSt09Rn9cknnxiOufL9zZ8/3/BvERwcLG644QZx5MiRjg++laZPny5CQ0OFUqkU4eHhYvr06SIlJcWw35k/v3qbNm0SAERycrLJPmf7/Op/X135qn8POp1OLFmyRAQHBwuVSiWuueYak/fdrVs3sWzZskZ1zX0PW4NECCGs1+lDREREZJkuv84IERER2ReTESIiIrIrJiNERERkV0xGiIiIyK6YjBAREZFdMRkhIiIiu2IyQkRERHbFZISIiIjsiskIERER2RWTESIiIrIrJiNERERkV0xGiIiIyK7+H3Ff77ohzBCkAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, 1/(1+np.exp(-x)),linewidth=4,clip_on=False);\n",
    "plt.plot(x, 1/(1+np.exp(-2*x)),linewidth=2,clip_on=False);\n",
    "plt.plot(x, 1/(1+np.exp(-.5*x)),linewidth=2,clip_on=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Activation Functions: tanh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjgAAAGdCAYAAAAfTAk2AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAABHWUlEQVR4nO3de1yUZf4//tfMwAygHOUwoAh4CDUPmCRhbSd4COmW7nZQ11ZlTTfTyrBU9rdJ6hZlfjpY7lqtx1+6WW2H7fChDLU+JaGiVhqSGIIIM6jIDAeBYeb6/uE6Oc09nGTOr+fjMY/kuq/7nvftHTMvr/u671smhBAgIiIi8iByZxdARERE1NsYcIiIiMjjMOAQERGRx2HAISIiIo/DgENEREQehwGHiIiIPA4DDhEREXkcBhwiIiLyOD7OLsAZTCYTqqurERgYCJlM5uxyiIiIqAuEEGhoaEBMTAzk8o7HaLwy4FRXVyM2NtbZZRAREVEPnD59GgMGDOiwj1cGnMDAQACX/oKCgoKcXA0RERF1hV6vR2xsrPl7vCNeGXAun5YKCgpiwCEiInIzXZlewknGRERE5HEYcIiIiMjjMOAQERGRx2HAISIiIo/DgENEREQehwGHiIiIPA4DDhEREXkcBhwiIiLyOAw4RERE5HHsGnC++uor3HnnnYiJiYFMJsMHH3zQ6Tp79+7FddddB5VKhSFDhmDLli1WfdavX4/4+Hj4+fkhJSUF+/fv7/3iiYiIyG3Z9VENTU1NGDNmDP70pz/h97//faf9y8vLMXnyZDz44IPYvn07CgoK8MADDyA6OhoZGRkAgJ07dyI7OxsbNmxASkoKXnrpJWRkZKC0tBSRkZH23B0iIo8nhIDRJNBuEjAYTWg3ChhMl/7bbhRoM5pgEgImISAEzP8VAhAQMIlL2zAJALjc55c2gV/6m4SA6GJNnfbp0s51pUsX3qsr2+lSQd5h1IBgRAX5Ofx9ZaIr/+f0xhvJZHj//fcxdepUm32WLVuGTz75BEePHjW3TZ8+HfX19cjPzwcApKSk4Prrr8err74KADCZTIiNjcXDDz+M5cuXd6kWvV6P4OBg6HQ6PouKiDyGySRwrqkVNfUtqGtuw4WmNtQ1taG+2YC65jY0tLTjYls7mtuMaGozmv98sc2IiwajOczwy5l6099nXodJo6J7ZVvd+f52qYdtFhYWIj093aItIyMDixcvBgC0tbWhuLgYOTk55uVyuRzp6ekoLCy0ud3W1la0traaf9br9b1bOBGRgwghUNvQip+0DfhJ24gT2gZU1jXjTP1F1NS3oM1ocnaJRC7BpQKORqNBVFSURVtUVBT0ej0uXryICxcuwGg0SvY5fvy4ze3m5eVh5cqVdqmZiMiejCaBo2d0OHCqDsUVF3Cw4gLONrR2viKRl3OpgGMvOTk5yM7ONv+s1+sRGxvrxIqIiGxrbTdib+lZ7PpRi93Ha1HX1ObskojcjksFHLVaDa1Wa9Gm1WoRFBQEf39/KBQKKBQKyT5qtdrmdlUqFVQqlV1qJiLqLaWaBrx1oBIfHD6DC80GZ5dD5NZcKuCkpqbi008/tWjbtWsXUlNTAQBKpRLjxo1DQUGBebKyyWRCQUEBFi1a5OhyiYh6xaHKC1i/uwwFx2vt9h4+chmC/X0RoFIgwNcH/koFAswvH/j5yqH0kcNHLoevQgYfhRy+chl8FfJLf1bI4CO/1O4jl0Euk0Emu3QBiVyGS3/Gr9pwxTKZDDLAvJ780gqQ4dKyznTe49L7dL6dLrxXl7bTBV3q5PmuiQp0yvvaNeA0NjairKzM/HN5eTmOHDmCsLAwDBw4EDk5OThz5gy2bdsGAHjwwQfx6quvYunSpfjTn/6E3bt34+2338Ynn3xi3kZ2djZmz56N5ORkjB8/Hi+99BKampqQlZVlz10hIup1J882YtVHP+LLn872eBsyGTAg1B+xoQGICfFH/xB/xIT4ITLID6EBSoQFKBHSxxeBKp8uBQkiT2HXgHPw4EHcdttt5p8vz4OZPXs2tmzZgpqaGlRWVpqXJyQk4JNPPsFjjz2Gl19+GQMGDMA///lP8z1wAGDatGk4e/YsVqxYAY1Gg6SkJOTn51tNPCYiclUtBiPW7ynDa1/+3K2rnuQyYERMEJLjwjCqfzAS1YEYHNEX/kqFHaslck8Ouw+OK+F9cIjIWY5V6/DwjsP4+VxTl/r3D/FH2vBI3D4sEtfHh6GPyqVmFhA5lNveB4eIyJPtKKrEUx8dQ1t7x6M2/r4KTB4djfuSY3F9fChPLRH1AAMOEZGdmUwCef9bgjf+r7zDfoF+Psi6MQFZE+IR2kfpoOqIPBMDDhGRHbUbTXjs7e/w0XfVNvso5DLMmRCPR9OHIsjP14HVEXkuBhwiIjtpN5rw6M4j+OT7Gpt9xgwIRt7vR2NEDOcDEvUmBhwiIjsQQmDZv3/oMNzMv3kQnshIhK9C7sDKiLwDAw4RkR28XHAC/z5UJblMqZDj+XtHY0pSfwdXReQ9GHCIiHrZR99V46UvTkguC1AqsGnO9bhhUD8HV0XkXRhwiIh60c9nG7H8399LLgtQKrAlazzGJ4Q5uCoi78MTv0REvaTFYMSiHYfR1Ga0WqaQy/CP+8cx3BA5CAMOEVEvefGLn/BjjV5y2eopI3HLNREOrojIezHgEBH1gh+qdPinjRv5zRgfiz+kDHRwRUTejQGHiOgqGYwmLP339zCarB/tlxgViBW/vdYJVRF5NwYcIqKrtP3bCpRInJryVciwbsZYPu2byAkYcIiIroKu2YCXCqQvCV942xAkqgMdXBERAQw4RERX5dU9J1DfbLBqHxrZFw/dOsQJFRERwIBDRNRjZ+ovYsu+U5LLcu+8FkoffsQSOQt/+4iIemjD3pMwGK0nFt8+LBI3DQ13QkVEdBkDDhFRD2j1Ldh58LRVu0Iuw18mDXNCRUR0JQYcIqIeeO3Ln9HWbrJqn5rUH0MiObGYyNkYcIiIuknXbMC/9ldatctlwMLbBjuhIiL6NQYcIqJu2nmwEhcN1s+b+u3oGAyK6OuEiojo1xhwiIi6wWgS2FZYIbls4W28LJzIVTDgEBF1Q0GJFlUXLlq13zikH2/qR+RCGHCIiLrB1ujN7NR4xxZCRB1iwCEi6qKqC8345uQ5q/YBof5IGx7lhIqIyBYGHCKiLnr/0BkI6/v64f4b4qCQyxxfEBHZxIBDRNQFQgi8e6jKql0hl+H31/V3QkVE1BEGHCKiLjhYcQEV55ut2m+9JgKRgX5OqIiIOsKAQ0TUBe9JjN4AwN3jBji4EiLqCgYcIqJOGIwm5B/VWLUH+/sibXikEyoios4w4BARdaLo5zpcaDZYtU8eHQ2Vj8IJFRFRZxwScNavX4/4+Hj4+fkhJSUF+/fvt9n31ltvhUwms3pNnjzZ3GfOnDlWyzMzMx2xK0TkhT75oUay/bejoh1cCRF1lY+932Dnzp3Izs7Ghg0bkJKSgpdeegkZGRkoLS1FZKT10O57772HtrY288/nz5/HmDFjcO+991r0y8zMxObNm80/q1Qq++0EEXmtdqMJnx2zPj3Vr48S4xPCnFAREXWF3UdwXnjhBcybNw9ZWVkYMWIENmzYgICAAGzatEmyf1hYGNRqtfm1a9cuBAQEWAUclUpl0S80NNTeu0JEXmh/eR3qmtqs2ideq4aPgmf5iVyVXX8729raUFxcjPT09F/eUC5Heno6CgsLu7SNjRs3Yvr06ejTp49F+969exEZGYnExEQsWLAA58+ft7mN1tZW6PV6ixcRUVd8/qNWsn0yT08RuTS7Bpxz587BaDQiKsryFuZRUVHQaKyHfH9t//79OHr0KB544AGL9szMTGzbtg0FBQV47rnn8OWXX+KOO+6A0WiU3E5eXh6Cg4PNr9jY2J7vFBF5DSEEdh+vtWoPDfDFDYN4eorIldl9Ds7V2LhxI0aNGoXx48dbtE+fPt3851GjRmH06NEYPHgw9u7di7S0NKvt5OTkIDs72/yzXq9nyCGiTp0824TKOomb+yVG8vQUkYuz629oeHg4FAoFtFrLIV6tVgu1Wt3huk1NTXjrrbcwd+7cTt9n0KBBCA8PR1lZmeRylUqFoKAgixcRUWf2SIzeAMBtw3jvGyJXZ9eAo1QqMW7cOBQUFJjbTCYTCgoKkJqa2uG677zzDlpbW3H//fd3+j5VVVU4f/48oqN5TpyIeo/U6SmFXIZbhkY4oRoi6g67j7FmZ2fjjTfewNatW1FSUoIFCxagqakJWVlZAIBZs2YhJyfHar2NGzdi6tSp6Nevn0V7Y2MjnnjiCXz77bc4deoUCgoKMGXKFAwZMgQZGRn23h0i8hL6FgMOnKqzah83MBTBAb5OqIiIusPuc3CmTZuGs2fPYsWKFdBoNEhKSkJ+fr554nFlZSXkcsucVVpaiq+//hqff/651fYUCgW+//57bN26FfX19YiJicHEiROxevVq3guHiHrNvrLzaDcJq3aeniJyDzIhhPVvsIfT6/UIDg6GTqfjfBwikrTiw6PYVlhh1Z6/+DcYpubnBpEzdOf7m5cBEBFJ+KbsnFVbRKAKiVGBTqiGiLqLAYeI6Fc0uhacPNtk1T5hcD/IZDInVERE3cWAQ0T0K4U/W4/eAMCNg8MdXAkR9RQDDhHRr3xTJv3olwlD+km2E5HrYcAhIrqCEAL7JObfxPULwIDQACdUREQ9wYBDRHSFyrpmVOtarNon8PQUkVthwCEiusKBUxck2ycM5ukpInfCgENEdIWDEncvBoDxCXx6OJE7YcAhIrqC1OMZYsP8ERXk54RqiKinGHCIiP7rfGOr5P1vro/j6A2Ru2HAISL6r+IK6fk3yfEMOETuhgGHiOi/DtoIONfHhzq4EiK6Wgw4RET/JTX/JiTAF4Mj+jqhGiK6Ggw4REQAWtuNOHpGZ9U+bmAo5HI+f4rI3TDgEBEBKKlpgMEorNqvi+PpKSJ3xIBDRATgu9P1ku1jY0McWgcR9Q4GHCIiAEckAo5MBowaEOz4YojoqjHgEBFBegRnSERfBPr5Or4YIrpqDDhE5PV0zQb8fM76Bn9jeHqKyG0x4BCR1/v+TL1kOwMOkftiwCEir2drgnHSgBCH1kFEvYcBh4i83pHT1ve/UfrIkagOdEI1RNQbGHCIyOtJ3eDv2pggKH34EUnkrvjbS0Re7XxjKzT6Fqv2Uf15eTiRO2PAISKvdqxaL9l+bUyQgyshot7EgENEXu1otfXpKQC4NoYjOETujAGHiLya1AiOr0KGa6I4wZjInTHgEJFXOyYxwXhoZCAnGBO5Of4GE5HXamgx4NT5Zqv2kf05/4bI3THgEJHXKqlpkGzn/Bsi98eAQ0ReS+r+NwBHcIg8gUMCzvr16xEfHw8/Pz+kpKRg//79Nvtu2bIFMpnM4uXn52fRRwiBFStWIDo6Gv7+/khPT8eJEyfsvRtE5GFKaqwnGMtkwDA1Aw6Ru7N7wNm5cyeys7ORm5uLQ4cOYcyYMcjIyEBtba3NdYKCglBTU2N+VVRUWCxfs2YN1q1bhw0bNqCoqAh9+vRBRkYGWlqsb9ZFRGTLT1rrU1Tx/fqgj8rHCdUQUW+ye8B54YUXMG/ePGRlZWHEiBHYsGEDAgICsGnTJpvryGQyqNVq8ysqKsq8TAiBl156CX/9618xZcoUjB49Gtu2bUN1dTU++OADe+8OEXkIk0ngJ22jVXsiLw8n8gh2DThtbW0oLi5Genr6L28olyM9PR2FhYU212tsbERcXBxiY2MxZcoUHDt2zLysvLwcGo3GYpvBwcFISUmxuc3W1lbo9XqLFxF5t8q6Zlw0GK3ar+EDNok8gl0Dzrlz52A0Gi1GYAAgKioKGo1Gcp3ExERs2rQJH374Id58802YTCZMmDABVVVVAGBerzvbzMvLQ3BwsPkVGxt7tbtGRG6uVOL0FAAMY8Ah8ggudxVVamoqZs2ahaSkJNxyyy147733EBERgddee63H28zJyYFOpzO/Tp8+3YsVE5E7KtVIBxzewZjIM9g14ISHh0OhUECr1Vq0a7VaqNXqLm3D19cXY8eORVlZGQCY1+vONlUqFYKCgixeROTdpEZwlD5yxPcLcEI1RNTb7BpwlEolxo0bh4KCAnObyWRCQUEBUlNTu7QNo9GIH374AdHR0QCAhIQEqNVqi23q9XoUFRV1eZtERFIjOEMj+8JH4XID20TUA3a/FjI7OxuzZ89GcnIyxo8fj5deeglNTU3IysoCAMyaNQv9+/dHXl4eAGDVqlW44YYbMGTIENTX1+P5559HRUUFHnjgAQCXrrBavHgx/va3v2Ho0KFISEjAk08+iZiYGEydOtXeu0NEHqC13Yjyc01W7byCishz2D3gTJs2DWfPnsWKFSug0WiQlJSE/Px88yThyspKyOW//IvpwoULmDdvHjQaDUJDQzFu3Djs27cPI0aMMPdZunQpmpqaMH/+fNTX1+Omm25Cfn6+1Q0BiYiknKxtgtEkrNoTOcGYyGPIhBDWv+UeTq/XIzg4GDqdjvNxiLzQ+4er8NjO76zaN2ddj9sSI51QERF1RXe+v3mymYi8TqnG+gZ/AC8RJ/IkDDhE5HVKNdY3+wzy84E6iKe5iTwFAw4ReR3JRzSoAyGTyZxQDRHZAwMOEXkVfYsBZ+ovWrVzgjGRZ2HAISKvcsLGIxp4iTiRZ2HAISKvUlYrPcGYj2gg8iwMOETkVU6etb7BHwAMZcAh8igMOETkVU5KjOCEBvgirI/SCdUQkb0w4BCRVyk7ax1wBkf0dUIlRGRPDDhE5DVaDEacrmu2amfAIfI8DDhE5DUqzjdD4hFUGBLJgEPkaRhwiMhrnJQ4PQUAgyP7OLgSIrI3Bhwi8hq2LhHnKSoiz8OAQ0ReQ2oER6mQY0BogBOqISJ7YsAhIq8hFXASwvtAIeczqIg8DQMOEXkFk0ngZK31Tf44wZjIMzHgEJFXqNG34KLBaNU+OIITjIk8EQMOEXkFqTsYA8BgjuAQeSQGHCLyCjYvEecVVEQeiQGHiLyCrYAziKeoiDwSAw4ReQWpCcb9Q/wRoPRxQjVEZG8MOETkFaQessnRGyLPxYBDRB6vocWAsw2tVu2cf0PkuRhwiMjjnTpn/QRxgCM4RJ6MAYeIPF75eev5NwAQ348Bh8hTMeAQkcc7dU464CSEM+AQeSoGHCLyeFIBR6mQIybE3wnVEJEjMOAQkceTOkU1sF8AH7JJ5MEYcIjI45VLjOBw/g2RZ2PAISKPVt/chvpmg1V7QniAE6ohIkdhwCEijyY1egMA8ZxgTOTRHBJw1q9fj/j4ePj5+SElJQX79++32feNN97Ab37zG4SGhiI0NBTp6elW/efMmQOZTGbxyszMtPduEJEbOmXjEvEEnqIi8mh2Dzg7d+5EdnY2cnNzcejQIYwZMwYZGRmora2V7L93717MmDEDe/bsQWFhIWJjYzFx4kScOXPGol9mZiZqamrMr3/961/23hUickPlNm7yl8Cb/BF5NLsHnBdeeAHz5s1DVlYWRowYgQ0bNiAgIACbNm2S7L99+3Y89NBDSEpKwrBhw/DPf/4TJpMJBQUFFv1UKhXUarX5FRoaau9dISI3JHWJuJ+vHFGBfk6ohogcxa4Bp62tDcXFxUhPT//lDeVypKeno7CwsEvbaG5uhsFgQFhYmEX73r17ERkZicTERCxYsADnz5+3uY3W1lbo9XqLFxF5B1tXUMl5iTiRR7NrwDl37hyMRiOioqIs2qOioqDRaLq0jWXLliEmJsYiJGVmZmLbtm0oKCjAc889hy+//BJ33HEHjEaj5Dby8vIQHBxsfsXGxvZ8p4jIbQghJEdweIk4kefzcXYBHXn22Wfx1ltvYe/evfDz+2U4efr06eY/jxo1CqNHj8bgwYOxd+9epKWlWW0nJycH2dnZ5p/1ej1DDpEXON/UhobWdqt2XkFF5PnsOoITHh4OhUIBrVZr0a7VaqFWqztcd+3atXj22Wfx+eefY/To0R32HTRoEMLDw1FWVia5XKVSISgoyOJFRJ7P9jOoeA8cIk9n14CjVCoxbtw4iwnClycMp6am2lxvzZo1WL16NfLz85GcnNzp+1RVVeH8+fOIjo7ulbqJyDPYugdOQnhfB1dCRI5m96uosrOz8cYbb2Dr1q0oKSnBggUL0NTUhKysLADArFmzkJOTY+7/3HPP4cknn8SmTZsQHx8PjUYDjUaDxsZGAEBjYyOeeOIJfPvttzh16hQKCgowZcoUDBkyBBkZGfbeHSJyI7bugRPPERwij2f3OTjTpk3D2bNnsWLFCmg0GiQlJSE/P9888biyshJy+S856x//+Afa2tpwzz33WGwnNzcXTz31FBQKBb7//nts3boV9fX1iImJwcSJE7F69WqoVCp77w4RuZFTEvfA6aNUIKIvPyuIPJ1MCCGcXYSj6fV6BAcHQ6fTcT4OkQebvO7/cKza8rYQI6KD8Omjv3FSRUR0Nbrz/c1nURGRRxJCoPK89QgOT08ReQcGHCLySPXNBslLxAeG8RJxIm/AgENEHqmiTvoZVAPDOIJD5A0YcIjII1XYuIIqrh8DDpE3YMAhIo8kNf8G4AgOkbdgwCEijyR1ispXIUNMiL8TqiEiR2PAISKPJDWCMyA0AAo+RZzIKzDgEJFHqqiznoPD01NE3oMBh4g8TovBCK2+1aqdE4yJvAcDDhF5nNO8RJzI6zHgEJHHqeAVVERejwGHiDyOrZv8xfXjXYyJvAUDDhF5nEobN/njCA6R92DAISKPIzWCExmogr9S4YRqiMgZGHCIyONI3QOHV1AReRcGHCLyKEaTQNWFi1btfIo4kXdhwCEij6LRt6DNaLJq5/wbIu/CgENEHoVPEScigAGHiDyMzaeIM+AQeRUGHCLyKDbvgcNTVERehQGHiDyK1AhOX5UPwvoonVANETkLAw4ReRRbTxGXyWROqIaInIUBh4g8itQIDq+gIvI+DDhE5DHqm9ugb2m3aucVVETehwGHiDyGzaeIM+AQeR0GHCLyGLavoOJdjIm8DQMOEXkMW08R5ykqIu/DgENEHkPqFJWPXIboYD8nVENEzsSAQ0Qeo1LiFNWAUH/4KPhRR+Rt+FtPRB5DKuDE8hJxIq/EgENEHqHFYIRG32LVzvk3RN7JIQFn/fr1iI+Ph5+fH1JSUrB///4O+7/zzjsYNmwY/Pz8MGrUKHz66acWy4UQWLFiBaKjo+Hv74/09HScOHHCnrtARC6u6kIzhLBu5xVURN7J7gFn586dyM7ORm5uLg4dOoQxY8YgIyMDtbW1kv337duHGTNmYO7cuTh8+DCmTp2KqVOn4ujRo+Y+a9aswbp167BhwwYUFRWhT58+yMjIQEuL9b/eiMg78B44RHQlmRBS/+bpPSkpKbj++uvx6quvAgBMJhNiY2Px8MMPY/ny5Vb9p02bhqamJnz88cfmthtuuAFJSUnYsGEDhBCIiYnBkiVL8PjjjwMAdDodoqKisGXLFkyfPr3TmvR6PYKDg6HT6RAUFNRLe/qLpibpS1UBQKFQwM/Pr0t95XI5/P39e9S3ubkZtg6tTCZDQEBAj/pevHgRJpPJZh19+vTpUd+WlhYYjcZe6RsQ8Mtzh1pbW9Hebn1n25709ff3h1x+6d8EbW1tMBgMvdLXz88PCoWi230NBgPa2tps9lWpVPDx8el23/b2drS2ttrsq1Qq4evr2+2+RqOxw3+E+Pr6QqlUdruvyWTCxYsX8f8XVeHZz8us+n608AaMiu1n0dcWHx8fqFQqAJdGipubpUNTd/t25/eenxHSffkZ4Z6fEb2tW9/fwo5aW1uFQqEQ77//vkX7rFmzxF133SW5TmxsrHjxxRct2lasWCFGjx4thBDi5MmTAoA4fPiwRZ+bb75ZPPLII5LbbGlpETqdzvw6ffq0ACB0Ol2P9qszAGy+Jk2aZNE3ICDAZt9bbrnFom94eLjNvsnJyRZ94+LibPYdMWKERd8RI0bY7BsXF2fRNzk52Wbf8PBwi7633HKLzb4BAQEWfSdNmtTh39uV7rnnng77NjY2mvvOnj27w761tbXmvg899FCHfcvLy819H3/88Q77Hj161Nw3Nze3w7779+83912zZk2Hfffs2WPu++qrr3bY9+OPPzb33bx5c4d93377bXPft99+u8O+mzdvNvf9+OOPO+z76quvmvvu2bOnw75r1qwx992/f3+HfXNzc819jx49KgCI0LT5Im7Zx1avxY8vNfctLy/vcLsPPfSQuW9tbW2HfWfPnm3u29jY2GHfe+65x+L/4Y768jPi0oufEb+83PkzorfpdDoBdO37266nqM6dOwej0YioqCiL9qioKGg0Gsl1NBpNh/0v/7c728zLy0NwcLD5FRsb26P9ISLX5ROitmozNl6AD2yPEBCR57LrKarq6mr0798f+/btQ2pqqrl96dKl+PLLL1FUVGS1jlKpxNatWzFjxgxz29///nesXLkSWq0W+/btw4033ojq6mpER0eb+9x3332QyWTYuXOn1TZbW1sthtP0ej1iY2N5iqoHfTn8zOFnVz1Fdeff9+PnX83DSRoQhLfnp1j1tYWnqC7hZ0TP+vIz4hJXOUXlY5cK/is8PBwKhQJardaiXavVQq22/tcWAKjV6g77X/6vVqu1CDharRZJSUmS21SpVOYPIke48hfNWX2v/MDpzb5XfkD2Zt8rP9B7s293jn13+iqVSvOXprP6+vr6dvlDpDt9fXx8zB9kvdlXoVB0+f/h7vS99MUdgCqddSAaFBFo8fcpl8u7vF2ZTGaXvgA/I3rSl58R3e/rCp8RzmTXU1RKpRLjxo1DQUGBuc1kMqGgoMBiROdKqampFv0BYNeuXeb+CQkJUKvVFn30ej2KiopsbpOIPJtG34K2duuRAF5BReS97B7BsrOzMXv2bCQnJ2P8+PF46aWX0NTUhKysLADArFmz0L9/f+Tl5QEAHn30Udxyyy34n//5H0yePBlvvfUWDh48iNdffx3ApX8pLV68GH/7298wdOhQJCQk4Mknn0RMTAymTp1q790hIhdk6xJx3uSPyHvZPeBMmzYNZ8+exYoVK6DRaJCUlIT8/HzzJOHKykrzeUgAmDBhAnbs2IG//vWv+Mtf/oKhQ4figw8+wMiRI819li5diqamJsyfPx/19fW46aabkJ+f361hSSLyHJV10nNPBvImf0Rey+73wXFF9r4PDhE51pr84/j73pNW7Qf/mo7wvo6bf0dE9tWd728+i4qI3F6FxEM2+ygV6Nena5MxicjzMOAQkdurlJiDM7BfH/PlvUTkfRhwiMjtVZy3noMTF8YJxkTejAGHiNxafXMb9C3WN1/jFVRE3o0Bh4jcGp8iTkRSGHCIyK1JTTAGgDheIk7k1RhwiMitVUrMvwF4iorI2zHgEJFbkzpF5SOXITqYN/4k8mYMOETk1qROUfUP9YePgh9vRN6MnwBE5NYk74HDS8SJvB4DDhG5rRaDERp9i1U7598QEQMOEbmt07yCiohsYMAhIrfFe+AQkS0MOETktmzeA4cBh8jrMeAQkduydQ8cTjImIgYcInJbUiM4EYEqBCh9nFANEbkSBhwicltSl4jzKeJEBDDgEJGbMpoETl+QuAcO598QERhwiMhN1eguwmAUVu28RJyIAAYcInJTUqenAF5BRUSXMOAQkVuydYk4T1EREcCAQ0RuytZN/jjJmIgABhwiclOVddb3wOmr8kFYH6UTqiEiV8OAQ0RuSWoEZ2BYAGQymROqISJXw4BDRG5HCCF9DxzOvyGi/2LAISK3c6HZgIbWdqt2TjAmossYcIjI7VTYeAYV74FDRJcx4BCR26nkU8SJqBMMOETkdmxdIs6niBPRZQw4ROR2pAKOr0KGmBB/J1RDRK6IAYeI3M4piTk4A0IDoJDzEnEiusSuAaeurg4zZ85EUFAQQkJCMHfuXDQ2NnbY/+GHH0ZiYiL8/f0xcOBAPPLII9DpdBb9ZDKZ1eutt96y564QkQs5dc464MRz/g0RXcHHnhufOXMmampqsGvXLhgMBmRlZWH+/PnYsWOHZP/q6mpUV1dj7dq1GDFiBCoqKvDggw+iuroa7777rkXfzZs3IzMz0/xzSEiIPXeFiFyEvsWA801tVu3x4byCioh+YbeAU1JSgvz8fBw4cADJyckAgFdeeQWTJk3C2rVrERMTY7XOyJEj8e9//9v88+DBg/H000/j/vvvR3t7O3x8fik3JCQEarXaXuUTkYuSGr0BgEEMOER0BbudoiosLERISIg53ABAeno65HI5ioqKurwdnU6HoKAgi3ADAAsXLkR4eDjGjx+PTZs2QQhhcxutra3Q6/UWLyJyT+U2Ag5HcIjoSnYbwdFoNIiMjLR8Mx8fhIWFQaPRdGkb586dw+rVqzF//nyL9lWrVuH2229HQEAAPv/8czz00ENobGzEI488IrmdvLw8rFy5smc7QkQuxWbA6ceAQ0S/6PYIzvLlyyUn+V75On78+FUXptfrMXnyZIwYMQJPPfWUxbInn3wSN954I8aOHYtly5Zh6dKleP75521uKycnBzqdzvw6ffr0VddHRM4hdYpKqZDzEnEistDtEZwlS5Zgzpw5HfYZNGgQ1Go1amtrLdrb29tRV1fX6dyZhoYGZGZmIjAwEO+//z58fX077J+SkoLVq1ejtbUVKpXKarlKpZJsJyL3Uy71FPF+vESciCx1O+BEREQgIiKi036pqamor69HcXExxo0bBwDYvXs3TCYTUlJSbK6n1+uRkZEBlUqF//znP/Dz8+v0vY4cOYLQ0FCGGCIvIH2JOE9PEZElu83BGT58ODIzMzFv3jxs2LABBoMBixYtwvTp081XUJ05cwZpaWnYtm0bxo8fD71ej4kTJ6K5uRlvvvmmxYTgiIgIKBQKfPTRR9Bqtbjhhhvg5+eHXbt24ZlnnsHjjz9ur10hIhdxoakNuosGq/ZBEQw4RGTJrvfB2b59OxYtWoS0tDTI5XLcfffdWLdunXm5wWBAaWkpmpsvDTkfOnTIfIXVkCFDLLZVXl6O+Ph4+Pr6Yv369XjssccghMCQIUPwwgsvYN68efbcFSJyAeU2niLOERwi+jWZ6Oj6ag+l1+sRHBxsvgSdiNzDv4ursOSd76zad8xLwYTB4U6oiIgcqTvf33wWFRG5DalnUAFAAu+BQ0S/woBDRG5D6h44fr5yRAV2fjECEXkXBhwichtSIzjx/fpAzkvEiehXGHCIyC0IIXDqnPU9cHh6ioikMOAQkVs419iGxtZ2q3Y+g4qIpDDgEJFbsPUMqgReIk5EEhhwiMgtSN3BGOAIDhFJY8AhIrdQdrZRsp1zcIhICgMOEbmFk7XWASfY3xfhfZVOqIaIXB0DDhG5hZMSIziDI/pAJuMl4kRkjQGHiFxea7sRlXXWl4gPiezrhGqIyB0w4BCRy6s43wyTxFPzBkcw4BCRNAYcInJ5UvNvAAYcIrKNAYeIXF6ZrYDDU1REZAMDDhG5PKkJxkqFHLGh/k6ohojcAQMOEbm8k2clHrIZHgAfBT/CiEgaPx2IyKUJIWxcIs7TU0RkGwMOEbk0jb4FzW1Gq3YGHCLqCAMOEbm0k7XSz6DiPXCIqCMMOETk0spqGyTbOYJDRB1hwCEilyY1wRgABkXwIZtEZBsDDhG5NKkJxtHBfuij8nFCNUTkLhhwiMil8QoqIuoJBhwicln6FgO0+lardk4wJqLOMOAQkcv6SWNrgjHn3xBRxxhwiMhllWqlA06iOsjBlRCRu2HAISKXVWpjBCcxKtDBlRCRu2HAISKXJRVw1EF+CA7wdUI1ROROGHCIyCUJISRPUSWqOXpDRJ1jwCEil1Tb0Ir6ZoNVOwMOEXUFAw4RuSTOvyGiq2HXgFNXV4eZM2ciKCgIISEhmDt3LhobrW/adaVbb70VMpnM4vXggw9a9KmsrMTkyZMREBCAyMhIPPHEE2hvb7fnrhCRg9kMOBzBIaIusOu9zmfOnImamhrs2rULBoMBWVlZmD9/Pnbs2NHhevPmzcOqVavMPwcEBJj/bDQaMXnyZKjVauzbtw81NTWYNWsWfH198cwzz9htX4jIsaTm38hlvMkfEXWN3QJOSUkJ8vPzceDAASQnJwMAXnnlFUyaNAlr165FTEyMzXUDAgKgVqsll33++ef48ccf8cUXXyAqKgpJSUlYvXo1li1bhqeeegpKpdIu+0NEjiU1ghMf3gd+vgonVENE7sZup6gKCwsREhJiDjcAkJ6eDrlcjqKiog7X3b59O8LDwzFy5Ejk5OSgubnZYrujRo1CVFSUuS0jIwN6vR7Hjh2T3F5rayv0er3Fi4hcl9EkcKJW4goqzr8hoi6y2wiORqNBZGSk5Zv5+CAsLAwajcbmen/4wx8QFxeHmJgYfP/991i2bBlKS0vx3nvvmbd7ZbgBYP7Z1nbz8vKwcuXKq9kdInKgyrpmtBhMVu2cf0NEXdXtgLN8+XI899xzHfYpKSnpcUHz5883/3nUqFGIjo5GWloaTp48icGDB/domzk5OcjOzjb/rNfrERsb2+Maici+SjXSo6wcwSGirup2wFmyZAnmzJnTYZ9BgwZBrVajtrbWor29vR11dXU259dISUlJAQCUlZVh8ODBUKvV2L9/v0UfrVYLADa3q1KpoFKpuvyeRORcpRrpqy05gkNEXdXtgBMREYGIiIhO+6WmpqK+vh7FxcUYN24cAGD37t0wmUzm0NIVR44cAQBER0ebt/v000+jtrbWfAps165dCAoKwogRI7q5N0Tkikq11iM4Kh854vrxKeJE1DV2m2Q8fPhwZGZmYt68edi/fz+++eYbLFq0CNOnTzdfQXXmzBkMGzbMPCJz8uRJrF69GsXFxTh16hT+85//YNasWbj55psxevRoAMDEiRMxYsQI/PGPf8R3332Hzz77DH/961+xcOFCjtIQeYiSGusJxkOj+kIhlzmhGiJyR3a90d/27dsxbNgwpKWlYdKkSbjpppvw+uuvm5cbDAaUlpaar5JSKpX44osvMHHiRAwbNgxLlizB3XffjY8++si8jkKhwMcffwyFQoHU1FTcf//9mDVrlsV9c4jIfTW0GFB+rsmqfbg6yAnVEJG7kgkhhLOLcDS9Xo/g4GDodDoEBfFDk8iV7C+vw32vFVq1r7zrWsyeEO/4gojIZXTn+5vPoiIil3KsWifZfm0M/zFCRF3HgENELuXoGesJxjIZMDyaAYeIuo4Bh4hcitQITkJ4H/RR2fXReUTkYRhwiMhltBiMOFFrfQ+ca2OCnVANEbkzBhwichk/aRtgNFlf9zCS82+IqJsYcIjIZRyrln5EA0dwiKi7GHCIyGX8cIZXUBFR72DAISKX8d3pequ2/iH+CO2jdHwxROTWGHCIyCW0GIw4rrF+RENSbIjjiyEit8eAQ0Qu4egZneQEYwYcIuoJBhwicglHJE5PAcAYBhwi6gEGHCJyCd9VWU8wlsuAkf05wZiIuo8Bh4hcgtQE42uiAhGg5B2Miaj7GHCIyOnON7aisq7Zqn3swBDHF0NEHoEBh4ic7nuJ01MAMGZAiGMLISKPwYBDRE5XXHFBsn00Aw4R9RADDhE53YFTdVZtfVU+SFQHOqEaIvIEDDhE5FRt7SbJS8SviwuFQi5zfEFE5BEYcIjIqY5W69DabrJqT44LdUI1ROQpGHCIyKkOSpyeAoDkeAYcIuo5BhwicqoDp6wnGPvIZXxEAxFdFQYcInIaIYTkCM61/YN5gz8iuioMOETkNGW1jbjQbLBqv57zb4joKjHgEJHT7Dt5XrI9OT7MwZUQkadhwCEip/mm7JxVm0wG3DCIAYeIrg4DDhE5hdEk8O3P1iM418YEISRA6YSKiMiTMOAQkVMcq9ZB39Ju1X7j4HAnVENEnoYBh4ic4psy6fk3E4Yw4BDR1WPAISKn2HfSev6Nr0KG63mDPyLqBQw4RORwLQaj5AM2xw4M5f1viKhXMOAQkcMVnjyPFoP186cmDO7nhGqIyBPZNeDU1dVh5syZCAoKQkhICObOnYvGxkab/U+dOgWZTCb5euedd8z9pJa/9dZb9twVIupFu4/XSrbflhjp4EqIyFPZdSx45syZqKmpwa5du2AwGJCVlYX58+djx44dkv1jY2NRU1Nj0fb666/j+eefxx133GHRvnnzZmRmZpp/DgkJ6fX6iaj3CSEkA054XxVG9Q92QkVE5InsFnBKSkqQn5+PAwcOIDk5GQDwyiuvYNKkSVi7di1iYmKs1lEoFFCr1RZt77//Pu677z707dvXoj0kJMSqLxG5vhO1jThTf9Gq/dbECMjlMidURESeyG6nqAoLCxESEmIONwCQnp4OuVyOoqKiLm2juLgYR44cwdy5c62WLVy4EOHh4Rg/fjw2bdoEIYTN7bS2tkKv11u8iMg5bJ2eun0YT08RUe+x2wiORqNBZKTlB5aPjw/CwsKg0Wi6tI2NGzdi+PDhmDBhgkX7qlWrcPvttyMgIACff/45HnroITQ2NuKRRx6R3E5eXh5WrlzZsx0hol5VUKK1avORy3DTUN7/hoh6T7dHcJYvX25zIvDl1/Hjx6+6sIsXL2LHjh2SozdPPvkkbrzxRowdOxbLli3D0qVL8fzzz9vcVk5ODnQ6nfl1+vTpq66PiLpPq2/BwYoLVu3J8aEI8vN1QkVE5Km6PYKzZMkSzJkzp8M+gwYNglqtRm2t5VB0e3s76urqujR35t1330VzczNmzZrVad+UlBSsXr0ara2tUKlUVstVKpVkOxE5Vv5RDaTOJmdcy/l0RNS7uh1wIiIiEBER0Wm/1NRU1NfXo7i4GOPGjQMA7N69GyaTCSkpKZ2uv3HjRtx1111deq8jR44gNDSUIYbIxX36Q41ke+ZIBhwi6l12m4MzfPhwZGZmYt68ediwYQMMBgMWLVqE6dOnm6+gOnPmDNLS0rBt2zaMHz/evG5ZWRm++uorfPrpp1bb/eijj6DVanHDDTfAz88Pu3btwjPPPIPHH3/cXrtCRL2gtqEF+yXuXjwuLhTRwf5OqIiIPJld74Ozfft2LFq0CGlpaZDL5bj77ruxbt0683KDwYDS0lI0NzdbrLdp0yYMGDAAEydOtNqmr68v1q9fj8ceewxCCAwZMgQvvPAC5s2bZ89dIaKr9JmN01N3cPSGiOxAJjq6vtpD6fV6BAcHQ6fTISgoyNnlEHmF3/39GxyurLdq/2b57egfwhEcIupcd76/+SwqIrK7stpGyXAzdmAIww0R2QUDDhHZ3b8PVUm2333dAAdXQkTeggGHiOzKaBJ4TyLgKH3kuHO09SNbiIh6AwMOEdnVVyfOQqtvtWqfOCIKwQG8uR8R2QcDDhHZ1dZ9pyTb7xnH01NEZD8MOERkN+XnmrC39KxVuzrID78Z2vlNPImIeooBh4jsZlvhKcn2P6bGQSGXObYYIvIqDDhEZBf6FgPePSg9uXj69bFOqIiIvAkDDhHZxbZ9p9DQ2m7VfufoGPTry+fGEZF9MeAQUa9ram3Hxq/LJZfNmRDv2GKIyCsx4BBRr3vz2wpcaDZYtd80JByjBgQ7oSIi8jYMOETUq/QtBrz21c+Syx6+fYiDqyEib8WAQ0S9av2eMtQ1tVm1j08IQ8qgfk6oiIi8EQMOEfWa03XN2Pz1Kcllj6YNdWwxROTVGHCIqFcIIbDq4x/RZjRZLbs1MQI3Dgl3QlVE5K0YcIioV3zyQw12/ai1alfIZfj/Jg13QkVE5M0YcIjoql1oasNT/zkmuWz69bEYGhXo4IqIyNsx4BDRVRFC4Il3v8O5RuuJxf36KLFkYqITqiIib8eAQ0RXZePX5fiipFZyWe5d1yKsj9LBFRERMeAQ0VX4+sQ5PPu/xyWXpQ2LxJ2jox1cERHRJQw4RNQjP2kbsODNYrSbhNWy8L4q5N09CjIZnxhORM7BgENE3VZ+rgl/3Fgk+TBNmQx4eXoSIgP9nFAZEdElDDhE1C1ltY2Y/nohtPpWyeWPpg3lPW+IyOl8nF0AEbmPwpPn8eCbxdBdtH6QJgBMSYrhHYuJyCUw4BBRp4QQ2Ph1OZ7LPw6D0XrODXDpWVNr7hnNeTdE5BIYcIioQ5Xnm/HXD4/iq5/O2uwzLi4UG2cnQ+WjcGBlRES2MeAQkaQWgxGvf/Uz1u8pQ2u79fOlLkuOC8XmrOsR6OfrwOqIiDrGgENEFhpb27H92wq88X8/S96d+EqTR0fjf+4dAz9fjtwQkWthwCEiCCHwXZUObx88jY+OVEte/n0lmQx4+PahWJw2FHI559wQkethwCHyUo2t7SiuuIA9x2ux60ctztRf7NJ6kYEqvDQtCRN4KTgRuTAGHCIv0NTajhO1jfhJ24BjZ3Q4WHEBJTV6SNyE2CaZDLg/JQ6PZyQi2J/zbYjItdkt4Dz99NP45JNPcOTIESiVStTX13e6jhACubm5eOONN1BfX48bb7wR//jHPzB06C/31airq8PDDz+Mjz76CHK5HHfffTdefvll9O3b1167QuSyDEYTLjS34UKT4b//bcO5pjZU1180v85cuIhqXctVvc91A0OQe+e1GBMb0juFExHZmd0CTltbG+69916kpqZi48aNXVpnzZo1WLduHbZu3YqEhAQ8+eSTyMjIwI8//gg/v0u3fZ85cyZqamqwa9cuGAwGZGVlYf78+dixY4e9dqVTzW3t2Fd23qJN6h/GQli3SveTeheJdSX6dXV74lc9u76trv2TvyvvabPfVdRi1XI1dUi+51VsTwi0mwTajZf/a0K7ScBoEjCYTDBebjeZzH1aDEZcbDOiuc2IZoMRF9va0dz2S9tFg1Giyt5zw6AwPHz7UEwY3I/3tyEityITXf3G6qEtW7Zg8eLFnY7gCCEQExODJUuW4PHHHwcA6HQ6REVFYcuWLZg+fTpKSkowYsQIHDhwAMnJyQCA/Px8TJo0CVVVVYiJielSTXq9HsHBwdDpdAgKCrqq/QOAivNNuOX5vVe9HSJXEOjngylJMbgvORajB4Q4uxwiIrPufH+7zByc8vJyaDQapKenm9uCg4ORkpKCwsJCTJ8+HYWFhQgJCTGHGwBIT0+HXC5HUVERfve730luu7W1Fa2tvzw3R6/X229HiNxQaIAvbhsWifThUbh9WCQv+yYit+cyAUej0QAAoqKiLNqjoqLMyzQaDSIjIy2W+/j4ICwszNxHSl5eHlauXNnLFRO5r8hAFZLjQzEuLgzJcaG4NiYIPgo+e5eIPEe3As7y5cvx3HPPddinpKQEw4YNu6qieltOTg6ys7PNP+v1esTGxjqxIiL7C1Aq0D/EHzEh/hgQ6o+hkX1xTVQghkYFIryvknNqiMijdSvgLFmyBHPmzOmwz6BBg3pUiFqtBgBotVpER0eb27VaLZKSksx9amtrLdZrb29HXV2deX0pKpUKKpWqR3UROZJcBvgo5PCRy+Ajl0Hpo0CA8tLL//J/fX0u/dlXgT4qH4T18UVoHyVCA/776uOL6CB/BPn7MMQQkdfqVsCJiIhARESEXQpJSEiAWq1GQUGBOdDo9XoUFRVhwYIFAIDU1FTU19ejuLgY48aNAwDs3r0bJpMJKSkpdqmrK6KC/LBjnvX7y2D95SL1fSP1FST1xdT1dSUau1BLl+vo4nt2df+lXM32rPfL8X+XUr1kskuhxUchg49cbvlnhQwKmYx3BSYi6iV2m4NTWVmJuro6VFZWwmg04siRIwCAIUOGmO9ZM2zYMOTl5eF3v/sdZDIZFi9ejL/97W8YOnSo+TLxmJgYTJ06FQAwfPhwZGZmYt68ediwYQMMBgMWLVqE6dOnd/kKKnvw81VgwmDe1ZWIiMhV2C3grFixAlu3bjX/PHbsWADAnj17cOuttwIASktLodPpzH2WLl2KpqYmzJ8/H/X19bjpppuQn59vvgcOAGzfvh2LFi1CWlqa+UZ/69ats9duEBERkRuy+31wXFFv3weHiIiI7K8739+8LpSIiIg8DgMOEREReRwGHCIiIvI4DDhERETkcRhwiIiIyOMw4BAREZHHYcAhIiIij8OAQ0RERB6HAYeIiIg8jt0e1eDKLt+8Wa/XO7kSIiIi6qrL39tdeQiDVwachoYGAEBsbKyTKyEiIqLuamhoQHBwcId9vPJZVCaTCdXV1QgMDIRMJuvVbev1esTGxuL06dMe+Zwr7p/78/R95P65P0/fR0/fP8B++yiEQENDA2JiYiCXdzzLxitHcORyOQYMGGDX9wgKCvLY/3EB7p8n8PR95P65P0/fR0/fP8A++9jZyM1lnGRMREREHocBh4iIiDwOA04vU6lUyM3NhUqlcnYpdsH9c3+evo/cP/fn6fvo6fsHuMY+euUkYyIiIvJsHMEhIiIij8OAQ0RERB6HAYeIiIg8DgMOEREReRwGnG56+umnMWHCBAQEBCAkJESyT2VlJSZPnoyAgABERkbiiSeeQHt7e4fbraurw8yZMxEUFISQkBDMnTsXjY2NdtiD7tm7dy9kMpnk68CBAzbXu/XWW636P/jggw6svOvi4+Otan322Wc7XKelpQULFy5Ev3790LdvX9x9993QarUOqrjrTp06hblz5yIhIQH+/v4YPHgwcnNz0dbW1uF6rn781q9fj/j4ePj5+SElJQX79+/vsP8777yDYcOGwc/PD6NGjcKnn37qoEq7Ly8vD9dffz0CAwMRGRmJqVOnorS0tMN1tmzZYnW8/Pz8HFRx9zz11FNWtQ4bNqzDddzp+El9nshkMixcuFCyvzscu6+++gp33nknYmJiIJPJ8MEHH1gsF0JgxYoViI6Ohr+/P9LT03HixIlOt9vd3+PuYsDppra2Ntx7771YsGCB5HKj0YjJkyejra0N+/btw9atW7FlyxasWLGiw+3OnDkTx44dw65du/Dxxx/jq6++wvz58+2xC90yYcIE1NTUWLweeOABJCQkIDk5ucN1582bZ7HemjVrHFR1961atcqi1ocffrjD/o899hg++ugjvPPOO/jyyy9RXV2N3//+9w6qtuuOHz8Ok8mE1157DceOHcOLL76IDRs24C9/+Uun67rq8du5cyeys7ORm5uLQ4cOYcyYMcjIyEBtba1k/3379mHGjBmYO3cuDh8+jKlTp2Lq1Kk4evSogyvvmi+//BILFy7Et99+i127dsFgMGDixIloamrqcL2goCCL41VRUeGgirvv2muvtaj166+/ttnX3Y7fgQMHLPZt165dAIB7773X5jqufuyampowZswYrF+/XnL5mjVrsG7dOmzYsAFFRUXo06cPMjIy0NLSYnOb3f097hFBPbJ582YRHBxs1f7pp58KuVwuNBqNue0f//iHCAoKEq2trZLb+vHHHwUAceDAAXPb//7v/wqZTCbOnDnT67Vfjba2NhERESFWrVrVYb9bbrlFPProo44p6irFxcWJF198scv96+vrha+vr3jnnXfMbSUlJQKAKCwstEOFvWvNmjUiISGhwz6ufPzGjx8vFi5caP7ZaDSKmJgYkZeXJ9n/vvvuE5MnT7ZoS0lJEX/+85/tWmdvqa2tFQDEl19+abOPrc8jV5SbmyvGjBnT5f7ufvweffRRMXjwYGEymSSXu9OxE0IIAOL99983/2wymYRarRbPP/+8ua2+vl6oVCrxr3/9y+Z2uvt73BMcwellhYWFGDVqFKKiosxtGRkZ0Ov1OHbsmM11QkJCLEZE0tPTIZfLUVRUZPeau+M///kPzp8/j6ysrE77bt++HeHh4Rg5ciRycnLQ3NzsgAp75tlnn0W/fv0wduxYPP/88x2eUiwuLobBYEB6erq5bdiwYRg4cCAKCwsdUe5V0el0CAsL67SfKx6/trY2FBcXW/zdy+VypKen2/y7LywstOgPXPqddIdjBVw6XgA6PWaNjY2Ii4tDbGwspkyZYvPzxhWcOHECMTExGDRoEGbOnInKykqbfd35+LW1teHNN9/En/70pw4f7OxOx+7XysvLodFoLI5RcHAwUlJSbB6jnvwe94RXPmzTnjQajUW4AWD+WaPR2FwnMjLSos3HxwdhYWE213GWjRs3IiMjo9OHlf7hD39AXFwcYmJi8P3332PZsmUoLS3Fe++956BKu+6RRx7Bddddh7CwMOzbtw85OTmoqanBCy+8INlfo9FAqVRazcGKiopyueP1a2VlZXjllVewdu3aDvu56vE7d+4cjEaj5O/Y8ePHJdex9Tvp6scKAEwmExYvXowbb7wRI0eOtNkvMTERmzZtwujRo6HT6bB27VpMmDABx44ds/uDhbsrJSUFW7ZsQWJiImpqarBy5Ur85je/wdGjRxEYGGjV352P3wcffID6+nrMmTPHZh93OnZSLh+H7hyjnvwe9wQDDoDly5fjueee67BPSUlJpxPh3ElP9rmqqgqfffYZ3n777U63f+X8oVGjRiE6OhppaWk4efIkBg8e3PPCu6g7+5ednW1uGz16NJRKJf785z8jLy/PZW+l3pPjd+bMGWRmZuLee+/FvHnzOlzX2cePLlm4cCGOHj3a4RwVAEhNTUVqaqr55wkTJmD48OF47bXXsHr1anuX2S133HGH+c+jR49GSkoK4uLi8Pbbb2Pu3LlOrKz3bdy4EXfccQdiYmJs9nGnY+duGHAALFmypMOEDQCDBg3q0rbUarXVTPDLV9eo1Wqb6/x6YlV7ezvq6upsrnO1erLPmzdvRr9+/XDXXXd1+/1SUlIAXBpBcMQX5NUc05SUFLS3t+PUqVNITEy0Wq5Wq9HW1ob6+nqLURytVmu34/Vr3d2/6upq3HbbbZgwYQJef/31br+fo4+fLeHh4VAoFFZXrHX0d69Wq7vV31UsWrTIfMFBd/8l7+vri7Fjx6KsrMxO1fWekJAQXHPNNTZrddfjV1FRgS+++KLbo57udOyAX77XtFotoqOjze1arRZJSUmS6/Tk97hHem02j5fpbJKxVqs1t7322msiKChItLS0SG7r8iTjgwcPmts+++wzl5pkbDKZREJCgliyZEmP1v/6668FAPHdd9/1cmW978033xRyuVzU1dVJLr88yfjdd981tx0/ftxlJxlXVVWJoUOHiunTp4v29vYebcOVjt/48ePFokWLzD8bjUbRv3//DicZ//a3v7VoS01NddlJqiaTSSxcuFDExMSIn376qUfbaG9vF4mJieKxxx7r5ep6X0NDgwgNDRUvv/yy5HJ3O36X5ebmCrVaLQwGQ7fWc/VjBxuTjNeuXWtu0+l0XZpk3J3f4x7V2mtb8hIVFRXi8OHDYuXKlaJv377i8OHD4vDhw6KhoUEIcel/zpEjR4qJEyeKI0eOiPz8fBERESFycnLM2ygqKhKJiYmiqqrK3JaZmSnGjh0rioqKxNdffy2GDh0qZsyY4fD9s+WLL74QAERJSYnVsqqqKpGYmCiKioqEEEKUlZWJVatWiYMHD4ry8nLx4YcfikGDBombb77Z0WV3at++feLFF18UR44cESdPnhRvvvmmiIiIELNmzTL3+fX+CSHEgw8+KAYOHCh2794tDh48KFJTU0VqaqozdqFDVVVVYsiQISItLU1UVVWJmpoa8+vKPu50/N566y2hUqnEli1bxI8//ijmz58vQkJCzFcu/vGPfxTLly839//mm2+Ej4+PWLt2rSgpKRG5ubnC19dX/PDDD87ahQ4tWLBABAcHi71791ocr+bmZnOfX+/jypUrxWeffSZOnjwpiouLxfTp04Wfn584duyYM3ahQ0uWLBF79+4V5eXl4ptvvhHp6ekiPDxc1NbWCiHc//gJcenLeuDAgWLZsmVWy9zx2DU0NJi/6wCIF154QRw+fFhUVFQIIYR49tlnRUhIiPjwww/F999/L6ZMmSISEhLExYsXzdu4/fbbxSuvvGL+ubPf497AgNNNs2fPFgCsXnv27DH3OXXqlLjjjjuEv7+/CA8PF0uWLLFI8Xv27BEARHl5ubnt/PnzYsaMGaJv374iKChIZGVlmUOTK5gxY4aYMGGC5LLy8nKLv4PKykpx8803i7CwMKFSqcSQIUPEE088IXQ6nQMr7pri4mKRkpIigoODhZ+fnxg+fLh45plnLEbbfr1/Qghx8eJF8dBDD4nQ0FAREBAgfve731mEBlexefNmyf9frxy8dcfj98orr4iBAwcKpVIpxo8fL7799lvzsltuuUXMnj3bov/bb78trrnmGqFUKsW1114rPvnkEwdX3HW2jtfmzZvNfX69j4sXLzb/fURFRYlJkyaJQ4cOOb74Lpg2bZqIjo4WSqVS9O/fX0ybNk2UlZWZl7v78RPi0gg8AFFaWmq1zB2P3eXvrF+/Lu+HyWQSTz75pIiKihIqlUqkpaVZ7XtcXJzIzc21aOvo97g3yIQQovdOeBERERE5H++DQ0RERB6HAYeIiIg8DgMOEREReRwGHCIiIvI4DDhERETkcRhwiIiIyOMw4BAREZHHYcAhIiIij8OAQ0RERB6HAYeIiIg8DgMOEREReRwGHCIiIvI4/w/6uwwO4MXe3wAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([-10,10],[0,0],'k--')\n",
    "plt.plot(x, np.tanh(x),linewidth=4,clip_on=False);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Activation Functions: ReLU\n",
    "Rectified Linear Unit: $\\sigma(z) = \\max(0,z)$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAGdCAYAAACyzRGfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/SrBM8AAAACXBIWXMAAA9hAAAPYQGoP6dpAAA4iElEQVR4nO3deXhU9dn/8U8WsgBZCCEJgQBhX0IiiqRA3VFEUFCgann6WGrVWtwBBRUQUcPm0qqPWn8t+DyugIKKCsUFUVkExATCvoclYU9CIPv39wdN2jgngcDMme39uq65LvO9z8y5T06GfJz7zCTAGGMEAABgk0B3NwAAAPwL4QMAANiK8AEAAGxF+AAAALYifAAAAFsRPgAAgK0IHwAAwFaEDwAAYKtgdzfwS5WVlTpw4IAiIiIUEBDg7nYAAMA5MMaosLBQiYmJCgys+7UNjwsfBw4cUFJSkrvbAAAA5yEnJ0ctW7ascxuPCx8RERGSzjQfGRnp5m4AAMC5KCgoUFJSUvXv8bp4XPioGrVERkYSPgAA8DLncskEF5wCAABbET4AAICtCB8AAMBWhA8AAGArwgcAALAV4QMAANiK8AEAAGxF+AAAALYifAAAAFvVO3wsW7ZMN954oxITExUQEKAFCxbUqBtjNHHiRDVv3lzh4eHq16+ftm3b5qx+AQDAeSouq9C+46fc3Ub9w0dRUZHS0tL06quvWtanT5+uv/71r3r99de1atUqNWrUSP3791dxcfEFNwsAAM7f8//cov4vLtO7q/bKGOO2PgLMBew9ICBA8+fP15AhQySdedUjMTFRo0eP1pgxYyRJ+fn5io+P1+zZs3Xbbbed9TELCgoUFRWl/Px8/rYLAABOsmb3MQ1/Y4Wqfutf1iFWU4emqkV0uFMevz6/v516zceuXbuUm5urfv36Va9FRUUpPT1dK1assLxPSUmJCgoKatwAAIDznC6t0Nh5WfrPlxu+23ZE/V9cpiUb82zvx6nhIzc3V5IUHx9fYz0+Pr669ksZGRmKioqqviUlJTmzJQAA/N6MxVu060iRw3ppRaXaNG1oez9uf7fL+PHjlZ+fX33Lyclxd0sAAPiMVTuPatbyXZa1R67tqA7xETZ35OTwkZCQIEnKy6v5Ek5eXl517ZdCQ0MVGRlZ4wYAAC7cqdJyh3FLlR6tonXXZW3tb0pODh/JyclKSEjQV199Vb1WUFCgVatWqXfv3s7cFQAAOItpX2zW3mOOb60NDQ7UzOFpCgoMcENXUnB973Dy5Elt3769+utdu3bp559/VkxMjFq1aqWHHnpIzzzzjDp06KDk5GRNmDBBiYmJ1e+IAQAArrdix1G9tWKPZW1s/05q16yxzR39W73Dx5o1a3TVVVdVf/3II49Iku644w7Nnj1bjz76qIqKinT33XfrxIkT+vWvf61FixYpLCzMeV0DAIBaFZWUa+y8TMtaz9ZNNLJvss0d1XRBn/PhCnzOBwAAF+bJBev19sq9DuthDQL1+QOXqa0LXvVw2+d8AAAA9/ph+xHL4CFJj/bv7JLgUV+EDwAAfERhcZkenZdlWevVJka/79PG3oZqQfgAAMBHPPf5Zu0/cdphPbxBkGYMT1Wgm97d8kuEDwAAfMCyrYf13o/W45ZxAzqrddNGNndUO8IHAABerqC4TI99aD1u+VXbGP3uV61t7qhuhA8AALzcMws36mB+scN6w5AgzRiW5jHjliqEDwAAvNg3Ww5pzpp9lrXHb+iipBj7/3Dc2RA+AADwUvmnyjSulnFL3/ZNNSK9lc0dnRvCBwAAXurphRuVV1DisN4oJEjThqYqIMCzxi1VCB8AAHihLzfm6cOfrMctTw7qqpZNPG/cUoXwAQCAlzlxqlSPz19vWbusQ6xuuzTJ5o7qh/ABAICXmfzpRh0qdBy3RIQGe/S4pQrhAwAAL7I4O1fz1+23rE0Y1FWJ0eE2d1R/hA8AALzEsaJSPVHLuOXKTs00vGdLmzs6P4QPAAC8xKRPsnXkZKnDekRYsKbe4vnjliqEDwAAvMAX6w/q08wDlrWnbuymhKgwmzs6f4QPAAA83NGTJXpywQbL2jWd43TLxS1s7ujCED4AAPBwEz/O1tEix3FLVHgDPXdLd68Zt1QhfAAA4MEWZh3QZ+sPWtYm39RN8ZHeM26pQvgAAMBDHS4s0YRaxi3XdY3X4IsSbe7IOQgfAAB4IGOMnlywXsdPlTnUmjRsoGdv9r5xSxXCBwAAHuiTzANanJ1nWZs8OEXNIkJt7sh5CB8AAHiYQwXFmvhxtmVtQEqCbkxtbnNHzkX4AADAgxhj9Pj8Dco/7ThuiWkUoilDUrx23FKF8AEAgAdZ8PN+fbnJetwyZXCKYht777ilCuEDAAAPkVdQrEm1jFsGpjbXQC8ft1QhfAAA4AGMMRr/0XoVFJc71GIbh2jK4BQ3dOUahA8AADzAvLX79PXmQ5a1Z4Z0V0yjEJs7ch3CBwAAbnYw/7SeXrjRsjb4okRdn5Jgc0euRfgAAMCNjDEa9+F6FVqMW5pFhOqpG7u5oSvXInwAAOBGc9bk6Nuthy1rz93cXU18aNxShfABAICb7D9xWlMWbrKs3dKjha7tGm9zR/YgfAAA4AZnxi1ZOlniOG6JiwjVJB8ct1QhfAAA4Abv/Zij77YdsaxNHdpdUQ0b2NyRfQgfAADYLOfYKT37mfW7W4Zd0lJXd/bNcUsVwgcAADaqrDR67MMsFZVWONQSIsM0YVBXN3RlL8IHAAA2emfVHi3fcdSyNnVod0WF++64pQrhAwAAm+w9ekoZX2y2rN12aZKu7BRnc0fuQfgAAMAGlZVGY+dl6pTFuCUxKkxPDOzihq7cg/ABAIAN/nfFbq3adcyyNm1YqiLCfH/cUoXwAQCAi+0+UqSpi6zHLb9Nb6XLOjSzuSP3InwAAOBCVeOW4rJKh1qL6HA9foP/jFuqED4AAHChWct3a/Xu45a1GcNS1Tg02OaO3I/wAQCAi+w8fFLTaxm3/O5XrdWnfazNHXkGwgcAAC5QUWk0Zm6mSsodxy1JMeEaN6CzG7ryDIQPAABc4O/f79RPe09Y1mYMS1MjPxy3VCF8AADgZNsPndTMf261rP2+Txv9qm1TmzvyLIQPAACcqLyiUqPnZqrUYtzSumlDPXp9Jzd05VkIHwAAONGb3+1SZs4Jh/WAgDPjloYh/jtuqUL4AADASbbmFerFJdbjlj/0TVav5BibO/JMhA8AAJygvKJSY+ZmqrTCcdzSNraRxlzHuKUK4QMAACd4Y9lOZe3Ld1gPDJBmDE9TeEiQG7ryTIQPAAAu0ObcAr30pfW45Y+XtdUlrZvY3JFnI3wAAHAByioqNXpOpsoqjEOtXbNGeuTajm7oyrMRPgAAuAD/880OZR8ocFgPDJBmDk9TWAPGLb9E+AAA4DxlH8jXy19vs6zdc0U79WjFuMUK4QMAgPNQWl6pMXOzVF7pOG7pENdYD/Xr4IauvIPTw0dFRYUmTJig5ORkhYeHq127dpoyZYqMcTw5AAB4q1e+2a5NBx3HLUGBAZo5PE2hwYxbauP0j1mbNm2aXnvtNb311lvq1q2b1qxZo5EjRyoqKkoPPPCAs3cHAIDtNuzP16vfbLes3XtFO6UlRdvbkJdxevhYvny5Bg8erIEDB0qS2rRpo/fee08//vijs3cFAIDtSsorNGZupiosxi2dEyJ0/zXt3dCVd3H62KVPnz766quvtHXrmfc7Z2Zm6vvvv9eAAQMsty8pKVFBQUGNGwAAnurlr7Zrc26hw3ow45Zz5vRXPsaNG6eCggJ17txZQUFBqqio0LPPPqsRI0ZYbp+RkaHJkyc7uw0AAJwuM+eEXvt2h2Xtz1e1V0qLKJs78k5Of+Vjzpw5euedd/Tuu+/qp59+0ltvvaWZM2fqrbfestx+/Pjxys/Pr77l5OQ4uyUAAC5YcVnt45YuzSN131WMW86V01/5GDt2rMaNG6fbbrtNktS9e3ft2bNHGRkZuuOOOxy2Dw0NVWhoqLPbAADAqV76cpu2HTrpsB4cGKDnh6cpJJhPrzhXTv9OnTp1SoGBNR82KChIlZWOf+UPAABvsG7vcf1tmfW45YFrOqhrYqTNHXk3p7/yceONN+rZZ59Vq1at1K1bN61bt04vvPCC/vCHPzh7VwAAuFzVuMVi2qKUFpG698p29jfl5ZwePl5++WVNmDBBf/7zn3Xo0CElJibqnnvu0cSJE529KwAAXO6FJVu143CRw3qDoDPvbmkQxLilvgKMh330aEFBgaKiopSfn6/ISF7GAgC4z9o9xzTs9RWy+k05tn8njeIi02r1+f1NXAMAwMLp0gqNmZtlGTxSW0bpnsvb2t+UjyB8AABgYeY/t2jXEcdxS0hQoJ4fnqZgxi3nje8cAAC/8OOuY/rHD7ssaw9f21Ed4iNs7si3ED4AAPgPp0rLNXZepuW45aKkaN11WbL9TfkYwgcAAP9h+qIt2nP0lMN6SHCgZjJucQq+gwAA/MvKnUc1e/luy9rY6zqpfVxjexvyUYQPAAAkFZWcGbdYuaR1E/3h14xbnIXwAQCApKlfbFbOsdMO66HBgZoxLFVBgQFu6Mo3ET4AAH5v+fYj+r+Veyxrj17fWW2bMW5xJsIHAMCvnSwp19h5WZa1Xm1iNLJPG3sb8gOEDwCAX3vu803af8Jx3BLWIFDTh6UqkHGL0xE+AAB+a9nWw3p31V7L2rjrO6tNbCObO/IPhA8AgF8qKC7TuA+txy3pyTH6795t7G3IjxA+AAB+6bnPNulAfrHDesOQIM0Ylsa4xYUIHwAAv7N0yyG9vzrHsjb+hi5q1bShzR35F8IHAMCv5J8u07gP11vW+rRrqhG9Wtnckf8hfAAA/MqUhRuVW+A4bmkUEqRpQ3l3ix0IHwAAv/H15jzNW7vPsvbEwK5KimHcYgfCBwDAL+Sfqn3cclmHWN3eK8nmjvwX4QMA4Bcmf5qtQ4UlDuuNQ4M1dWiqAgIYt9iF8AEA8Hn/zM7VR+v2W9YmDOqiFtHhNnfk3wgfAACfdryoVI/P32BZu6JjM/2mJ+MWuxE+AAA+7alPs3XkpOO4JSIsWFOHdmfc4gaEDwCAz1q04aA+/vmAZW3Sjd3UPIpxizsQPgAAPunoyRI9Ucu45erOcRp6cQubO0IVwgcAwCdN/CRbR4tKHdYjw4KVcQvjFncifAAAfM5nWQf1WdZBy9rkwd0UHxlmc0f4T4QPAIBPOXKyRBM+th63XNs1XkMuYtziboQPAIDPMMZowoINOmYxbolu2EDP3pzCuMUDED4AAD7j06yD+mJDrmVt8k3dFBfBuMUTED4AAD7hUGGxJtYybrm+W4JuSku0uSPUhvABAPB6xhg9MX+DTpwqc6jFNArRM4xbPArhAwDg9T7++YCWbMyzrE0ZnKLYxqE2d4S6ED4AAF4tr6BYkz7JtqwN7N5cA1Ob29wRzobwAQDwWsYYPf7ReuWfdhy3NG0UoqcHd3NDVzgbwgcAwGt9+NN+fbX5kGXtmSEpasq4xSMRPgAAXik3v1iTP7Uet9yUlqgB3Rm3eCrCBwDA6xhjNO6jLBUWlzvUYhuHavJNjFs8GeEDAOB15q7Zp6VbDlvWnrs5RU0ahdjcEeqD8AEA8CoHTpzWlIUbLWs392ih67ol2NwR6ovwAQDwGsYYPfZhlgpLHMctcRGhmnRjVzd0hfoifAAAvMb7q3P03bYjlrWMW7oruiHjFm9A+AAAeIV9x0/pmVrGLUMvbqlrusTb3BHOF+EDAODxKiuNHp2XpaLSCodafGSoJjJu8SqEDwCAx3vnx71avuOoZW3q0FRFhTewuSNcCMIHAMCj5Rw7pYzPN1nWbu2ZpKs6xdncES4U4QMA4LEqK43GzsvUKYtxS/OoMD0xqIsbusKFInwAADzW/63co5U7j1nWpg1NVWQY4xZvRPgAAHikPUeLNPWLzZa123u10uUdm9ncEZyF8AEA8DiVlUZj52bpdJnjuKVFdLieGMi4xZsRPgAAHmf28t36cbf1uGX6sFQ1Dg22uSM4E+EDAOBRdh4+qemLrcct//WrVurbPtbmjuBshA8AgMeoqDQaOy9LxWWVDrWWTcI1fgDjFl9A+AAAeIxZP+zS2j3HLWszhqWpEeMWn0D4AAB4hO2HTmrG4i2Wtd/3aaPe7Zra3BFchfABAHC7ikqjMXMzVVLuOG5p3bShHr2+kxu6gqu4JHzs379f//Vf/6WmTZsqPDxc3bt315o1a1yxKwCAD3jzu536OeeEw3pAwJlxS8MQxi2+xOln8/jx4+rbt6+uuuoqffHFF2rWrJm2bdumJk2aOHtXAAAfsC2vUC/8c6tlbWSfZPVKjrG5I7ia08PHtGnTlJSUpFmzZlWvJScnO3s3AAAfUF5RqTFzM1Va4ThuSY5tpLH9Gbf4IqePXT755BP17NlTw4cPV1xcnHr06KE333yz1u1LSkpUUFBQ4wYA8A9vLNupzH35Dutnxi2pCg8JckNXcDWnh4+dO3fqtddeU4cOHbR48WLde++9euCBB/TWW29Zbp+RkaGoqKjqW1JSkrNbAgB4oM25BXrpS+txyx9/nayebRi3+KoAY4xx5gOGhISoZ8+eWr58efXaAw88oNWrV2vFihUO25eUlKikpKT664KCAiUlJSk/P1+RkZHObA0A4CHKKip18//8oA37HV/tbtuskT5/4DKFNeBVD29SUFCgqKioc/r97fRXPpo3b66uXbvWWOvSpYv27t1ruX1oaKgiIyNr3AAAvu31pTssg0dggDRzeBrBw8c5PXz07dtXW7bU/JCYrVu3qnXr1s7eFQDAC208UKC/fr3Nsnb35e10cSveHenrnB4+Hn74Ya1cuVLPPfectm/frnfffVd/+9vfNGrUKGfvCgDgZUrLz7y7pazCceLfIa6xHurXwQ1dwW5ODx+XXnqp5s+fr/fee08pKSmaMmWKXnrpJY0YMcLZuwIAeJlXv9mujQcdxy1BgQGMW/yISz4ybtCgQRo0aJArHhoA4KU27M/Xq99st6z96Yq2SkuKtrchuA1/2wUA4HJV45bySsdxS6f4CD1wDeMWf0L4AAC43Mtfb9Pm3EKH9apxS2gw4xZ/QvgAALhU1r4T+p+lOyxro65sp+4to2zuCO5G+AAAuExJeYXGzM1UhcW4pXNChO67mnGLPyJ8AABc5i9fbtPWvJMO68GBAXr+N2kKCebXkD/irAMAXOLnnBN6/Vvrccv9V3dQt0TGLf6K8AEAcLrisgqNnvOzLKYt6pYYqT9f1c7+puAxCB8AAKd7cclW7Thc5LDeIOjMu1saBPHrx59x9gEATrV2z3H97budlrUHr+mgLs35A6L+jvABAHCa4rIKjZ2bKWMxbuneIkp/uoJxCwgfAAAnmrl4i3YecRy3hAQF6vnfpCmYcQtE+AAAOMnq3cf09x92WdYeuraDOsZH2NwRPBXhAwBwwU6X1j5uSUuK1t2XtbW/KXgswgcA4IJNX7xZu4+eclgPCQ7U88NTGbegBn4aAAAXZOXOo5r1w27L2pjrOqp9HOMW1ET4AACct6KScj06L8uydnGraN35a8YtcET4AACct2mLNmvvMcdxS2hwoGYMT1NQYIAbuoKnI3wAAM7L8u1H9L8r9ljWxvbvpHbNGtvcEbwF4QMAUG8nS8r16IfW45ZL2zTRyL7JNncEb0L4AADUW8bnm7Tv+GmH9bAGgZoxjHEL6kb4AADUy3fbDuudVXsta49d31ltYhvZ3BG8DeEDAHDOCovL9Fgt727plRyjO3q3sbcheCXCBwDgnD33+SYdyC92WG8YEqSZw9IUyLgF54DwAQA4J99uPaz3fsyxrI0f0Fmtmja0uSN4K8IHAOCs8k/XPm7p3bapRqS3trkjeDPCBwDgrJ5ZuFG5BY7jlkYhQZo+LJVxC+qF8AEAqNPXm/M0d+0+y9rjA7soKYZxC+qH8AEAqFX+qTKN/2i9Ze3X7WP1216tbO4IvoDwAQCo1eSF2corKHFYbxwarGnDUhUQwLgF9Uf4AABYWrIxTx/9tN+y9uTALmoRHW5zR/AVhA8AgIMTp0r1+HzrccvlHZvp1kuTbO4IvoTwAQBw8NQn2Tpc6DhuiQgL1rSh3Rm34IIQPgAANSzakKsFPx+wrE0c1FXNoxi34MIQPgAA1Y4VlerJBdbjlqs7x2nYJS1t7gi+iPABAKg28eMNOnKy1GE9MixYz93MuAXOQfgAAEiSPl9/UAuzDlrWnrqpmxKiwmzuCL6K8AEA0JGTJXpywQbLWr8u8bq5RwubO4IvI3wAgJ8zxmjCgg06VuQ4bokKb6Dnbk5h3AKnInwAgJ9bmHVQX2zItaw9Pbib4iIZt8C5CB8A4McOF5Zo4sfW45b+3eJ1U1qizR3BHxA+AMBPGWP05IL1On6qzKHWpGEDPTOEd7fANQgfAOCnPsk8oMXZeZa1pwenqFlEqM0dwV8QPgDADx0qKNbEj7Mtazd0T9Cg1OY2dwR/QvgAAD9jjNHj89cr/7TjuKVpoxBNGcy7W+BahA8A8DPz1+3Xl5sOWdaeGZKipo0Zt8C1CB8A4Edy84v11CfW45Yb0xI1oDvjFrge4QMA/IQxRuM/ylJBcblDLbZxiCbf1M0NXcEfET4AwE/MXbtP32w5bFl7Zkh3xTQKsbkj+CvCBwD4gQMnTmvKpxsta0MuStT1KQk2dwR/RvgAAB9njNG4j9arsMRx3NIsIlRPMW6BzQgfAODjPlido2VbrcctGTd3V3RDxi2wF+EDAHzYvuOn9Mxnmyxrt1zcQv26xtvcEUD4AACfZYzRuA/X66TFuCU+MlSTBjFugXsQPgDAR7374159v/2IZW3qLamKatjA5o6AMwgfAOCDco6d0rO1jFt+07OlruocZ3NHwL8RPgDAx1RWGj06L0unSiscas2jwvTkoK5u6Ar4N5eHj6lTpyogIEAPPfSQq3cFAJD09qo9WrHzqGVt6tBURYYxboF7uTR8rF69Wm+88YZSU1NduRsAwL/sOVqkjM83W9Zu75WkKzo2s7kjwJHLwsfJkyc1YsQIvfnmm2rSpImrdgMA+JfKSqOx87J0usxx3NIiOlyP39DFDV0BjlwWPkaNGqWBAweqX79+dW5XUlKigoKCGjcAQP29tWK3ftx1zLI2bWiqIhi3wEMEu+JB33//ff30009avXr1WbfNyMjQ5MmTXdEGAPiNXUeKNG2R9bhlRHor/bpDrM0dAbVz+isfOTk5evDBB/XOO+8oLCzsrNuPHz9e+fn51becnBxntwQAPq2i0mjs3EwVl1U61Fo2Cdd4xi3wME5/5WPt2rU6dOiQLr744uq1iooKLVu2TK+88opKSkoUFBRUXQsNDVVoaKiz2wAAvzHrh11as+e4ZW36sFQ1DnXJi9zAeXP6T+Q111yj9evX11gbOXKkOnfurMcee6xG8AAAXJgdh09qxuItlrU7erdWn3aMW+B5nB4+IiIilJKSUmOtUaNGatq0qcM6AOD8VVQajZmbqZJyx3FLq5iGemxAZzd0BZwdn3AKAF7q/323U+v2nrCszRiWqoYhjFvgmWz5yVy6dKkduwEAv7H9UKGeX7LVsjaybxult21qc0fAueOVDwDwMuUVlRo9N0ulFuOWNk0b6tH+jFvg2QgfAOBl/vbdTmXmnHBYDwiQZg5PU3gIF/bDsxE+AMCLbMkt1EtLtlnW7uybrJ5tYmzuCKg/wgcAeImyikqNmZup0grHcUvb2EYa07+TG7oC6o/wAQBe4o1vd2j9/nyH9cAAaeZv0hTWgHELvAPhAwC8wKaDBfrLV9bjlrsub6uLW/HXw+E9CB8A4OHKKio1ek6myiqMQ619XGM93K+jG7oCzh/hAwA83KvfbNfGgwUO64H/encL4xZ4G8IHAHiw7AP5euXr7Za1P13RThclRdvbEOAEhA8A8FCl5WfGLeWVjuOWjvGN9WC/Dm7oCrhwhA8A8FCvfL1Nm3MLHdaDAgP0/PCLFBrMuAXeifABAB5o/b58vbp0h2Xtz1e2U/eWUTZ3BDgP4QMAPExJeYVGz/1ZFRbjls4JEbr/asYt8G6EDwDwMH/9apu25p10WA8ODNDM4WkKCeafbng3foIBwINk5pzQa7WMW0Zd1V4pLRi3wPsRPgDAQxSXVWj03ExZTFvUtXmkRl3V3v6mABcgfACAh3jxy63afshx3NIgiHELfAs/yQDgAX7ae1xvLttpWXvg6g7qmhhpc0eA6xA+AMDNissqNKaWcUv3FlH605Xt7G8KcCHCBwC42fP/3KKdh4sc1kOCAjVzeJoaBPFPNXwLP9EA4EZrdh/T//t+l2XtwX4d1CkhwuaOANcjfACAm5wuPTNuMRbjlrSWUbrn8rb2NwXYgPABAG4yY/EW7T56ymE9JPjMuCWYcQt8FD/ZAOAGq3Ye1azl1uOWR67tqA7xjFvguwgfAGCzU6XlGjsvy3Lc0qNVtO66jHELfBvhAwBsNu2Lzdp7zHHcEvqvcUtQYIAbugLsQ/gAABut2HFUb63YY1kb27+T2jVrbHNHgP0IHwBgk6KSco2dl2lZ69m6iUb2Tba5I8A9CB8AYJOMLzZp3/HTDuthDQI1g3EL/AjhAwBs8P22I3p75V7L2qP9Oys5tpHNHQHuQ/gAABcrLC7TYx9mWdZ6tYnR7/u0sbchwM0IHwDgYs99vln7TziOW8IbBGnG8FQFMm6BnyF8AIALLdt6WO/9aD1uGTegs1o3ZdwC/0P4AAAXKahj3PKrtjH63a9a29wR4BkIHwDgIs8s3KiD+cUO6w1DgjRjWBrjFvgtwgcAuMA3Ww5pzpp9lrXHb+iipJiGNncEeA7CBwA4Wf6pMo2rZdzy6/axGpHeyuaOAM9C+AAAJ3t64UblFZQ4rDcODdbUod0VEMC4Bf6N8AEATvTlxjx9+JP1uOWJgV3UsgnjFoDwAQBOcuJUqcbPX29Zu6xDrG67NMnmjgDPRPgAACeZ/OlGHS50HLdEhAZr2tBUxi3AvxA+AMAJFmfnav66/Za1CYO6KjE63OaOAM9F+ACAC3SsqFRP1DJuubJTMw3v2dLmjgDPRvgAgAs06ZNsHTlZ6rAeERasqbcwbgF+ifABABfgi/UH9WnmAcvaUzd2U0JUmM0dAZ6P8AEA5+noyRI9uWCDZa1flzjdcnELmzsCvAPhAwDO08SPs3W0yHHcEhXeQM/dzIeJAbUhfADAeViYdUCfrT9oWZt8UzfFRTJuAWpD+ACAejpcWKIJtYxbrusar8EXJdrcEeBdCB8AUA/GGD25YL2OnypzqDVp2EDPMm4BzorwAQD18EnmAS3OzrOsTR6comYRoTZ3BHgfwgcAnKNDBcWa+HG2ZW1ASoJuTG1uc0eAdyJ8AMA5MMbo8fkblH/acdwS0yhEU4akMG4BzhHhAwDOwYKf9+vLTdbjlimDUxTbmHELcK4IHwBwFnkFxZpUy7hlUGpzDWTcAtQL4QMA6mCM0fiP1quguNyhFts4RE8PTnFDV4B3c3r4yMjI0KWXXqqIiAjFxcVpyJAh2rJli7N3AwC2mLd2n77efMiy9syQ7oppFGJzR4D3c3r4+PbbbzVq1CitXLlSS5YsUVlZma677joVFRU5e1cA4FIH80/r6YUbLWuDL0rU9SkJNncE+IZgZz/gokWLanw9e/ZsxcXFae3atbr88sudvTsAcAljjMZ9uF6FFuOWZhGheurGbm7oCvANTg8fv5Sfny9JiomJsayXlJSopKSk+uuCggJXtwQAZzVnTY6+3XrYsvbczd3VhHELcN5cesFpZWWlHnroIfXt21cpKdYXZWVkZCgqKqr6lpSU5MqWAOCs9p84rSkLN1nWbunRQtd2jbe5I8C3uDR8jBo1Shs2bND7779f6zbjx49Xfn5+9S0nJ8eVLQFAnc6MW7J0ssRx3BIXEapJjFuAC+aysct9992nhQsXatmyZWrZsmWt24WGhio0lA/nAeAZ3vsxR99tO2JZmzq0u6IaNrC5I8D3OD18GGN0//33a/78+Vq6dKmSk5OdvQsAcImcY6f07GfW724ZdklLXd2ZcQvgDE4PH6NGjdK7776rjz/+WBEREcrNzZUkRUVFKTw83Nm7AwCnqKw0euzDLBWVVjjUEiLDNGFQVzd0Bfgmp1/z8dprryk/P19XXnmlmjdvXn374IMPnL0rAHCad1bt0fIdRy1rU4d2V1Q44xbAWVwydgEAb7L36CllfLHZsnbbpUm6slOczR0Bvo2/7QLAr1VWGo2dl6lTFuOWxKgwPTGwixu6Anwb4QOAX/vfFbu1atcxy9q0YamKCGPcAjgb4QOA39p9pEhTF1mPW36b3kqXdWhmc0eAfyB8APBLVeOW4rJKh1qL6HA9fgPjFsBVCB8A/NKs5bu1evdxy9qMYalqHOryP30F+C3CBwC/s/PwSU2vZdzyu1+1Vp/2sTZ3BPgXwgcAv1JRaTRmbqZKyh3HLUkx4Ro3oLMbugL8C+EDgF/5+/c79dPeE5a1GcPS1IhxC+ByhA8AfmP7oZOa+c+tlrXf92mjX7VtanNHgH8ifADwC+UVlRo9N1OlFuOWNk0b6tHrO7mhK8A/ET4A+IU3v9ulzJwTDusBAdKM4WlqGMK4BbAL4QOAz9uaV6gXl1iPW/7QN1mXtomxuSPAvxE+APi08opKjZmbqdIKx3FL29hGGnMd4xbAboQPAD7tjWU7lbUv32E98F/jlvCQIDd0Bfg3wgcAn7U5t0AvfWk9bvnjZW11SesmNncEQCJ8APBRZRWVGj0nU2UVxqHWrlkjPXJtRzd0BUAifADwUf/zzQ5lHyhwWA8MkGYOT1NYA8YtgLsQPgD4nOwD+Xr5622WtXuuaKcerRi3AO5E+ADgU0rLKzVmbpbKKx3HLR3jG+uhfh3c0BWA/0T4AOBTXvlmuzYddBy3BAUGaObwNIUGM24B3I3wAcBnbNifr1e/2W5Zu/eKdkptGW1vQwAsET4A+ISS8gqNnpOpCotxS+eECN1/TXs3dAXACuEDgE94+avt2pJX6LAezLgF8DiEDwBeLzPnhF77dodl7c9XtVdKiyibOwJQF8IHAK9WXFahMXOtxy1dmkfqvqsYtwCehvABwKu99OU2bTt00mE9ODBAzw9PU0gw/8wBnoZnJQCvtW7vcf1tmfW45YFrOqhrYqTNHQE4F4QPAF6patxiMW1RSotI3XtlO/ubAnBOCB8AvNILS7Zqx+Eih/UGQWfe3dIgiH/eAE/FsxOA11m755je/G6nZe2hfh3VOYFxC+DJCB8AvMrp0gqNmZslYzFuSW0ZpXsub2t/UwDqhfABwKvM/OcW7TriOG4JCQrU88PTFMy4BfB4PEsBeI0fdx3TP37YZVl7+NqO6hAfYXNHAM4H4QOAVzhVWq6x8zItxy0XJUXrrsuS7W8KwHkhfADwCtMXbdGeo6cc1kOCAzWTcQvgVXi2AvB4K3ce1ezluy1rY6/rpPZxje1tCMAFIXwA8GhFJWfGLVYuad1Ef/g14xbA2xA+AHi0qV9sVs6x0w7rocGBmjEsVUGBAW7oCsCFIHwA8FjLtx/R/63cY1l79PrOatuMcQvgjQgfADzSyZJyjZ2XZVnr1SZGI/u0sbchAE5D+ADgkZ77fJP2n3Act4Q3CNL0YakKZNwCeC3CBwCPs2zrYb27aq9l7bHrO6lNbCObOwLgTIQPAB6loLhM4z60HrekJ8fov3u3sbchAE5H+ADgUZ5duEkH8osd1huGBGnGsDTGLYAPIHwA8BhLtxzSB2tyLGvjb+iiVk0b2twRAFcgfADwCPmnyzTuw/WWtT7tmmpEr1Y2dwTAVQgfADzClIUblVvgOG5pFBKkaUN5dwvgSwgfANzu6815mrd2n2XtiYFdlRTDuAXwJYQPAG6Vf6r2cctlHWJ1e68kmzsC4GqEDwBuNfnTbB0qLHFYjwgN1rShqQoIYNwC+BrCBwC3+Wd2rj5at9+y9uSgLkqMDre5IwB2IHwAcIvjRaV6fP4Gy9oVHZvpNz0ZtwC+ivABwC0mfZKtIyctxi1hwZo6tDvjFsCHET4A2G7RhoP6JPOAZW3Sjd3UPIpxC+DLCB8AbHX0ZImeqGXccnXnOA29uIXNHQGwG+EDgK0mfpKto0WlDuuRYcHKuIVxC+APXBY+Xn31VbVp00ZhYWFKT0/Xjz/+6KpdAfASn2Ud1GdZBy1rkwd3U3xkmM0dAXCHYFc86AcffKBHHnlEr7/+utLT0/XSSy+pf//+2rJli+Li4lyxyzp9t+2wSsoqbd8vgH87kH9aEz/Otqxd2zVeQy5i3AL4iwBjjHH2g6anp+vSSy/VK6+8IkmqrKxUUlKS7r//fo0bN67O+xYUFCgqKkr5+fmKjIx0Sj+9M77SQYs/0Q3A/aIbNtA/H75ccRG86gF4s/r8/nb62KW0tFRr165Vv379/r2TwED169dPK1ascNi+pKREBQUFNW4A/Mfkm7oRPAA/4/TwceTIEVVUVCg+Pr7Genx8vHJzcx22z8jIUFRUVPUtKYkPFgL8xfXdEnRTWqK72wBgM7e/22X8+PHKz8+vvuXk5Li7JQA2aBXTUM/cnMK7WwA/5PQLTmNjYxUUFKS8vLwa63l5eUpISHDYPjQ0VKGhoc5uA4AHu6JjM80YnqrYxjz3AX/k9PAREhKiSy65RF999ZWGDBki6cwFp1999ZXuu+8+Z+/unLzy2x4qKefdLoAnSIgMU9tmjd3dBgA3cslbbR955BHdcccd6tmzp3r16qWXXnpJRUVFGjlypCt2d1aXtI5xy34BAIAjl4SPW2+9VYcPH9bEiROVm5uriy66SIsWLXK4CBUAAPgfl3zOx4Vwxed8AAAA13Lr53wAAADUhfABAABsRfgAAAC2InwAAABbET4AAICtCB8AAMBWhA8AAGArwgcAALAV4QMAANjKJR+vfiGqPnC1oKDAzZ0AAIBzVfV7+1w+ON3jwkdhYaEkKSkpyc2dAACA+iosLFRUVFSd23jc33aprKzUgQMHFBERoYCAAKc+dkFBgZKSkpSTk+OTfzfG149P8v1j5Pi8n68fI8fn/Vx1jMYYFRYWKjExUYGBdV/V4XGvfAQGBqply5Yu3UdkZKTP/lBJvn98ku8fI8fn/Xz9GDk+7+eKYzzbKx5VuOAUAADYivABAABs5VfhIzQ0VJMmTVJoaKi7W3EJXz8+yfePkePzfr5+jByf9/OEY/S4C04BAIBv86tXPgAAgPsRPgAAgK0IHwAAwFaEDwAAYCufCh/PPvus+vTpo4YNGyo6Otpym71792rgwIFq2LCh4uLiNHbsWJWXl9f5uMeOHdOIESMUGRmp6Oho3XnnnTp58qQLjqB+li5dqoCAAMvb6tWra73flVde6bD9n/70Jxs7P3dt2rRx6HXq1Kl13qe4uFijRo1S06ZN1bhxYw0dOlR5eXk2dVw/u3fv1p133qnk5GSFh4erXbt2mjRpkkpLS+u8nyefw1dffVVt2rRRWFiY0tPT9eOPP9a5/dy5c9W5c2eFhYWpe/fu+vzzz23qtP4yMjJ06aWXKiIiQnFxcRoyZIi2bNlS531mz57tcK7CwsJs6rh+nnrqKYdeO3fuXOd9vOn8Sdb/pgQEBGjUqFGW23v6+Vu2bJluvPFGJSYmKiAgQAsWLKhRN8Zo4sSJat68ucLDw9WvXz9t27btrI9b3+dxfflU+CgtLdXw4cN17733WtYrKio0cOBAlZaWavny5Xrrrbc0e/ZsTZw4sc7HHTFihLKzs7VkyRItXLhQy5Yt09133+2KQ6iXPn366ODBgzVuf/zjH5WcnKyePXvWed+77rqrxv2mT59uU9f19/TTT9fo9f77769z+4cffliffvqp5s6dq2+//VYHDhzQLbfcYlO39bN582ZVVlbqjTfeUHZ2tl588UW9/vrrevzxx896X088hx988IEeeeQRTZo0ST/99JPS0tLUv39/HTp0yHL75cuX6/bbb9edd96pdevWaciQIRoyZIg2bNhgc+fn5ttvv9WoUaO0cuVKLVmyRGVlZbruuutUVFRU5/0iIyNrnKs9e/bY1HH9devWrUav33//fa3betv5k6TVq1fXOL4lS5ZIkoYPH17rfTz5/BUVFSktLU2vvvqqZX369On661//qtdff12rVq1So0aN1L9/fxUXF9f6mPV9Hp8X44NmzZploqKiHNY///xzExgYaHJzc6vXXnvtNRMZGWlKSkosH2vjxo1Gklm9enX12hdffGECAgLM/v37nd77hSgtLTXNmjUzTz/9dJ3bXXHFFebBBx+0p6kL1Lp1a/Piiy+e8/YnTpwwDRo0MHPnzq1e27Rpk5FkVqxY4YIOnW/69OkmOTm5zm089Rz26tXLjBo1qvrriooKk5iYaDIyMiy3/81vfmMGDhxYYy09Pd3cc889Lu3TWQ4dOmQkmW+//bbWbWr798gTTZo0yaSlpZ3z9t5+/owx5sEHHzTt2rUzlZWVlnVvOn+SzPz586u/rqysNAkJCWbGjBnVaydOnDChoaHmvffeq/Vx6vs8Ph8+9crH2axYsULdu3dXfHx89Vr//v1VUFCg7OzsWu8THR1d45WEfv36KTAwUKtWrXJ5z/XxySef6OjRoxo5cuRZt33nnXcUGxurlJQUjR8/XqdOnbKhw/MzdepUNW3aVD169NCMGTPqHJOtXbtWZWVl6tevX/Va586d1apVK61YscKOdi9Yfn6+YmJizrqdp53D0tJSrV27tsb3PjAwUP369av1e79ixYoa20tnnpPedK4knfV8nTx5Uq1bt1ZSUpIGDx5c6783nmDbtm1KTExU27ZtNWLECO3du7fWbb39/JWWlurtt9/WH/7whzr/kKk3nb//tGvXLuXm5tY4R1FRUUpPT6/1HJ3P8/h8eNwflnOl3NzcGsFDUvXXubm5td4nLi6uxlpwcLBiYmJqvY+7/P3vf1f//v3P+of5fvvb36p169ZKTExUVlaWHnvsMW3ZskUfffSRTZ2euwceeEAXX3yxYmJitHz5co0fP14HDx7UCy+8YLl9bm6uQkJCHK75iY+P97jzZWX79u16+eWXNXPmzDq388RzeOTIEVVUVFg+xzZv3mx5n9qek95wriorK/XQQw+pb9++SklJqXW7Tp066R//+IdSU1OVn5+vmTNnqk+fPsrOznb5H9Gsr/T0dM2ePVudOnXSwYMHNXnyZF122WXasGGDIiIiHLb35vMnSQsWLNCJEyf0+9//vtZtvOn8/VLVeajPOTqf5/H58PjwMW7cOE2bNq3ObTZt2nTWi6K8yfkc8759+7R48WLNmTPnrI//n9erdO/eXc2bN9c111yjHTt2qF27duff+Dmqz/E98sgj1WupqakKCQnRPffco4yMDI/++OPzOYf79+/X9ddfr+HDh+uuu+6q877uPoeQRo0apQ0bNtR5TYQk9e7dW717967+uk+fPurSpYveeOMNTZkyxdVt1suAAQOq/zs1NVXp6elq3bq15syZozvvvNONnbnG3//+dw0YMECJiYm1buNN58+beHz4GD16dJ2pVJLatm17To+VkJDgcMVu1bsgEhISar3PLy+yKS8v17Fjx2q9z4U6n2OeNWuWmjZtqptuuqne+0tPT5d05v+67fjFdSHnND09XeXl5dq9e7c6derkUE9ISFBpaalOnDhR49WPvLw8l50vK/U9xgMHDuiqq65Snz599Le//a3e+7P7HFqJjY1VUFCQwzuL6vreJyQk1Gt7T3HfffdVX3xe3//7bdCggXr06KHt27e7qDvniY6OVseOHWvt1VvPnyTt2bNHX375Zb1fLfSm81d1HvLy8tS8efPq9by8PF100UWW9zmf5/F5cdrVIx7kbBec5uXlVa+98cYbJjIy0hQXF1s+VtUFp2vWrKleW7x4sUddcFpZWWmSk5PN6NGjz+v+33//vZFkMjMzndyZ87399tsmMDDQHDt2zLJedcHpvHnzqtc2b97s0Rec7tu3z3To0MHcdtttpry8/Lwew1POYa9evcx9991X/XVFRYVp0aJFnRecDho0qMZa7969PfaCxcrKSjNq1CiTmJhotm7del6PUV5ebjp16mQefvhhJ3fnfIWFhaZJkybmL3/5i2Xd287ff5o0aZJJSEgwZWVl9bqfJ58/1XLB6cyZM6vX8vPzz+mC0/o8j8+rV6c9kgfYs2ePWbdunZk8ebJp3LixWbdunVm3bp0pLCw0xpz5oUlJSTHXXXed+fnnn82iRYtMs2bNzPjx46sfY9WqVaZTp05m37591WvXX3+96dGjh1m1apX5/vvvTYcOHcztt99u+/HV5ssvvzSSzKZNmxxq+/btM506dTKrVq0yxhizfft28/TTT5s1a9aYXbt2mY8//ti0bdvWXH755Xa3fVbLly83L774ovn555/Njh07zNtvv22aNWtm/vu//7t6m18enzHG/OlPfzKtWrUyX3/9tVmzZo3p3bu36d27tzsO4az27dtn2rdvb6655hqzb98+c/Dgwerbf27jLefw/fffN6GhoWb27Nlm48aN5u677zbR0dHV7zD73e9+Z8aNG1e9/Q8//GCCg4PNzJkzzaZNm8ykSZNMgwYNzPr16911CHW69957TVRUlFm6dGmNc3Xq1KnqbX55jJMnTzaLFy82O3bsMGvXrjW33XabCQsLM9nZ2e44hDqNHj3aLF261Ozatcv88MMPpl+/fiY2NtYcOnTIGOP9569KRUWFadWqlXnsscccat52/goLC6t/10kyL7zwglm3bp3Zs2ePMcaYqVOnmujoaPPxxx+brKwsM3jwYJOcnGxOnz5d/RhXX321efnll6u/Ptvz2Bl8KnzccccdRpLD7ZtvvqneZvfu3WbAgAEmPDzcxMbGmtGjR9dIvt98842RZHbt2lW9dvToUXP77bebxo0bm8jISDNy5MjqQOMJbr/9dtOnTx/L2q5du2p8D/bu3Wsuv/xyExMTY0JDQ0379u3N2LFjTX5+vo0dn5u1a9ea9PR0ExUVZcLCwkyXLl3Mc889V+NVql8enzHGnD592vz5z382TZo0MQ0bNjQ333xzjV/mnmTWrFmWP7P/+aKkt53Dl19+2bRq1cqEhISYXr16mZUrV1bXrrjiCnPHHXfU2H7OnDmmY8eOJiQkxHTr1s189tlnNnd87mo7V7Nmzare5pfH+NBDD1V/P+Lj480NN9xgfvrpJ/ubPwe33nqrad68uQkJCTEtWrQwt956q9m+fXt13dvPX5XFixcbSWbLli0ONW87f1W/s355qzqGyspKM2HCBBMfH29CQ0PNNddc43DcrVu3NpMmTaqxVtfz2BkCjDHGeUMcAACAuvnV53wAAAD3I3wAAABbET4AAICtCB8AAMBWhA8AAGArwgcAALAV4QMAANiK8AEAAGxF+AAAALYifAAAAFsRPgAAgK0IHwAAwFb/H66MigWVtnt4AAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x,x*(x > 0),clip_on=False,linewidth=4);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Networks\n",
    "\n",
    "<img src=\"imgs/tikz11.png\">\n",
    "\n",
    "Terminology alert: networks of neurons are sometimes called *multilayer perceptrons*, despite not using the step function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"ibpcnt\" style=\"width: 500px\"></div>\n",
       "<script>\n",
       "\n",
       "    var divid = '#ibpcnt';\n",
       "\tjQuery(divid).asker({\n",
       "\t    id: divid,\n",
       "\t    question: \"A network has 10 input nodes, two hidden layers each with 10 neurons, and 10 output neurons.  How many parameters does training have to estimate?\",\n",
       "\t\tanswers: [\"30\",\"100\",\"300\",\"330\",\"600\"],\n",
       "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
       "\t\tcharter: chartmaker})\n",
       "    \n",
       "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
       "\n",
       "\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div id=\"ibpcnt\" style=\"width: 500px\"></div>\n",
    "<script>\n",
    "\n",
    "    var divid = '#ibpcnt';\n",
    "\tjQuery(divid).asker({\n",
    "\t    id: divid,\n",
    "\t    question: \"A network has 10 input nodes, two hidden layers each with 10 neurons, and 10 output neurons.  How many parameters does training have to estimate?\",\n",
    "\t\tanswers: [\"30\",\"100\",\"300\",\"330\",\"600\"],\n",
    "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
    "\t\tcharter: chartmaker})\n",
    "    \n",
    "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
    "\n",
    "\n",
    "</script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Networks\n",
    "\n",
    "<img src=\"imgs/tikz12.png\" style=\"float:right\"> The number of input neurons corresponds to the number of features.\n",
    "\n",
    "The number of output neurons corresponds to the number of label classes.  For binary classification, it is common to have two output nodes.\n",
    "\n",
    "Layers are typically *fully connected*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Neural Networks\n",
    "\n",
    "The <a href=\"https://en.wikipedia.org/wiki/Universal_approximation_theorem\">universal approximation theorem</a> says that, if some reasonable assumptions are made, a feedforward neural network with a finite number of nodes can approximate any continuous function to within a given error $\\epsilon$ over a bounded input domain.\n",
    "\n",
    "The theorem says nothing about the design (number of nodes/layers) of such a network.\n",
    "\n",
    "The theorem says nothing about the *learnability* of the weights of such a network.\n",
    "\n",
    "These are open theoretical questions.\n",
    "\n",
    "Given a network design, how are we going to learn weights for the neurons?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Stochastic Gradient Descent\n",
    "<img src=\"imgs/valley_with_ball.png\" width=\"300\" style=\"float:right\">\n",
    "\n",
    "Randomly select $m$ training examples $X_j$ and compute the gradient of the loss function ($L$).  Update weights and biases with a given _learning rate_  $\\eta$.\n",
    "$$ w_k' = w_k-\\frac{\\eta}{m}\\sum_j^m \n",
    "\\frac{\\partial L_{X_j}}{\\partial w_k}$$\n",
    "$$b_l' = b_l-\\frac{\\eta}{m}\n",
    "  \\sum_j^m \\frac{\\partial L_{X_j}}{\\partial b_l}\n",
    "$$\n",
    "\n",
    "Common loss functions: logistic, hinge, cross entropy, euclidean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Loss Functions\n",
    "\n",
    "<center><img src=\"../files/lossfuncs.png\"></center>\n",
    "\n",
    "x = 1 is a correct prediction, x = -1 a wrong prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backpropagation\n",
    "<font size=\"3\">\n",
    "\n",
    "Backpropagation is an efficient algorithm for computing the partial derivatives needed by the gradient descent update rule.  For a training example $x$ and loss function $L$ in a network with $N$ layers:\n",
    "\n",
    "1. **Feedforward**.  For each layer $l$ compute\n",
    " $$a^{l} = \\sigma(z^{l})$$\n",
    " where $z$ is the weighted input and $a$ is the activation induced by $x$ (these are vectors representing all nodes of layer $l$).\n",
    " \n",
    "2. **Compute output error**\n",
    "$$\\delta^{N} = \\nabla_a L \\odot \\sigma'(z^N)$$\n",
    "where $ \\nabla_a L_j = \\partial L / \\partial a^N_j$, the gradient of the loss with respect to the output activations. $\\odot$ is the elementwise product.\n",
    "\n",
    "3. **Backpropagate the error**\n",
    "$$\\delta^{l} = ((w^{l+1})^T \\delta^{l+1}) \\odot\n",
    "  \\sigma'(z^{l})$$\n",
    "  \n",
    "4. **Calculate gradients**\n",
    "$$\\frac{\\partial L}{\\partial w^l_{jk}} = a^{l-1}_k \\delta^l_j \\text{ and } \\frac{\\partial L}{\\partial b^l_j} = \\delta^l_j$$\n",
    "    </font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Backpropagation as the Chain Rule\n",
    "\n",
    "&nbsp;\n",
    "\n",
    "$$\\frac{\\partial L}{\\partial a^l} \\cdot \\frac{\\partial a^l}{\\partial z^l} \\cdot \\frac{\\partial z^l}{\\partial a^{l-1}} \\cdot \\frac{\\partial a^{l-1}}{\\partial z^{l-1}} \\cdot \\frac{\\partial z^{l-1}}{\\partial a^{l-2}} \\cdots \\frac{\\partial a^{1}}{\\partial z^{l}} \\cdot \\frac{\\partial z^{l}}{\\partial x}  $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Deep Learning\n",
    "\n",
    "<center><img src=\"imgs/tikz36.png\"></center>\n",
    "\n",
    "A deep network is not more powerful (recall can approximate any function with a single layer), but may be more concise - can approximate some functions with many fewer nodes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolutional Neural Nets\n",
    "\n",
    "<center><img src=\"https://devblogs.nvidia.com/parallelforall/wp-content/uploads/2015/08/image1-624x293.png\"></center>\n",
    "\n",
    "Image recognition challenge results. Purple are deep learning methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolution Filters\n",
    "\n",
    "A filter applies a *convolution kernel* to an image.  \n",
    "\n",
    "The kernel is represented by an $n$x$n$ matrix where the target pixel is in the center.  \n",
    "\n",
    "The output of the filter is the sum of the products of the matrix elements with the corresponding pixels.\n",
    "\n",
    "Examples from [Wikipedia](https://en.wikipedia.org/wiki/Kernel_(image_processing)):\n",
    "\n",
    "<table style=\"border: 0px solid black; text-align=center\">\n",
    "<tr><td><img src=\"https://upload.wikimedia.org/math/7/a/d/7ad9f1c26f76bc1ee55fdf0ef8c6197a.png\"></td>\n",
    "<td>\n",
    "<img src=\"https://upload.wikimedia.org/math/3/0/c/30c42f667a353a5f0fbe92b0c4f2eee1.png\">\n",
    "</td>\n",
    "<td>\n",
    "<img src=\"https://upload.wikimedia.org/math/7/d/d/7ddb57480ef9fa57e2d1c2b3ffecb291.png\">\n",
    "</td>\n",
    "</tr>\n",
    "<tr><td>Identity</td><td>Blur</td><td>Edge Detection</td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Feature Maps\n",
    "\n",
    "We can think of a kernel as identifying a *feature* in an image and the resulting image as a feature map that has high values (white) where the feature is present and low values (black) elsewhere.\n",
    "\n",
    "*Feature maps retain the **spatial relationship** between features present in the original image.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolutional Layers\n",
    "\n",
    "<img src=\"../files/conv.png\" width=\"400\" style=\"float:right\"> A single kernel is applied across the input.  For each output feature map there is a single set of weights."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Convolutional Layers\n",
    "\n",
    "For images, each pixel is an input feature.  Each hidden layer is a set of feature maps.\n",
    "\n",
    "<center><img src=\"imgs/tikz44.png\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Pooling\n",
    "\n",
    "Pooling layers apply a fixed convolution (usually the non-linear MAX kernel).  The kernel is usually applied with a *stride* to reduce the size of the layer.\n",
    " * faster to train\n",
    " * fewer parameters to fit\n",
    " * less sensitive to small changes (MAX)\n",
    "<center><img src=\"imgs/tikz47.png\"></center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Consider an input image with 100 pixels.  In a classic neural network, we hook these pixels up to a hidden layer with 10 nodes.  In a CNN, we hook these pixels up to a convolutional layer with a 3x3 kernel and 10 output feature maps."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div id=\"iweightcnt\" style=\"width: 500px\"></div>\n",
       "<script>\n",
       "\n",
       "    var divid = '#iweightcnt';\n",
       "\tjQuery(divid).asker({\n",
       "\t    id: divid,\n",
       "\t    question: \"Which network has more parameters to learn?\",\n",
       "\t\tanswers: [\"Classic\",\"CNN\"],\n",
       "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
       "\t\tcharter: chartmaker})\n",
       "    \n",
       "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
       "\n",
       "\n",
       "</script>\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%html\n",
    "<div id=\"iweightcnt\" style=\"width: 500px\"></div>\n",
    "<script>\n",
    "\n",
    "    var divid = '#iweightcnt';\n",
    "\tjQuery(divid).asker({\n",
    "\t    id: divid,\n",
    "\t    question: \"Which network has more parameters to learn?\",\n",
    "\t\tanswers: [\"Classic\",\"CNN\"],\n",
    "        server: \"https://bits.csb.pitt.edu/asker.js/example/asker.cgi\",\n",
    "\t\tcharter: chartmaker})\n",
    "    \n",
    "$(\".jp-InputArea .o:contains(html)\").closest('.jp-InputArea').hide();\n",
    "\n",
    "\n",
    "</script>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<center><img src=\"../files/cnnex.png\"></center>\n",
    "The last features maps are typically connected to one or more fully-connected layers to produce the desired output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# <a href=\"https://pytorch.org/\">PyTorch</a>\n",
    "\n",
    "PyTorch creates the dataflow graph implicitly as operations are performed.\n",
    "\n",
    "* Dataflow graph can be parallelized\n",
    "* Dataflow graph maintains **autograd** information - how to compute gradients for backpropagation\n",
    "* Extremely flexible\n",
    "* Easier to debug and develop than static approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# PyTorch Tensors\n",
    "\n",
    "Tensor is very similar to `numpy.array` in functionality.\n",
    " * Is allocated to a device (CPU vs GPU)\n",
    " * Potentially maintains autograd information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[0.6449, 0.8679, 0.7000, 0.7213],\n",
       "        [0.7735, 0.5921, 0.2053, 0.5418],\n",
       "        [0.5145, 0.2320, 0.5560, 0.0838]])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch  # note package is not called pytorch\n",
    "\n",
    "T = torch.rand(3,4)\n",
    "T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(torch.Size([3, 4]), torch.float32, device(type='cpu'), False)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "T.shape,T.dtype,T.device,T.requires_grad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Modules vs Functional\n",
    "\n",
    "**Modules** are objects that can be initialized with default parameters and store any learnable parameters.  Learnable parameters can be easily extracted from the module (and any member modules).  Modules are called as functions on their inputs.\n",
    "\n",
    "**Functional** APIs maintain no state.  All parameters are passed when the function is called."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch.nn as nn\n",
    "import torch.nn.functional as F"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# A network is a module\n",
    "\n",
    "To define a network we create a module with submodules for operations with learnable parameters.  Generally use functional API for operations without learnable parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyNet(nn.Module):\n",
    "    def __init__(self):  #initialize submodules here - this defines our network architecture\n",
    "        super(MyNet, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=1, padding=1) \n",
    "        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1)\n",
    "        self.fc1 = nn.Linear(2304, 10) #mystery X\n",
    "\n",
    "    def forward(self, x): # this actually applies the operations\n",
    "        x = self.conv1(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.max_pool2d(x, kernel_size=2, stride=2)  # POOL    \n",
    "        x = self.conv2(x)\n",
    "        x = F.relu(x)\n",
    "        x = F.max_pool2d(x, kernel_size=2, stride=2) # POOL\n",
    "        x = torch.flatten(x, 1)\n",
    "        x = self.fc1(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from torchvision import datasets\n",
    "train_data = datasets.MNIST(root='../data', train=True,download=True)\n",
    "test_data = datasets.MNIST(root='../data', train=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(<PIL.Image.Image image mode=L size=28x28 at 0x7F25E2ED6FB0>, 5)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAABwAAAAcCAAAAABXZoBIAAABAElEQVR4nGNgGMyAWUhIqK5jvdSy/9/rGRgYGFhgEnJsVjYCwQwMDAxPJgV+vniQgYGBgREqZ7iXH8r6l/SV4dn7m8gmCt3++/fv37/Htn3/iMW+gDnZf/+e5WbQnoXNNXyMs/5GoQoxwVmf/n9kSGFiwAW49/11wynJoPzx4YIcRlyygR/+/i2XxCWru+vv32nSuGQFYv/83Y3b4p9/fzpAmSyoMnohpiwM1w5h06Q+5enfv39/bcMiJVF09+/fv39P+mFKiTtd/fv3799jgZiBJLT69t+/f/8eDuDEkDJf8+jv379/v7Ryo4qzMDAwMAQGMjBc3/y35wM2V1IfAABFF16Aa0wAOwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<PIL.Image.Image image mode=L size=28x28 at 0x7F25E2ED6260>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Inputs need to be tensors..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchvision import transforms\n",
    "train_data = datasets.MNIST(root='../data', train=True,transform=transforms.ToTensor())\n",
    "test_data = datasets.MNIST(root='../data', train=False,transform=transforms.ToTensor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0118, 0.0706, 0.0706, 0.0706,\n",
       "          0.4941, 0.5333, 0.6863, 0.1020, 0.6510, 1.0000, 0.9686, 0.4980,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.1176, 0.1412, 0.3686, 0.6039, 0.6667, 0.9922, 0.9922, 0.9922,\n",
       "          0.9922, 0.9922, 0.8824, 0.6745, 0.9922, 0.9490, 0.7647, 0.2510,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1922,\n",
       "          0.9333, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922,\n",
       "          0.9922, 0.9843, 0.3647, 0.3216, 0.3216, 0.2196, 0.1529, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706,\n",
       "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765, 0.7137,\n",
       "          0.9686, 0.9451, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.3137, 0.6118, 0.4196, 0.9922, 0.9922, 0.8039, 0.0431, 0.0000,\n",
       "          0.1686, 0.6039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0549, 0.0039, 0.6039, 0.9922, 0.3529, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.5451, 0.9922, 0.7451, 0.0078, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0431, 0.7451, 0.9922, 0.2745, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.1373, 0.9451, 0.8824, 0.6275,\n",
       "          0.4235, 0.0039, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3176, 0.9412, 0.9922,\n",
       "          0.9922, 0.4667, 0.0980, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1765, 0.7294,\n",
       "          0.9922, 0.9922, 0.5882, 0.1059, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0627,\n",
       "          0.3647, 0.9882, 0.9922, 0.7333, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.9765, 0.9922, 0.9765, 0.2510, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.1804, 0.5098,\n",
       "          0.7176, 0.9922, 0.9922, 0.8118, 0.0078, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.1529, 0.5804, 0.8980, 0.9922,\n",
       "          0.9922, 0.9922, 0.9804, 0.7137, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0941, 0.4471, 0.8667, 0.9922, 0.9922, 0.9922,\n",
       "          0.9922, 0.7882, 0.3059, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0902, 0.2588, 0.8353, 0.9922, 0.9922, 0.9922, 0.9922, 0.7765,\n",
       "          0.3176, 0.0078, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0706, 0.6706,\n",
       "          0.8588, 0.9922, 0.9922, 0.9922, 0.9922, 0.7647, 0.3137, 0.0353,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.2157, 0.6745, 0.8863, 0.9922,\n",
       "          0.9922, 0.9922, 0.9922, 0.9569, 0.5216, 0.0431, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.5333, 0.9922, 0.9922, 0.9922,\n",
       "          0.8314, 0.5294, 0.5176, 0.0627, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000],\n",
       "         [0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000,\n",
       "          0.0000, 0.0000, 0.0000, 0.0000]]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f25e3198250>"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAcTUlEQVR4nO3df3DU9b3v8dcCyQqaLI0hv0rAgD+wAvEWJWZAxJJLSOc4gIwHf3QGvF4cMXiKaPXGUZHWM2nxjrV6qd7TqURnxB+cEaiO5Y4GE441oQNKGW7blNBY4iEJFSe7IUgIyef+wXXrQgJ+1l3eSXg+Zr4zZPf75vvx69Znv9nNNwHnnBMAAOfYMOsFAADOTwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYGGG9gFP19vbq4MGDSktLUyAQsF4OAMCTc04dHR3Ky8vTsGH9X+cMuAAdPHhQ+fn51ssAAHxDzc3NGjt2bL/PD7gApaWlSZJm6vsaoRTj1QAAfJ1Qtz7QO9H/nvcnaQFat26dnnrqKbW2tqqwsFDPPfecpk+ffta5L7/tNkIpGhEgQAAw6Pz/O4ye7W2UpHwI4fXXX9eqVau0evVqffTRRyosLFRpaakOHTqUjMMBAAahpATo6aef1rJly3TnnXfqO9/5jl544QWNGjVKL774YjIOBwAYhBIeoOPHj2vXrl0qKSn5x0GGDVNJSYnq6upO27+rq0uRSCRmAwAMfQkP0Geffaaenh5lZ2fHPJ6dna3W1tbT9q+srFQoFIpufAIOAM4P5j+IWlFRoXA4HN2am5utlwQAOAcS/im4zMxMDR8+XG1tbTGPt7W1KScn57T9g8GggsFgopcBABjgEn4FlJqaqmnTpqm6ujr6WG9vr6qrq1VcXJzowwEABqmk/BzQqlWrtGTJEl1zzTWaPn26nnnmGXV2durOO+9MxuEAAINQUgK0ePFi/f3vf9fjjz+u1tZWXX311dq6detpH0wAAJy/As45Z72Ir4pEIgqFQpqt+dwJAQAGoROuWzXaonA4rPT09H73M/8UHADg/ESAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYGGG9AGAgCYzw/5/E8DGZSVhJYjQ8eElccz2jer1nxk885D0z6t6A90zr06neMx9d87r3jCR91tPpPVO08QHvmUtX1XvPDAVcAQEATBAgAICJhAfoiSeeUCAQiNkmTZqU6MMAAAa5pLwHdNVVV+m99977x0Hi+L46AGBoS0oZRowYoZycnGT81QCAISIp7wHt27dPeXl5mjBhgu644w4dOHCg3327uroUiURiNgDA0JfwABUVFamqqkpbt27V888/r6amJl1//fXq6Ojoc//KykqFQqHolp+fn+glAQAGoIQHqKysTLfccoumTp2q0tJSvfPOO2pvb9cbb7zR5/4VFRUKh8PRrbm5OdFLAgAMQEn/dMDo0aN1+eWXq7Gxsc/ng8GggsFgspcBABhgkv5zQEeOHNH+/fuVm5ub7EMBAAaRhAfowQcfVG1trT755BN9+OGHWrhwoYYPH67bbrst0YcCAAxiCf8W3KeffqrbbrtNhw8f1pgxYzRz5kzV19drzJgxiT4UAGAQS3iAXnvttUT/lRighl95mfeMC6Z4zxy8YbT3zBfX+d9EUpIyQv5z/1EY340uh5rfHk3znvnZ/5rnPbNjygbvmabuL7xnJOmnbf/VeybvP1xcxzofcS84AIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMBE0n8hHQa+ntnfjWvu6ap13jOXp6TGdSycW92ux3vm8eeWes+M6PS/cWfxxhXeM2n/ecJ7RpKCn/nfxHTUzh1xHet8xBUQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATHA3bCjYcDCuuV3H8r1nLk9pi+tYQ80DLdd5z/z1SKb3TNXEf/eekaRwr/9dqrOf/TCuYw1k/mcBPrgCAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMcDNS6ERLa1xzz/3sFu+Zf53X6T0zfM9F3jN/uPc575l4PfnZVO+ZxpJR3jM97S3eM7cX3+s9I0mf/Iv/TIH+ENexcP7iCggAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMMHNSBG3jPV13jNj3rrYe6bn8OfeM1dN/m/eM5L0f2e96D3zm3+7wXsmq/1D75l4BOriu0Fogf+/WsAbV0AAABMECABgwjtA27dv10033aS8vDwFAgFt3rw55nnnnB5//HHl5uZq5MiRKikp0b59+xK1XgDAEOEdoM7OThUWFmrdunV9Pr927Vo9++yzeuGFF7Rjxw5deOGFKi0t1bFjx77xYgEAQ4f3hxDKyspUVlbW53POOT3zzDN69NFHNX/+fEnSyy+/rOzsbG3evFm33nrrN1stAGDISOh7QE1NTWptbVVJSUn0sVAopKKiItXV9f2xmq6uLkUikZgNADD0JTRAra2tkqTs7OyYx7Ozs6PPnaqyslKhUCi65efnJ3JJAIAByvxTcBUVFQqHw9GtubnZekkAgHMgoQHKycmRJLW1tcU83tbWFn3uVMFgUOnp6TEbAGDoS2iACgoKlJOTo+rq6uhjkUhEO3bsUHFxcSIPBQAY5Lw/BXfkyBE1NjZGv25qatLu3buVkZGhcePGaeXKlXryySd12WWXqaCgQI899pjy8vK0YMGCRK4bADDIeQdo586duvHGG6Nfr1q1SpK0ZMkSVVVV6aGHHlJnZ6fuvvtutbe3a+bMmdq6dasuuOCCxK0aADDoBZxzznoRXxWJRBQKhTRb8zUikGK9HAxSf/nf18Y3908veM/c+bc53jN/n9nhPaPeHv8ZwMAJ160abVE4HD7j+/rmn4IDAJyfCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYML71zEAg8GVD/8lrrk7p/jf2Xr9+Oqz73SKG24p955Je73eewYYyLgCAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMcDNSDEk97eG45g4vv9J75sBvvvCe+R9Pvuw9U/HPC71n3Mch7xlJyv/XOv8h5+I6Fs5fXAEBAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACa4GSnwFb1/+JP3zK1rfuQ988rq/+k9s/s6/xuY6jr/EUm66sIV3jOX/arFe+bEXz/xnsHQwRUQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGAi4Jxz1ov4qkgkolAopNmarxGBFOvlAEnhZlztPZP+00+9Z16d8H+8Z+I16f3/7j1zxZqw90zPvr96z+DcOuG6VaMtCofDSk9P73c/roAAACYIEADAhHeAtm/frptuukl5eXkKBALavHlzzPNLly5VIBCI2ebNm5eo9QIAhgjvAHV2dqqwsFDr1q3rd5958+appaUlur366qvfaJEAgKHH+zeilpWVqays7Iz7BINB5eTkxL0oAMDQl5T3gGpqapSVlaUrrrhCy5cv1+HDh/vdt6urS5FIJGYDAAx9CQ/QvHnz9PLLL6u6ulo/+9nPVFtbq7KyMvX09PS5f2VlpUKhUHTLz89P9JIAAAOQ97fgzubWW2+N/nnKlCmaOnWqJk6cqJqaGs2ZM+e0/SsqKrRq1aro15FIhAgBwHkg6R/DnjBhgjIzM9XY2Njn88FgUOnp6TEbAGDoS3qAPv30Ux0+fFi5ubnJPhQAYBDx/hbckSNHYq5mmpqatHv3bmVkZCgjI0Nr1qzRokWLlJOTo/379+uhhx7SpZdeqtLS0oQuHAAwuHkHaOfOnbrxxhujX3/5/s2SJUv0/PPPa8+ePXrppZfU3t6uvLw8zZ07Vz/5yU8UDAYTt2oAwKDHzUiBQWJ4dpb3zMHFl8Z1rB0P/8J7Zlgc39G/o2mu90x4Zv8/1oGBgZuRAgAGNAIEADBBgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJhI+K/kBpAcPW2HvGeyn/WfkaRjD53wnhkVSPWe+dUlb3vP/NPCld4zozbt8J5B8nEFBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCY4GakgIHemVd7z+y/5QLvmclXf+I9I8V3Y9F4PPf5f/GeGbVlZxJWAgtcAQEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJrgZKfAVgWsme8/85V/8b9z5qxkvec/MuuC498y51OW6vWfqPy/wP1Bvi/8MBiSugAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAE9yMFAPeiILx3jP778yL61hPLH7Ne2bRRZ/FdayB7JG2a7xnan9xnffMt16q857B0MEVEADABAECAJjwClBlZaWuvfZapaWlKSsrSwsWLFBDQ0PMPseOHVN5ebkuvvhiXXTRRVq0aJHa2toSumgAwODnFaDa2lqVl5ervr5e7777rrq7uzV37lx1dnZG97n//vv11ltvaePGjaqtrdXBgwd18803J3zhAIDBzetDCFu3bo35uqqqSllZWdq1a5dmzZqlcDisX//619qwYYO+973vSZLWr1+vK6+8UvX19bruOv83KQEAQ9M3eg8oHA5LkjIyMiRJu3btUnd3t0pKSqL7TJo0SePGjVNdXd+fdunq6lIkEonZAABDX9wB6u3t1cqVKzVjxgxNnjxZktTa2qrU1FSNHj06Zt/s7Gy1trb2+fdUVlYqFApFt/z8/HiXBAAYROIOUHl5ufbu3avXXvP/uYmvqqioUDgcjm7Nzc3f6O8DAAwOcf0g6ooVK/T2229r+/btGjt2bPTxnJwcHT9+XO3t7TFXQW1tbcrJyenz7woGgwoGg/EsAwAwiHldATnntGLFCm3atEnbtm1TQUFBzPPTpk1TSkqKqquro481NDTowIEDKi4uTsyKAQBDgtcVUHl5uTZs2KAtW7YoLS0t+r5OKBTSyJEjFQqFdNddd2nVqlXKyMhQenq67rvvPhUXF/MJOABADK8APf/885Kk2bNnxzy+fv16LV26VJL085//XMOGDdOiRYvU1dWl0tJS/fKXv0zIYgEAQ0fAOeesF/FVkUhEoVBIszVfIwIp1svBGYy4ZJz3THharvfM4h9vPftOp7hn9F+9Zwa6B1r8v4tQ90v/m4pKUkbV7/2HenviOhaGnhOuWzXaonA4rPT09H73415wAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMBHXb0TFwDUit+/fPHsmn794YVzHWl5Q6z1zW1pbXMcayFb850zvmY+ev9p7JvPf93rPZHTUec8A5wpXQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACW5Geo4cL73Gf+b+z71nHrn0He+ZuSM7vWcGuraeL+Kam/WbB7xnJj36Z++ZjHb/m4T2ek8AAxtXQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACW5Geo58ssC/9X+ZsjEJK0mcde0TvWd+UTvXeybQE/CemfRkk/eMJF3WtsN7pieuIwHgCggAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMBFwzjnrRXxVJBJRKBTSbM3XiECK9XIAAJ5OuG7VaIvC4bDS09P73Y8rIACACQIEADDhFaDKykpde+21SktLU1ZWlhYsWKCGhoaYfWbPnq1AIBCz3XPPPQldNABg8PMKUG1trcrLy1VfX693331X3d3dmjt3rjo7O2P2W7ZsmVpaWqLb2rVrE7poAMDg5/UbUbdu3RrzdVVVlbKysrRr1y7NmjUr+vioUaOUk5OTmBUCAIakb/QeUDgcliRlZGTEPP7KK68oMzNTkydPVkVFhY4ePdrv39HV1aVIJBKzAQCGPq8roK/q7e3VypUrNWPGDE2ePDn6+O23367x48crLy9Pe/bs0cMPP6yGhga9+eabff49lZWVWrNmTbzLAAAMUnH/HNDy5cv129/+Vh988IHGjh3b737btm3TnDlz1NjYqIkTJ572fFdXl7q6uqJfRyIR5efn83NAADBIfd2fA4rrCmjFihV6++23tX379jPGR5KKiookqd8ABYNBBYPBeJYBABjEvALknNN9992nTZs2qaamRgUFBWed2b17tyQpNzc3rgUCAIYmrwCVl5drw4YN2rJli9LS0tTa2ipJCoVCGjlypPbv368NGzbo+9//vi6++GLt2bNH999/v2bNmqWpU6cm5R8AADA4eb0HFAgE+nx8/fr1Wrp0qZqbm/WDH/xAe/fuVWdnp/Lz87Vw4UI9+uijZ/w+4FdxLzgAGNyS8h7Q2VqVn5+v2tpan78SAHCe4l5wAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJggQAAAEwQIAGCCAAEATBAgAIAJAgQAMEGAAAAmCBAAwAQBAgCYIEAAABMECABgggABAEwQIACACQIEADBBgAAAJggQAMAEAQIAmCBAAAATI6wXcCrnnCTphLolZ7wYAIC3E+qW9I//nvdnwAWoo6NDkvSB3jFeCQDgm+jo6FAoFOr3+YA7W6LOsd7eXh08eFBpaWkKBAIxz0UiEeXn56u5uVnp6elGK7THeTiJ83AS5+EkzsNJA+E8OOfU0dGhvLw8DRvW/zs9A+4KaNiwYRo7duwZ90lPTz+vX2Bf4jycxHk4ifNwEufhJOvzcKYrny/xIQQAgAkCBAAwMagCFAwGtXr1agWDQeulmOI8nMR5OInzcBLn4aTBdB4G3IcQAADnh0F1BQQAGDoIEADABAECAJggQAAAE4MmQOvWrdMll1yiCy64QEVFRfr9739vvaRz7oknnlAgEIjZJk2aZL2spNu+fbtuuukm5eXlKRAIaPPmzTHPO+f0+OOPKzc3VyNHjlRJSYn27dtns9gkOtt5WLp06Wmvj3nz5tksNkkqKyt17bXXKi0tTVlZWVqwYIEaGhpi9jl27JjKy8t18cUX66KLLtKiRYvU1tZmtOLk+DrnYfbs2ae9Hu655x6jFfdtUATo9ddf16pVq7R69Wp99NFHKiwsVGlpqQ4dOmS9tHPuqquuUktLS3T74IMPrJeUdJ2dnSosLNS6dev6fH7t2rV69tln9cILL2jHjh268MILVVpaqmPHjp3jlSbX2c6DJM2bNy/m9fHqq6+ewxUmX21trcrLy1VfX693331X3d3dmjt3rjo7O6P73H///Xrrrbe0ceNG1dbW6uDBg7r55psNV514X+c8SNKyZctiXg9r1641WnE/3CAwffp0V15eHv26p6fH5eXlucrKSsNVnXurV692hYWF1sswJclt2rQp+nVvb6/LyclxTz31VPSx9vZ2FwwG3auvvmqwwnPj1PPgnHNLlixx8+fPN1mPlUOHDjlJrra21jl38t99SkqK27hxY3SfP/3pT06Sq6urs1pm0p16Hpxz7oYbbnA//OEP7Rb1NQz4K6Djx49r165dKikpiT42bNgwlZSUqK6uznBlNvbt26e8vDxNmDBBd9xxhw4cOGC9JFNNTU1qbW2NeX2EQiEVFRWdl6+PmpoaZWVl6YorrtDy5ct1+PBh6yUlVTgcliRlZGRIknbt2qXu7u6Y18OkSZM0bty4If16OPU8fOmVV15RZmamJk+erIqKCh09etRief0acDcjPdVnn32mnp4eZWdnxzyenZ2tP//5z0arslFUVKSqqipdccUVamlp0Zo1a3T99ddr7969SktLs16eidbWVknq8/Xx5XPni3nz5unmm29WQUGB9u/fr0ceeURlZWWqq6vT8OHDrZeXcL29vVq5cqVmzJihyZMnSzr5ekhNTdXo0aNj9h3Kr4e+zoMk3X777Ro/frzy8vK0Z88ePfzww2poaNCbb75puNpYAz5A+IeysrLon6dOnaqioiKNHz9eb7zxhu666y7DlWEguPXWW6N/njJliqZOnaqJEyeqpqZGc+bMMVxZcpSXl2vv3r3nxfugZ9Lfebj77rujf54yZYpyc3M1Z84c7d+/XxMnTjzXy+zTgP8WXGZmpoYPH37ap1ja2tqUk5NjtKqBYfTo0br88svV2NhovRQzX74GeH2cbsKECcrMzBySr48VK1bo7bff1vvvvx/z61tycnJ0/Phxtbe3x+w/VF8P/Z2HvhQVFUnSgHo9DPgApaamatq0aaquro4+1tvbq+rqahUXFxuuzN6RI0e0f/9+5ebmWi/FTEFBgXJycmJeH5FIRDt27DjvXx+ffvqpDh8+PKReH845rVixQps2bdK2bdtUUFAQ8/y0adOUkpIS83poaGjQgQMHhtTr4WznoS+7d++WpIH1erD+FMTX8dprr7lgMOiqqqrcH//4R3f33Xe70aNHu9bWVuulnVMPPPCAq6mpcU1NTe53v/udKykpcZmZme7QoUPWS0uqjo4O9/HHH7uPP/7YSXJPP/20+/jjj93f/vY355xzP/3pT93o0aPdli1b3J49e9z8+fNdQUGB++KLL4xXnlhnOg8dHR3uwQcfdHV1da6pqcm999577rvf/a677LLL3LFjx6yXnjDLly93oVDI1dTUuJaWluh29OjR6D733HOPGzdunNu2bZvbuXOnKy4udsXFxYarTryznYfGxkb34x//2O3cudM1NTW5LVu2uAkTJrhZs2YZrzzWoAiQc84999xzbty4cS41NdVNnz7d1dfXWy/pnFu8eLHLzc11qamp7tvf/rZbvHixa2xstF5W0r3//vtO0mnbkiVLnHMnP4r92GOPuezsbBcMBt2cOXNcQ0OD7aKT4Ezn4ejRo27u3LluzJgxLiUlxY0fP94tW7ZsyP2ftL7++SW59evXR/f54osv3L333uu+9a1vuVGjRrmFCxe6lpYWu0UnwdnOw4EDB9ysWbNcRkaGCwaD7tJLL3U/+tGPXDgctl34Kfh1DAAAEwP+PSAAwNBEgAAAJggQAMAEAQIAmCBAAAATBAgAYIIAAQBMECAAgAkCBAAwQYAAACYIEADABAECAJj4f4W4/AnknuSPAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.imshow(train_data[0][0][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "#process 10 randomly sampled images at a time\n",
    "train_loader = torch.utils.data.DataLoader(train_data,batch_size=10,shuffle=True)\n",
    "test_loader = torch.utils.data.DataLoader(test_data,batch_size=10,shuffle=False)\n",
    "\n",
    "#instantiate our neural network and put it on the GPU\n",
    "model = MyNet().to('cuda')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]],\n",
       " \n",
       " \n",
       "         [[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]],\n",
       " \n",
       " \n",
       "         [[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]],\n",
       " \n",
       " \n",
       "         ...,\n",
       " \n",
       " \n",
       "         [[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]],\n",
       " \n",
       " \n",
       "         [[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]],\n",
       " \n",
       " \n",
       "         [[[0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           ...,\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.],\n",
       "           [0., 0., 0.,  ..., 0., 0., 0.]]]]),\n",
       " tensor([3, 4, 2, 2, 0, 0, 4, 8, 4, 7])]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch = next(iter(train_loader))\n",
    "batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[-0.0618,  0.0725,  0.0696,  0.0907,  0.0204, -0.1858,  0.0752,  0.0432,\n",
       "         -0.1025, -0.0215],\n",
       "        [-0.0696,  0.0885, -0.0112,  0.1133,  0.0270, -0.1915,  0.1485, -0.0218,\n",
       "         -0.1089,  0.0137],\n",
       "        [-0.0315,  0.0997,  0.0130,  0.0563,  0.0575, -0.1342,  0.1309,  0.0970,\n",
       "         -0.0519,  0.0064],\n",
       "        [-0.0242,  0.0943,  0.0179,  0.0685,  0.0844, -0.2072,  0.0581,  0.1100,\n",
       "         -0.1413,  0.0031],\n",
       "        [-0.0648,  0.0661,  0.0950,  0.0312,  0.0466, -0.1491,  0.0859,  0.0665,\n",
       "         -0.0603,  0.0305],\n",
       "        [-0.0315,  0.0449,  0.0610,  0.1021,  0.0320, -0.1344,  0.1114,  0.0320,\n",
       "         -0.0648,  0.0369],\n",
       "        [-0.0270,  0.0672,  0.0473,  0.0491,  0.0293, -0.1694,  0.0731,  0.0291,\n",
       "         -0.1106,  0.0358],\n",
       "        [-0.0456,  0.0428,  0.0516,  0.0689,  0.0399, -0.2126,  0.0604,  0.0615,\n",
       "         -0.0844, -0.0294],\n",
       "        [-0.0477,  0.0740,  0.0625,  0.0808,  0.0134, -0.1671,  0.1014,  0.0406,\n",
       "         -0.1062,  0.0309],\n",
       "        [-0.0168,  0.0673,  0.0582,  0.0483,  0.0313, -0.1919,  0.0552,  0.0547,\n",
       "         -0.0937,  0.0116]], device='cuda:0', grad_fn=<AddmmBackward0>)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output = model(batch[0].to('cuda')) # model is on GPU, so must put input there too\n",
    "output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training MNIST\n",
    "\n",
    "Our network takes an image (as a tensor) and outputs class probabilities.\n",
    " * Need a loss\n",
    " * Need an optimizer (e.g. SGD, ADAM)\n",
    "   * `backward` does _not_ update parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor(2.3067, device='cuda:0', grad_fn=<NllLossBackward0>)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "loss = F.cross_entropy(output,batch[1].to('cuda')) #combines log softmax and \n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$L(x,class) = - \\log\\left(\\frac{e^{x_{\\mathrm{class}}}}{\\sum_j e^{x_j}}\\right)$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss.backward() # sets grad, but does not change parameters of model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Training MNIST\n",
    "\n",
    "**Epoch** - One pass through the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 44s, sys: 1.69 s, total: 3min 46s\n",
      "Wall time: 3min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.00001) # need to tell optimizer what it is optimizing\n",
    "\n",
    "losses = []\n",
    "for epoch in range(10):\n",
    "    for i, (img,label) in enumerate(train_loader):\n",
    "        optimizer.zero_grad()  # IMPORTANT!\n",
    "        img, label = img.to('cuda'), label.to('cuda')\n",
    "        output = model(img)\n",
    "        loss = F.cross_entropy(output, label)\n",
    "        loss.backward()\n",
    "        optimizer.step()\n",
    "        losses.append(loss.item())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x7f25e042d120>]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiMAAAGdCAYAAADAAnMpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/P9b71AAAACXBIWXMAAA9hAAAPYQGoP6dpAABKDklEQVR4nO3deXhU1f0G8HeyJ5KENQlLWBQB2TfB4IYVRaQK1Z+l1BakaqsFK8VKpVVRUWPdcGNRKaBVREUWBURi2CEsCUkgJIQlIfseMpN9mTm/P4AxQzLJ7OfemffzPHkeMnNn7nduhrnvnHsWjRBCgIiIiEgSL9kFEBERkWdjGCEiIiKpGEaIiIhIKoYRIiIikophhIiIiKRiGCEiIiKpGEaIiIhIKoYRIiIikspHdgGWMBgMyM/PR3BwMDQajexyiIiIyAJCCFRWVqJHjx7w8jLf/qGKMJKfn4/IyEjZZRAREZENcnJy0KtXL7P3qyKMBAcHA7j0YkJCQiRXQ0RERJbQ6XSIjIw0nsfNUUUYuXJpJiQkhGGEiIhIZdrrYsEOrERERCQVwwgRERFJxTBCREREUjGMEBERkVQMI0RERCQVwwgRERFJZVUYiY6Oxo033ojg4GCEhYVh+vTpSE9Pb/Mxa9euhUajMfkJCAiwq2giIiJyH1aFkb1792Lu3Lk4fPgwYmJi0NjYiLvvvhvV1dVtPi4kJAQFBQXGn6ysLLuKJiIiIvdh1aRnO3bsMPl97dq1CAsLQ0JCAm677Tazj9NoNIiIiLCtQiIiInJrdvUZ0Wq1AIDOnTu3uV1VVRX69OmDyMhITJs2DadOnWpz+/r6euh0OpMfIiIick82hxGDwYD58+fj5ptvxtChQ81uN3DgQKxevRpbtmzBF198AYPBgAkTJiA3N9fsY6KjoxEaGmr84SJ5RERE7ksjhBC2PPDJJ5/Ejz/+iAMHDrS5Et/VGhsbccMNN2DmzJlYsmRJq9vU19ejvr7e+PuVhXa0Wi3XpiEiIlIJnU6H0NDQds/fNi2UN2/ePGzduhX79u2zKogAgK+vL0aNGoVz586Z3cbf3x/+/v62lEZ2+iY+B706BmJC/66ySyEiIg9h1WUaIQTmzZuHTZs2YdeuXejXr5/VO9Tr9Th58iS6d+9u9WPJuVLytFi44QR+v+qI7FKIiMiDWNUyMnfuXKxbtw5btmxBcHAwCgsLAQChoaEIDAwEAMyaNQs9e/ZEdHQ0AOCVV17BTTfdhP79+6OiogJvvfUWsrKy8Nhjjzn4pZC98itqZZdAREQeyKowsmLFCgDAxIkTTW5fs2YNHnnkEQBAdnY2vLx+aXC5ePEiHn/8cRQWFqJTp04YM2YMDh06hMGDB9tXOREREbkFq8KIJX1d9+zZY/L70qVLsXTpUquKIiIiIs/BtWmIiIhIKoYRIiIikophhIiIiKRiGCEiIiKpPDqMGAwCKXla6A02TUJLREREDuDRYeTtnen49YcH8Pzmk7JLISIi8lgeHUaW7zkPAPjqaI7kSoiIiDyXR4cRIiIiko9hhIiIiKRiGCEiIiKpGEaIiIhIKoYRIiIiksqjw4hG88u/axv08gohIiLyYJ4dRpr9e/mec9LqICIi8mQeHUaaT7y69uAFaXUQERF5Mo8OI81V1jfJLoGIiMgjMYwQERGRVAwjzQjBBfOIiIhcjWGkmTd+PC27BCIiIo/DMNLMx/syoDd4buuIpvlYZyIiIhdhGLnKgm+SZJdARETkURhGrrIlKR8GD24dISIicjWGkVY06A2ySyAiIvIYDCNEREQkFcNIKy6UVcsugYiIyGMwjLTinvf2o66RC+cRERG5AsOIGXvPlMgugYiIyCN4dBhZeM9As/fVN7ETKxERkSt4dBi5e3C42fvOFlW6sBIiIiLP5dFhpH9YMObd0b/V+2ob2GeEiIjIFTw6jADAM3cPaPX2VQcyMfGt3cgoqXJxRURERJ7F48OIRqNBZvS9rd53oawGizaedHFFREREnsXjwwjQ9gJxjZyNlYiIyKkYRtrBlWyJiIici2GkHYwiREREzsUwctkbDwxr9XY2jBARUVuE4Erv9mIYuSzQz1t2CYpUXd+EipoG2WUQESnSp/sycONrscgs5Zpm9mAYuSw00LfV2zUefqFmyOKfMPKVGFTVN8kuhYhIcV7bnobSqnos2ZoquxRVYxi57PYB3Vq/w7OziFFmCVM/EZE5vFRjH4aRyzQaDe4b0aPF7VllPAkTERE5E8NIMx38fVrcVqSrl1CJHGwEIiIiGRhGmunXNUh2CURERB6HYaQZL47jJZVZfSATf1h1BHWNXNiRiNSLYaQZH6/Ww0h2WY2LKyGyzCtbU3HgXCm+PJItuxQiIpsxjDTz4Jherd6+9WS+iyshsk4Nh14TkYoxjDQTHND6XCNrDl5wbSFEREQehGHEAiWVnjOihoiIyNUYRq7Sq1Og7BKIiIg8CsPIVT6dNVZ2CURERB6FYeQqN3QPkV2CIglwqmMiInIOhhELvf1TuuwSiIiI3BLDiIU+2n1OdglERKRQGk6aaReGESIiIpKKYYSIiIikYhhpRWhg65OfERERkeMxjLTimbsHyC6BiIjIYzCMtMLHi4eFiIjso61pxNYT+VxV2wJWnXWjo6Nx4403Ijg4GGFhYZg+fTrS09sf8vrtt99i0KBBCAgIwLBhw7B9+3abC3aFPl2CWr09v6LWxZUQWYazwBApz6zVRzBvXSJe3ZYquxTFsyqM7N27F3PnzsXhw4cRExODxsZG3H333aiurjb7mEOHDmHmzJl49NFHkZiYiOnTp2P69OlISUmxu3hnmXBdl1Zvf27jSRdXQqQ8eoNATGoRiivrZJdCpGjJuVoAwJYkrvzeHh9rNt6xY4fJ72vXrkVYWBgSEhJw2223tfqY999/H/fccw+effZZAMCSJUsQExODjz76CCtXrrSxbOcyN15835kSF1dCpDzrjmbjhc0pCA30RfLiu2WXQ0RuwK7OEVrtpdTXuXNns9vExcVh0qRJJrdNnjwZcXFxZh9TX18PnU5n8kOXrD6Qid2ni12+X8HrAHTZrrQiAIC2tlFyJUTkLmwOIwaDAfPnz8fNN9+MoUOHmt2usLAQ4eHhJreFh4ejsLDQ7GOio6MRGhpq/ImMjLS1TLeSkHURr2xNxZy1x1yyvwul5i+/kXoUauugNzBNEpFy2RxG5s6di5SUFKxfv96R9QAAFi1aBK1Wa/zJyclx+D7UqFDr3Gv0V1+dmvL+fqfuj5zv4LlS3BQdi0fWHJVdChGRWTaFkXnz5mHr1q3YvXs3evXq1ea2ERERKCoqMrmtqKgIERERZh/j7++PkJAQkx+l+Me3ybJLcJlaDkdTvTUHLwAA9p8tlVsIEVEbrAojQgjMmzcPmzZtwq5du9CvX792HxMVFYXY2FiT22JiYhAVFWVdpQqxISEXBjZ5E5FCVNU3YdpHB/DRrrOyS/FoXCbPPlaFkblz5+KLL77AunXrEBwcjMLCQhQWFqK29pf5N2bNmoVFixYZf3/66aexY8cOvPPOOzh9+jReeuklxMfHY968eY57FUREHurLw1lIztXi7Z1nZJdCZDOrwsiKFSug1WoxceJEdO/e3fjz9ddfG7fJzs5GQUGB8fcJEyZg3bp1+OSTTzBixAhs2LABmzdvbrPTq9KxXYSIlKK+ySC7BCK7WTXPiLBgfOeePXta3PbQQw/hoYcesmZXREREboGXcNrHRVjMePn+IWbvazIY8F1CLnLKa1xYEZE67DpdhGkfHcC54irZpRgVaGst+jJF5K4KtXWK7u/IMGLG/SN6mL3vf3FZeObbZNz65m4XVkRknpLOs39aG4/kXC2e+ipRdikAgBV7ziMqeheW/swOnuSZdp4qxE3RsXhqvTL+T7aGYcSMa/zNX8H64USB2fvI/dQ36fmt2gY6hczQ+p8dpwEAH8Q6Noz8kJyPdUeyHfqcRM6wYu95AMA2BZ+7GEbM8PMxf2iScypcV4hCeOqpOPdiDQY+vwPzv06SXUqbhMf+heR56qtE/GvTSeRe5OVaInsxjBC14X+HswBw1U0yT1fbJLsEItVjGCEiq5hb1ZqIWsf/M+1jGGnDxIHdZJdARB5iS1IeHlp5CMU6565BRaREDCNt6BTkJ7sEIrvwC5l6PL0+CccuXMTr29Nkl0LkcgwjbVDa5zg7KRK5v8o69kEhz8Mw0oZRvTvKLkFVGpoMiEktgq5OGUM6iYiUgC2U7WMYacOwXh1ll6Aqb+44jcc/j8ecNcdkl0JERCrCMNIGTnRlnQ3HcwEACVkXJVdCRERqwjBCREREUjGMEBERkVQMI2QRXrKiK/heUBb2jSR3wDDSBn7kEtmOIwiIlEEN/xUZRsiIJw8iIpKBYaQNbI0mIiJyPoYRIjfgyuDMRb+IrMP/Me1jGGlDaKCP7BLIDrFpRbjnvX1IzdfJLkUad/oQ3JKUhxkfx6Gksl52KYqQXVaDBd8kIb2oUnYpRHZjGGlD/7Bg2SWQHR79LB6nCyvx5//Fyy6FHODp9Uk4klmO6B+5kBwAPPrZMWw8noetJwpkl+K2zhRVYtbqo0jMtm8iR7Ymto9hpB2zo/rILkE1KmqUuSZNVT0XHnMnulr3/ntaesXtbHGVU+sg4JHVR7HvTAl+s/yQ7FLcHsOIirBDretp3OpCBxFZI19bJ7sEj8Ew0g5/X2/ZJRCRB2H8VSdeibEPw0g7+AYj8gy5F2t4SY9IEoaRdvDSiHw55TWYu+44knMqZJdCbiqrrBq3/Gc3bnz1Z9mlEHkkhpF2BPkp8zJN3Pky5JTXuGx/MjPZ3HXHse1EAaYtOyixCnJnh86XAQBqG/VWP1Zw4QgiuzGMtOPRW/q1eb8rv603v2Q089PDuPXN3S7bt0yZJdXS9q2WE406qiTyTLKv9qthaDHDSDuCA3zbvH/fmRIXVUKeoKHJgKOZ5WjUG2SXYjcVfP4RkUIwjNjpnZgzskuwWnZZDXIvuu4SD1nuuY0n8NuP47Bka6rsUoiIXIZhxAJj+nSSXYLD1Dbocdtbu3HLf3ajyQ2+fbubjcfzAACfx2VJroSIyHUYRiwg2hlS888NJ5CkkpEeZdW/rOvRwDBCNrD06gsnjCMiSzGMOMDX8TmYzpEeJJHeYMA/vk3G+qPZJrez3wYRqQGXpSVyA9tPFiKztBobEnLxu3G9ZZdDRAqSkGXfQn+uwJYRUqXiyjpM/WA//ne4/b4VnjBxnbZWmYsUEsn0zs50/Gb5QdTZMH8MuRbDiAU84FzWwjfxObJLaNPbP6XjVL4OL2xOkV0KkUN54ueNs3y46xwSsyuwOTFPah28XNo+hhELeMI366st3HBCdgltqm1k51tZPPC/A6mca+btaStxMI20h2HEAr8dGym7BADOD0VtjX7wxEBW16jHx3szZJfhkZr0BnwYe1YV17qJyH4MIxaYcF0X2SWQBJskN+16snVHs/FOzBk8uOKQ7FJcTs3foS9WN+DOd/bgw9izskshlWEYsYBSF8uzhSpbOCz8dF61PwN/+yoReoNjXmRDEy8FyXKuuEp2CWSDT/Zn4HxJtSpnpia5GEYsEBYSILsEs2obbO8l7m6TUr26LQ3fJ+cjNq1IdinkVGpM1J7Bk2Z1bm8yTLIOw4jK3fDiDmhrOKyzuRo7ApotMkur+TdoBUcQKM+RjDIU6+pkl6F6lXWNuPXN3RzN50AMI24gLqPMJftJzL6IT/dlwOCgyyCOIrOeC6XVuOPtPRjxyk5pNRBZ4tD5Usz45DDGvR4ruxTV25CQi9yLtRbNc0SWYRixUN8uQS7f54ncCvzj22QUKeSbzG+WH8Jr29MU17Hzu+O50vZ99EK5tH0TWSPuvGVfWsqrG5xcifM5qt+YObxC43gMIxZ6cuJ1Lt/n/R8dxIaEXPzj22SX77st50qU1bnwRK5WdgmK5Yx+Qbz64r7WH83G6CUxeFfFHVDPFFVi8Is78N7P6n0NnohhxEId/H2l7TujpFravsnxDAaBYxfKUdPQJLsUIhP/2nQSAPCBiofmvrYtDfVNBrz3s3JeA/tPtY9hxEKCPfjJQVYfzMRDK+Mwe/VR2aUQeQR+eisfw4iFFNZnk1Rs3dFsAMCxC46bXZTDDIlIzRhGLKS0ESSup57Xf3UrFk/U6sNWbSLPwjBioT5WjKbR1jTi/o8O4L8HMp1Ykf3UfB1TCQEjIatc8QsKkvrIf2cTuR7DiIVG9e5k8bYr9p7HiVwtlmxNdWJF7iMppwIvfX8K2lp1TRz24Io42SWQAiggFxOpno/sAtxRXaNzZgA1/5mn7k/D6csOAgDqm/SIfmB4i/tV3IBDNtKoqNkuvbASO1IK8Zfbr0VwgLxRd2Se7MConnezPGwZcSBLJgu6WN2AzYl5dq0pY47aWhauZuviaPaOdNIbBP78ebyq5yVQdxyVz56TxTPfJuOj3ecQ/eNph9WjVmoKkaQsDCMONHpJDEoq69vc5o+rj2D+10lYss2xl3BW7c/AiJd3cnpiG+w7U4KdqUWKmpeA1CclzzGT7/F0rkxniyqRXVYjuwy3xTDiYDe+9nOb96fk6QAAW5PzHbrfV7elAQAXbrKBuctqSugkq2Y8qZI7uWvpPtz21m7ZZbgt9hlxAwlZjpuvwh3YOwV6Zmk1Xt+ehg7+/O9BjiGEwM7UIgwMD0bfrtfILsetKfFKkRJrUhp+2rqBT/dbPoSYX/bb99hnx3DemVPwu/BvwA9BZdiTXoK//C8BAHDhjamSqyFSHqsv0+zbtw/33XcfevToAY1Gg82bN7e5/Z49e6DRaFr8FBYW2lqz22vSG2SXYJGGJgNmrz6KFXvOO20feoOArs66NVzs7dCaU15r1+PdHQOO9Y5ns/WSqC1Wh5Hq6mqMGDECy5Yts+px6enpKCgoMP6EhYVZu2uPUF7dgFFLYrDgmyTZpbTr++R87D1Tgv/scN4ogphU9wutah/1RKQ27P+lfFZfppkyZQqmTJli9Y7CwsLQsWNHqx+nRva88Tck5KCyrgkbj+e1uM/pX0jb2EFrL6nWSfOpNFdV7/x9uFqZBUPAlYyf6+6Lf1qSxWWjaUaOHInu3bvjrrvuwsGDB9vctr6+HjqdzuRHTeIyymSX4FF4crRfRY3jA5Kz5pzg35tk46VKx3N6GOnevTtWrlyJ7777Dt999x0iIyMxceJEHD9+3OxjoqOjERoaavyJjIx0dpkOdabItsm7pPPQD3lrX7YQAsW6OqfUYit7TtDR29Mw8pUYfO/g4eZERJZyehgZOHAg/vKXv2DMmDGYMGECVq9ejQkTJmDp0qVmH7No0SJotVrjT05OjrPLdDnOVOg6js5Yr2xNxbjXYx38rPJ8vC8DAFy6ltLu08V4YPlBZJQoN7gnZF3E379OUlzwJHJHUob2jhs3DgcOHDB7v7+/P/z9/V1YkWX8vL3QoJKRLp7GlR3U1hy84LJ9qUlNQxOC/Cz7SJmz9hgA4On1SfjhqVta3K+ErP7gikMALl3CWjNnnORqXEMDj20gtZsS3rNqJmUG1qSkJHTv3l3Gru3y0NheskvwOD+nFrW7zZdHsjHj48MuqIbMef/nsxj84k+IseDv1dzFZn1VMkurkVOuvOm2s1w8BbiawwDPx62zdyJGT2B1y0hVVRXOnTtn/D0zMxNJSUno3LkzevfujUWLFiEvLw+ff/45AOC9995Dv379MGTIENTV1WHVqlXYtWsXdu7c6bhXQRazdw4OZ7r6P2xWWTV2nLJsaO/RC+XGfyu1g2Ndox71Tc5pWbNkuPDfvkrEezNGwsvL8R+MSy8vMmjrcgRV9U244+09AIDzr9/rqLKoGaX+v3A0dz/tCyFQXt2ALh2Ud/XAHlaHkfj4eNxxxx3G3xcsWAAAmD17NtauXYuCggJkZ2cb729oaMAzzzyDvLw8BAUFYfjw4fj5559NnkMtfJzwIW4NSz9LYtOKcPuAbvDxbrvhS+nNinkV7jX52OglMahxwmrNlvo+OR8PjumF2wd0k1aDOc0XmGwyGPhNktyKIz9rX9uWhlUHMvHejJGYPqqn455YMqvDyMSJE9u8Pr927VqT3xcuXIiFCxdaXZgSOeMbpTM8+lk8nrj9Ojw3ZZDxthO5FXhwxSH8dqzjRiap42goh8wgckWtC2vg+4OUwp0ahVYduLT8x2vb09wqjHDVXiv07hzk9H00/4Z4tRM5FRY/z+oDpuvV3P/RQTTqBb48km3mEeSOlN76RS3xT0aeiGHECn+4qY/Vj9EbLM/kq/ZntLno3aoDli+I5wrHmvXTICIishXDiBV82+mD0ZqFG060entr31hf3ZZm9fPLdOi8MmeaNdcaYDAI7D1TgrIq09YnT+nY5yhsbSG14f9x5WMYcbLvjue2uaptdlkN5q9PREqe1oVVWc8d/i9vTMzD7NVHcee7e122z7pGPf609hg+j7tg93OVVzfgSEaZ0+ZUaesSIamfO4bIBieNTiPXYxhxgdZWtb1yPnns82PYnJSPX39ofhI4cowrc5ZU1Lhu1dz1R7Ox63QxXtxyyu7nuv3N3ZjxyWGr5/IgO7nhSdwZvjqabZzN1xV2nS7CgOd/xH8VdvmabMMwIpkl69i42xBXJbjvwwP4wQVrsVTWNTnuueovPdeu08UOe05PlJxTgYdXHcapfGW3RspgT5vboo0nHVaHJf7+dTIA1y5jYCvm2fYxjKhEXaP1QzKFEDY9Tu0suYpxMk+Lp75KdH4xZBNnXlL4zfKDOHiuDL/7hLP2uoq2thHR29OQmu/8Fdi57pc6MYxY6e2HRsguwWKPfRaPQS/sMLvQl70TS/G/vGN96pImbnfo/WOfKwPcHNlqRW17dWsqPt6XgXs/2C+7FFIohhErdQt27BS8zgzxsZeb8zcl5jlvJ2qggvPvueJKvLZdXaOpWtPW+3nuuuOYt+64Zc/joHpIGU65oEVENr5n7cMw4qbcfXXhrScKHPZczdfrWbghWcqxc2WnWhlKq+qx7UQBtp4oQEWzxfGI1IjBw/EYRiThZc1WSDomL//wSwe4b+Jz5RTh5gyc6MFiPFLkiRhGJKmoaURNQ5NHJmy9QaC2QY8DZ0uRkCV/FlfOr0GWfDko0tUh2YolGdxZfkUtHl17DIfOlcouhdyE1QvlkeM8a2Z2Vnc39YP9OF1Y6fL9emJrlDuufuvqloMrR3D867EAgO1/uxWDe4S4uApl+ed3J7D/bCliTxfjwhtTZZdDboAtI1ay9aP9bFFli8l5tlnR78ElJ1IXnbfMBpHLZ5lGvYH9CiT565cJ7W5j8RUX98tBAID7PjqA3enuOdeLpX+yfAXMfWTNZ6LsS18cbtw+toy4yF1L98kuwS6uuOR/9PLCe5OX7kNGaTWWzlDPMGpz3os9a9F2ujpldGDdfrLQ6fvIvSj/RGYPvUFgzppjsssgcitsGZHMikV9PUZGaTUAYPfpEpser5S+kmkFOotXbV5z8IJzi7nMEcemrS95lnz/+94FM9+qGb9DO56z1nOSyd1eEsOIlWS1tk376KDNj43+seXaOPaqqm9CMTt+tsmaSbW44BeRYzDMqRPDiEo4o8OnsONK6kvf27/wW3uaDLadoGUFRiEEvjqabdMKzG72JcduvMSuftX1v4Rx/j2dK61Ah/SrzhFpBTp8su+8ar7osM8I2eTAWeuG9BkMAhsScjG6T0eLH2Nr/4WXf0jFA6N72fRYe+xMLTIuFubMEQZK/mB3VEc9dvhTvyVbU/HGg8Nll6EIjXoDhBBOeV/XNDRhyvuXptk/8+oU+PlcamO4cptaLuewZcRK7jhUEgC0NY3Yecp5nRc3JuZh4XcnMOld53fk1da23hnU2ee3My4arqyWDxcTaqyZ7LLfyi8s7qy4sh6Pf97+SDVbNJ+9ua6p5cKoKSqZip9hxEru+oXt4f8exldHc5z2/Jwsiq5w5n+hukY9pi07iDd3OK6flLu00lgSYpkZnefntCLZJSgaw4iVBkUEyy7BKVLyLE/Ppey46tYulFZj7cFM1DW2/JZlDRmn8IUbTiA5pwLL95yXsHcishX7jFgpyI+H7NuEluu3lFa5JqAU6epw6Dybf51p4tt7AABl1Q145u6BcouxQu7FGg4b9iDu0V5FV/DMSg4x9tWfXbKfe97bh4tWrnDLpmfbHLsgf90ga+SUK3cyNZ44na9Jb4CPNxv71YphhFTF2iDiSsm51g/pNXJWYrrqLHj1blJV0rnNkzA8W6+4sg63vbkb9w7tLrsUshFjJFnEHWcwdDQ1dlC794P9skuw2ukCBii1cXYf4PVHc1DXaMDGxDy3HWTg7hhGCJsT82SX4PFam4AuNq0Id7y9B0kqHYlkzyiUth6Zr62z+XndnUGh60u423cZdxlhpSQMI4RXt6W2u83Sn89Y9Zyv/JCKrSeU1ZlQiXPE/O9wFnaktD6/y6OfxSOztBpz1hx1cVXtUd5xVIIT9lymc5BRS2KweEuK7DLcmlIDn9oxjJBFDmdY15lx9cFMzFuX6KRq3McLm1PwxBdtT4ZU02DfEFtPYc/yBm1RU/TS1jbis7gs2WW4NbV17FbL+5dhhEgBbGm1aa8fT2lVPfam27bysa3yKpQ7okWm9mLSkYwyl9ThKJ58mcLSlbjJOgwjVnLWty93JYRAdX0Tj5uD1Tfp8fzmtpvj73p3L6rqLV85uD2F2jqbF90y99cv0pnp/+Fh57qM0mrjv5310i3JD2o67J4ciNwRh/ZaiaHYOrPXHMO+MyW4fUA3i7bPKa9xeA1V9U1uNRpIowEe+yy+3bU/HDkMOiVPi19/eAADwjugd+dr7HuyZueQRn37f5d9Z0pwm4XvH7JNka4Oy3efc+jnmzv9n1Mm9zq+bBmxEv+DWWffmUuXCfaesexywa1v7nZKHbkXXXv5QFfn3PlQXL0I2ZakSyOuzhRVuXS/ADBrtdI68Crf7NVHrZrOf966427d18SdP7abNxCp+XUyjFhJxX/rFuLOl+HRtcdQWtUguxSHyyqrNvm9ycVNWsNf2omKGsuPqzWXsez5wFHzh1VzCVkXZZegaHvPlGD90WyLt1fCSCBPkZBVjif+l+CUVmA142UaKwnbLpkr0iNrjskuwWlmt/Zt2gGXmK0Z1peQzROms/z583jcPSQCN/fvgl8P73HpRjcJWo5SraBRWM7u3qGm3iMProgDcGnW2I1/vVlyNcrBlhErsSOmOlwoc863juMMGC7R3uiisuoGfHU0m8PHqRVyoom1gSvHxZeOlY5hxErswe3ZGvTOaRpT1+UT1xfrqcMp//51Ep76ioGL3B/DiJVCA30xc1xv2WWQg7wbY/nMspOX7sPWEwVOrIbMcXYfkbZaYmR9/6ioacCmxDz8kJyPksp6OUUomDV/Fk9u0VbL92f2GbFB9APD8JUVncNIuT6IPWvxtulFlUgvqnRiNa3z3I/RXzQZ1NlZ65UfUqGra0RESECb2zVvGbvyT4PJbc59F6jxPaaWk6wz6eoa0WTB8Hg1YBghchJ7vs1qNL+coGydaKw93xzLccrzXuHqc4XSPpL1BoHVBzMBAL8ertyl7bU1jU57j5HzlFY1YPhLOxEc4B6ncV6mIXKSjcflrYb8fXL7ixQu/O6Ejc9uy9T1Nu7KTSi5z8vyvedkl0B2qKxz3CzLMjGMkEfQQF3D/+z1Nwd0enRWgFDi6smerE5BQ4A9nSdPqskwQh5Dyd8grv4IctZnklo68rE/AFlDje+Xq/+P/3vTSdzyn92odPLszUrFMGKjeXf0l10CWWH/uVKLp6RXuvVO7uuhRM9+a+slpbb973AWSquUOVLFVd+SnTZcvdm/3a01zBnh58sj2cirqJV6eVcmhhEbLbhrgOwSyApv7TgtuwSywtWf9XkVjpkgKv+q53lhc0rrs/UqjCNO5uayzVdHnR9uXdki5/RWEmfPJuvi3JZRUoVtJwqkXyJiGLGRl5d7JX0itbPks/RPa1sugXAqX+eEamzHTxaymQ154lfv7MXcdcexM7XI8fVYgWGEiBTFmV/QThe6fp6YttQ16vHA8oN4Z2e67FKs4sjA1NBkwIaE3BatVtZUUVbtfot9ulpSToXU/bvHAGUiUqzM0mok5bQ9g6q7tQbsO1OCRRtPtrvdlqQ8HM+uaPU+kz4X7naAmvlk33m8vfMMAny9cHrJFJueY83BC44tyo2o5a3DMEJETnXH23uc+vzz1ydi8X1DnLoPa82ysB9Kg5vMnmmPfWdLAQB1jc6beE1NI2bVVKsjMYyQR1D7AofP2TxBmWPY8gHpqkO+OSkf3l5e+M2oni3u++zQBaufTwihiPeLp56UnMEVh9LdRgy5GvuMEClAex9jnjic1xqt9Teoa9Rj8fenrH6uQ+fLHFGSQ3naaS42rQgncitkl+FkTJvNsWWESAFc9bHkym/b54qrXLcztBw+arDxxepqPXPSKWeyptXgXHEVHv0sHgBw4Y2pziqJFIYtI0TkFOaWY1HAFRBSsKyyatklkAQMI+QRahu5/oajOCNMrL2qb4eSAour+wKw8d4+x7PaHrklk+yJxZSMYYQ8gtKXSFfTh1SMgydHSsy+2DKMuFkviUYnTbnuycwFViXPOfK2BfPJqOmzwJHYZ4SIpCrQ1tn9HHEZZRjcI8QB1TjHz2nFrd/RxolHLYsakuWW7T4vuwTFYssIEbmF/x7IlF0CUp00tbwShhpby9Iv+Op7ZeqilveO1WFk3759uO+++9CjRw9oNBps3ry53cfs2bMHo0ePhr+/P/r374+1a9faUCoRkbIVV9rfymPU7GRuy+mkpqHJYaUQOZvVYaS6uhojRozAsmXLLNo+MzMTU6dOxR133IGkpCTMnz8fjz32GH766SeriyUi1zlXXIUtSXkecQ37fKlrhyG7wrLd51FR80v/CZV8QSYPZXWfkSlTpmDKFMvXD1i5ciX69euHd955BwBwww034MCBA1i6dCkmT55s7e6JqB2/+yTOIc9TVt2Ap9cnITjAtV3LNh7PQ87FGpfuM6fctftzlbjzZZgyrLtdz5FTXoPHP4/HY7dei/8b08tBlZEzMr6a+xk5vc9IXFwcJk2aZHLb5MmTERdn/gOzvr4eOp3O5IfInTnyI+RwRnmb+8kuq4GuzvKJvVLyXPv/71+bTmJLUr7dz2Prh73sFgSlNWAs/v4UThdW4h/fJtv8HLKPqSNP/BoHjfWqbzLg9e1pOJJhOuOvPaXGX2g5rHlTYp4dz+g6Tg8jhYWFCA8PN7ktPDwcOp0OtbWtLxkdHR2N0NBQ409kZKSzyyTyCFml1bjtrd0Y/UqM7FKMlH4VKL2oUsqlKqUcFlf3PVH6+8FRquqb8Mm+DMz45LDJ7ZsT8/D2T+mtvuf05mYSvOzvXyc5skSXUuRomkWLFkGr1Rp/cnK4Lge5t0Qzy8g7Wtzlb2FN7XyoebLWToa7080Mzb2Ko0YuXH0ickbLgrMDVqGuDq9uTXXa8yutBclRknO1+Gj3OcRfNXmb3iBw17t7JVXlfE4PIxERESgqMp0kqaioCCEhIQgMDGz1Mf7+/ggJCTH5ISL1MbQSelbsMZ1rYf2xbFeVY9b2k4Vt3p9R4pwpyluLA7GnizH+9VicLXJep9ptJwowekkMDp0rtenxloajVQcyUdPQ5DGtHY5UUWN6KbVAW4uMUvedKt/pYSQqKgqxsbEmt8XExCAqKsrZuyYiyYYsbn/U3P6ztp0QnaW+yYBT+VqpNRRX1mPRphN2P8+XR1oPenPXHcfFmkY8/N8jdu+jPe0FkebBZsE3SUjMcfx07mrs2HnQxqCoVlaHkaqqKiQlJSEpKQnApaG7SUlJyM6+9KZftGgRZs2aZdz+iSeeQEZGBhYuXIjTp09j+fLl+Oabb/D3v//dMa+AiCxW5+I1etS4JtD8r5Mw9YMDTnt+rYWrAhscMIP8gXOlSMkzH6yU0GLRvIaNx/NMZin9MPYsknMqXF8ULnX0bs2+syUu2f/VSyS4O6vDSHx8PEaNGoVRo0YBABYsWIBRo0bhxRdfBAAUFBQYgwkA9OvXD9u2bUNMTAxGjBiBd955B6tWrXKLYb1+PorsckNk1nEX9U3xZNHb09q8/4HlBwFcuoSlrTEfTAwOSgoOnYjNxd6JOYNpyw5K2ffrZv6OK/acR5Gu3sXVuD+rJxCYOHFimx2fWptddeLEiUhMTLR2V4o3cUA37HTwomFEStOkN7TZyG3pN31PoAHw8b6MNrc5X1KNAm0t/r0pBbtOm+8Y23zNHkcMJlVCK4iatBUGSyrdL4zIfn/wq70dXp42RHYJRE73wa5zbd4/4uWdLqrEfTz2WXybQeRqAgJHMsqsmh/map+0E5Jc5ZUfUlvt2EyejWHEDsEBvrJLICIXqKxz7Fwbp6xcUO/rYzmY8clhPLD8kNX7+jwuC+/GnEGD3gGdUGzUvJPq6oOZVgUxS56T1I9hhIjckiO/e78fe9aBz2a9zZdnpD1XbP1w3z3pJfhAcv1XK2+2Zo6lGvUGbD2Rb+wD45h5UD2PUvsQMYwQEZHN8itan0n7ivZaMI5kml++oLlP9mVg3rpEm0Y6NTY59rKQJUOFNyTk2r2fD2LPotKOS3OtGfdabJsjrGRhGLEDczkRebr565Ncsp+f0y4NFrCl8+iyPab9nirrGhH9Y5rxpJySp0WRzvIWg2/i2w4aOeU1dq3lc8W7MWcw7KWdyHTwZGcbjytvvRqGESIiB3H3fgytXSa6UGbDifJyw0JZleXB4oIdJ+SGJtP+Mv/ZcRof783Arz88gLNFlfj1hwcw/vVYM4+2XvORUI7wnYWtLGruFswwYgc1/+GJHOGHZPtX1yX1mPTuXuRX1CLvYtuXZpprq2/Hi1tOWfw8F9uYk8VaaQWVxn9vTlJeK0FrZA+9dTar5xkhIrriqa+UO3+QOzVSyHgt5dUNCAloeYpIzdehusG+mXVf256Ger0B50uct/5OWy4260DbfMZXe7h7q5izMYwQEalMTYNjhxq3ZvSSGAzuHoLQwLanMLDlC7u2thEvbE7BoIhg24qzk57znAAAdjtgiLWj8DINEbXrvwcyZZfg0a7+1n3GiSv6Npda0P58KCWV9diR0vaqx45uNPjxZAEKrehwSq2bt+647BKMGEaIiBzEmr4U7uSJLxLM3ueMyxdPfqmckyg5BsMIEbmlK0NBXckTWpAWf295p1MiSzGM2MGbPZaIFOtpF81/4Wny2pnkTMnaWuTVXlllNbj3/f3Y4oTRORqN41uYmvQG6Jsdj5V7HdOR11YMI3YI9POWXQIReYDma9m8uzNdYiWOdbqwsv2NHMRgEHhwxSE8/nm8RdtbG1tSC3RILdDh6fVJ7c5KK1uTwYCoN3ahrtF0/pUCrby6GUbs9PL9XLmXiC45a8PaMdb6YNc51DfaN7TW1ZQwduVMcSWOZ1cgJtX5l++WbE21antntti0JrO0utWZbK8OJ67EMEJEpDKvWHmyI9eNQAKAWivDYokVM9G6K4YRIiKVaX7Zhizzt2YT9CluNlOl1SMBwwgREZnVZLCv6Z79/MkSDCN2cvW1PiIiVzp24aLdz8E8Yr/KuvZn3VXz+YhhhIiIPIraWms0uLTScHvUG0UYRoiIyMOYa0AwGAR2ny5GqZ0dSpscvPZNk0Ego6Taoc+pNFwoj4iICMDX8TlYtPEkOl/jhzF9Otn8PA1Njh0iu3zPefQIDXDocyoNW0aIiIgA/Hx5DpLy6gbJlbSUr3XvhQEZRoiIiEgqhhEiInIaDcfSkAUYRuzUpYO/7BKIiBQrPqvcZEE2JRBmxp0oq0rr1TSoa5mA5hhG7HTvsO6ySyAiUqzff3oEKXnqmDG2wA36ZegdPJLHVTiaxk7eXhqEBvpCW9souxQiIrJDWoE6QlNbxr4ag18NCm9zm/1nS11UjeXYMkJERHQVhV1ZstjFmkZ8dzxXdhlWYxhxADVPwUtE5Gn4ka08DCNEREQkFcMIERF5FKWtTcOGGoYRIiIikoxhhIiIPEpjE9silIZhhIiIPEqhTlnziSjsqpEUDCNEREQkFcOIA6z4wxjZJRARkUrxohHDiEPc3L8rzr42RXYZRESkQo16g+wSpGMYcRBfby9MGRohuwwiIiLVYRhxoOG9OsougYiIHODntCLZJXgUhhEH+vVwruBLRERkLYYRB7rGn4sgExGRdTJLq2WXIB3DCBERkUR//O9R2SVIxzBCREREUjGMEBERkVQMI0RERCQVw4gDBQewAysRkaNNXrpPdgnkZAwjDuTrzcNJRORo6UWVsksgJ+PZk4iIiKRiGCEiIiKpGEaIiIhIKoYRB1v2+9GySyAiIlIVhhEHm8r1aYiIiKzCMEJERERSMYwQERGRVAwjREREJBXDCBEREUllUxhZtmwZ+vbti4CAAIwfPx5Hj5pf/njt2rXQaDQmPwEBATYXTERERO7F6jDy9ddfY8GCBVi8eDGOHz+OESNGYPLkySguLjb7mJCQEBQUFBh/srKy7CqaiIiI3IfVYeTdd9/F448/jjlz5mDw4MFYuXIlgoKCsHr1arOP0Wg0iIiIMP6Eh4fbVTQRERG5D6vCSENDAxISEjBp0qRfnsDLC5MmTUJcXJzZx1VVVaFPnz6IjIzEtGnTcOrUqTb3U19fD51OZ/JDRERE7smqMFJaWgq9Xt+iZSM8PByFhYWtPmbgwIFYvXo1tmzZgi+++AIGgwETJkxAbm6u2f1ER0cjNDTU+BMZGWlNmURERKQiTh9NExUVhVmzZmHkyJG4/fbbsXHjRnTr1g0ff/yx2ccsWrQIWq3W+JOTk+PsMomIiEgSH2s27tq1K7y9vVFUVGRye1FRESIiIix6Dl9fX4waNQrnzp0zu42/vz/8/f2tKY2IiIhUyqqWET8/P4wZMwaxsbHG2wwGA2JjYxEVFWXRc+j1epw8eRLdu3MNFyIiIrKyZQQAFixYgNmzZ2Ps2LEYN24c3nvvPVRXV2POnDkAgFmzZqFnz56Ijo4GALzyyiu46aab0L9/f1RUVOCtt95CVlYWHnvsMce+EiIiIlIlq8PIjBkzUFJSghdffBGFhYUYOXIkduzYYezUmp2dDS+vXxpcLl68iMcffxyFhYXo1KkTxowZg0OHDmHw4MGOexVERESkWhohhJBdRHt0Oh1CQ0Oh1WoREhIiu5x2zV+fiM1J+bLLICIistjuf0xEv67XOPQ5LT1/c20aJ1g6YyTefHC47DKIiIhUgWHECTQaDYL8vWWXQUREpAoMI06igUZ2CURERKrAMEJERERSMYwQERGRVAwjTqLhVRoiIiKLMIwQERGRVAwjREREJBXDiJP06hQouwQiIiJVYBhxkuG9OqJrBz/ZZRARESkew4gT/d+YSNklEBERKR7DCBEREUnFMOJEAopfg5CIiEg6hhEiIiKSimGEiIiIpGIYcaJenYJkl0BERKR4PrILcGe/uzES2WXViD1djIySatnlEBERKRLDiBP5envh31MHo1EvGEaIiIjM4GUaIiIikophhIiIiKRiGHEBITjfCBERkTkMIxL07sxRNkRERFcwjLgA20WIiIjMYxiRQKORXQEREZFyMIy4ALuMEBERmccwIsG8O/rLLoGIiEgxGEZc4K93XIfQQF/j7xGhARKrISIiUhaGERfoHhqIxBfukl0GERGRIjGMuIiXF3utEhERtYZhhIiIiKRiGCEiIiKpGEaIiIhIKoYRSaIfGIaIkACs//NNskshIiKSykd2AZ6ob5drcOv13TBzXG/ZpRAREUnHMOJCO/9+G8qqGhDJhfKIiIiMGEZcaEB4MBAuuwoiIiJlYZ8RBbiu2zWySyAiIpKGYUQBHpnQV3YJRERE0jCMKNjrvxkmuwQiIiKnYxhRoN/dGIk/33Ytfj+eo22IiMj9sQOrAvj5mGbCNx4cLqkSIiIi12PLiAJMG9lTdglERETSMIwoQICvN5bOGNHqfUumDXFxNURERK7FMKJwf4zqi/0L75BdBhERkdMwjCjEsJ4dzd7HGVuJiMidMYwoRP+wDtgy92Yceu5Xrd7/2Z/GubgiIiLyJHqDQdq+GUYUZERkR/ToGNjqfbcP6ObiaoiIyJPEpBZL2zfDiIoMDA+WXQIREbmpuka9tH0zjKjIC78eLLsEIiJyUxqNvH0zjKiIgJBdAhERuany6gZp+2YYUREfr1/+XG/+33D0D+sAgJdviIjIfo16eR1YOR28iozr1xkTB3bDdd064LdjI/Hg6F5oaDLgD/89Irs0IiJSueZfeF2+b2l7Jqt5e2mwds44k98D/bxNtol/fhI+j8vCzlOFOF1Yabzdz8cL4/t1xv6zpS6rl4iI1MPbS16nEV6mcTNdO/hjwV0DsGbOjcbbTrx0N868OgX/e3Q8Hr2lHwBgwV0DZJVIREQK5OvNMEJ2mBXVBwAw4bouxts6+P/S6OXfbFXg56fegN3/mIinftXfZN2bH5++1eQ5pw7v7qxyiYhIgXy8eZmG7DBtZE8M6RGC3p2vMd4WHOCLz/40Dj5eGvj7/HIpR6PRoF/XS9vdPSQCL2w5BQC4oXuIyXMuuGsApo3ogT//L8EFr4CIiGTzUdtlmmXLlqFv374ICAjA+PHjcfTo0Ta3//bbbzFo0CAEBARg2LBh2L59u03Fknn9w4Lh52P657x9QDfc3L+r2ceEhwTg4z+OwZePjTe53cdLg+u6dcDdQyKQGX1vm/t9+X6uKkxE5A5kdmC1es9ff/01FixYgMWLF+P48eMYMWIEJk+ejOLi1qeRPXToEGbOnIlHH30UiYmJmD59OqZPn46UlBS7iyf7TR4SYQwsyx8eja4d/EzCiUajwZ2DwgAAr04f2uLxsyf0xTd/iXJNsURE5DQlVXXS9q0RQlg1k9b48eNx44034qOPPgIAGAwGREZG4qmnnsJzzz3XYvsZM2aguroaW7duNd520003YeTIkVi5cqVF+9TpdAgNDYVWq0VISEj7DyCbCSGguWoavia9ARfKatA/rAMulFZj4tt7AADb/3YrBve49PfYe6YE/bpcgw3Hc1FaVY8AH2+sPphpfI6uHfzw9J3X47qwDujT5RpM++gASqtaTrATERKARfcOwvVhwbj3g/0AgMlDwqE3AD+nFQEA1s65EY+sOQYAWPb70cirqMF13TrgjoFhuPZfpq1u/cM6YMXDo9G7SxAGPr/D5L63/m84nt1wwo6jRUTkPpZMG4I/RvV16HNaev62Kow0NDQgKCgIGzZswPTp0423z549GxUVFdiyZUuLx/Tu3RsLFizA/PnzjbctXrwYmzdvRnJyskNfDLmGrq4RHfx84NXG9cVGvQHHsy5iZO+OqG3Qo2OQn8n9WWXVWLk3A3+57Vp07xiAH5ILcHP/Luge+stCgc2DUUOTAf/edBIPjumFm67tAnNKq+rxc2oRknMr8Pit1+Labh1M7lu1PxNDe4Zg0g3hCPA1HRY99tWYVgPSjX07YWBEMO4Z0h1pBTq8tj0NU4d3R9z5MqfNWLju8fH4/afOnz9mSI8QnMrXOX0/RKR8Ga/f2+bnui0sPX9b1YG1tLQUer0e4eHhJreHh4fj9OnTrT6msLCw1e0LCwvN7qe+vh719fXG33U6flgqSUiAb7vb+Hp7Yfzl0NC8A+0Vfbpcg+gHhhl//78xvVps07yFxs/HC289NKLd/Xbt4I/fjeuN343r3ep9z00ZZPaxB5/7FeqbDKhr0CPnYi1i04rw0NhIY4dfALjl+q54/LZrjb9raxuxOTEPN/btjME9QlCkq0Pna/zge7lXel2jHrraRqQXVSLA1xsNTQZMuK4L9IZLk/v7enshq6waz29OwZyb++JXg375v5K8+G406g3Ir6jFz2nFuGdIBPx8vHAitwKvbE3FB78bhWu7XYMeoYHILKtG/IVy3DEwDMm5Wrz102mM6dMZXx3NBgBc2+0a3DU4HPPu6I+GJgMOni/D5CHh8PfxxrniKpRV1SP2dDG2JuejuLIe00f1xNAeIbh9YBj6dA7ChuO5GNGrI97ZmY6dqZdaqHqEBiBfW4e+XYJwoawGQ3qE4KX7h8BLAxTr6vG39Yl4YFQvzLmlL+atS8S54iqT4/3oLf3wq0FheHjVEXz+p3G4bUA3PPVVIn5IzgcAvP+7kVi++zzSiy7Nl/P81BswfVRPTHhjFxqaWp8pctnvR2PuuuMIC/bHBzNHIb+iFgu+ufSl554hEaiobcDhjHKz74Gr9ewYiBu6B+PntEuXoT//0zh8EHsWPTsFYsFdA3DPe/tR26jH5CHh6Bbsj5N5OiTnVFj8/ERK4uggYg2rWkby8/PRs2dPHDp0CFFRv/QTWLhwIfbu3YsjR1p+k/Pz88Nnn32GmTNnGm9bvnw5Xn75ZRQVFbW6n5deegkvv/xyi9vZMkJ0SWuX02TQ1jQiJNDHobVc/dq0NY3w8ro0QuzK/QYBlFXXw1ujQUigL/QG0aKlq/nzZZZWo1/Xa6DRaFBaVY+uHfxNtmloMqCqvgkBvl4I8nPsIEO94dJHrEEIY0i9oq5Rj5LKekR2DmrxmCaDwSTIa2sbERLQ8lgbLj9/TaPeOKS/sq4R50uqER7iD2+NBmEhARBCQG8QOF1YiYERwTAIAYMBSMy5iKE9Q9HBz+fSbQI4dqEcY/p0QoCvd4u/R/PfhRBoMlx6Xebek2eKKhEc4GPS6glcOubZ5TUo0NYiPCQAhdo63HRtF/j5eCGnvAalVfWoazRgcI8QNOkN6BjkBw2AhstTlgf4eqO+6ZdVZv28vVBZ3wRfLy+UVtUjyM8bna/xQ+7FWvTqFIhzxVW4WNOI+ia9sXU1OacCg3uEGP/mtQ16ZJfXICI0AABQUlmP67pdA11tE/K1tfjXppNY/vBoXOPvA2+NBuXVDfD19kJdox5el197eKg/4i9cxJg+nbD1RAFG9e6Ivl2uQVZZNTQaDfp2CUKjXiCjtArHsyrwwOieCPD1RkVNA7LKanBdWAdc4+eN+suBWwigtlGPIl0ddLWNGN2nEwxCIPdiLb45loOQQF/8MaoPhACC/X1wtrgKKXlaDIwIRkVNI/K1tQgL9kfc+TLMHNcbHYN8ERLgiyOZ5QgO8IG/jxeu69bBaUFE1ZdpWmsZiYyMZBghIiJSEUvDiFWjafz8/DBmzBjExsYabzMYDIiNjTVpKWkuKirKZHsAiImJMbs9APj7+yMkJMTkh4iIiNyT1e2RCxYswOzZszF27FiMGzcO7733HqqrqzFnzhwAwKxZs9CzZ09ER0cDAJ5++mncfvvteOeddzB16lSsX78e8fHx+OSTTxz7SoiIiEiVrA4jM2bMQElJCV588UUUFhZi5MiR2LFjh7GTanZ2NryaTZwyYcIErFu3Ds8//zz+9a9/4frrr8fmzZsxdGjLOSuIiIjI81g9z4gMHNpLRESkPk7pM0JERETkaAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVAwjREREJBXDCBEREUnFMEJERERSMYwQERGRVI5dK9tJrkwSq9PpJFdCRERElrpy3m5vsndVhJHKykoAQGRkpORKiIiIyFqVlZUIDQ01e78q1qYxGAzIz89HcHAwNBqNw55Xp9MhMjISOTk5XPOmHTxW1uHxshyPleV4rCzHY2U5Zx4rIQQqKyvRo0cPk0V0r6aKlhEvLy/06tXLac8fEhLCN6uFeKysw+NlOR4ry/FYWY7HynLOOlZttYhcwQ6sREREJBXDCBEREUnl0WHE398fixcvhr+/v+xSFI/Hyjo8XpbjsbIcj5XleKwsp4RjpYoOrEREROS+PLplhIiIiORjGCEiIiKpGEaIiIhIKoYRIiIiksqjw8iyZcvQt29fBAQEYPz48Th69Kjskhxq3759uO+++9CjRw9oNBps3rzZ5H4hBF588UV0794dgYGBmDRpEs6ePWuyTXl5OR5++GGEhISgY8eOePTRR1FVVWWyzYkTJ3DrrbciICAAkZGRePPNN1vU8u2332LQoEEICAjAsGHDsH37doe/XntER0fjxhtvRHBwMMLCwjB9+nSkp6ebbFNXV4e5c+eiS5cu6NChAx588EEUFRWZbJOdnY2pU6ciKCgIYWFhePbZZ9HU1GSyzZ49ezB69Gj4+/ujf//+WLt2bYt6lPzeXLFiBYYPH26cICkqKgo//vij8X4eJ/PeeOMNaDQazJ8/33gbj9clL730EjQajcnPoEGDjPfzOJnKy8vDH/7wB3Tp0gWBgYEYNmwY4uPjjfer7vNdeKj169cLPz8/sXr1anHq1Cnx+OOPi44dO4qioiLZpTnM9u3bxb///W+xceNGAUBs2rTJ5P433nhDhIaGis2bN4vk5GRx//33i379+ona2lrjNvfcc48YMWKEOHz4sNi/f7/o37+/mDlzpvF+rVYrwsPDxcMPPyxSUlLEV199JQIDA8XHH39s3ObgwYPC29tbvPnmmyI1NVU8//zzwtfXV5w8edLpx8BSkydPFmvWrBEpKSkiKSlJ3HvvvaJ3796iqqrKuM0TTzwhIiMjRWxsrIiPjxc33XSTmDBhgvH+pqYmMXToUDFp0iSRmJgotm/fLrp27SoWLVpk3CYjI0MEBQWJBQsWiNTUVPHhhx8Kb29vsWPHDuM2Sn9vfv/992Lbtm3izJkzIj09XfzrX/8Svr6+IiUlRQjB42TO0aNHRd++fcXw4cPF008/bbydx+uSxYsXiyFDhoiCggLjT0lJifF+HqdflJeXiz59+ohHHnlEHDlyRGRkZIiffvpJnDt3zriN2j7fPTaMjBs3TsydO9f4u16vFz169BDR0dESq3Keq8OIwWAQERER4q233jLeVlFRIfz9/cVXX30lhBAiNTVVABDHjh0zbvPjjz8KjUYj8vLyhBBCLF++XHTq1EnU19cbt/nnP/8pBg4caPz9t7/9rZg6dapJPePHjxd/+ctfHPoaHam4uFgAEHv37hVCXDo2vr6+4ttvvzVuk5aWJgCIuLg4IcSl8Ofl5SUKCwuN26xYsUKEhIQYj8/ChQvFkCFDTPY1Y8YMMXnyZOPvanxvdurUSaxatYrHyYzKykpx/fXXi5iYGHH77bcbwwiP1y8WL14sRowY0ep9PE6m/vnPf4pbbrnF7P1q/Hz3yMs0DQ0NSEhIwKRJk4y3eXl5YdKkSYiLi5NYmetkZmaisLDQ5BiEhoZi/PjxxmMQFxeHjh07YuzYscZtJk2aBC8vLxw5csS4zW233QY/Pz/jNpMnT0Z6ejouXrxo3Kb5fq5so+RjrdVqAQCdO3cGACQkJKCxsdHkdQwaNAi9e/c2OV7Dhg1DeHi4cZvJkydDp9Ph1KlTxm3aOhZqe2/q9XqsX78e1dXViIqK4nEyY+7cuZg6dWqL18TjZers2bPo0aMHrr32Wjz88MPIzs4GwON0te+//x5jx47FQw89hLCwMIwaNQqffvqp8X41fr57ZBgpLS2FXq83edMCQHh4OAoLCyVV5VpXXmdbx6CwsBBhYWEm9/v4+KBz584m27T2HM33YW4bpR5rg8GA+fPn4+abb8bQoUMBXHoNfn5+6Nixo8m2Vx8vW4+FTqdDbW2tat6bJ0+eRIcOHeDv748nnngCmzZtwuDBg3mcWrF+/XocP34c0dHRLe7j8frF+PHjsXbtWuzYsQMrVqxAZmYmbr31VlRWVvI4XSUjIwMrVqzA9ddfj59++glPPvkk/va3v+Gzzz4DoM7Pd1Ws2kvkSnPnzkVKSgoOHDgguxTFGjhwIJKSkqDVarFhwwbMnj0be/fulV2W4uTk5ODpp59GTEwMAgICZJejaFOmTDH+e/jw4Rg/fjz69OmDb775BoGBgRIrUx6DwYCxY8fi9ddfBwCMGjUKKSkpWLlyJWbPni25Ott4ZMtI165d4e3t3aIndlFRESIiIiRV5VpXXmdbxyAiIgLFxcUm9zc1NaG8vNxkm9aeo/k+zG2jxGM9b948bN26Fbt370avXr2Mt0dERKChoQEVFRUm2199vGw9FiEhIQgMDFTNe9PPzw/9+/fHmDFjEB0djREjRuD999/ncbpKQkICiouLMXr0aPj4+MDHxwd79+7FBx98AB8fH4SHh/N4mdGxY0cMGDAA586d4/vqKt27d8fgwYNNbrvhhhuMl7XU+PnukWHEz88PY8aMQWxsrPE2g8GA2NhYREVFSazMdfr164eIiAiTY6DT6XDkyBHjMYiKikJFRQUSEhKM2+zatQsGgwHjx483brNv3z40NjYat4mJicHAgQPRqVMn4zbN93NlGyUdayEE5s2bh02bNmHXrl3o16+fyf1jxoyBr6+vyetIT09Hdna2yfE6efKkyX/wmJgYhISEGD842jsWan1vGgwG1NfX8zhd5c4778TJkyeRlJRk/Bk7diwefvhh4795vFpXVVWF8+fPo3v37nxfXeXmm29uMfXAmTNn0KdPHwAq/Xy3qrurG1m/fr3w9/cXa9euFampqeLPf/6z6Nixo0lPbLWrrKwUiYmJIjExUQAQ7777rkhMTBRZWVlCiEtDvzp27Ci2bNkiTpw4IaZNm9bq0K9Ro0aJI0eOiAMHDojrr7/eZOhXRUWFCA8PF3/84x9FSkqKWL9+vQgKCmox9MvHx0e8/fbbIi0tTSxevFhxQ3uffPJJERoaKvbs2WMytLCmpsa4zRNPPCF69+4tdu3aJeLj40VUVJSIiooy3n9laOHdd98tkpKSxI4dO0S3bt1aHVr47LPPirS0NLFs2bJWhxYq+b353HPPib1794rMzExx4sQJ8dxzzwmNRiN27twphOBxak/z0TRC8Hhd8cwzz4g9e/aIzMxMcfDgQTFp0iTRtWtXUVxcLITgcWru6NGjwsfHR7z22mvi7Nmz4ssvvxRBQUHiiy++MG6jts93jw0jQgjx4Ycfit69ews/Pz8xbtw4cfjwYdklOdTu3bsFgBY/s2fPFkJcGv71wgsviPDwcOHv7y/uvPNOkZ6ebvIcZWVlYubMmaJDhw4iJCREzJkzR1RWVppsk5ycLG655Rbh7+8vevbsKd54440WtXzzzTdiwIABws/PTwwZMkRs27bNaa/bFq0dJwBizZo1xm1qa2vFX//6V9GpUycRFBQkfvOb34iCggKT57lw4YKYMmWKCAwMFF27dhXPPPOMaGxsNNlm9+7dYuTIkcLPz09ce+21Jvu4QsnvzT/96U+iT58+ws/PT3Tr1k3ceeedxiAiBI9Te64OIzxel8yYMUN0795d+Pn5iZ49e4oZM2aYzJvB42Tqhx9+EEOHDhX+/v5i0KBB4pNPPjG5X22f7xohhLCuLYWIiIjIcTyyzwgREREpB8MIERERScUwQkRERFIxjBAREZFUDCNEREQkFcMIERERScUwQkRERFIxjBAREZFUDCNEREQkFcMIERERScUwQkRERFIxjBAREZFU/w+lnIlasBQa5AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the **batch loss**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing MNIST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy 0.9684\n"
     ]
    }
   ],
   "source": [
    "correct = 0\n",
    "with torch.no_grad(): #no need for gradients - won't be calling backward to clear them\n",
    "    for img, label in test_loader:\n",
    "        img, label = img.to('cuda'), label.to('cuda')\n",
    "        output = F.softmax(model(img),dim=1)\n",
    "        pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability\n",
    "        correct += pred.eq(label.view_as(pred)).sum().item()\n",
    "        \n",
    "print(\"Accuracy\",correct/len(test_loader.dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Some Failures\n",
    "\n",
    "*Not from this particular network\n",
    "\n",
    "<center><img src=\"imgs/ensemble_errors.png\"></center>\n",
    "Top label is correct.  Bottom is prediction from a CNN."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generative vs. Discriminative\n",
    "\n",
    "A *generative* model produces as output the input of a discriminative model: $P(X|Y=y)$ *or* $P(X,Y)$\n",
    "\n",
    "<p>\n",
    "<div style=\"font-size: 150%\">$y \\rightarrow$<div style=\"display: inline-block; background-color: black; width: 5em; color: white; font-size: 120%; padding: 1em; font-weight: bold\">Model</div>\n",
    "$ \\rightarrow X$</div>\n",
    "\n",
    "<p>\n",
    "<div style=\"font-size: 150%\"><div style=\"display: inline-block; background-color: black; width: 5em; color: white; font-size: 120%; padding: 1em; font-weight: bold\">Model</div>\n",
    "$ \\rightarrow X,y$</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Autoencoders\n",
    "\n",
    "A neural network trained to generate its input.\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/2/28/Autoencoder_structure.png\" width=\"600\">\n",
    "\n",
    "https://en.wikipedia.org/wiki/Autoencoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Latent Space\n",
    "\n",
    "http://blog.fastforwardlabs.com/2016/08/12/introducing-variational-autoencoders-in-prose-and.html\n",
    "\n",
    "<img src=\"http://fastforwardlabs.github.io/blog-images/miriam/tableau.1493x693.png\" width=\"800\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Latent Space Arithmetic\n",
    "\n",
    "https://arxiv.org/pdf/1707.05776.pdf\n",
    "\n",
    "<img src=\"../files/latentadd.png\" width=\"600\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Latent Spaces for Molecules\n",
    "\n",
    "https://arxiv.org/abs/1610.02415\n",
    "\n",
    "<table style=\"border:none\">\n",
    "<tr style=\"border:none\">\n",
    "<td style=\"border:none\">\n",
    "<img src=\"../files/latentchem1.png\" width=\"300\">\n",
    "1% - 70% of output valid SMILES \n",
    "</td>\n",
    "<td style=\"border:none\">\n",
    "<img src=\"../files/latentchem2.png\" width=\"450\">\n",
    "<img src=\"../files/latentchem3.png\" width=\"450\">\n",
    "</td>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generative Models of the Cell\n",
    "\n",
    "https://arxiv.org/pdf/1705.00092.pdf\n",
    "\n",
    "<table style=\"border:none\">\n",
    "<tr style=\"border:none\">\n",
    "<td style=\"border:none\">\n",
    "<img src=\"../files/cellgen1.png\" width=\"400\">\n",
    "</td>\n",
    "<td style=\"border:none\">\n",
    "<img src=\"../files/cellgen2.png\" width=\"400\">\n",
    "</td>\n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "https://drive.google.com/file/d/0B2tsfjLgpFVhMnhwUVVuQnJxZTg/view"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Generative Adversarial Networks\n",
    "\n",
    "https://arxiv.org/abs/1406.2661\n",
    "<img src=\"https://i.stack.imgur.com/UnKny.png\" width=\"600\">\n",
    "https://youtu.be/G06dEcZ-QTg\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "https://thispersondoesnotexist.com\n",
    "\n",
    "<img src=\"https://thispersondoesnotexist.com\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# pix2pix\n",
    "\n",
    "<img src=\"https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/imgs/edges2cats.jpg?raw=true\">\n",
    "\n",
    "https://affinelayer.com/pixsrv/\n",
    "\n",
    "https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# CycleGAN\n",
    "\n",
    "https://medium.com/coding-blocks/introduction-to-cyclegans-1dbdb8fbe781\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/896/1*dWd0lVTbnu80UZM641gCbw.gif\">\n",
    "\n",
    "<center>\n",
    "<img src=\"https://miro.medium.com/max/520/1*1Qaj9C79S3dGeBpbWLXMEg.png\" style=\"display:inline-block\"><img src=\"https://miro.medium.com/max/1258/1*XhdrXh3UfCM4CecRrTwMCQ.png\" width=\"400\" style=\"display:inline-block\">\n",
    "</center>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "https://techcrunch.com/2018/12/31/this-clever-ai-hid-data-from-its-creators-to-cheat-at-its-appointed-task/\n",
    "\n",
    "<img src=\"https://techcrunch.com/wp-content/uploads/2018/12/mapdetails.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Deep learning is not profound learning.\n",
    "\n",
    "But it is quite powerful and flexible."
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}