diff --git a/notebooks/Conjugate Gradient.ipynb b/notebooks/Conjugate Gradient.ipynb
index 4c5d7e0d..102a0bd0 100644
--- a/notebooks/Conjugate Gradient.ipynb
+++ b/notebooks/Conjugate Gradient.ipynb
@@ -1,505 +1,512 @@
{
- "metadata": {
- "language": "haskell",
- "name": "",
- "signature": "sha256:8332eed5b1a2647ecfe6b707d1d07de0e8798861c517cac876970de5eb31e43c"
- },
- "nbformat": 3,
- "nbformat_minor": 0,
- "worksheets": [
+ "cells": [
{
- "cells": [
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In the previous notebook, we set up a framework for doing gradient-based minimization of differentiable functions (via the `GradientDescent` typeclass) and implemented simple gradient descent for univariate functions. Next, let's try to extend this framework to a faster method such as nonlinear Conjugate Gradient, and see what modifications we'll need to make in order to accomodate it.\n",
+ "$\\newcommand\\vector[1]{\\langle #1 \\rangle}\\newcommand\\p[2]{\\frac{\\partial #1}{\\partial #2}}\\newcommand\\R{\\mathbb{R}}$"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Conjugate Gradient\n",
+ "===\n",
+ "Before diving in to Haskell, let's go over exactly what the conjugate gradient method is and why it works. The \"normal\" conjugate gradient method is a method for solving systems of linear equations. However, this extends to a method for minimizing quadratic functions, which we can subsequently generalize to minimizing arbitrary functions $f\\!:\\!\\R^n \\to \\R$. We will start by going over the conjugate gradient method of minimizing quadratic functions, and later generalize.\n",
+ "\n",
+ "Suppose we have some quadratic function\n",
+ "$$f(x) = \\frac{1}{2}x^T A x + b^T x + c$$\n",
+ "for $x \\in \\R^n$ with $A \\in \\R^{n \\times n}$ and $b, c \\in \\R^n$.\n",
+ "\n",
+ "We can write any quadratic function in this form, as this generates all the coefficients $x_ix_j$ as well as linear and constant terms. In addition, we can assume that $A = A^T$ ($A$ is symmetric). (If it were not, we could just rewrite this with a symmetric $A$, since we could take the term for $x_i x_j$ and the term for $x_j x_i$, sum them, and then have $A_{ij} = A_{ji}$ both be half of this sum.)\n",
+ "\n",
+ "Taking the gradient of $f$, we obtain\n",
+ "$$\\nabla f(x) = A x + b,$$\n",
+ "which you can verify by writing out the terms in summation notation.\n",
+ "\n",
+ "If we evaluate $-\\nabla f$ at any given location, it will give us a vector pointing towards the direction of steepest descent. This gives us a natural way to start our algorithm - pick some initial guess $x_0$, compute the gradient $-\\nabla f(x_0)$, and move in that direction by some step size $\\alpha$. Unlike normal gradient descent, however, we do not have a fixed step size $\\alpha$ - instead, we perform a line search in order to find the *best* $\\alpha$. This $\\alpha$ is the value of $\\alpha$ which brings us to the minimum of $f$ if we are constrainted to move in the direction given by $d_0 = -\\nabla f(x_0)$.\n",
+ "\n",
+ "Note that computing $\\alpha$ is equivalent to minimizing the function\n",
+ "$$\\begin{align*}\n",
+ "g(\\alpha) &= f(x_0 + \\alpha d_0) \\\\\n",
+ "&= \\frac{1}{2}(x_0 + \\alpha d_0)^T A (x_0 + \\alpha d_0) + b^T (x_0 + \\alpha d_0) + c\\\\\n",
+ "&= \\frac{1}{2}\\alpha^2 {d_0}^T A d_0 + {d_0}^T (A x_0 + b) \\alpha + (\\frac{1}{2} {x_0}^T A x_0 + {x_0}^T d_0 + c)\n",
+ "\\end{align*}$$\n",
+ "Since this is a quadratic function in $\\alpha$, it has a unique global minimum or maximum. Since we assume we are not at the minimum and not at a saddle point of $f$, we assume that it has a minimum. \n",
+ "\n",
+ "The minimum of this function occurs when $g'(\\alpha) = 0$, that is, when\n",
+ "$$g'(\\alpha) = ({d_i}^T A {d_i})\\alpha + {d_i}^T(A x_i + b) = 0.$$\n",
+ "\n",
+ "Solving this for $\\alpha$, we find that the minimum is at\n",
+ "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
+ "\n",
+ "Note that since the directon is the negative of the gradient, a.k.a. the direction of steepest descent, $\\alpha$ will be non-negative. These first steps give us our second point in our iterative algorithm:\n",
+ "$$x_1 = x_0 - \\alpha \\nabla f(x_0)$$\n",
+ "\n",
+ "If this were simple gradient descent, we would iterate this procedure, computing the gradient at each next point and moving in that direction. However, this has a problem - by moving $\\alpha_0$ in direction $d_0$ (to find the minimum in direction $d_0$) and then moving $\\alpha_1$ in direction $d_1$, we may *ruin* our work from the previous iteration, so that we are no longer at a minimum in direction $d_0$. In order to rectify this, we require that our directions be *conjugate* to one another.\n",
+ "\n",
+ "We define two vectors $x$ and $y$ to be conjugate with respect to some semi-definite matrix $A$ if $x^T A y = 0$. (Semi-definite matrices are ones where $x^T A x \\ge 0$ for all $x$, and are what we require for conjugate gradient.)\n",
+ "\n",
+ "Since we have already moved in the $d_0 = -\\nabla f(x_0)$ direction, we must find a new direction $d_1$ to move in that is conjugate to $d_0$. How do we do this? Well, let's compute $d_1$ by starting with the gradient at $x_1$ and then subtracting off anything that would counter-act the previous direction:\n",
+ "$$d_1 = -\\nabla f(x_1) + \\beta_0 d_0.$$\n",
+ "\n",
+ "This leaves us with the obvious question - what is $\\beta_0$? We can derive that from our definition of conjugacy. Since $d_0$ and $d_1$ must be conjugate, we know that ${d_1}^T A d_0 = 0$. Expanding $d_1$ by using its definition, we get that ${d_1}^T A d_0 = -\\nabla f(x_1)^TAd_0 + \\beta_0 {d_0}^TA d_0 = 0$. Therefore, we must choose $\\beta_0$ such that\n",
+ "$$\\beta_0 = \\frac{\\nabla f(x_1)^T A d_0}{{d_0}^T A d_0}.$$\n",
+ "\n",
+ "Choosing this $\\beta$ gives us a direction conjugate to all previous directions. Interestingly enough, iterating this will *keep* giving us conjugate directions. After generating each direction, we find the best $\\alpha$ for that direction and update the current estimate of position.\n",
+ "\n",
+ "Thus, the full Conjugate Gradient algorithm for quadratic functions:\n",
+ "\n",
+ "> Let $f$ be a quadratic function $f(x) = \\frac{1}{2}x^T A x + b^T x + c$\n",
+ "which we wish to minimize.\n",
+ "> 1. **Initialize:** \n",
+ "Let $i = 0$ and $x_i = x_0$ be our initial guess, and compute $d_i = d_0 = -\\nabla f(x_0)$.\n",
+ "> \n",
+ "> 2. **Find best step size:**\n",
+ "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via the equation\n",
+ "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
+ "> \n",
+ "> 3. **Update the current guess:**\n",
+ "Let $x_{i+1} = x_i + \\alpha d_i$.\n",
+ ">\n",
+ "> 4. **Update the direction:**\n",
+ "Let $d_{i+1} = -\\nabla f(x_{i+1}) + \\beta_i d_i$ where $\\beta_i$ is given by\n",
+ "$$\\beta_i = \\frac{\\nabla f(x_{i+1})^T A d_i}{{d_i}^T A d_i}.$$\n",
+ ">\n",
+ "> 5. **Iterate:** Repeat steps 2-4 until we have looked in $n$ directions, where $n$ is the size of your vector space (the dimension of $x$)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Nonlinear Conjugate Gradient\n",
+ "---\n",
+ "So, now that we've derived this for quadratic functions, how are we going to use this for general nonlinear optimization of differentiable functions? To do this, we're going to reformulate the above algorithm in *slightly* more general terms.\n",
+ "\n",
+ "First of all, we will revise step two. Instead of \n",
+ "\n",
+ "> **Find best step size:**\n",
+ "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via the equation\n",
+ "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
+ "\n",
+ "we will simply use a line search:\n",
+ "\n",
+ "> **Find best step size:**\n",
+ "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via a line search in the direction $d_i$.\n",
+ "\n",
+ "In addition, we must reformulate the computation of $\\beta_i$. There are several ways to do this, all of which are the same in the quadratic case but are different in the general nonlinear case. We reformulate this computation by generalizing. Note that the difference between $x_{k+1}$ and $x_k$ is entirely in the direction $d_k$, so that for some constant $c$, $x_{k+1} - x_k = c d_k$. Since $\\nabla f(x) = A x + b$, \n",
+ "$$ \\nabla f(x_{k+1}) - \\nabla f(x_k) = (A x_{k+1} + b) - (A x_k + b) = A(x_{k+1}-x_k) = cA d_k.$$\n",
+ "\n",
+ "Therefore, $A d_k = c^{-1} (\\nabla f(x_{k+1}) - \\nabla f(x_k))$. We can now plug this in to the equation for $\\beta_i$ and obtain\n",
+ "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
+ "\n",
+ "Conveniently enough, the value of $c$ cancels, as it is both in the numerator and denominator. This gives us the new update rule:\n",
+ "\n",
+ "> **Update the direction:**\n",
+ "Let $d_{k+1} = -\\nabla f(x_{k+1}) + \\beta_k d_k$ where $\\beta_k$ is given by\n",
+ "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
+ "\n",
+ "We can now apply this algorithm to any nonlinear and differentiable function! This reformulation of $\\beta$ is known as the Polak-Ribiere method; know that there are others, similar in form and also in use."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Line Search\n",
+ "---\n",
+ "The one remaining bit of this process that we haven't covered is step two: the line search. As you can see above, we are given a point $x$, some vector $v$, and a multivariate function $f\\!:\\!\\R^n \\to \\R$, and we wish to find the $\\alpha$ which minimizes $f(x + \\alpha v)$. Note that a line search can be viewed simply as root finding, since we know that $v \\cdot \\nabla f(x + \\alpha v)$ should be zero at the minimum. (Since if it were non-zero, we could move from that minimum to a better location.)\n",
+ "\n",
+ "There are many ways to do this line search, and they can range from relatively simple linear methods (like the [secant method](http://en.wikipedia.org/wiki/Secant_method)) to more complex (using quadratic or cubic polynomial approximations). \n",
+ "\n",
+ "One simple method for a line search is known as the **bisection method**. The bisection method is simply a binary search. To minimize a univariate function $g(x)$, it begins with two points, $a$ and $b$, such that $g(a)$ and $g(b)$ have opposite signs. By the intermediate value theorem, $g(x)$ must have a root in $[a, b]$. (Note that in our case, $g(\\alpha) = v \\cdot \\nabla f(x + \\alpha v)$.) It then computes their midpoint, $c = \\frac{a + b}{2}$, and evaluates the function $g$ to compute $g(c)$. If $g(a)$ and $g(c)$ have opposite signs, the root must be in $[a, c]$; if $g(c)$ and $g(b)$ have opposite signs, then $[c, b]$ must have the root. At this point, the method recurses, continuing its search until it has gotten close enough to the true $\\alpha$.\n",
+ "\n",
+ "Another simple method is known as the **secant method**. Like the bisection method, the secant method requires two initial points $a$ and $b$ such that $g(a)$ and $g(b)$ have opposite signs. However, instead of doing a simple binary search, it does linear interpolation. It finds the line between $(a, g(a))$ and $(b, g(b))$:\n",
+ "$$g(x) \\approx \\frac{g(b) - g(a)}{b - a}(x - a) + g(a)$$\n",
+ "\n",
+ "It then finds the root of this linear approximation, setting $g(x) = 0$ and finding that the root is at\n",
+ "$$\\frac{g(b) - g(a)}{b - a}(x - a) + g(a) = 0 \\implies x = a -\\frac{b - a}{g(b) - g(a)}g(a).$$ \n",
+ "\n",
+ "It then evaluates $g$ at this location $x$. As with the bisection method, if $g(x)$ and $g(a)$ have opposite signs, then the root is in $[a, x]$, and if $g(x)$ and $g(b)$ have opposite signs, the root must be in $[x, b]$. As before, root finding continues via iteration, until some stopping condition is reached.\n",
+ "\n",
+ "There are more line search methods, but the last one we will examine is one known as **Brent's method**. Brent's method is a combination of the secand method and the bisection method. Unlike the previous two methods, Brent's method keeps track of three points:\n",
+ "\n",
+ "- $a_k$: the current \"contrapoint\"\n",
+ "- $b_k$: the current guess for the root\n",
+ "- $b_{k-1}$: the previous guess for the root\n",
+ "\n",
+ "Brent's method then computes the two possible next values: $m$ (by using the bisection method) and $s$ (by using the secant method with $b_k$ and $b_{k-1}$). (On the very first iteration, $b_{k-1} = a_k$ and it uses the bisection method.) If the secant method result $s$ lies between $b_k$ and $m$, then let $b_{k+1} = s$; otherwise, let $b_{k+1} = m$.\n",
+ "\n",
+ "After $b_{k+1}$ is chosen, it is checked to for convergence. If the method has converged, iteration is stopped. If not, the method continues. A new contrapoint $a_{k+1}$ is chosen such that $b_{k+1}$ and $a_{k+1}$ have opposite signs. The two choices for $a_{k+1}$ are either for it to remain unchanged (stay $a_k$) or for it to become $b_k$ - the choice depends on the signs of the function values involved. Before repeating, the values of $f(a_k{+1})$ and $f(b_{k+1})$ are examined, and $b_{k+1}$ is swapped with $a_{k+1}$ if it has a higher function value. Finally, the method repeats with the new values of $a_k$, $b_k$, and $b_{k-1}$.\n",
+ "\n",
+ "Brent's method is effectively a heuristic method, but is nice in practice; it has the reliability of the bisection method and gains a boost of speed from its use of the secant method."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Implementation\n",
+ "---\n",
+ "\n",
+ "Now that we've reviewed the conjugate gradient method, let's revise our previous gradient descent framework to so that we can implement conjugate gradient (using Brent's method for its line search).\n",
+ "\n",
+ "Recall that in the previous notebook, we defined a class that allowed us to do gradient descent on arbitrary function-like data types:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Extensions and imports we'll need later.\n",
+ ":set -XTypeFamilies -XFlexibleContexts -XMultiParamTypeClasses -XDoAndIfThenElse -XFlexibleInstances\n",
+ "import Control.Monad.Writer\n",
+ "import Text.Printf"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "class Monad m => GradientDescent m a where\n",
+ " -- Type to represent the parameter space.\n",
+ " data Params a :: *\n",
+ " \n",
+ " -- Compute the gradient at a location in parameter space.\n",
+ " grad :: a -> Params a -> m (Params a)\n",
+ " \n",
+ " -- Move in parameter space.\n",
+ " paramMove :: Double -- Scaling factor.\n",
+ " -> Params a -- Direction vector.\n",
+ " -> Params a -- Original location.\n",
+ " -> m (Params a) -- New location."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This same class isn't going to work quite as nicely in this case, because we must be able to compute\n",
+ "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
+ "\n",
+ "Since both the gradients and the search directions are represented as vectors in the parameter space (`Param a`), we must be able to take the dot product of any two such vectors. We already have the capability to add and subtract them via `paramMove`, though.\n",
+ "\n",
+ "One option is to add something like `paramDot` to `GradientDescent`, and call it a day. One one hand, that is simple; on the other hand, it seems to conflate two independent notions - the ability to do gradient descent and the ability to use `Param a` as a vector space. Instead of doing that, we can require that the parameters form an inner product space:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- We will call this a vector space, though the definition actually\n",
+ "-- requires an inner product, since it requires an implementation of `dot`.\n",
+ "class VectorSpace v where\n",
+ " -- Add two vectors in this inner product space.\n",
+ " add :: v -> v -> v\n",
+ " \n",
+ " -- Scale a vector.\n",
+ " scale :: Double -> v -> v\n",
+ " \n",
+ " -- Take the inner product of two vectors.\n",
+ " dot :: v -> v -> Double\n",
+ " \n",
+ " -- For convenience.\n",
+ " minus :: v -> v -> v\n",
+ " minus a b = add a (scale (-1) b)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, instead of requiring `GradientDescent` instances to provide `paramMove`, we'll just require that the parameters form a vector space:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "class (Monad m, VectorSpace (Params a)) => GradientDescent m a where\n",
+ " -- Type to represent the parameter space.\n",
+ " data Params a :: *\n",
+ " \n",
+ " -- Compute the gradient at a location in parameter space.\n",
+ " grad :: a -> Params a -> m (Params a)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Great! Now we start implementing these methods. In order to avoid spending too much time on line searches, let's just go with a simple bisection search for the time being.\n",
+ "\n",
+ "The implementation is pretty simple:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- A point consisting of a value and the function at that value.\n",
+ "-- The stopping condition is implemented as a function\n",
+ "-- Point -> Point -> Bool\n",
+ "-- That way, the stopping condition can decide based on convergence\n",
+ "-- of the x-coordinate or of the function values.\n",
+ "newtype Point = Point {unPt :: (Double, Double)}\n",
+ "\n",
+ "bisectionSearch :: Monad m\n",
+ " => (Double -> m Double) -- What function f to find the root of\n",
+ " -> Double -- Starting point\n",
+ " -> Double -- Second starting point\n",
+ " -> (Point -> Point -> Bool) -- Whether to stop\n",
+ " -> m Double -- Approximate root location.\n",
+ "bisectionSearch f a b stop = do\n",
+ " let midpoint = (a + b) / 2\n",
+ " aValue <- f a\n",
+ " bValue <- f b\n",
+ " \n",
+ " -- Check if we're done with these two values.\n",
+ " if stop (Point (a, aValue)) (Point (b, bValue))\n",
+ " then \n",
+ " -- If we are, return their midpoint.\n",
+ " return midpoint\n",
+ " else do\n",
+ " -- If we're not done, change one of the values to the midpoint.\n",
+ " -- Keep the two values having opposite signs, though.\n",
+ " midvalue <- f midpoint\n",
+ " if signum midvalue /= signum aValue\n",
+ " then bisectionSearch f midpoint a stop\n",
+ " else bisectionSearch f midpoint b stop"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have our line search implemented, we can go ahead and implement the actual conjugate gradient algorithm."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "newtype StopCondition m a = StopWhen (Params a -> Params a -> m Bool)\n",
+ "\n",
+ "conjugateGradient :: GradientDescent m a =>\n",
+ " a -- What to optimize.\n",
+ " -> StopCondition m a -- When to stop.\n",
+ " -> Params a -- Initial point (x0).\n",
+ " -> m (Params a) -- Return: Location of minimum.\n",
+ "conjugateGradient f (StopWhen stop) x0 = go x0 Nothing\n",
+ " where\n",
+ " go x prevDir = do\n",
+ " -- Compute the search direction\n",
+ " gradVec <- grad f x\n",
+ " let dir = case prevDir of\n",
+ " -- If we have no previous direction, just use the gradient\n",
+ " Nothing -> scale (-1) gradVec\n",
+ "\n",
+ " -- If we have a previous direction, compute Beta and \n",
+ " -- then the conjugate direction in which to search.\n",
+ " Just (prevX, prevGrad, prevDir) ->\n",
+ " let diff = gradVec `minus` prevGrad\n",
+ " numerator = gradVec `dot` diff\n",
+ " denominator = prevDir `dot` diff\n",
+ " beta = max 0 $ numerator / denominator in\n",
+ " scale beta prevDir `minus` gradVec\n",
+ "\n",
+ " -- To minimize f(x + \\alpha d_k), we find the zero of\n",
+ " -- the dot product of the gradient and the direction\n",
+ " let lineVal alpha = do\n",
+ " let loc = x `add` scale alpha dir\n",
+ " gradient <- grad f loc\n",
+ " return $ gradient `dot` dir\n",
+ "\n",
+ " -- Stop when alpha is close enough\n",
+ " let stopLineSearch p1 p2 = \n",
+ " let val1 = fst $ unPt p1\n",
+ " val2 = fst $ unPt p2 in\n",
+ " abs (val1 - val2) < 0.1\n",
+ "\n",
+ " -- Find the best alpha value\n",
+ " alpha <- bisectionSearch lineVal 0 0.5 stopLineSearch\n",
+ "\n",
+ " -- Compute the new location, and check if we want to continue iterating.\n",
+ " let xNew = x `add` scale alpha dir\n",
+ " shouldStop <- stop x xNew\n",
+ " if shouldStop\n",
+ " then return xNew\n",
+ " else go xNew $ Just (x, gradVec, dir)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's try this out on a two-variable function. Since we do a line search, doing a single-dimensional conjugate gradient would be pointless."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- We need FlexibleInstances for declarations like these!\n",
+ "-- We must declare these instances together, because they have recursive dependencies on each other.\n",
+ "instance VectorSpace (Params (Double -> Double -> Double)) where\n",
+ " add (Arg a b) (Arg x y) = Arg (a + x) (b + y)\n",
+ " dot (Arg a b) (Arg x y) = a * x + b * y\n",
+ " scale s (Arg a b) = Arg (s * a) (s * b)\n",
+ " \n",
+ "-- In addition to our usual definition, let's log the number of function\n",
+ "-- gradient evaluations using a Writer monad.\n",
+ "instance GradientDescent (Writer [String]) (Double -> Double -> Double) where\n",
+ " -- The parameter for a function is just its argument.\n",
+ " data Params (Double -> Double -> Double) = Arg { x :: Double, y :: Double }\n",
+ "\n",
+ " -- Use numeric differentiation for taking the gradient.\n",
+ " grad f (Arg x y) = do\n",
+ " let dx = f x y - f (x - epsilon) y\n",
+ " dy = f x y - f x (y - epsilon)\n",
+ " gradient = (dx / epsilon, dy / epsilon)\n",
+ " tell [ \"Gradient at\\t\" ++ show' (x, y) ++ \"\\tis\\t\" ++ show' gradient ]\n",
+ " return $ uncurry Arg gradient\n",
+ " where epsilon = 0.0001\n",
+ " show' (x, y) = printf \"%.5f, \\t%.5f \" x y"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can define a function $f = x^2 + y^2 + 3$, which looks like this:\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This function has an obvious minimum at $f(0, 0) = 3$.\n",
+ "\n",
+ "Let's minimize this function using our conjugate gradient, and output the minimum and the gradient evaluation logs:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
{
- "cell_type": "markdown",
+ "data": {
+ "text/plain": [
+ "Min at x = 0.00078, y = 0.00054"
+ ]
+ },
"metadata": {},
- "source": [
- "In the previous notebook, we set up a framework for doing gradient-based minimization of differentiable functions (via the `GradientDescent` typeclass) and implemented simple gradient descent for univariate functions. Next, let's try to extend this framework to a faster method such as nonlinear Conjugate Gradient, and see what modifications we'll need to make in order to accomodate it.\n",
- "$\\newcommand\\vector[1]{\\langle #1 \\rangle}\\newcommand\\p[2]{\\frac{\\partial #1}{\\partial #2}}\\newcommand\\R{\\mathbb{R}}$"
- ]
+ "output_type": "display_data"
},
{
- "cell_type": "markdown",
+ "data": {
+ "text/plain": [
+ "Gradient at\t3.00000, \t2.00000 \tis\t5.99990, \t3.99990 \n",
+ "Gradient at\t3.00000, \t2.00000 \tis\t5.99990, \t3.99990 \n",
+ "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
+ "Gradient at\t1.50002, \t1.00002 \tis\t2.99995, \t1.99995 \n",
+ "Gradient at\t1.50002, \t1.00002 \tis\t2.99995, \t1.99995 \n",
+ "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
+ "Gradient at\t0.75004, \t0.50004 \tis\t1.49997, \t0.99998 \n",
+ "Gradient at\t0.75004, \t0.50004 \tis\t1.49997, \t0.99998 \n",
+ "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
+ "Gradient at\t0.37504, \t0.25004 \tis\t0.74999, \t0.49999"
+ ]
+ },
"metadata": {},
- "source": [
- "Conjugate Gradient\n",
- "===\n",
- "Before diving in to Haskell, let's go over exactly what the conjugate gradient method is and why it works. The \"normal\" conjugate gradient method is a method for solving systems of linear equations. However, this extends to a method for minimizing quadratic functions, which we can subsequently generalize to minimizing arbitrary functions $f\\!:\\!\\R^n \\to \\R$. We will start by going over the conjugate gradient method of minimizing quadratic functions, and later generalize.\n",
- "\n",
- "Suppose we have some quadratic function\n",
- "$$f(x) = \\frac{1}{2}x^T A x + b^T x + c$$\n",
- "for $x \\in \\R^n$ with $A \\in \\R^{n \\times n}$ and $b, c \\in \\R^n$.\n",
- "\n",
- "We can write any quadratic function in this form, as this generates all the coefficients $x_ix_j$ as well as linear and constant terms. In addition, we can assume that $A = A^T$ ($A$ is symmetric). (If it were not, we could just rewrite this with a symmetric $A$, since we could take the term for $x_i x_j$ and the term for $x_j x_i$, sum them, and then have $A_{ij} = A_{ji}$ both be half of this sum.)\n",
- "\n",
- "Taking the gradient of $f$, we obtain\n",
- "$$\\nabla f(x) = A x + b,$$\n",
- "which you can verify by writing out the terms in summation notation.\n",
- "\n",
- "If we evaluate $-\\nabla f$ at any given location, it will give us a vector pointing towards the direction of steepest descent. This gives us a natural way to start our algorithm - pick some initial guess $x_0$, compute the gradient $-\\nabla f(x_0)$, and move in that direction by some step size $\\alpha$. Unlike normal gradient descent, however, we do not have a fixed step size $\\alpha$ - instead, we perform a line search in order to find the *best* $\\alpha$. This $\\alpha$ is the value of $\\alpha$ which brings us to the minimum of $f$ if we are constrainted to move in the direction given by $d_0 = -\\nabla f(x_0)$.\n",
- "\n",
- "Note that computing $\\alpha$ is equivalent to minimizing the function\n",
- "$$\\begin{align*}\n",
- "g(\\alpha) &= f(x_0 + \\alpha d_0) \\\\\n",
- "&= \\frac{1}{2}(x_0 + \\alpha d_0)^T A (x_0 + \\alpha d_0) + b^T (x_0 + \\alpha d_0) + c\\\\\n",
- "&= \\frac{1}{2}\\alpha^2 {d_0}^T A d_0 + {d_0}^T (A x_0 + b) \\alpha + (\\frac{1}{2} {x_0}^T A x_0 + {x_0}^T d_0 + c)\n",
- "\\end{align*}$$\n",
- "Since this is a quadratic function in $\\alpha$, it has a unique global minimum or maximum. Since we assume we are not at the minimum and not at a saddle point of $f$, we assume that it has a minimum. \n",
- "\n",
- "The minimum of this function occurs when $g'(\\alpha) = 0$, that is, when\n",
- "$$g'(\\alpha) = ({d_i}^T A {d_i})\\alpha + {d_i}^T(A x_i + b) = 0.$$\n",
- "\n",
- "Solving this for $\\alpha$, we find that the minimum is at\n",
- "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
- "\n",
- "Note that since the directon is the negative of the gradient, a.k.a. the direction of steepest descent, $\\alpha$ will be non-negative. These first steps give us our second point in our iterative algorithm:\n",
- "$$x_1 = x_0 - \\alpha \\nabla f(x_0)$$\n",
- "\n",
- "If this were simple gradient descent, we would iterate this procedure, computing the gradient at each next point and moving in that direction. However, this has a problem - by moving $\\alpha_0$ in direction $d_0$ (to find the minimum in direction $d_0$) and then moving $\\alpha_1$ in direction $d_1$, we may *ruin* our work from the previous iteration, so that we are no longer at a minimum in direction $d_0$. In order to rectify this, we require that our directions be *conjugate* to one another.\n",
- "\n",
- "We define two vectors $x$ and $y$ to be conjugate with respect to some semi-definite matrix $A$ if $x^T A y = 0$. (Semi-definite matrices are ones where $x^T A x \\ge 0$ for all $x$, and are what we require for conjugate gradient.)\n",
- "\n",
- "Since we have already moved in the $d_0 = -\\nabla f(x_0)$ direction, we must find a new direction $d_1$ to move in that is conjugate to $d_0$. How do we do this? Well, let's compute $d_1$ by starting with the gradient at $x_1$ and then subtracting off anything that would counter-act the previous direction:\n",
- "$$d_1 = -\\nabla f(x_1) + \\beta_0 d_0.$$\n",
- "\n",
- "This leaves us with the obvious question - what is $\\beta_0$? We can derive that from our definition of conjugacy. Since $d_0$ and $d_1$ must be conjugate, we know that ${d_1}^T A d_0 = 0$. Expanding $d_1$ by using its definition, we get that ${d_1}^T A d_0 = -\\nabla f(x_1)^TAd_0 + \\beta_0 {d_0}^TA d_0 = 0$. Therefore, we must choose $\\beta_0$ such that\n",
- "$$\\beta_0 = \\frac{\\nabla f(x_1)^T A d_0}{{d_0}^T A d_0}.$$\n",
- "\n",
- "Choosing this $\\beta$ gives us a direction conjugate to all previous directions. Interestingly enough, iterating this will *keep* giving us conjugate directions. After generating each direction, we find the best $\\alpha$ for that direction and update the current estimate of position.\n",
- "\n",
- "Thus, the full Conjugate Gradient algorithm for quadratic functions:\n",
- "\n",
- "> Let $f$ be a quadratic function $f(x) = \\frac{1}{2}x^T A x + b^T x + c$\n",
- "which we wish to minimize.\n",
- "> 1. **Initialize:** \n",
- "Let $i = 0$ and $x_i = x_0$ be our initial guess, and compute $d_i = d_0 = -\\nabla f(x_0)$.\n",
- "> \n",
- "> 2. **Find best step size:**\n",
- "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via the equation\n",
- "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
- "> \n",
- "> 3. **Update the current guess:**\n",
- "Let $x_{i+1} = x_i + \\alpha d_i$.\n",
- ">\n",
- "> 4. **Update the direction:**\n",
- "Let $d_{i+1} = -\\nabla f(x_{i+1}) + \\beta_i d_i$ where $\\beta_i$ is given by\n",
- "$$\\beta_i = \\frac{\\nabla f(x_{i+1})^T A d_i}{{d_i}^T A d_i}.$$\n",
- ">\n",
- "> 5. **Iterate:** Repeat steps 2-4 until we have looked in $n$ directions, where $n$ is the size of your vector space (the dimension of $x$)."
- ]
+ "output_type": "display_data"
},
{
- "cell_type": "markdown",
+ "data": {
+ "text/plain": [
+ "... and so on ... (39 evaluations)"
+ ]
+ },
"metadata": {},
- "source": [
- "Nonlinear Conjugate Gradient\n",
- "---\n",
- "So, now that we've derived this for quadratic functions, how are we going to use this for general nonlinear optimization of differentiable functions? To do this, we're going to reformulate the above algorithm in *slightly* more general terms.\n",
- "\n",
- "First of all, we will revise step two. Instead of \n",
- "\n",
- "> **Find best step size:**\n",
- "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via the equation\n",
- "$$\\alpha = -\\frac{{d_i}^T (A x_i + b)}{{d_i}^T A d_i}.$$\n",
- "\n",
- "we will simply use a line search:\n",
- "\n",
- "> **Find best step size:**\n",
- "Compute $\\alpha$ to minimize the function $f(x_i + \\alpha d_i)$ via a line search in the direction $d_i$.\n",
- "\n",
- "In addition, we must reformulate the computation of $\\beta_i$. There are several ways to do this, all of which are the same in the quadratic case but are different in the general nonlinear case. We reformulate this computation by generalizing. Note that the difference between $x_{k+1}$ and $x_k$ is entirely in the direction $d_k$, so that for some constant $c$, $x_{k+1} - x_k = c d_k$. Since $\\nabla f(x) = A x + b$, \n",
- "$$ \\nabla f(x_{k+1}) - \\nabla f(x_k) = (A x_{k+1} + b) - (A x_k + b) = A(x_{k+1}-x_k) = cA d_k.$$\n",
- "\n",
- "Therefore, $A d_k = c^{-1} (\\nabla f(x_{k+1}) - \\nabla f(x_k))$. We can now plug this in to the equation for $\\beta_i$ and obtain\n",
- "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
- "\n",
- "Conveniently enough, the value of $c$ cancels, as it is both in the numerator and denominator. This gives us the new update rule:\n",
- "\n",
- "> **Update the direction:**\n",
- "Let $d_{k+1} = -\\nabla f(x_{k+1}) + \\beta_k d_k$ where $\\beta_k$ is given by\n",
- "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
- "\n",
- "We can now apply this algorithm to any nonlinear and differentiable function! This reformulation of $\\beta$ is known as the Polak-Ribiere method; know that there are others, similar in form and also in use."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Line Search\n",
- "---\n",
- "The one remaining bit of this process that we haven't covered is step two: the line search. As you can see above, we are given a point $x$, some vector $v$, and a multivariate function $f\\!:\\!\\R^n \\to \\R$, and we wish to find the $\\alpha$ which minimizes $f(x + \\alpha v)$. Note that a line search can be viewed simply as root finding, since we know that $v \\cdot \\nabla f(x + \\alpha v)$ should be zero at the minimum. (Since if it were non-zero, we could move from that minimum to a better location.)\n",
- "\n",
- "There are many ways to do this line search, and they can range from relatively simple linear methods (like the [secant method](http://en.wikipedia.org/wiki/Secant_method)) to more complex (using quadratic or cubic polynomial approximations). \n",
- "\n",
- "One simple method for a line search is known as the **bisection method**. The bisection method is simply a binary search. To minimize a univariate function $g(x)$, it begins with two points, $a$ and $b$, such that $g(a)$ and $g(b)$ have opposite signs. By the intermediate value theorem, $g(x)$ must have a root in $[a, b]$. (Note that in our case, $g(\\alpha) = v \\cdot \\nabla f(x + \\alpha v)$.) It then computes their midpoint, $c = \\frac{a + b}{2}$, and evaluates the function $g$ to compute $g(c)$. If $g(a)$ and $g(c)$ have opposite signs, the root must be in $[a, c]$; if $g(c)$ and $g(b)$ have opposite signs, then $[c, b]$ must have the root. At this point, the method recurses, continuing its search until it has gotten close enough to the true $\\alpha$.\n",
- "\n",
- "Another simple method is known as the **secant method**. Like the bisection method, the secant method requires two initial points $a$ and $b$ such that $g(a)$ and $g(b)$ have opposite signs. However, instead of doing a simple binary search, it does linear interpolation. It finds the line between $(a, g(a))$ and $(b, g(b))$:\n",
- "$$g(x) \\approx \\frac{g(b) - g(a)}{b - a}(x - a) + g(a)$$\n",
- "\n",
- "It then finds the root of this linear approximation, setting $g(x) = 0$ and finding that the root is at\n",
- "$$\\frac{g(b) - g(a)}{b - a}(x - a) + g(a) = 0 \\implies x = a -\\frac{b - a}{g(b) - g(a)}g(a).$$ \n",
- "\n",
- "It then evaluates $g$ at this location $x$. As with the bisection method, if $g(x)$ and $g(a)$ have opposite signs, then the root is in $[a, x]$, and if $g(x)$ and $g(b)$ have opposite signs, the root must be in $[x, b]$. As before, root finding continues via iteration, until some stopping condition is reached.\n",
- "\n",
- "There are more line search methods, but the last one we will examine is one known as **Brent's method**. Brent's method is a combination of the secand method and the bisection method. Unlike the previous two methods, Brent's method keeps track of three points:\n",
- "\n",
- "- $a_k$: the current \"contrapoint\"\n",
- "- $b_k$: the current guess for the root\n",
- "- $b_{k-1}$: the previous guess for the root\n",
- "\n",
- "Brent's method then computes the two possible next values: $m$ (by using the bisection method) and $s$ (by using the secant method with $b_k$ and $b_{k-1}$). (On the very first iteration, $b_{k-1} = a_k$ and it uses the bisection method.) If the secant method result $s$ lies between $b_k$ and $m$, then let $b_{k+1} = s$; otherwise, let $b_{k+1} = m$.\n",
- "\n",
- "After $b_{k+1}$ is chosen, it is checked to for convergence. If the method has converged, iteration is stopped. If not, the method continues. A new contrapoint $a_{k+1}$ is chosen such that $b_{k+1}$ and $a_{k+1}$ have opposite signs. The two choices for $a_{k+1}$ are either for it to remain unchanged (stay $a_k$) or for it to become $b_k$ - the choice depends on the signs of the function values involved. Before repeating, the values of $f(a_k{+1})$ and $f(b_{k+1})$ are examined, and $b_{k+1}$ is swapped with $a_{k+1}$ if it has a higher function value. Finally, the method repeats with the new values of $a_k$, $b_k$, and $b_{k-1}$.\n",
- "\n",
- "Brent's method is effectively a heuristic method, but is nice in practice; it has the reliability of the bisection method and gains a boost of speed from its use of the secant method."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Implementation\n",
- "---\n",
- "\n",
- "Now that we've reviewed the conjugate gradient method, let's revise our previous gradient descent framework to so that we can implement conjugate gradient (using Brent's method for its line search).\n",
- "\n",
- "Recall that in the previous notebook, we defined a class that allowed us to do gradient descent on arbitrary function-like data types:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Extensions and imports we'll need later.\n",
- ":set -XTypeFamilies -XFlexibleContexts -XMultiParamTypeClasses -XDoAndIfThenElse -XFlexibleInstances\n",
- "import Control.Monad.Writer\n",
- "import Text.Printf"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 1
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "class Monad m => GradientDescent m a where\n",
- " -- Type to represent the parameter space.\n",
- " data Params a :: *\n",
- " \n",
- " -- Compute the gradient at a location in parameter space.\n",
- " grad :: a -> Params a -> m (Params a)\n",
- " \n",
- " -- Move in parameter space.\n",
- " paramMove :: Double -- Scaling factor.\n",
- " -> Params a -- Direction vector.\n",
- " -> Params a -- Original location.\n",
- " -> m (Params a) -- New location."
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 2
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This same class isn't going to work quite as nicely in this case, because we must be able to compute\n",
- "$$\\beta_k = \\frac{\\nabla f(x_{k+1})^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}{{d_k}^T (\\nabla f(x_{k+1}) - \\nabla f(x_k))}.$$\n",
- "\n",
- "Since both the gradients and the search directions are represented as vectors in the parameter space (`Param a`), we must be able to take the dot product of any two such vectors. We already have the capability to add and subtract them via `paramMove`, though.\n",
- "\n",
- "One option is to add something like `paramDot` to `GradientDescent`, and call it a day. One one hand, that is simple; on the other hand, it seems to conflate two independent notions - the ability to do gradient descent and the ability to use `Param a` as a vector space. Instead of doing that, we can require that the parameters form an inner product space:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- We will call this a vector space, though the definition actually\n",
- "-- requires an inner product, since it requires an implementation of `dot`.\n",
- "class VectorSpace v where\n",
- " -- Add two vectors in this inner product space.\n",
- " add :: v -> v -> v\n",
- " \n",
- " -- Scale a vector.\n",
- " scale :: Double -> v -> v\n",
- " \n",
- " -- Take the inner product of two vectors.\n",
- " dot :: v -> v -> Double\n",
- " \n",
- " -- For convenience.\n",
- " minus :: v -> v -> v\n",
- " minus a b = add a (scale (-1) b)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 3
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, instead of requiring `GradientDescent` instances to provide `paramMove`, we'll just require that the parameters form a vector space:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "class (Monad m, VectorSpace (Params a)) => GradientDescent m a where\n",
- " -- Type to represent the parameter space.\n",
- " data Params a :: *\n",
- " \n",
- " -- Compute the gradient at a location in parameter space.\n",
- " grad :: a -> Params a -> m (Params a)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 4
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Great! Now we start implementing these methods. In order to avoid spending too much time on line searches, let's just go with a simple bisection search for the time being.\n",
- "\n",
- "The implementation is pretty simple:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- A point consisting of a value and the function at that value.\n",
- "-- The stopping condition is implemented as a function\n",
- "-- Point -> Point -> Bool\n",
- "-- That way, the stopping condition can decide based on convergence\n",
- "-- of the x-coordinate or of the function values.\n",
- "newtype Point = Point {unPt :: (Double, Double)}\n",
- "\n",
- "bisectionSearch :: Monad m\n",
- " => (Double -> m Double) -- What function f to find the root of\n",
- " -> Double -- Starting point\n",
- " -> Double -- Second starting point\n",
- " -> (Point -> Point -> Bool) -- Whether to stop\n",
- " -> m Double -- Approximate root location.\n",
- "bisectionSearch f a b stop = do\n",
- " let midpoint = (a + b) / 2\n",
- " aValue <- f a\n",
- " bValue <- f b\n",
- " \n",
- " -- Check if we're done with these two values.\n",
- " if stop (Point (a, aValue)) (Point (b, bValue))\n",
- " then \n",
- " -- If we are, return their midpoint.\n",
- " return midpoint\n",
- " else do\n",
- " -- If we're not done, change one of the values to the midpoint.\n",
- " -- Keep the two values having opposite signs, though.\n",
- " midvalue <- f midpoint\n",
- " if signum midvalue /= signum aValue\n",
- " then bisectionSearch f midpoint a stop\n",
- " else bisectionSearch f midpoint b stop"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 5
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we have our line search implemented, we can go ahead and implement the actual conjugate gradient algorithm."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "newtype StopCondition m a = StopWhen (Params a -> Params a -> m Bool)\n",
- "\n",
- "conjugateGradient :: GradientDescent m a =>\n",
- " a -- What to optimize.\n",
- " -> StopCondition m a -- When to stop.\n",
- " -> Params a -- Initial point (x0).\n",
- " -> m (Params a) -- Return: Location of minimum.\n",
- "conjugateGradient f (StopWhen stop) x0 = go x0 Nothing\n",
- " where\n",
- " go x prevDir = do\n",
- " -- Compute the search direction\n",
- " gradVec <- grad f x\n",
- " let dir = case prevDir of\n",
- " -- If we have no previous direction, just use the gradient\n",
- " Nothing -> scale (-1) gradVec\n",
- "\n",
- " -- If we have a previous direction, compute Beta and \n",
- " -- then the conjugate direction in which to search.\n",
- " Just (prevX, prevGrad, prevDir) ->\n",
- " let diff = gradVec `minus` prevGrad\n",
- " numerator = gradVec `dot` diff\n",
- " denominator = prevDir `dot` diff\n",
- " beta = max 0 $ numerator / denominator in\n",
- " scale beta prevDir `minus` gradVec\n",
- "\n",
- " -- To minimize f(x + \\alpha d_k), we find the zero of\n",
- " -- the dot product of the gradient and the direction\n",
- " let lineVal alpha = do\n",
- " let loc = x `add` scale alpha dir\n",
- " gradient <- grad f loc\n",
- " return $ gradient `dot` dir\n",
- "\n",
- " -- Stop when alpha is close enough\n",
- " let stopLineSearch p1 p2 = \n",
- " let val1 = fst $ unPt p1\n",
- " val2 = fst $ unPt p2 in\n",
- " abs (val1 - val2) < 0.1\n",
- "\n",
- " -- Find the best alpha value\n",
- " alpha <- bisectionSearch lineVal 0 0.5 stopLineSearch\n",
- "\n",
- " -- Compute the new location, and check if we want to continue iterating.\n",
- " let xNew = x `add` scale alpha dir\n",
- " shouldStop <- stop x xNew\n",
- " if shouldStop\n",
- " then return xNew\n",
- " else go xNew $ Just (x, gradVec, dir)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 6
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's try this out on a two-variable function. Since we do a line search, doing a single-dimensional conjugate gradient would be pointless."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- We need FlexibleInstances for declarations like these!\n",
- "-- We must declare these instances together, because they have recursive dependencies on each other.\n",
- "instance VectorSpace (Params (Double -> Double -> Double)) where\n",
- " add (Arg a b) (Arg x y) = Arg (a + x) (b + y)\n",
- " dot (Arg a b) (Arg x y) = a * x + b * y\n",
- " scale s (Arg a b) = Arg (s * a) (s * b)\n",
- " \n",
- "-- In addition to our usual definition, let's log the number of function\n",
- "-- gradient evaluations using a Writer monad.\n",
- "instance GradientDescent (Writer [String]) (Double -> Double -> Double) where\n",
- " -- The parameter for a function is just its argument.\n",
- " data Params (Double -> Double -> Double) = Arg { x :: Double, y :: Double }\n",
- "\n",
- " -- Use numeric differentiation for taking the gradient.\n",
- " grad f (Arg x y) = do\n",
- " let dx = f x y - f (x - epsilon) y\n",
- " dy = f x y - f x (y - epsilon)\n",
- " gradient = (dx / epsilon, dy / epsilon)\n",
- " tell [ \"Gradient at\\t\" ++ show' (x, y) ++ \"\\tis\\t\" ++ show' gradient ]\n",
- " return $ uncurry Arg gradient\n",
- " where epsilon = 0.0001\n",
- " show' (x, y) = printf \"%.5f, \\t%.5f \" x y"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 7
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can define a function $f = x^2 + y^2 + 3$, which looks like this:\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This function has an obvious minimum at $f(0, 0) = 3$.\n",
- "\n",
- "Let's minimize this function using our conjugate gradient, and output the minimum and the gradient evaluation logs:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Create a stop condition that respects a given error tolerance.\n",
- "stopCondition f tolerance = StopWhen stop\n",
- " where stop (Arg a b) (Arg x y) = do\n",
- " Arg dx dy <- grad f (Arg x y)\n",
- " return $ abs dx < tolerance && abs dy < tolerance\n",
- "\n",
- "-- A demo function with minimum at (0, 0)\n",
- "function x y = x^2 + y^2 + 3\n",
- "\n",
- "-- Note that we don't need to set an alpha!\n",
- "let tolerance = 1e-2\n",
- "let initValue = Arg 3 2\n",
- "let writer = conjugateGradient function (stopCondition function tolerance) initValue\n",
- " (minLoc, messages) = runWriter writer :: (Params (Double -> Double -> Double), [String])\n",
- "\n",
- "printf \"Min at x = %.5f, y = %.5f\\n\" (x minLoc) (y minLoc)\n",
- "mapM_ putStrLn (take 10 messages)\n",
- "printf \"... and so on ... (%d evaluations)\\n\" $ length messages"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "Min at x = 0.00078, y = 0.00054"
- ]
- },
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "Gradient at\t3.00000, \t2.00000 \tis\t5.99990, \t3.99990 \n",
- "Gradient at\t3.00000, \t2.00000 \tis\t5.99990, \t3.99990 \n",
- "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
- "Gradient at\t1.50002, \t1.00002 \tis\t2.99995, \t1.99995 \n",
- "Gradient at\t1.50002, \t1.00002 \tis\t2.99995, \t1.99995 \n",
- "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
- "Gradient at\t0.75004, \t0.50004 \tis\t1.49997, \t0.99998 \n",
- "Gradient at\t0.75004, \t0.50004 \tis\t1.49997, \t0.99998 \n",
- "Gradient at\t0.00005, \t0.00005 \tis\t-0.00000, \t0.00000 \n",
- "Gradient at\t0.37504, \t0.25004 \tis\t0.74999, \t0.49999"
- ]
- },
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "... and so on ... (39 evaluations)"
- ]
- }
- ],
- "prompt_number": 8
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "That concludes the **Conjugate Gradient** notebook. We've derived the nonlinear conjugate gradient algorithm as well as a few simple line searches for it, and then implemented it in our previously-discussed Gradient Descent typeclass heirarchy."
- ]
+ "output_type": "display_data"
}
],
- "metadata": {}
+ "source": [
+ "-- Create a stop condition that respects a given error tolerance.\n",
+ "stopCondition f tolerance = StopWhen stop\n",
+ " where stop (Arg a b) (Arg x y) = do\n",
+ " Arg dx dy <- grad f (Arg x y)\n",
+ " return $ abs dx < tolerance && abs dy < tolerance\n",
+ "\n",
+ "-- A demo function with minimum at (0, 0)\n",
+ "function x y = x^2 + y^2 + 3\n",
+ "\n",
+ "-- Note that we don't need to set an alpha!\n",
+ "let tolerance = 1e-2\n",
+ "let initValue = Arg 3 2\n",
+ "let writer = conjugateGradient function (stopCondition function tolerance) initValue\n",
+ " (minLoc, messages) = runWriter writer :: (Params (Double -> Double -> Double), [String])\n",
+ "\n",
+ "printf \"Min at x = %.5f, y = %.5f\\n\" (x minLoc) (y minLoc)\n",
+ "mapM_ putStrLn (take 10 messages)\n",
+ "printf \"... and so on ... (%d evaluations)\\n\" $ length messages"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "That concludes the **Conjugate Gradient** notebook. We've derived the nonlinear conjugate gradient algorithm as well as a few simple line searches for it, and then implemented it in our previously-discussed Gradient Descent typeclass heirarchy."
+ ]
}
- ]
-}
\ No newline at end of file
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Haskell",
+ "language": "haskell",
+ "name": "haskell"
+ },
+ "language_info": {
+ "name": "haskell",
+ "version": "7.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/notebooks/Diagrams.ipynb b/notebooks/Diagrams.ipynb
deleted file mode 100644
index f5d913e4..00000000
--- a/notebooks/Diagrams.ipynb
+++ /dev/null
@@ -1,92 +0,0 @@
-{
- "metadata": {
- "language": "haskell",
- "name": "",
- "signature": "sha256:d3903a8bf1d0cc71c024d88777d7a39da001586a55681e8e24409f5c12e0b13f"
- },
- "nbformat": 3,
- "nbformat_minor": 0,
- "worksheets": [
- {
- "cells": [
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- ":ext NoMonomorphismRestriction\n",
- "import Diagrams.Prelude\n",
- "import Data.Colour.SRGB\n",
- "import Data.List.Split (chunksOf)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 1
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Generate a board of size `size`\n",
- "board :: Int -> ((Int, Int) -> Diagram b R2) -> Diagram b R2\n",
- "board size sq = vcat rows\n",
- " where rows = map hcat $ chunksOf size diagrams\n",
- " diagrams = map (sq . tuple) indices\n",
- " indices = sequence [[0..size-1], [0..size-1]]\n",
- " tuple [a, b] = (a, b)\n",
- "\n",
- "-- Change visibility\n",
- "visible 0 diagram = diagram # opacity 0\n",
- "visible _ diagram = diagram\n",
- "\n",
- "-- Board square\n",
- "tile board (row, col) = visible value diagram\n",
- " where diagram = roundedRect size size 0.05 # fc color\n",
- " # pad padding\n",
- " padding = 1.15\n",
- " size = 1 / padding\n",
- " color = if value == 0\n",
- " then white\n",
- " else sRGB c c (c/3)\n",
- " c = (20.0 - value) / 20.0\n",
- " value = fromIntegral $ board !! row !! col \n",
- " \n",
- "-- Labels\n",
- "labels board (row, col) = visible value $ number `atop` sq\n",
- " where number = text (show $ 2 ^ value) # fontSize 0.5\n",
- " value = board !! row !! col\n",
- " sq = square 1 # opacity 0\n",
- " \n",
- "values = [[1, 2, 0], [0, 5, 6], [7, 0, 9]]\n",
- "\n",
- "diagram $ board 3 (labels values) \n",
- " `atop` board 3 (const $ square 1)\n",
- " `atop` board 3 (tile values)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "html": [
- "![](\"data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB3aWR0aD0iMzAwcHQiIGhlaWdodD0iMzAwcHQiIHZpZXdCb3g9IjAgMCAzMDAgMzAwIiB2ZXJzaW9uPSIxLjEiPgo8ZGVmcz4KPGc+CjxzeW1ib2wgb3ZlcmZsb3c9InZpc2libGUiIGlkPSJnbHlwaDAtMCI+CjxwYXRoIHN0eWxlPSJzdHJva2U6bm9uZTsiIGQ9IiIvPgo8L3N5bWJvbD4KPHN5bWJvbCBvdmVyZmxvdz0idmlzaWJsZSIgaWQ9ImdseXBoMC0xIj4KPHBhdGggc3R5bGU9InN0cm9rZTpub25lOyIgZD0iTSA2LjE3NTc4MSAtOC45MTAxNTYgQyA2LjQ2ODc1IC02LjQwMjM0NCA3LjYzMjgxMiAtNC42NzE4NzUgOS42Njc5NjkgLTMuNzEwOTM4IEMgMTAuNzEwOTM4IC0zLjIyMjY1NiAxMS45MTQwNjIgLTIuOTc2NTYyIDEzLjI4MTI1IC0yLjk3NjU2MiBDIDE1Ljg4NjcxOSAtMi45NzY1NjIgMTcuODEyNSAtMy44MDg1OTQgMTkuMDY2NDA2IC01LjQ2ODc1IEMgMjAuMzIwMzEyIC03LjEyODkwNiAyMC45NDUzMTIgLTguOTY4NzUgMjAuOTQ1MzEyIC0xMC45ODQzNzUgQyAyMC45NDUzMTIgLTEzLjQyNTc4MSAyMC4yMDMxMjUgLTE1LjMxNjQwNiAxOC43MTQ4NDQgLTE2LjY0ODQzOCBDIDE3LjIyMjY1NiAtMTcuOTg0Mzc1IDE1LjQzNzUgLTE4LjY1MjM0NCAxMy4zNTU0NjkgLTE4LjY1MjM0NCBDIDExLjgzOTg0NCAtMTguNjUyMzQ0IDEwLjU0Mjk2OSAtMTguMzU5Mzc1IDkuNDYwOTM4IC0xNy43NzM0MzggQyA4LjM3ODkwNiAtMTcuMTg3NSA3LjQ1MzEyNSAtMTYuMzc1IDYuNjg3NSAtMTUuMzMyMDMxIEwgMi44Nzg5MDYgLTE1LjU1MDc4MSBMIDUuNTQyOTY5IC0zNC4zNzUgTCAyMy43MDcwMzEgLTM0LjM3NSBMIDIzLjcwNzAzMSAtMzAuMTI1IEwgOC44MzU5MzggLTMwLjEyNSBMIDcuMzQ3NjU2IC0yMC40MTAxNTYgQyA4LjE2MDE1NiAtMjEuMDI3MzQ0IDguOTMzNTk0IC0yMS40OTIxODggOS42Njc5NjkgLTIxLjgwMDc4MSBDIDEwLjk2ODc1IC0yMi4zMzk4NDQgMTIuNDc2NTYyIC0yMi42MDU0NjkgMTQuMTgzNTk0IC0yMi42MDU0NjkgQyAxNy4zOTA2MjUgLTIyLjYwNTQ2OSAyMC4xMDkzNzUgLTIxLjU3NDIxOSAyMi4zMzk4NDQgLTE5LjUwNzgxMiBDIDI0LjU3MDMxMiAtMTcuNDQxNDA2IDI1LjY4MzU5NCAtMTQuODIwMzEyIDI1LjY4MzU5NCAtMTEuNjQ0NTMxIEMgMjUuNjgzNTk0IC04LjMzOTg0NCAyNC42NjQwNjIgLTUuNDI5Njg4IDIyLjYyMTA5NCAtMi45MDYyNSBDIDIwLjU3ODEyNSAtMC4zODI4MTIgMTcuMzE2NDA2IDAuODc4OTA2IDEyLjgzOTg0NCAwLjg3ODkwNiBDIDkuOTkyMTg4IDAuODc4OTA2IDcuNDc2NTYyIDAuMDc4MTI1IDUuMjg1MTU2IC0xLjUyNzM0NCBDIDMuMDk3NjU2IC0zLjEyODkwNiAxLjg3MTA5NCAtNS41ODk4NDQgMS42MDkzNzUgLTguOTEwMTU2IFogIi8+Cjwvc3ltYm9sPgo8c3ltYm9sIG92ZXJmbG93PSJ2aXNpYmxlIiBpZD0iZ2x5cGgwLTIiPgo8cGF0aCBzdHlsZT0ic3Ryb2tlOm5vbmU7IiBkPSJNIDQuNzg1MTU2IC0yNC43NTM5MDYgTCA0Ljc4NTE1NiAtMjguMTI1IEMgNy45NTcwMzEgLTI4LjQzMzU5NCAxMC4xNzE4NzUgLTI4Ljk0OTIxOSAxMS40MjU3ODEgLTI5LjY3NTc4MSBDIDEyLjY3OTY4OCAtMzAuMzk4NDM4IDEzLjYxMzI4MSAtMzIuMTEzMjgxIDE0LjIzNDM3NSAtMzQuODEyNSBMIDE3LjY5OTIxOSAtMzQuODEyNSBMIDE3LjY5OTIxOSAwIEwgMTMuMDExNzE5IDAgTCAxMy4wMTE3MTkgLTI0Ljc1MzkwNiBaICIvPgo8L3N5bWJvbD4KPHN5bWJvbCBvdmVyZmxvdz0idmlzaWJsZSIgaWQ9ImdseXBoMC0zIj4KPHBhdGggc3R5bGU9InN0cm9rZTpub25lOyIgZD0iTSAzLjQyOTY4OCAtNy44NTkzNzUgQyA0LjUxMTcxOSAtMTAuMDg5ODQ0IDYuNjI1IC0xMi4xMTcxODggOS43NjU2MjUgLTEzLjk0MTQwNiBMIDE0LjQ1MzEyNSAtMTYuNjQ4NDM4IEMgMTYuNTUwNzgxIC0xNy44NzEwOTQgMTguMDI3MzQ0IC0xOC45MTQwNjIgMTguODcxMDk0IC0xOS43NzM0MzggQyAyMC4yMDcwMzEgLTIxLjEyNSAyMC44NzUgLTIyLjY3MTg3NSAyMC44NzUgLTI0LjQxNDA2MiBDIDIwLjg3NSAtMjYuNDQ5MjE5IDIwLjI2MTcxOSAtMjguMDYyNSAxOS4wNDI5NjkgLTI5LjI2MTcxOSBDIDE3LjgyMDMxMiAtMzAuNDU3MDMxIDE2LjE5NTMxMiAtMzEuMDU0Njg4IDE0LjE2MDE1NiAtMzEuMDU0Njg4IEMgMTEuMTQ4NDM4IC0zMS4wNTQ2ODggOS4wNjY0MDYgLTI5LjkxNDA2MiA3LjkxMDE1NiAtMjcuNjM2NzE5IEMgNy4yOTI5NjkgLTI2LjQxNDA2MiA2Ljk0OTIxOSAtMjQuNzIyNjU2IDYuODgyODEyIC0yMi41NTg1OTQgTCAyLjQxNzk2OSAtMjIuNTU4NTk0IEMgMi40NjQ4NDQgLTI1LjYwMTU2MiAzLjAyNzM0NCAtMjguMDg1OTM4IDQuMTAxNTYyIC0zMC4wMDM5MDYgQyA2LjAwMzkwNiAtMzMuMzkwNjI1IDkuMzY3MTg4IC0zNS4wODIwMzEgMTQuMTgzNTk0IC0zNS4wODIwMzEgQyAxOC4xODc1IC0zNS4wODIwMzEgMjEuMTEzMjgxIC0zNCAyMi45NjA5MzggLTMxLjgzNTkzOCBDIDI0LjgwODU5NCAtMjkuNjcxODc1IDI1LjczMDQ2OSAtMjcuMjYxNzE5IDI1LjczMDQ2OSAtMjQuNjA5Mzc1IEMgMjUuNzMwNDY5IC0yMS44MDg1OTQgMjQuNzQ2MDk0IC0xOS40MTc5NjkgMjIuNzc3MzQ0IC0xNy40Mjk2ODggQyAyMS42MzY3MTkgLTE2LjI3MzQzOCAxOS41OTc2NTYgLTE0Ljg3NSAxNi42NDg0MzggLTEzLjIzMDQ2OSBMIDEzLjMwNDY4OCAtMTEuMzc1IEMgMTEuNzEwOTM4IC0xMC40OTYwOTQgMTAuNDU3MDMxIC05LjY2MDE1NiA5LjU0Njg3NSAtOC44NjMyODEgQyA3LjkxNzk2OSAtNy40NDUzMTIgNi44OTQ1MzEgLTUuODc1IDYuNDY4NzUgLTQuMTQ4NDM4IEwgMjUuNTYyNSAtNC4xNDg0MzggTCAyNS41NjI1IDAgTCAxLjU2MjUgMCBDIDEuNzI2NTYyIC0zLjAxMTcxOSAyLjM0NzY1NiAtNS42MzI4MTIgMy40Mjk2ODggLTcuODU5Mzc1IFogIi8+Cjwvc3ltYm9sPgo8c3ltYm9sIG92ZXJmbG93PSJ2aXNpYmxlIiBpZD0iZ2x5cGgwLTQiPgo8cGF0aCBzdHlsZT0ic3Ryb2tlOm5vbmU7IiBkPSJNIDE4LjE0MDYyNSAtMjEuOTM3NSBDIDE5LjIzMDQ2OSAtMjMuMDE5NTMxIDE5Ljc3MzQzOCAtMjQuMzA4NTk0IDE5Ljc3MzQzOCAtMjUuODA0Njg4IEMgMTkuNzczNDM4IC0yNy4xMDU0NjkgMTkuMjUzOTA2IC0yOC4zMDQ2ODggMTguMjEwOTM4IC0yOS4zOTQ1MzEgQyAxNy4xNzE4NzUgLTMwLjQ4NDM3NSAxNS41ODU5MzggLTMxLjAzMTI1IDEzLjQ1MzEyNSAtMzEuMDMxMjUgQyAxMS4zMzU5MzggLTMxLjAzMTI1IDkuODA0Njg4IC0zMC40ODQzNzUgOC44NjMyODEgLTI5LjM5NDUzMSBDIDcuOTE3OTY5IC0yOC4zMDQ2ODggNy40NDUzMTIgLTI3LjAyNzM0NCA3LjQ0NTMxMiAtMjUuNTYyNSBDIDcuNDQ1MzEyIC0yMy45MTc5NjkgOC4wNTQ2ODggLTIyLjYzMjgxMiA5LjI3NzM0NCAtMjEuNzAzMTI1IEMgMTAuNDk2MDk0IC0yMC43NzczNDQgMTEuOTM3NSAtMjAuMzEyNSAxMy41OTc2NTYgLTIwLjMxMjUgQyAxNS41MzUxNTYgLTIwLjMxMjUgMTcuMDUwNzgxIC0yMC44NTU0NjkgMTguMTQwNjI1IC0yMS45Mzc1IFogTSAxOC45MzM1OTQgLTQuNjUyMzQ0IEMgMjAuMjc3MzQ0IC01Ljc1IDIwLjk0NTMxMiAtNy4zOTA2MjUgMjAuOTQ1MzEyIC05LjU3MDMxMiBDIDIwLjk0NTMxMiAtMTEuODMyMDMxIDIwLjI1MzkwNiAtMTMuNTUwNzgxIDE4Ljg3MTA5NCAtMTQuNzIyNjU2IEMgMTcuNDg4MjgxIC0xNS44OTQ1MzEgMTUuNzE0ODQ0IC0xNi40ODA0NjkgMTMuNTUwNzgxIC0xNi40ODA0NjkgQyAxMS40NDkyMTkgLTE2LjQ4MDQ2OSA5LjczODI4MSAtMTUuODgyODEyIDguNDEwMTU2IC0xNC42ODM1OTQgQyA3LjA4MjAzMSAtMTMuNDg4MjgxIDYuNDIxODc1IC0xMS44MzIwMzEgNi40MjE4NzUgLTkuNzE0ODQ0IEMgNi40MjE4NzUgLTcuODk0NTMxIDcuMDI3MzQ0IC02LjMyMDMxMiA4LjIzODI4MSAtNC45OTIxODggQyA5LjQ1MzEyNSAtMy42NjQwNjIgMTEuMzI4MTI1IC0zLjAwMzkwNiAxMy44NjcxODggLTMuMDAzOTA2IEMgMTUuOTAyMzQ0IC0zLjAwMzkwNiAxNy41ODk4NDQgLTMuNTUwNzgxIDE4LjkzMzU5NCAtNC42NTIzNDQgWiBNIDQuNzYxNzE5IC0yMC40NTcwMzEgQyAzLjQ3NjU2MiAtMjEuNzYxNzE5IDIuODMyMDMxIC0yMy40NTMxMjUgMi44MzIwMzEgLTI1LjUzNTE1NiBDIDIuODMyMDMxIC0yOC4xNDA2MjUgMy43NzczNDQgLTMwLjM3ODkwNiA1LjY2NDA2MiAtMzIuMjUgQyA3LjU1MDc4MSAtMzQuMTIxMDk0IDEwLjIzMDQ2OSAtMzUuMDU4NTk0IDEzLjY5NTMxMiAtMzUuMDU4NTk0IEMgMTcuMDQ2ODc1IC0zNS4wNTg1OTQgMTkuNjc1NzgxIC0zNC4xNzU3ODEgMjEuNTgyMDMxIC0zMi40MTAxNTYgQyAyMy40ODQzNzUgLTMwLjY0NDUzMSAyNC40Mzc1IC0yOC41ODIwMzEgMjQuNDM3NSAtMjYuMjE4NzUgQyAyNC40Mzc1IC0yNC4wMzkwNjIgMjMuODg2NzE5IC0yMi4yNzM0MzggMjIuNzc3MzQ0IC0yMC45MjE4NzUgQyAyMi4xNjAxNTYgLTIwLjE1NjI1IDIxLjE5OTIxOSAtMTkuNDEwMTU2IDE5Ljg5ODQzOCAtMTguNjc1NzgxIEMgMjEuMzQ3NjU2IC0xOC4wMDc4MTIgMjIuNDg0Mzc1IC0xNy4yNDYwOTQgMjMuMzE2NDA2IC0xNi4zODI4MTIgQyAyNC44NjMyODEgLTE0Ljc1MzkwNiAyNS42MzI4MTIgLTEyLjYzNjcxOSAyNS42MzI4MTIgLTEwLjAzNTE1NiBDIDI1LjYzMjgxMiAtNi45NTcwMzEgMjQuNjAxNTYyIC00LjM1MTU2MiAyMi41MzUxNTYgLTIuMjEwOTM4IEMgMjAuNDY4NzUgLTAuMDcwMzEyNSAxNy41NDY4NzUgMSAxMy43Njk1MzEgMSBDIDEwLjM2NzE4OCAxIDcuNDkyMTg4IDAuMDc4MTI1IDUuMTQwNjI1IC0xLjc2OTUzMSBDIDIuNzg5MDYyIC0zLjYxNzE4OCAxLjYwOTM3NSAtNi4yOTY4NzUgMS42MDkzNzUgLTkuODEyNSBDIDEuNjA5Mzc1IC0xMS44Nzg5MDYgMi4xMTcxODggLTEzLjY2Nzk2OSAzLjEyNSAtMTUuMTcxODc1IEMgNC4xMzI4MTIgLTE2LjY3OTY4OCA1LjYzMjgxMiAtMTcuODMyMDMxIDcuNjE3MTg4IC0xOC42Mjg5MDYgQyA2LjM5NDUzMSAtMTkuMTQ4NDM4IDUuNDQ1MzEyIC0xOS43NTc4MTIgNC43NjE3MTkgLTIwLjQ1NzAzMSBaICIvPgo8L3N5bWJvbD4KPHN5bWJvbCBvdmVyZmxvdz0idmlzaWJsZSIgaWQ9ImdseXBoMC01Ij4KPHBhdGggc3R5bGU9InN0cm9rZTpub25lOyIgZD0iTSAyMi43ODkwNjIgLTMyLjA2NjQwNiBDIDI0LjMyODEyNSAtMzAuMDM5MDYyIDI1LjA5NzY1NiAtMjcuOTUzMTI1IDI1LjA5NzY1NiAtMjUuODA0Njg4IEwgMjAuNzUgLTI1LjgwNDY4OCBDIDIwLjQ5MjE4OCAtMjcuMTg3NSAyMC4wNzgxMjUgLTI4LjI2OTUzMSAxOS41MDc4MTIgLTI5LjA1MDc4MSBDIDE4LjQ0OTIxOSAtMzAuNTE1NjI1IDE2Ljg0Mzc1IC0zMS4yNSAxNC42OTUzMTIgLTMxLjI1IEMgMTIuMjM4MjgxIC0zMS4yNSAxMC4yODUxNTYgLTMwLjExMzI4MSA4LjgzNTkzOCAtMjcuODQzNzUgQyA3LjM4NjcxOSAtMjUuNTc0MjE5IDYuNTgyMDMxIC0yMi4zMjQyMTkgNi40MjE4NzUgLTE4LjA4OTg0NCBDIDcuNDI5Njg4IC0xOS41NzAzMTIgOC42OTkyMTkgLTIwLjY3OTY4OCAxMC4yMzA0NjkgLTIxLjQxMDE1NiBDIDExLjYyODkwNiAtMjIuMDYyNSAxMy4xOTE0MDYgLTIyLjM4NjcxOSAxNC45MTc5NjkgLTIyLjM4NjcxOSBDIDE3Ljg0NzY1NiAtMjIuMzg2NzE5IDIwLjQwMjM0NCAtMjEuNDUzMTI1IDIyLjU4MjAzMSAtMTkuNTc4MTI1IEMgMjQuNzYxNzE5IC0xNy43MDcwMzEgMjUuODU1NDY5IC0xNC45MTc5NjkgMjUuODU1NDY5IC0xMS4yMDcwMzEgQyAyNS44NTU0NjkgLTguMDMxMjUgMjQuODIwMzEyIC01LjIxODc1IDIyLjc1MzkwNiAtMi43Njk1MzEgQyAyMC42ODc1IC0wLjMyMDMxMiAxNy43NDIxODggMC45MDIzNDQgMTMuOTE0MDYyIDAuOTAyMzQ0IEMgMTAuNjQ0NTMxIDAuOTAyMzQ0IDcuODIwMzEyIC0wLjMzNTkzOCA1LjQ0NTMxMiAtMi44MjAzMTIgQyAzLjA3MDMxMiAtNS4zMDA3ODEgMS44Nzg5MDYgLTkuNDgwNDY5IDEuODc4OTA2IC0xNS4zNTU0NjkgQyAxLjg3ODkwNiAtMTkuNzAzMTI1IDIuNDEwMTU2IC0yMy4zODY3MTkgMy40NjQ4NDQgLTI2LjQxNDA2MiBDIDUuNSAtMzIuMjEwOTM4IDkuMjE4NzUgLTM1LjEwNTQ2OSAxNC42MjUgLTM1LjEwNTQ2OSBDIDE4LjUzMTI1IC0zNS4xMDU0NjkgMjEuMjUzOTA2IC0zNC4wOTM3NSAyMi43ODkwNjIgLTMyLjA2NjQwNiBaIE0gMTkuNDk2MDk0IC01LjMzNTkzOCBDIDIwLjY0MDYyNSAtNi44OTA2MjUgMjEuMjE0ODQ0IC04LjcyMjY1NiAyMS4yMTQ4NDQgLTEwLjgzOTg0NCBDIDIxLjIxNDg0NCAtMTIuNjI4OTA2IDIwLjcwMzEyNSAtMTQuMzM1OTM4IDE5LjY3NTc4MSAtMTUuOTUzMTI1IEMgMTguNjUyMzQ0IC0xNy41NzQyMTkgMTYuNzg5MDYyIC0xOC4zODI4MTIgMTQuMDg1OTM4IC0xOC4zODI4MTIgQyAxMi4xOTkyMTkgLTE4LjM4MjgxMiAxMC41NDI5NjkgLTE3Ljc1NzgxMiA5LjExNzE4OCAtMTYuNTAzOTA2IEMgNy42OTUzMTIgLTE1LjI1IDYuOTgwNDY5IC0xMy4zNjMyODEgNi45ODA0NjkgLTEwLjgzOTg0NCBDIDYuOTgwNDY5IC04LjYyNSA3LjYyODkwNiAtNi43NjU2MjUgOC45MjE4NzUgLTUuMjYxNzE5IEMgMTAuMjE0ODQ0IC0zLjc1MzkwNiAxMi4wMTE3MTkgLTMuMDAzOTA2IDE0LjMwNDY4OCAtMy4wMDM5MDYgQyAxNi42MTcxODggLTMuMDAzOTA2IDE4LjM0NzY1NiAtMy43ODEyNSAxOS40OTYwOTQgLTUuMzM1OTM4IFogIi8+Cjwvc3ltYm9sPgo8c3ltYm9sIG92ZXJmbG93PSJ2aXNpYmxlIiBpZD0iZ2x5cGgwLTYiPgo8cGF0aCBzdHlsZT0ic3Ryb2tlOm5vbmU7IiBkPSJNIDE2LjUyNzM0NCAtMTIuMzc4OTA2IEwgMTYuNTI3MzQ0IC0yOC4yMjI2NTYgTCA1LjMyMDMxMiAtMTIuMzc4OTA2IFogTSAxNi42MDE1NjIgMCBMIDE2LjYwMTU2MiAtOC41NDI5NjkgTCAxLjI2OTUzMSAtOC41NDI5NjkgTCAxLjI2OTUzMSAtMTIuODM5ODQ0IEwgMTcuMjg1MTU2IC0zNS4wNTg1OTQgTCAyMC45OTYwOTQgLTM1LjA1ODU5NCBMIDIwLjk5NjA5NCAtMTIuMzc4OTA2IEwgMjYuMTQ4NDM4IC0xMi4zNzg5MDYgTCAyNi4xNDg0MzggLTguNTQyOTY5IEwgMjAuOTk2MDk0IC04LjU0Mjk2OSBMIDIwLjk5NjA5NCAwIFogIi8+Cjwvc3ltYm9sPgo8c3ltYm9sIG92ZXJmbG93PSJ2aXNpYmxlIiBpZD0iZ2x5cGgwLTciPgo8cGF0aCBzdHlsZT0ic3Ryb2tlOm5vbmU7IiBkPSJNIDMuOTkyMTg4IC0yLjQ1MzEyNSBDIDIuMTI4OTA2IC00LjcyMjY1NiAxLjE5NTMxMiAtNy40ODgyODEgMS4xOTUzMTIgLTEwLjc0MjE4OCBMIDUuNzg1MTU2IC0xMC43NDIxODggQyA1Ljk4MDQ2OSAtOC40ODA0NjkgNi40MDYyNSAtNi44MzU5MzggNy4wNTQ2ODggLTUuODA4NTk0IEMgOC4xOTUzMTIgLTMuOTY4NzUgMTAuMjUzOTA2IC0zLjA1MDc4MSAxMy4yMzA0NjkgLTMuMDUwNzgxIEMgMTUuNTQyOTY5IC0zLjA1MDc4MSAxNy4zOTg0MzggLTMuNjcxODc1IDE4Ljc5Njg3NSAtNC45MDYyNSBDIDIwLjE5OTIxOSAtNi4xNDQ1MzEgMjAuODk4NDM4IC03LjczODI4MSAyMC44OTg0MzggLTkuNjkxNDA2IEMgMjAuODk4NDM4IC0xMi4xMDE1NjIgMjAuMTYwMTU2IC0xMy43ODUxNTYgMTguNjg3NSAtMTQuNzQ2MDk0IEMgMTcuMjE0ODQ0IC0xNS43MDcwMzEgMTUuMTY3OTY5IC0xNi4xODc1IDEyLjU0Njg3NSAtMTYuMTg3NSBDIDEyLjI1MzkwNiAtMTYuMTg3NSAxMS45NTcwMzEgLTE2LjE4MzU5NCAxMS42NTYyNSAtMTYuMTc1NzgxIEMgMTEuMzU1NDY5IC0xNi4xNjc5NjkgMTEuMDUwNzgxIC0xNi4xNTIzNDQgMTAuNzQyMTg4IC0xNi4xMzY3MTkgTCAxMC43NDIxODggLTIwLjAxOTUzMSBDIDExLjE5OTIxOSAtMTkuOTY4NzUgMTEuNTgyMDMxIC0xOS45Mzc1IDExLjg5MDYyNSAtMTkuOTIxODc1IEMgMTIuMTk5MjE5IC0xOS45MDYyNSAxMi41MzEyNSAtMTkuODk4NDM4IDEyLjg5MDYyNSAtMTkuODk4NDM4IEMgMTQuNTM1MTU2IC0xOS44OTg0MzggMTUuODg2NzE5IC0yMC4xNTYyNSAxNi45NDE0MDYgLTIwLjY3OTY4OCBDIDE4Ljc5Njg3NSAtMjEuNTg5ODQ0IDE5LjcyNjU2MiAtMjMuMjE4NzUgMTkuNzI2NTYyIC0yNS41NjI1IEMgMTkuNzI2NTYyIC0yNy4zMDQ2ODggMTkuMTA5Mzc1IC0yOC42NDQ1MzEgMTcuODcxMDk0IC0yOS41ODk4NDQgQyAxNi42MzI4MTIgLTMwLjUzNTE1NiAxNS4xOTUzMTIgLTMxLjAwMzkwNiAxMy41NTA3ODEgLTMxLjAwMzkwNiBDIDEwLjYyMTA5NCAtMzEuMDAzOTA2IDguNTkzNzUgLTMwLjAyNzM0NCA3LjQ2ODc1IC0yOC4wNzQyMTkgQyA2Ljg1MTU2MiAtMjcgNi41MDM5MDYgLTI1LjQ3MjY1NiA2LjQyMTg3NSAtMjMuNDg0Mzc1IEwgMi4wNzQyMTkgLTIzLjQ4NDM3NSBDIDIuMDc0MjE5IC0yNi4wODk4NDQgMi41OTc2NTYgLTI4LjMwNDY4OCAzLjYzNjcxOSAtMzAuMTI1IEMgNS40MjU3ODEgLTMzLjM4MjgxMiA4LjU3ODEyNSAtMzUuMDA3ODEyIDEzLjA4NTkzOCAtMzUuMDA3ODEyIEMgMTYuNjQ4NDM4IC0zNS4wMDc4MTIgMTkuNDEwMTU2IC0zNC4yMTQ4NDQgMjEuMzYzMjgxIC0zMi42Mjg5MDYgQyAyMy4zMTY0MDYgLTMxLjA0Mjk2OSAyNC4yOTI5NjkgLTI4Ljc0MjE4OCAyNC4yOTI5NjkgLTI1LjczMDQ2OSBDIDI0LjI5Mjk2OSAtMjMuNTgyMDMxIDIzLjcxNDg0NCAtMjEuODQzNzUgMjIuNTU4NTk0IC0yMC41MDc4MTIgQyAyMS44NDM3NSAtMTkuNjc1NzgxIDIwLjkxNDA2MiAtMTkuMDI3MzQ0IDE5Ljc3MzQzOCAtMTguNTU0Njg4IEMgMjEuNjEzMjgxIC0xOC4wNTA3ODEgMjMuMDUwNzgxIC0xNy4wNzgxMjUgMjQuMDg1OTM4IC0xNS42MzY3MTkgQyAyNS4xMTcxODggLTE0LjE5NTMxMiAyNS42MzI4MTIgLTEyLjQzMzU5NCAyNS42MzI4MTIgLTEwLjM1MTU2MiBDIDI1LjYzMjgxMiAtNy4wMTU2MjUgMjQuNTM1MTU2IC00LjI5Njg3NSAyMi4zMzk4NDQgLTIuMTk1MzEyIEMgMjAuMTQwNjI1IC0wLjA5NzY1NjIgMTcuMDIzNDM4IDAuOTUzMTI1IDEyLjk4ODI4MSAwLjk1MzEyNSBDIDguODU1NDY5IDAuOTUzMTI1IDUuODU1NDY5IC0wLjE4MzU5NCAzLjk5MjE4OCAtMi40NTMxMjUgWiAiLz4KPC9zeW1ib2w+CjwvZz4KPC9kZWZzPgo8ZyBpZD0ic3VyZmFjZTI2Ij4KPHBhdGggc3R5bGU9ImZpbGwtcnVsZTpub256ZXJvO2ZpbGw6cmdiKDU1JSw1NSUsMTguMzMzMzMzJSk7ZmlsbC1vcGFjaXR5OjE7c3Ryb2tlLXdpZHRoOjAuMDE7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlOnJnYigwJSwwJSwwJSk7c3Ryb2tlLW9wYWNpdHk6MTtzdHJva2UtbWl0ZXJsaW1pdDoxMDsiIGQ9Ik0gMi40MzQ3NjYgMi4zODQ3NjYgTCAyLjQzNDc2NiAxLjYxNTE5NSBDIDIuNDM0NzY2IDEuNTg3NTc4IDIuNDEyMzgzIDEuNTY1MTk1IDIuMzg0NzY2IDEuNTY1MTk1IEwgMS42MTUxOTUgMS41NjUxOTUgQyAxLjU4NzU3OCAxLjU2NTE5NSAxLjU2NTE5NSAxLjU4NzU3OCAxLjU2NTE5NSAxLjYxNTE5NSBMIDEuNTY1MTk1IDIuMzg0NzY2IEMgMS41NjUxOTUgMi40MTIzODMgMS41ODc1NzggMi40MzQ3NjYgMS42MTUxOTUgMi40MzQ3NjYgTCAyLjM4NDc2NiAyLjQzNDc2NiBDIDIuNDEyMzgzIDIuNDM0NzY2IDIuNDM0NzY2IDIuNDEyMzgzIDIuNDM0NzY2IDIuMzg0NzY2IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPHBhdGggc3R5bGU9ImZpbGwtcnVsZTpub256ZXJvO2ZpbGw6cmdiKDY1JSw2NSUsMjEuNjY2NjY3JSk7ZmlsbC1vcGFjaXR5OjE7c3Ryb2tlLXdpZHRoOjAuMDE7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlOnJnYigwJSwwJSwwJSk7c3Ryb2tlLW9wYWNpdHk6MTtzdHJva2UtbWl0ZXJsaW1pdDoxMDsiIGQ9Ik0gMC40MzQ3NjYgMi4zODQ3NjYgTCAwLjQzNDc2NiAxLjYxNTE5NSBDIDAuNDM0NzY2IDEuNTg3NTc4IDAuNDEyMzgzIDEuNTY1MTk1IDAuMzg0NzY2IDEuNTY1MTk1IEwgLTAuMzg0ODA1IDEuNTY1MTk1IEMgLTAuNDEyNDIyIDEuNTY1MTk1IC0wLjQzNDgwNSAxLjU4NzU3OCAtMC40MzQ4MDUgMS42MTUxOTUgTCAtMC40MzQ4MDUgMi4zODQ3NjYgQyAtMC40MzQ4MDUgMi40MTIzODMgLTAuNDEyNDIyIDIuNDM0NzY2IC0wLjM4NDgwNSAyLjQzNDc2NiBMIDAuMzg0NzY2IDIuNDM0NzY2IEMgMC40MTIzODMgMi40MzQ3NjYgMC40MzQ3NjYgMi40MTIzODMgMC40MzQ3NjYgMi4zODQ3NjYgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbC1ydWxlOm5vbnplcm87ZmlsbDpyZ2IoNzAlLDcwJSwyMy4zMzMzMzMlKTtmaWxsLW9wYWNpdHk6MTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAyLjQzNDc2NiAxLjM4NDc2NiBMIDIuNDM0NzY2IDAuNjE1MTk1IEMgMi40MzQ3NjYgMC41ODc1NzggMi40MTIzODMgMC41NjUxOTUgMi4zODQ3NjYgMC41NjUxOTUgTCAxLjYxNTE5NSAwLjU2NTE5NSBDIDEuNTg3NTc4IDAuNTY1MTk1IDEuNTY1MTk1IDAuNTg3NTc4IDEuNTY1MTk1IDAuNjE1MTk1IEwgMS41NjUxOTUgMS4zODQ3NjYgQyAxLjU2NTE5NSAxLjQxMjM4MyAxLjU4NzU3OCAxLjQzNDc2NiAxLjYxNTE5NSAxLjQzNDc2NiBMIDIuMzg0NzY2IDEuNDM0NzY2IEMgMi40MTIzODMgMS40MzQ3NjYgMi40MzQ3NjYgMS40MTIzODMgMi40MzQ3NjYgMS4zODQ3NjYgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbC1ydWxlOm5vbnplcm87ZmlsbDpyZ2IoNzUlLDc1JSwyNSUpO2ZpbGwtb3BhY2l0eToxO3N0cm9rZS13aWR0aDowLjAxO3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZTpyZ2IoMCUsMCUsMCUpO3N0cm9rZS1vcGFjaXR5OjE7c3Ryb2tlLW1pdGVybGltaXQ6MTA7IiBkPSJNIDEuNDM0NzY2IDEuMzg0NzY2IEwgMS40MzQ3NjYgMC42MTUxOTUgQyAxLjQzNDc2NiAwLjU4NzU3OCAxLjQxMjM4MyAwLjU2NTE5NSAxLjM4NDc2NiAwLjU2NTE5NSBMIDAuNjE1MTk1IDAuNTY1MTk1IEMgMC41ODc1NzggMC41NjUxOTUgMC41NjUxOTUgMC41ODc1NzggMC41NjUxOTUgMC42MTUxOTUgTCAwLjU2NTE5NSAxLjM4NDc2NiBDIDAuNTY1MTk1IDEuNDEyMzgzIDAuNTg3NTc4IDEuNDM0NzY2IDAuNjE1MTk1IDEuNDM0NzY2IEwgMS4zODQ3NjYgMS40MzQ3NjYgQyAxLjQxMjM4MyAxLjQzNDc2NiAxLjQzNDc2NiAxLjQxMjM4MyAxLjQzNDc2NiAxLjM4NDc2NiBaICIgdHJhbnNmb3JtPSJtYXRyaXgoMTAwLDAsMCwxMDAsNTAsNTApIi8+CjxwYXRoIHN0eWxlPSJmaWxsLXJ1bGU6bm9uemVybztmaWxsOnJnYig5MCUsOTAlLDMwJSk7ZmlsbC1vcGFjaXR5OjE7c3Ryb2tlLXdpZHRoOjAuMDE7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlOnJnYigwJSwwJSwwJSk7c3Ryb2tlLW9wYWNpdHk6MTtzdHJva2UtbWl0ZXJsaW1pdDoxMDsiIGQ9Ik0gMS40MzQ3NjYgMC4zODQ3NjYgTCAxLjQzNDc2NiAtMC4zODQ4MDUgQyAxLjQzNDc2NiAtMC40MTI0MjIgMS40MTIzODMgLTAuNDM0ODA1IDEuMzg0NzY2IC0wLjQzNDgwNSBMIDAuNjE1MTk1IC0wLjQzNDgwNSBDIDAuNTg3NTc4IC0wLjQzNDgwNSAwLjU2NTE5NSAtMC40MTI0MjIgMC41NjUxOTUgLTAuMzg0ODA1IEwgMC41NjUxOTUgMC4zODQ3NjYgQyAwLjU2NTE5NSAwLjQxMjM4MyAwLjU4NzU3OCAwLjQzNDc2NiAwLjYxNTE5NSAwLjQzNDc2NiBMIDEuMzg0NzY2IDAuNDM0NzY2IEMgMS40MTIzODMgMC40MzQ3NjYgMS40MzQ3NjYgMC40MTIzODMgMS40MzQ3NjYgMC4zODQ3NjYgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbC1ydWxlOm5vbnplcm87ZmlsbDpyZ2IoOTUlLDk1JSwzMS42NjY2NjclKTtmaWxsLW9wYWNpdHk6MTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAwLjQzNDc2NiAwLjM4NDc2NiBMIDAuNDM0NzY2IC0wLjM4NDgwNSBDIDAuNDM0NzY2IC0wLjQxMjQyMiAwLjQxMjM4MyAtMC40MzQ4MDUgMC4zODQ3NjYgLTAuNDM0ODA1IEwgLTAuMzg0ODA1IC0wLjQzNDgwNSBDIC0wLjQxMjQyMiAtMC40MzQ4MDUgLTAuNDM0ODA1IC0wLjQxMjQyMiAtMC40MzQ4MDUgLTAuMzg0ODA1IEwgLTAuNDM0ODA1IDAuMzg0NzY2IEMgLTAuNDM0ODA1IDAuNDEyMzgzIC0wLjQxMjQyMiAwLjQzNDc2NiAtMC4zODQ4MDUgMC40MzQ3NjYgTCAwLjM4NDc2NiAwLjQzNDc2NiBDIDAuNDEyMzgzIDAuNDM0NzY2IDAuNDM0NzY2IDAuNDEyMzgzIDAuNDM0NzY2IDAuMzg0NzY2IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPHBhdGggc3R5bGU9ImZpbGw6bm9uZTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAyLjUgMi41IEwgMi41IDEuNSBMIDEuNSAxLjUgTCAxLjUgMi41IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPHBhdGggc3R5bGU9ImZpbGw6bm9uZTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAxLjUgMi41IEwgMS41IDEuNSBMIDAuNSAxLjUgTCAwLjUgMi41IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPHBhdGggc3R5bGU9ImZpbGw6bm9uZTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAwLjUgMi41IEwgMC41IDEuNSBMIC0wLjUgMS41IEwgLTAuNSAyLjUgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbDpub25lO3N0cm9rZS13aWR0aDowLjAxO3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZTpyZ2IoMCUsMCUsMCUpO3N0cm9rZS1vcGFjaXR5OjE7c3Ryb2tlLW1pdGVybGltaXQ6MTA7IiBkPSJNIDIuNSAxLjUgTCAyLjUgMC41IEwgMS41IDAuNSBMIDEuNSAxLjUgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbDpub25lO3N0cm9rZS13aWR0aDowLjAxO3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZTpyZ2IoMCUsMCUsMCUpO3N0cm9rZS1vcGFjaXR5OjE7c3Ryb2tlLW1pdGVybGltaXQ6MTA7IiBkPSJNIDEuNSAxLjUgTCAxLjUgMC41IEwgMC41IDAuNSBMIDAuNSAxLjUgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbDpub25lO3N0cm9rZS13aWR0aDowLjAxO3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZTpyZ2IoMCUsMCUsMCUpO3N0cm9rZS1vcGFjaXR5OjE7c3Ryb2tlLW1pdGVybGltaXQ6MTA7IiBkPSJNIDAuNSAxLjUgTCAwLjUgMC41IEwgLTAuNSAwLjUgTCAtMC41IDEuNSBaICIgdHJhbnNmb3JtPSJtYXRyaXgoMTAwLDAsMCwxMDAsNTAsNTApIi8+CjxwYXRoIHN0eWxlPSJmaWxsOm5vbmU7c3Ryb2tlLXdpZHRoOjAuMDE7c3Ryb2tlLWxpbmVjYXA6YnV0dDtzdHJva2UtbGluZWpvaW46bWl0ZXI7c3Ryb2tlOnJnYigwJSwwJSwwJSk7c3Ryb2tlLW9wYWNpdHk6MTtzdHJva2UtbWl0ZXJsaW1pdDoxMDsiIGQ9Ik0gMi41IDAuNSBMIDIuNSAtMC41IEwgMS41IC0wLjUgTCAxLjUgMC41IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPHBhdGggc3R5bGU9ImZpbGw6bm9uZTtzdHJva2Utd2lkdGg6MC4wMTtzdHJva2UtbGluZWNhcDpidXR0O3N0cm9rZS1saW5lam9pbjptaXRlcjtzdHJva2U6cmdiKDAlLDAlLDAlKTtzdHJva2Utb3BhY2l0eToxO3N0cm9rZS1taXRlcmxpbWl0OjEwOyIgZD0iTSAxLjUgMC41IEwgMS41IC0wLjUgTCAwLjUgLTAuNSBMIDAuNSAwLjUgWiAiIHRyYW5zZm9ybT0ibWF0cml4KDEwMCwwLDAsMTAwLDUwLDUwKSIvPgo8cGF0aCBzdHlsZT0iZmlsbDpub25lO3N0cm9rZS13aWR0aDowLjAxO3N0cm9rZS1saW5lY2FwOmJ1dHQ7c3Ryb2tlLWxpbmVqb2luOm1pdGVyO3N0cm9rZTpyZ2IoMCUsMCUsMCUpO3N0cm9rZS1vcGFjaXR5OjE7c3Ryb2tlLW1pdGVybGltaXQ6MTA7IiBkPSJNIDAuNSAwLjUgTCAwLjUgLTAuNSBMIC0wLjUgLTAuNSBMIC0wLjUgMC41IFogIiB0cmFuc2Zvcm09Im1hdHJpeCgxMDAsMCwwLDEwMCw1MCw1MCkiLz4KPGcgc3R5bGU9ImZpbGw6cmdiKDAlLDAlLDAlKTtmaWxsLW9wYWNpdHk6MTsiPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC0xIiB4PSIyMDcuNDgyOTEiIHk9IjI2My41MDA5NzciLz4KICA8dXNlIHhsaW5rOmhyZWY9IiNnbHlwaDAtMiIgeD0iMjM1LjI5MDUyNyIgeT0iMjYzLjUwMDk3NyIvPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC0zIiB4PSIyNjMuMDk4MTQ1IiB5PSIyNjMuNTAwOTc3Ii8+CjwvZz4KPGcgc3R5bGU9ImZpbGw6cmdiKDAlLDAlLDAlKTtmaWxsLW9wYWNpdHk6MTsiPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC0yIiB4PSI1Ljg5NTk5NiIgeT0iMjYzLjUwMDk3NyIvPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC0zIiB4PSIzMy43MDM2MTMiIHk9IjI2My41MDA5NzciLz4KICA8dXNlIHhsaW5rOmhyZWY9IiNnbHlwaDAtNCIgeD0iNjEuNTExMjMiIHk9IjI2My41MDA5NzciLz4KPC9nPgo8ZyBzdHlsZT0iZmlsbDpyZ2IoMCUsMCUsMCUpO2ZpbGwtb3BhY2l0eToxOyI+CiAgPHVzZSB4bGluazpocmVmPSIjZ2x5cGgwLTUiIHg9IjIyMS4yNTI0NDEiIHk9IjE2My41MDA5NzciLz4KICA8dXNlIHhsaW5rOmhyZWY9IiNnbHlwaDAtNiIgeD0iMjQ5LjA2MDA1OSIgeT0iMTYzLjUwMDk3NyIvPgo8L2c+CjxnIHN0eWxlPSJmaWxsOnJnYigwJSwwJSwwJSk7ZmlsbC1vcGFjaXR5OjE7Ij4KICA8dXNlIHhsaW5rOmhyZWY9IiNnbHlwaDAtNyIgeD0iMTIxLjU5NDIzOCIgeT0iMTYzLjUwMDk3NyIvPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC0zIiB4PSIxNDkuNDAxODU1IiB5PSIxNjMuNTAwOTc3Ii8+CjwvZz4KPGcgc3R5bGU9ImZpbGw6cmdiKDAlLDAlLDAlKTtmaWxsLW9wYWNpdHk6MTsiPgogIDx1c2UgeGxpbms6aHJlZj0iI2dseXBoMC02IiB4PSIxMzUuNDYxNDI2IiB5PSI2My41MDA5NzciLz4KPC9nPgo8ZyBzdHlsZT0iZmlsbDpyZ2IoMCUsMCUsMCUpO2ZpbGwtb3BhY2l0eToxOyI+CiAgPHVzZSB4bGluazpocmVmPSIjZ2x5cGgwLTMiIHg9IjM1LjMxNDk0MSIgeT0iNjMuNTAwOTc3Ii8+CjwvZz4KPC9nPgo8L3N2Zz4K\"/)
"
- ],
- "metadata": {},
- "output_type": "display_data",
- "png": "iVBORw0KGgoAAAANSUhEUgAAASwAAAEsCAYAAAB5fY51AAAABmJLR0QA/wD/AP+gvaeTAAAgAElEQVR4nO3dd3gU1d4H8O9s300P6Y0QQiD0IiBKiXJRutgLKOALoqjXrvjqVdTL1cd2VVCviq+oqFwFlS4gEIqCSDeEEhJSSM+mbrZmd94/AgmTmd1sS3Yn+X2eZ5+wszO7h5zMd2fOnHOGAZB56UF8b96ln6t8WAbSat6ln6t8WAbSKgMAlvq2DOQKmaAvD3+SCaoPf7JU4usSEEKIsyiwCCGiQYFFCBENCixCiGg4E1iJQUHylRqNVCuTMWYALD2cf8jlElNgoKw8KEj+KYA4J37f7aH68LA+AgKk5YGBXqsP0olk7byeERgoW79oUYr63nt7ymNjVVCrpZ1SsK5Cr7cqior0UWvWFM37+OPzdzY2WqcD2Ofm22UEBEjXz5+frL7rrgR5TIwKKhXVhyv0equiuNgQtW5d8bzPP8/ztD5IJ2PQ3K1hqcBriWq19PSPP14TMHZsRKcWqqvas6cSt99+QGcwWNMBXBRYJfPSzwyB1xLVaunp1atHBYwZ06Ojitit7N9fhblz/3S3Pkjns9+tIShI/tJDD6UqKKy8Z8KESDzwQIoiMFD+gqvbBgXJX1qwoJeCwsp7xo6NwNy5yYrAQKnL9UF8w0EbFjtjzpwkeecVpXu4554khUTCznR9S3bGHXckUH142R13JCgYhnGjPogv2A0sg8Eanpio7syydAu9egVAr7dGurqdwWANT0jQdESRurXkZA0MBtfrg/iG3cCy2SCRSpnOLEu3IJUyYFnXu5PYbJBIJFQf3uZufRDfoIoihIgGBRYhRDQosAghokGBRQgRDQosQohoUGARQkSDAosQIhoUWIQQ0aDAIoSIBgUWIUQ0KLAIIaJBgUUIEQ0KLEKIaFBgEUJEgwKLECIa7d2Ewu9ZLDYYjTYEBsrA0HRRfkevt6KpiQUAMAwQFCT6PzniQ6L66zGZbNi5sxzr1hUjK6sOZWVG1NSYwbKATMYgNFSB9PQgjB8fiSlTYjBkSKivi9yt5eToMHnyPhgM1pZlJSXTfVgiInaiCaxPPsnDq69mo77eIvh6UxOLqioT9u0zYd++KixbdhqTJ8fg1VcHID09uJNLS8xmGxYvPsoJK0I85fdtWNXVZtxyy+94+ukTdsPKnl9+KcP11+/Brl0VHVQ6Ys+yZadx6lS9r4tBuhi/DiyrlcV99x3Cjh3ldtdhmObTQXt0uibcdtsBZGXVdUQRiYBduyqwcuUFXxeDdEF+fUq4bNlp7NlTyVt+zTU9cN99yZgwIRLR0UoAQF5eI06frsdHH+XiwAEtZ32LxYYnnjiB7dvHU8N8B6usNOHxx0+AZX1dEtIV+e0RVlmZEe++e46zTCpl8K9/DcK2beMxe3YSEhLUkMslkMsl6Ns3CLNmxWPbtvFYvnwY7xbuBw9q8e23hZ35X+h2WBZ4/PHjqKoy+boopIvy28D673+LYLVyv6affrovHn001eF2DAPMm5eMF15I57324otZMBqpEbijfPZZHnbvbj0idnSqTog7/DawvvmGezTUs6cGS5b0c3r7Rx9NxdCh3G4NVVUmZGdTQ3BHyMqqw7JlZ1qeh4crMHduMmcdOh0nnvLLwMrPb26PutLNN8e79I0tlTKYNSuetzwriwLL2wwGKxYvPgaLxday7J13Bre0LxLiLX4ZWAUFet6y6dPjXH6f9PQg3jK6Wuh9//jHKZw/r2t5Pnt2Em68McaHJSJdlV8GVnGxgbcsJSXA5fdJTuZvc/Ei/72J+zZtKuVczEhJCcArrwzwYYlIV+aXgVVSwg0VuVyCiAjXTy8uXGjkLevbl3/URdxTUmLAM8+cbHkulzNYsWIYNBqpg60IcZ9f9sO6+uoeeP31QS3PAwLcG9h89GgNb9ngwSGeFI1cYrWyeOSRY6irax198OSTabwLHYR4k18G1tixERg7NsKj92hoaMJ33xXxltOAaO/44IPzOHiwuuX56NHh7XY5IcRTfnlK6A3PPHMCRUXcxvsBA4LRq5frbWGE68iRGvz7362deoODZVi+fBgkEuq3QDpWlwysd945x+vHJZdL8MknI6gvkIfq65uwePGxljmuAGDZskFISFD7sFSku/DLU0J3NTQ04bHHjuGHHy7yXnvmmb50OugFS5b8xTlynTUrHrfeyu/vRkhH6DKBtWlTKZ566gTvCiPQ3Cb29NNpPihV1/L99xfx88/FLc/j49V4442BPiwR6W5EH1gFBXosWXISmzaVCr5+//298PbbgyGXd8mz306Tn9+IF17IankukTD44IOhCA6W+7BUpLsRbWDp9Va89dZZLF+eA5PJxns9JESO118fhHvv7emD0nUtFguLxYuPobGxqWXZQw+lYMyYHj4sFemORBdYLAusWVOIl18+hdJSo+A6d92ViGXLBiEqisayecObb57B8eO1Lc8HDw7Bs886PxCdEG8RVWCVlRmxcOFhZGbyJ/UDgH79gvDuu0MxbpxnfbhIq/37q/DRR7ktz9VqKVasGAa5nC63ks4nmsDavr0cixYdEZwcLipKiRdeSMfcucmQSmlH8qaPPsrlzB569dXhOHKkBkeO8EcRtHXyJHegOcs2z3PW1tSpsXT7L+IUv/8rsVhsWLo0G8uX5/Cm3VUqJfj73/vgySfTEBjo9/8VUbK1aR7cvbuSM0mfq5544gRv2ahR4RRYxCl+/1fy7LMnBW9ocMMN0Xj77SHUc52QbsSvA2vduou8sFIqJXj99UFYuDDFR6UihPiK3wZWXl4jHnnkGGeZQiHBjz9eg/HjI31Uqu5n9uwkTJjg3kWMo0drsXkzt3/cP/7Bn2s/PFzh1vuT7sdvA+v55/+CTtfa74dhgE8/HUFh1clmzIh1e9tvvinkBBbDAA891NsbxSLdlF92/66pMfNunvr442m49dYEH5WIEOIP/DKwfv65hHNDA6VSgkceobmWCOnu/PKUcO1a7mwLGRlRqK+3oL7eYmcL54WGyt2abpkQ4nt+GVhtpzbetq0M27aVeeW9H344FW+8Maj9FQkhfsfvTgkrK02cxnZCCLnM7wIrL49/pxtCCAH8MLByc3Xtr0QI6Zb8rg3rnnuScM89Sb4uBvGC2bOTMHs21SXxHr87wiKEEHsosAghokGBRQgRDQosQohoUGARQkSDAosQIhoUWIQQ0aDAIoSIBgUWIUQ0KLAIIaJBgUUIEQ0KLEKIaFBgEUJEgwKLECIaFFiEENGgwCKEiAYFFiFENCiwCCGiQYFFCBENCixCiGhQYBFCRIMCixAiGhRYhBDRsBtYEglYq5XtzLJ0C01NLBgGNle3YxiwNhvVh7e5Wx/EN+wGllot1RYU6DuzLN1CQUEjNBpppavbaTRSbWEh1Ye3FRbqoVa7Xh/ENxycEjIb16wpNHdeUbqHr78uNNtszAbXt2Q2rl17kerDy777rsjMsqwb9UF8wW5gNTRYlq5YkWveubOiM8vTpe3cWYGVK/NMOp3lNVe3bWiwLP3sswvmPXvoYMBb9uypxFdfFZh0OqvL9UF8Q+bgtWKj0Trj7rsPbnz00T7Ku+5KlEdGKjutYF1JVZUJq1blW1auzDMYDNbpAErceJtio9E64/77D29ctChFeeut8fKICKoPd2i1ZnzzTaHlq6/yDUaj2/VBfIABsPTSw574oCD5KwzDTjGb2bBOKVUXo1JJtTYbu6m+3rIMwEUHq2Ze+pnhYJ34oCD5KwA7xWy2UX24QaWSaK1WZpNO55X6IJ1nqaMjrMuKGxosCzq8KF2Y0Wj15ttRfXjIZKKLgmJF/bAIIaJBgUUIEQ0KLEKIaFBgEUJEgwGQf+lBfG8EmuvksK8LQgBQffibZDrC8i+MrwtAOKg+/NBSXxeAtMhEa98f4nuZoPrwJ0vpCIsQIhoUWIQQ0aDAIoSIBgUWIUQ0KLDEJ1Gjka5UqSRaqZQxA2Dp4fxDJmNMGo20XKORfgogzq0a4KL68LA+1Grn68OZwc/Ef2RoNNL1t9ySoJ4yJUYeEaGEUknfOa4wGq2K8nJT1I4d5fPWrr1456XpZfa5+XYZKpV0/fTpceq//S1K3qOHAgoF1YcrjEarorLSHJWZWTFvw4bSduvDmellSOfJvPQzQ+C1RKVScvrNNwcHDBkS2nkl6sKOHq3B88//pTOZbOkQnmYm89LPDIHXEpVKyemlS/sHDBwY0lFF7FZOnqzDq69mO6oP6tYgFhqN9KXbbktQUFh5z/DhYZg1K16h0UhfcHVbjUb60syZcQoKK+8ZPDgE06bFOqwPCiyRYBjMmDw5Ru7rcnQ1kyfHKBgGM93YdMbEiVFUH152/fVRDuuDAkskjEZbeHS0ytfF6HLi4tQwGm2Rrm5nMtnCacpw74uNVTmsDwoskWBZSCQSGtrmbRJJ8+/W1e2oPjqGRMI4rA8KLEKIaFBgEUJEgwKLECIaFFiEENGgwCKEiAYFFiFENCiwCCGiQYFFCBENCixCiGhQYBFCRIMCixAiGhRYhBDRoMAihIgGBRYhRDQosAghokE3oSB+oamJhclkhUYjA0PTTAlqaGiCXM5ApZJ2+mcbjVZYrc3/ZhhAo+n8MgAUWOSSnBwdfv21HGfONECrNUGrNYNlgZgYFWJiVIiNbf45cWIUIiI8m2nTYrHh0KFq7N5didxcHbRaMxoaLGBZQCplEBQkQ3JyAIYNC8U110SgT59AL/0vxYNlgT17KvHnn9UoKTGipMQAvb45MWQyBhERSgwYEIzhw0Nx7bURkEo7LuWLigx44onjMJlsLcs2bry2wz7PEQqsbu7cuQa88cYZ5OU1Cr6en9+I/PzW11auvICZM+Nwzz1J6NFD4fLn/fRTMVauvIDGxibB161WFrW1Fhw/Xovjx2vxxRf5GDOmBxYtSkFycoDLnydGf/xRja+/LkBBgV7w9aYmFmVlRpSVGbFzZwVWry7EAw+k4KqrwrxeFovFhrffPssJK1+iNqxubO3ai1i8+KjdsBJisdiwbt1F3H33QaxfX+L0dvX1Fjz77Em8/36O3bCy58ABLRYvPorDh2tc2k6Mli8/j3/+87TdsBJSWmrEa6+dRmZmpdfL8+WXBS79fXQ0CqxuavfuCqxYcR5NTaxb25vNNnzwQQ6ysuraXddmY7F0aTYOHaq2uw7DwOFpjV5vxZIlJ5Gbq3OrvGLw1VcF2L69XPC18HAF0tICERoqfKMem43F++/noKzM6LXyHDlSgw0bnP9S6gx0StgNlZQY8Pbb53jLw8IUGDcuAmlpQUhLC0RSkgbV1WYUFuqxaVMp9u+v4qxvtbJ45ZVsrFx5FUJC7N/x6osv8nH0KP/oaPDgEEydGovhw8MQHt58ellcbEB+fiPWrr2Iv/7ihmFTE4t//zsHy5cP63IN85s2leKHH7j3DmUYYMqUWMycGYv4eHXL8upqM44dq8Unn+TBYLC2LG9qYrFqVT6WLOnncXlqay34979zwLr3fdZhKLC6oW+/LeSdlsXFqfH224MRF6fmLY+LU+Pqq3sgO7seS5b8hfp6S8vrlZUmrFqVj8ce6yP4WVqtGd98U8hZJpEwePDBFNxxRyJv/Z49NejZU4Px4yOxeXMpPvggB2Zza/tJVlYdtm0rw+TJMS7/v/1VXZ0Fq1blc5YpFBIsWdIXI0eG89YPD1dg4sQo9OkTiNdeO805qvrtNy1On65Henqw2+VhWeC993JQV2dpf+VORqeE3YzRaMWuXRWcZcnJAVi+fBgvrNrq3z8YS5f25526nTnTYHebHTvKYbNxv6bnzEkSDKsrMQwwfXos5s9P5r32n//kckJM7NavL+E1aj/6aKpgWF0pKUmDZ5/ty1v+668VAms7b8OGEhw50npE3JFXIF1FgdXNZGZWtlwev+yxx/o4fcVv+PAwjBsXwVl24UIjL5Qu++WXMs7z2FgV5s5Ndrq8d96ZiLS0IM6y2loLLlzwn4ZgTzQ2NmHz5lLOspEjw5GR4dy9Xfv0CcTQoaGcZZ78bvLyGjlHe8HBckydyj2a9eXpOAVWN3PqVD3neXS0ivcH356+fbkBYjRaUVxs4K1XUmLgdIkAgIyMKJe+sSUSBhMm8Hfe8+e7RuP71q1lvC+QadNcO91t252hsFDvVtuTyWTDW2+d5VyI+fvfU1vaF/0BtWF1M5WVJs7z9PQgl78xhU4dq6vNSEzUcJYJXbG69toern0YgORkDW+ZP11q98TRo7Wc51FRSgwf7lp/qhtuiMagQSGcZSzLgnGxYj/9NA8XL7Z+8dx4YzRGjw5HUZHzXSw6GgVWN9M2sKKjVS6/R04O/+jmyqtY9j7L3nrtEQrIigrvXb73laYmFmfPctv/pkyJcfkLRK2WIiXFs061v/2m5XSpiI9XY8GCXh69Z0egwOpmJk2KxjXXtB7ljB7tuGFXSNvuBiqVFD168IfrtA0smYxBaKjrpxclJfzTzZ49xd/rPTdXx7t4MHZshJ21O05VlQkrVpxveS6TMXjqqTSfjFlsDwVWN3PXXY6vzrUnP78Rp09z28FGjQoXPCoYNCgEDz+c2vJcpZK41WArdBUyNVX84wvbtifKZAyiovjBX1FhQkWFEdXVFgQESJGUpEFkpGfjOS+z2Vi888456HSt3VzuvjvJb8dvUmARp506VY8lS05yjgrkcgkWLUoRXH/IkFAMGeJag35bjY1Ngr2//XWHckXbII6MVEIiaU50g8GK9etLsG9fFQoL+W1IGk1zcF1/fRQmT3b9NPKy77+/iKys1uAcMCAYt9+e4N6bdQIKLOKQyWRDXp4Oe/dW4aefimE0tl7RkkgYLFjQy612KWctX34e5eXc9qqUlIB2+4yJQXW1mfP8cnvisWO1WL78vGAb4GV6vRVnzjTgzJkGbNtWjkWLUpCeHmR3fSFnzjRgzZqilucBAVI8+WSaX48ioMAiHMeO1eL55/+CWi2F0WjlDP24UnS0Cv/7v/08PoJy5JtvCnn9uGQyBs8/n+7XO5WzrjwNA4DoaCX27q3C22+fdalbQm6uDs89dxJTpsTiwQdTnPrdNDZa8dZbZ2G1tn7QokW9BU9J/QkFFuExGq2cI6m25HIJnnwyrcPCqrGxCe++ew47d/J7bM+Z07NLnA4C/MCqqjLjvff44/fCwxVITtZALpegoECP8nIjbx2WBbZsKYVMxmDhwvav7n30US4qKlqP4MaPj8R11znXWdWXKLCIyywWG5577iQGDAjGI4+kejRura39+6vw3ns5qKrinw4NGRKKOXN6eu2zfK3teM4rh8MAwPjxEZg/P5k3YaLRaMVff9XhvffOc8Z1As3DaoYPD8WIEfb7cu3cWYG9e1unoomMVGLxYuF2SH9DPd0Jh1IpQXJyAHr1CkBYmMLh6cWpU/V48skTOHGi1v5KTiotNeLFF7Pw4otZgmE1c2Yc3nlnCGSyLnAuiObQsTe1j0IhwWuvDcAzz/QVnN1VpZJi5MhwvPPOYCQm8tvyvvqqwO4pZWmpEf/5T17Lc4YBnnyyDwICxHHsIo5Skk7Tv38wVq0a2fK8qYlFdbUZJ07UYv36Et78VwaDFc899xc++2wEr6e7M4xGK1avLsR//1sEi4U/oDkwUIbFi3tj6tRY1/8zfqztcJwrzZ3b06nhUjExKixdOgALFx7hjOXMy2vEhQuNvM6kTU0s3nrrLOd0/9ZbEzBwILeXvD+jIyzi0OW+QZMmRWPFimH4+OPhvG9jo9GKr78ucOl9WRbYvr0cc+YcwurVBYJhNWlSNL7+elSXCyuguR1QyMCBwZgxI87p94mKUgp2NhWaf+ybbwo5oxRSUwMxe3aS05/lDyiwiEvS04Px8sv9W/oLXbZzZ4XTs11qtWY89dQJ/OtfpwVP/5KTA/Dee0PxwgvpCAvzn4G33qRWC/ciHzcu0uUroDNn8gO97eDwEydqsW5d6wSBSqUETz2VJrpTbDolJC4bNSocN9wQzelyYLWyyM6uR0yM47GJf/xRjddfP43aWv7kcGFhCsyfn4zp02N5gdjVyGQMFAoJb2iOUJtUe3r1CgDDgNNuVVfHbdD/8cdizusDB4bg7NkG3lhGIW3Dj2UheAV3zJgeHX77Lwos4pa0tCBeHymhMX+XNTWx+OyzPHz/fRGvQVgul+CuuxJx991JPrvfnS8EBMhgNnM7jyYlud4OqFBIEBGh5HQ0bWjgfiHY2pxxHzlSw7sq6Yr33svhLUtPD4JG07EdeimwupGPPspFTk7rN2p0tMrt+b979+YPPi4vt98ze/nyHMG77IweHY7HHuvTJXquuyohQY2aGm5guTvgWKXitu501VNpCqxupLGxCceOtXZBkMslePbZvm6dfgntEEqlcJPorl0VvLCSyyV4+OHemDUr3uXP7ipSUgJ4M18UFxtcniqGZYGyMu6XRc+erh+piQEFVjfS9nTDYrGhtNTo1lhAodttCYVYcbEBb799lrNMLpfgzTcHY9iwjhvWIwa9e/N77BcV6V0OrMpKE+8qa9vAuvHGaAwf7t7v+8yZBvz+u5az7P77k3nrBQfbv3OSt1BgdSNC/aROnqxzK7CEJvETOk386KNcTp8jhgGef75ftw8roHn6nbaN5YcP1whOCe2IUFtU2/nCPJlna9u2ck5gMQxw882+OTKmbg3diNBUwxs3un6jzKoqE7Zs4d44QaGQ8Do71tdb8Mcf3G/mu+9OwvXXR7n8mV1RRISCN4Rmz55KnDvn/Hz1FosN33/PvZ/h5bGHXREFVjcSF6fGmDHcOdWzs+s5/XPaY7OxeO01freECRMieQ3Ge/ZUcoafyOUSv55ryRduvDGa85xlm+dWF+pIK2TDhlJeX7Z585KhUHTNXZtOCbuZBx/sjUOHqjnTiixffh6lpUYsXtzbYQN8Xl4jVqw4zxs7KJdLcP/9/BkC2t7/cMSIMOh0TbxZCtwRFCS3e9t2MRk9ugcGDQrhNL6fPduAp58+iWef7Wv3dJ1lm8cMrl3L/bJJTw92+hZhYkSB1c307KnB9OmxvKt2a9deRFZWHYYMCUVKSgBSUwMRGqpAcbEBRUV6/PVXHbZv598UFQAWLuyF2Fh+h9G2nRIPHtTi4EEtbz133HZbAh55JLX9Ff1c8+DjNDz66DFOkOflNeLxx09g2rQYpKYGtgxGLyhoRG5uIw4c0OLkyTreezk7H5ZYUWB1Q/PmJeO337S8U4nLM1i64uab4wXv4lxTY3Y4wJe0iohQ4IUX+uGf/zzDmXLGaLRi3bpip99n/vxkj++e4++65okucSgsTIEPPxyGhAT3O2vGxanx+uuD8NhjfQRfF7qxKrFv4MAQvPXWoHaHNgmRShk8/HBvn12560wUWN1UdLQKn3wyAnPnJrs0HEalkuJ//qcXvvxyJK8B/0oUWK5LTNTgww+HYfbsJKd7vI8cGY4VK4Zh8mTX7hYtVnRK2I0FBMgwf34ybr89AUeP1uDo0VqcP69Dfb0F9fUWKJVSJCaqkZCgQUKCGgkJavTtG+TUsI8bb4zBjTd2j53ImxSK5nGVN90UhyNHanDoUA3Ky42oqTGDYRiEhMgRHi7HoEEhGDkyvFPmYL/xxmje1UxfocAiCAyUYfz4SIwf33WvLomNWi3F2LERPrmxqj+jU0JCiGhQYBFCRIMCixAiGhRYhBDRoMAihIgGBRYhRDQosAghokGBRQgRDQosQohoUGARQkSDAosQIhoUWIQQ0aDAIoSIBgUWIUQ0KLAIIaJBgUUIEQ0KLEKIaFBgEUJEgwKLECIaFFiEENGgwCKEiAYFFiFENCiwRIJhwNpsrK+L0eVYrSwYBjZXt6P66Bjt1QcFlkioVBJtaanR18XocsrKjFCpJJWubqdUSrTl5aaOKFK3Vl5uhFJpvz4osESCZbFx+/Zys6/L0dVs2VJmZllscGPTjbt3V1B9eNmOHRVmwH59UGCJhF5vXbp2bZH5zz+rfV2ULuPPP6uxfn2xSa+3vubqtnq9den69SXmY8dqO6Jo3dKxY7XYsqXUYX3QrerFo9hkss148cWsjXfemaicNClaHhqq8HWZRKm21ozNm0stP/9cYjCZbNMBlLjxNsVms23GsmWnN958c7wyIyNSHhIi93ZRu4W6Ogu2by+3bNlSZjCbHdcHA2DppQfxvcxLPzMcrBOv0UhfYRhMsVjYsI4vUtejUEi0Nhu7Sa+3LgNw0cGqmZd+ZjhYh+rDQ3I5o2VZOFMfS+kIS3yK9XrrAl8XQszMZpcvCjpC9eEhswstgdSGRQgRDQosQohoUGARQkSDAosQIhoMgPxLD+J7I9BcJ4d9XRACgOrD3yTTEZZ/YXxdAMJB9eGHlvq6AKRFJlr7/hDfywTVhz9ZSkdYhBDRoMAihIgGBRYhRDQosAghouFMYCWqVNKVCoVEK5UyZgAsPZx/yGSMSaWSlKtU0k8BxDnx+ybikqhSSVfK5RKtREL7h6sPqZQxKZWScqVS4tT+0d7g5wyVSrp+8uQY9YQJEfKwMAUUCjooc4XJZFNUVZmifvtNO2/r1tI7L01nss/X5SJekaFQSNZfe20P9ciR4fKQEBnkcto/XGE22xQ1NZaoo0dr5u3bV3Xnpell7O4fjqaXSVQoJKefe65vQHp6cMeUtps5daoeb711Vmc229IhPI1G5qWfGZ1WKOJI5qWfGQKvJcrlktMLFvQK6N07oPNK1IWdP6/D55/n6ywWu/uH/W4NGo30pSlTYhUUVt4zYEAwbrghWqFSSV/wdVmIZ1Qq6UvjxvVQUFh5T2pqIK65JkKhVErs7h92A4tlMWP8+AiaQtHLxo2LVEgkmOnrchBPsTNGjgyn/cPLRo4MVUgkjN39w25gmc228IgImoLX26KjlTCZbJG+LgfxjNnMhoeF0f7hbT16KBzuH46OsCQSCQ2l8jaJhAHLUncSsWNZVsLQ7uF1lzLH7v5BOw4hRDQosAghokGBRQgRDQosQohoULqoImUAABLxSURBVGARQkSDAosQIhoUWIQQ0aDAIoSIBgUWIUQ0KLAIIaJBgUUIEQ0KLEKIaFBgEUJEgwKLECIaFFiEENEQdWBZrSwMBitYtvM+jxBin9XKwmjsuH2yvbvmuIRlgddfP42GhiYAQHp6MO67r6dX3ttiseHkyTocPKhFYaEeNTUWNDY2gWUBqZRBQIAMCQlq9O8fjOHDQ5Gc7Nlc2zpdEw4c0KKkxIjSUgNKSozQak1Qq6WIjVUhJkaFmBg1hg4NBc3rTS4rLTXi4kWDx+8TGipHnz6BHr8PywKffpqHxsbmfTIlJRCzZnnnbnNNTSzOnm3A8eO1KC01or7e0nIAIZEw0GikiI5WITU1AP37ByM+Xu3xZ3o1sHJyGpCVVd/yPDzcO1PIbt9eju+/L4JebxV83WplUV9vQXa2BdnZ9Vi79iKGDQvF3XcnISHBtV8SywJ791bi228LW4L3Snq9Fbm5jcjNbQQA/PjjRYwbF4G77kpCaChN8d3dHTigxe+/az1+n/T0YK8EVkGBHjk5upbnoaHe2Sd/+02LrVvLYDQK75M2Gwudrgk6nQ65uTps21aO/v2DMXVqDGJiVG5/rlcDyxsVdSWdrgkffpiLEydqXd722LFanD7dgCee6INBg0Kc2qa83IhPPsnDmTMNTn9Oc8BV4dChGsyZk4Trr49yuayk66iqMvu6CBzHjtV49f30eiu+/bbQpX3ksuzseuTm6jB3bjLS0twLY6+1YdXXW3DggPcCy2Zj8f77OQ7DimGaTwftMRqtePPNsygs1Lf7eRaLDe++e85hRTiaw9totOLzzy8gK6uu3c8iXZdWa/J1EVrodE04ftx7f482G4uvvipodx9xdC8Ik8mGzz+/gNJSo1tl8MoRlslkw1tvnRU8hXLX2rXFOHWqnre8X78gZGREYcCA4JZTsLKy5naDX34pw9mz3F+m1cri//4vHy+/3N9h4Pzww0UUFXHbHqRSBlOmxGDw4FDExakQGipHdbUZpaVGnDhRh23byjgN8SwLfPhhLl5/fRCdHnZDViuLmhpLy3OplEFIiHt/B0FBnu2aZrMNn3+e39J25Q3bt5fj/Hkdb3mvXgEYPTocqamBLeWuqjKjvNyIffuqcOFCI2d9q5XFunXFePjh3g73SSEe/VbMZhuOHKnB1q1lLW063lBba8GGDSWcZRIJg3vuScTUqbG89ePj1YiPV2PUqHDs3l2BL78sgMVia3n93LkG7NtXifHjhe8edO5cAzZvLuUsCw9X4Kmn0tCrF7dBPSJCiYgIJQYNCsH48RF4//0czrdFXZ0Fq1bl4/HH+7j8/ybiVlNjhs3W+gXWq1cAHnwwpVPLYLHYcOpUPfburUJRUftnFs6qr7dg165KzjKJhMG0aTGYMIG/X0VHKxEd3byfHDpUjZ9+KkZTU+vvJj+/EYcP12DkyDCXyuF0YFksNuTlNaK01IiSkuarZtnZ9XYb3Tyxf38Vp+IB4Kab4gTD6koMA1x/fRQaG6347rtCzmvffluEMWN6QC7nnwXv31/Fuwx73309eWHVVlKSBo88koqXXjrFOdLKyqoDyzo+hSRdT9v2q6goZYd+XlMTi6IiPSorTaioaH7k5upgMtna39hFR4/W8vbJiROjBMPqSgwDjB4dDr3eyjso2Ly5FMOGhUImc35HcTqwCgv1eOWVbKff2BN793KTPDJSiVtuiXd6+2nTYnDwoJZzKFpfb0FRkQEpKfwQunCB+03Us6cGo0aFO/VZvXoFYMKESOzaVdGyTK+3oqTE4JXLuEQ8qqq47VeRkR0bWCUlBnz4YW6HfsZlhw9zG+/DwxWYNMn5C0wTJkTg+PFaFBe3NrvodE0oLTUiMdH5/cTvOo5WVJh4/ViuvjrcYeN6WxIJIxg4Qo3vNhvLW56S4toVjL59g3jL8vK8d4pMxKHtEVZHB1Zn0WrNKCvjNpIPGRLisHG9LYmEwZAh/Kv1paWu9Vnzu8CqrORfZRkxwrXzXACC/a+EAqu62sxp7wKab5ftCqEGUrmczge7m7ZHWB19SthZamr4XTUGDHCuq9CVoqP5/a9KSly7Wuj0KWF8vBovvdTf4TqvvZbtcZf86mr+L8edjmZRUfxthC45h4UpIJMxnAZBVxsrCwr46wtVDunarjzCkskYhIV5p5OmPdHRKixe3NvhOh9/nOvxPllba+Eti4hw/f8mdCAg9N6OOB1YKpUU/frxT32uxDAMWA9/O20DSyplEBTk+qXhigp+csfF8Y+6pFIGsbFqTkjl5OjQ1MQ61RjIssDhw9WcZRIJ41FvXiI+NhvLORKJjFR2+EUXpVIi2CZ7JW/sk3V13FC5PBTOVVot/2AkOtq1o1Cv9nT3hrS0QMyZ0zr+UKWSuFXxQm1I9sYXDh0aygms6mozPv/8AhYtav+S9MaNJbwuHePGRUCtlrpYYiJmNTUWzpXiK9uvjEYrcnJ0qKoyQ6dr7hfVPBZVhZgYpeCVa3+SnByAmTNbxx8qFO7tk0JnLkIHEY74XWClpwcjPT3Yo/cwGKzYt6+Stzw5WSO4/i23xOPgQS2n/WzPnkowDHD77QmCh/Ymkw0//ngRW7aUcZbL5RLcdluCR+Un4iPUfqXVmrFxYwmysxt4XQIuY5jm8Jo2LbbdMxhf6d07wOMB/kajlXelEYDLV9L9LrC84csv83lXbBITNYLtWkDzofWDD6bgzTfPcvqwZGZW4vfftRg1Khzx8WpERChRU2NGSYkBWVl1vM+QShk88ECKy432RPza/i2cP69DZmYlp21UCMs2z/CwcuUFDB0aiptuivO4l7s/+vnnEl57VWysyuV9pcv9ZtavL8HevVWcZVIpg4ceSnF4GJueHoxXXx2ATz+9gNzc1uEHZrMN+/dX2d/wkogIJR54IAUDB3p2dEjEqe0RVn6+673Mjx+vxdmzDbjjjgSnB+yLwa5dFbyjK6mUwV13JXbu0Bx/YjA0Dz4WmjFi1qx4p+bHSkzU4Lnn+uKNN8641I+KYYCbb47DgAEUVt2VUIPyZVFRSqSnN499VSolqKgwoazMiIICPQwG7kgRg8GK1asLsXBhL6Smej69jC8ZjVasW1eMY8f4ExhMnBjlVsfqLhFYhw/XYNWqfMEuEenpwU5NWGazsfjll3L89FOxywNGWRb47LML2L27Eo88ktpl+t8Q5wn1H9RopLj33p5257W6PFzl0KFqTtcDq7V5VoQlS/pBoxHnxZusrHr89FMx7woj0NwmNnGie9MwiTqwKitN+PrrAsHGPKA5xefNS263l/zlwNmzh99QL5UyiI5WIS5OhchIJWpqLCgtNaCszMgbs3X+vA4vvpiFJUv6tXu5mXQdLMvvjtOjhwILFvRy2Ntdo5Hi9tsTMGxYKD799AKnYV6vt2LHjnLcdJN3ZgftLNXVZmzYUMKZyPNKY8b0wKxZcS6NXLmSKAPLZLLh55+LsWVLGa+XOtD8hzBnTk9kZDgemHnZ6tUFgmE1enQ4Zs/uKdhJzmi0YuPGUmzeXAqzubUMOl0TPv64eYoZVwZ1EvFqarJh0qRoGI1WGI1W2GzA1KkxCAx0bvdKTQ3EpElR2LatnLP84MFqTJ0a4/fdHoDmtt6dOyuwZ4/whQa1WooZM2KdHqNrj6gCi2WbZ1ZYs6ZIcLgAAIwdG4HZs5OcnoeooECPrVvLeMsfe6wPRo+2/8tVqZq/Ha+/Pgovv3yK8w1bXGzApk0lmDXL+QHbRLzkcgn+9jfPZpqdODEK2dn1nDnZLBYbcnJ06N/ff9tGWRY4erQGmzeXob5euNf6iBFhmD491itXP0UTWLW1Fnz00Xm7h5rx8WrMn5/scuVu2lTCWzZpUrTDsLpSjx4KPPRQb/zrX6c57RA//1yCG26IEW0bBOlcEgmDAQNCeJNIFhbq/Taw6ust+O67Is6c8VeKjlbhllvi0Lu39y4eiCKwjh+vxX/+kyeY4CEhctx2WwKuuy7SpdHjQPOp5cGD3GE1gYEy3HNPkkvvM2BAMCZMiERmZutppdlsw8WLeqSl+WdnQOJ/hIZzeXMWX286c6YBa9YUtfTcv1JQkAw33ND8pe/qPtkevw4sq5XFmjVF2LKllDeAUy6XYNq0WMycGQuVyr2jmMpKE+9eg8nJAVAqXW8zSEsL4gQW0HxqSIFFnCUUWEJttL5ktbLYsqUMe/dW8vZJmYxBRkYkrrsuyq19yBl+HVhfflmAX38t5y0fOjQUc+f29HhGBKFL0UlJ7k26JzQJ2ZWTlRHSHonAPu5vc2qtX18i2NexX78g3HxzfIeP8vDbwDpwQMsLK7lcgjlzkjBpUrRXPkOvFzqcde+mAT168P+w3L10S7qn8nL+DCP+NE3R8eO1vLCSyRjMnBmHa67p0Sll8MvAKi834rPPLnCWyWQMnnuur1cbIIOD+eHk7lFR2xkZgeY530nXlpvbiK+/LuAsu+OOBLf+TsvL+Uf8rk6/0lGqqsz44YeLnGVSKYMFCzq3R75fdvBYvbqQc3MLhgEeeqi316+WCM3CIDQZnzOEps5ISqLOo11dYqIaBoP10l2Omx9Hjrh+49/medW4HaAlEgYREf4RWBs3lnA6SjMMcPfdiZ0+fMjvAqv55o/cCp8+PQ5jxnj/kDM+Xs3rr1VUpBcc+9Setn9sMhmDuDj/OZwnHUOhkCAxkXsknZ1dz+lM7IzDh2sE5033h2YFvd7Ku3lqRkYkhg4N7fSy+F1gHTpUzblyJ5dLMHVqTId8FsMIzxf/xRf5Lt0qacuWMvz1F/cOuyNGhPnFHxvpeKmp3CNpi8WGrVvLnJ6auK7Ogm3b+POqTZvm+LZ2neXkyTrOPimTMXbv8dnR/K4Nq22j3oABwdDrrdDrPb//YUCAlNduNWNGLPbtq+JcPq6qMuGVV7KxcGEvh/cmtFpZ7NpVgTVruPdADAiQYe7cZI/LS8Rh8OBQ7N5dydmp9+2rQk2NGffckwSFwv5xwfnzOqxeXcjrzzRhQoTf3D287RlPnz6BLcOQPKVWS50ewgT4YWC1ndbl+PFa3i/MXVOmxODee3tylkVHqzBzZhzWreM2KObnN+If/ziFiROjkJYWiJgYNWJjVTCbbSgrM6KoqHlIj1Bj+733JvnNHxvpeHFxKtx+ewLWrCniLM/Kqsd77+Wgf/9gJCaqkZCgQXCwDOXlJpSUGFBYqMehQzW82UiDgmS47jrPhvp4U9v22dOnG3D69FmvvPe4cREuDfD2q8Cqr7d0yJ2k23PLLfGoq7PwulHYbCx27CjHjh38vmD2DB0a6rPDZeI7V10VhvJyI3bv5nYebr4jM39gvT0qlRT33dezwzpeukqna+qQO0m7y68CS+hopTMwDDB/fjISE9X48UfhOXycMXZsBBYu7OXdwhHRmDo1FlYri99+0/JGUDgjOFje8nfoL9pO/exrfhVYQv1QOgvDNA96Hj8+Elu3lmLTplKn283i49W4+eb4Tus8R/wTwwAzZ8Zh/PhI7NhRjj//5J/uCVGrpbjuukiMGxfhd1PJtJ362de8GlirV4/yaPtx4yIwblyEl0rjHqVSglmz4vG3v0Xj6NHalsn6Lj8YhkFQkAxhYQr06xeEgQODMWBASIffg46IR2ioHLffnoCJE6OQm9uIykpTywNovvij0cgQG6tCnz6BSErSdNgV5TffHOTR9lddFYarrnL9zusdxa+OsPxJYKAM48f7NjyJuIWHKxAeTndQ8ib/Ov4khBAHKLAIIaJBgUUIEQ0KLEKIaFBgEUJEgwKLECIaFFiEENGgwCKEiAYFFiFENCiwCCGiQYFFCBENCixCiGhQYBFCRIMCixAiGhRYhBDRoMAihIgGBRYhRDQosAghokGBRQgRDQosQohoUGARQkSDAosQIhqOAot15iaQxDVWKwuGgf/c+5u4hWEYlqXdw+tsNsf7h93AUiol2ooK/7rra1dQWWmCUimp9HU5iGcUCkar1frXbdy7Aq3WDLmcsbt/2A0shsHG/furqEa8bM+eSrPNhg2+LgfxFLPx6NEa2j+87NChajPL2t8/7AaWXm9dunVrmfnkybqOKVk3dPJkHXbsKDcZjdbXfF0W4hmj0bp0794q87lzDb4uSpdx7lwDDhyoNplMNrv7h6Nb1RebzbYZ7757buO0abHKsWMj5MHBdGd7d9TXN2HXrkrLr7+WGcxm23QAJb4uE/FYscVim/HFFwUbJ0yIUI4YESYPDKT9wx06XRP++KPa8vvvWoPF4nj/YAAsvfSwJ16lkr4ikWCKxcKGebeo3YNczmhtNmwyGq3LAFx0sGrmpZ8ZHV4o4ozMSz8zHKwTr1JJX2EYTLFYbLR/uEEul2htNnaTyWRrb/9Y6sxXQrHRaF3gpbJ1SxaLr0tAOhDtHx5qarI6vS71wyKEiAYFFiFENCiwCCGiQYFFCBENBs1XQjJ9WwxyybxLP1f5sAyk1bxLP1f5sAykVcb/Ay1wlOkEwQi2AAAAAElFTkSuQmCC"
- }
- ],
- "prompt_number": 4
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [],
- "language": "python",
- "metadata": {},
- "outputs": []
- }
- ],
- "metadata": {}
- }
- ]
-}
\ No newline at end of file
diff --git a/notebooks/Gradient-Descent.ipynb b/notebooks/Gradient-Descent.ipynb
index 98c4a8c7..77583a28 100644
--- a/notebooks/Gradient-Descent.ipynb
+++ b/notebooks/Gradient-Descent.ipynb
@@ -1,395 +1,401 @@
{
- "metadata": {
- "language": "haskell",
- "name": ""
- },
- "nbformat": 3,
- "nbformat_minor": 0,
- "worksheets": [
+ "cells": [
{
- "cells": [
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In supervised learning algorithms, we generally have some model (such as a neural network) with a set of parameters (the weights), a data set, and an error function which measures how well our parameters and model fit the data. With many models, the way to train the model and fit the parameters is through an iterative minimization procedure which uses the gradient of the error to find the local minimum in parameter space. This notebook will not go into the details of neural networks or how to compute the derivatives of error functions, but instead focuses on some of the simple minimization methods one can employ. The goal of this notebook is to develop a simple yet flexible framework in Haskell in which we can develop gradient descent methods.\n",
+ "\n",
+ "Although you should keep in mind that the goal of these algorithms (for our purposes) is to train neural networks, for now we will just discuss some abstract function $f(\\vec x)$ for which we can compute all partial derivatives.\n",
+ "$\\newcommand\\vector[1]{\\langle #1 \\rangle}\\newcommand\\p[2]{\\frac{\\partial #1}{\\partial #2}}$\n",
+ "\n",
+ "Gradient Descent\n",
+ "---\n",
+ "The simplest algorithm for iterative minimization of differentiable functions is known as just **gradient descent**.\n",
+ "Recall that the gradient of a function is defined as the vector of partial derivatives:\n",
+ "\n",
+ "$$\\nabla f(x) = \\vector{\\p{f}{x_1}, \\p{f}{x_2}, \\ldots, \\p{f}{x_n}}$$\n",
+ "\n",
+ "and that the gradient of a function always points towards the direction of maximal increase at that point.\n",
+ "\n",
+ "Equivalently, it points *away* from the direction of maximum decrease - thus, if we start at any point, and keep moving in the direction of the negative gradient, we will eventually reach a local minimum.\n",
+ "\n",
+ "This simple insight leads to the Gradient Descent algorithm. Outlined algorithmically, it looks like this:\n",
+ "\n",
+ "1. Pick a point $x_0$ as your initial guess.\n",
+ "2. Compute the gradient at your current guess:\n",
+ "$v_i = \\nabla f(x_i)$\n",
+ "3. Move by $\\alpha$ (your step size) in the direction of that gradient:\n",
+ "$x_{i+1} = x_i + \\alpha v_i$\n",
+ "4. Repeat steps 1-3 until your function is close enough to zero (until $f(x_i) < \\varepsilon$ for some small tolerance $\\varepsilon$)\n",
+ "\n",
+ "Note that the step size, $\\alpha$, is simply a parameter of the algorithm and has to be fixed in advance. \n",
+ "\n",
+ "Though this algorithm is simple, it will be a bit of a challenge to formalize it into executable Haskell code that we can later extend to other algorithms. First, note that gradient descent requires two things:\n",
+ "\n",
+ "- Something to optimize (a function)\n",
+ "- What to optimize over (the parameters)\n",
+ "- A way to compute partials of the function\n",
+ "\n",
+ "Note that we don't actually need to *call* the function itself - only the partial derivatives are necessary.\n",
+ "\n",
+ "We're going to define a single class for things on which we can run gradient descent. Although later we may want to modify this class, this serves as a beginning:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ ":set -XTypeFamilies\n",
+ "class GradientDescent a where\n",
+ " -- Type to represent the parameter space.\n",
+ " data Params a :: *\n",
+ " \n",
+ " -- Compute the gradient at a location in parameter space.\n",
+ " grad :: a -> Params a -> Params a\n",
+ " \n",
+ " -- Move in parameter space.\n",
+ " paramMove :: Double -- Scaling factor.\n",
+ " -> Params a -- Direction vector.\n",
+ " -> Params a -- Original location.\n",
+ " -> Params a -- New location."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In order to use some type `a` with our gradient descent, we require that it is an instance of `GradientDescent`. This class requires a few things.\n",
+ "\n",
+ "First off, we use type families in order to define a representation for the parameter space. We want to be able to operate on points in the parameter space of our function; however, while something like a list of values might be nice and simple in one case, it is inappropriate and inefficient when storing the weights of a neural network. Thus, we let each class instance decide how to store its parameters by defining an associated type instance. (We will see an instance of this later!)\n",
+ "\n",
+ "Next, `GradientDescent` requires a single function called `grad`, which takes the thing of type `a` and the current point in parameter space (via a `Param a`) and outputs a set of partial derivatives. The partial derivatives have the same form and dimension as the point in parameter space, so they are also a `Param a`. \n",
+ "\n",
+ "Finally, we must be able to move around in parameter space, so `GradientDescent` defines a function `paramMove` which does exactly that - it takes a parameter vector and an amount by which to move and uses these to generate a new position from an old one.\n",
+ "\n",
+ "Let's go ahead and create the simplest instantiation of this class and type family: a single-argument function. Note that this is just for demo purposes, and we're going to use numerical differentiation to compute the derivative."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- We need flexible instances for declarations like these.\n",
+ ":set -XFlexibleInstances\n",
+ "\n",
+ "instance Floating a => GradientDescent (a -> a) where\n",
+ " -- The parameter for a function is just its argument.\n",
+ " data Params (a -> a) = Arg { unArg :: a }\n",
+ "\n",
+ " -- Use numeric differentiation for taking the gradient.\n",
+ " grad f (Arg value) = Arg $ (f value - f (value - epsilon)) / epsilon\n",
+ " where epsilon = 0.0001\n",
+ " \n",
+ " paramMove scale (Arg vec) (Arg old) = Arg $ old + fromRational (toRational scale) * vec"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We're getting closer to implementing the actual algorithm. However, we have yet to define when the algorithm *stops* - and, in order to give maximum flexibility to the user, we'll let the stopping condition be an argument. This lets the user specify an error tolerance, as well as how they want to derive this error:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Define a way to decide when to stop.\n",
+ "-- This lets the user specify an error tolerance easily.\n",
+ "-- The function takes the previous two sets of parameters and returns\n",
+ "-- `True` to continue the descent and `False` to stop.\n",
+ "newtype StopCondition a = StopWhen (Params a -> Params a -> Bool)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With a single instance of our class, we can now implement our gradient descent algorithm:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "gradientDescent :: GradientDescent a => a -- What to optimize.\n",
+ " -> StopCondition a -- When to stop.\n",
+ " -> Double -- Step size (alpha).\n",
+ " -> Params a -- Initial point (x0).\n",
+ " -> Params a -- Return: Location of minimum.\n",
+ "gradientDescent function (StopWhen stop) alpha x0 =\n",
+ " let iterations = iterate takeStep x0\n",
+ " iterationPairs = zip iterations $ tail iterations\n",
+ " in\n",
+ " -- Drop all elements where the resulting parameters (and previous parameters)\n",
+ " -- do not meet the stop condition. Then, return just the last parameter set.\n",
+ " snd . head $ dropWhile (not . uncurry stop) iterationPairs\n",
+ " where\n",
+ " -- For each step...\n",
+ " takeStep params = \n",
+ " -- Compute the gradient.\n",
+ " let gradients = grad function params in\n",
+ " -- And move against the gradient with a step size alpha.\n",
+ " paramMove (-alpha) gradients params"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's go ahead and try this for some simple functions. First, we need to create a stopping condition. In this case, we're going to stop when successive updates to the parameters do not affect the outcome very much - namely, when the difference between the function value at successive parameters is below some $\\varepsilon$-tolerance. In other scenarios, we may want to use a more complicated stopping criterion."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Create a stop condition that respects a given error tolerance.\n",
+ "stopCondition :: (Double -> Double) -> Double -> StopCondition (Double -> Double)\n",
+ "stopCondition f tolerance = StopWhen stop\n",
+ " where stop (Arg prev) (Arg cur) =\n",
+ " abs (f prev - f cur) < tolerance "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's finally try to minimize something, such as the relatively trivial function $f(x) = x^2 + 3x$. It's graph looks like this:\n",
+ "\n",
+ "\n",
+ "![](\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAABDwAAAKSCAIAAACxzlwQAAAMQmlDQ1BJQ0MgUHJvZmlsZQAASA2tV3dYU9cb/u5IAiEJIxABGWEvUfaUvQUFmUIdhCSQMGIIBBX3KK1gHag4cFS0KmLVagWkDkTcFMVtHUUtCkotDlyo/M4Nwz592v9+93nOuW/e7zvffb/vnntyDoCmrUAuz8W1APJkhYr4iGD+pNQ0PuMesEAbmGAAxgJhgTwoLi4G/vN6cwMwynjVkYr1n27/btAWiQuEAFgcMmeICoR5CP8MQHKEckUhAK0Z8RYzCuUU7kRYV4EEIvyRwlkqTEfqQTdjAFuqfBLjQwDoXgBqLIFAkQXACUU8v0iYheJwRAg7yURSGcJrEPYXSgSI41xDeFRe3nSENREE24y/xcn6GxYIMoZjCgRZw3ggF2ooqIVKC+S5glmqH//PLi9XieqlusxQz5IoIuPRXRfVbVPO9GgKsxA+IMuYEIuwDsJHpVTGA7hVooxMQpjybxcWhKBaAg/h1yJBaDTCRgA4U5mTFDSIrQUKhFT+eLC0MCpxECcrpscPxsezZbkTqPmB4uBzJOKoIVwuLghLQDzSgGdnSsOjEEbvCt9dLElMQRjpxOuLpMkTEOYg3FyQk0BpoOJcKZaEULzKR6GMpzRbIr4zUxFO5Yh8CFZeAUKq+IS5UKB6lj7i3QoliZGIR2OJGJE4NAxh9FxikliWNKiHkMgLg6k4lH+xPFc1v5FOolycG0Hx5gjvKChKGBp7plCRSPGobsSNbME4ar4izcRTeWEcVRNKzzuIgRAIBT4oUcuA6ZAN0tbuum70a8ASDgJQQBaIwXGQGRqRorLIUJ8AxfAnyJBPwfC4YJVVDEWI/zTMDox1hEyVtUg1IgceoyfkkYakP+lLxqA+EDUX0ov0HhrH1xzSSQ+jh9Ij6eF0uyEGhEh1LmoKkP4LF41sYpSdAvWyoRy+xKM9prXRHtKu09pptyEZ/lBFGcx0mnSRYkjBcOTx0I6iDVRFjComg64hH9IaqXYng0k/pB9pJ3mkITiSbiiTIDIA5eaO2KHqUaqVw9q+1HKo7kN+lGr+33Ic5Dn2HPdBFRlDWaE3OVSJf0b5YpGCCHlF/9OT+JY4RJwlThLniaNEHfCJE0Q90UIco/Cg5nBVdbKGnxavqmgOykE65ONU49Tl9HHo13CuAsRQCqh3gOZ/oXhmIZp/EDJdPkshzZIU8oPQKizmR8mEo0fxXZycXQGoNZ3yAXjFU63VGO/CFy6/EcC7FK0B1HLKp7wABBYARx4DcN984Sxeok9qJcCxy0KlomjAj6RuNPRPoYm+DAMwAQuwRTm5gAf4QiCEwTiIhURIhamo6hLIQ6pnwBxYCCVQBithLWyErbAddsOPcBDq4CichDNwES7DdbiD5kYHPIMeeAN9GIYxMDbGxQwwU8wKc8BcMC/MHwvDYrB4LBVLx7IwGabE5mCLsTKsHNuIbcOqsZ+wI9hJ7DzWht3GHmBd2EvsA07gLFwXN8at8TG4Fx6ER+OJ+BQ8C8/Hi/El+HJ8PV6F78Vr8ZP4Rfw63o4/w3sJIDQIHmFGOBJeRAgRS6QRmYSCmEeUEhVEFbGPaEDv+irRTnQT70k6ySX5pCOan5FkEikk88l55DJyI7mbrCWbyavkA7KH/Exj04xoDjQfWhRtEi2LNoNWQqug7aQdpp1G304H7Q2dTufRbeie6NtMpWfTZ9OX0TfT99Mb6W30R/ReBoNhwHBg+DFiGQJGIaOEsYGxl3GCcYXRwXinpqFmquaiFq6WpiZTW6RWobZH7bjaFbUnan3qWupW6j7qseoi9VnqK9R3qDeoX1LvUO9jajNtmH7MRGY2cyFzPXMf8zTzLvOVhoaGuYa3xkQNqcYCjfUaBzTOaTzQeM/SYdmzQliTWUrWctYuViPrNusVm822Zgey09iF7OXsavYp9n32Ow6XM5oTxRFx5nMqObWcK5znmuqaVppBmlM1izUrNA9pXtLs1lLXstYK0RJozdOq1DqidVOrV5ur7awdq52nvUx7j/Z57U4dho61TpiOSGeJznadUzqPuATXghvCFXIXc3dwT3M7dOm6NrpRutm6Zbo/6rbq9ujp6LnpJevN1KvUO6bXziN41rwoXi5vBe8g7wbvwwjjEUEjxCOWjtg34sqIt/oj9QP1xfql+vv1r+t/MOAbhBnkGKwyqDO4Z0ga2htONJxhuMXwtGH3SN2RviOFI0tHHhz5mxFuZG8UbzTbaLtRi1GvsYlxhLHceIPxKeNuE55JoEm2yRqT4yZdplxTf1Op6RrTE6ZP+Xr8IH4ufz2/md9jZmQWaaY022bWatZnbmOeZL7IfL/5PQumhZdFpsUaiyaLHktTy/GWcyxrLH+zUrfyspJYrbM6a/XW2sY6xfob6zrrTht9myibYpsam7u2bNsA23zbKttrdnQ7L7scu812l+1xe3d7iX2l/SUH3MHDQeqw2aFtFG2U9yjZqKpRNx1ZjkGORY41jg9G80bHjF40um708zGWY9LGrBpzdsxnJ3enXKcdTnecdZzHOS9ybnB+6WLvInSpdLnmynYNd53vWu/6ws3BTey2xe2WO9d9vPs37k3unzw8PRQe+zy6PC090z03ed700vWK81rmdc6b5h3sPd/7qPd7Hw+fQp+DPn/5Ovrm+O7x7RxrM1Y8dsfYR37mfgK/bX7t/nz/dP/v/dsDzAIEAVUBDwMtAkWBOwOfBNkFZQftDXoe7BSsCD4c/DbEJ2RuSGMoERoRWhraGqYTlhS2Mex+uHl4VnhNeE+Ee8TsiMZIWmR05KrIm1HGUcKo6qiecZ7j5o5rjmZFJ0RvjH4YYx+jiGkYj48fN371+LsTrCbIJtTFQmxU7OrYe3E2cflxv0ykT4ybWDnxcbxz/Jz4swnchGkJexLeJAYnrki8k2SbpExqStZMnpxcnfw2JTSlPKV90phJcyddTDVMlabWpzHSktN2pvV+FfbV2q86JrtPLpl8Y4rNlJlTzk81nJo79dg0zWmCaYfSaekp6XvSPwpiBVWC3oyojE0ZPcIQ4TrhM1GgaI2oS+wnLhc/yfTLLM/szPLLWp3VJQmQVEi6pSHSjdIX2ZHZW7Pf5sTm7Mrpz03J3Z+nlpeed0SmI8uRNU83mT5zepvcQV4ib8/3yV+b36OIVuwswAqmFNQX6qLNc4vSVvm18kGRf1Fl0bsZyTMOzdSeKZvZMst+1tJZT4rDi3+YTc4Wzm6aYzZn4ZwHc4PmbpuHzcuY1zTfYv6S+R0LIhbsXshcmLPw10VOi8oXvV6csrhhifGSBUsefR3xdU0Jp0RRcvMb32+2fkt+K/22danr0g1LP5eKSi+UOZVVlH1cJlx24Tvn79Z/1788c3nrCo8VW1bSV8pW3lgVsGp3uXZ5cfmj1eNX167hryld83rttLXnK9wqtq5jrlOua18fs75+g+WGlRs+bpRsvF4ZXLl/k9GmpZvebhZtvrIlcMu+rcZby7Z++F76/a1tEdtqq6yrKrbTtxdtf7wjecfZH7x+qN5puLNs56ddsl3tu+N3N1d7VlfvMdqzogavUdZ07Z289/KPoT/W73Pct20/b3/ZATigPPD0p/SfbhyMPth0yOvQvp+tft50mHu4tBarnVXbUyepa69PrW87Mu5IU4Nvw+FfRv+y66jZ0cpjesdWHGceX3K8/0Txid5GeWP3yayTj5qmNd05NenUteaJza2no0+fOxN+5tTZoLMnzvmdO3re5/yRC14X6i56XKxtcW85/Kv7r4dbPVprL3leqr/sfbmhbWzb8SsBV05eDb165lrUtYvXJ1xvu5F049bNyTfbb4ludd7Ovf3it6Lf+u4suEu7W3pP617FfaP7Vb/b/b6/3aP92IPQBy0PEx7eeSR89OyPgj8+dix5zH5c8cT0SXWnS+fRrvCuy0+/etrxTP6sr7vkT+0/Nz23ff7zX4F/tfRM6ul4oXjR/3LZK4NXu167vW7qjeu9/ybvTd/b0ncG73a/93p/9kPKhyd9Mz4yPq7/ZPep4XP057v9ef39coFCoNoLEKjHMzMBXu4CYKeivcNlACZn4Myl8sAGzokIY4ONov+BB85llAHtIWBXIEDSAoCYRoAtqFkhzEJ3avudGAi4q+twQwx1FWS6uqgAxlKgrcm7/v5XxgCMBoBPiv7+vs39/Z92oL36bYDG/IGzHuVNnSG/N6BQy00E/3H9D+/EadV2+/VpAAAACXBIWXMAABYlAAAWJQFJUiTwAAABnmlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS40LjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlBpeGVsWERpbWVuc2lvbj4xMDg0PC9leGlmOlBpeGVsWERpbWVuc2lvbj4KICAgICAgICAgPGV4aWY6UGl4ZWxZRGltZW5zaW9uPjY1ODwvZXhpZjpQaXhlbFlEaW1lbnNpb24+CiAgICAgIDwvcmRmOkRlc2NyaXB0aW9uPgogICA8L3JkZjpSREY+CjwveDp4bXBtZXRhPgoauLcRAABAAElEQVR4AezdB7wU1dkHYLqgIoJgARv2hmLBkoiKJRr9gjUWYoXYUFRMNEoEa6xRRI0FC8TeYjeWxBqNhRALsUQlxgIWVFRUEC7e7z2sGuRe4Ja9u7O7z/zw797ZmTnnPGcpu+/OTPPq6upmTb+ccML90UivXstE7rjjmk3foBYIECBAgAABAgQIEKi3QPPmzWOfwrxHqHvnWtR9U1sSIECAAAECBAgQIECg8ALNC/Mu6sUX34uxnX32o5HXXLNnZO49XDywECBAgAABAgQIECCQEQGVloxMhG4QIECAAAECBAgQIFBKAq0K09m1114qGlpooTaRTz31VuSPfrR8pIUAAQIECBAgQIAAAQLzFnBOy7x9PEuAAAECBAgQIECAQJEFClRpyY1yjz3WiQc33/xCpEpLkWde8wQIECBAgAABAqUuMG1KjGDyjHaRHdvP/x/206ak7Vu1b58y/iudRaWldOZKTwkQIECAAAECBAhUpEBB32JtvvkKgXzJJU9FvvbapMhVVulSkewGTYAAAQIECBAgQKDhAlPGjY6d97no88jd13wpckznwZHD+60W+cNlYvx42T4/jTzk2hdnPbV35OMfjors3aWgbwdmtd6QUGlpiJp9CBAgQIAAAQIECBAomECB7tMy+3huvTW9w3v55Q8ihw3bZvanPCZAgAABAgQIECBAYF4CVa/Gs/u0Xj3y5x9WR/btkmopg5t3i+wzYUZa0/V/9ZO37jo71jyy9IGR+6+Xzma5f3DryN90Hxv5whHrRc6+uE/L7BoeEyBAgAABAgQIECBAoE4C/3sTVqfN87HRDjuk94XXXJPe202cmL6H17XrIvk4sGMQIECAAAECBAgQKHeBqenM8AnNTonc5NvTw1P9ZLW108BfeCddH6xv147ph1nLEn2OiP/v377ttz83a9ate594vHbnDt+vyf4D57Rkf470kAABAgQIECBAgEBFCxSh0tKuXfoWXd++a0bedNPzkYMHb1bRk2DwBAgQIECAAAECBOomMOX1dCXeR5qlOsl31ZNUaem5W6qfPBj//XBpO1uNJb7nFE/eccfKkYPvXPGHG2b6J5WWTE+PzhEgQIAAAQIECBAgUIRKSw59l116xIN99rkhsn//XpEdOrTLPSUJECBAgAABAgQIEKhVoH33DWJ9n2ZPRk6btUX7Zuk8lmdufST99JNZq2qJqlh3/5DfRK54+eWRsy4kVst2uVW5a4jVfLq6Ol2vrPCLSkvhzbVIgAABAgQIECBAgEA9BIpWaenYMdVVttxypcg//WlcZP/+G9aj4zYlQIAAAQIECBAgUIECHdO5KOs02yryqUlDI/t2SVWUz15Mlw/rtUI6vyVXg5k4cWo86jrrSmLjRh8Tj8dsdlrk0BXTuTDjH74rckavvpGr5XaKR98txaqofNf+nP9XaZlTxM8ECBAgQIAAAQIECGRKoHlx30VNmPBZcAwceFvkTTftHdm2bbq2mIUAAQIECBAgQIAAgbkJTP7n6HjqgOEfRu7eY3zkmKUHRw7vt1pk1VvXR7Ze/ubIJ5/YMvLHmx4Z+YPloFvix6mX7Rb53VXI0vO5s1mK+x4h9eOHi0rLDz38RIAAAQIECBAgQIBAxgSKXGnJaZx44gPxYO21u0buumu6qpiFAAECBAgQIECAAIH5CFSl64ZNnpK+qdSx4+z1kvnsN4+nVVrmgeMpAgQIECBAgAABAgQI1C6QiUrLK698EL076aS/RF5/fb/Ili19b632CbOWAAECBAgQIECAQNMJqLQ0na0jEyBAgAABAgQIECBQtgJFu0/L7KKrr75E/LjUUukC0Y88kq5+sPXWK8++gccECBAgQIAAAQIECFSsgG9hVezUGzgBAgQIECBAgACB0hDIRKUlR7Xnnj3jwRVXPBup0lIaLx+9JECAAAECBAgQIND0AiotTW+sBQIECBAgQIAAAQIEGiGQoTctG2+8XPyaOfOb+DVmzDvxqxHjsisBAgQIECBAgAABAmUikKE3LWUiahgECBAgQIAAAQIECORVIEPntOTGtdde68aDG254LrJXr2XyOlgHI0CAAAECBAgQIECg9ARUWkpvzvSYAAECBAgQIECAQEUJZK7SstVWK8UEXH75M5GvvTYpcpVVulTUlBgsAQIECBAgQIAAAQKzC6i0zK7hMQECBAgQIECAAAECmRNoXl1dnblONWt2yy0vRK9efvnDyBNP3CaDPdQlAgQIECBAgAABAuUn0Lx58xhU1t4jqLSU3yvNiAgQIECAAAECBAiUlUDmzmnJ6f7f/60RD6655p+R7733eeRSSy2Se0oSIECAAAECBAgQIFBRAiotFTXdBkuAAAECBAgQIECg9AQyek5LDvKKK9I1xL74YnrkUUf1Lj1dPSZAgAABAgQIECBQUgLOaSmp6dJZAgQIECBAgAABAgSyIZDpSssnn3wVSvvue2PkddftFdmhQ7tsuOkFAQIECBAgQIAAgTIUyFVaag6suNcTc05LzRmxhgABAgQIECBAgACBDAlkutKSczr33MfiQadOC0YecECvDOHpCgECBAgQIECAAIHyEnBOS3nNp9EQIECAAAECBAgQIFAQgRKotEyY8FlQDBx4W+RNN+0d2bZt64LgaIQAAQIECBAgQIBAZQmotFTWfBstAQIECBAgQIAAAQJ5EWiVl6M06UG6desQx+/Zs2vkvfe+Grnrrj2atEUHJ0CAAAECBAgQIEAgOwKuHpadudATAgQIECBAgAABAgRqESiBc1pyvX7llQ/iwUkn/SXy+uv7RbZs6R1XzkYSIECAAAECBAgQyI+Ac1ry4+goBAgQIECAAAECBAhUlEAJnNOSm4/VV18iHiy1VPvIRx4ZH7n11ivnnpIECBAgQIAAAQIECJSxgG9YlfHkGhoBAgQIECBAgACBchAomUpLDnvPPXvGgyuueDZSpaUcXoDGQIAAAQIECBAgQGB+Aiot8xPyPAECBAgQIECAAAECRRUosTctG2+8XPyqqvomfv3jH+/Er6LqaZwAAQIECBAgQIAAgSYXKLE3LU3uoQECBAgQIECAAAECBDImUGLntOT09torndlyww3PR26wwTIZI9UdAgQIECBAgAABAgTyKaDSkk9NxyJAgAABAgQIECBAIO8CJVlpyV037KqrxgTH669/FLnyyp3zTuOABAgQIECAAAECBAhkQUClJQuzoA8ECBAgQIAAAQIECMxVoHl1dfVcn8z2E7ffPi46OHbshMjTTtsu253VOwIECBAgQIAAAQIlINC8efPoZdbeI6i0lMBLRxcJECBAgAABAgQIVLJASZ7TkpuwHXZYPR5cffXYyDff/CSye/dOuackAQIECBAgQIAAAQJlI6DSUjZTaSAECBAgQIAAAQIEylOghM9pyU3IDTc8Fw/eeOPjyKFDty7PWTIqAgQIECBAgAABAgURcE5LQZg1QoAAAQIECBAgQIBAeQmU8DktuYnYaae14sEee1wbOWHCZ5HdunXIPSUJECBAgAABAgQIECgDAee0lMEkGgIBAgQIECBAgACBchYo+XNacpMzevSYeDBp0peRxxyzRW6lJECAAAECBAgQIECgXgK5c1pq7lLcO7eotNScEWsIECBAgAABAgQIEMiQQJlUWqZM+TpQ99rrushRo3aP7NJl4Qwx6woBAgQIECBAgACBUhBw9bBSmCV9JECAAAECBAgQIEAgYwJlUmnJqV566VPxYPr0mZFHHLFpxqh1hwABAgQIECBAgEDWBVRasj5D+keAAAECBAgQIECAQAYFyqrS8sknXwXxvvveGHnNNXtFduzYLoPoukSAAAECBAgQIEAgmwIqLdmcF70iQIAAAQIECBAgQCDTAmVVaclJjxjxt3jQrl3ryIMO2jjT/DpHgAABAgQIECBAIEsCKi1Zmg19IUCAAAECBAgQIECgRATKsNLywQdTAv+Xv7wl8oYbfhG58MILlMh06CYBAgQIECBAgACBYgqotBRTX9sECBAgQIAAAQIECJSoQBlWWnIzcdZZj8SDJZdsH7nffhuU6PToNgECBAgQIECAAIFCCqi0FFJbWwQIECBAgAABAgQIlIlA2VZa3n3305iiww67PfLGG/eOzF1PrEzmzTAIECBAgAABAgQINIGASksToDokAQIECBAgQIAAAQLlLtCqXAe49NKLxtDWX3/pyLvueilyjz16lutgjYsAAQIECBAgQIBAGQu0KOOxGRoBAgQIECBAgAABAmUgULbntOTm5j//+Tge/PrX90TeeGO6Z0ubNmVbXMoNWRIgQIAAAQIECBBosIBzWhpMZ0cCBAgQIECAAAECBCpXoMzLDiussFjM7eqrLx55772vRu6881qVO9tGToAAAQIECBAgQKAEBZzTUoKTpssECBAgQIAAAQIEKkmgzM9pyU3lq69+GA+GDn0gMndmS8uW3q1V0svcWAkQIECAAAECBOom4JyWujnZigABAgQIECBAgAABArMJlPk5LbmRrrZaOqdl2WXTnVsefPC1yJ/+dLXcU5IAAQIECBAgQIAAgYwL+JZUxidI9wgQIECAAAECBAhUukBFnNOSm+Tnn58QD37/+8cjr756z8gWLZrnnpIECBAgQIAAAQIECISAc1q8DAgQIECAAAECBAgQIFBvgYo4pyWn0rNnt3jQoUPbyMceGx/Zp89KuackAQIECBAgQIAAAQLfC+TqLd//GA+qq6tn/7HAj53TUmBwzREgQIAAAQIECBAgUD+BCqq05GD22We9eDBy5DORKi31e7HYmgABAgQIECBAoDIEiltXqWms0lLTxBoCBAgQIECAAAECBDIkUHFvWjbeeLn4FdcNi19PPfVW/MrQbOgKAQIECBAgQIAAAQI1BCruTUsNASsIECBAgAABAgQIEMi0QMWd05KbjdyZLddcMzZ+3GST5TI9RTpHgAABAgQIECBAoLIFVFoqe/6NngABAgQIECBAgEDmBSr0Tctmm60Qv778cnr8Gjv23fiV+ZnSQQIECBAgQIAAAQIVKlChb1oqdLYNmwABAgQIECBAgEAJClToOS25e3zut98GMWVXXvls5PrrL12C06fLBAgQIECAAAECBMpfQKWl/OfYCAkQIECAAAECBAiUtEBFv2nZYosV49dXX82IX88++078Kum51HkCBAgQIECAAAECZSlQ0W9aynJGDYoAAQIECBAgQIBAmQlU6DktuVls0aJ5PNh33/UjR48eE7nhhsvknpIECBAgQIAAAQIECGREQKUlIxOhGwQIECBAgAABAgQI1C7gTUuzPn1WjF+5e7Y4s6X2l4m1BAgQIECAAAECBIon4E1L8ey1TIAAAQIECBAgQIBAHQQq+pyWnE/uni3775/u2eLMljq8ZmxCgAABAgQIECBAoKACKi0F5dYYAQIECBAgQIAAAQL1FfCm5VuxH96z5e1nn327vpS2J0CAAAECBAgQIECgKQS8aWkKVcckQIAAAQIECBAgQCBvAs5p+ZYyd2bLfvvl7tnyj1i74YbL5o3ZgQgQIECAAAECBAgQaKiASktD5exHgAABAgQIECBAgEBBBLxp+QGzM1t+wOEHAgQIECBAgAABAhkQ8KYlA5OgCwQIECBAgAABAgQIzF3AOS0/sJn9ni2jRjmz5Qc4fiBAgAABAgQIECBQFAGVlqKwa5QAAQIECBAgQIAAgboKeNNSi9Tmm68Qv6ZNmxG/nnnm7fhVy0ZWESBAgAABAgQIEChTgfj+0RxLcQfqTUtx/bVOgAABAgQIECBAgMB8BJzTUgtQvK2Mtfvtt0Hk6NHpzJaNNnLPllqgrCJAgAABAgQIEChLgerq6kyNS6UlU9OhMwQIECBAgAABAgQIzCngTcucIt//7MyW7yk8IECAAAECBAgQIFBEAW9aioivaQIECBAgQIAAAQIE5i/gnJa5GuXObNl//3Rmy6hRYyKd2TJXLE8QIECAAAECBAgQaDIBlZYmo3VgAgQIECBAgAABAgTyIeBNy3wUN9tshfj19ddV8evpp9+KX/PZwdMECBAgQIAAAQIECORVwJuWvHI6GAECBAgQIECAAAEC+RZwTst8RGc/syV3z5aNN15uPvt4mgABAgQIECBAgACB/AmotOTP0pEIECBAgAABAgQIEGgCAW9a6oTqzJY6MdmIAAECBAgQIECAQBMIeNPSBKgOSYAAAQIECBAgQIBA/gSc01Iny+/ObOkVW48a9Y9IZ7bUCc5GBAgQIECAAAECBBotoNLSaEIHIECAAAECBAgQIECgKQW8aamH7mabdY9fM2bMjF9PPfVW/KrHzjYlQIAAAQIECBAgQKBBAt60NIjNTgQIECBAgAABAgQIFErAOS31kM6d2bLffhvEPrl7tmyyiXu21APQpgQIECBAgAABAvkUmDYljjZ5RrvIju3L+R/2Ki35fNk4FgECBAgQIECAAAECeRco5zdkecfKHTBOa4kHf/xjuoZY7rQW9ZYmonZYAgQIECBAgACBWgWmjBsd6/e56PPI3dd8KXJM58GRw/utFllzmfTqw7Hy9IFpm+5nPBp5xEYdI0tlUWkplZnSTwIECBAgQIAAAQIVKqDSUu+Jz53ZcsAB6cyWXL1FpaXeiHYgQIAAAQIECBBomEDVq7HfwLUPiOz/YXVk3y4TI8c07xZ51xYz0pquc/4jv/0SqQLTa+WIZu+mKLFFpaXEJkx3CRAgQIAAAQIECFSawJxvwipt/A0eb+/eK8S+V189NvKxx/4TufnmaY2FAAECBAgQIECAQBMKTJ0UB5/Q7JTITbrk2mkf/1tt7fT4hXfS9cT6dp3zfJW2HbumbVZM52aPj/9KbVFpKbUZ018CBAgQIECAAAECFSag0tKoCT/wwI1j/4sueiKyd+/0zrVFi+aNOqKdCRAgQIAAAQIECMxdYMrrT8WTjzTrENn2281SpaXnbn0iH/x2Tbn9T6Wl3GbUeAgQIECAAAECBAiUmYBKS6MmdMMNl4n9F1003YX0wQdfi9xuu1UbdUQ7EyBAgAABAgQIEJi7QPvu6Rq2fZo9GTlt1mbtm6XzWJ659ZH0009mrWp05K6XW/Mw1dXpemWFX1RaCm+uRQIECBAgQIAAAQIE6iGg0lIPrLlteuCBG8VTp532UOTWW68U2apVy7ltbD0BAgQIECBAgACBhgt0XDH2XafZVpFPTRoa2bdLVeRnL6bLh/VaIZ3fkqvBTJw4NR51nf1KYl9/HmsWaJ2+JTTvpVgVlbn1SqVlbjLWEyBAgAABAgQIECCQCQGVljxMQ48eS8VRunfvFHn33S9H7rxzjzwc1yEIECBAgAABAgQIzCmwXKwYNnZU5AFHnx35RY9055VPrrspcrsu6Z/3VW/dHNlt+ZSvzLgjPX7j/sjhw2ad97LAPfH4rZV3ilyufWm8HVBpicmyECBAgAABAgQIECCQXYHmWfu+Wnap5tezN974KDY59th7I6+/vl9k27at57eT5wkQIECAAAECBAg0VKAqXTds8pT0b86OHb+7a0utB6tK571UtZpVV6maddWxVmn7mnWW3HXDsvYeQaWl1lm1kgABAgQIECBAgACBrAiotOR5Jk4++S9xxJVX7hzZr9+6eT66wxEgQIAAAQIECBBoSgGVlqbUdWwCBAgQIECAAAECBMpUQKUlzxP7zjufxhEPP/z2yOuuS2e2LLzwAnluw+EIECBAgAABAgQINI2ASkvTuDoqAQIECBAgQIAAAQJlLaDS0iTTe/bZ6RrYiy22UOSAARs2SRsOSoAAAQIECBAgQCDfAiot+RZ1PAIECBAgQIAAAQIEKkBApaVJJvnDD7+I4w4YkO5CevXVe0V27NiuSVpyUAIECBAgQIAAAQL5E1BpyZ+lIxEgQIAAAQIECBAgUDECKi1NONUXXfRkHD13P9FBgzZtwpYcmgABAgQIECBAgEA+BFRa8qHoGAQIECBAgAABAgQIVJiASksTTvjkyVPj6Pvue0PklVfuHrn44gs3YXsOTYAAAQIECBAgQKBxAiotjfOzNwECBAgQIECAAAECFSmg0tLk037llc9GGx9//GXkscf2afL2NECAAAECBAgQIECgoQIqLQ2Vsx8BAgQIECBAgAABAhUs0KqCx16goe+xxzrR0i9+cX3kO+98GrnMMosWqG3NECBAgAABAgQIECh9gRalPwQjIECAAAECBAgQIECgnAWc01Kg2b3++ueipddf/yjyxBO3KVCrmiFAgAABAgQIECBQHwHntNRHy7YECBAgQIAAAQIECBCYJeCclgK9EHbZZa1oqV+/dGbLG2+kestKK3UuUNuaIUCAAAECBAgQIFDKAs5pKeXZ03cCBAgQIECAAAECFSDgnJaCTvLtt4+L9p555p3IM8/cvqBta4wAAQIECBAgQIDA/ARy57TU3Kq6urrmyoKtUWkpGLWGCBAgQIAAAQIECBBoiIBzWhqi1uB9/u//1oh9b7zxhch//ev9yLXWWrLBR7MjAQIECBAgQIAAgaYQKG5dpeaIVFpqmlhDgAABAgQIECBAgECGBJzTUoTJuP/+V6PV++77d+SIETsWoQeaJECAAAECBAgQIFCbgPu01KZiHQECBAgQIECAAAECBOYp4JyWefI0zZPbbLNKHPj665+LHDMmXUmsV69lmqYpRyVAgAABAgQIECBQ8gLOaSn5KTQAAgQIECBAgAABAuUtoNJShPlt2TK9VxwwYMPIyy9/JlKlpQjToEkCBAgQIECAAIESEVBpKZGJ0k0CBAgQIECAAAEClSrgTUvRZn6zzVaIX3EN7Fgef/w/8atoXdEwAQIECBAgQIAAgQwLeNOS4cnRNQIECBAgQIAAAQIEmjVzn5YivwrGjn03enDeeY9HXn31npG5M16K3C3NEyBAgAABAgQIVKSA+7RU5LQbNAECBAgQIECAAAECjRNw9bDG+TV67/XXXzqO0bXrIpF33fVy5M47r9XoozoAAQIECBAgQIAAgfIRcE5L+cylkRAgQIAAAQIECBAoSwHntGRiWseP/zj68etf3x153XX9IhdcsE0meqYTBAgQIECAAAEClSTgnJZKmm1jJUCAAAECBAgQIEAgTwIqLXmCzMdhzjzz4ThMly4LRw4YsGE+DukYBAgQIECAAAECBOohoNJSDyybEiBAgAABAgQIECBAICeg0pKhV8KkSV9Eb/r3vzly1Kg9Ijt3XihD/dMVAgQIECBAgACBchdQaSn3GTY+AgQIECBAgAABAgSaQEClpQlQG3fIkSOfjgN8+unUyGOP7dO4g9mbAAECBAgQIECAQD0EVFrqgWVTAgQIECBAgAABAgQI5ARagciawC9+sV50qV+/6yPffPOTyO7dO2Wtk/pDgAABAgQIECBAoGACLQrWkoYIECBAgAABAgQIECDQAAHntDQArRC7/OlP46KZZ599O/Kss3YoRJPaIECAAAECBAgQqHgB57RU/EsAAAECBAgQIECAAAEC9RdwTkv9zQqyx447rhHt3Hrri5HPPTchct11uxWkZY0QIECAAAECBAhUukCu3jK7QnV19ew/Fuxxrl3ntBQMXEMECBAgQIAAAQIECDREwDktDVEr2D6PPjo+2rruun9Gjhy5W2TNd70F64yGCBAgQIAAAQIEyl4ga+e0/PWvr4e5SkvZv/AMkAABAgQIECBAgEBpC6i0lMD8HXrobdHLXXZZK3KbbVYpgR7rIgECBAgQIECAQGkKZKfSMmPGzCDce+8bIlVaSvPVpNcECBAgQIAAAQIEKkbA1cNKYKoHDtwkennaaQ9Fbr75ipFt2rQsgX7rIgECBAgQIECAAIGGCtx++79i15VWWixSpaWhivYjQIAAAQIECBAgQKAgAiotBWFuXCM9eiwVB1h55c6Rt902LnLPPXs27pD2JkCAAAECBAgQIJBRgS+++Dp6dt11z0VeeOGOkSotGZ0q3SJAgAABAgQIECBAICeg0lIyr4SDDto4+jpo0O2RO+ywemT79guUTO91lAABAgQIECBAgEDdBHI1lt69u8fmyy7bMVKlpW5ytiJAgAABAgQIECBAoEgC7tNSJPiGNjt8+OOxa5s2qUR22GE/auhh7EeAAAECBAgQIECgFoHi3qflww+/iD4NGHBz5B//uGdkp04LRqq0BIKFAAECBAgQIECAAIHsCqi0ZHduau3Z5MlTY/2++6Y7g15++W6RSy65SK1bWkmAAAECBAgQIECgvgLFrbScccbD0eElllg4sn//Db/vvErL9xQeECBAgAABAgQIECCQRQFXD8virMyjTx07totnf/7zdSJHjnwmctiwbeaxvacIECBAgAABAgQIZF9g/PiPo5PPPvt25PXX/2KODqu0zAHiRwIECBAgQIAAAQIEsiWg0pKt+ahjb3bffe3Y8he/SGe2/PvfH0auuuriddzXZgQIECBAgAABAgSyJnDppU9Fl/bbb4PIdu1az9E9lZY5QPxIgAABAgQIECBAgEC2BFRasjUfdexN27bp3Wf//r0iL7kkvSs9//wd67ivzQgQIECAAAECBAhkR+DJJ/8bncndoeVnP1uj1o6ptNTKYiUBAgQIECBAgAABAlkRUGnJykw0oB8//elqsdett74Y+fTTb0VuvPFyDTiOXQgQIECAAAECBAgUXmDGjJnR6MUX/z1y8ODekS1b1l5TqX1t4XusRQIECBAgQIAAAQIECNQqoNJSK0tprGzRonl09NBDfxQ5YsTfIjfYYOnIVq1aRloIECBAgAABAgQIZFngttv+Fd1bdtlFIzfYYJl5dFWlZR44niJAgAABAgQIECBAoPgCKi3Fn4NG9mDDDdO70tw71D/9Kb1b3WOPdRp5TLsTIECAAAECBAgQaDqBTz+dGge//vp/Rl500c7zbUilZb5ENiBAgAABAgQIECBAoJgCzaurq4vZvrbzJPDOO5/GkQ4//PbIP/5xz8hFF22Xp2M7DAECBAgQIECAQKUING+ezpquueT3XcO55z4WTeTuPXjYYekM7XkvKi3z9vEsAQIECBAgQIAAAQJFFlBpKfIE5Lf53FWuv/pqRhz217/ePL8HdzQCBAgQIECAAIGyF8hVWvJbV5kd7T//+Th+HDz47sjrrtsrcuGFF5h9g1ofq7TUymIlAQIECBAgQIAAAQJZEXD1sKzMRF76se++68dx9tnnhsiddlozcqWVOuflyA5CgAABAgQIECBAoPECF1zwZBxkwIBekXWpseRaVGlpvLwjECBAgAABAgQIECDQhAIqLU2IW/hD596t9u+/YTR94YXpXeyIETsWvhtaJECAAAECBAgQIDCHwN/+9p9Y89ln6Q4tO+yw+hzPzvtHlZZ5+3iWAAECBAgQIECAAIEiC6i0FHkCmqL53DvXO+98KQ7+6KPjI7fYYsWmaMgxCRAgQIAAAQIECMxXYMaMmbHNH/7wVOSxx24R2bJl/Won9ds6GrAQIECAAAECBAgQIECgkAIqLYXULlBbLVqk+5gOGvTjyNNPfzjyRz9aLrJNG9MdDBYCBAgQIECAAIGCCtx664vR3oordopcb71uDWhbpaUBaHYhQIAAAQIECBAgQKBwAj56L5x1gVtaZ52u0eJqqy0eedNNL0Tus0+6i4uFAAECBAgQIECAQGEEJk9O1wq74YbnIy+5ZJcGN6rS0mA6OxIgQIAAAQIECBAgUAiB5tXV1YVoRxtFEnj//SnR8oEH3hI5atQekZ07L1SkvmiWAAECBAgQIEAg6wLNm6ezo/P1HuGccx6No+XuJXjooZs0ePAqLQ2msyMBAgQIECBAgAABAoUQUGkphHLR27jyymejD7mqy29/u1XR+6MDBAgQIECAAAEC2RTIV6XljTc+igEec8w9kdde2y9yoYXaNHjIKi0NprMjAQIECBAgQIAAAQKFEHD1sEIoF72NX/xi3ejD3nvfEPnyyx9ErrHGEkXvlQ4QIECAAAECBAiUq8AFFzwRQxswYMPIxtRYcj4qLeX6OjEuAgQIECBAgAABAmUioNJSJhM572G0bds6Njj44I0jc+96c9fJzn1ncd77epYAAQIECBAgQIBA3QUee2x8bPzll9Mjt99+9brvOI8tVVrmgeMpAgQIECBAgAABAgSKL6DSUvw5KFgPtt565Wjr9tv/Ffngg69FbrvtqgVrXUMECBAgQIAAAQLlLTB9+swY4MUXPxV5/PFbRrZoke760vhFpaXxho5AgAABAgQIECBAgEATCqi0NCFu1g6dO4Nl0KAfR8eGDn0gcrPNVohs1y6d8WIhQIAAAQIECBAg0BiBW255IXZfZZUukT17dm3MoebYV6VlDhA/EiBAgAABAgQIECCQLQGVlmzNRwF6s/rq6Q4t66+/dOS11/4z8sADNypAu5ogQIAAAQIECBAoV4FPPvkqhnbTTanScumlu+Z9mCoteSd1QAIECBAgQIAAAQIE8inQvLq6Op/Hc6wSEfj44y+jpwcccHPkZZeld8NLLbVIifRdNwkQIECAAAECBJpKYG738Zv3u4azznokOrToou0ic/cGzG//VFry6+loBAgQIECAAAECBAjkWcA5LXkGLZXDLbbYQtHV3XdfJ/KSS9K1tE85ZdtS6bx+EiBAgAABAgQINKnAvOsqszf92muT4sdnn30n8tpr95r9qTw+VmnJI6ZDESBAgAABAgQIECCQfwGVlvybltARc5WWffe9Ifr8/PMTInv27FZC/ddVAgQIECBAgACBYgl88006N/688x6PPOigdDXaprv7n0pLsWZZuwQIECBAgAABAgQI1ElApaVOTOW6UZs2LWNohx76o8gLL3wycuTI3SJbtvRuNhgsBAgQIECAAAECcxW4556X47nWrdO/J7fddtW5bpePJ/zbNB+KjkGAAAECBAgQIECAQJMJqLQ0GW3pHHjzzVeIzt55578i7747vWPeaae1Sqf7ekqAAAECBAgQIFBQgc8+mxbtXXXVmMjzzvtZAdpWaSkAsiYIECBAgAABAgQIEGi4QPO6X4O54Y3YsxQE3n57cnRz0KA7IkeN2iOyU6cFS6Hj+kiAAAECBAgQIJA3gebNm8ex5v0e4ZxzHo1tctcKO/zwH+et7bkfSKVl7jaeIUCAAAECBAgQIEAgAwIqLRmYhCx1YeTIp6M7kyZ9Gfnb326Vpa7pCwECBAgQIECAQJMLzLvS8vLLH0QPhg17IPKPf9wzcqGF2jR5n5o1U2kpALImCBAgQIAAAQIECBBouICrhzXcriz33Hff9WNc++13U+Tzz0+I7NmzW1mO1KAIECBAgAABAgTqLvDNN9Wx8fDhj0cecsgmkYWpseR6qNKSc5AECBAgQIAAAQIECGRUQKUloxNTrG61bds6mh40KF0F4rzz/hZ51VU/j2zVKt3r1EKAAAECBAgQIFCZAnfd9VIMPFdd2XrrlQuMoNJSYHDNESBAgAABAgQIECBQPwFXD6ufV0Vtffzxf47x9uixVGS/futW1NgNlgABAgQIECBQmQI1rx726adTgyJ3zvOIEX3j8fLLdyowjkpLgcE1R4AAAQIECBAgQIBA/QTqXWmp+d6rfg3aunQE3nvv8+jsIYf8KXLkyN0il1xykch53yE1NrCUpYDf+2U5rXUclNmvI1RZbmb2y3Ja6zgos19HqDLbrOa8n3nmIzHGDh3aRh56aLpuWOEXlZbCm2uRAAECBAgQIECAAIF6CJTY1cNqvvOrx1jztGlx+1DI1pdaKtVVdttt7cgLL3wyT34NP0whx16zl8VtPdefLPShpkxh1hR37MVt3ewX17+SWy/M7+55t1LJ/sUd+7znpTDPFlegkluffX7/9a/348exY9+NvPrqPWd/qkkf1/RXaWlScAcnQIAAAQIECBAgQKCxAiVWaWnscO1ff4E99+wZO+2//03139UeBAgQIECAAAECJSkwc+Y30e/hwx+PHDgwncfSrl26m1+xFpWWYslrlwABAgQIECBAgACBOgmotNSJqZI3at26ZQz/6KM3i7z++iQxfXpVZJs2XjxJw0KAAAECBAgQKD+BO+54KQa16KLtIvv0WanoA1RpKfoU6AABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgQIECBAgAABAgSaQqC5u5s3BWu5HvOjj76MofXvf3PkJZfsEtmtW4dyHaxxESBAgAABAgQqUCB3j5TLLnsqxn7QQRtnRMA5LRmZCN0gQIAAAQIECBAgQKB2AZWW2l2snYfAzTe/EM+OGfNO5Dnn/N88tvQUAQIECBAgQIBAqQi8+OJ70dV11ukaOXXq9Mi2bYt5b5bZ3VRaZtfwmAABAgQIECBAgACBzAk0ya02qqZNiYFWtWof2bZJWsicY0V1aNdde8R477//35GPPjo+costVvxeYMrkyfG4XceOkSb/e5ZSfTDr9/LkGeka7R3bz38+c7Pfun2afb/3S3XSv+93PWf/2/2mpT8B3vog3c2p23JdIuf/uvl2T//LkMC0KWkeZ7RLv5fr8Fs/Qz3XlcYL1Gf20+/0KVNStm/ftvFNO0IRBWbO/CZaHz788e/7kJ0aS65LKi3fT40HBAgQIECAAAECBAhkUWC+H4FNi16/+vCNkQO3uiPyjM9TbpSKKHMs6X323y44IHKzI6/9/rkRj0+Ix0f0Tt+Ns5SYQNWk6PDD154eudXV3SM/efiIyI4t03vdwYN7R55yyl8jN5r1grjpqIPi8Yfb7B45/tp/RR5+Y9q3hw9fQqHUlinjRkeX97no88jd10z3xB3TeXDk8H6rRf5wSa+T0QefEPn5NptHvn9tur7cemdeE7nbarX8YRHrLVkWqM/s58aRquv3X/DryLNfSnXXEw7fO3K5+M9SagLjRqff6Rd93ityzZfui+w8+OLIfvP8vTxt/OjYpt0u6U+MT16Y9TdFPLKUmkA9Zn/S32Jw+2y9WeS1L84a546Xxv8m3HFwpH/zzRIpsbjttnHR486dF8psv1VaMjs1OkaAAAECBAgQIECAQBKYb6UlbbDEamtHrtws1VjmtlS99ed46tFF0ic01dXpE9a37k+Plz/81sgDZn3u4hPXoCipJc3YauuvE7n28PT52exLjx5LxY/rr7905DmH7RZ5YushkdW7bRc5aeH0WcviR96f1lyW1lhKRqDq1ejqwLVT1bT/h9WRfbtMjBzTvFvkXVvMSGu6/u+Pjmmv3htrDnh6k8jqy/pFNlvh3Yh1rn45crfTN0prLKUiUM/Zb9ZsWozsrsGLRA7tlL4J/cJlqQZrKUWBXLVk1m/9Zh9Wp9/LXSYtHNl88WGRP64eHllL9awqndl4zoHpT4xmi6XP2i2lKFDf2b//0kdjmIOfSH9HXNPurcghrZePvPaf6ZVz7Hr+xRcMJbN8/PGX0dfrrnsu8qKLdor8/e+z2HmVlizOij4RIECAAAECBAgQIPC9wP8+Lv1+1Q8fpA06dl05cs0+6ZnpKWpblvhJrD1+//+dvrBEt+5pu+5dI9K1hywlJ9AqzWbXlVeP7N7ssVq7f8ghG8f6HTdNZzUcevwK32/Tvtua6fHI9EXXybMqLR2/f86DjAtMTbM5odkpkZt0yfV1Vs0tFVybvfBOOnuhb9f/zWfbldJr4KAX0+vk4Au+jlzs4fTJ6/CLj460lJhAPWd/2vjbYoA7np9GuePx6d7Jp56aKmw7HTIgskeX+f4VE1tZsiIw46MPoytrH/+zyO9+66daep9mD0e+n37rN1vuBx+gV8Wau4ZdGbnF5anOtveB6ZNaSykK1Hf2tzjm+Bhm22//0bdEPF5u1t8R7Ttk5Z4epTgLxerzeeel378777xW5NJLL1qsbsy33Rbz3cIGBAgQIECAAAECBAgQKKJAq4kT07fVa1lap+pI1y65z1PTpynzXlp993b7+83+fkP6bObSYaMifdr2PUtGHkybnD5N/2RqOj+h5tK6XadY2aXjrI9Qvt3kfzW02bdfdNH02lj1pVcjX7h7bHrqF6tEtF163cg+a/vULZFkbZn37H/9WromzCPNOkR+N+vpw9Weu6Vi64Px3xxLq9VixZA706duy+94SHpy7RERh8923ktaacmGQH5n/4OXxqRh7Zj+nL/8lD0j37ky5dqLp5rb59VHRP7go/n42VI8gcmT0t/4c/mDv9lrD12Xurbo1v/rYNtl4vF2a6fvTdT8nsWkh9NVxf770/R7v++K/408N/7z3YpZCBmM/M5+29lvxTXp0Rjvfd3TnwOjVvzu740MEuhSDYHHHvtPrHv33c8iTzopfWcqy4tKS5ZnR98IECBAgAABAgQIEGjW6pyTzwmGdovNftbJ1FgzddFtI085drvI+n5O9tZdQ2Kvu9dKn7kMXy9Xq4mHlgwJTHjmhujN2Q+8H/nDyf841iza59jIY/uuGPndMu27B3P8P7069j4ufQZ/1Mtpm/vv/3fkj7ulGssjL85tr3jSUjSB+cx+r97Rsz7N3ozMzV/7ZunL7M/c+kjqcc0PYialmmrfHdO3YD+ckb4Tf+8Bi0d2OzVdN6x6qKuHBUOGljzP/vSvYmx9fpS+zN5l1llwXQYMS2sOSfdseX1KqrS4jFAgZGBJv5ufGXVR5AOfpu784G/9j9Pf+722/Fl64oV0HaFvl2nvxIP7X0x/Gmz+3br0/6p0P4c9tjoystsp6TPaU297LfLOR66NPPKC9M34EUdsGelT90DIwJLX2f/BeN6Kn049MJ0Rceatp0f6N98PeDL8wxdfpHr4BRc8EXnyyemv9tatW2a4v6lrKi0ZnyDdI0CAAAECBAgQIFDpAs2rq6vrYJA+Zz17nX0iN3/mjsiNZvvwZPKss2Lade0a62eMGx058I4lI0cNTVWaqePTp7BPzugVud0876cbG1iyKDDtmejVlhul/POs++18N/npk5uJE9Pnc81ePC6i2x+3itz567Tm/INfivzZ/ek18MLw9HmbpXQE0idng5svH9nn2/u0TI7HpzbfIrLXh2Mjt+uSzmXKzX779y+Px4tclj5r//aePOMvi8fND1wgcurD+0d+95qJh5aMC9Rv9hefcV+Mp/Xy4yM/qR4a2XHa3yLXafdo5KO5NfHIUhICE++KbjbvlmqqE2bdlaXrlPQ3+DqLPBl536zZ7Dot/WkwcUrzyGafp8r857nzHr9Kf0cMXD+d43TGf1O1baPlfOYeDKWz1GP2U5Wua5c08aMPTjXVJYeMiNxuufS3/99ufS5yrd3S3/teAYGQ2SV3xbBc944+erM5+tm8efo9Xrf3CHPs2oQ/qrQ0Ia5DEyBAgAABAgQIECDQeIFWdTnEq/dfEpv95sU7I8+6J32assZuG0W2n3Uf3CO6rRSPt3rmr5F3bHRAZNou7oqaPmqJ5aj4740ZPmufhVFikT5Ru/+C9B3oR9INV5rd80z6FHa3jZaLrHrrtshuy98c+cqMUZGjbk+zf9eM9G3m3Qa/G3nTP817MJTckuZ32Ng0pwccfXbkFz1mfY5+3U3xeLtZd96oeivN+3ezf3E8HvHB3pFDRn8RueQLT0ReesZ5kWosgVBSS31nP9Xex44aHHnEqaMje737UOTAp9Ps+5w1EEpp6do3evviqEciTz57dOSK49L8/uaVGyPTtymaNRt/T/pzfqWLd4+c+nC/yNz6ZtM+iseL9Em5hhpLKJTcUs/Zf2yn9LfAASNn/Ytv5Mjvh3vQLW/E41Rtt2RVYNy496Jrf//7fyP/+Mc9s9rNWvql0lILilUECBAgQIAAAQIECGRHoE7ntFRVpe+vt2qVyjJV02Y9nv363NkZjZ7kWSDN9azJj9lPh56WTmOJO+DO+iE9rGX5+P10B5hDjnow8pije0duuOGytWxnVUkIVKXz2SZPSXc47pi7b888uz1lcqrONWufPmFvP6+XyTyP4smMCNRz9qumpNmf2nrW7CuxZWQSG9qN3GxOmTWbdfit39Bm7JdJAbOfyWnJQ6dmzJgZRxkwIFXJDjwwfWeqd+8Vaj2uc1pqZbGSAAECBAgQIECAAAEC8xKoU6VlXgfwHIHaBMaOfTdWn332o5GjR+8R2a5d+rTeQoAAAQIECBAgUHiB0aPT9f3eeCNd9++007abRwdUWuaB4ykCBAgQIECAAAECBAjULqDSUruLtXkROPPMdBWahRZKNZZBgzbNyzEdhAABAgQIECBAoO4Cb7+dzjkcNChdD/CKK34e2aXLwvPYXaVlHjieIkCAAAECBAgQIECAQO0CKi21u1ibF4EpU76O4+y3X7rGf+7bk2ussURejuwgBAgQIECAAAEC8xbI3dU+V2PZZptVYuMdd1xz3rvEs7lKS83Nckerub4wa9ynpTDOWiFAgAABAgQIECBAoIEC7qTQQDi71UWgffsFYrNBg34cmbuS2BVX7BaPW7VqWZfdbUOAAAECBAgQINBggbvvfjn2/eab6si+fdeo13GKW1ep2VWVlpom1hAgQIAAAQIECBAgkCEB57RkaDLKuytDhtwXA1x11S6R++23QXkP1ugIECBAgAABAkUU+PjjL6P1/v1vjhwxYsfI5ZfvVMf+uHpYHaFsRoAAAQIECBAgQIAAgf8JOKflfxYeNanA4MG94/i//OUtkVtssWLkcst1bNIWHZwAAQIECBAgUJkCI0Y8EQPPXSus7jWWLFs5pyXLs6NvBAgQIECAAAECBAg0c06LF0FBBe6886Vo78EHX4u86KKdIud2LfCCdktjBAgQIECAAIGyEHjyyf/GOC655KnIq67aPbJNm/pdtdU5LYFmIUCAAAECBAgQIECAQP0EnNNSPy9bN1Igd43wv/719TjOHXekqsvOO6/VyGPanQABAgQIECBA4KuvpgfC+ef/LfKEE7aKrG+NJcuGzmnJ8uzoGwECBAgQIECAAAECzVRavAgKKpD7luQxx2wRrQ4adHvkj3+8fOTiiy8caSFAgAABAgQIEGiYwOWXPxM7brjhMpHrrNO1YQfJ7F4qLZmdGh0jQIAAAQIECBAgQCAJuHqY10HRBK65Zmy0/fLLH0Seccb2ReuHhgkQIECAAAECpSyQ+9fUCSfcH4O4+uo9IxdeeIEGD8jVwxpMZ0cCBAgQIECAAAECBCpXQKWlcue+6COvqpoZfTjwwFsj99ln/cgtt1yp6L3SAQIECBAgQIBAqQjMnPlNdPWXv7wlct9907+m+vRp7L+mVFpKZfb1kwABAgQIECBAgACBDAm4eliGJqPSutKqVbo/629+0yfy+OP/HLnBBktHLrJI20gLAQIECBAgQIDAvAWuumpMbLD00otGNr7GMu+2ivusq4cV11/rBAgQIECAAAECBAjMR8A5LfMB8nRhBP7wh79HQ599NjVyyJB0D1cLAQIECBAgQIDA3AReffXDeCr3XZWrrtojHnfs2G5uG9drvXNa6sVlYwIECBAgQIAAAQIECCQB57R4HWRCYMCAXtGP/fe/KXLMmHcie/VK93O1ECBAgAABAgQIzC4wfXq6/urppz8cecQRm0bmq8YyeytZe+yclqzNiP4QIECAAAECBAgQIPADAee0/IDDD8UVGDv23ejA2Wc/Gjl6dPp2Zrt2rSMtBAgQIECAAAECOYFLL30qHrz//pTIk076Sd5ZnNOSd1IHJECAAAECBAgQIECg/AVUWsp/jktuhOef/7fo87RpMyKPO27Lkuu/DhMgQIAAAQIEmkLg5Zc/iMOecML9kaNG7R7ZoUN+rhg2e29VWmbX8JgAAQIECBAgQIAAAQJ1ElBpqROTjQopMHVqqrEMGHBz5OGH/zjyRz9aPtJCgAABAgQIEKhMga+/roqBDxhwS+TBB28U2bv3Ck1Ekau01Dx4dXV1zZUFW+PqYQWj1hABAgQIECBAgAABAg0RUGlpiJp9CiAwbtx70cqJJz4YOWpUupJYhw5tC9CuJggQIECAAAECWRP4wx/+Hl365JOvIocO3bpJu+eclibldXACBAgQIECAAAECBMpTQKWlPOe1bEZ1ySXpSuQffNBUVyIvGygDIUCAAAECBMpSIPfdk5NO+kuMLnfFsEUWadrvnqi0lOULyaAIECBAgAABAgQIEGhaAZWWpvV19EYKTJ8+M45w4IHpWhn77bdB5JZbrtTIY9qdAAECBAgQIJB9gdw96/r3L/T1VFVasv/a0EMCBAgQIECAAAECBDIn0CpzPdIhArMJtGnTMn4aMmSryN/85t7Inj27RnbqtGCkhQABAgQIECBQrgKXXfZMDK1Hj6Ui3bPOfVrK9XVuXAQIECBAgAABAgTKRMA5LWUykZUwjFGjxsQwX3ttUuQZZ2xfCUM2RgIECBAgQKACBZ5/fkKM+ne/ezgyd8WwhRdeoGAOzmkpGLWGCBAgQIAAAQIECBAoHwGVlvKZy7IfSVVVupLYoYfeFrnzzj0it99+tbIftQESIECAAAEClSMwdeqMGOwBB6Qrhg0e3Dtyo42WLfDwVVoKDK45AgQIECBAgAABAgTKQUClpRxmsaLG8Oabn8R4jzrqrsiRI3eNXGKJ9hUlYLAECBAgQIBAuQqcd97jMbTct0uOPbZPUYap0lIUdo0SIECAAAECBAgQIFDaAiotpT1/Fdv7669/Lsb+j3+8E3nuuT+LzH0qULEgBk6AAAECBAiUtMDYse9G/88665HIUaP2iFxooTZFGZFKS1HYNUqAAAECBAgQIECAQGkLqLSU9vxVbO+/+aY6xn744bdHbrPNypG564lVLIiBEyBAgAABAiUq8NVX06Pn++13U+Rxx6XzWNZff+kijkWlpYj4miZAgAABAgQIECBAoFQFVFpKdeb0OwTefffTyMMOS/WWiy/eJbJbtw6RFgIECBAgQIBAqQicc86j0dUWLZpH/upXmxe92yotRZ8CHSBAgAABAgQIECBAoPQEVFpKb870eA6B224bF2seeuiNyAsv3Cky91nFHJv5kQABAgQIECCQKYFnn307+pO7N0vuimHt2rUueg9VWoo+BTpAgAABAgQIECBAgEDpCai0lN6c6fEcAtXV6UpiRx99d+SGGy4Tudde686xjR8JECBAgAABAtkR+PzzadGZAQNuifztb7eM7NmzW0a6p9KSkYnQDQIECBAgQIAAAQIESklApaWUZktf5yHwwQdT4tmDDro18vzzd4zs3r3TPLb3FAECBAgQIECgWAJDhtwXTS+/fMfIgw7auFjdqLVdlZZaWawkQIAAAQIECBAgQIDAvARUWual47mSE/jzn1+NPueuJ3bZZbvG45YtW5TcKHSYAAECBAgQKFeBu+56KYZ2zz2vRF588c6RrVq1zNRgc5WWml3KnUVcc31h1vj3XGGctUKAAAECBAgQIECAQAMFVFoaCGe3LAscf/yfo3srr9w5sn//DbPcVX0jQIAAAQIEKkTg7bcnx0gHDboj8qKLUo1lmWUWzeDYndOSwUnRJQIECBAgQIAAAQIEsi6g0pL1GdK/Bgh8+unU2Kt//5sjhw3bOjI71z5vwHDsQoAAAQIECJS0wIwZM6P/hx56W+TOO68VucMOq2d2RCotmZ0aHSNAgAABAgQIECBAILsCKi3ZnRs9a6TAP/7xThzhrLMejbziip9HdujQNtJCgAABAgQIECikwMUX/z2ae//9dE+5U07ZtpBNN6AtlZYGoNmFAAECBAgQIECAAIFKF1BpqfRXQNmP/9JLn4oxvv32p5Gnn/7Tsh+vARIgQIAAAQLZEZj9ex9XXpm+97HIIln/3odKS3ZeP3pCgAABAgQIECBAgEDJCKi0lMxU6WjDBKqq0vU6Bg68PXL77VeL3GmndNUOCwECBAgQIECg6QQ++2xaHDx3LdMTTtgqHq+7bremay6PR1ZpySOmQxEgQIAAAQIECBAgUCkCKi2VMtMVPs4JqInrzQAAMaBJREFUEz4LgYED0/XRzz9/x8ju3TtVuInhEyBAgAABAk0nMGTIfXHw5ZfvGHnQQRs3XUN5P7JKS95JHZAAAQIECBAgQIAAgfIXUGkp/zk2wu8F7r//3/H4xhufjxw5ctfINm1aff+sBwQIECBAgACBxgvcdddLcZB77nkl8uKLd45s1apl4w9bsCOotBSMWkMECBAgQIAAAQIECJSPgEpL+cylkdRR4JRT/hJb5q6SftRRveu4l80IECBAgAABAvMWePvtybHBoEF3RF50UaqxLLPMovPeJYPPqrRkcFJ0iQABAgQIECBAgACBrAuotGR9hvQv7wJffjk9jjlgwM2RgwZtGvnjHy8faSFAgAABAgQINExgxox0X7hDD03XKd1553RHuB12WL1hhyr6XiotRZ8CHSBAgAABAgQIECBAoPQEVFpKb870OC8CL7/8QRzn+OPTNdSvvPLnkZ07L5SXIzsIAQIECBAgUGkCl1zy9xjyxIlTIk89dduSHr5KS0lPn84TIECAAAECBAgQIFAcAZWW4rhrNSMC11wzNnoyduyEyPPO+1lkixbNM9I33SBAgAABAgSyLzB27LvRyTPPfCQy992N3BVKs9/zufVQpWVuMtYTIECAAAECBAgQIEBgrgIqLXOl8UQlCHzzTXUM8+ij74rcYINlIvfee71KGLgxEiBAgAABAo0U+OyzaXGE/v3T9UhPOGGryHXX7dbIY2Zhd5WWLMyCPhAgQIAAAQIECBAgUGICKi0lNmG62xQCH330ZRx2wIBbIs8446eRa6yxRFM05JgECBAgQIBA2Qgcf/yfYyzdu3eKPOigjctmXLlKS83hVFen76cUa2lRrIa1S4AAAQIECBAgQIAAgboIqLTURck2FSHw5JP/jXFeeOETkVdeuXvkQgu1ibQQIECAAAECBGYXuPbaf8aPzzzzduT55/eNbNmyfCoBzmmZfa49JkCAAAECBAgQIECAQJ0EVFrqxGSjyhEYPvzxGOwXX0yPHDp068oZuJESIECAAAEC8xV4/vl0b7dTTvlr5MiRu0V27rzQfPcqrQ1UWkprvvSWAAECBAgQIECAAIFMCKi0ZGIadCI7AtOnV0VnDjroT5F77rlO5HbbrZad7ukJAQIECBAgUBSBTz75Kto98MBbI4cM2TJy/fWXLkpPmrpRlZamFnZ8AgQIECBAgAABAgTKUEClpQwn1ZAaL/DOO5/GQQ4//PbI3//+Z5Err9y58Yd1BAIECBAgQKDkBGbO/Cb6/Ktf3R2Zu+f9fvttUHKjqHuHVVrqbmVLAgQIECBAgAABAgQIfCug0uKlQGCuAo89Nj6eu+SSpyOvuCJdIWThhReY69aeIECAAAECBMpR4PLLn4lhvfbapMizztohskWL5uU40G/HpNJSxpNraAQIECBAgAABAgQINJWASktTyTpu2QhcdNGTMZaJEz+P/N3vtovMfQJRNgM0EAIECBAgQKBWgaeffivWn3tuuodb7jsXHTq0q3XLclqp0lJOs2ksBAgQIECAAAECBAgUSEClpUDQmildgaqqmdH5I464M3LTTbtH9uu3bukOR88JECBAgACB+Qp8+OEXsc1BB6W7spx2WvqexVprLTnfvcpjA5WW8phHoyBAgAABAgQIECBAoKACKi0F5dZY6QpMmpQ+cTn44D9FDhu2dWTPnt1Kdzh6ToAAAQIECNQqkPuGxaBB6RsWW2yxYuQee6xT65blulKlpVxn1rgIECBAgAABAgQIEGhCAZWWJsR16PITGDv23RjU7373UOTll6c7tyy22ELlN0wjIkCAAAECFStw4YVPxNg/+CB9wyJ3NkumKaZNie5NnpGuadaxfau8dFWlJS+MDkKAAAECBAgQIECAQGUJqLRU1nwbbV4ErrlmbBzn2WffiTz//L6RLVu2yMuRHYQAAQIECBAolsBjj42Ppi+77OnIkSPT9ykWXniBYnVmvu1OGTc6ttnnonQfud3XfClyTOfBkcP7rRZZc5n06sOx8vSBaZvuZzwaecRGHSNrLiotNU2sIUCAAAECBAgQIECAwHwE8vPVt/k04mkC5SWw997rxYDGjXs/cuTIZyIPPXST8hqi0RAgQIAAgQoSmDDhsxjteef9LfKcc3aIzHKNpVnVq9HDgWsfENn/w+rIvl0mRo5pnq5retcWM9KarnP+I7/9EqkC02vliGbvpiixxXdaSmzCdJcAAQIECBAgQIBApQnM+Sas0sZvvAQaIJD7rufQoVvHvr/85S2Ra621RGTv3is04Gh2IUCAAAECBIolMH16VTQ9dOgDkQMG9IpcZZUuxepMXdudOim2nNDslMhNvu1s+3i82trpAC+8k64n1rfrnOertO3YNW2zYvfIdO5OqS0qLaU2Y/pLgAABAgQIECBAoMIEVFoqbMINN38C7dunK4qccsq2kccee0/kCissFtmtW4dICwECBAgQIJB9gREjnohOrrBCp8i+fdfMSIenTU61lE+mprNTai5fv5bOvXmkWfr3Rttvn06Vlp679Yl88Ns15fY/lZZym1HjIUCAAAECBAgQIFBmAiotZTahhlNogVVXTV8mHTBgw8jcN2IvuWSXeLzAAn5zFXoutEeAAAECBOou8MAD/46Nx417L/Kyy9JdWbKzTHjmhujM2Q+k65Qulm52/90y9eN4tGiv3pF9mr0ZOW3WM+2bpfNYnrn1kfTTT2atanTkzuCteZjq6nS9ssIvKi2FN9ciAQIECBAgQIAAAQL1EPBhcD2wbEpgbgK5b8Hm7twyfHj6pulxx6XvlVoIECBAgACBrAm8+eYn0aVLL30q8rzz+ka2a9c6U51ccbsjoj+XbTe3Tr0VT7z3i30in5o0NLJvl3QNtM9eTJcP67VCOr8lV4OZOHFqPOo6+5XEvv481izQevbyTdq65lKsikrNnuTWqLTMTcZ6AgQIECBAgAABAgQyIdA8a++iMqGiEwQaJDBtWrrKxyGH3Bb585+nTzt22GH1Bh3JTgQIECBAgED+BSZPTpWHQw75U+TAgT+K3HzzFfLfTEGOOPmfo6OdA4Z/GLl7j3TnlTFLD44c3m+1yKq3ro9svfzNka/MuCOy2xv3Rw5c/aeRzc66JeK0Q3eKXK79nF+8yp3NkrX3CCotaeIsBAgQIECAAAECBAhkVkClJbNTo2OlKvD2259G1488Mn2qceaZ20euuuripToY/SZAgAABAmUhUFU1M8Zx1FF3RW6wwdKR++/fqxxGVpWuGzZ5Sjonp2PH7+7aUuvAqtJ5L1WtZtVVqmZddaxV2n7OOkuzZiottfpZSYAAAQIECBAgQIAAgXkJqLTMS8dzBBos8PTT6coev//9Y5G5O7d06bJwg49mRwIECBAgQKAxAuec82js/vnnX0eeckq6lcnc7kMST1X4otJS4S8AwydAgAABAgQIECBAoCECKi0NUbMPgToK3Hjj87Hlww+/EXnhhekaHQssUPO7o3U8mM0IECBAgACBegvcfvu42Ofuu1+J/MMfdo7M2l1Z6j2kJt5BpaWJgR2eAAECBAgQIECAAIFyFFBpKcdZNaaMCZx++kPRoxkzvok88cRtMtY73SFAgAABAuUp8NxzE2Jgp57618hcjWWppRYpz6HmdVQqLXnldDACBAgQIECAAAECBCpDQKWlMubZKIsqMH167trwd0YvNtlkuch99lm/qD3SOAECBAgQKGeB9977PIZ32GG3Rw4dunXkuut2K+cB53VsKi155XQwAgQIECBAgAABAgQqQ0ClpTLm2SgzIPDJJ19FLw455E+Rgwb9OLJ37xUy0C9dIECAAAEC5SMwdeqMGMzAgbdF7rjjmpE77bRW+QyvICNRaSkIs0YIECBAgAABAgQIECgvAbeMKK/5NJoMC3TqtGD07rTTtos85ph7I7t27RC54oqLRVoIECBAgACBxghUV1fH7r/7Xbpi51prLRmpxtIYz6zt2yJrHdIfAgQIECBAgAABAgQIzC7gnJbZNTwmUCCBRx55I1q67LKnIy+9dNfIRRdtV6C2NUOAAAECBMpR4Kqrno1hPffcxMjhw38W2apVy3IcaJOPyTktTU6sAQIECBAgQIAAAQIEyk9ApaX85tSISkbgyivTZ0LPP+8zoZKZMh0lQIAAgQwKPPbYf6JXF1/898jc9xc6dvT9hYZPlEpLw+3sSYAAAQIECBAgQIBAxQqotFTs1Bt48QVy1zkZOvSB6EqHDm0jjzlmi+J3Sw8IECBAgECJCIwf/3H09Oij74o899x0HstKK3Uukb5nt5sqLdmdGz0jQIAAAQIECBAgQCCzAiotmZ0aHasUgdy9ew877PYY8A47rB656649KmXwxkmAAAECBBok8NlnU2O/gw/+06zcOLJPn5UadCQ7zSmg0jKniJ8JECBAgAABAgQIECAwXwGVlvkS2YBAIQTef39KNDNw4G2RQ4ZsGbnBBssUomFtECBAgACBkhKoqpoZ/T366Lsj11mna+SAARuW1Aiy3tlcpaVmL3Pn4tZcX5g1LQrTjFYIECBAgAABAgQIECDQMAGVloa52YtAkwiMG/deHDd3PbGLLtopHi+99KJN0pKDEiBAgACB0hQYPvzx6PgCC7SKHDjwR6U5iEz32jktmZ4enSNAgAABAgQIECBAIJsCKi3ZnBe9qmiBP//51Rj/9dc/F3nppbtELrzwAhUtYvAECBAgQOC7vxkfeuj1wLjwwvR9hAUXbAMm7wIqLXkndUACBAgQIECAAAECBMpfQKWl/OfYCEtU4MILn4iev/32p5Fnnrl9ZMuWrpxRopOp2wQIECDQKIFcdeWyy56Oo1x8cfoOQufOCzXqiHaeu4BKy9xtPEOAAAECBAgQIECAAIG5CKi0zAXGagLFFvjmm+rowrBhD0S2a9c6Mnf/lrldPb3Y/dU+AQIECBDIv8Dzz0+Ig5544l8iR4zoG7n88p3y34wjziag0jIbhocECBAgQIAAAQIECBCom4BKS92cbEWgSALTp1dFy4MH5+77u1Q8PuigjYvUF80SIECAAIHCCfz3v59EY0ceeVfkySdvE9mzZ7fCNV/BLam0VPDkGzoBAgQIECBAgAABAg0VUGlpqJz9CBRQ4PPPp0Vrhx12e+Quu/SI3HnntQrYvqYIECBAgEDhBD766MtobODA2yIPPjh9v2CrrVYuXPMV35JKS8W/BAAQIECAAAECBAgQIFB/AZWW+pvZg0CRBN5/f0q0nKu3HHXUpvG4d+8VitQXzRIgQIAAgfwLfPXV9Djo4YffEbnNNqm6stde6+a/GUecp4BKyzx5PEmAAAECBAgQIECAAIHaBFrVttI6AgSyKLDkku2jW2edtX3kr351T+Sii7aL7NEjXVXMQoAAAQIESldg5sxvovNDh6a7k629dvp7TY2ldGezKXreoikO6pgECBAgQIAAAQIECBDIl4BzWvIl6TgECirwj3+8E+397ncPRY4YsWPksst2LGgPNEaAAAECBPIncPrp6W+0r76aEXnyyT+JbNnSZ+v5863PkZzTUh8t2xIgQIAAAQIECBAgQGCWgEqLFwKBEhZ48MHXovdXXvls5CWX7BLZqdOCJTweXSdAgACByhO46qr0t9izz6ZvEOS+O7DAAk66LubrQKWlmPraJkCAAAECBAgQIECgRAVUWkp04nSbwP8Err32n/HDo4+Oj7zggnR+y4ILtvnf0x4RIECAAIFMCtx77yvRr+uuS3+LXXxx+r5A7qqYmexsBXVKpaWCJttQCRAgQIAAAQIECBDIl4BKS74kHYdAkQXOO+/x6MHEiZ9HnnnmTyNbtWpZ5D5pngABAgQI1Cbw7LNvx+ozzng48qKLdo7s1q1DbRtaVwQBlZYioGuSAAECBAgQIECAAIFSF1BpKfUZ1H8C3wrk7iU8bFi6l/BCC6VzWoYM2YoOAQIECBDIlMDrr38U/fn1r++OPOOM7SPXWGOJTPVQZ1RavAYIECBAgAABAgQIECBQbwGVlnqT2YFAlgW+/roqujd48F2R663XLfKXv9woyx3WNwIECBCoEIEJEz6LkR555J2RRx3VO3LTTbtXyNhLa5i5SkvNPldXV9dcWbA1LQrWkoYIECBAgAABAgQIECDQAAGVlgag2YVA1gU++2xqdPHww++I3G23tSN33HHNrHda/wgQIECgTAXeey9d2fKII1KN5aCDUv1/m21WKdOxlsOwnNNSDrNoDAQIECBAgAABAgQIFFhApaXA4JojUDiBDz6YEo0NGpTqLfvtt0HkDjusXrjmtUSAAAECFS8wadIXYTBoUKqx9OvXM7JvX5X/rL8sVFqyPkP6R4AAAQIECBAgQIBABgVUWjI4KbpEIJ8Cs1+t5eCDN45D+yZxPn0diwABAgRqE5g8OZ1deeSRqdq/ww5rRO6xxzq1bWhd5gRUWjI3JTpEgAABAgQIECBAgED2BVRasj9HekggDwJvvz05jnLUUen+LUcema6Ov/nmK+ThuA5BgAABAgR+KDBlytex4qij0nksvXunv2v23z+dV2kpFQGVllKZKf0kQIAAAQIECBAgQCBDAq0y1BddIUCgyQSWXbZjHPucc/4v8le/ujuyTZuWkZtsslykhQABAgQINF7gq6+mx0GOOeaeyF69lolUY2m8qiPkBFqAIECAAAECBAgQIECAQJYFnNOS5dnRNwJNIvDvf38Yxz322D9HDhu2deT66y/dJC05KAECBAhUhsDXX1fFQHM1lu7dO8XjwYM3q4yhl+EondNShpNqSAQIECBAgAABAgQINLWASktTCzs+gYwKjBv3XvTshBMeiDz11G0j1157qYz2VbcIECBAIKsCM2bMjK4NGXJfZKdOC0Yed1yfyNyn9fHAUnICKi0lN2U6TIAAAQIECBAgQIBA8QVUWoo/B3pAoIgCzz03IVo/6aS/RJ5xxk8j11hjiSL2R9MECBAgUCoCM2d+E1098cQHI1u2TNd2yp0nmXtcKqPQz5oCKi01TawhQIAAAQIECBAgQIDAfARUWuYD5GkClSDwzDNvxzDPOOPhyN//Pt3LZaWVOlfCwI2RAAECBBog8M031bHX6ac/FPnFF+neLKedls6NbNUq3QHMUuoCKi2lPoP6T4AAAQIECBAgQIBAEQRUWoqArkkC2RR44ok3o2Pnnfd45Lnn/iwyd639bPZWrwgQIECgWALnnPNoND1x4ueRZ521fWSbNq0iLeUhoNJSHvNoFAQIECBAgAABAgQIFFRApaWg3BojkH2Bhx9+Izr5hz/8PXLEiL6RSy+9aPa7rYcECBAgUACBESP+Fq289tpHkblzINu1a12AdjVRSAGVlkJqa4sAAQIECBAgQIAAgTIRUGkpk4k0DAL5Fbj//lfjgFdeOSbyd7/bLnKVVbrktwlHI0CAAIESEjj55HRHr/HjP468+OKdIxdeeIES6r+u1l1ApaXuVrYkQIAAAQIECBAgQIDAtwIu9eClQIBALQLbbbdarK1OF+JvdvDBf4q89tq9Irt165BWWQgQIECgAgSqZ/01cMEFT8RY33prcuTo0XtEtmjRvAJGb4jZEmiRre7oDQECBAgQIECAAAECBH4o4JyWH3r4iQCBGgK564lddtnT8cx556X7t6i31ECyggABAmUlkLvnfe7OXW+++UmM7eyzd4hcaKE2ZTVOg6lNIHdOS81ncpW3musLs0alpTDOWiFAgAABAgQIECBAoIECKi0NhLMbgUoTuO++dD2xK654NjJ3bf7u3TtVGoLxEiBAoOwFZs78JsZ45pmPRE6a9EXkGWeke967H0sgVMji6mEVMtGGSYAAAQIECBAgQIBAPgVUWvKp6VgEyl7gkUfeiDHmriRz5pnps7dVV1287EdtgAQIEKgEgaqqmTHMU0/9a+TUqTMiTzst3aerTRsXmw2GClpUWiposg2VAAECBAgQIECAAIF8Cai05EvScQhUkMBTT70Vo8194/mUU34Sj9dZp2sFjd9QCRAgUF4C06enGsuJJz4Q2bJlukrTiSduE9m6dctIS6UJqLRU2owbLwECBAgQIECAAAECeRBQackDokMQqEyB556bEAM/+eS/RA4ZsmXkhhsuW5kURk2AAIESFZg2LZ278tvf3h+5yCJtI084YavIXL0lHlgqUEClpQIn3ZAJECBAgAABAgQIEGisgEpLYwXtT6DCBV5++YMQGDLkvsijj94scrPNVqhwE8MnQIBA9gW++mp6dPK44/4c2bXrIpHHHtsnskWL5pGWShZQaank2Td2AgQIECBAgAABAgQaKKDS0kA4uxEgMLvA+PEfx4/HHHNP5MEHbxy57barzr6BxwQIECCQEYEvvvg6enLssfdGrrRS58jBg3tH5j5fjweWChdQaanwF4DhEyBAgAABAgQIECDQEAGVloao2YcAgVoF3n7701j/61/fHdmv37qRO+20Vq1bWkmAAAEChRfI1ViOPPLOaHq99ZaOPOywHxW+G1rMuIBKS8YnSPcIECBAgAABAgQIEMiigEpLFmdFnwiUtMD7738e/T/66FRv+dnP1ojca69UdbEQIECAQLEEJk36IprOncfSu3f3eNy//4bF6ox2My6g0pLxCdI9AgQIECBAgAABAgSyKKDSksVZ0ScCZSDw8cdfxigGD071ls03T3duGTDAp3plMLGGQIBAiQn85z/p6o6/+U26H8uuu/aI3HPPniU2Bt0trIBKS2G9tUaAAAECBAgQIECAQFkIqLSUxTQaBIGsCnz22bToWu56Yj17dovHhx66SaQ7Lmd1xvSLAIHyERg79t0YzCmn/DXyqKM2jezTZ6XyGZ6RNJmASkuT0TowAQIECBAgQIAAAQLlK6DSUr5za2QEMiPw5ZfToy8nnHB/5EILtZn1eKvItm1bR1oIECBAIL8CDz74Whzw4ov/HnnqqdtG9uixVH6bcLQyFlBpKePJNTQCBAgQIECAAAECBJpKQKWlqWQdlwCBOQSqqmbGmnPOeSzyv//9JPKMM7aP7NRpwUgLAQIECDRe4JprxsZB7r33lcizz/6/yGWXXbTxh3WEihJQaamo6TZYAgQIECBAgAABAgTyI6DSkh9HRyFAoF4Cuc8C77775djrrLN2iOzevVO9jmBjAgQIEMgJzJz5TTw477zHI19//aPIM89Ux87ZyIYIqLQ0RM0+BAgQIECAAAECBAhUuIBKS4W/AAyfQDEFHn74jWh+xIgnIocN2zpy/fWXLmaHtE2AAIGSEpg6dUb096STHvy+1yed9JN43K6dazN+T+JBvQVylZaau1VXV9dcWbA1LQrWkoYIECBAgAABAgQIECDQAAGVlgag2YUAgXwKjBv3Xhxu2LAHIg88cOPI7bdfLZ8NOBYBAgTKTuCTT76KMR133J8jV1mlS+Tgwb0jW7b0eXQwWBol4JyWRvHZmQABAgQIECBAgACByhRQaanMeTdqApkTePfdT6NPxx57b2SfPitF/vKXG0bO7Zu18ZSFAAECFSjw1luTY9S/+U3603KHHVaP3Gef9SvQwZCbTkClpelsHZkAAQIECBAgQIAAgbIVUGkp26k1MAKlKPDZZ9Oi27/97X2Riy++cORxx20Z2aZNy0gLAQIEKlngpZfej+H/9rf3Rx522I8it9lmlUoGMfYmElBpaSJYhyVAgAABAgQIECBAoJwFVFrKeXaNjUCJCkyfPjN6fuaZD0e+//6UyDPO+Glkhw7tIi0ECBCoNIF7730lhnzFFc9EDhu2TeS663arNATjLZiASkvBqDVEgAABAgQIECBAgED5CKi0lM9cGgmBMhPI3Xn3qqvGxLgeeuj1yHPO+b/Ibt06lNlIDYcAAQI1BWbMSDXnCy54IvKFF9L9rE49ddvI5ZbrGGkh0HQCKi1NZ+vIBAgQIECAAAECBAiUrYBKS9lOrYERKCeB++57NYYzcuTTkb/61eaRm27avZwGaCwECBD4XmDSpC/i8bBhD0Z26bJQ5PHHp+sotmvXOtJCoKkFVFqaWtjxCRAgQIAAAQIECBAoQwGVljKcVEMiUK4C//73pBja0KHpHgVbbrlS5IEHbhTZsmWLSAsBAgRKXeCFFybGEE4++S+Ru+22dmS/fuuW+qD0v+QEVFpKbsp0mAABAgQIECBAgACB4guotBR/DvSAAIF6CXz++bTY/tRT/xqZu6PLiSemuxZ06rRgvY5jYwIECGRH4NZbX4zOXHfdPyNPOGHryPXXXzo73dOTihJQaamo6TZYAgQIECBAgAABAgTyI6DSkh9HRyFAoMACubu4/PGP/4h277473Sv6pJNSvaVHj6UK3BPNESBAoGEC06bNiB3PPffxyDff/CTytNO2i1xyyfaRFgJ1Epg2JTabPKNdZMf2rea7y7QpaftW7dNrbG5bq7TMl9EGBAgQIECAAAECBAgQmFNApWVOET8TIFByAs8++070+fTTH4rce+/1InNX3Sm5gegwAQIVIjBx4ucx0ty1EFdccbF4/OtfpztQtWkzt8++KwTGMOshMGXc6Nh6n4vSa2n3NV+KHNN5cOTwfqtF/nBJV6W7bJ+fRh5ybTp7Kv62jP8e/3BUZO8uc77qVFqSkIUAAQIECBAgQIAAAQL1ElBpqReXjQkQyK7A+++n7+kOG/ZAZNeu/9/evcBWWd5xHKd3ikChtEApQotQkDs4tslNucyAU5wZONdtYZdAtmQDWbKYuWxL3OayuUyzWyRmy5wTmOhEvOFEYMjEsSAgAootN7kUCq2l1dLSy37//mXAuIRDe07f857vk/jj7Xsu7/N+3gOm7/88z9NVee+9U5SsIS0EGgIIBERg06YD6skDD6xRzp37CeWddw4PSN/oRtwINLyjrn4l7XrlnGPNylm5VktZlJSvnHLIxkrN6nO2frJ/5S+1Z23fecqvjrXRLKsWpSnvLdys3LbAvqFwbqPScq4G2wgggAACCCCAAAIIIIDAFQmc/SXsip7OkxBAAIGgCvh8O7/73Z3q4G9+s0E5f/5TSp+Np3//7kHtOP1CAIGEEHj8cburvWKFjT3wf5eGD++dEGfOSba5QG253vJQh/uVN+b6u1v9ZMhI2972vn3vYFafs//X6zVlgfZ8tUtHe7il5RfaNxFG5mR9/HM8/JEcD52kjwgggAACCCCAAAIIIJC4AlRaEvfac+YIhFIgPT1F5+Xz8Kxa9a62FyxYoVy4cJJy6tSBShoCCCAQG4Gqqlod6Fe/spVYKis/Uj766GxldnYnJQ2BSwmcqrRaSkWtjU65sNXtfk0713awOsmZ6olVWkbPtvrJP/Tf+a3jOTWWDh1s9MuKFYOUi5697vwnBvonKi2Bvjx0DgEEEEAAAQQQQAABBKi08BlAAIHQCsyYMVjnNnCgrYHgs4q9+67du5o375PK1FSrydAQQACBaAj4+lG/+MVavfn06XZX+8c/nq7kX55oaIfvPQ/9e6lO6pcvlyl72GL3Z1rtCW11G2ffHZjSYa/yVMsjXTrYOJZ/P2Wftw63tOy6SDRo36r77lVe9+ijypaJxC7yPN/lc4hd+HBzs81XFvtGpSX25hwRAQQQQAABBBBAAAEEIhBgnZYIsHgqAgjEr0BNTZ06//Of29oIR4/WKH/wg2nKwsJsJQ0BBBBovUBdnd3JfuSRjcoNG/Yp/d+Z0aP7aJuGQNsJ7NdbLUoqUE75eJ2WSm3/JOlm5bhjNk/djFz7NB4+bKOq+rTMJLb9z4u0vaL3PcofzuivLF2zUnl63CzlEBsU83FjnZYzEvyJAAIIIIAAAggggAACCFyxAJWWK6biiQggEBaBF1+0tYQXL7a7ocXFY5Rz5oxSJicnKWkIIIBApAIlJcf1kvvvX60sKspR3nOPjTro3DlDSUMgGgKVb/5Zb/u1h44p7xpRqvxPX6ulPFQ8RNmwf4kyreBJ5b82TFVOmLhQeV6bv1w/1i62Ge3OzEJmj1NpMQUaAggggAACCCCAAAIIIBCRAJWWiLh4MgIIhEegrMzmWvFRLj4Xyve/b/ei8vK6huckORMEEIiaQFOTzaH0t79tVS5btk25YMEE5bRpNlcYDYEYCTTY/8sqq9OU3bufWy+5+uNTabl6O16JAAIIIIAAAggggAACCStApSVhLz0njgACJuA1luXL39L2E09sUc6f/ynlZz97vZKGAAIIXChw9Kjd237gAZuN0Nt991mdtlevcyZgOvMQfyIQdwJUWuLuktFhBBBAAAEEEEAAAQQQaH8BKi3tfw3oAQIIBERg374K9eRnP3tVmZNzjfJ737tZmZ3dSUlDAAEEVq9+Twi//e2/lHffbbMOfuELo5XMPSgEWmgEqLSE5lJyIggggAACCCCAAAIIIBA7ASotsbPmSAggEBcCDQ2N6udjj9mKws8/v1O5aNFk5eTJA5Q0BBBINIHa2tM65QcfXKcsKTmh/NGPpisHDrT1WGgIhE+ASkv4rilnhAACCCCAAAIIIIAAAlEXoNISdWIOgAAC8Suwa9dRdf6nP7VRLkOH9lJ+97tWdcnMtBnxaQggEG6BdetslfHf//515aRJhcpvfvPTyvT0VCUNgbAKUGkJ65XlvBBAAAEEEEAAAQQQQCCKAlRaoojLWyOAQDgE6uoadCKPPPKGcuPGfcpvf9vWvZ440e680hBAIEwChw+f1Ok8/PB6ZXn5h0qvr44YkRem0+RcELiMAJWWy+DwEAIIIIAAAggggAACCCBwcQEqLRd3YS8CCCBwUYEtWw5p/8MPv6b01a8XLpyo7fz8rIs+n50IIBB8gdOnbc7AZcu2Kpcvf0v5pS+NUc6ePVKZkpKspCGQOAJUWhLnWnOmCCCAAAIIIIAAAggg0GYCVFrajJI3QgCBxBFobGzSyT711HblkiVvKmfNGqb88pfHKjMymFlIDDQE4kBg61arnf761zaCpV+/7soFC6x22rNnZyUNgcQUoNKSmNeds0YAAQQQQAABBBBAAIFWCVBpaRUfL0YAAQSOH7f5hf7wB1vJYccOW9flO99hbjE+FwgEV6Cyslad87+z27Yd1vbChZOUEyYUKGkIIEClhc8AAggggAACCCCAAAIIIBCxAJWWiMl4AQIIIHApAf9+/EMP2dxivXt3Ufr345lb7FJi7EcgNgLNzc060HPP7VT+6U//Uc6cOUQ5d+4Nyo4d05Q0BBBwASotfBIQQAABBBBAAAEEEEAAgYgFqLRETMYLEEAAgcsLnJlbzFZ7eOKJLco77mBuscub8SgC0RLYu7dCb/3gg+uUvuKKr3BfWJgdrUPyvgjEuQCVlji/gHQfAQQQQAABBBBAAAEE2kOASkt7qHNMBBBIGAHmFkuYS82JBkjgxAmb0++xxzYrN2zYq/zGNz6pvPVWG8fid5G1QUMAgYsKXOrviI8Nu+hLYrAzOQbH4BAIIIAAAggggAACCCCAwFULUGm5ajpeiAACCEQmcGbtbZtbLD+/q/Jb3xqv7NevW2RvxLMRQOACgerqOu1butRGkfksYbfdNlTbxcVjlF26ZChpCCBwJQKMabkSJZ6DAAIIIIAAAggggAACCJwnQKXlPA5+QAABBKIt4HOL/f3v23Wgv/71TeXEiYVK/859dnanaHeA90cgNAKnTp3WuTz9tP1tevLJbcrJkwco5879hDIn5xolDQEEIhWg0hKpGM9HAAEEEEAAAQQQQAABBDpQaeFDgAACCLSbQE2NfQt/yRL7Fv4LL+xSzpgxWFlcPFaZldVRSUMAgXMFGhoa9eMLL7yj/MtfbH6wUaPylF6rzM/P0jYNAQRaI0ClpTV6vBYBBBBAAAEEEEAAAQQSVIBKS4JeeE4bAQSCJlBR8ZG69Pjjdud49er3lJ///IiWHKlk7iMh0BJWwFeHWLOmRAJ//OMmZd++VlGZN+/TykGDcpQ0BBBoKwEqLW0lyfsggAACCCCAAAIIIIBAAglQaUmgi82pIoBAvAiUlVWrqz632Lp1pdqeNctWnLjrrlHKbt0ylTQEEkFg06b3dZqLF29UZmSkKufPt+rK6NF9lDQEEIiGAJWWaKjynggggAACCCCAAAIIIBByASotIb/AnB4CCMS7QHl5jU7BZxh75RUb6zJz5hDl3Xdb1aVHD1aiEAMtJAK+itE//7lH5+Of+aamZm1//evjlL6iUUhOldNAIMACVFoCfHHoGgIIIIAAAggggAACCARVgEpLUK8M/UIAAQQuEPAZxpYu3apHXnrJ1qn4zGcGKefMsapLnz5dlTQE4kugvt7WXVm1yj7Py5bZZ9tXsv/iF8do+8Yb+ytpCCAQSwEqLbHU5lgIIIAAAggggAACCCAQEgEqLSG5kJwGAggkmkBVVa1O+Zln3lY+++wO5ciRNp/S7Nm2usuIEbZGOA2BYAr4p3flyp3qnn96i4pytV1cbNWV4cN7B7Pb9AqBBBGg0pIgF5rTRAABBBBAAAEEEEAAgbYUoNLSlpq8FwIIINAuAnV1DTruqlXvKpcvf0uZmWkrWsyePVI5bdpAZWpqipKGQHsJlJae0KFXrLDa4Nq1tvrQTTcNUM6ZY5/SgoJsJQ0BBIIgQKUlCFeBPiCAAAIIIIAAAggggECcCVBpibMLRncRQACByws0N9u6Fr6O+NNPb9f2O+8cU956q63ucscdw5R5ecwzJgZaFAXq6636t379XuXKlTbm6siRauXnPmefwNtuu16ZlZWppCGAQNAEqLQE7YrQHwQQQAABBBBAAAEEEIgDASotcXCR6CICCCDQGoHDh0/q5X63++WXbdyLjx+4/fah2p40qVCZlsaIFzHQWiXgNb0XX7QVV9asKVEOG9ZL6Z80X3ElJSW5VcfgxQggEH0BKi3RN+YICCCAAAIIIIAAAgggEDoBKi2hu6ScEAIIIHBpgcbGJj342ms20sDviPvd8alTbYaxW24pUg4danfHaQhcXqCq6pSe4LW7l16y6orPYjdzpo2emjFjsDI3t7OShgAC8SVApSW+rhe9RQABBBBAAAEEEEAAgUAIUGkJxGWgEwgggEB7CZSX1+jQr7zyntLvmtfVNWrbV3eZPn2QtgsLWUNDDAndamtP6/zXr9+jfPVVG6+yc+dR5cSJBUqvrowa1UfbNAQQiHcBKi3xfgXpPwIIIIAAAggggAACCLSDAJWWdkDnkAgggECQBUpKjqt7fjd97Vq7p96xY5ry5ptt/fJJkyyvu66HkhZWgZqaOp3axo37lb7WypYth7Q9alSecto0q7+NH99f6Z8NbdAQQCA0Al5pufB0fB2wC/fHZg8zD8bGmaMggAACCCCAAAIIIIDAVQpQablKOF6GAAIIJI7Arl02esHvuPvMY77e+fjxBdo/YYLlmDE2niE1lfVexBBn7eDBD9TjN944oNywYZ/Sq21jx+Zr21fy8avcqVO69tAQQCDcAoxpCff15ewQQAABBBBAAAEEEEAgKgJUWqLCypsigAAC4RbYv79SJ/j66/uUfod+9+5ybfv8UePGXattr70MGMDoF2EEolVX20gVH53i6dfu9GmbL87rZr5u/Q039NWe9HTqZoG4cHQCgRgLUGmJMTiHQwABBBBAAAEEEEAAgTAIUGkJw1XkHBBAAIF2F/D5pjZvPqiebN5sM035tu/3O/dehxkxorceLSiwtV+Sk5OUtLYVqKys1Rvu2FGm3L79iHLbNssDB2zsivv7FfGaGOvwiIWGAAL/E6DS8j8KNhBAAAEEEEAAAQQQQACBKxWg0nKlUjwPAQQQQOAqBMrLa/SqrVsPK/1+/9tvWwWgrKxaOWRIrnLYMKu9FBXZdlFRjjIvr6uSdikBX5++tPSEnuBzu+3ebavruK2PXRk6tKf2eHXL07WZ4e1SquxHAAEXoNLCJwEBBBBAAAEEEEAAAQQQiFiASkvEZLwAAQQQQKD1Aj7WZdeuY3qrM2mrwZSUWPXAH/Xai4+46N+/m/b369ddee21tp2Tc40yfKNivIpy8GCVzm7fvgrl++/bWJQ9e2x7zx7z8VErLjN4sFWoBg+2uopXV9zH75VqJw0BBBCISIBKS0RcPBkBBBBAAAEEEEAAAQQQMAEqLXwOEEAAAQQCJ+CjMkpLbZzG3r22JsyBA55Wc/DKg1cbevXqoj19+2Ype/bsrPQKjGePHp20p1u3ji2ZqezSJUMZ7ZXdGxubdJSTJ21dlJMnTykrKz9SVlTYvF4VFbZdXv5hS9qYHx/hc+TISW2fOtWg9DPymonXl7yuMmCAzbrmj1JLEQUNAQTaXIBKS5uT8oYIIIAAAggggAACCCAQfgEqLeG/xpwhAgggEEqB+npbx72szKoTXqnw9ArG8eNWx/jgg7OVjaoqq3tUV1vdo67OXtupU5oyM9PSV3/PyEjVttdJUlOTtZ2SYtncrLA/9V9Tk2Vysu2vr7eqiPfEx+H4O/s7eFUnK+tsnad7d6v29Ohho3G8FpSba9u9e1u9yOdM8+foRxoCCCDQLgJUWtqFnYMigAACCCCAAAIIIIBAfAtQaYnv60fvEUAAAQSuQqC5pXTy4Yf1eq3P1uUjSbxmUldn9ROvljQ2Wl0lKckO4ncfU1Lsh7S0FKVXZrxK4xUbr96kp1vFhoYAAgjEowCVlni8avQZAQQQQAABBBBAAAEE2lmASks7XwAOjwACCCCAAAIIIIBAcASotATnWtATBBBAAAEEEEAAAQQQiBsBm/yEhgACCCCAAAIIIIAAAggEVoBfWgJ7aegYAggggAACCCCAAAIImAC/tPA5QAABBBBAAAEEEEAAgUAL8EtLoC8PnUMAAQQQQAABBBBAAAF+aeEzgAACCCCAAAIIIIAAAoEW4JeWQF8eOocAAggggAACCCCAAAL80sJnAAEEEEAAAQQQQAABBAItwC8tgb48dA4BBBBAAAEEEEAAAQRSIUAAAQQQQAABBBBAAAEEzhVISko690dtNzc3/9+eWP5IpSWW2hwLAQQQQAABBBBAAAEEIhb4L95Q9ENbPk2jAAAAAElFTkSuQmCC\")
\n",
+ "\n",
+ "\n",
+ "This function has a minimum at $x = -\\frac{3}{2}$."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- A demo function with minimum at -3/2\n",
+ "function x = x^2 + 3 * x"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Finally, let's take the minimum. We're going to use a step size of $\\alpha = 0.1$, start at $x_0 = 12$, and stop with a tolerance of $1\\times 10^{-4}$:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
{
- "cell_type": "markdown",
+ "data": {
+ "text/plain": [
+ "-1.4892542376755242"
+ ]
+ },
"metadata": {},
- "source": [
- "In supervised learning algorithms, we generally have some model (such as a neural network) with a set of parameters (the weights), a data set, and an error function which measures how well our parameters and model fit the data. With many models, the way to train the model and fit the parameters is through an iterative minimization procedure which uses the gradient of the error to find the local minimum in parameter space. This notebook will not go into the details of neural networks or how to compute the derivatives of error functions, but instead focuses on some of the simple minimization methods one can employ. The goal of this notebook is to develop a simple yet flexible framework in Haskell in which we can develop gradient descent methods.\n",
- "\n",
- "Although you should keep in mind that the goal of these algorithms (for our purposes) is to train neural networks, for now we will just discuss some abstract function $f(\\vec x)$ for which we can compute all partial derivatives.\n",
- "$\\newcommand\\vector[1]{\\langle #1 \\rangle}\\newcommand\\p[2]{\\frac{\\partial #1}{\\partial #2}}$\n",
- "\n",
- "Gradient Descent\n",
- "---\n",
- "The simplest algorithm for iterative minimization of differentiable functions is known as just **gradient descent**.\n",
- "Recall that the gradient of a function is defined as the vector of partial derivatives:\n",
- "\n",
- "$$\\nabla f(x) = \\vector{\\p{f}{x_1}, \\p{f}{x_2}, \\ldots, \\p{f}{x_n}}$$\n",
- "\n",
- "and that the gradient of a function always points towards the direction of maximal increase at that point.\n",
- "\n",
- "Equivalently, it points *away* from the direction of maximum decrease - thus, if we start at any point, and keep moving in the direction of the negative gradient, we will eventually reach a local minimum.\n",
- "\n",
- "This simple insight leads to the Gradient Descent algorithm. Outlined algorithmically, it looks like this:\n",
- "\n",
- "1. Pick a point $x_0$ as your initial guess.\n",
- "2. Compute the gradient at your current guess:\n",
- "$v_i = \\nabla f(x_i)$\n",
- "3. Move by $\\alpha$ (your step size) in the direction of that gradient:\n",
- "$x_{i+1} = x_i + \\alpha v_i$\n",
- "4. Repeat steps 1-3 until your function is close enough to zero (until $f(x_i) < \\varepsilon$ for some small tolerance $\\varepsilon$)\n",
- "\n",
- "Note that the step size, $\\alpha$, is simply a parameter of the algorithm and has to be fixed in advance. \n",
- "\n",
- "Though this algorithm is simple, it will be a bit of a challenge to formalize it into executable Haskell code that we can later extend to other algorithms. First, note that gradient descent requires two things:\n",
- "\n",
- "- Something to optimize (a function)\n",
- "- What to optimize over (the parameters)\n",
- "- A way to compute partials of the function\n",
- "\n",
- "Note that we don't actually need to *call* the function itself - only the partial derivatives are necessary.\n",
- "\n",
- "We're going to define a single class for things on which we can run gradient descent. Although later we may want to modify this class, this serves as a beginning:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- ":set -XTypeFamilies\n",
- "class GradientDescent a where\n",
- " -- Type to represent the parameter space.\n",
- " data Params a :: *\n",
- " \n",
- " -- Compute the gradient at a location in parameter space.\n",
- " grad :: a -> Params a -> Params a\n",
- " \n",
- " -- Move in parameter space.\n",
- " paramMove :: Double -- Scaling factor.\n",
- " -> Params a -- Direction vector.\n",
- " -> Params a -- Original location.\n",
- " -> Params a -- New location."
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 1
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In order to use some type `a` with our gradient descent, we require that it is an instance of `GradientDescent`. This class requires a few things.\n",
- "\n",
- "First off, we use type families in order to define a representation for the parameter space. We want to be able to operate on points in the parameter space of our function; however, while something like a list of values might be nice and simple in one case, it is inappropriate and inefficient when storing the weights of a neural network. Thus, we let each class instance decide how to store its parameters by defining an associated type instance. (We will see an instance of this later!)\n",
- "\n",
- "Next, `GradientDescent` requires a single function called `grad`, which takes the thing of type `a` and the current point in parameter space (via a `Param a`) and outputs a set of partial derivatives. The partial derivatives have the same form and dimension as the point in parameter space, so they are also a `Param a`. \n",
- "\n",
- "Finally, we must be able to move around in parameter space, so `GradientDescent` defines a function `paramMove` which does exactly that - it takes a parameter vector and an amount by which to move and uses these to generate a new position from an old one.\n",
- "\n",
- "Let's go ahead and create the simplest instantiation of this class and type family: a single-argument function. Note that this is just for demo purposes, and we're going to use numerical differentiation to compute the derivative."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- We need flexible instances for declarations like these.\n",
- ":set -XFlexibleInstances\n",
- "\n",
- "instance Floating a => GradientDescent (a -> a) where\n",
- " -- The parameter for a function is just its argument.\n",
- " data Params (a -> a) = Arg { unArg :: a }\n",
- "\n",
- " -- Use numeric differentiation for taking the gradient.\n",
- " grad f (Arg value) = Arg $ (f value - f (value - epsilon)) / epsilon\n",
- " where epsilon = 0.0001\n",
- " \n",
- " paramMove scale (Arg vec) (Arg old) = Arg $ old + fromRational (toRational scale) * vec"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 2
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We're getting closer to implementing the actual algorithm. However, we have yet to define when the algorithm *stops* - and, in order to give maximum flexibility to the user, we'll let the stopping condition be an argument. This lets the user specify an error tolerance, as well as how they want to derive this error:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Define a way to decide when to stop.\n",
- "-- This lets the user specify an error tolerance easily.\n",
- "-- The function takes the previous two sets of parameters and returns\n",
- "-- `True` to continue the descent and `False` to stop.\n",
- "newtype StopCondition a = StopWhen (Params a -> Params a -> Bool)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 3
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "With a single instance of our class, we can now implement our gradient descent algorithm:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "gradientDescent :: GradientDescent a => a -- What to optimize.\n",
- " -> StopCondition a -- When to stop.\n",
- " -> Double -- Step size (alpha).\n",
- " -> Params a -- Initial point (x0).\n",
- " -> Params a -- Return: Location of minimum.\n",
- "gradientDescent function (StopWhen stop) alpha x0 =\n",
- " let iterations = iterate takeStep x0\n",
- " iterationPairs = zip iterations $ tail iterations\n",
- " in\n",
- " -- Drop all elements where the resulting parameters (and previous parameters)\n",
- " -- do not meet the stop condition. Then, return just the last parameter set.\n",
- " snd . head $ dropWhile (not . uncurry stop) iterationPairs\n",
- " where\n",
- " -- For each step...\n",
- " takeStep params = \n",
- " -- Compute the gradient.\n",
- " let gradients = grad function params in\n",
- " -- And move against the gradient with a step size alpha.\n",
- " paramMove (-alpha) gradients params"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 4
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's go ahead and try this for some simple functions. First, we need to create a stopping condition. In this case, we're going to stop when successive updates to the parameters do not affect the outcome very much - namely, when the difference between the function value at successive parameters is below some $\\varepsilon$-tolerance. In other scenarios, we may want to use a more complicated stopping criterion."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Create a stop condition that respects a given error tolerance.\n",
- "stopCondition :: (Double -> Double) -> Double -> StopCondition (Double -> Double)\n",
- "stopCondition f tolerance = StopWhen stop\n",
- " where stop (Arg prev) (Arg cur) =\n",
- " abs (f prev - f cur) < tolerance "
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 5
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's finally try to minimize something, such as the relatively trivial function $f(x) = x^2 + 3x$. It's graph looks like this:\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "\n",
- "This function has a minimum at $x = -\\frac{3}{2}$."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- A demo function with minimum at -3/2\n",
- "function x = x^2 + 3 * x"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 6
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Finally, let's take the minimum. We're going to use a step size of $\\alpha = 0.1$, start at $x_0 = 12$, and stop with a tolerance of $1\\times 10^{-4}$:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "let alpha = 1e-1\n",
- "let tolerance = 1e-4\n",
- "let initValue = 12.0\n",
- "unArg $ gradientDescent function (stopCondition function tolerance) alpha (Arg initValue)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "-1.4892542376755242"
- ]
- }
- ],
- "prompt_number": 7
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Monadic Gradient Descent\n",
- "---\n",
- "\n",
- "Although the above implementation of gradient descent works, we're going to run into problems when our functions are more complicated. For instance, suppose that computing the gradient required a lot of computation, and the computation required communicating with a distributed network of processing nodes. Or suppose that there were some regimes in which the function was non-differentiable, and we wanted to use the `Maybe` type to represent this. In order to support this, we can try to rewrite our class with *monadic* variants of its operations."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- ":set -XMultiParamTypeClasses\n",
- "class Monad m => GradientDescent m a where\n",
- " -- Type to represent the parameter space.\n",
- " data Params a :: *\n",
- " \n",
- " -- Compute the gradient at a location in parameter space.\n",
- " grad :: a -> Params a -> m (Params a)\n",
- " \n",
- " -- Move in parameter space.\n",
- " paramMove :: Double -- Scaling factor.\n",
- " -> Params a -- Direction vector.\n",
- " -> Params a -- Original location.\n",
- " -> m (Params a) -- New location.\n",
- " \n",
- " \n",
- "-- Since we've redefined GradientDescent, we need to redefine StopCondition.\n",
- "newtype StopCondition a = StopWhen (Params a -> Params a -> Bool)"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 8
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In order to utilize this, we're going to have to rewrite our instance to run all computations in a monad. The implementation will look quite familiar, but we won't be able to use as many built-in functions, as they do not have monadic variants in the base packages."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "gradientDescent :: (GradientDescent m a) => \n",
- " a -- What to optimize.\n",
- " -> StopCondition a -- When to stop.\n",
- " -> Double -- Step size (alpha).\n",
- " -> Params a -- Initial point (x0).\n",
- " -> m (Params a) -- Return: Location of minimum.\n",
- "gradientDescent function (StopWhen stop) alpha x0 = do\n",
- " -- Take the next step.\n",
- " next <- takeStep x0\n",
- " \n",
- " -- If we stop, do so, otherwise recurse.\n",
- " if stop x0 next\n",
- " then return next\n",
- " else gradientDescent function (StopWhen stop) alpha next\n",
- " where\n",
- " takeStep params = do\n",
- " gradients <- grad function params\n",
- " paramMove (-alpha) gradients params"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 9
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's try this for something simple. Suppose we're using our old $f(x) = x^2 + 3x$, but for some reason, we are incapable of differentiating if the function value is below zero. We'll use the `Maybe` monad to represent this - if the parameter to a function is negative, we return `Nothing`, otherwise, we return `Just` the derivative."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "instance (Ord a, Floating a) => GradientDescent Maybe (a -> a) where\n",
- " -- The parameter for a function is just its argument.\n",
- " data Params (a -> a) = Arg { unArg :: a }\n",
- "\n",
- " -- Use numeric differentiation for taking the gradient.\n",
- " grad f (Arg value) = \n",
- " if value > 0\n",
- " then Just $ Arg $ (f value - f (value - epsilon)) / epsilon\n",
- " else Nothing\n",
- " where epsilon = 0.0001\n",
- " \n",
- " paramMove scale (Arg vec) (Arg old) = Just $ Arg $ old + fromRational (toRational scale) * vec"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 10
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's go ahead and try this with the same example as before."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "stopCondition f tolerance = StopWhen stop\n",
- " where stop (Arg prev) (Arg cur) =\n",
- " abs (f prev - f cur) < tolerance \n",
- " \n",
- "let x0 = Arg initValue\n",
- "let stopper = stopCondition function tolerance\n",
- "case gradientDescent function stopper alpha x0 of\n",
- " Just x -> print $ unArg x\n",
- " Nothing -> putStrLn \"Nothing!\""
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "Nothing!"
- ]
- }
- ],
- "prompt_number": 11
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We saw in the original example that the minimum is at $-\\frac{3}{2}$, so this gradient descent tries to go into the $x < 0$ region - at which point the differentiation returns `Nothing`, and the gradient descent implicitly stops! This monadic gradient descent can be used to implement things such as bounded optimization, optimization that keeps track of the states it went through, optimization that uses networked IO to do its computation, and so on.\n",
- "\n",
- "That's it for now. In the next notebook, I'm going to try implementing conjugate gradient in this same framework."
- ]
+ "output_type": "display_data"
}
],
- "metadata": {}
+ "source": [
+ "let alpha = 1e-1\n",
+ "let tolerance = 1e-4\n",
+ "let initValue = 12.0\n",
+ "unArg $ gradientDescent function (stopCondition function tolerance) alpha (Arg initValue)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Monadic Gradient Descent\n",
+ "---\n",
+ "\n",
+ "Although the above implementation of gradient descent works, we're going to run into problems when our functions are more complicated. For instance, suppose that computing the gradient required a lot of computation, and the computation required communicating with a distributed network of processing nodes. Or suppose that there were some regimes in which the function was non-differentiable, and we wanted to use the `Maybe` type to represent this. In order to support this, we can try to rewrite our class with *monadic* variants of its operations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ ":set -XMultiParamTypeClasses\n",
+ "class Monad m => GradientDescent m a where\n",
+ " -- Type to represent the parameter space.\n",
+ " data Params a :: *\n",
+ " \n",
+ " -- Compute the gradient at a location in parameter space.\n",
+ " grad :: a -> Params a -> m (Params a)\n",
+ " \n",
+ " -- Move in parameter space.\n",
+ " paramMove :: Double -- Scaling factor.\n",
+ " -> Params a -- Direction vector.\n",
+ " -> Params a -- Original location.\n",
+ " -> m (Params a) -- New location.\n",
+ " \n",
+ " \n",
+ "-- Since we've redefined GradientDescent, we need to redefine StopCondition.\n",
+ "newtype StopCondition a = StopWhen (Params a -> Params a -> Bool)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In order to utilize this, we're going to have to rewrite our instance to run all computations in a monad. The implementation will look quite familiar, but we won't be able to use as many built-in functions, as they do not have monadic variants in the base packages."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "gradientDescent :: (GradientDescent m a) => \n",
+ " a -- What to optimize.\n",
+ " -> StopCondition a -- When to stop.\n",
+ " -> Double -- Step size (alpha).\n",
+ " -> Params a -- Initial point (x0).\n",
+ " -> m (Params a) -- Return: Location of minimum.\n",
+ "gradientDescent function (StopWhen stop) alpha x0 = do\n",
+ " -- Take the next step.\n",
+ " next <- takeStep x0\n",
+ " \n",
+ " -- If we stop, do so, otherwise recurse.\n",
+ " if stop x0 next\n",
+ " then return next\n",
+ " else gradientDescent function (StopWhen stop) alpha next\n",
+ " where\n",
+ " takeStep params = do\n",
+ " gradients <- grad function params\n",
+ " paramMove (-alpha) gradients params"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's try this for something simple. Suppose we're using our old $f(x) = x^2 + 3x$, but for some reason, we are incapable of differentiating if the function value is below zero. We'll use the `Maybe` monad to represent this - if the parameter to a function is negative, we return `Nothing`, otherwise, we return `Just` the derivative."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "instance (Ord a, Floating a) => GradientDescent Maybe (a -> a) where\n",
+ " -- The parameter for a function is just its argument.\n",
+ " data Params (a -> a) = Arg { unArg :: a }\n",
+ "\n",
+ " -- Use numeric differentiation for taking the gradient.\n",
+ " grad f (Arg value) = \n",
+ " if value > 0\n",
+ " then Just $ Arg $ (f value - f (value - epsilon)) / epsilon\n",
+ " else Nothing\n",
+ " where epsilon = 0.0001\n",
+ " \n",
+ " paramMove scale (Arg vec) (Arg old) = Just $ Arg $ old + fromRational (toRational scale) * vec"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's go ahead and try this with the same example as before."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Nothing!"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "stopCondition f tolerance = StopWhen stop\n",
+ " where stop (Arg prev) (Arg cur) =\n",
+ " abs (f prev - f cur) < tolerance \n",
+ " \n",
+ "let x0 = Arg initValue\n",
+ "let stopper = stopCondition function tolerance\n",
+ "case gradientDescent function stopper alpha x0 of\n",
+ " Just x -> print $ unArg x\n",
+ " Nothing -> putStrLn \"Nothing!\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We saw in the original example that the minimum is at $-\\frac{3}{2}$, so this gradient descent tries to go into the $x < 0$ region - at which point the differentiation returns `Nothing`, and the gradient descent implicitly stops! This monadic gradient descent can be used to implement things such as bounded optimization, optimization that keeps track of the states it went through, optimization that uses networked IO to do its computation, and so on.\n",
+ "\n",
+ "That's it for now. In the next notebook, I'm going to try implementing conjugate gradient in this same framework."
+ ]
}
- ]
-}
\ No newline at end of file
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Haskell",
+ "language": "haskell",
+ "name": "haskell"
+ },
+ "language_info": {
+ "name": "haskell",
+ "version": "7.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/notebooks/Homophones.ipynb b/notebooks/Homophones.ipynb
index e469f701..39b92774 100644
--- a/notebooks/Homophones.ipynb
+++ b/notebooks/Homophones.ipynb
@@ -1,615 +1,625 @@
{
- "metadata": {
- "language": "haskell",
- "name": ""
- },
- "nbformat": 3,
- "nbformat_minor": 0,
- "worksheets": [
+ "cells": [
{
- "cells": [
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "A few days ago, a friend of mine sent me [a fascinating problem](http://math.ucsd.edu/~justin/190hw.html). The problem goes like this:\n",
+ "\n",
+ "> The *homophony group* (of English) is the group with 26 generators `a`,`b`, `c`, and so on until `z` and one relation for every pair of English words which sound the same. Prove that the group is trivial!\n",
+ "\n",
+ "For example, consider the group elements **knight** and **night**. By the [cancellation laws](http://www.proofwiki.org/wiki/Cancellation_Laws), this implies that **k** must be the identity element. Recall that a trivial group is one which consists solely of its identity element, so our task is to show that each letter of the English alphabet is the identity element.\n",
+ "\n",
+ "Skipping all of the algebraic jargon, we want to show that if we set all homophones \"equal\" to one another, and do left cancellation, right cancellation, and substitution, we can show that all the English letters equal one.\n",
+ "\n",
+ "This is a fun exercise to do by hand, but I'd like to do it in Haskell. I've started by compiling a list of homophones in American English, starting with [this list](http://members.peak.org/~jeremy/dictionaryclassic/chapters/homophones.php) and removing all single letters (such as `j` being a homophone with `jay`) and all words with apostrophes and periods, as well as some less commonly used words.\n",
+ "\n",
+ "The contents of the file look like this:\n",
+ "```\n",
+ "ad add\n",
+ "add ad\n",
+ "arc ark\n",
+ "ark arc\n",
+ "...\n",
+ "```\n",
+ "\n",
+ "Each line is a space-delimited list of words. The first word in the list sounds identical to all the remaining words in the list. This is why you see repeats - `ad` sounds like `add` but also `add` sounds like `ad`. This repetition isn't necessary, as we could do it programmatically, but is convenient.\n",
+ "\n",
+ "Let's go ahead and load this list:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import Control.Applicative ((<$>))\n",
+ "import Data.List.Utils (split)\n",
+ "\n",
+ "removeEmpty = filter (not . null)\n",
+ "homophones <- removeEmpty . map words . lines <$> readFile \"homophones.list\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's take a look at a few more of these homophones."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
{
- "cell_type": "markdown",
+ "data": {
+ "text/plain": [
+ "adieu\tado\n",
+ "ado\tadieu\n",
+ "affect\teffect\n",
+ "aid\taide\n",
+ "aide\taid\n",
+ "ail\tale\n",
+ "air\terr\their\n",
+ "airs\terrs\theirs\n",
+ "aisle\tisle\n",
+ "ale\tail"
+ ]
+ },
"metadata": {},
- "source": [
- "A few days ago, a friend of mine sent me [a fascinating problem](http://math.ucsd.edu/~justin/190hw.html). The problem goes like this:\n",
- "\n",
- "> The *homophony group* (of English) is the group with 26 generators `a`,`b`, `c`, and so on until `z` and one relation for every pair of English words which sound the same. Prove that the group is trivial!\n",
- "\n",
- "For example, consider the group elements **knight** and **night**. By the [cancellation laws](http://www.proofwiki.org/wiki/Cancellation_Laws), this implies that **k** must be the identity element. Recall that a trivial group is one which consists solely of its identity element, so our task is to show that each letter of the English alphabet is the identity element.\n",
- "\n",
- "Skipping all of the algebraic jargon, we want to show that if we set all homophones \"equal\" to one another, and do left cancellation, right cancellation, and substitution, we can show that all the English letters equal one.\n",
- "\n",
- "This is a fun exercise to do by hand, but I'd like to do it in Haskell. I've started by compiling a list of homophones in American English, starting with [this list](http://members.peak.org/~jeremy/dictionaryclassic/chapters/homophones.php) and removing all single letters (such as `j` being a homophone with `jay`) and all words with apostrophes and periods, as well as some less commonly used words.\n",
- "\n",
- "The contents of the file look like this:\n",
- "```\n",
- "ad add\n",
- "add ad\n",
- "arc ark\n",
- "ark arc\n",
- "...\n",
- "```\n",
- "\n",
- "Each line is a space-delimited list of words. The first word in the list sounds identical to all the remaining words in the list. This is why you see repeats - `ad` sounds like `add` but also `add` sounds like `ad`. This repetition isn't necessary, as we could do it programmatically, but is convenient.\n",
- "\n",
- "Let's go ahead and load this list:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Control.Applicative ((<$>))\n",
- "import Data.List.Utils (split)\n",
- "\n",
- "removeEmpty = filter (not . null)\n",
- "homophones <- removeEmpty . map words . lines <$> readFile \"homophones.list\""
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 1
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's take a look at a few more of these homophones."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Control.Monad (forM_)\n",
- "import Data.List (intercalate)\n",
- "\n",
- "-- Show ten of the homophone sets\n",
- "forM_ (take 10 homophones) $ \\ homs -> \n",
- " putStrLn $ intercalate \"\\t\" homs"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "adieu\tado\n",
- "ado\tadieu\n",
- "affect\teffect\n",
- "aid\taide\n",
- "aide\taid\n",
- "ail\tale\n",
- "air\terr\their\n",
- "airs\terrs\theirs\n",
- "aisle\tisle\n",
- "ale\tail"
- ]
- }
- ],
- "prompt_number": 2
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that some of the sets have more than two elements, yet they are all on the same line.\n",
- "\n",
- "Let's convert this into a more usable format. We'll define a new type `WordPair` which represents a *single pair* of homophones, and convert this list into a list of `WordPair`s."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "data WordPair = WordPair String String\n",
- "\n",
- "-- Convert a list of homophones into a list of word pairs.\n",
- "-- Note that the wordpairs should only use the first of the \n",
- "-- list as the first word, since there will be repeat sets. \n",
- "-- For instance, the set [\"a\", \"b\", \"c\"] would only generate \n",
- "-- word pairs [WordPair \"a\" \"b\", WordPair \"a\" \"c\"].\n",
- "pairs :: [String] -> [WordPair]\n",
- "pairs (str:strs) = map (WordPair str) strs\n",
- "\n",
- "-- All pairs of words we consider homophones.\n",
- "wordPairs = concatMap pairs homophones"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 3
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now that we have this data in a usable form, let's use it to derive relations. \n",
- "\n",
- "The initial relations we have are simply the set of word pairs. However, we can use two operations in order to derive more relations:\n",
- "\n",
- "- `reduce`: The reduction operation will be the application of left and right cancellation laws. If a relation has the same thing on the left of both sides, we can take it off; same for the right side. This generates a new, simpler relation.\n",
- "- `substitute`: The substitution operation will be substituting identity relations in. For instance, if we've derived that `d` is the identity element, then we can remove `d` from all known relations to get new, simpler relations.\n",
- "\n",
- "In addition to each relation storing what strings it considers equal, we'd also like to be able to track what operations led to the creation of that word pair. So before defining a relation, let's define a history data type:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "data History = Reduce String String\n",
- " | Substitute Char"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 4
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, we'd like a relation to store all the transformations that were used to generate it, and also the two things it relates:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "data Relation = Relation [History] String String"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 5
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Since `Relation` and `WordPair` are slightly different, let's convert all our `WordPair`s to `Relation`s. This gives us our initial set of relations, which we will use to derive all other relations."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "toRelation :: WordPair -> Relation\n",
- "toRelation (WordPair first second) = Relation [] first second\n",
- "\n",
- "initRelations = map toRelation wordPairs"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 6
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Eventually, we're going to iteratively improve these relations until we have proven that all letters equal the identity. First, though, let's define our two operators, starting with `reduce`.\n",
- "\n",
- "When we `reduce` a relation, we apply the right and left cancellation laws. If we have the equation\n",
- "$$ab = ac$$\n",
- "we can use the left cancellation law to reduce it to $b = c$; similarly, using the right cancellation law, we can reduce the equation \n",
- "$$xa = ya$$\n",
- "to just $x = y$.\n",
- "\n",
- "Our `reduce` operator repeats these steps until it can no longer do so, and then the resulting strings are the reduced relation.\n"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "reduce :: Relation -> Relation\n",
- "reduce rel@(Relation hist first second)\n",
- " | canReduce first second = go (first, second)\n",
- " \n",
- " -- Note that we also have to be careful with the history.\n",
- " -- If the `reduce` does nothing, then we do not want to add\n",
- " -- anything to the history of the relation.\n",
- " | otherwise = rel\n",
- " \n",
- " where\n",
- " -- A reduction can happen if both strings are non-zero\n",
- " -- and share a common first or last letter.\n",
- " canReduce first second =\n",
- " not (null first) &&\n",
- " not (null second) &&\n",
- " (head first == head second ||\n",
- " last first == last second)\n",
- " \n",
- " -- Modified history including this reduction.\n",
- " hist' = Reduce first second : hist\n",
- " \n",
- " -- Base case: if we've reduced a word pair to an empty string \n",
- " -- and something else, we're done, as that something else\n",
- " -- is equivalent to the identity element.\n",
- " go (\"\", word) = Relation hist' word \"\"\n",
- " go (word, \"\") = Relation hist' word \"\" \n",
- " \n",
- " go (first, second)\n",
- " -- Chop off the first element if they're equal.\n",
- " | head first == head second\n",
- " = go (tail first, tail second)\n",
- " \n",
- " -- Chop off the last element if they're equal.\n",
- " | last first == last second\n",
- " = go (init first, init second)\n",
- " \n",
- " -- If netiher first nor last element are equal,\n",
- " -- we've simplified the relation down as much\n",
- " -- as we can simplify it.\n",
- " | otherwise =\n",
- " Relation hist' first second"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 7
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This looks pretty good. Next, let's define the `substitute` operator.\n",
- "\n",
- "The `substitute` operator removes a character from a relation. For instance, if we know that `d` is the identity, we can simplify the relation $$ad = dyd$$ to just $a = y$. \n",
- "\n",
- "Just like the `reduce` operator, we avoid modifying the `Relation`'s history if the `substitute` does nothing."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Data.List.Utils (replace)\n",
- "\n",
- "-- Generate a new relation by removing characters we know to be \n",
- "-- the identity. Make sure to update the history of the relation\n",
- "-- with this substitution!\n",
- "substitute :: Char -> Relation -> Relation\n",
- "substitute char rel@(Relation hist first second)\n",
- " | canSubstitute first second\n",
- " = Relation (Substitute char : hist) (replaced first) (replaced second)\n",
- " \n",
- " | otherwise = rel\n",
- " where\n",
- " canSubstitute first second = char `elem` first || char `elem` second\n",
- " replaced = replace [char] \"\""
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 8
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "With `substitute` implemented, we've finished all the machinery we're going to use for simplifying our relations. We're going to iteratively reduce and substitute until we've found that all the English letters are the identity element of the homophony group. We're still missing one thing, though - how do we know which letters we've proven to be the identity?\n",
- "\n",
- "Let's define a quick helper datatype for every identity we find. We're going to store the character that we've proven is the identity, as well as the history; that way, when we want to examine the results, we can see exactly how each letter was reduced to the identity."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "data FoundIdent = FoundIdent {\n",
- " char :: Char,\n",
- " hist :: [History]\n",
- " }"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 9
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's also define a function that extracts all the identity elements from a set of relations."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- mapMaybe = map fromJust . filter isJust . map\n",
- "import Data.Maybe (mapMaybe)\n",
- "\n",
- "identities :: [Relation] -> [FoundIdent]\n",
- "identities = mapMaybe go\n",
- " where\n",
- " go :: Relation -> Maybe FoundIdent\n",
- " go (Relation hist [char] \"\") = Just $ FoundIdent char hist\n",
- " go (Relation hist \"\" [char]) = Just $ FoundIdent char hist\n",
- " go _ = Nothing"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 10
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's finally put all of this together. We're going to start with our initial set of relations, `initRelations`, and then we're going to iteratively simplify them. Initially, we have no known identity elements.\n",
- "\n",
- "In each iteration, we\n",
- "\n",
- "- Substitute into each relation each known identity (replacing it with the empty string).\n",
- "- Reduce the resulting relations.\n",
- "- Collect all known identity elements."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Data.List (nubBy)\n",
- "import Data.Function (on)\n",
- "\n",
- "-- The iteration starts with a list of known identity elements\n",
- "-- and the current set of relations. It outputs the updated \n",
- "-- relations and all known identity elements.\n",
- "iteration :: ([FoundIdent], [Relation]) -> ([FoundIdent], [Relation])\n",
- "iteration (idents, relations) = (newIdents, newRelations)\n",
- " where\n",
- " -- Collect all the substitutions into a single function.\n",
- " substitutions = foldl (.) id $ map (substitute . char) idents\n",
- " \n",
- " -- Do all substitutions, then reduce (for each relation).\n",
- " newRelations = map (reduce . substitutions) relations\n",
- "\n",
- " -- We have to remove duplicate identity elements, because\n",
- " -- in each iteration we find multiple ways to prove that some\n",
- " -- letters are the identity element. We just want one.\n",
- " removeDuplicateIdents =\n",
- " nubBy ((==) `on` char)\n",
- "\n",
- " -- Find all identities in the new relations.\n",
- " newIdents = removeDuplicateIdents $ idents ++ identities newRelations"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 11
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's iterate this process until we have all the identities we want. We want 26 of them, so we can just check the length. (If this operation never finishes, we're out of luck!)"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Generate the infinite list of iterations and their results.\n",
- "initIdents = []\n",
- "iterations = iterate iteration (initIdents, initRelations)\n",
- "\n",
- "-- Define a completion condition.\n",
- "-- We're done when there are 26 known identity elements.\n",
- "done (idents, _) = length idents == 26\n",
- "\n",
- "-- Discard all iteration results until completion.\n",
- "-- Take the next one - the first one where the condition is met.\n",
- "result = head $ dropWhile (not . done) iterations"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [],
- "prompt_number": 12
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Woohoo! We're *done*! Let's take a look at the results!"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Data.List (sort)\n",
- "\n",
- "idents = fst result\n",
- "identChars = map char idents\n",
- "putStrLn $ sort identChars\n",
- "print $ length identChars"
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "abcdefghijklmnopqrstuvwxyz"
- ]
- },
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "26"
- ]
- }
- ],
- "prompt_number": 13
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Looks like we do indeed have every single letter mapped to the identity. \n",
- "\n",
- "Let's see if we can deduce, for each letter, how it was mapped to the identity. Instead of doing it in alphabetical order, we'll look at them in the order they were deduced, so it follows some logical flow."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "import Text.Printf (printf)\n",
- "\n",
- "forM_ idents $ \\(FoundIdent char hist) -> do\n",
- " printf \"Proving %c = 1:\\n\" char\n",
- " forM_ (reverse hist) $ \\op ->\n",
- " putStrLn $ case op of\n",
- " Reduce first second -> \n",
- " printf \"Reduce %s and %s\" first second\n",
- " Substitute ch ->\n",
- " printf \"Substitute %c for ''\" ch\n",
- " putStr \"\\n\""
- ],
- "language": "python",
- "metadata": {},
- "outputs": [
- {
- "metadata": {},
- "output_type": "display_data",
- "text": [
- "Proving e = 1:\n",
- "Reduce aid and aide\n",
- "\n",
- "Proving a = 1:\n",
- "Reduce aisle and isle\n",
- "\n",
- "Proving u = 1:\n",
- "Reduce ant and aunt\n",
- "\n",
- "Proving t = 1:\n",
- "Reduce but and butt\n",
- "\n",
- "Proving n = 1:\n",
- "Reduce cannon and canon\n",
- "\n",
- "Proving s = 1:\n",
- "Reduce cent and scent\n",
- "\n",
- "Proving h = 1:\n",
- "Reduce choral and coral\n",
- "\n",
- "Proving k = 1:\n",
- "Reduce doc and dock\n",
- "\n",
- "Proving l = 1:\n",
- "Reduce filet and fillet\n",
- "\n",
- "Proving w = 1:\n",
- "Reduce hole and whole\n",
- "\n",
- "Proving b = 1:\n",
- "Reduce plum and plumb\n",
- "\n",
- "Proving g = 1:\n",
- "Reduce reign and rein\n",
- "\n",
- "Proving c = 1:\n",
- "Reduce scent and sent\n",
- "\n",
- "Proving o = 1:\n",
- "Reduce to and too\n",
- "\n",
- "Proving i = 1:\n",
- "Reduce waive and wave\n",
- "\n",
- "Proving r = 1:\n",
- "Reduce air and err\n",
- "Substitute i for ''\n",
- "Substitute a for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving d = 1:\n",
- "Reduce awed and odd\n",
- "Substitute o for ''\n",
- "Substitute w for ''\n",
- "Substitute a for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving y = 1:\n",
- "Reduce bite and byte\n",
- "Substitute i for ''\n",
- "\n",
- "Proving z = 1:\n",
- "Reduce boos and booze\n",
- "Substitute s for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving q = 1:\n",
- "Reduce cask and casque\n",
- "Substitute k for ''\n",
- "Substitute u for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving x = 1:\n",
- "Reduce coax and cokes\n",
- "Substitute k for ''\n",
- "Substitute s for ''\n",
- "Substitute a for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving p = 1:\n",
- "Reduce coo and coup\n",
- "Substitute o for ''\n",
- "Substitute u for ''\n",
- "\n",
- "Proving f = 1:\n",
- "Reduce draft and draught\n",
- "Substitute g for ''\n",
- "Substitute h for ''\n",
- "Substitute u for ''\n",
- "\n",
- "Proving m = 1:\n",
- "Reduce damned and dammed\n",
- "Substitute n for ''\n",
- "\n",
- "Proving j = 1:\n",
- "Reduce genes and jeans\n",
- "Substitute g for ''\n",
- "Substitute n for ''\n",
- "Substitute a for ''\n",
- "Substitute e for ''\n",
- "\n",
- "Proving v = 1:\n",
- "Reduce felt and veldt\n",
- "Substitute l for ''\n",
- "Substitute e for ''\n",
- "Substitute f for ''\n",
- "Substitute d for ''"
- ]
- }
- ],
- "prompt_number": 14
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If you scan through the list above, there's a few weird cases, but for the most part, it seems legitimate. (I mildly question `felt` and `veldt`, but it depends on how you pronounce things. If you look at the British English list of homophones, it's totally different anyways!) \n",
- "\n",
- "So that's that! We've found the ways to reduce every letter to the identity, and shown how to do it.\n",
- "\n",
- "I wonder if other languages also have trivial homophony groups. It might be fun to try Spanish, French, Russian, and others, and see if the homophony groups tell us anything interesting about the language!\n",
- "\n",
- "**This work was done in [IHaskell](https://github.com/gibiansky/IHaskell), and what you're reading is the IHaskell notebook exported to HTML for viewing in the browser.**"
- ]
+ "output_type": "display_data"
}
],
- "metadata": {}
+ "source": [
+ "import Control.Monad (forM_)\n",
+ "import Data.List (intercalate)\n",
+ "\n",
+ "-- Show ten of the homophone sets\n",
+ "forM_ (take 10 homophones) $ \\ homs -> \n",
+ " putStrLn $ intercalate \"\\t\" homs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that some of the sets have more than two elements, yet they are all on the same line.\n",
+ "\n",
+ "Let's convert this into a more usable format. We'll define a new type `WordPair` which represents a *single pair* of homophones, and convert this list into a list of `WordPair`s."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "data WordPair = WordPair String String\n",
+ "\n",
+ "-- Convert a list of homophones into a list of word pairs.\n",
+ "-- Note that the wordpairs should only use the first of the \n",
+ "-- list as the first word, since there will be repeat sets. \n",
+ "-- For instance, the set [\"a\", \"b\", \"c\"] would only generate \n",
+ "-- word pairs [WordPair \"a\" \"b\", WordPair \"a\" \"c\"].\n",
+ "pairs :: [String] -> [WordPair]\n",
+ "pairs (str:strs) = map (WordPair str) strs\n",
+ "\n",
+ "-- All pairs of words we consider homophones.\n",
+ "wordPairs = concatMap pairs homophones"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now that we have this data in a usable form, let's use it to derive relations. \n",
+ "\n",
+ "The initial relations we have are simply the set of word pairs. However, we can use two operations in order to derive more relations:\n",
+ "\n",
+ "- `reduce`: The reduction operation will be the application of left and right cancellation laws. If a relation has the same thing on the left of both sides, we can take it off; same for the right side. This generates a new, simpler relation.\n",
+ "- `substitute`: The substitution operation will be substituting identity relations in. For instance, if we've derived that `d` is the identity element, then we can remove `d` from all known relations to get new, simpler relations.\n",
+ "\n",
+ "In addition to each relation storing what strings it considers equal, we'd also like to be able to track what operations led to the creation of that word pair. So before defining a relation, let's define a history data type:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "data History = Reduce String String\n",
+ " | Substitute Char"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now, we'd like a relation to store all the transformations that were used to generate it, and also the two things it relates:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "data Relation = Relation [History] String String"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Since `Relation` and `WordPair` are slightly different, let's convert all our `WordPair`s to `Relation`s. This gives us our initial set of relations, which we will use to derive all other relations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "toRelation :: WordPair -> Relation\n",
+ "toRelation (WordPair first second) = Relation [] first second\n",
+ "\n",
+ "initRelations = map toRelation wordPairs"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Eventually, we're going to iteratively improve these relations until we have proven that all letters equal the identity. First, though, let's define our two operators, starting with `reduce`.\n",
+ "\n",
+ "When we `reduce` a relation, we apply the right and left cancellation laws. If we have the equation\n",
+ "$$ab = ac$$\n",
+ "we can use the left cancellation law to reduce it to $b = c$; similarly, using the right cancellation law, we can reduce the equation \n",
+ "$$xa = ya$$\n",
+ "to just $x = y$.\n",
+ "\n",
+ "Our `reduce` operator repeats these steps until it can no longer do so, and then the resulting strings are the reduced relation.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "reduce :: Relation -> Relation\n",
+ "reduce rel@(Relation hist first second)\n",
+ " | canReduce first second = go (first, second)\n",
+ " \n",
+ " -- Note that we also have to be careful with the history.\n",
+ " -- If the `reduce` does nothing, then we do not want to add\n",
+ " -- anything to the history of the relation.\n",
+ " | otherwise = rel\n",
+ " \n",
+ " where\n",
+ " -- A reduction can happen if both strings are non-zero\n",
+ " -- and share a common first or last letter.\n",
+ " canReduce first second =\n",
+ " not (null first) &&\n",
+ " not (null second) &&\n",
+ " (head first == head second ||\n",
+ " last first == last second)\n",
+ " \n",
+ " -- Modified history including this reduction.\n",
+ " hist' = Reduce first second : hist\n",
+ " \n",
+ " -- Base case: if we've reduced a word pair to an empty string \n",
+ " -- and something else, we're done, as that something else\n",
+ " -- is equivalent to the identity element.\n",
+ " go (\"\", word) = Relation hist' word \"\"\n",
+ " go (word, \"\") = Relation hist' word \"\" \n",
+ " \n",
+ " go (first, second)\n",
+ " -- Chop off the first element if they're equal.\n",
+ " | head first == head second\n",
+ " = go (tail first, tail second)\n",
+ " \n",
+ " -- Chop off the last element if they're equal.\n",
+ " | last first == last second\n",
+ " = go (init first, init second)\n",
+ " \n",
+ " -- If netiher first nor last element are equal,\n",
+ " -- we've simplified the relation down as much\n",
+ " -- as we can simplify it.\n",
+ " | otherwise =\n",
+ " Relation hist' first second"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This looks pretty good. Next, let's define the `substitute` operator.\n",
+ "\n",
+ "The `substitute` operator removes a character from a relation. For instance, if we know that `d` is the identity, we can simplify the relation $$ad = dyd$$ to just $a = y$. \n",
+ "\n",
+ "Just like the `reduce` operator, we avoid modifying the `Relation`'s history if the `substitute` does nothing."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import Data.List.Utils (replace)\n",
+ "\n",
+ "-- Generate a new relation by removing characters we know to be \n",
+ "-- the identity. Make sure to update the history of the relation\n",
+ "-- with this substitution!\n",
+ "substitute :: Char -> Relation -> Relation\n",
+ "substitute char rel@(Relation hist first second)\n",
+ " | canSubstitute first second\n",
+ " = Relation (Substitute char : hist) (replaced first) (replaced second)\n",
+ " \n",
+ " | otherwise = rel\n",
+ " where\n",
+ " canSubstitute first second = char `elem` first || char `elem` second\n",
+ " replaced = replace [char] \"\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "With `substitute` implemented, we've finished all the machinery we're going to use for simplifying our relations. We're going to iteratively reduce and substitute until we've found that all the English letters are the identity element of the homophony group. We're still missing one thing, though - how do we know which letters we've proven to be the identity?\n",
+ "\n",
+ "Let's define a quick helper datatype for every identity we find. We're going to store the character that we've proven is the identity, as well as the history; that way, when we want to examine the results, we can see exactly how each letter was reduced to the identity."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "data FoundIdent = FoundIdent {\n",
+ " char :: Char,\n",
+ " hist :: [History]\n",
+ " }"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's also define a function that extracts all the identity elements from a set of relations."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- mapMaybe = map fromJust . filter isJust . map\n",
+ "import Data.Maybe (mapMaybe)\n",
+ "\n",
+ "identities :: [Relation] -> [FoundIdent]\n",
+ "identities = mapMaybe go\n",
+ " where\n",
+ " go :: Relation -> Maybe FoundIdent\n",
+ " go (Relation hist [char] \"\") = Just $ FoundIdent char hist\n",
+ " go (Relation hist \"\" [char]) = Just $ FoundIdent char hist\n",
+ " go _ = Nothing"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's finally put all of this together. We're going to start with our initial set of relations, `initRelations`, and then we're going to iteratively simplify them. Initially, we have no known identity elements.\n",
+ "\n",
+ "In each iteration, we\n",
+ "\n",
+ "- Substitute into each relation each known identity (replacing it with the empty string).\n",
+ "- Reduce the resulting relations.\n",
+ "- Collect all known identity elements."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import Data.List (nubBy)\n",
+ "import Data.Function (on)\n",
+ "\n",
+ "-- The iteration starts with a list of known identity elements\n",
+ "-- and the current set of relations. It outputs the updated \n",
+ "-- relations and all known identity elements.\n",
+ "iteration :: ([FoundIdent], [Relation]) -> ([FoundIdent], [Relation])\n",
+ "iteration (idents, relations) = (newIdents, newRelations)\n",
+ " where\n",
+ " -- Collect all the substitutions into a single function.\n",
+ " substitutions = foldl (.) id $ map (substitute . char) idents\n",
+ " \n",
+ " -- Do all substitutions, then reduce (for each relation).\n",
+ " newRelations = map (reduce . substitutions) relations\n",
+ "\n",
+ " -- We have to remove duplicate identity elements, because\n",
+ " -- in each iteration we find multiple ways to prove that some\n",
+ " -- letters are the identity element. We just want one.\n",
+ " removeDuplicateIdents =\n",
+ " nubBy ((==) `on` char)\n",
+ "\n",
+ " -- Find all identities in the new relations.\n",
+ " newIdents = removeDuplicateIdents $ idents ++ identities newRelations"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Let's iterate this process until we have all the identities we want. We want 26 of them, so we can just check the length. (If this operation never finishes, we're out of luck!)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Generate the infinite list of iterations and their results.\n",
+ "initIdents = []\n",
+ "iterations = iterate iteration (initIdents, initRelations)\n",
+ "\n",
+ "-- Define a completion condition.\n",
+ "-- We're done when there are 26 known identity elements.\n",
+ "done (idents, _) = length idents == 26\n",
+ "\n",
+ "-- Discard all iteration results until completion.\n",
+ "-- Take the next one - the first one where the condition is met.\n",
+ "result = head $ dropWhile (not . done) iterations"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Woohoo! We're *done*! Let's take a look at the results!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "abcdefghijklmnopqrstuvwxyz"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ },
+ {
+ "data": {
+ "text/plain": [
+ "26"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import Data.List (sort)\n",
+ "\n",
+ "idents = fst result\n",
+ "identChars = map char idents\n",
+ "putStrLn $ sort identChars\n",
+ "print $ length identChars"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Looks like we do indeed have every single letter mapped to the identity. \n",
+ "\n",
+ "Let's see if we can deduce, for each letter, how it was mapped to the identity. Instead of doing it in alphabetical order, we'll look at them in the order they were deduced, so it follows some logical flow."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Proving e = 1:\n",
+ "Reduce aid and aide\n",
+ "\n",
+ "Proving a = 1:\n",
+ "Reduce aisle and isle\n",
+ "\n",
+ "Proving u = 1:\n",
+ "Reduce ant and aunt\n",
+ "\n",
+ "Proving t = 1:\n",
+ "Reduce but and butt\n",
+ "\n",
+ "Proving n = 1:\n",
+ "Reduce cannon and canon\n",
+ "\n",
+ "Proving s = 1:\n",
+ "Reduce cent and scent\n",
+ "\n",
+ "Proving h = 1:\n",
+ "Reduce choral and coral\n",
+ "\n",
+ "Proving k = 1:\n",
+ "Reduce doc and dock\n",
+ "\n",
+ "Proving l = 1:\n",
+ "Reduce filet and fillet\n",
+ "\n",
+ "Proving w = 1:\n",
+ "Reduce hole and whole\n",
+ "\n",
+ "Proving b = 1:\n",
+ "Reduce plum and plumb\n",
+ "\n",
+ "Proving g = 1:\n",
+ "Reduce reign and rein\n",
+ "\n",
+ "Proving c = 1:\n",
+ "Reduce scent and sent\n",
+ "\n",
+ "Proving o = 1:\n",
+ "Reduce to and too\n",
+ "\n",
+ "Proving i = 1:\n",
+ "Reduce waive and wave\n",
+ "\n",
+ "Proving r = 1:\n",
+ "Reduce air and err\n",
+ "Substitute i for ''\n",
+ "Substitute a for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving d = 1:\n",
+ "Reduce awed and odd\n",
+ "Substitute o for ''\n",
+ "Substitute w for ''\n",
+ "Substitute a for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving y = 1:\n",
+ "Reduce bite and byte\n",
+ "Substitute i for ''\n",
+ "\n",
+ "Proving z = 1:\n",
+ "Reduce boos and booze\n",
+ "Substitute s for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving q = 1:\n",
+ "Reduce cask and casque\n",
+ "Substitute k for ''\n",
+ "Substitute u for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving x = 1:\n",
+ "Reduce coax and cokes\n",
+ "Substitute k for ''\n",
+ "Substitute s for ''\n",
+ "Substitute a for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving p = 1:\n",
+ "Reduce coo and coup\n",
+ "Substitute o for ''\n",
+ "Substitute u for ''\n",
+ "\n",
+ "Proving f = 1:\n",
+ "Reduce draft and draught\n",
+ "Substitute g for ''\n",
+ "Substitute h for ''\n",
+ "Substitute u for ''\n",
+ "\n",
+ "Proving m = 1:\n",
+ "Reduce damned and dammed\n",
+ "Substitute n for ''\n",
+ "\n",
+ "Proving j = 1:\n",
+ "Reduce genes and jeans\n",
+ "Substitute g for ''\n",
+ "Substitute n for ''\n",
+ "Substitute a for ''\n",
+ "Substitute e for ''\n",
+ "\n",
+ "Proving v = 1:\n",
+ "Reduce felt and veldt\n",
+ "Substitute l for ''\n",
+ "Substitute e for ''\n",
+ "Substitute f for ''\n",
+ "Substitute d for ''"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "import Text.Printf (printf)\n",
+ "\n",
+ "forM_ idents $ \\(FoundIdent char hist) -> do\n",
+ " printf \"Proving %c = 1:\\n\" char\n",
+ " forM_ (reverse hist) $ \\op ->\n",
+ " putStrLn $ case op of\n",
+ " Reduce first second -> \n",
+ " printf \"Reduce %s and %s\" first second\n",
+ " Substitute ch ->\n",
+ " printf \"Substitute %c for ''\" ch\n",
+ " putStr \"\\n\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you scan through the list above, there's a few weird cases, but for the most part, it seems legitimate. (I mildly question `felt` and `veldt`, but it depends on how you pronounce things. If you look at the British English list of homophones, it's totally different anyways!) \n",
+ "\n",
+ "So that's that! We've found the ways to reduce every letter to the identity, and shown how to do it.\n",
+ "\n",
+ "I wonder if other languages also have trivial homophony groups. It might be fun to try Spanish, French, Russian, and others, and see if the homophony groups tell us anything interesting about the language!\n",
+ "\n",
+ "**This work was done in [IHaskell](https://github.com/gibiansky/IHaskell), and what you're reading is the IHaskell notebook exported to HTML for viewing in the browser.**"
+ ]
}
- ]
-}
\ No newline at end of file
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Haskell",
+ "language": "haskell",
+ "name": "haskell"
+ },
+ "language_info": {
+ "name": "haskell",
+ "version": "7.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
diff --git a/notebooks/Static Canvas IHaskell Display.ipynb b/notebooks/Static Canvas IHaskell Display.ipynb
index 7545ab46..f5ca35df 100644
--- a/notebooks/Static Canvas IHaskell Display.ipynb
+++ b/notebooks/Static Canvas IHaskell Display.ipynb
@@ -1,316 +1,314 @@
{
- "metadata": {
- "language": "haskell",
- "name": "",
- "signature": "sha256:95812c4aac52ed0e9f86f92d457f4647bf9adb83e02b30f67ce5a9362a823c07"
- },
- "nbformat": 3,
- "nbformat_minor": 0,
- "worksheets": [
+ "cells": [
{
- "cells": [
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "Recently, Jeffrey Rosenbluth published (and showcased [on Reddit](http://www.reddit.com/r/haskell/comments/2vpf0t/announcing_staticcanvas_write_html5_canvas_in/)) a pretty cool Haskell package called [static-canvas](https://hackage.haskell.org/package/static-canvas). This package uses the free monad DSL pattern to make a DSL for programming for HTML5 `canvas`, restricted to fairly simple static use cases. While you can't use this to make user interfaces, it's still potentially a pretty cool tool, and there's a few very clear examples on the [GitHub readme](https://github.com/jeffreyrosenbluth/static-canvas)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "As with most things involving pretty graphics or pictures, I think this would be a whole ton of fun to experiment with interactively, making it a great fit for [IHaskell](http://www.github.com/gibiansky/IHaskell), an interactive notebook-based environment for Haskell.\n",
+ "\n",
+ "IHaskell allows the creation of \"addon\" packages to specify how to display various data types in its browser-based UI. These addons can render data types as text, as images, or even as HTML mixed with Javascript; they can even render them as interactive Javascript widgets that can evaluate Haskell code at will. All of this is done without GHCJS or similar Haskell-to-Javascript compilation tools.\n",
+ "\n",
+ "However, these display packages have mostly been written by only a few people, those fairly closely involved with IHaskell development. As the creator of IHaskell, I'd love to have more of these packages, but I obviously can't create display instances for all existing packages, and certainly can't anticipate what people might want for their own packages or new ones. Thus, I'd love to use this very neat library as a showcase and tutorial for how to make IHaskell display packages."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "## The Tools\n",
+ "In this section, I'll very briefly introduce you to the tools IHaskell provides for creating IHaskell display packages. If you'd like to get to the real meat of this tutorial, skip this, read the next section, and maybe come back here if you need to.\n",
+ "\n",
+ "IHaskell internally uses a data type called `Display` to represent possible outputs. The `Display` data types looks like this:\n",
+ "\n",
+ "```haskell\n",
+ "-- In IHaskell.Display\n",
+ "data Display = Display [DisplayData] -- Display just one thing.\n",
+ " | ManyDisplay [Display] -- Display several things.\n",
+ "```\n",
+ "In turn, the `DisplayData` data type from the `ipython-kernel` package specifies how to actually display the object in the browser:\n",
+ "```haskell\n",
+ "-- In IHaskell.IPython.Display\n",
+ "data DisplayData = DisplayData MimeType Text\n",
+ "\n",
+ "-- All the possible ways to display things.\n",
+ "data MimeType = PlainText\n",
+ " | MimeHtml\n",
+ " | MimePng Width Height -- Base64 encoded.\n",
+ " | MimeJpg Width Height -- Base64 encoded.\n",
+ " | MimeSvg\n",
+ " | MimeLatex\n",
+ " | MimeJavascript\n",
+ "```\n",
+ "\n",
+ "For example, to output the string \"Hello\" in red in the browser, you might construct a value like this:\n",
+ "```haskell\n",
+ "redStr :: Display\n",
+ "redStr = Display [textDisplay, htmlDisplay]\n",
+ "\n",
+ "textDisplay :: DisplayData\n",
+ "textDisplay = DisplayData PlainText \"Hello\"\n",
+ "\n",
+ "htmlDisplay :: DisplayData\n",
+ "htmlDisplay = DisplayData MimeHtml \"Hello\"\n",
+ "```\n",
+ "\n",
+ "You may note that `Display` takes a *list* of `DisplayData` values; this allows IHaskell to choose the proper display mechanism for the frontend. The frontend can be a console or the in-browser notebook, and the in-browser notebook may have different preferences for displays, so by providing different ways to render output, the best possible rendering can be chosen for each interface.\n",
+ "\n",
+ "Instead of always using the data types, `IHaskell.Display` exports the following convenience functions:\n",
+ "```haskell\n",
+ "-- Construct displays from raw strings of different types.\n",
+ "plain :: String -> DisplayData\n",
+ "html :: String -> DisplayData\n",
+ "svg :: String -> DisplayData\n",
+ "latex :: String -> DisplayData\n",
+ "javascript :: String -> DisplayData\n",
+ "\n",
+ "-- Encode into base 64.\n",
+ "encode64 :: String -> Base64\n",
+ "decode64 :: ByteString -> Base64\n",
+ "\n",
+ "-- Display images.\n",
+ "png :: Int -> Int -> Base64 -> DisplayData\n",
+ "jpg :: Int -> Int -> Base64 -> DisplayData\n",
+ "\n",
+ "-- Create final Displays.\n",
+ "Display :: [DisplayData] -> Display\n",
+ "many :: [Display] -> Display\n",
+ "```\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "## Creating a Display\n",
+ "\n",
+ "In order to create a display for some data type, we must first import the main IHaskell display module:\n",
+ "```haskell\n",
+ "import IHaskell.Display\n",
+ "```\n",
+ "This package contains the following typeclass:\n",
+ "```haskell\n",
+ "class IHaskellDisplay a where\n",
+ " display :: a -> IO Display\n",
+ "```\n",
+ "\n",
+ "In order to display a data type, create an instance of `IHaskellDisplay` for your data type – then, any expression that results in your data type will generate a corresponding display. \n",
+ "\n",
+ "Let's go ahead and do this for `CanvasFree a` from the `static-canvas` package."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "collapsed": false,
+ "hidden": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Start with necessary imports.\n",
+ "import IHaskell.Display -- From the 'ihaskell' package.\n",
+ "import IHaskell.IPython.Types(MimeType(..))\n",
+ "import Graphics.Static -- From the 'static-canvas' package.\n",
+ "\n",
+ "-- Text conversion functions.\n",
+ "import Data.Text.Lazy.Builder(toLazyText)\n",
+ "import Data.Text.Lazy(toStrict)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "Now that we have the imports out of the way, we can define the core instance necessary:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {
+ "collapsed": false,
+ "hidden": false
+ },
+ "outputs": [],
+ "source": [
+ "-- Since CanvasFree is a type synonym, we need a language pragma.\n",
+ "{-# LANGUAGE TypeSynonymInstances #-}\n",
+ "\n",
+ "instance IHaskellDisplay (CanvasFree ()) where\n",
+ " -- display :: CanvasFree () -> IO Display\n",
+ " display canvas = return $\n",
+ " let src = toStrict $ toLazyText $ buildScript width height canvas\n",
+ " in Display [DisplayData MimeHtml src]\n",
+ " \n",
+ " where (height, width) = (200, 600)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "We can now copy and paste the examples from the `static-canvas` Github page, and see them appear right in the notebook!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 34,
+ "metadata": {
+ "collapsed": false,
+ "hidden": false
+ },
+ "outputs": [
{
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
+ "data": {
+ "text/html": [
+ ""
+ ]
},
- "source": [
- "Recently, Jeffrey Rosenbluth published (and showcased [on Reddit](http://www.reddit.com/r/haskell/comments/2vpf0t/announcing_staticcanvas_write_html5_canvas_in/)) a pretty cool Haskell package called [static-canvas](https://hackage.haskell.org/package/static-canvas). This package uses the free monad DSL pattern to make a DSL for programming for HTML5 `canvas`, restricted to fairly simple static use cases. While you can't use this to make user interfaces, it's still potentially a pretty cool tool, and there's a few very clear examples on the [GitHub readme](https://github.com/jeffreyrosenbluth/static-canvas)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "As with most things involving pretty graphics or pictures, I think this would be a whole ton of fun to experiment with interactively, making it a great fit for [IHaskell](http://www.github.com/gibiansky/IHaskell), an interactive notebook-based environment for Haskell.\n",
- "\n",
- "IHaskell allows the creation of \"addon\" packages to specify how to display various data types in its browser-based UI. These addons can render data types as text, as images, or even as HTML mixed with Javascript; they can even render them as interactive Javascript widgets that can evaluate Haskell code at will. All of this is done without GHCJS or similar Haskell-to-Javascript compilation tools.\n",
- "\n",
- "However, these display packages have mostly been written by only a few people, those fairly closely involved with IHaskell development. As the creator of IHaskell, I'd love to have more of these packages, but I obviously can't create display instances for all existing packages, and certainly can't anticipate what people might want for their own packages or new ones. Thus, I'd love to use this very neat library as a showcase and tutorial for how to make IHaskell display packages."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "## The Tools\n",
- "In this section, I'll very briefly introduce you to the tools IHaskell provides for creating IHaskell display packages. If you'd like to get to the real meat of this tutorial, skip this, read the next section, and maybe come back here if you need to.\n",
- "\n",
- "IHaskell internally uses a data type called `Display` to represent possible outputs. The `Display` data types looks like this:\n",
- "\n",
- "```haskell\n",
- "-- In IHaskell.Display\n",
- "data Display = Display [DisplayData] -- Display just one thing.\n",
- " | ManyDisplay [Display] -- Display several things.\n",
- "```\n",
- "In turn, the `DisplayData` data type from the `ipython-kernel` package specifies how to actually display the object in the browser:\n",
- "```haskell\n",
- "-- In IHaskell.IPython.Display\n",
- "data DisplayData = DisplayData MimeType Text\n",
- "\n",
- "-- All the possible ways to display things.\n",
- "data MimeType = PlainText\n",
- " | MimeHtml\n",
- " | MimePng Width Height -- Base64 encoded.\n",
- " | MimeJpg Width Height -- Base64 encoded.\n",
- " | MimeSvg\n",
- " | MimeLatex\n",
- " | MimeJavascript\n",
- "```\n",
- "\n",
- "For example, to output the string \"Hello\" in red in the browser, you might construct a value like this:\n",
- "```haskell\n",
- "redStr :: Display\n",
- "redStr = Display [textDisplay, htmlDisplay]\n",
- "\n",
- "textDisplay :: DisplayData\n",
- "textDisplay = DisplayData PlainText \"Hello\"\n",
- "\n",
- "htmlDisplay :: DisplayData\n",
- "htmlDisplay = DisplayData MimeHtml \"Hello\"\n",
- "```\n",
- "\n",
- "You may note that `Display` takes a *list* of `DisplayData` values; this allows IHaskell to choose the proper display mechanism for the frontend. The frontend can be a console or the in-browser notebook, and the in-browser notebook may have different preferences for displays, so by providing different ways to render output, the best possible rendering can be chosen for each interface.\n",
- "\n",
- "Instead of always using the data types, `IHaskell.Display` exports the following convenience functions:\n",
- "```haskell\n",
- "-- Construct displays from raw strings of different types.\n",
- "plain :: String -> DisplayData\n",
- "html :: String -> DisplayData\n",
- "svg :: String -> DisplayData\n",
- "latex :: String -> DisplayData\n",
- "javascript :: String -> DisplayData\n",
- "\n",
- "-- Encode into base 64.\n",
- "encode64 :: String -> Base64\n",
- "decode64 :: ByteString -> Base64\n",
- "\n",
- "-- Display images.\n",
- "png :: Int -> Int -> Base64 -> DisplayData\n",
- "jpg :: Int -> Int -> Base64 -> DisplayData\n",
- "\n",
- "-- Create final Displays.\n",
- "Display :: [DisplayData] -> Display\n",
- "many :: [Display] -> Display\n",
- "```\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "## Creating a Display\n",
- "\n",
- "In order to create a display for some data type, we must first import the main IHaskell display module:\n",
- "```haskell\n",
- "import IHaskell.Display\n",
- "```\n",
- "This package contains the following typeclass:\n",
- "```haskell\n",
- "class IHaskellDisplay a where\n",
- " display :: a -> IO Display\n",
- "```\n",
- "\n",
- "In order to display a data type, create an instance of `IHaskellDisplay` for your data type \u2013 then, any expression that results in your data type will generate a corresponding display. \n",
- "\n",
- "Let's go ahead and do this for `CanvasFree a` from the `static-canvas` package."
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Start with necessary imports.\n",
- "import IHaskell.Display -- From the 'ihaskell' package.\n",
- "import IHaskell.IPython.Types(MimeType(..))\n",
- "import Graphics.Static -- From the 'static-canvas' package.\n",
- "\n",
- "-- Text conversion functions.\n",
- "import Data.Text.Lazy.Builder(toLazyText)\n",
- "import Data.Text.Lazy(toStrict)"
- ],
- "language": "python",
- "metadata": {
- "hidden": false
- },
- "outputs": [],
- "prompt_number": 12
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "Now that we have the imports out of the way, we can define the core instance necessary:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "-- Since CanvasFree is a type synonym, we need a language pragma.\n",
- "{-# LANGUAGE TypeSynonymInstances #-}\n",
- "\n",
- "instance IHaskellDisplay (CanvasFree ()) where\n",
- " -- display :: CanvasFree () -> IO Display\n",
- " display canvas = return $\n",
- " let src = toStrict $ toLazyText $ buildScript width height canvas\n",
- " in Display [DisplayData MimeHtml src]\n",
- " \n",
- " where (height, width) = (200, 600)"
- ],
- "language": "python",
- "metadata": {
- "hidden": false
- },
- "outputs": [],
- "prompt_number": 24
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "We can now copy and paste the examples from the `static-canvas` Github page, and see them appear right in the notebook!"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "{-# LANGUAGE OverloadedStrings #-}\n",
- "import Graphics.Static.ColorNames\n",
- "\n",
- "text :: CanvasFree ()\n",
- "text = do\n",
- " font \"italic 60pt Calibri\"\n",
- " lineWidth 6\n",
- " strokeStyle blue\n",
- " fillStyle goldenrod\n",
- " textBaseline TextBaselineMiddle\n",
- " strokeText \"Hello\" 150 100 \n",
- " fillText \"Hello World!\" 150 100\n",
- " \n",
- "text"
- ],
- "language": "python",
- "metadata": {
- "hidden": false
- },
- "outputs": [
- {
- "html": [
- ""
- ],
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "prompt_number": 34
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "As we play with this a little more, we see that this is a little bit unsatisfactory. Specifically, the width and the height of the resulting canvas are fixed in the `IHaskellDisplay` instance! I would solve this by creating a custom `Canvas` data type that stores these:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "data Canvas = Canvas {\n",
- " width :: Int,\n",
- " height :: Int,\n",
- " canvas :: CanvasFree ()\n",
- " }"
- ],
- "language": "python",
- "metadata": {
- "hidden": false
- },
- "outputs": [],
- "prompt_number": 26
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "Then we could define an `IHaskellDisplay` that respects this width and height:"
- ]
- },
- {
- "cell_type": "code",
- "collapsed": false,
- "input": [
- "{-# LANGUAGE TypeSynonymInstances #-}\n",
- "instance IHaskellDisplay Canvas where\n",
- " -- display :: Canvas -> IO Display\n",
- " display cnv = return $\n",
- " let src = toStrict $ toLazyText $ buildScript (width cnv) (height cnv) (canvas cnv)\n",
- " in Display [DisplayData MimeHtml src]"
- ],
- "language": "python",
- "metadata": {
- "hidden": false
- },
- "outputs": [],
- "prompt_number": 27
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "Then when we use this we can specify how to display our canvases:\n",
- "```haskell\n",
- "Canvas 200 600 $ do\n",
- " font \"italic 60pt Calibri\"\n",
- " lineWidth 6\n",
- " strokeStyle blue\n",
- " fillStyle goldenrod\n",
- " textBaseline TextBaselineMiddle\n",
- " strokeText \"Hello\" 150 100 \n",
- " fillText \"Hello World!\" 150 100\n",
- "```\n",
- "\n",
- "Sadly, it seems that the `static-canvas` library currently only supports having *one* generated canvas on the page \u2013 if you try to add another one, it simply modifies the pre-existing one. This is probably a bug that should be fixed, though!"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "hidden": false
- },
- "source": [
- "## Packaging IHaskell Display Addons\n",
- "\n",
- "Once you've made an IHaskell display instance, you can easily package it up and stick it on Hackage. Specifically, for a package named `package-name`, you should take everything before the `-`. Then, prepend `ihaskell-` to the package name. Finally, make sure there exists a module `IHaskell.Display.Package`, where `Package` is the first word in `package-name` capitalized. If this is done, then IHaskell will happily load your package and instance upon startup, making it very easy for your users to install the display addon!\n",
- "\n",
- "For example, the `hatex` library is exposed as an addon through the `ihaskell-hatex` display package and the `IHaskell.Display.Hatex` module in that package. The `juicypixels` library has an addon package called `ihaskell-juicypixels` with a module `IHaskell.Display.Juicypixels`. \n",
- "\n",
- "As I write this now, I realize that this protocol is a little bit weird. Specifically, I think that perhaps the rule that you take the first thing before the `-` is not too great, but rather that perhaps the `-` should be a word separator, and thus `package-name` would get translated to `ihaskell-package-name` and `IHaskell.Display.PackageName`. (We do need *some* standard!)\n",
- "\n",
- "If you have any opinions about this, or suggestions for how to improve this process, please let me know!\n",
- "\n",
- "Anyway, I hope that this brief tutorial / guide can show someone how to write small IHaskell addons. Perhaps someone will find this useful, and please get in touch if you have any questions, comments, or suggestions!"
- ]
+ "metadata": {},
+ "output_type": "display_data"
}
],
- "metadata": {}
+ "source": [
+ "{-# LANGUAGE OverloadedStrings #-}\n",
+ "import Graphics.Static.ColorNames\n",
+ "\n",
+ "text :: CanvasFree ()\n",
+ "text = do\n",
+ " font \"italic 60pt Calibri\"\n",
+ " lineWidth 6\n",
+ " strokeStyle blue\n",
+ " fillStyle goldenrod\n",
+ " textBaseline TextBaselineMiddle\n",
+ " strokeText \"Hello\" 150 100 \n",
+ " fillText \"Hello World!\" 150 100\n",
+ " \n",
+ "text"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "As we play with this a little more, we see that this is a little bit unsatisfactory. Specifically, the width and the height of the resulting canvas are fixed in the `IHaskellDisplay` instance! I would solve this by creating a custom `Canvas` data type that stores these:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "metadata": {
+ "collapsed": false,
+ "hidden": false
+ },
+ "outputs": [],
+ "source": [
+ "data Canvas = Canvas {\n",
+ " width :: Int,\n",
+ " height :: Int,\n",
+ " canvas :: CanvasFree ()\n",
+ " }"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "Then we could define an `IHaskellDisplay` that respects this width and height:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 27,
+ "metadata": {
+ "collapsed": false,
+ "hidden": false
+ },
+ "outputs": [],
+ "source": [
+ "{-# LANGUAGE TypeSynonymInstances #-}\n",
+ "instance IHaskellDisplay Canvas where\n",
+ " -- display :: Canvas -> IO Display\n",
+ " display cnv = return $\n",
+ " let src = toStrict $ toLazyText $ buildScript (width cnv) (height cnv) (canvas cnv)\n",
+ " in Display [DisplayData MimeHtml src]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "Then when we use this we can specify how to display our canvases:\n",
+ "```haskell\n",
+ "Canvas 200 600 $ do\n",
+ " font \"italic 60pt Calibri\"\n",
+ " lineWidth 6\n",
+ " strokeStyle blue\n",
+ " fillStyle goldenrod\n",
+ " textBaseline TextBaselineMiddle\n",
+ " strokeText \"Hello\" 150 100 \n",
+ " fillText \"Hello World!\" 150 100\n",
+ "```\n",
+ "\n",
+ "Sadly, it seems that the `static-canvas` library currently only supports having *one* generated canvas on the page – if you try to add another one, it simply modifies the pre-existing one. This is probably a bug that should be fixed, though!"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "hidden": false
+ },
+ "source": [
+ "## Packaging IHaskell Display Addons\n",
+ "\n",
+ "Once you've made an IHaskell display instance, you can easily package it up and stick it on Hackage. Specifically, for a package named `package-name`, you should take everything before the `-`. Then, prepend `ihaskell-` to the package name. Finally, make sure there exists a module `IHaskell.Display.Package`, where `Package` is the first word in `package-name` capitalized. If this is done, then IHaskell will happily load your package and instance upon startup, making it very easy for your users to install the display addon!\n",
+ "\n",
+ "For example, the `hatex` library is exposed as an addon through the `ihaskell-hatex` display package and the `IHaskell.Display.Hatex` module in that package. The `juicypixels` library has an addon package called `ihaskell-juicypixels` with a module `IHaskell.Display.Juicypixels`. \n",
+ "\n",
+ "As I write this now, I realize that this protocol is a little bit weird. Specifically, I think that perhaps the rule that you take the first thing before the `-` is not too great, but rather that perhaps the `-` should be a word separator, and thus `package-name` would get translated to `ihaskell-package-name` and `IHaskell.Display.PackageName`. (We do need *some* standard!)\n",
+ "\n",
+ "If you have any opinions about this, or suggestions for how to improve this process, please let me know!\n",
+ "\n",
+ "Anyway, I hope that this brief tutorial / guide can show someone how to write small IHaskell addons. Perhaps someone will find this useful, and please get in touch if you have any questions, comments, or suggestions!"
+ ]
}
- ]
-}
\ No newline at end of file
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Haskell",
+ "language": "haskell",
+ "name": "haskell"
+ },
+ "language_info": {
+ "name": "haskell",
+ "version": "7.8.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}