{"id":184,"date":"2016-05-19T13:14:25","date_gmt":"2016-05-19T13:14:25","guid":{"rendered":"http:\/\/deberker.com\/archy\/?p=184"},"modified":"2021-11-06T16:05:52","modified_gmt":"2021-11-06T16:05:52","slug":"deep-learning-textbook-chapter-2-probability","status":"publish","type":"post","link":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/","title":{"rendered":"Deep Learning Textbook Chapter 2: Probability"},"content":{"rendered":"

As part of the Max Planck Centre’s Deep Learning reading group, I’m working my way through the soon-to-be released Deep Learning<\/a> textbook. These posts are my informal notes for each chapter. Find my notes from chapter 1 here.<\/a>\u00a0Thanks again to Zeb & Toby for edits.<\/em><\/p>\n

This week we were back on slightly firmer ground, dealing with the chapter on probability. Most of this chapter will be fairly familiar to those who have done some statistics and\/or computational modelling. My notes are correspondingly sparse.<\/p>\n

Flavours of uncertainty in\u00a0modelling<\/strong><\/p>\n

    \n
  1. Inherent stochasticity, e.g. in the outcome of a coin flip (equivalent to irreducible uncertainty, which we wrote about here<\/a>)<\/li>\n
  2. Incomplete observability – we don’t have enough information to fully specify what’s going to happen<\/li>\n
  3. Incomplete modelling –\u00a0all models are wrong, some are useful (George Box). All models sacrifice some precision in order to make useful simplifications of the subject under study; these simplifications bring with them some uncertainty.<\/li>\n<\/ol>\n

    Frequentist vs Bayesian probability<\/strong><\/p>\n

    The textbook\u00a0has\u00a0a nice delineation of two ways of thinking about probability.<\/p>\n

    The frequentist tradition maintains that probability is the percentage of times that something has happened divided by all the times it might have happened, and, in the strongest interpretation, that these are the only kind of probabilities it makes sense to talk about. \u00a0We can therefore derive probabilities for rain on a day in June, but we can’t derive probabilities for the likelihood of the sun exploding, or for the probability of my having cancer.<\/p>\n

    A Bayesian treatment, however, allows us to use probabilities to describe various degrees of certainty<\/em>, or degrees of belief<\/em>. This is far more useful from a neuroscientific perspective, and allows us to do all sorts of neat stuff, like integrate sensory information optimally <\/a>or learn at appropriate rates.<\/a><\/p>\n

    Bayesian statistics also allows us to integrate priors – information about how likely something is,\u00a0a priori<\/em> (that is, without observing any new information). This can be pretty useful:<\/p>\n

    \"frequentists_vs_bayesians\"<\/a><\/p>\n

    Probability Mass Functions<\/strong><\/p>\n

    A Probability Mass Function (PMF) describes the probability of a randomly drawn\u00a0sample from x taking on a certain value x <\/em>(note the italics)\u00a0: P(x=x<\/em>).<\/p>\n

    PMF’s are used when variables are discrete. They generate nice blocky histograms, with the height of the bar telling us how likely we are to observe that value of x. For instance, this is the PMF for a (fair) die:<\/p>\n

    \"pmf\"<\/a><\/strong><\/p>\n

    Probability Density Functions<\/strong><\/p>\n

    Probability Density Functions (PDF’s) deal with the P(x=x<\/em>) again, but for\u00a0continuous\u00a0<\/strong>variables. This has one huge implication:\u00a0the height of the function no longer tells us the probability of observing that particular value of x, as the probability of seeing x of a particular value ~=0. <\/strong>This is because the function is continuous – so it has infinitely many values – and since the probability of any one value is going to be 1 \/ n_possible_values, we have 1\/infinity == 0.<\/p>\n

    Intuitively, this is not different from the material meaning of density – if we take an infinitely small volume, we can’t say anything about its density (since density = mass\/volume). We need to specify a volume in order to measure the density and therefore infer the mass.<\/p>\n

    Instead, we use integration with some set of limits to tell us how likely we are to\u00a0observe\u00a0x<\/em> in some range<\/strong>. So we can ask: ‘how likely am I to observe\u00a0x<\/em> between a and b?’:<\/p>\n

    \"PDF\"<\/a><\/p>\n

    Marginal probabilities<\/strong><\/p>\n

    We use marginal probabilities to isolate\u00a0the probabilities of a subset of variables. Say we have a bunch of people, and we split them up according to gender and by height. We now have them labelled as short\/tall, and male\/female, and we count them as<\/p>\n

    30 tall men<\/p>\n

    15 short men<\/p>\n

    20 tall women<\/p>\n

    35 short women<\/p>\n

    Now if we want to\u00a0know the probability that a random draw will produce a man, we can\u00a0marginalise<\/strong> over the height variable\u00a0to find p(male):<\/p>\n

    \"84ba7aca1e9f93f3151646d8b9717c35\"<\/a><\/p>\n

    Where X is male\/female, and Y is tall\/short. In order to isolate the probability distribution of gender, we need to ‘sum out’ the probability that\u00a0the person is tall or short.<\/p>\n

    Practically: p(male) = (0.3\/0.5)*(0.5) + (0.15\/0.5)*(0.5) = 0.45 (which is bang on – we have 45 men in the sample).<\/p>\n

    Conversely, p(short) = (0.15\/0.45)*0.45 +(0.35\/0.55)*0.55 = 0.5.<\/p>\n

    Note that in the two variable case, we can actually cancel out\u00a0the denominator\u00a0of the conditional with the total probability (i.e. in both cases, we’re dividing by p(y) and then immediately multiplying by it again).<\/p>\n

    In the case of continuous variables, this sum –> an integration.<\/p>\n

    Conditional probability<\/strong><\/p>\n

    Use a vertical line to represent this: p(y|x) means p(y=y<\/em>) given that x=x<\/em>.<\/p>\n

    So now we have marginal probabilities<\/em> (p(x)<\/strong>), conditional probabilities<\/em> (p(x|y)<\/strong>), and joint probabilities<\/em> (p(x,y)<\/strong>).<\/p>\n

    The chain rule of conditional probability<\/strong><\/p>\n

    If we want to decompose a joint probability distribution into components, we can repeatedly crack open our joint distribution using conditional probabilities:<\/p>\n

    P(a,b) = P(a|b)P(b)<\/p>\n

    So the probability of observing a AND b, is equal to the probability of observing a given that b has occured, multiplied by the probability of observing b in the first place.<\/p>\n

    We can do this recursively:<\/p>\n

    P(a,b,c) = P(a|b,c)P(b,c)<\/p>\n

    P(a,b,c) = P(a|b,c)P(b|c)P(c)<\/p>\n

    Disclaimer: I’m not clear yet why this is particularly useful.<\/p>\n

    Independence<\/strong><\/p>\n

    If two variables are independent, their joint probability is merely<\/p>\n

    p(x=x<\/em>,y=y<\/em>) = p(x=x<\/em>) * p(y=y<\/em>)<\/p>\n

    This leads on to the notion of\u00a0conditional independence<\/strong>.\u00a0There may be situations where we can’t predict A from B, but knowing the value of a third variable, C, gives us some predictive power. Here’s an (abridged, emphasised) example from stack exchange<\/a>:<\/p>\n

    “Say you roll a blue die and a red die. The two results are independent<\/strong> of each other. Now you tell me that the blue result isn’t a\u00a06<\/span><\/span><\/span><\/span>\u00a0and the red result isn’t a\u00a01<\/span><\/span><\/span><\/span>. You’ve given me new information, but that hasn’t affected the independence of the results<\/strong>. By taking a look at the blue die, I can’t gain any knowledge about the red die; after I look at the blue die I will still have a probability of\u00a01<\/span>\/5<\/span><\/span><\/span><\/span><\/span><\/span>\u00a0for each number on the red die except\u00a01<\/span><\/span>. So the probabilities for the results are conditionally independent<\/strong> given the information you’ve given me. But if instead you tell me that the sum of the two results is even, this allows me to learn a lot about the red die by looking at the blue die.<\/strong> For instance, if I see a\u00a03<\/span><\/span><\/span><\/span>\u00a0on the blue die, the red die can only be\u00a01<\/span><\/span><\/span><\/span>,\u00a03<\/span><\/span><\/span><\/span>\u00a0or\u00a05<\/span><\/span><\/span><\/span>. So in this case the probabilities for the results are not conditionally independent given this other information that you’ve given me. This also underscores that conditional independence is always relative to the given condition — in this case, the results of the dice rolls are conditionally independent with respect to the event “the blue result is not\u00a06<\/span><\/span>\u00a0and the red result is not\u00a01<\/span><\/span><\/span><\/span>“, but they’re not conditionally independent with respect to the event “the sum of the results is even”<\/p>\n

    Expectation, variance<\/strong><\/p>\n

    The expectation is basically the mean of a probability distribution – it’s the most likely value of\u00a0x<\/em> to obtain if we draw a random value. We denote it with a fancy E:<\/p>\n

    \"$\\mathbb{E}$<\/p>\n

    For discrete variables it’s simply:<\/p>\n

    \"\\mathbb{E}<\/p>\n

    Meaning the expectation of a random\u00a0x<\/em> from the probability distribution f(x) is the sum over the possible values of x multiplied by their respective probabilities. Pretty intuitive.<\/p>\n

    For continuous variables, we use… you guessed it, an integral version of the same:<\/p>\n

    \"\\mathbb{E}<\/p>\n

    If the expectation is like the mean of a distribution, the variance<\/strong> is equivalent to… well… the variance. It means exactly the same thing in this context: how much things vary around the expectation, which is to say, the ‘peakiness’ of the probability distribution.<\/p>\n

    \"Var(f(x))<\/p>\n

    This form\u00a0is intuitively\u00a0similar to the definition of the standard deviation: we are taking some average difference between each\u00a0data point and the mean (expectation in this case).<\/p>\n

    In the figure below, \u00a0\"x\"\u00a0defines the expectation, and\u00a0\"\\gamma\"\u00a0the variance:<\/p>\n

    \"360px-Cauchy_pdf\"<\/a><\/p>\n

    (from Wikipedia<\/a>)<\/p>\n

    Note that since the probability distribution has to integrate to 1,\u00a0changing the variance also changes the height of the distribution<\/em>.<\/p>\n

    This seems like a good time to introduce the dirac delta function<\/strong>, which is a weird function that exemplifies the above consideration: it has zero variance, and is 0 everywhere apart from x=0, but it still integrates to 1:<\/p>\n

    \"dirac3\"<\/a><\/p>\n

    \"dirac2\"<\/a><\/p>\n

    \"dirac\"<\/a><\/p>\n

    It’s useful for denoting very short events in time – we can use\u00a0it to denote a spike (action potential) in a neuron, for instance.<\/p>\n

    Covariance<\/strong><\/p>\n

    An important concept: how much two variance in two variables relate to one another (correlation is the covariance corrected for the magnitude of each variable).<\/p>\n

    This is similar to calculating the\u00a0variance (see above), but we sub in another variable instead of squaring everything:<\/p>\n

    \"Cov(f(x),g(y))<\/p>\n

    Covariance will be positive if two things tend to vary in the same direction (e.g. height and weight) and negative if two variables tend to vary oppositely (e.g. height and number of books you have to stand on to reach the top shelf). Note that covariance is a\u00a0linear<\/em> construct: two variables can be related in a funky non-linear way, and still have zero covariance.<\/p>\n

    A bunch of distributions<\/strong><\/p>\n

    Bernoulli<\/strong><\/em><\/p>\n

    A discrete probability distribution for describing the probability over a single binary variable, the most classic example of which would be heads or tails on a coin. It has a single parameter, \"\\phi\",which says how likely the binary variable is to attain state 1 (for an unbiased coin, \"\\phi\" is 0.5).<\/p>\n

    The\u00a0multinoulli<\/strong><\/em> distribution does the same thing over k states.<\/p>\n

    Gaussian (=normal)<\/strong><\/em><\/p>\n

    The old favourite. Random variables tend to the Gaussian distribution due to the central limit theorem\u00a0<\/a>. Despite being exceedingly common, you may never have actually seen the formula written down:<\/p>\n

    \"N(x;\\mu,\\sigma^2)=\\sqrt{\\frac<\/p>\n

    Phew, what a mouthful.\u00a0\"\\mu\"\u00a0is the mean,\u00a0\"\\sigma\"\u00a0is the standard deviation (\"\\sigma^2\"=variance).<\/p>\n

    They have a nice figure in the book (3.1):\"fig3.1\"<\/a><\/p>\n

    We can bang in lots more variables to the Gaussian distribution to produce a multivariate normal\u00a0<\/strong><\/em>distribution. This involves replacing our scalar mean with a vector of means (\u03bc –> \u03bc<\/strong>) and our variance with a covariance matrix, \u03a3.<\/strong><\/p>\n

    An example plotted for 2 dimensions;<\/p>\n

    \"MultivariateNormal\"<\/a><\/p>\n

    (from Wikipedia<\/a>)<\/p>\n

    Exponential & laplace<\/strong><\/em><\/p>\n

    Used to give us nice and spiky distributions. For (negative) exponential, this spike is at 0, whereas the laplace distribution gives us the ability to shift it around based upon the point\u00a0\u03bc. Both are characterised by a steepness\u00a0parameter \u03b3 (b in the laplacian graph):<\/p>\n

    Exponential:<\/p>\n

    \"exp\"<\/a><\/p>\n

    NB Exponential is a special subset of gamma distributions<\/a> (which look a bit like a mixture between an exponential and a normal distribution- they have a peak, and a long tail).<\/p>\n

    Laplace:<\/p>\n

    \"lapl\"<\/a><\/p>\n

    (from Wikipedia<\/a>)<\/p>\n

    A mixture of distributions<\/strong><\/p>\n

    We can combine a\u00a0multinoulli<\/strong> distribution with some set of the above to generate a distribution mixture. The multinoulli specifies which distribution we are likely to be in on each trial.<\/p>\n

    \"P(x)<\/p>\n

    Where P(c) is the probability distribution over component distributions. This tells us that the probability of obtaining a certain value (x) is a product of\u00a0the probability that we are in a certain state, and the probability of x given that state.<\/p>\n

    A common use of this is in\u00a0Gaussian mixture models<\/strong>. We use these in neuroscience when we think that a population might be composed of multiple groups.\u00a0For instance, the distribution of heights across the population is best described by two Gaussians, one for men, one for women. In this case gender is a\u00a0latent, or hidden variable<\/strong> – an extra piece of information not inherent in the data, but which allows us to understand our data better.<\/p>\n

    \"heigh\"<\/a><\/p>\n

    (from Disentangling Gaussians<\/a>)<\/p>\n

    Useful functions\u00a0<\/strong><\/p>\n

    When working with neural networks, we use functions which curtail the firing rates of our neurons to positive values (because neurons can’t fire at a negative rate).<\/p>\n

    Two useful ones: the sigmoid<\/strong> (this is what everybody used to use) and the rectifier<\/strong> &\u00a0soft plus<\/b> (I believe these is more in fashion now).<\/p>\n

    Sigmoid<\/strong><\/p>\n

    \"sig\"<\/a><\/p>\n

    Rectifier & softplus<\/strong><\/p>\n

    \"rect\"<\/a><\/strong><\/p>\n

    (from Wikipedia<\/a>)<\/p>\n

    Transforming distributions<\/strong><\/p>\n

    Sometimes we want to project one distribution through another. One common example from model-fitting: we want to fit a parameter in an unbounded space (-infinity to +infinity), but then when we use this parameter, we need to constrain to, for instance between 0 and 1 (if, for instance, we are fitting a learning rate). In this case, we’d fit the parameter then\u00a0pass it through a sigmoid.<\/strong><\/p>\n

    We can derive the new distribution by multiplying the original distribution by the derivative of the transformation:<\/p>\n

    q(x) = p(x) * g'(x)<\/p>\n

    Bayes rule<\/strong><\/p>\n

    A favourite of ours, it’s:<\/p>\n

    \"p(a|b)<\/p>\n

    A derivation:<\/p>\n

      \n
    1. p(x,y) = p(x|y)p(y)<\/li>\n<\/ol>\n

      2. p(x|y) = p(x,y)\/p(y)<\/p>\n

      Doing the same in reverse:<\/p>\n

      3. p(y,x) = p(y|x)p(x)<\/p>\n

      The substitute in\u00a0p(y|x)p(x) for p(y,x) in eq. 2, and we get:<\/p>\n

      p(x|y)=p(y|x)p(x) \/ p(y)<\/p>\n

      Woohoo!<\/p>\n

      A little bit of\u00a0information theory<\/strong><\/p>\n

      Information is defined as the amount of uncertainty we reduce when we\u00a0learn something. If you know something already, being told it again has an information content of zero. If something is very unlikely and you find out it has occurred, you gain a lot of information. If something was very likely and you find out it has occurred, you gain a little information.<\/p>\n

      Formally:<\/p>\n

      \"I(x)<\/p>\n

      So information is negatively related to probability: the more probable something is, the less we learn when we find out the outcome. Depending upon the type of logarithm you use here, information comes in different units:\u00a0nets<\/b> (for the natural log),\u00a0or bits <\/strong>or shannons\u00a0<\/strong>(log2).<\/p>\n

      We quantify the amount of uncertainty in a whole probability distribution (rather than merely associated with a single event, x) using the Shannon entropy<\/strong>:<\/p>\n

      \"H(x)<\/strong><\/p>\n

      Which is basically the expectation (i.e. the average) of the information.<\/p>\n

      Kullback-Leiber (KL) divergence<\/strong><\/p>\n

      A phrase which many have heard and, I suspect, few have understood. The KL-divergence quantifies the\u00a0difference between two distributions<\/strong>. More precisely, it quantifies\u00a0how much\u00a0information we use if we try and approximate one distribution with another distribution<\/strong>. Interestingly, it’s not symmetric – the amount of info we lose when we use A to approximate B \u2260 the amount of info lost when we use B to approximate A.<\/p>\n

      It has quite a simple formula:<\/p>\n

      \"D_{KL}(P\\parallel<\/p>\n

      Basically the average difference in probabilities of observing each sample based upon the two distributions!<\/p>\n

      Note that this means that the KL captures both\u00a0differences in means and\u00a0variances of distributions.<\/strong><\/p>\n

      A related concept is the cross-entropy:<\/p>\n

      \"Cross<\/p>\n

      If the two variables are completely independent, p(x,y) = p(x)p(y) [that’s the product rule], and we end up with\u00a0log(1), which is 0. If p(x,y) > p(x)p(y). then the joint probability is higher than the product of the independent probabilities, implying that the variables are related- and yielding a positive cross entropy.<\/p>\n

      Graphical\u00a0models<\/strong><\/p>\n

      Basically a way of drawing out probabilistic relationships. They consist of nodes<\/strong> (distributions, or functions), and\u00a0edges<\/strong> (links between them).<\/p>\n

      They can be\u00a0directed<\/strong>, in which case we are visualising conditional relationships, or they can be\u00a0undirected<\/strong>, which allows us to mix in non-probabilistic functions.<\/p>\n

      In undirected models, nodes\u00a0which are linked together by edges are referred to as\u00a0cliques<\/strong>.<\/p>\n

      A directed graph:<\/p>\n

      \"Graphical_model_example\"<\/a><\/p>\n

      (Wikipedia<\/a>)<\/p>\n

      One common usage is to describe transitions in a Markov model:<\/p>\n

      \"gmm\"<\/a><\/p>\n

      (Wikipedia<\/a>)<\/p>\n

      The advantage of having a graphical model is that it allows us to depict dependencies. If we learn these dependencies, then we save ourselves the effort of learning the whole covariance matrix across all of the variables. This is probably the kind of trick the brain uses to make things easier…<\/p>\n","protected":false},"excerpt":{"rendered":"

      As part of the Max Planck Centre’s Deep Learning reading group, I’m working my way through the soon-to-be released Deep Learning textbook. These posts are my informal notes for each chapter. Find my notes from chapter 1 here.\u00a0Thanks again to Zeb & Toby for edits. This week we were back on slightly firmer ground, dealing […]<\/p>\n","protected":false},"author":1,"featured_media":185,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"hide_page_title":"","_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[6],"tags":[],"class_list":["post-184","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-neural-networks"],"yoast_head":"\nDeep Learning Textbook Chapter 2: Probability - Archy de Berker<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deep Learning Textbook Chapter 2: Probability - Archy de Berker\" \/>\n<meta property=\"og:description\" content=\"As part of the Max Planck Centre’s Deep Learning reading group, I’m working my way through the soon-to-be released Deep Learning textbook. These posts are my informal notes for each chapter. Find my notes from chapter 1 here.\u00a0Thanks again to Zeb & Toby for edits. This week we were back on slightly firmer ground, dealing […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\" \/>\n<meta property=\"og:site_name\" content=\"Archy de Berker\" \/>\n<meta property=\"article:published_time\" content=\"2016-05-19T13:14:25+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-11-06T16:05:52+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1\" \/>\n\t<meta property=\"og:image:width\" content=\"1140\" \/>\n\t<meta property=\"og:image:height\" content=\"440\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"archy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@archydeb\" \/>\n<meta name=\"twitter:site\" content=\"@archydeb\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"archy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"19 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\"},\"author\":{\"name\":\"archy\",\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d\"},\"headline\":\"Deep Learning Textbook Chapter 2: Probability\",\"datePublished\":\"2016-05-19T13:14:25+00:00\",\"dateModified\":\"2021-11-06T16:05:52+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\"},\"wordCount\":2659,\"publisher\":{\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d\"},\"image\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1\",\"articleSection\":[\"Neural networks\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\",\"url\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\",\"name\":\"Deep Learning Textbook Chapter 2: Probability - Archy de Berker\",\"isPartOf\":{\"@id\":\"https:\/\/deberker.com\/archy\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1\",\"datePublished\":\"2016-05-19T13:14:25+00:00\",\"dateModified\":\"2021-11-06T16:05:52+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage\",\"url\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1\",\"width\":1140,\"height\":440},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/deberker.com\/archy\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Deep Learning Textbook Chapter 2: Probability\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/deberker.com\/archy\/#website\",\"url\":\"https:\/\/deberker.com\/archy\/\",\"name\":\"Archy de Berker\",\"description\":\"Building things with data\",\"publisher\":{\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/deberker.com\/archy\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d\",\"name\":\"archy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2021\/09\/freelance-logo.png?fit=359%2C311&ssl=1\",\"contentUrl\":\"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2021\/09\/freelance-logo.png?fit=359%2C311&ssl=1\",\"width\":359,\"height\":311,\"caption\":\"archy\"},\"logo\":{\"@id\":\"https:\/\/deberker.com\/archy\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/archydeb\"],\"url\":\"https:\/\/deberker.com\/archy\/author\/archy\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Deep Learning Textbook Chapter 2: Probability - Archy de Berker","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/","og_locale":"en_US","og_type":"article","og_title":"Deep Learning Textbook Chapter 2: Probability - Archy de Berker","og_description":"As part of the Max Planck Centre’s Deep Learning reading group, I’m working my way through the soon-to-be released Deep Learning textbook. These posts are my informal notes for each chapter. Find my notes from chapter 1 here.\u00a0Thanks again to Zeb & Toby for edits. This week we were back on slightly firmer ground, dealing […]","og_url":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/","og_site_name":"Archy de Berker","article_published_time":"2016-05-19T13:14:25+00:00","article_modified_time":"2021-11-06T16:05:52+00:00","og_image":[{"width":1140,"height":440,"url":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","type":"image\/jpeg"}],"author":"archy","twitter_card":"summary_large_image","twitter_creator":"@archydeb","twitter_site":"@archydeb","twitter_misc":{"Written by":"archy","Est. reading time":"19 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#article","isPartOf":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/"},"author":{"name":"archy","@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d"},"headline":"Deep Learning Textbook Chapter 2: Probability","datePublished":"2016-05-19T13:14:25+00:00","dateModified":"2021-11-06T16:05:52+00:00","mainEntityOfPage":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/"},"wordCount":2659,"publisher":{"@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d"},"image":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","articleSection":["Neural networks"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/","url":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/","name":"Deep Learning Textbook Chapter 2: Probability - Archy de Berker","isPartOf":{"@id":"https:\/\/deberker.com\/archy\/#website"},"primaryImageOfPage":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage"},"image":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage"},"thumbnailUrl":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","datePublished":"2016-05-19T13:14:25+00:00","dateModified":"2021-11-06T16:05:52+00:00","breadcrumb":{"@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#primaryimage","url":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","contentUrl":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","width":1140,"height":440},{"@type":"BreadcrumbList","@id":"https:\/\/deberker.com\/archy\/deep-learning-textbook-chapter-2-probability\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/deberker.com\/archy\/"},{"@type":"ListItem","position":2,"name":"Deep Learning Textbook Chapter 2: Probability"}]},{"@type":"WebSite","@id":"https:\/\/deberker.com\/archy\/#website","url":"https:\/\/deberker.com\/archy\/","name":"Archy de Berker","description":"Building things with data","publisher":{"@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/deberker.com\/archy\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/01cf8dd0f94a4ba124b26eeeeb59e67d","name":"archy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/image\/","url":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2021\/09\/freelance-logo.png?fit=359%2C311&ssl=1","contentUrl":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2021\/09\/freelance-logo.png?fit=359%2C311&ssl=1","width":359,"height":311,"caption":"archy"},"logo":{"@id":"https:\/\/deberker.com\/archy\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/x.com\/archydeb"],"url":"https:\/\/deberker.com\/archy\/author\/archy\/"}]}},"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/i0.wp.com\/deberker.com\/archy\/wp-content\/uploads\/2016\/05\/deepL_C2Graphic.jpg?fit=1140%2C440&ssl=1","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p4cGwe-2Y","jetpack-related-posts":[],"post_mailing_queue_ids":[],"_links":{"self":[{"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/posts\/184","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/comments?post=184"}],"version-history":[{"count":16,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/posts\/184\/revisions"}],"predecessor-version":[{"id":1167,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/posts\/184\/revisions\/1167"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/media\/185"}],"wp:attachment":[{"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/media?parent=184"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/categories?post=184"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/deberker.com\/archy\/wp-json\/wp\/v2\/tags?post=184"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}