Marginal Probability Density Function Continuous Uniform

Pairs of Random Variables

Scott L. Miller , Donald Childers , in Probability and Random Processes (Second Edition), 2012

5.6 Independent Random Variables

The concept of independent events was introduced in Chapter 2. In this section, we extend this concept to the realm of random variables. To make that extension, consider the events A = {X ≤ x} and B = {Y≤y} related to the random variables X and Y. The two events A and B are statistically independent if Pr(A, B) = Pr(A)Pr(B). Restated in terms of the random variables, this condition becomes

(5.37) $Pr\left(X\le x,Y\le y\right)=Pr\left(X\le x\right)Pr\left(Y\le y\right)\Rightarrow F_{X,Y}\left(x,y\right)=F_X\left(x\right)F_Y\left(y\right).$

Hence, two random variables are statistically independent if their joint CDF factors into a product of the marginal CDFs. Differentiating both sides of this equation with respect to both x and y reveals that the same statement applies to the PDF as well. That is, for statistically independent random variables, the joint PDF factors into a product of the marginal PDFs:

(5.38) $f_{X,Y}\left(x,y\right)=f_X\left(x\right)f_Y\left(y\right).$

It is not difficult to show that the same statement applies to PMFs as well. The preceding condition can also be restated in terms of conditional PDFs. Dividing both sides of Equation (5.38) by f_X (x) results in

(5.39) $f_{X|Y}\left(x|y\right)=f_Y\left(y\right).$

A similar result involving the conditional PDF of X given Y could have been obtained by dividing both sides by the PDF of Y. In other words, if X and Y are independent, knowing the value of the random variable X should not change the distribution of Y and vice versa.

Example 5.13

Returning once again to the joint PDF of Example 5.10, we saw in that example that the marginal PDF of X is

$f_X\left(x\right)=\frac{1}{\sqrt{2\pi }}exp\left(-\frac{x^2}{2}\right),$

while the conditional PDF of X given Y is

$f_{X|Y}\left(x|y\right)=\sqrt{\frac{2}{3\pi }}exp\left(-\frac{2}{3}{\left(x-\frac{y}{2}\right)}^2\right).$

Clearly, these two random variables are not independent.

Example 5.14

Suppose the random variables X and Y are uniformly distributed on the square defined by 0 ≤ x, y ≤ 1. That is

$f_{X,Y}\left(x,y\right)=\{\begin{array}{cc}1,& 0\le x,y\le 1,\\ 0,& \text{otherwise}.\end{array}$

The marginal PDFs of X and Y work out to be

$\begin{array}{cc}{f}_X\left(x\right)=\{\begin{array}{cc}1,& 0\le x,\le 1,\\ 0,& \text{otherwise},\end{array}& f_Y\left(y\right)=\{\begin{array}{cc}1,& 0\le y,\le 1,\\ 0,& \text{otherwise}.\end{array}\end{array}$

These random variables are statistically independent since f_{X , Y} (x, y) = f_X (x)f_Y (y).

Theorem 5.5: Let X and Y be two independent random variables and consider forming two new random variables U = g ₁ (X) and V = g ₂(Y). These new random variables U and V are also independent.

Proof: To show that U and V are independent, consider the events A = {U≤u} and B = {V≤v}. Next define the region R _u to be the set of all points x such that g ₁ (x) ≤ u. Similarly, define R to be the set of all points y such that g ₂(y) ≤ v. Then

$Pr\left(U\le u,V\le v\right)=Pr\left(X\in R_u,Y\in R_v\right)=\underset{R_v}{\int }\underset{R_u}{\int }{f}_{X,Y}\left(x,y\right)\text{d}x\text{d}y.$

Since X and Y are independent, their joint PDF can be factored into a product of marginal PDFs resulting in

$\begin{array}{c}Pr\left(U\le u,V\le v\right)=\underset{R_u}{\int }{f}_X\left(x\right)\text{d}x\underset{R_v}{\int }{f}_Y\left(y\right)\text{d}y=Pr\left(X\in R_u\right)Pr\left(Y\in R_v\right).\\ =Pr\left(U\le u\right)Pr\left(V\le v\right)\end{array}$

Since we have shown that F _{U, V} (u, v) = F_U (u)FV(v), the random variables U and V must be independent.

Another important result deals with the correlation, covariance, and correlation coefficients of independent random variables.

Theorem 5.6: If X and Y are independent random variables, then E[XY] = μ _Xμ_Y , Cov(X, Y) = 0, and ρ _{X ,Y} = 0.

Proof: $E\left[XY\right]=\int xy{f}_{X,Y}\left(x,y\right)\text{d}x\text{d}y=\int x{f}_X\left(x\right)\text{d}x\int y{f}_Y\left(y\right)\text{d}y={\mu }_X{\mu }_Y.$

The conditions involving covariance and correlation coefficient follow directly from this result.

Therefore, independent random variables are necessarily uncorrelated, but the converse is not always true. Uncorrelated random variables do not have to be independent as demonstrated by the next example.

Example 5.15

Consider a pair of random variables X and Y that are uniformly distributed over the unit circle so that

$f_{X,Y}\left(x,y\right)=\{\begin{array}{cc}\frac{1}{\pi },& x^2+y^2\le 1,\\ 0,& \text{otherwise}\text{.}\end{array}$

The marginal PDF of X can be found as follows:

$\begin{array}{cc}{f}_X\left(x\right)={\int }_{-\infty }^{\infty }{f}_{X,Y}\left(x,y\right)\text{d}y={\int }_{-\sqrt{1-x^2}}^{\sqrt{1-x^2}}\frac{1}{\pi }\text{d}y=\frac{2}{\pi }\sqrt{1-x^2},& -1\le x\le 1.\end{array}$

By symmetry, the marginal PDF of Y must take on the same functional form. Hence, the product of the marginal PDFs is

$\begin{array}{cc}{f}_X\left(x\right)f_Y\left(y\right)=\frac{4}{{\pi }^2}\sqrt{\left(1-x^2\right)\left(1-y^2\right),}& -1\le x,y\le 1.\end{array}$

Clearly, this is not equal to the joint PDF, and therefore, the two random variables are dependent. This conclusion could have been determined in a simpler manner. Note that if we are told that X = 1, then necessarily Y = 0, whereas if we know that X = 0, then Y can range anywhere from –1 to 1. Therefore, conditioning on different values of X leads to different distributions for Y.

Next, the correlation between X and Y is calculated.

$E\left[XY\right]=\underset{x^2+y^2}{\int }\underset{\le 1}{\int }\frac{xy}{\pi }\text{d}x\text{d}y=\frac{1}{\pi }{\int }_{-1}^{1}x\left[{\int }_{-\sqrt{1-x^2}}^{\sqrt{1-x^2}}ydy\right]\text{dx}.$

Since the inner integrand is an odd function (of y) and the limits of integration are symmetric about zero, the integral is zero. Hence, E[XY] = 0. Note from the marginal PDFs just found that both X and Y are zero-mean. So, it is seen for this example that while the two random variables are uncorrelated, they are not independent.

Example 5.16

Suppose we wish to use MATLAB to generate samples of a pair of random variables (X, Y) that are uniformly distributed over the unit circle. That is, the joint PDF is

$f_{X,Y}\left(x,y\right)=\{\begin{array}{cc}\frac{1}{\pi },& x^2+y^2<1,\\ 0,& \text{otherwise}.\end{array}$

If we generated two random variables independently according to the MATLAB code: X=rand(1); Y=rand(1); this would produce a pair of random variables uniformly distributed over the square 0 < x < 1, 0 < y < 1. One way to achieve the desired result is to generate random variables uniformly over some region which includes the unit circle and then only keep those pairs of samples which fall inside the unit circle. In this case, it is straightforward to generate random variables which are uniformly distributed over the square, –1 < x < 1,–1 < y < 1, which circumscribes the unit circle. Then we keep only those samples drawn from within this square that also fall within the unit circle. The code that follows illustrates this technique. We also show how to generate a three-dimensional plot of an estimate of the joint PDF from the random data generated. To get a decent estimate of the joint PDF, we need to generate a rather large number of samples (we found that 100,000 worked pretty well). This requires that we create and perform several operations on some very large vectors. Doing so tends to make the program run slowly. In order to speed up the operation of the program, we choose to create shorter vectors of random variables (1000 in this case) and then repeat the procedure several times (100 in this case). Although this makes the code a little longer and probably a little harder to follow, by avoiding the creation of very long vectors, it substantially speeds up the program. The results of this program are shown in Figure 5.3.

Figure 5.3. Estimate of the joint PDF of a pair of random variables uniformly distributed over the unit circle from the data generated in Example 5.16.

(For color version of this figure, the reader is referred to the web version of this chapter.)

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123869814500084

Advances in Geophysics

Wojciech De¸bski , in Advances in Geophysics, 2010

4.8.5 PDF in the $\mathcal{D}$ space

Although solving inverse problems we are most commonly interested in the m parameters, occasionally we may need also to explore the marginal PDF in the $\mathcal{D}$ space. The general formula for this marginal PDF is given in Eq. (67). Let us consider now two special cases of ${\sigma }_{\mathrm{pos}}^d$ (d).

First let us consider the case when no theoretical information is available:

(114) ${\sigma }_{\mathrm{th}}\left(\text{m,d}\right)=\mu \left(\text{m d}\right)$

so the integral in Eq. (67) evaluates to a constant and ${\sigma }_{\mathrm{pos}}^d$ (d) reads

(115) ${\sigma }_{\mathrm{pos}}^d\left(\text{d}\right)={\sigma }_{\exp }^d\left({\text{d};\text{d}}^{\mathrm{obs}}\right)\frac{{\sigma }_{apr}^d\left({\text{d};\text{d}}^{\mathrm{apr}}\right)}{{\mu }_d\left(\text{d}\right)}.$

The a posteriori PDF is given by product of the observational and a priori PDFs as expected.

Next, let us assume that there is no observational data so

(116) ${\sigma }_{exp}^d\left({\text{d};\text{d}}^{\mathrm{obs}}\right)={\mu }_d\left(\text{d}\right).$

In this case we arrive at the following formula for ${\sigma }_{\mathrm{pos}}^d$ (d):

(117) $\begin{array}{cc}{\sigma }_{pos}^d\left(\text{d}\right)={\sigma }_{apr}^d\left({\text{d};\text{d}}^{\mathrm{apr}}\right)& {\displaystyle \underset{\mathcal{M}}{\int }{\sigma }_{apr}^m}\left(\text{m}\right)\frac{{\sigma }_{\mathrm{th}}\left(\text{m,d}\right)}{\mu \left(\text{m d}\right)}\text{d}\text{m}.\end{array}$

This formula is the counterpart of Eq. (63) and shows that in the case of missing observational data the a posteriori PDF in the data space is the product of the a priori PDF in $\mathcal{D}$ space and the function which describes how the a priori information about m is projected into the $\mathcal{D}$ space.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S0065268710520016

Multiple Random Variables

Scott L. Miller , Donald Childers , in Probability and Random Processes (Second Edition), 2012

Section 6.1 Joint and Conditional PMFs, CDFs, and PDFs

6.1

Suppose we flip a coin three times, thereby forming a sequence of heads and tails. Form a random vector by mapping each outcome in the sequence to 0 if a head occurs or to 1 if a tail occurs.

(a)

How many realizations of the vector may be generated? List them.

(b)

Are the realizations independent of one another?

6.2

Let X = [X_l , X ₂, X ₃]^T represent a three-dimensional vector of random variables that is uniformly distributed over a cubical region

$f_X\left(x\right)=\{\begin{array}{cc}c,& \left|x_1\right|\le 1,\left|x_2\right|\le 1,\left|x_3\right|\le 1,\\ 0,& \text{otherwise}.\end{array}$

(a)

Find the constant c.

(b)

Find the marginal PDF for a subset of two of the three random variables. For example, find f _{X 1} , _{X 2} (x ₁, x ₂).

(c)

Find the marginal PDF for one of the three random variables. That is, find f _{X 1} (x ₁).

(d)

Find the conditional PDFs $f_{X_1|X_2,X_3}\left(x_1|x_2,x_3\right)$ and $f_{X_1,X_2|X_3}\left(x_1,x_2|x_3\right)$ .

(e)

Are the X_i independent?

6.3

Suppose a point in two-dimensional Cartesian space, (X, Y), is equally likely to fall anywhere on the semicircle defined by X ² + Y ² = 1 and Y ≥ 0. Find the PDF of Y, f_Y (y).

6.4

Suppose a point in three-dimensional Cartesian space, (X, Y, Z), is equally likely to fall anywhere on the surface of the hemisphere defined by X ² + Y ² + Z ² = 1 and Z ≥ 0.

(a)

Find the PDF of Z, f_z (z).

(b)

Find the joint PDF of X and Y, f _X,Y (x, y).

6.5

Suppose N ₁ is a discrete random variable equally likely to take on any integer in the set {1, 2, 3}. Given that N ₁ = n ₁, the random variable N ₂ is equally likely to take on any integer in the set {1, 2, …, n ₁}. Finally, given that N ₂ = n ₂, the random variable N ₃ is equally likely to take on any integer in the set {1, 2, …, n ₂}.

(a)

Find the two-dimensional joint PMF, P _{N 1,N 2} (n ₁, n ₂).

(b)

Find the three-dimensional joint PDF, P _{N 1,N 2,N 3} (n ₁ n ₂, n ₃).

(c)

Find the marginal PDFs, P _{N 2} (n ₂) and P _N3 (n ₃).

(d)

What are the chances that none of the three random variables are equal to 1?

6.6

Let X = [X ₁, X ₂, X ₃]^T represent a three-dimensional vector of random variables that is uniformly distributed over the unit sphere. That is,

$f_X\left(x\right)=\{\begin{array}{cc}c,& \Vert x\Vert \le 1,\\ 0,& \Vert x\Vert >1.\end{array}$

(a)

Find the constant c.

(b)

Find the marginal PDF for a subset of two of the three random variables. For example, find f _{X 1,X 2} (x ₁, x ₂).

(c)

Find the marginal PDF for one of the three random variables. That is, find f _{X 1} (x ₁).

(d)

Find the conditional PDFs _{f X1|X 2,X 3} (x ₁|x ₂,x ₃) and _{f X1,X 2|X 3} (x ₁,x ₂|x ₃)

Extra: Can you extend this problem to N-dimensions?

6.7

Let X = [X ₁, X ₂, …, X_N ]^T represent an N-dimensional vector of random variables that is uniformly distributed over the region x ₁+ x ₂ + … + X_N ≤ 1, x _i ≥ 0, i = 1, 2, …, N. That is

$f_X\left(x\right)=\{\begin{array}{cc}c,& {\sum }_{i=1}^N{x}_i\le 1,x_i\ge 0,\\ 0,& \text{otherwise}.\end{array}$

(a)

Find the constant c.

(b)

Find the marginal PDF for a subset of M of the N random variables.

(c)

Are the X_i independent? Are the X_i identically distributed?

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123869814500096

Additional topics in probability

Kandethody M. Ramachandran , Chris P. Tsokos , in Mathematical Statistics with Applications in R (Third Edition), 2021

3.6 Chapter summary

In this chapter we looked at some special probability distribution functions that arise in practice. It should be noted that we discussed only a few of the important probability distributions. There are many other discrete and continuous distributions that will be useful and appropriate in particular applications. Some of them are given in Appendix A3. A larger list of probability distributions can be found at http://www.causascientia.org/math_stat/Dists/Compendium.pdf, among many other places. For more than one random variable, we learned the behavior of joint probability distributions. We also saw how to find the probability (mass) density function and cumulative distribution for the functions of a random variable. Limit theorems are a crucial part of probability theory. We have introduced the Chebyshev inequality, the law of large numbers, and the CLT for the random variables.

We now list some of the key definitions introduced in this chapter:

■

Bernoulli probability distribution

■

Binomial experiment

■

Poisson probability distribution

■

Probability distribution

■

Normal (or Gaussian) probability distribution

■

Standard normal random variable

■

Gamma probability distribution

■

Exponential probability distribution

■

Chi-square (χ²) distribution

■

Joint pdf

■

Bivariate probability distributions

■

Marginal pdf

■

Conditional probability distribution

■

Independence of two random variables

■

Expected value of a function of bivariate random variables

■

Conditional expectation

■

Covariance

■

Correlation coefficient

In this chapter, we have also learned the following important concepts and procedures:

■

Mean, variance, and mgf of a binomial random variable

■

Mean, variance, and mgf of a Poisson random variable

■

Poisson approximation to the binomial probability distribution

■

Mean, variance, and mgf of a uniform random variable

■

Mean, variance, and mgf of a normal random variable

■

Mean, variance, and mgf of a gamma random variable

■

Mean, variance, and mgf of an exponential random variable

■

Mean, variance, and mgf of a chi-square random variable

■

Properties of expected value

■

Properties of the covariance and correlation coefficient

■

Procedure to find the cdf of a function of random variable using the method of distribution functions

■

The pdf of Y  = g(X), where g is differentiable and monotone increasing or decreasing

■

The pdf of Y  = g(X), using the probability integral transformation

■

The transformation method to find the pdf of Y  = g(X ₁, …, X _n )

■

Chebyshev's theorem

■

Law of large numbers

■

CLT

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128178157000038

Beyond Wavelets

Bertrand Bénichou , Naoki Saito , in Studies in Computational Mathematics, 2003

9.3.2 Statistical Independence

The statistical independence of the coordinates of Y ∈ R ⁿ means

$f_Y\left(y\right)=f_{y1}\left(y_1\right)f_{y2}\left(y_2\right)\cdots f_{yn}\left(y_n\right),$

where fY _k (y_k ) is a one-dimensional marginal pdf of f_Y. Statistical independence is a key property for compressing and modeling a stochastic process because: 1) an n-dimensional stochastic process of interest can be modeled as a set of one-dimensional processes; and 2) damage of one coordinate does not propagate to the others. Of course, in general, it is difficult to find a truly statistically independent coordinate system for a given stochastic process. Such a coordinate system may not even exist for a given stochastic process. Therefore, the next best thing we can do is to find the least statistically-dependent coordinate system within a basis dictionary. Naturally, then, we need to measure the "closeness" of a coordinate system (or random variables) Y ₁,…, Y _n to the statistical independence. This can be measured by mutual information or relative entropy between the true pdf f_Y and the product of its marginal pdfs:

$I\left(Y\right)\triangleq \int f_Y\left(y\right)log\frac{f_Y\left(y\right)}{{\prod }_{i=1}^{n}f{y}_i\left(y_i\right)}dy=-H\left(Y\right)+\sum _{i=1}^{n}H\left(Y_i\right),$

where H(Y) and H(Y _i) are the differential entropy of Y and Y _i respectively:

$H\left(Y\right)=\begin{array}{cc}-\int f_Y\left(y\right)log{f}_Y\left(y\right)dy,& H\left(Y_i\right)-\int f_{Y_i}\left(y_i\right)log{f}_{Y_i}\left(y_i\right)d{y}_i.\end{array}$

We note that I(Y) > 0, and I(Y) = 0 if and only if the components of Y are mutually independent. See [5] for more details of the mutual information.

Suppose Y = B ^-1 X and B ∈ GL(n, R) with det(B) = ±1. We denote this set of matrices by SL±(n, R). Note that the usual SL(n, R) is a subset of SL± (n, R). Then, we have

$I\left(Y\right)=-H\left(Y\right)+\sum _{i=1}^{n}H\left(Y_i\right)=-H\left(X\right)+\sum _{i=1}^{n}H\left(Y_i\right),$

since the differential entropy is invariant under such an invertible volume-preserving linear transformation, i.e.,

$H\left(B^{-1}X\right)=H\left(X\right)+log|det\left(B^{-1}\right)=H\left(X\right),$

because | det(B ^-1)| = 1. Based on this fact, we proposed the minimization of the following cost function as the criterion to select the so-called least statistically-dependent basis (LSDB) in [21]:

(3.3) $C_H\left(B|X\right)=\sum _{i=1}^{n}H\left({\left(B^{-1}X\right)}_i\right)=\sum _{i=1}^{n}H\left(Y_i\right).$

The sample estimate of this cost given the training dataset T is

$C_H\left(B|T\right)=-\frac{1}{N}\sum _{k=1}^N\sum _{i=1}^{n}log\hat{f}{y}_i\left(y_{i,k}\right),$

where $\stackrel{⌢}{f_{Y_i}}$ (y_i,k) is an empirical pdf of the coordinate Y_i, which must be estimated by an algorithm such as the histogram-based estimator with optimal bin-width search of Hall and Morton [11]. Now, we can define the LSDB as

(3.4) $B_{LSDB}=B_{LSDB}\left(T,D\right)=\underset{B\in D}{argmin}{C}_H\left(B|T\right).$

We note that the differences between this strategy and the standard independent component analysis (ICA) algorithms are: 1) restriction of the search in the basis dictionary D; and 2) approximation of the coordinate-wise entropy. For more details, we refer the reader to [21] for the former and [3] for the latter.

We now demonstrate the fact that the sparsity and the statistical independence are two intrinsically different concepts using a simple example.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/S1570579X0380037X

Probability Theory

P.K. Bhattacharya , Prabir Burman , in Theory and Methods of Statistics, 2016

1.10 Independent Random Variables and Conditioning When There Is Dependence

Random variables X ₁,   …, X _k are said to be mutually independent if

(8a) $\begin{array}{l}\hfill F_{X_1,\dots ,X_k}(x_1,\dots ,x_k)=\prod _{i=1}^k{F}_{X_i}(x_i)\text{for all}{x}_1,\dots ,x_k.\end{array}$

Equivalently, for mutually independent rv's,

(8b) $\begin{array}{l}\hfill f_{X_1,\dots ,X_k}(x_1,\dots ,x_k)=\prod _{i=1}^k{f}_{X_i}(x_i)\text{for all}{x}_1,\dots ,x_k,\end{array}$

holds for their joint pdf or joint pmf. The conditional pmf or pdf of X _r+1,   …, X _k given (X ₁,   …, X _r ) = (x ₁,   …, x _r ) when $f_{X_1,\dots ,X_r}(x_1,\dots ,x_r)>0$ is

$\begin{array}{l}\hfill f_{(X_{r+1},\dots ,X_k)|(X_1,\dots ,X_r)}(x_{r+1},\dots ,x_k|x_1,\dots ,x_r)=\frac{f_{X_1,\dots ,X_k}(x_1,\dots ,x_k)}{f_{X_1,\dots ,X_r}(x_1,\dots ,x_r)}.\end{array}$

In particular, for (X, Y ) having joint pmf/pdf f _XY (x, y),

$\begin{array}{l}\hfill f_{Y|X}(y|x)=\frac{f_{XY}(x,y)}{f_X(x)}=\frac{f_{XY}(x,y)}{{\int }_{-\infty }^{\infty }{f}_{XY}(x,y)dy},\text{when}{f}_X(x)>0.\end{array}$

On the other hand, from the marginal pdf f _X of X and the conditional pdf f _Y|X of Y given X,   the joint pdf of (X, Y ) is obtained as

(9) $\begin{array}{l}\hfill f_{XY}(x,y)=f_X(x)f_{Y|X}(y|x).\end{array}$

If X ₁,   …, X _k are mutually independent rv's, then

$\begin{array}{l}\hfill \text{E}\left[g_1(X_1)\cdots g_k(X_k)\right]=\prod _{i=1}^k\text{E}\left[g_i(X_i)\right]\end{array}$

provided that $\text{E}\left[g_i(X_i)\right],i=1,\dots ,k$ exist. This follows immediately from the definition of independence (8a, b). It follows that if X, Y are independent, then ${\sigma }_{XY}=\text{Cov}\left[X,Y\right]=0,$ and ρ _XY = 0,   and therefore,

$\begin{array}{l}\hfill {\sigma }_{aX+bY}^2=a^2{\sigma }_X^2+b^2{\sigma }_Y^2.\end{array}$

However, ρ _XY = 0 does not imply that X, Y are independent.

Suppose that X ₁,   …, X _n are mutually independent and each X _i is distributed as X. We then say that X ₁,   …, X _n are independent and identically distributed (iid) as X,   and if $\stackrel{-}{X}=\frac{1}{n}{\sum }_{i=1}^n{X}_i,$ then

$\begin{array}{l}\hfill {\mu }_{\stackrel{-}{X}}={\mu }_X\text{and}{\sigma }_{\stackrel{-}{X}}^2=\frac{{\sigma }_X^2}{n}.\end{array}$

If X ₁,   …, X _k are independent rv's and if the mgf $M_{X_i}(t)$ exists for each X _i ,   then

$\begin{array}{l}\hfill M_{X_1+\cdots +X_k}(t)=\text{E}\left[{\text{e}}^{t\sum _{i=1}^k{X}_i}\right]=\prod _{i=1}^k\text{E}\left[{\text{e}}^{t{X}_i}\right]=\prod _{i=1}^k{M}_{X_i}(t).\end{array}$

Going back to the general case of (X, Y ) with joint pdf f _XY (x, y),   let

$\begin{array}{l}\hfill m(x)=\text{E}\left[g(Y)|X=x\right]=\int g(y)f_{Y|X}(y|x)dy.\end{array}$

We now denote by $m(X)=\text{E}\left[g(Y)|X\right]$ the conditional expectation of g(Y ) given X. This conditional expectation is a function of the rv X and therefore, is itself an rv, which takes the value m(x) when X = x. Hence

$\begin{array}{ll}\hfill \text{E}\left[\text{E}\left[g(Y)|X\right]\right]& =E\left[m(X)\right]=\int m(x)f_X(x)dx\hfill \\ \hfill & =\int \left[\int g(y)f_{Y|X}(y|x)dy\right]f_X(x)dx\hfill \\ \hfill & =\int g(y)f_Y(y)dy=E\left[g(Y)\right],\hfill \end{array}$

because $\int f_{Y|X}(y|x)f_X(x)dx=\int f_{XY}(x,y)dx=f_Y(y)$ using Eq. (9). Next consider

$\begin{array}{ll}\hfill \text{E}\left[h(X)g(Y)|X=x\right]& =\int h(x)g(y)f_{Y|X}(y|x)dy\hfill \\ \hfill & =h(x)\int g(y)f_{Y|X}(y|x)dy\hfill \\ \hfill & =h(x)\text{E}\left[g(Y)|X=x\right]\text{for each}x.\hfill \end{array}$

Hence $\text{E}\left[h(X)g(Y)|X\right]=h(X)\text{E}\left[g(Y)|X\right]$ . We thus have the following important results:

$\begin{array}{l}\hfill \text{E}\text{E}\left[g(Y)|X\right]=\text{E}\left[g(Y)\right]\text{and}\text{E}\left[h(X)g(Y)|X\right]=h(X)\text{E}\left[g(Y)|X\right].\end{array}$

We next consider the conditional properties of variance.

$\begin{array}{ll}\hfill \text{Var}[Y]=& \text{E}\left[{\left\{Y-\text{E}\left[Y\right]\right\}}^2\right]\hfill \\ \hfill =& \text{E}\text{E}\left[{\left\{(Y-\text{E}\left[Y|X\right])+(\text{E}\left[Y|X\right]-\text{E}\left[Y\right])\right\}}^2|X\right]\hfill \\ \hfill =& \text{E}\text{E}\left[{(Y-\text{E}\left[Y|X\right])}^2|X\right]+\text{E}\text{E}\left[{(\text{E}\left[Y|X\right]-\text{E}\left[Y\right])}^2|X\right]\hfill \\ \hfill & +2\text{E}\text{E}\left[(Y-\text{E}\left[Y|X\right])(\text{E}\left[Y|X\right]-\text{E}\left[Y\right])|X\right].\hfill \end{array}$

The three terms in the last expression are

$\begin{array}{l}\hfill \text{E}\text{E}\left[{(Y-\text{E}\left[Y|X\right])}^2|X\right]=\text{E}\left[\text{Var}\left[Y|X\right]\right],\end{array}$

since $\text{E}\left[{(Y-\text{E}\left[Y|X\right])}^2|X\right]=\text{Var}\left[Y|X\right]$ ,

$\begin{array}{l}\hfill \text{E}\text{E}\left[{(\text{E}\left[Y|X\right]-\text{E}\left[Y\right])}^2|X\right]=\text{E}\left[{(\text{E}\left[Y|X\right]-\text{E}\left[Y\right])}^2\right]=\text{Var}\left[\text{E}\left[Y|X\right]\right],\end{array}$

and the third term is 0, using $\text{E}\left[h(X)g(Y)|X\right]=h(X)\text{E}\left[g(Y)|X\right]$ .

Summary

Besides all the properties analogous to the properties of expectation, we have proved the following important properties of conditional expectation.

Proposition 1.10.1

(i)

$\text{E}\left[\text{E}\left[g(Y)|X\right]\right]=\text{E}\left[g(Y)\right].$

(ii)

$\text{E}\left[h(X)g(Y)|X\right]=h(X)\text{E}\left[g(Y)|X\right].$

(iii)

$\text{Var}\left[Y\right]=\text{E}\left[\text{Var}\left[Y|X\right]\right]+\text{Var}\left[\text{E}\left[Y|X\right]\right].$

Definition 1.10.1

The function $m(x)=\text{E}\left[Y|X=x\right]$ is called the regression function of Y on X. In particular, if m(x) is a linear function of x,   then we can represent Y as

$\begin{array}{l}\hfill Y=\alpha +\beta X+\epsilon \text{with}\text{E}\left[\epsilon |X\right]=0,\end{array}$

and if ε is independent of X,   then this is called the linear regression model and $\text{Var}\left[\epsilon \right],$ if it exists, is the residual variance. More generally, if the dependence of Y on a k-dim rv X = (X ₁,   …, X _k ) is such that $m(x_1,\dots ,x_k)=\text{E}\left[Y|\mathit{X}=\mathit{x}\right]$ is a linear function of (x ₁,   …, x _k ),   then

$\begin{array}{l}\hfill Y=\alpha +{\beta }_1{X}_1+\cdots +{\beta }_k{X}_k+\epsilon \text{with}\text{E}\left[\epsilon |\mathit{X}\right]=0,\end{array}$

and ( X , Y ) is said to follow a multiple linear regression model if ε is independent of X .

Example 1.10.1

The joint pdf of (X, Y ) is

$\begin{array}{l}\hfill f_{XY}(x,y)=\left\{\begin{array}{ll}C(x^2+2y^2)& 0<x,y<1\\ 0& \text{otherwise}.\end{array}\right.\end{array}$

Find the constant C,   the marginal pdf's of X and Y,   the conditional pdf of Y given X = x and then find the means μ _X , μ _Y ,   the variances ${\sigma }_X^2,{\sigma }_Y^2,$ the correlation coefficient ρ _XY and the conditional expectation $\text{E}\left[Y|X=x\right]$ . Also find $P\left[X>Y\right]$ .

Solution

$1=C{\int }_0^1{\int }_0^1(x^2+2y^2)dxdy=C\left[1{\int }_0^1{x}^{2}dx+2{\int }_0^1{y}^{2}dy\right]=C\left[\frac{1}{3}+\frac{2}{3}\right].$

Thus C = 1 and f _XY (x, y) = x ² + 2y ²,   0 < x < 1,   0 < y < 1. Now

$\begin{array}{ll}\hfill f_X(x)& ={\int }_0^1(x^2+2y^2)dy=x^2+\frac{2}{3},0<x<1,\hfill \\ \hfill f_Y(y)& ={\int }_0^1(x^2+2y^2)dx=2y^2+\frac{1}{3},0<y<1.\hfill \end{array}$

The conditional pdf of Y given X = x is

$\begin{array}{l}\hfill f_{Y|X}(y|x)=\frac{f_{XY}(x,y)}{f_X(x)}=\frac{x^2+2y^2}{x^2+\frac{2}{3}},0<y<1\text{for}0<x<1.\end{array}$

Next, we evaluate the means, the variances, and the correlation coefficient:

$\begin{array}{ll}\hfill {\mu }_X& ={\int }_0^{1}x{f}_X(x)dx={\int }_0^{1}x\left(x^2+\frac{2}{3}\right)dx\hfill \\ \hfill & =\frac{1}{4}+\frac{2}{3}\cdot \frac{1}{2}=\frac{7}{12},\hfill \\ \hfill {\mu }_Y& ={\int }_0^{1}y{f}_Y(y)dy={\int }_0^{1}y\left(2y^2+\frac{1}{3}\right)dy\hfill \\ \hfill & =2\cdot \frac{1}{4}+\frac{1}{3}\cdot \frac{1}{2}=\frac{2}{3},\hfill \\ \hfill {\sigma }_X^2& =\text{E}\left[X^2\right]-{\mu }_X^2={\int }_0^1{x}^2\left(x^2+\frac{2}{3}\right)dx-{\left(\frac{7}{12}\right)}^2\hfill \\ \hfill & =\left(\frac{1}{5}+\frac{2}{3}\cdot \frac{1}{3}\right)-{\left(\frac{7}{12}\right)}^2=\frac{19}{45}-\frac{49}{144}=\frac{59}{720},\hfill \\ \hfill {\sigma }_Y^2& =\text{E}\left[Y^2\right]-{\mu }_Y^2={\int }_0^1{y}^2\left(2y^2+\frac{1}{3}\right)dy-{\left(\frac{2}{3}\right)}^2\hfill \\ \hfill & =\left(\frac{2}{5}+\frac{1}{3}\cdot \frac{1}{3}\right)-{\left(\frac{2}{3}\right)}^2=\frac{23}{45}-\frac{4}{9}=\frac{1}{15},\hfill \\ \hfill {\sigma }_{XY}& =\text{E}\left[XY\right]-{\mu }_X{\mu }_Y={\int }_0^1{\int }_0^{1}xy(x^2+2y^2)dxdy-\frac{7}{12}\cdot \frac{2}{3}\hfill \\ \hfill & =\left(\frac{1}{4}\cdot \frac{1}{2}+2\cdot \frac{1}{2}\cdot \frac{1}{4}\right)-\frac{7}{12}\cdot \frac{2}{3}=\frac{3}{8}-\frac{7}{18}=-\frac{1}{72},\hfill \\ \hfill {\rho }_{XY}& =\frac{{\sigma }_{XY}}{{\sigma }_X{\sigma }_Y}=\frac{-\frac{1}{72}}{\sqrt{\frac{59}{720}\cdot \frac{1}{15}}}=-\frac{1}{72}\sqrt{\frac{10,800}{59}}=-0.1879.\hfill \end{array}$

The conditional expectation of Y given X = x is

$\begin{array}{ll}\hfill \text{E}\left[Y|X=x\right]& ={\int }_0^{1}y{f}_{Y|X}(y|x)dy={\int }_0^{1}y\frac{x^2+2y^2}{x^2+\frac{2}{3}}dy\hfill \\ \hfill & =\frac{1}{x^2+\frac{2}{3}}{\int }_0^1(x^{2}y+2y^3)dy=\frac{x^2\frac{1}{2}+2\cdot \frac{1}{4}}{x^2+\frac{2}{3}}\hfill \\ \hfill & =\frac{x^2+1}{2x^2+\frac{4}{3}}.\hfill \end{array}$

Note that

$\begin{array}{ll}\hfill \text{E}\text{E}\left[Y|X\right]& ={\int }_0^1\text{E}\left[Y|X=x\right]f_X(x)dx={\int }_0^1\frac{x^2+1}{2x^2+\frac{4}{3}}\left(x^2+\frac{2}{3}\right)dx\hfill \\ \hfill & ={\int }_0^1\frac{1}{2}(x^2+1)dx=\frac{1}{2}\left(\frac{1}{3}+1\right)=\frac{2}{3}={\mu }_Y,\text{as it should be}.\hfill \end{array}$

Finally,

$\begin{array}{l}\hfill P[X>Y]={\int }_0^1\left[{\int }_0^x(x^2+2y^2)dy\right]dx={\int }_0^1\left(x^2+2\frac{x^3}{3}\right)dx=\frac{5}{3}\cdot \frac{1}{4}=\frac{5}{12}.\end{array}$

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780128024409000011

Feature Selection

Robert Nisbet Ph.D. , ... Ken Yale D.D.S., J.D. , in Handbook of Statistical Analysis and Data Mining Applications (Second Edition), 2018

Feature Ranking Methods

Simple feature ranking methods include the use of statistical metrics, like the correlation coefficient (described in Chapter 4). A more complex feature ranking method is the Gini Index (introduced in Chapter 4).

Gini Index

The Gini Index can be used to quantify the unevenness in variable distributions and income distributions among countries. The theory behind the Gini Index relies on the difference between a theoretical equality of some quantity and its actual value over the range of a related variable. This concept was introduced by Max O. Lorenz in 1905 to represent the unequal distribution of income among countries.

The empirical formula for the Gini score is

(5.1) $G=\frac{n+1}{n}-\frac{2\underset{1}{\overset{n}{{{\displaystyle \sum }}^{\text{}}}}\left(n+1-i\right)x_i}{n\underset{1}{\overset{n}{{{\displaystyle \sum }}^{\text{}}}}{x}_i}$

where x_i is the value of I-variable, sorted from least to greatest. For example, suppose $12 is distributed among five people as follows: Two people receive $3, and three people receive $2. In this scenario,

the bottom 20% own $2 or 16.7% of the wealth,

the bottom 40% own $4 or 33.3% of the wealth,

the bottom 60% own $6 or 50% of the wealth,

the bottom 80% own $9 or 75% of the wealth,

the bottom 100% own $12 or 100% of the wealth.

The Lorenz curve is shown in Fig. 5.1.

Fig. 5.1. The Lorenz curve for personal income example.

The Gini coefficient for this data set is

(5.2) $6/5-\left(2\times 33\right)/\left(5\times 12\right)=1.2-1.1=0.1$

Source: http://www.had2know.com/academics/gini-coefficient-calculator.html.

The theoretical even distribution of income among the people in this data set is symbolized by the straight line through the center of the figure. The inequality in incomes at any percent level of the people is plotted as the curved line below the line of perfect equality. The total inequality among all persons is represented by the area between the diagonal line and the curved line (colored yellow). If curved line remained near the bottom of the figure until the 80th percentile, for example, it would represent a population with a few very rich people and a lot of very poor people.

Corrado Gini incorporated the Lorenz concept in 1912 to quantify the change in relative frequency of income values along the range of a population of the countries. For example, if you divide up the total number of households into deciles (every 10%), you can count the number of households in each decile and express the quantity as a relative frequency. This "binning" approach allows you to use a frequency-based calculation method instead of an integration method to find the area under the Lorenz curve at each point along the percent of household axis (analogous to the X-axis in Fig. 5.1). Many analytic tools provide facilities to calculate Gini scores for only part of the Lorenz curve for each variable, which represent the relative inequality among bins of the range of each variable. One use of this information is to determine the cut-point in the range of a variable in building decision trees.

You program the Gini score in Perl, Python, C++, or SQL. A Perl program to calculate the Gini score can be found on the book website (GINI.plx). You can use this method as a guide in selecting a short list of variables to submit to the modeling algorithm. For example, you might select all variables with a Gini score greater than 0.6 for entry into the model. The disadvantage of using this method is that it combines effects of data in a given range of one variable that may not reflect the combined effects of all variables interacting with it. But that is the problem with most feature ranking methods.

A slightly more integrative approach is to use bivariate methods like the scatterplots and web diagrams described in Chapter 4.

Bivariate Methods

Other bivariate methods like mutual information calculate the distance between the actual joint distribution of features X and Y and what the joint distribution would be if X and Y were independent. The joint distribution is the probability distribution of cases in which both events X and Y occurring together. Formally, the mutual information of two discrete random variables X and Y can be defined as

(5.3) $I\left(X\text{;}Y\right)=\underset{y\in Y}{{{\displaystyle \sum }}^{\text{}}}\underset{x\in X}{{{\displaystyle \sum }}^{\text{}}}p\left(x\text{,}y\right)log\left(\frac{p\left(x\text{,}y\right)}{p_1\left(x\right)p_2\left(y\right)}\right)$

where p(x,y) is the joint probability distribution function and p ₁(x) and p ₂(y) are the independent probability (or marginal probability) density functions of X and Y, respectively. If you are a statistician, this all makes sense to you, and you can derive this metric easily. Otherwise, we suggest that you look for some approach that makes more sense to you intuitively. If this is the case, you might be more comfortable with one of the multivariate methods implemented in many statistical packages. Two of those methods are stepwise regression and partial least squares regression.

Multivariate Methods

Stepwise Linear Regression

A slightly more sophisticated method is the one used in stepwise regression. This is a classical statistics method that calculates the F-value for the incremental inclusion of each variable in the regression. The F-value is equivalent to the square root of the Student's t-value, expressing how different two data samples are, where one sample includes the variable and the other sample does not. The t-value is calculated by

$t=\text{difference}\text{in}\text{the}\text{sample}\text{means}/\text{standard}\text{deviation}\text{of}\text{differences}$

and so

$F=\surd t-\text{value}$

The F-value is sensitive to the number of variables used to calculate the numerator of this ratio and to the number of variables used to calculate the denominator. Stepwise regression calculates the F-value both with and without using a particular variable and compares it with a critical F-value either to include the variable (forward stepwise selection) or to eliminate the variable from the regression (backward stepwise selection). In this way, the algorithm can select the set of variables that meets the F-value criterion. It is assumed that these variables account for a sufficient amount of the total variance in the target variable in order to predict it at a given level of confidence specified for the F-value (usually 95%).

If your variables are numeric (or can be converted to numbers), you can use stepwise regression to select the variables you use for other data mining algorithms. But there is a "fly" in this ointment. Stepwise regression is a parametric procedure and is based on the same assumptions characterizing other classical statistical methods. Even so, stepwise regression can be used to give you one perspective on the short list of variables. You should use other methods and compare lists. Don't trust necessarily the list of variables included in the regression solution, because their inclusion assumes linear relationships of variables with the target variable, which in reality may be quite nonlinear in nature.

Partial Least Squares Regression

A slightly more complex variant of multiple stepwise regression keeps track of the partial sums of squares in the regression calculation. These partial values can be related to the contribution of each variable to the regression model. Statistica provides an output report from partial least squares regression, which can give another perspective on which to base feature selection. Table 5.1 shows an example of this output report for an analysis of manufacturing failures.

Table 5.1. Marginal Contributions of Six Predictor Variables to the Target Variable (Total Defects)

Summary of PLS (fail_tsf.STA) Responses: TOT_DEFS Options—NO-INTERCEPT AUTOSCALE
	Increase—R 2 of Y
Variable 1	0.799304
Variable 2	0.094925
Variable 3	0.014726
Variable 4	0.000161
Variable 5	0.000011
Variable 6	0.000000

It is obvious that variables 1 and 3 (and marginally variable 2) provide significant contributions to the predictive power of the model (total R ²  =   0.934). On the basis of this analysis, we might consider eliminating variables 4 through 6 from our variable short list.

Sensitivity Analysis

Some machine-learning algorithms (like neural nets) provide an output report that evaluates the final weights assigned to each variable to calculate how sensitive the solution is to the inclusion of that variable. These sensitivity values are analogous to the F-values calculated for the inclusion of each variable in stepwise regression. Both IBM SPSS Modeler and STATISTICA Data Miner provide sensitivity reports for their automated neural nets. These sensitivity values can be used as another way to determine the best set of variables to include in a model. One strategy that can be followed is to train a neural net with default characteristics and include in your short list all variables with greater than a threshold level of sensitivity. Granted, this approach is less precise than the linear stepwise regression, but the neural net set of variables may be much more generalizable, by virtue of their ability to capture nonlinear relationships effectively.

Complex Methods

A piecewise linear network uses a distance measure to assign incoming cases to an appropriate cluster. The clusters can be defined by any appropriate clustering method. A separate function called a basis function is defined for each cluster of cases. A pruning algorithm can be applied to eliminate the least important clusters, one at a time, leading to a more compact network. This approach can be viewed as a nonlinear from of stepwise linear regression.

Multiple Adaptive Regression Splines (MARS)

The MARS algorithm was popularized by Friedman (1991) to solve regression and classification problems with multiple outcomes (target variables). This approach can be viewed as a form of piecewise linear regression, which adapts a solution to local data regions of similar linear response. Each of the local regions is expressed by a different basis function. MARS algorithms can also be viewed as a form of regression trees, in which the "hard" splits into separate branches of the tree are replaced by the smooth basis functions. The MARS algorithm is implemented in STATISTICA Data Miner by the MARSplines algorithm, which includes a pruning routine—a very powerful tool for feature selection. The MARSplines algorithm will pick up only those basis functions (and those predictor variables) that provide a "sizeable" contribution to the prediction. The output of the MARSplines module will retain only those variables associated with basis functions that were retained for the final solution of the model and rank them according to the number of times they are used in different parts of the model.

You can run your data through a procedure like the STATISTICA MARSplines module to gain some insights for building your variable short list. Refer to Hastie et al. (2001) for additional details.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780124166325000050

The Basis of Monte Carlo

William L. Dunn , J. Kenneth Shultis , in Exploring Monte Carlo Methods, 2012

2.3 Multiple Random Variables

In most Monte Carlo applications multiple random variables are involved that often depend on each other. In this section some of the important properties of such probability functions are summarized.

2.3.1 Two Random Variables

The concept of a probability density function of a single random variable can be extended to probability density functions of more than one random variable. For two random variables, x and y, f (x, y) is called the joint probability density function if it is defined and non-negative on the interval x ∈ [a, b], y ∈ [c, d] and if

(2.35) ${\int }_a^b{\int }_c^{d}f\left(\text{x},y\right)d\text{x}dy=1.$

The functions

(2.36) $F_{\text{x}}(\text{x})={\int }_c^{d}f(\text{x},y)dyand{f}_y(y){\int }_a^{b}f(\text{x},y)d\text{x}$

are called the marginal PDFs of x and y, respectively. The joint PDF can be written in terms of these marginal PDFs as

(2.37) $f\left(\text{x},y\right)=f\left(\text{x}|y\right)f_y\left(y\right)=f\left(y|\text{x}\right)f_{\text{x}}\left(\text{x}\right),$

where f (x|y) is called the conditional PDF of x given y and f (y|x)| is called the conditional PDF of y given x. Equation (2.37) can also be used to define the conditional PDFs, namely,

(2.38) $\begin{array}{lll}f\left(\text{x}|y\right)=\frac{f\left(\text{x},y\right)}{f_y\left(y\right)}\hfill & \text{and}\hfill & f\left(\text{x}|y\right)=\frac{f\left(\text{x},y\right)}{f_{\text{x}}\left(\text{x}\right)}\hfill \end{array}.$

The mean and variance of x are given by

(2.39) $〈\text{x}〉\equiv {\int }_a^b{\int }_c^d\text{x}f\left(\text{x},y\right)d\text{x}dy={\int }_a^b\text{x}{f}_{\text{x}}\left(\text{x}\right)d\text{x}$

and

(2.40) ${\sigma }^2\left(\text{x}\right)\equiv {\int }_a^b{\int }_c^d{\left(\text{x}-〈\text{x}〉\right)}^{2}f\left(\text{x},y\right)d\text{x}dy={\int }_a^b{\left(\text{x}-〈\text{x}〉\right)}^2{f}_{\text{x}}\left(\text{x}\right)d\text{x}.$

Similar expressions hold for 〈y〉 and σ²(y). A measure of how x and y are dependent on each other is given by the covariance, defined as

(2.41) $\text{covar}\equiv {\int }_a^b{\int }_c^d\left(\text{x}-〈\text{x}〉\right)\left(y-〈y〉\right)f\left(\text{x},y\right)d\text{x}dy.$

Finally, if f(x, y) = g(x)h(y), where g(x) and h(y) are also PDFs, then x and y are independent random variables, and it is easily shown that covar(x, y) = 0.

Example 2.2 Isotropic Scattering

In simulating particle transport, one must keep track of a particle's direction. For instance, as neutrons migrate through matter, they scatter from nuclei. The probability of scattering from one direction into another direction is often expressed as a joint PDF of two angles, a polar angle θ (measured from the initial direction) and an azimuthal angle ψ. Generally, neutron scattering is rotationally invariant, meaning that the angle ψ is independent of the scatter angle θ and all azimuthal angles are equally likely. Thus, θ and ψ are independent and the joint PDF for neutron scattering can be expressed as

$f\left(\theta ,\psi \right)=g\left(\theta \right)h\left(\psi \right),$

where

$\begin{array}{ll}h\left(\psi \right)=\frac{1}{2\pi },\hfill & \psi \in [\mathrm{0,2}\pi ].\hfill \end{array}$

The form of the PDF g(θ) depends on such things as the type and energy of the radiation and the frame of reference in which the scatter angle is defined. These matters are more fully developed later, in Chapters 9 and 10, and in references such as Shultis and Faw [2000].

Bayes theorem

Bayes theorem, which follows from the axioms of probability, relates the conditional probabilities of two events, say x and y, with the joint probability density function f(x, y) just discussed. For two random variables, this theorem states

(2.42) $f\left(\text{x}|y\right)=\frac{f_{\text{x}}\left(\text{x}\right)f\left(y|\text{x}\right)}{f_y\left(y\right)}.$

This result is easily verified by using the definitions of Eq. (2.38). This theorem and its application to nonclassical statistical analysis is discussed at greater length in Chapter 6.

2.3.2 More Than Two Random Variables

The concept of a PDF for two random variables can be generalized to a collection of n random variables x = {x₁, x ₂, …, x_n }. Note that each component random variable x_j , j = 1, …, x_n can be either discrete or continuous. The function f (x) is the joint PDF of x if it obeys both f (x) ≥ 0 for all values of x ∈ V and ∫ _V f (x)dx = 1, where V defines the "volume" over which x is defined. If x is decomposed such that x = {x_j , x_k }, where x_k ∈ V_k and where j + k = n, ² then

(2.43) $f_{{\text{x}}_j}\left({\text{x}}_j\right)={\int }_{V_k}f\left(\text{x}\right)d{\text{x}}_k,$

is called the marginal PDF of x_j and

(2.44) $f\left({\text{x}}_k|{\text{x}}_j\right)=\frac{f\left(\text{x}\right)}{f_{{\text{x}}_j}\left({\text{x}}_j\right)}$

is called the conditional PDF of x_k , given x_j. Also, in analogy to Eq. (2.40), each x_i has a variance and, in analogy to Eq. (2.41), each pair of random variables has a covariance. Finally, if

(2.45) $f\left(\text{x}\right)=f_{{\text{x}}_1}\left({\text{x}}_1\right)f_{{\text{x}}_2}\left({\text{x}}_2\right)\dots f_{{\text{x}}_n}\left({\text{x}}_n\right)=\prod _{i=1}^n{f}_{{\text{x}}_i}\left({\text{x}}_i\right),$

then all n random variables are independent and all covariances between any two pairs are zero.

Again, suppose z represents a stochastic process that is a function of the random variable x, where x is governed by the joint PDF f(x). Then z(x) is also a random

variable and one can define its expected value, by analogy with Eq. (2.16), as

(2.46) $〈z〉\equiv {\int }_{V}z\left(\text{x}\right)f\left(\text{x}\right)d\text{x},$

and its variance, by analogy with Eq. (2.17), as

(2.47) ${\sigma }^2\left(z\right)〈{[z(\text{x})-〈\text{x}〉]}^2〉\equiv {\int }_V{[z(\text{x})-〈z〉]}^{2}f(\text{x})d\text{x}.$

This last result can be reduced to

(2.48) ${\sigma }^2\left(z\right)=〈z^2〉-{〈z〉}^2.$

The quantity 〈z〉 is properly called the population mean, while σ²(z) is the population variance and $\sigma (z)=\sqrt{{\sigma }^2(z)}$ is the population standard deviation. However, the formality of this nomenclature is often ignored and the terms mean, variance, and standard deviation are commonly used, respectively, for these quantities.

2.3.3 Sums of Random Variables

The purpose of a Monte Carlo calculation is to estimate some expected value 〈z〉 by the population mean or average given by Eq. (2.19). But to know how good this estimate is, the variance of $\overline{z}$ , which is also a random variable, is also needed.

Begin by considering the random variable $\xi ={\sum }_{i=1}^N{z}_i$ where, in general, the z_i are distributed by a joint PDF f (z ₁, z ₂, …, z_N ). The expected value of a sum can be shown (see problem 6) to be the sum of expected values, i.e.,

(2.49) $〈\xi 〉=〈\sum _{i=1}^N{z}_i〉=\sum _{i=1}^N〈z_i〉=\sum _{i=1}^N\mu i.$

The variance of ξ is calculated as

(2.50) $\begin{array}{l}{\sigma }^2(\xi )={\sigma }^2\left(\sum _{i=1}^N{z}_i\right)=〈{\left(\sum _{i=1}^N{z}_i-〈\sum _{i=N}^N{z}_i〉\right)}^2〉\\ =〈\left(\sum _{i=1}^N{(z_i-〈z_i〉)}^2\right)〉=〈\sum _{i=1}^N\sum _{j=1}^N(z_i-〈z_i〉)(z_j-〈z_j〉)〉\\ =\sum _{i=1}^N〈{(z_i-〈z_i〉)}^2〉+\sum _{i=1}^N\sum _{\begin{array}{c}j=1\\ j\ne i\end{array}}^N〈(z_i-〈z_i〉)(z_j-〈z_j〉)〉\\ =\sum _{i=1}^N{\sigma }^2(z_i)+2\sum _{i=1}^N\sum _{\begin{array}{c}j=1\\ j<i\end{array}}^N\text{covar}(z_i,z_j).\end{array}$

Now if the z_i are all independent, then $f(z_1,z_2,\dots ,z_N)={\prod }_{i+1}^N{f}_i(z_i)$ , and it is easy to show that covar(z_i , z_j ) = 0 (see problem 7) so that

(2.51) ${\sigma }^2(\xi )=\sum _{i=1}^N{\sigma }^2(z_i)$

One final property of the variance of a random number is needed, namely, for α = a constant,

(2.52) ${\sigma }^2(\alpha \xi )=〈{(\alpha \xi -〈\alpha \xi 〉)}^2〉={\alpha }^2〈{(\xi -〈\xi 〉)}^2〉={\alpha }^2{\sigma }^2(\xi ).$

Now apply these results to the Monte Carlo estimate of the sample mean $\overline{z}=(1/N){\sum }_{i=1}^{N}z({\text{x}}_i)$ . Here, the z(x_i ) ≡ z_i are all identically distributed because the z_i are independently sampled from the same PDF f(z). Thus, all the z_i have the same variance σ²(z) given by Eq. (2.18). Then the variance of the sample mean is, from Eqs. (2.51) and (2.52),

(2.53) $\begin{array}{l}{\sigma }^z(\overline{z})={\sigma }^2\left(\frac{1}{2}(z_1+z_2+\cdots +z_N)\right)\\ =\frac{1}{N2}\left[{\sigma }^2(z)+{\sigma }^2(z)+\cdots +{\sigma }^2(z)\right]=\frac{1}{N}{\sigma }^2(z).\end{array}$

Although σ²(z) is not known, it can be approximated by the sample variance s ²(z), as given by Eq. (2.23), so that

(2.54) $\sigma (\overline{z})\simeq \frac{s(z)}{\sqrt{N}}=\frac{1}{\sqrt{N}}\sqrt{\frac{N}{N-1}(\overline{z^2}-{\overline{z}}^2)}\simeq \frac{\sqrt{\overline{z^2}-{\overline{z}}^2}}{\sqrt{N}}\text{for}\text{large}N.$

Hence, the Monte Carlo estimation of 〈z〉 ± σ (z) is calculated as

(2.55) $〈z〉\pm \sigma (z)\equiv {\int }_{V}z(\text{x})f(\text{x})d\text{x}\pm \sqrt{{\int }_V{(z-〈z〉)}^{2}f(\text{x})d\text{x}}\simeq \overline{z}\pm \sqrt{\frac{\overline{z^2}-{\overline{z}}^2}{N}}.$

Here, $\overline{z}=(1/N){\sum }_{i=1}^{N}z({\text{x}}_i)$ and $\overline{z^2}=(1/N){\sum }_{i=1}^{N}z{({\text{x}}_i)}^2$ , where the x_i are distributed according to f(x) over the "volume" V.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780444515759000026

Feature Selection

Robert Nisbet , ... Gary Miner , in Handbook of Statistical Analysis and Data Mining Applications, 2009

Feature Ranking Methods

Simple feature ranking methods include the use of statistical metrics, like the correlation coefficient (described in Chapter 4). A more complex feature ranking method is the Gini Index (introduced in Chapter 4).

Gini Index

The Gini Index can be used to quantify the unevenness in variable distributions, as well as income distributions among countries. The theory behind the Gini Index relies on the difference between a theoretical equality of some quantity and its actual value over the range of a related variable. This concept was introduced by Max O. Lorenz in 1905 to represent the unequal distribution of income among countries, and can be illustrated by Figure 5.1.

Figure 5.1. The Lorenz curve relating the distribution of income among households in a population.

The theoretical even distribution of income among households is symbolized by the straight line through the center of the figure. The inequality in incomes among households is shown by the red line below the line of perfect equality. If the red line remained near the bottom of the figure until the 80th percentile, for example, it would represent a population with a few very rich people and a lot of very poor people.

Cerrado Gini incorporated the Lorenz concept in 1912 to quantify the change in relative frequency of income values along the range of a population of countries. For example, if you divide the % Households into deciles (every 10%), you can count the number of households in each decile and express the quantity as a relative frequency. This binning approach allows you to use a frequency-based calculation method instead of an integration method to find the area under the Lorenz curve at each point along the % Households axis (the x-axis). With this approach, you can calculate the relative mean difference (RMD) of all the binned values (frequency of a bin/mean frequency across all bins), and divide it by 2 × the mean frequency value, expressed for a population of size n, with a sequence of values y_i , i = 1 to n:

$\mathit{RMD}=\frac{\mathit{MD}}{\text{arithmetic mean}}$

where

$\mathit{MD}=\frac{1}{n^2}{\sum }_{i=1}^n{\sum }_{j=1}^n|y_i-y_j|$

You can use this method as a guide in selecting a short-list of variables to submit to the modeling algorithm. For example, you might select all variables with a Gini score greater than 0.6 for entry into the model. The disadvantage of using this method is that it combines effects of data in a given range of one variable that may not reflect the combined effects of all variables interacting with it. But that is the problem with most feature ranking methods.

A slightly more integrative approach is to use bi-variate methods like the scatterplots and web diagrams described in Chapter 4.

Bi-variate Methods

Other bi-variate methods like mutual information calculate the distance between the actual joint distribution of features X and Y and what the joint distribution would be if X and Y were independent. The joint distribution is the probability distribution of cases where both events X and Y occur together. Formally, the mutual information of two discrete random variables X and Y can be defined as

$I(X;Y)=\sum _{y\in Y}\sum _{x\in X}p(x,y)log\left(\frac{p\left(x,y\right)}{p_1(x)p_2(y)}\right),$

where p(x,y) is the joint probability distribution function, and p ₁(x) and p ₂(y) are the independent probability (or marginal probability) density functions of X and Y, respectively. If you are a statistician, this likely all makes sense to you, and you can derive this metric easily. Otherwise, we suggest that you look for some approach that makes more sense to you intuitively. If this is the case, you might be more comfortable with one of the multivariate methods implemented in many statistical packages. Two of those methods are stepwise regression and partial least squares regression.

Multivariate Methods

Stepwise Linear Regression

A slightly more sophisticated method is the one used in stepwise regression. This classical statistics method calculates the F-value for the incremental inclusion of each variable in the regression. The F-value is equivalent to the square root of the student's t-value, expressing how different two samples are from each other, where one sample includes the variable and the other sample does not. The t-value is calculated by

$t=\text{difference in the sample means}/\text{standard deviation of differences}$

and so

$\text{F}=\sqrt{t-\text{value}}$

The F-value is sensitive to the number of variables used to calculate the numerator of this ratio and for the denominator. Stepwise regression calculates the F-value both with and without using a particular variable and compares it with a critical F-value to either include the variable (forward stepwise selection) or to eliminate the variable from the regression (backward stepwise selection). In this way, the algorithm can select the set of variables that meets the F-value criterion. It is assumed that these variables account for a sufficient amount of the total variance in the target variable to predict it at a given level of confidence specified for the F-value (usually 95%).

If your variables are numeric (or can be converted to numbers), you can use stepwise regression to select the variables you use for other data mining algorithms. But there is a fly in this ointment. Stepwise regression is a parametric procedure and is based on the same assumptions characterizing other classical statistical methods. Even so, stepwise regression can be used to give you one perspective on the short-list of variables. You should use other methods and compare lists. Don't trust necessarily the list of variables included in the regression solution because their inclusion assumes linear relationships of variables with the target variable, which in reality may be quite nonlinear in nature.

Partial Least Squares Regression

A slightly more complex variant of multiple stepwise regression keeps track of the partial sums of squares in the regression calculation. These partial values can be related to the contribution of each variable to the regression model. STATISTICA provides an output report from partial least squares regression, which can give another perspective on which to base feature selection. Table 5.1 shows an example of this output report for an analysis of manufacturing failures.

TABLE 5.1. Marginal Contributions of Six Predictor Variables to the Target Variable (Total Defects)

Summary of PLS (fail_tsf.STA) Responses: TOT_DEFS Options: NO-INTERCEPT AUTOSCALE
	Increase - R2 of Y
Variable 1	0.799304
Variable 2	0.094925
Variable 3	0.014726
Variable 4	0.000161
Variable 5	0.000011
Variable 6	0.000000

It is obvious that variables 1 and 2 (and marginally, variable 3) provide significant contributions to the predictive power of the model (total R² = 0.934). On the basis of this analysis, we might consider eliminating variables 4 through 6 from our variable short-list.

Sensitivity Analysis

Some machine learning algorithms (like neural nets) provide an output report that evaluates the final weights assigned to each variable to calculate how sensitive the solution is to the inclusion of that variable. These sensitivity values are analogous to the F-values calculated for inclusion of each variable in stepwise regression. Both SPSS Clementine and STATISTICA Data Miner provide sensitivity reports for their automated neural nets. These sensitivity values can be used as another reflection of the best set of variables to include in a model. One strategy that can be followed is to train a neural net with default characteristics and include in your short-list all variables with greater than a threshold level of sensitivity. Granted, this approach is less precise than the linear stepwise regression, but the neural net set of variables may be much more generalizable, by virtue of their ability to capture nonlinear relationships effectively.

Complex Methods

A piecewise linear network uses a distance measure to assign incoming cases to an appropriate cluster. The clusters can be defined by any appropriate clustering method. A separate function called a basis function is defined for each cluster of cases. A pruning algorithm can be applied to eliminate the least important clusters, one at a time, leading to a more compact network. This approach can be viewed as a nonlinear form of stepwise linear regression.

Multivariate Adaptive Regression Splines (MARS)

The MARS algorithm was popularized by Friedman (1991) to solve regression and classification problems with multiple outcomes (target variables). This approach can be viewed as a form of piecewise linear regression, which adapts a solution to local data regions of similar linear response. Each of the local regions is expressed by a different basis function. MARS algorithms can also be viewed as a form of regression tree, in which the hard splits into separate branches of the tree are replaced by the smooth basis functions. In STATISTICA Data Miner (for example), the MARSplines algorithm includes a pruning routine, which provides a very powerful tool for feature selection. The MARSplines algorithm will pick up only those basis functions (and those predictor variables) that provide a sizeable contribution to the prediction. The output of the MARSplines module will retain only those variables associated with basis functions that were retained for the final solution of the model and rank them according to the number of times they are used in different parts of the model.

You can run your data through a procedure like the STATISTICA MARSplines module to gain some insights for building your variable short-list. Refer to Hastie et al. (2001) for additional details.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B978012374765500005X

Conditional Probability and Conditional Expectation

Mark A. Pinsky , Samuel Karlin , in An Introduction to Stochastic Modeling (Fourth Edition), 2011

Examples

(a)

Queueing Let N be the number of customers arriving at a service facility in a specified period of time, and let ξ _i be the service time required by the i th customer. Then, X = ξ₁ + · ·· + ξ _N is the total demand for service time.

(b)

Risk Theory Suppose that a total of N claims arrives at an insurance company in a given week. Let ξ _i be the amount of the i th claim. Then, the total liability of the insurance company is X = ξ₁ + · ·· + ξ _N .

(c)

Population Models Let N be the number of plants of a given species in a specified area, and let ξ _i be the number of seeds produced by the ith plant. Then, X = ξ₁ + · ·· + ξ N gives the total number of seeds produced in the area.

(d)

Biometrics A wildlife sampling scheme traps a random number N of a given species. Let ξ _i be the weight of the ith specimen. Then, X = ξ₁ + · ·· + ξ _N is the total weight captured.

The necessary background in conditional probability was covered in Section 2.1 for when ξ₁, ξ₂, … are discrete random variables. In order to study the random sum X= ξ₁ + · ·· + ξ₁ when ξ₁, ξ₂, … are continuous random variables, we need to extend our knowledge of conditional distributions.

2.3.1 Conditional Distributions: The Mixed Case

Let X and N be jointly distributed random variables and suppose that the possible values for N are the discrete set n = 0, 1, 2, …. Then, the elementary definition of conditional probability (2.1) applies to define the conditional distribution function F _{x | n} (x | n) of the random variable X, given that N = n, to be

(2.23) $F_{X|N}\left(x|n\right)=\frac{Pr\left\{X\le x\text{and}N=n\right\}}{Pr\left\{N=n\right\}}\text{if}Pr\left\{N=n\right\}>0,$

and the conditional distribution function is not defined at values of n for which Pr{N = n} = 0. It is elementary to verify that F _X | _N (x | n) is a probability distribution function in x at each value of n for which it is defined.

The case in which X is a discrete random variable was covered in Section 2.1. Now let us suppose that X is continuous and that F _{X | N} (x | n) is differentiable in x at each value of n for which Pr{N = n} > 0. We define the conditional probability density function f _{X | N} (x | n) for the random variable X given that N = n by setting

(2.24) $F_{X|N}\left(x|n\right)=\frac{\text{d}}{\text{d}x}{F}_X\left(x|n\right)\text{if}Pr\left\{N=n\right\}>0,$

Again, f _{X | N} (x | n) is a probability density function in x at each value of n for which it is defined. Moreover, the conditional density as defined in (2.24) has the appropriate properties, e.g.,

(2.25) $Pr\left\{a\le X<b,N=n\right\}=\underset{a}{\overset{b}{\int }}{f}_{X|N}\left(x|n\right)p_N\left(n\right)\text{d}x$

for a < b and where p_N (n) = Pr{N = n }. The law of total probability leads to the marginal probability density function for X via

(2.26) $f_X\left(x\right)=\sum _{n=0}^{\infty }{f}_{X|N}\left(x|n\right)p_N\left(n\right).$

Suppose that g is a function for which E [| g (X)|] < ∞. The conditional expectation of g(X) given that N = n is defined by

(2.27) $E\left[g\left(X\right)|N=n\right]=\int g\left(x\right)f_{X|N}\left(x|n\right)\text{d}x.$

Stipulated thus, E [g (X)| N = n] satisfies the properties listed in (2.7) to (2.15) for the joint discrete case. For example, the law of total probability is

(2.28) $E\left[g\left(X\right)\right]=\sum _{n=0}^{\infty }E\left[g\left(X\right)|N=n\right]p_N\left(n\right)=E\left\{E\left[g\left(X\right)|N\right]\right\}.$

2.3.2 The Moments of a Random Sum

Let us assume that ξ _k and N have the finite moments

(2.29) $\begin{array}{l}E\left[{\xi }_k\right]=\mu ,\text{Var}\left[{\xi }_k\right]={\sigma }^2,\\ E\left[N\right]=v,\text{Var}\left[N\right]={\tau }^2,\end{array}$

and determine the mean and variance for X = ξ₁ + · ·· + ξ _N as defined in (2.22). The derivation provides practice in manipulating conditional expectations, and the results,

(2.30) $E\left[X\right]=\mu v,\text{Var}\left[X\right]=v{\sigma }^2+{\mu }^2{\tau }^2,$

are useful and important. The properties of conditional expectation listed in (2.7) to (2.15) justify the steps in the determination.

If we begin with the mean E [X], then

$\begin{array}{l}E\left[X\right]=\sum _{n=02}^{\infty }E\left[X|N=n\right]p_N\left(n\right)(\text{by}2.15)\\ =\sum _{n=1}^{\infty }E\left[{\xi }_1+\cdot \cdot \cdot +{\xi }_N|N=n\right]p_N\left(n\right)(\text{definition}\text{of}\text{X})\\ =\sum _{n=1}^{\infty }E\left[{\xi }_1+\cdot \cdot \cdot +{\xi }_n|N=n\right]p_N\left(n\right)(\text{by}2.9)\\ =\sum _{n=1}^{\infty }E\left[{\xi }_1+\cdot \cdot \cdot +{\xi }_n\right]p_N\left(n\right)(\text{by}2.10)\\ =\mu \sum _{n=1}^{\infty }n{p}_N\left(n\right)=\mu v.\end{array}$

To determine the variance, we begin with the elementary step

(2.31) $\begin{array}{l}\text{Var}\left[X\right]=E\left[{\left(X-\mu v\right)}^2\right]=E\left[{\left(X-N\mu +N\mu -v\mu \right)}^2\right]\\ =E\left[{\left(X-N\mu \right)}^2\right]+E\left[{\mu }^2{\left(N-v\right)}^2\right]\\ +2E\left[\mu \left(X-N\mu \right)\left(N-v\right)\right].\end{array}$

Then,

$\begin{array}{l}E\left[{\left(X-N\mu \right)}^2\right]=\sum _{n=0}^{\infty }E\left[{\left(X-N\mu \right)}^2|N=n\right]p_N\left(n\right)\\ =\sum _{n=1}^{\infty }E\left[{\left({\xi }_1+\cdot \cdot \cdot +{\xi }_n-n\mu \right)}^2|N=n\right]pn\left(n\right)\\ ={\sigma }^2+\sum _{n=1}^{\infty }n{p}_N\left(n\right)=v{\sigma }^2,\end{array}$

and

$E\left[{\mu }^2{\left(N-v\right)}^2\right]={\mu }^{2}E\left[{\left(N-v\right)}^2\right]={\mu }^2{\tau }^2,$

while

$\begin{array}{l}E\left[\mu \left(X-N\mu \right)\left(N-v\right)\right]=\mu \sum _{n=0}^{\infty }E\left[\left(X-n\mu \right)\left(n-v\right)|N=n\right]p_N\left(n\right)\\ =\mu \sum _{n=0}^{\infty }\left(n-v\right)E\left[\left(X-n\mu \right)|N=n\right]p_N\left(n\right)\\ =0\end{array}$

(because E [(X − nμ)| N = n] = E [ξ₁ + · ·· + ξ _n − n μ] = 0). Then, (2.31) with the subsequent three calculations validates the variance of X as stated in (2.30).

Example The number of offspring of a given species is a random variable having probability mass function p (k) for k = 0, 1, …. A population begins with a single parent who produces a random number N of progeny, each of which independently produces offspring according to p (k) to form a second generation. Then, the total number of descendants in the second generation may be written X = ξ₁ + · ·· + ξ _N , where ξ k is the number of progeny of the k th offspring of the original parent. Let E [N] = E [ξk] = μ and Var[N] = Var[ξ k] = σ². Then,

$E\left[X\right]={\mu }^2\text{and}\text{Var}\left[X\right]=\mu {\sigma }^2\left(1+\mu \right).$

2.3.3 The Distribution of a Random Sum

Suppose that the summands ξ₁, ξ₂, … are continuous random variables having a probability density function f (z). For n ≥ 1, the probability density function for the fixed sum ξ₁ + · ·· + ξ _n is the n-fold convolution of the density f (z), denoted by f ⁽ⁿ⁾(z) and recursively defined by

$f^{\left(1\right)}\left(z\right)=f\left(z\right)$

and

(2.32) $f^{\left(n\right)}\left(z\right)=\int f^{\left(n-1\right)}\left(z-u\right)f\left(u\right)\text{d}u\text{for}n>1.$

(See Chapter 1, Section 1.2.5 for a discussion of convolutions.) Because N and ξ₁, ξ₂, … are independent, then f ⁽ⁿ⁾(z) is also the conditional density function for X = ξ₁ + · ·· + ξ _N given that N = n ≥ 1. Let us suppose that Pr{N = 0} = 0. Then, by the law of total probability as expressed in (2.26), X is continuous and has the marginal density function

(2.33) $fx\left(x\right)=\sum _{n=1}^{\infty }{f}^{\left(n\right)}\left(x\right)p_N\left(n\right).$

Remark When N = 0 can occur with positive probability, then X = ξ₁ + · ·· + ξ _N is a random variable having both continuous and discrete components to its distribution. Assuming that ξ₁, ξ₂, … are continuous with probability density function f (z), then

$\begin{array}{l}Pr\left\{X=0\right\}=Pr\left\{N=0\right\}=p_N\left(0\right),\\ \text{while}\text{for}0<a<b\text{or}a<b<0,\text{then}\end{array}$

while for 0 < a < b or a < b < 0, then

(2.34) $Pr\left\{a<X<b\right\}=\underset{a}{\overset{b}{\int }}\left\{\sum _{n=1}^{\infty }{f}^{\left(n\right)}\left(z\right)p_N\left(n\right)\right\}\text{d}z.$

Example A Geometric Sum of Exponential Random Variables In the following computational example, suppose

$f\left(z\right)=\{\begin{array}{cc}{\lambda \text{e}}^{-\lambda \text{z}}& \text{for}z\ge 0,\\ \text{0}& \text{for}z<0,\end{array}$

and

$p_N\left(n\right)=\beta {\left(1-\beta \right)}^{n-1}\text{for}n=1,2,\mathrm{\dots .}$

For n ≥ 1, the n-fold convolution of f (z) is the gamma density

$f^{\left(n\right)}\left(z\right)=\{\begin{array}{cc}\frac{\lambda n}{\left(n-1\right)!}{z}^{n-1}{\text{e}}^{-\lambda \text{z}}& \text{for}z\ge 0,\\ \text{0}& \text{for}z<0,\end{array}$

(See Chapter 1, Section 1.4.4 for discussion.)

The density for X = ξ₁ + · ·· + ξ _N is given, according to (2.26), by

$\begin{array}{l}{f}_X\left(z\right)=\sum _{n=1}^{\infty }{f}^{\left(n\right)}\left(z\right)p_N\left(n\right)\\ =\sum _{n=1}^{\infty }\frac{{\lambda }^n}{\left(n-1\right)!}{z}^{n-1}{\text{e}}^{-\lambda \text{z}}+\beta {\left(1-\beta \right)}^{n-1}\\ =\lambda \beta {\text{e}}^{-\lambda \text{z}}\sum _{n=1}^{\infty }\frac{{\left[\lambda \left(1-\beta \right)z\right]}^{n-1}}{\left(n-1\right)!}\\ =\lambda \beta e^{-\lambda z}{\text{e}}^{\lambda }\left(1-\beta \right)z\\ =\lambda \beta e^{-\lambda \beta z},z\ge 0.\end{array}$

Surprise! X has an exponential distribution with parameter λβ.

Example Stock Price Changes Stochastic models for price fluctuations of publicly traded assets were developed as early as 1900.

Let Z denote the difference in price of a single share of a certain stock between the close of one trading day and the close of the next day. For an actively traded stock, a large number of transactions take place in a single day, and the total daily price change is the sum of the changes over these individual transactions. If we assume that price changes over successive transactions are independent random variables having a common finite variance, ^* then the central limit theorem applies. The price change over a large number of transactions should follow a normal, or Gaussian, distribution.

A variety of empirical studies have supported this conclusion. For the most part, these studies involved price changes over a fixed number of transactions. Other studies found discrepancies in that both very small and very large price changes occurred more frequently in the data than suggested by normal theory. At the same time, intermediate-size price changes were under represented in the data. For the most part, these studies examined price changes over fixed durations containing a random number of transactions.

A natural question arises: Does the random number of transactions in a given day provide a possible explanation for the departures from normality that are observed in data of daily price changes? Let us model the daily price change in the form

(2.35) $Z={\xi }_0+{\xi }_1+\cdot \cdot \cdot +{\xi }_N={\xi }_0+X,$

where ξ₀, ξ₁, … are independent normally distributed random variables with common mean zero and variance σ², and N has a Poisson distribution with mean v.

We interpret N as the number of transactions during the day, ξ_i for i ≥ 1 as the price change during the i th transaction, and ξ₀ as an initial price change arising between the close of the market on one day and the opening of the market on the next day. (An obvious generalization would allow the distribution of ξ₀ to differ from that of ξ₁, ξ₂, ….)

Conditioned on N = n, the random variable Z = ξ₀ + ξ₁ + · ·· + ξ _N is normally distributed with mean zero and variance (n + 1)σ ². The conditional density function is

${\phi }_n\left(z\right)=\frac{1}{\sqrt{2\pi \left(n+1\right)\sigma }}exp\left\{-\frac{1}{2}\frac{1z^2}{\left(n+1\right){\sigma }^2}\right\}.$

Since the probability mass function for N is

$p_N\left(n\right)=\frac{{\lambda }^n{\text{e}}^{-\lambda }}{n!},n=0,1,\mathrm{\dots },$

using (2.33) we determine the probability density function for the daily price change to be

$f_Z\left(z\right)=\sum _{n=0}^{\infty }{\phi }_n\left(z\right)\frac{{\lambda }^n{\text{e}}^{-\lambda }}{n!}.$

The formula for the density f_Z (z) does not simplify. Nevertheless, numerical calculations are possible. When λ= 1 and ${\sigma }^2=\frac{1}{2}$ , then (2.30) shows that the variance of the daily price change Z in the model (2.35) is Var[Z] = (1 + λ)σ²= 1. Thus, comparing the density f_Z (z) when λ= 1 and ${\sigma }^2=\frac{1}{2}$ to a normal density with mean zero and variance, one sheds some light on the question at hand.

The calculations were carried out and are shown in Figure 2.2.

Figure 2.2. A standard normal density (solid line) as compared with a density for a random sum (dashed line). Both densities have zero mean and unit variance.

The departure from normality that is exhibited by the random sum in Figure 2.2 is consistent with the departure from normality shown by stock price changes over fixed time intervals. Of course, our calculations do not prove that the observed departure from normality is caused by the random number of transactions in a fixed time interval. Rather, the calculations show only that such an explanation is consistent with the data and is, therefore, a possible cause.

Read full chapter

URL:

https://www.sciencedirect.com/science/article/pii/B9780123814166000022

millvaged1936.blogspot.com

Source: https://www.sciencedirect.com/topics/mathematics/marginal-probability-density-function

Share this post