This post is about Stein’s lemma, which first appeared in the landmark paper of Stein in 1981. This lemma is leads to Stein’s unbiased risk estimator and is useful for proving central limit theorems. It has also been called `Gaussian integration by parts’, which is in fact a high level description of the proof.
Lemma 1 (Stein’s lemma) If
follows the standard normal distribution, then
if the expectations are well-defined.
![\displaystyle \mathbb{E}\left[x f(x) \right] = \mathbb{E}\left[f'(x)\right],](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cmathbb%7BE%7D%5Cleft%5Bx+f%28x%29+%5Cright%5D+%3D+%5Cmathbb%7BE%7D%5Cleft%5Bf%27%28x%29%5Cright%5D%2C+&bg=ffffff&fg=000000&s=0&c=20201002)
Proof: If 
![\displaystyle \begin{array}{rcl} \mathbb{E}\left[f'(x)\right] &=& \int_{-\infty}^{\infty} f'(x) \frac{\exp(-x^2/2)}{\sqrt{2\pi}} dx\\ &=& 0 - \int_{-\infty}^{\infty} f(x) \left(-x \frac{\exp(-x^2/2)}{\sqrt{2\pi}} \right) dx \\ &=& \mathbb{E}\left[x f(x)\right]. \end{array}](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Cbegin%7Barray%7D%7Brcl%7D++%5Cmathbb%7BE%7D%5Cleft%5Bf%27%28x%29%5Cright%5D+%26%3D%26+%5Cint_%7B-%5Cinfty%7D%5E%7B%5Cinfty%7D+f%27%28x%29+%5Cfrac%7B%5Cexp%28-x%5E2%2F2%29%7D%7B%5Csqrt%7B2%5Cpi%7D%7D+dx%5C%5C+%26%3D%26+0+-+%5Cint_%7B-%5Cinfty%7D%5E%7B%5Cinfty%7D+f%28x%29+%5Cleft%28-x+%5Cfrac%7B%5Cexp%28-x%5E2%2F2%29%7D%7B%5Csqrt%7B2%5Cpi%7D%7D+%5Cright%29+dx+%5C%5C+%26%3D%26+%5Cmathbb%7BE%7D%5Cleft%5Bx+f%28x%29%5Cright%5D.+%5Cend%7Barray%7D+&bg=ffffff&fg=000000&s=0&c=20201002)
It is also convinient to denote 
Stein’s lemma can be generalized to exponential family distributions. In particular, for multivariate normals, if 
![\displaystyle \textrm{cov}\left(\hat{\mu}_i, y_i\right) = \sigma^2 \mathbb{E}\left[\frac{\partial \hat{\mu}_i}{\partial y_i}\right].](https://s0.wp.com/latex.php?latex=%5Cdisplaystyle++%5Ctextrm%7Bcov%7D%5Cleft%28%5Chat%7B%5Cmu%7D_i%2C+y_i%5Cright%29+%3D+%5Csigma%5E2+%5Cmathbb%7BE%7D%5Cleft%5B%5Cfrac%7B%5Cpartial+%5Chat%7B%5Cmu%7D_i%7D%7B%5Cpartial+y_i%7D%5Cright%5D.+&bg=ffffff&fg=000000&s=0&c=20201002)
This is Equation (12.56) in `Computer Age Statistical Inference’ by Efron and Hastie.
This post is the second in the category `trivial matters’, where I formally write down some notes to myself about identities, tricks, and facts that I repeatedly use but (unfortunately) need to re-derive everytime I use them. Although these posts are short, they discuss important topics. The term `trivial matters’ is used as a sarcasm, because my blood boils everytime I see terms like `obviously’ or `it is trivial to show that …’ when I grade students’ homeworks.
Phyllis with data 