10 Characterising Random Fields

Consider a random field $f(𝒙)$ – i.e. at each point $f(𝒙)$ is some random number – with zero mean, $⟨f(𝒙)⟩=0$. Correlators of fields are expectation values of products of fields at different spatial points (and, possibly, different times). They are important because typically they can be calculated theoretically, and can be estimated observationally; they are the essential link between observation and theory (see Section 6 below).

The two point correlator $\xi$ of the field $f$ is defined by

$${\xi }_f(𝒙,𝒚)\equiv ⟨f(𝒙)f(𝒚)⟩\equiv \underset{N\to \mathrm{\infty }}{lim}\sum _{i=1}^N\frac{f_i(𝒙)f_i(𝒚)}{N}\text{.}$$

(266)

where again $f_i(𝒙)$ is the field $f$ evaluated at $𝒙$ in the $i$th notional universe.

We can now explore some general properties of ${\xi }_f(𝒙,𝒚)$. Statistical homogeneity means that the statistical properties of the translated field are the same as the original field. For the two-point correlation, this implies

	${\xi }_f(𝒙,𝒚)$	$={\xi }_f(𝒙-𝐚,𝒚-𝐚)\mathit{\hspace{1em}}\text{ for any displacement }𝐚$
	$\Rightarrow \mathit{\hspace{1em}}{\xi }_f(𝒙,𝒚)$	$={\xi }_f(𝒙-𝒚,0),$		(267)

so the two-point correlator only depends on the separation of the two points. As a shorthand, we therefore can write ${\xi }_f(𝒙,𝒚)={\xi }_f(𝒙-𝒚)$, discarding the trailing $0$.

Statistical isotropy means that the statistical properties of the rotated field are the same as the original field. For the two-point correlator, we must have

$${\xi }_f(𝒙,𝒚)={\xi }_f(𝖱𝒙,𝖱𝒚)\mathit{\hspace{1em}}\text{ for any rotation matrix }𝖱.$$

(268)

Combining statistical homogeneity and isotropy gives

${\xi }_f(𝒙-𝒚)$

$=\xi \left(𝖱(𝒙-𝒚)\right)\mathit{\hspace{1em}}\text{ for any rotation matrix }𝖱$

(269)

so the two-point correlator depends only on the distance between the two points. We therefore write ${\xi }_f(𝒙,𝒚)={\xi }_f(|𝒙-𝒚|)$, both discarding the trailing $0$ (as before) but also being clear that $\xi$ is a function only of the scalar separation of $𝒙$ and $𝒚$, not of orientation. You can verify that this holds even if correlating different fields (or the same field at different times).

We can repeat these arguments to constrain the form of the correlators in Fourier space, which we will use in preference to the real-space results above. We adopt the symmetric Fourier convention, so that

$$f(𝒌)=\int \frac{d^3𝒙}{{(2\pi )}^{3/2}}f(𝒙)e^{-i𝒌\cdot 𝒙}\mathit{\hspace{1em}}\mathrm{and}\mathit{\hspace{1em}}f(𝒙)=\int \frac{d^3𝒌}{{(2\pi )}^{3/2}}f(𝒌)e^{i𝒌\cdot 𝒙}.$$

(270)

Note that we don’t use a different notation for the function $f(𝒌)$; whether we are talking about the function evaluated in the Fourier or real-space domains is normally very clear from the context. While some textbooks use a different symbol such as $\tilde{f}(𝒌)$, this becomes extremely fussy for complicated expressions and so we will not bother.

For real fields $f(𝒙)$ it follows immediately from the definitions above that $f(𝒌)=f^{*}(-𝒌)$. Also recall that spatial derivatives turn into factors of $i𝒌$ in the Fourier transform, e.g. the fourier transform of $\nabla f(𝒙)$ is

	${\displaystyle \int \frac{d^3𝒙}{{(2\pi )}^{3/2}}(\nabla f(𝒙))e^{-i𝒌\cdot 𝒙}}$	$=-{\displaystyle \int \frac{d^3𝒙}{{(2\pi )}^{3/2}}f(𝒙)\nabla e^{-i𝒌\cdot 𝒙}}$
		$=i𝒌{\displaystyle \int \frac{d^3𝒙}{{(2\pi )}^{3/2}}f(𝒙)e^{-i𝒌\cdot 𝒙}}$
		$=i𝒌f(𝒌).$		(271)

Similarly the Fourier transform of ${\nabla }^{2}f(𝒙)$ is $-{𝒌}^{2}f(𝒌)$. These properties will be important to us when we develop perturbation theory in Sections 11 and 13, allowing us to replace differential equations in real space $𝒙$ with algebraic ones in Fourier space $𝒌$.

For now, we need to focus on a different property; namely, under translations, the Fourier transform acquires a phase factor:

	$f_{\mathrm{translated}}(𝒌)$	$={\displaystyle \int \frac{d^3𝒙}{{(2\pi )}^{3/2}}f(𝒙-𝐚)e^{-i𝒌\cdot 𝒙}}$
		$={\displaystyle \int \frac{d^3{𝒙}^{\prime }}{{(2\pi )}^{3/2}}f({𝒙}^{\prime })e^{-i𝒌\cdot {𝒙}^{\prime }}{e}^{-i𝒌\cdot 𝐚}}$
		$=f(𝒌)e^{-i𝒌\cdot 𝐚}.$		(272)

Let’s consider the two-point correlator in Fourier space. We could write this $⟨f(𝒌)f({𝒌}^{\prime })⟩$ but actually we instead by convention consider $⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩$. (The two options contain equivalent information, but the second turns out to lead to simpler mathematical expressions.) Invariance of the two-point correlator in Fourier space under translations implies

	$⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩$	$=⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩e^{-i(𝒌-{𝒌}^{\prime })\cdot 𝐚}\mathit{\hspace{1em}}\text{ for any displacement }𝐚$
	$\Rightarrow \mathit{\hspace{1em}}⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩$	$=F(𝒌)\delta (𝒌-{𝒌}^{\prime }),$		(273)

for some function $F(𝒌)$, where $\delta (𝒌-{𝒌}^{\prime })$ indicates the Dirac delta function which is zero when $𝒌\ne {𝒌}^{\prime }$. The upshot is that different Fourier modes are necessarily uncorrelated due to statistical homogeneity. Under rotations,

	$f_{\mathrm{rotated}}(𝒌)$	$={\displaystyle \int \frac{d^3𝒙}{{(2\pi )}^{3/2}}f(𝖱𝒙)e^{-i𝒌\cdot 𝒙}}$
		$={\displaystyle \int \frac{d^3{𝒙}^{\prime }}{{(2\pi )}^{3/2}}f({𝒙}^{\prime })e^{-i𝒌\cdot ({𝖱}^{-1}{𝒙}^{\prime })}}$

where ${𝒙}^{\prime }=𝖱𝒙$. Now, for any rotation matrix $𝖱$, we have ${𝖱}^{-1}={𝖱}^{\top }$. So, $-i𝒌\cdot {𝖱}^{-1}{𝒙}^{\prime }=-i(𝖱𝒌)\cdot 𝒙$, and

	$f_{\mathrm{rotated}}(𝒌)$	$={\displaystyle \int \frac{d^3{𝒙}^{\prime }}{{(2\pi )}^{3/2}}f({𝒙}^{\prime })e^{-i(𝖱𝒌)\cdot ({𝒙}^{\prime })}}$		(275)
		$=f(𝖱𝒌)\text{.}$		(276)

So demanding invariance of the two-point correlator under rotations implies

$$F(𝖱𝒌)\delta (𝒌-{𝒌}^{\prime })=F(𝒌)\delta (𝒌-{𝒌}^{\prime })\mathit{\hspace{1em}}\text{ for any rotation matrix }𝖱.$$

(277)

(To get this result you need to use property of the Dirac delta function $\delta (𝖱𝒌)=\det 𝖱\delta (𝒌)=\delta (𝒌)$.) Equation (277) can only hold if $F(𝒌)=F(k)$ where $k\equiv |𝒌|$. We can therefore define the power spectrum, $P_f(k)$, of a homogeneous and isotropic field, $f(𝒙)$, by

$$\boxed{⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩=\frac{2{\pi }^2}{k^3}{P}_f(k)\delta (𝒌-{𝒌}^{\prime })}.$$

(278)

The normalisation factor $2{\pi }^2/k^3$ in the definition of the power spectrum is conventional and has the virtue of making $P_f(k)$ dimensionless if $f(𝒙)$ is.

The correlation function ${\xi }_f(r)$ and the power spectrum $P(k)$ carry the same physical information, just in a different form. To get the detailed relationship requires a bit of algebra. First, we write down the correlation function and substitute the inverse Fourier transforms (270):

$${\xi }_f(𝒙,𝒚)=⟨f(𝒙)f(𝒚)⟩=⟨f(𝒙)f^{*}(𝒚)⟩=\int \frac{d^3𝒌}{{(2\pi )}^{3/2}}\frac{d^3{𝒌}^{\prime }}{{(2\pi )}^{3/2}}⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩e^{i𝒌\cdot 𝒙}{e}^{-i{𝒌}^{\prime }\cdot 𝒚}$$

(279)

where we made use of the fact that $f(𝒚)$ is real to replace it with its own complex conjugate. (This is just a minor trick to simplify the algebra a little.) The term $⟨f(𝒌)f^{*}({𝒌}^{\prime })⟩$ can be replaced by the power spectrum using equation (278), after which the ${𝒌}^{\prime }$ integral can be performed. We also write $𝒌=|k|\widehat{𝒏}$ where $\widehat{𝒏}$ is a unit vector containing the direction information in $𝒌$. This gets us to the following expression:

${\xi }_f(𝒙,𝒚)$

$={\displaystyle \frac{1}{4\pi }}{\displaystyle \int \frac{\mathrm{d}k}{k}{P}_f(k)\int {\mathrm{d}}^2\widehat{𝒏}{e}^{ik\widehat{𝒏}\cdot (𝒙-𝒚)}}.$

(280)

We can evaluate the integral over $\widehat{𝒏}$ by setting $\widehat{𝒏}\cdot (𝒙-𝒚)=|𝒙-𝒚|\mu$, the $\widehat{𝒏}$ integral then reduces to

$$2\pi {\int }_{-1}^1𝑑\mu e^{ik|𝒙-𝒚|\mu }=4\pi \frac{\sin (k|𝒙-𝒚|)}{k|𝒙-𝒚|}.$$

(281)

It follows that the sought-after relationship between $P_f(k)$ and ${\xi }_f(r)$ is:

$${\xi }_f(r)=\int \frac{dk}{k}{P}_f(k)\frac{\sin (kr)}{kr}.$$

(282)

Note that this only depends on $r=|𝒙-𝒚|$ as required by the translational and rotational invariance that we discussed earlier.

Inflation predicts fluctuations that are very nearly Gaussian and this property is preserved by linear evolution. The cosmic microwave background probes fluctuations mostly in this linear regime (see right panel of Fig. 12).

When we say that the field is Gaussian, we mean first that the probability of finding a particular value at a given location can be written as a Gaussian:

$$P(f(𝒙))=\frac{1}{\sqrt{2\pi }\sigma }\exp \left(-\frac{f{(𝒙)}^2}{2{\sigma }^2}\right)\text{.}$$

(283)

Since, from the definitions above, we must have $⟨f{(𝒙)}^2⟩={\xi }_f(0)$, the value of $\sigma$ is just given by ${\sigma }^2={\xi }_f(0)$. By saying the field is Gaussian we also mean that the probability density for the field taking on specified values at multiple points – say ${𝒙}_1$, ${𝒙}_2$, $\mathrm{\cdots }$, ${𝒙}_N$ – is given by a multivariate Guassian, with covariance given by:

(284)

i.e. fully specified just by the correlation function. So the real point of a Gaussian field is that these 2-point correlations capture all the information. There is no further information in higher order correlators.

Why should the fluctuations emerging from inflation be nearly Gaussian? If you have studied quantum field theory (QFT), the answer is pretty clear: when we calculate the 2-point correlator of a quantum field $⟨\varphi ({𝒙}_1)\varphi ({𝒙}_2)⟩$, the result is given to us by the QFT propagator. But when we calculate higher-order correlations such as $⟨\varphi ({𝒙}_1)\varphi ({𝒙}_2)\varphi ({𝒙}_3)⟩$, we turn to interacting field theory where there are not just propagators but also interaction vertices; these carry at least one factor of the interaction strength which is typically small.

Non-linear structure formation at late times destroys Gaussianity and gives the filamentary cosmic web (see left panel of Fig. 12). So the late time universe is certainly non-Gaussian. But searching for primordial non-Gaussianity to probe departures from simple inflation is a hot topic; no convincing evidence for primordial non-Gaussianity has yet been found.

Physical fields (in a flat universe) can be represented in Fourier or real space as described above. We’ve seen that an ability to work in Fourier space is essential because it allows us to encode notions like homogeneity and isotropy of the field statistics in a succinct way – equation (278).

However observations at their root consist of a field associated with position on the sky – we can think of observations as measured fields on the surface of a sphere. For that reason we need to be able to express fields on the surface of a sphere and manipulate them with the same sophistication as we can for fields in flat space.

Spherical harmonics are basis functions on the sphere that are the closest possible equivalent to Fourier modes. Functions $f$ of position on the sphere $\widehat{𝐧}$ can be expanded as

$$f(\widehat{𝐧})=\sum _{\mathrm{\ell }=0}^{\mathrm{\infty }}\sum _{m=-\mathrm{\ell }}^{\mathrm{\ell }}{f}_{\mathrm{\ell }m}{Y}_{\mathrm{\ell }m}(\widehat{𝐧}).$$

(285)

In this expression and others involving spherical harmonics we normally don’t use the summation convention, i.e. repeated indices do not implicitly mean a sum. We instead write sums explicitly where required, as in Eq. (285). The $Y_{\mathrm{\ell }m}$ are familiar²³²³ 23 Just to ward off potential confusion – there are various phase conventions for the $Y_{\mathrm{\ell }m}$; here we adopt $Y_{\mathrm{\ell }m}^{*}={(-1)}^m{Y}_{\mathrm{\ell }-m}$ so that $f_{\mathrm{\ell }m}^{*}={(-1)}^m{f}_{\mathrm{\ell }-m}$ for a real field. from quantum mechanics as the position-space representation of the eigenstates of ${\widehat{L}}^2=-{\nabla }^2$ and ${\widehat{L}}_z=-i{\partial }_{\varphi }$:

	${\nabla }^2{Y}_{\mathrm{\ell }m}$	$=-\mathrm{\ell }(\mathrm{\ell }+1)Y_{\mathrm{\ell }m}$
	${\partial }_{\varphi }{Y}_{\mathrm{\ell }m}$	$=im{Y}_{\mathrm{\ell }m},$		(286)

with $\mathrm{\ell }$ an integer $\ge 0$ and $m$ an integer with $|m|\le \mathrm{\ell }$. This implies a typical wavelength of fluctuations for a given $\mathrm{\ell }$: because ${\nabla }^2{Y}_{\mathrm{\ell }m}\sim {\mathrm{\ell }}^2{Y}_{\mathrm{\ell }m}$, we are talking about fluctuations with an angular scale of

$$\delta \theta \sim \frac{2\pi }{\mathrm{\ell }}\text{.}$$

(287)

Note the close analogy with Fourier modes in a flat space where ${\nabla }^2{e}^{i𝐤\cdot 𝐱}=-{𝐤}^2{e}^{i𝐤\cdot 𝐱}$ and ${\partial }_x{e}^{i𝐤\cdot 𝐱}=i{k}_x{e}^{i𝐤\cdot 𝐱}$. In the Fourier case, the scale of fluctuations is $\sim 2\pi /|𝒌|$. So $\mathrm{\ell }$ acts a bit like the wavenumber $|𝐤|$ whereas $m$ is a bit like $\widehat{𝐤}\equiv 𝐤/|𝐤|$ (i.e. $m$ contains information about directionality).

Just like the Fourier modes, the spherical harmonics have a lot of special properties that help us manipulate them. They are orthonormal over the sphere,

$$\int 𝑑\widehat{𝐧}{Y}_{\mathrm{\ell }m}(\widehat{𝐧})Y_{{\mathrm{\ell }}^{\prime }{m}^{\prime }}^{*}(\widehat{𝐧})={\delta }_{\mathrm{\ell }{\mathrm{\ell }}^{\prime }}{\delta }_{m{m}^{\prime }},$$

(288)

so that the spherical multipole coefficients of $f(\widehat{𝐧})$ are

$$f_{\mathrm{\ell }m}=\int 𝑑\widehat{𝐧}f(\widehat{𝐧})Y_{\mathrm{\ell }m}^{*}(\widehat{𝐧}).$$

(289)

In turn, this implies a completeness relationship

$$\sum _{\mathrm{\ell }m}{Y}_{\mathrm{\ell }m}(\widehat{𝐧})Y_{\mathrm{\ell }m}^{*}({\widehat{𝐧}}^{\prime })=\delta (\widehat{𝐧}-{\widehat{𝐧}}^{\prime })\text{.}$$

(290)

where $\delta (\widehat{𝐧}-{\widehat{𝐧}}^{\prime })$ is the Dirac delta function on the sphere.

There is a generalisation of equation (290) which sums only over the $m$s, not the $\mathrm{\ell }$s. It is known as the spherical harmonic addition theorem and states that

$$\sum _m{Y}_{\mathrm{\ell }m}(\widehat{𝐧})Y_{\mathrm{\ell }m}({\widehat{𝐧}}^{\prime })=\frac{2\mathrm{\ell }+1}{4\pi }{P}_{\mathrm{\ell }}(\widehat{𝐧}\cdot {\widehat{𝐧}}^{\prime })$$

(291)

where $P_{\mathrm{\ell }}(x)$ is the $\mathrm{\ell }$’th Legendre polynomial. Many different proofs of this statement are available e.g. in quantum mechanics text books. They’re a bit tedious so here we’ll just take the result on trust (we’ll only need it once anyway).

What is the implication of statistical isotropy for the correlators of $f_{\mathrm{\ell }m}$? We first need to know that, under a rotation, $Y_{\mathrm{\ell }m}\to {\sum }_{m^{\prime }}{R}_{m{m}^{\prime }}^{\mathrm{\ell }}{Y}_{\mathrm{\ell }{m}^{\prime }}$ for some rotation coefficients $R_{m{m}^{\prime }}^l$; any given rotation of a general function $f(\widehat{𝒏})$ can therefore be effected by transforming its spherical harmonic coefficients $f_{\mathrm{\ell }m}\to {\sum }_{m^{\prime }}{R}_{m{m}^{\prime }}^{\mathrm{\ell }}{f}_{\mathrm{\ell }{m}^{\prime }}$. There are a few helpful properties of the $R_{m{m}^{\prime }}^{\mathrm{\ell }}$:

• •

  They do not mix different $\mathrm{\ell }$s (this is already built into the notation). If you’re curious to see why, you can show it from the commutation of ${\widehat{L}}^2$ with the individual rotation generators.

• •

  It’s relatively straight-forward to show that ${|R_{m{m}^{\prime }}^{\mathrm{\ell }}|}^2=1$ – this can be derived from the orthonormalisation of the $Y_{\mathrm{\ell }m}$s themselves.

• •

  There is an extension of the previous statement that is even more powerful: namely, the $R$’s are orthonormalized such that ${\sum }_m{R}_{m{m}^{\prime }}^{\mathrm{\ell }}{R}_{m{m}^{\prime \prime }}^{*\mathrm{\ell }}={\delta }_{m^{\prime }{m}^{\prime \prime }}$. Demonstrating this requires application of the completeness relation (290) and is recommended for enthusiasts only.

• •

  For a rotation of angle $\varphi$ around the z axis, $R_{m{m}^{\prime }}^{\mathrm{\ell }}={\delta }_{m{m}^{\prime }}{e}^{i\varphi }$. This follows directly from equation (286).

😇 Exercise 10A Use the above properties to show that, for $⟨f_{\mathrm{\ell }m}⟩$ and $⟨f_{\mathrm{\ell }m}{f}_{{\mathrm{\ell }}^{\prime }{m}^{\prime }}^{*}⟩$ to be invariant under rotations, it is sufficient that $⟨f_{\mathrm{\ell }m}⟩$ $=0\mathit{\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{0.5em} }(\mathrm{\ell }\ne 0)$ (292) $⟨f_{\mathrm{\ell }m}{f}_{{\mathrm{\ell }}^{\prime }{m}^{\prime }}^{*}⟩$ $=C_{\mathrm{\ell }}{\delta }_{\mathrm{\ell }{\mathrm{\ell }}^{\prime }}{\delta }_{m{m}^{\prime }}\text{.}$ (293) for some $C_{\mathrm{\ell }}$s which we call the angular power spectrum of $f$.

Exercise 10B Show that the above conditions are not just sufficient but also necessary for isotropy.

Equation (293) is the equivalent of equation (278); these are the two most useful and important results in this Section.

Random fields are defined by their ensemble average properties. If some element of randomness is involved in physics (e.g. because of quantum mechanics) we would ideally want to measure ensemble average outcomes rather than an individual outcome.

However in cosmology, this isn’t an option – we have just one universe and we’re stuck with it. Yet, if one makes sufficiently strong assumptions (normally homogeneity and isotropy), it’s still possible to make inferences about the random process. In effect, one splits up the data into lots of chunks and regards them as independent “trials” of the random process. This introduces inherent limits to what can be achieved – especially when we’re talking about the very largest scales of the universe. If and when we find something that seems strange or doesn’t fit the model, we must bear in mind that we might just be looking at an “unlucky” draw from the random process.

Later in the course we will especially look at the cosmic microwave background, which is an example of a random field on a notional sphere (the “surface of last scattering”). We want to be able to measure $C_{\mathrm{\ell }}$, but in practice we can’t because $C_{\mathrm{\ell }}\equiv ⟨{|f_{\mathrm{\ell }m}|}^2⟩$ – we’re unable to take the required ensemble average from within our own universe.

Instead we construct an “estimator for $C_{\mathrm{\ell }}$” which does what the name suggests: it estimates $C_{\mathrm{\ell }}$ from a single observation. Such estimators are normally written ${\widehat{C}}_{\mathrm{\ell }}$; the simplest (and, in ideal circumstances, the best) example of which is:

$${\widehat{C}}_{\mathrm{\ell }}=\frac{1}{2\mathrm{\ell }+1}\sum _{m=-\mathrm{\ell }}^{\mathrm{\ell }}{|f_{\mathrm{\ell }m}|}^2\text{.}$$

(294)

The exercise above shows that on average, ${\widehat{C}}_{\mathrm{\ell }}$ will be close to $C_{\mathrm{\ell }}$, especially as $\mathrm{\ell }$ becomes large (this corresponds to small scales where there are lots of independent modes each sampling $C_{\mathrm{\ell }}$). But for small $\mathrm{\ell }$ – the largest accessible angular scales – the standard deviation becomes comparable to $C_{\mathrm{\ell }}$ itself. This inherent limitation in our ability to estimate $C_{\mathrm{\ell }}$ (or other statistical properties of fields we observe) is known as “cosmic variance”.

• •

  Many of the predictions made by modern cosmological theories are statistical in nature;

• •

  Quantum fields have near-Gaussian random fluctuations which, in the inflationary picture, seed structure in our Universe;

• •

  Gaussian fluctuations can be characterised by two-point correlations alone;

• •

  Statistical homogeneity and isotropy put stringent limitations on the form that two-point correlations can take;

• •

  The predictions then take the form of a power spectrum $P(k)$, measuring fluctuations on comoving scales $\sim 2\pi /k$ – or, equivalently, a correlation function $\xi (r)$;

• •

  When measured on a sphere, the power spectrum is discretised into $C_{\mathrm{\ell }}$’s measuring fluctuations on angular scales $\sim 2\pi /\mathrm{\ell }$

• •

  Power spectra can be calculated from theories but cannot be measured directly from observations because they involve an average over an infinite series of universes…

• •

  …however we can estimate power spectra observationally by averaging over a number of observed modes. The theory/observation comparison will then have an irreducible level of uncertainty known as “cosmic variance”. This is most problematic when probing large scales due to the small number of accessible large-scale modes.