nonlinear hierarchal model
y~gv 1ðÞ ,h 1ðÞ
/C16/C17
zz 1ðÞ
v 1ðÞ ~gv 2ðÞ ,h 2ðÞ
/C16/C17
zz 2ðÞ
/C1/C1/C1
v mðÞ ~z mðÞ
ð38Þ
This is the same as Equation 33 but with unknown causes. Here,
the D-Step performs a nonlinear optimisation of the states to
estimate their most likely values and the M-Step estimates the
variance components at each level. As mentioned above, for static
systems, Dt = ‘ and n = 0. This renders it a classical Gauss-Newton
scheme for nonlinear model estimation
Dm~{ e
T
v Pev
/C0/C1 {1
eT
v Pe ð39Þ
Empirical priors are embedded in the scheme through the hierarchical
construction of the prediction errors, e and their precision P,i nt h e
usual way; see Equation 11 and [15] for more details.
Linear models and parametric empirical Bayes. When
the model above is linear, we have the ubiquitous hierarchical linear
observation model used in Parametric Empirical Bayes ( PEB;[ 8 ] )
and mixed-effects analysis of covariance ( ANCOVA) analyses.
y~h
1ðÞ v 1ðÞ zz 1ðÞ
v 1ðÞ ~h 2ðÞ v 2ðÞ zz 2ðÞ
..
.
v mðÞ ~z mðÞ
ð40Þ
Here the D-Step converges after a single iteration because the
linearity of this model renders the Laplace assumption exact. In this
context, the M-Step becomes a classical restricted maximum
likelihood ( ReML) estimation of the hierarchical covariance
components, S
(i)z. It is interesting to note that the ReML objective
function and the variational energy are formally identical under this
model [15,18]. Figure 3 shows a comparative evaluation of ReML
and DEM using the same data. The estimates are similar but not
identical. This is because DEM hyperparameterises the covariance
as a linear mixture of precisions, whereas the ReML scheme used a
linear mixture of covariance components.
Covariance component estimation and Gaussian process
models. When there are many more causes then observations, a
common device is to eliminate the causes in Equation 40 by
recursive substitution to give a model that generates sample
covariances and is formulated in terms of covariance components
(i.e., hyperparameters).
Figure 3. Example of estimation under a mixed-effects or hierarchical linear model. The inversion was cross-validated with expectation
maximization ( EM), where the M-step corresponds to restricted maximum likelihood ( ReML). This example used a simple two-level model that
embodies empirical shrinkage priors on the first-level parameters. These models are also known as parametric empirical Bayes ( PEB) models ( left).
Causes were sampled from the unit normal density to generate a response, which was used to recover the causes, given the parameters. Slight
differences in the hyperparameter estimates ( upper right), due to a different hyperparameterisation, have little effect on the conditional means of
the unknown causes ( lower right ), which are almost indistinguishable.
doi:10.1371/journal.pcbi.1000211.g003
Hierarchical Models
PLoS Computational Biology | www.ploscompbiol.org 11 November 2008 | Volume 4 | Issue 11 | e1000211