Figure 2 has demonstrated that in the case of data that conforms to the model, the model order can be inferred accurately. In the case of real FMRI data, estimation of the number of source processes is a much more difficult task. In order to assess the dependency between the estimated number of source processes and the spatio-temporal accuracy of the estimation, we performed ROC analysis on spatial maps obtained after projecting the data into subspaces of increasing dimensionality7.
Figure 7 shows the results of the temporal correlation and the final false-positive rates and false-negative rates over the range of possible dimensions for the data set with peak level activation, where the spatial maps were thresholded at the level. Both for the spatial and temporal accuracy these plots suggest that the quality of estimation does not improve once the source signals are being estimated in a subspace with more than about 30 dimensions. These results appear to be consistent for both artificial activation patterns and time courses. Reducing the number of sources below 30, however, will lead to increasingly poor estimates.
Overfitting would necessarily result in an increase of the false-negative rate, an effect that is shown in figure 7 (iii) for the auditory activation cluster. For this particular data set the effect is very subtle since the data has been generated without any voxel-wise variation of the temporal signal introduced into resting-state FMRI data. The artificial time courses are consistent within the clusters and therefore these clusters are less likely than in 'real' FMRI data to be incorrectly split into different spatial maps. Though the quality of estimation does not degenerate badly with increased dimensionality it is still essential to find a good estimate of the lower dimensional subspace. It not only dramatically decreases the computational load but more importantly provides better estimates for the noise, which is essential as part of the inferential steps. For this data set, the Laplace approximation to the evidence for model order (figure 7(iv)) appears to work well.