Correcting a pseudo-correlation matrix to be positive semidefinite

In a recent LinkedIn conversation, the topic of correlation between multiple financial indices was raised. While the actual details are not relevant, the discussion reminded me of one of the concerns I have whenever multivariate correlation is used—how to populate the correlation matrix.

First, some background. Unfortunately, most financial random variables are not normally distributed—they are more severe and have thicker tails than the normal distribution does. When dealing with a joint distribution of multiple random variables, with each of the marginals being thin-tailed, the problem is compounded as the joint distribution of those many thin-tailed marginals has no chance at being thick tailed. When dealing with financial variables, another family of multivariate copula should usually be considered.

Nevertheless, assuming we are dealing with an elliptical copula (thicker tailed than the normal or not), the correlation matrix needs to be populated. When people discuss correlation, they almost always mean linear correlation, or, more precisely, the Pearson product-moment correlation coefficient. This correlation serves naturally for the normal and multivariate normal distributions. However, linear correlation is not necessarily the best metric when dealing with copulas. The Pearson product-moment correlation is notoriously sensitive to outliers (Lindskog 2000)^[1]. Moreover, it really isn’t even a true measure of concordance (Sweeting 2011)^[2]. According to Scarsini’s axioms (1984)^[3], if variables are absolutely co-monotonic, their measure of concordance must be 100%. Now consider the case where we have two vectors of variables, $\vec{X}$ and $\vec{Y}$ . If we let $X_i = Y_i$ then the Pearson correlation coefficient is 100%, as it should be. Now define a new variable $Z = \ln(X)$ . The natural logarithm is a strictly increasing function, so if $X$ increases, $Z$ must increase. Nevertheless, the Pearson correlation is not 100% as since the increase is not linear, showing that the Pearson correlation is not a true measure of concordance.

When dealing with copulæ, we want a measure of correlation that is a true measure of concordance and not unduly affected by outliers. There are two other common measures of correlation which have both of these qualities: Spearman’s $\rho$ and Kendall’s $\tau$ . Spearman’s correlation can be thought of as the linear correlation of the ranks of the data, as opposed to their values. Kendall’s rank correlation can be thought of as the percentage increase that the set of concordant pairs have over the set of discordant pairs—which can be negative if the set of discordant pairs is larger. Of the two, Kendall’s $\tau$ is more frequently encountered when dealing with copulæ as there is a direct functional relationship between its value and that of both the generating function of Archimedean copulæ and the correlation of any elliptical copula, which both the multivariate normal and multivariate t copulæ are members (Demarta and McNeil 2005)^[4]. The relationship for elliptical copulæ is $\tau = \frac{2}{\pi}\arcsin \rho$ , so given the Kendall $\tau$ value we can calculate the needed correlation as $\rho = \sin\left(\frac{\pi}{2}\tau\right)$ . This allows us to calculate pairwise Kendall $\tau$ values for each of the variables and convert them to the corresponding $\rho$ for use in the elliptical copula we choose.

This leads us to another potential problem—it is not always the case that the matrix composed of the pairwise converted Kendall $\tau$ values is itself a valid correlation matrix. Correlation matrices have to be positive semidefinite. A correlation matrix is simply a scaled covariance matrix and the latter must be positive semidefinite as the variance of a random variable must be non-negative.

There are a number of ways to adjust these matrices so that they are positive semidefinite. The method I tend to use is one based on eigenvalues. This method has better properties than simpler shrinking methods and is easier to apply than scaling methods, all of which are described and discussed in Rousseeuw & Molenberghs (1993)^[5]. The eigenvalue method decomposes the pseudo-correlation matrix into its eigenvectors and eigenvalues and then achieves positive semidefiniteness by making all eigenvalues greater or equal to 0. If truly positive definite matrices are needed, instead of having a floor of 0, the negative eigenvalues can be converted to a small positive number. Afterwards, the matrix is recomposed via the old eigenvectors and new eigenvalues, and then scaled so that the diagonals are all 1’s. A simple R function which reads in a pseudo-correlation matrix and returns a positive semidefinite correlation matrix after adjusting the eigenvalues and rescaling is:

CorrectCM < - function(CM)
  {
  n <- dim(var(CM))[1L]
  E <- eigen(CM)
  CM1 <- E$vectors %*% tcrossprod(diag(pmax(E$values, 0), n), E$vectors)
  Balance <- diag(1/sqrt(diag(CM1)))
  CM2 <- Balance %*% CM1 %*% Balance  
  return(CM2)
}

CorrectCM < - function(CM)

{

n <- dim(var(CM))[1L]

E <- eigen(CM)

CM1 <- E$vectors %*% tcrossprod(diag(pmax(E$values, 0), n), E$vectors)

Balance <- diag(1/sqrt(diag(CM1)))

CM2 <- Balance %*% CM1 %*% Balance

return(CM2)

}

To see it in action, here is a pseudo-correlation matrix:

##         [,1]     [,2]     [,3]     [,4]    [,5]
## [1,]  1.0000 -0.15548 -0.15639  0.17273  0.7062
## [2,] -0.1555  1.00000 -0.16063  0.07662  0.1895
## [3,] -0.1564 -0.16063  1.00000  0.08714  0.3326
## [4,]  0.1727  0.07662  0.08714  1.00000 -0.5027
## [5,]  0.7062  0.18945  0.33256 -0.50269  1.0000

## [,1] [,2] [,3] [,4] [,5]

## [1,] 1.0000 -0.15548 -0.15639 0.17273 0.7062

## [2,] -0.1555 1.00000 -0.16063 0.07662 0.1895

## [3,] -0.1564 -0.16063 1.00000 0.08714 0.3326

## [4,] 0.1727 0.07662 0.08714 1.00000 -0.5027

## [5,] 0.7062 0.18945 0.33256 -0.50269 1.0000

The initial eigenvalues are:

## $values
## [1]  1.8123  1.1800  1.1569  0.9974 -0.1465
## 
## $vectors
##           [,1]     [,2]     [,3]    [,4]    [,5]
## [1,]  0.549247  0.63648 -0.04933  0.1147 -0.5269
## [2,]  0.006179 -0.03264  0.67403 -0.6952 -0.2475
## [3,]  0.164849 -0.36532 -0.67875 -0.5254 -0.3203
## [4,] -0.329193  0.67247 -0.28180 -0.4493  0.3977
## [5,]  0.750164 -0.09036  0.05606 -0.1600  0.6327

## $values

## [1] 1.8123 1.1800 1.1569 0.9974 -0.1465

## $vectors

## [,1] [,2] [,3] [,4] [,5]

## [1,] 0.549247 0.63648 -0.04933 0.1147 -0.5269

## [2,] 0.006179 -0.03264 0.67403 -0.6952 -0.2475

## [3,] 0.164849 -0.36532 -0.67875 -0.5254 -0.3203

## [4,] -0.329193 0.67247 -0.28180 -0.4493 0.3977

## [5,] 0.750164 -0.09036 0.05606 -0.1600 0.6327

After applying the correction, the correlation matrix is now:

##         [,1]     [,2]    [,3]     [,4]    [,5]
## [1,]  1.0000 -0.13310 -0.1281  0.13765  0.6263
## [2,] -0.1331  1.00000 -0.1473  0.06122  0.1611
## [3,] -0.1281 -0.14725  1.0000  0.06720  0.2922
## [4,]  0.1376  0.06122  0.0672  1.00000 -0.4476
## [5,]  0.6263  0.16112  0.2922 -0.44759  1.0000

## [,1] [,2] [,3] [,4] [,5]

## [1,] 1.0000 -0.13310 -0.1281 0.13765 0.6263

## [2,] -0.1331 1.00000 -0.1473 0.06122 0.1611

## [3,] -0.1281 -0.14725 1.0000 0.06720 0.2922

## [4,] 0.1376 0.06122 0.0672 1.00000 -0.4476

## [5,] 0.6263 0.16112 0.2922 -0.44759 1.0000

and its eigenvalues are:

## $values
## [1] 1.729e+00 1.148e+00 1.141e+00 9.824e-01 2.887e-15
## 
## $vectors
##           [,1]     [,2]     [,3]    [,4]    [,5]
## [1,] -0.546460  0.55609 -0.32051  0.1084 -0.5269
## [2,] -0.002108  0.26674  0.63673 -0.6812 -0.2437
## [3,] -0.171375 -0.61762 -0.44239 -0.5416 -0.3164
## [4,]  0.338972  0.48589 -0.53656 -0.4534  0.3944
## [5,] -0.746395 -0.04542  0.09076 -0.1590  0.6382

## $values

## [1] 1.729e+00 1.148e+00 1.141e+00 9.824e-01 2.887e-15

## $vectors

## [,1] [,2] [,3] [,4] [,5]

## [1,] -0.546460 0.55609 -0.32051 0.1084 -0.5269

## [2,] -0.002108 0.26674 0.63673 -0.6812 -0.2437

## [3,] -0.171375 -0.61762 -0.44239 -0.5416 -0.3164

## [4,] 0.338972 0.48589 -0.53656 -0.4534 0.3944

## [5,] -0.746395 -0.04542 0.09076 -0.1590 0.6382

One of these days I’d like to implement the scaling technique discussed in Rousseeuw & Molenberghs (1993)^[6] and compare the results. I’d be interested in hearing if anyone else has some experience or anecdotes about adjusting correlation matrices.

References

Filip Lindskog (2000): Linear Correlation Estimation. RiskLab 2000.
Paul Sweeting (2011): Financial Enterprise Risk Management. Cambridge University Press, 2011, ISBN: 978-0-521-11164-5.
Marco Scarsini (1984): On measures of concordance. In: vol. 8, no. 3, pp. 201–218, 1984.
Stefano Demarta and Alexander J. McNeil (2005): The t Copula and Related Copulas. In: vol. 73, no. 1, pp. 111–129, 2005.
Peter J. Rousseeuw and Geert Molenberghs (1993): Transformation of non positive semidefinite correlation matrices. In: Communications in Statistics - Theory and Methods, vol. 22, no. 4, pp. 965–984, 1993.
Peter J. Rousseeuw and Geert Molenberghs (1993): Transformation of non positive semidefinite correlation matrices. In: Communications in Statistics - Theory and Methods, vol. 22, no. 4, pp. 965–984, 1993.

2 Responses

Nick Higham November 29, 2013 at 2:39 PM | Permalink

For more on the nearest correlation matrix problem mentioned above and in [5], see
http://nickhigham.wordpress.com/2013/02/13/the-nearest-correlation-matrix/
1. Avraham December 2, 2013 at 10:08 AM | Permalink
  
  Yes, Dr. Higham. Subsequent to my posting this, a colleague of mine pointed me towards your paper and the nearPD function in the Matrix package which implements the algorithm. I should post a followup comparing the two; thank you!

Comments are closed.