  C RUBY-ON-RAILS MYSQL ASP.NET DEVELOPMENT RUBY .NET LINUX SQL-SERVER REGEX WINDOWS ALGORITHM ECLIPSE VISUAL-STUDIO STRING SVN PERFORMANCE APACHE-FLEX UNIT-TESTING SECURITY LINQ UNIX MATH EMAIL OOP LANGUAGE-AGNOSTIC VB6 MSBUILD # Match information from a correlation matrix according to their p-value cut off  » r » Match information from a correlation matrix according to their p-value cut off

By : user2949983
Date : November 17 2020, 04:28 AM
like below fixes the issue I have used rcorr function of Hmisc library for calculation of correlations and p-values. Then extracted pvalues to Pval matrix and correlation coefficients to corr matrix. , You could try code :
``````indx <- which(Rvalue==1 & Pvalue < 0.05 & !is.na(Pvalue), arr.ind=TRUE)
d1 <- data.frame(rN=row.names(Rvalue)[indx[,1]],
cN=colnames(Rvalue)[indx[,2]], Pval=signif(Pvalue[indx],
digits=4))

#     rN    cN Pval
#1 41700 41699    0
#2 41701 41699    0
``````
`````` f1 <- function(Rmat, Pmat, cutoff){
indx <- which(Rmat==1 & Pmat < cutoff & !is.na(Pmat), arr.ind=TRUE)
d1 <- data.frame(rN=row.names(Rmat)[indx[,1]],
cN=colnames(Rmat)[indx[,2]], Pval=signif(Pmat[indx],
digits=4))
d1}

f1(R1, P1, 0.05)
#  rN cN  Pval
#1  B  A 0.021
#2  C  A 0.018
#3  D  A 0.001
#4  A  B 0.021
#5  A  C 0.018
#6  E  C 0.034
#7  A  D 0.001
#8  C  E 0.034

f1(R1, P1, 0.01)
#  rN cN  Pval
#1  D  A 0.001
#2  A  D 0.001

f1(R1, P1, 0.001)
# rN   cN   Pval
#<0 rows> (or 0-length row.names)
``````
``````set.seed(24)
R1 <- matrix(sample(c(0,1), 5*5, replace=TRUE), 5,5,
dimnames=list(LETTERS[1:5], LETTERS[1:5]))
R1[lower.tri(R1)] <- 0
R1 <- R1+t(R1)
diag(R1) <- 1

set.seed(49)
P1 <- matrix(sample(seq(0,0.07, by=0.001), 5*5, replace=TRUE), 5, 5,
dimnames=list(LETTERS[1:5], LETTERS[1:5]))

P1[lower.tri(P1)] <- 0
P1 <- P1+t(P1)
diag(P1) <- NA
`````` ## List Highest Correlation Pairs from a Large Correlation Matrix in Pandas?

By : African Web Develope
Date : March 29 2020, 07:55 AM
I wish this helpful for you You can use DataFrame.values to get an numpy array of the data and then use NumPy functions such as argsort() to get the most correlated pairs.
But if you want to do this in pandas, you can unstack and sort the DataFrame:
code :
``````import pandas as pd
import numpy as np

shape = (50, 4460)

data = np.random.normal(size=shape)

data[:, 1000] += data[:, 2000]

df = pd.DataFrame(data)

c = df.corr().abs()

s = c.unstack()
so = s.sort_values(kind="quicksort")

print so[-4470:-4460]
``````
``````2192  1522    0.636198
1522  2192    0.636198
3677  2027    0.641817
2027  3677    0.641817
242   130     0.646760
130   242     0.646760
1171  2733    0.670048
2733  1171    0.670048
1000  2000    0.742340
2000  1000    0.742340
dtype: float64
`````` ## Computing the correlation between the auto-correlation and cross-correlation for each pair of rows in a matrix

By : Kris Cochrane
Date : March 29 2020, 07:55 AM
this one helps. Make sure you set plot to FALSE for acf, ccf. Then, you can just wrap your code in a call to outer to provide every pair of i and j values. Note that since outer expects a vectorized FUN (e.g. *), we need to vectorize your function:
code :
``````set.seed(1)
m <- 4
n <- 10
mat <- matrix(rnorm(m*n), nrow = m, ncol=n)
lag <- 5

outer(1:nrow(mat), 1:nrow(mat),
Vectorize(
function(i, j) {
acf.i <- acf(mat[i,],lag.max=lag, plot=F)
ccf.i.j <- ccf(mat[i,],mat[j,],lag.max=lag, plot=F)
cor(acf.i\$acf,ccf.i.j\$acf[(lag+1):(2*lag+1)])
} ) )
#            [,1]        [,2]         [,3]        [,4]
# [1,]  1.0000000  0.47035200 -0.006371955 -0.85880247
# [2,]  0.4133899  1.00000000 -0.462744858 -0.13327111
# [3,] -0.3573965  0.01882691  1.000000000  0.09358042
# [4,] -0.8570117 -0.58359258  0.249930947  1.00000000
`````` ## Extract Pairwise Correlation from Correlation Matrix in Spark

By : jmcalderon235
Date : March 29 2020, 07:55 AM
Hope this helps With some help I was able to find a solution:
Get the result into a local Array:
code :
``````import scala.collection.mutable.ListBuffer

val pairwiseArr = new ListBuffer[Array[Double]]()

for( i <- 0 to correlMatrix.numRows-1){
for(j <- 0 to correlMatrix.numCols-1){
pairwiseArr += Array(i, j, correlMatrix.apply(i,j))
}
}
``````
``````case class pairRow(i: Double, j: Double, corr: Double)

val pairwiseDF = pairwiseArr.map(x => pairRow(x(0), x(1), x(2))).toDF()
display(pairwiseDF
`````` ## How to apply a Pearson Correlation Analysis over all pairs of pixels of a DataArray as a Correlation Matrix?

By : user3275975
Date : March 29 2020, 07:55 AM
I wish this help you I have managed to solve my question. The script has become a bit long. Nevertheless, it does what it was previously intended.
The code is adapted from this reference ## Python Scipy spearman correlation for matrix does not match two-array correlation nor pandas.Data.Frame.corr()

By : Florin Miron
Date : March 29 2020, 07:55 AM
hope this fix your issue It appears that scipy.stats.spearmanr doesn't handle nan values as expected when the input is an array and an axis is given. Here's a script that compares a few methods of computing pairwise Spearman rank-order correlations:
code :
``````import numpy as np
import pandas as pd
from scipy.stats import spearmanr

x = np.array([[np.nan,    3.0, 4.0, 5.0, 5.1, 6.0, 9.2],
[5.0,    np.nan, 4.1, 4.8, 4.9, 5.0, 4.1],
[0.5,       4.0, 7.1, 3.8, 8.0, 5.1, 7.6]])

r = spearmanr(x, nan_policy='omit', axis=1)
print("spearmanr, array:           %11.7f %11.7f %11.7f" % (r[0, 1], r[0, 2], r[1, 2]))

r01 = spearmanr(x, x, nan_policy='omit')
r02 = spearmanr(x, x, nan_policy='omit')
r12 = spearmanr(x, x, nan_policy='omit')

print("spearmanr, individual:      %11.7f %11.7f %11.7f" % (r01, r02, r12))

df = pd.DataFrame(x.T)
c = df.corr('spearman')

print("Pandas df.corr('spearman'): %11.7f %11.7f %11.7f" % (c, c, c))
print("R cor.test:                   0.2051957   0.4857143  -0.4707919")
print('  (method="spearman", continuity=FALSE)')

"""
# R code:
> x0 = c(NA, 3, 4, 5, 5.1, 6.0, 9.2)
> x1 = c(5.0, NA, 4.1, 4.8, 4.9, 5.0, 4.1)
> x2 = c(0.5, 4.0, 7.1, 3.8, 8.0, 5.1, 7.6)
> cor.test(x0, x1, method="spearman", continuity=FALSE)
> cor.test(x0, x2, method="spearman", continuity=FALSE)
> cor.test(x1, x2, method="spearman", continuity=FALSE)
"""
``````
``````spearmanr, array:            -0.0727393  -0.0714286  -0.4728054
spearmanr, individual:        0.2051957   0.4857143  -0.4707919
Pandas df.corr('spearman'):   0.2051957   0.4857143  -0.4707919
R cor.test:                   0.2051957   0.4857143  -0.4707919
(method="spearman", continuity=FALSE)
`````` 