logo
down
shadow

Match information from a correlation matrix according to their p-value cut off


Match information from a correlation matrix according to their p-value cut off

By : user2949983
Date : November 17 2020, 04:28 AM
like below fixes the issue I have used rcorr function of Hmisc library for calculation of correlations and p-values. Then extracted pvalues to Pval matrix and correlation coefficients to corr matrix. , You could try
code :
indx <- which(Rvalue==1 & Pvalue < 0.05 & !is.na(Pvalue), arr.ind=TRUE)
d1 <- data.frame(rN=row.names(Rvalue)[indx[,1]], 
               cN=colnames(Rvalue)[indx[,2]], Pval=signif(Pvalue[indx],
                                                                digits=4))

head(d1,2)
#     rN    cN Pval
#1 41700 41699    0
#2 41701 41699    0
 f1 <- function(Rmat, Pmat, cutoff){
   indx <- which(Rmat==1 & Pmat < cutoff & !is.na(Pmat), arr.ind=TRUE)
    d1 <- data.frame(rN=row.names(Rmat)[indx[,1]], 
              cN=colnames(Rmat)[indx[,2]], Pval=signif(Pmat[indx],
                                                            digits=4))
 d1}

 f1(R1, P1, 0.05)
 #  rN cN  Pval
 #1  B  A 0.021
 #2  C  A 0.018
 #3  D  A 0.001
 #4  A  B 0.021
 #5  A  C 0.018
 #6  E  C 0.034
 #7  A  D 0.001
 #8  C  E 0.034

 f1(R1, P1, 0.01)
 #  rN cN  Pval
 #1  D  A 0.001
 #2  A  D 0.001

 f1(R1, P1, 0.001)
 #[1] rN   cN   Pval
 #<0 rows> (or 0-length row.names)
set.seed(24)
R1 <- matrix(sample(c(0,1), 5*5, replace=TRUE), 5,5, 
            dimnames=list(LETTERS[1:5], LETTERS[1:5]))
R1[lower.tri(R1)] <- 0
R1 <- R1+t(R1)
diag(R1) <- 1


set.seed(49)
P1 <- matrix(sample(seq(0,0.07, by=0.001), 5*5, replace=TRUE), 5, 5,
       dimnames=list(LETTERS[1:5], LETTERS[1:5]))

P1[lower.tri(P1)] <- 0
P1 <- P1+t(P1)
diag(P1) <- NA


Share : facebook icon twitter icon
List Highest Correlation Pairs from a Large Correlation Matrix in Pandas?

List Highest Correlation Pairs from a Large Correlation Matrix in Pandas?


By : African Web Develope
Date : March 29 2020, 07:55 AM
I wish this helpful for you You can use DataFrame.values to get an numpy array of the data and then use NumPy functions such as argsort() to get the most correlated pairs.
But if you want to do this in pandas, you can unstack and sort the DataFrame:
code :
import pandas as pd
import numpy as np

shape = (50, 4460)

data = np.random.normal(size=shape)

data[:, 1000] += data[:, 2000]

df = pd.DataFrame(data)

c = df.corr().abs()

s = c.unstack()
so = s.sort_values(kind="quicksort")

print so[-4470:-4460]
2192  1522    0.636198
1522  2192    0.636198
3677  2027    0.641817
2027  3677    0.641817
242   130     0.646760
130   242     0.646760
1171  2733    0.670048
2733  1171    0.670048
1000  2000    0.742340
2000  1000    0.742340
dtype: float64
Computing the correlation between the auto-correlation and cross-correlation for each pair of rows in a matrix

Computing the correlation between the auto-correlation and cross-correlation for each pair of rows in a matrix


By : Kris Cochrane
Date : March 29 2020, 07:55 AM
this one helps. Make sure you set plot to FALSE for acf, ccf. Then, you can just wrap your code in a call to outer to provide every pair of i and j values. Note that since outer expects a vectorized FUN (e.g. *), we need to vectorize your function:
code :
set.seed(1)
m <- 4
n <- 10
mat <- matrix(rnorm(m*n), nrow = m, ncol=n)
lag <- 5

outer(1:nrow(mat), 1:nrow(mat),
  Vectorize(
    function(i, j) {
      acf.i <- acf(mat[i,],lag.max=lag, plot=F)
      ccf.i.j <- ccf(mat[i,],mat[j,],lag.max=lag, plot=F)    
      cor(acf.i$acf,ccf.i.j$acf[(lag+1):(2*lag+1)])
} ) )
#            [,1]        [,2]         [,3]        [,4]
# [1,]  1.0000000  0.47035200 -0.006371955 -0.85880247
# [2,]  0.4133899  1.00000000 -0.462744858 -0.13327111
# [3,] -0.3573965  0.01882691  1.000000000  0.09358042
# [4,] -0.8570117 -0.58359258  0.249930947  1.00000000
Extract Pairwise Correlation from Correlation Matrix in Spark

Extract Pairwise Correlation from Correlation Matrix in Spark


By : jmcalderon235
Date : March 29 2020, 07:55 AM
Hope this helps With some help I was able to find a solution:
Get the result into a local Array:
code :
import scala.collection.mutable.ListBuffer

val pairwiseArr = new ListBuffer[Array[Double]]()

for( i <- 0 to correlMatrix.numRows-1){
  for(j <- 0 to correlMatrix.numCols-1){
    pairwiseArr += Array(i, j, correlMatrix.apply(i,j))
  }
}
case class pairRow(i: Double, j: Double, corr: Double)

val pairwiseDF = pairwiseArr.map(x => pairRow(x(0), x(1), x(2))).toDF()
display(pairwiseDF
How to apply a Pearson Correlation Analysis over all pairs of pixels of a DataArray as a Correlation Matrix?

How to apply a Pearson Correlation Analysis over all pairs of pixels of a DataArray as a Correlation Matrix?


By : user3275975
Date : March 29 2020, 07:55 AM
I wish this help you I have managed to solve my question. The script has become a bit long. Nevertheless, it does what it was previously intended.
The code is adapted from this reference
Python Scipy spearman correlation for matrix does not match two-array correlation nor pandas.Data.Frame.corr()

Python Scipy spearman correlation for matrix does not match two-array correlation nor pandas.Data.Frame.corr()


By : Florin Miron
Date : March 29 2020, 07:55 AM
hope this fix your issue It appears that scipy.stats.spearmanr doesn't handle nan values as expected when the input is an array and an axis is given. Here's a script that compares a few methods of computing pairwise Spearman rank-order correlations:
code :
import numpy as np
import pandas as pd
from scipy.stats import spearmanr


x = np.array([[np.nan,    3.0, 4.0, 5.0, 5.1, 6.0, 9.2],
              [5.0,    np.nan, 4.1, 4.8, 4.9, 5.0, 4.1],
              [0.5,       4.0, 7.1, 3.8, 8.0, 5.1, 7.6]])

r = spearmanr(x, nan_policy='omit', axis=1)[0]
print("spearmanr, array:           %11.7f %11.7f %11.7f" % (r[0, 1], r[0, 2], r[1, 2]))

r01 = spearmanr(x[0], x[1], nan_policy='omit')[0]
r02 = spearmanr(x[0], x[2], nan_policy='omit')[0]
r12 = spearmanr(x[1], x[2], nan_policy='omit')[0]

print("spearmanr, individual:      %11.7f %11.7f %11.7f" % (r01, r02, r12))

df = pd.DataFrame(x.T)
c = df.corr('spearman')

print("Pandas df.corr('spearman'): %11.7f %11.7f %11.7f" % (c[0][1], c[0][2], c[1][2]))
print("R cor.test:                   0.2051957   0.4857143  -0.4707919")
print('  (method="spearman", continuity=FALSE)')

"""
# R code:
> x0 = c(NA, 3, 4, 5, 5.1, 6.0, 9.2)
> x1 = c(5.0, NA, 4.1, 4.8, 4.9, 5.0, 4.1)
> x2 = c(0.5, 4.0, 7.1, 3.8, 8.0, 5.1, 7.6)
> cor.test(x0, x1, method="spearman", continuity=FALSE)
> cor.test(x0, x2, method="spearman", continuity=FALSE)
> cor.test(x1, x2, method="spearman", continuity=FALSE)
"""
spearmanr, array:            -0.0727393  -0.0714286  -0.4728054
spearmanr, individual:        0.2051957   0.4857143  -0.4707919
Pandas df.corr('spearman'):   0.2051957   0.4857143  -0.4707919
R cor.test:                   0.2051957   0.4857143  -0.4707919
  (method="spearman", continuity=FALSE)
Related Posts Related Posts :
  • Conditionally removing rows from a matrix in R
  • Using a loop to find P(-1.5<Y<1.5) for a range of sample sizes
  • R-Count and list the maximum count row by row
  • Include Iverson Bracket in R documentation
  • update a data frame and environment in R
  • How to write dynamic cumulative multiple in R
  • format time using as.POSIX in R
  • Change the class of multiple columns
  • Remove period and spaces within column headings nested in a list of data frames
  • R: error message --- package error: "functionName" not resolved from current namespace
  • labels with geom_text ggplot2
  • Passing mongodb ISODate in R
  • Importing "csv" file with multiple-character separator to R?
  • Change row names of a table obtained from a lm regression using xtable function
  • R language iterate over R object
  • How do you delete the header in a dataframe?
  • Re coding in R using complicated statement
  • accumulating functions and closures in R
  • How do you combine two columns into a new column in a dataframe made of two or more different csv files?
  • Twitter authentication fails
  • Summing Values of One Vector Conditional on Values of Another Vector
  • draw cube into 3D scatterplot in RGL
  • lme4 translate formula to code in 3-level model
  • How to draw single axis plot in R
  • Combine geom_tile() and facet_grid/facet_wrap and remove space between tiles (ggplot2)
  • Use snpStats with R version 3.0.1
  • Makefile gives strange error while compiling markdown file into .docx file
  • How to determine whether a points lies in an ellipse
  • Summarize data already grouped in r
  • Is the bigvis package for R not available for R version 3.0.1?
  • Operator overloading in R reference classes
  • How to enable user to switch between ggplot2 and gVis graphs in R Shiny?
  • Is there an easy way to separate categorical vs continuous variables into two dataset in R
  • Correct previous year by id within R
  • Installation of rdyncall package for R
  • ggplot2 plot that evaluates the percentage and mean of a third variable at intersecting points
  • Error Handling with Lapply
  • data.table - split multiple columns
  • How to compute the overall mean for several files in R?
  • R: Graph Plotting: Subscripts in the legend like LaTeX
  • Restructuring data in R
  • Distance of pointsfrom cluster centers after K means clustering
  • R incorrect value of date function
  • Package "Imports" not loading in R development package
  • r - run a user defined function several times by taking column elements as parameters
  • Create input$selection to subset data AND radiobuttons to choose plot type in Shiny
  • Generate crosstabulations from dataframe of categorical variables in survey
  • Restructure output of R summary function
  • New behavior in data.table? .N / something with `by` (calculate proportion)
  • search certain number vector in R
  • R version doesn't support quartz graphic device - RStudio won't plot
  • Referencing a function parameter in R
  • How to synchronize signals using a cross-correlation and FFT in R?
  • Plotting coefficients and corresponding confidence intervals
  • passing expressions to curve() within a function
  • More effective merging of matched column with duplicates in data.table
  • Easy way to export multiple data.frame to multiple Excel worksheets
  • R Foreach Iterator - Walkforward
  • Table format and output in R
  • Restructuring data and duplicating rows in R
  • shadow
    Privacy Policy - Terms - Contact Us © ourworld-yourmove.org