tttjjj/elt_bench huggingface dataset:
To find the correlations between the constructs you need numeric data for each construct.  So, do you have numeric data?  What are the scale values for each of the items? Are those scale points numeric or categorical?  Do you plan to compute one overall value for each construct, perhaps by adding up the values for each of the 3 items?  Or do you want to consider each item separately?  Also, it might be helpful if you would tell us why do you want to find the correlations between the constructs, since the purpose drives the choice of an analysis.
For example, fit the model: $\\lambda_1 p + \\lambda_2 (1-p)$, where $p = 1/(1 + exp(-\\beta t) )$? That would work I believe, and would allow for smooth transitions. You are correct that inference on $\\beta$'s slope could determine if a switch point exists. I really like this, you should explore it more.
so it means that we need to correct the distance estimated from  standard normal distribution hypotheses for non-standard data?
@user815423426 I think the comparison depends on the loss function you have. Diebold and Mariano (2002) have a nice paper studying your question. They proposed some statistical tests comparing the "generalization" performance. I don't know how to set up a link in comments. The paper is: Diebold, Francis X., and Robert S. Mariano. "Comparing Predictive Accuracy." Journal of Business & Economic Statistics 20.1 (2002): 134-144.
That's a good explanation, but not for a nontechnical layperson. I suspect the OP wanted to know how to explain it to, say, the MBA who hired you to do some statistical analysis! How would you describe MCMC to someone who, at best, sorta understands the concept of a standard deviation (variance, though, may be too abstract)?
Can you be more specific? It is hard to tell what you want to know from the question. Also the one tag seems to bear no resemblance to the question.
@gung A big +1 for your thorough comment. I've been teaching R for a long and I finally came to similar conclusion: people having no lightweight understanding of programming or a command-line prompt will likely face some difficulties dealing with R. It gets worse when we have to make students familiar with statistics using R in few hours...
Can you elaborate on this short notice?
I am just writing a general purpose stats library so I guess I would want it as accurate as possible while still considering run-time complexity. The arguments would simply be the degrees of freedom and the alpha value. Could you point me in the direction of one or two of these formulas  since all my Google searches lead to excel and general t-test tutorials. thx
I thought about this because you have been using $R^2=SS_{reg}/SS_{tot}$, while I was using the equation I posted in the OP. These two equations are equivalent to each other only when the sum of the prediction errors is zero. Hence, in my OP, $SS_{res}=\\sum_i(f_i-\\bar y)^2$ does not change while $SS_{tot}$ changed, and hence the $R^2$ is changed.
This is, almost verbatim, what I had in mind. (+1)
thanks for your reply :)
In plain English, if the log likelihood of the data increased (as in AIC decreased), then that means ... ?  FYI, the reason for slightly modifying the data is that I have what I measured, but some folks think that one should apply a dynamic correction based on location.  I want to be able to say whether that makes the fit better or not.
"physics experiment. [...] y-intercept [...] would lead to completely incorrect stuff." But if the experimental data indicates an intercept (btw, you could check whether the confidence interval for the line covers the origin), this would make me think *very hard* where the intercept comes from. I'm analyitical chemist. In analytical chemistry, we also have a bunch of relationships that *should* be linear without intercept. But they hardly ever are in practice, because of the nitty-gritty details of instruments and measurements. Thus, we usualy see suppressing the intercept as a very bad idea.
The R package `AICcmodavg` has a QAIC function & references Burnham & Anderson (2002), *Model Selection & Multi-Model Inference*.
Voting to close. There are several questions that address related matters. Here are some: [Evaluate definite interval of normal distribution](http://stats.stackexchange.com/questions/7200), [Is it possible to analytically integrate x multiplied by the lognormal probability density function?](http://stats.stackexchange.com/questions/9501), [Probability distribution value exceeding 1 is OK?](http://stats.stackexchange.com/questions/4220) and [How can a probability density be greater than one and integrate to one](http://math.stackexchange.com/questions/105455), to give you a start.
A number of distributions will fit this bill. Two other candidates that come to mind are Weibull and Poisson distributions. Googling "how to fit a distribution in r" spits out this pdf: http://cran.r-project.org/doc/contrib/Ricci-distributions-en.pdf
How are you determining suitability? There are also other distance metrics that might be more general.
I believe this question is answered at http://stats.stackexchange.com/questions/9131/obtaining-a-formula-for-prediction-limits-in-a-linear-model/9144#9144, which provides the correct formula for regression prediction intervals.  Implement that and let us know how it compares to the `R` output.
Great -- I think I really like this answer, but I need to think about it a little more.  Thanks!
@gung seems like a legitimate statistical question to me. The GBM package supplies the Deviance used for adaboost but it is not clear to me either what f(x) is and how to back transform to a probability scale (perhaps one has to use Platt scaling). http://cran.r-project.org/web/packages/gbm/vignettes/gbm.pdf
As there are 15 variables, combinations can vary from 2 to 15 combinatory sets (e.g. var1+var2 or var1+var2+var3+var4+var5+var6.......+var15). So the number of combinations can be very large....
+1, I think it would help the OP if you would put some of that into your answer & unpack it a little bit. Eg: how what is posted differs from the logic of (say) the Hausman test, etc., whether this test is likely to be useful / accurate, etc.
Thank you Rob. Can you confrm that I must increase the nMChdr value? Tank you.
Interesting idea.  It seems that "negative bursts" where the frequency suddenly goes *down* would also be detected this way.  It's not clear whether this is intended, but the multiple references to "popular" in the question suggest not: a burst is solely a sudden temporary *increase* in the frequencies.
@RobertHarvey I don't think you read my question right; I already have the average accuracies and their corresponding standard error. I'm trying to come up with a proper way to do hypothesis testing using these values to figure out the significance of what I am observing.
@shadi: i have difficulty understanding what you mean by 'track the outliers'.
Excellent comments on a pervasive issue. It's important to propagandise for thoroughly quantitative thinking here. There is already too much emphasis on crossing thresholds, e.g. above some level disaster, below some level comfort.
Decrease your alpha. You are diverging during gradient descent
It is unclear what "explanatory insights" can be gleaned in this way, if by that phrase you imply causality.
It is p(A or B).  I'll correct my questions.  Thanks.
I wouldn't suggest resorting to python to accomplish this task unless you need it to be standardized to run on multiple datasets with different variable names. The UCLA website is full of useful examples though of interest to anyone who uses SPSS.
(-1) $\\sum_{i=1}^n Y_i/n$. $\\sum_{i=1}^n \\left(Y_i - \\sum_{i=1}^n Y_i/n\\right)^2/n$. You're welcome. :)
You will get better advice if you state why you want to constrain the coefficients. Because you can take any linear transformation to the variables and change the coefficients (but still have the same fit) the constraint is not always meaningful unless transforming the variables would also change the constraint.
@ Greg, Thank you for your reply. What are the cases when the median cannot be used with ordinal level data?
By the way, I guess that in the context of one single predictor, logistic regression and ANOVA would produce the same result.
If you're trying to look at post-hoc for a K-W, take a look at Dunn's test
The answer is that you are summing the deviations of values around their mean and that sum is zero.  But what, precisely, are $x$, $m$, and $m_i$?  How are $m$ and $m_i$ related to $\\mu$ and $\\mu_i$?  The quality of answers will depend on how accurately we guess but you're forcing us to do an awful lot of guessing!
The question is ill stated. Without any measure of the accuracy/reliability of the classifiers (such as true positive fraction), we can't be sure that even if A classified x as LA1 100% of the time, x is ever an element of LA1. You should also specify whether these classes are mutually exclusive (like died/living)
Hi again,

I know really a little in math, so I have problem with the notations and terms here in making contrast matrix: I don't know what tensor product is? so can anyone explain what these formulae mean? 
M_1 : M_2
M_1 * M_2
Maybe some explanation with examples related to above case study.
Thanks
Bioinformatics is simply the use of computers to study biological questions. Disciplines are usually defined by the questions they ask, not by their methods, so bio-informatics should be part of biology in my opinion. It has a special name because biologists are very bad with computers, so people who can do it must have a special label. I am not sure that in 50 years, when biologists are better at IT and math, bio-informatics will still exist.
See also this paper for a more extended discussion: http://stata-journal.com/article.html?article=st0187.
*How* does the dependence manifest itself in your case? Can you describe your situation in more detail?
I wonder if the answer key has a typo (if I just did it right, I think it should be nearer to 0.213). Another possibility might be if a continuity correction was used.
Your question is well-phrased and interesting, although I'd challenge the implication of a distinction between a *programatic solution* versus a *statistical or machine learning* solution". Any machine learning or stats based approach will be in the form of a program.
Why do you want to combine them?
Please take a while to ponder and double check your answer. This is about the 4th different expression you provide as the answer.
To get the values back just do `y = y_esc * sd(y) + mean(y)`, but that would mess with the model properties i guess, so i'm also waiting a more technical answer too!
@Pardis: approximately 2.6k observations
Since I wrote the stuff about Bessel functions etc in an answer to the question pointed out by @MichaelMayer, let me add that for the special case of iid Gamma rvs with shape parameter $2$, no Bessel functions come up because we are evaluating integrals of the form $$f_{X-Y}(z)=  \\int_{0}^\\infty (y+z)\\exp(-(y+z))\\cdot y\\exp(-y)\\,\\mathrm dy$$ which can be evaluated without resorting to Bessel functions; Gamma functions, or even more simply, factorials, suffice.
"Accurate" is probably a red herring; it's no good saying "well by the dictionary it's the right way to say it" if few people understand you. What's really important is to convey your message - it matters whether you're *correctly understood*. And that's why my answer emphasizes that and discusses what might be misunderstood (most especially, I was trying to get you to avoid the common-but-bad "takes 300% less time" or "takes 3 times less time" type of expression, but avoided actually saying them). In this case, "4 times faster" is both actually correct and likely to be understood correctly.
I would fix a general enough structure and select the best model for the mean of the response; and then tweak the covariance structure to improve efficiency.
"Ordinal" simply means "ordered": it does not invariably mean the data are discrete! The continuity assumption says otherwise. And yes, of course "equal to" is meaningful: but the import of the continuity assumption is that the chance of obtaining two equal values will always be zero, no matter how much data you have. Therefore, if your data have an appreciable number of ties, the sign test is not applicable to them. (A few ties can be tolerated; people have developed adjustments in the test statistic to handle them.)
I might begin by estimating a logit/probit type model by including the lags . However, I believe there are issues with correcting for autocorrelation in these types of models, so I would hesitate to draw any statistical inferences.
@ ttnphns, thanks!  I got it!  I was reading up on centering matrix (ya, I am weak in this subject) and updating the question and then only see your answer after edit.  Many thanks!  I have learned a lot from you and @whuber.  Very much appreciate the time and kindness from both of you.
A Dirichlet process under what concentration parameter? It is kind of equivalent to the same original question, k-means under what k? Though I agree that we understand better the Direchlet distribution that the behavior of some complex algorithm on some real-world data.
@curious_cat: Or, perhaps to be even more evocative, drop the `y` altogether. :-)
@SideshowBob See edits to my answer. I think you really need a verification dataset in your hypothetical example.
I found one very interesting report: [Fast computation of the exact null distribution of Spearman's $\\rho$ and Page's $L$ statistic for samples with and without ties](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.7124&rep=rep1&type=pdf). The authors cite many classical references for the computation of the exact distribution of $\\rho$ for small samples, and they also provide an algorithm. I would be interested in your opinion on this article, @koskos!
I posted some sample data and updated the question with my goal. I think you have the answer I just dont understand your response yet.:)
Hi Jacob and welcome! [This post](http://stats.stackexchange.com/a/13173/21054) provides some excellent explanations of `lmer`s formulas and their interpretation. What the "right" model is depends greatly on your research question and if it makes sense given your data.
It is unclear what you are testing in the second line.  Exactly what is being compared to what?  Treatment after *vs* treatment before? Treatment after vs. control after? Something else?
@Bob: ;-) see also http://go.helms-net.de/stat/images/boxplots.gif
Done Kruskal-Wallis, and it seems to have worked. Hopefully that'll be ok! Thanks a lot!
@Mark In light of the revised question, I agree that looking at a copula-based description is a good suggestion.  There's nothing terribly difficult about copulas: because you have the marginals, you can apply the probability integral transform to make the three variables (R, G, B) uniform on [0,1].  A copula merely describes their multivariate distribution in these terms.  In effect, it eliminates the parameters from the marginals to allow you to focus on describing their multivariate dependencies.  It affords the non-parametric approach you're asking for.
Alternatively, you can use the properties of the normal distribution to replace the last line by `samples <- rnorm(N)*sds[components]+mus[components]`. I find it easier to read :)
(+1) Good description. I think the other reason the former notation is so popular is that it more naturally translates to intuitive notions in applications. (Voted up several hours ago.)
A moving average MA(1) model for a timeseries $X_t$ leads to a specific tridiagonal covariance matrix. This reads as $X_t = \\varepsilon_t - \\theta  \\varepsilon_{t-1}$ where $\\varepsilon_t$
is Gaussian White Noise and $\\theta$ the MA coefficient. The two parameters  $\\theta$ and the  variance $\\sigma^2_\\varepsilon$
can have independent priors, e.g. uniform and gamma.
By specifying a slowly varying $\\theta$ and/or variance, you will 
get more general tridiagonal matrices. 
This may be a hint for a hierarchical formulation of your model
if the timeseries formulation makes sense in your context.
the test won't be meaningful unless you know the full sample size.  Proportions of 1.0 / 0.0 / 0.0 / 0.0 mean very different things if they are from a sample of size 1 as opposed a sample of size 100.
@suncoolsu thanks
@DilipSarwate Thanks for that suggestion. Yes, it was ambiguous; I have made the intent clearer.
Well that is changing a continuous variable into a categorical one it still throws out information in the distance measure.  It is better than binary but not as good as using the distance itself.  The skewness can be dealt with,  As gui11aume suggested a log transformation might work or you could try to find a best Box-Cox power transformation.
@levon9 General idea is to make title show what the question is about, preferably using proper words. The sentence capitalization policy is just to establish some level of order (people do spit less on clean floors).
How about white noise that you could record on some unused radio band?
+1 for the effort
That is correct assuming $(A, B_1, \\ldots, B_k)$ are *uncorrelated.*  Biologue, see http://stats.stackexchange.com/questions/31177 for an account of properties of variances and [search our site](http://stats.stackexchange.com/search?tab=votes&q=[variance]%20sum) for even more information.
Consider http://stats.stackexchange.com/questions/67571/how-can-i-group-numerical-data-into-naturally-forming-brackets-e-g-income
The point of my question, Adhesh, which I should have made more clearly, is that there is a difference between using these values as if they were continuous variables and using them as categorical variables.  I suspect you might be treating the codes as actual numbers (continuous variables).  That would make most of your interactions next to meaningless and could explain why they give similar results.
The later is true: I'm given parameter sets for distributions.
There are many possible models. The choice depends on the nature of your real problem.
Perhaps moving standard deviations would be more informative.
Usually, when people use transformations to improve characteristics of the fit (like the normality of the residuals), the transformation is applied just to the LHS / $y$. Of course you *can* transform the RHS too, but that typically addresses different issues, such as curvature in the data that the model is missing. However, your non-linear model ought to be flexible enough to capture whatever curvature is there, I would think. Note also, that there is no reason the *same* transformation needs to be used on both sides.
@gues43434 Thanks for reading so carefully but no, I didn't mean that: "$B$" stands for any single intermediate node between $A$ and $C$.
This is like template matching. One problem with this is that the input histogram bin size is dynamic, and the convolution will require creating a template histogram of a given size dynamically.
I have never seen economic significance defined like that. Do you have any references for that use? In my experience, economic significance means something akin to "the estimated effect size / coefficient is relevant from a policy and / or theory perspective, and is not only a statistical curiosity".
It appears that your signs are munged up since the last two displayed equations are _inconsistent_..  If by $\\sigma_{XY}$ you mean $\\operatorname{cov}(X,Y)$, then $$\\begin{align}\\operatorname{cov}(X,Y) &= E[XY]-E[X]E[Y] &= E[XY]-\\mu_X\\mu_Y\\\\ E[XY]&= \\operatorname{cov}(X,Y) + E[X]E[Y] &=\\sigma_{XY}+\\mu_X\\mu_Y\\end{align}$$
What is the meaning of stationary? The condition that $X_{n+2}=-X_n$ puts a constraint on the random variables since there are really only two random variables $X_{\\text{odd}}$ and $X_{\\text{even}}$ in this random process.
A monotonic increasing function will have positive correlation even though it is nonlinear.But the correlation will be weaker than a linear function with the same residual noise variance. The Pearson correlation does measure degree of linearity. However the statement that for Pearson's r nonlinear functions are treated as noise is not a correct interpretation.  Do look at the influence function for the correlation to find any specific points that have high influence on the correlation either for increasing or decreasing it.
The original scale for $\\rho$ is the best, and is directly related to a "majority concordance of triplets of observations" index.  But I also don't have much of a problem communicating $\\rho^2$.  A (much) different way to view this is the $\\rho$ is almost a special case of a regression test arising from the proportional odds ordinal logistic model, and that model has a generalized $R^2$ measure based on log likelihood explained, that is akin to variance explained.
$(n/p+n/[1-p])^{-1}$ is what you have above. Just factor the $n$ out, put on a common denominator and then take the reciprocal.
See any of these (only a sampling of many): [1](http://stats.stackexchange.com/questions/56194/is-it-possible-to-use-a-two-sample-t-test-here) [2](http://stats.stackexchange.com/questions/53053/mann-whitney-or-two-tailed-t-test) [3](http://stats.stackexchange.com/questions/54913/t-test-equivalent-for-samples-given-a-distribution-of-unknown-form) [4](http://stats.stackexchange.com/questions/52047/when-sample-size-is-small-and-distribution-is-not-normal-should-i-use-wilcoxon-o) [5](http://stats.stackexchange.com/questions/49465/mann-whitney-for-non-normal-distributions-with-n20)
Do the errors come in $x$ or in $y$ or both?  In what form do you expect noise to enter (multiplicative, additive, etc.)?
Ok..that was helpful
@COOLSerdash you might want to give the answer in excel since that is what they seem to be using.
@Roronoa Zoro: and so is the answer. It is explaining how to find the parameters for radial basis functions based SVMs (C and \\lambda in Smola's blog post) in 3|Cs| time as opposed to |\\gammas||Cs| like it is done in the case of grid search.
Is there another way or place I could ask this?
@Zach: Please feel free to reconsider your check now that cardinal and shabbychef have taken the time to give full answers.
It's old homework, so I have the answers, but I don't understand why. The answer to that question is 0.0032. How do you calculate joint probability?
Thanks a million for your valued comments. They have certainly helped
@whuber. Thanks, I went through that thread, and while interesting (and maybe I am missing something, since I don't fully understand some of the answers), it wasn't clear to me how they addressed the difference between `nonimal` vs `actual` coverage probabilities.
yes indeed this post is adressed only to people able to access this book. The proof i refer to is really lengthy, so it would be pretty hard to state it completely, especially because i had to explain all the notation and I do not know how to include LaTex style formulas. Maybe i can include some pdf in my next post. ANYWAY...thanks for trying to help!
Hi Paul, welcome to CV. I have corrected some of the spelling. Please check that it still says what you want.
Ok, thank you. I'look into it.
@StasK: Just noticed this comment. How true. SAS is based on punched cards and JCL from the 1970's or earlier. I've never gotten far with it because of the smell. Just bought Stata 13 yesterday and I really like it, though as a programmer I find it's not as flexible as R. (The whole dataset versus return variables thing makes me a bit claustrophobic.) Still, it's my second statistical tool now and I'd recommend it.
Take a look at multiple logistic regression.
Correct the term inside the expectation.
For more context, you can see what I am doing here: http://marktheballot.blogspot.com.au/2013/04/towards-more-integrated-primary-vote.html
"... way to determine if the violation is excessively non-normal is to take a look at how similar the correlations between r and rho are". This is wrong statement. Magnitude of both coefficients has nothing to do with notion of _normality_ (still magnitude of Pearson has something to do with shape of distribution)
Are you forecasting, or evaluating a recent result?