tttjjj/elt_bench huggingface dataset:
@probabilityislogic: The SEM of the three experiment means is 5.168  (not 2.98 as you wrote), and the confidence interval I gave in the original post (17.4 to 61.9) is correct. The SEM is computed from the SD (8.95) by dividing by the square root of n (square root of 3). You divided by n (3) instead.
"any test on contingency tables" doesn't account for the nature of the response variable (which is ordinal in this case) nor the particular design (Fisher's test assumes that margins are fixed). A trend test might be interesting, depending on the sample size per strata.
Actually, I think it really is about statistics, so it's probably good that it came here.
Hi, just wanted to let you know that I've not found any more relevant examples of this approach. I was trying to give you something that would be relatively simple to implement give your sampling scheme. Whuber is rather more advanced in stats than me, so I'd also take a look at the referenes given in the comment if you have the time. I tried to steer away from spatial statistics and go for something more basic.
Wow, I would like to up vote 10 times for you! Thanks!
Interpretation: The three variables are all estimates of the same thing (let's call it _the thing_). $A$ is known to be the best measure, but for various reasons (e.g. time to calculate), $B$ and $C$ are preferable. I'm assuming that a better correlation with $A$ means a better correlation with _the thing_. I am not really interested in the correlation between $B$ and $C$, but given their high dependence, it will be high.
@Michael: Might be, but something in the phrasing of the comment triggers old memories of reliability analysis & shifted failure-time distributions.
Thanks for your comment; I think you are correct.  However, in practise it seems that Ward's linkage is often used with non Euclidean distances.  I am still not sure what the implications of this might be.
I've trawled the internet but I can't find anything that really answers my question - leaps or AIC?
Also for the SVD you can write (or just use) the algorithm for $\\mathbf X^T$ which has the opposite runtime behaviour (because the decomposition has symmetry wrt. transposing $\\mathbf X$ ) and otherwise the same numeric behaviour. Numeric stability is an important point (+1) - though I guess it depends a lot the data whether that matters. E.g. I typically far too few cases and noisy measurements - so the stability of my models is typically not limited by the numeric stability of underlying SVDs but by the patients/cell culture batches or instrumental noise.
Are these a time series (e.g. 36 months), in which case we're going to have autocorrelation issues, or are they different districts, in which case we may (or may not) have spatial correlation issues?
`lm` uses the residual error as the error term for all effects; when there are effects that should use a different error term, `aov` is necessary (or instead, using the results from `lm` to compute the F-stats manually).  In your example, the error term for `factor` is the `id:factor` interaction, which is the residual error term in an additive model.  Compare your results to `anova(lm(value~factor+id))`.
These read like [routine textbook-style](http://stats.stackexchange.com/tags/self-study/info) questions, suggesting that perhaps they should be tagged `self-study`
Couldn't you run N (where N is the number of products) separate logistic regressions, one per product?
This way you'd remove the dependent units problem, but you'd have to interpret 4 different models.
@whuber: Hi, Thanks for your comments. I am not that familiar with the Gamma, but I would like to learn more from you. Basically, those parameters estimates come from a paper "Understanding individual human mobility patterns"[Nature](http://www.nature.com/nature/journal/v453/n7196/abs/nature06958.html). (Equation 1) I want to generate the random set for step size.
@Bill I don't know if I understand what you mean. Why use a variance analysis when I know variance is huge and not reproducible from one experiment to another?
Yes, inverting the matrix in the normal equations isn't the best solution and isn't the way most packages actually solve linear regression. But it's still true that any way of doing it is more or less dimensionally-agnostic when expressed in matrix form. It is usually a good idea to consider ridge-regression or lasso regression rather than standard regression.
By random I mean that each value in the range 0-N inclusive has an equal probability (I'm sure there's a better term for that I just do know the words). What I should have said with regards to your second comment is the child at index i receives a random value in the range 0-N. And yes N is the same as it was in the first paragraph. Thank you for helping me add some detail to this question!
In what form is the probability measure specified? This will determine what is "easy" to do and what is not. For instance, let $X,Y$ be independent standard normal variables and $A_1: X^4+Y^4\\ge 2500,$ $A_2: X^2Y^2\\ge 625.$  This is almost impossible to do with brute-force sampling from the joint distribution of $(X,Y)$, since $\\Pr(A)\\lt 10^{-11}$, but--because that distribution can be manipulated analytically--it is easy to compute the ratio (which is $0.591\\ldots$) numerically or even with clever Monte-Carlo simulation. Would such methods be accessible in your circumstance?
(+1) For spotting this. [Baklizi and Eidous (2006)](ftp://metron.sta.uniroma1.it/RePEc/articoli/2006-1-4.pdf) give a reference on this `The Mann-
Whitney statistic (Lehmann, 1975) is defined as ... A version corrected for ties is also given in Lehmann (1975).` The reference is [Nonparametrics: Statistical Methods Based on Ranks](http://www.amazon.co.uk/Nonparametrics-Statistical-Methods-Based-Ranks/dp/0387352120). Unfortunately I have not had the chance to have a look at it.
Daily Net Margin:  11, 12, 13, 14, 15, 16, 17

First day of 5-day Simple Moving Average: (11+12+13+14+15) / 5 = 13

Second day of 5-day Simple Moving Average: (12+13+14+15+16)/5 = 14

Third day of 5-day Simple Moving Average: (13+14+15+16+17)/5 = 15
Please, don't cross-post on [SO](http://stackoverflow.com/questions/6241181/gap-statistics-matlab-implementation) and [MatlabCentral](http://www.mathworks.fr/matlabcentral/answers/8811-gap-staticstics-implementation)!
@dsmcha, sorry to say this, but I don't think I like that example too much. The response is identical to two of the predictors? That's beyond pathological, in my view.
Yeah, I totally agree. I'm a biologist. My latest data set has 16 samples and over 400,000 variables.
Thanks @Stat for your answer. How to get these SS of squares using Linear Model (matrix approach in this case) approach?. Thanks
This seems reasonable to me. However, I am a bit confused by their example. They say that "The z-scores will be normally distributed with mean equal to zero and a standard deviation of one." but the inverse normal transformation they apply actually results in scores with a standard deviation of 1.486. Am I missing something or is there an error in the example?
If they are independent, you could draw the substitution rates from Beta distributions.
@gogurt: Right! You need a size alpha test in order to have infimum coverage probability $1-\\alpha$ (as can be seen from the last line in my answer)... so it's a typo of some sorts - probably the authors getting their own terminology confused :)
You should really post a sample of the data you are trying to analyze, as well as give a little more information about where your data is coming from.
I am not sure if this question is duplicated. The link provided did not answer my above question. I am confused with the word "repeated", hence the reason I am posing this question. I tend to confuse "within" with "repeated". Any clarification would be useful. Thanks
@Steve I added some details of the simulation.  You can use them directly in *Mathematica* or use them as a guide to implementing the simulation on some other platform.
+1, though I would recommend R's `lme4` package, where I believe your model would be `fm2 <- lmer (distance ~ age + (age | Subject), Orthodont)` and you could get the coefficients with the same `ranef` and `fixef` calls. Also, package `arm` provides the `fixef.se` and `ranef.se` functions that I find useful.
Is this a homework question? If so, you'd be better off posting the actual problem, especially since it may be a trick question along the same lines as [Bertrand's Paradox](http://en.wikipedia.org/wiki/Bertrand_paradox_(probability)). Actually, you should *first* take a look at [this](http://stats.stackexchange.com/tags/self-study/info) post and *then* proceed with the post-editing...
I do not understand the situation here - you might be confused about some definitions. How does a state have a probability distribution? What is the process here? I suggest you add some information about what you are trying to do.
Hi sean, I like this answer - can you please elaborate some more on your understanding here as it pertains to the example given? Thanks!
Of possible interest: [Empirical Relationship Between Mean, Median and Mode](http://stats.stackexchange.com/q/3787/930).
Hi Kyle, thanks for your answer. However, answers which consist essentially of little more than a link, or which are only about a sentence in length are generally not considered answers, but comments. In particular, link-only answers suffer from link-rot, so answers should have enough information to be useful even if the link no longer works. Can you please expand your answer a little more, perhaps giving a very brief outline of what it is/how it relates to factor analysis more generally?
Comparing/ranking variables using the stress-strength coefficient $P(X<Y)$ might not be a good choice. This has been discussed in terms of the [Pitman](http://en.wikipedia.org/wiki/E._J._G._Pitman) closeness because this method leads to [paradoxes](http://www.jstor.org/stable/2290692) and [might not be transitive](http://xianblog.wordpress.com/2011/11/15/on-pitman-closeness/).
@Ojtwist thanks, the first value is 1, second 0, for example for intensity={normal,high}, normal=1, high=0 ,  for entropy = {high,low} , high = 1, low = 0, for contrast ={yes,no}, yes=1, no=0  - I believe there isn't a -1 involved at all. Would be great if you do take the shot :)
Not exactly equal, but maybe about equal. If you want to let the data set the spacing, you could look at ridits.
Even better is NX, which has generally much improved performance characteristics.
You need to state what your goal of analysis is. Sampling is used to estimate a parameter of the population or the process, such as the mean or a proportion of certain response. "A probability model" sounds too generic to describe what it is that you want to do with these data.
I see. The points that I'm interested in evaluating tend to fall towards the extremes where I have relatively few actual samples in my empirical distribution. I figured that I might be able to make a 'better' estimate of the true CDF of the underlying distribution than I could get by computing the ECDF from the ranks of my samples by assuming that the PDF was smooth and applying KDE. In any case, I'm somewhat reassured that my estimated CDF ought to tend to overestimate p-values in the tails, since this will just make my hypothesis test overly conservative and I care more about type I errors.
By prediction score you mean $f(x)=(w,x)+b$? What about data normalization, is it used/not used in both cases? What about default parameters of the functions that you are calling, do they have the same values in both cases?
@Glen Thanks for explaining why it turns out that the formula makes sense, along with your matrix derivation below.
Shouldn't the variance rather be 100,000*0.3*0.7?
Come on, once you've got the tree, how hard is it to build the forest? And you're welcome.
"*and don't most models assume this?*" -- do they? What makes you say so?
Could you please tell me what is the R code when  ρ12 is the correlation between d1,2 and d2,t instead of two error series?
Many thanks Peter.
@gmacfarlane what is the difference? A data point can only be outlying with respect to a model!
16 regions. But the problem is solved, i don't need to reproduce R results anymore, just update the DB with the forecast values.
This looks like an interesting paper.  The Welch papers are classics.  They provided a frequentist approach. This is commonly used today with a t approximation which usually will have non-integer degrees of freedom.  It is commonly used in packages such as SAS and you even can find it in the Excel t test of its data analysis packages.  The work of Fisher is unfortunately not referenced.  I don't think the fiducial approach is emphasized.  The Segal 1938 paper is one that I am not familiar with but must represent the Fisherian approach at the time before fiducial inference went out of favor.
It returns `y2_predicted = []`!?
This question appears to be off-topic because it is about a request for code. Nor are such questions widely welcomed on Stack Overflow. It may be that software-specific forums would be more receptive, but in general "please give me complete code" questions have low success probability anywhere unless tutorials already exist (in which case you should be able to find them). For general advice, see http://stats.stackexchange.com/help/on-topic Rewriting this question to increase its statistical content might increase its appeal and pertinence here.
This is really a Stackoverflow question as it has to do with learning the R programming language.  With the current wording the question is only associated with statistics by virtue of R's focus on statistical analysis.
+1 This is interesting, thank you. I've updated the update.
+1 It occurs to me that your sense of "best" fit might not be the same as the OP's, who might only be seeking a parsomonious description of the data that gives a reliable picture of the tails. In that case, there are no small-sample concerns and asymptotic convergence is irrelevant (although nice).  BTW, did you notice that your answer is a special case of the probability plotting approach I suggested? The import of that is (1) you can easily compute the AD statistic using, say, `lm` without recourse to `optim` and (2) you can just as easily handle other families of distributions.
More precisely, that it _has existed_
Thanks Peter. I think i'm clear on my 1st Q now. SC stands for Switching Costs, CS 4 Customer Satisfaction, & CRI for Consumer Repurchase intention. By theory, SC positively effects CRI and negatively moderates the relationship between CS & CRI. In my results, although the strength of those relationships has been approved (p<.05) but the direction of the relationship is not confirmed & turned out 2 b vice versa which doesn't make sense. I don't know what's wrong here! I added the scatterplot to my Q & included some other specs in my 2nd comment 2 Aniko's response. Hope they will help!
look at this [PCA vs SVD](http://www.cs.princeton.edu/picasso/mats/PCA-Tutorial-Intuition_jp.pdf) tutorial, it explains how to estimate PCA through SVD. PCA may seem more intuitive for some people. It also discusses the dimensionality reduction
I've edited your answer to use mathjax formatting. Please double-check that I haven't inadvertently introduced errors.
Are you saying that ctree or tree in my example actually treats this unordered factor as an ordered factor and thus sends it into the "0" branch?
@Daniel: What you are calling a "Bayes multiplier" is not a probability
My apologies for lack of clarity. What I meant is that I am utilizing a separate method for determining the C values. Imagine a physical experiment where A is the force leftwards on a 1kg mass, B is the force rightwards, and C is determined as the net acceleration. According to Newton's law, F=ma, C should equal B-A, but it might not. In the case that it isn't, I need a method to determine the best estimate of A,B and C. Is that clearer?
This sentence: "*But data is not available on a single scale i.e. some are of form 0.00x while others are integers*" is a bit cryptic, especially if your data (as you write below) are concentrations of different elements. Why integers? Why ratios? What are the values?
Not quite sure what you mean by "which" bootstrap - it should just be the boot.rq function unless your data are censored.
not sure what the difference between the 2 numbered bullet points is. But 'logistic regression' seems to be the way to go. you can google it.
A possible related question, http://stats.stackexchange.com/questions/825/any-suggestions-for-making-r-code-use-multiple-processors.
That is not finding the tangency portfolio. It is just discretizing the returns space and plotting points of min variance on the graph.
9 observations per tank. Balanced design.
(1) What is "P hat"? (2) The chance in the second question calls for a Binomial *prediction limit*.  This question is more complicated than it might look at first sight because the *true* proportion of returning birds is unknown and the 7/100 data give only a very rough sense of what that proportion really is.
I see, so it is enough to check the error distribution, it helps a lot!! I was confused esp. because of some sources like the first: "Simple linear regression allows us to look at the linear relationship between one normally distributed interval predictor and one normally distributed interval outcome variable"... It's confusing...
@Henry Because if I added them together it would give each point within the cluster as much weight as a point outside of the cluster. I want the cluster to be weighed against other results as if it was 1 data point.
Unfortunately the measurements cannot be considered independent within individuals. They are multiple measurements made over a 24-hour period using a continuous sampling method (EMG muscle activity data).
thanks for that. Yeah, I did consider the conditional logit model, but for the same reasons you described I had to drop the approach.
It is a lot of data, that's true. Part of the mcl software (disclaimer - I wrote it) is a program called 'mcxarray'. It can parallelise using both threads (CPUs) and jobs (machines) and integrate partial solutions later. I find that brute force is still the easiest solution, if one has access to good compute resources. Three more comments: - objects with many tags could well be problematic. - I would test on smaller (carefully chosen) subsets before going large-scale. - Wayfair has done something similar I think: http://engineering.wayfair.com/recommendations-with-markov-clustering/ .
A search for Likert on this site will reveal several threads.
Yes, I was thinking about independent random variables. Sorry.
@ Dilip: Thanks for the discussion. I was trying to write a question and was having doubt about second part as the sentence is starting with Poisson distribution.
0.09% as a maximum implies to me an average of the order of 0.05% and so ~2000 variables. One of us is wrong, I suggest.
For Example, one is Multinomial Logistic Regression, another one is Hierarchical Gamma.
Is this a homework problem? If so, please add the `[self-study]` tag, & note that we treat such questions differently (see our [help page](http://stats.stackexchange.com/help))--although, of course, they are still welcome.
What would be the reason for the distance calculation? That ought to play an important role in evaluating whether your choice of distance is useful or appropriate.
Welcome to the site, @Luis. In general, we are cautious about answers that are primarily composed of links to external sources, because linkrot is so common on the internet. Would you mind expanding on this idea & including the most important parts here?
$\\rho$ biased means that its value is unreliable, it could be anything from -1 to 1. If you accept to give an interpretation to it , then you are implicitly assuming your universe has 2 variables of interest only. If you suspect there might be others, why to calculate a bivariate $\\rho$ ? e.g. a universe must be defined before starting the analysis and yours is multivariate (>2) From that point of view, a bivariate analysis suffers from an omitted variables issue.
Can you tell me about the PhD source of this information?
I need to bring sources to a project that I'm working on...
`two-sample t-test seems dodgy in this case` Please say why you think so. For me, using an independent-sample test when you actually know the samples are paired, is fair, only it is less powerful. As long as it is right to stand for H0 (and againts own wish to find effect), it is just a conservatism, not an error. Another approach can be based on random matching, but it is asymptotically the same as independent-sample test.
I don't think this question belongs on SO. Although it is couched in terms of `R` code (& will require that potential answerers can read the `R` code), this Q is really about what is going on, not about how to code in `R`. Moreover, the code implies confusion about variable selection and cross validation. Clearing up that confusion is what would truly constitute answering this question.
Muthen frequently recommends a more flexible model that resembles EFA where are manifest observations are allowed to load on all latent variables to test the fit of the CFA model. See [here](http://statmodel2.com/download/EFACFA6.pdf) for one example.
+1 for any advice _a fortiori_.
There are many ways to compute a lagged correlation.  One standard formula for the autocorrelation function (ACF) appears at http://stats.stackexchange.com/questions/81754.  Is this what you are looking for?
The numbers seem wildly off, even bizarre, since 0.18% is about 1/550 (for a typical sized poll that would be *two people* -- you can't imply you 'know' the probability from two  "approve" answers) and 0.84% is about 1/120 (maybe 8-10 people in a typical poll). Do you mean 18% and 84% instead? (i.e. 0.18 and 0.84 as proportions; the second percentage is twice too high to be the actual figure but at least a believable alteration to reality to make a question).
BGreene, calculating confidence intervals from repeated k-fold cross validation not be a good idea according to some researchers: https://lirias.kuleuven.be/bitstream/123456789/346385/3/OnEstimatingModelAccuracy.pdf A bias-corrected bootstrap confidence interval might be a better idea.
Thank you for your reply.
However, I am doing that already. I want to know the way in which the TFIDF values should be changed for those BOLD_ or HEAVY_FEATURE_ terms.Is there any standard way by which I can increase TFIDF value? @Christopher
I can't tell from your 2nd paragraph, but it's worth noting that Cronbach's alpha is meaningless if there is >1 factor.
Can you give more details about the null hypothesis? i.e. What do you know about the "typical" distribution of counts?
This sounds like a strange thing to want to do. WHY do you want to see if two series are "significantly correlated"? Also, time series are tricky; if you "struggle with statistics" you may want to find a consultant who knows more.
The source of the (summary) data in that presentation is [Freedman, Pisani, & Purves *Statistics*](http://www.amazon.com/Statistics-4th-Edition-David-Freedman/dp/0393929728) (any edition).  It is, however, only a summary, not an account of *all* the results requested here. Kerrich published his results in 1946 in a small book; Google has [digitized it](http://books.google.com/books?id=JBTvAAAAMAAJ).
@ttnphns I've attempted to do both - I hope it is clearer now.
Just to clarify (there seems to be some confusion in the answers below), you have a regression problem where you trying to predict y (a real number) given a multivariate x.... and you are interested in using a kernel regression approach, correct?
This is a great answer Harvey - thanks for writing it!
As Stephan guessed correctly, the problem is your huge sample size. It has the same problem as the normality tests described here: http://stats.stackexchange.com/a/2498/54099