tttjjj/elt_bench huggingface dataset:
I agree, and wonder if @user40767's expression for `Q` generates a 13-way interaction, or just the 2-ways. If the latter, I'd think Stata could handle it, given the abundance of memory and speed in today's PCs.
That would work to some degree, but in the above plot you can see the means dip downwards. This is not a real effect, but rather that the top gaussian gets overwhelmed by more statistics in the lower. By parameterizing the means with a form that doesn't dip downwards it would prevent that effect.
Which one would you recommend? Each comes with its own null hypothesis, and the sampling design suggests that at least one margin if fixed in the two-way table.
I suspect they're not log-odds *ratios* but log-*odds*. See http://en.wikipedia.org/wiki/Substitution_matrix#Log-odds_matrices
:Wayne filtering by differencing is a particular form of an autoregressive (ARIMA Model). Any filter that is used should be empirically identified and never assumed :see the Slutsky effect. For example if you difference a white noise process you create a series that is an MA(1) series. Your comment about injecting pulses and creating multiple pulses is to this point.
Welcome to the site, @Robert. We prefer if you don't sign your posts; note that your avatar & username (w/ a link to your userpage) are added automatically to your posts. Since you're new here, you'll want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info like this about CV.
And I can get exactly which individual is visiting the page at the moment.
Did you make sure that, when SPSS generates its own dummy codes with BY, the reference category is the same category as the one with a value of 0 on your own dummy codes? (i.e., did you make sure that the reference category is the same when using BY vs. WITH)
@whuber I think you are going far deeper into this than I intended with my remark.  I just want to say that just because we commonly connect the word "confidence" with confidence interval, it does not mean that the OPs term confidence score could not be used to mean a probability (maybe like the Bayesian view of probability as a subjective level of belief but not necessarily).
Is it possible to refine your question somewhat, @Bijoy? Are you just looking for a tutorial on decision tree methods? Note that such a tutorial would be beyond the scope of CV.
A typical ratio would be 1042:42. 1042 turn left, 42 turn right.
@AndrewKlaassen: See my edit, where I explain local FDR in detail, why it's the way to calculate your value "A" (though you might want to change .05 while you're calculating A), and also why it is rarely used. Anyway, to clarify one point that doesn't really fit in the answer: your example with the lottery ticket misunderstands the point that I and others have been making. We weren't getting hung up on the idea of "do probabilities change with new information" (both Bayesians and frequentists have their interpretation of that): the point is just that you weren't changing them the right way!
What do you mean by "statistical features"?
Why do you say *un*-paired? The hospitals surely act as blocks, don't they? Also, by doing separate tests to 'improve power' what you're actually doing is improving power by ramping up your overall type I error rate. If you are *actually prepared to do that*, make the type I error rate higher in the ANOVA and use that instead. These sound like scaled counts; if so, the t-test assumptions may be dodgy (why not analyze the counts with their exposures via something like a Poisson or quasi-Poisson regression model)?
(+1) I would add that in case of factors, labels should be encoded directly in the csv file, when possible; otherwise, rely on a separate codebook. (This might also be used to provide extended labels, i.e. more explicit description of the variable than a 16 char. long descriptor). Likewise, units of measurement for a given variable should be constant across records (when possible), and documented in a separate file. If providing the user with a single file is mandatory, we can use lines before or after the header line to give all those information; this might be skipped when reading the file.
Three values are plotted in 2006, so it looks like March is being plotted to me. But unless you give replicable code, nobody can help further. The timeSeries package does not contain an object called boardings. I think you probably mean the TSA package. When I plot the boardings object from TSA, it is identical to what is plotted in your graph including the March 2006 figure.
Thank you, Joris. So "my definition" was just partially correct. ;) Do you probably know the asyptotic power of Friedman test (non-parametric repeated measures ANOVA)?
@Prob the question is, why?  What conceivable conditions would have to apply for the noise to increase backwards in time?  My analysis, non-rigorous as it is, nevertheless indicates the kinds of assumptions and gross simplification that may be needed.
May I know the reason behind saying that they have different probability distributions if we cannot reject the hypothesis of these correlation coefficients being zero?
Interesting question. Does Professor Ng give reasons why "PCA should only be performed on the training data and then the mapping is used to transform the cross validation and test sets"? My intuition is that the impact should be minimal, as train/cv/test data are assumed to come from the same distribution, and PCA does not involve the output/dependent variable(s).
Because this question includes a concrete example drawn from methods comparison studies, I think there's no need to convert this thread as CW, as suggested, because there may well be a good and acceptable answer.
"starting point" = "initial state," yes. Your interpretation of my RNG comment is also correct. Two chains with the same initial state might technically still not be independent after any finite number of draws, but assuming the chains have mixed reasonably well by the time burn-in is over they can be considered independent for all practical purposes.
Thanks. I am trying to find if the relationship between study hours and marks are different between males and females.
Thanks. yes each person has 3 means for responding to 3 different displays, it means that I have one whithin subject and 1 between subject factor. So I need to compare the interaction between gender and display type, main effect of gender and main effect of displays(I hope it makes sense), shall I call it mixed design ANOVA?
+1 Thank you for your suggestions. Could you also point me towards some work on Fourier transformations? And finally, I was wondering - are there any practical implementations out there that I can try out? I mean, some databases that actually implement this in action.
Ah indeed, I conveniently read over "hierarchical"
@Rahul, only the last sentence is about how you would adapt the procedure for continuous moderators, the bulk of the last paragraph explains how to do this for continuous moderators.
Is the difference between 0 and 1 the same as between 1 and 2 and between 2 and 3? In typical cases, these differences are not the same, and you should avoid using mean/standard deviation. Median should still be okay, though.
Gelman and Hill (2007) say that there is nothing wrong with having groups with few individuals (even when there is only one). But I am not sure whether I can add an intercept for the deputies or not.
@Abhi previous code does plot value of Phi correlation coefficient in different values of cut-off from 0 to 1.
Please see our [faq] concerning questions about datasets: "Questions about obtaining particular datasets are off-topic (they are too specialized)."
Did you see that I extensively revised (again) my answer to the original question at http://stats.stackexchange.com/questions/25946/goodness-of-fit-for-2d-histograms, which may render this whole approach of using a contingency table of bin counts not necessary?
I think this has been asked before: http://stats.stackexchange.com/q/8557/10026
Note that you did not need to invoke the chi-square distribution, as you can use the same $E[x^TP_1x]$ but with $P_1$ replaced by $P_0$  then you get $tr(\\Sigma_0^{-1}\\Sigma_0)=tr(I_k)=k$
Apart from the description of the Hessian, I cannot find any definitions of anything in or near the second paragraph. Exactly what definition are you referring to?
Is it churlish to wince when I read "data is"?
@GreenNoob - Scortchi has given a pretty thorough guide to testing the difference between two means.  Every term used (except for "mean" itself) has been defined.  You can simply plug your values into the equations given to obtain results.  Can you be specific about what else you are looking for?
No. No, except for eventual differences due to the care taken in the implementation (the loss function is convex).
`lm` uses OLS. The basic algorithm (the same one used in most packages) is described [here](stats.stackexchange.com/questions/90402/algorithm-for-minimization-of-sum-of-squares-in-regression-packages). There's no `ols` or `mle` function that I could find in the distribution of R; there may be more than one such function elsewhere -- what packages are you talking about?
This formula assumes there is negligible serial correlation among the returns. This often is not the case and when it is not, the formula can be grossly incorrect, regardless of the length of time.
jupp, the permutation approach is similar to the bootstrap approach - keeps the repeated measures together.
The answer in that other thread don't really explain why 2 is a better value than other values that are very near to 2 but are no natural numbers.
I realized that my question misrepresented the number of people in the control group. By "+- 10", I meant 50-70 people in each control group, not ~10 people in each control group. Sorry for the confusion.
@robin: done. might need to be checked.
Superimposing quantile and box plots is attractive too. It underlines that the box plot is a reduction of the quantile plot, although to some the box plot might then seem redundant. For strong emphasis on the relationship between the two graphs, see e.g. Parzen, E. 1979. Nonparametric statistical data modeling. 
_Journal of the American Statistical Association_ 74: 105-121
@PatrickCoulombe Yes, that's what I thought. Thank you. Also, I revised the problem. Could you check my new models?
Can you say anything about your data, such as where they come from, or if they're bound by 0 & 1?
(+1) Thanks for sharing your experience. I often observed that a good compromise is found just by looking at the scree plot while considering Kayser's rule as a lower limit and simulated data as an upper limit. What I like in `psych` is that it displays simulated scree plots from both PCA and FA.
+1 - reading the bean plot article now. I would also suggest a working paper by Hadley Wickham, [40 years of boxplots](http://vita.had.co.nz/papers/boxplots.pdf). It goes through a description of a variety of plots with more of less the same goals as bean plots.
I understand that MAE will never be higher than RMSE. I have been using both error estimates and looking at the difference between values to give an indication as to the impact of outliers. I.e when they are close great, when they further apart i investigate to see whats going on. Ultimately i want to predict parameters that best suit the data, and e.g. 9% error sound better than 12% - i just wanted to make sure i'm picking the right one for the right reason. Cheers for your advice
Ah, yes, of course, sorry! Does it work?
With an appropriate setup, Poisson regression _is_ the way to obtain inference for rate ratios, so you will need to provide more details about the methods used.
I took the liberty of adding the self study tag since your question is unambiguously a standard textbook problem being done for your own study. Please read its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info)
Box-Cox gave me lambda close to 0 so this mean log transformation, still nothing close to normality in qqplots of residuals
@Scortchi Am I alone in thinking it quite bizarre that evidence isn't a standard statistical term? I suspect the reason involves a mistrust of the likelihood principle. If the likelihood principle is the only proper route to evidence then discarding it precludes a statistical notion of evidence. (This may be worthy of a question.)
Try taking something simple. Perhaps start with a collection of Bernoulli random variables with $p=0.9$. Imagine you observe $k$ 1's (and $n-k$ 0's); what does the LHS work out to be?
At first sight, it does seem absurd that something that simple would work, [but it does](http://amzn.com/0387978674).
+1 Very good!  I didn't think that the CI on the intercept would reveal the mean, but of course it does: the CI depends on the squared mean, as you point out.  Once we are reminded of that, the rest is algebra.
Hi C. Pieters. Thanks. wld try out ur suggestion. Never used Cronbach's alpha but wld gv a go. Thanks again
Thanks for the edit Nick
The following question is quite similar, and might be a good reference.
http://stats.stackexchange.com/questions/44066/correlation-and-multicollinearity/44067#44067
@whuber Thanks for your comment. I added a bit of details in order to clarify its meaning and relationship with statistics.
It's simple: Ward's method (and centroid, and so called "median" methods) are involved in computing geometrical centroids in euclidean space. They do it in a way that requires squared euclidean distances.
@ziggystar the links looks promising, but if i apply all values to the density (so that a real density is generated) my computer hungs up for some minutes before i can see the whole plot because there are just too many values (>1.5 million) - i mean it looks very promising but is too slow
One thing you could do is just use 2 of the three in your imputation, but the three of them in your age-period-cohort analysis. Also, if the missing data is on either age, period, or cohort, and you have the other two, then obviously you can just calculate the missing one directly.
Another point perhaps worth mentioning is that this calculation allows one to read off immediately what the UMVU estimator of the standard deviation is in the Gaussian case: One simply multiplies $s$ by the reciprocal of the scale factor that appears in the proof. This generalizes to UMVU estimators of $\\sigma^k$ fairly readily.
Yes, I mean the one about parameter constancy. Below, Stat interprets correctly the meaning of my question.
Is this homework? If so, it should be tagged as such.
That sounds right. The intercept would then represent the value of the omitted level. Coefficients of other levels would be added to the intercept to get the fit for those other levels.
@Srikant: I tried to clarify the question. Better?
I found your description of the conditions and problem a little difficult to understand at first. Type I and type II are technical terms that are different to your usage of type1 subgroup and type2 subgroup. As far as I can tell you are applying a t-test to data from a distribution with a mixture of means. Is that right?
I'm not sure I understand your question. AMOS / SPSS will output tests by default. Can you clarify what you are after?
It has nothing to do with normality. Simply factor analysis perfomed on standardized data (=correlations), centered data (=covariances), scaled data (=cosines) or raw data (=dot-products) often leaves quite different results. You shouldn't rely on default specification but think over what type of data narmalization will suit your task
Thank you Dr. Lenth. Actually I have cited your articles in my dissertation. I am honored I was in any way able to be of help to an established statistician like you.
I would think the variance needs to be small for the mean to be a meaningful representative.
If I'm reading your second paragraph correctly: a random sample of raters will assess a random sample of emails, where raters do not rate the same set of emails (conversely, emails are only rated by a subsample of the pool of raters). Is that right? If so, things simplify a lot if this fall under a [balanced incomplete block design](http://apm.sagepub.com/content/5/1/105.abstract).
Sadly this is also incorrect, the problem of over-fitting is likely to arise whenever you minimise a statistic over a finite sample of data.  If the statistic has a non-zero variance, some degree of over-fitting is inevitable if the statistic is minimised fully.  If you perform feature selection independently in each fold, the resulting cross-validation estimate is (almost) unbiased, but that doesn't mean the model won't be over-fit, just that the performance estimate accounts for the effects of the over-fitting.
Thank you a lot for all those information.
SD > mean doesn't necessarily imply skewness or outliers.  If the data is non-negative then that argument could be made but I just think it should be noted that it's perfectly possible to have this occur.  But with something like 'change in weight' I might expect low means and high standard deviations...
Could you give the context in which they were being compared, and the exact nature of the comparison? A quote, and if possible a reference or link would help, if there is one.
I'm a bit confused, perhaps by the title. Are you wanting to interpolate between two time points or are you asking about something like smoothing (as opposed to filtering)?
Within vast limits, scaling makes no difference whatsoever for floating point calculations: it simply amounts to a shift of exponent with no loss of precision.  Where scaling can help is where a calculation involves sets of data that are themselves on different scales.  My guess is that the time series formulae are using some time measurement (milliseconds?  years? just integral steps?) that may have a hugely different range than the range of the data.  Good stats SW will internally scale its matrices to avoid loss of precision; this could explain the differences between FP and Minitab.
42 variables, as for solid, I just meant "of which results can be trusted".
@Joe No, $P(A, B) = P(A\\cap B)$.  It is the joint probability of both events happening.
The [manual](http://cran.r-project.org/web/packages/lme4/lme4.pdf) and three vignettes for the `lme4` package can be [found on CRAN](http://cran.r-project.org/web/packages/lme4/)
It would make more sense to use the latent variable representation of a mixture and to add those latent variables as a secondary HMM with an independence constraint...
No. I would not consider an aggregate statistic an experimental value.
I have seen random forests of gradient boosted trees applied to dimensionality reduction for that type of data.
you can get coefficients for linear models only
You sound like R is inferior to propriety software. :)
As you say, a staircase pattern is an inevitable side-effect of discreteness, but that is the only obvious limitation. The rule otherwise remains that departures from sameness of distributions are shown by departures from equality of quantiles.  Quantile-quantile plots are also often better on transformed scales, but that is true of continuous (or not rounded) variables too. Incidentally, quantile plots also work well for discrete and rounded data. An example is included within http://www.stata-journal.com/sjpdf.html?articlenum=gr0053
(ctd)... That convergence confers some calculation advantages that make appropriate [interpolation](http://stats.stackexchange.com/questions/64538/how-do-i-find-values-not-in-interpolate-in-statistical-tables/64539#64539) work very well, and at sufficiently large denominator d.f. let you just use the asymptotic values.
@whuber, It refers to the article by Davar - meta-analysis of sample correlations: The validty of Hunter,Schmidt and Jackson (1982) availble on EBSCOHOST. Also, the article - Meta-anlysis:An improved procedure of Hunter, schmidt and Jaxkson framework" may be relevant when we have bimodel findings.
You appear to confuse probabilities with probability densities: see http://stats.stackexchange.com/questions/4220/a-probability-distribution-value-exceeding-1-is-ok/4223#4223 (*inter alia*) for a discussion. Also, what are you really doing in your analysis? Do you have data concerning locations of breakage? If so, those data are likely inconsistent with a hypothesis of random breakage (and testing that hypothesis would not be informative). Wouldn't you want instead to *estimate* the probability distribution of breaks along the cable?
"The amount that you add is specified by the delta rule. Note that wij doesn't appear on the right-hand-side of the equation." - This is only true for neural networks **without** hidden layers! But you mentioned two other points, that are good arguments against initializing an ANN with *equal* weights.
Assuming that you are interested in this sport, you should be able to apply your qualitative knowledge. Is it really true that the response (whatever it is) is _highly_ predictable in principle making high $R^2$ achievable? I don't follow any sport closely, but surprise after surprise about outcomes is what fuels some large fraction of sports journalism (setting aside tittle-tattle about the players).
I want the st.dev of the data within each bar
:wayne How many trends did you het ? How long was each trend ? Were there level shifts as well ?
I usually verbalize $\\sim$ as "*is distributed as*", and $|$ as either "conditionally on" or perhaps more often, "given", so "$x_i|\\theta_i \\sim F(\\theta_i)$" reads "$x_i$ *given* $\\theta_i$ *is distributed as* $F(\\theta_i)$"
@cardinal only previous terms
+1 for the point that computing power is not the challenge here - just the problems with translating statistical inference to large datasets.  (I also gave +1 to @mpiktas; it only took around 40 seconds to run a simulated example of this size on my far-from-cutting-edge laptop).
Please provide a specific simplified example of your desired result.
X and X are not independent.
About RLM: the default parameters are defined, as usual, with the normal distribution plus outliers as the reference case. I haven't seen data like yours yet, so I don't know which options for robust estimation would produce the best results. (maybe concentrated weights in the center and then flat weights from there.)
@ren: Please consider formulating an answer based on your discovery. I, for one, will be happy to upvote. Cheers. :-)
The regression only tells you the strength of the possible association, it says precisely nothing about causation.  It can't even tell you the direction of the causal relationship.  In this case, there are multiple confounding variables, so interpreting the regression in this way is very risky and should be accompanied by plenty caveats.
As it is shown the "margin of error" might have the dimension of cm^2/s or m^2/s. Simply, I calculated variance of error.
I'd think you should not be combining shapefiles with different distributions of size. If you want to condition on the current sample's size/shape distribution (which would make sense in many situations), you'd randomly locate the objects to get the distribution of your test statistic under the null. It really comes down to what you want to condition on and what assumptions you want to make.
Hi, thanks for your answer. It is very helpful. One more question: what happens if I use Wilcoxon with a non symmetric distribution?
$X$ is a matrix, $F$ is a function, how exactly do you put them together into matrix $\\tilde{X}$?
@IrishStat - I'm trying to outline a work-plan for creating a program... I don't have data yet and even if I had, the program still needs to be robust enough to handle data that may behave considerably different. 
So basically I'm just trying to understand what are the most generic sensible approaches to this problem.
More specifically, you've mentioned time-series methods and intervention detection... :) I'm looking into those, but I still didn't quite manage to understand how I should apply them in this case (started reading books on the time-series analysis just a few days ago)
I do not have enough rep to up-vote your comment. I believe it is clear to me know. Thank you kindly.