tttjjj/elt_bench huggingface dataset:
Can you maybe provide a link to the pages you are looking at. I think by *momentum* you probably mean (sample) *moment* and I suspect that by *standardized* you might mean *central*, but I am not as sure of the latter. It's not *quite* clear to me (yet) what your exact question is. Sometimes giving a concrete example helps.
Note that Tukey's test controls the experiment-wise error rate across all 3 comparisons, & that's *not* conditional on the result of the ANOVA F-test. See ["Do we need a global test before post-hoc tests?"](http://stats.stackexchange.com/questions/9751/do-we-need-a-global-test-before-post-hoc-tests).
@Akavall could you please give some pointers on weighting the features properly? they are all binary (predictors and class variable). i would like to stick with PPV as the primary weight, but i could also use mutual information as well. or i suppose i can use any number of contingency table association analysis.
I hope you believe me that every new example is  linearly independent from all the ones before (except for the same $x$). In $\\mathbb R^n$ you cannot do that: Every point beyond $n$ must be linearly dependent on the others.  For the Gaussian RKHS, if you have 100 different examples, they span a 100 dimensional subspace of the infinite dimensional space. So the span is finite dimensional, but the features space they live in is infinite dimensional. The infinity is uncountable, because every new point in $\\mathbb R^n$ is a new dimension and there are uncountably many points in $\\mathbb R^n$.
@psj This is a csv file
@filups21 similarly to stackoverflow, questions like this are welcome on cross validated, especially if (i) you do some research for existing answers (which you have), and (ii) put some effort into the question (which you have).
@probabilityislogic Yes, you're right that leaving out these parameters does not simplify the problem.
Sure. I added a sample problem.
The first link you provided (hierarchical/multilevel models) seems broken now.
A bit more context might help.
With this question you've made me a fan.
Can you say more about what you mean by "how similar... the usage pattern" is? Can you provide a small set of example data? I'm not sure if this question is answerable as stated.
Did you include the main effect of Lot1 in the final analysis you show? You should.
Welcome to the site! What did you intend to say with your second sentence (since, as stated, it's clearly false)?
I doubt it would have a serious effect on performance, but I would read files into a list then use `rbind` or `cbind` through `do.call` to "merge". That way you don't have extra objects lying around.
Is it a problem that the number of security guards is not randomly distributed but highly correlated to the input variables?
Possibly the warning that comes with `ez` could be re-worded; it actually has two parts that are important: (1) that data is being aggregated and (2) stuff about partial designs. #1 is most pertinent to the discrepancy as it explains that in order to do a traditional non-mixed-effects anova, one must aggregate the data to a single observation per cell of the design. In this case, we want one observation per subject per level of the "direction" variable (whilst maintaining group labels for subjects). ezANOVA computes this automatically.
@babelproofreader I believe the posteriors are Bayesian updates, via the training data, of priors.  It's unclear how a kde could be construed similarly, but I'm no expert in this area.  Your question is interesting enough that you might consider posting it separately.
F Tusell, my intent is to estimate the blue line. would it make sense to introduce another underlying state and use both my red and blue lines as observations of that state?
(Although it does make sense, because the 'underlying state' is the 'actual' inflation and both the red and blue lines are simple observations of it.)
It seems to be a homework exercise so I have added a tag. The exercise is simple though, first find the pdf of $I^2$ then the pdf of the product (try to do it yourself, what are the difficulties solving the exercise?). By the way solution could be found in H.Hsu. nice book for self-study http://www.amazon.com/Schaums-Outline-Probability-Variables-Processes/dp/0070306443
"This process can provide no information not already provided by the effect size." this is incorrect, the p-value provides some information about how unusual this effect size would be under the null hypothesis, thus it provides an element of calibration of effect size.  Don't misunderstand me, I think Bayes factors are more useful, but it is hyperbole to say that the p-value is a worthless statistic.
yes, my question was poorly worded and ambiguous. to your last point: there are well known techniques for comparing Sharpe given paired observations. My Q is for independent observations, applicable for:
1. non-overlapping times, or
2. the case where summary statistics are provided by a third party (e.g. in a fund prospectus).
You can treat the topic mixture vector for each document as its position in this latent topic space. Simply run your clustering using this as the input data. What is the point of the hard clustering though?  The topic mixtures already give you a lot of information about how documents are similar/different.
@whuber Could provide us some references to those dedicated books. Thank you.
The rigamma function in R in the LearnBayes package allows you to draw directly from an inverse $\\gamma$ distribtion. 

http://www.inside-r.org/packages/cran/LearnBayes/docs/rigamma
Sorry for answer you late.I read your blog ,great work
@stan repeated measures seems appropriate for your design.
I'm basically talking about the case where the domain expert or the system that is using the model basically has no idea at all about the hyperparameter. A~N(mu,sigma) -> there is no idea about the value of mu.
Indexing with `gender[i] + 1` doesn't seem as problematic as indexing with `gender[i]` and `health[i]` in this case.
Yeah we can but definitely can't afford the time :)!
Adding to @mzuba 's good answer (+1), the reason an interaction won't test what you want is that an interaction tests whether the effect of one IV on the DV is different at different levels of the other IV. That isn't your question.

I am not an SPSS person, but, at the least, you could run the two regression models, get the log likelihoods, subtract one from the other and then test using a chi-square with df = 1. But SPSS can almost certainly do this for you.

You might also look at AIC and BIC, which penalize for the complexity of hte model
+1 for a good, clear answer.
but the support vectors *are* those observations for which the corresponding alpha_i's>0 (strict). This is why by reducing the # of non-0 alpha's you also reduce the # of support vectors. Is it clear?
I added the parameter estimates to my post. There the significance level is 0,05. What would be the right interpretation of the model hypothesis test values of my output?
(1) What is a "Spearman normal distribution"? (2) What distinction are you making, if any, between a CDF and a "normal CDF"? (3) Could you explain how this method introduces any correlation at all? I'm afraid your general uses of "variable" and "distribution" make your description rather vague, so it is hard to tell what it's really doing. Could you reword your answer to be more precise?
Is the categorical variable unordered? Approximately how many cases do you have? What is the frequency distribution across the categorical variable?
Neat!  But why use my head when I can use R ?  ;-)
K-means requires numeric data
Look at R's `factor` function. (That is, get help with `??factor`.) Obviously, `Data` will end up being a `data.frame`, which you can also look up.
Yes.  The bottom line is that ANOVA deals very poorly with many non-normal distributions, whether this is in the form of outliers from an otherwise Normal distribution or of a skewed distribution.  Particularly in the small sample sizes that are common in certain fields.  Modern robust methods are a must.
To add to Peter's questions, univariate analysis can cause an amazing amount of damage when done before multivariable analysis, because there is a temptation to use the uivariate results in guiding model building.
The cut points are in `q`. I added some information to show them. Also if you want to keep the numeric data you can use `split`. There are thousands of ways to use this.
Hi gung, thank you for your continued help. I have taken your advice and posted a new question to Stack Overflow asking specific questions about the code. I have also run the code and varied some of the parameters which has been somewhat instructive. I am experienced in programming and programming concepts but just haven't had to apply these to statistical problems before. Hopefully some more trial and error with the code I have and some help from Stack Overflow will help me to make this power analysis happen. Thanks again.
But I'm interested in overdispersion in the context of logistic regression. For each value of a predictor variable in logistic regression, there isn't n trials, there is only one trial. And the result of that one trial is either success or fail
At first glance this seems like a very roundabout and probably inefficient way of performing parameter estimation; can you give any intuition as to why you might prefer a simulation-based approach over e.g. [maximum likelihood estimation](http://en.wikipedia.org/wiki/Log-normal_distribution#Maximum_likelihood_estimation_of_parameters)?
Are you fitting all three models to one data set? Why would youbwant to average those parameters?
this is where I can not understand, if for positive Theta value the process is more regular and for negative value process is more smoother than white noise, why variability(variance) is equal for both positive and negative values. for example for Theta=-.5 and Theta=-.5 we will get the same variance.
thanks, that seems valid. Actually, that is what I did: I conducted the 5x2 ANOVA first and proceeded to checking the separate 2x2 ANOVAs. However, I also need to report and interprete the main effects of the four ANOVAs. Would I have to adjust across all main effects and interactions together? Or can I have separate sets of adjustments for separate hypotheses (the hypotheses concerning the main effects are largely separate from the interaction hypotheses)...?
I am trying to perform maximization on historical data to calibrate GARCH parameters. So I guess there are enough deterministic methods to achieve the same? Do I need stochastic methods? Cheers
BTW Such question would be very on-topic on yet to be created [Computational Science SE](http://area51.stackexchange.com/proposals/28815/computational-science?referrer=4pEy7Pj-D8kbaDTh4NmFiQ2).
What does a plot of these data tell you?
By zero-truncated, I mean that the probability of a zero-count is zero. My data are the count of items purchased _per transaction_ at a convenience store over a five-day period--obviously there's at least one item being bought per purchase (but usually fewer than, say, 20). The schema looks something like: DATE, TRANSACTION_ID, COUNT_OF_PURCHASES As to Sichel, I do have reason to believe that that distribution might provide a good fit, but I'm trying several others as well (e.g., ZTNB). Thanks for your reply. --Jeff
not a full answer, but what may help you is to "go backwards" and learn a more basic model, and then how this one "generalises" that more basic model.  You may need to do this in several steps.  do you get to choose $S$? like in a mixture model?  This does look like a mixture model, where each "s" defines a "component".  Kind of like saying "I have $S$ models, but I don't know the state which each individual observation is in.  So I will average over these states"
@MikeLawrence: Thanks. The question was probably worded too specifically to receive responses. My main question is just if there is a way to compile R code. That doesn't involve any knowledge of mixed models or EM algorithm.
Thx a million.. Appreciate your efforts.
See http://stats.stackexchange.com/q/577/159 for some discussion.
@user27736 You could accept the answer, then.
I've just one again checked it in SPSS. McNemar = Sign test for dichotomous data (asymptotic = asymptotic, exact = exact).
I just saw your answer- that was tremendously helpful, thanks so much!
+1, I like this idea; however, randomization / permuting the data will only break relations b/t variables, this wouldn't work if there is clustering w/i a single variable.
More to the point, it is not the case that the logarithm of the model $y = \\exp(b_0+b_1 x_1+b_2 x_2) +\\varepsilon$ can be written as $\\log(y) = b_0+b_1 x_1 + b_2 x_2 + \\eta$. The problem, when we are explicit about the error structure (as reflected by the random variables $\\varepsilon$ and $\\eta$), is apparent: the logarithm of a sum is not the sum of the logarithms. The two models are different.
The intercept aside, you're estimating the same population parameters when you down-sample but with less precision - except the intercept, which you can estimate when you know the population prevalence of the response. See Hosmer & Lemeshow (2000), *Applied Logistic Regression*, Ch 6.3 for a proof. Sometimes you can introduce separation, though not commonly, as you down-sample the majority response.
My understanding is that you wouldn't _always_ standardize before doing PCA (you always subtract the mean however). If, for example you were trying to model salaries of different occupations and you cared primarily about the \\$ accuracy of approximating with a reduced set of components you might not standardize, choosing to accept higher relative errors on low salaries. OTOH if the variables are on different scales you'd always standardize. Discussed e.g. [here](http://stats.stackexchange.com/questions/62677/covariance-v-correlation-based-pca-theoretical-view/62699#62699) in more detail.
Have you seen this question on mathoverflow: [Correlation between 3 variables](http://mathoverflow.net/questions/57998/correlation-between-3-variables%20Correlation%20between%203%20variables)?
Aaah, gotcha!  That makes sense, at least in theory...  But running it gives me these results: 

http://i.imgur.com/cPIb3AX.png

Where it seems like pretest/posttest are just considered 2 random variables; is there a way to make Mr Kruskal and Mr Wallis contrast the changes between the groups?  Or am I missing the point completely (seems likely!)
I would suggest changing your strategy from *testing* to *exploration.* Start by summarizing and visualizing your data. A natural beginning to visualization (for the purposes of identifying relationships) is a scatterplot. (When you're still at the stage of wondering what type of relationship two variables might have, formal statistical tests give misleading results. Search the literature for "data snooping" and ["data dredging"](http://en.wikipedia.org/wiki/Data_dredging).)
$$\\chi^2=\\frac{1}{N}\\sum_{i=1}^{N}\\frac{(T_{exp}-T_{fit})^2}{\\sigma^2_{exp}}$$
I think they actually did it slightly differently: Rank the states by EV, then see what happens if each candidate wins, going down the tree. So, you don't need to generate $2^9$ and then prune.
$Q'(\\hat{\\beta})=0$. Yes, it, is one part of M-step in shared frailty model. I am trying to do it in a way, but facing some problem for double summation and do not get the exact value.
Overfitting happens when you have not enough training data not when you have too much. There is probably something wrong with your learning step though.
If you think these events will move the price in a particular direction (even if some events move it up and some move it down), then absolute price change might be a good metric. Suppose a stock falls 2% and then rises 4%, so at the end of the day it is up 2%. Do you want to think of that as a 2% change or a 4% change?
I'm not sure what an "adequation test" is. It's not a term I've heard before, and a google search almost solely returns results written in french. Is it the same as a peason's chi-squared test?
What are you referring to where you note (at the beginning) that regression requires that "factors should be continuous"? I am not aware of any such assumption. Indeed, because ANOVA is an application of (least squares) regression, and *all* of its factors are discrete, it would be terrible indeed to discover that it is not valid!
Asking about statistical methodologies is Off Topic for StackOverflow.
@Metrics The phenomenon of the existence of a unit-root is just one of the ways that non-stationarity can be induced. In the specific series, a deterministic trend is clearly present. In theory, they could co-exist, so a unit-root test could be applied to the detrended series, but I don't see any real gain in that, since the deterministic trend is the leader here.
That is correct, but only if the original variables were standardized prior to PCA. `pca.object$rotation` contains eigenvectors of the covariance matrix (i.e. *principal axes*), presumably normalized to unit length. If the first eigenvector is $\\mathbf{v}$, then covariance between original variables and standardized principal components are proportional to the elements of this vector (coefficient of proportionality is given by the standard deviation $s$ of this PC). But this is equal to correlation only if the variables are standardized.
@user Your edit suggests the interpretation implicit in the reply by @Macro is the one you want, not the interpretation I made here.  The R code appears to deal with sequences, not with random variables.  It is not at all clear what would constitute a "transformation."
I like to use [trellis displays](http://cm.bell-Ulabs.com/cm/ms/departments/sia/project/trellis/display.examples.html) for this kind of visual investigations. You'll find them in the `lattice` package.
@coulminer No matter which level as the base, the model is essentially the same. Only the estimate value is different, but the variable meaning (difference to the base) is also different correspondingly. It is not necessary to change the base level to make all the result table "look" lots of "***". This means that even though some of the level in the factor is not "significant" itself, we should not simply drop them directly.
I think for each question you can look for the most frequently occurring score and perhaps the mean score to get a sense of how the respondents felt about each question. Also the Kappa statistic can be used to measure the level of agreement between students to the answers to the questions. Also you can look to see if any particular question stood out as having a high frequency of agree and strongly agree answers.  I think the analysis should be mostly exploratory rather than confirmatory.
Any chance you could elaborate on why using a subset of the data is problematic? A single hypothesis usually doesn't explain *everything* at once. Maybe they hypothesized that drug A works even better in patients also taking drug B, and so their subset is restricted to patients taking A, or A+B.  

I'd agree that the quality of the hypothesis matters a lot, but that was also stipulated in the question.
This is definitely not correct. An ROC curve is always increasing. The true positive rate aka sensitivity (y-axis) can never decrease as per your picture.
Did you try your formula, did it work? Can you give a numeric example of your attempt to use it?
Kernel smoothing is an option, but it's not clear to me how to build a kernel which will guarantee the property I want without being too restrictive. I think I would also run into the same problem of favouring the mean of the interval.
sometimes they infect the population just to see what happens...
That answer doesn't speak to the difference when one creates the one-hot-encoding themselves though. If RF is truly respecting the fact that a factor is one thing, if you one-hot prior, that would be lost (I think).
@onestop what I meant was the specific choice of (1,10) written above doesn't give values of $p$ that are very small, to the point that the normal approximation looks pretty good.  If a person wanted the Poisson to come out then they would need to try something else;  it sounds like your choice of (1,999) does a good job, yes?  I had also thought to make $\\alpha < 1$, say, 0.25, but I haven't tried that.
I'm not sure I follow your statement. In general, when there's an interaction, we *don't* interpret the main effects, though. Instead, you interpret the *simple effects*. Ie, you stratify your model on 1 of the terms (whichever makes more sense for you) and explain / interpret the relationship b/t the other term & the response at each level of the stratified term. If your covariates are continuous, you can specify the relationship b/t x1 & y @ the mean of x2 & +/- 1SD. For general info on interactions my answer [here](http://stats.stackexchange.com/questions/49002//49701#49701) might help.
Tomas, a "precise" model does not mean the estimates will be more precise.  As an extreme example, take any data-free model you like, such as one that predicts all responses are zero.  This model is *absolutely certain* in its estimate of zero.  It is therefore as precise as one could possibly get--but it's probably as wrong as possible, too.  Giving a model greater scope to fit parameters therefore usually means those parameters are fit with *less* precision, not more.  A better model, because it can quantify the uncertainty not captured by a worse model, often has larger standard errors.
I really prefer the R framework. There are many ways to construct a confidence interval for a Gaussian mean, a confidence interval for a binomial proportion, a confidence interval for a Poisson rate... For example the binom.test() function provides the Clopper-Pearson binomial test with its associated confidence interval, as shown by ?binom.test. And there is no reason to impose a "default" confidence interval to statisticians who want to understand what they are doing.
Are you certain that you want to use A*B and not A+B? Using the t-test is interesting. Have you also calculated precision and recall?
Statistics, like just about any other science, is full of bizarre terminology, notation, and analysis habits just copied from others.
After 5 month I finally update this question. I missed some basics since I can have the same coefficients with different explained variance
"*didn't know how to add the hat*" -- you can always ask, or you can google things like *mathematical symbols in LaTeX* and find many pages like [this](http://web.ift.uib.no/Teori/KURS/WRK/TeX/symALL.html) (under 'math mode accents') or [this](http://en.wikibooks.org/wiki/LaTeX/Mathematics)
Wolfgang, you were as close to the problem as possible without knowing the problem domain. Thank you
I'm not sure what I am misunderstanding.  Reducing the number of dimensions in an image reduces the number of pixels (doesn't it?).  For example, applying the first x principal components to an image would make an image of lower dimension.  mMy question is how to visualize this image
Can you be more specific about what you need help understanding? Are you familiar with Beta-Binomial models?
I think you 'rage' is justified, but you can't do much. If the only 'hammer' you have is a Gaussian distribution, then you have to fall to F-test, R-Sq, etc. It is definitely not non-sense to report $R^2$ if the CLT is satisfied, which may not be true in your case. It is good that you question such issues, instead of just using them.
@user603: Thank you... but I am interested in the standard errors... not in the residuals...
I think that @"James Stanley" answers that very well. In any real life case, you take a sample, calculate the sample mean, and that's your estimate.
Does this distinction have any practical significance? What can you do with an unbounded varience that you can not do with an undefined one? I've never seen this before. How does one know an undefined value from an unbounded one? Is 1/0 unbounded or undefined?
Not necessarily... to establish it's Gaussian, I would use a QQ-plot (http://en.wikipedia.org/wiki/Q%E2%80%93Q_plot), or use QQ-plots of your data vs. various distributions.
@love-stats I'm not sure I follow your comment