tttjjj/elt_bench huggingface dataset:
"To use maximum likelihood it needs to be uncorrelated": See this post http://stats.stackexchange.com/questions/72669/auto-regressive-process-maximum-likelihood-estimator/72719#72719
@MichaelChernick, Yes, he does. Note that you'd model, in my example, the probability for each observation that it received treatment 0, then proceed as suggested. Propensity scores would help find the matches, but you'd still be left with having to do something about the multiple treatments.
please supply more details, which software you are using, how exactly are you running the regression, etc. Reproducible example always help. As it stands now, the question is unanswerable.
Thanks for your response. I do want my error term to have a mean of zero. I had the same problem when simulating multiple AR(1) price series. I simulated the Ornstein-Uhlenbeck process instead - the continuous time version of an AR(1) process that is guaranteed to be positive. I'll see if I can convert the O-U process into a GARCH process and hopefully I'll be able to post an answer.
This reminds one of model-adaptive control.
I have now added more explanation to ma answer. Hopefully it is clearer to you now. Yes, you were correct in your link to the `predict.prcomp` help.
This is a possibility I have considered. I do not like leaving out data though.
Overly regular cells would be cells with equal counts or nearly equal counts. This constitutes a departure from multinomial data with equal probabilities, since in practice, one typically gets a lot of variation from cell to cell. A p-value is calculated with reference both to the null and alternate hypotheses. So if the alternate is one-sided, the test is incapable of detecting an alternative from the wrong side.
Thanks, cardinal. Yes, the states are observable.  There will probably be 5 to 10 states. (It's still uncertain, but I don't expect a very large state space.)  Right now, we have a list of about 30 additional covariates that we intend to investigate, although most of them will probably end up having little effect.  Some are continuous, and some are discrete.
@darkcminor - I am happy to elaborate, but your question asked about quantifying the error.  So if you could add some info into your question about the rule you are using to classify, then that would make it much easier for me.
Am I too naive thinking that, _maybe_, discriminant analysis could be employed here?
Yes, that is correct. This isn't (or shouldn't be, anyways) MCMC, just direct Monte Carlo.
It just means that the sd is zero, it's not a problem (usually).
I have made several edits to my answer.
Em... not really debated... Putted simply (and paraphrasing Julian Faraway's "Extending the Linear Model with R", section 8.4) : If you compare two LME models with different fixed effects, the linear combinations, $k$, of them will be different and the obtained maximum likelihoods will be incomparable. I haven't seen Zuur's book but are you sure he is referring to model comparison? In your case it is a bit shady cause you have actually the same $X$ matrix in the LME and the GLS model but even then you assume uniqueness of $k$.
These concepts and keywords helped me get a good start. Thanks. I'll report back here if I make progress.
possible duplicate of [Logit with ordinal independent variables](http://stats.stackexchange.com/questions/5387/logit-with-ordinal-independent-variables)
And `mtry`, the number of variables the algorithm draws to build each tree, by default the square root of the number of features total.
@john That's because neither the question nor the answer are about ANCOVA and rather arbitrary notions of what should or should be called that. In fact, my point is that it doesn't matter at all (but I provided a link to a question in which you touch upon the issue).
Flagged to migration.
To add to this. Before adding interaction variables, center the variables so that $0$ becomes a typical value, and better, a value with some special meaning in the context---so that the estimated coefficients become interpretable!
I can't say I've ever seen it called a scalar model. I get why you'd want to call it that, though to me that would be perhaps also suggestive of a line through the origin. I'd be more likely to call it a 'constant model', but terms tend to differ in different application areas.
It would be nice, in any case, if you present some references about the `multiple ways` and the `ongoing discussion`, - do show these alternatives.
@mbq: Thank you for editing it to make it better.
I'm tired too, and i'm feeling more unsure than i was, so i hope a third party will confirm or deny.
I agree, there is no object named 'descr'. I'm betting that it is an object in Kuhn's example and you need to translate it to yous. Perhaps you need mutagen[inTrain, ]?
There are quite many ways to compare distributions. Most typically it is some sort of [distance between the two CDFs](http://stats.stackexchange.com/a/25777/3277). So, decide what is "tail" for you (say, percentile rank > ...) and consider tail as the entire distribution. Then compare.
please add the `self-study` tag and check its [tag wiki info](http://stats.stackexchange.com/tags/self-study/info)
What does "busy" mean?  What is a "pulse"?  What information do you especially want to convey to readers--*who* is "busy," *when* they are busy, who is busier than whom, what the *trends* are over time, something else? Do you want readers to be able to assess this information quantitatively or is a gross qualitative depiction sufficient? Must the visualization be updated in real time?
@Macro, Thanks for the link.
Well put.  Seeking a cutoff (unless utilities are trivial) is equivalent to failing to understand the problem.
asjohnson, bootstrapping can be either parametric or nonparametric: it is not inherently nonparametric.  See http://en.wikipedia.org/wiki/Bootstrapping_%28statistics%29#Parametric_bootstrap.
I see no way to test how a sample is drawn from a population when all you have is the sample.
+1 You could also sample from an appropriate Gamma distribution and transform that result (take the reciprocal square root). Since [Gamma RNGs](http://en.wikipedia.org/wiki/Gamma_distribution#Generating_gamma-distributed_random_variables) are easier to find than inverse-Gamma RNGs this option widens the possible implementation choices.
One for @Frank Harrell ?
"Estimating Rates of Rare Events with Multiple Hierarchies
through Scalable Log-linear Models" looks promising.
[link](http://users.cis.fiu.edu/~lzhen001/activities/KDD_USB.../p213.pdf)
@Xi'an so binomial distribution is the correct one for this scenario? Also can you please explain: "Dirichlet distributions are distributions on the simplex .. so they are certainly not appropriate for your binary variables".
Oops, thanks. That is what I had written down on paper, I just typed it up incorrectly. I'll use the excuse that it was my first time writing an equation in LaTeX :P
The key thing your answer gave me was the realization that looking for a single value was where I was getting hung up. Once I started thinking in terms of distributions the Kruschke text and everything else began making much more sense. Thank you!
If the series has one or more shifts in the mean , I believe some if not most of your recommended tests will falsely conclude about the need for differencing as a remedy . Differencing is one from of remedy for non-stationarity but ny no means (play on words) is not the only form. See the following article on the flaws of differencing http://www.autobox.com/makridakis.pdf. Other possible causes for apparent non-stationarity that do not require differencing are 1:) time-varying parameters and 2) time varying error variance
I found that too, but where do I put it?  Before the summary() call?  Inside it?  I tried both and it still won't show the whole table.
Thanks IrishStat.I'll keep your argument in mind while working with this kind of analysis.
Do you just not know how to do this in R? Or do you not know how to do this in any language? If you want suggestions for methods on clustering categorical data, you're better off asking at [stats.se]; that is not a specific programming question.
People *do* use Bayesian techniques for regression. But because the frequentist methods are very convenient and many people are pragmatic about which approach they use, so often people who are happy to use either will use ordinary regression if there's no need for something more complicated. But as soon as you need to deal with a bit more complexity, or to formally incorporate prior information, or any number of other reasons then the modest additional work in Bayesian approaches start to look good.
@Michael, yes I know - my point was more that you (and I) asked for clarification but you didn't wait for the clarification before posting your answer. Re: your second comment, you can investigate the coverage properties of any interval in this way, regardless of whether it was based on the normal approximation or not. If you think there's something distinct to add that is missed by the existing answers then please edit your answer so we can all learn.
I believe there's no need to do PCA after removing correlated columns
Can you show me where on that page you link to it suggests a permutation test? I tried searching but couldn't locate 'permutation test', 'randomization test' or 'resampling test' on the page.
google makes $65B a year not caring about the difference...
If $\\beta$ is a polynomial, what's $L$? Why does $\\beta_1$ appear twice? Why is $y$ on the right hand side?
I'm not familiar with that book. If you're doing something analogous to regression, you should correlate predictors or you'll get incorrect estimates.
I also made a p-p plot for step 3. Or at least, in my interpretation this is what I need (a linear regression with normal probability plot): http://i.imgur.com/EZ3Ic.png Any comments?
**Hint:** you can solve this using the [properties of covariance](http://en.wikipedia.org/wiki/Covariance#Properties), specifically bilinearity.
1) Either actually, but I think it would be much more educational for me to actually be able to generate data in practice.  2) I'm okay with anything: R, Matlab, SAS, Mathematica, hand-coded C.  3) Population.  4) I don't care whether the data is normal.  It's fine if it is.
For some reason it just didn't seem right that the same sentences would have different pre scores in the data, but you're right - these aren't 3 versions, but 3 sets of answers using the same scale. I'll proceed with a mixed model. Thanks for this!
Good discussion. See also references in http://stats.stackexchange.com/questions/62815/line-chart-vs-area-chart-noir
Related: http://stats.stackexchange.com/questions/22/bayesian-and-frequentist-reasoning-in-plain-english
I'm not sure what you're asking.  As you've written it above, both commands fit exactly the same model (the /// comments out the remainder of the line and continues on to the next).  I suspect that the line continuation originally broke the line with the residuals() option appearing on the next line, and then it got moved up to the first line without removing the line continuation character.  Remove the /// and you should be all set.
In your dataset with 132 observations, is there an associated count and offset term, implying it is actually a weighted dataset with many more than 132 observations? Because of the mean variance relationship in Poisson RVs, large counts can lead to "model selection" properties which are unfavorable due to the "large sample size".
Bump for some help!
@Richard80  'you can say after the 1st flip: "after 3 more flips of Heads, a Tail will appear)" ' -- such things are only possible post-hoc. Before you toss, the probability is unaltered by the history of the coin toss. *The coin has no memory*
@David But can't you give some guidance, such as typical ranges?  For instance, if the sum of the p's ranges between 1 and 100 that's useful information and suggests some efficient solutions, but if it can get up to 10,000 that could exclude some approaches.
@Brian, it's what locfit does for many other models. I hope this helps.
Very clear explanation! You might add that in the case the mean of an exponential is equal to $\\lambda^{-1}$.
Do you mean perform separate correlation measurements for the cheap and expensive datasets, and then measure the correlation between these correlations, like this? http://bit.ly/e9DoEk
@Anton If you trust your judges but not all of them and do not know whose opinion might be biased, you can use trimmed mean as a central tendency, such as in figure skating for that purpose.
It is maybe due to the smoothing. Did you read the documentation of spec.pgram? http://stat.ethz.ch/R-manual/R-patched/library/stats/html/spec.pgram.html
Could you confirm that "1- vs. 8-dimensional" means we are dealing with uni- vs. multidimensional models?
I am very sorry, but this question appears to be off-topic because it is only about running software: please consult our [help] concerning what kinds of questions are on-topic here.
@PeterFlom, one thing you can do is scale the coefficients by the variance of the linear predictors (plus $\\pi^{2}/3$, the variance of the standard logistic distribution) - this puts them on the same scale. Of course, they are no longer interpretable as odds ratios once you do this.
Is this a question *only* about what's happening w/ R, or also about some of the statistical content? It appears to me to be the former; if not, could you edit it to make the latter more apparent? If the former, it doesn't really belong here, but rather on [Stack Overflow](http://stackoverflow.com/) (although nothing in that means it isn't a perfectly good question).
Many thanks Tal Galili. Could you please explain more about the permutation test? I really have no idea what it is.
@Peter Flom: I think that there is mistake in Your code, which leads to bad result. After applying round function You don't sample from uniform distribution. You will get digit 0 only when sampled number is lower than 0.5, I have the same concern about 9 - round will return 9 only when number is greater that 8.5. It would be better to use floor(ruinf(64, 0 , 10)) or sample(0:9, 64, replace=TRUE). After the change probability is a bit lower than 0.1.
+1. Many things you discuss here were covered, slightly more compressed, in my [answer](http://stats.stackexchange.com/a/48859/3277). I have added a link to your present answer in the body of that my old one.
If this is the variables (features) that are binary, I think you may do PCA to reduce the dimensionality. Please see my opinion [here](http://stats.stackexchange.com/questions/19591/factor-analysis-on-binary-data-and-summated-scales/19595#19595) with a further link therein. _Why_ do you think PCA will be inappropriate?, I'm really interested - can you clarify it in your question and/or give a link?
Thanks for the links. I actually requested your book from my local university. :)
I believe it is useful to separate two essential concepts behind the bootstrapping method.  The bootstrap itself ought to be thought of as a way of modifying an estimator to produce another (hopefully better) estimator.  It can be computed theoretically, exactly, and (sometimes) in closed form.  The simulation is *not* an inherent part of the bootstrap!  However, in many cases simulation is a natural and easy way to *approximate* the bootstrap estimator.  See the introduction and chapter 1 in Hall, *The Bootstrap and Edgeworth Expansion.*
There is too much to explain in a single answer. I advise you to check out semi-supervised learning.
The p-value depends heavily on the tail behavior of the assumed distribution of the test statistic under the null hypothesis.  The observed p-value depends havily on normality.  Perhaps normality is incorrect.  A distribution with heavier tail behavior could have a much larger (relatively) value than the value of  0.0000000000034 that you got.  I never take such p-values very seriously.  It really is just an indication that the result is highly significant.
Have you tried using cov instead of var?
Yes, I agree that it is a harder problem than many would like to admit.  In fact, it's not even trivial to define what it means to "attend"; do we mean just stopping by for a second, staying all day long?, etc.  I think the actual problem is too hard to solve exactly, and I am happy to hear about first order approximations.  :-)
A complete Gaussian curve covers the range $(-\\infty,\\infty)$. If you consider *any* finite interval, it will only contain a subset of the actual curve. If that subset is narrow relative to the curvature at the peak\\*, then you *can* only see the very peak of the curve in that interval. $\\quad$ \\*(given that it's centered within the interval, as it is in your problem) $\\,\\,$ I really don't see the problem here.
$10^{12}$ is for the trillion fair coins, and 1 is for the unfair coin.  And I haven't confused $\\alpha$ and $1-\\alpha$ the Clopper Pearson interval listed [here][1]
You should always look at the data before you run a regression, yes.  Doesn't have to be a histogram, though.  In the two-variable case you describe, I would plot a scatterplot (maybe with a histogram on the margins).  And look at it for a while.  Because maybe there's a crazy outlier observation.  Because maybe $y$ only takes on three values.  Because maybe the $y$ are all positive, or all between 0 and 1.  Because maybe the relationship between $y$ and $x$ looks less like a line and more like a hump or a circle or a hyperbola.  Or ...  Who knows?  If you don't look, you certainly won't know.
The terms you're using makes this somewhat unclear. We typically use the *generalized* linear model (eg, logistic regression) to classify, not linear regression. If you're referring to *discriminant function analysis*, that can certainly be used (it's more efficient than logistic reg if the response variables are multivariate normal), but DFA isn't usually called multiple linear regression in my experience. If you want to classify cases into more than 2 categories, you might check out [multinomial logistic regression](http://en.wikipedia.org/wiki/Multinomial_logit).
+1 for introducing the idea of loss.  (But aren't "exponential", etc., *distributions*, not loss functions?) Historically linear loss was the first approach formally developed, in 1750, and there was a straightforward *geometric* solution available for it.  I believe Laplace established the relationship between this and the double-exponential distribution in an 1809 publication (for which the MLE will minimize absolute error, not squared error).  Thus squared loss is not uniquely distinguished by the criteria of having an MLE and being mathematically easy.
As i understood...I have to assume a SD by myself regarding the 7point Likert Scale...Maybe it's better to ask, how we can calculate a Standard Deviation for 7.point Likert scale?
Thanks very much for your answer gung. I'll have a look at this test this evening.
It depends on your goal with the normalization. What do you want to achieve with the normalization?
This might be useful: http://stats.stackexchange.com/questions/23117/which-mean-to-use-and-when
Aim: Investigate if morphological variation exists among 4 locations (location is sole IV). Co-variate (CV) is body size, which does not significantly differ among locations. When I run the glm on all traits & include the interaction (IV, CV & IV*CV), a significant interaction is recorded (between location and body size). I usually 1) perform this test, 2) see no interaction, 3) re-run glm excluding interaction. Text books tell me that a significant interaction = MANCOVA model is inappropriate. Although some online tutorials disagree, I am yet to find a key reference.
+1 - What is your definition of spurious though? If you include a time trend, e.g. $Y_t = a_0 + a_1X_t + a_2t + e_t$, $a_1$ should go to zero asymptotically.
Why wouldn't you resample participants with replacement and then resample their data as well?  That seems to be more in keeping of the spirit of a multilevel model with one distribution nested within another.  Another point is that there is a potential problem with binomial data because the extreme ends of the samples will be less likely to converge.
1. No this is not homework.
2. That's true, I should have been more specific: The equation in my model is actually $\\frac{1}{1+y}$ so it does not diverge.
You might get more responses if you upvote or accept solid answers to your previous questions.
Thank you. Since the $\\epsilon$ is iid, its covariance with $y_{t-1}$ is zero right? How can I show that the $Cov(y_t,y_{t-1})=Var(y_{t-1})$?
How do i calculate perplexity for pLSA. I have datamatrix $X$ which has the count and by TEM algorithm $p(d)$ and $p(w|d)$ are calculated.
$P_Z x$ is the orthogoanl projection of $x$ on the linear subspace $Z$. And $P^\\perp$ denotes projection on the orthogonal.
@downvoter - the idea is to write *why* you downvote so that answers *improve* over time.
It's clearly explained under the table what was done and why. The 0.05 level was the *overall* type I error rate; the per-test type I error rate was therefore lower.
@Glen_b Yes, my mistake. I was focusing on the title and overlooked the details of the question.
This question appears to offer insufficient information for an answer. On what basis are we to compute a probability for correctly guessing that a word has changed? Are you assuming that this guess is made by a mechanism tantamount to flipping a fair coin? If so, why is that a reasonable model for "chance" in this experiment?
I confirm your function needs to return a scalar, otherwise what is there to maximize? Also note that `nlm` does a minimization so you'll have to negate your objective function.
+1 It might also help the OP to observe that the very same correlation coefficient appears in three distinct kinds of analysis: correlation of $X$ and $Y$, regression of $Y$ on $X$, and regression of $X$ on $Y$.  This gives it three different possible interpretations, which may be the source of some confusion that motivates the question.
Apologies, I thought the intention was 1-sentence answers. I have edited above.
1. What do you know about the stochastic process generating the points? 2. What do you mean; how to *extrapolate* the data?
I had experience with SQL. I learned myself the rest and got a base certification. Not sure it will be any benefit to get the advanced from an employers perspective.
I think you should spell these out. People from many countries read these and acronyms can be hard to read in translation.
It may make sense to just throw the variable out and compare the model with the factor only to the model without anything. It makes little sense to, e.g., compare the model with the interaction to a model with the interaction *but without the main effect from the factor*. As I wrote, it depends on what "matters" means to you. You may want to look at AIC or cross-validation to compare models (which I would simply call "comparing models" ;-).
With my limited knowledge, it's to: 1. to remove heteroscedasticity
2. to solve the problem of different dimensions
"*If you don't calculate your confidence interval you've got something similar to the hidden coin and it has a 95% probability of containing mu just like the coin has a 50% probability of being heads.*" -- I think you got the analogy wrong here. "Calculating the CI" doesn't correspond to revealing the coin, it corresponds to calling "Heads" or "Tails", at which point you *still* have a 50-50 chance of being right. Revealing the coin corresponds to *seeing the population value of $\\mu$, at which point you can answer the question of whether it's in the "called" interval. The OP's puzzle remains.
The www.r-project.org/useR-20xx  URLs are preferable.
Just to be clear from the start - none of the items are actually normally distributed, even the ones stated in the question to be normal.
Define what is considered a noisy label. Instance duplicates with different labels?
Sorry, I mistyped logit when I meant probit. Thanks