tttjjj/elt_bench huggingface dataset:
It also helps to rewrite the model as $r_{t+1} = (\\alpha + \\beta_{2} r_{t-1}) + (\\beta_{1} + \\beta_{3} r_{t-1}) r_{t}$. This is a simple regression model $r_{t+1} = \\alpha' + \\beta' r_{t}$ where the absolute term $\\alpha'$ and the regression weight $\\beta'$ are both linear functions of $r_{t-1}$. $\\alpha'$ and $\\beta'$ are often called "simple intercept" and "simple slope", respectively. To choose values for $r_{t-1}$ for the interaction plot, I've often seen people use the mean, and $\\pm$ 1 standard deviation.
"on the other hand the programmer in me wants an absolute and 100% correct answer". I'd suggest changing your user base to automatons of some sort.
@StephanKolassa Maybe there's something I am missing, but it seems fairly straightforward to generate the most extreme ones. That's what the various specialized 'exact' permutation test algorithms try to do (eg various papers by Mehta, Patel and some of their coauthors, as well as a number of others) - specifically to move through the extreme permutations of various statistics in order from most extreme, in efficient ways. Is there a reason why such approaches can't be used/adapted to work with tau-b? Ties complicate it, but I don't think they'll prevent those approaches from being used.
Do you mean the **standard error** for each proportion? If this is what you are looking for, then this webpage might be of interest for you: [Interval Estimate of Population Proportion](http://www.r-tutor.com/elementary-statistics/interval-estimation/interval-estimate-population-proportion)
This is called 'suppression'. Please check out the following CV post: ["Suppression effect in regression: definition and visual explanation/depiction"](https://stats.stackexchange.com/questions/73869/suppression-effect-in-regression-definition-and-visual-explanation-depiction).
Inquest, because you are in a small minority who uses this convention, I have edited your question so it will be understood by others as you intended it.  (+1 for an interesting question.)
I am not confusing the two. We would say that group 1 and group 1 are collinear if $\\text{group 2} = \\lambda_0 +\\lambda_{1} \\cdot \\text{group 1} + \\varepsilon$ where $\\varepsilon$ is some noise. If $\\varepsilon$ is sufficiently small, then we would want to remove one the variables.
You can estimate both; there are a number of methods. Besides what's already mentioned in answers, take a look [here](http://en.wikipedia.org/wiki/Mixture_model#Parameter_estimation_and_system_identification)
You can apply a test like Ljung-Box on each individual series to get p-values for each case.  Now these so-called raw p-values need to be adjusted because of multiple testing.  There are many ways to do adjustments that can be found in books on multiple testing/simultaneous inference including resampling approaches given by [Westfall and Young](http://www.amazon.com/Resampling-Based-Multiple-Testing-Adjustment-Probability/dp/0471557617/ref=sr_1_1?s=books&ie=UTF8&qid=1345904571&sr=1-1&keywords=Westfall+and+Young)
An easy way to see this is : you will take each element (1), or not (0). So you can represent all binary number with n bits : 2^n. And this amounts to all combination with one item removed, plus all combinations with 2 items removed,and so on.. = sum of C(k/N).
This is far and away the aspect of Excel that infuriates me the most.  Data storage needs explicit data types, not formatting.
Thanks for responding! I updated my original question because I ran out of characters in this comment box. It would be great if you could give some feedback for some of the details I wrote in the update.
Think this way: if your statistics is right on 1.645, then p-value = 0.05, which is the area left of the right hand side of 1.645. Now, slowly slide the calculated statistics to the left (towards negative), what would happen to the p-value when you stopped at 0? And what would happen to the p-value when you slide the calculated statistics even further left, into the negative?
Using X_1, X_2 as independent copies of X as above is there anything simple to be said about X_1 + X_2?
Simpler and very likely to be seriously close to what you would get doing it the "right" way.
@onestop: I am sorry i missed your question. The comparison is on log odds ratio. Thank you for your interest.
That seems like a question suited for the [GIS site](http://gis.stackexchange.com/). While the R users on the site are limited, potentially other programs have compiled that data into other formats suitable for geoprocessing tools. I have seen several web mapping examples use that info, and so I suspect some on the GIS site do have experience with it (or at least knowledge of it).
I don't really understand what you mean by the comment in italics, what I have done from 'there' (whatever there might be) is written under **My answer**. My teacher does exactly the same and *I understand* what he does, my question only is why he used a wrong form of the formule **he taught me** Also, the two-sample t-test is significantly different from what we've both done, so.. (besides, we are doing very basic hypothesis testing, no t-tests etc.)
I can't understand how what you are doing crosses with your question title. Multiplying orthonormal data by Cholesky root of a covariance matrix makes the data covariate right according that cov matrix. And if the data is a random sample from orthonormal population, then the same trick yields data which is a random sample from population with the cov matrix. But sorry, I can't trace, how it connects to drawing an ellips.
Terms random effects, fixed effects, latent growth might mean different things in different contexts. Concerning the second one Andrew Gelman had a blog post with examples of several definitions. So it would be great if you provided the links to the definitions of these models. In general I think you are right in your assumption. Time trends are usually treated separately, since the usual assumption that variance of the regressors is bounded does not hold, so you have to show that for time trend this does not actually change anything in terms of model estimation and interpretation.
Please, use $\\LaTeX$ (see [help](http://stats.stackexchange.com/editing-help#latex)) to format your equations (use the `align` environment), and check their correctness.
+1 Good advice and good discussion.  But *lower* kurtosis?  Don't you mean higher?  You can experiment (in R) with `library(moments); apply(matrix(1:5,5,1), 1, function(p) kurtosis((1:100)^p))`: notice how the kurtosis increases as the right tail stretches out under higher powers.
Absolutely! That would be very helpful.
What data do you have or could you collect?
Good answer but one note. Not all software uses reference cell coding by default (although I think this is the best option). SAS in PROC LOGISTIC uses [effect coding](http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_logistic_sect006.htm) and I vaguely recall that S+ at one point (or maybe it was an earlier R?) used Helmert contrasts.
@MarcClaesen Your statement `"stratified random constitutes 1 point in ROC space"` isn't correct. The same could be said for a uniform random classifier. A stratified random classifier could output scores with a Beta distribution fitted to the label information of the training data, for example. You would be able to get a curve, and not just a single point. So it **does make sense** to talk of the AUC of a stratified random classifier. The question is: what AUC can we expect for such a classifier, and if the comparison is useful (e.g. would it outperform a uniform random classifier?)
@user52672: My point was that you should use two-way MANOVA, and not one-way. But if I were you, I would not rely on any single test. In addition to this MANOVA, do two-way ANOVAs on height and on weight separately; make scatter plots of height vs weight in different groups; make histograms and look at them. **Plot the data and look at it!**
This is great. Could you explain some of the logic/intuition behind the decomposition ordering?
thanks Peter for the feedback, i've edited the answer to give a little more details
Thank you for the clarification.  The point is that at each stage you only have to consider the probability distribution over (W,L,T) and not how it was derived.
Does the rule for determining significance by whether the confidence intervals include 0 or not apply only for normally distributed data?
What you are probably looking for is known as random subspaces (RS). This is the universal approach that has been published even before the random forest (RF). However, RF proposed using out of the bag samples for building the ensemble.
There was a similar question with good answers: http://stackoverflow.com/questions/7545962/data-mining-for-significant-variables-numerical-where-to-start
That pattern is not noise: it suggests a lack of a linear relationship.  There are not enough data to tell us whether that suggestion is correct.
If you want a numerical answer it would help if you uploaded your data. You could post it inline too.
One could look at this as two completely separate questions.  (1) When $X=0$, how would you predict $y$? (2) When $X=1$, how would you predict $y$?  If you have a dataset of $(X,y)$ pairs, these two questions deal with disjoint subsets of the dataset--in effect, two different datasets. In each dataset $X$ does not vary.  Thus each is the same question and that common question boils down to "I have a set of $y$ values. How do I predict them?"  That is a standard, simple *univariate* problem--and it has many more solutions than those offered by regression models.
In such a nonlinear model, this is not fully well-defined, e.g., if you fixed all the other variables at their medians you'd get a different "marginal effect."
@PAUL I would suggest uploading the actual data rather than some arbitrary manipulation of it.
I am not sure if it is clear, but the histograms are of the 30 values of variance, and the barplots are the raw values of the variance, i.e. `var <- c(0,0,1,3,10,100,150), hist(var), barplot(var)`, so I interpret this as a few parameters explain most of the variance, not that most of the explained variance is in the tail. Does that make more sense? Sorry if it was unclear.
Let the intended mode be $(x_1,\\ldots,x_K)$ with $x_1+\\cdots+x_K=1$. Pick any positive number $N$ and set $\\alpha_i=Nx_i+1\\ge 1$ for the parameters. Because $\\sum_i\\alpha_i-K=\\sum_i(Nx_i+1)-K=N\\sum_i x_i+(K-K)=N,$ the mode of a Dirichlet$(\\alpha_1,\\ldots,\\alpha_K)$ distribution has components $(\\alpha_i-1)/(\\sum_i\\alpha_i-K)=(Nx_i+1-1)/N=x_i$.
This thread might help you: [How to generate correlated random numbers given means, variances, and degree of correlation](http://stats.stackexchange.com/questions/38856/). AFAIK, there's nothing special / required about having a normal error term (eg, see: [Pearsons or Spearmans correlation with non-normal data](http://stats.stackexchange.com/questions/3730/)), but $r$ can't reach 1 or -1 if the marginal distributions differ.
Hey I have `/path/to/results.csv` and `/path/to/command.r`, and it gives me `source("/path/to/command.r")
Error in file(file, "rt") : cannot open the connection
In addition: Warning message:
In file(file, "rt") :
  cannot open file 'results.csv': No such file or directory`
I don't think the discreteness of the data makes a big difference here, unless your counts are very small.  The issue is rather about the distribution of the data.  For binary data the prediction is a continuous probability between 0 and 1, but the response is always 0 or 1.  In their example the force the cost to be "right" or "wrong" and count the number of rights and wrongs.  Are you particularly worried about continuity?  I'll add a P.S. to my answer to address continuity
It is never lupus.
We have included the previous state now as some other predictors were affected by its omission.
Can you say a and b are independant?i.e. p(a)*p(b)=p(a and b)
Did you understand the purpose of "Shrinkage" in LASSO?
The correlation between $X$ and $X + Y$ will depend on their variances and their covariance. That's evident from the formula, but it also follows that there is no general effect. The point about the correlation between $X$ and $X + Y$ when $X, Y$ are uncorrelated is that it provides a benchmark for the inbuilt correlation even where there is no "real" relationship. I don't see how you can get further without either special assumptions or simulations for plausible cases. It is evident from the case $Y = a + bX, b > 0,$ that the correlation could be as high as $+1$.
@EconomiCurtis, If you mean to program computation of a binary (such as Jaccard) measure this question might help:  http://stats.stackexchange.com/q/49453/3277
A convergence in probability result is rather straightforward (as I guess you know, plim goes places that the expected value can't).
Hi Shinobi. As things stood, your original question "How do I make Package-A do calculation-X?" did not appear to be on topic for CrossValidated. I have edited it to be more clearly on topic (as well as fixed up some of the formatting). If you disagree with my edits, you can revert it to undo my changes, but then you should edit subsequently to present a statistical question rather than a "what buttons do I press in my package" question, or you can revert it without editing and flag it so the moderators can move it to StackOverflow. If you only disagree with some of my changes, you can edit it
Are these predictions on new data not in the training set?  If so, have a look at [this paper](http://www.csie.ntu.edu.tw/~cjlin/papers/plattprob.pdf).
@desperate, doing cluster analysis on components is wide-spread practice, but a bit risky one: sometimes clusters differ mostly on those junior components that you discard. So, why not cluster on all initial 40 items?
Thanks, Peter. I'll consider a GLMM, although I would like to capture other characteristics of the situations within which these recurrences happen, so perhaps that would call for a HLM.
I think you're describing two related, but different questions here - 1) how does your respondent space group together to find cluster of participants who behave in a similar fashion? This is primarily addressed through a latent cluster analysis technique (of which there are many). 2) Can you reduce the attribute space from 250 questions down to a smaller number of questions? This is primarily done through a factor or principle component analysis. One of the key outputs to this is how your observed factors "load" onto the unobserved factors. This sometimes is used as an input into clustering.
Thanks a lot, I see much more clearly now. I already had many doubts regarding Mann-Whitney / censored data. Thanks again
Many thanks whuber!!!! I'll take a look
I don't think I am wrong. Just think about it like this: The MSE is actually the sample variance since $\\mbox{MSE} = \\hat\\sigma^2 = \\frac{1}{n}\\sum_{i=1}^n (x_i-\\hat x_i)^2$. The only difference is that you divide by $n$ and not $n-1$ since you are not subtracting the sample mean here. The RMSE would then correspond to $\\sigma$. Therefore, the population RMSE is $\\sigma$ and you want a CI for that. That's what I derived. Otherwise I must completely misunderstand your problem.
Your question touches on a number of essential aspects of color science. For a suitable answer, it would be good to know what you are ultimately trying to achieve with your analysis.
A useful answer to a related question to your new question is [here](http://stats.stackexchange.com/questions/55663/denotation-of-probability-symbols/55665#55665)
If we believe in models, then model-based estimates are ideal, and they are the most powerful/sensitive/precise. There are various classes of measures, such as explained variation measures like $R^2$ and generalizations of that.  Other measures concentrate on the variety of predictions achieved by the model.  A histogram of predicted values goes a long way.  ROC curves envision different cutoffs.  Cutoffs are misleading and dangerous; they give rise to categorical thinking, i.e., treating all persons in a group as if they have identical characteristics. Another approach: partitioning deviance.
Just found out that it's not possible to specify constraints on parameters in flare package
@Elvis If there are 0's in an array .. how are we supposed to handle that situation?
Looks like $\\delta$ is making the link between discrete and continuous. $X_k^i$ is in the discrete domain and the $\\delta$ function defines an infinitely small interval around $x_k$ for the purpose of making $\\pi$ a valid density. But still, I don't see what is the probability of $x_k$...
Thanks a lot for your help! That's a very interesting idea, I didn't know that this is also allowed. Just by chance, do you have a reference in mind who did such a double between/within permutation?
@whuber I was certainly assuming a model, not numeric derivatives. The OP doesn't seem to have a theoretical model for the problem, though I am sure there are theoretical models for this. [I'm not really in a position to give the specific subject area advice - at least not without researching it myself, which I think goes beyond the scope of what's usually to be expected (we can't be chemists and physicists and biologists and astronomers and geologists and psychologists and economists and actuaries and linguists and ... ).
I don't know the answer, but +1 for asking an interesting question! :)
I understood the equation. My question was: If storing the dictionary is no problem then: Instead of storing |T| idfs one stores |T| denominators (of the equation) + number of documents. Incremental update is no problem then and idf is calculated on the fly. I have the feeling that I have overlooked something.
@whubber, why? It is perfectly relevant. Ok, I changed the post a little to be more clear.
+1 `MASS::Null` should also get this answer.
Thank you, my confusion begins to lift.
Thank you but there is no factor $\\sigma$ in the exercise as it is a standard normal density we are concerned with. I appreciate your insight.
Robert Gallager's DRAFT OF  STOCHASTIC PROCESSES: THEORY FOR APPLICATIONS (http://www.rle.mit.edu/rgallager/notes.htm) is a good free alternative for an introduction to stochastic processes including a discussion of the Poisson process
Consider: what events (situations) are included in "At least seven females"?
@nclfinance Please read the FAQ and don't treat this place as a forum.
I'm not an image processing person, but perhaps you could try running a sobel or canny operator on the image to detect edges first?  Also, how familiar are you with SVM?  I ask because it is not a technique that generally works well 'out of the box' it requires some parameter tuning, kernel selection and feature scaling, etc..  If you aren't familiar with that and are getting poor results it could be due to that and not necessarily to the data or features themselves.  You may already be doing that of course.
did you try google/ wikipedia first ? 
http://en.wikipedia.org/wiki/Normal_distribution
Unfortunately, if you aren't interested in statistical inference, or are scarce on time and/or resources, the ritual does seem very appealling...
While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference.  Link-only answers can become invalid if the linked page changes.
Look into equivalence testing.
That depends on the accuracy you want in your p-values! I often regard 10,000 as a minimum - near the 5% level, the standard error in the estimated type I error rate for a given critical value is $\\sqrt{0.05\\times 0.95/10000}=0.0022$, meaning we should be within about $0.005$ of the correct rejection rate; I often do more - sometimes many more if I need to construct a table or something, sometimes fewer if I am not fussy about my actual rejection rate (not doing a formal test, for example). It's not unusual for me to do a million simulations if it's fast. I use small runs (~100) to time &debug
Nope, the likelihood will remain *exactly the same* because a jacobian factor will come out of the transformation to make them equal (dx to 0.1 dx)
Thanks! My point is that for one of the model, I cannot access the model but only to its "output" (thus n% interval). So BIC/sensitivity analysis/Bayes Factor seem not possible. The calibration study (with different levels for the HPD interval) seems a good choice. Have you any guidelines or recommendations to perform such an analysis ? Thanks a lot.
This is what @whuber calls a prediction interval in the answers to http://stats.stackexchange.com/a/44476/3601
looks like this will be a list / community wiki
"easiest" kind of depends on a bunch of things. But they can be at least fitted fairly easily (in R, for example - there's the packages `mgcv` and `gam`, to get started with).
It was just testing 1 strain vs the other. The trait in question really does tend to be higher for one of the lines, but there is some overlap--the distributions aren't totally separated.
Would you mind developing this answer a bit?
Can you please provide an example explaining what you mean.
@whuber: Personally, I like to see both if possible (mostly since I "read" histograms easier than QQ-plots). But you are right and I stand corrected.
@StéphaneLaurent: I apologize for the confusion. Meanwhile I have corrected the oversight.
**This is the answer given by my professor:**

The k-means algorithm proceeds until convergence by computing the mean of each cluster and assigning data objects to the closest cluster. If the clustering in D were a solution, the two cluster means would be around -1.8 and 0 on the PC2 axis, which would force the data objects between -0.9 and -1.8 on
the PC2 axis to be grouped into the first cluster in the next iteration of the k-means algorithm. Thus, D can not be a solution.
I didn't know about your cross-posting. Had you asked "How can I determine when two histograms are similar?" you would have got a very different answer.
In addition to the lognormal family there are many other common distributions with positive essential support: gamma, Weibull, F, half-normal, etc.  Moreover, *no* distribution is truly "continuous" if you're going to be this strict about definitions: your measurements must be real numbers with finite representations and as such are necessarily *discrete*!
wtt if they all have equal values of the cost function, in what *other* sense do you mean 'best'? What's 'best' for your purpose?
It seems it was set as a one-sided test. The p-value represents the area on the left of the test statistics. There is only 9.426% of area at the left of a z-score -1.3149, but there is 93.769% of area at the left of a z-score 1.5356. The difference in the sign of your z-statistics is due to in test one signups in higher in variant but in test two signups is lower.
Mann-Whitney U is effective for non-normal distributions, including discrete distributions with two values (i.e. binary).  If *all* data were binary, then maybe another test would work better.
What if my data isn't all normally distributed? The NDVI is negatively skewed, and my NDWI is positively skewed. The SWI is normally distributed. Would I still be able to create a linear model?
LibLINEAR is designed for large datasets which can't fit in main memory - but its limited to the linear kernel.  Generally linear kernels work well for text classification and often datasets become very large and are very sparse, so LibLINEAR works well in that case.
I have reshaped the data, put it in SPSS and done the analysis. Looks like the $\\eta_P^2$ that SPSS calculates is the same as you get from doing the calculation from the F value. Perhaps this is because I only have two levels in each of my factors?
@gung is right to ask for the syntax (if necessary, use "Paste" in the regression dialog box), and in order to help you people might also need to see any output tables that list the country categories.
+1, but I'd love to hear more detail about this. To me it seems like odds ratios would always be better, but I also don't deal w/ that sort of thing in the matter that / as much as epidemiologists do.