tttjjj/elt_bench huggingface dataset:
...And also, how many variables? Two or more?
Thanks @wolfies.  That's what $\\bar{W}$ is, but my question is to specifically address the case of $\\bar{W}'$
I guess so? but the comparison seems strange in contrast with SPSS. It sounded like you were saying SPSS was more of a "language" than R...
I mean that for any continuous random variable, the probability mass function of any discrete point has value 0.
I don't get why you are throwing away essentially all the reputation you have accumulated so far to hope for an answer to this.  The reason it is not getting attention is probably because it is so specicalized that no one who has seen it thus far has the expertise to answer it.  If you get no answer for the next six days you lose 50 reputation points and get no benefit.
You might find [this question](http://english.stackexchange.com/questions/7894/x-times-as-many-as-or-x-times-more-than) and [this question](http://english.stackexchange.com/questions/91263/why-is-a-100-increase-the-same-amount-as-a-two-fold-increase), located on the EL&U site, to be helpful.
This is a wonderful example of an estimator that exists in theory but seems unusual in practice (until you look for this sort of thing): given *any* set of data, it returns a predetermined number (80% in this case). It is easy to compute, very inexpensive (data collection costs can be reduced to zero) and has zero variance. It is Bayes (for an atomic prior) and admissible. There will still be nagging questions about its bias and consistency that can be difficult to address and won't go away by avoiding a "detailed analysis."
Hi @mattia, I think these two posts ([1](http://gpjacob.blogspot.com/2010/06/multinomial-logit-in-stata-and-r-ii.html), [2](http://gpjacob.blogspot.com/2010/06/multinomial-logit-in-stata-and-r-iii.html)) may be helpful, to translate between `Stata` and `R`.
Hi Nick, thanks for your answer.

Re the multiplication of data. This is just an initial test, using already existing data. I am currently collecting data for the actual research, but trying to get a model running so when the data comes in i can sub out the old (pointless) data and sub in the new data. But what your saying makes sense, and I agree with it. 

Thanks for the feedback on the lack of useful information in my original post, I'll provide more usefull information in the future. 
I will also get the syntax instructions I made for this model next time I am in at UNI.
One way which may make it easier to build your model is to use polar co-ordinates instead of cartesian ones.  If you set the origin equal to the enemy ship, and have the ship facing north always, then you could say something like position of the sub at time $t_j$ is $(r(t_j),\\theta(t_j))$ with $r$ being distance and $\\theta$ being angle.  Now we expect $|\\theta|$ to be small because the sub is usually in front of the ship, and $r$ should be small but not close to zero (else the sub crashes into the ship).  You also have $r$ getting large for ships that pause.
Hey, I'm just glad you still care (after all, your original post was a couple months ago...). Definitely check out that first article--it's by Hal Varian, who is now the chief economist at Google (plus, **JEP** articles tend to be much more readable than your typical Economics fare).
@andrea - BINGO!  I just changed my output to be displayed to 6 decimal points, and sure enough, the estimate for HHINCOME is now 0.000068, with a standard error of 0.000028. Seems really obvious now, and I thank you for pointing this out to me.
I suspect your understanding of "moving average" (as a time series *model*) may differ from the OP's understanding (which might be the more ordinary sense of a moving-window weighted average).
@kkp Strange, did you run the first three lines defining `x`, `y`, and `lims` before continuing with the second example?
@Thierry: Assume that they get the flu on Monday, Tuesday, and Wednesday of the first week.
I tried `STDEVP` (and `STDEV.P`.) But still doesn't match the `correl()` result.
@Jonathan, that is probably a good idea. The classification problem can also just be downright difficult.
I have partially answered my own question in learning that the name for this alternative-based model I am looking for is the conditional logit model. I have successfully run my model using clogit(survival) and mclogit(mclogit) after removing some collinear variables identified via condition numbers and VIF. The clogit function runs with no warnings, mclogit returns a "fitted rates numerically zero" error, and mlogit returns wither "missing value where TRUE/FALSE needed" or "computationally singular", depending on the number of variables included.
I edited my question to illustrate with an example of what my data looks like. When the facility value is high, the item is easy. I would like to know if people in general perceived the right degree of difficulty for the items... If I had only one value each time on the likert scale, I would have paired values. But it the answers on the likert scale are not that easily interpreted... I hope I am clear enough, as english is not my first language (I speak french). Again, thanks a lot for the help, I appreciate it a lot.
Cohoz, thank you this confirms what I had since learned. The issue is the random forest being built using the target variable. There is a missForest package in R with paper that can be used for unsupervised imputation: http://www.ncbi.nlm.nih.gov/pubmed/22039212
+1 for finding someone else who works more with statistics
By the way with Random oversampling I'll get approximately the same confusion matrix for all 3 cases (1. balancing with undersampling, 2. balancing with oversampling and 3. without balancing) and the same test-error. I think this would confirm my assumption that if you have perfectly separated responses (e.g. we have it in the iris data, or in the olives data here), there is no need to balance. I still hope finding agreement :). But I also would appreciate disagreement (especially with a explanation, why balancing still would be necessary).
@B_Miner Hi, sorry for the delayed response. The answers: 1) Well, yes, we usually fill the missing values with zero before running SVD. However, I usually recommend to fill it with non-zero rating - for example, you can fill the missing values by the average rating that the user has given so far. 2) SVD-based approach is for only known users and known items. It cannot handle new users or new items. And how can it, if a new user comes in, we don't know anything about him in this framework to predict.
I've added the `stata` case
I don't understand the multiple regression model, why didn't you include “superordinate identity” in the model?
+1 for the tool for graph scraping!
Exponential scale looks very possible. If y is bounded by 0 and 1 but neither is attainable then logit scale is also possible. More generally, why be put off by a scale being unusual? The more it helps, the more you get credit for a unusual and good idea. But it is best to  ensure that readers, including you, see text labels such as 0.1(0.1)0.9 on the axis. I don't know how to do that in R, which I don't use routinely, but it should be trivial.
Judging from this quotation, the paper is written at a very abstract level and is intended for people who are (a) knowledgeable about statistical problems in the first place and (b) conversant with the theory of Hilbert and Banach spaces and functional analysis. It sounds like this does not match your background--your questions have to do with linear operators, not statistics--and so you would be better off reading a different introduction to statistics. Might I suggest [searching our site](http://stats.stackexchange.com/questions/tagged/references?sort=votes&pagesize=15) for recommendations?
Could you please elaborate on the sense in which this approach would be "more meaningful?"
Did you know there was a mixed methods email group, that especially covers R and Douglas Bates (for example) reads it and posts there? See https://mailman.stat.ethz.ch/mailman/listinfo/r-sig-mixed-models
+1 Nice answer.  A simple rendering of the math is that the null and its alternatives are assumed to yield disjoint sets of outcomes; *e.g.,* either there is a zebra in this room or there isn't.  Of course "prove" here implicitly includes "conditional on the model," which itself is never established with the same rigor as, say, a mathematical theorem; it implicitly includes "conditional on the accuracy of the observations;" and it implicitly includes that the hypotheses can be unambiguously interpreted.  (For criticism of the latter, see George Lakoff's *Women, Fire, and Dangerous Things.*)
I'm currently working on something very similar to this but the theoretical distribution is very hard to calculate accurately, so I'd like a Monte Carlo method that works here.
Can you provide the data to make a reproducible example? I think what you want to do is make a graph of e.g. p(Move) vs. age, for fixed values of the other covariates? I would like to do the same thing for my multinomial model. The difference between your fitted model and the empirical data could be because your empirical plot is not accounting for all the other predictors that your model is. Fit Choice ~ 1 | Age and see what that looks like.
@Srikant, a (hypothetical) Bayesian will have strong disagreements with your answer. She would have done something like this: prior  $\\propto f(\\alpha_1,\\beta_1|-) \\alpha + f(\\alpha_2,\\beta_2|-) (1-\\alpha)$ and then put prior on $\\alpha$. However, your answer will be a little less flexible than the Bayesian's answer.
Welcome to the site, @Jon. This does answer the question of what asymptotic omega is, but I think the OP might like to know a bit more. Eg, which omega should the OP use? Would you care to elaborate on this a little bit?
+1 The illustrations and explanations are very clear and well done.  Thank you for the effort and research that (obviously) went into this answer.
I like "zero-added beta distribution" as suggested by @Hibernating
@Robby McKilliam: But what data do you use?  This question arises before any data have been collected.  If you collect values one at a time and compute a CI after each new one is added to the dataset, you cannot use standard formulas for the intervals due to the correlated multiple comparisons you are making.  Thus, you need a *stopping rule* that optimizes the sum of the statistical risk of your estimator and the cost of collecting each additional sample.
CORRECTION: I actually have only about 84 cases. My previously conclusion resulted from a gross tabulation error (i.e., only 84 of my 4000 cases apparently have post scores, which I was not previously aware of). Regardless of the practical meaningfulness of the differences, I would like to know how I could calculate for statistical differences, for this paper and for the future.
Aniko, Thanks for your reply. I have a couple of things to say and ask. So, this is going to take more than 1 comment.
1) Yes, you're right. There is no p-value here. From my RNA-Seq data, I have some million reads. Suppose you have a gene region E1,I1,E2,I2,E3 (I=intron, E=exon, this is not the whole gene, just a portion). Every E and I have a start and end coordinate. Some spliced reads map to I1 (meaning part maps to E1 and other part to E2), and some spliced reads map to I2 (meaning a part maps to E2 and other part to E3). But sometimes a read maps across I1 and I2 (reads across E1 and E3)
You probably need the same thing as this questioner: http://stats.stackexchange.com/questions/53407/which-statistical-test-to-use-for-likert-scale-questionaire/53409#53409.  You also need to be careful because it sounds like you are thinking of doing many many tests to identify which factors are significant on possible multiple response variables.  You come up against problems from the data mining field (which, intuitively amount to, that you are bound to find some associations by chance if you look at enough possibilities).
To compare mean differences between/among groups by, let's say, just subtracting the averages is not sufficient to establish statistical significance. Rather, it is important to capture the variability of each of your groups that you're comparing, as well. To do this, ANOVA compares the raw data of each group to corresponding group means and an overall (total) mean.  These comparisons yield "sums of squares" that when divided by degrees of freedom give you "mean squares." These mean squares are, for all intents and purposes, variances. In the end, you compared means but you analyzed variances.
@ Alexis See e.g. this post, http://stats.stackexchange.com/questions/80611/problem-understanding-the-logistic-regression-link-function/80623#80623, and the wikipedia article, http://en.wikipedia.org/wiki/Logistic_regression#As_a_latent-variable_model. This approach clarifies also that it is the assumption we make on the error term of the underlying model that determines what model we will obtain at the Probabilities level. For another example, if we assume that the underlying error follows a uniform, we obtain the Linear Probability Model, see, http://stats.stackexchange.com/questions/81789
@naught101. Thanks for the direction. I thought it would be easier to answer if I wrote in abstract ideas rather than specific variables but here goes. I'm a statistician so just wrote
Are you talking about loglinear models for proportions (generally in tables) or loglinear models for something else?
Is there an appropriate alternative?
@F. Tusell: that was a good description. I already puzzled this together from other posts, but this summarized things well for a beginner like me.
+1.  Maybe the ranking depends on the order in which H was computed.  E.g., one could be KL(green, red) and the other could be KL(red, blue).  Morever, tail behavior can have a profound effect on the value: what we really need to see are plots of the *logarithms* of the densities, not the densities themselves.
BTW, Wittgenstein's [family resemblance](http://en.wikipedia.org/wiki/Family_resemblance) is where early clustering (such as hierarchical methods) stems from.
I don't know what you mean by common practice in the literature. Among statisticians it is considered bad practice.
+1 Good thorough answer: it provides options and useful comments.
Once you manage to put your data in R, there is a fast track to try two de-facto state-of-arts: SVM with Gaussian kernel (single call to `ksvm(decision~.,data=yourdata,cross=10)` from kernlab) and Random Forest (`randomForest(decision~.,data=yourdata)` from randomForest) -- both one-liners will also give you a reliable approximation of error.
I should ask before I begin: is this homework/assignment? Can you show me your thoughts?
Could you expand a bit on what you mean by "scale"?  After all, an EMST is a set of edges associated with sets of points in a Euclidean space, whence they have coordinates implying a built-in "scale" and natural methods of plotting.
What question are you trying to answer?  How would you analyze the data if none were missing?  We need more details to have any hope of providing a helpful answer (I can think of at least one useless answer without additional info, but guess you would prefer real help).
Wow, this helped. Thanks!
Since all the parameter estimates are the same, just scrambled, it looks like you have a coding error somewhere.
Thank you for your quick answer. The original coefficient is -.3365 but I think thats okay as exp(-.3365) roughly is .7 as well right?!
Have you heard of INLA? may be what you're after.
Oh that is brilliant. I had not considered edge.betweenness, but it makes total sense. Thanks micans.
I am also looking for similar examples - `http://stats.stackexchange.com/questions/104481/hierarchical-text-classification-in-r` . Did you find some thing on this, can you help
This is pointless. Bayesian methods are not a panacea. In survey statistics, the random elements are the indicators of sample inclusion, not the measured values, and the likelihood of any SRS is just $(C_N^n)^{-1}$, i.e., non-informative.
the "unique solution" argument is quite weak, it really means there is more than one value well supported by the data.  Additionally,  penalisation of the coefficients, such as L2, will resolve the uniqueness problem, and the stability problem to a degree as well.
I added some example code to show the basic process.
hmm, thanks! sas even has a proc arima.  I'll look into the Chow test, haven't seen that before.
I think [this](http://people.duke.edu/~rnau/seasarim.htm) might help you.
Actually if we suppose we have n periods, in the first one the constant part is equal to μ while in the second one equals to nμ
You can multiple the two values, and this gives you a log odds.
I wholeheartedly agree with @whuber, that this data classification scheme is "far too limited and outmoded to be a useful framework". Here is a preprint of a paper you might find helpful: http://www.cs.uic.edu/~wilkinson/Publications/stevens.pdf
AFAIK, the conditional is computed this way, $p(x|y)=p(x,y)/p(y)$, but Bayes formula is $p(x|y)=p(y|x)p(x)/p(y)$. I don't see why finding conditional is using Bayes formula, could you please be more specific?
I simply don't have more than a single point for many of the predictions. I don't want a model to be unfairly penalized because it never served the same prediction more than once (which I assume using a statistic that prefers a larger sample size may do).
@StasK is there a Bartlett-type decomposition for the inverse-Wishart? I only know the one for the Wishart.
Warning, the slide from that presentation on "Best Practice for Reporting Model Fit" does not seem to me to be good practice.  Step one is to use cross-validation to identify the best model and step 2 is to report the RMSE and MAPE from the cross-validation procedure.  This will underestimate the true RMSE and MAPE as the cross-validation statistics will be optimistically biased because they have been used to choose the best model.
Thanks I will have a look in more detail.
Now I am wondering if 1) the specified model above is still valid for analyzing this dataset (for an example please see my first comment to your answer)? And 2) does this design fall under the term incomplete block design? My answers: 1) Yes, since "site" is still a random component and not a fixed effect (which is "clone"); 2) I really don't know... Thanks in advance for any answers!
Out of curiosity, is it intentional that in your discussion of regression for prediction (starting p. 33) you choose predictors (step 1) *before* partitioning into training and validation datasets (step 3)?  I have thought that the most objective and honest procedure would be to partition at the outset, even before looking at scatterplots (step 2).  If regressors are chosen based on the entire dataset, wouldn't that inflate apparent significance levels on many tests even when they are subsequently applied to the validation data?
Thanks for your answer :) I already tried that but now I haven't a satisying number of observation for a lot of levels. So the trees are really unstables.
Since the connections have a direction (arrows) it could be called a directed graph.
+1 - In addition to the many groups notes - with only 15 values box-plots (while still useful in general) can be misleading (e.g. bimodal data). Super-imposing the points on the box plot can show such discrepancies (examples [here](http://stats.stackexchange.com/a/22733/1036) and [here](http://stackoverflow.com/q/9232331)).
@dmcnelis, actually this IS **a crucial assumption**, that all you have at hand are two (probably coming from poorly specified model) got by $OLS$ estimates of the parameters. I have had several suggestions what to do (testing changes in parameters, if a structural break occurs, weighting on the basis of relative usefulness (forecasts accuracy) of past data to forecast next years values, etc.), but under such an assumption any suggestions a la $(k\\cdot B + A)/(k+1)$ are just fine here, where $k$ is your subjective opinion.
In that case you should be okay. You can consider each of the samples A,B,C to be experimental units, and each of the categories a,b,c to be some measurement taken on each sample (with the restriction the measurements sum to 1.) This could be helpful when you have a lot of samples, not so much when you have 3 or 5.
I think I see what you mean. Part of the solution is how many ways I can choose N + 1 - L subsets from  K draws. The next part is in how many of those subsets does each number occur. I am yet sure how to consolidate the fact that I know there are M possible subsets with how many of those subsets I know contains come number Xi. It does follow a pattern, but I don't yet know how to express it.
This question appears to be off-topic because it is not about Statistics
Please, add more details -- the crucial thing is WHY do you have multiple values if there should be one: are these accumulated over last timestep? Coincident in output from many detectors?  Something else?
For what hypothesis test? Can you ask the person who told you this?
What's wrong with newton rhapson?
Do **not** use k-means with other metrics, it may stop converge and will yield suboptimal results: The *mean* is inappropriate for (most) other metrics, it does *not* yield an optimal representative for other metrics than squared Euclidean.
You can specify the limits of the fill scale: `scale_fill_continuous(limits = c(0, 0.05))`
But how about an hmm based threshold model for gesture recognition described here : http://herin.kaist.ac.kr/Publication/PS/hklee_PAMI_i09611.pdf . They create a threshold model which is an ergodic hmm that has states of each hmm combined together. "The threshold model acts as a base-line. A candidate gesture is found when a specific gesture model rises above the threshold" But my problem is that I am using java and jahmm library and I don't think it has the option for ergodic hmm.
Beta regression does *not* require 0 < y < 1, it is simple to transform the data.
This looks like homework. If so, it should have the `self study` tag.
@Galled, question not really understood. Both R functions compute valid residual deviances and AICs, but the absolute values can not be compared between the functions. This is a matter that one should always be careful about. AIC is only defined up to an additive constant and if you compare two models in terms of AIC you better make sure that you use the same constant (in technical terms, the same reference measure).
The question is about transformation of a variable not a test, so I don't think your answer applies. Also, it is probably more informative if you give the full reference instead of just a name-year reference and some people object against SHOUTING.
I think you are on exactly the right track
Yes: that's a fundamental data limitation.  I would still trust `lme4` more than `glmmPQL`.  I also tried this problem with the `glmmLasso` package -- as expected, it throws out all the PC variables ...
In addition to @SvenHohenstein's good answer below, it may help you to read my answer here: [Interpretation of simple predictions to odds ratios in logistic regression](http://stats.stackexchange.com/questions/34636//34638#34638), which contains additional basic information about probabilities & odds. Note that the logit can be understood more abstractly as a link function; you can read more about that here: [difference-between-logit-and-probit-models](http://stats.stackexchange.com/questions/20523//30909#30909) (although this answer might be a bit more technical).
(+1) Obviously, this will also work for the y-axis, as requested by the OP :-)
Ahh, I see your point. I was thinking of an approach to find the non-zero clusters in a matrix. I was thinking we could find the largest cluster of zeros, say it is of size $n_{1} \\times n_{2}$. I can now use this clustering algorithm recursively as follows: Use it on the two sub-rectangles above and below the red rectangle and on the sub rectangle on the right etc.
(1) What's the variance of $X_i$? (2) "*Why not just use the student's z-distribution? After all isn't that the true distribution for the sample mean*?" -- what is this belief based on?