tttjjj/elt_bench huggingface dataset:
This is what my atmospheric scientist friend and I had basically come to agreement about, but we didn't have a name for it.  I would still like to leave the question up in case some academic with experience doing this wants to chime in.  Still, 100% helpful, thanks!
@mark999 - I did write something on that recently but deleted it when the question was clarified and turned out that the ratios in question were not proportions.  In the event of a response that is between 0 and 1 you could still use a glm with a binomial response but you might get over- or under-dispersion (depending on what has generated the variable).  So a quasi likelihood model instead, still with a logit link function, might be more appropriate.
@Russ You might want to expand a little on how you are interpreting this question.  Your comments suggest you could be thinking of it differently than some other readers. For instance, in light of the OP's latest comment, would you object to, say, a Tukey HSD test?  Because it is based on an ANOVA, it sounds like you would, but why?
@KennyPeanuts Yes, that's what I was suggesting.  

You'd want to compare the *marginal effects* of day between years rather than the model coefficients directly to really make the inference.  A good paper discussion and figures is here: http://homepages.nyu.edu/~mrg217/interaction.html  

The other issue is that you don't say what the distribution of diapause is over days of a year, so I assumed as a default that it is rising or falling.  If in fact it peaks at some time of year then you may want days^2 in the model as well (or some other polynomial or smooth term).
I agree with @NickCox - the only purpose I can see to this exercise would be if it were part of an introduction to finding MLEs numerically by beginning with an example you can also easily do by hand. Which implies you're doing this for some subject. Is that the case?
Great post! Note that Vapnick had a PhD in statistics. I'm not sure there are a lot of computer scientist that know the name Talagrand and I'm sure 0.01% of them can state by memory one result of talagrand :) can you ? I don't know the work of Valiant :)
I've added the tag self-study. Normally you are expected to show your ideas so far, not just ask us to do your homework for you.
How do you know it's normal? It doesn't look normal to me. It looks left-skewed.
+1. Re: small-medium-large, as a 1st pass, if you have no relevant knowledge or context whatsoever, these 't-shirt sizes' are OK, but in reality, what is a small or large effect will vary by discipline or topic. Moreover, just because an effect is 'large' doesn't necessarily mean it's practically important or theoretically meaningful.
It is still biased. If you take the second estimate, and consider N to be much, much larger than n, then the factor is almost 1 and you are left with the same estimate. It is a correction to account for the fact that in that setting not all data from the finite population are observed.
+1: I like it.  I assume you mean $Y_n$ is distributed…?
@ Shane♦ Thank you. As I have no experience with time-series analysis also, could you leave a few pointers of where should I start?
I changed my first answer :)
@ Michael Mayer  Thanks for your comment. Yes, it is the probability of having an adverse event at the time point where the follow up ends (for both censored and non-censored data). I wonder if we could calculate the probability for every single individual in the survival analysis? –
Actually, come to think of it: what do you mean by uniform distribution in this case? If e.g. L is 4 and d is 2 while n is also 2, the only solution is to take positions 1 and 3. In what way can this ever be 'distributed uniformly'?
@Glen_b I agree that significance does not measure strength of association, but only that there is an association. I plan to report odds ratio in part 1, and predictive accuracy (perhaps via cross-validation) in part 2, to indicate strength. Are there better means to report strength?
This happens e.g. if one rater is consistently providing slightly higher ratings than the other one.
@SideshowBob: I don't know if I can *formalize* the approach, it was a kind of *seat of the pants/back of the envelope* suggestion to help you put some numbers to your findings. I don't know your specific problem, but if each of your proposed models has a random element (e.g. normally distributed errors), then you can simulate from each of them using Monte Carlo and then compare the fits of each model under the data from each to see how often they fit each others data.
I edited out the reference to the answer above. Answers can change order and in fact the answer that was above is at this moment below, and that can change. So cross-refer to posters, not position.
(+1) Reminds me of an answer by [G Jay Kerns](http://stats.stackexchange.com/a/6134/1036) on this site to another question that ends up having a data distribution like the OP describes.
Centroids aren't very meaningful for sparse vectors IMHO.
@Calimo: Thanks - corrected.
"The intersection point will necessarily be to the right of 3(1−Φ(c−75))=1;". Sorry if it is trivial but would you mind explaining that a bit?
You should also quote the title attribute of the image: "Correlation doesn't imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing 'look over there'."
@user946850 - yes, though actually, if I were writing it myself, I would omit the phrase "using a simple multinomial model".
How did you derive a ratio when there is only one case? Wouldn't they be either 0 or undefined? I think in order to get a useful answer, you may need to explain the question a bit more. E.g. what is the design, **why** do you need to balance them, and what kind of ratio you were referring to.
Previous question was http://stats.stackexchange.com/questions/62535/anova-and-levene-test-problem
This seems very vague to me.  What kind of data, and what kind of analysis?  Also, this should be community wiki if it is subjective.
Hm... I _can't_ calculate it directly, since I don't know the property for the rest of the population, I just know that sample is a significant part of it. Look, if I had 15 entities, and data for only 14 of them, surely I wouldn't use sample standard deviation, right? :-)
I think it's considered to be an AI algorithm.
Generally not. Are you thinking about assuming that $\\prod_{i=1}^I{x_i}$ is negligible relative to $\\prod_{i=1}^I{y_i}$ ?
@cardinal: the latter!
Thanks for your answer. The two groups actually overlap and there is a strong relationship between age and balance (older individuals tended to be less stable than the younger ones). How do I adjust my data to account for the effect of age?
Midpoints are just one of several possible treatments. Depending on the shape of the distribution, midpoints would be too low in some cases and too high in others. So, using midpoints implies a need for some sensitivity analysis. You haven't told us the number of bins (or indeed the currency units!), but if your questionnaire is typical of many, there won't be more than about 10 bins. There's quantitative juice in your lemon, but it is hard to squeeze out.
I have some background in calculus and linear algebra and I understand the essence of probability and distributions over words etc… The hardest part is to really understand how I can use this in practice with the tools mentioned (JGibbLDA, WEKA/MEKA, etc…)
Yes, I forgot the log on the left hand side, or rather I added unnecessary log on the right hand side. Motivation for $p_i =1/(1+\\exp(f(x_i)))$s comes from the fact, that $f(x_i)$ is unbounded, but $p_i$ is constrained to interval $(0,1)$, so we need to transform $f(x_i)$ in order to succesfully model $p_i$. This transformation does exactly that. Look for more explanation in wikipedia under logit function.
Is X (the discrete variable) ordinal or not?
I hope they represent the probability of reaching state 0 before 1, and the probability of reaching state 1 before 0. Starting in state 5.
I'm not sure if it should be community wiki as I think that votes on good answers should count towards rep, and there's a chance you'll get better answers if such is the case.
John, I think this should stay here. You may have to wait some days before getting a response.
Yes, you are right. That is exactly what I meant.
Do you assume your time-point 3 measurements are MCAR? MAR? NMAR?
@Nick ... For sure ... Since Poisson variables are restricted by 0 this is so. Thanks ...
Glad to help. An additional precision, regarding terminology: you can actually find people calling this type of network "heterogenous" in the literature, but the term is not only used for multipartite graphs. Sometimes, it just means there are different types of nodes, without implying any constraint on the way they are connected. I've also seen it used to name multiplex graphs (graphs with different types of links). By the way, if you find any implementation, it'd be nice if you could post it here, it can be a useful thing to know. cheers!
@teucer, look at the coefficients, lag coefficient is 0.9, this is very large. This means that almost everything is explained by the lagged value. If the values of x1 and x2 are small they contribute little, since the corresponding coefficients for the them and the constant are small.
Can you present examples of items? The likely answer is that you calculate the alpha separately.
with R you do get a score matrix. in my case with 2 columns/components. I tried that and I get some weird scatter plot. I plot variable1 vs variable2*score.component1
(1) Stock value is usually a poor metric.  Changes in stock price relative to an appropriate index will almost always be better.  (2) Given that intervention analysis uses *known* dates of events and structural change needs to *estimate* events, and you know the history of bank mergers, doesn't that answer your question?
@NickStauner, it is essentially an exact duplicate, but in this case, the OP could hardly have been expected to have found that one. Questions like this will inevitably become one-offs.
Thanks, that book contains quite a lot of relevant results!
Do you have access to previous auction results for product X ?
@FrankHarrell: could I use the same command on SPSS (Unianova) specifying in the 'design' command the interactions between event and the variables? I want to be able to do special contrasts and I find it easier this way than using the standard regression command. Thank you.
I think it's now clear that McNemar's test answers your original question. Fisher's exact test is not relevant here.
+1: Very helpful, special extra kudos for the R. Thank you
As I understand, (from the paper of Petersen and Bernasconic) for a d-dimensional ball, one can generate the radius by raising a U(0,1) variate to the (1/d) power, and the last angle as a U(0,2$\\pi$) variate. The intermediate angles can be obtained as $C. asin(\\sqrt[k]{u})$, where ${C^{-1}}$ is $$\\frac{\\sqrt{\\pi} \\Gamma(\\frac{k}{2} + 0.5)}{\\Gamma(\\frac{k}{2} + 1)}$$.  To me this sounds rather simple. What I'm wondering is this: if I use a quasi random sequence for my uniforms, will I get the niceness in the ball as well?
Thank you. I saw that previously, but I'm looking more for a big picture view rather than the mathematics. I understand the math just fine, and based on that, my definitions seem to be more on point.
There are a couple of issues, the Poisson distribution is common for counts when things are fairly independent.  Instead of thinking about the table as having a fixed total and you are distributing the values between the cells of the table, think about just one cell of the table and you are waiting a fixed amount of time to see how many responses fall into that cell, this fits with the general idea of the Poisson.  For large means you can approximate a Poisson with a normal distribution, so the test statistic makes sense as a normal approximation to the Poisson, then convert to $\\chi^2$.
The "61% of votes counted" are not 61% of votes throughout the state. They are 61% of precincts, and the order in which they report is probably not random. For example, many polling places stayed open beyond the official time in urban areas where lines were extremely long, so rural areas would be more likely to be in the 61%.
thanks Mr. Stauner
Is there a name for a suggested model it could be used?
This seems like semantic nitpicking that intentionally avoids using what is clearly the intended interpretation of the question.
(1) [Search here](http://stats.stackexchange.com/search?q=%2Bcubic+%2Bspline) for information on splines.  (2) Consider using a [generalized linear model](http://cran.r-project.org/web/packages/pscl/vignettes/countreg.pdf) for these data.
I'm not to say I disagree with you, but I (a kind of) feel lack of clearness in your response. For example, you don't explain why transformation of a 6-point rating scale is `unlikely to help` and what you mean under "help" here. Also `skewed distributions from ordinal-level items` looks like farrago. Linear FA don't have ordinal input, it treats everything as interval.
Sandile, I have taken the liberty of making your answer super explicit by including the term your reference.
Here is a +1 for Macro
Maximum likelihood may be possible. However, ML estimators are not unbiased in general. You have some data that is called right-censored. Do you know anything about the distributional form of your time-to-event data? If so, you can use that. If not, you will need to use a nonparametric fit such as Kaplan-Meier.
And the reason a ttest on ranks is not giving the exact right pvalue is because ranked data is not normal?
I'm curious as to the kind of questionnaire you are using: 41 factors for a total of 142 items means you may have factors with very few items, as you pointed out. I would neither trust the % of explained variance, nor Kaiser's rule (which is known to overestimate the number of factors). Parallel analysis should be ok, providing your factors make sense. What kind of factors extraction are you using (ML, principal components, principal axis, etc.)? Please note that Ruscio and Roche's code is available as [R code](http://www.tcnj.edu/~ruscio/taxometrics.html#EFAwithCD).
@Glen_b The comment at http://stats.stackexchange.com/questions/67422/volume-of-the-95-confidence-ellipsoid/67429#comment130300_67429. The question starting that thread explains the chi-squared origin of these numbers and contains a couple of links.
But you already used that variable in creating the index, so you already know how good a predictor it is. It is as good a predictor as you made it.
I really like this question! I have not read of any example in the literature. I will answer something that might be of interest.
Is this it? http://anson.ucdavis.edu/~mueller/carglmbiometrikarevfinal.pdf
As said - I'd love it to be.
I don't think you want to take groups. You can look at the predicted value for the DV at any level of the IV. But only look at levels of the IV that actually exist.

If you want a number, you could compare the predicted value of the DV at the 25%tile and 75%tile of the IV; but, really, I think a graph is the best way to interpret.
@blubb I would consult a doctor that has no clue but 100% success rate with a confidence interval [98,100] :)
Record the target positions *and* the error - plotting one against the other may show something of interest.
OK, it appears that rules are different in different countries here. Sorry about that. The game I play has a small straight 1-2-3-4-5 and a large straight 2-3-4-5-6.
Using a 3-period weighted moving average does not appear to help me as I will still have some exceptional values which do not accurately reflect the original calculations.
I thought Deming in the MethComp package was TLS - what's the difference?
You appear to be expressing your p-values in an unusual way: normally, using a p-value of 0.95 means you expect to make false positive determinations in *95%* of a set of *independent* tests.  Do you perhaps mean to write "0.05" instead of "0.95"?
I can't quite figure out why looking over the code at first glance, but the pdf's and the cdf's are switched.
That comment makes a two-parameter gamma seem qualitatively as well as quantitatively inappropriate. See again the comment of @whuber on three-parameter relatives.
Perhaps you could post the name of the paper as its quite hard to tell from the problem at hand. One way they could have done this is to include three lags of $X_{2}$.
It is difficult to conceive of what a definitive or objective answer would be. This question invites subjective comments, anecdotal evidence, and discussion. The principal decision I had to make, as moderator, was whether to make it CW or close it.  (You will notice it has collected some close votes already and could still be closed by the community.)
This question should also be of interest: [do-all-interactions-terms-need-their-individual-terms...](http://stats.stackexchange.com/questions/27724/).
Could you explain what you mean by "control for"?  That term, as commonly understood, does not seem to make sense in this context.  Are you perhaps asking how to find the variance-covariance matrix of the terms in an estimated correlation matrix?
Did you read http://en.wikipedia.org/wiki/Cohen's_kappa, paragraph "Kappa maximum"? I expect that the ratio of observed kappa to its possible maximum in your table may be a measure of effect size.
thanks for your help and thoughts: I need to get the answer of COOLSerdash in this post: http://stats.stackexchange.com/questions/66861/standard-errors-of-hyperbolic-distribution-estimates-using-delta-method working. He uses the vcov function which does not work, I get an error message, therefore I need a substitute?!
It depends, what did you read specifically about $R^2$ that makes it a non-ideal indication of fit in your case?
@Luke That was not clear to me from your question. You might want to look at `lme4::lmer` or `nlme::lme` and their family argument. Substracting a bit more than the min avoids negative or zero values when taking the log.
My impression is that Johansson is not talking about comparing p-values from different experiments. In light of that & @Glen_b's comment, would you mind clarifying your post, Emil? It's fine to raise a related point ('I think J's wrong in context A, but it would have some merit in context B'), but it needs to be clear that that's what you are doing. If you *are* asking a question or commenting, please delete this post & make it a comment.
I find the illustration confusing because of the huge Lie Factor built into it: the successive levels of the "funnel" use different scales that change irregularly. Thus the widths of the bands are *not* determined by the absolute numbers in each--at least not in any easily understood or visualized way. So what are you asking: whether there are better ways to visualize such data or how to create this graphic in Python?
Thanks a lot for such a descriptive answer. Thanks to StackExchange, I think it's time I'm not ashamed of asking stupid questions because they trigger such beautiful answers.
I guess the `lm` object is the way to go! Thanks for a great summary... sometimes in the application I get too far from the theory.
I agree with @rolando2. One quick thought though, I am not sure it is sound to rely on a factor analysis for 20 items based on 70 respondents.
I edited the question with further details regarding the covariate. I hope it is more clear now. Please let me know if further info is required.
@Prometheus you wouldn't if you only care about the predicted *label*. In practice you often care about the confidence of your prediction, which is exactly why you need decision values. In your case, when you need to combine predictions from multiple models, you need to have a measure of *confidence* in each prediction.
+1 Great explanation and commentary.  Welcome to our site!.
@Glen_b My categories are "political categories": far-right, right, centre, left, far-left. 50-50 in categories 1 and 5 is more heterogeneous than 50-50 in categories 1 and 2, but is equally heterogeneous as a distribution with 20 in each category.
@user3509 TeX is enabled in comments as well as questions and replies: surround it with $\\$$, as in "$\\$$E\\{y_1,y_2\\}=\\int\\int s(x_1)s(x_2) p(x_1,x_2) dx_1 dx_2$\\$$" gives  $E\\{y_1,y_2\\}=\\int\\int s(x_1)s(x_2) p(x_1,x_2) dx_1 dx_2$
I am not sure what to compare? If you mean the training and testing set, the fraction of neutrals is 50 % (training) and 60 % (testing). When adding this feature in the second SVM, only 14 % of instances in both training and testing set have that feature above  (in other words, have that feature at all).
As I do not understand the question, as it stands at the moment, I can't tell which model is better. However, I do not completely agree with your first sentence. As @Henrik said, MANOVA was suggested as an alternative to RM ANOVA when the sphericity assumption is not met; it has its caveats as do Greenhouse and Geisser or Huynh-Feldt corrections. So, these two methods are not necessarily so different. For more information, I like to refer to [A Bluffer’s Guide to ... Sphericity](http://bit.ly/R8HWTi) (PDF) and [An introduction to sphericity](http://bit.ly/SyyRQw).
(+1) I like the first, and especially the last one (pretty much the same as the *Elements of Statistical Learning*, by Hastie and coll., but with other examples and a discussion of biplots and correspondence analysis).
An example is ridge regression, which is OLS with a bound on the sum of the squared coefficients. This will introduce bias into the model, but will reduce the variance of the coefficients, sometimes substantially. LASSO is another related method, but puts an L1 constraint on the size of the coefficients. It has the advantage of dropping coefficients. This is useful for p>>n situations Regularizing, in a way, means "shrinking" the model to avoid over-fitting (and to reduce coefficient variance), which usually improves the model's predictive performance.
If the documentation (or the package vignette) don’t tell you what procedure is used for the test or for the confidence interval, the only thing to do is to look at the functions code to find it...
You don't have an understandable or answerable question until you can describe, in sufficient detail, what you are doing and what you wish to accomplish. It's not even clear from the image you have posted what you mean by "trend": after all, in both cases there are increases in prices between the two times.
do you mean "bcp" function of the "bcp" library in R? Can I use this method with financial time series? Is it good for that?
+1 for describing an interesting (and amusing) teaching tool (and wrapper of the runif command).
Maybe I'm being dense, but surely there's more than one way to roll 50 ones and 50 twos. In fact, I would expect there to be 100 choose 50 ways to do that, which is about 10^29...
@chl I'd assert that it depends on what you want to show for a large data set. For example, transparency is very good at showing areas of high density, while still preserving outlying values, which things like binning and KDE sometimes struggle with.
Thanks. I think it's equivalent to what I am doing, but conceptualized better :)
+1 Great Question. I'll be keen to see the answers.
+1, great answer. I always wondered why the original formula is used for $R^2$ instead of square of correlation. For linear regression it is the same, but when applied to other contexts it is always confusing.
Even if the predicted click probability is exactly equal to the true parameter $p$, the Pearson correlation coefficient $r$ will not necessarily equal 1, so, while this method does have some merits, it can be hard to interpret / prone to misinterpretations.
So the difference is "repeated measure", which means what assumption?
Both are technical (regularity) conditions that make more sense in the context of the proofs of the theorems in which they are involved. Hence, my advice would be to study the proofs of the Lehmann-Scheffé Theorem, Bahadur's Theorem and Basu's Theorem.