tttjjj/elt_bench huggingface dataset:
Having admitted you do not yet understand the math, this might be helpful for you http://stats.stackexchange.com/questions/41794/bayesian-updating-for-a-discrete-rating-value/43048#43048 and for practicle use you might wish to look at this http://stats.stackexchange.com/questions/43471/examples-of-bayesian-and-frequentist-approach-giving-different-answers/43498#43498
I was not sure if thats sensible, as I thought (think?) proportion does not equal probability? Is this the common approach?
Thanks, Huber. I was double counting! I hope it's OK now.
(+1) Very nice, interesting, informative answer.
Good find! Even if the results are off, it's nice to see the claim explicitly stated somewhere.
@DilipSarwate: The words "random" and "independent" are words I found in textbooks I am reading at the time. Unfortunately they are not being explained in the context of this subject. Thank you for your explanation.
VAR is unbiased for any distribution with a finite variance, not just for the normal distribution.
Dear Kumara: For clarify, it would help to better explain your notation, specifically the meaning of $g_{i,j}$. Also, I think your expression is missing a constant factor of $1/2$ in front of the first term of the right-hand side of the display equation. Note that what Kullback himself called *divergence* (using the notation $J(\\cdot,\\cdot)$) is the symmetrized version of what is know called the KL divergence, i.e., $J(p,q) = D(p \\| q) + D(q \\| p)$. The KL divergence was denoted $I(\\cdot,\\cdot)$ in Kullback's writings. This explains the factor of $1/2$ as well. Cheers.
This question can be solved using a result in http://stats.stackexchange.com/questions/77110, whose answer gives the density of the maximum of $n$ iid Normal variables.  More discussion of this circumstance also appears at http://stats.stackexchange.com/questions/44139.
I don't think it has a name, but, sure, it shows up on occasion. Here is one case: Let $X$ be a nonnegative random variable in $L_2$. Then, $\\mathbb P(X > 0) \\geq (\\mathbb EX)^2 / \\mathbb EX^2$. This is a straightforward consequence of Cauchy-Schwarz.
If Naive Bayes was an option then logistic regression is a better one, with weaker requirements and no pesky conditional independence assumption.
Wow! (I have learnt so much on this site). Thanks @Glougloubarbaki. An example will be great - a simple one at that!
Some more details about your particular study might help gathering precise and helpful answers.
+1 Thanks for calling attention to the truncated/censored distinction.  I added some comments to elaborate on that.
Dear @Mysterious, did you attempted to try it out yourself before asking? (Just asking)
This [answer](http://math.stackexchange.com/a/442916/86468) might be useful.
+1 This package also has non-Bayesian solutions if you want to do some classical variography yourself without specifying priors.
According to an [elementary, rigorous account of random variables](http://stats.stackexchange.com/questions/50/what-is-meant-by-a-random-variable/54894#54894), $X$ is a way of writing numbers on tickets in a box. A function $f$ is a rule to erase the value $X(\\omega)$ and replace it by $f(X(\\omega))$ on each ticket $\\omega$. That should make it plain that (a) functions *can* be applied to random variables, (b) the sense in which they are applied, and (c) there actually is a technical condition they must satisfy: namely, for every $x$, $\\{\\omega|f(X(\\omega))\\le x\\}$ must be measurable.
Each row is the total number of usages of the same structure by two different people talking to each other. So, for instance, if I were counting passive sentences, then the number in column A would be the number of passive sentences used by speaker A, and the number in column B the number of passive sentences used by speaker B. So each row is a different conversation. Each conversation has exactly two participants.
Hint: Use the Factorization Theorem...and don't forget the indicator function in the density when you do.
There's a link given [here](http://stats.stackexchange.com/questions/68077/are-categorical-variables-standardized-differently-in-penalized-regression) that may be helpful.
I think that Spiegelhalter's book is a classic and a good point to start (Markov Chain Monte Carlo in Practice)
I've added the tag [tag:sequential-analysis] as it sounds like you want to do sort of [interim analysis](http://en.wikipedia.org/wiki/Interim_analysis). I may well be wrong, so feel free to revert to RM ANOVA if that applies.
Thanks for the comment. And for the useful complement approach to verify that indeed my computation is correct.
As stated, the hypotheses apply to the *observations* - in effect you observed the population quantities about which you wish to make a conclusion, in which case no statistics is required - they plainly differ (in effect the p-value is exactly 0). If that  doesn't satisfy your needs, before even attempting inference, you need first to consider these issues: What is the population of interest?  What is the model? (Without that, what does the question even mean?)
@fabee: Thanks for the valuable comment,
I think it's a great article. It highlights a lot of the incentive problems with graduate school these days. Another interesting article is [The Rise of the Post-Doc as Principal Investigator](http://papers.ssrn.com/sol3/papers.cfm?abstract_id=891041). I'm not sure why you think the article is poorly researched, can you provide some more specific criticisms? I agree that it's important to avoid confirmation bias. But as a reader, I don't much care if the writer is in confirmation mode as long as the sources are good. As far as I'm concerned, there should be more bitter rants about academia.
I would write "the last two references".
Have you noted that $h(t)$ is the derivative of $- \\log S(t)$ ?
I mean that you can call the entire vector as an estimator
If you're looking for something more advanced, you probably want a specific area of stochastic processes or something. A good read is, [Stochastic Differential Equations](http://www.amazon.com/Stochastic-Differential-Equations-Introduction-Applications/dp/3540047581/ref=sr_1_1?ie=UTF8&qid=1355243349&sr=8-1&keywords=stochastic+differential+equations), another one in [Brownian Motion](http://www.amazon.com/Brownian-Stochastic-Calculus-Graduate-Mathematics/dp/0387976558/ref=pd_bxgy_b_text_y).
Yes, and I would say it is unlikely that _any_ cluster analysis - not K-means only - will give the solution D (unless maybe when inproperly tuned).
For simplicity, lets assume that there are only two classes, Like and Dislike. Like being 1 and Dislike being 0. Have changed the problem statement to reflect this.
@Andy edited in the granger-causality tag, which shows that there are lots of questions on the topic on this site. Have you looked at any of them? Are you sure you have a new question? Have you studied basic texts and internet sources?
The [documentation](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.beta.html) calls the last two "location" and "scale" parameters.  Thus the fourth is the scale parameter.  Location and scale have standard statistical meanings.  One interpretation in this context is given explicitly in the [NIST handbook](http://www.itl.nist.gov/div898/handbook/eda/section3/eda366h.htm).
I believe that most typical algorithms can be adapted to this setting. The problem is that it is not usually done/well known, since the problem is not that relevant. Anyway you are right, generally it is not possible (e.g. I would not know how to do it in the case of an SVM).
Without any loss of generality, you may assume that the means of the $x$ and $y$ are zero (center the variables).
Yes, this formula looks reasonable but somehow the result I get is always below the non-recursive moving mean. And the difference looks constant. Perhaps I have a bug in my code, the difference would hardly be constant otherwise.
Fair enough, I have some programming work ahead to implement and test this though. Could you please point to a source that explain all this ? Thank you !!
See, for example, [here](http://courses.engr.illinois.edu/ece313/fall00/ppt/Lecture39.pdf) for the bivariate normal density formula.  The barebones equation for the likelihood ratio is the ratio of the two bivariate densities under the two hypotheses. The covariances are zero according to the FYI statement and so $X_1$ and $X_2$ are independent normal random variables while the means are there subtracting off from x_1 and x_2 in the formula I gave. I took the log of the likelihood ratio to get the formula.
"ML" is not an algorithm but a criterion. MLE doesn't find a minimum, it's the value of (a monotonic function) of the thing you try to optimize. Minimizing $J$ probably\\* *is* ML (in that its argmin will correspond to ML. \\*(I haven't done more than taken a quick glance at the paper, but you should have at least mentioned you were doing some form of nearest-neighbor modelling here, and better still you should write more details of your model in your question.)
Thanks @Glen_b. Am I confusing the 'smoothing' in time series data to remove noise with the 'smoothing' of frequency polygon to obtain a density function? Are they related or two completely different concepts with same names? My real goal is to understand kernel density estimation and eventually convolution which I am using to remove noise from traffic flow data.
However you want... small enough to make your simulation stable at different starting values, but large enough that it doesn't take all year.
@user900978 - Please be clearer about exactly which difference you want to test for significance.  Apparently you don't want to test the difference between X and Y, as I think Glen_b (and I) initially thought.  If you are testing the first 10 values of Y vs. the last 4, then it is important whether you are doing so based on some prior hypothesis or merely because, after collecting the data, that is where you have noticed the greatest difference.  If the latter, any statistical test will be clouded by the "data snooping" that you have done.
If an estimator is biased, increasing the sample size will not necessarily make it any less biased. It depends on what kind of bias there is.
On another note, are you sure you need ordinal logistic regression? Age strikes me as something that could be sufficiently normal that OLS could be used. Even if it's not exactly normal, the Central Limit Theorem may cover for you if your N is high enough.
Continuity corrections may apply pretty much any time you approximate discrete cdfs with continuous ones; as such the "general" results tend not to be very general. For example, even with something as standard as the normal approximation to the binomial, if $p\\neq 0.5$, it's not necessarily an improvement; correction in the extreme tails can be worse than not correcting. For "smooth" unimodal situations, they often help a lot for ranges of probabilities near the mode, but the situation as you move further away seems to very much depend on circumstances. I'm unaware of any truly general results
@metro No problem.  It took some research to figure out where it best belonged.  Usually a single posting is best; if the question doesn't fit your first choice site, it will get migrated to a better site :-).
Thanks 4 your answers:)
Do you have a reference?  Please note that the so-called "spherical" variogram model has *nothing* to do with spatial correlation on the sphere!  Also, where you write "spatial autocorrelation function," did you really mean "spherical autocorrelation function"?  The former is a very general question whereas the latter is more consistent with the rest of your question.
No. The idea here is to account for time series "complications" . Forecasting is not in play for you but it could be !
Ok and formally how do you prove it? I understand the intuition but searching for a formal proof. Furthermore why the fdr power is good when the number of samples is increased and the feet gets worse when increasing the number of samples? Thank you!
Thanks!  This is exactly the sort of thing I was looking for.
Thanks, I'll try that. The relative survival methods I've looked at so far require that the background distribution must be almost free of the cause of death in the sample, but I'll keep looking.
I don't get what you mean by 'autocorrelation of residuals across the cross-section'. Can you clarify?
So cutoffs are never a good idea??
That's exactly what I mean. In this weird case, the absence of any explanatory variables is a signal, and given the lack of data, it's useful.

I could just add an if statement that is "if there is no explanatory variable, AAA is the projected outcome, otherwise use the model" but it would be ideal if I could have a single model object.

Maybe I could just add a factor variable "has_data" to the test set, and set it to level 1 if there is data and level 2 if there is not?
@gung I expect OP means [this kind of phantom](http://en.wikipedia.org/wiki/Imaging_phantom), but OP should confirm.
I simulated the clusters to be drawn from a Gaussian mixture, so I think my assumption is valid. The goal here is not to reduce complexity or come up with a decision criterion for choosing k, but to compare how well k clusters models the data when k is actually incorrect. Some incorrect choices might model the data better than others, and I'm trying to quantify this degree of misfit with some calculation (like KL divergence, but easier to implement for Gaussian mixtures).
I cannot tell whether it's right because I find it meaningless: although you call this formula "Bayesian" you haven't provided a probability model, you haven't defined a "team", and you haven't (yet) formulated a clear question.
This question was crossposted on the Mathematics site.
No one can direct me to a resource?
The trace is invariant under cyclic permutations.
@gung the wikipedia page you linked to has entries for both "riffle shuffle" and the "overhand shuffle" the OP was talking about. Its good to read the links you link to :)
This brings to mind: http://xkcd.com/882/
Heh, I had a similar idea before I realized you'd already been down that route. To a first approximation, it's close to quadratic. Given the y axis is a proportion/percentage, some things to check are "is it a logit?" (nope, indeed because 100 is on-scale, clearly not, though the other points fit pretty well); "is it arcsin-sqrt?" (nope; and I realize it doesn't make sense to use it here, but someone might be confused enough to do that).
I must confess your actual situation is unclear to me then. Could you make the actual situation explicit in your question please?
@guest Thanks. I read and answered the question too quickly; updated.
If you obtain a satisfactory answer yourself, we would be grateful if you would share it here: it's ok to answer your own question.
May I asked why you offered to delete all this reply with your suggested edit, @nina?
If I may humorously disagree with your comment: Excel can be used effectively for statistical work, if you're a statistical wizard who can prove from first-principles what should be done, then reverse-engineer Excel's methods to determine what it's actually doing. When you're that good, most any tool will do. Though I'd also point out that in this thread, 100% of the wizards involved in this discussion so far do not have access to the latest version of Excel, so are unlikely to actually use it.
I was bit sloppy in my explanation. Of course it was square root of Chi-square distributed random variable divided by its degrees of freedom.
Here's a function to make all the prplots for a given lm fit, using ggplot2: https://gist.github.com/3041812.
(+1) I always like to see primary-source material. Nice addition.
Tangent: the fact that differential entropy can be negative, creates a puzzle that I have posted in this question: http://math.stackexchange.com/questions/454078/non-zero-conditional-differential-entropy-between-a-random-variable-and-a-function-of-it If you have something to contribute...
(+1) Nice review, thanks for sharing.
"each interview was conducted in the same order, but the order was randomized once as part of the experiment design." So, first a number of stimuli were defined, then they were randomized, and then the interviews were performed? Do I understand this correctly?
@Josh You added the tag `ridge-regression`. Does it mean that you have a large number of predictors and/or that you suspect that it has something to do with the observed difference in prediction accuracy?
"$L_1$" really is the $L_1$ norm.  The median is any point at which the sum of the $L_1$ norms between the points and the median is minimal, just as the mean is any point at which the sum of the $L_2$ norms and the mean ("centroid") is minimized.
And you gave a really nice answer then, OP should carefully read it (+1).
Why criticise the chi square test when it is clearly following "intuition" for both cases used in the question?  Additionally, using the "extreme case" is not called for here without first qualifying when it won't occur - can you provide an example of the "bad" behaviour using numbers of similar magnitude to the data given in the question?
+1 Johnson & Kotz has been a great resource for me for decades, but the price is appalling.  It would be nice to find an affordable version.
+1.  The posterior probability that Mike, with distribution $B(\\alpha,\\beta)$, is better than John, with distribution $B(\\gamma,\\delta)$, is proportional to a generalized hypergeometric distribution.  This can be computed either as a polynomial or as a rapidly converging power series, giving far greater computational accuracy than any simulation.
I don't think that this is on-topic, but I will wait for others to vote to close it.
Thankyou for your answer. I was hoping that there was some way I could "extrapolate" the per-entry type given some samples, however your suggestion will also work!
Ok thanks! I will try that
You should be able to figure it out from the covariance matrix. But another way would be to regress a set of variables on one of the covariates.  When you get an R square =1 you will have the linear combination.
@Jeromy @chl @rm999 @Thomas I thank you for your help. I hope now to use a more precise language.

What I have is a question like the following:

"Q: For what services do you contract company X:

* Cleaning;
* Transportation of waste;
* Surveillance;
etc.

And I would like to test which are the most important services for which company X is contracted.
I would agree, and add that Matlab has a horrible syntax (if you can call it that) and costs a pretty penny.
Hi Jeremy. Thank you for your response. So, you mean that go to "Transform"->"Compute variable" and either create summing items variables or mean items variables? Which is more common to use? To me, mean items seem more make sense.Last step seems after creating these new variables, treating them like observed variables...I can see that why item parceling is so controversial..Several articles have mentioned that I should not use item parceling if the EFA result indicated that factor is not unidimensional..can I ask why it should be unidimensional?
@mark999: Could be, but it doesn't negate the importance of performing some sort of power analysis *before* doing the experiment, which I understand to be Matt's point.
@jerad woo no... do you have any links to good resources?
Let me clarify with more details. In my scenario, a failure is: complete component failure, sub-component failure and yes, a sub-component failure will indicate a complete component failure. What I was worried about was the notion of **repairs**. When a component is repaired and put back into operation again, wouldn't it be incorrect to ignore the fact that this is an old device that was repaired (which means, its lifetime was affected by the repair)?
@asdir What you tried in Stata is not clear, but see the help on `permute` for an example of use with `ranksum` or `search somersd` for more general approaches.
Not quite sure if I am in the right direction, however, maybe this helps. For your first question, would **entropy** help? Assuming you have a normal distribution under which the heights must be (law of large samples helps you with this assumption), you can estimate the probability of a particular height and from your sample, you could estimate the probability from your real data and then compute the [entropy](http://en.wikipedia.org/wiki/Entropy_(information_theory))?
@NickCox: I thought natural numbers are ordered. What would many users think of Poisson regression as then?
I have the same problem as the OP, and I have to say, I didn't found out the detailed derivations in GPML book. And I was confused further after I read the lecture notes as I posted in the above comment. In that notes, the posterior $p(u|S)$ given out by Rasmussen is different from the one in OP's equation $(5)$. I did the derivation myself, and I agree on the posterior $p(u|S)$ being the same as equation $(5)$, I even think Rasmussen's lecture notes might be wrong at this point. If I miss something or make some mistake, please correct me. And I hope that you could elaborate on the derivation.
The moment estimators can give nonsensical (e.g. negative) answers.
@what [This post](http://stats.stackexchange.com/questions/18988/do-null-and-alternative-hypotheses-have-be-to-exhaustive-or-not) deals exactly with your question. To back my claim up further, [Young and Smith](http://www.amazon.com/Essentials-Statistical-Inference-Probabilistic-Mathematics/dp/0521548667/ref=sr_1_1?ie=UTF8&qid=1373536792&sr=8-1&keywords=young+smith+essentials+inference) write (page 65) that possibly, *but not necessarily,* $\\Omega_{0}$ and $\\Omega_{1}$ satisfy $\\Omega_{0} \\cup \\Omega_{1} = \\Omega$.
@Milktrader: From the data above, for a 24 day delta, you have 268 Ups, 98 Downs, and 676 Nulls (1042 Total Events). Assuming no structural changes, each trading day in the FUTURE, before the trading day, you face the probabilities of 268/1042 Ups, 98/1042 Downs. The remaining 676/1042 Nulls will show up more often. All of this deals with the future. After the Close, you'll know if it's a "Preferred Day", but again this is after the close (not the future). Your "Preferred Probabilities" only apply after-the-fact (in the past). I added a diagram in my answer above to help explain.
(+1) That sounds good. Thanks for sharing your experience with IRT modeling and FA. Apart from the graphics functionalities, the conditional approach in eRm is more in line with the initial thinking of theta by Rasch (as a fixed parameter).
@user603: No, I don't think I am confusing or even conflating these. I said "make most sense"; I didn't say and didn't mean that rejection rules make absolutely no sense in other situations.
Since the normal is continuous, specifying 'discrete' already means it can't be normal. It is *not* a general property of the Poisson that $P(X>0) = P(X>1|X>0)$ (you may be partially confusing it with the geometric?)
@AlexWoolford did it help you?
@StéphaneLaurent Personally, I am not interested in experimental contexts. I'm interested in contexts in observational studies with numeric predictors. I'm interested in the differences between assuming such data are the only predictor values of interest, versus acknowledging the  that the predictor values are drawn from a distribution.
A regression doesn't add new information, but it could perhaps make more effective use of the information you have.  However, I have answered your question on the basis that each estimated point simply reflects the sample proportion of people who would buy the good at that price.
Hi. I still have not resolved this issue. The values still change with re-leveling, and I am now noticing that this occurs on both old (pre-1.0)  and new (post 1.0) versions of lme4. One thing I have noticed is that in the old versions of lme4, adding random slopes for Condition is a significant improvement to the fit, but in new versions it is not significant. Whenever I omit Random slopes, the problem with re-leveling goes away. it was suggested above that I am over-fitting my models. But I am not really sure how to verify that my models are not over-fit. Is anova not reliable?