tttjjj/elt_bench huggingface dataset:
My distribution seems to be a special case of the generalized Beta distribution. With parameters $p=2+1/a$ , $q=3$ and $b=1$, the remaining part should be equal, since both distributions have a volume of 1. So $|a| \\cdot(b^{ap} B(p,q))^{-1} = ((4a+1)^{-1} - 2(3a+1)^{-1} + (2a+1)^{-1})^{-1}$
Thanks, This is a very similar case, however our problem is finding the exact probability to each vector, and not classify it
@rocinante I believe the OP has to give further clarifications. He does not ask for "confidence X% that _the probability of_ obtaining Tails will be YY" - (which could point to a confidence interval around an estimated probability). He specifically writes "confidence X% that _we will obtain Tails_", and it is not clear what he means.
@Łukasz Can you say anything more about the parameters $n$, $\\alpha_i$, and $\\beta_i$?  It's possible to obtain exact expressions for $\\sum_j{X_j}$ and thereby approximate the expectations of the ratios, but for certain combinations of the parameters one could exploit Normal or saddlepoint approximations with less work.  I don't think there will be a universal approximation method, which is why additional restrictions would be welcome.
There is also [this post](http://stats.stackexchange.com/questions/56066/wald-test-in-regression-ols-and-glms-t-vs-z-distribution) which is quite similar to your question but deals with that in the framework of regression. Maybe you'll find some useful information there too.
The set to $25$ games means you might win by a score of $25-20$ which would be $45$ games.
No, sorry, you got it wrong. It's not that the posterior distribution over parameters does not exist or is improper. It's just that BP is a weak method which can handle very limited number of models only, because it attempts to compute marginals exactly. It can be possible for discrete or Gaussian models, but that's pretty much it. EP, on the other hand, approximates marginals and, therefore, is out of trouble. And in discrete and Gaussian models EP will work exactly as BP.
This set of questions is probably too broad to be answerable here. You might break them down into a series of questions that you ask as each is answered. In addition, I think some of this material is already well covered on the site.
Radek, I think the idea is this: suppose the price $Y(t)$ depends on time but only changes at discrete times $t_1,t_2,\\ldots$. We conceive of this as the manifestation of some unobserved underlying variable (the "real price") $Z(t)$ and we *hope* that between times $t_i$ and $t_{i+1}$ $Z(t)$ will always lie between $Y(t_i)$ and $Y(t_{i+1})$. In effect, then, we view the observed price at any time $t$ in this interval as being $Z(t)$ as *censored* both at the left and the right by $Y(t_i)$ and $Y(t_{i+1})$.  (I must emphasize "hope": this is the "risky statement" referred to.)
@user No, and I doubt there is one.  There's no magic software bullet here, by the way: almost all the work involves thinking through the issues and exploring specific trade-offs in a controlled fashion.
In the terms you've described it, it's very hard to understand what you're asking. If the $\\beta$ is a regression coefficient, then of course it can be over 1, although it's not clear since you use the term "beta weight". A more basic problem is that there seems to be some fundamental lack of understanding on your part about what model you're fitting that I think needs to be clarified before asking specific questions about the model output. Perhaps you can consult the SPSS documentation and modify your question accordingly.
Macro, as a rule of thumb in homoscedastic errors-in-variables regression, if the errors in the IVs are small compared to errors in the DV, you can safely ignore the former and resort to ordinary regression. That gives you a quick, simple way to triage the problem.
(Made some edits to the post): My question is how to make the redboxed formula differentiable in a way that is theoretically justified. I see your point about the non-appropriateness of the machine learning stuff.
I guess the important question is: is it baised. I say it is not, because all the test are independent, so one can really 'abuse' all the information of one test and use to make predictions about other tests. What is not allowed is to use the test's p-value information on itself! that is why the cross validation must be done.
A not for later readers -- statisticians might be more familiar with this kind of plot being called a *Tukey mean-difference* plot (equivalently, a Tukey-sum-difference plot -- the plots are identical, aside from the scale on one axis), since this was well known in statistics long before Bland and Altman popularized it in the medical literature.
I think that you may use chi-square test, if it is not exact (based on binomial distribution) but is asymptotic, as usual. The Chi-square distribution is continuous and is ok for fractional counts. (I your particular example, though, counts are too low for a chi-square test.)
Because this is [cross-posted](http://linguistics.stackexchange.com/questions/2399/dimensional-reduction-with-synonomy-and-polysemy) and is collecting answers on the linguistics site, let's keep it there.
@jokel I think that what you're suggesting is a special case of the "relaxed lasso", and it can work very well for some purposes.  I'm not sure you can trust the p-values you'd get from the procedure you've described, though, since your F statistic or t-statistic won't "know" about the variable selection step you did, and your Type-I error rate will be inflated. One way to think about this: what would the correct number of degrees of freedom be for an F statistic? The total number of variables in the LASSO regression? The number of variables in the secondary regression? Something in between?
yes, it is that simple
Failure to punctuate and make the question title proper case, - jk ;)
After trying different inputs, I came to the conclusion that lmp() seems to return the mean of the y values as the intercept. This can't be right, right?
This answer sounds highly introspective to me, and it offers kind of a recipe derived from observed physical properties (causes or consequences). Could you link the aforementioned facts to a precise mathematical model?
Can anyone figure out why link #2 is not working correctly?  Maybe there's a problem with using square brackets elsewhere in the text?
@SeanEaster, Thank you for reading my question. I may have posted this question prematurely due to frustrations. Currently, I am working hard to track down bugs and other possible issues. I'm considering removing "Adaptive Metropolis". How useful is "Adaptive Metropolis" and when do we use this function ?
But you can apply a Poisson-like model for non integers.  All you need to do is modify the likelihood function to use the log gamma function for the factorial.  Whether this is fully sensible and whether it is fully interpretable is another matter, but it quite possibly could be.  I agree with trying a negative binomial model where an additional scale parameter is estimated to relax the $E(Y) = Var(Y)$ constraint in Poisson.
You might find the discussion (look at *all* the answers) at http://stats.stackexchange.com/questions/7308/can-the-empirical-hessian-of-an-m-estimator-be-indefinite to be illuminating: it concerns a situation where multiple, strikingly-different optima of a nonlinear function were found.  An example function studied there is qualitatively similar to yours (it has no constant term and minimizes a sum of fourth powers instead of squares).
@Zach, @rolando2 Make answers!
ahh!, my initial idea of it was very vague but something along these lines. this example cleared it up nicely, thank you!
Your fitted model links its input with your observations. If you transformed the input before fitting the model, it will not make sense for untransformed inputs. So: yes, you do need to apply the same transformation to the test set inputs.
Actually this does not appear to be what I am looking for - is the Chow Test only used for looking for structural breaks in time series models? That is all I can seem to find references on. I am wanting to compare the time series of two separate groups, starting from the same time point.
+1 for the classic examples and mentioning collinearity
Do you mean the skewness and kurtosis of the dependent variable or of the *residuals*? If it's the former, your problem is complicated by the fact that those moments depend on the regression coefficients themselves and on moments of the IVs.
+1 - It might be good to distinguish between prediction intervals and confidence intervals, as it wasn't specified by the OP (although I agree most forecasting software will likely give prediction intervals by default).
@StéphaneLaurent Both $k$s are *known*, not estimated. That's given in the question.
I see many people have this problem- Does anyone understand the response to this persons query?https://stat.ethz.ch/pipermail/r-help/2006-April/103836.html
The problem you discuss (representing 4d points as 2d points) is called **dimensionality reduction**.  Google or read wikipedia articles about the topic.
"Visibly different" doesn't necessarily mean much. Consider not curves, but just means. Two fitted means can be "visibly different" but not *significantly* different, if the sample sizes aren't large enough to tell the difference from random variation. If you plot standard errors for the curves you may be able to see why one pair are different while the other pair are not.
Maybe I wasn't clear, I edited it.  I am working with hypothesis testing, are you talking about regression models?  Thanks for your answer.
At present, this is more of a comment than an answer, would you mind expanding it a little? Why might the OP want to take a look at the paper? Can you provide a summary of the relevant information in the paper?
Isn't this what @Felix said in his reply?
Great answer, thanks very much!
thanks for you answer, but the problem is, that boot.se leads to very different results every time I run the procedure?
Although homework exercises are welcome here, we ask that you explain what you have tried and that you formulate *specific* questions concerning where you might need help.  You are invited to edit this question so it meets these guidelines.  For more information, please see our [faq].
Some algorithms have extra constraints on the distances. For k-means, the *mean* must be consistent to the distance function. This mostly holds for L_p norms. I read somewhere it only works for Euclidean distance, but I'd assume it works for any distance where the mean is optimal.
I don't know about Matlab, as I don't use it. There are lots of crappy implementations around that are tied to particular distance functions. ELKI is pretty interesting here, see the tutorial on their web page on adding custom distance functions. They can be used almost everywhere then.
yes you are right. I have already found the function `rfe()` in `caret`. But is this the best way to handle it? Are there other functions more recommended? Does anybody have further insights?
I think you'll get more useful help if you take a few steps back, and ask us what scientific question  you are trying to answer. Why are you looking for the smallest normalized SD?
@user862: I figured it might just be a simple situation like this. But, I thought the background of the question might be interesting enough to others that I'd put together a quick exposition. That way the question and answer are more self-contained, especially in case that link ever goes down.
Here's a paper with better formatting: [paper](http://arxiv.org/pdf/bayes-an/9504001.pdf).
@renrenthehamster I have added relevant material. I don't believe there is a standard recipe for all cases, to find these series.
@Harlan, there are considerable complications evaluating the model as initially proposed. The terms within the parenthesis would be considered formative measurement models, and would need further information to identify those parameters anyway. See [Bollen & Bauldry (2011)](http://dx.doi.org/10.1037/a0024448).
Thank you @Fer. I think this will help me in time and I will likely officially accept the answer soon. May I be more direct about my second question? *When* may I make a Bayesian type of interpretation? It sounds like you've answered something to the effect of (though probably much more nuanced and presently beyond me) 'when the parameters are sampled from the posterior distribution'. Is that essentially right? Or is there other machinery (so to speak) that must be in place? How crucial is MCMC here? Thank you again.
Thanks, whuber! Could you recommend some books/tutorials that have some good cover on antithetic methods?
If the minimum service time is very small compared to the mean, the results may still be somewhat useful but in many cases a better approximation to service time might be a shifted-exponential. As to when that (or indeed any other) approximation may be useful, that depends on what aspects of the system you're investigating and how much approximation is acceptable
+1 for the paragraph about how the last layer does logistic regression on top of the hidden layer's feature selection. That is a nice way to think about NNs.
Do you have reason to believe that the matrix has low rank?
I just updated the code so it will report valid/missing n, and then omits the missing values for the functions which were broken by missing values.
Are you asking about doing boxplots by hand or about how some software does this? As @Glen_b said, different software will do different things.
Is it possible to include all PCs?
Added a third paragraph to describe how I handle such problems in my research. That is, I assume your problem is just like one I've had, in that the labels are nominal...?
See, for example, [this comment](http://stats.stackexchange.com/questions/31036/what-is-the-difference-between-a-consistent-estimator-and-an-unbiased-estimator#comment60184_31038) and the related discussion.
+1 Although you might be more explicit in how this answers the OP's question (it's not so obvious).
Could you clarify how you propose to use the "entire sample's mean" in your t-test? To what *independent* set of data would that mean be compared?
Do you know what a population is (e.g. did you read say the first sentence of the link I gave)? What's the population in that situation?
+1: Having looked at this problem before, I find this answer fascinating.
+1 for taking the time to reproduce the problem and lay it out very clearly.
Hint: by "randomly" you probably mean that all such sequences have the same probability.  So: how many possible sequences of four bits are there?
@Peter Flom is correct. Essentially for any sized effect you can find statistical significance with a large sample size. John Myles White did an interesting write up on this in the context of social science: http://www.johnmyleswhite.com/notebook/2012/07/17/criticism-5-of-nhst-p-values-measure-effort-not-truth/
@nico, I don't think this would have been off-topic / better suited for SO even before the changes. The request for code was specified as optional, & `data-visualization` is part of our mandate.
Notice that the proportional odds assumption looks as 1 versus more , 2 versus more , etc. So it does _not_ look at 1 versus 2, but 1 versus (2, 3, 4,  ...)
Thanks for your answer. I thought about this models, you are talking about, too. But I got the impression, that these models do not really respect the "nestedness" of my data. I mean in this data I got kind of two repeated measure variables nested in each other: **Each** dose is used to **each** rat on the first level. On the second level I got the three measurements for each dose.
Take a look at the chapter 2. Section 2.6 - which is partially available on books google - might in particular help you. http://books.google.se/books?id=BQYR6js0CC8C&pg=PA70&source=gbs_toc_r&cad=4#v=onepage&q&f=false (Ref: Model Selection and Multi-Model Inference by Kenneth P. Burnham and David R. Anderson. Springer Verlag)
I don't understand why I should use the product's ID in the model. What I want to know is if products of category A are better than products of category B using the grade and it's popularity (views).

I'm not sure either if a regression would be the best way to compare both categories ..
(ctd) ...  An alternative argument might start with the beta and use say a series expansion around it which values are then ranked and then try to argue that the beta-binomial is an approximation to the resulting process (such an argument would be harder to make the details work for, obviously). Or there might be other ways to argue it.
The term *IID* stands for '*independent and identically distributed*'. Absence of autocorrelation doesn't say *anything* about the second part, but only about a particular kind of dependence (i.e. it only addresses the first letter in 'IID'). Consequently, the values of those sample autocorrelations alone cannot allow you to make statements the plausibility of the IID assumption. You should probably edit to ask more specifically about the independence assumption.
+1 for the coin toss paper--really clear explanations.
(+1) The Gamma function is defined for all *complex* numbers except the non-positive integers, so this solution certainly works for positive non-integral reals.  However--a little nit-pick--the calculations done here are meaningless until one changes $|x^k|$ to $|x|^k$ or else makes some assumptions about branches of the complex logarithm. :-)
@kwak I'm not sure why changing the title was necessary...the "homework" tag should be sufficient.
Well, the sign of $r$ tells you the direction of the relationship, while $R^2$ does not.
I believe I outlined an example of bootstrap bias reduction in my preceding comment, Michael, which is a form of improvement in an estimate.  If you would like to pose this as a question, I will endeavor to provide a more detailed reply.  In the meantime, you may consult the first chapter of Hall's book or Chapter 10 of the Efron & Tibshirani book (*An Introduction to the Bootstrap*, Chapman & Hall, 1993).
@Abigail First you need to have a very clear idea what hypotheses you want to test. Are you trying to test whether `Social Support` moderate the effect of `Stress` on `Depression`, `Anxiety`, and `QoL`? Or trying to build a measurement model for `Anxiety` including `Worry` as an indicator amongst others? Or trying to build a latent variable `Wellbeing` that incorporates `Depression`, `Anxiety`, and `QoL` and see how `Stress` affects `Wellbeing` with `Social Support` as a moderator? You need to think about how things are measured, and put together carefully before start drawing your models.
@Baltimark I'm not too sure myself tbh, I've just been basing the level of my answers on how the questions are asked, so e.g. if someone asks a question showing that they have some knowledge of the field, then you can say things that won't scare them away. This question seemed basic so I assume a very simple reply is appropriate. I'm sure http://meta.stats.stackexchange.com/ has some more guidance.
Exactly one, or at least one? (I'd think that the other one is actually more interesting)
How do you define X' ? Existence of X' would imply that X' is square, which, unless I am mistaken is not the case.
The libsvm website <http://www.csie.ntu.edu.tw/~cjlin/libsvm/> includes a Java applet where you can also change parameters.
Thanks for your answer, Nick. I found the example especially helpful.
thank you very much. You are right, that with "apply" the script could be optimized. I just used my script as a minimal-example in order to get the message through... Thx a lot, your answer is exactly what I was looking for!!
+1 As an aside: the characteristic function is the one that's more closely related to the Fourier transform (in that case, again, there's the small issue of a minus sign) - the c.f. is $E(e^{itX})$, while - up to multiplicative constants - the usual Fourier transform would be $E(e^{-itX})$. These connections prove to be quite useful at times, such as finding lists of useful properties of Fourier or Laplace transforms that usually carry directly over, or being able to look up extensive tables of Fourier or Laplace transforms when finding MGFs or cfs.
Isn't it great to be able to verify theoretical results in an artificial environment (simulation), and not, say, having first to build a bridge that will collapse in order to realize that somewhere in the calculations a minus sign was taken as a plus sign? Thanks for the verification!
Whenever there's a case where people seem determined to insist on some kind of test - and thereby a probabilistic sampling argument - it usually turns out to be because they want their inference to apply *outside* the populations at hand, in some way. If that's not the case, then the sampling we'd normally do is in order to make inferences about the unknown population quantities of interest but if those quantities are actually known, there's nothing to do, since you can immediately see their values.
(+1) Very good work, thx.
+1 Lots of nice points.  The calculations add quantitative insights to the previous replies.
Actually, the variable we are looking at here is a gamma variable from which I have substracted the mean, and divided by the standard deviation. The reason why I need to identify the PDF is that this would allow me to identify outliers (using QQ plot).
Thank you jbowman and Michael Chernick. Now I get it...
I've added some references.  Thanks for the question which gave me a chance to include an excerpt of my favorite description (Welsh's) of the inference problem!
What's the equivalent of what you are looking for, if the measures are continuous. A measure of composite reliability might be what you're after.
Do you have the same number of subjects in different groups (particular age + particular gender), or substantially different?
I think this is probably a statistical issue rather than a software problem, since I am able to get a solution with this model on slightly different data in SPSS.
Here, the main model is concerned with cars; the secondary one, with dealers. I'm glad to have seen your reference on double hurdle models, but don't those models only fit when the two hurdles are to be "jumped over" by the same units of analysis?  E.g., when the same set of individuals are facing two decisions.
1 - (rank/size) gives you same as excel percentilerank
:Zach OK got it ! You, at this time, are not concerned with tests of sufficiency or necessity of your coefficients.
As for a general tutorial, I always like to refer to [A visual guide to item response theory](http://www.metheval.uni-jena.de/irt/VisualIRT.pdf) (it does not cover the MCMC approach).
Hi Maarten, when you say numerical problems do you refer to convergence or such? If the regression converged and appears to do well on a test set - do you conclude the situation is not a problem?
PCA doesn't invert the correlation matrix, so it doesn't worry about the matrix being positive definite. Factor analysis does, so it won't work.   Your analysis is not (necessarily) flawed. People doing Q-sort analyses do PCA with more variables than cases as a matter of course.
Does the presence or absence of a given point help in classifying the data?
Try to migrate this question to stats.SE, you'll get better answers there. To migrate, click "flag" to attract moderator attention and ask for migration.
Possibly. What software are you using? Do you need to understand something about this, or do you just need a pointer or sample code?
This isn't really any different from lots of cases where the data are described numerically in the text, but people like to present the information visually as well.
+1 to @StephanKolassa Splines are good. Another possibility here, mentioned in the question, is polynomial terms. The curve in the question looks like it would be well fit by a cubic. One advantage of polynomial terms is that they can be easier to interpret.
This is indeed something I wondered rather often: What is the best color gradient to use ?
+1 The abstract of the paper linked is interesting.
Actually it's the alternate hypothesis in both cases