tttjjj/elt_bench huggingface dataset:
As the error message says, you're getting this error because there are negative values in your response vector $y$. When the Box-Cox procedure determines which transformation to use, it uses the [geometric mean](http://en.wikipedia.org/wiki/Geometric_mean) $(y_1\\cdot y_2\\cdots y_n)^{1/n}$ in the computation. The geometric mean is only defined when all $y_i$ are positive, as taking roots of negative numbers may lead to [imaginary/complex numbers](http://en.wikipedia.org/wiki/Imaginary_number). Therefore all $y_i$ must be positive in order to use Box-Cox.
$||x-y||^2 = ||x||^2 + ||y||^2 - 2\\langle x, y \\rangle$, if $x$ and $y$ are orthogonal, then $||x-y||^2 = ||x||^2 + ||y||^2$. Therefore, if I had to assign $y$ to a particular cluster mean $x$, I just have to chose the mean with the smallest norm. Since this is independent of $y$, the behavior of 90 points getting assigned to 1 cluster in the first iteration is expected. Now, however, after the mean is recomputed, this probably should not continue to happen. So, you probably should check what happens to the recomputed mean of the cluster to which the 90 points have been assigned.
Eric, for some reason your site is very poorly rated with Web of Trust. Apparently for malware? 
http://www.mywot.com/en/scorecard/predictiveanalyticsworld.com?utm_source=addon&utm_content=rw-viewsc
@PeterFlom thank you for your attention.
I have your solution! I'd recommend polytomous latent class analysis, in which the outcome is a set of factors that are assumed to group in one or more latent classes. Membership in these classes is predicted based on a multinomial logistic regression. See `?poLCA` in R for more information on fitting this model.
Thanks. I will refer to your research work.
Please clarify the question: what is theta? The inverse hyperbolic sine function is $asinh(z)=\\log(z+\\sqrt{z^2+1})$ - no theta in sight.
Welcome to this site, JPC. Please, don't cross-post but rather ask a moderator (by flagging your post) to migrate your question. The [duplicate on SO](http://stackoverflow.com/q/14738037/420055) is unlikely to be well received as it stands, but see [this discussion](http://meta.stats.stackexchange.com/q/1335/930) on our Meta. I agree with @gung that this question could well be on-topic here if emphasis was put on the statistical aspect of model evaluation rather than programming.
Are you saying that _all_ your observed responses come from bots? If so, all I can think of to help you is using Bayesian statistics: get a uniform prior for the probability (or whatever is a typical noninformative prior for probabilities), apply the data, and this will give you the posterior distribution for the probability. From this you can gather a credible interval. I don't know of a way in frequentist statistics to support a case that is not in your data (without cheating)...
or [How to compute precision/recall for multiclass-multilabel classification?](http://stats.stackexchange.com/questions/21551/how-to-compute-precision-recall-for-multiclass-multilabel-classification)
@Aly i've also added what i believe is a necessary factor (n/(n-1)**2)...
(-1) It is strange that your answer proposes $R^2$, but fails to relate it to the residuals and then devotes itself to explaining why it shouldn't be used!  (This ground has been covered in the thread at http://stats.stackexchange.com/questions/13314/r-squared-useful-or-dangerous.)
@glassy $z$ has (asymptotically) unit variance *by construction*: the difference $\\hat{p_1}-\\hat{p_1}$ has been standardized by dividing it by its estimated variance.
I believe this is incorrect. Please see my answer for the explanation why.
I find two questions which, appropriately interpreted, appear to be duplicates (and therefore already provide answers): http://stats.stackexchange.com/search?q=valuation+rank.  Are these adequate?  If not, please help us understand what is special about your situation.
+1 Finally somebody gets it right... ;-)  The statement to be proved *is* correct; it's just not tight.  Incidentally, you may also pick your units of measurement to make $\\sigma^2=1$, which further simplifies the calculation, reducing it to about two lines.
@whuber:  Wow, nice diagram!!  (Thanks for adding it!)
Was the answer below of any help?
I must say I have to disagree about R being simpler to learn than SAS. It may be because I learned SAS and SPSS first, but I think SAS, SPSS(PASW now), and Stata are all easier languages to pick up than R. It is a subjective argument though.
+1 It's a big gun, but a very flexible one.
(B)In the explanation following (5.1),there are terms "distances" mentioned -r_l &r_u.If u may kindly have a look at the paper found in this link http://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CC0QFjAA&url=http%3A%2F%2Frepository.tudelft.nl%2Fassets%2Fuuid%3Abd5339a0-24e5-4362-b378-adc7c60cac99%2Faps_schouten_1994.pdf&ei=Ao-bUvrHGJH8oATm-oKYDw&usg=AFQjCNH2zOq5JJw-G8xJCo_qi8JSiD1YiA&sig2=XYPBmhxJIgk0RtigrN8AlA&bvm=bv.57155469,d.cGU .
Your premise is false - a CI for an OR that includes 1 doesn't imply "no difference". It implies a failure to detect a difference. That may simply be lack of sample size for the size of the difference you have,
A correction to the above comment: The $L$ vector should sum to $0$, so if you use $L=(3/4,-1/4,-1/4,-1/4)$ then $L\\beta=A-(A+B+C+D)/4$ which is group A versus the average of the other four groups.
The equation might be erroneously including a summation sign. If the observation is $y$, then the likelihood of $p(y|t_i, k_i)$, the joint probability is the likelihood times the prior probability $p(t_i,k_i)$. Thus, $p(y) = \\sum_j p(y|t_j, k_j)p(t_j,k_j)$ is the probability of the observation and is the sum above. The posterior probability is $$p(t_i|y) = \\frac{p(y|t_i, k_i)p(t_i,k_i)}{p(y)}$$ and is thus proportional (with proportionality factor $[p(y)^{-1}$) to the _joint probability_ $p(y|t_i,k_i)p(t_i,k_i)$. It is _not proportional to, but rather inversely proportional_ to the sum $p(y)$.
**Bounty is over**. Thanks to all contributors and the excellent ideas! This bounty generated a bunch of interesting approaches. Several answers contain a lot of brain work, and sometimes it's a pity that bounties cannot be split up. Finally, I decided to award the bounty to @whuber, as his solution was the algorithm that seemed to me the most comprehensive concerning the features it captures, and as it is easy to implement. I also appreciate that it was applied to my concrete examples. Most impressive was its ability to assign numbers in the exact order of my "intuitive ranking". Thanks, F
Is y =ax$^2$ with a>0 the onlt way to get a perfect inverted U?  I don't think so.  If the inverted U corresponds to a different function then there will not be an a that gives a perfect fit to y=ax$^2$.  Then it would be possible for some other model to fit better.
If you have one response (dependent variable) Y your question is more simply posed as being about multiple regression. Please read the tags for multiple regression and multivariate regression to see why.
Thank you for the tips on writing a custom export and the useful functions.
I've recently used `lqmm` too, and it does exactly what the OP wants. +1
There is no such a menu in SPSS `Analyse> generalized linear model> repeated measures`. Are you speaking of general linear model?
Could you give the context in which you heard the term "**test error**"? Because there *is* something called 'test error' but I'm not quite sure it's what you're looking for... (it arises in the context of having a **test set** and a **training set**--does any of that sound familiar?)
@whuber: I think it must be quite intuitive, but I was unable to prove it. Could you please expand your comment into an answer? Specifically, the term *mixture* was new to me. I looked it up in Wikipedia and understood the concept, but it's not immediate for me how $f$ is a mixture of identical distributions, and how this proves the theorem.
@Alexis Using the probability integral transform, Donald Knuth derives their (common) distribution on p. 57 and exercise 17 of TAoCP Volume 2. I quote: $$\\Pr\\left(D_n^{+}\\le \\frac{t}{\\sqrt{n}}\\right)=\\frac{t}{n^n}\\sum_{0\\le k\\le t}\\binom{n}{k}(k-t)^k(t+n-k)^{n-k-1}.$$  It is asymptotically Normal for large $n$.
@NickCox I mean if you don't mind please add answer to this thread question so that I can mark as accepted. I don't need an answer for the question in the comment. Sorry if I'm troubling you.
@henry I would be really happy if you can give a look at my new question here: http://stats.stackexchange.com/questions/15202/create-a-dataset-with-condition-probability-2-2-symptoms. It is a generalization of this question, with 2 symptoms.
maybe partial regression?
@JMS, Why would the matrix still be positive semidefinite necessarily? I don't believe your statement is true.
Welcome to our site, Martino!
Nice. I had to look up what a confusion matrix is. Let me clarify on what I have understood here. I'll make the matrix with my output centroids and the actual values of centroids [calculated in some way or the other] and find some measure of similarity between each centroid and the other. In the example you have given, if **1**,**2** and **3** are output centroids of my code and **A**,**B** and **C** are the true centroids, **1** corresponds to **C**, **2** corresponds to **A** and **3** corresponds to **B** right? Then maybe I can use 1-distance between centroids as the values in the matrix?
Any model for the way the test depends on temperature that doesn't use results from the test run at different temperatures would appear to be pure guesswork.
+1 Great illustrations. They make one think: on a computer there will be limits to the size of numbers that can be represented and usually they will be symmetric. In that case we actually have a truncated distribution. By the same argument, it *does* have a mean and the mean is zero!  I am moved to ask: to what extent should one be concerned about a behavior that truly happens at infinity when working on practical problems that necessarily use finite numbers? I'm not fishing for an answer--I can think of plenty of rejoinders--but just want to put that idea out there for general consideration.
@nth what exactly do you denote by (x,y)?
Twitter is a particularly difficult dataset for topic modeling not only due to the small size of the 'documents', but also due to the type of text. People tend to use various texting shorthands which makes identifying co-occurrences even more  difficult.
a similar question: http://stats.stackexchange.com/questions/28139/why-squaring-r-gives-explained-variance?rq=1
What will the end result of estimation procedure will be? An equation with a time varying coefficient on $Y_{t-1}$? If yes, this time-varying coefficient will have to be expressed as a _function_ of something so that it can change as time passes - a function of the index $t$ or of its own past. How the estimation algorithm will produce one or the other?
I mean independence *between calculations*. If, for instance, you are testing the correlations between random variables $X$ and $Y$ and $X$ and $Z$, then those tests are not independent because they share data. It is worthwhile giving some thought to the underlying statistical model of random variation in the values because that is a good indicator of how to proceed with testing.
Nice answer. Another thing that could be used for comparison might be large music festivals, and there's usually something localish that might suffice..
There are some R packages for this goal such as PtProcess or ETAS. I am also working on this topic and have some problems. I appreciate if you could give me your email address to discuss this issue more.
I think we'll need a formal definition of "inactivity" or "rest". It's OK if you state it in terms of what the fish does rather than in terms of what the plots look like (we can translate later), but it has to be formal. If a formal definition is exactly what you need help with, here are some things to consider: 1. I don't think there is any standard definition of "inactivity in time series" that you could just take and use; 2. it could be useful to know more context about your experiments to come up with a definition;
How does this differ from [your previous question](http://stats.stackexchange.com/questions/28996/principal-component-analysis-in-r)?
You can run all the models you want.  Merely calculating a model isn't a test.  A test is a logical procedure, not a mathematical one.  The fact that much software by default returns p-values for parameter estimates as if you had done some sort of test doesn't mean one was.
Why would a z-score help? Do you have any kind of guess as to which "beads" are heavier than others? Would the distribution of the number of beads have any relationship to the mass of the beads? (If not, then there is no better solution than just summing them).
Do you know the value of $\\rho$ or not?  By "large sample" do you mean (as suggested in the question) that $n\\to\\infty$ or are you taking large iid samples of $(X_{12}, \\ldots, X_{(n-1)n})$ for a fixed $n$?
I think that if any of those 800 students were to read this question, they might be offended. How well did they perform? How much learning was accomplished? That is what a grade should reflect, not some arbitrary statistical summary of their position in a group. IMHO this question should be recast in terms of teaching objectives, not statistical procedure, such as "what is a good way to convert raw scores to grades in a way that respects student accomplishments and advances the learning objectives of this class?" Statistics can help, but blind statistics--like standardization--will not.
Re "Of course it isn't."  At a guess, a lamb has on the order of $10^5$ hairs on each side.  If there are an even number of such hairs and they are distributed randomly with equal chances on both sides and the sides are clearly delineated, then the chance that both numbers are *exactly* equal is 0.178%.  In a large flock of several hundred, you should *expect* to see such a perfectly balanced lamb born at least once each decade (assuming an even number of hairs occurs about 50% of the time).  Or: just about *every* old sheep farmer has had such a lamb!
I thought that the problem Alex was having was precisely on how to do interpolation, so I answered that, assuming he would not have any issues with the rest. I'll update the answer with some insights on how to do the rest of the computation, then!
For many problems, resampling is computationally infeasible. For example, if you simply want to calculate CIs for a large 3D matrix, or for a long time series.
I might redirect you to a long and amusing discussion (especially comments) about PCA http://stats.stackexchange.com/q/62677/3277. In this respect, it doesn't matter that your question is about factor analysis, not PCA: the "dilemma" is the same.
This is correct, but not efficient.  It is a linear transformation, so you would precalculate `a` and `b` constants, and then just apply `newvalue = a * value + b`.  `a = (max'-min')/(max-min)` and `b = max - a * max`
You could do that: no rules against it. I'd recommend against calling these constructs factors when they are principal components. It's not clear that PCs would be better used in Spearman correlations when they are based on Pearson correlations.
I think mean, median, lower and upper quartiles jointly would describe the data better. You can find some other descriptive statistics [here](http://stats.stackexchange.com/questions/28431/what-are-good-data-visualization-techniques-to-compare-distributions/28442#28442).
@StasK - A few years ago, I asked a question similar to what's in your second paragraph above.   Here's the link: http://stats.stackexchange.com/questions/7036/number-of-trials-required-from-a-binomial-distribution-to-get-the-desired-odds
And the solution is?
@StéphaneLaurent This problem strikes me as being interesting in many ways. What is the best estimate for the mean and standard deviation and how do you construct best confidence intervals. Are the mles good estimates? Do bootstrap confidence intervals work in this case? I see that [X$_($$_n$$_)$ - X$_b$]/3 is a consistent estimate for sigma where X$_b$ denotes the sample mean. Also [X$_b$-X$_($$_1$$_)$]/3 is consistent for sigma and the average of the two is unbiased based on symmetry considerations.
This is a good start, in the right direction, with two caveats: (a) $h(t)$ is not equal to the expectation shown (check the sign in the exponent) and (b) this appears to restrict attention to the case where $x$ and $y$ are scalars and not vectors. I've upvoted in the meantime, because the exposition is nice and clean and I'm sure you'll quickly plug these small gaps. :-)
Welcome to the site, @MohSahx! It would be clearer if you can edit your formulae in `latex`, especially to revise the symbols like ▒.
This is my best answer based on the research I've done. I don't feel confident enough to accept it though -- I would really appreciate if someone who has a deep grasp could sanity-check this!
How do you get ties if $k$ is odd? What do you mean by "only using variables with >2 levels"?
This post isn't about designing toxins, its about discovering which compounds are toxic, so that we can stop using them.
Can you define "easy-to-use" ;-)? Are you willing to do some programming (like in Python or R), or do you only want something w/ a point & click interface?
Perhaps I am being picky, but statistical tests do not **prove** causality, one way or another, but they can show strong evidence if conducted properly.  Second, social scientists often show causality through regressions, for example in an IV regression.
The answers depend on why the values are missing and on the pattern of missingness. (Are all three variables entirely missing from the dataset? Are they missing only from some cases? If they are, are all three always missing in those cases or are only some of them missing?) Answers also depend on the values of non-missing variables in the cases with missing values. They also depend on your method of multiple imputation. Finally, whether any of this matters depends on what you will be using the OLS results for. Please edit your question to clarify these points.
Now that I think of it, when merging the categories, I would implicitly conclude that feeling 'unsafe' and 'very unsafe' are similar feelings, but feeling 'very safe' and 'safe' are meaningfully different feelings. Something you can not conclude just from the sparseness. This is probably in favour of dropping it instead of merging.
Thanks for the pointers B_Dev. It's not exactly MAUP problem in this case since my areas are not mutually exclusive. And for more pointers about regionalization methods see my other question: http://gis.stackexchange.com/questions/3189/what-tools-for-regionalization-redistricting-can-you-recommend
I've edited the question to try to make a little clearer, check that the meaning is still accurate; and consider if there are any more details you can add to answer the queries from the other commenters.
@loganecolss, if I understand you correctly, I think that's right. We don't usually measure bias & variance from our data, though; those are *theoretical* properties of the estimator.
Sounds OK to me.
I just noticed that this is the first example from the Matplotlib web page: http://matplotlib.org/
Correction: an earlier example starting on the bottom of page 88 describes this approach to posterior sampling for bivariate posterior distributions in detail.
Are the residuals conditional on the predicted random effects or unconditional and in the simulations are the predicted random effects constant or varying. Remember try to get a sense of this for the default simulation methods in LME4 and not being able to (but the project was cancelled before I sorted it out).
Comparisons to the "real world" are next to impossible except in contrived situations: after all, we compute the CIs precisely because we do *not* know the values they target. However, there are a huge number of simulation studies of coverage of CIs: it is practically unimaginable that there exists any CI in existence that has not been studied in that way.  Having said that, there are some notable exceptions, such as a study of the [CIs for the speed of light provided by physicists](http://books.google.com/books?id=ajd1V305PgQC&pg=PA58#v=onepage&f=false).
Thank you for the reply. FWIW, indeed I am using R's auto.arima with xreg.
I've used a negative binomial regression model due to the overdispersion. I'm also going to look at using a hurdle model to address the large number of zeros in the dependent variable. Many thanks for your suggestions.
I was referring to the model residual variance which is different from the measurement variance. If you expect a different residual variance and different coefficients for the regression. the first sensor should not come into play.
If I am interpreting your comment correctly, by "observable periodicity" you mean that if $n$ is the number of points in your sample, then the period $T$ of a periodic signal satisfies $T \\ll n$. Yes?
Thanks for making that clear. The  KS test should be fine, but if you know your data will actually be be normally distributed, its not as efficient as it could be at picking differences. It's not clear how you could know this though (approximately normal, perhaps).
Good to know that. Though "non-standard", it is realistic in practice.
Can you point me where I can find documentation on how to do the Fisher test using SPSS? Thanks
(1) Use subject matter knowledge before looking at the dataset; (2) Use redundancy analysis/data reduction blinded to Y; (3) Use a method that adequately penalizes for the huge multiple comparison problem caused by doing feature selection (see elsewhere on this page).
BTW, a [set of lecture notes](http://www.mast.queensu.ca/~stat455/lecturenotes/set5.pdf) takes a similar approach. On p. 238 it mistakenly asserts that an expression like $\\exp{t\\mathbb{Q}}$ is "meaningless." On the contrary: this expression is meaningful algebraically (as a convergent sum), analytically (as a solution to a system of differential equations), and especially geometrically (as the equation of geodesics in the neighborhood of the origin of a Lie group).
http://stats.stackexchange.com/questions/46019/why-squared-residuals-instead-of-absolute-residuals-in-ols-estimation is essentially a duplicate.
I think we're talking about two different things here. My method calculates Mahalanobis distance, which I've verified against a few other formulas. My formula has also now been independently verified by `Matteo Fasiolo` and (I assume) `whuber` in this thread. Yours is different. I'd be interested in understanding what you are calculating, but it is clearly different from the Mahalanobis distance as typically defined.
Without any further restrictions, the answer is: any class. That's because you could have a neuron for each function in your function class.
Your linked output says the mean difference for Vraag 6 is 5.33, not zero. What is your basis for saying they have the same mean?
Can you add some details about this TMA to better place it in ML context? The background about the purpose is not too relevant.
(+1) "I would make that decision of which variables to include based on substantive knowledge." this is good advice that people often decline in favor of some automatic method.
Aha! Thanks for clarifying. I had made a typo here initially (apologies).  In my simulations, my formula for pooled variance for the sex:disease interaction were within ~15% of MSE; is this good enough?  I just want to answer the question: is the main disease severity effect common to both genders significantly bigger than the male-female difference in disease severity?  This will help in interpreting our results when there are sex biases. Many thanks @JakeWestfall !
Thank you very much for your answer. Now I understand that my distribution is very skew and normal approximation is not valid. However in my opinion proportions 87/88 and 48/60 look very different. Can I use another test to prove difference between proportions?
Hi, Prof.Rob, the code works perfect. Thanks so much for your solution. I really appreciate your help!