tttjjj/elt_bench huggingface dataset:
Good beginning! I fixed a subtraction error in the final formula. As a result, you might want to revisit your speculations in the last paragraph. To get started, note that the equation is linear if and only if all the quadratic coefficients are zero, which is equivalent to $a=1$, $b=1$, and $c=0$. (There is another special case where the equation represents a *pair* of lines.) Whether the locus is elliptical or hyperbolic depends on the sign of the determinant $(a-1)(b-1)-c^2$; both signs are possible.
Hmm, the formulation "whether a set of measures falls within a certain range around the population values(s)" describes very well what I want to relate to, I should perhaps take this into my question. But my problem is also a paedagogical: I hoped I could derive the rationale of the t-test from that of the chi-square test, which I'm to introduce before and with which I make a great deal about comparing an empirical (discrete) distribution (frequencies of a for instance dichotome variable) with an expected one.
@user51964  Sorry for the slow reply.  I have been vacationing.  No, you would not minimize that quantity.  Usually, you would run OLS first.  This gives unbiased, consistent, asymptotically normal estimators even in the presence of heteroskedasticity.  Nowadays, most economists at least would stop there.  We would just use heteroskedasticity-robust (or Huberized, or Huber-White) standard errors along with the OLS estimates.  You could, however, use the OLS estimates to construct weights and then use feasible generalized least squares.
I have had this checked by a couple of people, external to this forum and they have confirmed that it is correct.

Also I have transferred the equations to a downloadable spreadsheet located here:[link] (http://www.genroe.com/resources/free-downloads/net-promoter-margin-of-error-calculator)
You could string them together so you get `if (octet1 == "206") {label <- "OK"}` and `if (octet123 == "193.64.11")  {label <- "SPAM"}`
What kind of individual effects do you want tot take into account? If the answer is few or none, you can use a standard gmm function. If it's more than that you're likely going to have to program parts of it yourself.
I'm a bit skeptical when I hear "stock prices" and "accuracy" in the same sentence. Stock price prediction is the deep, dark morass of statistics, and I'm not sure that anything in the market can be considered linear.
I like your idea of a 'best fitting' estimation strategy, but what is being fitted best?  Usually the issue is about the fit of the strategy to the state of the researcher's confidence in her knowledge of the various parametric assumptions she'd like to make, and the level of anxiety she has about them being quite wrong.  The econometrics literature is quite explicit about this as a motivation ML vs GMM vs Robust, etc.
Surely it's way too late for an answer (you posted this 1.5 yrs ago), but there is not /repeated line in this case. What you put in the /repeated line should have gone in the /random line.
The "$\\Sigma$" is called [summation notation.](http://en.wikipedia.org/wiki/Summation#Capital-sigma_notation) It asks you to plug in first the value $i=1$, then $i=2$, etc., into *all* occurrences of its argument. When you do that for the $w_i$ you will not have any more questions.
**Hint**: What is the definition of *symmetric*?
I doubt any regression models book has that formula for the coefficient, but $(X^\\prime X)^{-1}(X^\\prime X)Y=Y$, which is the original data, is not correct. The second formula is the correct Ordinary Least Squares estimator of $b$ when $(X^\\prime X)^{-1}$ exists.
Is this sampling with replacement? Or without replacement?
Is $t$ a constant? I'd interpret the notation here so, but @NeilG's comment at emcor's answer considers a random $t$.
(Updated to reflect corrected code.)  A simulation with nIterations=10^7 yields 0.647 23 (that one took 40 minutes over lunch :-).  The first 3.5 digits of this result should be correct, strongly indicating the Poisson approximation (0.645 9) is actually *more* accurate than the Binomial (0.639 9).
@mbq: awesome! thanks for that.
It depends on the software you use, but generally [dummy coding](http://bit.ly/Ote3a6) is used, whereby the intercept includes the mean of the reference category and all other coefficients for your categorical variable reflect mean deviation from that value. If you have other predictors in your model, replace 'mean' by 'adjusted mean'. Here is an overview of coding systems with R: [R Library: Contrast Coding Systems for categorical variables](http://bit.ly/LFkFBg).
+1 For emphasizing the value of learning from simulations.
"Ideal" here means something other than asked for in the question.  The question seeks to stabilize the variances, whereas the Matlab solution seeks to make them as close to normally distributed as possible.
You can even apply PCA when $p$ is *infinite*. For example, your observations might be periodic signals and you could choose to treat them as functions (rather than in terms of some discretized version of those functions). Nevertheless, the set of all real linear combinations of those functions is a finite dimensional vector space endowed with a Euclidean metric (the $L^2$ distance): it makes perfect sense to seek principal directions within this "point cloud."
*Data* doesn't have a p-value. A p-value is associated with the test statistic in a particular *hypothesis test*. So start by working out which particular hypothesis and alternative you might be dealing with and which hypothesis test goes with it. It looks like a one-sample test of means might be required, but it's not at *all* clear what the hypothesized mean might be. Did you leave information out?
Oh, man, that's a mess. I will try to arrange it better.
@Procrastinator: I have updated the question.
First you say the "only possible logically internally consistent" answer is a uniform distribution and then you proceed to propose an alternative.  That sounds illogical and inconsistent to me :-).
Are you basing your assessment of distribution on the *raw* dependent variable or on its *residuals*?  Only the latter is relevant.
What is "SSPS", and how do you read this pea-soup of text?
A useful discussion by Jeanette Mumford appears at http://mumford.bol.ucla.edu/lin_cont_illus.pdf .
@Kyle Thanks so much for the grammar and syntax correction.
Could you indicate which book/online material offers such distinction?
@dmcdougall I am trying to remove the dependence.
If you're choosing those based on having seen the data, you can't calculate the distribution of the test statistic given the data. If those are chosen *a priori*, there's no problem. I'd suggest separating the issue of testing uniformity and the inference relating to your choice of endpoints into two stages. ... "*What should I do then with all these Ui*" -- you choose and then perform a goodness of fit test on them.
Maybe a count-weighted average of that values would give a reasonable daily values for that estimates.
Can you give more context? I'm as confused as you are from the isolated statement you pasted.
I think the idea behind sex as a DV was to predict "sex of observed carnivore" from location of observation.
@cardinal, could you indicate what you edited? I am just curious. I am happy to give a detail description if required.
Have you tried looking at the outliers tag?
In november 2013 the [third edition](http://www.wiley.com/WileyCDA/WileyTitle/productCd-0470387378.html) of Hollander & Wolfe appeared and a corresponding GNU R package [NSM3](http://cran.r-project.org/web/packages/NSM3/index.html) is available.
CONTD For me, this is the "have your cake and eat it too" issue of the hybrid approach. For example, an N-P approach without power-test calculations should be unthinkable, but all the time we see test posed in the N-P framework, but no mention about power calculations.
I'm confused what your question is. See the other questions with the same [tag:kolmogorov-smirnov] tag for plenty of examples and discussion on how to interpret the test, e.g. [this example](http://stats.stackexchange.com/a/105095/1036) is pretty straightforward and provides an example in R.
Just my 2¢: [RColorBrewer](http://cran.r-project.org/web/packages/RColorBrewer/index.html) or [colorspace](http://cran.r-project.org/web/packages/colorspace/index.html) offer far better options to handle diverging palettes of colors.
What model? What link function? You have not posted any model.
Great question, and one that seems to be ignored. Would be great if someone has references. I frequently see manuscripts based on complex models and <2k observations: http://www.ncbi.nlm.nih.gov/pubmed/23364879stacked models
+1 The question is a very important one.
Please note, not to be confused with testing the test for heteroskedasticity using the F-distribution. I'm talking about restricting a number of variables in the model under robust standard errors.
I'm not sure I understand, @user164846.  A straight line has no inflection points.
Because $X$ and $Y$ are positive a.s., the statement $X\\gt Y$ is algebraically equivalent to $(X/1)/(Y/10)\\gt 10.$  Does that help?
@RobHyndman, this link is very useful. It says “if I have a fore­cast­ing prob­lem where I want to use covari­ates, I tend to use regres­sion with ARMA errors.” (Why don’t you write that in an answer?) I have read that double exponential smoothing is somwhat equivalent to ARIMA (0,2,2). However, that does not take into account the panel character of my data.
I believe Peter Flom is referring to getting to know your data and how it might be generated and what kinds of limits might make sense. For example, if you have a variable that is a person's age, you know it can't be negative and reasonably won't be greater than, say, 120. Automatically throwing out everything greater than 3 SD away from the mean could well throw out some 90-year-olds who snuck into your dataset, yet also keep some people with an age of -4.
+1 in anticipation :-)
Here is a question you might be interested in: http://dsp.stackexchange.com/questions/3393/extracting-audio-track-to-determine-the-song-beat
Is this homework? If so please tag as such
You have included random effects for ID which is unnecessary because every data point has a unique ID (i.e., get rid of '(1|ID)' term). Run as just fixed effects and also try running without area as a covariate and compare AIC/BIC of models. Also I'm not sure you are incorporating make correctly, you probably want to use it as a factor variable but it is treating it as continuous. That could just be me reading it wrong though.
Largest/maximum spacing statistics are used in different statistical fields. An asymptotic result is e.g. http://www.jstor.org/discover/10.2307/2946462?uid=3737760&uid=2129&uid=2&uid=70&uid=4&sid=21103293232591
Thanks! I have learnt so much from your detailed responses.
I'm also looking for change-point analysis in Python. Did you find anything useful (e.g. using RPy?).
Thank you DL. The link was very useful!
@kype: Why not use your first formula for $P(A^c\\mid B^c)$? Then $P(A^c\\mid B^c)=1-P(A\\mid B^c)$ which you can compute.
This doesn't explain why you couldn't just take the *absolute value* of the difference. That seems conceptually simpler to most stats 101 students, & it would "take into account both its distance from the mean and its (normally speaking) rareness of occurrence".
This question appears to be off-topic because it is about interpreting the results of a statistical model. It is not a questions about programming since the code seems to be running fine. As already suggested, this is likely a better fit for [stats.se]
Thanks! replicate works perfectly.
Thank you, your solution adresses my question very well. Categorization has been performed, because percentiles are a common reference in the field. The only thing still bothering me though, is performing chi-square in a small sample, where I would expect to observe <5 observations at one of the variable's levels.
See http://stats.stackexchange.com/questions/14002/whats-the-difference-between-principal-components-analysis-and-multidimensional
Cool! Cool! Cool!
Thanks for the comment! As for your first point: that's definitely true. It's just that $\\mu_1=\\mu_2,\\ldots,\\mu_N$ is a stronger hypothesis than the one I would like to test. I don't care whether they're all the same, just that $\\mu_1$ is different than each individually.

For your second point, that would definitely work, but then I would end up with $N-1$ separate p-values, right? Is there a way to, in a sense, "combine" them?
@JohnMoeller: "difference of proportions" would be a search term, independent or dependent we don't know yet. If it's paired: McNemar's test. I'm guessing that t-test means rather small sample size, so possibly normal approximation is not a good idea. I'd go for [Statistical Methods for Rates and Proportions](http://onlinelibrary.wiley.com/book/10.1002/0471445428) to look up details.
I dropped the subscript $n$ in the final formulas, so $\\Pr(i=2)$ is $p_{n2},$ for example.  The matrices have dimensions $b+1$ by $b+1$.
By a model, I mean things like a generalized linear model (such as Gaussian linear model, a logistic model, ...), a generalized mixed linear model. By a model which is bigger, I mean that GEE can also be applied to models which are not generalized linear models.
Have you searched for similar questions? This has been discussed quite extensively, viz. right-hand bar with "Related" questions. What do you use for classification?
+1 Your "Alternatively" section is _exactly_ the same as the "SVD" used in netflix, with the number of dimensions collapsed to 1.
I have encountered another post that suggests a similar approach - take the interaction to be significant and use your noodle (rather than extensive further testing) to interpret it.

http://stats.stackexchange.com/questions/18873/graphing-versus-further-testing-interactions
Or any regression text, for that matter; it's not specific to mixed models.
Maybe you can write the equations, up the point you have solved?
The two together don't tell you anything more than the second one would alone!  The main effects are uninteresting and misleading when there is interaction present.  The second model tells you all you need to know.  Describe your variables more (what are they, what are their ranges like, are they both numeric or are they factors?) and someone might help you think about what that interaction means/looks like.
I find 848 posts (so far) related to interpreting regression coefficients.  A highly voted one that covers all aspects of your question is at http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output.
There's no magical pattern finding algorithm. There's not even a clear definition of what a pattern is. There are statistical tests for various hypotheses that you may be able to perform on your data, but choosing one is a statistical question, and not a programming question. You might want to seek advice from [stats.se]. Of if you know what test you want to perform, edit your question to focus on the specific programming question.
@whuber Yes. Perhaps even using some "unknown/coded text book example" which could be used as a backdrop.
You can use ordinal logistic regression instead of dichotomizing the scale.
+1 thanks for the suggestions!
[citation needed]. In other words, intriguing answer, although it doesn't jive (at least) with a lot of ML literature.
I edited the question to make it more precise. Can you check whether I have changed the meaning accidentically ?
The mean change can be found by subtracting the final mean from the baseline. But if this table you show is all the information you have, then you cannot calculate the standard deviation for the changes from baseline just from that. Have a look at [this page](http://handbook.cochrane.org/chapter_16/16_1_3_2_imputing_standard_deviations_for_changes_from_baseline.htm), it explains possible ways to impute the missing standard deviations for changes from baseline.
Is there a trend in both $P_i$ and $T_i$? Differencing the series will detrend those series and if that was what was causing the correlation, there goes your correlation.
Didn't you already ask a very similar question?
How did you deem them as "not statistically significant"? Did you bootstrap your model for example? Or do they seem as being a couple of scale of magnitude smaller than the residuals errors? Just plain old good AIC? AICc maybe? What exactly? Please provide additional information to your variable selection procedure.
@PeterFlom, I agree with what you say, but I am curious from your penultimate statement: what effect size would you suggest paying attention to and what parameter(s) would you compute a confidence interval for?  I would also be curious how a Bayesian would approach the comparison of 2 distributions without apriori information about what distribution or family of distributions would be used (I have heard of mixtures of Dirichlet distributions or Polya trees, but have not been able to wrap my head around those yet).
Yes, it makes sense that you really only care about the precision of your forecast. I am accustomed to the intervals returned by predict.lm. Thanks for taking time to explain.
Will [OVERALS](http://stats.stackexchange.com/a/31297/3277) - linear canonical analysis which optimally scales (monotonically transforms) variables to maximize canonical correlations - be to your liking?
It is no accident that $\\exp(0.2243)\\approx 1.25$. You got exactly what you asked for. :-)
@ChamberlainFoncha The best thing to do is not to do any classification at all. Toxicity is a continuous variable (measured in percentage or absolute terms). From your description, I don't see why it's necessary at all to collapse this into 3 categories, let alone 2.
I suppose you meant $\\alpha=0$ and $\\beta=1$ in your third question?
Note that there is nothing special about this being a proportion: a CI for the estimate of anything will be interpreted in a similar manner.  (However, different procedures may be used to construct the CI, depending on what is being estimated.)  Consequently, this question is exactly the same as previous questions asking for interpretations of CIs.
I don't understand why you want to use ML at all. Can't you just use $\\frac{P(1 boy, 4 girls|m>1000)}{P(1 boy, 4 girls|m\\le 1000)}$?
Update: I haven't found any evidence that this is possible, so I've taken the means for each individual animal and will use that as my data matrix and slashed the variables I'm using.  Unfortunately, the data are non-normal and variances are not homogeneous across groups, for biological reasons. I've seen that there are workarounds for heterogeneity (hda / qda), and that qda may also work on non-normal data. I'm wondering if I'm going too far down the statistical rabbit hole with all these workarounds and if there's another obvious method that I'm missing..
A *negative* $R^{2}$ signifies you definitely did something wrong in your implementation.
There's a clear difference from a frequentist perspective too - it's just that asymptotically it doesn't matter, & arguments are often made for conditioning on one or both margins in any case.
Also this is not really related to anomaly detection but since your membership matrix contains probabilities which sum to 1, one criteria you might consider to assess the quality of your clustering, is what proportion of points have membership probability greater than some threshold (say 95%).
Thank you Sven.
This approach is significantly faster than "split-compute-rbind". Furthermore it allows grouping on more than one factor. For a c. 650,000 row (8, narrow, columns) the "order-duplicated" approach took 55 seconds, the split-compute-rbind... 1h15minutes.  Of course when the aggregate computation is other than selecting or filtering duplicates, the latter approach or similar plyr-based approaches are needed.
114 is not *that* large.
@COOLSerdash I saw I forgot to mention that $x_i$ are like a random walk. I.e. they are successive. Thus, my question in the context of David's solution, is if there is a name for the sum of the covariances/variance in the context of random walks. In any case, thanks a lot for the link!
@guckogucko IEEE journals are still very important in the domain, despite having a paywall.
First, N is not >> 80. It's about 120. However, it is matched on age and sex. What differences model are you suggesting? I would want the permutation test to simply say "The chance that these differences would occur, given these correlations but no difference between control and treatment is XXX" I know it will be very small (having looked at the data) but I need something more than "Ecce! Differences!"
@Eric: $11-9=2$ which is an even number.  If $a+b=2k$ then $a-b=2(k-b)$ which is even.
thank you for your response! It looks like a great product!
@newhotter First of all: No, I did not solve the problem. But I think I found the reason for it: In my case the change in the log-likelihood is only around the third digit after decimal mark. So the change in the log-likelihood due to a change in the parameters is very small. This means, the log-likelihood must be extreme flat. I don't know what people do in such a case.
I disagree somewhat. Revolution does a great sales job in getting mindshare (as evidenced by your post) but as of right now there is very little in the product you would not already get with the normal R (at least on Linux). Intel MKL, sure, but you can get Goto Blas.  On Windows, they offer doSMP which helps as multicore cannot be built there.
As with most things, it depends on the context. For example, if you recruited students that were nested in schools, it might be very important to understand how much variability in the outcome is explained at the school level. I have reported estimated variances from RE models where I was interested in how community of residence was related to child growth. Hopefully these examples will help you find others, because there are plenty published papers that do this out there.
It appears your function is intended to represent a *cdf*. As such, I've replaced $f$ with $F$, but if that's wrong you should alter your question (in several places) and make the exact question clear (no matter whether you read it as a mis-scaled pdf or a correctly-scaled cdf, there are things wrong with the original question). Please carefully check the question.
I wonder if you can accomplish this by including a $\\ln (pop)$ with the constraint that its coefficient equals $1$. This assumes that housing production is proportional to population: $\\frac{E[y \\vert x]}{pop}=\\exp\\{x'\\beta\\},$ which is algebraically equivalent to a model where $E[y \\vert x]=\\exp\\{x'\\beta+\\ln {pop}\\}$. The term for this is a logarithmic offset.
The result is basically dealt with [here](http://stats.stackexchange.com/questions/34956/a-proof-involving-properties-of-moment-generating-functions)
@NickCox, okay, for the pirates example we could talk about industrialization possibly having a causal effect on both variables. The lemons example linked in my last comment is a better one.
Did your variable fulfill the assumptions of t-test?
Your two fits don't tell me much without data to compare them to.
Hi Michael. I tried editing your answer to make the equations more readable, using tex, but your last equation is wrong somehow - you have one too many `[` or one to few `]`. I guess you mean something like http://math.tntech.edu/ISR/Introduction_to_Probability/Joint_Distributions/thispage/newnode6.html though, right? You can probably just copy and paste the tex from those pages (highlight and ctrl+v the image, don't copy the image), and put it between `$` signs.
It's still an interesting point.  Thanks for bringing it up.
Why was this question closed? It relates to data visualization and generated several worthwhile responses.
Taking reciprocals is an order-reversing operation. It might help to draw a picture of the relevant tails of the two distributions.