tttjjj/elt_bench huggingface dataset:
Gene, welcome to the site. You should migrate questions, not ask them on multiple sites. You need to find a way to close the original math.SE question if you expect to get an answer here. Your question per se is unclear though. The alternative hypotheses are formulated as $\\mu=\\mu_1$ or $\\mu \\in \\Omega_1$, while $\\bar X$ may be a part of the test statistic, but not the hypothesis.
thanks for the direction to Sobol indices. Again, I'd like to specify that I'm looking for a general rather than a specific answer here. I'm not asking about a specific set of data but rather trying to explain a problem I've been having with a number of different sets.
so there is nothing to do with forcing the data to the $[0, 1]$, and in general I do agree with standardization if the author really looks for it :) I answered before the discussion appeared, so probably delete my answer if this one is correct :)
Does my edit answer your question?
An optimistic possibility is that your correlations are, in effect, data which you can feed into (e.g.) an analysis of variance (and that here as elsewhere behaviour might be simpler on the z scale).
Aha, this should be my problem, because I did not split my data into bins/cells and I am trying to calculate chi-squared directly from function values. I am going to read more carefully about this topic, thank you for help.
Given that you do not know the values of the $a_k$ nor of the $b_k,$ you will have to find extreme values that are consistent with what you do know. Your information includes the mean and the fact that the $a_k$ are non-negative, but you haven't given us any information about the possible values of the $b_k$ nor about the possible upper bounds on the $a_k$: both of these are essential for bounding the inner product.
[See](http://stats.stackexchange.com/questions/72285/confusion-about-moving-averagema-process) + [1](http://stats.stackexchange.com/questions/52877/trying-to-fit-a-model-after-detrending) [2](http://stats.stackexchange.com/questions/50151/how-to-tell-if-residuals-are-autocorrelated-from-a-graphic) [3](http://stats.stackexchange.com/questions/59694/autocorrelation-and-partial-autocorrelation-interpretation) [4](http://stats.stackexchange.com/questions/49571/help-understanding-the-following-picture-of-acf) [5](http://stats.stackexchange.com/questions/56824/help-interpreting-acf-and-pacf-plots)
Don't forget autocomplete on browser history in the address bar. I type the letter "f" into Chrome or Firefox and it autocompletes to facebook.com as the first suggestion. This feature has been active for several years now.
**Quick and dirty**: `sqrt(n)*randn(1)/norm(randn(n,1))` will generate a $t$-distributed variate with $n$ degrees of freedom. This is probably the fastest to code, but certainly not the fastest to execute, especially if you need a very large number of them.
This is a good answer (+1) but I'm not sure I agree with the statement "It tends to be studied in math departments". In my department (a stat dept) in grad school there was a lot of mathematical statistics research. In the math department there was plenty of probability/analysis research being done but none that I would call "mathematical statistics". Perhaps my University was not the norm.
That is a good paper - especially figure 1. I'm saving a copy of that for future reference.
@JMS - $\\overline{H}_{i}$ stands for the logical statement "$H_{i}$ is false", or that one of the other hypothesis is true.  So in "Boolean algebra" we have $\\overline{H}_{i}\\equiv H_1+H_2+\\dots+H_{i-1}+H_{i+1}+\\dots+H_{n}$ (because the hypothesis are exclusive and exhaustive)
It's easy enough to experiment and to find out that PCA based on a covariance matrix is necessarily affected by a change of magnitudes. Whether this is a problem depends on why magnitudes were changed and what your aim is. PCA with a covariance matrix only makes sense if the values are comparable (same units of measurement **and** you are interested in actual magnitudes); otherwise use a correlation matrix.
The predict error shows their ability, which is the difference of predict earnings and actual earnings. However, the horizon might affect the predict error. All I do now is to rank them into deciles and then take the average. Do you think are they comparable?
Definitely appreciated, thanks.
(+1) Good, simple, concise description. :)
+1 I like this interpretation! could you clarify " p below e "? why do you take small e ? you say "the probability of making the opposite mistake is" it is an upper bound  or exact probability? If I remember, this type of approach is due to Chernoff, do you have the references (I find your first reference is not clarifying the point :) ) ?
@Shabbychef: Right, your stipulation of standard normals establishes you want to know $f$ in a neighborhood of the origin that doesn't extend more than a few units away. RSM occurs to me because it includes methods that *efficiently* explore the behavior of a function, so if evaluating $f$ is expensive, RSM methods may be helpful. If evaluating it is not expensive, you have a purely mathematical problem. If each evaluation for a given set of arguments can produce a variety of results, you need the $\\epsilon$ term--but maybe you can eliminate it by averaging multiple evaluations at each point?
"Each of the dependant variables are discrete but are probably distributed normally." is contradictory. You may mean 'approximately normally', but you can't mean 'normally'. The discreteness might (or might not) be less important than the issues of independence and constant variance.
@whuber If you selected a "bucket size" that small then there is a "false autocorrelation" as you will have tons of zeroes and it will appear that one zero can predict another/ When you have "sparse data" or "intermittent demand/failure data" then one has two random variables ( the time bwtween events and the size of the event). Care should be taken to not make the time interval too small or too large. Some judgement is required in not making the interval too fine or too coarse.
How did you come up with -0.00037 for the covariance?
Ok I understand now what is happening, thanks so much for all the answers, I really hope that this will help someone else. I wish I could have selected all right answers. Thanks to everyone
thanks @ Peter Flom @ Nick Cox
@whuber: I actually thought questions on numeric optimization were on-topic. The goal function is a measure of different correlation matrices, but there is no clear statistical connection. Is there a better site to ask this question on?
You have not explained what you mean with $p^{target}_t$ and $p^{base}_t$. In the question the O.P. has only one mortality rate per year ($p_t$).
+1 Good advice.
You may find that the insight afforded by the answers to [Making sense of PCA...](http://stats.stackexchange.com/questions/2691) will help you. Some of the descriptions and images, especially in the highest-voted answers, really do answer your first two questions.  As far as the third goes, there is no connection at all between data *quality* and variance "explained" by principal components, but there are things that can be said about your data based on the statistics you report.
@Glen_b at 10:20
If the lm residuals don't work out a glm with a Poisson distribution will likely do the trick (for the original data)
FYI, _outlier_ is a valid scientific term.
whuber makes an important point about it being subtle - studying an interaction within the same study raises less concerns about that implicit assumption of common error/lack of fit. It is good that you have the same IVs in both, but were there measured with the same accuracy over the same range? This type of extra concern about heterogeneity may justify thinking of it as a meta-analysis. Your situation is briefly addressed in Greenland S, O' Rourke K: Meta-Analysis. Page 652 in Modern Epidemiology, 3rd ed. Edited by Rothman KJ, Greenland S, Lash T. Lippincott Williams and Wilkins; 2008
As I pointed out to @EpiGrad, correlation is *not* a necessary condition for causation. There is a widespread conception in data analysis that a relationship between variables always refers to a monotonic relationship, which is tacitly assumed by suggesting that correlation is a necessary condition for causation.
I think it is impossible. When you simulate $X_1, \\dots X_n$ you have no information about $Y_1, \\dots Y_m$ so it is unlikely that the draw $x_i^{(k)}$ can be considered as a subset of a  sample from $f(Y_1, \\dots Y_m, X_1, \\dots X_n)$.
I may refer to my own answer [there](http://stats.stackexchange.com/a/88553/3277) which is not in consent with Anony-Mousse. It seem that the whole domain is dwelt by opinions, not one truth. "Gold standard" approach, too. Have you seen absolute, non-relative gold standards?
Just wanted to let you know that (at least) one site user is giving this question some thought. I'll try out some ideas and let you know if I find anything that looks like it might work.
The OP may wish to see my answer to a very similar question here: http://stats.stackexchange.com/questions/71525/critical-effect-sizes-and-power-for-paired-t-test/71530#71530
It is possible for *all* the data to be one or more SDs from their mean. However, *at least* three-quarters of the data will lie within two SDs of the mean.  If that's not the case, then you have made a computational error.
Depends on why it has an upper bound: is it like a binomial?Describe your outcome!
If you compute the minimum Mahalanobis distance for each point, and plot them as a histogram then you will find the threshold that you are seeking.  http://stat.ethz.ch/R-manual/R-patched/library/stats/html/mahalanobis.html
This question appears to be off-topic because it is about how to use MATLAB.
This is a valid comment, but that's why included the phrase "data of sufficient quality". Let's assume signal/noise ratio is average... there is noise, but not so much to pollute the data. The complexity of the model can also be regarded as sufficient complex to learn the data but not too complex to make the training to difficult.
...(cont'd)... there may be other features in the posterior which make more accurate predictions.
Would it be possible to transform your random field to be Normally distributed instead, so you could use Hermite polynomials?  Alternatively, are you sure the random field is lognormal, and can't be represented using, e.g., Gamma distributions instead?  I doubt there is a handy orthogonal polynomial basis for the lognormal, thanks to the $\\exp\\{-\\ln(x)^2\\}$ term in the pdf; I certainly wasn't able to find one in my references, but then I'm not an expert.
You are correct: the formula is an elementary plane geometry exercise.  The lines create three triangles at the left, top, and right.  The left and right ones are similar, whence the estimated mode partitions its bin in the same proportion as the bases of these triangles.  In the notation of the formula, one base is *fm-f1* and the other is *fm-f2*, so the fraction of the way to go across the bin equals *(fm-f1)* / (sum of the two bases).  The interpolated value is obtained by starting at the left endpoint *L* and adding this fraction of the bin width *h*.
You can still estimate the coefficients that way; the parameter estimates are still consistent, but their standard errors *aren't*. Any hypothesis tests, confidence intervals and prediction intervals are not 'valid'.
@BrashEquilibrium Thanks again .. that was useful. The only thing I won't be able to do is collect more data as I'm working of a given dataset that was collected two years ago.
Jochem, I've merged your two accounts (thanks to @Macro who noticed that). You still need to register your account once and for all.
Thanks for editing that into your question. One approach would be to use a generalized linear mixed model (GLMM), treating the association between the groups as a random effect.
Indeed. Sorry about that, thanks for spotting the mistake. I have now edited the answer to fix that.
@Cam.Davidson.Pilon You can find the algorithm in page 272 of Introduction to Mathematical Statistics, 7th edition by Hogg, Craig McKean. Unless they are mistaken, I am not either.
@Elvis I think you mean Frank Harrell's answer there (whuber edited it)
GoF=goodness of fit, GRM=graded response model, right?
Thank you for your comment, I will look at your links more thoroughly.
is the "partial selfing" hypothesis exclusive to the "pollinator" hypothesis? If so, then your 2nd model would simply be a binomial model with probability p and size=4.
@whuber I'll paste a sample original surface data and some python code to render it. It is a line-structured point cloud over which I perform interpolation with `scipy.interpolate.rbf` (not included for now).
Great, thanks for this thorough answer!
Can you write down the likelihood function?
@Glen_b this is not homework. Besides, I think that the meta of this stackexchange has a preference for not using that tag anymore?
It is difficult to reconcile your summary information with the description of the histogram as "bell-shaped," because it sounds rather skewed.  Perhaps, to clarify your question, you could post the p-values or an image of the histogram?
No, "class" is not restricted to recursively defined distributions.
@Bogdanovist: I rather say that overfitting has to do with having too few training cases for too complex a model. So it (also) has to do with numbers of training cases. But having more training cases will reduce the risk of overfitting (for constant model complexity).
Just run the same code through glmnet(), not cv.glmnet() and the returned object should have the training error.
You should not transform to z-scores! A common approach if you want to do hypothesis testing would be to use ANOVA to test for equality of means.
+1 It might be worth pointing out that the error of the $n$th Taylor expansion can be understood in terms of the $(n+1)$th derivative. See [here](http://en.wikipedia.org/wiki/Taylor's_theorem#Explicit_formulae_for_the_remainder). This gives a case-by-case basis for determining how useful the approximation is in a worst case scenario.
Hope my edit preserved your meaning. [The FIL Methods group's page on SPM](http://www.fil.ion.ucl.ac.uk/spm/) states, "The sequences can be a series of images from different cohorts, or time-series from the same subject." This implies the data don't have to be a time series; one could compare different individuals or groups instead, if I understand correctly...[Wikipedia](http://en.wikipedia.org/wiki/Statistical_parametric_mapping#Statistical_comparison) also makes time series analysis sound optional.
As to why some of the predictions are outside (0,1): this is because the result of `predict.glm()` is by default on the link scale, not the response scale. Try `rbind(yhat=predict(mod, newdata=df, type="response"), y=df$y)` and `?predict.glm`, and enlightenment will follow.
Ahh, right, sorry. The 0.876 coefficient is high and worrisome, but it might be reduced when you add fixed effects. I don't know whether trading volume is typically thought of as a unit root process or not. There are unit root tests for panel data that you might employ, but try running the model with fixed effects and see what happens.
Do you know what the distribution of a ratio of independent chi-squareds is?
Welcome to our site, Remo! I have used $\\TeX$ markup to make your notation a little more readable. While doing that I noticed what appeared to be a typographical error and changed $\\mathbb{E}(a\\cdot b)$ to $\\mathbb{E}(a\\cdot C)$, understanding your question to be about generating a random vector $C$ whose expectation is orthogonal to two given vectors $a$ and $b$. If that is incorrect, please make any necessary changes to the question.
Did you try *drawing a picture* of the likelihood or the log likelihood?
Excellent - thanks very much, Nick, for the comprehensive and informative answer. Now I've got some reading to do!
@Youloush What is it you think is inexact here? (Note that the use of approximation before rejection sampling doesn't make the result inexact.)
(+1) Nice illustrations.
You appear to have posted three questions that are variants of each other. This won't help any one.
Maybe you'll find help here: Howell, G. T., & Lacroix, G. L. (2012). Decomposing interactions using GLM in combination with the COMPARE, LMATRIX and MMATRIX subcommands in SPSS. _Tutorials in Quantitative Methods for Psychology, 8_, 1–22. Retrieved from: http://www.tqmp.org/Content/vol08-1/p001/p001.pdf
@`mpiktas`, suppose i wanted to fit with a mean model of p,d,q 2,3,4 what function do you suggest `rugarch` or `garchFit` or is there others? or do i need to do further differencing until reaching zero `d` before fitting?
@rolando2 I've done this in the past (e.g., with one of onestop's [answer](http://stats.stackexchange.com/a/4077/930))--otherwise I put my reply as CW. I have no problem with this. As I said, my main concern is when edits might alter the original meaning of the post. I like illustrations a lot; they add a lot of intuitive and insightful perspectives, IMO. I appreciated whuber's initiative because it leads to one unique answer which is sufficient in itself, although I can understand that you may feel embarrassed: no need for that, we are all here to make this site better :-)
(When referring to someone with the `@` sign make sure that there is no space between it and the user's name. Otherwise SE ignores it and people do not get alerted about the ongoing conversation).
Question is misleading as you ask about unique combinations of factors and then in details you ask about summary by unique combinations.
@Spencer You'll find python/R packages for all the tools in Weka. See the updated answer.
Great response, @Nameless! Thank you very much!
+1, good question, and you're in good company with it, see commentary by Andrew Vickers "Whose data set is it anyway? Sharing raw data from randomized trials" (http://www.trialsjournal.com/content/7/1/15)
Can't believe I overlooked this fact! It was on my tougths at first, but I discarded it in order to try to come up with a quick answer. I'll be more careful from now on :-), thanks!
Because I think that it is good if people can fix typos, incorrect wordings... I like the idea of continuous improvements of answers. Is this against best practice?
+1 I think one point should perhaps be emphasized and stated more forcefully: It's not merely a matter of understanding the mechanics to inform the model, whether we know the “laws“ of the market, etc. There is a fundamental between reflexive and non-reflexive phenomena. The weather does not “care” about predictions made about it, at least not from one day to the next, whereas most financial series result in part from the actions of many other people armed with ARMA models and the like.
I don't see any distinction.
At present, this is more of a comment than an answer. Would you mind expanding on it a little bit? Eg, you could state why you think the package is a good option in this case.
The way you simulate your data looks weird to me. If you want, for an alternative more standard way, you can have a look here: http://stats.stackexchange.com/questions/12857/generate-random-correlated-data-between-a-binary-and-a-continuous-variable/12858#12858
As you press me, my personal view is that a good acceptable question will just cite the video as a reference and be focused on questions that other people might ask or difficulties others might express. This looks too much like "Here are some examples I have tried; are they correct?" There are many questions on mixed effects on this site; I can't see this being one of the most useful. Also, you are presupposing some acquaintance with a particular dataset, `mtcars`.
So should the probability for 21 heads out of 21 be 1 ? :)
Exactly the p-value `lillie.test` gives. Yeah, I was wondering about the adjustments...
Hierarchical clustering seems to optimal for you. There are plenty (dis)similarity measures for binary data. Choose one that will suit you best and go to h. cl. As for your point 3,  it is feasible only for an extreme cluster. If a cluster is between other clusters, its points will, naturally, differ in that respect.
In spite of your assertion of the contrary, Likert scales are *clearly* not normal. Your 'here is what I would like to test' is unclear. When you put a '-' between two variable names are you trying to test some form of equality of location or something else? Could you clarify what you're trying to achieve in ordinary English?
Shouldn't there be an inequality in the LM test?
I agree that his books are full of bluster, but the ideas are IMHO valid, and in many cases, scary.
@Damien There is a changepoint package in R. It is important for you to know all the caveats i.e.all the assumptions underlying the tests that are conducted. The BIG PRINT giveth and the small print taketh away. You will find a lot of small print.
Perhaps a link:  http://mcfromnz.wordpress.com/2011/03/02/anova-type-iiiiii-ss-explained/
It works very well this codes. Thanks
There could be trouble if X can be zero.
Try it with some $x_i$ equivalent to `FLOAT_MAX-1.0` or with $x_i-x_{i+1}$ very small but $x_i-x_{i-1}$ very large.
@Staty Despair: No, its in my answer: "...are weighted by the inverse variance of the estimate...", i.e. $w_i = \\frac{1}{SE_i^2}$.
How many individuals are used to estimate the item intercorrelations? What software is used for computing both correlations?
As a word of caution, and may be you cover that in your chapter: the interpretation of the model results as subpopulations may be way too strong. What a mixture model may be capturing instead is non-normality of the underlying distribution of random effects. E.g., you'd approximate a $t$-distribution with two normal distributions that have the same mean and different variance, and you'd approximate a skewed distribution by a sequence of normals with offset means and mixture shares that are monotone in the means. Dan Bauer had some papers on this equivalency (indeterminacy?) 5-10 years ago.
+1. Very good non-technical explanation of EM.
@EngrStudent, it actually generalizes really nicely. In the paper, they use the Pascal VOC task, which has ~20 categories. 

To extend our animal example, you'd have a subunit for "Fido" vs. (all birds, cats, andfish), another subunit for "Rex" vs. all non-dogs, and so on for each dog. For the birds, you'd train "Tweety" vs. (all cats, dogs, fish), "Polly" vs. all non-birds, and so on. There would also be a subunit for each cat and fish example, trained against all non-cats and non-fish, respectively. 

You end up with 1 SVM per labelled example, regardless of the number of classes.
Thanks Ragy for the example. However from what I can see your answer did not cover the calculation of prediction intervals using bootstrap. As I said in my answer I already understand how to use bootstrapping to calculate confidence intervals - which your code appears to do.
Not clear what you are asking. (If, for example, you performed FA in a sample and extracted 2 factors, there is no sense to compare these two factors with each other; btw, both means are supposed to be 0.)
This question has a bunch of problems:
1) you never accept the null, you just sometimes fail to reject it
2) Increasing the sample size makes it more likely to reject the null, not fail to do so
3) As @DavidRobinson said, it's like cheating.
My pleasure.  I'm not sure the Wilcoxon test would answer something you want to know - but if you are interested in over all dropout "earlier" in the stages of one group vs the other - that seems to me to be a good way to test in.  Cheers.
Hierarchical clustering relies on a dissimilarity metric that determines the distance from a point to a cluster. Why don't you use the labeled points as clusters, and calculate, for each unabeled point, the dissimilarity to all clusters? Then, you can either make a hard assignment using the smallest dissimilarity, or you can make a soft assignment using relative dissimilarities.
"Allows us to say, with 100% precision, exactly how big our lack of precision is"
i know that problem... its a old birthday problem... i am not talking about birthdate but brithweek over here...