tttjjj/elt_bench huggingface dataset:
@hoyem thanks for your suggestions. it's the same TV station which advertizes the different products, so I can't classify different advertisments in this way
And one more question. How can I simulate each of the three last populations 30 times without writing rnorm(1000,M1,SD1) thirty times?
What does the response consist of? Are you aggregating the answers into some single score, or a few scores, retaining answers along many dimenstions, or something else? Do you have other covariates to adjust for?
@Glen_b My current goal is to generate three random Matrices from matrix normal distributions with zero mean and different covariance structures, also adding correlation between those three. I translate this problem into multivariate cases by vectorising the matrices. Normally for generating correlated samples I would simply multiply the samples by the desired covariance structure but I'm not sure whether I could do this here?
Correct, but once again you allude to the fact that the estimate is only good for some people, but not the others. We epidemiologists love to keep the error non-systematic (aka random). If we estimate the conception date with one reference data, error can be assumed to be equal in all sexual lifestyles; if we asked them for the date of having sex, different groups will provide answers with various accuracies, and that can potentially induce bias, as the relationship is now confounded by the frequency of sex, which can either be observed or worse, unobserved (aka we never asked the mother.)
In both cases, unless your $n$ is very large, you may want to have a look at the variance as this is in many practical situations the dominating factor for the total uncertainty.
I realize that this is 6 months old at this point, but I do have a question regarding the equation for $z_i$. In section 3 of the Friedman paper they have it listed as $z_i = \\widetilde{\\beta}_0 + x_i^T\\widetilde{\\beta} + \\dots$ whereas you have it listed as $z_i = \\widetilde{\\beta}_0 + \\widetilde{\\beta}^TX + \\dots$. I'm confused where the difference between the whole set ($\\beta^TX$) and the single row ($x_i^T\\beta$) comes from. Any thoughts?
Thanks for the suggestions! I'm quite sure that lme is able to handle unbalanced data, but I'll try `plm` out to see how the results look there. I'm also pretty sure that scaling isn't the problem, as all continuous variables are log-transformed and hence vary more or less on the same scale (sorry for forgetting to state this in the original question!).
I'm perfectly aware of the problems with the LPM, I'm looking for a good statistic to quantify the quality of predictions. Since I am considering maybe 10,000 simulations, evaluating fit from a scatterplot will be rather difficult.
Thank you SO much for this, and the helpful code, which is a real life-saver.  It amazes me that R can do so much with so little code. The current research question is: "if variation between speakers in the amount of codeswitching is just as great as that between 
groups of speakers, using groups to characterise code-switching is unsound", but that may need to be refined,  and your point about controlling for conversation length is well-made.  I will run your code against some of our real data, and post again when I have done that.  Thank you again for such a helpful reply.
BTW, I'm trying to use Sparse PCA and it cannot be achieved by classical PCA since constrians other than minimizing square error is applied.
First question, still don't get it.  My mistake was subscript $x_i$?  Second question.  Thank you, that was my reading comprehension.  I thought it was sum of probabilities ....  Instead as you point out it's sum of K-1 and other terms.
Can you provide an academic reverence for your answer regarding the possibility to perform repeated measures anova on different scales by using z-scores?
Thanks in advance;
Tomer
Can you show code and a reproducible example for what you did? That way it's easier for someone to modify it.
@Abe: you might also be interested in http://stats.stackexchange.com/questions/24978/halton-sequence-vs-sobol-sequence
But in a case where the variable for 'animal' and the variable for 'weight' are both significant, but their interaction is not, I wouldn't even include the interaction in the model. The multiple regression estimates the effect of 'weight' independent of what the value for 'animal' is.
However, I see nothing wrong in removing physically impossible RTs.
I'm afraid I don't really understand your question. You wrote "What is the correct way to compute the predicted probability from a mixed effects logit model, or think about it statistically?" What is missing from the sample output to answer that question? Are you interested in the random effects?
There are many questions here. What is your response variable exactly - counts of the numbers of fish species or something else (e.g. abundance for individual species)? If they are species counts; what is the range of the counts? What are your observations (rows in your dataframe I assume) - information for different sites or repeated sampling from the same site (or a mixture of the two)? You should also try to phrase the question more conceptually or in mathematical formulas, and not relying on R code (which only make sense if you know the program and the specific packages).
Quoting a friend of mine: if you want to "hide" something in a paper, put it in the text rather then in a figure. If you want to make sure nobody ever reads it put it in a table! ;) Just joking of course, but having a website with interactive maps for users to click etc. all of that to get a table... well that would be disappointing!
You need to square your terms $n_k/N$ in the first definition. Also, with varying degrees of mild incorrectness, this measure is also attributed to Gini, Turing, Hirschman and Herfindahl, among others. The name Simpson's index appears most common in ecology.
I don't think this is "pretty sure" at all. Only Pearson correlation is a measure of linearity; arguably the main point about other kinds of correlation is that they have more relaxed ideas about what counts as perfection in relationships.
Could you elaborate on what "normality assumption" you are referring to?  (I am not aware of any such assumption for logistic regression.)
It's the multivariate analog of the usual 'noninformative' prior; a more succint/modern way to write it would be $p(\\mu, \\sigma)\\propto \\prod_{i=1}^k 1/\\sigma_i$ (with $\\mu$ and $\\sigma$ k-vectors). Like @probabilityislogic says, the $d$'s are differentials and come from the proper definition of a pdf. I think they ought to be typeset $d\\mu_i$. Pop in a one for them and you should be fine :) You're unlikely to see that notation in either of those texts, I think.
I will add an answer below their comments, but I also wanted to not leave the question unanswered, given the fact that the previous posters did not provide an answer themselves (just in comment form).
@guest43434 I'm looking at the distances between the three distributions, so I think $P3$ is still relevant here.
Two normal distributions centered around 0, one with larger variance (compare 0.1 and 0.9 quantile)?
By residual are you referring to (actual values after treatment applied) - (time series model prediction)? In that case, it seems reasonable to me. mb3041023 is right about wanting to use something like ARIMA:  `arima()` and `predict.arima()` in R could be suited to that.
I don't mind the downvote, but a comment would be appreciated (so I can learn myself, understand what was wrong, and improve my future responses).
Yes you can apply the delta method in this case.
This probably should be split up into separate answers.
I wouldn't say I'm at the cutting edge of statistical theory or the like, but one that's pretty easy to envision is the use of multiple imputation for missing/incomplete information problems, or MCMC based Bayesian models for many variables in very large datasets. In Epidemiology, data like the entire Medicare/Medicade database, or other national claims databases in countries with national healthcare systems. This can all be done now, but it's *slow* and requires things like clusters, which aren't accessible to some and...intimidating to others.
I see. If you are a Bayesian, the only way for you to deal with heteroskedasticity is by modeling it. Permutation methods are at the heart of frequentist inference, so you may never be able to get them, as well as most other heteroskedasticity-robust techniques. In permutation, like in finite population sampling, the random variables are not the values of the response variables, but of the indicators, so this is a finite probability space from which you sample your particular allocation of 0s and 1s (with uniform measure on that space), and sampling distributions are defined over this space.
Can you say more about your situation, your setup, your variables & your goals? It is going to be impossible to answer this question in the abstract. To help you formulate your question, you may want to read our [FAQ](http://stats.stackexchange.com/faq) & this blog post: [how to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/).
In general, I'd always use a robust measure of scale to identify outliers.  If you don't, extreme outliers tend to mask moderate outliers.  I'd also recommend moving past the MAD to something more efficient like Qn or a 2-stage Tau estimate of scale if you're serious about this.  Package `robustbase` in R provides many of these alternatives and more.
Please delete the t-table: it is pointless.  It would suffice to say your are using the upper 97.5 percentile for your critical region.  Of more import is a description of how the testing works and some examples of how the timings are distributed.  These variances differ so greatly that it makes no sense to test for equality of means.
The problem is that you are not looking at the full model, which is $Y=\\beta_0+\\beta_1X+\\epsilon$ with $\\text{Var}(\\epsilon)=\\sigma^2.$ When you invert this algebraically you obtain $X=(Y-\\beta_0-\\epsilon)/\\beta_1$. It can indeed be rewritten in the form $X=\\alpha_0+\\alpha_1Y+\\delta$ but now $\\text{Var}(\\delta)=\\sigma^2\\alpha_1^2$: the variances of the residuals depend on the slope! If $\\sigma^2$ is appreciable, the least squares fit is not good and its variance estimates are all wrong.
Do you mean you need a CI for the difference of  the prediction?
Just an argument to make your edit a bit more elegant, you don't really need to appeal to integration here.  Just use the fact that $\\frac{\\Gamma(\\alpha)}{\\Gamma(\\alpha)+\\Gamma(1)}\\sim Beta(\\alpha,1)$, plus that $\\Gamma(\\alpha)+\\Gamma(1)\\sim \\Gamma(\\alpha+1)$.  These are both standard properties of the beta and gamma distributions.  Also, when $\\alpha\\approx 0$ we have roughly $y\\sim expo(1)$, which may be quicker to simulate ($-\\log(u)$) than a general $\\Gamma(\\alpha+1)$ random variable.
Thanks for posting the examples. This post on CV shows some nicer code for the [ParSets plots in R](http://stats.stackexchange.com/a/12036/1036) (sorry should have pointed to that first). My foray into the ggparallel package suggests it is pretty rough around the edges so far (although random data like you show won't tend to look nice IMO for ParSets).
Thank you! Your last point is very interesting, it should be trivial, but I hadn't even thought about it. I will definitely try and check the distances between two cluster centers and an observation to the nearest cluster center! I validated the result of the clustering algorithm by printing out the top 15 words from each cluster and seeing if they made any sense, which is of course not a very precise way to do it.
Cpk = MIN [  (μ - LL) / 3 sigma , (HL - μ)  / 3 sigma ] sigma here is robust sigma ie sigma=[(p90-p10)/2.56]
Perhaps I can make my question clearer: Let's say I'd like to test the mean difference between the 25th and 75th percentile of my sample of cars from 2000 to 2010. One inefficient way I could do that would be to simply look at the cars from those two time points, excluding all others, as you suggest. I was wondering if there was a way to use the rest of the data to test that same hypothesis.
@Nick autobox classified them as 12-5-3 based upon their numeric order
Jeampz - If you paste your data from an ascii file in what looks like a good format before you paste (hint: avoid tabs), once it's pasted in, select the whole table and click the **{}** button above the edit window. Now save.
You have dealt with neither part of my question but simply raised more questions. What do you mean by "how much fit", which is still in your question. What do you mean by "exponential distribution hypothesis testing" - testing what, in particular? If you're trying to see if data fits an exponential distribution with specified parameter, the second part of my initial question applies -- *WHY* would you do that? This is the most central question. If you're actually interested in knowing whether it's okay to use an exponential model for inference, formal testing of it may not be the best idea.
TY for the reference to the paper
You may want to also ask @chl what source he pulled the above picture from ([in this post](http://stats.stackexchange.com/a/9944/1036) on the site). You never know maybe the authors made have stated how they made the picture. You could do it in a spreadsheet program such as excel as well.
Hi Glen, say 95%. Something which sprang to mind recalling my school stats is the rule Sigma = sqrt(N). So a very rough estimate might be: 100/5=20; 20^2=400. I guess this means I need to count images until I reach 400 bubbles to reach within 5% of the population mean? I suspect that is wrong but I do not know of a more rigorous approach.
@user2520918 Yes, `lmer()` will recognize that these predictor variables have no within-group variation. In fact, if you do try to fit a model with random effects of v3 and v4, the model estimation will fail. (Or at least it *should* fail... if it doesn't fail, then something is seriously amiss, because those random effects cannot be estimated even in theory given these data!)
By model, do you mean what is going on molecularly to produce this result?  If so, then I think I know exactly what is going on.  My assay is detecting both my target and a similar piece of DNA, that is why I see signal in the absence of target.  For this reason I had to tweak the assay so that it detects this non-target molecular at as low efficiency as possible, which had the side effect of reducing the efficiency of true target detection, ergo the 54% efficiency. Does that explain things?
Thanks for the tip, the book seems pretty good. The lme() approach will probably help find a better model for testing the main effects of the whole experiment. Do you have any tips on calculating significance of each time point for each comparison of treatments? Thanks!
This is the most relevant answer, at least the first part! In many MCMC simulations, the (log-)likelihoods are computed for all proposals and can thus be stored. Or the highest value so far and its argument can be stored. Provided the MCMC algorithm has converged over the number of simulations you ran, this is a valid approach.
It's a lovely podcast that I can strongly recommend but it doesn't deal that much with statistics. I've found the [Radiolab](http://www.radiolab.org/) also has some statistics - their show about [stochasticity](http://www.radiolab.org/2009/jun/15/) is awesome.
I see the point. However, since you answered so specifically about parametric comparison, and my data is non parametric, does the case  "There is no dependency on the range involving zero" apply to non parametric data, too?
As it stands, I can no longer easily expand on my original suggestion and explain how it solves the problem, because that problem it is most relevant to isn't there any more. I'm inclined to delete the comment.
For bias it kind of depends what bias you mean - you get bias with any smoothing, and ASH is smoothing (for that matter, so are plain histograms). For more general distributions, the Anderson-Darling test is still going to generally be better than chi-square, though its null distribution needs to be adjusted for parameter estimation (on the other hand, if I recall correctly, even so, it tends to converge fairly quickly so in large samples it may not matter much)
Please rather use edits and comments than new question if you only want to ask for extending the answer.
The problem is that I do not want to know the overall quality of the model or overall importance of my features but rather I need estimates at each level of a feature separately.
Because the expectation of the square of a random variable does not equal the square of its expectation.
Hi Shiva. No problem. But if you'd like to accept my answer, that would be nice.
@rocinante I see your point but now: 1) is it possible to calculate AIC (or other stats) when using the model to predict values from new observations) 2) why then RMSE seems to be a common metric used in CV?
Not everyone here uses R to read in your file format. Please give your data in more convenient format (plain text, Excel?) or publish it in your question if it's not too lengthy.
Can you give an example of transformation to vector? From the mathematical point of view your data is already a collection of vectors in three dimensional space, clearly you want something else. So the example would be very welcome.
You might get better luck if you specified how you are defining an inverse-Wishart distribution, as the term inverse has several competing common definitions.
note that AIC is also equivalent to ML when dimension doesn't change.  Your answer makes it seem like this is only for BIC.
it's an idea that's been used frequently since 25 years at least. You can read more about it [here](http://en.wikipedia.org/wiki/Kernel_trick).
you could look in the guts of the `coda` function `geweke.diag` to see what it does ...
@Lars I don't see any way to have different clustering scheme (distance + method) for rows and columns, after a quick look at the code. Some suggestions were provided on this [related question](http://stats.stackexchange.com/questions/6890/plotting-a-heatmap-given-a-dendrogram-and-a-distance-matrix-in-r); otherwise, the simplest solution is to play with the code and add your own modifications. I can show you how to do it if you want.
John, I have edited the question.  I hope I've addressed all of your questions.
Likelihood is a probability density? Could you supply either a citation to a source where this is stated (or proved) or edit your answer to include a proof?
By the way, although it doesn't affect the result, I think you miscalculated the standard deviation.
What is `I` in `time*I(time>=0)` ?
I am not familiar with R but this sounds like the design matrix is signular and hence you cannot get a unique solution.
Check this out - http://www.mpia-hd.mpg.de/Gaia/publications/probcomb_TN.pdf
Actually, the fifth paper does **not** use LSTM, but shows that even for **simple** RNN the gradient does **not** vanish for 100 time steps. Hence the question.
1.44 1.94 2.45 3.14 3.71

sorry no that was for 6, because I was assuming n-k-1 degrees of freedom. for 8 it is:

1.4 1.86 2.31 2.9 3.36
@onestop, Arun: Basically, it's a proprietary algorithm that gives a p-value for similarity.  I would like to assume that the algorithm has an appropriate null hypothesis and that it has an accurate way of placing whatever test statistic it generates in the distribution of test statistics (for the null hypothesis) and thus is able to give an accurate p-value so far as it goes (if this makes any sense).  Assume also that this test statistic distribution is not normal (and probably empirically constructed with no closed form) ... is the answer to my original question a yes or a no?
Excellent answer. I'd add that I think the general trend in the computer vision community is to use convolutional nets (at least) as inputs, but there was a time way back in 2006 when spatial pyramid matching kernels showed promise. On one-hand, you're hard-coding a feature extraction procedure that a neural networks are able to learn. On the other, it's more robust wrt hyperparameter selection. See: www-cvr.ai.uiuc.edu/ponce_grp/publication/paper/cvpr06b.pdf
(1) I have a funny feeling that this is homework. (2) I see no question to the community. but wait, ahh, now I see. You want us to do your homework. I dont think so...
This site supports $\\LaTeX$. It'd be helpful to retype your math using that. Many of your sentences are actually fragments. Attention to good composition is one quick way to attract answers.
Perfect answer.  Many thanks!
Thanks to you, Elvis and guest. So this is a consistent estimator of the cumulative hazard rate but not th hazard rate.
@whuber, Oops. Yes it should be $\\times$ not $+$. I was wondering what you were talking about ... This same typo appears in the original question - odd how one can sometimes be completely unable to see things like that since they know what they meant to write :)
Cumming, G., & Finch, S. (2005). Inference by Eye: Confidence Intervals and How to Read Pictures of Data. American Psychologist, 60(2), 170–180. doi:10.1037/0003-066X.60.2.170
You'll enjoy this.
I'm not sure of a particular name, but I would describe this model as, "a log-level linear model with month and year fixed effects".
The cheeky one liner: Unbalanced = hard maths.  Balanced = easy maths.  :)
Can you explain what the actual research question is? For example, whether diameter and depth differ between occupied and unoccupied holes? Or are you thinking more of diameter and depth being potential predictors for whether a hole is occupied or not?
Er... don't envy, I meant.
So the distribution of $y_i$ is conditioned on a certain value of $x$, is that correct?
@MarcClaesen thanks anyways man! Been a huge help. Found some promising leads
Interesting but the standard deviation of both groups is the same sqrt(36 * 1/6 * 5/6). Please change your answer as this sounds useful
My short answer to your first question  about "similarity distance" would be No , you may refer to similar question on a different matter here: http://stats.stackexchange.com/questions/32890/can-p-values-be-used-to-show-impact-of-treatment/32897#32897 . So this posts would suggest other "divergence" or distance measures. I guess it's better to first adress your first question.
What have you tried so far? If you don't show any sort of effort, you're not very likely to get much help.
How much would you recommend? You can employ super-consistency, but then too the number of observations should be around 20. All the proofs associated with detecting unit roots I've seen rely on the asymptotics on $T$, unless you are ready to make a strong assumptions on the errors. But to make such assumptions you need to test them, and for that usually more data points are needed, not less.
I replaced "delta" with "\\Delta". Hopefully that's what you intended. If not, you can edit. You should include an edit or additional discussion to clarify your question (see Alexis' answer).
+1 However, all this seems to assume that the computer is able to compute with perfect accuracy, and that we always choose formulas that allow it to calculate with the best possible accuracy. The first is never true and the second only occasionally true at best. If we very carefully set up our code it should be possible to always avoid negative numerator and denominator in the F ratio ... but in practice it's quite clear that some programs have not been so carefully set up. This is less a problem now than it was 25 or 30 years ago, but it's still sometimes an issue... ctd
@Avraham Question is not quite clear, OP using correlation and association simultaneously and not clear for multiple or partial or pairwise correlation, I think we should wait OP response.
No.  The $p$ value is not the probability **that** a given observation or more extreme is from the original distribution.  The $p$ value is the probability **of** a given observation or more extreme from the original distribution.