tttjjj/elt_bench huggingface dataset:
Thank you! Just to add, I found in the sampleSelection R package, under the help page of the inverse mills function, the ability to extract inverse mill ratios for bivariate probit models which can then be used in the final outcome equation. This doesn't resolve the panel data issue, but it does address the multiple selection one. FWIW.
@whuber - I meant equality, the boundary points only
I added a few more hints.
You may be interested in this related question, http://stats.stackexchange.com/q/2628/930.
(Cov)variance formula based on `n-1` is the unbiased estimate; based on `n` is biased maximul likelihood estimate. So both are valid estimates. The true population parameter is not known since we don't have the whole population at hand.
Yes, I agree, _if_ by "equal shape" to mean `identical distribution apart from ... shift` Then of course, the test is just the test of that shift (and hence of any quantile, including that one corresponding to mean).
@Leeor - Yes, more info on any attempts to legislate/impose policy or even sway public opinion in a concerted effort would make this a very good parallel to the issue I am looking at - if I can find more on that related to that aspect. Both answers thus far are great. Thanks.
@Sam, you're welcome. I have even fixed my typos for you :-)
check the edit.
Thanks @Momo for the suggestions and references (esp. the one by Rusch & Zeileis)! I will have to see how they perform on my data. Sorry, the site doesn't seem to allow me to accept two answers.
Thanks! This seems the solution i was looking for.
Hi thanks for the response, I was really thinking of material covered in a Grad Statistics course: Topics include one- and two-sample tests and confidence intervals for means and medians, descriptive statistics, goodness-of-fit tests, one- and two-way ANOVA, simultaneous inference, testing variances, regression analysis, and categorical data analysis.
Fitting an alternative model with only two variables gives me some non significant results...Now I can't predict the probabilities in this way. About the idea of a weighted mean: do you think to weight the mean with the number of observations? I didn't found anything about the mean of the variance. Is this allowed?
Oh yes,sorry!!!
Thanks a lot for your input @whuber. I did recreate your example and it all makes sense. However, I wasn't able to recreate on my own data of n=56 of which the head is left <- c(860, 516, 430, 1118, 860, 602) and right <- c(946, 602, 516, 1204, 946, 688). I get this warning message: "1: In pnorm(log(right), mu, sigma) : NaNs produced 2: In pnorm(log(left), mu, sigma) : NaNs produced" when fitting with the optimizer to extract the mle estimates. That brings me back to my earlier problem of having negative probabilities when calc. the likelihoods step by step and substracting.
It is a good deal better, thank you. However, you do still leave a number of totally subjective things (such as 'a better result') unspecified. Better in what way? What makes it better? How are we to guess? Many things might be optimized, but what do you want it to achieve?
See @stachyra's answer. (If you just want to use the standard errors to get approximate confidence intervals you can transform the end points.) For negative binomial modelling using `MASS` see [here](http://stats.stackexchange.com/questions/70619/dispersion-in-summary-glm/77594#77594) for the parameterization.
Yes, there is a wikipedia article. It says to use MLE if the data are normal and PAF otherwise. It doesn't say much about the merits or otherwise of the other options. In any case, I would be happy to know what the members of this site think about this issue, based on their practical experience.
@chl Yes, thanks for the correction. Was a bit more than a typo to be honest, more of a thinko. This conversion requires some assumptions, of course - i think they amount to requiring that the predictor should have (approx) the same variance in those with and w/o the outcome after a transformation, if required, to make the distribution of the predictor approximately normal/logistic (strictly logistic, but the two are hard to distinguish without a huge dataset).
Aha that's a good point - I clarified the question. I'm interested in a scenario when you're confronted with time/space restrictions and "cannot afford" mini-batch learning.
Hi again,I was going through an article with similar context and found that data aggregation may be what I need to do. I attach the relevant portion of the article herewith. If you could please take a look and give your comments, that will be great. Comments are very much appreciated. Thanks. Screenshot: https://www.sugarsync.com/pf/D728823_61176011_47943
`teaching statistics better` is pretty broad - can you perhaps ask for more specific advice? This is IMO [not on topic](http://stats.stackexchange.com/help/on-topic), but if you can be more specific I suspect it will be closer to on topic.
Is the score something that has to be between 0 and 100?  In that case you could divide by 100 and do a logistic regression on the resulting variable, which would always be between 0 and 1...feels a bit odd doing things that way, and I'm not sure how sensible it is, but maybe that's what the reviewer is suggesting?
Very helpful description, thanks for the direction and the detail on my subquestion as well.
Thanks a lot Rob, I'm trying to reproduce R results which I got using 'predict' command. Results are almost the same in first lead and diverge for higher leads. According to your explanation, it could be because of error terms which I don't have them available. Is there any way to get access to internal R process which calculates error term in each step?
You mean an explanation that goes beyond what is described in [Hothorn's paper](http://cran.r-project.org/web/packages/party/vignettes/party.pdf)? I remember some nice illustrations from [Carolin Strobl](http://www.stat.uni-muenchen.de/~carolin/) at the [IMPS2009](http://www.thepsychometricscentre.com/) conference; I think there're some handouts on her website.
According to the [book "Design and Analysis of Gauge R&R Studies" (Richard K. Burdick, Connie M. Borror, Douglas C. Montgomery)](http://books.google.be/books?id=lsYAJWg6stcC&dq=gauge%20RR&hl=fr&source=gbs_navlinks_s) this question is discussed in Section 7.2 of the [book "Confidence intervals on variance components" (Richard K. Burdick, Franklin A. Graybill)](http://books.google.be/books?id=XpDNvGEAe7UC&dq=%22confidence%20intervals%20on%20variance%20components%22&hl=fr&sa=X&ei=6KiVT9PyFseM8gO42JmYCg&redir_esc=y).
You should have posted the fit. It is a warning rather than an error so there was a result and if may be informative to look at the coefficients. In this case the se.est is zero.
"*so as the variance of the laplace distribution increases to 1, you lose all moments including the mean*" --- can you please explain why you think this follows? I'm pretty sure all moments of the Laplace exist no matter what the value of the scale parameter.
Multicollinearity is not necessarily a function of a single bivariate correlation. In addition to @Greg's answer, you might look at other posts here tagged `multicollinearity`, including [this](http://stats.stackexchange.com/q/70899/3277) one.
@ayush (1) the test statistics (e.g. Wald) depend on the sample size (the standard error decrease with increasing sample size, and you're likely to get lower p-values with a larger sample). (2) yes, although AIC may be used to compare non-nested models, here I was thinking of it as a way to compare different models of increasing complexity.
Here is a thought experiment to consider before going too much further: Assume two scenarios, A and B. In scenario A, each recipient receives their mail, opens it, then flips a coin with probability $r_i$ to decide whether to respond. In Scenario B, your recipients look at the mail they receive, flip a coin with probability $r_i$, and open the mail if it comes up heads; they eagerly respond to any offers contained in the mail they open and discard any mail they don't open.
This goes back to my point of unwarranted causal interpretation...
I'm using the standard `prcomp` function from R.
@Luna, it sounds like you're describing a "horn" shape, indicating heteroskedasticity. If this is a linear model, then I don't think the heteroskedasticity will bias your estimates but it will affect your inference (i.e. $p$-values and confidence intervals) so you'll want to take care of it if you plan to do any inference. You may want to consider generalized least squares (the `gls` function in `R`), which is a common remedy for heteroskedasticity.
Is this question so difficult to answer? or is it so simple that no one want to answer? If its difficult i wonder how people are implementing HMM in different applications and if its simple then may be its simple for you guys.. but i really need help to understand it.. if someone can answer..
Does the answer need to have confidence intervals?
I think I'm still being unclear.  I'm not trying to use $|N(0,1)|$ as a proposal distribution for $Q(Y;X)$.  I am given a realization $y$ of $Y$ drawn from $Q(Y;X)$ for unknown $X$ and would like to approximate the posterior $P(X|y)$ with particles.  The support of $X$ is $\\mathbb{R}$ so all samples from $|N(0,1)|$ that are greater than or equal to $|y|$ COULD potentially have generated $y$, it's just that some are more likely than others.  My primary difficulty is that I don't know if $y$ was generated from one of the deltas or from a continuous portion of $Q(Y;X)$.
In which time interval are you interested?  From $0$ to $\\Delta t$ or from $t_1$ to $t_0$?
I guess the "empirical" in your graph should correspond to "blue" and to "red" again. Then, it looks like you may have a symmetric around zero distribution (=no skewness) that exhibits excess kurtosis. As a start, you can see this post http://stats.stackexchange.com/questions/11821/interpret-t-values-when-not-assuming-normal-distribution-of-the-error-term
As far as intuition of the score, do you understand the intuition of a derivative?
Dave, wouldn't glmmADMB work for this problem (as a start)?
Another option is to store the data in a database (SQL or otherwise). Often, an analysis will only require a subset of data (variables 1:10, but not 1:1000) which will fit in memory. If the subset is still larger than memory, the analysis can be done in chunks (load in the data from the database 1000 observations at a time, appropriately collate the results together at the end).
I personally like this very concise description, by J.K. Vermunt: [Latent Profile Model](http://spitswww.uvt.nl/~vermunt/ermss2004f.pdf).
Most such *ad hoc* statistical procedures perform poorly compared to extant, well-known ones.  What have you done to verify that this procedure has any useful properties?
Many thanks.  I know about concentration bounds (method of bounded differences and the like). But this particular bound seems particularly awkward to tackle using those techniques. For example, I have no problem to prove concentration bounds for the maximum of $p(x)$. Thanks for the pointer, anyway.
To me, an outlier test (like Grubbs) seems more approriate. Run this code and take a peak to see why your test "seems" to fail. `r.cus <- efp(r~1, type="Rec-CUSUM")` and then `plot(r.cus)`. Here's a related paper on the topic. http://blogperso.univ-rennes1.fr/arthur.charpentier/public/strucchange.pdf
@user603 given the restriction to symmetric distributions in the first sentence of the question, a symmetrically trimmed mean would be a reasonable estimator for the population mean in a very wide range of such circumstances (assuming the mean exists).
Although simple and straightforward, the robustness of this solution is questionable: it assumes that the gold particles and vessels will be located at identical parts of the images when the particles are within the vessels. If, for instance, one image is just slightly rotated, shifted, or expanded relative to the other, then the correlation can be completely changed--even its sign can change.
Great, thanks for the info. Looks like I'm gonna have to do some more research first.
+1 for the side note and the link
The final line of code should be "sum(x<y & y<z )/N".
A reference is Tobler, W.R. 1965. Computation of the correspondence of geographical patterns. _Papers in Regional Science_ 15: 131–139. doi: 10.1111/j.1435-5597.1965.tb01318.x
" chi-square statistics are the sum of terms which each have the flavour $(value − mean)^2/variance$ or $[(value − mean) / SD]^2$." In Wikipedia, the chi square statistic is $(value − mean)^2/SD$ instead
What's R0? In what context does this question arise?
You write "You might ask: what exactly do you want to glean from your data?" Indeed! But you then answer by saying you are new to this type of analysis. OK. But try to explain it to us in non-statistical terms. What do you want to find out?
Well, I meant 'variation' without implying affine properties relative to it's arguments, so I didn't expect the second equation to hold.
Good answer.  With such a large sample size and a basically regular distribution it is implausible that 16 has just ended up at zero at random with nothing special about it.  A bayesian approach could help quantify this but the answer would certainly be "no chance".
The approach of data splitting is highly inefficient.  Until both training and test sets are enormous, the mean squared error for an estimate of likely future performance for a predictive model is smaller with bootstrapping or with 100 repeats of 10-fold cross-validation, assuming the resampling procedures had access to all modeling steps that involved $Y$.  Use data splitting when you also need to validate the measurement process, survey instrument, or other procedures related to the meaning of the data.  A good use of data splitting is when instrumentation varies by country.
Because of what I have seen via Rcpp, I am even more impressed by Julia---about 50, 60, 70 fold increases for simple looping as in MCMC, and several hundred fold for "degenerate" examples like fibonacci are essentially the same as Rcpp got!  But I also know that with Rcpp I still get access to the 3700 CRAN packages---as well as countless C++ libraries---whereas Julia right now has almost nothing. That said, the promise of Julia is huge. But maybe there is a "then" as well as a "now". Time will tell.
@chi thanks. Can you please explain your comment a bit more.
@tnaser A bigger concern is that you already know the answer: **There is certainly a relationship between season and win/loss rate**. It might be so small as to not be interesting, but it's certainly not 0.00000...infinite zeros. This means that the result of "significance" depends largely on the size of your sample. With enough data, you will eventually detect a "significant" difference, even if it's not practically important.
You are expecting us to Google that, read the paper and find "the formula".... Please provide more information.
@user1806 This is basically what I answered in my comment. Now, the point is that the OP want to know how it reads in plain english, which depends *a priori* of the kind of parametric assumptions that are made in the model.
@Anh every second there is new data to add. The theory should predict the future. In astronomy there was predicting the positions of comets for example. Also you calculate the power for an expected parameter value. So in the case of the quote, they would be referring to the power to test a theory that predicted a correlation of at least r=.5.
No, it's not a homework question. I wondered how it would be to get a max probability.
Thanks for the clarification.  I'll check out `MGCV`.
This is a one-sample Z test. You do indeed do calculations based on the sample mean. [example](http://www.statisticslectures.com/topics/onesamplez/)
I don't have any problem justifying the model, I'm just wondering about the technical problem in the last sentence of the question. Should the non-significant main effect be treated as zero or the small non-zero estimate for the sake of reporting the significant within site slopes? It probably doesn't make a practical difference, but which way is correct?
@whuber: I get your point. However, reading the question I saw it as "how should I call my variables?", which to me sounds more like a programming matter and not about statistics... On second thought, it is also true that the audience here is closer to that which will use the actual data than that of SO.
I tried this before, but the problem is that sleep does not have sufficient granularity to support this.
@zyx I haven't a clue what you meant in that comment. :-) Maybe it would be constructive to note that the column of 1's is not intended to have any interpretation: it is there to expedite the computations, nothing more. Although it is possible to think of many alternative approaches to this problem, I simply cannot fathom what you might actually mean by doing something "by hand" or by "other criteria." If you would care to elaborate in your answer--comments are not the place for that--then I'm sure it would be a welcome improvement.
Well take $\\varepsilon$ to come from other distribution than normal. Take the distribution which does not have second moments, then the expectation of the $L_2$ norm need not exists. This is a contrived example, but it might give general idea, i.e. that except in normal case MSE and LSE are usually not equal.
@ onestop, thanks for your help.  I do not know how to set burn-in.   In Sample Monitor Tool, I just use the default setting of beg of 1, and end of 1000000. Is that for burn-in setting? Thanks again
Want to post it ?
Can anyone suggest an answer for this as I am having same question on my mind
@Glen_b Each experiment would be done with different cells, i.e. starting with "100%" attachment and applying a different value of shear stress.
@whuber Ah sorry. Currently I am happy with $x_i$ being univariate, but I think I might have to extend it to the multivariate case later. I did an update to the notation to make it more clear, but reverted it again since then its out of line with the comments.
+1, Why do you disagree w/ change scores? Wainer, eg, has advocated them as more likely answering the question people have in mind than using a baseline covariate.
thanks Ahmed. very helpful
z-estimation isn't included in first usages identified in http://www.stat.iastate.edu/preprint/articles/2011-10.pdf If you have access to www.jstor.org that is often helpful for this kind of task. As always, a supposed first date can often be corrected by an earlier one, but the latter is never confirmed absolutely.
I would think that distance would be a measure of the separation of the mean vectors.  But that would require a common scale.  If the covariance matrices were equal I would think the Mahalanobis distance might be what the OP is referring to.
Thanks for your answer. That is an empirical/ iterative solution I was thinking about as well. For my simulations, however, I need a more analytical solution without a costly fitting procedure. Fortunately, I just found a solution which I will post shortly ...
I'll take a look at these references, thanks!
This is given in every textbook on time series. See Chapter 3 in [Shumway and Stoffer](http://www.stat.pitt.edu/stoffer/tsa3/) for cross-disciplinary examples.
@whuber. Yes. But non-linear re-expression of X is really a different problem. The "right" way to do this is to assume a latent process Y that is partially observed and try to model that using a non-linear state space model. But I figured that was more complicated than the OP would want. My suggestion will work ok provided the cap is not hit very often. Otherwise it will underestimate the forecast variance.
I'd suggest that making clearer what you are doing would have a better chance of eliciting a good answer than offering a bounty. According to this you are estimating 600 parameters ... from what data? What kind of features? Examples and concrete details are needed.
I agree with rolando2 that the rank of difference is useful. I was thinking that the p values would give an index of the magnitude of the differences scaled for their 'convincingness'. It would be interesting to see if there was any important differences in the ranks assigned by p values from, say, t-tests and the standardised residuals suggested by rolando2.
It's a very general observation. I can point to technical discussions of effects in particular situations such as http://www.pokerroad.com/forums/red-pro-strategy-discussions/3936-shoving-mathematics.html, but I don't know whether that helps you.
Your dilemma is false. Partial correlation is just another way to standardize regression coefficient `b`, and they share the same p-value. http://stats.stackexchange.com/a/76819/3277.
@Aniko, you are right, it is better to say that I have a random variable. I just wanted to abstract from the details. In my case the real numbers are created by different person (one "random number generator" in my case is a human).
Thanks, I will read those papers. I wonder what definition you think is most proper for DBN, if the Wikipedia one isn't?
@Behacad  My question arose when i was studying epidemiology.As far i know , epidemiology relates to statistics.
I am using this approach in a forecasting project I'm developing, but don't yet have any results to share... You could try a few variations on the approach and see what effect they have on your forecast accuracy.
thanks so much for your answer! It was really helpful for me!
If you have a dependent and an independent variable, then why not use regression?
(+1) nitpick: $\\sqrt{R^2}$ only equals the observed correlation coefficient when it is positive :)
Furthermore, the next line (53) is going to cause you even more problems, when it tries to allocate a 1.25e9 x 150 element array. By my calculations (which could be incorrect) this array will require almost 700GB of RAM!
People often forget another use of the bootstrap: to identify and correct *bias*. I suspect that if you were to include a bias correction within your bootstrapping you might get much better performance out of the CI.
@Isaac, the list I've provided was not intended to overwhelm. It was intended simply to potentially help with brainstorming. Depending on the nature of the project, I would think that handling 1-2 of them would be the most that could be expected. Cheers.
Let's not confuse "intuitive" with "nontechnical".
Why don't you work out a small example. Say, $(x_1,x_2)$ = $((-1,-1),(-1,1),(1,-1),(1,1))$.
@Anony-Mousse I think that this will affect my clusters at the end. I think that small clusters will be lost when I take a sample from the data set.
No problem.  A big benefit of cross validation compared to a holdout is also that you will get a standard deviation of your estimate.  This comes in very handy when you are comparing different algorithms and techniques.  A model with the same mean performance but less variance may often be more desirable.
Thanks @gung for changing it into a nice format!
Normally, people don't use bands around PDFs as in your illustration, because they are unrealistic: the upper band would have total probability greater than $1$ while the lower would have total probability less than $1$. A [standard technique](http://en.wikipedia.org/wiki/Probability_box) draws bands around the *cumulative* distribution functions. Would such a technique be acceptable to you?
> ks.test(y, y_norm)

        Two-sample Kolmogorov-Smirnov test

data:  y and y_norm 
D = 0.9386, p-value < 2.2e-16
alternative hypothesis: two-sided 

Warning message:
In ks.test(y, y_norm) : cannot compute correct p-values with ties
>
Perhaps we have different takes on partialling out? In my mind it would involve i) e1 <- resid(lm(y ~ Sex)), ii) e2 <- resid(lm(X ~ Sex)), and finally iii) lm(e1 ~ e2) . i) residualises y with respect to Sex, ii) residualises the other covariates (X) with respect to Sex, iii) fits the partial regression. In that case it doesn't matter how one codes Sex. In the above, we aren't really interested in the effect of Sex nor the interpretation of the coefficients. If we are model building, i.e. controlling for Sex as a Null, then how we parametrise the model *is* an important consideration, however.
A big +1 for sharing this approach.