tttjjj/elt_bench huggingface dataset:
HI, thanks for the advice. I am currently working on some simulations so I hope they work out. As far as I know heteroscedasticty does not affect estimation of regression coefficinets, but it can over- or under-estimate the standard errors of these estimates.
@whuber, yeah I have some Y1 and Y2 values missing. But when calculating the MSE, correlation, I have only used those values where both Y1,Y3 and Y2,Y3 pairs are available. Lets say I have 100 Y1,Y3 pairs available with both Y1 and Y3 available and 200 Y2,Y3 pairs available with both Y2 and Y3 available
IMHO, $L_1$ space is not a Hilbert space, so I doubt there is a kernel for manhattan distances.
Can someone answer @SRKX question about why some implementations (e.g., `fGarch` in `R`), don't give us the fit for the $a$ parameter? (However, EViews does for some reason).
(+1) Fantastic! Now, how long did it take for you to put all of this together?!
There is also an age divide in some industries, e.g. some younger actuaries prefer R, older actuaries have been using SAS for years and prefer not to change.
@chl definitely I will do that. I will remove one post once I get a favorable response in either of the sites.
I found that it is more sensitive to changes to values at the extreme ranges. For example:
(0,0), (0.5,0.6), (1,1) gives 0.9901, and
(0,0.1), (0.5,0.5), (1,1) gives 0.9890.
Both sets have a single point deviating the same amount from the diagonal, but give different results. Like I said this is not a deal-breaker, but something to keep in mind.
Do you know how that stacks up against the Agresti-Coull CI? (Agresti, A. and Coull, B. A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. *The American Statistician*, 52(2):119–126.)
Definitively, you learn me many things. Thanks !
@EngrStudent, I added some sample code to the answer above.
Have you considered a multi level model?
To generate a Gaussian copula with t-margins, the easy way is to generate a multivariate normal, transform to uniform margins (giving the copula) then transform to t-margins.
This set of videos looks great. I [created a question](http://stats.stackexchange.com/questions/8239/introductory-statistics-video-courses) to popularise it and see if there are any other great quality video courses on introductory statistics.
+1, this is a good answer, much more thorough than mine. I'm cautious about the chi-squared test, though. The thing about the chi-squared test is that it doesn't assume one of the variables is an explanatory variable & one is a response variable, so if that's true, you have less power than say a logistic regression w/ all categorical predictors. I describe this in the other answer I link to in my answer here.
@Alecos: $\\text{Pareto}(\\alpha,\\lambda)$ is the distribution with density $f(x)=\\frac{\\alpha\\lambda^\\alpha}{(\\lambda+x)^{\\alpha+1}}$.
(+1) There was also an interesting thread about the causality issue in this related question, [From a statistical perspective, can one infer causality using propensity scores with an observational study?](http://stats.stackexchange.com/questions/3400/from-a-statistical-perspective-can-one-infer-causality-using-propensity-scores-w).
Ok, fixed. BTW, these typos do not exist in my original script.
Normally the binomial coefficient would be written with the arguments in the other order. See the last line of the introductory section (just before the 'Contents' section) [here](http://en.wikipedia.org/wiki/Binomial_coefficient)
But you shouldn't take p-values as your sole input for decision. In most problems a coefficient of -3.348e-15 is meaningless, regardless of the the p-value! If you needed that kind of precision, then you would have to use the proper instrument.
How about the references mentioned in the documentation page you link to?  The Yee/Wild paper seems like it explains it fully and has a non-proportional odds model example.  The Yee/Hastie paper specifically references this library and has several examples, and there is code for at least one in the appendix. Plus the documentation page you link to also has several examples.  What more are you looking for?
Do you mean dummy variables to indicate people?  (Or whatever your units of analysis are). This is what fixed effects models too, if you set them up properly. I think we need more information to know what you are doing.
Are you sure the Frechet distance is between ordered pairs? the [wikipedia page](https://en.wikipedia.org/wiki/Fr%C3%A9chet_distance) says that a point in one path can be matched to multiple points in another. Also, surely there is more than one such metric. What about the sum of the distances between ordered pairs?
With apologies for the self-citations, here are a couple of related questions: (**1**) [Birthday-coverage problem](http://math.stackexchange.com/q/26772/7003) and (**2**) [What is a tight lower bound on the coupon collector time?](http://stats.stackexchange.com/q/7774/2970). The first asks a version of your same question and the second asks something related, with answers and comments providing the information necessary to construct bounds. The bounds are a little weak, but are nonasymptotic.
Are all of your class 0 observations at x~=75?
@baogorek Off the top of my head I can't think of any cases where MoM beats MLE in small samples, but there could well be some cases. It might make an interesting question here - to ask if there are any known cases. If nobody else asks it, perhaps I will.
Can you clarify the part about $X_1$ and $X_2$ being "not independent"? Eg, variables can be not independent in the sense of being collinear w/o causing any problems relative to this issue. OTOH, there can be a real interaction b/t 2 variables & have *those* 2 variables be completely independent.
For the record, this result is referred to in [many threads on this site](http://stats.stackexchange.com/search?q=cauchy+stable) and is explicitly asserted at http://stats.stackexchange.com/a/74272.  I find one (very pretty) demonstration in Douglas Zare's post at http://stats.stackexchange.com/a/36037, but there are many others available.
@user2998757 See the update.
MCMC is not really Bayesian method.  You could simply draw estimates from your target likelihood (not posterior) if convergence is the issue.
Great. Works like a charm. Thanks again!
Its also worthwhile to point out that the $p_i\\frac{\\partial}{\\partial p_i}$ trick works for higher order moments as well.  For example $\\left(p_i\\frac{\\partial}{\\partial p_i}\\right)^2=p_i\\frac{\\partial}{\\partial p_i}m_ip_i^{m_i}=m_i^2p_i^{m_i}$ and so on by replacing "2" with $k$ gives you the $k$th order moment $E[m_i^k]$
@Jens the call should be `anova(mod1$lme, mod2$lme)`. If you are fitting a non-Gaussian model then this might not work as there isn't a true log-likelihood in PQL methods, hense the quasi-likelihood in the name PQL. This is one reason to use **gamm4**, but then you need to do something about the correlation structure as **lme4** doesn't allow them.
This phenomenon has little or nothing to do with Spearman's $\\rho$: given any dataset of 26 cases that exhibits any variation at all, I can substantially change the analysis if you allow me to remove four of them at will.  Spearman's $\\rho$ is robust against wild changes in a few of the *values* in the dataset, but not against motivated deletion of data!
Exactly - I'm looking for an expression for the 1st two moments. I updated the question to remove the term asymptotic.
I am not sure but my guess is that it probably has something to do with using an $n$ or an $n-1$ in the standard deviation formula. Possibly Excel uses an $n-1$ by default, or something like that.
Please don't change the content of your question so severely, @Lolly. We want the questions & answers on CV to remain as a permanent repository of statistical information, so that someone else can learn from this later. You can always add comments below your question, or update your Q w/ additional info below the main portion that already exists.
The Z test you've calculated is probably a Z test of the null hypothesis that the true value is 0?
Any non-zero effect size - & what effect sizes in the world are really, exactly, zero?
Odds ratios and Fisher's exact test are not the same thing. The function for using Fisher's exact test in R is `fisher.test`. A description of how it computes the p-values is found in the help file `?fisher.test`. See also http://en.wikipedia.org/wiki/Odds_ratio#Statistical_inference for statistical inference on odds ratios.
The data is not based on one company in multiple years, but one year with multiple different companies. Therefore inflation doesn't need to be taken into consideration.
This may be an option but I have never seen this method used in my field. Often a fold-change from rest is used but I don't agree with not showing the variability which occurs at rest.
A problem I see with this is not being able to show significance in the change in the measure from rest as there would be no resting measure, and then again not being able to show the variance at rest.
(+1) Re the edit: Try using 536 replications of 52.15 :-).
You basically want to find a (nonlinear) regressor for the sum, based on `x1`,`x2`,`x3` and `x4`. Googling for regression should get you started.
Anthony, you should have the required reputation now.
`?ks.test` in R (Kolmogorov-Smirnov)
Some empirical data might make an interesting alternative from which to build a model.
Do you mind expand on that? I hear the similar comment on p-value all the time but most of them focus on the drawback of p-value instead of clearly providing better ways.
Using Hammersley-Clifford, I get $$Pr(X)\\propto \\phi_1(A,B) \\phi_2(C|A,B) \\phi_3(D|A,B)$$ but this does not use the first independence assumption...
I got it and this was really helpful, thanks.
Actually, I think you've demonstrated that removing outliers can increase the ability to get significant mean differences :) I suppose it should in general if it decreases the standard error of the mean. IOU +1 in 13 hours (SE has my voting finger on cooldown), but FWIW, it's not clear that this is a question of mean differences. An NHST of a correlation might behave differently due to inclusion/exclusion of outliers. E.g., with `i=c(rnorm(50),10);j=rnorm(51,5);inoout=i[i<8];cor(i,j);cor(inoout,j[1:50])` you'll sometimes get stronger correlations with the outlier, and sometimes without.
What makes a correlation causal is a contentious issue in philosophy of science. The "gold standard" is to perform an experiment where the treatment variable is assigned at random, thus ensuring other potentially confounding covariates are not systematically related to the treatment. However, in many fields and for many questions, experiments are impossible. Some turn to so-called natural experiments for exogeneity. You may be interested in Paul Holland's classic article, "Statistics and causal inference". Journal of the American Statistical Association, 81, 945-970.
That cannot be a correlation matrix: as $|x_i-x_j|$ gets large, its values grow without bound.
Often the parameterization involves terms of the form $(1-\\theta_1B)$ and $(1-\\Theta_1B^{12})$. If you are using a computer package, check which parameterization is implemented.
(1) how would you use PROC GLM? cluster robust SE? (2) Do both GEE and Mixed, compare results. Should be very similar. (3) is this a conditional/marginal model question? that is so subject field specific doubt you'll get a good answer here (answer likely =conditional) (4) any interest in within/between cluster effects? http://www.ncbi.nlm.nih.gov/pubmed/15213496
I recommend you change the title. The answer to the question 'is this result good or bad' is subjective, and would depend on what the predictor variables are - as well as the values of the respondent. Car mechanics / salesmen / manufacturers might prefer more crashes, a car driver might prefer less. The size of the estimate of a standard error is just that - there is no reason to think that smaller is better without context. Perhaps you want to know if your results support your hypothesis or not, or if your results are valid, in which case you would be able to find useful answers here.
Stask's point is that you can create 1 variable *via* any arbitrarily large number of intermediate additional variables (interactions, transformations, etc) so that by the time you have n of them you are guaranteed to have the maximum correlation with profit.
Aaron - that link is dead, could you post a fresh one?
of possible interest: http://stats.stackexchange.com/questions/4700/what-is-the-difference-between-fixed-effect-random-effect-and-mixed-effect-mode
@Macro good point. thanks. I changed the wording a little to "researchers typically ..."
Thank you very much Glen_b. It is very helpful.
Both the mean and the SD computed with the nominal data values will be biased low if you use $x_5=5$; if you use any other value for $x_5$, you're being arbitrary.
So for example, if I want to learn about the moments of the sample kurtosis when sampling from a $t_{12}$ distribution (which is an example the kind of thing I think the OP is asking about), I won't be able to use an MGF for that.
+1 Clear, pithy analysis.
Basically a duplicate of [this](http://stats.stackexchange.com/questions/48627/sum-of-squares-of-residuals-instead-of-sum-of-residuals?lq=1), where in turn [this](http://stats.stackexchange.com/questions/46019/why-squared-residuals-instead-of-absolute-residuals-in-ols-estimation) is linked as 'essentially a duplicate'. Another near duplicate is [this](http://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia?lq=1). There are others as well.
So you want a plot that shows a boxplot plus the original raw data at the same time? In what package? (In R, I've come to really like the `beanplot` package, using `jitter` to display the raw data.
@Nayan: I need to determine where we can start in terms of common tools/knowledge.  Do you use SPSS or R?  Do you know how to do regression?  Do you know how to calculate a Z-score?
Yes, it is the hinge loss. The hinge loss has been removed from the objective and made into a bunch of constraints (their number equalt to the number of examples, $l$ in your notation). In particular, $\\max[0,1-y_iw^Tx_i]$ is the loss on example $i$.
Please give more details on what data you have. Do you have repeated surveys? Or do you have the same people measured multiple times? How many time points have you got measurements for (one per year? More?) How were the variables measured? (e.g. "heart attack ever?") Do you have covariates (e.g. age, sex, exercise, weight)?
@user603 I agree with you in general that we do not want to send programmers away as a policy. The problem here is that the question *explicitly* asks just for `R` code. If it were changed to request an *algorithm* I think there would be no controversy.
Care to share the citation to that paper, Brett?
Did you tune the cost parameter ?
Well the problem is that tm library does not scale well. RWeka provides access to only select WEKA functionality (no way to call WEKA feature selection methods, for example). I don't see any other packages that would let you do pre-processing on text data in a scalable fashion. And then there's always this question of how to store and process highly sparse data...
Many people ignore the autocorrelation in logit.
How would I approach this. I do know how to perform non-linear regression, however I do not have experience with 3 variables and would not know where to start.
Do you mean mean absolute prediction error there, or the actual mean of the (signed) prediction errors?
+1 Man, this app made my switch to Linux really hard.
Just because one variable correlates more highly with overall satisfaction does *not* mean that changing it will have the greatest effect on overall satisfaction. I agree with @NickCox. If the problem you have stated is not your real problem, then please withdraw your post and re-post your real problem. If it *is* your real problem, listen to Nick.
I don't think you'd expect the SD to change with sample size. As you point out, the standard error would decrease (given constant SD).
Imagine a world in which it was necessary to rule out the placebo effect on cancer!  :D
You can constrain things with least squares by reparameterizing (let y* = y-x2 and x* = x1-x2. Fit y* = ax* + c and then let b = 1-a. However, that won't deal with your errors-in-variables. In some cases at least, though,  reparameterizing might work there as well.
Your transformation would need to be a bit weird in that it would need to pull the tails in -- preferably in a smooth way.  But I'm not convinced of the advantage.  What are you doing that you think you need a normal distribution?
Thank! in Your answer there are two important points:
1) the evaluation of the uncertainty. Your suggestion about log transformation is advantageous. The uncertainty of the concentration A could be fixed with the combined standard deviation.


2) the problem of only one measurement for the soil could be insurmountable because of the small amount of each sample.
This is very useful, but not quite what I'm after. Up voted anyway.
what if there were *lots* of 2s and 4s, and relatively few other ratings? It is hard to imagine this happening in reality, but would one really want to call that polarized?
I think that behind the score there is also a constant will of @whubert to create and generate constructive connexions and exchanges. impressive !
Andrey, this is a pretty broad question and unlikely to get you a satisfying answer.  Perhaps you can be a bit more specific about what type/field of psychology you work in and what types of data sets you keep and work with. (also what types of questions you're trying to answer) Then people might be able to help and let you know what types of methods work best on those types of questions.
"Hyperbole" refers to the text and "exaggeration" to the graphic.
@whuber I've edited the question. In its current form, if it's answered by something else on the site, it's probably best to just go ahead and close as a duplicate since I think this expresses pretty precisely what I'd like to know.
Nope. But the converse almost works: normality (and heteroskedasticity) of errors plus one of @Aerandal's principles does imply ordinary least squares.
Thanks.  I did think about 3-fold cross validation or just 2 samples.  But don't you think the training set would be too small for logistic regression model?  The larger picture is: Why do we divide the set?  To create a model that can predict an event.  My main focus is not to build a prediction model but just to show that some coefficients are in line with a hypothesis.  Though at the same time I agree that using 2 samples - train and test can prove the over all model performance, which makes coefficients & hypothesis sound solid.
Did the idea to look at the shift come *after* you saw that these data were like this, or before? (If it came before, why did you do a chi-square?) If you mainly want power against simple alternatives like a shift in the mean, *why on earth do a vanilla chi-square*? It has terrible power against such alternatives.
I assume that when you say that your test had 95% confidence after 15 days you mean that the p-value was less than .05?
@steadyfish, oh, I see. In fact, I was sloppy with the transpositions and whether the vectors were row or column vectors. I have made that precise now and added a little about the relation between the empirical covariance matrix and the theoretical covariance matrix. The theoretical is not defined in terms of the observations.
Not to be nit picky, but "ensembling" is right in the title of my post.  I'm very specifically looking for an R package for ensembling arbitrary models, which doesn't seem to exist.  Thanks for posting the code, though.  Maybe I'll write my own package!
@chl: it doesn't make much sense, but omitting it might be misleading. If I put in 123.4, someone like you will see the extra digits and just disregard them, no harm done. If I put in 120, many readers will think this is accurate to 3 digits - bad.
There is precisely no Stata-specific issue here, just a question of the algebra of linear regression.
Maybe this question is more suitable for Stackoverflow since it's more related to exact implementation rather than statistical background?
We went with cross-validating the confidence limits. It was computationally expensive.
In your 3rd answer "the likelihood under stated assumptions" What assumptions here to make the likelihood factorizable?
By the way, the solution outlined above is written out in more detail at the bottom of this other problem: http://stats.stackexchange.com/questions/3194/how-can-i-test-the-fairness-of-a-d20
"Hiring a statistician after the data have been collected is like hiring a physician when your patient is in the morgue. He may be able to say what went wrong, but he is unlikely to be able to fix it" = George Box (I believe)
I'd add that the mean, sd, max and min provide some evidence regarding the issue of whether assuming the underlying distribution is symmetric is a reasonable assumption; if the distribution is symmetric then the population mean and median will be the same.
For me, he is the best statistical thinker ever.  And also one of the most interesting and distinctive writing styles I've seen.  Never seen anyone use the word "common sense" like he does!
Could you explain what is meant by Adstock variable in statistical terms? Wikipedia reads like it is some kind of distributed lag model, like the Kyock transformation?
+1 Reading this answer is really amazing and helpful. I personally learned lots of thing. It is one of the most complete answers that I have ever received. I really appreciate it.
If without replacement then you'll need to specify exactly how the probabilities change when you remove a sample from the pool. The obvious way is to just renormalise the probabilites $P$ and repeat, until you've drawn $q$ samples (so you'll probably end up building a $for$ loop over $q$ iterations, each time calling randsample to draw a single weighted sample from the remaining available ones). But be sure this renormalisation makes physical sense for the system - it's entirely concievable that some systems may need a more involved re-calculation of sample probabilities.
@steffen I made it more clear what the transactions are. Also added a bounty if you are still interested in helping. Thanks.
All these suggestions are interesting, I agree that random forest should handle natively categorical variables, but scikit-learn doesn't... I think that's one of scikit's main flaw btw. I'll try your code on my data to see what happens, and I'll see about your other suggestions !
@whuber That makes sense, but in that case...what do you suggest? How do I get a measurement of the evidence provided by the bootstrap technique of these two samples having evolved differently?
If the regression includes a constant, the mean residual is guaranteed to be zero.  Normally one doesn't formally test residuals for symmetry: the purpose is usually exploratory or diagnostic or both.  It's straightforward to check with a folded histogram or an m-letter summary.  For additional suggestions see [EDA short course part 1](http://www.datavis.ca/courses/eda/eda1.html).
@Neil, you are definitely right.  I guess I should have said it would have to have at least two parameters.
Is machine learning *only* about classification? Can't machine learning be used to serve other goals?
what do distributions have to do with optimization problem? do you first fit a function, then optimize it?
Indeed this is very useful for me. Thanky you both and everyone else very much, your help is much appreciated.
To inspect this visually, you want to look at plots of EW vs A at different levels of MT. You want to find the level of MT where the red regression line is as close as possible to perfectly horizontal. Based on the 3 plots shown, that appears to be the last plot where MT = 0. B/c the slope gets flatter as MT gets lower, I'm guessing that it would be perfectly flat if MT were negative (although I'm guessing that's not possible in reality).