tttjjj/elt_bench huggingface dataset:
I would prefer using Python. It seems Pandas is a useful addon. Yet, I hope some mathematical understanding together with some library functions will do the job. So it seems to me programming language is not so much of an issue.
A pleasure to serve. :)
I can't imagine a better place for such a question, although it might not fit the usual frame. E.g. it is to be remembered that newer Intel chips provide PRNGS in HW, which might speed up most MCMC computations, and it would be interesting to know about AMD.
lm, as you used it, is minimizing the sum of squared residuals, i.e. $(y - a - b*x)^2$ for each data point, called OLS (ordinary least squares). I couldn't find a nice picture for linear OLS, but maybe [this one](http://upload.wikimedia.org/wikipedia/commons/1/17/MDKQ1.svg) is still illustrative to you. OLS is minimizing the square of the green lines, lm is doing this with a line. In comparison, look at a  [linear total least squares picture](http://upload.wikimedia.org/wikipedia/commons/8/81/Total_least_squares.svg).
To be able to be averaging in the first place, you have already decided to treat your variables as interval scale. It doesn't become less interval-scale after averaging.
The function `predict()` uses by default all the components. However, you can limit the number of components that are returned, e.g., `predict(pr, USArrests)[,1:2]. Would that work for you?
Well you should separate your training dataset in train and test and see for yourself which classifier has the best accuracy, f1-score etc... You do not need manual checking.
Well, the vocabulary isn't completely clear. According to (this paper)[http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2819361/] confounding and mediation are the same thing. MacKinnon is one of the gurus of mediation, so his word is probably good.
No idea. I'm afraid you will have to look for the original data contained in Berger's *Experimental design with application in managament, engineering and the sciences* yourself.
(+1) Good overview and nice, illustrative counterexample. It is interesting that if we sample from $p(x,y)$ we can recover a pair of iid standard normals $X^\\star$ and $Y^\\star$ by letting $X^\\star = \\epsilon_1 X$ and $Y^\\star = \\epsilon_2 Y$ where $\\epsilon_i$ are independent random variables with $\\mathbb P(\\epsilon_i = \\pm 1) = 1/2$.
Absolute sign is meaningless, but relative sign contains the same kind of information as the distinction between a negative correlation and a positive correlation. This may become clearer if you consider doing PCA on two variables rather than millions :-).
Sorry for leaving out the details. CEP - stands for circular error probable or 50% of the population should fall within certain parameters defined.  I have defined in a 95% confidence interval and 3% margin of error for my attempt to solve this problem
How did you test the increase for significance? Was a control group used?
Hi, I'm a bit lost by your question. Normally we compare groups on the basis of outcome (percent tumour reduction?) directly, so I'm not sure why you mention comparing p-values - we get one overall p-value for the comparison. Could you give some more detail about the variables you have rather than the methods you're thinking of?
Both age and education influence this cognitive scale performance. So do you think a low Durbin-Watson in my regression may represent a problem due to correlation between age and some other factor (like education)?
@Benjamin Could you explain how that solves the OP's problem?
I am afraid I cannot help you with code. Stochastic frontier models are almost always estimated by Maximum Likelihood, after making a distributional assumption. The symmetric error term is assumed always normal, and the asymmetric error term is assumed, half-normal, exponential, or gamma. The density of the composite error  has been derived for these cases (in the case of a half-normal it is  a skew normal density), and researchers implement them directly in the context of ML estimation.
Some ideas: logistic regression? Elasticnet with categorical variables?
(+1) Better explanation than mine!  (as usual...)
@whuber About the time series, I've tried to use lagged versions of the predictors (around 0.10 correlation for some) but they don't give me a significant prediction improvement. The seismic measure is represented by a linear scale calculated aggregating a number of factors. But in general my question is about robust regression and the Huber estimator when facing a low fit model.
Can you create an animated gif from the matlab plot at different angles? I do this in R with the animate() package.
All this means is that your events $A$ and $B$ are not independent. So knowing that $A$ has occurred tells you something about the probability of $B$ occurring, and vice versa.
Take a look at Ash's "Probability and Measure Theory", second edition, section 1.6, problem 3 (answered at the end of the book).
They're some great examples. For anyone who is interested, the letter to the editor with the figure is here: http://onlinelibrary.wiley.com/doi/10.1002/bimj.200800256/abstract
Deciding the specific covariates to include in a model does commonly go by the term model selection and there are a number of books with model selection in the title that are primarily deciding what model covariates/parameters to include in the model.
Thx for the quick reply, if i have understand it right the one-step forecasts are then provided by
splitting a time series in two sequences: [60 Periods] t1<-[36 periods] fitting sample t2<-[24 periods] test sample.....both tseries objects. fit the model within the first periods: model1<-ets(t1) then using the model to the out-of-sample data t2: test<-ets(t2,model=model1) and the one-step-ahead forecast with seeds in t=37,38 is preserved by
fitted(test). am i right? or how can i get the one-step forecasts to an test data-set
using the model i estimated from a fitting data-set. thx in advance
T
+1, nice answer! Welcome to CV, we hope we'll see more of these in the future.
An explanation of your notation would help people understand what you're asking.  What are "R" and "[K]"?  Note, too, that there is no such thing as a "correct" loss: that is something adopted beforehand by the decision maker.
@cardinal Thanks very much for the offer, but I'm fairly happy at this point - I awarded the bounty to egbutter. I think I've got the intuition. My main purpose in posting was to see if someone jumped in and said "No no no you've got it all wrong because of..." :-) Cheers.
It is always possible to interpret alpha in terms of a quasi-average correlation, and vice-versa. alpha/(1-alpha) = k*r/(1-r), where k = the number of items and r = (average covariance)/(average variance). From this we get alpha = k*r/(k*r + 1-r) and r = alpha/(alpha + k(1-alpha)).
@benrolls and EMS>> In this case the standard deviation under the null is 315, meaning if the null is true you'd expect about 95% of runs to give resulting in the range $1.0 \\cdot 10^5 \\pm 730$. The normal approximation will be fine. (In fact, if you're happy with a 5% significance level, that's your hypothesis test right there.)
Feel free to ask that as a separate question, the comments are not really meant for dialogue and follow questions like this. If you think my answer was valuable, please upvote it. If you think my answer was correct, please mark it as the accepted answer. Thanks!
Calculate some characteristics of shape for each time series and then cluster on them. One simple solution would be to do principal component analysis and do clustering on loadings on first few principal components.
The correlation will stuff up the variance of the mean; this may be *much* be more important than the issue of non-normality (because it doesn't go away with increasing sample size).
Did you switch the definitions of $\\alpha$ and $\\beta$ partway through your post?
Can you make your dainty buttons clickable? :-)
Number of features to use *for what purpose?*  What is the context?  (In its current state, this question is to broad to answer.)
Hmm... that is interesting. I'd love to see his or her explanation.
(+1) Can you provide a link to the mentioned book ?
@Skolnick, I've missed that at first, so I changed my answer accordingly
@Fred123 Please register an account -- then I will be able to merge it with the vapor user that owns this Q, so you'll reclaim it.
*"it's not the case"* - I don't understand what you are trying to say here.  Can you re-phrase?
You may want to look up the different kinds of sums of squares. Specifically, I believe linear regression returns type III sum of squares, while anova returns a different kind.
The fact that it is a copula does not matter on principle. You have the joint distribution of $(X_1,X_2,Y_1,Y_2)$, from which you can derive the joint distribution of $(V,W)$ by a linear change of variables.
$\\sqrt{x}$ works even better than $\\frac{3}{2}\\sqrt{x}$ because $\\sin\\sqrt{x}=\\sqrt{x}-x^{3/2}/6+O(x^{5/2}) \\le \\sqrt{x}$ in the interval $[0,1]$.
@jorges The sum will be normal and therefore is symmetric about a single mode.  When the two means are sufficiently far apart (compared to their SDs), the mixture distribution is *bimodal*.  In no circumstance is the mixture normal except in the degenerate cases (equal means and equal SDs).  This is also apparent from the formula for $\\varphi_m$: this average of exponentials cannot be written as the exponential of a quadratic form.
@Unwisdom well, given a CR you can always do the hypothesis test: $\\theta \\in CR$, then take the data, form a CR and check if your parameter is in the region or not and reject or not based on that. BTW: can you give a practical example of a non-measurable acceptace region generated by a test?
hi thanks for answer but I am not interested in logistic regression but in general model. While all the examples are related to logistic regression I am wondering if there is something similar for others classification models such as decision tree and svm. Sorry if the question was not clear
Eating lots of oranges and no apples makes one thinks so.
it's not a survey. I've just calculated the benefits of an irrigation system and tried to change the inputs by a certain range  to observe how the benefits would change therefore my output is Y=benefits and I have about 50 repetitions (changing 8 inpupts 6 times)
KL divergence is an asymmetric measure of the difference between **two** distributions. What is the second one in your question?
I don't think "good fit?" is easy to answer without knowing purpose (Maarten's point) and without knowing what other distributions you tried. Did you look at the gamma distribution as a more general alternative, for example? Note that recasting a distribution fit problem as a regression problem is a little dubious: points on the distribution function are hardly independent. But the graphical evidence is clearly pertinent. As you have systematic departure from the exponential, two-parameter distributions should give closer fits, but you must judge whether that's worthwhile.
If you would like a relevant explanation, you will need to tell us more about how this t-test is applied. At least please provide a link to the paper if you can.
@fhucho: You can build your probe set by taking the last 3 ratings of each user, and as mentioned above, leaving out movies which occur only in the probe set. After validation, as alessandro-negro suggested, you can build your model using all data points.
Welcome to the site, @M.Buchler. Your post is being automatically flagged as low quality, because it is a little short, but I think it is a real answer. Would you mind expanding a little on the implicit argument behind this answer?
I am not sure this deals with the "probability mass function" part of the question.  You might need to translate $R=P+Q$ into something like $r(n) = \\sum_i p(i) q(n-i)$
@Joe, I get that. But I'm old fashioned, and when I studied pure mathematics, back the day, infinite series were always addressed as the limits of finite series. So concretely, you are always dealing with a finite series. But even if you could take an infinite series, typically those convergence results only apply on a compact set. Outside that region, the approximand could differ a lot ielike feedback loops, tipping points and singularites, are essentially non-linear behaviours, which perhaps deserve non-linear models.
I was able to get results similar to  your initial pathological results by putting in certain 'init' values with the example dataset. That was when I started asking about initial values.
Actually what I meant is inter-subject within the same object. I'm not trying to compare the users.
How you would create it is probably more of a [Stack Overflow](http://stackoverflow.com/) question, but at a minimum, we would need to know what software you would use.
Could you ("hold my hand") and elaborate.  I don't see it still even if i plug in 1/u.
@user1275515 no problem. Glad that helped.
Do you suggest that using linear regression after obtaining factors from Categorical principle component analysis (CATPCA), is the best option? means is it right to convert categorical data into continuous by CATPCA and run linear regression (not logistic) to find relationships.
1. There is no reason to develop "standard testing" for the Bayes factor. It is a quantity that reflects the ratio of evidences, not a _p_-value. 2. No symmetry: $\\log B_{12}=-\\log B_{21}$ but one side is integrated in $m_1$ and the other one in $m_2$...
For now, the audience is me. I'm statistically informed and want to see the raw data. The key issue is the binary/categorical nature of the data--I'm not aware of anything that mentions anything one might want to do differently if the data are binary/categorical. I don't think Tufte really discusses this either. If I remember right, he mentions the tick plots in a general discussion of sparklines, but doesn't really discuss them much.
Did you try to transpose one of the two matrix of loadings? Because, it seems MYRESULT ~ t(R_).
But that results in a ridiculously large number (7765253). As you can see, I'm confused...
At the end I have R values for four technologies. However for example minimum possible R value in technology A happens because of increase in feed price, while the minimum possible R value for technology B happens when the cost of fuel increases. It means that all minimum values of the criterion is not because of the same reason and so we should not compare just the minimum values with each other and maximum values with each other, there can be a possibility to have minimum value of this criterion for technology A when we have the max for technology B (I am comparing the error bars).
" Roughly speaking, this is the prior for which the Kullback-Leibler divergence between the prior and the posterior is maximal." Interesting, I did not know that.
I have only the Twitter topics and a count of tweets referring it. Currently I can predict events using the Grubbs test and frequency thresholds. But.... the thresholds are set but hand. So my main problem is how to use ML for such a problem.
@whuber, that's right.
Just to be sure, if I want to use for instance the "wilcox.test", I would treat all students from a semester (3 different exams) as one group?
This is a great answer.  One of the vignettes for the package even goes through and answers a bunch of questions from the Train book.
Using $\\alpha = 0.98$ you also get the mean = 4.98, which is equally useless. :) Using such coefficient, you put almost all weight on the last measurement. More realistic values of $\\alpha$ are close to zero, in that case they account for long-range average. For your example, try $\\alpha = 0.2$, but in practice you will probably need to average more measurements, so the values around $\\alpha = 0.01$ are more realistic.
Great example, thank you.  So Markov Chains concern themselves only with the number of transitions, not their placement, correct?  For example, would `AAABBBA` have a a same matrix as `ABBBAAA`?
+1 The plots make it clear that some such transformation would help and that the logarithm is too strong. This one ought to work well as a point of departure.
Please show actual attempts - show any working and clearly identify where your specific problems are. From the tag wiki:
• *Make a good faith attempt to solve the problem yourself first. If don't seem to be making a genuine attempt, your question might be voted down or closed.*
• *Ask about specific problems you have encountered in your initial efforts. If you can't do that yet, try some more of your own work first or searching for more general help.*
I appended my thought in the answer
I missed that point. Your answer is indeed correct: you only need to add a constant to a confidence limit for the coefficient of variation $\\mu/\\sigma$. (+1 to your answer for that.) However, I have long wanted to have an exposition of the basic theory of tolerance intervals somewhere on this site, so if you don't mind, I would like to keep my answer open: in that way this thread will provide a reference for several kinds of tolerance intervals.
Could you, just for my personal interest, make a couple of citations to those papers? (+1)
(For statistical reasons, slightly different normalizing factors often show up, but hopefully the intuition is clear. If you're still worried about "small" standard deviations, ask yourself how the standard deviation and the correlation each change, respectively, if you replace $x_i$ by $a x_i + b$, for example.)
It would be helpful to provide a full citation in the text of your answer in case the link were ever to break. Also, it would be good to be sure that the link provided meets copyright and fair-use guidelines.
Why isn't the size of the training sample fixed?
@whuber I think min Xi and max Xi are sufficient statistics for the uniform distribution. Wouldn’t that mean that using other order statistics can’t improve things?
No you calculated the ML 95% CI for the mean wind speed but the problem asks for the interval for p. You calculated the interval for the wrong parameter.
if I have _n_ [**dependent** groups](http://stats.stackexchange.com/q/11887/5003), do I have to check normality for _n_ batches of residuals or just 1 time for all residuals?
@Andy McKenzie Re #1: That's what I am currently doing (i.e. IPD meta-analysis). Given that my outcome statistics come from regression models, this seems the most useful approach to me. I know that you wrote "most of the time" :-)
No, if you had F fine classes and C coarse classes the network would hvae F+C classes. each  training example would then have 2 labels. The goal would be to try and predict both the coarse and fine label correctly at the same time. In this way you might be able to learn both tasks better than if you trained these separately. A decision tree could also work.
@euphoria83, sorry I didn't mean to be obnoxious or anything. I've edited my answer.
Mann-Whitney and its direct extension to multiple groups Kruskal-Wallis do _not_ require the distributions be of the same shape. The tests compare gravity locations, assess stochastic dominance. But if you choose to add the equality-of-shapes assumption then the tests are about the difference in shift (i.e. in any quantile's position).
The joint distribution is a multivariate density -so the phrase "the joint distribution of $y_t$" makes no sense. Do you mean "the joint distribution of $t$ $y$-random variables"? And what you found, is it perhaps the marginal distribution of the _single_ r.v. $y_t$? Because it appears to be univariate, not multivariate.
You aren't actually making a distributional assumption when calculating a Pearson correlation; it just measures the strength of a linear relationship between two variables.  You could have a bivariate Poisson dist'n or a bivariate Normal distribution (or one Poisson, one Normal)...
@ Glen_b I mean I still have the b1_hat floating around. I guess if I know b1_hat i could plug values in after expanding the expression but it would not be very nice. Sorry I am just not seeing it even though I actually spent about 3 hours today trying some different, and probably very wrong, methods.
This does not appear to be a contingency table setting, but rather a test of a uniform distribution.  The usual $\\chi^2$ test, using the permutation distribution of the null to compute the p-value, will be one of your best bets and actually has some appreciable power to detect variation with such small datasets. (An adequate answer to support this opinion would require an extensive simulation study, which I have only partially carried out.)
Questions about obtaining particular datasets are off-topic here: please see the Help Center.
I may be struggling with your notation and your language.  "P(X is from population 1)" would seem to be *what you are trying to find,* and so you do not appear to obtain a solution at all. I do not see any place where you refer to the prior probabilities of the two populations. Part of the struggle may be that "$X$" has no explicit definition. Another source of my difficulty is that "no prior knowledge" means you simply don't know what the prior distribution is; it does not automatically allow you to assume that both populations are equally likely.
@rolando2, the above edited question is exactly what I am looking for and I have added an additional question, can you throw some light on that.
Great! I will look into it.
Actually belongs to stackoverflow (purely programming related). Why is the off-topic-belongs-other-site-button gone ?
The [Durbin-Watson statistic](http://en.wikipedia.org/wiki/Durbin%E2%80%93Watson_statistic) is a test for serial correlation: that is, to see whether *adjacent error terms* are mutually correlated.  It says nothing about the correlation between your X and your Y!  Failing the test is an indication that the slope and p-value should be interpreted with caution.
I like the coniditional idea, but I must admit I'm confused by Gaussian couple answer. $\\psi_1/(1+\\psi_1)$ isn't restricted to [0,1], so I don't see how it applies.
For further elucidation, Wikipedia's article on Accuracy and Precision would make good reading for the original poster.
In what sense do you have a measured value and a known true value? Is the response yes/no, or something like that?
In this case, using SPSS I selected "Data/Split file..." and entered the grouping variable. Is this correct? This way, I found a slightly significant ANOVA (p = 0.044) for the control group, but Bonferroni (it doensn't allow me to do Tukey) comparisons are all non significant... How should explain this? Is the ANOVA result a I type error?
I think this oversimplifies the "aim" of Fisher's $z$, which is partly a matter of purpose as well as mathematics. Skewness or not is only part of the picture; $z$ transforms a bounded distribution to an unbounded one, which is important for confidence intervals. In fact, I would argue that unless a null hypothesis of zero correlation is the scientific question too, the use of Fisher's $z$ for confidence intervals is much more fruitful than trying to get a P-value.
@ Bob.  Without knowing the problem context I can't tell what scale you should be using.  This is a caution anyone using Box-Cox transforms needs to consider.
Could you post example papers with: "I've seen many different approaches to this - using Chi-squared tests, t-test, ANOVA with post-hoc testing etc."? I'm really interested in that.
If you have a well-defined criterion (for instance maximizing precision) then this can be automated.  But a good tradeoff for one problem might be lousy for another.
OK @meepmeep, then couldn't it be that you have more variables than cases? It leads to singular matrix, not generally appropriate for FA. If not, other exact collinearities between your variables are also probable.