tttjjj/elt_bench huggingface dataset:
@DimitriyV.Masterov That's why I said "Suppose the coefficient for USA remains 1.12 in the revised model." I will edit it to make it more explicit. Thanks!
Actually, see http://stats.stackexchange.com/questions/45050/diagnostics-for-logistic-regression
@whuber. Thanks for your comments. However, I don't understand it very clearly. So, let $X_i$ are i.i.i. $N(0,1)$ rvs, one can calculate the PDF of Z according to the integration.(Using convolution?) When there are constants, $Y_1$ and $Y_2$ are independent, what if there are more than 2 rvs, that is $Z=a_1*X_1+...a_d*X_d/\\sqrt(X_1^2+...+X_d^2)$?
So for example, in component $A$ you are measuring EDA for each person? Another thing you might try is to apply changepoint methods to each individual in each component and find the "average" changepoint.
Do you know anything about the shape of the distribution of the $Y$-values?
Sorry - maybe I'm a being a bit dense, but can you explain further what stitching data frames together means? Is it safe to assume that `process()` is some internal function you've written up yourself? If so, do you need to apply `process()` by each level in the column `name`? Or is there more to it?
I'm curious - how, mechanically, would you go about combining the 1000 sets of results using spss?
Thanks a lot for this additional comment
A meta-comment here as on many, many of your questions is that you seem to be reacting to one (unstated) source. So (1) please give a reference (2) note that anything tied to one specific reference is hard to answer unless it is accessible; even with a mass of definitions, the context can be important (3) try reading something else if one source is obscure to you.
For your first comment, if $f(x)$ is the $N(\\theta,\\sigma^2)$ density, then $E_{\\theta}^{X}(c-\\eta_{X})^2=\\int(c-\\eta_{X})^{2}f(x)dx$.  Since $f(x)>0$ for all $x\\in \\mathbb{R}$, and the integral is zero, then the term $(c-\\eta{X})^2$ must be zero except on a set of lebesgue measure zero.  Well done on the choice of $\\eta$ for the second case.  I am going to write up that part right now.
As it stands this answer is a little brief. Could you expand a little further about how the Bayesian Lasso works and perhaps give a reference which discusses it?
SEM on 25 observations of 15 variables, huh...I can hardly imagine fitting a model like that. I really wouldn't call SEM a *qualitative* analysis, nor would I say it's meant to handle multicollinearity by itself, especially in small samples (see [Grewal, Cote, & Baumgartner, 2004](http://www.personal.psu.edu/rug2/Grewal,%20Cote,%20&%20Baumgartner%20MKS%202004.pdf)).  SEM is too often fit to excessively small samples, producing bad estimates ([Westland, 2010](http://indigo.uic.edu/bitstream/handle/10027/7655/Lower%20bounds%20on%20sample%20size%20in%20structural%20equation%20modeling.pdf)).
That will be a trade-off. Setting a lower value will give you more false-positives but fewer false-negatives, i.e. between flagging some points that are fine, and missing some points that aren't. In practice, a suitable value will depend on the number of points in your data set. I'd suggest maybe trying 4 to start with.
The fact that the values appear in the same input vector already conveys the information that they correspond to the same instance (entity).
+1 for David Howell, because his Intro to Statistics for Psychologists is the best you can get as an undergrad!!
Indeed pretty normal. Good to know. The usual conditions for CLT were never an issue here, my question was whether there were other, perhaps subtle issues that twisted the asymptotic result and required a modified CLT. Your simulation shows that indeed the discrete discontinuity becomes negligible in probability as more variables enter the sum.
By 'such event' do you mean exactly 3 aces, or at least 3 aces?
what kinds of variables did you collect data on?  What types of variables (ie. continuous or binary/categorical)?
What you're asking about centers on the concept of partial correlation, which is dealt with, among other places, at http://stats.stackexchange.com/questions/17336/how-exactly-does-one-control-for-other-variables .
Are `result1` and `res1` in `AIC` and `BIC` different results in your notation?
Why not take a set of inputs where it goes infinite/nan and stop a few steps before that happens and watch the calculations at each step? You'll note that you have divisions and matrix inverses (which you won't do by actual inversion, hopefully, but by solving systems of equations by more stable methods). These are where your problems are most likely to arise.
@Yevgeny, for some reason I don't get emails when comments are made. In response to this, I don't know - why not make it an answer and let others decide? I suppose that one could average the values at each time point per cluster and use a distance matrix.
It makes sense to differentiate between the two groups via the interaction as you did in your model above because the effect for the two groups is different. Basing model and variable selection on standard errors is generally not a good idea because it can be misleading. If you have some theoretical/economic intuition about why the effect differs between the two groups you will be able to make a stronger argument.
This question appears to be off-topic because it is about application debugging rather than about stats.
I was looking to the code and I find that "testing" was not defined yet and therefore the code:
"final.list<-testing[-removed] #removing variables" gives the error: object not found So looking to the code I suppose that instead of using "testing" it should be used "cp.list" so that the code will be: final.list<-cp.list[-removed] #removing variables
final.list<-c(final.list,duplicates) #adding in those vars which were both removed then added later Let me know if this is correct Kind regards
Simulate some data with $x$ symmetrically spaced around $0$ and try again :-).
For (1), the q-value of an individual hypothesis test is the minimum pFDR or minimum FDR at which the test may be called significant. So isn't it that the bigger the q-value is, the more likely the null is rejected?
@Mohammed: if they are not all positive, you should consider whether either a geometric or a harmonic mean is meaningful
@AlecosPapadopoulos I just found it in pdf form, here http://www.colorado.edu/economics/morey/7818/MoodGraybillBoesBook/MGB3rdSearchable.pdf . Feel free to take a look.
(without checking your answer yet): You're not necessarily doing anything wrong, since likelihood is actually only defined up to a multiplicative constant; two people can calculate log-likelihood and get different numbers (but differences in log-likelihood is the same).
The `lme4` package in `R` can estimate mixed effects models with more than one random effect. Because a mixed effects model isn't exactly analogous to a linear model without random effects, there isn't an easy direct equivalent for $R^2$. However, this paper by Gelman and Pardoe might be useful to you: http://www.tandfonline.com/doi/abs/10.1198/004017005000000517
SPSS version 15, unfortunately. While I'd love to use R, or SAS, or another stats package, this isn't an option in this case.
Thanks! Simpson's index of diversity is what I'm looking for! Is there a way to tell if the largest segment is significantly dominant?
What operational distinction are you making about "fake" *versus* "true" growth?  By definition, Google Trends "compute[s] how many searches have been done for the terms you enter, relative to the total number of searches done on Google over time."  And that's that: it's neither fake nor true, it is what it is.
"what this could mean is that your manipulation “moves” the mean of the whole group but in each group those with the highest European identification are not necessarily those who would pay more." Thank you very much for this comment. It really makes sense! For example, it could be that after a certain point in the identification scale, higher identification actually reduces the amount of tax you will pay (just an example)? Or it could be that my manipulation affects tax, and affects how much identify with Europe, but this identification does not impact directly on willingness to pay ...
(1) you might have made a mistake in writing out your question. it appears that the test set metrics are not as good as your training set metrics, as expected. (2) there aren't any (many) absolutes.
What is the concept you have in mind for correlation between two sets?  There are concepts such as testing equality of covariance matrices if the two sets had the same dimension and represented the same variables.  You can always calculate individual pairs for correlation also.  But it sounds like you have in mind some other kind of relationship which may not exist in the statistics literature.
Since no other answered, I will accept your answer.
Thank you again. I have to admit that I don't understand all of what you just wrote but I'm going to use it as my starting point for further study on this matter.
How about specifying a covariance structure to account for heterogeneity of residuals; probably by specifying some `weights` when using `lme`? Not to send you in a wild-goose chase though: Have you used the "homoskedastic" version of your Mix.Eff. model and you saw heteroskedasticity in the residuals it gives? (Also you could probably transform your response variable but it wouldn't be my first pick...)
1) Uniform on what? 2) Just to double check - $\\overline{X^{−1}}$ or $\\bar{X}^{−1}$ ?
Here's a how-to for filtering: http://blog.bjornroche.com/2012/08/basic-audio-eqs.html
My Research Methodology teacher made a comment that she would use ROC for this problem. I still have to hear her out on the details and will report back here then, but any input on this? Why this would be better or worse than the ones you proposed?
If you have trouble you might turn your comment into a more specific version of your question - I suspect it's one that would be likely to get an answer.
Does the percentage of failures for a particular test converge to some number as the number of tests increase? I'm assuming here that test x is done say, 200 times this week, 500 next week and 4000 times the following week (or something like that.) If that's indeed the scenario, you could use the converged value as your error rate.
Wayne My Training data is [1:260].
Thanks for the great answer; how you presented it was very understandable.
There are. A lot. Thus, you need to be tremendously more specific before this question becomes suitable to the site.
There is a generated quantities section at the end that is supposed to be able to handle simulation, but the documentation (as of ver. 1.3, but ver. 2 should be out soon) does not explain in much detail how to accomplish this. You might consider asking the mailing list.
@Ayush. Afraid you lost me on what the integration of these means. Either you could provide a deeper explanation or else perhaps someone else can provide their knowledge.
Have you considered Linear programming (http://en.wikipedia.org/wiki/Linear_programming) for solving this problem?
Thank you for crystal clear explanation. Further study on your explanations leads me to [Clopper-Pearson method](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval). 
There is however a difference: the definition of  [regularized beta function](http://en.wikipedia.org/wiki/Regularized_incomplete_beta_function#Incomplete_beta_function) seems to use $a=k+1$ and $b=n-k$ (not $n-k+1$). I do not know how is correct. However I have an implementation of the inverse of regularized beta function and it seems to give the same results as in Quinlan's book, using formula with $b=n-k$.
I probably would have given an answer like this.  The point that it doesn't matter in modeling is a good one too. We could just as easily use base 2.  The difference is only a constant factor
http://www.biomedcentral.com/content/pdf/1471-2105-10-S1-S22.pdf Using random forest for reliable classification and cost-sensitive learning for medical diagnosis
It's sometimes useful to remember Peter Huber's comment to the effect of: a standard Cauchy distribution truncated at +/- 1 million is indistinguishable, practically speaking, from a standard Cauchy distribution, but all its moments are finite.
A "trivial" observation: If $X_1$ takes values in $[a,c]$ and $X_2$ takes values in $[c,b]$, then whether they are independent or not, $X_1 + X_2$ takes values in $[a+c,b+c] \\neq [a,b]$. Have I misunderstood your question? If not, and there is a way to clarify your question, e.g., by shifting the range of $X_2$ appropriately, I'd be interested in seeing it. Cheers.
@jsk - one theoretical argument to suspect that a NB model will always fit better than Poisson, is that you can write NB as a poisson-gamma compound distribution.  So you have $( y_i|\\lambda, v_i, r)\\sim Poisson (\\lambda v_i) $ and then $(v_i|\\lambda, r)\\sim Gamma(r, r) $ gives a negative binomial model $ ( y_i|\\lambda, r)\\sim NB (r,\\frac {\\lambda}{r+\\lambda} )$.  Now the addition of those $ v_i $ parameters allows the model to make the predicted mean closer to the observed value (when $ y_i>\\lambda $ you would see $ v_i> 1$ , reducing the residual.)
Could you please add some explanation concerning how this answers the question?  What specifically are you proposing as the answer?
Are 2 downvotes due to the fact that I cross-posted? I did read meta stackexchange on this topic, and it suggested it's ok if I link the questions and wait for a reasonable time before cross-posting. Given that the question was downvoted on the other forum too, I'll delete both of them. I just joined stats.stackexchange, and feel really unwelcome here.
Ok that makes sense to me.  I was a bit confused because while reading some books on Cox regression, some authors occasionally log-transformed variables in their examples but never really mentioned WHY they were doing it.
Your reference is broken - you haven't provided a URL/link.
I agree with you. Unfortunately there seems to be a discrepancy between advances in psychometrics and their reception by more applied fields. If you are interested you can read [this article](http://link.springer.com/article/10.1007/s11336-006-1447-6/fulltext.html). Also, I found a [freely available source for CTT](http://www.personality-project.org/r/book/#chapter6) on [this site](http://www.personality-project.org/r/book/).
I like your approach. However, to get a more general answer it may be worth to take a bit larger symmetry group - identity, 3 rotations and 4 reflections (i.e. $D_4$, http://en.wikipedia.org/wiki/Dihedral_group). Then count differences ($d$) between all pairs (i.e. $8* 7$) and as a measure of randomness $r=k\\frac{1}{8*7}\\frac{25}{2n(25-n)})$, where $n$ is the number of black stones. For purely random shapes one should get $r\\approx 1$, while for very symmetric $r \\approx 0$. The good thing is that the formula for $r$ holds for different number of stones on the board and has the B-W symmetry.
First of all, you need to find the sequence $x_n$ approaching x in order that for sequence $f(x_n)\\rightarrow f(x)$ when $n\\rightarrow \\infty$. Otherwise, it is not easy to find a "good approximation" for function $f(x)$. Anyway, density distribution  for $f(x)$ (this is not dist) should be given if you are going to talk about probability for it. Right?
@gung I agree, I was thinking yesterday after I commit the message that those tests won't work well for continuous variables. However I do not know if deleting a response after publish is a way to go, so I will simply correct that
@bayerj Good question. The prior is MVN, but the likelihood isn't.  I think that limits my options.
Apologies for not knowing what I'm doing, but I was hoping that a model wouldn't fit all values of df$tonnes exactly onto the regression line. I thought it would instead adjust all values of df$tonnes by adding the mean residual for a particular dist_fr_middle to each value at that dist_from_middle
I think that it is a textbook question, which sometimes gives this type of information claiming it to be normal, especially if it is a social science statistics book...
I don't really understand why this option was added to `hist()` (`labels=T`) and not `barplot()`.
Please accept your own answer. Also: maybe you could de-abbreviate NPD and VIF. It could also be useful for the potential google searches if you wrote which software you were using.
Random effect is a very vague term, that means different things to different people. But yes, they can be the same.
Did my answer answer your question? If not, what else do you need to know? If yes, please check the accept button below the voting arrows.
Generally speaking, merely deleting/ignoring incomplete records is not recommended. More details would allow more specific advice.
What is your null hypothesis? What is the difference between a control and a case?
If the $F$-statistic can be computed in the Wald form, correction for clustering can be made. See also [discussion on statalist](http://www.stata.com/statalist/archive/2009-09/msg01179.html).
Though I have not mean normalized it, my values lie in between 0 and 1. That is not the problem. I will ask in the forum of the library too.
Bootstrapping is factored out from the command.  It should not care what the variables are.
I am facing the same problem. I think mlogit does not allow offset as you can see here : https://github.com/cran/mlogit/blob/master/R/mlogit.R so it ignores the offset variable.
It would be more suggestive to write your formula as $\\binom{4}{2}\\binom{1}{1}\\binom{n-5}{m-3}/\\binom{n}{m}$. Note that you must assume the variables are [exchangeable](http://en.wikipedia.org/wiki/Exchangeable_random_variables).
You'll need to say more to clarify what you mean by "outlier" in this context for this question to be answerable. I wonder if you are thinking about noise points in DEBSCAN clustering?
@DimitriyV.Masterov is correct, see a pretty straight forward discussion here on the Stata blog, [Use poisson rather than regress; tell a friend](http://blog.stata.com/2011/08/22/use-poisson-rather-than-regress-tell-a-friend/). The expectation of the exponentiated error term *is not one*.
+1 for two reasons: first, it is the only reply (so far) to explain the merits of its nomination, and second, despite the barrage of vague business-speak in the blurb ("data supply chain," "data intelligence," "hardcore," "leading edge"), a look at the conference schedule, the topics, and the presenters reveals the involvement of substantial, serious players from government (NASA, e.g.) as well as industry.  It's very top-heavy in presentations from software and services purveyors, though, suggesting an orientation toward marketing commercial solutions.
As an additional note, if you do `p.adjust( c(0.3803, 0.3577, 0.03746) )` in R then you see adjusted p-values that are more in line with the Anova result.  Just confirms that this is an example of inflated alpha from multiple comparisons.
+1 like for the answer.
Are your patterns global or local in time, i.e. if you have a flat signal with triangular spike at 10s, you would call it similar to a flat signal with rectangular spike at 10s or to a signal with exactly the same triangular spike at 500s?
Argl. I may have just found the cause: as most forms of DA assume (?) multivariate normality per outcome class, they don't make much sense for categorical predictors (i.e. factors), right? This may be turning into more of a stats.stackexchange question afterall...
Thanks Sebastian! Your answer seems correct. But waiting for some other relies as well.  :)
Also note that extreme values are the least robust statistic. They are maximally sensitive to outliers. [wikipedia](http://en.wikipedia.org/wiki/Sample_maximum_and_minimum). First of think of why you want to use them. A much robust alternative are deciles or some more general quantile of interest that can be as close to the extrema.
To get the behavior you wanted, try using a simple CART.
@László - I added a bit more regarding the local regression aspect.
@Tomas: Please read my comments. The WinBUGS model is a Poisson-Log-Normal model and *not* a quasi-Poisson model. The dispersion parameters cannot be compared, they will never be the same. So don't try to get 1.47 using WinBUGS, you won't. The overdispersion parameter in the quasi-Poisson glm is calculated as follows: `1/(n-k)*sum((C.OD-fitted(glm.fit.no.OD))^2/fitted(glm.fit.no.OD))` where `n` is the sample size and `k` the number of parameters (i.e. 2).
When using PCA, keep in mind that it's a linear transform. That means that many kinds of data might actually end up being even less useful after PCA-ifying them. In those kinds of cases, techniques derived from expert knowledge on the structure of the data are almost guaranteed to outperform PCA-based approaches.
@Eric Dong: I think the distinction is more that the probability that the flip you just made landed heads is 1, but the parameter in the underlying Bernoulli distribution, $\\theta$, is - and was - 0.5. "That coin *had*" means we do not condition on the actual outcome..
If you do not have any pairing of the X's with the Y's, then what information *do* you have about their relationship?
Do you just want the pairwise euclidean distances in PC scores for all languages (based on the first two PCs)? If so, do you just not know how to do this in your stats program? It will help if you can provide more details, such as what software you are using.
This question appears to be off-topic because it is about how to do something in R, & not about any related statistical issues.
I thought Jenk's method of optimization *was K-means* in one dimension (see this comment by [whuber](http://stats.stackexchange.com/q/26700/1036)). I believe there are multiple "Jenk's" methods though, so it could be some confusions around this. Your second comment also seems to be confused, if you use K-means clusters in one dimension it doesn't assign groups in non-contiguous rank order.
But it seems to me that the problem your model is trying to solve is actually different that you described. If w was a variable which we do not observe and we are trying to figure out what the parameters of w's distribution are then w would be a latent variable. Parameters you would be after would be the one that describe its distribution. I'm not sure if that helps, but hopefully gives some food for thought:)
On its face, this procedure is invalid because $E/P$ is emissions per capita, $E/A$ is emissions per unit area, and $E/G$ is emissions per unit of economic output: the three are incommensurable, so adding them is not well defined. (You would get different results merely by changing GDP from pounds to euros, for instance.) However, incommensurable values *can* be added (with weights) in order to *rank* the cities. That's a question of *your* valuation; it cannot be answered with the information given. Please see http://stats.stackexchange.com/questions/9358 for more about valuation.
@ttnphns: that is both correct, yes.
+1 I like it when more is given than just a reference to previous questions and answers.  You are being very explicit, presenting concepts in simple terms and being helpful.
Is your continuous variable really continuous or discrete? That is, can it take values such as 2.5, or only 0, 1, 2, ...?
I think this should be treated as two independent samples, since you cannot estimate pairwise differences. You can check normality assumption and use t-test.
Nice. Along the lines of your 2nd-to-last paragraph I'd recommend Fooled by Randomness or The Black Swan by Nassim Nicholas Taleb.
When you do regression in any common statistical program, you get a bunch of standard output. This will include F test for the overall model and t-tests for the individual effects. For example, in R, the syntax would look something like

m1 <- lm(purchase~research, age, sex, age*research, sex*research)

in SAS, you could use PROC GLM.