tttjjj/elt_bench huggingface dataset:
Thank you for the reference. However i currently do not have access to jstor. Thank you for the idea as well about partitioning the chi-square.
"Commonly accepted" where? To use EV as a principle under *all* circumstances would be to state that risk aversion was somehow a mistake. What makes a decision 'right' is if it respects the individual's own preferences relating to risk and money. If losing one dollar means I can't make my rent (which I have to pay today), I'd be insane to play. But my neighbor has a few hundred dollars to spare, and losing a dollar will mean almost nothing to her, so maximizing expectation would make much more sense in her case, were she so inclined.
Closely related: http://stats.stackexchange.com/questions/1850/difference-between-cohens-d-and-hedges-g-for-effect-size-metrics?rq=1

That said, I believe the initial definition for Cohen's d didn't specify how to compute the SD, so the de facto formula (with the biased version) might just be a historical accident.
slightly simpler would be: `Response ~ .^2 -1, data = DF`, but excellent point!
If I am not mistaken, you can't estimate FDR, you can control it. There are ways to determine local fdr, but it isn't the same as FDR.
You probably don't want to use ordinary least squares on this problem - it's for the case where only the y's have errors. If the x's have errors as well, it's biased, sometimes quite badly. Good search terms include `errors in variables regression`, `Deming regression`, `Model II regression`, `Total least squares` and `orthogonal regression` (which are not all exactly the same thing but are related in one way or another). Searching on those here and more widely may help you identify some more appropriate approaches.
I'll give a couple of links here, see e.g., http://stats.stackexchange.com/questions/40542/how-to-report-the-results-of-fishers-exact-test-for-larger-than-2-by-2-tables?rq=1 and http://yatani.jp/HCIstats/ChiSquare. Much of the specifics of reporting depends on what you are aiming at. But including the table and the test results (as shown in the documents behind the links) would be the minimum, I would say.
Seems like a reasonable question.  Only the 'etc' makes it really vague. 'What should I read on' and 'Where can I find code to do' questions seem to be pretty regularly asked and answered.
(+1) This is by far the easiest way to obtain moments of the lognormal distribution.
@PeterFlom, these general strategies certainly might work (+1), but I think it's worth noting that if someone fits a model, finds it non-significant, & then does some of these things & tests again, the p-value is no longer valid.
Do you only want to have someone review and polish your code? If so, this is not about statistical analysis per se and is off-topic here. The code review SE may be a better fit. You can flag the post to have a moderator migrate it. If there is a statistical question lurking behind, please edit the question so as to bring it to the forefront.
This is impossible to answer as is. Please provide a *lot* more context. What are your variables? What models were tested with SBIC? What did autocorrelation and partial autocorrelation look like? How much serial correlation is there? A plot of the variable would also help.
Thanks, exactly what I needed. The reference to derangements was quite useful.
Re: "how to paste a block of R code". You have access to a [small menu bar](http://i.imgur.com/jIAiJ.png) when editing your post.
+1, @Richard. I've deleted my previous comments.
It doesn't look like there is a trend at all (a spline wouldn't help with that)
Several of the questions under "Related" in the sidebar to the right -> are duplicates or near-duplicates. e.g [here](http://stats.stackexchange.com/questions/26024/moving-average-model-error-terms), or [here](http://stats.stackexchange.com/questions/64033/how-do-i-estimate-the-e-t-from-a-moving-average-model)
If you define loss = -reward, aren't they the same?
thank you very much for your response. I do have an option to run separate regressions. I was actually trying to replicate what a consultant's "black box" did for us, and I was unable to proceed any further. I like you blog http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/ I try to adopt your suggestion as much as I can. Thanks
"*I would like a "smoking gun" to say YES, YOU HAVE THE RIGHT FAMILY*" - nothing will tell you this. The best you can hope for is a family that is not clearly wrong. There are many ways you can choose a distributional family, but in general it tends to involve a combination of a priori or theoretical considerations and the indications from the data itself.
Based on your notation I don't think it's what I'm looking for. My features $x$ aren't binary (they're counts in a given time period), but I have a binary split of observations. I suspect that each subset of the data tends toward a different part of the feature space, and I would like to characterize that difference. Since the feature space is high-dimensional, I turned to dimension reduction. I asked about PCA because I'm most familiar with PCA.
`left-handed individuals are more likely to be homosexual than right-handed individuals?` Yes. `a person who is left-handed is more likely to be homosexual than heterosexual` Ooh.
I might just mention the obvious thing that there's no wonder or surprise, because comparing groups and correlating two variables (at least one of which is the grouping one) are two sides of the same coin and they are counterpart, being a significance test and a effect size, respectively. So there exist many examples of such "pairs". Most evident, perhaps, is ANOVA and Eta.
Generalized Pareto distribution (GPD) is a good model to accomplish this
@JohnJiang Thanks for your response.  I'm not that familiar with the differences between supervised and unsupervised, so I wasn't aware that you could only use some of these techniques if you didn't know what class they really belonged to....
Hello, I'm very new in time series, but yes I'm trying to make an AR-model from the data and my goal is to forecast the NO, NO2-quantities for the future =) Hope this clarifies a bit, thank you for your answer! =) I'm using MATLAB to do this
Thank you for your interest in guiding me. I will try this and get back
(-1) the answer is a) partly so generic, it could be applied to any classification question b) it is not explained why PCA is recommended (over any other dimension reduction technique).
If the value of the Spearman correlation is meaningful for you, then certainly calculate it. (If you're just after some general measure of monotonic association, you might like to consider the Kendall correlation which might be more intuitive in interpretation).
@Alexis can you point me a textbook where I can find this one because I didn't understand why the first equality is true
In addition to Mike's answer, [this post](http://stats.stackexchange.com/questions/56645/what-is-the-fastest-method-for-determining-collinearity-degree) might also be of interest.
The presence of errors in the values makes it hard to say much of anything because their comments may be based on numbers other than the ones either of us are looking at. Maybe they made yet another table between the original figures and the published table and their comments refer to that - if they are so careless, how could we tell what else may have happened? You'd need to ask the authors.
Given that you have equal degrees of freedom, and these are all within the classes of GLMs, I would use the model with the highest likelihood.
Although this didn't completely answer my question, it lead me to a possible answer : http://math.nist.gov/MatrixMarket/data/misc/xlatmr/xlatmr.html
How can one smoking status be regressed upon another? At the individual level, isn't someone either a daily smoker or an occasional smoker? Are you aggregating these data up to some level to obtain proportions of individuals who are of either status at each "group" level? What's the scientific question here?
@Kjetil start at http://stats.stackexchange.com/editing-help and follow links there for further information if required.
You say you have surveyed your entire population of interest, but also refer to possible temporal variation.  This suggests that it might be helpful to define the population of interest not only in terms of which individuals belong but also in terms of temporal limits, implying that only a sample will have been surveyed, and re-interpreting temporal variation as a form of stochastic variation.  If you were really interested only in a set of individuals, each at a defined point in time, then temporal variation would not be relevant.
I have some thoughts that I hope will help add context why you're making some decisions in about your research. Keep in mind that your $\\chi^2$ results will also be sensitive to how you transformed the continuous variables into categories. Rebinning could change the kinds of results that you obtain. Generally it's best practice to keep the data in the original units, so that the texture of the data is retained. Also, it's unclear to me why you want to do a $\\chi^2$ test when you have binary outcomes and continuous predictors -- isn't that what logistic and similar regressions are for?
[Separation](http://stats.stackexchange.com/questions/11109/)?
"So for a., the probability of G I get it just by summing p(B,G) + p(G,B) = 0.5" This is not correct. Remember, you neighbour has told you he has **at least** one child who is a boy. What options does that rule out? What then are the odds of 1 girl?
You use weights when you want to obtain a mixture of answers not conditioning on covariates (e.g., average over males and females where oversampling was done on females).  If you want conditional estimates (e.g., sex-specific) then it is a good idea not to use weights.
can you pls be a bit more specific, multivariate logistic regression is a special case of GLMM. AFAIK.
Hi Ciri, can you give some examples of the metric's you refer too in the first part? If I understand correctly though the alternative that you show will just illustrate that the small eigenvalues Principal Components add little to the total variation which is a nice graphical way of validating the procedure we use currently. My point is that even if these eigenvalues are small they may still be contributing important information  and I would like a way to test for this?
@kwak: Least squares!  The standard example of a compact subset is a finite set of points and the L^2 norm sums the squares of residuals.  With measurable compact subsets you need to *integrate* the squares of residuals.  There's also a classical L-infinity (sup norm) theory of approximating functions by polynomials (Weierstrass' Theorem) or even with sets of arbitrary real powers (Muntz-Szazs Theorem).  Rudin's textbook on real and complex analysis has an accessible account (I'm looking at a 1974 printing).  I believe these extend from R^1 to higher dimensions.
If the pairing makes sense and you retain the pairs when you generate bootstrap samples the bootstrap distribution under the null hypothesis that the paired differences have 0 mean should give approximately the same result as the paired t test (but not necessarily if the differences are not normally distributed). The result you get will be a Monte Carlo approximation to the bootstrap and therefore has a small component of error in approximation due to having B (number of bootstrap samples finite).  There could many possibilities for differences.
@Brani Try `nrow(na.omit(cbind(y,x1,x2)))` (following your notation) or `length(lm.fit$model$y)`, to get the number of observations used when estimating model parameters. BTW, you're better off making a data.frame, e.g. `df <- data.frame(y,x1,x2)`, and then use `lm(y ~ ., data=df)`.
I am tempted to say "R for Stata users", but I would get voted down for this :)
Assuming (i) independence and (ii) that there's no heterogeneity in click through rates
They are different because ${\\bf K}_X + {\\bf K}_Y$ is the sum of two covariance matrices while ${\\bf K}_{X+Y}$ is the covariance matrix of the random variable $X+Y$. To see why the two matrices are different, use the [bilinearity of covariance](http://en.wikipedia.org/wiki/Covariance#Properties) to see that 
$$ [{\\bf K}_{X + Y}]_{ij} = [{\\bf K}_{X}]_{ij} + [{\\bf K}_{Y}]_{ij} +  {\\rm cov}(X_i, Y_j)  + {\\rm cov}(X_j, Y_i) $$

i.e. the cross-covariances are missing from ${\\bf K}_X + {\\bf K}_Y$ (note I assume $X,Y$ are of equal dimension to ensure that question makes sense).
you might want to rephrase your last comment, which is a little confusing. Continuous CDFs do not necessarily have a (proper) inverse. (Can you think of a counterexample?) So your proof requires additional conditions to hold. The standard way to get around this is to define the pseudoinverse $F^{\\,\\leftarrow}(y) = \\inf\\{x: F(x) \\geq y\\}$. The argument becomes more subtle, too.
(1) It's not particularly useful to talk about Poisson *errors*; better to simply refer to the Poisson distribution of the observed response. (2) $y=ax+b$ isn't simple proportionality, that would be $y = ax$. (3) when you say errors, do you mean *standard errors* or something else? (4) I am not sure TLS (in its usual form) is suitable for Poisson data
@das It depends on what you need the data for and how accurate you need them to be.  If you have related questions that will primarily be of interest to GIS people, why not post them on GIS.SE?  You can reference this thread to provide context, make progress, and avoid repeating yourself.
@SRKX, why don't you simply drop the correlated factors from further analysis? It is easy to do. Moreover, *stepwisefit* will do this for you.
@ttnphns: By "joint significance", does Adhesh mean F-statistic for the model, as opposed to F or T-statistic for individual predictors?
Here's some sample code for time-series cross-validation (or rolling out-of-sample forecasts) that might be useful: http://robjhyndman.com/researchtips/tscvexample/
Thanks! I wonder mathematically why power increases or nondecrease with sample size when random sampling?
You should give more background, like what the variables represent and why you want to transform them so that they are positively correlated. It sounds like you are fishing for results in the data that aren't there.
You clearly have to adjust your paired tests for multiplicity. Did you do?
Yes, my values are counts.
This post is difficult to follow due to (apparently) missing words and lack of formatting. Regardless, there does not seem to be anything in it that is related to the question, which has nothing to do with proportions. Please edit it to clarify these points.
It looks like it has to do with some caching system (available in [knitr](http://yihui.name/knitr/) or [Sweave](http://www.stat.uni-muenchen.de/~leisch/Sweave/)) or [memoization](http://en.wikipedia.org/wiki/Memoization) which avoid running computation again if nothing has changed (and is more elegant than saving full R session with timestamp).
All agreed to Nick. In fact, I already used some curve-fitting software (Sigma-plot) that returns tons of fits all in the $R^2$ range of .95 or above. I did not find this very useful.
Turns out in this case though, that there is a theoretical debate on whether an exponential, hyperbolic, or hyperbolic with constraint fit is most appropriate. This is in the area of "social discounting". Leonard Green (Wash U) published on this and I often see arguments with regards to $R^2$ and eyeball fit.
I have been searching for an answer similar maybe you guys can help me. I am also trying to interpret categorical variables with more than two classes. Some are significant whilst other classes are not. what can I infer from the insignificant ones?
does this mean the insignificant ones and the reference category are equally influence the dependent? For example:
ETHNICITY
Reference- Indian Other Asian - Sig = .273	Exp (b) = 1.123
African - Sig = .000	Exp (b) = .148 Many Thanks!
I looks like some values of 'var1' are missing values...
Are you sure that you are not confusing a distributional statement with an equality statement? The above formulas are statistics (=functions of random variables), that have the same distribution (chi-square) -and the symbol that should be used is ~, not =. They are not necessarily the same expression written differently.
You have not posted a reproducible example, since base R does not load object `maxLik`, but check that the expression inside of `log()` evaluates to a positive number.
Hi cardinal - thank you for the link.  The question I ask is more specialized to a specific case (which is not answered in the link you give).  I am not sure there is an answer, but still.  With regards, Tal
You're going to need to be more specific (e.g. the error distribution of...? What model are you using?) but the second thing you said sound closer to the law of large numbers than the CLT.
Thanks! I very much appreciate all of the statsmodels work!
@JuhoKokkala
Can easily do in the future. Would you recommend deleting this one and migrating the other?
@cardinal, this type of question belongs to group of questions which are asked from time to time and will be asked almost indefinitely. And yes OP only talks about constant variance and mean, which is weaker than weak stationarity if you want to name it.
@Wayne: if you used dummies and the full set of interactions in the second, the point estimates should be the same. Standard errors may differ because the second method may presume higher degrees of freedom, but you would want to check whether that is a correct modeling assumption.
Yohn, no INLA can not do what RJMCMC can do.
@Almo I love this site:)
I am not sure the group size is the main problem here, more details could be useful. Does every teacher teach the same students in turn, were they randomly attributed to a class or are they just teaching whatever students they have at the moment? Where do this frequencies come from? A fixed/known number of learners' error? How and why do you want to use a Chi2 test? What's the underlying question?
This question is more adequate for a place like StackOverflow.com If you are running a Linux server, this is ridiculously easy to do with a Bash script using 3, maybe 4 lines of code. See `?mutt`, `?sleep`, `?mv`, and use `R CMD BATCH` to execute workhorse functions.
It isn't clear from the original question whether they're trying to directly calculate $t+12$, which is possible if you structure the problem correctly, or whether they have a DLM that won't forecast with the default procedure because it's variable.
@Hal (con'td) The only requirement I put on the probability function is that it is an increasing function of the number of known common attributes between X and Y.
I agree, but I feel happy to extend my answer to residual plots, etc. More generally, any way to judge this could run into the same objection, e.g. significance tests _might_ fail to reject equal performance, $R^2$ _might_ be practically identical. Having published on comparison of different measurement methods with real data, I do assert that choice of method will usually find a best method, but if two methods appear about equally good, then that's the answer, not an insoluble problem leading to methodological anguish.
I use R's `RODBC` all the time. Highly recommend. http://cran.r-project.org/web/packages/RODBC/RODBC.pdf
Like @whuber, I'd be curious to know what the source is for your weights (0.4, 0.3, 0.3). 

I would think that one of the "official" conversions would be better, like CIE 1931 (0.2126, 0.7152, 0.0722) or the Rec 601 (0.299, 0.587, 0.114). You'd have to deal with gamma expansion/compression too, if you wanted to be 100% correct, but that's verging into overkill.
How would this be different from pairwise comparisons?
informative! thanks.
No worries, good to know -- I think @zbicyclist's point is that if you choose a reference category which has really infrequent outcome, then all of the parameters for that factor might be affected by the quasi-complete separation (whereas choosing a level with more frequent outcomes will prevent this being a problem for all parameters). [FYI, which you might already know -- you can change the reference level if needed: in R, one would use e.g. `kraj <- relevel(kraj, ref = "JHC")` if you wanted to use JHC as the reference level instead.]
@Wayne, please see my edit.
+1 This is a question that is difficult to provide a good answer to as it seems to be such a highly specialized paper.  But in addition to claiming to lose the bounty on a previous similar question, the OP added a question begging for solutions that was migrated to meta.  I made similar comments to yours in my answer there.  I am particularly dubious about regarding sample from the multiply imputed set of data.
I'm not sure I completely agree with your first sentence. A rejection region is a set, while a test statistic is ... a statistic; they are different animals. Yet the second pair of things are both probabilities, and can be compared. I think it's really the value taken by the test statistic at the *boundary* of the rejection region (the *critical value*) that is to test statistic as significance level is to p-value. The rest of it is well put.
May explaining the context and data would help
Hi Ted - perhaps you could ask a couple of more specific direct questions of the form "How would I ...?" that would help fill in what you don't yet know how to do. We don't know what you can and can't do.
you can use data.matrix() for that http://stat.ethz.ch/R-manual/R-devel/library/base/html/data.matrix.html
Can you give more specifics? Perhaps some of the relevant text?
That's a reasonable guess.  It still does not measure dispersion,though!
The Breusch-Pagan test is clearly described in the [Wikipedia article](http://en.wikipedia.org/wiki/Breusch%E2%80%93Pagan_test): you regress the squared residuals against the original (independent) variables.  What, then, are you trying to ask about rep(0,N)?  Because it never makes sense to regress *anything* against a constant value such as 0 (except as a complicated way to compute an arithmetic mean), I am wondering whether this question correctly communicates what you want to ask.
Start please with formulas of these related statistics http://stats.stackexchange.com/a/18136/3277.
(+1) Note that model (1) as it is written assumes that `x1` and `x2` are continuous predictors, not categorical ones. (Why not denoting the intercept by $\\beta_0$?)
Thanks! At the moment I'm fully sure what I would want the outcomes to mean; each of your two suggestions seems reasonable to me. But perhaps I should be satisfied with a given outcome if the means of the continuous outcomes for each of the prediction bins are in the neighborhood of the corresponding bin centers. Would that mean adding a third category to your answer?
I am comfortable with PHP for programming and I have been able to make some visual basic work within excel...
Maybe I'm misunderstanding your issue, but what about fitting a mixture model to your data, and comparing the "non-zero part" to the distribution of the non-bounded protein?
Thanks Rob.  2 quick things though.  I converted by zoo object to ts using as.ts.  That didn't seem to do the trick as I got the same error.  I then backed all the way up created a new object from the raw CSV, converted to ts using ts(object).  Functions ets and auto.arima both found model, but I got an error when trying to plot the fitted auto.arima object (Error in xy.coords(x, y, xlabel, ylabel, log) : 
  'x' and 'y' lengths differ).  Thanks again for your help!
Oh sorry, now I understand what you are asking.  Yes, if you specify both then it applies both forward and backward and chooses the one with the best criterion.
This seems to be related to [Queuing theory](http://en.wikipedia.org/wiki/Queueing_theory) or time series/stochastic processes, possibly with heavy tailed queues. Ask one of the employees or the manager, they tend to have great empirical understanding of this behaviour.