tttjjj/elt_bench huggingface dataset:
I guess I am having confusion with P(word|sense_i).  I though the denominator would be sum_w[count(w,sense_i)] and not just count(sense_i).  i.e. total number of words in sense_i examples, not total number of sense_i examples.  Can you explain why this is not the case?
I am also interested in this question, as the autocorrelation will cause the Chow test to be more likely to detect structural breaks that don't actually exist, so a way to compensate for this would be very useful.
$X_{1},...,X_{n}$ are from simple random sampling (normal distribution)
You've applied *twice* the inverse logit.
note that you can accept your own answers if they solve your problem (simply click the checkmark below the up/down-vote button)
Thank you very much for your answer - I do now use a combination of the functions coef and drop1 in order to identify which interactions can be savely omitted before computing the ANOVA - this works in most instances except for some cases, where drop1 leads to NA results for the F- and p values, which probably means that there are not enough data points for the covariate.
Yes, given you know/assume they're independent, it's correct. As for issues, there's not nearly enough information to guess at potential problems... it really depends on what you're doing, and why, and how.
... Oh also I should indicate that if working with proc mixed those get solved ..The only reason I would like to keep working on proc glimmix is the advantage of having the ilink option since most all of my variables needs to be transformed to reach normality, this because with proc mixed the transformation back can not be done directly, so doing manually I am not able to calculate back he standard error of the mean. If any one knows the formulas to transform back SEM from inverse square root and other common transformations will also be appreciated.
Many thanks! I had done the analysis with helmert contrasts as well and was just about to post a follow-up, since they yield different results:
@PeterFlom Always a good question; I made an assumption, but it might not be the right one.
"equivalent" is a broad term. There exists a formula to convert. ;-) I've been mostly using cosine *similarity* actually; not cosine distance, because there is more than one "angular distance".
(ctd) ... that is, the LS problem converts to optimizing [$\\sum_{i=1}^n (Y_i - \\hat\\mu(x_i))^2 + \\lambda \\int_{x_1}^{x_n} \\hat\\mu''(x)^2 \\,dx$](http://en.wikipedia.org/wiki/Smoothing_spline) where $\\lambda$ acts as a tuning parameter that allows you to choose the tradeoff between fit and smoothness. Penalized splines are basically the same thing, but where fewer than $n$ knots are initially chosen. Smoothing splines have a very close connection to older ideas like Whittaker-Henderson smoothing. ...(ctd)
It is not clear to me whether you are having technical issues editing your questions or whether you are purposefully vandalizing them. If the latter, *please* stop.
You really only compared two means? If you got the Tukey CI from a post-hoc comparison of two means following an ANOVA of $>2$ means, this would be fairly simple to explain...
I have edited the title to (hopefully) make your point more clear.
Have you thought instead about using techniques that are more powerful and insightful than a correlation coefficient? For instance, have you drawn side-by-side boxplots of the continuous variables, partitioned by their position on your scale? Such graphics will not only display the nature of the association, they can also suggest ways to re-express both the continuous values and the numeric codes in your scale to make subsequent linear modeling more applicable.
@ZivkoJuznic-Zonta: sorry, it was not clear from your original question. Now, if you cannot fix a specific family of distributions from knowledge about the problem at hand, you should indeed demonstrate by simulation that the fact that the errors are not normal is not very influential on the resulting CIs if the errors are from another family, e.g. $t$. Or Laplace.
This question appears to be off-topic because it is about how to use R's table function.
Dear @Alecos: I don't see how this question has much of anything to do with nonmeasurable sets (beyond the fact that we can't always construct a space where all subsets are measurable). Indeed, all of the sets referenced *are* implicitly measurable (since we can apply $P(\\cdot)$ to them). Provided the probability space in question is sufficiently rich (e.g., $([0,1], \\mathcal B[0,1], \\lambda)$), there are, of course, plenty of distinct measurable sets with probability zero (and, hence, *equivalently*, plenty with probability one as well).
Dear Nick, What is your interpretation about undefined ρ?
@Lao Tzu true :)
Associated to this.  Does the distribution even matter?  I think the 2 standard deviation rule applies regardless.  Fat tails just don't matter at all.
Also of interest: [Significance-of-coefficients-in-linear-regression:-significant-t-test-vs-non-sigificant-F-statistic](http://stats.stackexchange.com/questions/24720/).
I don't, but as @Frank Harrell suggests, there's not much need or use for anyone's imprimateur saying "this rsq is sufficient."  Normally there's much more one wants to accomplish with such research, such as estimating coefficients, specifying standard errors of estimate, and so on.
+1 for mentioning sensitivity analysis. You should know how much your results are dependent on the priors used...
I think this example--although it will likely provide useful insights upon further analysis--may be somewhat misleading, Jeremy. Your conclusion "it's not a problem" actually belies a *huge* problem: the $R^2$ barely changes even when the new variable is highly significant because *the system is very nearly collinear.* (The condition number is over $100$, which is enormous for such a small design matrix.)
If $\\theta < 0$, then $\\sqrt{\\bar{X}}$ is complex-valued with (arbitrarily) high probability as $n$ increases, so in that sense, there *is* a problem. However, your intuition regarding the MLE on the restricted parameter space is correct. How would you prove it? (Hint: Use the definition of "maximum likelihood" directly.)
For your questions, I'm 80 (but not 100) % sure of the following: re. #2, Yes, if you don't estimate the correlation, you force it to be 0; for the rest, if the correlation is actually not *exactly* 0, then you are mis-specifying the non-independence of your data. The betas can nonetheless be unbiased, but the p-values will be off (& whether they are too high or too low depends & may not be knowable). Thus, interpretations of the betas may be able to proceed as normal, but interpretations of 'significances' will be inaccurate.
See responses on [this question](http://stats.stackexchange.com/questions/30394/how-to-perform-two-sample-t-tests-in-r-by-inputting-sample-statistics-rather-tha)
Thank you for the answer, I added image of the components and variables. So, my last PCs got high weight and this helped. If PC1 looks like a sum of variables, I am wondering what information goes to PC2 and PC3, or it depends on the image?
An editor interested in clarity would remove the implicit double negative in the first statement ("difference ... unless") by restating it more positively, like "When the population effect is nonzero, it can be detected with a sufficiently large sample." The rest is confusing because although the first statement refers to a *population* effect, the example refers to an effect *observed in a sample.* A better example would suggest that a *population* correlation coefficient of 0.1 will show up as a significantly positive correlation in a sufficiently large sample (but maybe not in a small one).
Good first question! I'm not convinced each person's three measurements on the same specimen can really be treated as independent though. Did they know (or were they likely to realise) that the specimen was the same? Did they know the 'true' results when they made the measurements? Also do you know how many of the 'true' results were positive and how many were negative?
Are you asking 'how can I be sure Bayes rule is true?' or something else?
Have you considered having a look at the log-rank test that compares two Kaplan-Meier curves?
@Glen_b I think I understand the problem with the arima output that you encountered. The forecast package had been loaded on my machine and this has the effect of changing the output of the arima() function! Not ideal behaviour and I only noticed it today, myself. Not loading the forecast package and calling arima() returns only the aic in the output - not the BIC and AICc. Compare the outputs when the forecast package is loaded and not loaded to see what I mean. Apologies for not pointing this out earlier - I've now added a note to my answer to highlight the issue.
There is no way to tell a priori; you have to try it out.
The graph I am talking about is on p 7 of The Elements of Graphing Data.
+1, thanks. (1) For acf to dacay to zero, did you mean that wide-sense stationarity and ergodicity are necessary and sufficient condition, or just necessary or sufficient? Also see [my question and replies](http://stats.stackexchange.com/q/85902/1005) (2) When the process is a ARMA(p,q) process, when will its acf decay to zero? [When will it be ergodic](http://stats.stackexchange.com/q/95713/1005)? (3) Same questions for a white noise (defined as uncorrelatedness, mean zero and variance one): when its acf decays to zero, and [when it is ergodic](http://stats.stackexchange.com/q/86695/1005)?
Looking at your figure, I can suggest that standartization by genes may be helpful. Try it and see whether the resulted heat-map supports your inferences more clearly. Note that mir-1826 expression is constant, and thus may be deleted from figure, but mentioned in text. Also, you may add mean expression value near each gene in order to save the possibility of gene comparison after Z-scoring
@COOLSerdash You may be interested in [this][1] question, where I ask which of the grid results or `rstan` results are more correct. [1] http://stats.stackexchange.com/questions/114366/is-rstan-or-my-grid-approximation-incorrect-deciding-between-conflicting-quanti
Thanks for adding the original reference.
Thanks! Really helpful.
`These are made by people who find it more profitable to sell software to confused day traders than to use their own systems` That on its own would have been worth an upvote.
(+1) This is identical to the convolution solution I posted, except the convolutions are performed directly rather than with a built-in convolution function. There's little reason to prefer one method over the other for 15 probabilities, but with, say, 150000 of them, you might want to look at using an FFT. I had the same suspicions as you about numerical error creeping in, which is why I did the *Mathematica* comparison: it used exact arithmetic and its answers--fractions with large numerators and denominators--were converted to machine precision only at the end. The agreement was great.
I would put it: "Of the X% American/Australian/French/etc. adults who intend to buy a television, Y% intends to buy brand A, Z% intend to buy brand B,..." So you immediately see both information in one sentence.
Just because something is a random variable doesn't mean it can't be a parameter. Welcome to the wonderful world of bayesian probability :)
I don't think survival analysis is mentioned in the link Peter found.  If you accumulate time between earthquakes at a certain region that could be used as the basis for a survival curve.  But what is missing is the total number of earthquakes that will eventually occur. That prevents us from determining the magnitude of the steps down in say a Kaplan-Meier curve.  Also even if we are given a survival curve how would we use that for prediction.Perhaps we could use median survival time as the prediction for the next event.  But that doesn't seem to be a good way to forecast.
(+1) Nice summary! [Forrest Young](http://forrest.psych.unc.edu/teaching/p230/p230.html) (already cited by @whuber in his comment), [Yoshio Takane](http://takane.brinkster.net/yoshio/) and [Jan de Leeuw](http://gifi.stat.ucla.edu/) have some very good papers on MDS.
@JohannesMorfeldt: Andreas Alfons has a package for k-fold CV. k-fold CV is not per se better than LOO-CV. But LOO does not offer the possibility of iterating/repeating, so it does not allow you to measure model instability. See also the papers I linked in http://stats.stackexchange.com/a/71192/4598 for out-of-bootstrap vs. cross validation. They also discuss LOO
Although this Q mentions R, it appears to be ultimately about how the predicted values are calculated (not the R code to retrieve them). That may be enough to make it on-topic here.
The short answer may seem unhelpful, but it is whatever works and is valid, so long as it helps your goals.  Assuming all your values are positive, then all three transformations you've tried are well defined and it's clear that cube root works best _if it is important that you have a symmetric distribution_.
The independent variables can't tell you anything more than the model "the DV is always 1". You might regard that as 'can predict', but you can't argue that the IVs contribute anything.
Yes, $t$ and $p$ values are the same. Concerning the correlation coefficient (Pearson's). Denote $b_{Y.X}$ the slope of the regression with $Y$ being the dependent and $X$ the independent variable. Further, denote $s_{X},s_{Y}$ the standard deviations of the variable. Then the correlation coefficient is: $r = b_{Y.X}\\cdot\\frac{s_{X}}{s_{Y}}$. See [here](http://stats.stackexchange.com/a/104577/21054).
Wow, that is interesting.
@JasonO.Jensen: Each device is measuring from the same input but is calibrated differently. For instance, at T1, given that `x` is the true value, device 1 might report a measurement of `x` while device 2 reports `x+5`, so at T2, with `y` as the true value, device 1 might report `y` and device 2 might report `y+5`. Device 2 is thus calibrated by +5 from the true value, but is accurate when adjusted for calibration (i.e. `T2-T1` is approximately equal for both devices). I'm trying to test whether the adjusted measurements of some device 3 are significantly similar to those of the other devices.
He's clumsy enough to change his name in the middle of a question at least! ;)
Sounds to me like this is more so a multivariate classification problem, as opposed to being a Linear Model.
This is a difficult question to answer, given the information provided above. Without knowing the number of attributes (and levels) in your experimental design and the number of questions you plan to ask of each respondent, I cannot provide a justifiable answer. Do keep in mind that the quality of your experimental design can greatly affect the variance of your parameter estimates and, therefore, increasing the efficiency of your design is equivalent to increasing your sample size.
My goal is to find whether there are associations between alcohol and BMI (body mass index). I have to control though for potential confounding factors (such as economic status, index of deprivation, physical activity level, age) and one of them is smoking. Apart from alcohol and the confounders that I always add, when I put smoking as a factor my associations are similar compared to when I do not put smoking. When I separate in smoking groups (and ran the generalized linear models with BMI as the dependent variable and all the rest as factors and covariates) I find much weaker associations.
@nickflees answer is right on. For future reference, you would only want to use something like a Gaussian process or neural network if you don't know the the functional form, i.e, you have a bunch of $(x_i, f(x_i))$ pairs but don't know anything (helpful) about $f$.
+1. But I think you exaggerate the limitations of (1): you are limited only by your skills and imagination. For instance, you can take two *real* datasets and form a mixture, using one to "contaminate" the other with [known] "outliers." The singular advantage of (1) is that you can use it to understand--in a precise quantitative way--how the underlying distribution and the amount, size, and distribution of any outliers all affect the results of a statistical procedure. That insight is not possible with method (3) and can be only semi-quantitatively obtained with (2).
Thanks Michael. But I couldn't understand sorry , Can you explain clearly ? Hmmm maybe using maths ?
@gd047. LaTeX code: `$\\sigma_{\\epsilon}$`
If cross-posting, please do pay attention to http://www.stata.com/support/faqs/resources/statalist-faq/#crossposting
Is there a reason you can't use [?rnbinom](https://stat.ethz.ch/R-manual/R-patched/library/stats/html/NegBinomial.html)?
I don't think what you're asking for is a standard mathematical concept.  Maybe if you can explain why you need this, we can recognize the concept you are thinking of.
If the 3 for you is not the same as a 3 for somebody else, then you don't even have a scale: you have a collection of incomparable measurements and there's little meaningful you can do to summarize them.  What makes a scale ordinal is that (a) values *can* be compared, so your 3 and my 3 mean the same thing, but (b) numerical differences of values are meaningless apart from their signs, so that (say) two 3's, a 4 and a 2, or a 5 and a 1 could be put *in any order,* although numerically each pair of ratings has the same mean and median.
First, to answer the question about whether your system is accurate, you need to have a measure that you consider accurate.  But I don't see that in your description.  Do you have such a measure?  Second, in comparing your measure to measures B and C, are you most interested in absolute agreement (same score value) or relative agreement (same ordering of scores)?  Third, do you consider the data interval or ordinal?
Firstly, you have got the terms dependent and independent the wrong way around. Also, please do not use this syntax to estimate fixed effects regressions. It is obsolete (`xi` has been replaced by explicit support for factor variables since Stata 11) and numerically cumbersome (use `xtreg, fe` or `areg` instead). Lastly, omission of variables is not an error, but intended behavior when the design matrix is singular. You need to fix this by making sure that your regression is well-specified.
I did this and it is not linear.
(-1) I don't believe you've quoted your source accurately.  Besides that, people will need to see more of the context in order to answer your question.
I wouldn't say it's "not sensible" to do what you have done.  However, because you already know the factors which determine GPA - namely the grades in various courses, it may make more sense to model those factors rather than GPA.  Perhaps a "better question" might be, for example: *What factors are most strongly associated with high math scores?*.
There are many great R packages for ensembles of trees: e.g. randomForest (classic algorithm), party::cforest (random forest using conditional inference trees), gbm (gradient boosting of trees) to name a few. I read the OP as wanting to implement classifier/regression agnostic ensembles. The simplest procedure is of course to average predictions.
`group` is a numerical vector, is this on purpose? Normally, grouping factors should have class `factor`, such that the transformation to contrasts can be handled automatically by functions like `lm()`. This will become apparent once you have more than two groups, or use a coding other than 0/1 for your `group` variable.
Actually, that does help! I was previously considering multiple estimation methods, so it would have been hard to distinguish, but then decided to consider only the MMSE for simplicity. Thanks!
http://www.mathworks.com/help/gads/genetic-algorithm.html
In addition, the way I meant can't isn't really "cannot".  You could somehow come up with a single equation that calculates one confidence interval for everything.  It just wouldn't mean anything sensible.  That's what I meant for can't.
A related question I asked a while ago: http://stats.stackexchange.com/questions/961/statistical-test-for-n-x-m-contingency-tables
You could use it, although in the big picture of things, you can only hope for consistency. I am pretty sure there are various small sample correction factors have been developed and applied akin to those used for [heteroskedasticity only corrections](http://www.sciencedirect.com/science/article/pii/0304407685901587), but I am not following that literature very closely.
@JoshO'Brien I took the first derivative of the $P_{\\text{once}}$ and set it to zero and solved it. I used my calculator to do this though. The first derivative is: $2(n-1)^{2k-1}(1/n)^{2k}(1+2k(\\log(n+1)+\\log(1/n)))$. Sorry that I can't give you more details.
Any help would still be very appreciated.
Hi and welcome to the site, GrantD71! I have tried to incorporate the equations into your question, please make sure that they are still correct. Further: Your data are displayed quite messy, could you please re-format them so that they are easier to read?
Can you be a little more specific regarding *which* arcsine law you are interested in? What you currently describe almost sounds more like the *ballot theorem*. There *are* generalizations to the arcsine law regarding the proportion of intervals between ties in which one player is leading. Note that in an biased coin setting there is a nonnegligible probability of *never* returning to zero.
Can you say more about what you'd like to know? (This question is fairly simple / sparse.) You may also be interested in [this recent question](http://stats.stackexchange.com/questions/31690/how-to-test-the-statistical-significance-for-categorical-variable-in-linear-regr).
-1 for "never", see example 1 of Joshuas' answer
Are you actually interested in the distribution of the metric or in whether there is a significant difference between the distribution of the metric when the class is 0 and when it's 1?  There's a subtle difference between the two; you can test for a significant difference even if you don't know the underlying distribution.
Thanks for the advice Glen. Will make more conscious attempts to improve my problem solving methods. I am confident I will be able to continually improve and contribute to this awesome community which has really helped me a lot in my quest of self-learning stats.
I want to identify how similar are the samples to each other.
I'm not sure if this is exactly the sort of procedure you sketch, but there is [BART](http://cran.r-project.org/web/packages/BayesTree/index.html). Here's a link to a [PDF](http://astro.temple.edu/~msobel/courses_files/chipman-george-mculloch_BART.pdf)
@ttnphns One way I know is checking the sign of b'x-k and depending on the sign the population is chosen. Is this the same or different?
So how would you use `poisson.test()` with the x given in this answer (`rpois(100, 5)`? This seems overly complicated and wrong: `poisson.test(as.integer(mean(x)))`
Sure, you can do it: nothing will prevent your software from running. But the significance levels and p-values will be utterly meaningless.
I looked up AVAS and it looks like thats what I need. However, I'm a little stuck on implementing it. I took x = matrix with height, age etc, y = weight and a = avas(x,y). however, only plot(a$x, a$tx) and plot(a$y, a$ty) work i.e. not plot(a$tx, a$ty). Also I'm unsure of how to interpret the charts and determine if a transformation is needed as my graphs do not look anything like the graphs presented in the example. Can you help me please?
Well on behalf of your students (and everyone else) I thank you for not using the original colors in the picture!
...As a monotone transformation, we expect that $\\sum_{i=1}^{n}\\log f\\left(X_{i};\\theta_{0}\\right)\\geq\\sum_{i=1}^{n}\\log f\\left(X_{i};\\theta\\right)$.
By the LLN, we then can expect $E_{0}\\left(\\log f\\left(X;\\theta_{0}\\right)\\right)\\geq E_{0}\\left(\\log f\\left(X;\\theta\\right)\\right)$. Things make better sense now.
@Andy: Can you please elaborate "... which sometimes resolves the negative test statistic..".? In particular, when is it not appropriate?