tttjjj/elt_bench huggingface dataset:
This is more about education than about statistics. Cultures vary all the way from (a) students being expected to ask if they don't understand what is being sought to (b) this kind of question being all the instructions you will get and the expectation being that students need to do **all** of the thinking with the idea that what they have been taught (and, who knows, **expected to read**) is the whole of the context they need. So, this is more for some educational website where people talk about learning and teaching. The statistical detail here seems incidental.
(cont.) (**4**) It is clear that you are using the notation $\\mathbb P(X = x)$ to refer to the density here, but a little caution might be advised. (+1) :-)
All four graphics suggest a nearly linear relationship between t[,2] and x. (Given the relatively narrow range of t[,2] this could be consistent with any of the standard links in logit regression: logit, probit, log, complementary log-log, and even a linear link.) It would help to size the dots to be proportional to the square roots of their bin counts, so that visual area is directly proportional to precision.
Is that Matlab code? In any case it would be more decent and promising for you to show the spectrogram - the picture or the data, and ask your question upon explaining it.
Hmm, yet another interpretation of added value. May I  point to a recent paper (and R package) of mine that combines continuous classifier perfromance measures (like Brier's score) with the idea of sensitivtiy, specificity, etc.: http://softclassval.r-forge.r-project.org/2013/2013-01-03-ChemomIntellLabSystTheorypaper.html ?
You say 10 to 20 is good, but would this also depend on the size of the error variance (possibly relative to other things)? For example, suppose there was just one predictor variable. If it was known that the error variance was really tiny, then it seems that 3 or 4 data points might be enough to reliably estimate the slope and intercept. On the other hand, if it was known that the error variance was huge, then even 50 data points might be inadequate. Am I misunderstanding something?
The only advice I could give is that predicting the change scores tends to be much harder than predicting the levels (so it is not surprising in many situations that a low `R^2` occurs). See some nice discussion of [pre-post designs here](http://stats.stackexchange.com/q/3466/1036). Although that still totally does not answer your question!
Have you tried simulation instead? Don't ask for a hybrid approach--it does nothing different--but specify `simulate.p.value=TRUE` as per the manual page.
(i) since your subgroups have smaller samples, even if they all shared the same effect size you wouldn't expect them to have a similar p-value to the aggregate. (ii) Also see [Simpson's paradox](http://en.wikipedia.org/wiki/Simpson%27s_paradox).
@John thanks for your comments. I may not be using the terms correctly.  I have an experiment where 2 categorical factors (light/no-light and ambient/elevated CO_2) and one continuous variable ([DOC]) were manipulated.  To evaluate the effect of these factors on the response, I used ANCOVA, since there was a mix of categorical and continuous factors. However multiple regression with dummy variable coding can also be used to test the effect of a mixture of cont. and cat. factors on a response.  I am hoping to learn more about when it is appropriate to select one or the other.
have you thought creating a dummy for 'candidate's gender' or anything else you want to check?
I agree with the majority of this point, but I think "Keep it simple." could be unclear. Your main point is that you should know what you want the chart to convey. "Keep it simple." brings up some other ideas, like "The data:ink ratio should be high.", which Tufte encourages, and "Present no more than three variables.", which Tufte discourages.
For both $a$ and $b$ it has a spike at $0$ to allow for very small values and has a Cauchy tail to allow for very large values. The behavior at $0$ makes it improper, but if we  bounded away from $0$ it would still have infinite mean. These are somewhat desirable properties for a "noninformative" prior to have.
Jake, I look forward to hear your comment on this. Thanks!
Anyone experience using the `dse` package by Paul Gilbert. I just stumbled upon it. It contains an `error.weights`parameter is the ARMA function and others. "If error.weights is greater than zero then weighted prediction errors are calculated up to the
horizon indicated by the length of error.weights. The weights are applied to the squared error at
each period ahead."
@bullmose - Regarding additonal observations increasing uncertainty we can look at the precision $\\frac{1}{S_{t+1}}=\\frac{1}{N_{0}}+\\frac{1}{D+S_{t}}$.  For ease of example i will use $\\sigma_0^2>>N_{0}>>D$ so we have $\\frac{1}{S_{0}}\\approx\\frac{1}{N_{0}}$ this means that $\\frac{1}{S_{1}}\\approx\\frac{1}{N_{0}}+\\frac{1}{D+N_{0}}\\approx\\frac_{2}{N_{0}}$.  this is a doubling in precision.  For next observation we get $\\frac{1}{S_{2}}\\approx\\frac{1}{N_{0}}+\\frac{1}{D+N_{0}/2}\\approx\\frac_{3}{N_{0}}$.  hence each observation increases precision by $\\frac_{1}{N_{0}}$.
@Momo, you are right at your comment. "tomous" ("anatomy") is "cut", so ordinal scale is polytomous, because it is categorical. Multinomial is a nominal scale - with multiple unordered categories. Polytomous is any categorical scale with more than 2 categories, so it is opposed to "dichotomous".
You can "build" a curve from no data points if you have a theory in which you are sufficiently confident. Thus a good answer needs to make reference to what quantities the coordinates on this curve represent and how they are measured.  Please give us this information for your "yield curve" (whatever it may be).
Just to add to this a bit, if responses like 'don't know' exist in your data, and you believe that these still contribute to your latent trait estimation (say, 'don't know' is sometimes a higher/lower response than 'no') then fitting models like the nominal/nested-logit may be informative to see how the response curves look when including these options (nested-logits still include information about which is most 'correct'). Otherwise, as @doug.numbers suggests, it might be better just to set them as missing and let the IRT software just skip those particular data points.
Why are you using chi-squared tests? Age is a continuous variable; unless it's grouped (which is good to avoid). Also, I am not sure what you mean by "residue"; perhaps "residual" but the usual test with chi-square is chi-square.
@whuber Okay, thanks, I can certainly do that, and I'll get rid of my rambling comments as well. I'm not thinking clearly this morning ... in respect of your last sentence, can you give me a hint about what sort of circumstances that comes up in?
Just to make a link with an earlier related question: [The Two Cultures: statistics vs. machine learning?](http://stats.stackexchange.com/q/6/930)
I've added details in my question. If you need more feel free to ask.
A couple of the tricks that @StasK refers to are discussed in M.L. Buis (2012) "Stata tip 106: With or without reference", The Stata Journal, 12(1), pp. 162-164. http://www.maartenbuis.nl/publications/ref_cat.html
If you forgot your login credentials, visit our Help center in order to merge your accounts.
A reference I have found helpful for Tex is this [online latex equation editor](http://www.codecogs.com/latex/eqneditor.php).
I have just finished a draft for a second answer based on the idea that you might want also a LDA, so I'll post it in a while using your data as the example.
I'm very sorry that I forgot to accept this answer for over a year.  Thanks very much.
As an archival note: Section 5.4.2 of Kuncheva's book gives a Bayesian validation for the sum rule proposed by Kittler et al.
Because at a fundamental level QM is *deterministic*--the Schrodinger equation precisely describes how a system evolves over time and there's no uncertainty about that--and it is *linear*, it would seem that reversibility in the evolution of individual states would immediately imply reversibility in any distribution of such states. I would therefore like to see your *mathematical* justification of your assertion to the contrary, because it will show more clearly what you are now only implicitly assuming about the dynamical equations.
@Raffi: You could add the ratio of correctly classified examples of the minor class (i.e. precision or true negative rate respectively). However, I think it should be enough that you state your awareness of this problem and that you check that the model is not just predicting the major class. But this is just my opinion.
Yes, I want the former. But I wonder if there is not another way (without the root of the MSE) to obtain the SD from regression standard errors? Thank you for pointing out lincom; I didn't know it yet. Perhaps there is a way to translate the standard errors obtained there into group standard deviations..
as such, the spread of multivariate gaussian doesn't make sense. However, depending on your needs, there might exists approaches to answer your question. Trace of the matrix is one of the many ways, but you would be ignoring correlations, which may make a huge difference. Eigen values, PCA, etc. might be much better. Therefore, could you please elaborate on your needs?
Have you tried looking at the definition of the moment generating function?  Its an extremely straight forward calculation: http://en.wikipedia.org/wiki/Moment-generating_function
crosspost: http://datascience.stackexchange.com/questions/851/computing-image-similarity-based-on-color-distribution
I think this would work to test whether one distribution is significantly more skew than another.
BTW, I'm surprised not to see more upvotes for this question: it has attracted three answers, which is *prima facie* evidence that it is a worthy and interesting question. It's especially remarkable that nobody who has answered has upvoted the question! What's up with that, people?
Although not exactly the answer to my question, I am selecting since it gives pointers to techniques I didn't know and seem relevant. I know Jolliffe's book well, and it has nothing to that effect.
Technically, the problem is that singular covariance matrix means that some subset of variables is perfectly correlated, so probability density should be exactly 0 in some areas, but that's not possible with a Gaussian. One solution is to instead look at conditional density, conditioned on the fact that random variable lies in a feasible region. This looks like what they are doing in the link. Never heard the term "G- inverse", I'm guessing it's Penrose-Moore pseudo-inverse?
"*I believe it is necessary to find the distribution of the data*" --- whereas I believe it's impossible to find the distribution of the data, and in any case, usually pointless since -- for the models you mention -- the unconditional distribution of the data is not something assumptions apply to. What are your response variables, and what are your predictors?
How would you talk about something being "by chance" if chance isn't operating, but deliberate choice is involved?
@Glen_b Thanks for the correction :)
One more thought: a mixed model with a random effect of student, fixed effects of $\\delta$ and cumulative number of exams taken, and a variance effect of $\\delta$ (in `nlme`: using `weights=varFixed(~delta)`
Try asking in the Structural Equation Modeling group (SEMNet), which specializes in just this sort of thing.  
https://listserv.ua.edu/archives/semnet.html
Another non-statistical example from Edward Tufte, [Powerpoint does Rocket Science](http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001yB&topic_id=1). Although it is bit more closely related to logical progression from statistical thinking in general than the metric mishap you mention. Also are you familiar with this book, [The Cult of Statistical Significance](http://statisticsforum.wordpress.com/2011/10/18/the-cult-of-statistical-significance-book-review/)?
Multiply your second inequality by the denominator. Those are your bounds if you only have two variables. If you have k>2, you need the covariance matrix to be positive definite to generate like that (unless it's written especially cleverly, in which case it can be positive semidefinite). There are a variety of ways to get a positive semidefinite matrix; one common way is via working with its choleski decomposition.
A couple of clarifying questions: 1. you have 38 subjects under category P and 38 subjects under category M, such as length(y) = 76? 2. Could you give more background information about the rationale for the hyperparamters and the experiment? It's a bit confusing to me.
And R isn't really strictly pass-by-value.  Lazy evaluation and some clever optimization means that often data isn't copied unless it has to be.
Sure. The three issues you'll have to consider are (1) heteroskedacity (2) autocorrelation and (3) seasonality. There are a number of methods of addressing all three concerns. For short time series seasonality is less of an issue, for longer time series  the limitations of using static seasonal components may become more apparent.
Repeated measures ANOVA makes strong assumptions, especially sphericity, that are unlikely to be met in longitudinal data. Have you considered a multi-level model?
Great answer, thanks! This phenomenon is quite interesting. In my specific application, using Gamma functions makes sense. So you are saying that MLE is reasonable for a large $N$? I am trying to use an EM-type algorithm (for MLE) by alternating between estimating $N$ and estimating $p_i$ (which are more complicated that presented above).
Boosting doesn't solve the asynchronous nature of the data streams. It's a great way to combine a lot of "weak" classifiers into a strong classifier.
tmp$V7 = p.adjust(tmp$V6,method="BY")

tmpV6[1:5]

[1] 0.040461040 0.001250268 0.037407030 0.009361665 0.006866937

tmpV7[1:5]

[1] 0.31423224 0.08659423 0.29570166 0.14456543 0.13611251

tmp$V7 = p.adjust(tmp$V6,method="BH")

tmp$V6[1:5]

[1] 0.040461040 0.001250268 0.037407030 0.009361665 0.006866937

tmp$V7[1:5]

[1] 0.05252837 0.01447545 0.04943072 0.02416616 0.02275313

plz take a look at this, tmp$V6 is the original pvalue, while tmp$V7 is after correction.

However, it seems that after BY correction, I get even worse result than BH(or fdr) method
Can you clarify what you mean by irregular? I had initially taken it to mean that you had two series of different discrete time intervals.
one question: suppose you have 3 clusters and you find 2 vectors using LDA. What do you do to classify? Do you project over both vectors? I can only understand LDA with one vector, where you project the samples onto it and see which cluster center is closest. Thanks
@Andy $j$ and $k$ index two distinct treatments.  I should have written "For $j,k \\ne 0$..."; I will fix that typo.  Thanks for catching it.
Theoretically, because these are inherently orthogonal (non-correlational) polynomial codes, I am favouring the separated non-correlated random slopes syntax.  Furthermore, an empirical test of models with/without correlated slopes does not show a significant difference in fit (using ANOVA).  Anything else to consider?
Increasing sales imply increasing demand, and increasing demand leads to increasing price
@prob @JMS Why don't we let the OP, who I believe is quite knowledgeable about stats, try the transformation route first?  Then, if that doesn't work, it would be fruitful to commence a new thread in which the problem is presented less narrowly.  Your comments would be appropriate in that context.
Yes it is duplicate. I did not found these question before asking. Sorry for that. But I also want to point, that the link provided in the answer of this question by **Pio** gives better explanation of the topic then answers in other questions.
And I would like to plot the curve of the marginal effects. How can I do it in R? What I have tried does not work: plot(outcome ~ variableA, data=ml)
Part of the issue here is that, IIRC, the reason we select the no-effect as the Null is because the parameter for that effect is known, it is `0`. If you want to turn this around and have some non-zero effect as the Null, then we'd need to know in advance what the value of this parameter was for the entire population, and if we knew the value of the parameter for the population there would be no point in testing.
Well, here is source on it.http://www.dartmouth.edu/~chance/teaching_aids/books_articles/probability_book/Chapter7.pdf
How exactly are the categories labeled?  By default, SAS-GLM orders categories in the design matrix by the sort order of the class variable.
It depends on what the error bars are to represent; different things are possible. For example, it's quite common in this situation to use standard error bars to represent the standard deviation. So what, specifically, are your error bars intended to indicate? (In the previous incarnation of your question, now deleted -- I can't find it - it might have been possible to back the definition being used out. If you can point me to it, that might help.)
*"Pie charts are a terrible way to visualized data"* - depending on your objective >:)
Dimitriy: I can use SPSS or GraphPad Prism. I have access to almost any statistical software that is user-friendly at the university. I am, however, not able to use R well so would prefer to avoid it.
@Emilio You get an expression involving an error function upon doing that integral (as you would expect), but then the derivative with respect to $\\rho$ reduces it back to an exponential.  Numerical integration and differentiation confirmed that result for $-0.98\\le\\rho\\le 0.98$ (beyond that range there are difficulties with floating point roundoff error, even when using tricks like taking logarithmic derivatives).
Can you say more about your situation, data, & goals here? It may be that what you're trying to do is best accomplished some other (easier) way, but I can't tell. You may find this blog post helpful: [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/comment-page-1/).
I may well be missing something, but if you're splitting a sample of 44 cases into 10 subsamples, doesn't that leave only 4 or 5 per subsample?  If so, it's unfortunately a situation where the software makes something possible that doesn't have any practical worth.  Perhaps the procedure you're using is more like bootstrapping or jackknifing, where you select most of the 44 each time, sampling with replacement?
@Nick That sounds like a good resolution, given that various authorities do use "standardization" broadly but none (AFAIK) ever use "studentize" in such a broad sense.
You are asking a one-dimensional question!  Because $\\Lambda$ is diagonal, your expectations and variances are computed component by component.  Therefore your last line gives the answer and there's nothing left to show.
I agree with this and other similar remarks. I don't think much purpose would be served by leaving this question in the forum under its present name. Anyone facing the difficult and delicate problem of detecting fraud, or of handling allegations of fraud, would not find it helpful or pertinent.
@nico:  Sure it can be negative, but there's some finite limit to it because there are only so many protons and electrons in the Universe.  Of course this is irrelevant in practice, but that's my point.  Nothing is **exactly** normally distributed (the model is wrong), but there are lots of things that are close enough (the model is useful).  Basically, you already knew the model was wrong, and rejecting or not rejecting the null gives essentially no information about whether it's nonetheless useful.
It is not clear to me that LASSO would serve my needs: I am not in a $p > n$ situation.
I wouldn't necessarily say that confidence intervals are preferred.  In fact the confidence intervals correspond to hypothesis tests.  TOST can be achieved by looking at the confidence intervals obtained by intersecting the two one sided confidence intervals that correspond to the two one-sided t tests that are used in the procedure.
Great answer! You might worry that gas stations are not independent even if the sample was random. One example might be the chains. Taking natural log of the price might be useful here to deal with the skew.
You need something that works symmetrically on (variable $-$ "middle") while also respecting sign. Nothing that you tried comes close if you don't have a "middle". Use median for "middle" and try cube root of the deviations, remembering to implement cube root as sign(.) * abs(.)^(1/3). No guarantees and very ad hoc, but it should push in the right direction.
Thank you, really helpfull.
Are you sure you have this right? For an inequality $P(X\\ge 5) = 1 - P(X \\le 4)$. For a strict inequality $P(X>5)$ means the probability that $X$ takes values $6,7,8,9\\ldots$, which is the same as $1$ minus the probability of X taking $0,1,2,3,4,5$. So, $P(X>5) = 1-P(X<6)$.
Given that you have derived the joint distribution of $X_1$ and $X_2$ you can easily determine the distribution of $X_1|X_2$. See http://en.wikipedia.org/wiki/Multivariate_normal_distribution#Conditional_distributions
An actually unbiased estimator of a transformed random variable can't be given in general without supplying the original distribution; in some circumstances, it may be possible to give approximations if some information is available.
If you want pairwise comparisons of the different concentration levels, and are testing gene expression levels over many genes, then SAM can be used for each pairwise comparison, and will give you honest statements of statistical significance for each comparison. You may then, should you wish, use these to rank the comparisons. Also, Welch's test is not fundamentally flawed. Sure, it just doesn't work well with n=3, but this is not what it claims to do. A bicycle pump is hopeless for peeling potatoes, but this doesn't mean you can conclude it is "fundamentally flawed".
http://cran.r-project.org/web/packages/circular/circular.pdf
The formula for $R^2$ looks incorrect, not just because it's missing some characters in the denominator: those "$-1$" terms don't belong.  The correct formula looks much more like an $F$ statistic :-).
Those are not the only two options
Yes in this case you should go for binomial test instead of using asymptotic chi squared test.
Thanks for posting this - this is surprisingly easy to understand, considering how technical it is - well done.
David: and so, what is $H_0$ ?
Why not compare to a poisson distrubtion for waiting times?
At the page most are categorical but there is an figure at the top of the page one could imagine a variety of derivatives of... Specifically, if you have a categorical and continuous predictor then the continuous predictor could easily flip sign if the categorical one is added and within each category the sign is different than for the overall score.
I need the proper and exact way for implementing it in matlab.
Apart from looking at various other posts that discuss consistency of OLS, the short answer is: No, if this "orthogonality" condition does not hold, then the OLS will be inconsistent, meaning that it will converge to a constant different than the true value of the parameter, i.e. it will have an asymptotic bias.
Thanks, this is what I was looking for
@suncoolsu I edit my post, maybe it is more clear now. @Macro: I am referring to something more general than that. I want to know what happens statistically when we constrain our model fitting according to properties of the model that we know are true.
I would say "highest variance" rather than "most informative". "Informative" can mean different things; sometimes the second (or later) PC is the most informative in a particular context. E.g. sometimes the first PC is a sort of general property of the observations that is already known while later ones are new discoveries.
Now you can post an answer :-)