tttjjj/elt_bench huggingface dataset:
+1. Fortunately, for explaining the condition number we already have [an outstanding thread](http://stats.stackexchange.com/questions/2691/making-sense-of-principal-component-analysis-eigenvectors-eigenvalues) on this site: it's the maximum distortion found in the second-order description of the design variables as a point cloud.  The greater the distortion, the more the points tend to lie within a subspace.  This geometric insight also shows why the conditioning of a centered design matrix is better than that of the raw design matrix itself.
`...but it should be higher` Why? And what is `simple MLR`? and what is $r$ in your context?
See [What, precisely, is a confidence interval?](http://stats.stackexchange.com/q/6652/32036)
Brilliant. This is exactly what I wanted and it's a lot simpler than I had anticipated. Thank you so much for both clarifying my problem and providing an elegant solution.
Scikit-learn is a popular machine learning library that also has some GMM support. I am not sure it fits your needs but it has the benefit of having other learning algorithms and framework (e.g. cross-validation, model composition).
+1 Really nice and patient explanation.
@whuber oh, right, good point thanks, I fixed it above.
I don't understand the meaning of your statement that you want to use the distribution "to predict a p-value for every element in this dataset."  That makes no sense.
Great, exactly what I was looking for, thanks a lot Glen_b!!! And indeed, seems to confirm that this can be done with a QR!
I am unclear what is the issue or the problem you are addressing.  Given that, I am not sure that linear regression is the proper methodology.  Maybe an hypothesis testing framework using ANOVA to figure out if those groups are statistically different would be a better approach followed up with related Post Hoc test to measure the statistical difference for each paired combination of States.
Your data show that timings for algo1 tend to be around $5171$ give or take $\\sqrt{782}=28$ while timings for algo2 tend to be around $3753$ give or take $\\sqrt{1920}=44$.  The mean timings are over $1400$ ms apart, which is more than an order of magnitude greater than the typical variation.  Such a difference is so blatantly obvious that no further testing is needed.
Yeah I did, but you know how it is when you sit with a problem too long, or with something that you _know_ that you know, but still can't do it. Anyway, my eyes were opened =)
I have started a new bounty to award this answer.
@Tomas Thanks for your interest, sorry the question did not live up to the title. There are however many conflicting ideas on this subject and I thought perhaps useful question. Although i am interested in you statements regarding testing the random effect, i.e. if the variance is zero and whether or not the definition of deviance on the `glm` and `glmer` are the same. Best.
http://statweb.stanford.edu/~tibs/ElemStatLearn/ There you can find an online version of the book.
(+1) Your 2nd paragraph is particularly well formulated (I initially interpreted the question as an experimental study with *a priori* hypotheses); I also like the 3rd paragraph, and I recommend this approach when finding interesting but unexpected interaction effects. Finally, I don't know about any application of cross-validation in ANOVA designs: do you have any references?
@M.Berk Isn't how we choose to define $0^0$ irrelevant for this question?
I know nothing about LASSO, but assume that the slope should not equal one. You haven't just flipped the axis of your calibration plot and the slope is >1?
@alto thanks for your useful comments. i think its because of my data that the net does not train well.
Hi, Cam. It would be helpful to see some output to potentially help diagnose things. Note that what you've done is recoded things by creating two main effects and effectively removing an interaction (though the model is equivalent). Also, are you controlling for exposure or just modeling raw counts? If the latter, that would probably explain it: There just aren't many pipe/cigar smokers compared to nonsmokers and cigarette smokers in the first place.
Thanks for such a quick answer, I guess this is fair enough. I know that if I increase the number of nodes in my hidden layer I get better convergence as there is less chance of having local minima to get stuck in... I think I wanted some kind of a quantitative answer: like when do I stop increasing the number of nodes? I guess I can just rely on common sense and if added computational time for more nodes gains little improvement, than it is OK to stop. Would you mind to migrate it to stats forum if you think it is best? I'll mark your answer if I get no helpful replies...
Where do these data come from? Why do you need to know what distribution they are (ie, what is your end goal)? What are the "really different results", & what is the impact of them? Do you need a formal test, or are you looking exploratatively for a better understanding?
Are you after something like a (binomial) [confidence interval for a proportion](http://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval)? Or do you want error bars that represent something else?
@OSE - in case you are disinclined to follow gung's pointer, - FA explains, by a few factors, pairwise covariances (or correlations) between the variables. PCA explains, by a few components, the bulk of the multivariate variance. PCA _may_ explain covariances too, sometimes, but it's not guaranteed.
I have different experiences with NCA -- it usually converges quite qickly for me. Checkout "dimensionality reduction via learning an invariant mapping" by LeCun and "Minimal Loss Hashing for Compact Binary Codes" by Norouzi.
@F.Tusell: Forgot to mention, *T* and the set of time series in *U* are all non-stationary. I have updated my question accordingly.
And the dirichlet with the multinomial
Naturally there are other ways to do this. One is show fractions rather than amounts. Another is to show each time series separately rather than cumulatively. Another is to use a transformed scale. These methods can be combined.
It's unclear what you mean by BMI categories, but it sounds like you want to correlate physical activity level with the discrepancy between body image perceptions and body image reality (which may be fairly independent of BMI – seems it would've been good to have data on body fat and muscle percentage)...This would give you an effect size estimate for the relationship between activity and some operationalization of body image accuracy. How to perform that calculation depends on the levels of measurement, which it would be helpful to specify as well. Are body image perceptions based on surveys?
+1 nice answer.  I was sure org mode could do this but I didn't know how
For those previously unfamiliar with random variables or ticket-in-a-box models, a quick interactive tutorial on my Web site at http://www.quantdec.com/envstats/notes/class_06/tutorial.htm provides practice and some additional concepts.
Why would one need to invoke the CLT when sampling from a known (or assumed) Gaussian distribution?  The mean of even a sample of one will be Normally distributed!
If you have another data set with more than 1000 data points. Here a pca model with 1000 components might contain more information.
Have you tried [Random Forests] (http://en.wikipedia.org/wiki/Random_forest) as well?  They are based on the combination of decision trees. It might get better results.
Are these data responses to a Likert-type item? If so, could you please tell us a little more on the questionnaire that's being used?
@Glen_b I think it's conditional, you pick out a card, and then you work out the probability of the next one being greater.
Because if I know the distribution of ${(X-\\mu_X)} \\over \\sigma$, (normal(0, 1), of course), I want to be able to use the distribution of ${(X-m)} \\over s$, which is the propre prior distribution to use when these $X$ are themselves parameters in a Bayesian model. See an example of this use in Gelman's regression textbook (2007) about IRT models, when he uses this trick to get a better convergence of the IRT coefficients ; I'd like to have an explicit derivation instead of blind faith in BUGS sampling. PS : you're right $m \\over s$ has a $t(\\mu_x, 1, N-1)$ distribution.
Just here: http://www.bayesianregression.com/bmr.html
+1, good question.
Unfortunately, this answer is not relevant to the question asked which is about a _mixture_ distribution (weighted sum of two Poisson pmfs with weights summing to $1$) and not about the sum of two Poisson random variables.
I'm not afraid of for loops, it's everything else that throws me off :). What is this continuity correction everyone is talking about? According to Wikipedia, you add 1/2, not subtract it. I also don't understand the purpose of your code. You made a matrix that is 101 columns wide by 606 rows?
Because in my particular case I had only the possibility to determine the total sum of the $x_i$. And without knowing the $x_i$ I was trying to minimize the scoring function which was the absolute deviation from a fixed number I provide.
@Tom Yes, what you say is evident in a plot of the data. (Indeed, a plot shows three separate sudden level shifts, not just one). But characterizing this as differing from a "trend" does not do justice to your analysis, which I know will reveal subtle details in the behavior of this time series. I would like to suggest the O.P. would be better served by clear characterizations of the data behavior rather than by discussions of possible definitions of "trend." She asks for an alternative to testing a least squares slope--and that is a *de facto* indication of what *she* means by "trend."
I've added an EDIT section to my answer to address this issue, I hope it helps...
Why do you think that you need to remove the outliers? Do you have reason to believe that they are "bad" data, i.e. data entry errors, etc? In general, 0 is a reasonable number of outliers to remove.
Your question is based on an incorrect characterization of linear classifier and as such will be difficult, if not impossible, to answer in any meaningful way: the value of a linear classifier, *by definition*, is a [linear combination of a feature's numerical characteristics.](http://en.wikipedia.org/wiki/Linear_classifier)  This is independent of its correctness.
Perhaps you need to think about whether you are trying to model whether a customer orders at all (logistic regression). Or whether your work is about modelling which product (if any) a customer orders (multinomial logistic regression). You might want some extra columns in your data (e.g. price).
You never fail to amaze me.
Thanks for your reply and for the statistical background
1. What's the size of n in your model? 2. whats j? a covariate, right? 3. I thought your dependent (response) variable was binary. 4. Why mu.y and demVote? 5. Fit a simple regression (not hierarchical) in  Bugs and compare with classical regression. They should be similar. And fit a quick hierarchical model in R with lmer function in lme4 pacage...
@RobertKubrick, are you aware of [Wold decomposition theorem](http://en.wikipedia.org/wiki/Wold_decomposition_theorem)? Each stationary process has its corresponding innovation process, that is from where terms $e_t$ come.
Thank you for the editing! I did not know how to format the references. It looks much nicer now.
These are two very different classification problems. Why do you think it make sense to compare the success rates?
Just with my eyes. For example one instrument can have one very large outlier, second instrument can have more outliers but less large. For me second instrument is better.
Ok, I got it ! Thanks a lot. I'll wait for an answer to the first question before accepting an answer.
Is this related to your question here? http://stats.stackexchange.com/questions/57468/issues-with-solving-large-sparse-linear-equations In general are you totally positive that there isn't something you are trying solve? Usually getting $A^{-1}$ is a bit hard. If you try to replicate a paper you might as well mention it so we might be able to help more substantially. (Also try the http://scicomp.stackexchange.com/ you might be pleasantly surprised.)
Man, AP tests are just getting harder and harder... or maybe students are getting smarter.
Thanks for this answer and also for the very interesting and informative discussion by telephone earlier. The Autobox forecasting environment is remarkably sophisticated.
> thanks, first time i've eared of this idea.
In light of @rolando2's comment, it may be more helpful to try what you can, & then ask a series of well-focused questions about the places where you're stuck (assuming those issues haven't already been covered elsewhere on CV). For each question, you would state what you've done thus far, where you're stuck, & what you understand about that issue based on your own study of statistics.
See the answers too http://stats.stackexchange.com/questions/108705/deriving-the-bivariate-poisson-distribution/108724#108724
This is exactly what I needed, thank you! I used lsqcurvefit in _Matlab_, it does not output the confidence interval or standard error. It gives the Lagrange multipliers (?), the residuals and the squared 2-norm of the residuals. Now with that and your answer I can calculate what I need!
Thanks!  If I understand correctly, the GLM model will give us a way to infer the parameter of the Binomial (e.g., $p_i$ or $p$); this might be done, e.g., by logit or probit regression on the equation $x \\sim 1$ (no independent vars). Now how do I get a $p$-value for the null hypothesis?  And how do I characterize, quantitatively, the degree of internal consistency among the reviews?  Can you elaborate on that at all?  Also, how is logit/probit regression preferable to simply taking $p_i=X_i/10$ or $p=(X_1+\\dots+X_{60})/600$ as our estimate of the parameter of the Binomial distributions?
@A Friend, thanks for that post-very short but quite precise and easy to understand. So I guess its the function A(Z) that makes it non-linear. Quite different from the definition of linear/non-linear in differential equations.
@Frank Harrell - Agreed. The paper I referenced applies to the area I work in sometimes and I have used the bootstrap for the scenario you stated: assessing a particular model's variability due to sampling error. But the uncertainty due to model _selection_ itself is even harder to assess and the bootstrap model averaging approach could be useful as an aid, especially for practitioners like myself who lack the experience/background to reformulate problems for Bayesian model averaging.
Incidentally - with such a large sample size, even trivial differences in distribution will show up as statistically significant. You should focus on what matters in practical terms.
It's not clear what you actually need. When you say "*In which way can I give a number "similar" to the covariance of these vectors?*", but don't appear to want the covariance, it's not at all clear what you *do* what, or what you would regard as 'useful' by contrast. Please clarify your question.
This problem is solved for any positive integral $n$ by taking the $n^\\text{th}$ root of the characteristic function of the given Gamma distribution.  This yields a c.f. which--by inspection--is easily seen to correspond to (another) Gamma distribution (with the same scale parameter but a different shape parameter).
Is the category identified before looking at the data? Or do you mean 'in any category'?
I have made some (minor, I believe) formatting edits to your post, @JoeKing. Please make sure you are OK with the results.
It is just a difficult topic (perhaps too large in scope to easily answer on this site). In my experience it is more common for people to ignore such error than to deal with it. Slightly different context (error is from interpolation, not admitting initial measurement error), but here are two papers I am familiar with ([Anselin & Lozano-Gracia, 2009](http://dx.doi.org/10.1007/978-3-7908-2070-6_2); [Foster et al., 2012](http://dx.doi.org/10.1111/j.1467-9876.2011.01030.x)) that may be of interest.
Well i am new to spss so pardon me if i am unable to get much of it  , but what i get from the last para is to convert my ordinal IVs to scale variable and treat them as numerical value to apply ordinal regression. Also i have only 15 cases in my data so i guess that is a pretty small sample. please, Correct me if i am wrong
+1 It's a long post, but at least it's clear that you're really thinking carefully about the problem.  I agree wholeheartedly with @John's answer, although this can be a hard sell with some traditionally trained bosses / supervisors / reviewers.  If you do decide on multiple comparisons, you can use `p.adjust` with (say `type="BH"` to control false discovery rate -- very popular nowadays because it's not as conservative as methods that control experimentwise error ...)
David, can you explain to me please what's the difference between full matching and 1-to-1 matching?
Ah yes and of course the constant. But thank-you. This confirms that I have done something else wrong. If you would like to form that as an answer I will accept it :) Thank-you for your advice, and again sorry for not being specific enough initially.
$A$ and $B$ can't magically change with time to make $X(t)$ constant. That's not how random variables work.
@Yeti It would be best if you could include those plots into your question by editing it. Just from your description, it is hard to judge what's going on.
@mark999 Personality traits and management constructs like LMX are typically measured using multi-item scales (e.g., a 10 item scale with Likert responses with 5 options for each item). Researchers typically perform analyses that treat these variables as numeric variables (e.g., multiple regression and correlation). For most purposes, I think this is preferable. I'd be a little bit more worried about the DVs in this example. Amount of sales and repeat sales could be quite positively skewed.
: Second, if we know the
future value of some covariates (e.g. blood pressure), the existence of the values
implies that the subject has not reached a death endpoint. This fact implies that
one cannot estimate the survival curve from the observed and future values of a
quantity such as blood pressure. The existence of a positive value would imply
that the subject was still alive. Knowing the covariate would imply knowledge
at each stage of the vital status.
`ggplot(x, aes(value, fill=variable)) + geom_density()` like that?
+1: It doesn't mention Winnie the Pooh, but it is quite illustrative none-the-less.
Why would you be squaring the variances in the second code snippet? Or are your variables just oddly named?
(+1) This answer hits all the main points.
[David Spiegelhalter](http://www.statslab.cam.ac.uk/Dept/People/Spiegelhalter/davids.html) has been working on risk and uncertainty and how to communicate these ideas. I do not believe there is a general answer to this question because it strongly depends on the context, mathematical-statistical tools developed at the moment, understanding of the phenomenon in question, ...
I like this, and I think a slight tweak will improve it further: suppose that if the coin is "heads", one will be awoken on Monday, and if it's tails one will be awoken on Tuesday and again on Wednesday.  There are three days when one may wake up, and all three are equally likely.  The times one wakes up on Monday, the coin will have been heads; on Tuesday or Wednesday, tails.
Depends what is your goal and what constrains do you have. For example, if you have fixed sample size Neyman Allocation will allow to estimate population total with minimum variance.
I posted a reply for almost the same question [here][1]\\n\\n\\n  [1]: http://stats.stackexchange.com/questions/9253/nu-svm-parameter-selection?rq=1
@Nick: As single measurement per device per sample. While measurements are not taken over time, there are two groups of samples (n = 50+) with a differing property, it is this property which is being explored. No current error model for the devices.
Ok so the variable selection procedure resulted in retaining only betas which were well above their real value (0.5). But presumably it isn't always the case that retained betas are well above their actual value? I presume their closeness to their real value depends on sample size and variation in the response variable?
thanks Cam, will take a look at it.
I assumed that $X_i \\sim N(6, 4)$ meant ${\\rm var}(X_i) = 4$ and used ``pnorm(8, 6, 2)`` in ``R``.
Sounds like a good idea (at least to get baseline). So when I build a classifier to distinguish class $X$ from the rest of the classes (say $Y$) what would the training data for $Y$ be? I mean for class $X$, I have documents that are labeled as class $X$, but what about $Y$?
Have you tried doing a parallel analysis?  My understanding is that it can empirically provide a criteria that is more stringent than Eigenvalues == 1 (which is known to over factor).
Assuming you already have a scatterplot of your two variables, x *vs.* y, with origin at (0,0), simply draw two lines at x=mean(x) (vertical) and y=mean(x) (horizontal): using this new system of coordinates (origin is at (mean(x),mean(y)), put a "+" sign in the top-right and bottom-left quadrants, a "-" sign in the two other quadrants; you got the sign of the covariance, which is basically [what @Peter said](http://bit.ly/v1WX5s). Scaling the x- and y-units (by SD) lead to a more interpretable summary, as discussed in the [ensuing thread](http://bit.ly/vjOfeG).
Good-Turing should have a hyphen - not least, to distinguish the first name from the adjective. You may also want to replace \\`N_1/N\\` with \\$N_1/N\\$ which will be rendered as $N_1/N$.
Indeed, post edited.
What would it mean for this table (or another table like it) to be valid or invalid?
@Moderator, thank you for migrating this question as a duplicate to  CrossValidated. There are many smart people here, hopefully one or more will offer a reply to this question as I still don't have answers to this.
Correlated with what? Each obs to the one before it?
@cathy you might have noticed that my comment was a joke capitalizing on play of words in your qiestion
You added the `spss` tag, but your syntax looks like you are looking for an R solution. Correct me if I am wrong...
Because ensemble learning doesn't always give better performance. Both bagging and boosting work in some cases but can severely degrade performance in others.
@DWin, @David This is *not* a repeated measurement situation.  The data format only makes it look like that.  The measurement for each subject consists of a single interval during which a transition was observed.
it is deterministic and continuous.
@Henrik for small $n$, the t-statistic is appropriate, and as you said, as $n$ increases, t approximates Z. See also http://math.stackexchange.com/q/23246/3733
A linear interpolator will surely work :-).
It seems you could use the result of [your previous question](http://stats.stackexchange.com/questions/14916/how-can-it-be-proved-that-if-number-of-samples-are-less-than-d1-then-the-samp) to develop a brute force $O((n-d-1)\\binom{n}{d+1})$ algorithm.
If your approach is to treat the "significance" of a single pair as enough to conclude that the gene is significant as a whole then I think you do need to be jointly testing all three pairs (e.g., using an ANOVA if your data is appropriate).  To test each separately and controlled under separate FDR seems to me to be an unambiguous violation of what you are wanting to achieve by controlling the FDR.
Dear Karl, thanks a lot. I found your comment very useful. Many thanks
@whuber Please let me know if something remains unclear.
@drnexus: I did not want to bias the results by mentioning my technique. I learned about the g-and-h and the g-and-k distributions from Haynes et. al, http://dx.doi.org/10.1016/S0378-3758(97)00050-5 , and Fisher & Klein, http://econstor.eu/bitstream/10419/29578/1/614055873.pdf
Your intuition is correct. Experiment on it a bit and you'll get it. Having said that, I think the StackOverflow is the appropriate audience for this post and not CrossValidated.