tttjjj/elt_bench huggingface dataset:
Can you say how many rows & columns you have in your dataset? Are your rows observations & your columns measurements (as is typical)? How many factors are you trying to extract?
"the formula for finding $E[g(Z)]$ using the law of $Z$" is called the _[the law of the unconscious statistician](http://en.wikipedia.org/wiki/Law_of_the_unconscious_statistician)_.
From what you have shown it appears you do not have enough data to support estimates in a model of that complexity. All of the table entries that you have shown are zeroes. You have 1656 main effects and that doesn't even include the 2-way and three-way interactions.
Please, don't cross-post. Choose one site once and for all. I will delete this one by the end of the week without action from your part.
@Francesco PCA operates on the covariance matrix already. The rotation matrix is just the eigenvectors of the covariance matrix of the data you provided.
The question has ANCOVA in the title... is this in any way related to the problem?
If you have the density function for the minimum, then it's exactly the same as doing the calculation for an arbitrary random variable, $X$. Are you asking how to derive the density of the minimum?
Check out the CUV library or Eblearn (more here http://deeplearning.net/software_links/) The Python libs might be a good alternative to R.
Thank you! Ended up going with this http://www.icpsr.umich.edu/icpsrweb/ICPSR/studies/29182?permit%5B0%5D=AVAILABLE&q=exit+polls&paging.rows=25&sortBy=7
It seems to me that your variance ratio depends on the units of measurement of X and Y and therefore is arbitrary.  You have concluded Y is "weaker" simply because its scale has a smaller range than that of X.
@stan No. (Note: Unplanned (post-hoc) tests should be performed after the ANOVA showed a significant result, especially if it concerns a confirmatory approach.)
Are you doing one mixed within-between ANOVA or are you trying to compare between two separate RM-ANOVAs?
Wrong site - I think you want math.stackexchange, tag statistics...
When you say "the same but with a time shift" you mean $y(t) = x(t-\\tau)$ (where the shift is $\\tau$), with no error, no rescaling? Or something else? Can you make your question more precise? What sort of alternatives would you want power against?
Why 2 times the square root, when the canonical link of the poisson family in a glm is log?  Shouldn't it be exp() of the linear predictor?  But I don't see what the problem is with plotting residuals against the linear predictor itself, which I think is what is being done here - perhaps you could expand on that.
Basically I have a data set comprising of: Groups (A,B,C...) - each group contains 3 different genotypes (1,2,3,..). The same genotype doesn't occur in two different groups. I then have several measurements of height for each genotype. I had to perform an analysis that can test for differences between the three groups as well as for the genotypes within each of the groups. I chose nested ANOVA, but now I'm slightly doubting myself. A huge thank you for any help!
It's not quite clear what you seek to do. Without narrowing the problem down, there's no obvious distribution over nominal categories that would induce an ordering *other* than one based on the empirical frequencies.
It also seems as if $y$ is always larger than some linear function of $x$, $y>C x$.
The spherical symmetry immediately implies all marginals are uncorrelated and have equal covariances, reducing the problem to calculating the variance of a single marginal.  But first, could you clear something up?  What is $n$ and what precisely do you mean by a "uniform spherical distribution"?  Is this uniform on the *unit ball* in $\\mathbb{R}^n$ or on the *unit sphere* (in $\\mathbb{R}^n$, presumably)?  I think it's the latter, because (by definition) the sum of squares equals $1$, immediately implying the expected sum of squares of one marginal is $1/n$ and that's the variance you seek.
@I'll ask Santa Klaus
Leave-one-out estimators are unbiased, while 10fold cross-validation will tend to give you a biased (towards lower errors). However, the unbiasedness comes at the price of a high variance.
This is a Markdown-enabled website with $\\LaTeX$ support, further info can be found here: http://stats.stackexchange.com/editing-help.
Further to Jeromy's comment, could I recommend that you have a read  Ziliac and McCloskey's essay on the cult of statistical significance. It's not the most mind-blowing statistics, but it does provide thoughtful--and entertaining--discussion of why effect sizes, practical significance, and loss functions are extremely important. http://www.deirdremccloskey.com/docs/jsm.pdf
@Glen_b, yes. In the example given in the question, I have 3 categories and each category has a fixed success probability of 33.33%. So, it can be considered a multinominal distribution. To generalize the question, it is exactly what you said: calculate the sample size (or number of trials) to assure that the range of a multinomial distribution lies within given bounds with a given confidence.
Velcome to the site!
@leonardo I am not clear on what you mean by "divide the parameter space into sections" or "walking through the parameters". Minimizing the rmse sounds fine as a goal, but how to achieve it? This seems like it may be a problem good for gradient descent, but I am not sure how I would code it.
I think you'd be hard-pressed to conclude _anything_ is normally distributed from just 8 observations....
@Shelagh Perhaps it is "trendless over all" but may contain local trends. Perhaps there are "level/step shifts" . Perhaps you could post an example time series. Often free software in the area of time series is not worth what you paid for it.
Thank you Teresa !
In case you gather enough experience on this to answer your own question I would really appreciate. This is one of the classic areas where research papers do not explain the important details. (Also, thanks for the bounty :)
Also, with regards to functional form assumptions in RD when using a polynomial, how do you choose the polynomial?  Can some polynomials be "wrong" to choose?
May I suggest that you edit your main question to include this information?
Indeed, as I was pointed out, the interest in (3) is more conceptual than computational. But this leaves my first question open: what makes e.g. (2) more "tractable" than (1) or (3)?
The Chow test assumes that there are no gaussian violations in either the sub-models or in the composite model. Have you verified the gaussian assumptions?
I am not sure I understand your interrogation. Could you have a look at http://stats.stackexchange.com/questions/11956/why-is-the-first-postulate-of-the-poisson-process-that-lambda-dt-is-the-proba?rq=1 and see if it answers your question?
I made significant edits to your question, especially regarding title and tags. Please, make sure I didn't alter its original meaning.
As a counterpoint to this pessimistic assessment (which does have some justification in rules of thumb such as requiring 5-10 times as many observations as variables), it is noteworthy that entire fields of modeling and data analysis have grown up around problems that have few observations and many predictors, such as DACE (design and analysis of computer experiments).  See http://www.ressources-actuarielles.net/EXT/ISFA/1226.nsf/9c8e3fd4d8874d60c1257052003eced6/f84f7ac703bf5862c12576d8002f5259/$FILE/Jones98.pdf for a seminal paper.
@ARTICLE{pet89ord,
  author = {Peterson, Bercedis},
  year = 1989,
  title = {Re: {Ordinal} regression models for epidemiologic data},
  journal = Am J Epi,
  volume = 129,
  pages = {745-748},
  annote = {proportional odds model; partial proportional odds}
}
@ARTICLE{mcc80reg,
  author = {{McCullagh}, Peter},
  year = 1980,
  title = {Regression models for ordinal data},
  journal = JRSSB,
  volume = 42,
  pages = {109-142},
  annote = {ordinal logistic model}
}See also Whitehead Stat in Med 1993 p. 2257
If you want to know about how much ice cream men and women bought, you have to look at how much ice cream men and women bought. Means, standard deviations, medians, t-tests and so on.
$D^{+}$ and $D^{-}$ are common names for the one-sided statistics (cf http://www.springerlink.com/content/y460k248m8w1822j/ ).  In what context does your $D'$ appear?
@Simon - Sorry for the lack of clarity.  I need the number of free parameters of G for inclusion in the BIC.  The number of nodes is fixed, but the number of edges can vary arbitrarily (with optional limits on the number of parents).  I could think of a few different reasonable ways to consider the number of free parameters (free parameters = # edges seems to be a weak criteria, free parameters = 1 parameter (for the number of edges) + # possible locations for these edges tends to work better), but do not know the mathematically correct model.
Already noted here: http://stats.stackexchange.com/questions/1906/data-mining-conferences
Thanks, this was certainly helpful.  However I'm still unclear which of the two answers (pooled vs separate) is more "correct".  Unfortunately, the differences only arise when you are so far away from the mean that it's a bit hard to simulate.
About 15-20 observations in each group. SD is higher than the mean in most cases. For instance, Group1: mean 10, sd 11, Group2: mean .20, sd .40.
I agree with @MichaelBishop - plotting is essential, especially when there is an interaction
Maybe the cross-correlation function (correlation between two time series at various lags) would give you what you want. The function is `ccf()`. Run `?ccf` for details on how it works.
Thanks, it works great. Did not know doBy had those capabilities.
This is most definitely a forest plot: http://en.wikipedia.org/wiki/Forest_plot  They are most commonly used in meta analysis to compare estimates from different studies.
If your aim is to judge how much that citing article is "important" or "good" by itself, you could always try reading it...
@Ananth So you must only manually make a recommender from a model.
Also, incidentally see my parallel analysis package `paran` for R (on CRAN) and for Stata (within Stata type findit paran).
The ymd, dmy, etc. functions in this package look close, but don't quite solve the problem.  Better than the base POSIX commands, though...
@kjetil I am so desperate for answers that I didnt explain my problem clearly. Sorry for that. Question is edited.
If you just highlight the data you're interested in displaying and then click on insert you'll be presented with a bunch of options you can experiment with.You need to describe more precisely what you want your graph to represent, e.g. the graph will display a single variable. the x-axis will break the range of the variable into bins and the y axis will represent the number of cases which fit in each bin.
@gung This seems like an answer to me, albeit perhaps a little too brief.  Please allow me also to welcome you, Carmelo.  Perhaps you could be persuaded to amplify your answer a little by (a) explaining how that computation would be carried out, (b) explaining how to compare the result to a reference, and (c) explaining why you think this would be an effective or useful method?
Thanks! Very instructive. What do you mean by *"it only is in the normal linear model"* ? For a Fisher test when $B_0$ and $B_1$ are linear subspaces, then the deviance is a monotone function of the Fisher statistic, and it is only *asymptotically* $\\chi^2$.
Hi, I think the question is clear: I need the correlation between model's scores and human scores (the way I got the model scores is irrelevant). If we had only one human, the solution is trivial. Now we have more than one human, what is the appropriate way to get the correlation of all human scores with model scores?
@berkay A simple algorithm for finding the No. clusters is to compute the average WSS for 20 runs of k-means on an increasing number of clusters (starting with 2, and ending with say 9 or 10), and keep the solution that has minimal WSS over this clusters set. Another method is the [Gap statistic](http://gremlin1.gdcb.iastate.edu/MIP/gene/MicroarrayData/gapstatistics.pdf). But if you already have labelled instances, then why are you trying an unsupervised method?
@whuber I wonder if user30682 is talking about fusing different secondary datasets. For instance, taking some county-wide health data from CDC and merge it with Census' 2010 county data for analysis, etc Since both datasets have their own sampling errors, the challenge is then how  the analyst should evaluate the merged data's sampling error.
Why don't you read over the page you linked to in your comment? It does a wonderful job of describing how to perform, interpret, and check an ordinal logistic regression with `R` (using the `MASS`, `HMisc`, and `GGPlot` packages).
@Mynah: weal, you've added a few restrictions to the question...
@Peter Eilis : my observations are monthly discharge time series so  it is univariate and yes by innovations I mean residuals of my ARMA model (in my case I have fitted an AR(1) )
@DWin, I have just edited my question.
Please don't re-post the same question multiple times. It sometimes takes a while before someone provides an answer.
To do this analysis properly, I suggest taking a look at Andrew Gelman & Hill's "Data Analysis using Multilevel/Hierarchical Models." However, it seems that the number of different 10 card sets is too big to take into account individually.
@Andre The *data* have a Binomial$(1,p)$ distribution while a particular *statistic*--their sum--has a Binomial$(n,p)$ distribution. To that extent the language in this answer is correct. It also contains a lot of relevant information--thank you, Sofia! In its present form it falls a little short of addressing the question, though. It should be pointed out that because $\\sqrt{p(1-p)}\\le 1/2$ and $p+\\sqrt{p(1-p)}$ never exceeds $1.20711$, the "standard deviation bars" of the question cannot possibly extend to $1.3$. Thus *we need clarification from the OP* in order to develop good answers.
interesting ! there are testing images to try it, exercices to learn how to use it...
Thanks.  You have a typo, then in that you say PCA reduced from 1500 to 30. You mean 500 to 30.
@tomka makes some good points, but I have some additional comments. Look at the residual correlation matrix. Fit statistics in SEM can be misleading, because they only quantify model mis-specification in one value but do not tell you where it is. Both this, and looking at which items have large error variances, is a great way to detect problematic items. And to add another question, what is your sample size?
The issue has been explained in a comment to the question by @Macro: that your models 1, 2, and 3 are all different is made clear by writing them in a way that makes the nature of random variation explicit. This is no different than noticing that regressing $Y$ on $X$ differs from regressing $X$ on $Y$.
Although I've accepted an answer, I would recommend that interested people should look at all the answers.
"Should" applies to regular cases, up to convergence tolerance. However, in "irregular" cases like near singular design matrix or perfect separation, the difference in implementation will show up. But then the statistical result is ambiguous anyway.
Hello rvl, how is the interpretation after analysis with transnformed variable? can we interpret in terms of original scale of variable?
@whuber The quiz offered is dependent on whether the results from the previous day were (G)ood or (B)ad. The transition matrix is what is given in my notes (it took me quite some time to confirm the probabilities). This is all the information I have. Steps 2 and especially 3 are my main worry.
What hypothesis? What model? What is it you're actually trying to do, in plain, simple terms?
@Tom - you will need to be more specific. This is just a linear regression equation with a set of variables. It should be pretty simple to estimate the equation in any software once you have your data.
A t-test is just an ANOVA with only 2 groups. ANOVA is also parametric. The assumption of random selection isn't related to parametric vs. non-parametric; it's related to how generalizable the results are. You need more variables to try to control for the non-random selection.
Conclusions drawn from the mixed model are intended to be generalised to the population from which your experimental units come from. Conclusions drawn from the "individual model" are only valid for one specific unit. It is a little bit like comparing some average $\\bar{x}$ with one single observation $x_i$.
@whuber Reg chi-square tests: Done. I feel that any mention of KS test and alternatives to chi-square should be added to your answer instead of mine as the idea is yours.
If you could share a fragment of your R code, at least the part that calls JAGS, that would help construct some R code that would address your "how it is done" question in the context of your code, as there are several ways you could be doing this.
Two-way ANOVA doesn't seem quite right, as the goal seems to be to compare hospital and provider rather than look at their separate effects and potential interaction.
I guess sum(range(your_variable))/2 should do the trick
Hmmm.  I see. There's probably an analytic solution, although I don't know what it is. You could also simulate it. I think it might depend on size of parties and number of people who were at both, but that's just my intuition.
+1 Thanks! However, I can see how this approach could get out of hand if the Bayesian Network had 5+ nodes and arc.
@vikkor I expanded my answer to add a few remarks about this.
I could not be more grateful. `schools.sim$summary[,3]` gives exactly what I want. Thank you very much
Thank you John for your answer, I edited my question above adding more informations as you suggested me.
I would bet he assumes the process is *Gaussian,* right?
Clair, I want to acknowledge that it's obvious you did put a lot of thought & effort into formulating the question as best as possible. If only every CV user did the same! However, statistical questions can hinge on details that are not obvious unless you are very thoroughly versed in the underlying ideas. When you're trying to learn these things, you are not thoroughly versed in this manner, by definition. The resulting conversation & proffered answers are intended to help bring these obscure underlying details out, not as criticism.
I second whuber: in matlab you have an estimator of regression with the same breakdown that can handle exact fits [here](http://www.riani.it/MATLAB/FSDA_working_paper.pdf) and also [here](http://automatica.dei.unipd.it/public/Schenato/PSC/2010_2011/gruppo4-Building_termo_identification/IdentificazioneTermodinamica20072008/Biblio/Articoli/LIBRA%20a%20MATLAB%20Library%20for%20Robust%20Analysis.pdf)
(+1) But why do you expect to get exactly 95%? Clopper Pearson for example is guaranteed to be conservative. For your $n$ and $p$, I get that the CI should cover the true value 95.3648% of the time.
+1 for beginning your question with a statement of your research objective. Questions on this site can often benefit from an opening statement such as this, putting into context the statistical detail that follows.
The overall test across multiple years and both classes would be a chi-square test of homogeneity. Across a pair of years, @jsk's suggestion of a two sample proportions test makes sense; it's equivalent to a 2x2 chi-square but is easier to make one tailed.
@NickSabbe In order for the individual items to be added, or scaled and added - as the OP has already done to obtain the factors - the items have *already been assumed* to be interval-scaled. I see no issue with then computing correlations with the resulting quantities; no additional assumption would be needed.
I have the impression that the data simply do not conform to the model, e.g., you have some extremely high correlations between the factors. Try to look at a standardized solution to get correlations instead of covariances (and at standardized loadings, too). Maybe you want to collapse some factors? Maybe you want to add a method factor for the reverse-coded items if you have some--that often improves fit considerably.
"Inefficient" is in the statistical sense (i.e., low precision/high standard error) not in the computational sense.  100 repeats of 10-fold cross-validation, like the bootstrap, can be quite efficient.
(+1) Although I have offered a different answer--on the premise it might indeed be "fun to watch"--there is much wisdom in this one; careful study of its points would be worthwhile.
+1 Great idea - I guess I could try out combinations of words / counts and remove my common words like And The A etc.
Good answer. But can you please tell me how i should to choose range for attributes. Should be range 1-10, 0-1 or 1-100. How can i decide about that.
:chi . Now I see ..It looks like a nearly identical answer ! Thanks for pulling it.
This question seems relevant to your problem: http://stats.stackexchange.com/questions/203/group-differences-on-a-five-point-likert-item
As Glen pointed out, no. However, to solve your problem you can do one of two things. (1) At each iteration, change the scale so that all values fit or (2) scale the mean from the beginning.If you know the database values can't be larger than say, 100, then at each iteration divide the mean by 100. That way you don't have to mess with the y scale every time.
Where did you hear this statement?
In what way are you using **optim** (with what options, with or without supplying a gradient function, etc)?? Logistic regression is a smooth convex problem. It's readily solved using Newton's method or similar. In fact, to get an estimate of the covariance matrix, you *need* to do (something close to) this.
It would be nice to see the height-weight scatterplot of your data, with two sexes coloured differently.
One suggestion I can make is that you should not use varimax rotation (or any other rotation that assumes orthogonal factors), it is *much* better to use an oblique rotation, such as direct quartimin.  First, it is not clear why the underlying factors *must* be orthogonal; second, oblique methods allow you to estimate the factor correlations; and third, if the factors are orthogonal, an oblique rotation will return fundamentally the same results as varimax.  On the other hand, if the underlying factors are *not* orthogonal, then varimax can yield dangerously flawed results.
Not a good idea to demonstrate an untrustworthy platform, in my humble opinion.  Plus you will limit yourself quit a bit in terms of features implemented.
But "always" comprises exactly what scenarios?  *You* need to stipulate the circumstances and assumptions to be made, not your respondents! Otherwise some people will assume you are thinking of $X$ as a random variable and others will assume it is not random; some may assume it could be a vector of values and others that it is just a single (scalar) value; etc.
Classification is not a good goal and is not the way logistic regression was developed.  It is the notion of classification that causes all the problems listed here.  Stick to predicted probabilities and proper accuracy scoring rules
Suppose you are updating $p_j$ to $p_j^{\\textrm{new}}$. How about just setting $p_i^{\\textrm{new}}=\\frac{p_i}{p_j^{\\textrm{new}}+\\sum_{k\\neq j}p_k}$?
When n < p, LASSO selects at most n variables.
Thanks a lot, that sounds quite promising. I'll take a look into it.
+1 Thanks, Douglas!
+1 As I know MATLAB, I think your hypothesis is (sadly) true.
+1. There are many, many names for this plot. Dot and strip plots or charts are two of the most common. As "dot" is widely used in other senses, the name element "strip" is perhaps preferable, at least in searching for implementations (there are several, not just in R but also in all leading packages or languages, e.g. Stata `stripplot`).
As far as I can tell, there is no way to do it without $\\sigma$.
Thanks for sharing, but I don't really see that as solving the problem. As I understand it, he is still classifying two to n distinct classes over the original region of interest (by sampling over smaller subsets and inferring discrimination boundaries), but never actually identifying then discarding or recoding previously classified low probability regions.