tttjjj/elt_bench huggingface dataset:
Based on the results above, the third model seems by far the best.  Also based on my domain knowledge, it seems reasonable.  But when I test it using real betting lines it gets destroyed.  The simple model two (also reasonable based on my domain knowledge) tests the best by far when using real betting lines.
+1 Apparently, it's "lapply(xcomb..."
Scott, I think @mbq is referring to the fact that $R^2 = {\\rm cor}({\\rm predicted \\ values}, {\\rm actual \\ values})^2$
"Data-generating process", not data gathering process.  Thats the term I learned anyway for whatever phenomenon is responsible for creating the data that you in have.  A rather implicitly frequentist conception, now that I think about it.
I really can't follow this. Can you state what you are trying to do more plainly & w/o the jargon that you might expect statisticians want to hear? What are the states & traits here? If you had these scores & metrics, what would you want to do with them? What are your actual data & what is your end goal?
Welcome to the site, @MarkA.Greenbaum. There's no need to sign your posts, in fact we'd rather you don't, your avatar & a link to your user profile page are added automatically. Since you're new here, you may want to read our [FAQ](http://stats.stackexchange.com/faq), which covers site policies like this.
@nameless . That's what I was thinking. However, I too am not well versed in this sort of regression and am unsure if it is a methodologically sound procedure.
+1. How do you impute missing data on predictors in JAGS *while* accounting for the uncertainty in the imputed values? (i.e. much like multiple imputation does vs. single imputation)
This is a fairly common problem but even a tiny bit of regularization can fix it.  Standardized coefficients in the +/- 5 range will shift probability from .01 to .99.  For sensible priors, see the bayesglm function in the ARM package and Gelman's paper on the subject: http://www.stat.columbia.edu/~gelman/research/published/priors11.pdf
is there a problem with [ESL2](http://statweb.stanford.edu/~tibs/ElemStatLearn/)?
Do you have any suggestions how I might be able to test to see if the trends (or lack thereof) for individuals are different between groups (=Region)?
+1 for the sensible discussion; a few points though.  Inferential machinery is unavailable for population analysis, but in many modeling cases, I'd question whether one ever has *the* population data to begin with -- often, it's not very hard to poke holes.  So it's not *always* an appeal to a super population as the means to deploy inference.  Rather than "super population", the better way is to assume a data generating process yielding, for example, the year to year test taking cohorts in question.  That's where the stochastic component arises.
@MaximKhesin Logistic regression can be trained with many methods, some of which involve matrix inversion, e.g. [IRLS](http://www.cs.cmu.edu/~ggordon/IRLS-example/). (But that's kind of irrelevant to the problem here.) Anyway, it *seems* like logistic regression is deciding that "this sux", when it's not part of "this sux s*t" or similar, is actually a negative indicator. Are you sure that's not the case in your problem? You might also try running e.g. L2-regularized logistic regression, if you're not currently regularizing.
I am no expert on the matter but the basic idea of having a dendrogram is that it induces a non-arbitrary covariance function between your points (eg. OU - the "Hansen" model). A very "rough" way to check if there is "phylogenetic signal" would be to use a second, star-phylogeny, that assumes no complex phylogenetic relations between your tips and compare your phylogenetic means. Additionally you could keep the same dentrogram and mixing the tips at the reading and then estimate their likelihood under your model of evolution. Simplify the terminology you use to get better answers.
cross-posted to [r-sig-mixed-models@r-project.org](https://mailman.stat.ethz.ch/pipermail/r-sig-mixed-models/2014q1/021527.html) ; some of us prefer it if you pick one venue or the other, rather than cross-posting ...
+1. An alternative understanding of the problem supposes the original $N$ cells are still in the population, requiring $N+(N-1)x+1$ cells to be chosen to assure at least one of each of the progeny is included. An answer that satisfies both readings is that you need to take enough cells so that *no more than $x-1$ remain* (for otherwise--clearly--the remaining cells could include all $x$ progeny of a single cell).
@O_Devinyk Thanks that was very helpful.
Ah, sorry, I left its definition out. It's just a sequence of x-values at which to compute fitted values so we can do smooth looking plots. So something like `xx <- seq(0,2.6,0.05)` should work okay. Make the last number smaller (like 0.02) if it looks a bit angular (piecewise linear). You just need to define it before the first use of `xx`. I will edit.
This question is not appropriate for this site. It is not specifically about programming. If you need general statistical advice, try [stats.se] instead. Be prepared to define what exactly you mean by "good fit". That's a very subjective term.
well RBugs will interface with Win/Openbugs, which you can basically write any model you want. Jags uses the bugs language, with some nice additions, but also lacks the ability to us autoregressive priors (if you care about that), and mcmcpack is just a general suite that uses native rcode to estimate bayesian models, if you want pure generality, and don't mind bugs, then I would suggest RBugs, my 2 cents
Looking for complementary alternatives.
@guest43434 Yes, that's exactly what it means.
The wisdom or lack thereof depends on the range of the dependent variable of interest. Consider car braking data where you have speeds and stopping distances. You could fit a quadratic model with or without intercept. Speeds of interest usually start around 50 km/hr and go up to, say, 130 km/hr. Fitting a quadratic *with* intercept in this case makes more sense, I think, as forcing the intercept to zero can incur (practically) significant lack-of-fit problems. The fact that the "braking distance" of a stopped car is zero is not particularly relevant for the modeling problem at hand.
It sounds like you have it pretty much in hand. I made a few additional comments but nothing that adds anything of much value to what you seem to be doing already.
The dataset $(2.18501, 3.86374, 7, 9.29171, 18.0595)$ has *exactly* these statistics but has a sample size of five. A dataset with two copies each of $(1.76835, 3.67465, 7, 9.28261, 18.6744)$ also has these statistics but has ten values. We can easily create similar examples with $20$ or $25$ values.  I suspect you have forgotten to quote at least one relevant statistic, most likely a standard error or confidence interval.
As shown by the numerous answers (none being accepted by Scott Weinstein btw), the question of which diversity index is optimal seems to be trickier than it appears at first sight. In this context, I thought that a link towards that paper would be of some interest, as it aims at clarifying the topic in an axiomatic way. But you are right, it is maybe far too theoretical compare with what Scott was looking for. (And sorry for replying so late!)
@Brandon The 1st axis is an axis of "dominance" (light -> dark) for both modalities, but we can also see that the 1st axis opposes blue and green eyes to brown and hazel eyes (their coordinates are of opposite signs), and red hair/green eye combination--which is quite uncommon--contribute mostly to the 2nd factor axis. As this axis only explains 9.5% of the total inertia, it is rather difficult to draw firm conclusions (esp. wrt. genetic hypotheses).
Internal evaluation is not the best "unsupervised" approach. Essentially, you are making it a "self-supervised" approach, supervised by the criterion. k-means is a good example of this. It does a reasonably good job at optimizing a particular criterion. But it's also very easy to construct a data set, where the "optimal" result is completely useless. You *do* want to a large extend to have the evaluation driven by "usefulness", which unfortunately needs some kind of external evaluation. Yet, with labels, we could as well do supervised learning. So we need yet something else.
Can you show us the original time series?  It looks like there might be an increase in variability and perhaps some other nonstationary effect. But it could be some short term blip due to an intervention also.  It would help to put this into context.  What do these series represent? Is there something that you would expect would happen?
How did you determine that "the value has a high probability between -1 and 1" in the first set? The sets are disjoint, and moreover have more numbers outside of the [-1,1] range than inside of the range, so I don't understand what you mean.
See [here](http://stats.stackexchange.com/questions/43159/how-to-calculate-pooled-variance-of-two-groups-given-known-group-variances-mean) and [here](http://stats.stackexchange.com/questions/19350/how-to-calculate-overall-standard-deviation-from-standard-deviation-of-sub-perio)
It seems that you have graph values, probably from a social network. I would add some edges or vertex feature like centrality, betweeness, etc...
Ben: regarding overdispersion in binary data, you might want to respond to this question: http://stats.stackexchange.com/questions/91597/overdispersion-in-logistic-regression
I suppose it depends on what you mean by *to sample*.
@Richard: My problem is to solve speaker recognition. Yes , I am constrained to solve as per the above decided method. Even then if you have any other good means of other solutions at hand I ll surely go through to attain beneficiary. My primary motive is to combine MFCC technique and LPC for improved recognition.
This seems a little vague to me. Can you list more of your current phrasing &/or the criticism? What you have seems fine to me as far as I can tell.
I agree - 'define' there probably means 'write down the model in symbolic terms, for this specific situation'. Another way to find the assumptions is to use the search box at the top of the page.
Please tell if you oppose adding the picture
This was a great explanation. Another example of your "tangible" data would be A survey with many questions and you want to know which questions on the survey are most important and perhaps some combination of them is really asking about one topic.
At best, this is a caricature of the differences in the two approaches.
+1 I found plenty of references (in support of this answer) by Googling [rifle "group size" statistic](https://www.google.com/search?q=rifle+"group+size"+statistic).
I appreciate the intro to analyzing split-splot designs with mixed-effects models and further background info! It certainly is the preferred way to carry out the analysis. I've updated my question to emphasize that I'd still like to know how to do this "the old way".
@SeanEaster I mean "latent Dirichlet allocation"
(1) Are you applying the JB test to (a) an dependent variable, (b) an independent variable, or (c) residuals in a regression?  It probably is most useful in case (c); its interpretation and your response could be quite different in cases (a) or (b).  Could you clarify?  (2) "Market Value Added" sounds like a difference of two variables.  If you do elect to re-express its values, you are best off re-expressing the original two variables and then recomputing MVA.  For example, you might compute the log of the ratio of the original two variables.  Do you have these variables?
That is how I view it.
The "Gaussian" family with a linear link is just ordinary least squares (OLS) regression; methods to check such fits are probably discussed in a thousand questions on this site (I do not exaggerate).
Can you describe your data, both the concepts behind what is being measured, and also the kinds of variables?
I'm just using it for simple bivariate correlations. I'm not sure why it is claimed that normality is required. The stats texts I've read always list normality as an assumption of the Pearson's correlation and advise to use Spearman's for conditions in which non-normality holds.
@Cagdas - I dunno if I'd call it a gimmick. It's just measuring a different thing. And as cardinal points out, it has some uses. As for whether it'll break when applied to the binominal distribution, well, depends how you're going to apply it :). Probably worth starting a new topic if you're not sure.
The _constraint_ is just the test. Stata complains when there are redundant or inconsistent test specifications. Read **[R]test**, which has a few examples on this.
Thanks for your help, I've penned a quick answer but if you want to put your comments in as an answer then will select as accepted answer.
This answer is great. But I think the given tables are wrong, I will try to explain it as a new answer. Would you like to check it and add something about the covariance or should I just select your answer as the accepted?
Are there any tricks you can share for modelling this type of response variable with multinomial logistic? Row deletion? Asymmetric loss functions? Guessing you'll recommend to just leave as-is assuming no violations of assumptions.
Clustering is probably less efficient and less effective than the actual segmentation or classification. Because in this setting, you do have training data; and then supervised methods always outperform unsupervised methods. And most likely, the key idea is to use OCR to turn it into text and work with that. Because the majority of documents will be printed, and then OCR works quite good, at which point you are in text and no longer in images.
it depends on what structure you are willing to assume for $V$
+1 for adding context and the interesting points you raised. Is this a validated scale? If so, I assume a cutoff has already been established because it should not be sample-dependent, for interpretation purpose. Good point about the PPV and prevalence.
I don't think so. Just write out the definition of the binomial coefficient, and you'll see that the equality works. Note that ${n\\choose k} = {n\\choose {n-k}}$
If you introduce a random effect I would think it would be a variable that you would automatically be include or else why make it random?  I also don't see why you would not have any variables that you are sure you need to include.
@Michael , I think this paper might be relevant  http://www.ornl.gov/~webworks/cpr/rpt/82307.pdf  . Makridakis in http://www.insead.edu/facultyresearch/research/doc.cfm?did=46900 critically reflects on differencing while suggesting what I believe is an inferior solution. Another reference is Nathan Balke Balke,N, Detecting level shifts in time series, JBES,1993).
While its a brilliant software, it cant be used for custom datasets. But a good find for later use perhaps.
In case anyone wants to read more, there's an active literature about non-convex penalty functions that are alternatives to the L1 norm (e.g., recently, http://papers.nips.cc/paper/4904-regularized-m-estimators-with-nonconvexity-statistical-and-algorithmic-theory-for-local-optima).
There are so many possibilities...could you be more specific as to what you are trying to achieve?
I was able to create the seven models you described. However, in this dataset, the `Vision*Sound*Touch` interaction seemed to dominate the M4, M5 and M6 models. That is, the negative impact on the odds of success observed in that condition results in these pair models showing a negative effect when the raw counts are higher than the individual conditions. Thank you for suggesting this analysis, I did learn something from trying the series of models.
I've reformatted your question. Is the table now correct?
After the question was modified my answer above does not seem to apply.
is there a way to connect these CI ? (with straight lines ? 3 straight lines)
I hope and trust that Greene is more nuanced: small changes in data *can* lead to large changes in estimates--but they do not necessarily do so. Coefficients can have implausible signs or magnitudes--but they do not necessarily do so.  He's (apparently) just trying to list some of the possible consequences of high standard errors: multicollinearity *tends* to produce high SEs (but does not necessarily do so!) When multicollinearity is viewed as an issue concerning *how the variables are encoded,* rather than about the model, the answers to the first two questions are clear.
Your last line is correct: computing a pseudo-$R^2$ is contrary to the spirit of Loess, which is to *explore, identify patterns, and smooth* data.  Computing a measure like this misses the point and, IMHO, is an abuse of the tool.  Instead, if you want to assess the fit, continue in the spirit of EDA and analyze the residuals (the "rough" in Tukey's language).  Although you might wind up looking at m-letter statistics, IQRs, etc., which could be construed as serving in a role a bit like $R^2$, the analysis proceeds in an entirely different spirit.
Hi Greg! This is a very interesting idea. I would like to try it out. Are you aware of any examples in published articles that make use of this way of power analysis?
sir thank you so much for your valuable response in my question. i was really enlightened with what you have explained. thank you.:)
Related to your third bullet point: http://stackoverflow.com/a/15984310/684253
OK thanks I'll update the question
How familiar are you with Gibbs sampling in general?
@COOLSerdash You can always calculate or simulate to obtain p-values based on the pattern of ties; essentially a permutation test (or randomization test).
+1 Thanks. I finally got the idea. Sorry about the notation issues... Deleting comments...
[This](http://acl.ldc.upenn.edu/C/C96/C96-2141.pdf) paper just calls it 'word alignment' (see fig 1 & 2), though it adds lines joining the pairs, while [this](http://www.statmt.org/wmt08/pdf/WMT03.pdf) calls it an 'alignment matrix' (see the discussion under Figure 1), which makes sense.
Thanks for the quick response. Right now I'm calling LIBSVM via the command line for training but eventually I'll be using the C# lib in my trading app. For now I just need to figure out the data format. Now, you said that I need to produce a long single column. How does LIBSVM know how to separate them all? Or is it agnostic to that? If I produce a single row of 5x5 params that'll get me to 25 then - does that include all date columns/params? And I assume I need the ideal at the beginning of the row, correct?
The only problem is, I have to be able to reproduce this with SPSS output. :( I'm still going to try it though!
@user643722 I am in chat "ten fold" now. Please join if you want to continue the discussion.
Perhaps you can expand a little bit and make this answer more self-contained.
+1 Interesting points, thanks.
To apply Rao Blackwell, I need an unbiased estimator (for example, $X_1 -1$) and I need to find the distribution of the sufficient statistics given this estimator But it not seems so easy to find the distribution of $X_{(1)}|X_1-1$
can you please be more specific about what you are doing exactly? I think you are missing something here as @dsimcha mentions.
You may have a look at this book for a description of kNN http://statweb.stanford.edu/~tibs/ElemStatLearn/
why exactly are you testing for non-linearity? And what are you trying to model? What is IV?
If you take the weighted mean of a rating of 5 with a weight of 0.05 with a rating of 1 and a weight of 0.25, you get (5*0.05 + 1* 0.25)/(0.05+0.25) = 1.67. That's a lot nearer 1 than 5, so the low-score high-value rating hasn't overwhelmed the high-score low-value rating at all.
Thank you for sharing that, Hamed.  Welcome to our site!
We are entering an era of a revival of experimental research in behavioral sciences. In the 1950s, almost all of statitsics was experimental research, with applications in agriculture. But around 1980s, people have recognized these techniques are not of much help with observational data, which is all you can do in most social sciences. Now, at least in the niches of online marketing research, if you are Amazon or Google or Bing, you can run experiments and get the cleanest form of causal inference possible.
This answer is wrong.  Unfortunately @steadyfish has asked for "any distribution".  Let $X$ be the result of rolling a die.  Then append `1.00` to the front of $X$, so that the sample space is 1.001, 1.002, 1.003, and so on.  Truncating to two digits would leave you with a degenerate random variable.
For prediction it doesn't matter if you use colinear predictors - colinearity is a problem for $\\beta $ but not for $ x_i^T\\beta $
Class is the categories you are trying to distinguish between; i.e. normal and abnormal. Cases is the number of subjects or measurements. To see which features are stable between trials try intraclass correlation coefficients
You can calculate it from the individual sample information.
Thanks for your answer too. It is an interesting way to see the question. I had never think about normality in such a way (i.e., "that having normality over [the] entire raw y usually means [we] have no effect").
The answer appears in [your previous thread](http://stats.stackexchange.com/questions/54896/regression-on-means-of-log-transformed-variables): the methods are not equivalent. Consider, too, that a few thousand observations is by no means "large." But your final question looks worthy of consideration: what should be done when you have so many observations that they are more than you need and more than your computer can cope with? The answers depend partly on the purpose of the regression: is it for exploration, testing hypotheses, prediction, or perhaps something else?
Since Spearman's rho is just a Pearson correlation type computation on the ranks of the data and ranking makes sense for ordinal data Joel is quite right. For continous data pearsons correlation estimate will be 1 or -1 if the data fall on a straight line.  For continuous, discrete or ordinal data Spearman's rho will be 1 or -1 if the data monotonically increase or decrease respectively.  So you can think of Pearson as measuring the degree of linearity and Spearman as measuring tendency fro both variable is to increase together or one tends to decrease as the other increases.
In the tables SPSS outputs for `T-TEST` the sample means are right there to tell you which group is higher or lower.
If there is a dependence structure in time, it is usually a good idea to think about modelling it explicitly. Time series people are defined by their thinking there is no other way to do it.
I think he means there's a monotonicity structure to the question that should be reflected in a model of the answers. The thought is that, despite the unfortunate wording, the question is trying to discretize a daily rate of how often subjects eat at McDs by asking about thresholds.  So if you eat at least 3 times a week then you also eat at least once a week, and at least once a month, etc. which seems reasonable - it's just an ordinal response. But he does, as I tried to suggest, have to treat it like a response, not a independent variable.
What is the "filtering" supposed to do? How would you interpret its results?
Thank you so much for explaining it to me in detail, you have cleared up a lot of the confusion. Much appreciated! :)
**For future reference:** this answer generalizes to the case where the $X_i$ may have entirely different distributions altogether, provided they are continuous.  (Care is needed in dealing with ties for distributions that are not continuous.) The independence assumption is still essential.  (Without this assumption, solutions can still be obtained with numerical integration or simulation, but both are more difficult to carry out and require the dependence to be explicitly represented.)
Your data are non-negative ratios, so take the logarithm and model on $\\mathbb{R}$. A non-parametric method will probably serve you well. You might need to [mix](http://en.wikipedia.org/wiki/Mixture_model) in a point mass at 0% (on the original scale, corresponding to $-\\infty$ on the log scale).
Apologies, I've edited it now.