tttjjj/elt_bench huggingface dataset:
@jonsca, thank you for all the edits! They are a vast improvement on what I originally wrote.
ok, thanks a lot for your feedback. I'll try adding some dropped variables again. I don't want to change the link for now as i would be venturing in water far above my waist. 

i must say i am surprised that the model has no problem at all dealing with the ceiling performance (for example, a repeated measure ANOVA on the number of "correct" responses completely failed on this data, showing absolutely all effects and interaction highly significant with ridiculous p values)
As you stated yourself, variables are not really independent and some of them "shadow" others. Sex itself is a strong predictor, but in presence of more fine-grained Title* variables it loses some of its importance. However, many "weaker" variables may give better results than few "stronger". That's why I proposed to evaluate models first. In general, if variables are highly correlated, coefficients may not really reflect importance of features.
A *definition* is usually a good place to begin. What definition(s) do you know for expectation, variance, and SD?
Do you mean you have *paired* data $(x_i,y_i)$ and you would like to determine whether to use a paired test or unpaired test to compare $\\overline{x}$ to $\\overline{y}$? If you have no such pairing, how do you propose to make any sense of a covariance in the first place? My concern is that if you base a preliminary decision to use a paired test or not by inspecting these data, then you will change the size and power of the test you do choose (in a complicated way), thereby invalidating any p-value it produces.
This sort of question is probably more suited for [Stack overflow](http://stackoverflow.com), as there is no statistics involved. I answered two of them because the answers were short, but please send more of these type of questions over there. Better yet: read the introduction to R, which holds the answer to at least your previous question...
Pressed by @Wesley, I deleted my quick thoughts about autocorrelation functions and periodogram (may be he indeed is a frequency domain analysis guru, but I personally don't think Bartlett, while working with autocorrelations in time domain), but still think that my second suggestion about `bootspecdens` may be helpful.
Your idea is ok but i need to compare various algorithm which needs feature matrix.
You have illustrated one of the problems with histograms. But why use a histogram at all?  What are you trying to do?
Have you looked at the literature? Andrew Beyer, James Quinn, Steven Davidowitz and others have written technical books on handicapping. Beyer is one of the main originators of speed figures. 
Just "position at finish" is going to be very inadequate as a measure of form. There's distance, class, pace, track condition and a bunch of other factors to consider
Look at it this way: how could you formulate the relationship in the other direction?  Do you have in mind, say, a mathematical formula that will give you a player's name when you plug in their home run rate? And if you somehow succeeded in doing that, what would it tell you (that just looking up the data in a table doesn't)?
Think of this: suppose the covariate were practically *equal* to the interaction. Then, upon introducing the covariate, there would be no need for the interaction at all.
The answer varies according to your reason for considering the transformation: whether it is for exploratory or confirmatory purposes, whether it is a dependent or independent variable in a regression, and so on.  Could you perhaps share some of that relevant information with us so we can give you appropriate, focused answers?
Why not? It might provide valid information, but plots with individual points or pairs are not really popular in medical research. That's why parallel plot (which provide more information) are so unpopular. Medical researchers want averages and "normal ranges" (whatever that is).
Hi Mathlete, this appears to be standard bookwork and looks like it's probably related to some course. Would you mind adding the `self-study` tag please? \\[You might like to read through the [faq](http://stats.stackexchange.com/faq#questions) item relating to `homework` (a synonym for `self-study`) and the [self study tag wiki](http://stats.stackexchange.com/tags/self-study/info) for a description of how those questions are dealt with. In particular, there are some expectations of both askers and answerers that differ slightly from the other kinds of questions.\\]
Very useful @Dimitriy V. Msterov, thank you.  I will read up more on finite mixture models.  Also found this which seems to be an alternative approach to this kind of problem:http://cran.r-project.org/web/packages/poisson.glm.mix/poisson.glm.mix.pdf
You seem to be telling us contradictory things. Do you have *data* (a finite set of ordered pairs of numbers) or do you have a joint PDF (which is a mathematical function)?  What do you mean by the "type" of a random variable?
The interpretation of an OLS coefficient is a slope. See Kutner, Nachtsheim, Neter, Li "Applied Linear Statistical Models".
If you check [here](http://en.wikipedia.org/wiki/Autoregressive_model#Yule-Walker_equations) under "Examples for some Low-order AR(p) processes", then for an AR(1) it's the case that $\\rho_1 = \\phi_1$.  This is when the series given at that first place converges, which is when $|\\phi_1|<1$ (the part about stationarity under 'Definition' at the top is needed; when that's not the case, it's really an ARI model)
Ari, what do you mean by "remove the time-series variance from the panel data"? You can certainly *analyze* the data as though they were cross sectional. And you can do so in ways that treats within-panel variance as a nuisance parameter (e.g. using a random intercept mixed effects model), but you cannot make the data not have been panel data in the first place, with the potential issues that may arise around, for example non-stationary/integrated processes in your data.
You just have to read the first page. So theta (or k in anscombe) is the shape parameter of the negbin distribution and it manages, if the distribution is closer to gamma (k -> 0) or poisson (k -> infinity). But what does it mean to the fit? How can I interpret theta for example for the cars estimation?
+1 for this analysis and checking the interpretation of extreme value distributions.
There's even an R package for that book: [ElemStatLearn](http://cran.r-project.org/web/packages/ElemStatLearn/index.html) :-)
Any particular reason you're not using the standard Pearson or Deviance residuals?
Good luck!  It's a stimulating problem, and it would be interesting to know how it works out.
@Cardinal The trick is to probe further concerning the actual application, for as you know, in many cases inverting the matrix is unnecessary (and time-consuming and error-prone).
You might want to change $B-A$ to $A-B$ in the first line of your last paragraph.
Good question.  This is related to why Douglas Bates doesn't include p-values in `lmer` output. See his reasoning [here](https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html).
Interesting approach. I'll try give it a crack shortly.
But should the last line be $\\text{Cov}[\\epsilon_i, \\epsilon_i] = \\sigma^2$ by gauss markov conditions?
By definition, if you assume [additive white Gaussian noise](http://en.wikipedia.org/wiki/Additive_white_Gaussian_noise), $e_0$, $e_1$, and $e_2$ are independent and generally will not be equal. That is what the "white" in the "white noise" means, uncorrelated (independent for Gaussian noise) and mean zero.  You could choose to have the same noise for each speaker if that makes sense for your application (I wouldn't think it does though), but that would not be white noise.
You described a scale that is a weighted average where all of the weights are $1/3$ (not a weighted product). In that case, yes, 0-4 and 1-5 are effectively the same. The weighted average only has to be $\\leq$ to highest level observed if the sum of the weights is $\\leq 1$. In your first example in the question, the sum of the weights is 1, but in the second example, the sum of the weights is $4/3$.
Well, this thread is subjective, so who knows...  Now seriously - FDR stands for false discovery rate (wikipedia it)
How does this question arise?
Note that even though it is rather common to do thinning to reduce the autocorrelation, it is more often better not to thin. If the chain is long enough, you get a good approximation of the posterior even with autocorrelation (and usually better than from the thinned chain without autocorrelation). See for example here http://doingbayesiandataanalysis.blogspot.de/2011/11/thinning-to-reduce-autocorrelation.html. However, thinning can make sense if you want to make further computional intensive calculation on your MCMC sample or if you run into memory samples.
What about if you're getting the posterior predictive over multiple periods? I have been using $\\theta^{\\left(i\\right)}$ for each $x_{n+j}$, but I can see why it might make sense to redraw a new theta.
Could you get to the same result by first clustering with k-means, then constructing your 'groups' by taking M points from each cluster?
Sir, can an explanation be given in detail specifically with respect to the shape and scale parameters?
That is definitely harder. I missed the end of the post. But I think I should leave my answer up as it does answer part of the question
Maybe this one helps? http://biol09.biol.umontreal.ca/borcardd/partialr2.pdf
I edited my question a bit, thank you very much for your answers.I don't know exactly what intervals I am supposed to compute, i was told just 'intervals' so far, so.. i assumed it with be confidence intervals. Still, the quantiles range (I took the interval between 75% and 50% quantile ) seem to make a lot of sense with the data. I think the request has to do more with describing the data that I have , rather than making inferences about the whole population, so I will stick with this for now. Thanks.
Are your 8 covariates perhaps a set of mutually exclusive dummy variables?
It's not just that its inefficient; its estimated standard errors are biased, so any inference based on those will be wrong.
The reasons why (b) is plausible should also indicate why the other answers are not good. So what are your reasons for favoring (b)?
Right, but there are applications of statistical methodologies in every discipline.  Why does biostatistics exist (in the US, at least) as a semi-distinct discipline?
It is problematic to (mis)interpret a p-value as a probability in this sense. You seem to be applying a Bayesian-like approach but without the usual care needed to consider and defend a prior distribution.  What justification can you provide to support your recommendations?
1. $\\theta\\in [0,2]$ still does not work in $\\sqrt{1-\\theta}$!
Does "millions of those" refer to the number of observations, or the number of predictors? How many of each do you have? What is the purpose of the regression, do you want to predict? How did OLS yield bad & unstable results?
@Henrik @Brandon There were already some discussions, headed under the [scales](http://stats.stackexchange.com/questions/tagged/scales) tag, about the nature and the way to treat Likert scale/item.
...where practice is large $n$.
I was wondering why you say you have data for minimum & maximum & maximum values; then later that you have information on only expected minimum & maximum. Which is it - observed or expected?
If you want to randomly generate a covariance matrix you can also randomly generate an upper triangular matrix (the cholesky factor of a random covariance) and calculate $LL'$.
(+1) I would suggest the same myself.
Thanks for this! Could you elaborate a little more about the relationship between the effectiveness of MCMC and the correlation between the variables? I had always thought that MCMC would provide an advantage over exisiting numerical integration techniques when the variables are correlated - though I cannot provide a good reason for why it would.
+1 for Graphviz which is often missed and really useful
@Colin Have you tried any clustering analysis by now? (k-means, hierarchical clustering, partitioning around medoids...)
Ah, that does make some more sense, thanks.
@whuber: Thanks. My understanding about that paragraph is limited. My question is rather "why" than "how". My confusion comes from that one is for survival function, and the other is for hazard function.
@user4229 I'll second @whuber's remark and think you'll find useful clues about MT issues with correlation measures, if not an exact answer to most of your questions here. Now, if you're really interested in discussing FDR vs. single-step methods (such as the Bonferroni method), FP vs. FN issues in semi-exploratory studies, or multiple testing when dealing with correlated predictors, then I would suggest you rewrite your question in that particular direction, making this question a less obvious duplicate of some of the earlier threads.
Don't systematically search for relationships without verifying them on independent data. You risk "data dredging". At 10^2 pairs of variables, and just 100 observations, you are bound to find some "correlations" just by chance.
I have been using tutorteddy for long for my statistics problems.
@whuber Yes, if my days were 96 hours long I would be right on it :) If you can read italian the article on the WW test is a bit more detailed. Before assessing the power, defining good alternative models is not so easy... the worst case is that of most modern RNG, fully deterministic periodic sequences for which these tests will have no power at all.
Thanks, that gives some food for thought. Do you have any good links describing transfer function models? My current idea is to put together a state-space model (West&Harrison- or Durbin&Koopman-style), plug in a diffuse prior, no state variance (except on the constant and trend), and turn the crank, but that method tends to depend a lot on the exact variances chosen.
What will be linear in the parameters *depends on the model*. It's perfectly possible for that linearity to be on the original scale or the square root scale or some other scale. Even the - useful/important - 'maps to the real line' property isn't unique to the log function. The reason the log link is 'natural' is because of the way it simplifies the GLM by having a sufficient statistic of $X'y$.
No, sorry! I just meant that it is a random sample of length $p$.
This looks like a standard textbook problem. Please review the [tag wiki info](http://stats.stackexchange.com/tags/self-study/info) of the `self-study` tag and add it if at all applicable.
I can't answer your question, but [this comment](http://ba.stat.cmu.edu/journal/2006/vol01/issue03/gelman.pdf) by Gelman has a nice discussion of priors for variance parameters in the hierarchical setup you have here. In particular it seems like you could put a non-informative prior $p(\\mu, \\log \\sigma^2) \\propto 1$.
@Tal I've +1 this response because I'm pretty sure there's a way to achieve what you want although I didn't take time to investigate package options in depth (I used it once to display several models as you said).
As I understand his question, he might want to run PCA on these new variables instead of the "divided" attributes. To cite the question: "any way to handle them together, or group them for PCA"
(+1) Censoring is the key idea; time is just a particular (and the most common) application.  For instance, [Dennis Helsel](http://www.practicalstats.com/nada/) has applied survival analysis to chemical measurements having detection limits (left censoring).  There's no time involved, but the model and the techniques are still useful.
I was speaking in general terms, not specifically for the above graph. Each chart type has its merits. A certain chart type is not appropriate in *all* situations.
@TrynnaDoStat ordering is critical in correlation between ordered variables.
Can you give a number of how many effects you are visualizing? IMO using forest plots with small multiples is the easiest way to pack alot of information into a small space. It takes some arbitrary decisions though how best to arrange the panels and effects within the panels to provide the clearest picture.
Well that's interesting to see CI in the graphing dialogue -- can SPSS report  numerical values for these CIs for proportions in a frequency table? (Excluding the possibility of using any add-in packages for reasons of convenience/practicality for students.) I keep telling students that SPSS doesn't calculate confidence intervals for proportions and would be slightly happier to be proven wrong.
+1, great overview.
(cont...) In a somewhat similar manner, when dealing with vector autoregressions, one tends to interpret the impulse response functions (IRFs) rather than the estimated coefficients; the coefficients are often too difficult to interpret, but sense can usually be made of the IRFs. Out of curiosity, have you found many papers in which the author(s) dedicated much attention to interpreting the coefficients in an ARIMA model?
I have taken few courses in statistics, I've studied advanced numerical analysis using SAS. It is okay if its complex, could you point me to some resources? and How I should approach such problems? Any help is much appreciated.
@charles, I can use anova to see if the means are same or different, but I really like something more complicated. I was curious to see what other approaches I can use for this analysis?
The wikipedia article you linked has the definition both for one- ($n$) and two- ($m, n$) sample cases. The one-sample case applies to paired differences. That is, the answer for your case (median of pairwise averages) is already discussed [in the article](http://en.wikipedia.org/wiki/Hodges%E2%80%93Lehmann_estimator#Definition) you link to.
Honestly, I just kept graphing values in R until it looked right.
I'm pleased the model is working. Best of luck with your research.
I would ask this as a separate question to draw more attention, I think!
It's natural to go right into conducting a test, but consider this.  Such tests compare results to what would likely occur if chance alone were at work and if there were no difference in the population between the breakdowns for young and elderly customers.  But did chance produce this sample, or is it a convenience sample?  If the latter, the findings from the Chi-square test will not apply.  In other words, there will be no statement one can make about statistical significance for these data.
Still asking for a Matlab solution? In this case, I let you add this tag. And feel free to add or delete the edits I made if they don't exactly reflect your design.
Reader @xan makes some *very* interesting comments after the (beautiful) answer by [Accidental Statistician](http://stats.stackexchange.com/a/81634/919) in this thread. Those comments point to a logical gap in my argument: it is necessary also to show that the two kings can (in some intuitive sense) truly move independently of each other. Xan's example concerns kings that cannot make diagonal moves: if one king starts in state $E$ and the other in $C$ or $M$, then they *never* can occupy the same square! This gap can be fixed by considering a transition matrix for ordered pairs of states.
@JCL Renders fine for me in Chrome (v 5.0.375.125) on Ubuntu 10.04 running from a USB drive on my laptop.
Is it _absolutely necessary_ to use $m$ (in a subscript) to denote the maximum and also to use $m$ as the parameter of the Poisson distribution? I mean there are more than $40$ unused upper-case and lower-case letters breathlessly awaiting a call from you to serve as symbols.
@IhaveCandy Please see my comment above, Michael Mayer is making the same point.
(2) Although it's possible that my results are due to a bug, I think it's more likely that they're due to some sort of error on my end (and if this does seem to be the case, please tell me!).  I'm using the `topicmodels` package in R, which is essentially an R interface to the original algorithm implemented by Blei and colleagues.
But, again, you can use very similar arguments to not reject null. There is nothing special about 0.05, if you had chosen 0.06 as your limit you would probably not be asking the question, but the situation would not be that much different... Rather in these situations I would ask: "what is the real-life meaning of this result?". For instance if this was a biological experiment I would look for the biological significance of the specific result, report the p-value as it is and rather comment on the biology.
+1 For noticing the strange treatment of critical values in the OP's code!
I take it that you looked up the book?
A dictionary definition of 'subjective' (straight from Google) says: *based on or influenced by personal feelings, tastes, or opinions. e.g. "his views are highly subjective" synonyms: personal, personalized, individual, internal, emotional, instinctive, intuitive, impressionistic*.  Notice that this reflects a folk theory that a 'view' (ie a belief) is subjective (sense 1: impressionistic, intuitive, weird etc.) *because* it concerns the *internal* state of a particular subject (sense 2: personalized, individual, etc.) rather than being public ie *impersonal*.
The Federalist paper is very interested and describes similar to above answer.  Thanks.
Thanks for the comment.  Indeed, y = a+b with no error (or more precisely, the error is ignorable and ignored).  Even more precisely, y, a, and b are all undefined below zero.  So my regression where I model a cannot give me $\\hat a$ that has elements less than zero.  I've been getting around this by bottom-coding the fitted values (and top-coding so that they don't drive b--> <0), but this is a hack and there might be more elegant solutions.
"* hence the dynamics of each process should dictate the dynamics of the ratio of the processes...right?*" -- not just the two marginal dynamics, but also their dependence.
I have noticed that you have not accepted any answers to your previous questions. Consider accepting some of them if you find them helpful. This will motivate responders :).
Thank you, What I meant about the mean is just that in market studies class (and from several other books) we are instructed that for qualitative variables (nominal and ordinal ones-even binary in my case) mean cannot be calculated- only mode and median are appropriate descriptives.  But I will check out the Fischer's tests.
It may indeed be clearer, but it still doesn't specify well-defined questions.  I assume the issue is that somewhere in either step 1 (estimating the CDF's) or step 2 (calculating race model bounds).  With that said, looking at step 2, I don't see how that would possibly be a race model bound.  It implies that the bound is as high as BOTH reaction times combined?  I would think that the bound would be similar to min(X,Y) not X+Y.
I'm a coder, primarily, so my chief concern is how to do the statistical analysis. I'm sure I can figure out how to make the database connection.
I know you can set these parameters in the libsvm library. I'm not sure about the one you're using...
Hi, thx for the links. Respective to the eigen value decomposition, this decomposition does not help, because from there you get complex eigenvalues for square root matrix, but I need reell valued matrix.
Not an answer to your question, but my currently favorite answer to the underlying problem: When one has a continuous predictor variable, traditionally we are taught to either assume its effect as linear (losing power to detect non-linear effects) or quantize it (losing power to detect linear effects). An elegant alternative is generalized additive modelling (gam), which permits the data itself to determine the degree of non-linearity that the model attempts to fit. The `mgcv` package in R implements gam with cross-validation to avoid over-fitting, and has revolutionized my work.
@whuber - could you please remove the duplicated tag?
This question is based on an incorrect assertion.  When $\\lambda=0$, $P(0) = \\exp(0)(0)^0/0! = 1$, not $0$. The problem is the use of a *log link,* because $\\log(0)$ is undefined.