tttjjj/elt_bench huggingface dataset:
Thank you. You're right -- after thinking about it for a while I realize my objective is not achievable. Treating the samples as entirely separate RVs dos not give us any population to argue about, and thus we need to assign posterior distributions, as you suggest. However, my application would probably do badly with posterior distributions. Instead, I opt to make some assumptions on the $\\mu_i$'s -- to assume they are independently chosen from a distribution. This is not perfect for my application, but it'll probably do. I'll opt to edit the original question. Thank you! for the answer!
I believe that the sums need to be normalized (e.g., use the mean score) for the distribution to tend to normality.
@caracal Yes, thanks for catching that error.  I have fixed it.  (The calculations in the example were nevertheless correct.)  Independence of A and B is asserted by the OP in the second sentence of the question.
SVD is used (amongst other uses) as a preprocessing step to reduce the amount of dimensions for your learning algorithm. This why you would introduce a choice of $p << M$, which basically allows you to learn in the reduced $p$-dimensional space. Here, $p$ is a design choice. If you are familiar with PCA (which I recommend you should be), this would be the equivalent of dropping the $M-k$ least important eigenvectors. Setting $p$ equal to the original dimensions asin your example, allows for a flawless reconstruction, but no dimensionality reduction.
Well really I mean that its shape can take more obviously different forms depending on that shape parameter than the log-normal, which is at least built from a self-similar location-scale family (the normal). Technically they both have two parameters so it's not that much more flexible, but in a narrow sense... There is also a distribution with an additional parameter, which I forget right now, that includes the gamma as a special case.
+1, however, I'm mixed about this sort of thing. Often they seem to exist so that someone can not know much about the analyses in question, but still scroll through the list, find a name that meets their conditions and then run through the procedures. In short, I fear they lead to 'cookbooking' data. In addition, I suspect they reinforce the idea that these are distinct tests w/o an underlying continuity, & that the test (p-value) is all that's important. IE, they help to solidify misconceptions & conceptual biases about statistics. Nonetheless, they do have some value...
What exactly do you want to compare?
Can you post a link to his notes?
Thanks very much, Arthur. Your explanation was exactly what I needed! I appreciate it.
Thanks for that clear answer!
The good news is that you have at least 24 friends, otherwise distinct subsets of them could not add up to 95% and 75%.
This is problematic and should be avoided.
@kjetilbhalvorsen The studies I've seen - including some simulation I've done myself (and I haven't looked at any for a good while; you may well have seen something I have not), the majority of the effect on power seemed to be mostly pushing the level up and down (which didn't affect the Wilcoxon). Given the generally good power properties of the Wilcoxon in these circumstances (particularly with heavy-tails), that's enough to have the Wilcoxon win on power - if you adjust the levels so they're similar, it surprised me how well the t- often did.
I believe now that I have to interpret it multiplicatively: exp(Intercept)*exp(height) would be the income with a 1 unit increase in height.
Thank you nonetheless! :)
In SPSS "inverse" variously means "reciprocal" (i.e., the transformation $x \\to 1/x$), of which there is only one (making it doubtful you would be asked for a "type" in this context), and "functional inverse" (i.e., the inverse of $f:x \\to y$ is the function $f^{-1}:y \\to x$), which is very general and conceivably could have many "types."  The latter shows up in the context of variable transformations and link functions for generalized linear models, among other places.  Could you tell us what SPSS procedure you are using?
OK, so we gradually learn about the data. Is the survey new or an existing one? What was it intended to do? Why are there 171 questions on the survey? How will factor analysis help shorten it? Factor analysis finds latent variables - but latent variables are *latent*.
Alright thanks for your thoughts! Yes, the author definitely assumed that the hypothesis was 'chosen at random' which is not correct.
I think the OP tried to calculate the $z$-score, but one needs to know how to utilize that $z$-score to obtain the desired probability...
Note that $y_i$ can always be rescaled to integer values when it is rational, eg measure pence/cents rather than pounds/dollars. Though you may want to round to the nearest pound/dollar anyway since the distribution of the pence/cents part of the price of goods will likely be very uneven (ie mostly 99).
I think the notation is innacurate, he uses scalars where vectors are needed to refer to the fact that between firms covariances are not zero so his notation implies $\\epsilon_{it} = [\\epsilon_{it}]_{i=1,...,N}$ is a N rows vector. Another interpretation is that he is refering to $ij$ element of $\\epsilon_{.t}^T\\epsilon_{.t}$. In both cases he means the same thing, but as it's not a quantitative journal ambiguities in math notation happen...
If you only want the visual reminder, you could also set ``to="none"``, but then use ``options(na.action="na.pass")`` before creating the forest plot. See [here](http://www.metafor-project.org/doku.php/tips:handling_missing_data) for further details.
i can try with Q-Q plots, but to compare the distributions i should know the theoretical one, isn't it? or i'm wrong?
@ryu576: Not if you can calculate SST before you do the regression (it is related to the variance of the dependent variable in the data).
@Nick Cox Thanks for informing me, I added the self-study tag.
Thankyou that is extremely helpful. I posted this quite a while ago and ended up using a negative binomial model, so it's interesting to hear another person who suggests the same thing. I agree with you that the probability of extraction is not a constant. This is something that I disagreed with my supervisor about however my modelling showed that the negative binomial model had a better fit than modelling a binary distribution (bernoulli trials). I still have an issue modelling the large number of zeros but the project is now on the back burner.
Your answer implicitly assumes the autocorrelation is irrelevant and as such it fails to address the key point of the question, and may even be misleading.
@rbm Don't mind about that.
To get realizations of long memory time series I would recommend Wornell's 1996 book (link : http://books.google.cl/books/about/Signal_processing_with_fractals.html?id=o_1SAAAAMAAJ&redir_esc=y ), :-). Although the tractability of long memory processes it's not "that" easy, you can still do it.
Caveat, with such a large sample size, one would expect even tiny differences in occurrence to produce "strong" significance levels. See [Why does frequentist hypothesis testing become biased towards rejecting the null hypothesis with sufficiently large samples?](http://stats.stackexchange.com/questions/108911/why-does-frequentist-hypothesis-testing-become-biased-towards-rejecting-the-null/)
You would want to take a look at `caret` package which can do repeated cv and tune for both alpha & lambda(supports multicore processing!). From memory, I think the `glmnet` documentation advices against tuning for alpha the way you doing here. It recommends to keep the foldids fixed if the user is tuning for alpha in addition to the tuning for lambda provided by `cv.glmnet`.
Most of the answers in that *Mathematica* thread address the different problem of regression with errors in $x$. The summary of [my answer there](http://mathematica.stackexchange.com/a/13087/91) explains the technique as plainly as possible: use weighted least squares with weights inversely proportional to the variances in the $y_i$.  This capability is built into most statistical software, so if your favorite software does not implement it, just look elsewhere.  (`R` is free and many of its commands, such as its basic regression command `lm`, accept weights.)
@deps You are right, but it seems likely that people are comparing angles, not areas, in circular and near-circular pie charts, so the situation is not quite as bad as with area perception. Pie-like charts and "spinners" have been found to be good ways to help people think correctly about probabilities and proportions, too, so there is some possible merit in the use of pie charts. (See [Decision making from a cognitive perspective](http://books.google.com/books?id=PIOidpRPz18C&lpg=PA307&ots=W2ibugZrUe&dq=spinner%20probability%20cognitive&pg=PA307#v=onepage&q&f=false) (Busemeyer et al), p. 307)
For cash flows, I might convert to the level of cash and take log changes. For confidence, I would use changes if everything else is log changes. After performing the PCA and you were concerned with forecasting, I might apply a Factor Augmented VAR (many of the variables are likely not random walks).
+1. Another good mathematical exposition is here: http://terrytao.wordpress.com/2008/10/10/small-samples-and-the-margin-of-error/
I think this bears some resemblance to what is know in ecology as “capture/recapture” or “mark and recapture”. Using these keywords might help you find relevant information.
I see the confusion, I added some links/comments on that. Basically, yes, residuals are enough, much in the same way that in a straightforward neural network approach or an ARIMA model you would only use observations of the same process. The distinction between inputs and outputs is time-dependent: You want to use past observations or residuals to predict future observations or residuals.
Words uninteresting in normal English (articles, etc.) may actually be relevant in tweets: that would be the case, for instance, if you change the style in which you write depending on the positive or negative nature of the message. (That may be one of your reasons not to use a stoplist.)
Welcome to the site, @user29503. This isn't an answer by CV's standards. Would you mind expanding it? It's fine to have a link as a reference, but consider the probability of linkrot. If it helps you, we have a guide to answering questions on CV [here](http://meta.stats.stackexchange.com/questions/1390/how-should-questions-be-answered-on-cross-validated).
Why do you think a linear regression is suitable to capture the relation between date and time? It looks like there is no linear relation between memory and date at all.
The plot actually isn't so bad.... but how to improve it depends on what you want to emphasize.  Do you want to just plot weekly data? Do you want to add a smooth line? 

You ought to change the x-axis labels, certainly....
+1 A very nice response.
Ok if you say, thanks
So this is a case of correlation not causation between this period's salary and distant past's salary, yes? In contrast, the case of AR(1) as MA($\\infty$) is actually causation, as in my boss mood in distant past does have a causal impact on my salary, albeit a diminished one. Plus, is there any intuition for the alternating sign?
If you just want to answer the question: "which hyperparameters are the best", then doing model selection on the full dataset is ok. If you want to answer the question "what is the performance / score of the model with tuned hyperparameters", then you should split the dataset into a development set and an evaluation set, run grid search with cross validation on the development set, retrain a model with the best hyperparameters on the development set and evaluate it on the evaluation set. Otherwise you might run the risk of hyperparameters overfitting.
I would not report any of these distributions as symmetric given the outliers and long tails. 1000 is a small sample size for bootstrapping these days. The perfect case would be assessed by simulating from a hyperbolic distribution, but you have four parameters and the shape of the sampling distributions is likely to depend on all of them. "OK" is no easier to answer than "reliable".
You might find this thread helpful: [which pseudo r2 measure is the one to report for logistic regression?](http://stats.stackexchange.com/questions/3559/).
Question was cross-posted to StackOverflow and answered there: http://stackoverflow.com/questions/9675936/matrix-manipulation-with-sas-iml/9685583#9685583
Actually, if the samples are independent, that's a highly significant difference.  Consider m1 and m4: $p_1=0.312$ and $p_4=0.274$, but with those sample sizes, the std. error of $p_1$ is only about 0.008 and of $p_4$ about 0.007.  Consequently the two are roughly 3.6 std. errors (of the difference) apart.
All models can be approximated by a Taylor series, i.e. a polynom of sufficiently high order. However, if you know the model from the science, you should use it. In fact, the second fit shows that the `dcl ~ I(J^(-0.5))`relationship is significant. Now you need to consider how Re could factor into this. Could it possibly modify the relationship between dcl and J? Than look at interactions.
@Akavall yes, I thought this seemed like a natural extension / generalization of what you had outlined.
@wok: oops, I haven't seen your last paragraph!
The basic question is: how do I construct interaction effects between a continuous variable and a categorical variable with 5 categories?
Can you please define "stable?" There are a number of definitions for this concept from control theory, if they apply then I can answer your question.
By "covariates are not in all the models" I mean that between the three top models one covariate could be in one of the three but not in the other, or an interaction for example. I was wondering if the model averaging command in R would average my coefficients without me having to add a 0 coefficient when some of the variables are not in one of the models. I don't know if I am being clear...I am running a count response variable without zeros and my covariates are continuous except for one categorical with two levels.
Could you explain what you mean by "model" a distribution?
@Durrett, According to your examples you are averaging $Z_{I_{k-1}+1},\\dots,Z_{I_k}$. May be you could add some story behind the formal setup? And what do you want to do with it ? Possible tasks would be to disaggregate $Z$ from $X$ or at least to estimate the distribution it comes from, but since $X$s distributions are the convolutions of $Z$'s distribution, it won't be easy.
@TroyofHelen Nice! If $P(AB) = p = P(A^cB^c)$ and $P(AB^c)=q, P(A^cB)=1-q-2p$, then I get $P(A)=p+q$, $P(A^c) = 1-p-q$ and $P(B)=1-p-q$ giving $$P(B^c\\mid A^c) = \\frac{P(A^cB^c)}{P(A^c)} = \\frac{p}{1-p-q} = P(A\\mid B) = \\frac{P(AB)}{P(B)} = \\frac{p}{1-p-q}.$$  Also, interestingly enough, $P(A^c) = P(B)$.
@Penguin_Knight; i was reading the wikipedia article on the Wilcoxon signed-rank test..is there an example with a software package that i can follow such as Matlab/Excel/SPSS, particular to my case? i am getting confused by the volume of stats jargon(being a software dev and not a ux specialist)..thanks
Thank you. I very much appreciate the reply. I guess my main issue is the first point under section b)... i.e., how the "second branch" of the pdf is defined. It is a discrete pmf and doesn't really define a pdf from the definition of a pdf. Could this section be explained further? Thanks very much.
That's rather critical information in your comment there. It's probably better to also add it to your post.
I don't think you have given enough context here. Otherwise put, this does not seem to differ notably from why are variables often log transformed and getting closer to linearity is perhaps top reason. The fact of being a covariate is immaterial here; for example, outliers and even skewness in covariates can be more difficult to handle than the opposite.
thanks for helping me
The $t$ value and associated $p$ value may be for the $\\beta$ coefficient, for the constant $\\alpha$, or for Pearson's $r$. Basically, each of these estimated statistics, has an associated Wald-type test associated with it. The OP needs to understand _which_ $t$ and $p$ are being requested.
@whuber, from what I found, the Gauss-Newton regression is non-linear regression approximation. Instead of model $y=f(x,\\beta)+\\varepsilon$ model with Taylor expansion of $f$ is fitted: $y=f(x,\\beta_0)+\\frac{\\partial f}{\\partial \\beta}(x,\\beta_0)(\\beta-\\beta_0)+u$. Here is the [link](http://russell.vcharite.univ-mrs.fr/EIE/fchap6.pdf) where this regression is described. Unfortunately (if you do not know French) it is French translation of [this book](http://qed.econ.queensu.ca/dm-book/).
Why not just average the individual time series?
What assumptions of the linear regression model (I assume that you're referring to OLS) is violated by skewness in the <data>?
Did you run it both for $H_0$ true and for $H_1$ true before computing the rejection rate? You should get a $5~\\%$ rejection rate if you _only_ run it under $H_0$. That's what is meant by type I error rate. The probability of rejection is larger under $H_1$, so if you run it under both hypotheses the rejection rate should be above $5~\\%$ (depending on how many times you run it under each hypothesis).
Thank you I will look into global sensitivity analysis for sure!
@nupul Well, the opposite of "up" is "down" and the opposite of "positive" is "negative".  I tried to give a one sentence answer. Yours is much more complete.  Even your "how two variables change together" is more complete, but, I think, a little harder to understand.
Because a mode is arbitrary--its calculation depends on how the data are grouped--and, when used to infer the mode of a population, can be very uncertain--the mode is of little use in assessing symmetries. Instead use the mid-quantiles, defined as the averages of complementary quantiles (such as the average of the upper and lower quartiles). A plot of the mid-quantiles will be perfectly constant for symmetric distributions; deviations from a constant indicate (and measure) departures from symmetry. Because "bell curve" is too vague to be of any quantitative use, that issue cannot be addressed.
This is too general to allow good answers. People in this field will surely want to know (a) what you are doing precisely? (b) what do you mean by "more robust"? (c) why not Dempster-Shafer?
$X$ and $Y$ also should be independent, which is worth mentioning.
floor((year-1900)/10) will produce a factor to summarize over.
@O_Devinyak. I have added my code. Yeah, the NN tool is performing minmax normalization to make the output in the range [-1 1]. However, I am using nn's sim function to predict and compare it with the actual values. So I guess it should have been fine
I agree with Dikran Marsupial that you cannot be that of a purist! You actually start your analysis by considering that the normal distribution is not the "true" one. The idea of using a Bayes factor is then relevant: you could run in parallel similar analyses for different families and let the Bayes factor pick the most appropriate for your data. That you cannot run this for all possible families and that you only consider the most likely cases is a constraint of time and pragmatism. No [sin of using the data twice](http://xianblog.wordpress.com/2012/01/13/on-using-the-data-twice/) there!
GLMNET is another R package
@FabioF, it should rejected $5\\%$ of the time when the null hypothesis is true - that's the definition of the significance level, $\\alpha$.
(+1) *Technical note*: gamma, beta, and hypergeometric functions are analytic almost everywhere in the complex plane.  They're actually fast and easy to calculate to high accuracy.  *All* functions "require numeric evaluation" at some point!
@Kyle. This file grows continuously and can get very large. So i want a way to do on-demand estimation very fast.
The model fitted is not logistic. It grows from 0 and approaches an upper asymptote exponentially.
caret support for `Matrix` objects would be wonderfull, but it doesn't currently exist.  Everything gets coerced to a data.frame.
Could you be more specific which chi square test you use and to what end? For example, what are those parameters, what is the issue with them being correlated? We need more info about the problem you're trying to solve. Or are you simply referring to a problem solvable by a McNemar test http://en.wikipedia.org/wiki/McNemar%27s_test ?
Your file is 20K. Why on earth would you save it as binary and compress it as rar when you could simply upload an ascii or CSV file and save people a lot of mucking about? In fact, the *best* thing to do would be to grab a small subset of your data with the same issues and put the data in your question, so your question doesn't break when that file becomes inaccessible.
You seem to have quite a high N, right? This may just be a consequence of the fact that with a sufficiently large N, *any* deviation from H0 will get significant. I'd assume that, for instance, the large number of residuals clustered around -200 should "not happen" under normality.
@whuber Oh I agree, and I've seen some work on interval estimation of late, but for some reason it just feels weird in my brain to write "point estimate of an interval" even though such a beast is perfectly well defined :) (well maybe it isn't - it's not a point! - but I think you get the idea...)
@MichaelChernick Yes you're right I don't have a very good grasp of statistics, my goal is to remedy this. That frequentist idea is actually not intuitive to me at all.. :/ I was hoping if it is presented in full rigor with $\\forall$'s and $\\epsilon$'s and functions between sets and such, then I could make sense of it. Bayesian statistics is much easier since I can just think about conditional expectation of random variables.
Got that! I am also trying to mathematically show that the probability of drawing a certain red ball before the green one is independent of what other red balls having already been chosen. But, since probabilities are assigned based on the weight of remaining balls, I do not see how these are independent events. Any clarifications would help.
Related: http://stats.stackexchange.com/q/105387/930.
If you have such a sample, you can assess the suitability of say a gamma model of interarrival times with perhaps a [gamma probability plot](http://www.itl.nist.gov/div898/handbook/apr/section1/apr165.htm) (QQ plot), or by comparing a histogram with a fitted disribution, or some measure of goodness of fit. Another possibility would be to see if the cube root was close to normal. You can check for a shifted exponential by subtracting the smallest observation from all the others and checking an exponential QQ plot. You could check for serial dependence by looking at an ACF
Related: [Intuitive explanation of unit root](http://stats.stackexchange.com/questions/29121/intuitive-explanation-of-unit-root).
Readers may also find this thread: [number-of-significant-figures-to-put-in-a-table](http://stats.stackexchange.com/questions/8734/) of interest.
@awshepard: Try http://stats.stackexchange.com. They also cover data mining, which I would say this falls under. Also, a "which algorithm should I use" question typically requires a lot more information about the size and shape of your data than you've given here.
Thanks, this is helpful. I would also be happy to hear what other people think.
@whuber this is pretty amazing. You are a wonderful teacher. I appreciate both yours and rpierce's answer, I'll still give him credit as it did solve our problem, and you have shown why the behaviour occurs. Ty!
An image of *the very first graphic* (a manuscript draft in, I believe, Mendeleev's own hand) is online at http://images-of-elements.com/l/draft.jpg.  For subsequent (and preliminary) developments, see [the presentation by David Hobart of LANL](http://periodic.lanl.gov/mendeleev.shtml). The table had acquired its standard (modern) appearance by 1938.  Some innovative visualizations appear on slides 46-48.
Note that the EM algorithm relies on the inequality $\\log\\left(\\int\\right)\\geq\\int\\log()$.  This would be a good place to start.