tttjjj/elt_bench huggingface dataset:
@Arthur Indeed, it has been updated! (The contrary would have been surprising.) Not a big deal, though: you can follow the above steps by noting that the inner loop now starts around line 97 (depending on your version of `DAAG`; mine is 1.15, packaged on 2012-07-26 10:02:12 UTC).
Did you actually read Dougal's reply? It covers everything you suggest that has any theoretical justification: he describes the "noise," writes the system of linear equations, and shows how to use least squares regression ("polynomial" is unnecessary and inappropriate).
This question appears to come out of an unstated context in which so much is taken for granted that in its present location it is incomprehensible.  Is $R$ perhaps a radius?  Is a "circle" the one-dimensional boundary of a disk or the entire disk?  Is a "range" a distance or an interval of angles?  What is the "resulting graph"?  What is a "component"?  Although readers can make reasonable guesses about these, we cannot suppose everyone will guess the same. Therefore, *please edit this question to provide the particulars needed to make it answerable.*
If you really only want to use the names of the teams, you could use the kind of rating system which is used for chess or other two-player games, for example [Elo][1] or [Trueskill][2]. There was also a [Kaggle contest][3] about this. You could perhaps start with one of these systems and then enhance it using the other covariates. 


  [1]: http://en.wikipedia.org/wiki/Elo_rating_system
  [2]: http://en.wikipedia.org/wiki/TrueSkill
  [3]: http://www.kaggle.com/c/chess
@Glen_b  yes library(glmnet) in R
Aha! I see this now. I got lucky because the binomial coefficient in this case is 1. Thank you both
Glad I could help. Keep @PeterFlom's advice in mind though--the problem here isn't the evaluation method; it's the classifier.
Hi again.first of all let me say that I'm a new graduate. A random process is composed of several deterministic signals.it's called  Random processwe don't know which one of these deterministic signals will happen in a trial.
The title of this question almost read like the short description of plyr :)
(1) Could you please tell us what you mean by "DV"? In this context most of us would read "dependent variable," but then it makes no sense to say it is "significant." (2) What do you mean by "interaction"? I suspect you intend that to refer to the "composites" you have created, but ordinarily in a regression context an "interaction" refers to a *product* of one or more independent variables. As a result of these issues, I'm unsure what you are asking and suspect others may be equally puzzled.
@Legend You can also try a [rootogram](http://rss.acs.unt.edu/Rdoc/library/vcd/html/rootogram.html) (don't know if it overlaps with @Chase's response on SO). Now, I don't understand why you want to try and fit every discrete distribution to your data. Either you have *a priori* knowledge or hypothesis about the law of your outcomes, or you don't. In the former case, you might want to explain why the observed data don't fit the model. In the latter case, you're left with exploratory data analysis (and, potentially, non-parametric density estimates, mixture models, etc.)
How do you expect these ages to vary? Is there a maximum age? For example, would it be more appropriate to model the age distribution for each disease as a multinomial or a poisson? My first thought would be to use some kind of dirichlet process clustering on the diseases with respect to the age distribution, but that may be too involved if this is supposed to be a quick task?
@Antonio2100: if you do univariate $z$ scores, you change the multivariate shape of your data. To preserve the multivariate shape of your data you should do a multivariate standardization (see my answer).
*That* it's reasonable to ask in comments at the original; it's a clarification of his meaning, not a whole new question. However, I may offer the suggestion that since $n+1$ looks rather like the more usual $n-1$ (or $n$ for that matter), one might be tempted to think he simply made a sign error or something, and *sic* is perhaps intended to emphasize that, no, even if it might appear possible that he meant $n-1$ or $n$, he really does mean $n+1$, just as originally typed.
*the sparsity is an empirical property of natural signals* Could you explain this statement? Given an image, why should it be sparse? For example, is the lena image sparse?
They aren't the same! See [Land & Deane, 1992](http://www.jstor.org/stable/270997).
Thank you very much for your reply. If p is the success rate, isn't that the formula for expected cost should be as follows? ∑n (1-p)$^n$$^-$$^1$ p c Define q = (1 - p) and assuming n tends to infinitely, we've got the following ∑n (1-p)$^n$$^-$$^1$ p c = p c (∑ d/dq (q$^n$))
= p c (d/dq (∑q$^n$)) = p c / (1-q)$^2$
= c/p
(1) I asked. It's fine if you have constraints preventing you from disclosing any more information, but please be aware that is likely to limit the scope and quality of answers you can expect. (2) On an *implementation* level they will be draws. But let's ignore that: conceptually, you are asserting that `Y[i]` and `alpha[group[i]]` are *identical* and that `1/(SE[i]*SE[i])` and `eps_tau` are *identical*. Neither I nor, evidently, the software itself can find any evidence that assures this. Might I suggest that the first step in a solution would be to clarify *conceptually* what your model is?
@bil_080, the OP does not mention R, and help page for Box.test in R mentions the correction and has an argument to allow for the correction, although you need to supply it manualy.
Given the many extant definitions of "matrix," and the many more ways in which they can be applied to representing data, what exactly do you mean by a matrix "having an individual" in it?  What do your matrices represent?  What do you mean by "cross-matrix relations"?  What exactly does "correlated" mean in this context?
@AdamO Given the syntax used, I'm pretty sure this is `gam()` from package **mgcv**, not the same from package **gam**.
I suggest you start with the introductory econometrics book by [Woolridge](http://www.amazon.com/Introductory-Econometrics-Approach-Applications-Solutions/dp/0324581629) for panel data. If you are using `Stata`, see the helpful notes [here](http://dss.princeton.edu/training/Panel101.pdf) and for R, see [here](http://dss.princeton.edu/training/Panel101R.pdf). Also, please post the output, sample data, and code whenever possible along with the software you used for the analysis to get helpful feedback.
What do you mean when you say fit a model with one independent variable, controlling for the other -- as opposed to fitting a multiple regression model with both of them as predictors? Including model would help explain a lot of variation, so it should be included. How big is your database? Wouldn't you want to just use all the data -- or all the data for a certain range of model years?
With a uniform prior distribution wouldn't guessing 30 be equivalent to guessing 35, since both will have 65 actual test grades that will result in a passing grade?
But couldn't you express a facial image in terms of, say, the first 3 eigenfaces?  The result would be an image of reduced dimensionality, wouldn't it?
Is this homework (it kind of reads like it)? If so, please add the `homework` tag. For such questions, we will provide hints, but not full solutions, in general. It is helpful if you also edit the question to include the work you've done and what specifically you are finding challenging.
Certainly; it's just multivariate multiple regression instead of univariate multiple regression.
@Zen that's probably worth expanding (just a little; you basically cover the underlying issue quite well) into an answer.
I suspect this reply would be of higher interest if you could supply additional details on the reasons to use the `coin::wilcoxsign_test` R function. Also, are you referring to the OP formula, $\\text{effect size} = z / \\sqrt{N}$?
* threshold could be like 30* for this instance
You have already received lots of good advise in the thread on Statalist starting here: <http://www.stata.com/statalist/archive/2013-05/msg00321.html>
Looks like image can accept transparent colors after all. See: http://menugget.blogspot.de/2012/04/adding-transparent-image-layer-to-plot.html . In case you still want to use interp, you can generate your xy coordinate combinations for your z levels with : "grid <- expand.grid(x=x,y=y)"
Can you say where you read this? Some context would help. Can you explain why you have used the 'paired comparisons' tag? Where does the pairing arise?
Under perfectly satisfied assumptions (normal distribution within groups, equal variances, independence), a t-test works even with group sizes of just 2. (Of course you can't check the conditions.) This is in contrast to a Wilcoxon rank-sum test that needs at least four observations per group to produce a p value below 0.05. Thus, a Wilcoxon test is of no use here I am afraid.
@what: It doesn't expect it - it's optional - & it's for something other than the Wilcoxon signed-rank test itself: the computation of a confidence interval for the pseudomedian. It's explained in the help page.
I would suggest you remove the duplicate on MathOverflow.
According to the conditions in the first equation, shouldn't the upper limit in your integral be $p_i$?
In fact, the results are not statistically significant. The P value with Fisher's exact test is 0.26, and 0.38 with a chi-square test (with Yate's correction).
a lot of good info in this message - thanks very much @Kruger
Hi mpiktas, thank you for your response but the R result is very different from SAS proc mixed procedure. The SAS output is the same with the answer in a textbook(Kuehk RO. 1999. Design of experiments: statistical principles of research design and analysis, 2nd edition). The dataset is        method day level
1   1 142.3
1   1 144.0
1   2 134.9
1   2 146.3
1   3 148.6
1   3 156.5
1   4 152.0
1   4 151.4
2   1 142.9
2   1 147.4
2   2 125.9
2   2 127.6
2   3 135.5
2   3 138.9
2   4 142.9
2   4 142.3
My reading of that section is that you cannot *guarantee* positivity: the choice of $\\bar{\\alpha}=10$ appears to be an empirical one, selected so that "an unstable backwards heat equation" does not occur. I think that for any deeper insight into this choice you would have to to consult with the author of that tutorial.
I suggest you don't try to distribute fewer than 1 ball. What does negative balls mean in terms of the original problem?
Awesome ... thanks for the link
Standard trees do not handle continuous predictors well.
The sum of two covariance matrices is positive semidefinite; the eigenvalues are non-negative.
Because your notation is rather abbreviated, it might be worth pointing out that it implicitly refers to some Borel set $A$, subject to a universal quantifier, and that a fuller rendering of your question therefore would be whether it is the case that $$\\forall_A\\ \\mathbb{P}(Y\\in A|X=x)=\\mathbb{P}(g(X,Z)\\in A|X=x)=\\mathbb{P}(g(x,Z)\\in A).$$
Benjamin, I've tried density and it doesn't do what I want. Moreover, the plot I got its nothing like a fit of the data. I don't know about EM algorithm "or something"...
Why boringly?! This is one of the most frequent uses of statistical methods.
Are you scaling the out of sample test set in the same way? If you are normalizing, are the mins & maxs dissimilar? If you are standardizing, are the means & SDs dissimilar?
**Sure**. Let $(X_i,Y_i)$ be your iid bivariate sample. Let $X_i$ have an *arbitrary* distribution $F$ and take $Y_i = X_i + Z_i$ where $\\{Z_i\\}$ are iid $\\mathcal{N}(0,\\sigma^2)$.
It is possible. It means that any tree other than the root node is overfit; that is, your best tree has no branches.
Yep, that explains it. I was sure that ecdf at 0 matched the fraction of x<0 and was also not expecting that **my** `x` would contain that much 0's. Thank you
@dbr I am still not sure I completely understood your question. I have now started thinking that you want just gaps in weekends and connected lines for each five days?
I have to disagree with my friend @StasK here about Mostly harmless econometrics. It's the kind of book professionals love and imagine they would have loved if they had read it when they were students. That's probably true for the students who will become professional econometricians, but I'd predict that for typical social scientists setting that book would be more likely to provoke rebellion. It's too difficult unless students have traversed much of the material before, and very thoroughly. (In any case, the question of the OP is about a bachelor's programme.)
I recognize that that question was about one-way ANOVA & you have a two-way ANOVA, but most of the methods will generalize to >1 factor. OLR is the most general non-parametric approach. All you need is for your data to be ordinal in nature, ie you need to be able to confidently say .66>.31, if you can say that, you can use OLR.
Well-written question, and I learned a few things from reading it. I do wonder if random effects analysis is doomed in the long run, perhaps to be beat out by more straightforward smoothing techniques. If I do try to answer #1, I'll go for a situation with high variability, many subjects, and small sample sizes within subject. Then the within-subject fixed estimates will be all over the place.
@Queops, R code `pbinom(5,20,1/5)-pbinom(1,20,1/5)` gives 0.7350325, so yes.
Hi @EngrStudent probably the histogram is not the best. The stepclass function from klaR has cross validation, but I could not figure out how to use it.
@Tom, example given
I would simulate y1 from the first exponential distribution, y2 from the second exponential distribution, and I would store the minimum between y1 and y2 along with a binary variable indicating which event occurred (1 if y1 < y2 and 2 if y2 < y1).
@Erik "$\\cdot$" denotes multiplication and "$*$" denotes convolution.
+1 Thank you for joining us, Patrick!
Cool. I have a text that deals with left truncation in a discrete time hazard model, simply by setting the customer's time (or whatever is the unit being measure) t= their tenure at the truncation date. All previous dates are excluded. Then, they only contribute to the likelihood (assuming fitting with a logistic regression) where observed.
This looks like a conditional probability. For example, $f_{A}(e)$ looks something like the conditional probability that event $e$ came from $A$ given that we know it came from either $A$ or $B$.
It's more that a paired sample t-test doesn't answer all your questions and doesn't make full use of the data.
Thank you, Elvis! I modified my code by correcting the value of sigma = 0.0745, and I got the same result. Thanks for the advice, I'll keep this in mind :-)
There are many libraries out there. If you are using Java, there is Apache Mahout. For javascript I just found this: http://ganeshiyer.net/blog/2013/10/22/machine-learning-with-javascript/. In python you have scikit-learn. Be careful, thought. You need to understand some basics before using any m-l library (e.g.: overfitting, training/test sets, Cross validation, to interpret the results such as the parameters of the model, etc).
Since you're fitting GLMs, the [deviance](http://en.wikipedia.org/wiki/Deviance_%28statistics%29) for GLMs would be the relevant thing.
So, the model is $g(\\mu) = \\beta_0 + \\beta_1 X_1 + \\beta_2 X_2 + \\beta_3 X_1 X_2$? I think the reasonable thing to do would be to interpret the effect of $X_1$ conditional on $X_2$, i.e. look at $\\beta_1$ for $X_2 = 0$ and $\\beta_1 + \\beta_3$ for $X_2 = 1$. Essentially, interpret the regression separately for $X_2 = 0$ and $1$.
@user2991243 : For which case? Do you have two data sets (i.e. two vectors with samples) or one sample and a prespecified distribution?
It can be obtained by algebraic manipulation beginning with a definition of covariance. What definition are you using and where are you stuck?
@Josh, I updated my answer above.
That helped me work through the problem, thank you. Turns out it didn't want the first column of names in the file, once those were removed it worked perfectly.
"Trying to find often repeated cases and such" seems like a very limited and vague objective. Could you be a little more expansive about what kinds of "relationships" you are looking at and what the project aims to accomplish?
I agree. The question is either bogus or has been misinterpreted in some way from some other source into the form presented. It doesn't make much sense as it stands.
@jwdink, for simple cases your are correct, and possibly even some medium cases, but some of the harder cases (including many real problems) it is not that simple.  In some cases you can easily define the correct conditional distributions to do gibbs sampling, but difficult to find the actual posterior.  In other cases you may find the posterior, but it has multiple local modes over many dimensions, you may find a point that has a gradient of 0, but how do you know that it is **the** maximum rather than a local high or saddle point?
@act00 What I mentioned is that the tails of those distributions are different *independently* of the sample. The tails of the log-t are heavier than those of the lognormal and they become more similar as the degrees of freedom parameter goes to $\\infty$. If you have estimators of both distributions, say $\\hat F$ and $\\hat G$, then an informative graphical tool is the graph of $\\dfrac{1-{\\hat F}(x)}{1-{\\hat G(x)}}$.
Well, it’s the $1-\\alpha$ quantile of the `permstat` vector...
@whuber.  Indeed, my comments are only on the differential geometry of the situation and the tangent space.  I am not at all clear on what conditions on the $p$ would be necessary  to make the map an isometry.  But as I understood the question, it was really getting at what was the difference between an identification ('the same') and an isomorphism.
I would ask you to reflect on what it means for two predictors of a third value to be "independent" and consider explaining that to your readers.  In particular, it is *strongly* counter-intuitive that combining two "independent" predictors would actually *lower* the chances predicted by each predictor separately.  I suspect you may have interpreted this question in a radically different way than I read it.
Are these functions from some parametric distribution? Or how did you get them? And what happens/what error do you get when you run your code?
@Affine  Not a problem at all.  I appreciate your taking the time to provide the explanation.
Your problem is far worse than non-constant variance: obviously the means of the residuals vary strongly with the 'index'.  GARCH will not be the cure!
@gung I am reluctant to convert this reply into a comment because it seems that it *can* be interpreted as an answer: it asserts that the existence of a conjugate prior is of little importance apart from simplifying computations.  (I believe there may be reasons to dispute the validity of that assertion, but being incorrect is not the same as not answering!)
(This would be more appropriate as a comment than a reply.)
Another great illustration of why not to use Excel for anything statistical. :-)
Could you tone down the criticism in this a little?  Your point is fair, but I would love to see Gary get involved in the site (if at all possible)...
Clarified. Sorry for the loose language.
You're right, that doesn't make any sense. Who told you that? Are you sure they didn't mean that you need to *include* those factors? (Eg, perhaps they meant that you need to eliminate *the effect* of those factors *on the dependent variable*--ie, you need to be sure to include them.)
Thanks for the answer. It is not possible for me to install xttest1?

After writing this question I found one note saying that xttest0 has been modified and can be used for unblanced panels too?
this seems closest to what I was looking for.
Perhaps the following intuition might help: Similar as with splines, you might think to approximate a function by concatenating many local functions that are bounded and monotonic and then smoothing this series. The function (e.g. a sigmoid) can be made differently steep by the linear function in the weights and x. Think of a polynomial of order 2 and how to approximate it with two S shaped functions where one is the mirrored version of the other. Does this help?
@whuber my take away from the answer is that a correlation based on non-stationary data yields a random variable, which may or may not be useful. Correlation based on stationary data converges to a constant. This may explain why traders are attracted to "x-day rolling correlation" because the correlated behavior is fleeting and spurious. Whether "x-day rolling correlation" is valid or useful is for another question.
Can't you run kNN on any distance matrix? Just compute what you find adequate. A ball tree is just a data-structure and doesn't give you a metric, no?
I suggest this question be migrated to Stats
You might be interested in [Projection Pursuit](http://en.wikipedia.org/wiki/Projection_pursuit). (Available in the [GGobi](http://www.ggobi.org/) software.)
Not sure I expressed this quite right - my 'data' is the logged daily change, then after this I have taken one month (the last 30 logged daily change), 1 year (last 365 etc.) then i've fit a different distribution to each time period. For example, the daily returns over a month follow a t-distribution, the daily returns over 1y follow a skew-normal etc.
@Julian:  I wish I had more time to provide a longer explanation, but maybe it would be helpful to read the wikipedia entry on entropy under "Characterization — additivity".  Remember that an infinite surprisal corresponds to a zero probability.  Are you okay with a product of terms some of which may be zero?  After studying the characterization of entropy, I wonder if your intuition about what you "think is not good" will shift.
Are you asking a statistics question or an SPSS question?
To be more specific I plotted the above data and R and added a trend line geom_smooth(method="lm") which results in an upward trend. I need to find if that upward trend is significantly different from no change.
+1 for warning about collider bias.
As far as I know McNemar totally ignores diagonal frequencies. P-value must be the same whether you have diagonal `12207 12200` or `122070 122000` or even `12207 120`. Check this.
It's a bit of a tautology: the z-score $Z$ is the value such that the quantile is $\\alpha$, here 0.05 (or $\\alpha/2$ if you consider both tails). In that sense 1.64 corresponds to a 10% critical value ("z-score") with 5% in each tail.  For 5% in both you need 1.96 corresponding to the two-sided 2.5% tails.
From what I wrote before, I think neither tool correctly finds the effect of $drug_a$. Your situation can be due to the fact that within both groups (treated and non-treatment) people with higher hormone_a have higher hormone_b. But if you must choose between those two, you should take the regression, because it somewhat takes into account the other factors, and you should look at the coefficient of $drug_a$ to see if the drug affects the level of hormone_b.
@whuber Are you referring to question 2 or 1? If 2, $X^{-1}(B) \\in \\mathfrak{F} \\forall B \\in \\mathfrak{B}$. Hmmm...does the 1st part of 2 follow from the fact that $X^{-1}(\\sup(y \\in \\mathbb{R}: F(y) < \\omega)) \\in \\mathfrak{F}$ and the fact that $\\sup(y \\in \\mathbb{R}: F(y) < \\omega) \\in \\mathfrak{B}$? No idea for 2nd part of 2. If 1, X^{-1}(B) \\in \\mathfrak{F} \\forall B \\in \\mathfrak{B}$. What may I ask is the relevance?
(+1) Good response. I would add that these are often particular contrasts that are interesting in DoE as analyzed through ANOVAs, and that the default ones (as returned by `summary.aov(lm(...))` or `anova(lm(...))` are not of particular interest.
Can you please elaborate on that sentence? What I understood from it is that the signif. level (=alpha) can be figured from the rejection rule - and that's what I did and how I really found this alpha. Are you saying that the rejection zone is just $C_a=[168,\\infty)$ ? If that's true why do I need the alpha in order to answer the question at all?