tttjjj/elt_bench huggingface dataset:
Questions like this are a little problematic. They don't have single, factual answers, but tend to elicit opinions. That is considered a reason for closing a question as off-topic (see our [help center](http://stats.stackexchange.com/help/on-topic)), note that CV is not intended as a discussion forum. I do think this is a gray area, & that there is some room for questions like this. At a minimum, you should make this Q community wiki, but be aware that it may still be closed.
+6, impressively clear, comprehensive, & thoroughly annotated.
@David After reading this question over many, many times I still don't understand it.  Are you using the two observations and the scientist's opinion to estimate a prior for a Bayesian analysis?  Is your prior going to be based on the scientist's opinion only and then you want to update it with the observations?  What is the distinction between "unsurprising" and having zero probability?
One application of this: hypothesis testing when you have many samples from either $\\mathcal{D}_0$ or $\\mathcal{D}_1$ (and you want to determine which).  See also [Hypothesis testing and total variation distance vs. Kullback-Leibler divergence](http://stats.stackexchange.com/q/17300/2921) and also [this](http://stats.stackexchange.com/a/23978/2921).
Error in solve.default(sig0) : 
  system is computationally singular: reciprocal condition number = 1.938e-21
[This post](http://stats.stackexchange.com/questions/27033/in-r-given-an-output-from-optim-with-a-hessian-matrix-how-to-calculate-paramet)  will help. see espeically the answer and the different relationship between Hessian and var-cov depending on the objective function.
It seems answers to this question are not forthcoming because it is a specialized circumstance and to construct useful responses we would need to know the purpose and type of the analysis, information about the amount of data, and much more.
Dear @Zen: Note that a careful accounting of the $N=0$ case will allow you to replace the strict inequality $>$ by $\\geq$ everywhere without changing the final bound.
¿  Why does my markup $\\mathcal N$ not work above?
@Saul I added some graphics based on your data.  Note that an interlaboratory CV of 9% and an independent patient temporal variability of 10% combine to give an effective CV of $\\sqrt{.09^2 + .10^2}$ = 13.5%, exactly high enough to explain the ratio of 2.5/1.7 with 95% confidence.  I would actually expect more variation in the patient over a 10-day period if the *daily* variation were 10%, further increasing the plausibility of the chance mechanism.
@Hong Because the title calls these Gamma variables, and Gamma distributions have non-negative support, we deduce that $I_{(0,\\infty)}$ must be an *indicator function*: $I_{(0,\\infty)}(x)$ is equal to $1$ when $x\\in (0,\\infty)$ and is equal to $0$ otherwise.
Hello Gung,  Thank you for your posting.  It is very interesting and opened my eyes in some areas.  However, I am having trouble reading between the lines of the posting you linked to.

So, my question remains: In a linear multiple regression equation, if the beta weights reflect the contribution of each individual independent variable over and above the contribution of all the other IVs, where in the regression equation is the variance shared by all the IVs that predicts the DV?
@Michael There are alternative, more generous interpretations of that kind of reporting.  For example, the author might be aware that readers might want to apply their own thresholds of significance and therefore do the flagging of p-values to help them out.  Alternatively, the author might be aware of possible multiple-comparisons problems and use the differing levels in a Bonferroni-like adjustment.  Perhaps some portion of the blame for misuse of p-values should be laid at the feet of the reader, not the author.
link: http://socialnetworks.mpi-sws.org/
I'm on too slow a connection to google links now, but I will do so tomorrow.  Thanks for the suggestion.
@AndyW Thanks for the response. That does answer the second bit: don't use change as an outcome, use response and adjust for baseline. I wrote out the algebra and convinced myself why that's a better approach. I also feel fairly confident about treating the correlation structure by using a GEE approach. I see this person treats scores continuously and Frank didn't comment on the validity of that. I'm just a bit curious about the cumulative logit link approach.
Can you give an example, what does the data look like?
This is very helpful -- I didn't realize the statement was in reference to the sample size in the limit.  But is this just another way of saying that the CLT extends to MLEs?  Or that a MLE done where `x --> \\infty` will give us a normal distribution?
It might not be that complicated if you focus just on the three statistics you mention. For instance, the estimated sample variance of a proportion $\\hat p$ in a random sample of $n$ is $\\hat p(1 - \\hat p)/n$. For the $n$ versus $n-1$ issue with the sampling SD of the sample mean, see *inter alia* http://stats.stackexchange.com/questions/11707 or [search our site](http://stats.stackexchange.com/search?tab=votes&q=standard%20deviation%20unbiased).
see also http://glmm.wikidot.com/faq#modelspec
How large are the four groups? Are the error bars representing standard deviations of individual observations within group or standard errors of group means?
What does “replenish a model” means? Also, you might be interested in “semi-supervised learning”.
Thank you very much for reformatting it for me. I really appreciate it and updated my question using your suggestions.<br/>
I enlisted in the help of another researcher, and we tried using Excel with the DDXL Add-in, but we encountered the same limitation GraphPad had with only being able to go up to 2-way ANOVA. She also said she thought I would need to use 4-way ANOVA because she feels I have 4 treatment variables, not 3. I still don't see it that way/why she feels there are 4, not 3 treatment variables. I was hoping someone might be able to shed some light on which is correct?
The quotation is only partly correct: the log likelihood is given by *negative one-half* the square root of the Mahalanobis distance *plus* one-half the log of the determinant of the inverse covariance matrix, $\\det(S^{-1})$, *minus* $(n/2)\\log(2\\pi)$ (in $n$ dimensions). For the purpose of comparing likelihoods with the same covariance matrix the additive factors can be dropped, but the initial factor of $1/2$ is crucial for implementing and interpreting certain tests.
1. How do I know whether a problem is linear or not? According to the training data? 2. Could you recommend some materials about "how to make a good choice of kernel and tune the hyper-parameter"?
@MichaelChernick, I agree with you. I have edited my answer a little to address these issues. Let me know if you think it still needs more work.
Could you help us make sense of the apparent contradiction between the assertions that "the data represent the entire population" and "events are drawn from a differently behaving population"?  Either the data are the entire population or they are drawn from other population(s), but it is difficult to see how both models could pertain at the same time.  Did you intend this to mean that the data are a *sample* but it's the only sample you can obtain?
@kerkejnrke, if you model the distribution of $Y$ conditional on some specific level of a set of variables $\\bf X$, you are doing *multiple regression*. This is a little more complicated, but ultimately the same thing. The mean would be: $\\hat y_i=\\bf X_i \\boldsymbol{\\hat\\beta}$, where $\\boldsymbol{\\hat\\beta}=(\\bf X^TX)^{-1}X^TY$.
What is the reason for the missing data?  This could play an important role; it certainly affects how to interpret the statistical output.
@whuber: i interpreted the question to be, "these univariate distributions are not normal; how can they be made more so?"  The earth's tilt is treated as merely a background factor affecting the temperature distributions; tilt is not plotted.
I suggest a prior that is supported only on $E[f]>\\tilde{f}$, since this is a better summary of your prior information. [Pareto](http://en.wikipedia.org/wiki/Pareto_distribution) with shape parameter increasing in $n$ would be a reasonable place to start. Even better estimates could be computed if you have an upper bound on $E[f]$. If you give more detail about the distribution of $X$ and function $f$, I expect more informative answers would be possible.
Thanks for a nice consise summary with some useful terms to search for. I forget how fortunate I am to work in a field where if one has insufficient data you simply take more.
Ah, I see it's there in scipy, thanks!
"Classifying documents with Bayes" sounds like you'd like to resurrect the poor reverend and make him help you sort a pile of papers...
@gung the issue is not the size of the sample, but the problem of generating an unbiased sample for a dataset which has a huge number of attributes. And generalization is usually done by Machine Learning algorithms, trained on a portion of the data set. The online data streams constantly coming in makes the issue of sampling secondary as batch updates can be used to change the parameters.
?predict.glmnet says only "The object returned depends on type."
+1, in addition to @MichaelChernick's answer, you may find it simpler to use a $\\chi^2$ test of independence for a 2x2 contingency table, which is, in essence, a special case of the GLiM where both variables are binary.
Glen_b raises a good point that I did not consider.  The true analysis may require treating the data as distinctly multivariate.  Although, for poking around in a preliminary way, univariate testing can still give some guidance if you keep in mind that such testing lowers the power to detect weak effects.
Some of this is discussed at http://stats.stackexchange.com/questions/13314/r-squared-useful-or-dangerous/13317#13317.  In light of that, it's good that you focus on the "unexplained" 45% as opposed to fixating on $R^2$ itself.  But just how large an error does that represent?  And does that amount of error matter for the decisions you make or what you recommend?  If not, then you're in good shape.
Douglas Bates seems to agree, at least with respect to the hazy mapping of R-square definitions in mixed models to their non-mixed analogs: https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q1/003363.html
All treatments (a fixed effect) are represented in all blocks. I have a schematic of this that would be most helpful if I could post it :)  Again, thanks for your help!
FWIW, for spirometry the industry standard is to run each patient 3x. If the measures vary by <10%, it is considered a good test. Most advocate averaging the measurements, but a minority advocate using the best under the theory it's a more accurate measurement of what their lungs can do. From a stats point of view, either is a good measurement as long as the same method is used for everybody, & using the average yields a 58% reduction in measurement error relative to best. There isn't an appreciable trend except for patients w/ COPD b/c the huffing & puffing leads them to clear their airways.
Disclaimer: I didn't look at the file. Your link obliges download of an executable, and I don't want to do that. But you seem to have answered your question at the same time as asking it. You can and may calculate a correlation but in so far as it appears not to reflect the structure of the data, you are looking in the wrong direction. The way forward is to seek a model of the data that somehow uses the group structure that is evident to you. No single summary can do justice to such data.
@Aaron @Web These data (V1 through V7) are not tied.  An error must have been made somewhere in obtaining p-values of 1.  I get a lot of p-values of 0.036 when comparing them with the Wilcoxon signed-rank test.
It is the human tendency. Robert Millikan was no exception. I am very happy that so many new things have been enlightened and the philosophy behind a statistical model is emphasized.
+1 Thanks for the link (and the retag). I'll look into this direction.
I don't understand why you have a first random effect as Technique is also part of your second.
Again in a very intuitive way: the more variation, the more information you need.
Thanks for your answer.

As far as I can tell, the algorithm in _A Practical Guide to Splines_ is the same as the one given by Reinsch.

I haven't yet looked at your other reference.

Regarding Kalman filters, I believe they would require additional structure, such as knowledge about transitions between data points.
e.g., Python (http://j.mp/9TVyhv), Perl (http://j.mp/by85El), Octave (http://j.mp/aVQ5Xz), Clojure (http://j.mp/dkj9Z9, not the default one), Haskell (http://j.mp/aWK7kK), lua (http://j.mp/bSD2vO), or even SQL (http://j.mp/aOPJMW, for an overview) :-)
A couple of thoughts: Your `aov()` formula is a bit redundant, `weight ~ Time*Diet + Error(Chick/Time)` is equivalent. Note that the design is unbalanced - `xtabs(~ Time+Chick, data=ChickWeight)` shows that chicks `incl <- c(8, 15, 16, 18, 44)` provide incomplete data. Running `summary(aov(weight ~ Time*Diet + Error(Chick/Time), data=ChickWeight, subset=!(Chick %in% incl)))` then causes `Time` only to be tested against the `Chick:Time` SS. Finally, `Time` is not a factor in this dataframe, but shoud be. I don't know how to replicate the Statistica results though.
Related: http://stats.stackexchange.com/q/6795/2970
At present, this is more of a comment than an answer. Can you expand it a little, perhaps by discussing the nested LASSO penalty, etc?
Welcome to our site!  I appreciate the care and quality that you have put into your first answer here.
Boxplot? This requires some manual work in Excel, though: http://support.microsoft.com/kb/155130 The R Graph Gallery offers a few alternatives here: http://gallery.r-enthusiasts.com/graph/The_boxplot_friends,102
what is "this book" you speak of
It sounds to me that you want to fit a mixture of a given fixed number of normal distributions.  If so there is a lot of theory for doing this. One approach is maximum likelihood.  I expect that there is a lot of software available to do this.  However, the ENORMSEP program that you describe sounds like it is a very different approach to this problem.
$\\sigma^2$ is the variance of $s$, which is the sum of $n$ independent poisson
Some quantitative analysis of this recommendation, or at least one example of an actual application, would be welcome. I suspect that the accuracy of your proposal is *heavily* dependent on the shape of the kernel. This would make me reluctant to rely on such a calculation without substantial study of its properties.
What do you mean by "center *earnings* before taking the logarithm"? It can't be to subtract the sample mean earnings or you'd be trying to take logs of negative numbers.
Hello - could you define/explain 'seemingly unrelated'?  Typically differences in coeff. by group would be tested using interaction terms.
As @Glen_b hinted, a completely general demonstration is much simpler than this: from the definition of the inverse function, $\\Pr_U(F^{-1}(U)\\le x)$ = $\\Pr_U(U\\le F(x))$. The definition of a uniform variate states that provided $0 \\le F(x) \\le 1$ (which is axiomatically true for all $x$) this latter value equals $F(x)$, QED.
If I understand you correctly, ordering the variables by exogeneity, i.e. causal priority will change the order of the coefficients in $B^{-1}$ matrix and, hence, the Choleski decomposition. We thereby attach economic meaning to our choice of restriction.
The order of the dependent variables is then of course only an issue when $n>2$ and the correlation coefficient between the variables is high enough.
Very detailed. I loved it. Thank you and I'll consider it as an answer unless someone comes up with something else.
(+1) for "First, the other IV may absorb some of the residual variability and thus increase the power of the statistical test of the initial IV", which @whuber gave a nice example of
All features should be scaled or normalised. After that, from algorithms' perspective it doesn't matter if features are continuous or binary: for binaries, it sees samples that are either "far" away, or very similar; for continuous there are also the in between states. Hope that helps.
What are your assumptions? It is almost always the case that $\\mathbb{E}[e]=0$, whence your bias term reduces to zero in the usual application where $X$ is fixed. But if you're viewing $X$ as random, your expression for the expectation doesn't simplify in the way you write.
Stupid question -- can't you just partition the space into regions cut by those lines and count dots inside each?
Thank you for reply. Is it possible to get centroids from this type of clustering? – user974514 9 mins ago
Good call. I see now that collapsing to cue to a difference score prior to analysis is equivalent to permitting cue effect to vary by participant.
@PrOno I discussed an approach  http://stats.stackexchange.com/questions/33923/how-to-perform-pooled-cross-sectional-time-series-analysis/33932#33932 . My comment about "no outliers" reflected my incredulity about the analysis that your software reported.
I believe that I have figured this out in R, but I am still unsure of the theory.  In the `MCMCpack` the output is samples of the joint posterior distribution on all parameters, not the marginal posterior distributions of each individual parameter.  So the difference in means can be displayed as `plot(postDist[,"y1"]-postDist[,"y2"])`.
It seems to me that you have *counts*, not the kind of continuous data for which the t-test (paired or otherwise) is appropriate. You have values for the number of cases that were opened in each situation, do you know the total number of cases that could *possibly* have been opened? If so, we can treat these as binomial, if not, we can treat them as Poisson.
Short answer: no. The error doesn't appear in that image, and is not observed. Your best estimate of the error is the residuals (but sometimes it's a demonstrably poor estimate).
I summarized the "several methods" from Bertrand et al as well as added a link.  Thanks for the pointer.  I'd seen this paper before but it hadn't come to mind when planning this particular analysis.
Note that `FORMAT x y 17.12` will show both `x` and `y` as 17 characters long (including the decimal and, when present, the negative sign), and 12 of those characters will be digits after the decimal point. Numbers with more characters or greater decimal precision may not display correctly.
@StéphaneLaurent You can model group membership as a covariate, and - yes, you can add covariates. There can be different events' intensity for one group and different for the other. I've never done something like that, though.
If this is for homework, you should add the `self-study` tag.
@user603 That's always a good reminder to make, thanks.  In this case IrishStat has been clear about his affiliation with Autobox and, over a long time, has consistently been generous with his application of this software on behalf of users who post their data *publicly.*  Sure, it may be good marketing, but (IMHO) it's sufficiently transparent and helpful to fit in with our site, too. If I'm missing the point, or perhaps misinterpreting the FAQ, let's take this up on meta, shall we?
In most applications this will bias the results and in some applications it can have terrible effects. For instance, it will not always flag the outliers unambiguously: up to one-ninth of the *good* data can exceed 3 SDs. Your statistical software, if it is any good at all, can represent missing values (and flag suspect values) and it will handle such values automatically in a meaningful way.
@robin girard: MLE is a good idea for question 1. I suspect that for Gaussian errors, MLE will give identical estimates as my _ad hoc_ minimization. As for $g$, as I noted, we can assume it is positive-valued and twice-differentiable. We can probably assume it is convex as well, and maybe we can assume it is analytic.
FYI, in case you followed the link and hit the back button b/c the web page is in German, Johann Bacher's posted lecture notes are in English.
This seems like a homework problem; if so, it should have the `self-study` tag. See [this help page](http://stats.stackexchange.com/tags/self-study/info)
+1. Given edits to the question, which now suggests that "bell shaped" means unimodal with a mode in the interior of the interval $[0,1]$, the first part of the analysis provides a simple direct answer: $0\\lt (a-1)/(a+b-2)\\lt 1$, equivalent to both $a$ and $b$ strictly greater than $1$.
Thanks for the input. It sounds like all I need is a spreadsheet. Would this also allow me to, given a column "gender", to get the "reason" counts per gender?
(+1) Very good list!
Check out the BTYD R package, which has some models for calculating CLV based on RFM statistics. http://cran.r-project.org/web/packages/BTYD/index.html
Welcome to the site, @Jayran Choopan. Would you mind giving a summary of the info in the linked paper to help readers decide whether it's what they need & in case the link goes dead?
Unless you have perfect contradictions (which are almost absent in real, usable data), domains do not change anything since the classifier should also produce noise instead of exclusively bad class for objects outside its domain -- this way only different partial classifiers will be "good" in each of the domains. Anyway, it is obvious that a sharp, perfect signal will be silenced by the noise (hopefully not completely), but this is the trade-off for having no information about what partial classifiers work for what objects.
I think that it is not proper to do this unless the cost function has been specified from subject matter experts.  It is not a statistical quantity, and often varies with subjects.
... If I remember right, I think I previously had a one sample case work out here (i.e. that the smallest denominator that would yield the observed percentage was still significantly different from the null hypothesis proportion). Most people say "sorry, you can't do anything and that's that"... but there is at least this one thing that can be tried, and once in a while works out, even with percentages rounded to whole numbers.
I mean, take the US, China, and Cuba.  The price of American goods in Cuba will be high, but not because of distance.  The price of Chinese goods in the US will be low, but not because of distance.  The effect of distance that you are estimating is a residualized effect.  If your mode was fit with only data from those three countries, you might find that the effect of distance decreases the price (if you didn't have country dummies).
@antecessor Check out the rpsychi package.
@Peter Flom. No unfortunately not and I don't think a subset would be of any use in this case.
@BenBolker Thanks a lot for the clear answer.
I think there are a number of questions similar to this on the site already ([try this search](http://stats.stackexchange.com/search?q=test+two+[time-series]) eg). Have you read through those & still have a question? If so, could you edit your Q to clarify what you still need to know? Note that, if this Q is *only* about how to get this done in SPSS, it is probably off-topic for CV (you may want to check out our [FAQ](http://stats.stackexchange.com/faq) on that issue).
@Thomas The different answers are $p=ar+b$ and $p=(r-b')/a'$ from the two regressions.  The slopes *always* differ whenever the correlation is not 1 or -1: that's just another manifestation of "regression to the mean."
Any improvements to this answer are welcome.
The numbers and presentation are slightly suspicious: nobody seems to have left in the first year at all, and in educational terms, many teachers leave exactly at the end of an academic year, so the number employed at the end of one year is usually not equal to the number employed at the beginning of the next.