tttjjj/elt_bench huggingface dataset:
@luckyi for simpler algorithms like KNN you could probably create test cases in a straightforward way "by hand". The other suggestion for more complicated scenarios would be to try and replicate previous anlyses (either from intro texts/blogs or academic articles introducing the model). Of course this wouldb't apply when building your own unique algorithms
@ Dmitrij Celov: it is your answer, not mine, but the wiki link you provide has a nice, unambiguous formulation [emphasize mine]: "the Phillips–Perron test makes a non-parametric *correction* to the t-test statistic. The test is robust with respect to *unspecified autocorrelation and heteroscedasticity in the disturbance process of the test equation.*" Here it is clear that the functional form of the residuals is assumed Gaussian, yet allowing for varying (but stationnary) auto-correlation and variance.
+1 The point about good experiments needing more than just unbiased estimates seems to get to the core of the controversy.
+1 and let me link to an older response of yours here, http://stats.stackexchange.com/questions/6005/an-r-function-for-performing-searches, because this particular question looks "closely related".
Does the inverse of parameter $\\nu$ ('degrees of freedom') not satisfy your desire for a scale parameter?
This question isn't quite clear. Are you interested in estimating the exponential *rate parameter* $\\lambda$ and incorporating the final *censored* observation in that estimate?
Thank you very much for weighing in! Your perspective certainly does put a very direct motivation for cross-validation in causal models I had never cogently formulated myself. IMO your even selling yourself a bit short by using the label of over-fitting. For instance, in the initial exploratory set I may look at model fit between equations using an independent variable on the initial scale versus the log scale. I decide the model with log scale fits better, and then use that in the hold out model. This wouldn't be typically considered over-fitting (choosing between one or the other), cont...
@AlfredM. that is a fair characterization of that book
@FrankHarrell - isnt that the whole idea of dropping insignificant terms?  You trade increased bias for decreased variance.
This is not a "derivation" in any significant sense, because it starts with the result. A *meaningful* derivation might begin with the construction of the Poisson as a [limit of Binomial$(\\lambda/n, n)$ distributions](http://en.wikipedia.org/wiki/Poisson_distribution#Derivation_of_Poisson_distribution_.E2.80.94_The_law_of_rare_events) as $n$ grows large. Because the PGFs of these distributions are $\\left(1 + \\frac{\\lambda}{n}(s-1)\\right)^n$, their limit as $n\\to\\infty$ is $e^{\\lambda(s-1)} = e^{-\\lambda}e^{\\lambda s},$ QED. (Use of characteristic functions makes this argument rigorous.)
@Chl Your edit cleared up the question and answered all of my questions.  Without that help I would have voted to close.  Instead I think I can now give a helpful opinion.
thanks. Perhaps these routines are also available in R?
@Glen_b yes you are right since the correlation will be the same even if the means of $x$ and $y$ are different.  What do you suggest instead? Mean square error?
Adjusted R^2 is not to be used for this problem. From the values I calculated, I have: for X1: R2 = 8%, MSE=5900; for X2: R2 = 80.1%, MSE = 1880, for X1X2: R2=80.6%, MSE=1940. Which model do you feel is best? It appears to me as if it's a clear race between X2 and X1X2(combined), but which?
These are examples of procedures used in meta-analysis when the individual studies only provide p-values or the data cannot be combined but each study has a p-value calculated.  Also Fisher's combination method and inverse normal are ways to construct stopping rules in adaptive designs.
Although this particular point is not relevant to the general question you ask, it might be worth mentioning that applying any formal test of normality to regression residuals is usually not a good idea.  You should instead be looking at appropriate diagnostic plots to find out *how* the residuals depart from normality and evaluating them based on how sensitive your formal testing will be to departures from normality. Many tests in the least squares context are rather *insensitive* to departures, especially when the residual distribution is approximately symmetric.
Why not accept-reject based off a Poisson proposal?
+1.  Is there anything published on how bootstrap performs on problems like this?  (That final "8" in "0.4733268" is of course colossally silly: this is where the noise totally overwhelms the signal.)  I wonder if anyone's used Bayesian methods in which the prior on $\\lambda$ puts all the weight on rational numbers, and gives more weight to those with small denominators?  That way things like $0.5$ are automatically favored over things like $0.4733268$, and in physical data there seem to be reasons for doing things that way.
Ampleforth, I am not sure that is the case.  I think you can test all sorts of data with the Chi Square test.
You appear to have a census of the person's number of publications. Why do you wish to use statistics?
You might be able to ask a better question.  You are asking yes/no is it in the goal.  That is a binary question, and doesn't pre-stage you for the next step.  You can ask questions so that the answer for step one also answers step 2.  What is the location that it crosses the plane of the goal?  If your KF gives you a best estimate of that then not only do you know IF it is a goal, but you know where you should put your goalie.
I didn't get the analytical chemistry example, but I do want to do something of the sort. Where can I read more about an approach like this? By the way I added more info about my problem, maybe it helps to understand my issue.
I'm not sure I fully understand what you're suggesting. Would you mind elaborating a bit? Thanks!
Open an account with dropbox.com and post the data there and then the link here.  Post any output that you have in Excel or text file....actuals, your causals fit and your model.
Nothing changes when you add polynomial terms.  OLS can't distinguish between x^2 and some other variable that takes on the same values as x, squared.
Ah I see. 55.5 is 55.5% That is 55.5 out of 0 to 100. 60 is 60 out of 11.111 to 100. The former is a common enough way of thinking about proportions that just about every numerate human being on the planet does so (thinks in percentages) several times a day. The later exists only for this one example. That is how.
Answer to first one is yes. I ovelooked the way i wrote that, as the matrices in question would not have matching dimensions for multiplication. 

For the second part i did not exclude the diagonal elements but the correlation values here would be 1 as for the diagonals we are just multiplying the standard deviations of the same rv with 1. 

also by full choleski, did you just mean computing the upper and lower triangle matrices which mutliply to give the original matrix or is there any other steps i am missing ?
Hi sir, thank you for your answer, I have a very long time series (about 10000 integere values). Im am not sure, if I can attach the file here.  Which method do you find better to find anomalies? FFT, Wavlet transform or moving average- Is there any difference between anmolay and outlier in time series?
duh, I didn't read the fine manual. Just do `corrs <- cor(t(A),B)`. Look at the help for `cor`.
"*Which one is the sample size used to calculate degrees of freedom?*" -- what is the model being fitted?
@StasK If only reading was as easy as using Stata. Excellent point, and fixed.
an enlightened customer said "try proving anything without them..."  :)
No need to apologize! I have flagged the qn to the moderator for migrating to stats.SE.
Just in case anyone else should misread the above as I did at first.  The 20:1 in the question is the ratio of negative to positive observations.  The 15:1 in Frank Harrell's answer is something else: the ratio of positive observations to candidate independent variables.
Is it a linear model?
Knowing the problem you're trying to solve makes anything easy.
BTW please note that `EMMEANS` subcommand displays, for marginal means, _estimated_ means, not _observed_ ones. When there are more than on factor in the model it will make difference.
My main doubt about making fixed-effects at cluster level is that in some simulated samples it can happen that we have not selected some of the initial clusters, so that the related cluster-specific intercepts cannot be identified. If you have a look at the code I posted, it should not be a problem from a "mechanical" point of view (at each iteration we can fit a different FE model with just the sampled clusters' intercepts). I was wondering whether there is a "statistical" issue in all of this
What you call "the standard upper confidence interval, lower confidence interval, mean, and data range-showing box plots" are not standard box plots at all. The standard box plot shows median, quartiles, minimum and maximum and often other details.
+1: Thank you for illustrating the application of GLS. In your example, to match the code in the question, the argument `data=your.data.,` should be removed.  It's also a good idea to use orthogonal polynomials rather than direct computation of the square of `t`.
What is your data, and what do you want to do with the anonymized data?
My answer (which I deleted) presupposed that for each gene you had multiple pairs of values so that you could do as many t tests as you have genes to compare s1 with s2. Then you can get an adjusted p-value by say a bootstrap adjustment for multiplicity or a Bonferroni bound or other multiplicity adjustment.  But it appears that you only have one pair per gene.  I don't think there is any test that you can apply with one pair per gene.
This interpretation is sort of on the right track, at least at the beginning. I'm not sure who Gaines is or how it relates to this problem. But, see my hint, which provides you the answer. If needed, I can expand on it a bit. I would *definitely* not equate asymptotic relative efficiency to resources used, as you have tried to do in your last paragraph.
You may also look at http://en.wikipedia.org/wiki/MaxDiff
60 year-old dudes read this forum too....
Can you provide some details about this article, and explain why you recommend it?
Cluster membership would reflect the factor, yes. So if you have a factor that is happiness, and your interpretation of the clusters is that one group is the "happy" group, then yes the number of people in that class would reflect the number of "happy people". That being said, you need to be careful in your interpretation of classes. The classes may or may not be "real" in truth. So the happy people may be the happiest in your data, but that doesn't mean that they are truly happy people or that happy people even exist. You may benefit from reading up on K-means if you think this is an issue.
@Glen_b Than `C++` tag should be added in question. I read "an alternative implementation".
For the record, Wimpey's solution in the comments above is incorrect and tests something completely different to what he/she originally asked.
I use Emacs + ESS so I guess I would not be of much help here. However, for your information, there will be a [special issue](http://www.jstatsoft.org/misc/CFP-JSS_SV-GUIs_in_R.pdf) on R GUIs (expected for mid 2011).
Welcome @Ryan You could improve your answer & help the OP (and others) by telling him/her which ones he can use? Perhaps include a short example illustrating how?
here is a 5-part video series: http://vancouverdata.blogspot.com/2010/11/text-analytics-with-rapidminer-loading.html
Welcome to the site, @Wendan Zhong. It's not clear to me if you are asking a question that is primarily about the underlying statistical issues, or an R how-to coding question. Note, if it's the latter it would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/). If your question is mostly about statistics, please edit to clarify; if it's mostly about coding, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
I agree that SD vs. SE is potentially a pretty silly critique (absent any other justification). A good way to plot the original data is to make the lines very thin and semi-transparent, this allows one to see the distribution of the raw trajectories. Can you give an example of what you mean by "individual differences"? Is this a time-varying covariate - or are you just talking about varying individual growth trajectories?
Have you worked out any explicit examples for small $n$, including $n=1$ and $n=2$?  Drawing graphs of this process can be helpful, too.
Thanks for your answer. As for the last question, that's because I want to "learn" the pattern/knowledge from last year's data, and then use the data from first half of the year to predict the sales of this year's end. Simple concatenation may weaken the connection.
@user570593 I see. In that wikipedia page, the random variables are indicator functions of k=1 and c=1. That's the confusion.
Can you point me, which of the dataset there relates to network (eg ethernet, LAN, WAN) topology? I don't see anything there.
The first question is probably answered at http://stats.stackexchange.com/q/10947 .
It's very easy to highlight a subset of points in R, by using points() on a subset of the data with different settings for point character or colour from the original plot you are adding the points to.  So if you have a way of working out which are the "non-dominated points" (and sorry, I have no idea what this is) it is easy to highlight them.
If you mean the *Maxwell-Boltmann* distribution, then the [Wikipedia article](http://en.wikipedia.org/wiki/Maxwell%E2%80%93Boltzmann_distribution) provides a simple method where it points out its square is the sum of three iid standard Normal variates: so just generate three such variates (using, say, the Box-Muller method) and compute the root of the sum of their squares. If you mean something more general than this (Boltzmann distributions [run a wide gamut](http://en.wikipedia.org/wiki/Boltzmann_distribution)) then please include the particulars in your question.
@Nixuz: as I said in the answer, I usually use PCA, mostly because I know how to do it. I like the fact that it treats all variables equally, which is a good start in many situations. I use linear regression only when there is a clear response variable.
It strongly depends on want do you want to say. If you want to report only the difference - the "t" and p-value may be sufficient. If you want to emphasize confidence - add CI. Either way I recommend you to read the "author guidelines" of journal where you want to publish your work.
Still it looks promising.
-1 to the question, since it has become clear that what was sought was for someone to do the work rather than to teach the method(s).  +1 to the answer, though we don't want to "rub it in" that someone's question is a basic one.  Those are welcome here, just as the more advanced ones, are, right?
Could you indicate what size this matrix has and what form of PCA you need to do? It is possible that standard software can perform the work.  (Two possibilities immediately come to mind, and I am sure there are others: one is to base the computations not on the design matrix $X$ but rather on the possibly much smaller $X'X$; another is to subsample $X$.)
Factor it: because it is symmetric positive-definite, you can write it as $X^TX = UU^T$ for an invertible $p+1$ by $p+1$ matrix $U$.
I don't have time to give a complete answer, but here are some thoughts: You can make confidence intervals for each individual entry using the usual $\\hat{\\mu} \\pm 1.96 \\hat{\\sigma}/\\sqrt{n}$ intervals. For joint confidence intervals, there are many options: you could use the [fisher information matrix](http://en.wikipedia.org/wiki/Fisher_information#Multivariate_normal_distribution), for one thing. There is a classic paper **Stein, C. (1962) Conﬁdence sets for the mean of a multivariate normal distribution (with
discussion). JRSS-B, 24, 265-296** that discusses exactly this subject.
Would you mind expanding your last points: How do this dataset help teaching statistics? (AFAICT, the iris dataset only has one categorical variable, namely iris class.)
I'm not sure how this constitutes a helpful answer beyond what was already written here. How about you expand more on kurtosis and peakedness of the curve?
@Jimj: As Jeremy Miles said, and as I mentioned in your previous question, pairing influences your expected t-ratio (and hence power) by a factor of $1/\\sqrt{1-r}$, with r being the correlation. If $r$ is close to 1, you take a huge hit to power by ignoring pairing. If $r$ is close to 0, you don't lose much at all. So it's hard to make the kind of general statements that you want. That said, I would guess that typically the proportional loss in data from the incomplete pairings would have to be quite severe indeed before this outweighed the power benefit of incorporating the correlation
The rule of thumb there is highly useful. Thanks for that.
I think you forgot to add the Grade variable as IV
Note eta-squared can be obtained also by `summary(lm(y ~ x, xy))$r.sq`so this question is equivalent of asking how to calculate r-squared by hand
Hmm, not sure if I am in the position to collect more data. Would AUC from the ROC curve be a measure of "robustness" of the resulting classifier output?
What do you mean by assigning an uncertainty? Do you want something like a standard error or confidence interval for each bin?
@whuber, Partly I want to estimate the number of students admitted out of a specific pool of applicants. Also of more interest though I may not have phrased it correctly, is it possible to assign a probability on the average success rate estimate from the logistic model. For example if the estimate success for a case is 85%, is there any way to find out the probability that this could be say less than 60% instead? Does it make sense now? I thought maybe I could use the standard error on the prediction to answer that, would that be correct?
You start out well with a correct distinction between estimation and prediction, but then the last two-thirds of the reply appears to confound prediction with estimation once again.  Introducing the example of gender gets more confusing still, because it's not related to the initial distinction (in fact, it's nonsensical, because underlying it is a shift of statistical model between the estimation and prediction step).
Here is a reference for estimating the parameters of the Gompertz model, see http://www.weibull.com/RelGrowthWeb/Parameter_Estimation_for_the_Gompertz_Models_Using_Least_Squares_in_Nonlinear_Regression.htm
Could well be separation as @tiantianchen suggests: see [here](http://stats.stackexchange.com/questions/11109/how-to-deal-with-perfect-separation-in-logistic-regression/68917) for a discussion of ways to deal with it.
Or, the alternative: _If_ you use clustering based on euclidean measure or you use k-means, multiply each variable by the sq. root of its weight. This multipication should be, of course, done _after_ any pre-processing (such as standardization) you might want to do before clustering.
@charles The sampling method imposes no limits whatsoever on the *analysis,* but it places severe restrictions on what the analysis *means.* With convenience samples it is next to impossible to validly infer anything about the population (unless the sample is almost as large as the population itself). Things like p-values and other probability-related outputs of standard statistical analyses are utterly without meaning unless one can (at least in principle) compute the prior chance that the sample one actually obtained would have arisen.
Hmm... isn't MAXENT supposed to be used to find out what our priors ought to be in the first place? I'm fuzzy on this part, sorry. So my priors depend specifically on the hypotheses I have considered and don't include cases for all the hypotheses I haven't?
Welcome to the site, @nikos. I don't think your question is answerable in its current form. Can you tell us more about your data, your situation, & your goals?
Thank you for your contribution. Welcome to our site!
Some points: (1) Those zeros are probably nondetects; there's *always* some quantity of everything in any sample. Knowing the limits of detection (and how they were determined) can be crucial. This is a *left censoring* problem. (2) Chemical concentrations frequently are best analyzed in terms of their logarithms (which have important interpretations in chemical kinetics). (3) Many people have successfully used PCA (on the logs) for addressing such problems.
I have edited your title so it matches your question  - which seems to be about observations, not variables.
Fractional L_p norms only hide the problem. I believe the result then tends towards something like the minimum attribute difference, which for a high number of dimensions becomes just as meaningless in practise. It only solves the problem of the numbers getting larger and larger.
Dimension reduction works in some cases, but consider the case when it doesn't get you any further. What then? Plus, dimension reduction essentially is "640k dimensions ought to be enough for anybody". Text is commonly in the 10^5 range. What about video?
Even if the same data (response variable values) are used in two models removing covariates could change coefficients simply because a covariate removed is correlated with one that remains.  I believe that this can be avoided if the covariates can be constructed to be orthogonal such as with orthogonal polynomials.
I believe this issue was addressed here: http://stats.stackexchange.com/questions/4354/distributions-other-than-the-normal-where-mean-and-variance-are-independent
Not really so, sources of heterogeneity are many: differences in parameters (random parameters for instance), variables (usual regression thing), residuals (idiosyncratic part that may belong to different distributions, or distribution parameters may be different, heteroscedasticity belongs here, thus it is a separate case of heterogeneity), differences in functional form. So I would leave just the more particular term - heteroscedasticity.
Just a comment - when doing hypothesis testing you don't set type-I error rates after seeing what the p-value is. For example, suppose you decide a priori that you want to control the type-I error rate at 5 percent and you end up with a p-value of 0.00294. You don't get to then go ahead and set the alpha level at 0.01 since you were going to accept anyway for all p-values less than 0.05. In effect - even though you received a low p-value of 0.00294, the type I error rate is still 0.05. Not sure if this was what you were trying to say, but I thought I'd write about this common fallacy.
@Alexis All right, I've changed the title
Simple question, but why should the null hypothesis be rejected? What are you comparing the result of 0.015625 to conclude that the nullH should be rejected?
I see. I think this would be clearer if you did a double-y plot. One of the axes would be S&P 500, the other would be counts per day. You could test if one is a good predictor of the other using Granger causality. Testing for correlation between time-series can be done using cross-correlation, but it leads to spurious correlations that don't mean much.
@NickCox Indeed: [Thirteen Ways to Look at the Correlation Coefficient](http://www.jstor.org/discover/10.2307/2685263?uid=3739808&uid=2&uid=4&uid=3739256&sid=21102313286497)
@Atticus29 Yes, its in **car** - my mistake. However, it is not based on a strict linear model - you can use `leveneTest(y ~ as.factor(group), data= datafile)` for a test of difference in variance between groups, and if you use the option `center="median" it is more robust to non-normality. Strictly, I think its called *Brown–Forsythe test* if based on the median.
that looks like a really promising start :)