tttjjj/elt_bench huggingface dataset:
I second this answer. A variant: Randomly permute your red and green labels under the null hypothesis that there is no difference between them, form 2 clusters, and calculate the average silhouette value (http://en.wikipedia.org/wiki/Silhouette_%28clustering%29). Repeat this 10k times and compute the fraction of average silhouette values that are at least as large as the average silhouette value of your actual data -- this is the probability of observing a dataset at least as extreme as your dataset, given that the null hypothesis is true.
+1 In fact, @CroadLangshan mentioning his degree in "hard science" or knowledge of some statistics (or even just know the jargon -- words like base rate, or Bayesian decision theory, or [prosecutor's fallacy](http://en.wikipedia.org/wiki/Prosecutor%27s_fallacy)) when he has not been asked about it explicitly makes it very much more likely that he will be excused and returned to the jury pool room to await the next call.
Not only were some responses recorded twice, but it's also likely some responses were not recorded at all and many more may have been recorded with an error in at least one field.  You should treat this 603/600 discrepancy as a red flag that signals the possibility of pervasive quality problems and therefore not limit your actions to fixing only the errors that were detected: if you care about these data, you should take more comprehensive action to detect and fix the other errors that are likely present.  See http://stats.stackexchange.com/q/11659.
+1, like them undergraduates at OR.
Perhaps you could explain why you find a non-significant interaction parameter surprising or problematic?
You say that you have 8 measurements per individual, but list 6 measurement occasions, can you clarify this?
@Membran Aggregation doesn't have to be meaningful: you're free to combine bins any way you please.  A subtle problem, though, is that *post-facto* aggregation casts the p-values in doubt; the aggregation ought to be independent of the data.
Please do not cross-post your questions.  This one is more suitable for the Math site, because it is purely a question of linear algebra.
This question has been considered before. Here is a link: http://stats.stackexchange.com/questions/51292/bayesian-parameter-estimation-of-a-poisson-process-with-change-no-change-observa Also, here is a preview of a paper by Buck, Pignatiello, and Raz that deals with your question: http://www.tandfonline.com/doi/abs/10.1080/07408179508936756#preview
Please don't cross-post, http://stackoverflow.com/q/9695704/604456. And if the answer on SO doesn't meet your needs it is best to elaborate there than it is to ask the exact same question again here.
Thank you very much. I actually had the wrong idea that NP require $P_0$ and $P_1$ to have densities with respect to lebesgue.
This is a little off-topic, but I would urge you to consider whether you really want to be doing this in the first place. If you are implementing this QR  factorization, e.g., as a project in a course on numerical methods, that is fine and good. For (almost) *any* other reason, you should strongly consider using a standard numerical linear algebra package. For example, `numpy`, $R$ and MATLAB are all using LAPACK under the hood.
@whuber I didn't realize I could access the individual points through `$x` and `$y`. Things make sense now. Unfortunately, my reputation doesn't allow me to +1 your comments, but thanks a lot!
Is there a common name for the test that prop.test() performs?
According to the textbook, the probability of (b) happening is 100 * 1/100 = 1/10000, as suggested by Comp_Warrior. It does make sense now, there are n combinations of ways they can pick the same number
How exactly are you creating forecasts from this model? Are you using smoothing spline estimates of the Schoenfeld residuals to estimate the baseline hazard function? If so, do you do this *before* or *after* eliminating "insignificant" variables? If your goal is prediction, some very prognostic markers may have inference on hazards ratios which are not significant.
FYI, there's an [entire stack exchange site dedicated to quant finance](http://quant.stackexchange.com/).
@whuber thanks for the link, I'll check it out. I'm aware that gradient descent has a lot of limitations and weaknesses. My motivation for this question was to confirm that my understanding of gradient descent and line search is correct.
your intuition is great, but I am looking for a mathematical derivation. Thanks anyway.
I think you meant "Sheldon, Sheldon, Sheldon". :-)
Can you provide a little more information about your situation & setup?  What are RC & HS, for example?
As stated the question doesn't make sense. "So it is recommended in statistics books" can you give an example? I suspect a misunderstanding.
GPS units *require* time information to work, accurate to the nanosecond, and are capable of reporting it.  Could you not just collect the timestamps from the GPS then?
Not to mention robust estimates of location. Trimmed means, for example, usually include more values than the median but fewer than the mean. (They equal the mean and median as limiting cases.) The 25% trimmed mean, or midmean, is easy to explain to users of boxplots as the mean of values inside the box, at least approximately.
The plates are grouped in the center of each rectangle. And coloured based on concentration of bacteria. If we didn't have a contour or mapping plot of any type what would a statistical test be to see if distance from the injection point was important in influencing the results?
@whuber - the differences aren't 0, they are all the same.  Having said that, unless the sample size is very small, I have a hard time believing this isn't due to a data entry error of some sort, e.g., coding "Yes, this is a pre-intervention observation!" or "No, this isn't a pre-intervention observation!" instead of the variable itself.
(1) I don't have the answer (2) you might get a better answer - or even work out the answer - if you explain your choice of performance criteria
@Jason, what you're describing is the [memoryless property](http://en.wikipedia.org/wiki/Memorylessness) and the Poisson does not have that property. In fact, memorylessness _characterizes_ the exponential (continuous case) and geometric (discrete case) distributions, so no other distributions _can_ have that property.
Can you say any more about how exactly you'd determine whether bias or variance is the problem? With a binary outcome the mean and variance are linked so I'm curious how you'd quantify this.
I am not an SPSS expert, but it would almost surely help if you posted your code. It sounds like you are doing something wrong, but I am not sure.
Yes, sometimes the easiest problems are the hardest!
Many thanks Scortchi. I do not know the relation between Q & R. You suggest that I include log(R) as a dependent?  Your "βrlogR term" is not clear to me.
And yes, the tanh function. The Sierra Chart is just a charting software, it does have nothing to do with the NN, I probably shouldn't have mentioned the program at all.
You bring up an interesting point (+1).  Multivariate GLMs certainly exist.  On the other hand, giving `lm` a matrix for a dependent variable should probably be seen more as syntactic sugar, than as the expression of a multivariate model: if it *were* a multivariate (normal) model it'd be the one where the errors are 'spherical', i.e. one where you could have run separate regressions on each element of the dependent variable and gotten the same answer.
+1 For clear, useful advice. But some clarifications are in order, of which the most important goes to the last bullet: **I would like to challenge you to an SD prediction game.** We observe the SD of $n$ iid samples of, say, a Normal distribution. *I* will predict whether the SD is going to be higher or lower after another $100*n$ samples, say. You pay me a dollar if I'm correct, otherwise I pay you a dollar. (With correct play--which I invite you to figure out!--the expectation of this game is positive for me, getting as high as about $.18$ dollar when $n=2$.)
It is simply not appropriate to seek cutoffs on inputs.  Optimum decisions are made using no cutoffs at all, or, if needed before decision time, by making cutoffs on the predicted probabilities.  Utilities (loss/cost) are needed to solve for the optimum cutoff on predicted risk.
I think if anyone is to answer this, they're going to need a lot more information about the data.  Are you trying to determine the distribution of a data set from the raw data?
Thanks, I guess that is exactly what I was looking for.
Homework, in other words?
If you have the mean *of the differences*, and its standard error, yes. If you only have the means of the first and the second groups and their separate standard errors, then no.
(+1) Nice catch!
@mpiktas- thanks for editing my post. Are you suggesting that I need a dataset with more time points or lesser time points? any specific reason why you say for 4 time points, autocorrelation cannot be estimated well?
btw also have a look at the http://en.wikipedia.org/wiki/Benford%27s_law if you are interested in the occurencies of certain numbers.
A problem with PCA is interpretation the final model. Say PC1 and PC2 are significant terms in the model. To explain the relationship between share2 and the original data to gain understanding on it, you need to pick apart the loadings of the variables on PC1 and PC2. This likely won't matter too much is prediction is all Brandon is interested in though.
A more general statement is that in a OLS model $\\ln y = x' \\beta + u$, we have that $\\mathbf E[y \\vert x] = \\exp (x'\\beta ) \\cdot \\mathbf E[\\exp (u)]$. If $u \\sim \\mathcal{N}[0,\\sigma^2]$, then $\\mathbf E[\\exp (u)] = \\exp (\\frac{1}{2} \\sigma^2)$, which is where the $e^{0.5}$ comes from. To estimate, just replace with $\\sigma^2$ with its consistent estimate $s^2$. If the errors are not normal, though still *iid*, you can use the average of the exponentiated residuals to estimate $\\mathbf E[\\exp (u)]$.
This seems only to push the question back one inconsequential step: *why*, then, do people use odds in betting rather than (say) differences of chances? Moreover, you need to go one step further: once the use of odds to express chances is justified, why would one use odds *ratios* to compare chances rather than, say, odds differences or something else?
There is a paired permutation test. Google it to get details.
Auto correlation has a precise definition. As far as I know, "trend" does not. So, the answer will depend on how you define trend.
I guess I have seen both in articles. Some articles claim the inverse and some claim the standard prediction. Is there a difference between the claims -- is one stronger than the other?
Thanks. I'm aware of the progress that has been made in data mining and machine algorithms. For those, there is in fact an explosion of resources. What I meant was computational issues in statistics. Something as simple as calculating variance is a nontrivial problem (see for instance Welford's algorithm.)
Do you want a [contingency table](http://www.psychstat.missouristate.edu/introbook/sbk28m.htm)?   You can try one out [here](http://vassarstats.net/newcs.html).
An Exponential variate is [Gamma(1)](http://en.wikipedia.org/wiki/Gamma_distribution#Summation); Gamma variates enjoy nice relationships under addition.
Sharpie, I wonder if it's time to accept a "best answer"or at least explain why you are not satisfied with the answers on offer so far.
"... so that for a Gaussian $X$ you can find that $E(X^2)=\\sigma^2+\\mu^2$."  Is it absolutely necessary that $X$ by Gaussian for this relationship to hold?
@user8648: I just had a glance at that paper, 1st thought: empirical error looks very much like MAE. 2nd thought: if you have enough cases, also inefficient measures will eventually work. 3rd thought: Does it actually work? They validate with % error rate and a test set of 2000 cases.  The binomial 95% confidence interval for 4.5% errors observed over 2000 test cases is ca. $\\pm$ 0.9%. That means they cannot distinguish *all* presented models are actually quite central within that window, so based on the random uncertainty of their testing results, they cannot distinguish the models.
I'm sorry, you're not going to pull me into one of these endless debates.  The answers to your question are obvious.
Adhesh Josh, the null hypothesis can be any fixed value specified beforehand.  That's another feature of the CI over straight NHST.  It's very easy to use when you want to test against a hypothetical value other than 0.
As BWilliams writes, doing these types of problem in Excel is like showing up to the Tour de France with a tricycle.

Try R or - if that makes life a bit difficult - give Python a go (with packages like scipy / numpy etc).

I suspect R is your best bet.
Perhaps, asking on Stackoverflow may get better answers.
Does [this](http://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Test_of_independence) help?
Expected value is calculated as Sigma x.p(x), summing over each of the m elements in set T. Each common item you find adds 1 to the common item total, so in our case, x = 1 and p(x) is m / n.
In general, I would (and have!) agreed that people often bin or discretize their data unnecessarily. However, it might pretty reasonable here. I've heard it claimed that the economic crisis in 2008 made it much harder to get loans. If that is the OP's hypothesis, splitting on either side of 2008 might be defensible, though something like a regression discontinuity design might be more powerful.
@whuber: Yes, you're right.  I initially wrote it up as the expectation is either zero or doesn't exist, but then he edited his question and it was clear that it did exist.  Macro has a nice explicit answer.
Oh. I have one response with more than one predictors.
The explanation is right, but it's purely qualitative: the precise form of the multiplicative factor is still a mystery.  The -1/2 power simply appears magically.  Thus, at some level, you have to do the same thing that Calculus does: find the rate of change of the square root function.
Thanks, I'm glad you found my post helpful. If you require shorter code I can edit it down for you or else you can try the following (after importing the data). Call: acf(inventories), run the bart.error loop, then call lines(2*bart.error[1:15]). This will add the Bartlett approximations to the acf() plot. I hope this is a clear enough explanation. There should be enough info there to be able to tweak it around to the way you want it.
@amoeba That's an excellent point. I will amend my comment.
@IrishStat Thank you so much sir, I have tried to post the result in the form of a table however they have become in text format and is not readable. I shall send the same through e mail Could you please help My mail Id is drsurajs@yahoo.com Looking forward for your help and suggestions. Thanking you with gratitude - Suraj
This won't do, because mutual correlations can (and often do) mask or even completely alter the apparent relationships. These issues are extensively discussed on this site: look at threads concerning *multiple regression.*
It's not because something is "lost" but because MDS is not a "forcasting" method to be able to find coordinates for _new_ points on an _old_ map. Consider using PCA or Correspondense analysis which can do it.
If this is for some subject, could you add the `self-study` tag please?
http://pic.dhe.ibm.com/infocenter/spssstat/v20r0m0/topic/com.ibm.spss.statistics.help/idh_fact_sco.htm
See (your own post?) [here](http://diffuseprior.wordpress.com/2013/08/18/endogenous-spatial-lags-for-the-linear-regression-model).
Good point! If I recall correctly, the general advice is just to include as much information as possible in the imputation process. That may have a limit, of course. For example, I have no idea what happens to the imputation process if we were to add (say) 1000 generated random normal variables, unrelated to everything else.
How do you propose using numerical integration, given non-numerical  $a, b, c$ and $d$?
@Cyan Appreciate your help! Actually I'm dealing with multivariate Gaussian however chose to use bivariate and eliminate matrices to make it simple to represent and understand. Looking forward to your result.
This is not a place for hiring tutors. Please read the [FAQ](http://stats.stackexchange.com/faq).
Would you please clarify the distinction you are making between "linear regression" and "least square"? (To most people, the default method of linear regression is that of ordinary least squares.) And, if you would like this answer to have any real informative content, please elaborate on what your "several measures" are and explain what you mean by "masking" and "swamping."
This is impressing! Thanks a lot, whuber!!!
Cool question; welcome to CV!
is this homework? if it is please tag as such. Please let us know what you have tried as well.
+1 for explaining *why* we can't back transform the betas.
That first paragraph is a parody of the classical approach to hypothesis testing.
Actually, the opposite. We have more information about a random variable when it's entropy is smaller. Minimizing the entropy of the distribution of the residuals from the regression is like maximizing homoscedasticity.
In fact, if $X$ and $Y$ are independent, then $\\mathrm{pmi}(x,y)$ is constant, regardless of the marginal distributions. So there's a whole class of distributions $p(x,y)$ for which $\\mathrm{pmi}(x,y)$ obtains its maximum value for every $x$ and $y$.
There is also a series of articles in the *International Journal of Epidemiology* in response to Robinson's original article (see the [related articles](http://ije.oxfordjournals.org/content/38/2/337.extract)).
Thanks, I had not run across either of these; the latter article seems particularly interesting.
I think you're onto something but a goodness of fit chi-square comparing the actual counts in each cell against an expected count of equal cells would be what you'd want.  Using a contingency test would NOT test if there was uniform distribution among your cells, only if row depended on column.
There would seem to be no issue of "mathematical" correctness: Euclidean distance is a *bona fide* distance. Aren't you rather concerned about whether using it is *appropriate* for your analysis? If so, this seems to be a question that is more about your objectives and the subject matter than it is about statistics. For it to be answerable I believe you would need to tell us more about your objectives and the data you have.
(+1) Here is a **fun** nitpick: One needs the extra assumption that $u(a)$ is continuous at $a$ to be able to conclude $F_{Y_n}(a) \\to u(a)$, so, in particular, the first inequality ($a \\geq 0$) also needs to be strict ($a > 0$), like the second one. A canonical example is to take $X_i$ iid uniform on $\\{-1,+1\\}$ and show that $F_{Y_n}(0) \\to 1/2$. Showing this for the subsequence of *odd* $n$ is easiest (just symmetry), but it's not much harder to show it holds also for the subsequence of even $n$.
Have you considered making the lines semi-transparent by changing the alpha that's being used to plot them?
Moved the "1-" inside the square root which I believe is the correct relationship
@juampa Yes, how would I be able to do that?
You mentioned ROC AUC which requires probabilistic outputs (or something that is a monotonic function of same).  The best predictive methods have probabilities under the hood.
Do you really seek a common transformation of *both* the independent and dependent variable? This is so extremely unusual that it would be interesting and useful to know why.
It is difficult to comment here, so I will edit the question.
Hm, I see. Why not try logistic regression with elastic net penalty than? You can cut the regularization path at the regularization parameters where you have exactly 15 predictors. The $L_2$ regularization takes care of the correlations and the $L_1$ regularization will select the fifteen predictors. The reason I'm skeptical with pipelines is that often it is better to solve the problem directly. 200k and 1000 predictors should be feasible computationally.
The gamma function is not the gamma distribution. See http://en.wikipedia.org/wiki/Gamma_function: "the gamma function ... is an extension of the factorial function, with its argument shifted down by 1"
For a given patient with disease, this gives the percentage of normal patients with weight less than or equal to it, which is simply my understanding of what you're asking for. If you mean something else, could you make your request more precise? Said another way - if you mean something else, could you define quantile so I can figure out what you mean? Even if I did understand correctly, it could reasonably be argued that I should average "<" and "<=" results when approximating the population percentiles, though in the example data it makes no difference (there being no ties across samples).
@guy Yes, that could be - and then $\\bar X$ would be the _expected value_. But then, using the sample-mean notation to denote $E(X)$ is just confusing practice.
That depends on whether you care about overall Type I error rates being above some particular level. I think people tend to worry more often than strictly necessary, but perhaps more important than my opinion is the opinion of your intended audience.
@williamyarberry Register here and on maths with the same OpenID to recover the ownership of your question (if you think it is worth it, of course ;-) ). You can do it by clicking "log in" at the top bar.
I had thought about the geometric mean, but I'd missed the analogy to my case. This of course begs the question of when the geometric mean is appropriate (which I haven't yet settled in my mind), and maybe the messier question of when "something in-between" is appropriate. Not to mention the possibility of applying some other other concave monotonic transformation. Is there any good resource to maybe help build intuition on when such means might be useful? This "sqrt mean" is less sensitive to outliers than the arithmetic mean... but is that all?
Using `ci.type="ma"` also works with the `Acf` function from the `forecast` package.
It's possible to construct infinitely many distributions set up in ways that have parameters that are somewhat akin to or are functionally related to correlations. The most popular would be the family of [elliptical distributions](http://en.wikipedia.org/wiki/Elliptical_distribution), (which includes the multivariate t and the multivariate normal), which has a $\\Sigma$ matrix representing the dependence structure.
In the help page for rtnorm it says mean and sd are for the untruncated distribution.
(+1) Nice follow-up.
Correct, there are no other variables at play here besides what I've included in the original post. I'll read the Oxford paper and see if that clears things up for me.
@ccapizzano: Check out the `party` package in **R**--that may be more what you're looking for...
The general answer is a resounding, emphatic *no* for reasons Stephan Kolassa has outlined (but perhaps not forcefully enough). Unless you have *theoretical* reasons to suppose that the spline ought to extrapolate properly--and that seems unlikely, given that splines are general-purpose and therefore arbitrary--then you should expect the spline to fail, perhaps dramatically, outside the calibration region.
@ttnphns I think they differ only in how to initialize centroids. Or something  else?
Please go back to the original question and then re-post the correct problem as a new one.  The original problem got good answers and may be helpful for other people.  As is, the answers are nonsensical
In case anyone is unfamiliar with this, OP means [this](http://en.wikipedia.org/wiki/Rule_of_three_(statistics)) "Rule of Three".
@John, your answer would be improved with a reference or two, and possibly a formula or two. On the edge of my seat to +1 that answer here.
@Matthew I think the specification for that case is "inconclusive".
The reasoning might become clearer if you made this problem *larger*.  If there were $7$ men and $3$ were feuding, then how would you go about selecting a committee to *exclude* from the valid possibilities?