tttjjj/elt_bench huggingface dataset:
You assumed a uniform distribution, so why do a simulation?  An exact answer is straightforward to obtain.  (Indeed, my facetious comment still applies: by assuming a uniform, you know *a priori* that the mean is 50.5, so you don't need to pick *any* numbers at all!)  But if *all* you know is that the numbers are in the range [1,100], you have to plan for the worst case, which will require up to 9000+ draws (when half the values are 1 and half are 100).  First we need to hear from the OP concerning what assumptions are valid to make here.
I typically do something similar with other classification methods, but I use these as weights against each class's f1 score computed on some large unseen test sample.  I can keep updating the test sample and f1 score as new data come in and keep track of drift, rebuilding the models when thresholds or accuracy falls.
@Gary: unfortunately I don't know a (published or online) reference; it's part of the folklore of my field, spelled out in courses but written off as "simple and well-known" in papers.  The proof is easy, though.  Given the tail estimate, the moment estimate follows from integration by parts (i.e. $\\mathbb{E} |X|^p = \\int_0^\\infty p t^{p-1} \\mathbb{P}(|X| > t) dt$) and Stirling's formula. Given the moment estimate, the tail estimate follows by applying Markov's inequality and optimizing over $p$.
Could you expand on how the dummies work for the OP?
+1 Finding all those discrepancies was well done (and reflects unusually dedicated attention to code details).
Sure, but differences will occur for a variety of reasons, including just by random variation (noise makes the individual estimates differ even when their population values may not). One might use fixed effects to account for intercept or slope differences in some particular situations, or simply treat the variation as part of the overall error in other cases. Often the consideration is *a priori*, rather than *post hoc*, and arguably should usually be so.
Thank you for this - your third point in particular as we've found significant standard deviations (eg mean of 6, standard deviation of 3)
I'd suggest you to be more precise about the cross-validation. Do you mean any crossvalidation (as your title implies) or just leave-one-out crossvalidation (also known as U-method crossvalidation) or another specific method of cv.
MikeP; Thanks for your help. There are priors for fuel and spark plugs given in the second paragraph. (Big) I think, P(NS|SPD) =  1*0.001 + 0.1*0.999 is correct but i would like to express it, along the lines of P(NS|SPD and F)P(F) + P(NS|SPC and NF)P(NF) - but correctly.
Are you trying to test whether or not two independent samples come from the same distribution.  If the sequences are assumed to be independent realizations from a given distribution then there are standard parametric and nonparametric two sample tests that could be applicable.  If the data are time series or pairwise associated it changes the picture.  You need to provide a lot more information.
@Drazick `I could '-1' your answer since the argument it is based is unproven - "Let Z=XcosΘ+YsinΘ. Since X and Y are independent of Θ, their conditional joint density given that Θ=θ is the same as their unconditional density`  You should feel absolutely free to do so. Your action will reveal more about your understanding of the issues involved than you might want others to know.
@FrankHarrell Because in many cases your predictions might not be better than random - in which case the usefulness you report is actually nil. Sure, reporting a confidence interval of the usefulness measures (sensitivity and specificity) would be more useful. But testing the difference between two groups is commonplace in clinical literature at least (and in fact there the groups often don't differ) and I saw reviewers asking for it specifically.
The short answer--which is well understood by pollsters and even the news media--is that you cannot tell from this information alone, because the nine million votes already in are almost always *not* a random, or even representative, sample of all ten million votes. Without additional information about how the vote returns may be related to voting behavior, statistics has little or nothing to offer apart from this negative assessment.
I still don't really understand what you are trying to accomplish, but have you looked at correlations?
Why would rescaling the response distribution lead to overfitting? The ordering relationship between any two of the forests outputs will retain the same order after pushing it through a polynomial fit to rescale it, so decision making from the model will not be impacted if you base your decision on quantiles of the output.
Of possible interest (although it focus on genetics): [How does quantile normalization work?](http://stats.stackexchange.com/q/10744/930)
I deleted some previous comments that were no longer relevant. I will try to update this answer with a more general one in the next few days if you are still interested.
This seems more like a statistical modeling question than a specific programming question. I've voted to migrate to [stats.se].
Did you notice that *two* points in the diagram are labeled "$x_2$" and none are labeled "$x_1$"?
I've had quick look at the data, I think you need to start with how to group the data, I guess your interested in seeker/hider behaviour? I guess the question is: do they choose same or different color depending on their role? Your variable no.RED has in this sample no meaning since they are all 5. The questions you want to answer is the best place to start, what you describe is 4 different proportions and that is a fairly small amount of possibilities and is perhaps easiest displayed as bar charts.
I think the second answer to this question http://stats.stackexchange.com/questions/6/the-two-cultures-statistics-vs-machine-learning also answers your question.
How you getting on with this? I have some Fortran code from Gary White which I have working (mostly) wrapped in R: https://github.com/barryrowlingson/telemetr - the 95% CI ellipses given correlation matrices is on the todo list...
I'm not aware of a pre-packaged way of combining MI output in SPSS, but I once followed the guidelines from the link your provided.  It required me doing some coding (or you could hand calculate some things feasibly).  You apply the rules the same for multiple regression as for any other regression:  for each estimated quantity (a coefficient or p-value, for example) let Q represent that quantity of interest (Q_i ranges over 14 values in your case) and then run through the calculation steps in that link.
Thanks for the refs, will take a look
(My first comment was based on misreading your question.) PCA gives you zero covariances; you can standardize the PCs afterwards if you wish. It sounds an odd thing to do, but you can do it.
If one needs symmetry, you can use Mutual Information, which as hinted at, can be defined in terms of KL.
(1) Use of PCA to reduce the dimensionality of the predictor set in this way involves an assumption that the PCs explaining most of the variability in predictors relate most strongly to the response - an assumption which needs to be justified in any particular case. There's no general reason why the response shouldn't be strongly related to the principal component with the _smallest_ eigenvalue and only weakly to the PCs with larger eigenvalues.
When I read the question, I checked ?cph and I knew I already heard about the author... Now I know where I heard about you ;-)
Thanks. I'm new to automated model selection. Could anyone point me to any comprehensive discussions of why manual selection is superior?
The probability density of $Y$ is __not__ independent from $y$!
What do you mean by "independent" Brownian motion?  The terms $Z_t$ and $Z_s$ normally refer to values at "times" $t$ and $s$ within the *same* Brownian motion.  Although (by definition) this has independent Normal *increments,* it is not the case that $Z_t$ and $Z_s$ are independent random variables.
please add context when providing link. Also if advising using some function, please specify from which package it comes from.
@robin - thank you for your comment, take a look at: http://www.inside-r.org/packages/cran/changepoint/docs/cpt.var
Is it your contention that the estimate of the intercept parameter must equal the sample mean? (You should be explicit what you think should happen and why you think it.)
For a familiar (to some) example, the mean result of someone playing blackjack for one hour might be negative \\$25 but with a standard deviation of say $100 (numbers for illustration).  This large coefficient of variation makes it easier for someone to be tricked into thinking they are better than they really are.
+1 thanks for the link, ill check it out.
Can you show your code?

However, the HL test and the GINI test are for different things. HL is a test of goodness of fit of the model, GINI is a test of how much the model improves your prediction of the DV. KS usually stands for Kolmoogorov Smirnov and I am not sure what you are using it for, here.
The customer groups are a 5-level factor either way, you just don't have an interaction w/ it.
What does "scientific proof" mean in this context?
What do you mean by "the independant variable is with log link"? Are you actually talking about logging Y (the **dependent** variable) and doing linear regression, instead of Poisson regression? Or are you talking about logging one of the predictors in a Poisson regression?
A missing contrast does matter. I don't know about the order, try it.
do you have one outcome or multiple outcomes?
Do bear in mind that you can estimate any contrasts, regardless of the coding scheme you use; it's just convenient to use a coding scheme where coefficients correspond to something of primary interest.
@Marcin I wish to use the density metric to evaluate the quality of clustering. And I wish to know, if we could calculate density of each cluster in K Means clustering or not..
Thanks so much. Actually I want to use a graph to show that the log likelihood function cannot be differentiated with respect to k in order to find the maximum likelihood estimator (MLE). How can I express the third graph in terms of parameter k?
@Akavall, I want to rank them by how strong they are cointegrated. As the pairs are very different, have different statistical caracteristics, i was not sure if a DF t-statistic for one pair could be comparable with a t-statistic for another pair.
If $Y = aX + b$, then $E(Y) = a E(X) + b$ and $Var(Y) = a^2 Var(X)$. Does this help?
Similarity of $R^2$ doesn't of itself imply any kind of connection, no.
I haven't been taught about referent groups, nor indicators for referent groups so I don't understand your comment, but it seems like what you're saying is that there is no way for me to reference these interaction terms in a separate model.
In that case you are actually removing the time effect (year). I am not sure how that can be fitted to the anova test as it needs some assumptions (like normality ...). You have a time series data ... so you need to go with the models that are appropriate for time series data.
There is no general requirement of similar group sizes when comparing independent groups by descriptive or inductive methods. If it makes sense from a medical point of view, I can easily use a Kruskal-Wallis test to compare the groups in your examples as long the variances are not too different between groups.
I think it's important to establish what Bayes theorem *means* both intuitively and mathematically. For the latter, a good presentation using Venn diagrams makes clear the relationship between the conditional probabilities and the probability of the intersection. From there the conclusion of the theorem using basic algebra will seem "obvious".
Hi Dave,

It is indeed helpful.

I will build these 23 predictive series based on the time the adverts are shown.

Thank you.
I want to determine the variance of the misclassification rate. If I'm doing a regular N-fold cross validation, then I will a vector of mis-classification rates and then I can just compute the variance. With leave-one-out that vector contains only 100 and 0, so the variance is very large. I'm assuming there is something else for loo.
This question appears to be off-topic because it is about Stata. It should probably be asked on a list devoted to Stata.
I am sorry that I did not correctly guess that your reference to "symbols" meant "$\\mathbb{E}$;" I had to guess which ones needed explanation and apparently I guessed wrong. "Simple linear regression" is not a term conventionally applied to this situation. The solution derived here is indeed an example of a *generalized least squares* estimator, as well as an application of *unbiased linear estimation.* In the same vein, perhaps it would be helpful to know that this is an extremely simple example of a Kriging estimator.
I hear what you are saying. Basically observed frequencies will be my matrix... In simle words!
If you google your own question ("Why do people use the squared error method"), you will be directed [here](http://stats.stackexchange.com/questions/118/why-square-the-difference-instead-of-taking-the-absolute-value-in-standard-devia)
Maybe if you could post some of your own work, someone could point out where you are going astray?  Else the calculation is a fairly standard one found in most textbooks, and people might not want to bother typing it all in here (unless they have a canned answer in their files that they can paste in here easily).  Looks a lot like homework too, and if so, please add the homework tag.
I think the correct formula for R-squared should be
`1 - sum(errors^2)/sum((y.actual-mean(y.actual))^2)`
Please edit this question to explain what you mean by "variant length on rolling" and "uncertainty values."  These do not appear to be standard, well-known terms, so they need clear definitions for your question to be understandable by most people.
One more: `m %*% diag(1/colSums(m))`
Should this be tagged trimmed instead of truncated?
It isn't orthogonal, one model is unlikely to be better than another without being different.  In situations where over-fitting is likely, an exaustive search based model is likely to be unstable, i.e. if you collect a different 500 samples, you are likely to get a different set of features.  LARS on the other hand is liklely to be more stable.  Whether 50 features and 500 samples is likley to result in overfitting depends on the nature of the dataset, but it is certainly possible.  Exhaustive search is liable to pick features that explain the variability peculiar to this sample; LARS less so.
To clarify one of my earlier points, if your residuals are non-normal, and for whatever reason you decide that transformation is called for, it will be the data, not the residuals that you transform.
I've been meaning to provide an answer to this Q, but maybe I should let you since you've already started. LR is clearly the right answer here, but I wonder if you'd expand on this a bit (I suspect the OP isn't very statistically sophisticated). Can you talk a little about the interaction terms she'll need to include, given her data & her goals, & how she'd interpret (eg the signs of) the coefficients that result?
Closely related: http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous.  Also see our threads with the [tag:AIC] and [tag:BIC] tags, as well as threads about [tag:model-selection].
try the ellipse function in car. You have to set the  radius to the square root of the desired quantile of the $\\chi^2_{2}$...
I appreciate your response.  I have added a histogram of the p-values and some descriptive statistics, including the dispersion for p-vals < and > .2.  They are concentrated near zero.  They were generated via permutation testing.  In general, I would expect this to be true. The method I am characterizing basically asks the question, can I differentiate between two subtypes with this network, where there are 249 networks.
But $p$ is a probability and $y2$ is a label (either 0 or 1) correct? Perhaps you need to explain more about what you mean by classifier accuracy in your question.
@Daniel, please revert this to the previous question, and ask the new question separately, linking to this one. Also please see the FAQ and properly format the code, now it has an error in it. The answer to your second question is very simple, but you must figure it out by yourself. This site is for learning, not for getting your job done by somebody else. If somebody does not show inclination to learn (as you do), there is no incentive to help.
In a word, no. The only reason that this might be a problem is if prior theory has argued that they should be correlated, but even then you are contributing to the literature by finding a counter-example
Is this a homework question? If so, it should be flagged as such.
This assumes that the reader has Excel. This is often incorrect for people on many platforms. More interestingly, why do you say that $R^2$ makes no sense for moving average trend lines? $R^2$ as the square of the correlation between data and smooth is perfectly well defined. Whether it is useful or interesting is a different question.
(1) binary inputs aren't an issue with any glm including logistic regression (2) the issue is non-independence of observations (iid) (3) do you really have 30 features or just 1? (seems like 1 to me)
@Filippo OP=Original Poster (http://www.urbandictionary.com/define.php?term=op). In this case, you. You may want to start with the wikipedia entry: http://en.wikipedia.org/wiki/Kernel_density_estimation.
I suspect multicollinearity. The confidence interval of `SEXNULL` is also extremely wide. Could you explain how `SEXNULL` and `ASA` are related? The `R` [package `rms`](http://cran.r-project.org/web/packages/rms/index.html) has a function called `vif` that calculates the variance inflation factors which are used to spot multicollinearity. It works with a `glm`. It may be helpful if you posted the output of `vif(fit)`.
Recall the definition of the expected value for a discrete random variable: $\\mu = \\mathbb E Y = \\sum_{y=0}^\\infty y \\mathbb P(Y = y)$. So for $y = 0$, the term in the expected value is $0 \\cdot (\\pi + (1-\\pi)e^{-\\lambda}) = 0$.
I agree with you. My post does not argue otherwise. What I tried to do was to "predict" when the actual implementation of the MLE _algorithm_ through the attempt to satisfy the usual conditions, will numerically break down under perfect separation, and when it won't. But my post does not discuss the usefulness/meaningfulness of the actual estimates we will obtain, in the case where the MLE algorithm does provide results, and I have wrote this clearly in my post. I see my examples as a warning: "beware -even if perfect separation exists, don't trust the MLE algorithm to tell you so".
If you mean that you have a density function expressed as a linear combination of Epanechnikov densities and you wish to compute its quantiles directly using the inverse CDF, the answer is no in general.  (Conceivably some special configurations of data--equal spacing comes to mind--could be amenable to such an approach.)  You're best off integrating your density using the CDF of the kernels and numerically finding its quantiles.
This is exactly how social scientists think about mixed models (or rather the models they call multilevel). See say [Raudenbush and Bryk's book](http://www.amazon.com/Hierarchical-Linear-Models-Applications-Quantitative/dp/076191904X).
@whuber - The birth ratio in this problem is an invariant. The variant in the problem is the number of births per family. The question is a distraction, it is not part of the problem. <br/> Lateral thinking, is the ability to think creatively, or “outside the box” as it is sometimes referred to in business, to use your inspiration and imagination to solve problems by looking at them from unexpected perspectives. Lateral thinking involves discarding the obvious, leaving behind traditional modes of thought, and throwing away preconceptions. [fyi> I am a principal scientist in Lab]
Outcome is continuous. I wasn't thinking of creating a regression model - are you suggesting that it may help? I want to build the best model possible out of all the variables.
And, for completeness, `summary(lm(decrease~-1+treatment+rowpos,data=OrchardSprays))` does the same thing with a slightly different command.
Thanks for the k-fold cross-validation idea.
The Marshall-Olkin bivariate distribution is not absolutely continuous, this is $P[Y_1=Y_2]>0$, therefore the result you are observing is natural.
Can you paste the first few rows of your data set?
Precicely. `summary.lm` gives the uncentered R-squared for models without intercept and the centered R-squared otherwise, as is appropriate.
I wouldn't *quite* say never. It should be avoided where practicable. Yes, the plot won't be linear unless the proportion barely changes with $x$, or $x$ has a very limited range. It *can't* be straight over the full range of proportions, because the bounding will force the function to curve as you approach the boundary (or, alternatively, it must have a hard kink in it as it hits the bound, but that's a very rare situation indeed), and simultaneously the variation about the line will reduce.
For each scale that you use there is a manual describing how score is calculated (usually mean or sum of corresponding items) Did you made that computations?
By "participate" do you mean put in a bid in the auction?  So if a buyer puts in 6 bids, their value is 6 for that auction?  So you would still need a variable for each buyer, it just will have a bit more information than a dummy.
See also: http://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless .
I have all the data, I can use much more individuals at no extra cost (apart from some effort on my behalf). Surely using more data will yield more reliable results, e.g. less (risk for) bias and narrower confidence intervals. I want to get the best possible result with my resources.
I changed the tag from "truncated-gaussian" to "truncation" because most answers will be potentially useful in situations involving other distributions.
All the approaches I've seen for tf-idf normalize the term count into term frequency *before* the idf weighting.  I'm not sure why this approach is taken instead of yours, but it makes me wonder. Thanks though.  This is a great answer.
Maybe you can start with an old paper: Sparse Bayesian Learning and the Relevance Vector Machine http://jmlr.csail.mit.edu/papers/volume1/tipping01a/tipping01a.pdf
Any thoughts would be appreciated !
I have given presentations on this paradox in which the game is actually played with the audience, with real amounts of money (usually a check to the host institution).  It gets their attention...
Ok thank you @Anony
Your example data shows a mixture of data types: Sex is dichotomous, Age is ordinal, the other 3 are interval (and those being in different units). Doing linear PCA is right for interval data (but you have first to z-standardize those variables, because of the units). It is debatable whether PCA is appropriate for [binary or dichotomous](http://stats.stackexchange.com/q/16331/3277) data. You should not use ordinal data in linear PCA. But the main question with your example data: why _at all_ to do PCA with it; what sense could it make in this case?
Thanks. Why "high correlation" is needed? Any reference for that?
Can you clarify regarding "central tendency"? Are these assumed to be some sort of location-scale family? From what it sounds like, the answer is "no". So, is your "parameter" of interest the mean, median, mode? Something else?
@Arushi, if you include all the dummy variables then you will either see an error (something like a singular x-matrix) or the program will drop the extra dummy variables (and may or may not tell you).  I don't remember which SPSS does.