tttjjj/elt_bench huggingface dataset:
+1 People tend to think intuitively and well about actual numbers, rather than probabilities, so this method of converting the information into (hypothetical) numbers often helps with Bayesian problems.
Hi, have you already solved this problem? Thanks
In that last paragraph aren't you effectively arguing against any use of p-values?  Why, then, use a hypothesis test at all?  Why not just estimate the effect sizes and be done with it?
Also, since OP is interested in prediction, he is going to want to look at posts on the re-transformation problem.
Do you mean to condition $X_t$ on $Y_t$ as well ($\\xi_\\alpha(X_t \\mid Z_t)$ versus $\\xi_\\alpha(X_t \\mid Y_t, Z_t)$, where $\\xi_\\alpha$ denotes the $\\alpha$-th quantile)? That makes the computed conditional quantiles endogenous.
I lost you at the mention of lm().  Up to that point the question appears to concern identifying the die or dice attaining the maximum value within a collection of 20.  After that, the references to lm() and p-values indicate you want to perform some kind of regression, which is a completely different question requiring a context you haven't described.  Regression of what against what?  What do you *really* want to "test" about your simulation?
@ttnphns, ones used in spatial analysis are the nearest neighbor test (aka Clark and Evans test), or Ripley's K. See the R library [spatstat](http://www.spatstat.org/spatstat/) or the [CrimeStat documentation](http://www.icpsr.umich.edu/CrimeStat/download.html) for examples. Another possibility based on simulation are "scan" tests, but these aren't based on average distances.
If you look at whubers comment, his idea can lend credibility to yours.  If they are 10 points each then maybe this is an artifact of rounding policies.
Further context, the reason why my descriptors are of varying dimensionality is that they are edge-based directional features of different kinds, as opposed to the standard d-dimensional SIFT local descriptors ubiquitously used in image classification.
Re your first comment: collinearity suggests some of the explanatory variables (IVs) may be redundant, but this is not *necessarily* the case. What matters are the relationships among the IVs and the dependent variable (DV). It's possible for one of the IVs to be heavily dependent on the other IVs, yet contain uniquely useful information related to the DV. This is a critical concept: no amount of analysis of relationships among the IVs *alone* is going to tell you which variables best explain the DV. Lack of collinearity--a property *solely* of the IVs--doesn't reveal anything about the DV.
I think this is a perfectly reasonable question on computation statistics.
How are you relating the VIF to $5\\sigma^2$?  (BTW, you can mark up the math with $\\TeX$ by enclosing it in dollar signs \\$.)
Could it be as simple as the fact that tanh becomes non-linear as it starts to flatten out approaching the value 1?
May I know how to test the homogeneity of variance? Is it by Levene's test on the residual by each predicted variable? I am using spss, but the linear mixed model does not contain this test. Many thanks!
Can you cite some examples-docs of the approach you are mentioning?
Thank you very much. That's exactly what I was trying to do.
mod-tip: post the second part as a comment to the Dirk's answer.
You can do whatever you want; the bigger question is, can you explain what you've done to whomever you will be presenting your results for them to develop their policies.
+1 for the idea of splines.  One has a lot of flexibility in their use: in my experience the model tends to be less sensitive to the positions of the knots and numbers of splines than it would be to the choices of bins.  It allows for more continuous transitions (by construction).  As @cardinal rightly points out, though, the fundamental question is still out there: what guidance is there concerning the numbers of splines and locations of the knots?  To what extent might selecting them *post hoc* affect p-values?
Actually, `P(y | grassland)` would be `exp(-2.8) / (1 + exp(-2.8))` A point in the forest would be `exp(-2.8 - 3) / (1 + exp(-2.8  - 3))`

However, you could estimate those probabilities without doing a logistic regression. What would be more interesting would be to look at `exp(-3)`, or `1/exp(-3)`. This says that an animal is 20x more likely on grassland than forest, and if the p-value is <0.05, then that difference is significant.
I don't get it: by default, `sample` obtains an equiprobable sample. So you seem to be asking whether explicitly specifying equal probabilities gives the same result as this default.  (The answer, surprisingly, is no, because the algorithms differ and therefore the actual numbers returned differ even when starting from the same random seed; but in the sense of *statistical processes* the answer is yes, the two methods are the same, as the documentation claims.  But that's purely an `R` implementation issue.)
Excellent points, @Daniel.  But "irrational" may be overlooking something: if I wanted to create a strong password generator, I might indeed generate uniformly distributed strings--but I would then screen out anything I thought likely to be tested by cracking programs, including everything in dictionaries, dates, etc., thereby creating a generator that is *not* completely uniform!  The point is that strength lies not entirely in high entropy, but in an extremely low probability that an opponent could crack the output.
Agreed. R's `survival` package is under close scrutiny by a large community.
Yikes - woe unto the person that wants to prove a non-deterministic function is monotonic.
@JDav The sum is Poisson with the rate parameter equal to the sum of the individual rate parameters.  But the OP scales by 1/n and then wants to truncate the the the integer just below Y.  I don't know exactly what that does to the distribution.
Does "one dataset by one dataset" mean line by line? Ie one dataset = 10000 features? If that is the case, then online algorithms might be something that you are looking for, see: http://en.wikipedia.org/wiki/Online_algorithm and http://en.wikipedia.org/wiki/Online_machine_learning. There exists online version for many machine leaning algorithms, for example SVM and random forests.
I would be using R's `table` and `cut` functions: `with(colon, table(cut(time, seq(0, max(time), by=500) ), status, rx) )` ... further processing to subtract the events from the sum of entrants within `rx` categories to get the size of the risksets at cut-times. (Note: I do not think the output you display matches up with any numbers other than those of time=0.)
Thank you very much. This (fortunatly) confirms what I already assumed.
Welcome to the site, @Duuuusty. Please don't phrase your posts like a Nigerian email scam.
For large $N$ and small lags, sample ACF decreases close to linearly, but to see the bigger picture, try something like `acf(S, lag.max=N)` and you will probably see a curve which falls from almost $1$ and heads towards $0$, but where the pattern is rather unpredictable at least on the right hand side and which will change if you run it again with a different seed the $S_n$.
Thank you, very clear explanation. This solved it for me. I added the solution I use in my original post.
An appropriate model might be taken to mean a model that effectively uses all pre-specified variables (accounting for nonlinearity, etc.).  I hope your instructor understands that stepwise variable selection is invalid.  Few do.
Mathematics can definitely be useful but it's certainly not true that everyone working in applied statistics, economics, modeling or related technical fields has either a degree in pure math or access to a card-carrying mathematician. So what's the claim here?
$\\Pr(C=c)$ is a constant for given $c$ so you can cancel that from the numerator and denominator.
I'd look at a document on how to read and interpret confidence intervals. I generally am not much interested in tests per se. Check out Cumming and Finch 2005 as a good primer.
(+1) Surely you mean "endogenous" in the first paragraph instead of exogenous?
Division by $n-1$ is not ML. The MLE has division by $n$.
Definitively my favorite cartoon about Data Mining
If the files are for "completely different systems," then what exactly does it mean to talk about the "probability of ... an error"?
This makes out ANOVA to be a testing procedure and regression to be a modelling procedure in which you can carry out tests. But ANOVA also has an underlying model, regardless of whether this is emphasised in all introductory treatments. So, this answer does not capture any difference between them. Nor is it addressed at the question, which is why they are taught as different regardless of strong similarities.
+1 for the how-to link.
The idea is similar, though I think you could make the visualization much nicer. It looks like there the ordering is chronological, which also brings out some interesting features.
@user1125946 - The bootstrap can be used to estimate the sampling distribution of the coefficient estimates, which distribution may itself be skewed.  This does not change the minimum variance property of OLS under the Gauss-Markov assumptions, however.  It just avoids the perhaps-inappropriate use of the (symmetric) Normal or t distributions when calculating, e.g., confidence intervals for the coefficients.
doesn't this depend on how you determine accuracy? If you're using your training set to determine accuracy this is an issue. Similarly with split sample. But if you're using some form of cross-validation you might well obtain a distribution of OOB estimates from which to generate some sense of the reliability of estimate. (The key being that the training set accuracy is deterministic, but you're interested in the OOB accuracy that can only be estimated.)
I don't have a definitive answer on this, nor the expertise... but I believe the σ2 is essentially factored out because the diagonal variances in the matrix might differ in some predictable way and the covariances in the off-diagonals may also have some pattern. That way, the matrix contains the information for the pattern only. I am not so sure this notation works for a completely unstructured matrix, which would require a different variance and covariance parameter in each cell of the matrix that might not be directly relatable to a single σ2.
Take a look at this paper by Murdoch et al (2008) [p values are random variables](http://xa.yimg.com/kq/groups/16412409/1018781877/name/Pvalues.pdf‎) in *The American Statistician*. (Link is to a self-archived version of paper.)
Thank you for this. It's just what I needed. Regarding your edogeneity comment. That makes sense, but it seems that if I do the conservative Bonferroni correction on the correlation p-values, even if there is endegoneity, the bonferroni correction should account for it no?
Glad to see Eureqa being used!
I think you should have run an experiment with more power if any of what you have are really meaningful effect sizes.  :)  That said, for ANOVA such things almost never matter much and it's clear in your findings that's the case.  Given your question, and the likelihood that you should be presenting all condition means with confidence itnervals, I'd go with the full model anyway.  But there are certainly arguments that can be made to the contrary.  If you were genuinely concerned about presenting what you feel is the most parsimonious model I'd go with the one way.
You had 10 `X` variables and 11 groups. Hence, you extracted min(10,11-1)=10 discriminants `LD`. It looks like "group means" are indeed themselves. Why not? "Coefficients" are the regressional weights to compute the LDs by the Xs. I can't tell, without having data, what is "proportion of trace", it may be related with the eigenvalues of the extraction. Please see my [LDA of iris data](http://stats.stackexchange.com/a/83114/3277).
Achilles, you appear to be conflating the ideas of "sample" and "population."  The fraction nonconforming in the *population* could be 0.02, but there is no guarantee that a *sample* of that population will also contain the same fraction of nonconforming units. In principle, the fraction in the sample (if it is a simple random sample with replacement or if the population size is $2500$ or greater) could be any one of $0/50, 1/50, \\ldots, 50/50$.
@DouglasZare: We don't know exactly, unfortunately. Most likely there are in the millions. Other than the issue that the distribution is actually discrete, would it matter that much, though, or could we normalize numbers by the total (since we can get a decent idea about the ratios)?
Apropos your real concern, you might find some useful information by searching our site for [multivariate outlier](http://stats.stackexchange.com/search?q=multivariate+outlier).
@Gaetan @chl To summarize my understanding: The features of SPSS and XLStat whereby you can specify the measurement scale (nominal, ordinal etc) decreases the data file size. However, in both instances, the software uses the correct coding scheme (e.g., expand a nominal variable with J categories into J-1 dummy variables) as part of the estimation process in the background. Would that be a fair assessment of the situation?
I suspect part of the confusion here is that many people use the term 'mediator' to refer to the middle term in a causal chain.  So if A causes B, and B causes C, then B would be a mediator in this sense.  However, this is not quite the same as what is implied by a significant test for mediation.  In a common cause model, where B causes both A and C, but there is no causal connection between A and C, inclusion of B into the model would eliminate the effect of A on C, just as with the causal chain model.
+1. Yes but it is expressible in terms of the _conditional_ correlations, knowledge of which would require more information about the association structure.
Hello Joseph, as with most (_not in general_) results in classical statistics: parametric bootstrap converges around rates $n^{\\frac{1}{2}}$, where$n$ is the sample size. In your case $n$ corresponds to $M$, as you are taking average of $M$ bootstrap estimates. This is equivalent to the asymptotic normality (or _CLT_). The assumptions and the details of the result can be found in: Hall, P. 1988. Rate of Convergence in Bootstrap Approximations. Annals of Probability.
Ah, I didn't know that densities were internally represented as (x,y)-series. That makes sense, thank you.
Could you give us the actual time series? For R, you can use `dput(temp)` and edit your question to put the result in there.
This mixes together various different problems, most prominently combining measures on different scales. Why you think it's worth collecting quantitative data and then degrading to ranks and quintiles is not clear.
When X and Y are both centered variables, $r$ is [exactly related](http://stats.stackexchange.com/a/36158/3277) to the summed squared difference (i.e. squared euclidean distance). Also, since $r$=1 if X=k*Y+c, if you z-standardize both variables then it appeares that $r$=1 if X-Y = constant 0.
@Benincasa How 'citable' a reference are you looking for? Book/journal or website?
You should clarify *in your question* what quantity you're discussing the sampling distribution *of*, under what circumstances. In particular, you can't rely on links to other pages still being there in the long term. While it's fine to link for context, your question should be able to stand on its own. Take care not to simply copy chunks of text though - brief quotes may be okay.
Thanks!! This helped very much. You were right, there was a problem with the data base, but I fixed it. Now there is no collineality and most of the variables are significant. As well, now I understand  how to read the results :)
I appreciate your effort.  I'm still confused because some of the phrases seem inconsistent (the "don't" looks particularly out of place) and much of the answer has no clear connection with the question: how is "distance-matrix" related to the covariance matrix of PCA?  Does "NMDS" refer to non-metric MDS and if so, what does that have to do with PCA?  It seems like you're taking a completely different approach.  This is fine, but it would help to explain where you're coming from and why it's applicable.
@MichaelMcGowan Fair enough.
Thanks! I got so focused on association rules that I forgot completely about decision trees.
Again, ADF and PP tests are not designed to guard against structural breaks. Hence they should not be used.
If your values are discrete, especially if they're squished up one end, there may be no transformation that will make the result even roughly normal. But formal hypothesis tests of normality don't answer the right question, and cause your other procedures that are undertaken conditional on whether you reject normality to no longer have their nominal properties.
Logistic regression (also probit, etc) does not make any assumptions about the distribution of your RHS variables. LR is perfectly fine.
ETA: figured it out over here http://stackoverflow.com/questions/7924166/converting-time-to-readable-form-in-r-x-posted-on-stats. Thanks.
What I mean is, presumably there was some legitimate reason to run the study in the 1st place. Current theoretical knowledge &/or recent studies suggested your hypothesis might be true. Your hypothesis isn't likely to "lead to meaningless data analysis" unless it is incoherent. Potentially interesting surprises / features of your data could very well be discovered post-hoc; the fact that they are *surprises* implies you didn't know they would occur when you planned the study. The issue regarding "post-hoc" is whether to believe the surprises--they need to be confirmed by future research.
That is a crucial observation. It implies you need to take more care than usual, because initially you will be obtaining a "robust" estimate of the mean high outliers. By continuing to update that estimate, you could wind up throwing out all the lower values. Thus you will need a data structure in which key parts of the entire distribution of data are recorded and periodically updated. Check out our threads with keywords "online" and "quantile" for ideas. Two such promising ones are at http://stats.stackexchange.com/questions/3372 and http://stats.stackexchange.com/q/3377.
Like majority of clustering criterions, Silhouette criterion requires, as input, 1) the distance matrix (between objects, the matrix on which clustering was performed) and 2) cluster solutions (which you want to compare). A cluster solution is a grouping variable which groups objects into clusters. So, since each tree cut is a cluster solution, make various cuts, constructing grouping variable for each. And there you are...
You use the variables "p" and "p0" to denote the response rates. To avoid the possibility that some people will see this as a P value, I'd suggest using a different variable name.
"plot(x,y) is pretty much a straight line. " wouldn't that be also true if you replace the first line by rnorm(100,1,1)?
That's a homework?
I'll look more carefully the derivations of BIC and DIC and read about the log-predictive scores. However, as I understand it, none of them evaluate what I'm asking for: the KL divergence between the 'true' posterior and my set of posteriors. I didn't get the contradiction you where referring about :-(.
Why do you insist on crossposting here and on SO?
I should let @frank-harrell answer, since you asked him, but you may also want to look at this: http://stats.stackexchange.com/questions/18638/model-selection-logistic-regression/18641#18641 although it does not use the phrase, stepwise selection is what is being discussed there.
@whuber It wouldn't be contrived at all. Maximum and minimum _potential_ prices in a market arise naturally as those reflecting different evaluations by market participants. Among the buyers, there is conceivably one that would pay maximum price for a given house. And among the sellers there is one that would conceivably accept minimum price. But this information is not public and so actual observed transaction prices are affected by the existence of incomplete information. (CONT'D)
Thank you Greg Snow.
Ah, very good! In that context, your proportions are equivalent to counts and you know exactly how the proportions should be weighted (i.e., equally since there are an equal number of members in each group---this is what I was hinting at in my comments in the answers). This is a textbook scenario for logistic regression. :)
Very definitely yes .
The zero is an observable value, that is your predictor can take such a value?
Indeed, the measures of kurtosis are typically applied to symmetric distributions. You can calculate it for skewed ones as well but the interpretation changes since this value varies when the asymmetry is introduced. In fact, these two concepts are difficult to separate. Recently, a skewness-invariant measure of kurtosis was proposed in [this paper](http://amstat.tandfonline.com/doi/abs/10.1198/tast.2011.10194).
ANOVA does not assume the data are normal it assumes the residuals are normal.
I don't know if this can be of any help, but - for example - quadratic loglikelihood penalization (squared $L_2$ norm) corresponds to having independent normal  priors on the model parameters. I don't know the deatils of the paper you're referring to, but it's definitely not impossible to "exploit" PL for that purpose - Or maybe I've misunderstood your question?
I think I'm missing a step. So my initial input is a two-column file with customerID,sales. Once I get that file in I do this upd.ts <- ts(data=upd.df$X_c1, start=c(2012, 8), end=c(2014, 5), frequency=12) which ends up stripping my customerID out. What I need to do is pass my whole file (100s of customerIDs) into my time series and run the forecast (at least I think that's how it'll work). Unfortunately, I'm stuck on how to get that working, which I believe precedes your suggestion above.
Many things; for example, the order of the observations, the sample skewness and kurtosis, the correlation with other variables.
@user1566: I formatted your equations using LaTex. Would you double check that I didn't introduce any errors?
(+1) In case SEM or factor analysis seem daunting (e.g., given your time frame), you could assess the extent to which your components hang together by inspecting a matrix of their correlations.  One step up from this would be to compute Cronbach's alpha using SPSS's Scale...Reliability command.  As gung said, you first may need to reverse-score some variables so that in all cases "high=good."
@topepo: I actually don't want to only optimize `alpha`. I want to assign a fixed `alpha` to fit only ridge or only lasso, but have `caret` use the `lambda` that's generated from `glmnet` as the grid. The custom model approach doesn't seem like it will work. Please see my edit. Thanks.
Interesting question, though it required a trip to the bookshelf (which I should have done anyway).  I've added a bit to my answer on this point.
Are you looking for an answer that Dr. Sheldon Cooper might appreciate, or one that Leonard or Raj would be satisfied with?
That "relative frequencies" converge almost surely to the true probability of the event is a particular *subcase* covered by the LLN, though.
Can you please elaborate a bit? Like write few-word intro to those algorithms?
Thanks a lot for this solution. I, at least, know one method that can solve it half the time as compared to `solve` :-)
Sometimes - though incorrectly - quantile-terms are used for the between-quantile groups. Thus you'll see expressions like "scored in the upper quartile" or a reference to income "in the [sixth decile](http://www.google.com/search?q=%22in+the+sixth+decile%22)" or similar expressions. This seems to be especially common with deciles. I'd be inclined to say "inter-quintile means" (but I wonder whether the not-strictly-correct "*intra*-quintile means" might not more often convey the correct impression).