tttjjj/elt_bench huggingface dataset:
Am I missing something? You have >10K features but <1K samples? This means the last 9K components are arbitrary. Do you want  all 1K of the first components?
Thanks for your answer. I just found a useful paper on that issue: http://dx.doi.org/10.1007/BF01897163. It compares the usefulness of different normalization approaches with each other and with no normalization.
If you want to guess the value of an individual with given properties, then look at the prediction interval. If you want to guess the average of all individuals in the population having these properties, check the confidence interval.
A "perfect fit" is so far from being attainable in any realistic logistic regression that it seems unfair to use it as a reference or a standard.
@anna wrote: `When you prescribe "random number" by definition it is the uniform distribution`  ... not at all. There are infinitely many distributions with domain of support on (0,1). A random variable whose distribution has not been specified is just a random variable whose distribution has either not been specified, or is not known.
@Aaron, I don't know about "never will" handle corr/var structures -- I am interested in adding these features and don't think it would be *that* difficult -- but I would certainly say "don't hold your breath".  I'm not familiar enough with panel data to know what would be required for `lmer` to handle them ... Hong, can you add a brief explanation to the question that describes the necessary statistical properties in a little more detail, or gives pointers?
I'm not sure the OP is after PCR. PCA is also a good way to summarize multivariate datasets (not necessarily in order to perform data reduction for subsequent use in a modeling framework), that is approximate the VC matrix to a lower-order one while retaining most of the information. The question seems to be: Am I right when interpreting the first few eigenvalues and PCs (as linear combinations of the original variables) even if there were some collinearity effects? Your response doesn't seem to address directly the OP's question.
This paper is helpful, regarding concentration and dispersion measures for the log normal: "Log-Normal Distributions across the Sciences: Keys and Clues
Author(s): Eckhard Limpert, Werner A. Stahel, Markus Abbt
Source: BioScience, Vol. 51, No. 5 (May, 2001), pp. 341-352 at http://www.jstor.org/stable/1314039
I think that this question and answers would be of help:
http://stats.stackexchange.com/questions/48662/reasonable-assumption-in-bayesian-linear-regression/48682#48682
if you have some notion of location and spread there (like mean and standard deviation) you could perhaps use moment matching or some equivalent.
That is the intuition, a proof maybe must be different. I am out of time now, but note that the multivariate normal distribution can be specified by specifying the (normal) distribution of all linear combinations of the components of $Y$. When the covariance matrix is the projection $P$, choose $u_1, \\dots, u_k$ as an orthonormal basis of $U$. $P$ can be written $P=\\sum u_i u_i^T $. Choose as coefficient for the linear combination on of the $u_i$, you will see the variance is one. Choose for the coefficient an length-one vector orthogonal to $U$, you will see the variance is zero.
BTW your `Current GPA` variable could be safely considered as interval because you supply the numeric bounds. You might recode it into 1=3.88, 2=3.63, 3=3.25, for example.
Thanks.  My features aren't words but a set of real numbers for each feature.  But, I think I can adapt some of these techniques.
Let Xi ~ beta (ai, bi), i = 1, 2, be independent standard beta
variables. The random variable w=X1/X2 has density... The formulas do not copy and paste. I can re-type them but if I am misinterpreting then my re-type will show the wrong thing.
@rg255 I have added a bit at the end to address your comment. Also perhaps the conclusions from the _Strong encouragement_ reference would be relevant.
I think we can agree that the desiderata stated by the OP are contradictory: preserving the original distribution of values and transformation are antagonistic.
+1 This is a question I wish I'd asked myself. Your edit also shows you've tried to research your problem. I'm leaving a note to (hopefully) remind me to promote this question when it's had a couple of days.
@NickStauner This touches on the core issue which is "how much uncertainty do we accept in our decisions"? And this is not a question that Science can answer. In Statistics the scientific community has settle in "guarding the null" by using very small Type I errors. In Accounting, International Financial Reporting Standards accept a principle of _more likely than not_ which is at the 50-50 threshold. Criminal Courts have the "beyond reasonable doubt" principle which appears closer to Statistical practice, while civil courts follow "preponderance of evidence", which is something in between, etc
I think the statistics tells the story.  Both results should be disclosed but as long as the data is correct and the correlation can be explained the paired test is more accurate because it takes the correlation into account.
"*To estimate the standard deviation of my non-normal data, I will need either 30k or 300k measured data points*" -- Your question was about constructing a "99.993666%" interval. If a standard deviation gave you the limits, it would be different and you could get away with less. *It doesn't.* Even the Chebyshev theorem doesn't save you.  Neither sample standard deviation nor population standard deviation will get you that interval you asked for.
Agreed. If you want to try a spline, perhaps `loess` with `span=0.9` or something like that would be worth it. I really don't think the two-polynomial approach would work: among other things it doesn't look (eyeballing it) like 2008 is the turning point.
@Wei: $\\mathrm{E}\\{g(X)\\}$ exists only if $\\mathrm{E}\\{\\, \\left| g(X)\\, \\right| \\} < \\infty$.  Without this restriction,the expectation is not uniquely defined for some CDFs.
While that's true, changing both units in concert doesn't serve to reveal the distinction between slope and correlation (but instead disguises it), so it's unhelpful as a response to the question at hand. (In general $y$ and $x$ are not even in the same *kind* of unit, so there's no particular reason to expect them to change together.)
@Analyst has given the best answer so far. FN may equal FP, but you have to accept some utility function before choosing cutoff.
Well, in my opinion it's better if the long term range doesn't include the short one, the information is clearer. But In my company we often use long term range including short term range because it's simpler for us and the client. For exemple, if you use the second method you will have to handle variable such "mean over last year excluding front week" and/or "mean over last year shifted by 1 week" and/or "mean over last year excluding front month" and/or "mean over last year shifted by 1 month". As you can see "mean over past year" is way simple/clear.
The two sample proportions test was quite similar, though I did it two tailed.
Just to be clear, I referred to stepwise only because a lot of relevant issues are discussed on those questions through the critique of those techniques. In relation to the current question, I wanted to emphasize the importance of repeating feature selection when one is carrying out resampling. It sounds like that was already your plan.
See http://stats.stackexchange.com/q/1883/159 for a similar question (which was closed as subjective & argumentative).
I think this depends on what you really hope to achieve with the analysis. In your case (given how you describe your data), both methods will be descriptive. Regression will help you answer a question such as _which features have the strongest impact on price?_, whereas clustering (such a MCA) will help you answer questions like _which features are shared by printers of different price range?_.
Brier score (used since 1951 for judging accuracy of rainfall forecasts) is the mean squared difference between predicted probability and 0/1 outcome.
@steve. I created a new df with less noise in x1 and   x2.df=data.frame(y=rank(rnorm(50,0)),x1=rank(y/2),x2=rank(y*rnorm(50,0)),x3=rank(rnorm(50,0))). Now the lines seem more appropiated for the plot with original values than for the plot with residuals..I woudl expect the inverse would happen..
I went away and tried to take the z-score approach with this but am not sure that I have understood the approach or whether it is valid e.g. (avg.diff seattle - avg.diff NFL)/standard deviation of differences to league average HFA. In the case of Seattle: (3.47-0)/1.32 = approx 2.64. On the basis that this is above 2.0 can I treat it as significant over 6 years? Incidentally the only other teams that were close to 2, were Baltimore (at virtually exactly 2.0) and Miami (at -1.98). 5 other teams had a 1 or -1 value. Everything else was between 1 and -1.
A general approach to transformation are [Box-Cox transformations](http://en.wikipedia.org/wiki/Power_transform#Box.E2.80.93Cox_transformation). What you could do is the following: 1. Fit your regression model with `lm` using the untransformed variables. 2. Use the function `boxcox(my.lm.model)` from the `MASS` package to estimate $\\lambda$. The command also produces a graphic that you could upload for our convenience.
We have answered at the same time ;-)... (+1)
The idea was to stop simulation if the 95% confidence interval of estimated $p$ value is located above 0.05 after particular number of permutations. I am using the normal approximation to obtain the CI in this case. So, no, I don't keep simulating in this case. Thank you for pointing out the SPRT approach.
Initially, I performed around 2500 permutations in order to estimate p-value of order of 0.01.
Thanks for the links and commentary. I think that generally answers can be improved by using the `[manuscript title](uri)` [link markdown](http://stackoverflow.com/editing-help/#links). After a long day, I find coming across answers with long hyperlinks can be subconsciously jarring, and unfortunately might bias a reader against an otherwise good answer.
Judging by the name, that's probably doing smoothing, not interpolation - the difference being whether the original points are preserved exactly, or not.
I recommend doing something about the outlying data first, if you can: I suspect they might give ARIMA some trouble.
@Shankar, It is not prudent to relate causality and gene expression of microarray data. It is close to saying "correlation $\\Rightarrow$ causation"
@rosser Also had this idea, but I cannot get rid of the feeling that if one treats "drug" as a predictor just as any other patient attribute, then "drug" may be undervalued when the model is fitting the problem "globally". I cannot express this explicitly, I might be wrong.
@Miroslav: Water content, higher level of metabolic activity in the particular tissue?
thank you, this is a great answer and will help me explain it to people who are not statistically savvy.
@Scortchi I agree: NOIR is not nonsense and would not have survived otherwise. 1 = strongly disagree, 5 = strongly agree are importantly different from 1 sheep, 5 sheep and many points made under the heading of NOIR are serious and practical.
If you are trying to estimate the total liver volume from contours of the liver on slices taken throughout its extent, use the number of slices that you would normally use to estimate the volume by the manual method, and compare volumes. If you mainly care about contours from individual slices as independent observations, choose the number of slices based on the power calculations recommended by @O_Devinyak. You should, however, examine whether your method works as well for slices through the middle of the liver as it does for slices near the tip.
So the dummy variable interaction is subdividing the group of private school students into male and female?
Consider the expected value of what you found. Then multiply by $1/T$, and consider what happens when $T\\rightarrow \\infty$.
Thanks @drknexus. Will have a look into it. Just doing a little more research myself and this article also might lead in the right direction: http://stackoverflow.com/questions/4807398/how-to-generate-distributions-given-mean-sd-skew-and-kurtosis-in-r
Here's a little thought experiment.  (1) What do you think the human population of the world was on June 1, 2013?  (2) What do you think the population will be next year?  (3) What do you think it will be in the year 3000?  In which answer do you have the most confidence?  The least?  *Why?*
@ Jeromy. I understood that if you are using ordinal data "When the sample size is even" and the numbers on either side of the cut point are different then it is not possible to calculate the median. This is because there is no clear mid point between two ordinal values. That is, your values are ordinal not ratio/interval because there are no even distances between them.
@MathEE Since this question havn't enjoyed much attention here I'm grateful for any pointers and/or discussion!
+1 for "Computers will only make statisticians obsolete when strong AI makes humans as a whole obsolete."
Thank you Dr. Snow.
Judgement is a good word, since all we can ever observe is correlation. All that experiments and/or clever statistics can do is allow us to exclude some alternative explanations for what could have caused an effect.
It doesn't matter. You should first be thinking about the scientific questions you would like to answer with these data and what the numbers mean about those questions.  For additional guidance, search our site for [related threads](http://stats.stackexchange.com/search?q=%2Binterval+%2Bordinal). That might help you to ask an improved question whose answers may be more relevant and useful for your research.
@whuber: I don't think RAM is the problem. First because I have 70GB of RAM in my workstation ;), and second because if I monitor the process using top there's barely a blip in the RAM usage... If anyone knows what R is doing here I would be very curious to know...
http://journal.r-project.org/archive/2010-1/RJournal_2010-1_Guenther+Fritsch.pdf   the paper on neuralnet i read
For reference would you please provide the full reference as provided by Castella? It is an interesting quote.
Welcome to CrossValidated! You could try using Amazon's Mechanical Turk to collect data from users, but you will have to compute the results yourself. For most voting systems, you can probably do this in Excel fairly easily.
And what happens if your population (not sample) increases a lot? Will the generalization capability of your algorithm decrease?
Thank you for your answer.
+1 It looks like replacing `uniroot` by a direct solution speeds things up by three more orders of magnitude: you should have no trouble obtaining around $10^7$ variates per second.
Isn't it possible that your independent variable that obtains negative coefficient in multiple-prediction circumstances serves as a suppressor variable?
If statistically challenged, the best strategy is to back off and focus on explaining your scientific or substantive problem. In this case, any variable coded 1-2-3 for yes-no-sometimes is unsuitable both for ANOVA and for correlation. I would certainly recommend coding yes-sometimes-no as 1-2-3; that makes your scales ordinal (if this term is unfamiliar you need to read about it). That said, what your other variables are is very unclear. There are just passing references to "more games" and "age group".
Good point; "weak" is yet another term that statisticians use slightly differently than other English users.
Thank you for this insightful answer! Your point about independence is a very important one that I realize for the first time.
(+1) If I recall correctly, Cover at one point was a consultant to the California State Lottery.
If your monthly Sharpe is 0.75, then your annualized Sharpe is 2.6, which is rather high! The above rule should probably be amended to state that you should probably have at least 20 samples. (I assume you have access to daily marks. In that case, the _length_ of time remains the same, but the number of samples changes). So barring the small number problem, yes, in 4.8 months, you should have a reasonably powerful test *under that value of the alternative*. Most likely what will happen is you will fail to reject the null and be somewhat disappointed.
Could you please make your question a little more understandable?
Related to: [How to do ANOVA write-up when there is no significant effect at all?](http://stats.stackexchange.com/q/41678/930)
Yes: see http://stats.stackexchange.com/questions/34396.  The SAS details, if you need them, are best asked on SO rather than here.
Thanks for the reply. So I understand from this that  clustering never takes training data. Is it true?
(+1) As an alternative, we might consider a permutation test. Good that you indicated to report ESs.
Since you may take $\\rho(x) = x$ and then $\\mu_m$ is the sample mean, that means it could be even not strictly convex, but strictly increasing yes, thus... I would put either strictly convex or strictly increasing, both seems to be sufficient, though have still to prove this. Intuitively strict convexity ensures unique global minimum, for strictly increasing it's the gaussianity assumption that matters.
I only care about the groups, but I wanted to make sure that I was not confounding something/doing it incorrectly by creating measures versus finding some complicated interactions
Have you checked this (aggregated) thread: http://stats.stackexchange.com/questions/12386/machine-learning-cookbook-reference-card-cheatsheet I think you'll find it quite helpful.
Because I misinterpreted the question to be $B_i = A_i + ki$ instead of just $k$.  I will edit.
It's a good question (+1), but I would urge caution: unless those companies are using some standard intervals for reporting their statistics, there may be substantial bias introduced by their choices of breakpoints. You're probably familiar with the myriad sports statistics flung about the media of the form "the team has won five of their last seven games--they're on a streak!" that exist because, of course, the team *lost* the eighth game back. Aggregation can be similarly jiggered to make data look better (or worse) than they really are. Statistical procedures won't cure that problem.
I'll try to use Chebyshev bounds but I would have to see how they can help with my [original problem](http://stats.stackexchange.com/questions/83967/transform-moments-of-a-random-variable-to-fit-the-moments-of-another). I was hoping that getting the probability of a value under one set of moments I could get a quantile using a different set. @Glen_b
IF number of articles written and number of events applied to constituted variables that, in your judgment, should determine acceptance rate, then once you regress the latter on these 2 predictors, you can see to what extent you're getting a good fit to the data and you can study the residuals to see what patterns there might be to the poorly predicted cases.
maybe you could add a word on what specifically make InvivoStat relevant to this kind of analysis?
@dchandler: The parameters and probabilities were chosen arbitrarily, for the sake of illustration.
I'm not sure I understand what you need.  If referring to the second paragraph, let $X$ = the single predictor or the predicted probability that $Y=1|X$ and $c$ be a cutoff (naively) chosen to trigger a "positive" decision about $Y$, where $Y=1$ indicates that the event occurs and $Y=0$ that it did not. Sensitivity is Prob$(X>c | Y=1)$ which is in backwards time (and thus information flow) order.  What is needed is Prob$(Y=1 | X=x)$ where $x$ is the observed value for the object to be decided upon.  Once we know $X=x$ it is not helpful to know that $X>c$ for some cutoff $c$.
@John, there are similarities, but there is one important distinction: Cohen's d (etc.) is calculated from the relationship between 2 distributions, whereas z-scores are calculated within 1.
You can have a look here: http://onlinelibrary.wiley.com/doi/10.1111/j.1541-0420.2007.00960.x/abstract. 

I would be interested to know how you checked proportionality in the presence of a frailty term. Indeed, caution is needed when interpreting population hazards in the presence of heterogeneity in the population due to the "selection effect".
Sorry, I should have clarified my previous comment. I'm reposting it with explanation. The link to Rob Hyndman's post on [Cross-validation](http://robjhyndman.com/hyndsight/crossvalidation/) was in answer to Q2
Have you looked at Max Flow algorithms?
You don't need labels to visualize your data, in the example above, labels were used only to paint points with different colors. In your case you'll see clusters of points where all the points will have same color. Then you might get some insights depending on what you see.
Frank, thanks for your response. I don't understand why $p$ would be 1. The same model is being applied to a holdout set, so the same weights and their values are being used.

Also, this isn't the formula for R^2, but rather, for adjusted R^2.
ANOVA is based on the homogeneity-of-variances (= homoscedasticity) assumption. Anova is fairly robust to violation of the assumption unless the group means correlate with the group variances (i.e. the wider is the variance the higher is the mean). If that correlation is present, power transform of the dependent variable (to attain homoscedasticity) before ANOVA is a good cure. More sophisticated Box-Cox transform is another option. A nonparametric analysis (suggested by @Peter in his answer) is yet another option.
EFA is easier to do: just dump everything into the kitchen sink. With CFA, you have to get yourself into trouble of writing some syntax of grouping variables into factors, or drawing the diagrams in Stata or AMOS GUI. Few people have the extra 5 minutes to do that... although they do seem to have the couple of months to revise the papers thus written.
How do you define the "risk" of an estimator? By expected squared error loss?
+1 Just a quick note that odds ratio has the property of "invertibility" if you change the level of the outcome -- e.g. if smoking (compared to not smoking) has an OR of 2 for having disease X, then smoking has an OR of 1/2 for NOT having disease X. This is not true of relative risks (whether this is trivial or not depends on the application.) For a worked example see [Wikipedia on odds ratios](http://en.wikipedia.org/wiki/Odds_ratio#Invertibility_and_invariance)
(+1) Very nice!!
(+1) Welcome to our community, Ido!
Are they trying to reconstruct an image? Or does "best" mean looks the most human appealing? (i.e. very subjective)
You could retain the similarities as weights: http://stats.stackexchange.com/questions/2948/how-to-do-community-detection-in-a-weighted-social-network-graph
I think I understand, thanks very much!
Regarding the first question a quick side note: Mean in statistics is just _the first moment of a population_, while median is not. Seeking to use CLT, law of large numbers, etc. you are again linked to the existence of finite moments. Though taking for example Cauchy distribution: median exists, while mean doesn't ;)
Glad to help.  To understand why you should keep a 'non-significant' interaction (or any other term), it's best to read the answer I linked above.
@meri: there's an example on the page I linked to.
Thanks Tom! You are a star. You should visit more often :-)
Depends on your background. If it's economics, then a mainstream econometrics book. If it's not, you need a modern regression text (too many to choose from) followed by a generalised linear models text (start with Dobson and Barnett, not McCullagh and Nelder). But don't expect any of them to start talking about spatial gravity models as an example; for that you may need journal literature.
Already posted here: http://stackoverflow.com/questions/23969100/clustering-with-proportional-threshold
The **cumulative**distribution function tells you the value of $P\\{X \\leq x\\}$, not of $P\\{x \\leq X\\}$.