tttjjj/elt_bench huggingface dataset:
From my initial read of the documentation, this seems like what I was looking for; I will investigate further.
Right, so as far as I understand your second suggestion this is somehow similar to measure the KS distance. I wanted to do 1+2 as you proposed and measure the greatest distance between the cumulative density functions.

However, I am more interested in the BIC/AIC method. With your answer and the answer above I found the following manuscript that might help for future references: 
http://www.cs.cmu.edu/~epxing/Class/10701-06f/project-reports/yu.pdf
Since the timestamps are irregular one can only make guarantees on the dt between data points. The actual points depend on the starting point of the series. If you fix your starting point to a round hour it will behave as you like... (assuming you have points every N minutes)
It's intuitive but only really makes sense if you accept the normalized likelihood as a pdf -- I know enough stats to know there's a lot I don't know and others on CV are better qualified to judge than me. I don't think that plotting the cumulative function gives you much more than plotting the original. Not sure what you mean by "individual weights" but you  might be looking for confidence intervals, in which case you can either get a confidence interval for a binomial parameter directly or derive it from the likelihood ([example](https://onlinecourses.science.psu.edu/stat504/node/39)).
We both get the same answer but as I said we do need to assume that if p is the probability that task i takes x days then 1-p is the probability that it will take longer than x days.
Also since there were not equal items in each of the categories (I had to enter the data in the 'long' format into SPSS), then I could not just enter the raw number of vocabulary items correct for each combination of the variables, instead I entered the % of items correct.
This question seems to be contingent on how you define *more than strictly necessary*.
Knowing nothing about the dependent variable except that it's a number makes it basically impossible to give you advice.  You hint that it's a contingency table in long form, where e.g. there are 1101 people with black hair and brown skin.  Is that the case?
I don't understand your question: Yes, data are needed in order to test a given hypothesis. Are you interested in preliminary [tag:power-analysis] which will give you 'confidence' in your decision about test outcome, or are you asking about power of statistical tests with observational studies as the tags you used would suggest?
@Stat do you want to post the comment about Eq q as an answer? Otherwise I will append it to my answer.
I'm sorry see @Suncoolsu's answer was deleted.  It and the comments that followed clarified a difference between multicollinearity and ill conditioning.  Also, in a comment Suncoolsu pointed out how preliminary standardization can help with polynomial regression.  If it happened to reappear I would vote it up ;-).
@Michael, $(X_1,\\ldots,X_n)$ is multivariate lognormal if and only if $(\\log(X_1),\\ldots,\\log(X_n))$ is multivariate normal.  ptikobj: This gives you many ways to fit a distribution--just use your favorite method to fit a multivariate normal distribution to the logarithms of the observations. (You are in trouble if any of the observations actually equals zero, because that is inconsistent with the lognormal assumption.)
+1.  Beat me by 22 seconds... :-)
I absolutely hate wikipedia answers for stats. There are rambling, symbolic messes. I am looking for a gem of an answer that can explain the answer in plain English, as I believe that that shows a deeper level of understanding than a math equation. There are many popular "plain English" questions on here, and for good reason.
Hi @Ahdee, please could you describe in greater detail which question you want to answer from your data, what your hypotheses are, what "is about 50% percent in each group", what are the bins, how were the data collected? Perhaps even share a plot or a snapshot of the data so we can see what is going on?
Good DBSCAN and OPTICS implementations should use index structures to run in O(n log n). In ELKI, I recommend using STR bulk loaded R*-trees, these work best for me.
I get the point. I figured F can be seen as a quotient between two Chi-squares (which should in turn be sums of squared normal distributions) - but my prerequisites about Beta are so hazy that I should have actually retreated to "Why can weird forms of F be used to compute CIs for binomial distributions?" (Naively assuming that there is an easily comprehensible explanation.)
Cyclic variables are best thought of as elements of the unit circle in the Complex plane.  Thus, it would be natural to map the days of the week to (say) the points $\\exp(2 j \\pi i / 7)$, $j=0, \\ldots, 6$; *i.e.*, $(\\cos(0), \\sin(0))$, $(\\cos(2 \\pi/7), \\sin(2\\pi/7))$, ... $(\\cos(12\\pi/7),\\sin(12\\pi/7))$.
@Daniel It means replacing aspects of the real world with idealized random processes.  In my preceding comment I treated the answers *as if* they had been obtained by drawing values out of a well-mixed box containing equal amounts of zeros and ones.  That's a probability model.  It describes a situation where respondents are independently guessing the answers to the questions.  A more complex model would be needed when it's likely answers will be correlated.  E.g., two questions might ask the same thing, so a *consistent* guesser would give identical answers.
I don't know what you mean by "level-form," Anna, but it seems your questions about interpreting coefficients when the dependent variable is a logarithm have been answered in many similar contexts: please search this site for [regression log interpretation](http://stats.stackexchange.com/search?q=regression+log+interpretation).
It's not all that hard to decide which segment of the boundary the minimum lies on. Then given that constraint, it's easy enough to cast it as a problem which does have a local minimum ... but @user603's suggestion of using a standard minimization algorithm like gradient descent can be quite useful as a general approach.
+1 Some interesting ideas. An example of a spatial nature (which might therefore have some application here) that takes a few steps in this direction recently appeared at http://gis.stackexchange.com/questions/52502/how-to-represent-trend-over-time.
Thanks for the feedback. I get that the IVs that are important will vary depending on the quantile. What I'm not sure about is if I can interpret the t-stats for the various quantiles in the same way I would for a normal linear regression. For each quantile I run 10k quantile regressions with different combinations of the IVs. At the extremes I find multiple combinations of IVs that are significant at the 5%-level(t-stat), while at the middle there are very few IVs that are significant. Is this due to may data or how the t-stats are evaluated at the quantiles?
But now there is one:  http://comments.gmane.org/gmane.comp.lang.julia.devel/15153   did'nt try it (yet).
You have worked far too hard and have extended the spread-vs-level plot beyond its breaking point.  It should be used just *once* and its estimate of the Box-Cox parameter should never be taken as gospel.  In most cases, you should round it to the nearest multiple of $1/3$ or $1/2$. In this case, the initial result is good evidence that no transformation was needed, because $1.12$ is comfortably close to $1$. Stop with that and go on your way with the diagnosis of your model's fit.
Define "true", please.
Brian, please spell out "GMM" acronym in your _question_.
As I understand, there is no variability at all in your sample. How then do you plan to compute a confidence interval?
(Oh, btw, you have a typo - it's "winsorized", not "windsorized". Took me a moment to figure it out, since I hadn't heard about it before.)
And the prove indeed is more than trivial with equal variances :( $Cov(X+Y,X-Y) = E((X-\\mu_X)+(Y-\\mu_Y))((X-\\mu_X)-(Y-\\mu_Y)) = Var X - Var Y = 0$... Good luck, Rob.
Something is seriously wrong when you have to use Box-Cox transformations this strong.  Moreover, there's not much practical difference between -2.8 and -2.3: you could safely use -2.5 in both situations.  I suspect you may have one or more outliers to deal with.  Box-Cox transformations really shouldn't be used in such an automated way: they are more suited for exploration.  Draw a ladder of probability plots of transformed residuals for each model, varying $\\lambda$ from $-1$ to $2$ in units of $1/2$, to see what might be happening.
Thanks. I like the example, it is very helpful.
Hint: divide the numerator and denominator by $\\sigma^2$. Note that if $Y\\sim Normal(0,\\sigma^2)$, then $Y/\\sigma\\sim Normal(0,1)$. Investigate the distribution of the square of a standard normal variable (hint':chi-squared) and then the distribution of the sum of these variables when they are independent. Finally, find the distribution of the ratio, there is a well-known relationship. You do not need to do all the calculations, just find the relationships. Keywords: standard normal, chi-squared distribution, F-distribution.
To me, it does not make much sense to model your DV as a latent variable measured by two indicators. I'd only use one and choose the more informative one. This should be sufficient and reduce unnecessary complexity.
By the way, this is the earliest place I have found mention of Cramer's phi.  Amazing how a lovely little sidenote in that book became a well known method many decades later.
+1.  *Finally* someone has pointed out what perhaps is the most important aspect of this question: in most situations, "normality" is important in regard to residuals or to sampling distributions of statistics, not in regard to the distributions of the populations!
The reference does not include any information on KNN. Is it the right one?
I considered the preferable way too, ie put dosage explicitly in the model as a continuous variable. However given the fact that experiments have already been done for only 3 dosage values (0.1 0.5 and 0.9), which way will generate better predictions, especially for those dosage values not tested? Thanks!
Read the "and" like "plus". They include a variable $age^2$ plus a variable $driver dummy$. That is, "the baseline specification contains (a quadratic in age) + (drivers dummies)".
I don't think it becomes any simpler.
I don't see any direct need for the first two tags in that list.
Frank Harrell's *Regression Modeling Strategies" might be a good choice for an applied perspective.
I appreciate very much your answers, I'm studying them. Then my question would be: what am I really doing when I follow the steps which I've listed in my R code?
only if $x_i$ is independent from $(\\bar{mu},\\bar{S})$
Are you sure about equality of the dimension of $y'y$ and $\\hat{y}y'$?
@robin: thanks for comment. i haven't seen much research in this area and the input from data provider was minimal [so far they were not interested/capable of investigating it further]. after initial exploration of data it was quite clear for me that there are couple of distinctive patterns [examples being 'heavy downloaders' in just a few locations or 'frequent hoppers' with lots of small sessions in large number of locations].
my goal at this stage was to try to use data itself to tell me how can i divide it best and minimize subjective input.
@PeterFlom I did but to no avail. I was recommended the multcomp package but there seems to be no solution to creating the tukey intervals there.
Well said @jbowman.  Unless all the trees turn out to have the same structure and give the same predictions, none of them is representative.  And the typically poor cross-validating trees is why random forests, boosting, or bagging are more appropriate.  I can't see that fitting one tree on the whole sample is an adequate summary, if the tree structure doesn't replicate over cross-validations.
This is what I did some time ago using the packages `risksetROC` (by P.J. Heagerty) and `boot`. CIs are obtained, as said, trough bootstrapping (x-axis is time in months).  http://i.imgur.com/Gf0g5.png
Thanks, I have one more related question: how should I setup and test the null hypothesis that the probability of developing a condition is gradually increased from one category to another?
@gung -hello!  What about this point of view:  In a RCT, the 2 groups are all we have.  Of course they come from the same population: there are not 2 populations about which to make any errors, Type I or otherwise.  So statistical significance is irrelevant, but large differences could well matter and could well require adjustment via the use of covariates.
Loess is a variable kernel REGRESSION method. The question asked about variable kernel DENSITY estimation.
Thanks for that. A customised solution might work best.
Indeed, if they have the same shape, the signed rank test can as legitimately be interpreted as a test of means (as long as means exist), just as it can be a test of medians, or 10th percentiles, or any number of other location-measures.
Thanks Glen. Looks alright. However I did not mention in my question that after I standardized the X variables, I fitted the model without intercept, i.e., $\\beta^*_0=0$. This is generally done in multiple linear regression. I am not sure whether this trick hold for logistic regression as well or not. If holds, then intercept estimate will be obtained easily by using your formula. But if not, then I have to do my work.
The following question on ["intuitive explanation of multicollinearity"](http://stats.stackexchange.com/questions/1149/is-there-an-intuitive-explanation-why-multicollinearity-is-a-problem-in-linear-re) may be of some help in the above context.
I don't understand what do mean by the if/else clause (especially if $f$,$g$,$e$ are real functions).
duplicate of [this](http://stats.stackexchange.com/q/4708/603) question?
+1. On Google: "the univariate distribution of" has 25,600 hits.  "the joint distribution of": 1,080,000. "the multivariate distribution of": 85,100.  "the bivariate distribution of": 89,800.  This sounds like the "joint" version is popular with "univariate," "bivariate," and "multivariate" occasionally used, each with similar frequencies.  These likely are used in circumstances requiring clarification.  (I have often seen "the univariate distribution of" used in this sense.)
+1 for good points. It frankly did not even cross my mind to make a difference between the metric and the non-metric trees, since I was all the time thinking of *not* making the assumption of a molecular clock.
Thanks! I tried the example on the FAQ and it provided a nice-looking plot.
@Ken - It would be inaccurate to describe economics psychology or AI as shadow arms of statistics - even if statistics is used heavily within each to analyze many of the problems these fields are interested in. You would not want to suggest medicine is a shadow arm of statistics even if most medical conclusions rely heavily on data analysis.
It's worth noting that some authors like to make a distinction between DM and ML depending on the magnitude of $k/n$. I personally like Radford Neale's approach, in his course on [Statistical Methods for Machine Learning and Data Mining](http://www.utstat.utoronto.ca/~radford/sta414/): Many machine learning problems have a large number of variables, Data mining applications often involve very large numbers of cases.
Hi Frank. Sorry for my ignorance, but what do you mean by "unblinded to Y"? The model selection I employ to get best model (eg prey) starts with the full set of prey-related variables and removes non-significant in a backward stepwise fashion (anova() in ML). Then I compare with best habitat (for example) model, obtained with same procedure. In final model comparison, I want to use AIC (from ML - see  above question) to compare all (ie prey alone, habitat alone and prey+habitat). I obviously dont want biased parameter estimates etc. so want to be clear on what you have pointed out. Thanks.
I'm not sure.  If you look at your data, does the first sample collected in each of the three time series correspond to the same point in time in the underlying process?  If so, then they're "in phase".  Or is there a lag between when one series is collected and the other two are collected?  If there is a lag, then they would be out of phase. There's a cute picture on [this Wikipedia](http://en.wikipedia.org/wiki/Phase_%28waves%29) page which might provide some insight.
Some clarification might be helpful.  You say want to build a 'predictive model', what is it you want to predict?  Also, what constitute the data that went into the CA?  A set of explanatory variables, a set of response variables, all the variables you had access to, etc.
The weighting seems a bit weird; why not simply use all the data points at once to find an overall error rate per member of team 1? Of course, this error rate could be updated for every member of the first team when they add another 10 data points.
Could you elaborate on how Jeffreys's prior and its relation to the Fisher information matrix means that Bayesian inference is not subjective? As I understand it, the main reason to use Jeffreys's prior is that it is invariant to alternative parameterizations of the model. Additionally, in a multidimensional setting, these Jeffreys's prior can become highly informative, and the results are controversial (Gelman, BDA 3, p. 53). Does this undermine its 'objectivity'?
I don't understand. 
If ID is just an ID, then you don't want to transform it. 

And what are you averaging X4 over? That is, what is the denominator? X4 seems to have one value for each ID.
+1 for mentioning about lower precision, though it is far from being an enginnering prerogative.  The lower the precision the more likely we are to make bad decisions. This is closely tied to Type I/II error and spans several disciplines but is mostly relevant to statistics, decision science and economics.  Utility functions should be thought of ahead of time and part of the thought process to identify a suitable methodology.
For a data set of $n$ nonnegative numbers, the _coefficient of variation_ -- the ratio of the standard deviation to the mean -- can take on values as large as $O(\\sqrt{n})$ with the maximum value occurring in the extreme case when all the numbers are $0$ except for one (see [this question](http://stats.stackexchange.com/q/18621/6633) for details). Thus, the standard deviation exceeding the mean should not be regarded as an exceptional case requiring lots of explaining away.
@dimitrisfekas That's exactly what this would do?  If it was 88 instead then the score for dog would be $\\frac{88-45}{\\sqrt{88+45}} = 3.73$ which is still a lot higher than 0.816 for cat?  Cat would only outrank dog if dog's second day was 53 or lower ($\\frac{53-45}{\\sqrt{53+45}} = 0.808$)
Write down the *log* likelihood: obviously it is a sum of squares of things. Algebraically attempt to rewrite that in terms of $\\bar{x}$; if you're successful, you will see that the likelihood depends on the sample only through the sample mean *and* you will see precisely how it depends on that mean. A little reflection should then indicate how to answer your "aside" question. Your third question, when inverted (given $\\alpha$, to find an $x$ for which $\\Pr(X\\gt x)=\\alpha$) is a *tolerance limit*; there are several kinds that cope variously with uncertainties in $\\mu$ and $\\sigma$.
I think you should write out some equations, to make your reference to "the RHS" clearer.
You can check whether predictions is valid or not by Cross Validation, for example. But I didn't get your point about Q2: why do you need to reduce the effect of a good predictor? In most practical cases that would mean that you reduce classification quality?
It is standardized to be in the same unit as the mean
This does not seem to have any data about number of people who work as either Interaction Designers or User Experience Designers
IV is used plenty by non-statisticians, and reduced-form econometrics uses plenty of techniques for causal inference other than just IV (diff-in-diff, regression discontinuity, etc.).  See [this paper by Imbens](http://www.nber.org/papers/w19983.pdf?new_window=1) for a reconciliation of econometrics IV with recent non-econometric statistical IV developments.
Andrew Ng's class is great, but I think his way of explaining CV is somewhat confusing. Check out [this](https://class.coursera.org/ml-005/forum/thread?thread_id=2229) thread on Coursera for a discussion on k-fold CV vs. CV as taught by Ng.
It seems that the cubic spline used in proc expand is inadequate due to the Jagged nature of the cyclic pattern in the data.  With December observations revealing the peak storage levels after harvest, the September observations are usually the low point with steady usage occurring throughout the year.
Thanks.  This is basically what I'm looking for.
(+1) I have been kind of on the fence about whether this question should be closed as a duplicate or not. But, I think good answers can make very similar questions worth keeping. Having everything cross-linked helps.
Could you elaborate in your answer on what you mean by "well scaled"? For instance, suppose the data arise from calibration of an aqueous chemical measurement system designed for concentrations between $0$ and $0.000001$ moles/liter which can achieve a precision of, say, three significant digits. Your "relative error" would therefore be constantly zero except for obviously erroneous measurements. In light of this, how exactly would you rescale such data?
Based on your labelled dataset, you learn not only the transition matrix between states (stable/unstable) but also the observation probabilities per state. In other words, given [speed1=3, speed2=2, Pressure=1, acceleration=0.7], are you more likely to be in a stable or unstable state? You learn this based on MLE on your labelled set. Then, when it comes to labelling the unlabelled series, you know what your initial states is, and the transition matrices and the observations (and therefore likely) state at each time point.
Brilliant, thanks very much for the help.
Great :). I answered this question because I am studying the theory of SVD / PCA at the moment and wanted to get to grips with how it works with an example: your question was good timing. After working through all the matrix  calculations I was a bit surprised that it turned out to be an R problem -- so I'm glad you appreciated the matrices aspect of it too.
@ttnphns Thank you for the suggestion; I added few sentences to clarify the matter.
And a perfect example of how posting your data really, really helps get good answers!
About how many individual cells are subjected to each of the 6 treatments?
The key is not the absolute size of an estimated coefficient but how that compares with its standard error. The units of measurement of $\\beta_2$ here are (units of $Y$)/(units of $X$ squared) and one reason the estimate could appear small is just using particular units. In particular, comparing it with the estimate of $\\beta_1$ or $\\beta_3$ is unsound unless you consider differences in units. If any coefficient appears very small, it might be practical to change the units, although no statistical point is implied or raised thereby.
I believe that strategy 2 is also a step towards not choosing a more complex model if its superiority cannot be proven.
For what it's worth I ended up doing it both ways and in this particular, simple case the clusters ended up being the same regardless of if standardization was done or not.
Many, many people have used such approaches and they are extensively discussed in publications.  Check out the top hits in [this Google search](https://www.google.com/search?q=prior+elicitation+probability+density+decision+analysis).