tttjjj/elt_bench huggingface dataset:
Hint: the log PDF is given (up to a constant depending only on $\\alpha$) by $(2\\alpha-1)\\log(1-x)+(\\alpha-1)\\log(x).$  Can you algebraically rewrite that in the form $f(\\alpha)g(x) + h(x)$?  If so, then all the information about $\\alpha$ available through the data comes via $g(x)$ and you're well on your way to a solution.
Look up "monotone likelihood ratio" families; *e.g.,* in [this presentation](http://www.stat.wisc.edu/~shao/stat710/stat710-21.pdf).
The key point here is about "similarity". Do you have any ideas about what sort of similarity you want?
@roland - the "memoryless" property is a bit confusing because it is conditional, not absolute.  More work/effort would result in a higher rate parameter, and thus a smaller amount of time to finish a project.  the memorylessness would be assessed by looking at the distribution of the "late tasks".  Now why would a task be "late"?  One way I can think of is that the amount of work required to finish it was under-specified - adding in these additional tasks - seems like we are back to the start - so memorylessness may be a good property.
Standard error is a measure of the expected quality of your estimate while a standard deviation is a measure of variability.  The former decreases with n while the latter does not.
FYI, for this regression, p=.002.
Can you provide me with a tutorial/paper that mentions that? I'm not able to find any systematic tutorial/paper!
(+1) good resources. Especially Kohavi did a great job writing about the optimization he performed at amazon.
Dividing by $n$ happens (often) when you take the variance of an estimator.  The object you are considering, $\\sqrt{\\textrm{SSR}/(n-k)}$, is not the variance of an estimator.  Rather, it is an estimator of the standard deviation of the regression error.  It is analogous to $\\overline{X}$ not to $V(\\overline{X})$.  Closely related is the estimator of the variance of the regression error, $\\hat{\\sigma}^2_{OLS} = \\textrm{SSR}/(n-k)$.  When the regression error is distributed normal, the variance of this estimator is $V(\\hat{\\sigma}^2_{OLS})=2\\sigma^4/(n-k)$.  There, the $n$ in the denom is back!
@GaëlLaurans The outcome is campaign gross sales (similar to monthly sales, each campaign lasts 3 weeks) and orders. Each area consists of a variable number of postcodes (they may have varied slightly for each area from year to year). I have historical data by area: orders, sales, number of representatives, removals(leaving the company) and new representatives. Ideally I may get sales by area broken down by categories and get a sales forecast even by category, but getting an accurate prediction of sales and orders would suffice.
I think that testing for a null hypothesis that the data is not normal vs an alternative that it is means using a criteria for closeness to normality like is done with the chi-square goodness of fit test or the various tests that compare the empirical cdf such as the Kolmogorov-Smirnov test.  It is not the same as simply applying the test in the standard way because the null and alternatives are reversed. Of course in practice this has the same problems as a traditional hypothesis problem in that data really never follow a particular distribution exactly.
@Nick Cox: I agree.  I've edited my response to be more clear.  (And you're right, my commuter example doesn't work quite as well for walkers, bikers, or people on roller skates, etc.)
This reply does not appear to account for the supposition that $x_1$ and $x_2$ are accompanied with statements (or at least estimates--it's unclear which) about their errors.
It is difficult to understand this description.  Could you perhaps post a simple example of your dataset?  And please clarify what you mean by "represent": does that mean a logical file format, a graphical visualization, a statistical summary, or perhaps something else?  What is the intended purpose of this representation?
I am reading through the paper but am stuck, do you think you can elaborate on these calculations?
@Kirk I edited the above answer to show the guts of the spreadsheet.  This should be enough for you to recreate it.
The answer to the uniqueness question is both yes and no. It is "yes" in the sense that the eigenspaces and eigenvalues are mathematically well and uniquely defined. It is "no" in the senses that (a) there are multiple ways to *represent* those eigenspaces (even a normalized eigenvector can be negated and there are many choices of basis for degenerate eigenspaces) and (b) different *algorithms* may produce results that differ due to accumulation of floating point error in the calculations.
we have 4 classes, not really
Since distances cannot be negative, it's difficult to see how they could "average out" to zero!
You should edit your [original question](http://stats.stackexchange.com/q/33456/1036) with this new information.
Thank you for your answer, I guess a moving time-window should work just as well as my approaches. I can't get your example to run though, could you please give an example based on my sTRACE example so that I know exactly how to implement it?
How much heteroskedasticity is there (roughly, $\\max \\sigma_i / \\min \\sigma_i$ as a guess?  If it's less than about 3 or so, the efficiency loss is pretty small, or so I was taught as a rule of thumb...
@Ejs. Do you refer to the standardisation step? By the way, does my answer help ? Have I misunderstood the question ?
Thanks, but what do you mean by "planned" tests? In Howell's book, he contrasts "post hoc" comparisons to "prior" comparisons, and both are planned either after or before data collection. Also do you think Howell's is correct?
@JoFrwld: I like ribbons too, though I tend to reserve them for cases where the x-axis variable it truly numeric; my version of the "don't draw lines unless the x-axis variable is numeric" rule that I profess violating in my answer above :Op
Thanks.  Let’s say in my example, my R2_val is 0.565 and adjusted R2 val stays the same as 0.565 when I add a new variable due to the large same size.  But then how could I tell if adding the new variable is significant (ie. opening question)?
http://en.wikipedia.org/wiki/Coefficient_of_determination 
The adjusted R2 can be negative, and its value will always be less than or equal to that of R2 .  Adjusted R2 does not have the same interpretation as R2—while R2 is a measure of fit, adjusted R2 is instead a comparative measure of suitability of alternative nested sets of explanators.
(+1) I'd also recommend [scikits.statsmodels](http://statsmodels.sourceforge.net/) (eventually, with [pandas](http://pandas.sourceforge.net/) for data structure), rather than just `scipy`; but @whuber's remarks are worth thousand of words.
@os @Ralph A *histogram* represents probability (or frequency) by means of graphical *area,* whereas a "bar chart" (which is what we're looking at here) uses *length*.  Of course histograms can have blank spaces between bars, when there's no probability or frequency there!
Not sure if this applies in your context, but perhaps of some interest: [1](http://archive.nyu.edu/bitstream/2451/14775/1/SOR-97-12.pdf) [2](http://www3.stat.sinica.edu.tw/statistica/oldpdf/A19n223.pdf)
(+1) As usual, you've posted an interesting a well-formed question. Here is an **exercise** to help with some intuition: (i) Show that, unless $f(X)$ and $g(X)$ satisfy very special conditions, then the estimate you've proposed is biased. (ii) Consider the bias if you looked at something like ${n \\choose 2}^{-1} \\sum_{i < j} f(x_i) g(x_j)$. (iii) Work out the variance in the case of (ii). It will (likely) be a bit of a mess and involve moments of the form $\\mathbb E f^k(X) g^{4-k}(X)$ for $k \\in \\{1,2,3,4\\}$.
Not only if the null is true will *p* have any importance, but only then is it an accurate description of the probability of obtaining results as least as extreme. The null hypothetical scenario is useful as a simple basis for comparison in a very wide variety of applications where it is not true. If the null is true, you're just as likely to calculate a low *p* value as a high one. When the null is true (and other elements of analysis are handled properly), *p* values are distributed uniformly across replications.
Are you asking for anything beyond explication of the [set theory notation](http://www.rapidtables.com/math/symbols/Set_Symbols.htm)?
You can still use KL. a,b,c are independent but their joint distribution f(a,b,c)=f(a)f(b)f(c) is still a valid distribution. I would say the L2-Norm and the KL distance make sense. I am not so sure about the the T-test given the assumptions required for that.
@PeterFlom: let's hope that it is indeed declining.
It seems you need some since I'm not familiar with, so I hope someone else will give you a more satisfying answer; nevertheless getting/assuming distribution would be a inevitable starting point for such an analysis.
My understanding is that PCA, LDA (Scikit learn implementations) would require normalized data. 
More than that, I need to normalize the scale of each feature somehow, since values can vary immensely. (An example feature: Length can have values between 120 and  33,000, with most having under 9,000. The frequency of the appearance of a specific sequence motif on the other hand is binary, or ~0-2, while the frequency of a given amino acid-letter bigram is varies depending on the physical length of the underlying protein due to biophysical constraints..)
(I plan to extract variables + RF/SVM).
@StéphaneLaurent, ok so where exactly do they deduce it from?
would you say that a Fisher test is appropriate for this contingency table?
I can't describe the differences between a linear discriminant analysis, support vector machines, and a GLM LASSO in a way that makes sense to a doctor. So I built a logistic regression model for breast cancer risk prediction using a handful of carefully adjusted covariates. When presented, the doctors immediately launched into an enlightening discussion about their effect sizes. The discrimination of my "science" model was very comparable to more sophisticated ML techniques (overlapping 90% CIs for AUC based on bootstrap in validation sample), and I'm not the only one with such a case report!
I am confused what you are asking. As stated, your question is about "a good experimental design." But this doesn't seem right because you aren't in a position to design an experiment; the data is already collected and you are simply analyzing it. So then perhaps your question is about what is a good statistical model for handling these data. But then this is also confusing because you already wrote a model in your post which you say is the model you desire. So then it's not clear what you need from us. Could you give some hints about what a satisfactory answer to your question might look like
Sure but why to take $C_{k}$ as opposed to $C_{j}$?
I google searched time-based boxcar and came up with very little. Can you point us to a worked example?
@WillBeason i put in my whole code so that people could run it and see exactly what could be changed and where as before i have been told i didnt put enough information in
I also ask there for important clarification: why is your experiment two-stage? Why do you separate the samples (people) from the outcome (height, their amount of children) but not make your experiments 3-stage? You could easily sample a city at first, then sample an individual from that city and, finally, sample his/her height. You could divide the experiment into arbitrary amount of stages. Why two is right good?
Did you include an interaction term in the two-way ANOVA?  What contrast/dummy variable encoding was used?
I'm not aware of a good *general* internal evaluation measure. They all make very strong assumptions on what a cluster should look like, and none is general enough to make sense for arbitrary algorithms.
You can do external evaluation though, if you have labels.
I found this article provides a good strategy for quantile aggregation: http://metamarkets.com/2013/histograms/
All three of the answers below are very good (+1 to each). If you want another source of information, I discuss this topic here: [Estimating $b_1x_1+b_2x_2$ instead of $b_1x_1+b_2x_2+b_3x_3$](http://stats.stackexchange.com/questions/58709//58712#58712), & illustrate it w/ a simulation in `R`.
Although others might find enough here to respond to, I do not yet see that there is enough information to support a well-reasoned answer. Would it be possible to be a little more specific about the data you plan to combine?
Differencing can certainly lead negative autocorrelation in several ways - (i) outliers can do it, as you suggest, as can (ii) overidfferencing, but it can also be quite legitimate in a perfectly appropriate model, such as (iii) an IMA(1) model where (in the differenced series) the MA coefficient leads to negative serial correlation at lag one.
PCA is a good way to reduce dimensionality and does not make the stronger assumptions required of the Fourier analysis. PCA can be performed on any matrix of any size (up to certain computational limits), so if you have a set of parallel time series, you can arrange them in a matrix and proceed.  [Search our site](http://stats.stackexchange.com/questions/tagged/pca?sort=votes) for more information.
I would recommend first reading all the subsections under "Significance Tests" and the section "Multiple Comparisons" under "Analysis of Variance of Dallal's [Little Handbook of Statistical Practice](http://www.jerrydallal.com/LHSP/LHSP.HTM).
Don't think there's a definite answer: how different does one approach have to be from another before it forms a separate "school"? The frequentist school could be divided into one for design-based inference & one for model-based inference. The latter might well be divided into decision theorists (headed by say Wald) vs likelihoodlums (headed by say Sprott). Bayesians can choose to adhere to the subjective school or take their pick from among the objective schools using different types of reference prior.
*does the difference in sample size pose a problem?*  -- **No**
$R^2$ can also be negative. This happens when other methods are used to calculate the coefficients in linear regression model specification, and the usual $R^2$ formula is used. I think $R^2$ can also get negative if regression is without the intercept.
In an effort to meet the high standards and expectations of this site, I use simulation (as well as theoretical derivations and illustrations) in every answer that would benefit from it. A large fraction of my recent replies will include some form of simulation, especially almost any reply to a question with the [tag:r] tag.  For examples, you can look through them from the [search page](http://stats.stackexchange.com/search?tab=newest&q=user%3a919%20%2b[r]) if you're interested.
It seems to me model b makes more sense, since you could imagine fixed developmental effects occurring at particular ages in all mice, but occasion specific conditions (captured by the random intercept, grouped by time) may be more thought of as random perturbations.
"*the linear model that gives this very significant p-value does not explain much of the total variance in fish color and size.*" --- this may be explained by the large sample size.
Why do you `decompose` RevenueSeasonallyAdjusted again? You already removed the seasonal part, so you should have all you need. If you are asking why you are still getting the seaonal part, it because it is calculated by default in the `decompose` function. Type `decompose` in the console and you will see that it is always calculated either by `season <- x - trend` (for an addative decomposition) or `season <- x/trend` (for multiplicative decomposition)
Firstly, thank you so much for such a detailed response.
This is very broad. The time series aspect alone is broad enough. Focusing on day of the week alone is going to be a good idea if variations within weeks are the main component of variation: we can't say, but sounds unlikely. If you don't get a good answer, you may need to make this much more specific. Meanwhile, there are many, many questions on this site about prediction: did you look at any?
For testing linear hypotheses there is an R function linearHypothesis from the the car package.
Why call a model an algorithm? Blurs a key distinction!
Pretty much any website on Poker hand probabilities seems to agree. E.g. http://en.wikipedia.org/wiki/Poker_probability
$x$ is distributed according to, e.g., a Normal law with mean $\\mu$ and standard deviation $\\sigma$.  When, for example, $\\mu=0, \\sigma=1$, $f(0) = 0.3989...$.  When $\\mu=1, \\sigma=1$, $f(0) = 0.2419...$.  The value of $f(x)$ is conditional upon the values of $\\mu$ and $\\sigma$.  I think, BTW, we are using the word "conditional" in two slightly different ways; you are limiting it to "conditional upon some random event occurring", and I am using it to mean that or just "given", as in "$f(x)$ given (specific values of) $\\mu$ and $\\sigma$".
Thank you for recommending the book
There is a chi-square test function in Matlab Central File Exchange: http://www.mathworks.com/matlabcentral/fileexchange/4779
I want to get Wilks lambda for a priorly done LDA on the same data set and from what I have read so far there is no other easy way to get Wilk's Lambda in R for LDA - and the manova generated wilk's works just as good as a measure for that? I'd be happy if you would correct me!
Also, during the Bayesian Data Analysis session at the recent annual meeting of the Psychonomics Society, it was mentioned that Bayesian approaches can capture scenarios like this where you have two tasks but want to analyze the data simultaneously, letting the data from each task influence inference about the other. I'm working my way through the happy puppy Bayes book (http://www.indiana.edu/~kruschke/DoingBayesianDataAnalysis/), but still not to the point where I can describe how to achieve this analysis.
No it's not http://stats.stackexchange.com/a/12991/2488
Sorry about the delay responding - yes, $\\beta_{NGroups}$ is still just as interpretable, and in the same manner, as the other $\\beta_j$.  They are all subject to the caveat that the zero-sum constraint was imposed on them, but they are all still "real" $\\beta$s.
I second this. Also, there is quite a lot of suggestions for further reading which I think are useful in the book.
In the last paragraph, you state that your hypothesis was that students in the experimental group would perform better in the **pre-test** than students in the control group. Did you mean **post-test**?
I know that `Match` in the `Matching` package does not require you to supply a `Y` variable.
I have about 2.3 million records, out which about 88k are known and rest are unknown (that i want to predict) i will be dividing into 10 clusters, and have 8 predictor variables
@Glen_b In fact there were 5 options in the questionaire but I have combined for my analysis two of them in one -responsibility. These were taking part as a partner which means shared responsibility and dominating role which means taking complete responsibility for the desicions on the topic. So, you can consider them as four at the end. The reason to comine was that the there were very low frequences on the domination role like 2 or 3 people and the other reason was that combinig the last two options in one responsibilty gives me the chanse to comapre my results with other study.
@Dave, I didn't understood your idea: do you suggest to calculate $\\hat{\\theta}_{EST}$ using a saddle-point approximation? This is exactly MAP (maximal a-posteriori), no? What kind of bounds this may provide?
There is an interesting linguistic ambiguity coming to light here: in almost any other context, "the sample mean" would be taken in the sense of "the mean of *this* sample," whereas the text intends that it be understood in the sense of "a random variable determined by the *process* of obtaining a sample and computing its mean."  (However, the quotation was taken from a more extensive text which uses various locutions in an effort to avoid this ambiguity, such as the phrase "standard deviation for this procedure.")
Not the answer you seek, perhaps, but I'd say that the best normality test is a normal probability plot, i.e. a quantile-quantile plot of observed values versus normal quantiles. The Shapiro-Wilk test is indeed often commended, but it can't tell you exactly how your data differ from a normal. Often unimportant differences are flagged by the test, because they do qualify as significant for large sample sizes, and the opposite problem can also bite you.
Yes, I make sure to obey the time order. So having a test set (that obeys the time order) is out-of-time testing.  Makes sense.  I understand the difference with out-of-sample now.  Thank you @inzl .
Is this a duplicate of http://stats.stackexchange.com/questions/7948/when-is-it-ok-to-remove-the-intercept-in-lm
I've graphed the timings as you suggessted and discovered two neat peaks (cache hits and cache misses). Since cache hits are not very significant, I've removed them leaving two nice slanted gaussian peaks. I'll update my question with the data and the graph. Thank you.
I think this is pretty much it. The discrepancy for values > 36 might be algorithmic, but maybe `loess()` can be adjusted to produce predictions between -0.5 and 0.5 (as in the SAS plot) instead of -0.5 and 1.5 (as in the R plot) (all values just roughly inferred from the plots).
If you were to write "generalized linear" instead of "linear" and *parameters* instead of *predictors,* this would be correct.  (Many logistic regression models are *not* linear in the predictors.  For instance, no logistic regression with an interaction term will be linear in the prdictors.)
Thanks for the pointers to R packages/functions. I will take a look at them.
(+1) I suspect medical research stands nearly alone in human endeavors with regard to the need and desire to have more rats.
Your note about Bayes factors performing well in high dimensions, but BIC performing poorly, is a likely consequence of ignoring the determinant term that the BIC approximation makes.  BIC takes an approximation as $\\log|A_n|\\approx\\log|nA_1|=p\\log n+\\log|A_1|$ where $A_n$ is observed informative and $A_1$ is expected information.  When the dimension of the parameter space is large, $\\log|A_1|=O(1)$ is a poor approximation, especially if there is very large variation in the parameter dimension across models.
Fair enough. I am looking into "latent budget" or "end-member" analysis, which is a sort of reformulation of the latent class model that might be useful for clustering compositional data. Apparently they are "usually estimated by maximum likelihood under the assumption that the frequencies are generated by a product multinomial distribution" (from Ch 4 of Applied Latent Class Analysis - Hagenaars - which can still be found online http://ewd.zozlak.org/ZlozoneModeleSkalowaniaLiiniowego/HagenaarsMcCutcheon_2002_AppliedLatentClassAnalysis.pdf)
Please define what is "appropriate" for your task, your question is very unclear.
How can I add precisions in order to help you to answer the question ?
@mio: you don't necessarìly need to do it with every time point. Instead, you could e.g. randomly select 1000 time points an train models each up to the particular time point and test with the time points after the chosen one. That would be a time-series versiion of repeated/iterated set validation.
@whuber I consider this part of the question. There are some nice ways to define the covariance matrix which might be extensible to higher rank however. One such definition of the covariance is the expectation of the outer product of the random vector (x-mu) with itself.
Thank you David. I guess Hierarchical gives different results from kmeans because the definitions of distance between two clusters are not the same.  It may not be easy to determine which metric to use, and whether the variance should be included. It seems like different groups of people developed their own metrics on their own problem. The method just gave such problem a good result, yet it lacked theoretical support on the option of clustering methods.