tttjjj/elt_bench huggingface dataset:
I took the liberty of formatting your equations with the $\\LaTeX$ markup features that the site supports. Please ensure it still says what you want it to. Also, you say you "need to derive", is this for a class assignment? If so, please add the `[self-study]` tag & read its [wiki](http://stats.stackexchange.com/tags/self-study/info).
The uniform distribution has maximum entropy when the only information you know is the range. So in the example you give, if you know only that the distribution has support on [0,1] then the uniform distribution (which will by definition have $\\ \\mu=0.5, \\sigma=\\frac{1}{12}$) will have max entropy. But if you characterise the moments then the uniform distribution is no longer the MaxEnt solution.
It is often better to generalize the question to ask if a randomly selected observation from A tends to be larger than a randomly selected observation from B.  This is the essence of the highly efficient and robust Wilcoxon-Mann-Whitney two-sample rank test.
Little remark -- entropy uses logs, what can be a computational time issue.
@ htrahdis ,you are saying that replace mean or median to replace missing data but i think they will alter the behaviour of time series data.I have red few articles but I am not able to understand,one of them is: A missing values imputation method for time series data:an efficient method to investigate the health effects of sulphur dioxide levels.Please read this.
The distributional assumptions underlying the calculation of p-values include "no model selection based on the data". Therefore, only the original model really fulfills those assumptions. (However, it may not fulfill normality requirements, which in turn we only can assess by looking at the data... which is part of why NHST is problematic.)
That graph is *obviously* outdated. Either that or there is bias due to lack of surveyors available to sample in the [Gulf of Aden](http://en.wikipedia.org/wiki/Gulf_of_Aden)
Have you tried to specify your dummy variable with all the levels as a factor (use `as.factor`) and then entered it into `plm` as `factor(mydummy)`? That was how `Year` was entered into the model on p.17 of the package vignette: http://cran.r-project.org/web/packages/plm/vignettes/plm.pdf
@Vass. You will need to write your own Matlab function. It should be about 3 lines at most. For the version with squared x_i, you will need to derive the estimators and then write the corresponding function.
@whuber - 2D.  By path I mean there are sets of points that roughly resemble a movement over time to some final location, its a trace of location data, there will be one or more of these.  Also, maybe obvious, but the points consisting of a path aren't independent (time t will be close to time t+1), but I'm ignoring that here.  My goal is to determine whether the entire set of points looks uniform
@ttphns I don't think so; neither of those addresses the OP's question about getting a test for 2x2 in SPSS.
This question is off topic according to [a meta discussion](http://meta.stats.stackexchange.com/q/486/919).  However, many of the answers are on topic and useful.  Therefore I suggest we leave this thread open and I'm leaving the present comment in case the issue comes up with future questions.
@Dbr, glad to help.  Usually we think of the response variable as being on the vertical axis and the predictor variable on the horizontal axis.  With this setup, everything is vertical--regression is minimizing the vertical distances between the predictions and the response variable (SSE).  Likewise, the residual SD is a measure of vertical dispersion after having accounted for the predicted values.  Finally, R^2 is the ratio of the vertical dispersion of your predictions to the total vertical dispersion of your raw data.
@Mike Can you please register your account? You are quite an active user and we have to merge it manually over and over again. The process is quite easy; just visit http://stats.stackexchange.com/login
$u_i$ is the error term and $SST_x$ is the total sum of squares for $x$ (defined in the edit).
(1) The role of the indexes in this question is unclear: why are they there?  *E.g.,* for the given values of $X$ and $W$, with what frequencies should $b_0$ and $b_1$ *individually* be chosen? (2) Are you sampling with or without replacement? (3) Must the proportions in the $10,000$ runs be *exactly* the values in $W$ or should the *expectations* of the proportions be the values in $W$? Must successive samples ("runs") be independent or not?
Thanks for the quick reply! Actually I have 120 variables in my spss file. For each question, the person had to rate (from 1 to 7) six different Values (~ A principle, standard, or quality considered worthwhile or desirable).
If you want to compare the three matched samples you'll have to make the observations comparable, i.e. unify the scales in some way.
SPSS, since this is what Shantanu asked for ^^
No need to acknowledge me, assuming you don't disagree with the sentiment. If you'd rather it say something else, of course you should do so.
Like it, though it does remind me of http://en.wikipedia.org/wiki/Swiss_Toni
You might actually add an energy analysis as well.
@Glen_b then (1)is it illogical to find point estimator under pdf?(2)Would you explain me the matter $\\bar X_i$ are normal?(3)an estimator cannot estimate exactly parameter.there must be some error.Then why this $P[T=\\tau(\\theta)]=0$ is not also for the case of discrete?
Are you sure this is a squared CV and not the ratio of the sample variance to the sample mean?  For a Poisson distribution, this ratio should be close to 1, so it could make sense to plot it against the mean as a check for departures from "Poissonness": the plot should be approximately level.  Plotting the squared CV would instead produce a hyperbolic curve, which is not a good visual reference for exploring and understanding data.
Are the students different each semester? If so couldn't a difference in the capabilities of the students be a confounding factor? Are you comaring yourself with the same class given in the previous semester by a different professor.  Please make it clear what you are comparing.
@Jeromy, I don't think the work I cited would discourage people from using such methods to identify latent groups. I would say the point of the work is that you can't use such methods to solely infer the existence of groups, because you will always find groups, even in random data. It is up to more subjective interpretation whether those groups you find are real or are simply artifacts of the method. You could identify some logical theories that generate such processes and then see if the groups identified fit within those theories.
I'd try searching using the terms "logistic regression" and either "narrow distribution" or "rare event."
You are correct to be concerned.  A lot of inference is based on the assumption that we have the true model.  I've been running regressions a long time and I've never had the true model.  For my purposes it rarely makes sense to even think that one true model exists.  Instead, ask yourself what the goals of your modelling are (prediction in sample, prediction out of sample, estimating the average causal effect of x3, data summary, etc.) because your goals will indicate which modelling strategies are best.
Nonparametric statistics is a large field, but I'd guess *All of Nonparametric Statistics* by Larry Wassermann should be a reasonable starting point. I don't know if I would call it "advanced level" but he sketches the proofs of many of the results in the book. Topics include the bootstrap, smoothing techniques, density estimation, regression, and lots of other things. No nonparametric Bayes, however.
Could you describe a bit why having fast RTs in these tasks confounds your results?
How to convert from xts to ts if my original xts doesn't have exactly the same number of data per cycle?
For instance if I have daily data every weekday => every year there is not the same number of daily data... is there a best practice around that?
"we also notice that there are some points whose Euclidean distances are quite small, but whose cophenetic distance is quite large " I think this an important point as it seems to suggest should use the original distance function, in this case euclidean instead of cophenetic if the euclidean distance function offers a more accurate value for how similar two users are.
OK - i see what's happening (at least I think I do.) It's treating this as a categorical variable instead of continuous. How do I tell R they're continuous? (I'm a new user so sorry for novice questions!)
If I understand correctly, you now have independent observations and KW ANOVA followed by any suitable nonparametric technique for pairwise comparisons should be ok. However, what is your classification factor?
You're right, I fixed the degrees of freedom. I'll look into ordinal logistic regression.
@Macro, I agree about blind use of the CLT. But requiring Normal data for confidence intervals and $p$-values - and not just requiring light-tailed data - is considerable overkill, and encourages e.g. inverse-Normal transforms, that just make interpretation of the output harder. The balance to strike is between answering the right question approximately versus the wrong one precisely; if the right one involves comparison of population means, use of OLS is the right way to go.
Don't categorize it. Why would you do that? It throws away data.
Do you have 4 pre-post pairings, or just 1 4 years apart? What do you mean by "scores divided by question... "? Can you post an excerpt of your data?
Have you looked into survival analysis?
Could you post a plot of the data?
@jbowman I want the error distribution to depend on $X$, so that doesn't give us what we want immediately. But yes, If I tweak the idea I could do that if I was willing to impose symmetry and unimodality for all realizations of $X$. Actually, that might be a reasonable solution if I was worried about heteroscadasticity but otherwise okay with symmetry and unimodality. I could drop it down to "just" symmetry by setting up the distribution for $\\eta$ as above and setting $\\epsilon_i = \\eta_{i1} - \\eta_{i2}$.
Classify as in supervised learning a map to a given target class label? Or did you mean clustering?
@Glen_b I added some of the information you needed and changed the variable (see the new results above). The variable is a time series with 380 consecutive observations, measured from t = 1 to t = 380. I cannot make a plot because I don't know how to do it (also I need to hand in my report on these measures very soon so I don't have much time left). So if the hypothesis is normally distributed data, both of these results indicate the variable stems from a not normally distributed population, right? Btw, the mean is 4.67 and the sd 2.89 if that helps.
+1 to @Glen_b who highlighted this already, I just wanted to make sure the proper economics term was used.
Any intro to stats textbook will cover how to run a basic chi-squared test. If you want a more specialized treatment, the first 2 chapters of Agresti's [Introduction to Categorical Data Analysis](http://www.amazon.com/Introduction-Categorical-Analysis-Probability-Statistics/dp/0471226181) will do the trick.
@chl, OP uses postscript, so the plotting must end with dev.off(). This resets all the par settings. The same holds for usual plots. If dev.off is called between plot's, settings are reseted.
I think this should be left open. It's about choosing a statistical program, not programming.
Although the population mean may equal the population median, the sample mean and sample median are not always the same and do not have the same sampling distributions.
**Hint:** Just replace $a$ by $a+1$ in $f(x)$.
Note that there is an ongoing discussion on meta about migrating threads to SO, @Penguin_Knight. In light of what has been discussed so far, I suspect this question might not fit well on SO. It might need to go to SPSS tech support or an SPSS forum.
thank you for your answer, I was trying to avoid folding it again myself since my comp is slow. But seems like there's no shortcut then!
wonder if [this](http://glmm.wikidot.com/faq) partly answers your question.
@Budric Well, that's not what they are doing!  Obviously they are assuming a common covariance structure around each center.  This proves your original interpretation is correct.  But why does a common covariance "make no sense"?  It's a good place to start.  You can do *post hoc* tests to see whether it's a reasonable assumption.  If it's not, then it sounds like you might need to generalize the X-means approach yourself :-).
@Nesp Since facts are facts, do you ever use $(4\\pi/3)r^3$ as the volume of an actual sphere (say a ball bearing) of radius $r$ and ever worry that the surface is not perfectly smooth as the surface of a mathematical sphere must be, and that the division into arbitrarily small pieces that is needed for the derivation of $(4\\pi/3)r^3$ is inapplicable to an actual sphere that exists in real life as opposed to the mathematical world? Or is $(4\\pi/3)r^3$ close enough not just for gummint purposes but for yours as well?
That's a very good answer, thank you a lot. Indeed I was not mainly interested in bias, but thought of bias as an example of a criterion which could be used to prefer one approach over the other. So, I was generally interested in knowing if there are theoretical reasons to rather prefer the ML estimate, since as you say intuitively it makes more sense to use all the information available.
Sure it makes sense to differentiate: this log likelihood is differentiable within the interior of the region where it is defined. But, as all Calculus students are admonished--because this is so easily forgotten in the excitement of learning about the power of differentiation--extrema can occur at the boundaries of domains of functions, not just at the critical points, and so the boundary must be tested too.
@whuber The task says we are looking for probability of rolling 2,3,5,7 and 11, so the person who wrote this task might not assume number one as a prime number. Can you at least tell me whether my procedure of solving this is correct?
What are your results? Are you sure that you can classify your dataset? 800 items is not that small and you should be able to do it.
When you say "will best take into account", what do you mean by "best"? E.g. do you want the variables to be weighted equally, or do you want to give some variables more weight than others? Or a different way to put it, what would be the purpose of the score you'd like to get?
The idea is that what you are interested in is to find or estimate the population parameters, when you know population mean than use that value to estimate population value, if you do not, than use sample mean which is an estimation of population mean.
I mean, what is the actual meaning of the numbers - what do they represent in your original application.
One other thing: i'm not sure that a proper analytical solution to this problem even exists. I've looked at some additional literature, but this problem is elegantly looked over everywhere. I've also noticed that yucel&demirtas (in the article i mentioned, page 798) write:“These multiply imputed datasets were used to estimate the model […] using the R package lme4 leading to 10 sets of (beta, se(beta)), (sigma_b, se(sigma_b)) which were then combined using the MI combining rules defined by Rubin.”
*landslides induced by 13 typhoon events.

the wakeby function can be shown as [link](http://www.itl.nist.gov/div898/software/dataplot/refman2/auxillar/wakpdf.htm) _italic_ **bold** `code`

[link](http://www.johndcook.com/inverse_gamma.pdf) _italic_ **bold** `code`
Hi Peter, thanks for sharing your thoughts. Unfortunately I dont have the dataset the means are based on; thus making the comparison that bit more difficult. Any more ideas what one might do to test if the means differ?
As @fileunderwater points out, my question concerns repeat measurements of the same variable (that have been normalized). I have two populations, each made of $n$ individuals. I'd like to know if individuals within each population are the "same", and also if there are any significant differences between the populations (see my response to fileunderwater below for more detail). If needed, some sample data on a minimal working set could be provided.
1) DA implies classes; what are classes in your question. 2) DA is multivariate analysis; is there multivariate data in your question?
If `Y` is your dependent variable, compute a new variable `Y2` which equals `Y` in your model sample and `sysmis` in your hold out sample. Run the regression on `Y2` (without split files), and then save the predicted values. These will be the predicted values from the model sample, extended to the hold out sample.
Thanks Rob.  That was the first paper I read when trying to find an existing package that implemented the stat space model this way.  It didn't seem like any of them worked similar to my textbook after reading the underlying documentation, but maybe I missed something.  After reviewing the paper that I am trying to replicate again, it seems like I might need to write something myself regardless.
$d(X,Y)=\\mathbb{E}\\left[ \\sup_t |X(t)-Y(t)| \\right]$
Thank you, that was very useful!
Right, of course, sorry about that. You might try the resource that I added in an edit.
1) Could I get a quick description (or a link) of what is an autocorrelation and how it leads to bias? 3) So any statistical test is inappropriate here because of census? How could I answer my question then? 4) What test are you talking about: regression or chi-squared? In the latter I focused on the last data-point only -- the most recent census.
If you define your CI's as having *at least* coverage $1-\\alpha$m everything works. It's often impossible to get exactly $1-\\alpha$ (consider a binomial, for example - the sample is discrete so the distribution of possible values that the MLE of the continuous parameter takes is discrete).
oh, that's amazing nice. I'm going to test it, but before I find out, you deserve at least upvote ;-) Thanks
Very neat answer. Thanks!
Just say that the model is saturated, and goodness of fit tests are not available/not applicable.
@whuber, I think it is possible to use recursion here. Maybe it is even possible to get analytic solution. I would be pretty hopeful that there is some combinatorical trick which provides it.
"Is sampling a proper technique here?" -- could you clarify what you mean by this? Also, why do you want to use Mann-Whitney in the first place? ... Note: If you're going to try taking the means of subsamples of your data then you have to assume your data is either interval-scaled or ratio-scaled (i.e. you're implicitly assuming that your data is *more* than merely **ordinal-scaled**).
Thanks Erik, that's really helpful. I has a hunch there was a problem but couldn't work out the solution. I'm now buried in "Biostatistics - the bare essentials" in the chapter on Repeated Measure ANOVA. Hopefully this will be the sophisticated solution I was after. I'm trying to see whether any of the treatments enhance wound healing in comparison to no treatment and if so which treatment is significantly better than the others so I suspect the Repeated Measure ANOVA is the way forward, I just need to get my head round it. Thanks so much for your response I really appreciate it! Katie
If you are worried about outliers or influential points in your original or transformed model, it is worthwhile to perform a proper set of regression diagnostics. If you are doing your analysis in R you may find http://www.gmw.rug.nl/~boomsma/apstatdata/Regrdiag_R.pdf helpful, although it doesn't cover all diagnostic approaches. What is the reason you chose to perform a log transformation - was there a known theoretical result, or a pattern in your data, that made you suspect this is the correct functional form?
@mark999 LaTeX simply is not an option for collaborating with non-mathematical people.  As Gavin Simpson explains, high-quality graphics can be incorporated within Word documents by choosing appropriate formats.
Have you looked into using [coherence](http://en.wikipedia.org/wiki/Coherence_(signal_processing)) (in the signal processing sense, not the statistical), a cross-spectral measure?  I'm not sure if that's the type of measurement that you're looking for.
A condition number of 6 with 600 points is not a problem at all.  Collinearity should be near the bottom of your list of things to deal with for these data.  Higher up on the list would be which variables to include, outliers, influence, heteroscedasticity, goodness-of-fit, interactions, and even higher-order interactions. Monitor collinearity only to make sure results are not affected by numerical instability.
Glad I could help. ;)
@whuber. Via the second argument of the `autocode` function you have to specify the number of intervals you want to create, rather than the width of the intervals. Thus, here you would have to type `gen i = autocode(x, 28, 0, 280000)` or `gen i = autocode(x, 27, 10000, 280000)`, depending on whether you start at 0 or 10000.
Note that if $\\sigma^2$ doesn't vary across $i$, it won't affect the maximization, so the weighted least squares results would be the same as the log-likelihood results.
Thank you for the references!  Very nice review.
Thank you for this idea! I will give this a shot within the next couple of days. And thanks @ssdecontrol for the note on overdispersion.
@Szabolcs: It helps to smooth the data before computing their Fourier Transform.  The Welch method, which is a running weighted average (with weights that linearly decrease with distance) is described in *Numerical Recipes*.
No I don't think this is correct.  Look at it as 15 simulations of 10 possible groups (number of items is unknown).  In the first simulation, almost all items were assigned to 7.  In the second simulation, almost all items were assigned to 6 and 9.  And so on...
Looks like a great contribution to a topic that had many of us puzzled for a quite a while.
I think I might post a more complete description of the problem I'm trying to solve as a separate question, since I still feel like my whole approach is almost certainly overly conservative.
I'd say that this shows that the mutants don't differ when age and sex are controlled for.
@ttnphns: Thanks for your reply. I am new to PCA. Would you please elaborate your last comment? Will I get any benefit, if I apply PCA to my data-set?
Reporting the two different effect sizes, or confidence intervals for the two different effect sizes, would be a good idea.  It would be easier to interpret this if the outcome in each of your two datasets was the same (is it?).
Do you know if that formula uses kurtosis or excess kurtosis (i.e. does Normal dist have 4 or 0)?
Thanks for the elaborate answer. However I am still not sure what would have to be my primary indicator to which method to use. How can I determine which theoretical distribution my outcome is most inclined to?

Does it help if I say the mean is 2.34 and the variance is 18.39. I also added a barplot of my outcome variable to the initial question.
Indeed, to see it more directly, note that for this $X$, then the joint probability must factor as $P(X=x,Y=y) = P(Y=y)$ if $X=\\hat{x}$ for the singleton $\\hat{x}$ receiving all the probability mass, and zero otherwise, which is the same as $P(X=x,Y=y) = P(X=x)P(Y=y)$. Plugging this into the formula for MI you'll see it's zero. More intuitively, the mutual information between any random object $Y$ and any deterministic object $X$ needs to be 0 .. how can a deterministic quantity tell you something about the entropy of $Y$?
I'm glad you found it useful.  You might consider accepting my answer (or Glen's if you prefer that one).
thanks for your helpful feedback
@Peter Was it really "Hiring a physician after the data ..." or should "statistician" be in there somewhere?