tttjjj/elt_bench huggingface dataset:
I have been told that imputing the outcome leads to Monte Carlo error. Will try to find a link later. Don't forget that you need to make sure to include the outcome in the imputation models for the covariates.
Thanks. I guess it all boils down to an issue of interpretation, and whether it makes sense to conceptualise the interactions as ratios of ratios of rates.
Even a weighted sum of i.i.d. variates with finite mean and variance tends towards Normality as long as, loosely speaking, the maximum weight (when the weights sum to 1 regardless of sample size) goes to zero as the sample size goes to infinity.
@whuber: I add some small example for question, hope it helps me explain this question.
Thanks for this good perspective.
Thank you so much all of you! I am really grateful! :D
(1) It appears that the cosine similarity does not entirely capture your sense of "difference" among patterns. (2) How these data are compared will depend on how the "scores" are measured or computed.  What do the raw data look like and how are scores derived from them?
Can you clarify a bit more?  You have "program" in parentheses after "first questionnaire". Are these terms the same?  What are the first and second programs?  Is either the "intervention program" you mention?  Also, what correlational analysis are asking about?
I don't really know about using wavelet coherence. One of the vectors in the plot is an extracted scale from a continuous wavelet transform and the other vector is a measured series. Therefore I don't know if wavelet coherence is useful here.
I need to ask for a clarification, which is entirely due to my lack of knowledge in this area. If I want to see whether the change in # of successes after the switchpoint (316 successes, compared to 127 before, to use the above example) is significant, then wouldn't the binomial test take: `x = 317`, `n=79076`, and `p=443/114541` (in other words: x = # of successes after switchpoint, n = # of trials after switchpoint, p = hypothesized probability of success across the whole time period, i.e. `total successes / total trials`)?
1/ I tried different values of learning rate, but it seems that it's not so crucial ; 2/ That's what I'm asking: I tried it separately and nothing worked, I'm curious why it worked out previous time when they were together 3/ About the variable: most of them are continuous, but there are few discrete having values e.g. 0,..,5. I scaled all of them.
@Srikant, more-or-less correct; individuals that conduct the polling *may* remember who said what, but there is no physical record between response and respondent. Polling is more continuous than punctual: roughly 6 interviews are conducted per week though this can vary wildly as external factors that disrupt access to the group prohibit a more consistent sampling. We have 3 years of data. Since I arrived I've been enforcing putting dates on the results, though many old records are un-dated.
Do you mean externally studentized residuals? These are calculated by deleting one observation, which would explain a loss of 1 degree of freedom.
`library(fortunes)` `fortune(106)`
It was not tailor for GA and its definition led to the binary output. 1/0 refers to existing or missing different atoms or bonds in the compound. You can generate other fingerprints that shows the repetation or concentraton of a specific characteristic in the compound. So, you end up with a real value. You may, also, check the following link for generating other real values: http://www.rdkit.org/docs/GettingStartedInPython.html#descriptor-calculation
yeah probit regression might work here
I am afraid that does not clarify what you would like to do.  What is the relationship between a "user" and the movie information given in the question?  What is the *meaning* of the intended "similarity"?
@Aksakal I would probably say not.  In fact, I would like to guess that the trend will decrease leading to a relatively normal distribution, but the sample I have says nothing about this, it is just my intuition.  Is there a way to treat this like a normal distribution despite the lack of data?
Is this homework? If so, please use the Homework tag.
thanks for change the title Jeromy. better this way.
As you mentioned, the joint pdf of two random variables is $f_{X,Y}(x,y)=f_X(x)f_Y(y)$ if they are independent. So if $X,Y\\overset{iid}{\\sim}\\text{Geometric}(p)$ then $f_{X,Y}(x,y) = (1-p)^{x-1}p(1-p)^{y-1}p = (1-p)^{x+y-2}p^2$
Also, it is slightly tedious, but not terribly difficult to fit BT and related models "manually" in $R$.
(+1) Good advice in light of the OP's comment to @Peter Flom's response.
Welcome to the site, @jokel. This Q was flagged as being a little sparse. Could you expand on your question a little bit? EG, it's common in MA to make a [funnel plot](http://en.wikipedia.org/wiki/Funnel_plot) to check for publication bias, are you asking if / how that could be done w/ single case studies?
I just changed the magnitudes in the regression equation. It is my mistake. Sorry. By the way, how you coded factors is not correct, I believe. You need to use (-1, 1) not (0, 1). Because in order to get predicted values or response values for each run (in this case, it's 16) you need to multiply every row in the sign table with the corresponding β coefficients of active effects in the regression equation and then, to add error that is normally distributed.
The [Kruskal-Wallis](http://en.wikipedia.org/wiki/Kruskal%E2%80%93Wallis_one-way_analysis_of_variance) test is the appropriate generalization of the Mann-Whitney test to consider when comparing more than two groups. With Likert scales and the possibility of many ties, the results of either test have to be evaluated critically because it's likely their assumptions will be (strongly) violated.
Re the edit (+1): The nonparametric prediction interval can be understood as a permutation test under the iid null hypothesis. In that case, there is only a $2/31\\approx 6.45$% chance that the 31st value would be the largest or the smallest of all 31 values. The associated *test* concludes that the 31st value is inconsistent with the other 30 when it is either smallest or largest. This test has a size (in the [usual sense](http://en.wikipedia.org/wiki/Size_%28statistics%29)) of $6.45$%. It's the smallest size one can achieve for a (two-sided) test with 30 data values.
"Better" kind of depends on your criteria. I think that randomization test is a great way to do it (though I'd use a larger sample, 10 times as big or more). Speaking for myself, to be honest, with those sample sizes, unless it was *really* badly skew or *really* heavy tailed (and in both those cases I'd question the value of using the mean anyway), I'd just do a one-sample t-test against the mean of the population (or a two-sample-t if I decided to treat it as a big sample) - the central limit theorem and Slutsky's theorem together should give you a nice normal distribution for the statistic.
Good point, Azeem.  On the other hand, you wouldn't want to weight visual fit too highly, per the concern of over-fitting the data.  The beauty of BIC and AIC is they weight models by the number of parameters, and so can deem a model with a worse fit (discerned visually), but fewer parameters, the more likely source of the data.
(-1) This answer was looking good until it completely misinterpreted the p-value and therefore came to the wrong conclusion.  (It also obtained an incorrect p-value.)
I think that in ANOVA-type interaction diagrams the estimated marginal means would be plotted. These are equivalent to the means of the predicted (conditional) means. Not sure if this helps.
P(H) + P(no H) = 1. So does P(D) + P(no D). However, here there are 4 possibilities: H only, D only, H+D, and neither. Those 4 together have to = 1.
Here's some criticism of that paper in 4 from the guy who runs eager eyes:
http://junkcharts.typepad.com/junk_charts/2010/05/8-red-flags-about-the-useful-chartjunk-paper.html

http://junkcharts.typepad.com/junk_charts/2010/05/8-red-flags-about-the-useful-chartjunk-paper.html
Thanks @Procrastinator. As you can see I answer my own question. Starting a bounty when you have a cold is not a good idea.
@Hugo See my response with `pch`.
where you see the mu = 1 and the sigma = 0 is where the function f() is defined. The way it is written the 0 and the 1 are just the default values. The default values are ignored because the simple.sim passes the two fives in as mu and sigma
Thank you! Sorry but I'm pretty new with this
Have you considered beta regression?
Smoothness and differentiability?  It achieves the "interpolation" between points, but in a more probabilistic manner than direct interpolation.  If you don't need a smooth c.d.f. then the bootstrapped empirical CDF sounds as good an approach as any other.
@Gaeten - AIC & BIC are more general than F-tests and adjusted R-squared which are usually limited to models fit by least squares. AIC & BIC can be used for any ,model where the likelihood can be calculated and degrees of freedom can be known (or estimated).
Great to hear. You might consider posting a brieft pointer for other SPSS users if you do get it running. Good luck.
Please see the comments to the OP. :)
(+1) What is the basis for your winsorization rule $(x_j−median(x_j))/mad(x_j)<3$, $j={1,2}$? Also you told me we have to use both univariate and bivariate winsorization but that's not what you decided to do?
Tremendous explination, I will have to thank you again for the additions you've made. Very informative indeed.
This is more a matter of convention than of logic. In practice, at least in my experience, loss functions are functions with zero or positive value. I don't think anything stops anyone calling something a loss function when that something might be negative. It's just like thinking that you want to minimise expenditure, but if you get some net income that is better yet than zero expenditure.
But I am still looking for some rules of thumb to tell effect of A is greater in B(1) or B(2). That could facilitate my interpretation.
@Scortchi Would you like to elaborate that for a second answer to the question?
example: get a list of  flickr images with the according metadata. someone sorts them in to two groups (the ones he likes and the one hi does not like so much). then i want to find out if there is possible metadata upon which i could have guessed his choice.
Just a code comment. Use broadcasting not loops. E.g. `adjusted_samples = samples - samples.mean(axis=0)` is the same as using a loop.
You can get the standard error for the parameters of the MLE via the Hessian but not a standard error for the MLE.
The general procedure is illustrated in the answer at http://stats.stackexchange.com/questions/32620/dividing-a-uniform-by-a-normal-random-variable-whats-the-distribution.
You could try numerical convolution, or simulation.
Thanks MBQ!  As I mentioned to Henry, I can already table out all of the different possibilities using a macro, I should be able to create the tree that you suggest.
@Sebastian That's on the right track.  Going further, using about the same space as the triptychs that are shown, one would be able post a stack of *scatterplots* (of GOMOS vs. LIDAR), one for each kilometer or two.  Or you could post a single scatterplot with the dots shaded by altitude.  Or, on a single set of axes, you could plot a *curve* of GOMOS vs. LIDAR (parameterized by altitude) for each monitoring station.  Any of these would be rich and revealing.
I agree with Tukey although you might think from my answer to this thread that I am one of the "ossified" statisticians he challenges. The problem is that you have taken his quotation out of context. Tukey is specifically addressing the question of how to evaluate the properties of procedures "when the hypotheses on which they are customarily developed do not hold." This in no way changes the *definitions* of things like parameters, estimators, and estimands. In particular, a parameter still is *not* "whatever we are estimating."
(+1) It is also worth noting that testing individual polynomial terms can be problematic because, except in special cases (like when the predictor,$X$, is symmetric around zero, ${\\rm cor}(X,X^2)=0$), they are (often very highly) collinear with the other polynomial terms.
Hi Ronaf.  This came up when I learned about Cramer's V statistic for contingency tables.  I was wondering what the CI for it are, and found a way to get them - but hoped to validate my results using bootstrapping.  Cheers, Tal
@Scortchi You know the graphs they show you of a regression line with the normal graph extended outward towards you to show, that is where I got the bread slicing idea.  If, each $x_i$ is normal,then why isn't the variablity of the regression line normal because of that?
@whuber I think the OP is looking for the use of PCA as a way to account and adjust for population stratification when modeling a given outcome (continuous phenotype or case/control studies) and DNA markers (SNP). I gave a reference here: http://stats.stackexchange.com/questions/1708/variation-in-pca-weights/2848.
I should be careful here: correlation may, or may not be a problem depending on the situation. It is possible that highly correlated observations will cause issues when exploring the optimization space (if 1000 observations are all similar then you might push the wrong direction). In cases like that you can filter out observations (say only take 1 out of every 100) or do mini-batch after a large number of observations. Or you could decide it just doesn't matter.
Nope, that's the key advantage of AIC over e.g. backward selection. As long as the data are the same you can compare AIC between different models even if they aren't nested. You do have to be careful that the software is calculating likelihoods without leaving out constants, but within a single function you can compare non-nested models with ease.
+1 for Davidson & McKinnon. I really appreciate its clarity.
+1 for the nice answer and the link to the article. Did not know that it is feasible to use such approximations. I would have resorted to numerical integration without even trying to get the closed form solution.
I see, so you're saying the probability is $\\frac{k}{\\binom{n+1}{2}-\\binom{k}{2}}$? I guess I will have to prove it by induction by first looking at the urn with only the balls in $\\{k\\ldots n\\}$ and then proceed by adding balls. Thank you very much!
Did some simple regressions to predict returns which should work ok, compared to predicting a price-value. Besides predicting return have to predict a large enough return to beat spread&commish etc. Would be cool to talk more, couldnt find your mail in your profile. Mine is in the pastie I posted in the previous comment.
@Aaron Weights are not always defined to sum to unity, as exemplified by the weights given in this question!
What's your sample size? Sounds like you had as many df's used as you had. If so, then the second model is also overfit.
Not exactly.  AIC is a criterion to minimize when selecting between models. What I was referring to is really a subjective assessment of the change in fit or the change in the AIC if you want to look at it that way.
The following Qs are strongly related & will be of interest: [How to define a distribution such that draws from it correlate with a draw from another pre-specified distribution?](http://stats.stackexchange.com/questions/13382/) & [Generate a random variable with a defined correlation to an existing variable](http://stats.stackexchange.com/questions/15011/).
@Daniel Okay. I've tried to update your question to make this clearer; feel free to make it even clearer; at first I thought you were comparing individual items; It's a very different question if you are comparing mean proportions.
nice explanation and plot
The bootstrap is almost always good for getting standard errors in complex modeling situations.  I am not as familiar with the bootstrap literature on this topic.  I do know that naive application of the bootstrap does not work on the extreme order statistics.  I am not sure what difficulties this might impose on this particular problem.
I would like to invite you to draw those diagrams and include them in your answer: it would make it (and the one you reference) clearer and more accessible.
+1. I agree with you both that thresholding a continuous variable is often a bad thing but in some situations it can simplify to the point of being beneficial. E.g., if your continuous measurement shows difficult-to-model heteroskedasticity that can bias your inference, it can be very beneficial to categorize the data instead. In this way, you're at least getting unbiased inference about the categories. Another example is random processes - in practicality it is often much easier to model a discrete state process than a continuous state one - even in the simple case of a Markov chain.
Andrew Gelman (who is not an economist), lists 5 distinct definitions in his ANOVA paper: http://www.stat.columbia.edu/~gelman/research/published/banova7.pdf.
You presumably mean $\\sigma_i^2$ rather than $\\sigma_i$ for the variance
@Yoel I have updated my post and included some references. However, I am not sure if this is really what you are looking for...
At the very least, it looks like I could use it as a model a viable 2d alternative, although I don't want a minimum spanning tree for within-clusters representation. I want a between-clusters representation for the protoclust linkage outcome. Maybe, since I only have 6 clusters, I'll cobble together a subset data frame and make creative use of the "lines" command, especially, since in a protocluster solution, the "center" is always a specific leaf node.
Thank you for answer! Yes, I am not worrying about precision at this moment. Just want to  know the accuracy, "deviation of the forecast from actuals". I am not worrying about running few models calculate forecast errors and pick best model. My only aim is find out the deviation between actual and forecasted values. Our model is constant here. Irrespective of our model is good or bad for the data set, we just need the number of deviation. This question is not related to parameter fine tuning (or) model selection. I hope now I am clear. Please let me know if anything missing.
What else were you looking for? I feel my two answers, while not 100% satisfactory, do provide good intuition and starting points.
@Nick: Indeed I mean Bias.
@NickCox thanks for the comments, I'll see what I can do.
@gui11aume: you mean something like this?: http://dirk.eddelbuettel.com/code/rcpp.html
You can look at my answers, whenever they contain Monte Carlo simulations, they were done on Mac OS X using [R](http://www.r-project.org).
Isn't parsimony a good thing in itself ?
The way I see it, a model with variables that add little or no explanatory power in a clinical sense, is worse than a smaller model without those variables, even if those variables are significant in a statistical sense
Analysis of deviance is applied to generalized linear models (McCullagh and Nelder, 1989). `anova()` deals with both types of analyses, see the help page. The deviance is -2 times the log-likelihood ratio (http://en.wikipedia.org/wiki/Deviance_(statistics))
For googling, it might help that option 2 is often called the "model-based bootstrap" or "model-based resampling" with modifications like the "wild bootstrap". It is typically discussed in the context of bootstrapping linear models.
The very starting point may be the concept or intuition of cross-product (covariance is only the extension of it). If we have two series of numbers X and Y of the same length, and we define the summed cross-product as Sum(Xi*Yi), then it is maximized if both series were sorted in the same order, and is minimized if one series were sorted ascendingly and the other descendingly.
That sounds like two hypotheses, and thus two tests: one for the (alternative) hypothesis that female instructors achieve better results than male instructors, and one for the (alternative) hypothesis that female students benefit more from female instructor than male students. Which hypothesis do you want to test?
for this purpose you can see the stringr package of H. Wickam
Then I believe you ought to apply the specialized tools of SNA.  I did some of this, but that was a decade ago and I am not up to date on the latest. But I know there are packages (in R and standalone) to do SNA.
Illustrations elsewhere on SE address the first two bullet items: http://gis.stackexchange.com/a/20428; http://gis.stackexchange.com/a/15567
In [this e-mail](https://stat.ethz.ch/pipermail/r-sig-mixed-models/2010q4/004969.html) Ben Bolker translated this function to "lmer-land".
Because the penalty in AIC does not vary with the sample size I would have thought it would be subject to the same problem as ordinary significance testing - the saturated model will eventually be favoured if the sample is made big enough.
@Erik I can always change the age groups of users to be as 1,2,3,4 where in 1 determines the age-group 0-5, 2 determines the age group 5-15 etc. The hypothesis would be that people in age group 4 (25-35) perform better than people in other age group ?
@gung, thx, hope the abstract does the job.
Thanks a lot. I actually have some estimate of the correlation (we replicated the original study and used our correlation). From this I ended up using the mean of the sd (i.e., totally ignoring the $n$ in the formula) as it was indeed completely balanced. With means, sds and correlation you can make a power analysis using the fabulous G*Power 3: http://www.psycho.uni-duesseldorf.de/abteilungen/aap/gpower3
Are those generic keywords or are they specific to a certain domain? In the latter case maybe specific databases do already exist!
Maybe it's just me but I'm confused by your description of the basic problem in the first paragraph, so I'm unable to address the questions that follow.
@Stephen, I'm outside my domain here because I haven't used this method.  That said, the relationship between x and the outcome is modeled as linear (though, like logit regression, the outcome is probably transformed so that relationship can be modeled as linear).  It's just like OLS regression, the effect of a 1 unit decrease in X is just the opposite of a 1 unit increase.  This is called symmetry and it's an implicit assumption that most never discuss.  I actually have a paper under review now looking at precisely this issue.
+1. I like your "optimisation is the root of all evil in statistics" message...
No doubt @PeterFlom is right (I don't know much about social network analysis). From a standard cluster analysis perspective, you can consider your data matrix as a *similarity* matrix; you simply need to convert it into a *distance* matrix. Given that the similarity of each node w/ itself is 2, the simplest way would be to subtract each value from 2 (the diagonals become 0's). Then there are a number of clustering algorithms that operate over a distance matrix / don't need a raw data matrix.
@Gaël Laurans: Reusing the pilot data introduces dependency among data, so in theory the whole procedure is invalidated. Of course in practice one might still account for that and compensate, but you must know where, how and how much, and afaik already for the latter any dependence estimate will itself be also biased, so the whole procedure is not as trivial as it might seem at first.
@kleptog, the result, though maybe not explicitly stated as such, is well-known via the characterizations of a Poisson process. Section 2.6 of Durrett, 3rd ed. quickly covers the Poisson process, but leaves this result as an exercise at the end. Resnick's book on stochastic processes has a more complete and general treatment. You can also just derive it directly by transformation of variables since $X_i$ and $\\sum_{j\\geq 2} X_j$ are independent and have known distributions. If you want, I can post the details in a separate answer because it won't fit here.
(+1) Thanks for the references.
Wow, I never know it's also called "Duration Analysis," which sounds so much more politically correct.
Many thanks for this detailed answer, it makes a lot of things clearer and I think I will follow your suggestion regarding PCA.