tttjjj/elt_bench huggingface dataset:
Cosmo, that's because your normalizing changes the locations and scales so that you have two *new* distributions guaranteed to be approximately alike by virtue of that normalization.  But please pay attention to @Peter's comment: the KS test is designed for *unbinned* data and doesn't really work properly with binned data to begin with.  (You don't even give it information about bin counts, which is crucial.)  Do not be overly concerned with what's common in your field: it is not in the least unusual for some (apparently) common practices to be statistically inferior--or worse.
Thanks for showing how to plug this in.  It did set me on the right path.  I took your advise but made an adjustment.  I first combined my two data frames together with rbind.  Then I used the subset() function to indicate d1 and d2.

d1 <- subset(data[indices,],Time=="initial") #sample for boot
d2 <- subset(data[indices,],Time=="final") #sample for boot
(+1) The hourly minimum wage example is excellent.  The very phrasing of your question suggests a good answer: namely, that connecting points on a graph is not valid when it would cause the reader to make inaccurate (or altogether invalid) interpolations. Making a distinction between discreteness and discontinuity would help with further analysis: the donut consumption is discrete while the minimum wage is discontinuous. Each deserves a different form of plot.
I think a general wiki for where to get data is great, with a section for survey data.
That seems to be a nice solution. I would not have thought on the lines of mgf and gamma distributions given my noob status.
but the covariance matrix of multinomial in question is singular, you showed it yourself...
[An example of observational data](http://xkcd.com/925/)
I don't have any requirements how correct my ASR should be. For example, when I change my model a bit my error goes down from 30% to 29.8%. For that change I want to now if it's significant
I'm afraid this won't work, since it ignores the periodic nature of circular data. For example, suppose half your values are 1 degrees and the rest are 359 degrees. These should have a fairly low variance--(some are just above the x-axis, the others are just below), but it will actually have a much larger variance than a data set split between 0 and 180 degrees, which are actually opposite directions.
If you could divine the correct terms to use, a search of this site would give you abundant answers!  Try ["delta method"](http://stats.stackexchange.com/search?q=%22delta+method%22&submit=search).  The answer will depend on whether $x_i$ are viewed as random or not.
But if Z changes as a function of Y by only a tiny amount this might not account fully for the effect of X.
I completely agree @Henrik.  If you're concerned about profiling, Matlab has an **excellent** profiling tool (even back in version 7.1!!).  Rprof on the otherhand leaves a lot to be desired.  I end up profiling by executing each command several times in a for loop and comparing the `system.time` difference amongst different versions.  [Here is an interesting case study](http://jeromyanglim.blogspot.com/2010/02/case-study-in-optimising-code-in-r.html)
Are you trying to predict improvements/declines based upon past performance or just determine at a given point whether there has been improvement?  Also, what type of "sports data" do you have?  A sample would be helpful.
@peter-flom this time you were a couple of seconds faster...
@GregSlodkowicz, OK well perhaps you can pass this matrix to the function you're using. Also, if you've found this answer helpful please consider an upvote (or accepting the answer if you consider it definitive) :)
@Stefan The likelihood function weights the guards in the proper proportion to how many times they flipped the coins. More flips is more evidence and so should be more heavily weighted. The function I give weights each flip equivalently as there is no information in the fact that more than  one guard was involved (under the assumptions I state).
Thank you that is what I needed to hear.
@Brandon: If there is a missing interaction, there will be patterns in the residuals conditional upon values of the covariates. Plotting residuals against the covariates may help determine where an interaction might be appropriate.
The instrument used was essentially dynamic light scattering, but using two detectors - one detector gives autocorrelation for short lag times, the other detector gives autocorrelation for long lag times.  I want to combine the autocorrelation output from both detectors.
Do you have the confidence interval? SE can be back calculated from that. Also, you say that you don't have the betas, but I suppose that you have the OR's. In that case the betas can be calculated by taking the natural logarithm of the OR's.
How do the columns sum to 100?
+ NP. You put it nicely.
That question refers to knowing which predictor was relevant for a particular categorical value of the dependent variable.  This question is more broad (variable importance/ranking without detecting to which nominal value it effects).  Since that question wasn't answered I thought it worth to phrase it in a more general way in the hopes that someone might be able to help ...
@Glen_b: I think "bit (logical) vector y" does imply 0/1 response.
Thanks. I will check MCMC later.
I gather your answer is either *always* or 'it will be *always* soon'. This is the answer of a Bayesian partisan.
Walter, that's a great improvement (+1). Instead of proposing a solution that puzzles us, you have explained your problem clearly in your own terms. That helps us understand your needs better and broadens the scope and potential value of the answers you might receive.
Thanks for the recommendation! I'll look into statsmodels. Too bad that scikit-learn doesn't provide this sort of output.
Ah...so you are indeed interested in a description of the data that might account for a possible asymmetry around the mean. This suggest @Gung is on the right track: don't use semivariances, use *robust* statistics instead, such as the differences between the quartiles and the median. Indeed, these are the basis of the familiar boxplot and the five-letter summary from which it is developed.
Other than some basic misunderstanding about what a LM statistic is, and that auxiliary regressions are one way to conduct LM tests, your Stata code is incorrect. To save your residuals after a regression, you need to use the `predict` command. For example, `predict epsilon, resid`. Then you can use the residuals, which are now available as the variable `epsilon` in further regressions.
Since it is homework, please add the "self-study" tag.
**Crosspost**: http://mathoverflow.net/q/150127/17114
Notice that if $4/5$ of the numbers are $11-\\sqrt{12}\\approx 7.54$ and $1/5$ are $11+\\sqrt{4/3}\\approx 12.15$, the mean is $11$ and the SD is $2$, so *none* lie between $8$ and $12$. On the other hand, let $p\\ge 4/13$ and suppose $100p\\%$ of the numbers are $a=(11p-2\\sqrt{p-p^2})/p$ and the remaining numbers equal $11 + 2p/\\sqrt{p-p^2}$. The mean is $11$ and the SD is $2$. Because $8\\le a\\lt 11$, *at least* $100p\\%$ of the values lie between $8$ and $12$. Thus the possible answers range from a low of $0\\%$ all the way up to (but perhaps not including) $100\\%$.
is there a faster way instead of this leuk.diag <- glm.diag(leuk.mod) it is taking for ever with my data to execute.
@AndyW Perhaps in this picture there is no need to project the data onto a single line, however, I have three reasons why I'd like to know how to do so. 1] I have many parameter sets to compare **A**,**B**,**C**, etc..., 2] I'd like to apply some qualitative analysis to the different "average" curves and I think a projection might be the best place to start. 3] This can't be a unique problem, I think I'd learn something from other approaches.
(+1) this is a great question !
Is this not just another version of the distinction between *statistical* significance & *practical* significance?
The terminology here is misleading. Trimming means ignoring extreme values, some fraction in each tail. That doesn't imply deletion or dropping of values in the tails, not least because you might, and usually should, include them in other analyses.  The term truncation is best reserved for other meanings. See e.g. http://en.wikipedia.org/wiki/Truncation_(statistics)
Could you clarify what you mean by "single data element".  If you mean use $ p (\\theta|y_1) $ as a prior for processing $ y_2,\\dots,y_n $ then what you describe is perfectly reasonable to do.
Something strange happens at the beginning of the second paragraph: whereas the first paragraph refers to the expected value of an *estimator* $\\hat\\theta,$ the second paragraph refers to the expected value of a *parameter* (which in the theory of unbiased estimators makes no sense: a parameter is a *constant*).  That may be why you're stuck.  As far as the "just in case" part goes, what is $\\mu$?
High covariance between the factors indicates a violation against construct validity (discriminant validity). The questions for each factor are intercorrelated. Thus, construct validity is not given.
Maybe as a graph - see my trials at https://github.com/stared/tag-graph-map-of-stackexchange/wiki and http://stared.github.io/tagoverflow/. (Is that what you have in mind?)
Thanks for the reply.

 To give a specific example. I am aiming to compute effect sizes from published data, which is a repeated measures design. Suppose the data only reports M and SD for two conditions, to which all subjects are exposed: 

 Con 1(M=651,SD=87), Con 2 (M=611,SD=80)

It seems that (according to Dunlop) the best/only way to compute d is M1-M2/pooled sd. (pooled sd is the square root of the average of the squared deviations- as used in calculators such as this: http://www.uccs.edu/lbecker/index.html)

How are confidence intervals constructed for a d computed in this way?
@Glen_b You're quite right - it looks a lot more like [this](http://stats.stackexchange.com/questions/37732/when-someone-says-residual-deviance-df-should-1-for-a-poisson-model-how-appro).
I answered this in the follow-up you posted to previous Q. Given that, there was no need for this. I mentioned that you shouldn't edit Q's to follow-up *for future reference*!
I can apply Bradley-Terry models in SAS because I am familiar with SAS But I feel that SAS not very useful in Bradley-Terry models. My Superviser ask me to use R. My information about it is not enough. I do not do more constricts in R when I apply in my data example .
Biostat, could you clarify WHY you would recommend those books, above others?
+1. Moreover, there are no effective bounds on the probability, either. When the number is Uniform$[.5,.6]$ the probability is $1$ and when the number if Uniform$[0,.1]$ the probability is $0$; all probabilities are possible under some circumstances.
I'll take a look, but (looking at the title!) my question is after feature selection. So the scenario is basically after feature selection and cross-validation, I've the model and I would like to know which features are more important to the classification.
@whuber I intended $p$ to be a random variable in the first two as well. To me $x$ is just a syntactic entity to express the equation more clearly.
@Macro Not exactly.  I would say that the underlying assumption is that at least most of the variability and hence pattern of the data is concentrated in some lower dimensional space.  I can view a parabola very well in a 2-dimensional space with orthogonal components.  I think nonlinear shapes can be viewed in two or three dimensions.  If the data come from a multivariate Gaussian distrbution then in some subspace the points should look like an ellipsoidal cloud.  The distribution does not have to look like an ellipsoid for its view in the subspace of the high PCs to be interesting.
@user11852, now you can retract it (edited). I admit that the core principles of the answer might be the same, but not at first sight at least :-) Thanks for your help.
sounds like the way to go. You don't happen to recall a default implementation?
Hi Bayesian-believer, thanks for the directions. I will relook my problem and shall definitely read the paper that you pointed out. Do you know any means through which one shall implement noisy OR, multiplexer, sigmoid CPDs in R. I hope this also helps to reduce the dimension of the problem right.
@Zen: Of course i know what a partial derivative is, but $\\theta$ is not a variable, it is a shorthand for a list of $d$ variables, and in this case i'm not sure i understand what $\\frac{\\partial}{\\partial\\theta}$ means.
From your comment to @Learnerbeaver's reply below, I gather that this is homework/self study, which is why I added the relevant tag. So a few hints: what have you learned about the effect on R^2 when adding a variable to a model? How do R^2 and MSE relate? What does this imply about how useful R^2 and MSE are in selecting models (and why Learnerbeaver recommends adjusted R^2)? Finally, look at your specific values of R^2 and MSE for the three models in the comment below. You may be able to figure out the answer yourself by now.
What do you mean by "its pattern"? The analyst should have an idea of what to look for. What is the research problem?
John Tukey wrote otherwise (back in 1960) in a monograph "Data Analysis and Behavioral Science" (published in Collected Works v. III).  One result he obtained is that if you're getting better than about 10% test-retest agreement, your scale isn't narrow enough!
Even if correct, this is more of an assertion than an answer according to CV's standards, @Jeff. Would you mind elaborating / backing up your claim? Even just a page number & edition of Draper & Smith would suffice.
I don't understand that comment. I see only two levels: species and families. Where is that 3rd and 4th level comming from? As long as the levels are hierarchical, than indicator variables at the lowest level (e.g. species) will absorbe all the variance of all the higher levels (e.g. families).
Given frequencies below 5 in some cells, [Fisher's exact test is preferable](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test#Problems), but [not uncontroversial](https://en.wikipedia.org/wiki/Fisher%27s_exact_test#Controversies).
Did you mean Bartlett (for Barnett)?
The coefficient must change when you scale, or the fitted values wouldn't be the same. In what way would normalizing help?
Hi Jeromy, Thanks for your reply. I have got the Borenstien book and have started going through it.
Thanks for the response. But is there a way to perform this on SPSS?
First of all, I see in your question several misunderstandings: from Bayes theorem you get not posterior estimate, but entire posterior distribution.The second point is that this posterior distribution does not depend on "the sum of squares of the likelihood". It simply depends on the size of your sample (namely, n) and the sample values, which is perfectly natural and reasonable. These dependencies affect your posterior estimates of mean, variance, etc. For example you posterior mean ofvariance parameter is equal to $$\\frac{1}{n-2}\\sum \\left ( y_{i}-\\mu \\right )^{2}$$
$x_{n+1}$ isn't on $f(x)$, it's the input to $f(x)$. $f(x_{n+1})$ is on $f(x)$.
You seem to have overlooked the expectation operators in drawing your conclusions. Here is a simple analogy: when the expectation of a random variable is positive, does that imply the variable cannot have any negative values? Your conclusions are tantamount to assuming that it cannot.
Hi Adrian, thanks for getting back to me. Good question, as far as I know I should be using the same features. When I output my coefficients, I get the same number of coefficients as features. I left the LinearSVC as default, no specific kernal selected. Could it still be transforming my features behind the scenes?
Sorry, your question is pitch-dark. What are "unique columns" you grieve for? On what "this" you're going to run Gower similarity calculation? How can PCA ever leave you different types of scales (nominal, ordinal, etc.) to justify computation of the Gower? Was that this macro found on my web-page that you failed to cope with?
Just want to make sure you're clear that "both are the same" and "both are fair" are very different questions.
@gung: I do have multiple measures but I'm trying to keep it simple to start off by using the mean values.
Most certainly, $y_i=Q(\\alpha , \\beta ,x_i) +u_i \\Rightarrow y_i-Q(\\alpha , \\beta ,x_i) =u_i$ and not just "some noise", but noise for which we assume that it follows the normal distribution and that it is independently distributed (otherwise the likelihood function wouldn't be just the product of the individual densities).
...There is the observed data likelihood $p(x|\\sigma)=\\int p(x|\\sigma,\\gamma)p(\\sigma|\\gamma)\\mathrm{d}\\sigma$, it is the complete data likelihood with $\\sigma$ integrated out, sometimes known as the integrated likelihood. If you look at your equation 4, the second part of the equation does not involve the data: it is a parameter prior distribution. Similarly, $p(\\gamma)$ is a hyperprior. Therefore, I don't think equation 4 represents the likelihood of the data. Reference: Applied Bayesian Hierarchical Models, Peter D. Congdon, 2010.
Simply put, replication involves same technique on different sample
Thanks for the input.  I'm not sure I understand your suggestion or your question.  I created this example to try and understand why glmnet seemingly can't find a good solution in my real data (which has a dimension ~1,000,000).  It produces a cvfit where lamda.min=lambda.1se and no non-zero coefficients.  Essentially it fails to find any fit.  Though if I take subsets of my data, it can find fits.  So, I'm confused.
No, all Xi have different probability P and I am assuming  the weight parameter W is constant for all Xi`s it doesn`t change. If Xi is highly probable then corresponding weight Wi will take large value between 0 and N. Thanks inadvance, Glen_b
This question appears to be off-topic because it is essentially a request for advice on how to use an R package. See the help center for advice on software-related questions.
At least part of the answer will depend on whether you wish only to identify the relative proportions, or if you wish to ask questions/take measures of the individual units and try to determine if the results are significantly different based on unit colour. For the latter, multidimensional cell sizes can influence the minimum sample numbers.
Why did you model `Temperature` as a factor? Don't you have the actual numerical values? I would use them as a continuous variable & then this entire issue is moot.
@PeterFlom Yes I assume that's what Erel wants, together with any other advantages/disadvantages of those methods.
@cardinal: right you are, that was a typo, since fixed (I'm not at a computer so I'll have to adjust the figures later in response to that change). Lindsay, what are the dimensions of your s$v matrix?
They were empirical smoothing factors. Sort of a hack for a non-parametric model. I ended up not going this direction with my work. I ended up fully-specifying a model in which I was able to derive a minimum MSE estimator of my quantity.
Thank you for taking the time! I appreciate this!
+1 a good answer, but I just want to point out that a LOWESS/LOESS type smooth isn't *necessarily* overfitted (it might be in this case, but I am not sure it is) -- it has a smoothing parameter, and it's perfectly possible for it to underfit (to smooth too much, failing to follow 'real wiggles' in the data). It - and local regression type methods more generally - is a very useful tool in a wide variety of situations. It's not so hard to come to grips with local regression methods, there might be a good question to be had on how LOWESS works...
Cassie, PCA works with variance-covariance matrix, where variances are its diagonal. Sum of variances is overall multivariate variability. It is this POV is computed of. For more, see [here](http://stats.stackexchange.com/questions/22569/pca-and-proportion-of-variance-explained/22571#22571)
Dawid provides amazing intuitions on the subject: http://edlab-www.cs.umass.edu/cs589/2010-lectures/conditional%20independence%20in%20statistical%20theory.pdf
The two lines leading up to the question ("A pair..." and "Let X...") seem not to have any relevance to the question itself. Is something perhaps missing in its statement?
Hi, if they pick say "12" , is it possible 12 can be called again before success?
I think you are confused, because you don't know how computers represent real numbers. Read up on floating point numbers and check `help(".Machine")` and `help("format.pval")`. "exact" has a specific statistical meaning, i.e., the value is not an approximation. That is however unrelated to the precision to which the computer can calculate the value.
$1/p(X)$ scales the quantity $p(X|C)p(C)$ so that it is a probability on the [0,1] interval.
The sum of the frequency-weighted residuals should be zero.
http://www.stata-journal.com/sjpdf.html?articlenum=st0147 is just one source for the terminology zero-inflated beta. I don't see a case for disturbing previous terminology here: zero inflation to me means that the probability of zeros is (possibly) increased over the default model.
If the data are dependent you need to figure out what is nested within what, that will tell you the nesting structure. Then you will need to explore different models for the mean (of your DV) and different variance structures (for your random effect).
Welcome to R :-) Seriously, most of the descriptive stuff (that is how responses are distributed) can be done within Excel. But, I was suggesting R, for you could grab a lot of ideas from here (look for [scales](http://stats.stackexchange.com/questions/tagged/scales), [questionnaire](http://stats.stackexchange.com/questions/tagged/questionnaires), and [ordinal](http://stats.stackexchange.com/questions/tagged/ordinal) tags). Your question suggests you just want to summarize the data, which might be done in a purely univariate manner.
@cardinal I think that the order was reversed and that may explain why I thought Stephane's comment came in first but it does seem to be corrected now.
Is that a verbatim statement of the question?
+1 I agree with COOLSerdash. Lots of good information here! In addition to the recommended Google search, I'd specifically recommend a textbook called Econometrics by Example by Gujarati. Chapter 12 covers the Poisson regression model and the negative-Binomial regression model. As the title of the book suggests, there are examples. Data used in the book is available from the books companion website and so to is a [summary of Chapter 12](http://www.palgrave.com/economics/gujarati/students/chaptersummaries/chapter12.html) itself. I recommend that the OP checks this out.
What's the nature of the hierarchical data? Does it allow you to use the leaves of the data as your data points?
+1 Good reference.  Your answer to #1 does not appear to be effective, though, except when exactly one point is being interpolated!  What is being sought is a way to compare a (usually large) *list* of *correlated* interpolated values to the true surface.
"Simpson's Reversals show that there are numerous ways of partitioning a population that are consistent with associations in the total population. A partition by gender might indicate that both males and females fared worse when provided with a new treatment, while a partition of the same population by age indicated that patients under fifty, and patients fifty and older both fared better given the new treatment. Normalizing data from different ways of partitioning the same population will provide incompatible conclusions about the associations that hold in the total population."