tttjjj/elt_bench huggingface dataset:
Theorem 2 on page 4 of  [this link](http://jekyll.math.byuh.edu/courses/m321/handouts/mean_var_indep.pdf) will provide one way of showing this. But this may not be the only way!
excellent, +1 from me.
They all use the same method for computing the Hessian, the optimHess() function.  What varies though, is how close to the maximum they get. But this depends on the function. BFGS and L-BFGS-B are generally quite good. For some functions, I've resorted to writing an analytic Hessian, and used a Newton method I wrote myself, but this isn't always feasible.  For MLEs, an alternative is to use the Fisher matrix in place of the Hessian.
@AlecosPapadopoulos the question is updated...
@NickCox by gini index i meant http://en.wikipedia.org/wiki/Gini_coefficient
(1) Out of curiosity, what regression algorithm requires binning?  (2) Is the binned variable an independent variable in the regression or a dependent variable?
@RickyRobinson, you wouldn't want to sacrifice considerable power! :) Perhaps I should've used "you will sacrifice" rather than "you can sacrifice" (edit made). The point was that, if the variances are equal, then it improves your inference to use that information. Re: your second question, I'm sorry but I'm not very familiar with scipy.stat. If you're using the normal distribution it is simple enough to convert a 2-sided to a 1-sided $p$-value using symmetry -  $p/2$ (or $(1-p)/2$) depending on the direction
Welcome to our site! To help us focus our replies, could you please tell us the *purpose* of the information display?  It would also help to indicate you have spent at least a little time researching this problem: the capabilities of statistical packages are well-advertised and easy to discover, so your last question is not really appropriate here (and, given R's extensive capabilities, is too broad).
Thanks for the response. I don't have the book that you cite, but often these properties require existence of the moments being estimated. I'm not sure that the reciprocal of a normal has the requisite moments. I should have made this point clearer in my question.
For the time being (if nobody will try to give more details), I mark this one as correct, but what I would personally like to know is a practical guide, what rule-of-thumbs and graphics proved to be useful, and many other how-to-things from those who dig much deeper than me. Say I'm a bit lazy kind of person to read the manuals, but if you say do it, probably I should, thanks to the links below...
Just random subsets of the test data. Say I have two datasets, A and B.  I want to know how a model built on dataset A will predict dataset B.  So I train the model on A, then score it on B.  Say I've already done a 5-fold cross validation on A and I've averaged the model weights together, and now I need to score this model on B.  Should I just score it on the entire dataset of B, or should I do some sort of sampling and score on B multiple times.
An implementation of the model I mentioned (even based on the same references) is in `flip`. 
[**Although I do not find `flip` too friendly, so I've prefered either running the original code that appears in the book (for SAS) or re-writting it for R.] 
But I still don't have a sense of where to include corrections for non-heteroskedasticity, if necessary...
Thank you very much StasK. Finally I ended up the anova cammand and completed the analysis.
Can you give the reference for the paper you refer to?
The RLRsim package is a really convenient way to test random effects using simulation-based likelihood ratio tests.
Ummm, $\\hat{\\beta}$ are not variables, they are parameter estimates. And $x$ is usually regarded as a matrix of independent variables (each observation includes both the DV and these IVs).
How did you arrive at your (2)? Can you think of an example of an unbiased statistic that is complete but *not* sufficient? (**Hint**: Bernoulli.) What does sufficiency buy you, particularly when evaluating quadratic risk of unbiased estimators?
(+1) Googling turns up at least three distinct definitions (Matsushita, Morisita, and Weitzman).  Your implementation is Weitzman's.
I think there is insufficient information; you need to also know how many chickens and penguins there are.
@AndreSilva: Thank you, I had forgotten about this question. I've written an answer and a blog post, see below. There have been some updates since the previous post and I therefore removed the original code.
"how many trials I should expect" would normally be interpreted to be "the mean of the distribution of the number of trials" so you're being nice and consistent. The mean should be relatively easy.
Correct, the ANOVA p-value does not include that information.  The hypothesis for ANOVA is just that there is a difference.  COIem-easy is zero because R is defining the effects of the other levels relative to COIem-easy.
The short answer is **no**. It might help you to read my answer here: [Model evaluation and comparison for selecting the best model](http://stats.stackexchange.com/questions/22902//22905#22905), which is fairly closely related to your question. A candidate solution is described [here](http://stats.stackexchange.com/questions/78973/zero-inflate-models-vs-generalized-mixture-model/78977#78977). For more general understanding, you might try reading some of the threads on the site categorized under the [tag:model-selection] tag.
Although this question looks focused, it really is extremely broad due to the huge disparate collection of solutions available.   So that the question can be narrowed and made effectively answerable, could you please indicate what the *immediate* problem is that you are attempting to solve?
You posed that as an assumption in the original problem. http://en.wikipedia.org/wiki/Moment_generating_function#Examples has some examples; for instance, the Poisson's mgf is $e^{\\lambda (e^t - 1)}$. By our result here, the pgf is then $e^{\\lambda (t - 1)}$.
The mean bias deviation as you call it is the bias term I described.  It measures how far the aimpoint is away from the target.  Bias contributes to making the shot inaccurate.
Answering questions about effect sizes requires knowledge of the coding used, which you have not provided. (In fact you haven't even told us what statistical package has this "glme4" function.)
is there any reason why some observations are dots and other crosses? --otherwise, i would say a complementary log-log function (after appropriately re-scaling the vertical axis to get things in 0-1), because of the strong assymetry [you will find it in R in the VGAM library, function cloglog()]
i'll try myself when I have time.
Shalizi has some fun blog posts on fitting power law distributions, e.g. [this one](http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/491.html).
For now, I'm trying a scatterplot matrix of 16 graphs (focal species with 16 other species) with the data from all 10 sites posted with different symbols. Gets hard to differentiate because the graphs are small and things get clustered in the lower left. I also supplemented this with a table showing r for each species at each site and indicating significant relationships in bold. Gives a feel for the data, but it all seems so subjective.
I do not understand this 
"This leads to underfiltering in quiet periods and overfiltering during more busy times"
care to explain ?
I actually was able to get that far in the time I've been trying at this problem since. One of the confusions that I had the first time around was that I was treating the entire term, $\\mathrm{E}[...]$ to be squared, rather than just squaring the inside, i.e. $\\mathrm{E}([...]^2)$. I understand why $\\mathrm{E}[f(X) - \\hat{f}(X)]^2$ becomes $[f(X) - \\hat{f}(X)]^2$ since it is just a number, and the expected value of a real number is just the number. What I don't understand is how $2\\mathrm{E}[f(X)-\\hat{f}(X))\\epsilon] + \\mathrm{E}[\\epsilon^2]$ becomes $\\mathrm{Var}(\\epsilon)$...
Here is a picture of the result of the normality tests:
http://s11.postimage.org/5udsqz2ub/image.jpg
(you can't see the p = 0.187 vs. p = 0.026 in the image because those are from another test, actually to simplify I wrote about a 2x3 anova but I need to conduct two anovas, one 2x2 and one 2x3, but the problematic is the 2x2)
+1 Nice analysis. These data actually *do* look homoscedastic: you have to bear in mind that where log(LOC) is large, there is little data. The deceptive parts are the linear sequences of points: they probably reflect groups of closely related projects, indicating the points in this plot are not independent.
no problem; i should mention that although the terms "standardization", "normalization", and "rescaling" are used more or less interchangeably, what i did above is technically "standardization"; "normalization" (again, the strict definition) means dividing a data row by the vector norm so that its Euclidean length is equal to one.
I would not use an histogram because the binning can add patterns to the graph that are not present in the data. Instead I would plot but the deviation between the observed coverage (the proportion of instances in which the p-value is less than the nominal siginficance) and the nominal significance against the nominal siginficance. You can add a confidence interval to such a graph, like I have done here: <http://www.maartenbuis.nl/software/simpplot.html>
Thanks for all your comments !
Can you clarify how an error is measured or quantified?
+1, a nice suggestion. A (complex) extension of this idea would be to use a multi-level model: that is, a model where the relationship between manager-level information and personnel-level information is estimated taking into consideration the fact that one or more personnel may "belong" to the same manager. Thus you could explicitly account for the similarities (or non-independence) of responses of the different personnel working for the same manager.
Consider the second picture above.  How would you deal with interpreting the numbers within it. I want "to assess the consistency between raters, that is, how well their ratings correlate", so I agree with that.
@Harvey True enough, but irrelevant - you asked about fitting normal vs log-normal distributions to the *same* data, & this is what whannymahoots is answering.  Because the number of free parameters is the same for both models, comparing AICs or BICs reduces to comparing log-likelihoods.
(But you need to make sure the mathematics is right first...)
Please read the guidelines for [answering self-study questions](http://stats.stackexchange.com/tags/self-study/info). This goes far beyond the recommended 'give helpful hints' and looks like it just does someone's homework for them. Once it's clear the OP has solved it, or in any case, after several days, this level of detail would be a good thing to put in an answer.
@Thomas Added the link.
@JMS - thanks for that. have edited it accordingly
I think you would have more luck asking this on [stats.se].
If A and B are clusterings, then d(A,B) can be split up as d(A,B) = d(A,X) + d(B,X) where X is the largest clustering that is a subclustering of both. In Suresh's notation we have that d(A,B) = f(A)+f(B)-2f(X). This can be rewritten as f(A)+f(X)-2f(X) + f(B)+f(X)-2f(X) = d(A,X) + d(B,X). Above I have written the two components d(A,X) and d(B,X) separated by commas. The biggest difference between the two by far is the quadratic characteristics of Mirkin/Rand. If you look at the Top/Bottom and Left/Right examples, the Top-Bottom distance is huge; this is entirely due to the size of Top.
Unless you have some special clever way to process these codes in your statistical procedures, this method will have the doubly unfortunate characteristic of (a) actually allowing the procedures to produce output that (b) is wholly erroneous.
@shabbychef Then check out the interface from R called rggobi http://www.ggobi.org/rggobi/
You should specify if you want to oversample based on covariates or on the outcome. But the general idea is that if your event of interest is very rare, then your results might be biased under random sampling due to finite nature of the sample. A good reference on the bias of a logistic model under rare events can be found here: [Gary King's website](http://gking.harvard.edu/category/research-interests/methods/rare-events).
Thanks!  I will look into that
Alternately, *do the actual driver/victim pairs have the equivalent degrees of separation as known murderer/victim pairs?*
@user721975: (2) "single" LR is a bit ambiguous: at least if there are more than 2 classes you'd have a multinomial model. Maybe you need to tell us more about your application in order to get more detailed answers.
"*have been told that this method is not appropriate*" - on what basis?
Why don't you look at the original plot, Michael?  It will make it abundantly obvious why at least one additional parameter is needed.  Please note, too, that in a comment to the question the OP has asked, "what would be a more flexible nonlinear model?" One implication of the initial analysis offered in this answer is that it should be considered out of the ordinary to fit an exponential with fewer than three parameters: there must be some inherent constraint operating in such cases (such as an intrinsically determined unit of measurement or an intrinsic location for $x$).
This seems like a homework problem. If it is, please add the `homework` tag so that we can treat it accordingly. Also add some comments on how you've tried to solve the problem and where you got stuck.
Haven't read the Wikipedia entry (which one?), but: this problem is not fully specified. In the absence of more information I would say you should calculate P(H>=14 OR H<=6), i.e. a two-tailed test (the probability, under the null hypothesis (H=10) that the value deviates by as much or more than observed **in either direction** from the expected value under the null).
Mr. Miles, Thank you so much for your good answer.
@Josh Sorry, I should have closed this earlier :-(
Thank you very much for your reply! Would you please be so nice to explain your Mathematica code a bit. When I just apply the R-code from the last example along Pr(A > B | A > C) = Pr(A > max(B,C)) / Pr(A > C) it yields the wrong result. What does for example Max/@Tuples@{b,c} do? Thank you!
I was looking for a way to get a better expression of the mean in presence of repetition, but i think i'm off-road...Thank again for the solution
Thanks, Masato! This finally brought me on the right track. Now `summary(glht(model, lsm(pairwise ~time|cond)))` does what I want :-)
This explanation, although it is clear and technically correct, does not explicitly involve any distributions whatsoever, and thus appears not to be relevant to the quotation in the question.
Are the events independent or dependent? 

Also, if this is homework, it should have the homework tag
I like the title you propose !
Many questions just like this one have been answered at [GIS.SE], *q.v.*
This is correct - thank you very much for correcting my beginner's mistake. David
How related are the different metrics? 

I'm thinking in combining these metrics, there are a couple of potential pitfalls to avoid. 

One is mix change - if one group processes a lot of transactions related to one metric that overall all groups score lower on, their overall score may look worse but it may not be attributable to performance.

Another is autocorrelation - if these metrics are highly correlated with each other on a single transaction, then performing a simple weighted average for each metric would favor groups that had transactions that qualified for the most metrics.
In general no. You are right about the special case of zero SD and your sum, but SD is a root mean square and if it could be re-written as a weighted sum of deviations that would be public knowledge. I suspect at most there is an inequality lurking here, related say to Jensen's inequality, but don't bet on that and I am not the one to flesh that out even if it is true.
P(C|D,λ) refers to the probability distribution of classes to every instance in the dataset, given a particular weight vector λ.  EG, the probability that instance 1 has class 3 and instance 2 has class 1 and instance 3 has class 1 is a single entry in that distribution.  If there are m classes and n instances, there are m^n entries in the distribution.
In the former case, I'm expressing my prior belief that the beta1[] are drawn from a distribution with zero mean. In the latter case, I want to constrain the posterior means of each beta1[j] such that if I take the mean of these posterior means, I'll get zero.
There is one problem with doing such modification -- people are used to standard boxplot definition and will most likely assume it when looking at the plot whether you like it or not. Thus, this may bring more confusion than gain.
It's not clear to me what you have and what you are asking for. There are tests of whether a proportion is 0, but that's not a meaningful test for accuracy - accuracy of 0 would be perfect in a way - always wrong!
Also of interest: [surprising characterizations of the Gaussian distribution](http://stats.stackexchange.com/questions/4364/)
As I wrote above: I don't see how clustering can help you here. Clustering is inherently an unsupervised learning algorithm, but you do have a target variable, the spend per customer, so you should use a supervised learning technique. I suggest you go with ANOVA. I would not advise using a technique you do not understand.
Lots of things. The 1st answer would be sampling error, I suppose. Can you say anything more specific? This question is too broad & unspecified to be answerable beyond this comment.
I would still like to hear you about estimating association, though. Even just some concise remarks! Thanks!
You need to describe your motivation for feature selection, and review the literature (and huge number of posts to this site) on its pitfalls.  If you really want to proceed you might consider using the bootstrap to quantify the volatility of this process.  For example it is easy in R to get bootstrap confidence intervals for the importance ranking of each predictor based on partial $R^2$ or on marginal correlations.
Note that when you're using AICc (or AIC) to compare models with the same no. parameters, you're just comparing likelihoods; which you might be happy to do even if you didn't have any compelling arguments for using AICc.
It is difficult for people to answer questions that are based explicitly on false assumptions, as this one is, and even more difficult when the question is broadly stated and vague.  Please edit your title and the text to make your question more specific.
You guys are after the function `prop.table()` which comes with base R, it's essentially a wrapper for what you've got above. With your example, you'd want something like `prop.table(table(x))`
You want numerator d.f. p not error d.f. n-p-1
Good that you cite the whole book from Pearl.
+1 What an excellent answer!
Equivalence testing is well established, and you misunderstand its null hypotheses, which are generally of the form H$^{-}_{0}: |\\theta-\\theta_{0}| \\ge \\Delta$. This is an _interval hypothesis_ which can translate, for example, into two one-sided tests (TOST): H$^{-}_{01}: \\theta-\\theta_{0} \\ge \\Delta$, or H$^{-}_{01}: \\theta-\\theta_{0} \\le -\\Delta$. If one rejects H$^{-}_{01}$ & H$^{-}_{02}$, then you must conclude that $-\\Delta < \\theta - \\theta_{0} < \\Delta$, i.e. that your groups are _equivalent_ within the interval $[-\\Delta,\\Delta]$.
Can you clarify what you mean by a weight variable? See the distinction between survey weights & frequency weights discussed [here](http://stats.stackexchange.com/questions/22989/). For frequency weights you do in fact have a greater sample size.
Also, you will get much better answers if you write down what the distributions are.
I later found that there is "AddCluster" filter in Weka tools that performs the clustering on the data set and stores the cluster number for each of the data points.
@January: you forgot to mention the DoE "take all samples we can get - they'll anyways be too few" - which is the most frequent DoE I encounter.
"Sine" and "sinusoid" are the usual English words. It implies periodicity. In your case, the peaks at 9, 18, 27, 36, ... imply a period of 9. What does AR(1) have to do with your question?
Do you strictly have to do it with Matlab? Otherwise I suggest using Hugin http://hugin.sourceforge.net/ . The software is released under GPL so you may also peek into the source code and see how the magic happens.
The short answer is "yes".
Also of possible interest http://stats.stackexchange.com/a/22331/3277
Thanks for the comments. The negative value is indeed correct unfortunately. Is there a way to deal with negative values? I have another dataset with the same x values and negative y values (it is basically a reflection of this data in the x-axis) so I could really do with an approach that can handle these.
It sounds like you have a time series data. If the data are time series in nature then you may have to look for VAR (Vector Auto Regressive) or VARMA models.
The examples are really helpful to visualize the data. Another question regarding your caveat: "the values clump at the floor and the ceiling and that the data are not normal." Can you tell me if (a) the data on the lower end of the scale look normal and (b) the data on the upper end of the scale look normal?
ARGH. I wanted to add the figure directly, but not enough mojo/reputation/bla bla bla.
I am not sure if I understand your question, but if you have only two data points for each compound, you can't do anything. But I doubt that is what you have.

Could you post some sample data?
Surely it can't be a random walk because sales can only go up or stay the same (equivalently stocks can only decline or stay at the same level) -- something more like a renewal process.  Also I'm not sure how you figured out that exactly 3 points are outliers.  (BTW the 'analysis' I suggested is only a generalised version of his own, with an unsustainable linearity assumption, so I personally wouldn't be content with it.)
@whuber: it is positive skewed data, and actually I want to compare the length between object A and object B. Or should I simply calculate the CI using t-distribution?
It's perhaps worth pointing out that you can use multinomial logit if you want to.  You just have to re-define the choices.  There would be 4 alternatives: {nothing,brand1,brand2,both}.  Now, the assumption of independent errors that MNL uses is pretty untenable for this choice space, so you should probably use Generalized Extreme Value (GEV) instead of MNL regression.
@DimitriyV.Masterov I think of something like firms' R&D expenditures. In a survey some firms report to spend >=0, some firms report to spend zero and the third group just does not answer the question.
Sounds like a potential mediation problem. Might be worth performing the usual mediation analyses if V5 would make theoretical sense as a mediator.
$\\frac{x\\pm\\sqrt{x^2-4}}{2}$ has a $\\pm$ and so gives two maxima
Thank you so much for your quick response! And so how does my outcome variable factor in there? I'll have gain scores for my primary outcome measure (depression). I thought I would need to make the model with my alliance scores predicting outcome scores, moderated by attachment scores. Thanks!
I'm not saying its computable but that is the calculation you need to make.  Also, here is the reference for Bayes Theorem http://en.wikipedia.org/wiki/Bayes%27_theorem
Is there a question in here somewhere that I missed?
Duplicate of:  http://math.stackexchange.com/questions/754401/test-the-mean-difference-of-two-gamma-distributions
Thanks for writing up such a comprehensive response! (Apologies if comments like this are frowned upon in this community)
Isn't that what AR and MA are for (in ARMA)?  You might also look up "long memory process", such as ARFIMA.