tttjjj/elt_bench huggingface dataset:
In Australia up until around 10-15 years ago, the conservative parties usually started strong in the early counting, with the progressive parties making a late comeback. The TV networks probably knew what was going on, but the variability probably made for more drama. It all changed when an analyst name Antony Green started using the booth by booth results to account for the fact that small booths in rural areas tend to get their counts done and results in early, and they tend to vote more conservatively. Antony famously correctly called an election result hours before anyone else using this.
@Roland I think it’s an unfortunate (though valid) definition in the textbook. Whether you define the support of *X* as an open, half-open or closed interval, the distribution of *X* is the same; that is, the cumulative distribution functions are identical.
I might be understanding this incorrectly, but do you mean you have a binary classifier where all the outputs are equal? If this classifier gives you actual accuracies of 95% and 80%, doesn't this just imply that 95% and 80% of inputs were indeed "stayer"? I would suppose this would probably not be a fluke in the sense that it is not unlikely that the next year will be similar. However, it is then not a very sophisticated and it might be possible to do much better (depending on your data).
After reviewing the API, it doesn't look like there is a direct method to do it (please correct me if I'm wrong, that's why I posted this question here!). I'm thinking of using some sort of collection where the dependency edges would be stored as indexed entries (maybe the relationship or head word would be indexed), and then use generic Java search patterns to find the desired edges?
That appears correct. The extra question here as to why the null hypothesis is a point value not a range or inequality is more interesting! In the commonly-encountered cases at an elementary level with a single parameter of interest it always is, I think, but i believe things get a bit more complicated in the multiparameter setting. But I'm not sure so I'll leave someone better qualified to expand!
@sjm.majewski : Thanks for you reply!

Using your result for $X+Y$ and the definition of conditional probability, I get:

$$ P(X=k|X+Y=n) = \\frac{P(X=k,X+Y=n)}{P(X+Y=n)} $$

Applying this to the above question gives:

$$ P(X=k|X+Y=n) = \\frac{\\frac{\\mu_{X}^{k}}{k!}e^{-\\mu_{X}}{\\frac{\\mu_{Y}^{n-k}}{(n-k)!}e^{-\\mu_{Y}}}}{\\frac{(\\mu_{X}+\\mu_{Y})^{n}}{n!}e^{-(\\mu_{X}+\\mu_{Y})}} $$

Is this correct? If so, how do I get this in the form of the binomial distribution?
An efficient negative binomial generator bypasses the Poisson/Gamma representation, using instead [optimised accept-reject techniques](http://www.mendeley.com/research/computer-methods-for-sampling-from-gamma-beta-poisson-and-binomial-distributions/).
I agree that an answer does not necessarily need to incorporate a complicated formula. I still think that the answer, as it stands now, would be better suited as a comment rather than an answer. The [FAQ](http://stats.stackexchange.com/help/how-to-answer) clearly states that fuller answer that are a bit more fleshed-out are desired. I do not - in any way - say that your answer is wrong. But a bit more explanation why you propose the linear term would be helpful.
You can either (i) fit two models, one where prep2 and prep3 have the same dummies, and one where they have different dummies and compare them using an F test via the information I indicated, or (ii) reparameterize so that you have a single dummy representing the contrast (that difference in the two preparations) and test it via a t-test or F-test. (Well there are other ways but those should work okay). Failing that, if you post some data I could see if I could explain how to do it using your data as an example.
Time for me to do some reading... Thanks for the explanation; I'll either accept or comment again once I have a clearer idea of how to reformulate my ideas.
@ttnphs Thanks so much! What is the new procedure in SPSS then?
How did you cluster the standard errors? Many of the canned approaches use cluster-robust standard errors, which also take heteroskedasticity into account.
What type of expanded outcome measure did you have in mind (instead of the 2x2 outcome matrix)? And could you provide a source to the last claim about correlaton between d' and Hit Rate-False Alarm Rate?
Thank John,your method works in assessment of 2 group's means as t-test. I don't know whether your method is applicable in comparing correlated AUC.
Thank you for the paper. I shall read it tonight when I have some peace and quiet.
I like the question: can you say more about the *kind* of data other than that they are non-negative? For example, are you explicitly referring to *count* data? Not that you are, but if you were, you could perform hypothesis tests based on the Poisson distribution (or negative binomial distribution), rather than the *t* distribution. Also: I took the liberty of edit the last sentence in your first paragraph to avoid a misinterpretation of non-parametric tests.
@whuber Quite right! I stick by the model I set out in the first para, but my equation was screwed up in several ways... Goes to show I haven't actually used log-linear modelling of contingency tables since my MSc, and I haven't got the notes or books to hand. I believe I've fixed it now though. Let me know if you agree! Apols for the delay. Some days my brain just doesn't do algebra.
An R issue this one: `read.csv`, which you probably used, returns a `data.frame` which is a not a `matrix`.  So you need to convert it to being a `matrix` with `as.matrix` before making it one long vector with `c()` and giving the results to `cor`.  Here it is in one line: `cor(c(as.matrix(matrix1)), c(as.matrix(matrix2)))`
If this is homework, and it really sounds like it from the "please consider" remark, then you should do it yourself, or failing that, say exactly what you have thought about and use the homework tag as stated in the FAQ.
The table shows multiple rows of data. Are the rows from different units of analysis (e.g., people)? And if yes, can they be assumed to be independent?
I have no clue what SPSS  does, and frankly would prefer understanding what a package does before using it. Stratifying by age and sex would sound good enough for me, unless there are some other covariates known to affect the outcome that can be stratified upon. Matching that is too picky may not generalize well for the controls that it could never pick, although one can argue that only the individuals at risk in the general population that are "close enough" to cases are of the primary interest, anyway. (Economists like to put forth that sort of argument in their local treatment effects).
+1 Nicely presented MansT.
The reason I want to buffer the curve to intercept at negative 5 is due to the fact this is the temperature that methane flux terminates in both the data I have and also the value in a number of papers which investigated this in lab/field tests.

Thanks for the advice though, I'll look into it!
@Vincent The classes are equally divided. For output 1 and 2 I used a linear SVM (C=1.0) and achieved ~63% and ~80% classification rate respectively. Using the same classifier for output 3 I only get a 8.3% rate. (I also used scikit-learn "scale" on the features). Data can be found here: https://docs.google.com/file/d/0BzpEPYc7dTD0QjN4X3NWRVRWSUk/edit?usp=sharing
I think you can bring more than opinion to bear on this question. In particular, you can talk about what the interval actually means (this is a somewhat subtle issue that's worth the time to understand), and the importance of having a representative sample (as well as how that might be achieved, given the assumptions under which your interval was derived). There's some scope for opinion, but one that relies on understanding the statistical issues.
Could you tell us what criterion you would use to define whether a peak is "significant" or not? It would also help to provide some context: what is the purpose of this peak identification? What would it cost you to fail to identify a "significant" peak?  What would it cost to identify something as "significant" that actually arises out of the background noise alone?
From now I already tried it on several dataset which is public available like MNIST or my own dataset, all of them give worse result than standard MLP with sigmoid. I use same layer configuration but for ReLu I tried lower learning rate. So I just want some rule of thumb for nn with dropout & relu hyperparameter (like bigger or smaller hidden layer size, bigger or smaller learning rate) compared to standard sigmoid MLP
For discrete distributions the exact methods will pick integre end points but it is possible that no such interval has exact 95% coverage.  You may have one with 94% coverage and adding the next integer to the interval gives 97%.  So the one that gives 97% would be defined as your exact 95% confidence interval.  Just like the normal approximation to the binomial approximates a discrete distribution with a continuous one, so too does the chi square do that in approximating the Poisson.
Solved. Ref:  Wahba, G., and Craven, P.. "Smoothing Noisy Data with Spline Functions. Estimating the Correct Degree of Smoothing by the Method of Generalized Cross-Validation.." Numerische Mathematik 31 (1978/79): 377-404.
I think the article says it pretty clearly. One may write a syntax to name a cut point, run the two regression, export the two $R^2$, and repeat the process with another cut point, so on so forth. When all repetitions are done, plot the two $R^2$ against the cut point, and look for the one that optimize the models (aka give highest $R^2$ for both models). Unless you have spotted a very clear inflection in $x$, there can be multiple optimums, which will cause some problems.
@Mike Lawrence So, when you take a mixed effect modelling approach, you don't collapse your data (in this case for subject and category) to a single mean but rather use all the observations? I'm taking a signal detection approach for some of my data, and you obviouslty can't talk about concepts such as "hit rate minus false alarm rate" for single observations. Also, in the paper, they mention that this model is robust against violations against sphericity and homoscadasticity. Does this mean I can cancel my upcomming question about alternatives to RM-ANOVA in cases of non-sphericity of data?
@Macro: I *think* the notation is odd/unusual, but correct (i.e. equivalent to what you suggest.)  I've expressed the random effect terms as part of $\\mu$. It would probably be clearer/more familiar if I wrote it out in multilevel notation ($Y \\sim \\text{Normal}(X \\beta + Z u, \\sigma^2); u \\sim \\text{MVNormal}(0,\\Sigma); \\Sigma=f(\\theta)$).
Thank you for your answer. Those are very good points.  However, my goal is not to make an exhaustive list of score functions, but rather finding other existing ones that I am not yet aware of. Then I can see whether they would fit my application or not.
Welcome to the site. "Stationary" is, I believe, used with time series data. Do you have such data? If so, OLS regression might not be best.
@PeterFlom: yes. But often with high dimensional data, most of the variance is concentrated in a few component (in the PCA sense). Outlier detection algorithms have been developed which explicitly aim to take advantage of this. These have numerical complexity polynomial in $k$, the number of component (instead of $p$; the size of the original space). Of course, all this hinges on the user being able to set a upper bound on $k$.
I've updated the question, also could you please explain your last sentence a little bit? Thanks!
can you describe in much more detail what you're trying to do? that is, describe each of the steps you want to take and then let people respond to them?  if you're unclear on the steps you want to take, then please try to add much more detail to your desired conclusion or output  :)
Yah, makes sense. I suppose I was hung up on the fact that the coefficients were the same, but I suppose the standard errors themselves are important.
With `lm()`, you are using $\\sqrt{\\hat\\sigma}$ instead of $\\hat\\sigma$.
$f$ is already a probability distribution. Now I think I know where your objection come from, so I would attempt to explain it. If you just grab some random blocks and add together, it need not be a distribution. But we start with an $f$ already being a distribution. The value of $X$ is invariant under multiplication of $f$ by a nonzero constant anyway.
If you feel like the question has been addressed satisfactorily at this point, you may want to accept one of them by clicking the check mark next to it.
Hi there, could you please add your sample size and an explanation of your variables to this question. I've not seen this error in `lmer` but in another context it tends to mean you have run out of degrees of freedom - i.e. sample size too small for number of variables in model. Not sure if that applies here.
In R there is the `profdpm` package, and an overview of available indices (incl. Fowlkes & Mallows's $B_k$) to compare HCs is available here, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.164.6189&rep=rep1&type=pdf.
You (or other readers) may enjoy: A. Renyi (1961), [On Measures of Entropy and Information](http://projecteuclid.org/euclid.bsmsp/1200512181), *Proc. of the Fourth Berkeley Symposium on Mathematical Statistics and Probability*, vol. 1,
 547-561.
What makes you say that every null is rejected if the samples are large enough?  As the sample size increases we also expect Dnn' to decrease.
@Felix, I understand that. I think if you're looking for the correct terminology the information you're leaving out about how you actually perform each step (how does the information from step 1 figure into step 2) is going to be important. For example, does the second classifier give you $P(y=k|x)$ for $k=1,\\ldots,N$ and you just pick $\\max_{k\\in M}\\{P(y=k|x)\\}$ for the final classification, essentially allowing the first to veto any decision made by the second? Or is the second classifier doing something more like $\\max_{k\\in M}\\{P(y=k|x,M)\\}$?
What about the third option: measure the difference of the final values, adjusting for the baseline values as a covariate. I think this will be better than option two in terms of estimating the [treatment] effect size. This equates (in a simple form) to analysis of covariance.
I can't resist mentioning this -- Kernighan & Plauger famously put it: “*Floating point numbers are like piles of sand; every time you move them around, you lose a little sand and pick up a little dirt.*”. When taken across many calculations, accumulated error can sometimes become quite problematic (e.g. a stock index being famously out by 44%)
@Kevin, maybe you could describe your general problem and what you are hoping to learn from this analysis.  You may be asking an overly focused question.  If you really want to know whether there is periodicity or how to account for it in a model, zero crossings and zero correlations are not the best approach.
I do not think the use of Bayes factors for testing hypothesis is "occasional", see for example [this reference](http://www.stat.cmu.edu/~kass/papers/bayesfactors.pdf).
I believe some help is provided in the [Wikipedia article on "energy distance"](http://en.wikipedia.org/wiki/Energy_distance).
Can you give a definition or a link to the definition of the different test. Can you state the hypothesis you want to test (otherwise it is difficult to discuss the power ...).
I am voting up your answer. Thanks for the source document I will check it and return back.
Yep, visualization is what I want. I have added some more info in the question.
"most different" means what you did. The inclusion of the qualifier 'significantly' implies the person asking the question is looking for which one is the most standard errors from zero, or which one has the lowest p-value.
Experiment results are given and I have to find a way to deal with it. Agree with you that treating dosage continuously is a better approach and will definitely suggest it if I'm consulted for the next experiment.
Thanks a lot, now I see why normality is required.
It would be polite, especially to new users posting their very first question, for downvotes to provide some constructive feedback on how they think the question could be improved.
1) the scikit-learn link does k-means with scipy sparse arrays (although I agree with Anony-Mousse, "k-means is bound to fail"); 2) start with 10 or 20 clusters, small enough to understand on paper / plot; 3) can you hand-classify say 100 points ? Then look at semi-supervised SVMs.
My features assume boolean value, so I don't need normalization. It is probably due to the fact that the classifier gets the information from other highly correlated features I have in the dataset
I've included some R code by way of explanation. So you're saying I'm not applying the wiki formula correctly? Presumably by assigning the probabilities, no?
I wonder if the regression is using sampling weights (weights that denote the inverse of the probability that the observation is included because of the sampling design) and the t-test is using importance weights. That is what I get with Stata using your data.
@gung: actually there is even a closer thread: http://stats.stackexchange.com/questions/68162/two-independent-variables-both-correlate-with-the-dependent-variable-but-none-a. Definitely a duplicate.
Your questions touch on statistical analysis but are mostly about economics.  I believe you'll find the best answers looking through the literature on employee retention.  I'll bet there are two or three journals devoted chiefly to this topic.
Note that EM (at least when it converges nicely - for example, doesn't get stuck in a local maximum) yields ML estimates, and in general ML estimators are biased. This is a property of ML; EM is just a way of computing ML estimates.
This question, in a more specific form, might get some great answers on the [Mathematica site](http://mathematica.stackexchange.com/). For instance, it's possible that a single command like `FindSequenceFunction[{1,1,2,3,5,8,13}]` (with the values replaced by yours) would immediately solve your problem.  *Mathematica* calculates symbolically and with arbitrary-precision and rational arithmetic; numbers of 100,000 digits will barely slow it down. Otherwise, I suggest posting it on the Math site, because *exact* fitting isn't really a statistical problem--it's a mathematical one.
Do you mind elaborate a bit on this bayesian approach? To be more specific, say if we are talking about a second order markov chain with 4 states, I want to estimate the probability that a "3,2" will be followed by a "1". But in the time series I've seen so far this has never happened. How can I esitimate this probability?
See [CFA factor scores AMOS](http://stats.stackexchange.com/q/66656/32036) and [IBM's support on this topic](http://www-01.ibm.com/support/docview.wss?uid=swg21600240).
Not "other than using t-test," because 1/0 variable is not suitable for t-test. Try searching [comparing two proportions](https://www.google.com/search?q=comparing+two+proportions) or [chi-square test](https://www.google.com/search?q=two-way+chi+square+test).
@mbq I did read the Wikipedia article but I wanted to confirm and verify it
Well, in a large sample you can detect very small differences from the null hypothesis.  That doesn't mean they're necessarily of any practical significance; or even that they're real, given that there will also be unknown systematic errors.
(+1) Nicely presented.
I don't know what to tell you. The optimal *k* parameter can vary greatly based on your data. Cross-validation or visual examination of the data is the "legitimate" way.
Are you sure your predictors aren't separating `y` perfectly? That's a problem for a lot of logistic multiple regression models.
This question will likely be removed as off-topic given that it has nothing to do with statistics. For Stackoverflow this will also be too broad or likely already has an answer.
The documentation says:
"By default, the numerical algorithm numericDeriv is used. It is of course strongly suggested to provide this function when it is possible. This gradient is used to compute the asymptotic covariance matrix of \\hat{θ} and to obtain the analytical gradient of the objective function if the method is set to "CG" or "BFGS" in optim and if "type" is not set to "cue""
So obviously R solves this numerically if I don't provide it!?
I do not recognize any difference in performance, so letting R do the job removes at least the error source of getting the gradient wrong. Comments?
(+1), nice answer. @MYaseen208, you can find the identities used here in the matrix cookbook, chapter 6 - http://orion.uwaterloo.ca/~hwolkowi/matrixcookbook.pdf
It can, for a certain definition of "statistical power". You might like to look at section 7 of http://socrates.berkeley.edu/~maccoun/PP279_Lenth.pdf
I was reading through an old textbook from the 1950s and found this example as a way of showing what was and what wasn't a t-distribution. Unfortunately, the textbook didn't come with any solutions.
I think the reason I hesitated, is that your answer, while certainly more constructive the other answer, it appears to make a lot of big assumptions.  I fear that the approach you describe is therefore no better than simply saying "no, in general there is no way to calculate or bound the probability without knowing more about how (as @Bitwise puts it) the phenomenon generates observations."  Is that a bad interpretation of your answer?
Shouldn't $\\alpha_i$ be a single, known and fixed probability between 0 and 1 (like 0.5 if you are modelling the median)? (I am referring to intuition and also to http://dx.doi.org/10.4236/ojs.2012.25068)
@bill: The OP is asking about *statistical* interpretation of the results. Indeed, the question is interesting and involves several easy sources of confusion. While results of such polling should not be viewed in a vacuum, and it is important to try to ferret out potential sources of bias, the main intent seems to be to uncover some statistical subtleties. For this reason, I'm hopeful that a nice answer might appear. It's a worthy question, in my view.
It depends entirely on the context. The indicator $\\mathcal{I}\\{X>Y\\}$ is a random variable that has a Bernoulli distribution with success probability $p = P(X>Y)$. So, $X>Y$ gives you a single realization from that distribution - a sample of such values will give you information about $p$ - whether or not this is meaningful depends on the context - can you provide more info?
I erred in my first draft so let's look at a simple example, say $l=2$. The probabilities for outcomes $(M,F)$ are $\\Pr(1,0)=1/2$, $\\Pr(1,1)=1/4$, $\\Pr(0,2)=1/4$. The limit and the stop-at-one-male rule prohibit all other possibilities. Expectations are probability-weighted values: $\\mathbb{E}(M,F)$ = $(1/2)(1,0)+(1/4)(1,1)+(1/4)(0,2)$ = $(3/4,3/4).$ For a population of $n$ breeding pairs (at least $2n$ people), we obtain $3n/4$ for the expected numbers of Ms and Fs in the next generation. Thus the expectation of $\\min(M,F)$ must be less than $3n/4$ = $3/4$ the number of breeding pairs.
I agree strongly with @user603. Density estimation is at first sight a very odd and indirect way of trying to find outliers. Your answer would be enhanced by summarizing how that is applied in practice -- and why you think it works well.
I am not sure of the correct terminology here, there are probably better ways to tag the question.
Dear @StéphaneLaurent: We can't know this because it's not true.