tttjjj/elt_bench huggingface dataset:
Logically if the mean (or other location measure) of $A$ differs from $A+B$ then it differs from $B$, and vice versa. The same applies to a more general U test, even one comparing $P(A<B)$ to 0.5. Rejection of one form of hypothesis implies rejection of the other. If you did form something like a t-test for comparing dependent samples like that, you'd end up with something that was in effect doing the $A$ vs $B$ comparison anyway.
Actually I am trying to replicate Labauch and Williams (2003) paper on natural interest rate. It involves estimation in 3 stages, each stage requires inclusion of exogenous variables.

Say:

Measurement equations:

$y_t = y_t^* + \\alpha_1 (y_{t-1} - y_{t-1}^*)-\\alpha_2 (r_{t-1} - r_{t-1}^*) + e_{1t}$,

$\\pi_t = \\beta_1 \\pi_{t-1} + b_y (y_t - y_{t-1}^*)+b_f (\\pi_{t-1}^o-\\pi_{t-1})+e_{2t}$

State equations:

$r_t^* =c g_t +z_t,

g_t =g_{t-1} +e_{5t},

y_t^* =y_{t-1}^* +g_{t-1}+e_{4t},

z_t = \\gamma_1 z_{t-1} + e_{3t}$

For now only measurement equations have exogenous variables ($\\pi_{t-1}^o$).
Your statement about "one dimensional data" is only sometimes correct and often is wrong. What is true is that the degrees of freedom *usually* are equal to the data count minus the number of parameters used to compute the expected values. For instance, if the five observations in the question are bin counts and the five expected values are based on fitting (say) a three-parameter lognormal distribution using maximum likelihood, then there are only two degrees of freedom. For more about this, please visit [How to Understand Degrees of Freedom?](http://stats.stackexchange.com/questions/16921).
Some of the methods used to identify multivariate outliers can be adapted to ranking the data (often by identifying "outliers", removing them, and iterating).  You might therefore have some interest in reviewing our thread at http://stats.stackexchange.com/questions/213/what-is-the-best-way-to-identify-outliers-in-multivariate-data.
Yes, asymmetry and outliers can affect the PCA decomposition. For an example, have a look [here](http://stats.stackexchange.com/a/33602/603)
Great thanks. Is there anything to say about how each approach would cope with a systematic or random error in the results (due to poor predictor or outcome variable data quality for e.g.)?
How well would this method work for the pathological blue line case?
Thanks, It works.
Framing this as a sample size determination is helpful but I wonder whether an approach based on sequential sampling theory might be even more fruitful.
I have seen lots of horrible notation in this territory. Composite symbols such as $ss$ are difficult to read, but if you are determined to use them roman ss is preferable and most of the precedents seem to use upper case, e.g. SS. An analogue is that exp is fine as a composite, while $exp$ is classifiable as learner error.
@mlxa Properties of the binomial and poisson distributions indicate the variance of a count is roughly proportional to that count, whence the square root is a variance-stabilizing transformation.  This makes it legitimate to compare variability across different heights in the plot.  The log downweights variation in the high bars whereas using the raw data downweights variation in the lowest bars. // Since you're concerned about bad scores, why not just look at the proportion of bars falling below a critical threshold?
I've explained examples of the kinds of mechanical procedure used to find what the mixture distribution is from the specified distribution of $X$ and $Z$ (in your notation) quite directly. Two possible lines of calculation in answer to the explicit question "I was wondering how one would go from the prescription above for Y to a formula for the distribution" are provided above. Specify $X$ and $Z$, and do some algebra. ... (ctd)
Interesting related question: http://stats.stackexchange.com/q/61531/5509
Are there any papers showing feature selection methods like Information Gain based ranking are more likely to overfit ?
Please explain what you are trying to do. Do you mean using windowing with spectral estimation, or with kernel smoothing, or something else? And what sort of anomalies are you looking for? It is impossible to answer such a general question.
Just for clarification, do the points at the 0.0 line really mean no observations within that category (from -30 to -15) or a very small odds ratio (same for the NA category at the rightmost of the plot)?
Maybe [this](http://en.wikipedia.org/wiki/Amazons) one.
Hi Glen,
Thank you very much. This was really helpful.
You're trying to infer whether there's a difference between correlations in 1 group versus another. Now, there are some very subtle wrinkles to this. However, using a t-test to infer whether the average of the group correlations in one group is different from another should be approximately valid. Most software does this automatically and doesn't require you to calculate any averages.
Alternatively, for low expected values, the same test statistic could be used but the distribution of the test statistic (conditional on the margins) obtained by simulation. Some packages (including R) offer this option.
How do you work out the sampling distribution of a mean without knowing what the distribution of a sum is first? When you say 'the samples are related' ... do you mean 'the observations are related', or do you have multiple samples each containing many observations?
All passengers can be boarded on the $i$-th flight if $X_i = 0$.
+1 for a detailed answer; with regards to (1): it might be useful to note that the expected MSE loss from an estimator is $\\text{Bias}^2 + \\text{Variance}$, without factoring in irreducible error from the regression estimation
Can you give a little more information about your data? Like what is the treatment, is the treatment variable something numeric or categorical (e.g. 1 or 0)?
That Poisson distribution seems like a rather restrictive assumption, no?
Those notes appear to be here now, in Chapter 7: http://cseweb.ucsd.edu/~elkan/255/dm.pdf
After application of Factor analysis, if we have used orthogonal rotation, we will be sure that there is no correlation between factors
Knowing the *counts*--not just the proportions--is crucial no matter what you do.  But once you have the counts, the first model to consider, even if it's just a point of departure, is logistic regression.
I'm doing more than just precision and recall; those two metrics were just cited as examples of whatever it is I'm talking about.
I'm confused... If the summary reports them, I suppose they must be useful for something? Also, the main reason I'm computing a model is that I want to find out which level can be said to have what kind of influence with a certain degree of certainty. So if they tend to be "meaningless" there's little reason for me to do this. Or would you recommend another method to do this?
You can "plug values" into cubic splines (and many other local interpolators), so what's the problem with them really? After all, if your data exhibit complicated behavior, you should expect to need a lot of parameters to reproduce it.
Can you re-phrase your question to be more specific?  The way you've phrased it, I would suggest you make a table of users and functions, but I don't think this is what you're after.
@highBandWidth you are correct, I have changed my answer. Perhaps it would be better to rephrase the question 'Is there a conjugate prior...'
So is the thought that since $E[h(X)] = \\int_{-\\infty}^{\\infty} h(x)f(x)dx$, it may not be clear which probability density function, $f(x)$, we're taking the expectation with respect to? So for example, if we have two pdfs, a(x) and b(x), that we potentially could take the expectation with respect to, we would write either  $E_{a}[h(X)] = \\int_{-\\infty}^{\\infty} h(x)a(x)dx$ or $E_{b}[h(X)] = \\int_{-\\infty}^{\\infty} h(x)b(x)dx$, to make it clear which one we want to use?
I checked all your links and downloaded the last one but to me all look elementary statistics as I categorized them as a.
It doesn't strike me as valid to say that \\$12 is a count variable with 12 currency units as the realization. You probably want something like Gamma.
Thanks Peter ! the forest ensemble will pick the two features randomly if they provide the same split. There should not be any gains in keeping both features. I could remove one of them and I should get the same classification result.
As you have stated you have just one contingency table and hence AUROC are not applicable that is true. In this multinomial case the terms like sensitivity and specificity should be avoided. You can simply term them correct prediction: = $P(Y=i|D=i)$, where $i$ corresponse to surgery,...,. $Y$ is the Top recorded plan( Predicted)  and $D$ is the Recorded plan (Known).
Dependence. What I wanted to do is to learn the graph from data. I thought about calculating Pearson correlate coefficient between all pairs of variables and use a threshold on those coefficients to decide if they are dependent or not, but it does not sound like a correct solution. Thanks for the link.
Question 2 is a simpler version of http://stats.stackexchange.com/questions/52004/overlapping-sets-3-significance-test, which concerns three samples. If you take any one of those three to be all the contents of the container you will obtain the answer.  Question 1 is a special case of question 2.
Thanks you very much @whuber.
There are no controls so it is neither, rather a descriptive study
Sorry, but I can't upvote because I don't have enough reputation.
It will also partly depend on the nature of the questionnaire and what you are trying to do and on the distributions of the other questions. Is this an ability test? Attitude? Are you interested in people who are at the extremes?
@GavinSimpson - honestly, I am not following what you're saying - I'm no specialist in the topic, and this just sounds too complicated to me. So please correct me - the way I see it is that *analysis of variance* relies on normal distribution, because of the *variance* which is a parameter in that distribution. Therefore, I think, you can't do AOV on data that is not normally distributed, because you're estimating a parameter from normal distribution.
Can you say anything about the distribution of the numbers in $S$?  One worry is that $S$ could contain, say, $N-1$ values in the interval $[0,1]$ and another value near $N^2$.  Unless the sample is large enough to contain that last value with high probability, neither the sample mean nor the sample variance will be anywhere near the mean and variance of $S$ itself.
Ah, I see it: in the notation used by Ng, $J$ is a cost function which includes the summation, see again here http://class.coursera.org/ml-003/lecture/view?lecture_id=105, minute 7:30.
@Aksakal, I have never heard the word [multinomial](http://en.wikipedia.org/wiki/Multinomial) used that way in English. When you are predicting 1 of several classes given some information about predictor variables, you can use [multinomial logistic regression](http://en.wikipedia.org/wiki/Multinomial_logit) though, so that may be what you are thinking of. But we still say that you have only 1 response variable, just one w/ multiple levels.
is this a one-sided test?
(2) is actually a measure of relative *precision,* not relative error, because it only incorporates variation in the estimate $\\tilde n$ without accounting for a possible systematic difference between $\\tilde n$ and $n$.
Hi, @timbp, would you mind elaborating on this a little? It's nice to have a bit more information.
To whoever downvoted: It might help to improve my answer if you clarified what you thought the problem was. I'd appreciate an opportunity to give a better answer.
That probably won't be Gaussian.
From your description, I'm guessing you mean that corruption is your *dependent* variable, and I'm guessing that it is ordinal as well at press freedom & democracy. Is that right?
So, in R parlance you're doing something like `lm (S ~ T, data=X)` where X is the data for one day? And the relationship between S and T changes at irregular intervals because of repair work and also changes throughout each day? The changes that take place during the day are due to what? (That is, the temperature changes during the day, but what else affects it? Precipitation, who-knows-what, rail traffic?) If you know what causes the changes within each day, you may be able to add that to your model. Knowing some more details would help shape an answer.
(I am completely in favor of doing permutation tests with the most suitable statistic for your problem; that comment was simply clarifying that the Wilcoxon isn't distinct from permutation tests.) If you're looking for robustness to outliers, simply construct a permutation statistic that has the robustness properties you want, while retaining the other properties you need it to have.
Are you trying to compute a standard error of some quantity? What quantity is it?
With respect, this looks like a direct copy-and-paste from Wikipedia, and doesn't really answer the question.
err: 'but this in no way implies that the standard deviation is, in general, connected to those percentiles.' A bit too [strong?](http://en.wikipedia.org/wiki/Chebyshev%27s_inequality#Statement)
Excellent analysis.  I would simply add that this assumes the distribution for the random variable is bimodal about two different means and the same variance about each.  Suppose not only the mean value, but also the variance (or the shape of the distribution) changed based on x.  Then I think you would have to model both populations separately as whuber suggests above, and the impact of x may not be measured with a single number.
I would personally omit steps 3 and 5, as standard methods don't really need either of them (SVM, RF, ANN) and removing information is usually a bad idea.
Thought I'd quote your comment to the accepted answer. (I think editing your question would be helpful.) "I should have mentioned in the post that the 156,000 was the maximum transactions observed in one hour over a four month period."
Sounds like a case of [Simpson's paradox](http://en.wikipedia.org/wiki/Simpson%27s_paradox) (- see the diagram in particular). What are these subsamples? Are you splitting on the values of one dichotomous variable?
I noticed that I could use the simpler approach long ago, but I was determined to dig deep and come up with the same answer using different approaches, in order to ensure that I understand the concepts. I realise that first $\\sum_j \\hat{u_j} = 0$ from normal equations (FOC from least square method), so $\\bar{\\hat{u}} = \\frac{\\sum_i u_i}{n}=0$, plus $\\bar{\\hat{u}} = \\bar{y} - \\bar{\\hat{y}} = 0 $, so $\\bar{y} = \\bar{\\hat{y}}$. So there won't be  the term $\\sum_j \\frac{u_j}{n}$ in the first place.
According to ROCR manual - no, there is no elegant solution. There are no density function already built-in.
Perfect as far as what I was hoping for. It is amazing how well the data matches the curve. Thank you!
A very closely-related idea is given in the adaptive estimation of [Mukherjee and Chatterjee 2008](http://onlinelibrary.wiley.com/doi/10.1111/j.1541-0420.2007.00953.x/abstract), who give a point estimate that moves between a robust/less-robust version, depending on how the assumptions of the less-robust version are satisfied. They give standard errors too, so you could turn it into a test with the general characteristics you seek. As far as I know, the idea has not yet been generalized.
Thanks chl, the version number was a type (zeros were supposed to be nines), but your tip helped. plotLMER.fnc works again after the update. So, my immediate issues are solved. However, I think I'll leave the question here anyway (unless a mod thinks otherwise?) if anybody wants to discuss other ways of getting intervals outside of the plotLMER function.
Corrected title. Cheers for the spot.
@SteveS: I updated my "summary", hope it will be less confusing now.
@Aksakal I have updated my question with the real problem and some more details. Thank you.
About your second bullet: the random $\\Theta$ is (a.s.) the limit of $\\bar{X}_n$, as stated in De Finetti's LLN. So, when some Bayesian says that my prior for $\\Theta$ is $\\mu_\\Theta$, he means that this distribution represents his uncertainty about this limit, before having access to data. Different Bayesians may have different priors, but, with suitable regularity conditions, they will have $\\textit{a posteriori}$ agreement about $\\Theta$ (similar posteriors), as they get more and more information about results of the tosses.
@loganecolss - that's why I'm a Bayesian.
aside...Mplus can also generate bootstrap standard errors - advised to try and account for the extra variability in polychorics - see ch12 of mplus examples on how to implement.
By default, most mathematicians consider a "vector" to be a column vector so that matrices operate from the left. Regardless, the context usually suffices to tell.  In this case, interpreting $X$ as a row vector makes the statement utterly trivial (a test for convergence of $J$ values is replaced by a test for convergence of $J^2$ values, which subsumes as special cases the original $J$ tests), whereas interpreting it as a column vector gives it some content.
I suppose it's to some extent a philosophical question, but it's not necessarily the case that _any model has to have a strong background in discipline-specific theory._ Models that are theory-driven are useful in some instances, but one could cite "theory-dependence" as a disadvantage. At the other extreme, there are data-driven models and these, too, have their uses, however, "sample-dependence" may be cited as a disadvantage in this case. Perhaps one should find a middle-ground to reduce both theory and sample dependence? Arguably, discipline-specific theory is not necessary nor sufficient.
Because this doesn't point to material, it isn't an answer, but it may give you some insight: L1 regression of $n$ values against a constant fits *a* median (which might not be unique), whereas L2 regression fits the mean.  Finding the mean is obviously a $O(n)$ computation needing only $O(1)$ local storage.  Finding a median naively takes $O(n^2)$, requires $O(n\\log(n))$ with the best available sorting algorithms, and actually needs only $O(n)$ time and $O(n)$ storage provided a clever algorithm is used.  That algorithm is considerably harder to code than one for the mean.
You will get better responses if your question is more specific. What are the models you have in mind and what do you want to achieve with this approach? What are your initial ideas to approach this problem?
Thanks. I'm using the following to interpret the posterior. Might not be the most efficient way.

`import numpy as np`
`c = [10, 7, 2]`
`alpha = [.5, .5, .5]`
`alpha_post = np.array(c) + np.array(alpha)`
`X = np.random.dirichlet(alpha_post, 100000)
np.apply_along_axis(lambda x: all(x[0] > x[1:]), 1, X).mean()`
Okay so then if I assume you that you worked that out for the chi square distributions then as you pointed out since X2=Z-X1 X2/Z=1-X1/Z which is just a constant minus a random variable that is independent of Z and hence is independent of Z.
(+1) Nice answer (and awesome new gravatar :-)
I don’t get the rationale of your definition of $\\widetilde p_i$. I would have put $Z_i = \\Phi^{-1}(1-p_i/2)$, and use as a score $\\sum Z_i^2$ which follows a χ²(k) distribution.
+1. There is a long-run difference in the performance of your analyses vs. Test Everything's analyses, however. Every time this analysis is run, you will use the same strategy, based on what was written in the literature (which doesn't fluctuate experiment by experiment). OTOH, the data *are* a random sample, & the output from Box-Cox testing will fluctuate study by study.
What is your "Var(X)"?  It's definitely not the variance!
That depends on your clustering algorithm; what are you using? If you want inspiration try [this](http://www.inf.ufrgs.br/~vbogorny/acmsac2008.pdf).
Linear regression requires normal errors??
StatTistician, why define $r_t$ at the start only to call the same quantity $a_t$ on the very next line? You don't need two symbols for the same thing.
(1) I do not see in your reference where it claims the gray area is the pointwise confidence interval.  It seems pretty clear from the examples that the gray area is instead a confidence interval for the *curve*.  (2) Nobody would reasonably declare the large proportion of points beyond the gray area as "outliers"; there are just too many of them.
Yes thanks to your books yes...now I know...Thank you for your answer! :)
By the way, if you are familiar with k-means, it handles well large-ish data. Maybe you should explore ensembles of just k-means runs. Another thing, mini-batch k-means updates centers based on random samples of data so it is suitable for very large data and is supposed to sometimes converge to the same optimal points (you can check the paper and implementation in the scikit learn python package). There are a few packages in R for cluster validation associated to good theoretical sources: fpc, clValid. You can also maybe look a bit into separability measures like Jeffries-Matusita.
A "plain vanilla" NN will keep everything, just use low weights on irrelevant information. (Of course there are tons of different flavors of NN, some of which may well include discarding inputs altogether - I am not an NN expert.)
You explain FDR vs. Bonferroni, but don't define "positive dependency" but rather just reword it to "positively correlated" but I don't understand. Consider that I have 1000 P values, testing expression of 1000 different genes comparing people with and without some disease. I use the BH method to decide which of these comparisons are "discoveries". What does "positive dependency" mean in this context?
And what you said is largely correct. CSI could have modified the effect of Ads, and Ads could have modified the effect of CSI on the factor score. When there is a significant interaction, interpret the main effect AND the interaction because main effect alone is no longer universally applicable.
Basically if the mean varies over time, one way to model that is with ARMA. The residuals from a ARMA regression would (ideally) no longer have a time-varying mean. If it is important to do both, then you can take a pseudo-maximum likelihood approach, fit the ARMA model, get the residuals, then fit the Garch model to the residuals.
@Yves: Thanks for your comments. To be clear, the procedure I've outlined gives an *exact* realization of the continuous-time process sampled on the corresponding partition; in particular, there is *no* discretization error like there is in typical Euler-scheme approximation to more general SDEs. The inverse Cholesky, as shown by the construction in the answer has nonzero terms only on the diagonal and lower off-diagonal, so it's a little simpler than tridiagonal.
Thanks for your explanation. BTW, the error I wrote above is the average I got performing k-fold validation (with k=10).
What do you mean by "the significance changes according to the order of `a`, `b` and `a*b` in the model"? Is that `a`, `b` and `a*b` have different p-values? Or the p-values will change if you change the order of `a`, `b` and `a*b` in the model setting?
Please: make us know, briefly, what is "LDA", what is "penalized", which package you use. Then show the output and point to the difficulties there.
Well, you should not use the sign rank test. I would say that the sign test would be appropriate. I learned my nonparametrics out of [Lehman](http://www.amazon.com/Nonparametrics-Statistical-Methods-Based-Ranks/dp/0387352120). Using the rank test makes sense, because you have before and after treatment data. The one thing you need to take into account would be a control group, as time would probably play a factor in pain levels.
@gung I'm sorry you're getting flak for trying to help--please accept my thanks, at least, for all you have been doing for us. I have deleted the comment in question.
You should expand your question on how you will analyze similarity: Are you using a `tanimoto similarity score`? Euclidean distance? Clustering? Pearson correlation? IMO that needs to be stated in order for us to best answer.