tttjjj/elt_bench huggingface dataset:
The p-value the `wilcox.test()` reports states the probability of finding data *as extreme or more extreme than yours, **IF** the null hypothesis is true*, nothing else. Specifically, it does not suggest "that the alternative hypothesis is true". I'm sorry for being a little pedantic, but it's important to be clear on exactly what statistical significance means.
I take "sensitive" to cover a lot of reasons - ranging from "I'm not answering this because it's too personal" to "This survey is boring me now so I will skip the rest of the questions". If you get different conclusions, I would be conservative and stick with the non-imputed results because I don't think imputation algorithms were designed to address non-response due to sensitivity. But that's just a personal practice of mine; I don't think there's a hard rule in the community. Ultimately, I think it has to do with how confident you are in your imputation algorithm.
@pnuts: Thanks. Glad to try to help. :)
Great thanks! That makes sense. 
So if I want to analyze data in the binomial format (as in my first example), how do I do a test which takes into account that there are actually more than 1000 data points?
Thank you very much. I am bookmarking this.
Your question is not clear. Can you say more about your situation, your data, & your goals? Are you asking if you can use Tukey's HSD for *a-priori* comparisons, or if you can use a lot of $t$-tests instead of Tukey's HSD for many post-hoc comparisons?
Thanks for this nice answer. I appreciate it that you took the time to write it up that nicely. Maybe you could mention it in the discussion on GitHub; I am sure that would be of help to fix the LDA in the next version of sci-kit
@suncoolsu, I edited the latex code for better readability. If it is not ok, please feel free to rollback.
Could you provide some context? What are your data? What do you want to know?
Your statement that a random effects model is unproblematic with only 13 higher level units (in this case countries) is not shared by everybody. See for example this simulation study: https://www.iser.essex.ac.uk/publications/working-papers/iser/2013-14
I would prefer a closed form formula so I can easily apply it to different values of n, m and k.
Have you tried this strategy on the dataset shown [here](http://stats.stackexchange.com/a/50780/603)? More fundamentally, your strategy amounts to claiming that outliers can be found reliably from the results of a chain of fits minimizing a convex loss function, which is a known [fallacy when there is more than a single outlier](http://projecteuclid.org/euclid.aos/1176343347) (this links shows this for the related problem of finding multivariate outliers but the results apply to regression as well).
Thanks! That is something I'm searching for. One of good new articles. It's a pity I can't vote-up =(.
Can anyone suggest any source regarding my query please?
The first paragraph above assumes that the cutoff is correct and does not vary across subjects, i.e., that the utility function is trivial.
Coin flipping can absolutely be considered as a sample from the space of the possible coin flips.
Alas, the book has no reference to SPSS nor any other software. It is also B&W.
@Jyotirmoy, what if the minimum happens at the boundary of your parameter space?
Is your call to `glm` meant as pseudocode? I can't get that to work with simulated data. Also, is there any possibility that these 99 dependent variables are dependent (even after conditioning on the predictor variables)?
Duplicate of [How can I calculate $\\int^{\\infty}_{-\\infty}\\Phi\\left(\\frac{w-a}{b}\\right)\\phi(w)\\,\\mathrm dw$](http://stats.stackexchange.com/q/61080/6633)
A discussion following a now-deleted reply noted a **possible ambiguity** in this question: do you seek the SD of the monthly averages or do you want to recover the SD of all the original values from which those averages were constructed?  That reply also correctly pointed out that if you want the latter, you will need the numbers of values involved in each one of the monthly averages.
Welcome to the site, @user24799. Would you mind saying a little bit about this package? Why do you think it's the best?
As you mentioned correctly, I should use a filter to estimate the future state of the target considering both the ideal dynamic model and error. The only thing that I still can't understand is that how should I use the corrected model to predict the position of the object after a long time?
please define $r$, $S_x$ and $S_y$.
+1, this is really nice.  It addresses some of the points I was trying to make much more clearly and thoroughly than I did, and also brings in additional ideas that are quite germane.
@cardidal: thanks for the editing suggestion!
As stated, this question does not follow our guidelines for posting questions, see our [FAQ](http://stats.stackexchange.com/faq). This is too broad and it's hard to tell what's being asked there. Please, try to reformulate your question such that objective and arguable answers can be proposed; otherwise, we will close it.
Thanks DWin. I personally think the segments should be made based on business logic but wanted to know if there was a statistical way to determine this. Also, hese be introduced as independent variables or should i include the interaction terms. If so, how do i determine if the interactions need to be added? I plotted the average response (binary) against the interaction terms to check for a trend. Some interaction terms have the same average response rate. For e.x. East-Male and East-Female have similar response rate. Can i use them as independent variables? Thanks!
What do you mean by "asymmetry"? There is no need to try an ECM based hypothesis test on the cointegrating vector since a linear combination will be $I(0)$.
"Finally, use the fact that the probability of the union of two disjoint events is the sum of their probabilities."  This is not one of the "axioms" that the OP is given but needs to be deduced from the two that say `P(False) = 0; 
P(A v B) = P(A) + P(B) - P(A ^ B)`  In other words, there is more work to be done for the proof "using axioms and substitutions" that the OP wants.
Agreed on the last 2 paragraphs! One doubt:

`What can be a limitation in much of statistics, the sensitivity of means and variances to squared deviations, can be a virtue in cluster analysis in so far as clusters are tight and compact.`
Can you be a little more elaborate on this? What sensitivity are you talking about?
Look into [three-way component analysis](https://ppw.kuleuven.be/okp/_pdf/Kiers2001TWCAP.pdf)
I admit that this may not be extremely helpful at this point in time, but I would suggest that the next time, you ask yourself this question _before_ performing the study. If only to allow you to do a proper sample size requirement analysis. That said, I just voted +1 on the answer by @Marcus Morrisey below.
Actually, I think I was misled by your title; while your predictors are binary, the part that matters in terms of figuring out what procedure is required - the response - is not binary, but ordinal. You might do that with ordinal logistic regression, for example.
yes, but my point is: you do not remove the denominator. You can ignore it when sampling the new value of $z_{i}$. Another thing is when you want to calculate estimates of probabilities.
Clever, and I certainly can't think of a better way of doing this.  But for Q1 to make sense we need another constraint; otherwise the smaller $n'$ gets the better it meets your criterion, with $n'=0$ meaning you don't have do any new sampling at all.
I'm not sure I recognize the upshot of this question.  It is true that if you just have a vector of numbers in R, there isn't a lot of metadata associated with it, but why would that bother you?  Why would you need that / what would you want to do with it?  Suppose it did have such, it would only be helpful to the extent that you were to pass that vector to a function with specific methods for Gaussian data vs. other.  I don't know of any (although I'm hardly the world's most expert R user).
**NB** Please do not attribute these thoughts to me!  I'm just doing my best to articulate and defend the stated model, as requested.  Whether this model is appropriate, applicable, well-fit, etc., is a separate issue altogether.
OK, that is good to know ! Thank you ! I am familiar with Bonferroni adjustments from our statistics clinics, is that appropriate here? I would have to divide my alpha level of 0.05 by three, making the new alpha level .017?
I don't know the class of distribution. Is there any general approach? Binomial distribution does not work for some mean and variances (0.6,2.64 for example).
Simplest thing I can think of: `s1 = [100, 120, 40, 200, 1000, 1400, 1300, 1500]; np.diff(s1).mean()` That is, obtain first order difference of your series and then calculate the mean. Repeat for every series. Finally, normalize to get a value [0,1]. In order to compare them (assuming series of equal length), use euclidean distance.
This is a very good answer (+1), but could be improved in terms of the ordering of the approach.  We start with saying we want a linear combination $z=Cx=C_1x_1+C_2x_2$ of the whole vector that is independent/uncorrelated with $x_2$.  This is because we can use the fact that $p(z|x_2)=p(z)$ which means $var(z|x_2)=var(z)$ and $E(z|x_2)=E(z)$.  These in turn lead to expressions for $var(C_1x_1|x_2)$ and $E(C_1x_1|x_2)$.  This means we should take $C_1=I$.  Now we require $cov(z,x_2)=\\Sigma_{12}+C_2\\Sigma_{22}=0$.  If $\\Sigma_{22}$ is invertible we then have $C_2=-\\Sigma_{12}\\Sigma_{22}^{-1}$.
I didn't see it as basically difficult but it would be nice if there was something like the boxcox tranforms that automatically plot out the best selection for lambda.  Yes, not terrible to implement...
@asd123 1) when you write $\\Sigma$ it suggests that $x_1$ and $x_2$ are vectors, in which case the product $x_{1}x_{2}$ isn't defined as written (unless $n=1$).  Do you mean $x_{1}^{\\mathrm{T}}x_{2}$? If not, what do you mean? 2) Without independence it is not necessarily true that $(x_{1},x_{2})$ is jointly normal, so it would seem that you would need more information about their joint distribution (and/or variance/covariance matrix) before you could say anything definitive.
In R, to see which confidence intervals don't contain the mean, use `which(results==0)`. Then it's the _columns_ of `tsamples` which contain the data, so you can use `hist(samples[,j])` to get a histogram of the `j`th sample -- set `j` = the first number returned from `which(results==0)`.
The exponential is a special case of the Gamma, so you can actually construct a test by looking at the confidence interval for the MLE of the shape parameter of the Gamma distribution.  If it includes "1", then you can't reject the null hypothesis that the dist'n is actually exponential at whatever level of confidence corresponds to the confidence interval.
I would consider varying intercept fixed effects, see http://www.stata.com/support/faqs/stat/xtreg.html . It probably isn't a bad idea to check out those Stata FAQ's to see if they have anything pertinent to your question as well.
If your knowledge of machine learning or statistics is low I suggest you to avoid neural network and to use a simpler tool. For example random forest.
Also, if you are using R, just want you to be aware that there is a great built in function called p.adjust which will adjust your p-values and has all of the methods you mentioned (and more) available.
I am revising my answer since when i went to get you the statistical reasoning i realized i hasted into a conclusion that was not right
@Josh The peak is ~2x the plateau which seems like a hint. Perhaps the protein is dimerizing. A quick search indicates the florescence of GFP is suppressed by dimerization, which is a function of concentration: http://www.ncbi.nlm.nih.gov/pubmed/8578587
Thanks I'll take a look at this
(+1) very well done, again, and spot on dead-center.
Do you have a measure for how latent they are?
Classification and clustering have much in common.
1) What kind of variables are L and D? Continuous? Categorical? What values can they take? 

They are continuous probabilities, so can take values between 0 amd 1?

2) Given know P(L|D) and P(D|L) exactly, could you tell us what they are? 

P(L|D) = 0.385, P(D|L) = 0.735

3) What models have you used to estimate P(L) and P(D)? Why?
It's the same thing.. the diagonal is rolled into the other two matrices when they talk about two matrices.
not in the last question.
I'm not sure why this post http://stats.stackexchange.com/questions/17129/how-to-average-spearman-correlations was merged with this one.  They are actually asking two different questions in my opintion -- there are two different goals.
Google "Unsupervised learning". If it's time-series; wavelet power spectrum, visualization of correlation matrix, running window PCA and plot the eigenvalues to see how the global correlations change over time, big table of descriptive statistics (quantiles, moments, etc).
I have to say, I had a hard time reading past the first "paradox"; an author who opines on statistics and decision-making while, it would appear, having no knowledge of statistical decision-making, is not to be trusted on the applicability of statistics in general.  Also, as Russell and Whitehead showed, logic is a part of mathematics, and of course so is probability theory, so they can't be inconsistent with each other - unless mathematics itself is internally inconsistent.  As for paradox #2, ask any actuary or gambler about whether probability can be applied to real life.
I'm afraid I can't follow what you're doing. Perhaps you could illustrate with a tiny version of your data, showing us more clearly what you mean by a "dataset," what you mean by a "sample," exactly how you perform the "splitting," how you manage to obtain *two* p-values for each t-test, and what distinction (if any) you are making between "p-value," "significance," "significants," and "alpha."
I missed your 20 Jan comment. Take the feasible combinations of $(X_{(1)},\\; X_{(n)})$ (that are qualitatively different, eg. both negative, one negative and one positive, one negative one zero etc), and calculate for each combination the $\\max \\{-X_{(1)},\\; X_{(n)}\\}$, which is $\\hat \\theta$. You will see that it will always be a non-negative value, and also, by assumption, lower than $\\theta$.
This is (obviously) two applications of the result at http://stats.stackexchange.com/questions/61080/how-can-i-calculate-int-infty-infty-phi-left-fracw-ab-right-phiw.
It's still not entirely clear what you're doing. I start out feeling like I have understand the situation, but then as I pass from your third paragraph to your fourth, I find myself going "huh?". Repeated re-readings don't appear to reduce this remaining confusion. Can you clarify further, please?
@moderators: Community Wiki?
@Macro: +1 to you as well. Variance and interval width are of course more or less equivalent here. Great minds think alike - and so do ours. ;)
Thanks for your answer. Your last comment on the interaction term was very helpful. Are there any non-linear models that include such interaction terms automatically?
Non-parametric ANCOVA is available in the [sm](http://cran.r-project.org/web/packages/sm/) R package (`sm.ancova`).
Thanks. I have found algorithm. here: http://www.stata.com/manuals13/rsignrank.pdf
@Placidia Round is indeed the lowest level, and Investment is the response measured each round. In including Round in the model, I am following the instructions by [Bates](http://lme4.r-forge.r-project.org/lMMwR/lrgprt.pdf) for a longitudinal (two-level) model with correlated random effects (sidenote: just realised I wrote "correlated fixed effects" in the question; I edited that). The code Bates supplies (using the sleepstudy data set) is

`fm06 <- lmer(Reaction ~ 1 + Days + (1 + Days|Subject), sleepstudy, REML=FALSE)`
@Jaap a 'random slope' would mean that the direction / slope, as well as the magnitude of an effect could vary by person.  This sort of consideration takes off from where the discussion at http://conjugateprior.org/2013/01/formulae-in-r-anova/ stops.
In this application the "total number of things that can happen" is infinite. How, then, do you apply this "definition" of probability?
Also, I suggest you watch http://videolectures.net/mlss09uk_jordan_bfway/ by Michael I. Jordan, a prof at berkeley who is quite balanced in his views on the supposed Bayes vs Freq. "war". I can't really comment on the second half of the first paper b/c I don't know any of the ecological references. I'll read the second one later.
@Michael Bishop-If i transmit packet then it is represented by "1" otherwise "0".Every row i transmit in total 9 packets that's why every row there is nine "1".
Could you outline the exact procedure how the coefficients are calculated? From what I know about gram-schmidt ortogonalisation and how it can be applied to regression problem, I can presume that by using the gs procedure you can get fit of the regression, but not the original coefficients. Note that regression fit is the projection to the space of the columns. If you orthogonalise the columns you get the  orthogonal base  of the space spanning the columns, hence the fit will be linear combination of this base, and also linear combination of original columns. It will be the same...
Maybe you want to do something Bayesian?  The results from the previous study would be used to set your prior for the parameters in the regression, and then you update the priors with the data from the new study.
Can you post the papers you are referring to?
By default in this case Stata uses a structure that "allows a distinct variance for each random effect within a random-effects equation and assumes that all covariances are zero". Stata calls it an _independent_ structure, but afaik it's also known to be a _variance components_ (VC) structure. So, it should be the same.
(+1) Incidentally, as I was first considering this problem, the fact that $\\max_i |X_i - \\bar X| \\leq S\\sqrt{n-1}$ was actually the initial clue that the sample inequality should be tighter than the original. I wanted to squeeze that into my post, but couldn't find a (comfortable) place for it. I'm glad to see you mention it (actually a very slight improvement on it) here along with your very nice additional elaboration. Cheers.
"Contrast matrix" is used to represent categorical IVs (factors) in modeling. In particularly, it is used to recode a factor into a set of "contrast variables" (dummy variables being just an example). Each type of contrast variables has its own corresponding contrast matrix. See for example my own related [question](http://stats.stackexchange.com/q/63639/3277), not answered yet.
was this what you wanted?
How many datapoints do you have? What kind of SVM are you uisng -- linear or Gaussian kernel (or some other kernel)? Kernel methods in libsvm are much slower.
You can do two cases... Maybe, an information is missing, yes.
@user27271 - Following the link above you could prepare a simple *select case* procedure that would accept your first 4 moments, compute B1 and B2, and identify a potential distribution.  This could be combined with a histogram for inspection purposes.  It would be fairly straightforward to do this in R.
Related -perhaps- http://math.stackexchange.com/questions/16455/how-to-evaluate-probability-estimators-with-only-external-information/31245
I have tried that, it doesn't help. I also used step-halving in the code, doesn't help, either.
For other ways to think about the problem (especially versions with more points in the scatterplot) please see the varied answers to a generalization of this situation at http://stats.stackexchange.com/questions/33078.  (In the generalization the information about the colors is lost.)
I figured it out today. so all good! by contributions i mean the portion of the predicted explained by each independent variable. BY process, i mean in an additive model, these contributions can be added up to find the predicted. In a log-log aka a multiplicative model, the process of calculating the total contribution is different. Which i could not figure out, but now have! Cheers :)
For background literature, in case you aren't aware: http://en.wikipedia.org/wiki/RAND_Health_Insurance_Experiment
@Tomas I tried to sum up the main difference and when each CI might be appropriate.
For each of your binary variables, you may consider setting true to be +\\alpha and false to -\\alpha, where alpha is an adjustable parameter, then use any of the similarity measures you mention. You can then fix \\alpha by eye or cross validation to be more rigorous.
It's not "wrong" to use the multinomial logistic model. True, it doesn't take advantage of the ordinal structure in the data but, as I said, the ordinal model is a submodel of the multinomial model. Therefore, any fit achievable with the ordinal model is achievable with the multinomial model. If the proportional odds assumption does hold, you're sacrificing parsimony by using the multinomial model.
Exactly as Nick suggests (and that was why I asked) - the lower p-value in the second case (even though the differences in density estimate look similar) will be largely due to the larger sample size, since a more precise estimate serves to reduce the possibility that the observed difference is due to random variation.
At the moment I only have access to SPSS, but because of the way it works with cases there is a problem when I combine the two sets to data in order to run any models. The existing cases with voter info on each line end up having a random vote put on the same line and SPSS seems to interpret this as that vote belongs to the rest of the info on that case. Any idea how I would get around this?
IMO it is always a good idea to cite the paper you are discussing. (I see no conflict here - it is not as if you are being critical anyway.)
Are you trying to come to some overall conclusion about the first 15 ms having a higher rate, or is this for a collection of separate conclusions about the initial period (say under different conditions where some might and some might not have higher rates)?
I don't think "Is $R^2$ high enough?" to be a meaningful question. In some cases, the *true* (population) model can have $R^2=0$. Is the correct model "good enough"? (I would have thought so.) --- and if that's so, on what basis would we conclude that substantially higher $R^2$ values *aren't* 'high enough'? I suspect your real question of interest is probably not about $R^2$.
How would using a static/compiled language save you from an API change?
If you want to know if the fit is better, take a look at measures of fit for both models. A common measure is the area under the ROC curve (AUC). If you are wondering about the models' out of sample performance, you can cross-validate to get estimates of out of sample AUCs. I wouldn't worry too much about the z-scores in either case.
@MrAlpha I used AUTOBOX available from Automatic Forecasting  Systems (www.autobox.com) . I am one of the developers .
Your title and contents show some confusion between the terms "dependent" and "independent". Please check that my edit preserves your intended meaning. The fact that people get confused about which is which strengthens the case for more evocative terminology, such as "response" or "outcome" rather than "dependent variable". Finally on abbreviations note that for many people "IV" means **instrumental variable**.
ok: I already noticed one big mistake: My "from" vector in hyperbChangePars is wrong, so I have to use the order of the output. But my question stays the same!
It depends on the suitability for the "taste" of visualization. For example, to explore/show time series the IPCC web have enough data and is widely used (obviously for analyzing the climate change), to show spatial data the Human Development website contains a lot of space-related data as well as data related to time.