tttjjj/elt_bench huggingface dataset:
This seems to be a very common question. You may also want to look at these related Q's on CV: [what-if-interaction-wipes-out-my-direct-effects-in-regression?](http://stats.stackexchange.com/questions/5450/); [interaction-suppresses-the-main-effect-how-to-interpret-it](http://stats.stackexchange.com/questions/16178/); & this is also related (albeit from the other direction): [how-to-interpret-inconsistent-beta-values-in-different-steps-of-hierarchical-reg](http://stats.stackexchange.com/questions/32653/).
Indeed, I used a parametric model (Weibull) to draw the plots. H0(⋅) can be estimated non-parametrically (cf. Breslow estimator). I think that this is what you did with 'stcurve'. However, computing the jacobian is the non-parametric case is not obvious to me...
I think he tries to do basically the example from chapter 1 of the Duda-Hart book. There are two classes, points from one are distributed normally with parameters x, y, points from another are normal with params a, b. What is the posterior probability of an observation given each class, and what is the decision rule.
I wanted to test for a difference in values between my categories, and was going to use a kruskal wallis test. After reading this post: http://stats.stackexchange.com/questions/70207/advantage-of-using-anova-over-a-kruskal-wallis-test (also linked to my original question), I thought another test may be more appropriate if my values are ordered.
I think this is a bad example in which to include the point `which implies that $X$ and $Y$ are uncorrelated; but the converse is false)` because for _Bernoulli_ random variable $X$ and $Y$, $E[XY] = E[X]E[Y]$ _does_ imply that $X$ and $Y$ are independent.
@FloraZhou Welcome to the site! Please note that it is a Q&A not a forum, so please consider [asking a new question](http://stats.stackexchange.com/questions/ask). You can read more about how this site works in the [FAQ](http://stats.stackexchange.com/faq).
How exactly are you fitting the data and how do you combine all the variables into 1?  Is your purpose prediction or identifying the important variables?
You have some "zero inflation" here (nearly half of your counts are 0). Also, if the data have been entered in the order in which they were observed, there is some systematic effect (a long stretch of 0s, and three of the four high observations all occurred close to each other).  That said, you might try a bootstrap computation for the confidence interval for the mean. The relevant R packages are "boot" or "bootstrap". I think the t.test CI is definitely not right.
Why MiniBatch instead of regular k-means?
I love the comment "it allows you to do advanced computations without having the... knowledge required...".  This is, in fact, the primary competitive advantage of SPSS products, and reflects its market strategy; in exchange for this you pay a hefty price, as you note.  However, being able to run advanced analyses without the required understanding does not bode well for the quality of the resulting findings.  Although R requires a steep, up-front learning curve, it will benefit you in the long run.  In addition, there are plenty of tutorials and learning materials available, and it's free.
Thanks @whuber - I've updated my post so that it is more specific.
please don't hesitate to post an answer because you don't think i will understand it. first-- it won't hurt to provide more information, even if takes me some research in order to grasp it. second-- the stackexchange model assumes that this question and answer will benefit others in the future. in other words, don't dumb down your answer based on how much you think the OP knows, or you'll be doing others a disservice.
+1 For this literature compilation. Some of them are frequentist, rather than classical (Fisherian), but still very interesting. Are you suggesting to "change the approach" in order to solve "what can we do"?
@AtillaOzgur there are many links talking about supervised clustering, I added some of them to my post:

  [1]: http://www.cs.uh.edu/docs/cosc/technical-reports/2005/05_10.pdf
  [2]: http://books.nips.cc/papers/files/nips23/NIPS2010_0427.pdf
  [3]: http://engr.case.edu/ray_soumya/mlrg/supervised_clustering_finley_joachims_icml05.pdf
  [4]: http://www.public.asu.edu/~kvanlehn/Stringent/PDF/05CICL_UP_DB_PWJ_KVL.pdf
  [5]: http://www.machinelearning.org/proceedings/icml2007/papers/366.pdf
  [6]: http://jmlr.csail.mit.edu/papers/volume6/daume05a/daume05a.pdf
I can't really tell. Sometimes these things happen by accident. As for example, everybody knows about the bias-corrected version for the variance, but we don't think much about bias for higher moments. So the function was built by using the sample analogue for the fourth moment and the already created function for the variance -which had the bias correction term.
This question is answered (somewhat opaquely, because the details are buried in `R` code) at http://stats.stackexchange.com/questions/10947/formula-for-autocorrelation-in-r-vs-excel.
It's also worthwhile to note that effects of market manipulation have been experimentally studied in prediction markets (see [here](http://hanson.gmu.edu/biashelp.pdf) and [here](http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0010/5968/biastest.pdf)), and while more work needs to be done it appears that participants can easily compensate for malicious manipulators. The empirical results suggest that it would be extremely difficult to 'game' the system, as you mentioned in your [other comment](http://stats.stackexchange.com/a/28516/8927)
Thanks for that. It's a c-SVM. Based on the 2nd paper (Batuwita & Palade) the reason SVMs suffer from class imbalance is due to the "weakness of the soft margin optimization problem". This basically describes how the separating hyperplane (SH) becomes skewed towards the minority class. We want to maximize the margin (either side of the SH) and minimise misclassifications. The low number of minority samples makes them appear further from the SH resulting in the skew towards this class. With oversampling, the misclassification costs increase causing the sep hyperplane to shift back?
I think the question is **"why volatility is an important topic in financial econometrics?"**. I do not see how your answer relates to this.
I'm afraid I don't follow the premise of your question. Can you say more about your situation, your data, and your goals? In asking a statistical question, you may find it helpful to read this blog post: [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/), & our [FAQ](http://stats.stackexchange.com/faq).
The LR test and the H-L test assess completely different things, though!  The LR test merely assesses whether including all the variables does any good at all; the H-L test assesses whether the response (as transformed by the logit link) is approximately a linear function of the variables.
I think what you are looking for is a "two Stage Least Squares". In R, there is package called [twosls](http://cran.r-project.org/web/packages/Zelig/vignettes/twosls.pdf) for this. You can have a look at subsection 1.0.6 in above link.
Edited slightly for clairity - anti-conversative slipped in there because its currently being used in a paper open in another window. *sheepish*
@ EpiGrad: Iam using STATA 10.1; What kind of plot would be the most helpful or do you mean a smaller or simplified dataset? To the model: my cross sectional variable are the region, my time variable are months. So if I understand that right I am adding a fixed effect on top of the random effect.
What happens when any of events $y$, $z$, $k$, or $p$ doesn't occur? Do you win/lose money? How much? What is the objective criteria to determine "if it's worth it"? Expected value (strictly) positive?
Simulation is always an option - i.e. simulate data under a particular alternative hypothesis and sample size and re-fit the model many times to see how often you reject the null hypothesis of interest. From my experience this is quite (computer) time consuming as it takes at least a few seconds for each model fit.
Yes, if you do the "double intercept approach", you can also model distinguishable dyads.
@Yves Thanks for the suggestion.  I already tried scaling it but it made no difference.  It is block maxima, but as you say, not a good fit.  I'll look into the options you suggest.
Thanks for the response. Please have a look at new question: http://stats.stackexchange.com/questions/92732/within-subject-anova
I'd look into something called multiple kernel learning.
Adam, I feel that you misread the OP's question and so your answer may be irrelevant. No speach was about partial correlations. And "shared variance" was not thought of as covariance.
Maybe you don't need to mention GLM at all. If you did e.g. linear regression of the DV on several covariates, and based on that regression's output performed Wald tests of particular hypotheses, then just say that. (In fact there are so many ways to use GLMs that you *need* to say that much, to be precise about what was actually done.) Citing the piece of software you used to do the fitting would be courteous, but for use of GLM-fitting software is unlikely to be a big issue.
The answer to this is the same as the answer to your corresponding question on linear regression. Please review the answer given to that question.
why not follow an introductory textbook? Jim Albert's Bayesian Computation with R (Use R!) is one I can recommend (it is certainly full of simple examples).
Both the dependent and the independent variable are chronological age measured in years
Could you be more specific about the sense in which PCA is "equivalent" to estimating a correlation matrix? Suppose I stop my PCA after $k$ principal components. That requires estimating $k$ eigenvalues and $(p-1)+(p-2)+\\cdots+(p-k)$ independent eigenvector coefficients, all totaling less than $pk$ parameters, which could be quite a bit less than $p(p-1)/2$.
@PeterFlom: Perhaps you understood the problem from the start. I was confused, but feel like the issue is very clearly spelled out now. (See my answer and the comments to it.) And I feel like I'm not qualified to address the problem beyond my opinions. Any further thoughts?
There's a lot of good information here. I especially like bullet points #4 & 5.
I would use LASSO regularised Cox regression and then use bootstrapping to generate the confidence intervals.  I would have thought that the two-stage approach would give invalid confidence intervals as the same data has already been used to select the features prior to application of Cox regression.
Please pay close attention to dmk38's comments, below. *Who* is rating each speaker, and how much variability there is in each speakers' ratings make a difference. That 4.5-rated person has a higher mean than someone rated, say 4.3, but depending on whether their audiences had significantly different personal grading criteria and how much variation in grades there was around each mean, there may be no real-life difference between the two. The numbers will look authoritative, since you ran them through a computer, but the question is: are they saying what you think they're saying?
Dear Cagdas, Implicit in the specification of (vanilla) ARMA is the feature of conditional homoscedasticity. There *are* ARMA-GARCH combined models (I was not disputing that), but that is an entirely different model. If your question is: Should I use GARCH to estimate the conditional heteroscedasticity of a (vanilla) ARMA process, the answer is *no*. If your question regards (a) *what an appropriate model is* for some problem you are facing or (b) *here is my* (non-vanilla) *model, how do I fit it*, then we need more information about the actual problem you're considering. Cheers. :)
This test is done using the `poisson.test()` function in R.
@Memming : assuming I use the canonical link, what is your proof/source that it will be convex for those distributions?
(+1) It might be worth noting *why* this is the case, as it may not be clear to all readers. For example, it follows from Markov's inequality.
( p*exp(r1*x) + (1-p)*(exp(-r2*x)) ) / ( p*exp(r1*x) + (1-p) )  is the equation I'm trying to fit.  Basically, I am varying x in my experiment, and recording the response, y.  I am doing this under two different conditions in which I expect the parameter "p" to change, but not the parameters r1, r2; i.e. r1 r2 should be fit "globally" across the two datasets, whereas p should be fit individually, to each datset.
This should run out of the (tool)box at least the statistics toolbox. Have a look at [lasso](http://www.mathworks.de/de/help/stats/lasso.html?s_tid=doc_12b) at bottom of the page under the header "Elastic net".
@Aniko The event can happen during the year but is only recorded at the end of the year (I am not considering other variables for time being). The probability is surely time dependent, but I am interested in a "average" probability over the observation window. Actually one of the question I have is which observation window should I choose? I will use the probabilities for prediction, I need to balance accuracy versus stability...
You may want to consider accepting some of the prior questions you asked.
Do you only have (estimated) coefficients from one model including all of the independent variables? Can you exclude one or more of the IVs and re-calculate coefficients?
For the record: There was a [Letter to the Editor](http://informahealthcare.com/doi/full/10.3109/00207454.2011.648763) regarding this paper. It challenged some of the (post-hoc?) classifications of severity used in Table 3a (Experiment 1 and 2), warned of publication bias risks and the need to replicate. It didn't complain about the data in Table 5.
By searching our site for keywords like "maximum likelihood," "variance," and "error" you will find the standard theory explained and illustrated for many different cases. Other useful keywords are "Wald" and "score," as described at http://stats.stackexchange.com/questions/359.
At first sight, this just looks like a Python-related issue. Give us more information on your dataset and the statistical question you are trying to answer, and this will be on-topic on CV.
What is the chance that the rooks occupy all $8$ ranks?  By symmetry that's the chance they occupy all $8$ files, too. If you can show these two events are independent then all you need to do is multiply the chances :-).
The density is indeed the derivative of $\\Pr(V\\le v)$: in your last line you lost a negative sign and mistakenly got the correct result.  (Just draw a graph. $F(v)$ is increasing, whence $1-F(v)$ decreases and therefore so does $(1-F(v))^n$: thus, its derivative *must* be negative and cannot be a density.)  This is starting to sound like a question about using the Chain Rule rather than about probability or statistics...
Note that $\\log_{10}(X)+\\log_{10}(Y)=\\log_{10}(XY)$ and $a\\log_{10}(X)+b\\log_{10}(Y)=\\log_{10}(X^a Y^b)$.  Does this look meaningful in your exercise?
@Silverfish I would phrase it as extreme negative market outcomes are more likely than extreme positive market outcomes. Markets also have asymmetric volatility. Market volatility generally increases more following negative returns than positive returns. This is often modeled with Garch models, such as GJR-Garch (see Arch wikipedia entry).
Hi @user40398, welcome to Cross Validated. Your question seems to be a little bit hard to understand the way it is written now. Could you make it a little bit clearer? Perhaps give an example, or more details? You can make the changes by clicking the "edit" button below your question.
This is a property of the Normal distribution, to be sure, but it does not *characterize* it, because plenty of other distributions also have this property.
as I understand it, it works well if the covariance matrix is well-estimated. Can someone comment on this? thanks.
Also, I tried using fourier to take care of seasonality but returned the same error "Error in optim(init[mask], armafn, method = optim.method, hessian = TRUE, : non-finite finite-difference value [1]" even when used as a matrix on its own with no other exogenous variables. Please could someone help me with an explanation as to why that would be the case?
This is a [FAQ](http://stats.stackexchange.com/search?tab=votes&q=%22normality%20test%22%20p-value). A good overview of what you are up against appears at http://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless/2498#2498.
Connection signifies a pointer or in simple terms the row is "related" to the matrix.

r Matrices refers to the fact that each matrix will have only 'r' matrices they are connected to. If I can, imagine each row having a "pointer" (Computer Science terminology) to a matrix. This signifies that the matrix is related to a pointer.

The values of the matrix as told are just os and 1s.

I seek a relationships between the columns of these matrices bearing the connections and the distribution of values. 

If this is still vague will uploading a file of drawings help?
How many times was each subject tested/sampled? Without knowing the substantive aspects of your research, if Stata tells you that the variation within subjects is 0.000000718 (with a standard error of 0.378) and R tells you that it's 0.000, isn't the story here that there really isn't any variation at the subject level? Also note that Stata doesn't give you a confidence interval for the subject variation.
+1 Very nice answer to a very poor question.
I believe that if you consult your favorite definition of conditional distribution, this question will resolve itself easily. In particular I am thinking of the relationship $\\Pr(A|B)\\Pr(B)=\\Pr(A)$ where $A$ is the event $X\\le x$ and $B$ is the event $X\\le q_\\theta$.
It doesn't satisfy the triangle equality, so it is not a distance function. It also is not well-defined for the vector 0.
Be patient, with a little luck, admins might migrate the question (shortly).
Do you have spike-in data for absolute normalization between arrays? If not, your analysis will be almost impossible. There are numerous normalization methods you can use without spike-in data, but they are often predicated on either rank or normalizing by a gene that "doesn't change much" (constitutively active). The former fails spectacularly when there are global changes to expression, and the later fails spectacularly because the choice of a (or even a cluster) of constitutively active genes is arbitrary, unverifiable, and small changes in these references distort the entire array analysis
Nicely enough, I have downtime in both continuous and binary formats.
In R, you can try glmRob() in CRAN package robust to get robust quasiPoisson regression.
Categorization of BMI is invalid, and the use of percentiles for categorization defies the laws of physics.  Do you think that increasing BMI puts physical strain on the cardiovascular system or do you think that the number of other persons in the sample who have BMI lower than yours affects your cardiovascular system?  That is what is assumed when percentiling.
You might consider creating a new question asking more specifically about ways of dealing with convergence problems in MI (if that issue hasn't been addressed in another CV question).  That seems like a question that would be appropriate for CV.  I think you would get more responses than on this question.
You might want to investigate the use of orthogonal polynomials.
It appears that you are looking at Cohen's $d$ as a measure of effect size; at least, your definitions of "low", "medium" and "high" are the ones Cohen proposed for his $d$. As per [Alexis' answer](http://stats.stackexchange.com/a/113521/1352), you can simply flip signs for negative $d$, keeping the interpretation in mind. However, note that $d$ is not constrained to be below 1 - $d>1$ is perfectly possible!
@ Michael, unfortunately, the link you provided is not working!
Where did you see this term used? First search result for me is a restaurant in Spain...
Among other spatio-temporal techniques, the *Kriged Kalman filter* can
be a solution. The spatial smoothing captures the spatial low
frequencies thanks to the *Principal Kriging Functions* and seems
comparable to that of thin plate splines. RBF are used if the covariance
kernel is square exponential.

http://www.southampton.ac.uk/~sks/research/papers/sahumardia05.pdf
Sorry, I was unaware of that tag.
Your post isn't too clear.  Perhaps you could write out the equations you're estimating?  

Also, this is the first I've seen of "median splitting" -- but if you're arbitrarily dividing a continuous variable and creating a binary variable, you might imagine that you are muddying your data.  Perhaps drop those in the 40-60 percentile?  

Finally, it is possible that your coefficients are not significant because they are either collinear, or they arenit in fact good predictors.
Interestingly, most researchers in genetics studies are encouraged or pleased (it depends on the review) to make data available. I also remind of @csgillepsie nice answer about *reproducible research*, http://stats.stackexchange.com/questions/1980/complete-substantive-examples-of-reproducible-research-using-r/1993#1993
If $W$ & $Z$ are random noise, regressing $Y$ on $X$ gives a strong relationship. Whence the asymmetry that labels one regression spurious rather than the other without consideration of what the variables even mean? This kind of thing was debated between Pearson & Yule ([Aldrich (1995)](https://projecteuclid.org/euclid.ss/1177009870)) & I'm with Yule: what's spurious isn't the correlation but the claim of a causal relationship based on that correlation.
This may also be useful, if you want to specify classes by variable name (e.g., for assigning class to only one variable): http://stackoverflow.com/a/8185739/1169233
Jimmyb, I will try the glm! Thanks in the meantime! :-)
This is an incredibly answer whuber, thank you so much. It'll take me a while to fully process everything you've written here but I completely trust you calculations which means I've marked the question as resolved. Cheers.
(+1) I found a way to make this idea work.  It requires two techniques: first, use FFT for the convolutions; second, don't do them sequentially, but divide and conquer: do them in disjoint pairs, then do the results in disjoint pairs, etc.  The algorithm now scales as $O(n\\log n)$ rather than $O(n^2)$ for $n$ probabilities.  For instance, *Mathematica* can compute the *entire* distribution for 40,000 probabilities in just 0.4 seconds.  (1,000,000 are calculated in 10.5 seconds.)  I will provide code in a followup comment.
Do you mean that common-causal variable(s) or omitted variable(s) is NOT inevitable?
I estimate a Markov-Switching model on fiscal policy in the United States and thus allow for structural breaks in my series (I break my regression in two parts : the one pertaining to, say, regime 1 and then the one pertaining to regime 2.) I fit my series with the coefficients of regime 1 when the model identifies regime 1 and so forth. I end up with a value of Log-Likelihood which I would like to compare to a simple R² for the linear case (when I do not allow for switches) and see whether the difference is large or not. I doubt that comparing corr(y,y_hat)^2 for the two models is enough.
Hi @hbaghishani; I'll just quote Tukey "Far better an approximate answer to the right question, which is often vague, than an exact answer to the wrong question, which can always be made precise." 
:-). It's not completely apropos here, but it is at least partially on target
Welcome to the site. Why not expand this a little--it does seem like a potential answer.
Just to throw a couple of ideas on the subject, if the study discloses standard regression statistics you could focus on the t stats and p values of the coefficients.  If the RSquare of the model is high; but, one or more of the variables have a t stat < 2.0; this could be a red flag.  Also, if the sign of the coefficients on some of the variables defy logic that is probably another red flag.  If the study does not disclose a hold out period for the model, that may be another red flag.  Hopefully, you will have other and better ideas.