tttjjj/elt_bench huggingface dataset:
Thanks! I added some info to my original question
@Downvoter: It is good practice to give a reason for the downvote and, even better, give an idea how to improve the answer - Thank you.
The Stata command for a Wald test is `test`. The last hypothesis $H_0 : A = 1/B$ is a non-linear Wald test, the Stata command for that is `testnl`.
Is it implied that the time series is ergodic and not directly forecastable? How about Bayesian approach of credible interval?
@mpiktas - well, it was just the result of a google search on max-margin clustering, the "original" paper on it seems to be http://webdocs.cs.ualberta.ca/~dale/papers/nips04.ps.gz (Xu et al, NIPS 2004)
Maybe you can give more specifics to the problem, as what is the purpose of the model?
If you're trying to predict future values, if you're trying to estimate k, or if you're trying to validate/reject the model, there would be different answers depending on what you want to do.
Thank you very much.. I forgot about the basic assumptions of ANOVA. And thanks also for your last link!!!
So essentially you want to fit a piecewise linear function to the data? I.e., have intervals $(x_0,x_1), (x_1,x_2), \\ldots, (x_{N-1},x_N)$ such that it is linear in each interval but the slope changes at each boundary?
I don't know that there is necessarily a best way. Sometimes you have to try a couple of different starting points, if it fails to converge or if you aren't sure you're in the global minimum. I think a common way programs pick a starting point is to use the OLS estimates, even though they aren't appropriate & you know you'll have to move from there.
This is immediate from the definition of the p-value as the [probability integral transform](http://en.wikipedia.org/wiki/Probability_integral_transform) of the test statistic using the distribution under the null hypothesis.  The conclusion requires that the distribution be continuous.  When the distribution is discrete (or has atoms), the distribution of p-values is discrete, too, and therefore can only approximately be uniform.
Just a thought.. but, if zipcodes are distributed geographically then you could geographically represent zipcodes in a map and represent them with their location. With that you could also see which zipcodes are closeser..
**Hint**: Can you think of an example where this is the case, say, in the context of a family of normal distributions? How would you prove it?
You just *look* at it! When an absolutely continuous PDF $f(x,y)$ factors as $f_x(x)f_y(y)$, then (a) $X$ and $Y$ are independent and (b) [$f_x$ and $f_y$ are the conditional distributions](http://en.wikipedia.org/wiki/Conditional_probability_distribution#Continuous_distributions).
@gerrit, if the job had a component where the applicant had to interpret customer specifications for example, then pointing out that flaw might be the right answer that the interviewer probably didn't hope to receive.
Good luck with the rest of you analysis.
I disagree with your last point.  People who say my data are normally distributed are usually not referring to the residuals. I think people say that because they think every statistical procedure requires all data to be normal.
(+1) If one country is a great predictor, then the whole group of countries will be significant and it will be retained.  @Matt Parker is correct: keep groups of dummy variables together and base the entering and leaving criteria on the joint significance level for the entire group, not on the significance levels for the individual dummies.
+1 Excellent observations.
@ATJ I hope that this clarifies my point. There are other arguments that one could advance, but the real sticking point for me was the implicit assumption that Bayesian stats is subjective *in ways that other paradigms are not.* For example, I would argue against the characterization in Bernd's quote because it seems just as "personal" to favor a method of unbiased point estimation over a framework of posterior variability.
A rough estimate of the minimum sample size needed to do what you are doing is that you need the minimum of the number of events or the number of non-events to exceed 15 times (12 + 5), assuming you had 17 **candidate** terms in the model.  If you did any screening of predictors, making use of $Y$, all bets are off.
Of possible interest: [How to transform ordinal data from questionnaire into proper interval data?](http://stats.stackexchange.com/q/27927/930) or [Measuring the Mind](http://en.wikipedia.org/wiki/Measuring_the_Mind), by Denny Borsboom.
@chl Thank you for noting! Added outliers and correction of interval. Code maybe is not so beautiful, but works well.
For more closely related threads please [search our site](http://stats.stackexchange.com/search?q=probability+book+wiki%3Ayes+is%3Aquestion).
@onestop: thanks, I fixed my answer to note the mean/censored issue.
@whuber Thanks! -- although a large part of the credit goes to the OP for suggesting the elegant cutpoint approach. Also I agree your approach should remain -- arguably it is the only one that belongs here not on stackoverflow. Two comments: (1) interesting that your approach isn't much slower than mine; (2) your approach of plotting your results was extremely helpful, as it illustrated that a [second approach I tried wasn't the optimization that it appeared to be](http://stats.stackexchange.com/questions/67521/why-do-two-equivalent-approaches-to-selecting-numbers-that-sum-to-100-produce-di).
Those are all interesting topics.  The way I interpret either the Dickey-Fuller or the Augmented Dickey-Fuller is that their t statistic (called tau statistic) should be very statistically significant with a negative sign.  This indicates that the dependent variable is indeed mean reverting.  Regarding whether R Square is useful or dangerous... well it can be very useful.  It can also be very dangerous.  I remember reading a book on econometrics recently, and the author/professor stated if he sees any econometrics model with an R Square of 0.95 or above... it is very likely mispecified.
You don't *have* to do anything. If you are lost and in a hurry and since the design is not very complicated and all effects are significant, just interpret the pattern of means directly.
Good concise statement. Much more is easily accessible on this site e.g. http://stats.stackexchange.com/questions/16381/what-is-a-complete-list-of-the-usual-assumptions-for-linear-regression/16460#16460
DV is brand attitude.  IVs will be organic-orientation (two levels: high low) and size (two levels: large small).  So i have one brand that is organic and large, one that is organic and small, one that is non-organic and large and one that non-organic and small.  I provide participants with some health information.  I want to assess changes is brand attitude for each brand, but need to control for existing brand attitude differences.  So i want to add in Time 1 attitudes as covariates, but it seems odd just to put them all together.
Some things are very hard to predict.  This is one of the events that Taleb called "Black Swans" that made his first book that claimed statisticians can't predict financial events a New York Times best seller.  Ability to predict is based on adequacy of the model and available information.  At the time of 9/11/2001 there was intelligence information that al Qaeda was planning a major attack on the US in that time frame.  The same terrorists had bombed the World Trade Center previously. Getting the intelligence information to the analysts and then the decisionmakers is what matters.
@DWin: I believe truncation is distinct from censoring (e.g., in tobit regression) and dichotomization (e.g., in probit regression), though all are forms of limiting a variable's information based on a cutpoint.  Work on limited variables, missing data, or coarsened data might provide insight into my question.
@mbq the edit is a definite improvement, but I don't see the need to change '*linearization*' to '*linearisation*'; the first is perfectly correct as it stands, so the other spelling (while also legitimate) is not better.
Can you edit and add more context? It's hard for me to tell from your current description what you are doing and what your concerns are.
I am voting to close. Please see here: http://meta.stats.stackexchange.com/questions/1032/data-sourcing-we-need-to-make-up-our-mind
If it is worse than linear, it is an optimization issue. Try to play with parameters such as the learning rate. (if you are using stochastic gradient descent... if you are using something more sophisticated, try using SGD).
@whuber - $s$ is the standard deviation MLE, $s^2=N^{-1}\\sum_{i=1}^{N}(Y_i-\\overline{Y})^2$.  I thought it was the usual notation for standard deviation, just as $\\overline{Y}$ is usual for average - which I have incorrectly written as $\\overline{x}$, will edit accordingly
regarding the comment from @cbeleites, I think there is a theoretical answer here - an expansion of your suggestion that linear combinations are preserved. However, in practical application terms, it is very hard to draw a general conclusion on the validity of an approach, and there would need to be a specific example.
@BabakP I have to use table in exam hall. In practical exam i may be allow to use software.
Is the value of the paired difference exactly the same for every pair? That is what **"the standard error of the difference is 0"** seems to mean.
Yes Glen, sorry about that, it is indeed the $i$th iterate of the EM algorithm.
What exactly does it mean that you have a "profile" for instrument 1? Is that a set of weights that lets you combine the responses to the 20 questions into a single score? Do you want to show that the 2nd instrument measures the same latent factor as the 1st?
@Randel I agree that this question is not explicitly about statistics and would get several good answers quickly on the math site. However, it does point out a statistical connection and there certainly exist solutions that are motivated by statistical concepts and a few that even use statistical ideas. Thus I am not voting to migrate this one, in the expectation that our community will provide answers with more statistical interest than one might hope to see on a purely mathematical forum.
Isn't the Bayesian network $A \\rightarrow B \\rightarrow C$ supposed to be _locally Markov_ in the sense that given $B$, the sole parent of $C$, $C$ is conditionally independent of all _nondescendants_ (such as $A$), that is, $P(A\\mid B,C)$ should equal $P(A\\mid B)$?  While Henry's equation is correct in the general case, the probability assignment in his counterexample is not compatible with the Bayesian network $A \\rightarrow B \\rightarrow C$.  For the probability assignments required by the locally Markov property in this case, @ArKitect's solution is correct.
"I can re-parametrize constants on either pdf to make them look the same" — did you try it?  You can't.  The binomial distribution doesn't even have a pdf; it has a pmf.
Please don't assume people are aware of abbreviations from other disciplines. You say "the cocomo81 dataset" like we should know what that is. Don't make that assumption. Is it something to do with [this](http://en.wikipedia.org/wiki/COCOMO)? If not, can you give some explicit idea as to what this dataset is?
The quotation says correlations and variances. I've edited the question very slightly to be consistent with that. You can revert if you think I really mangled your question.
That should work. Unless i get into data imputation techniques.
+1, in addition, I might note that "cut[ing] off insignificant features" is unlikely to be a good idea.
yes it doesn't crash, but depending on the formula (e.g. in a saturated model), some of the parameters can have effectively infinite standard errors. My question is whether this is a problem when doing a likelihood ratio test. You can still calculate a likelihood even if some parameters aren't estimated, those parameters just won't contribute to the likelihood. What's the standard practice and why?
The title of the question does not quite correspond to the body.
I am beginning to believe I should be multiplying the likelihood intervals by some quantile of either the normal or chisquare distribution to get the corresponding confidence interval..
Hi Nick. Thanks for the link. This is not a homework question, but rather a topic I'm very interested in. The data above are real-life numbers for the New Jersey Devils this season and I'm just hoping to understand it better. I did look at using confidence intervals and found ranges for 95% and 99%, but I wasn't very clear what the results were telling me.
@whuber - I'll add that I didn't "propose" the skewness/kurtosis tests; I was reacting to what user9065 proposed.  Maybe my response was not so sound on that point, but I see my larger point on the subjective factor in normality testing is being echoed by the consensus argument forming at http://stats.stackexchange.com/questions/2492/normality-testing-essentially-useless.
I visited the link and read your praise of the decision tree.  Have you considered the way this method treats all main effects (save the first) as if they were interaction effects?  I have found that highly misleading and counterproductive on many occasions.
In more academic contexts one may also ask: 'have a look at this model output in this paper *that you (co-)authored*.  Tell me what it means.'  Underwhelming answers are then fatal because there are no unfamiliarity excuses available, yet dismayingly common.
One more thing, consider comparing sensitivity and specifity. For the ensemble source, check the book attached at my post http://stats.stackexchange.com/questions/47075/defintion-for-model-diversity/47129#47129
(ctd) ... the next most likely thing is that you don't understand how hypothesis tests work (that you reject the null for small p-values, not large ones); the third most likely thing is that you have a fairly strange distribution (such as A and B having severe skew in the opposite direction to R). $\\quad\\quad\\quad$ ... perhaps you should clarify what you're actually trying to achieve in plain words.
@whuber, peaks of the two populations are nearly contemporaneous. One is in temperate latitudes, and one is in tropical latitudes. Our hypothesis is that the tropical population has a narrower ecological niche than the temperate population (i.e., a wider host of predators and pathogens pressures the population into a narrow emergence time). Does that help?
:Bruder I should have written "If you assume (initially) that the ARIMA component, remember the ARIMA component encompasses differencing and ARMA structure DOESN'T DOMINATE.
I need to measure the correlation of two vectors, not two daat points in vectors. "as.dist(1-cor(t(X)), diag=T, upper=T)" do the measure in this way ? Thanks !
@mzuba: Indeed, I realize "I do not have enough data"... but I am looking for an argument that is more rigourous...if any... thx
This is a resourceful approach and yet I wonder whether its benefits are outweighed by the problems it might pose for interpretability for some audiences.
The probability you are looking at is a run-length probability, but what you want is a run-count probability.  The former takes a particular run and calculates the probability that its length is $r$, but the latter takes a sequence of $n$ elements and calculates the probability of seeing $r$ runs.
@ArekPaterek Your short answer looked like a joke and so those comments which don't apply to your revised answer does not apply to the revised one.  So my guess is that a moderator probably removed them.  Still calling fully Bayesian approximate is puzzling.
Very late to the party, but this answer will not solve any problems due to a hierarchical nature of the data set.
Thanks for this clarification. I guess one problem for this misunderstandings comes from the R output of the sort where it reads "(Dispersion parameter for Negative Binomial(theta) family taken to be 1)". Here dispersion refers to the $\\phi$ in $cov(\\hat{\\beta})=\\phi(X^T\\hat{Ŵ}X)^{-1}$, which you seem to call scale. So many people opt for calling the $\\theta$ differently. Also derived from the R function, I tended to call $\\mu$ location and $\\theta$ "shape"-the latter of which I don't find unreasonable as it certainly influences the shape.
I've taken the liberty of adding the `python` tag. Where possible one should indicate the language you give code in, both for those people that don't instantly recognize it, and also to facilitate searches.
In general there's no closed-form expression for the coefficients in the logistic regression model, but [this paper](http://czep.net/stat/mlelr.pdf) provides an equation for the variance-covariance matrix associated with a particular beta value in equation (16). This could get you started on deriving the updated variance-covariance matrix for a particular regularization term.
Thanks. Actually I already do that
(https://github.com/barrycarter/bcapps/blob/master/bc-get-cycle-files.pl),
but that data only goes back 24 or so hours and isn't archived as far 
as I can tell. I am looking for international data. As an example, 
http://ns1.data.barrycarter.info/HOURLY/ is what I have so far. I took 
a look at the berkeleyearth.org site. The dataset sounds exciting, 
though I wonder if "4 daily and 6 monthly" means they have no hourly 
data?
@gung No I don't. Topic is probably the wrong word here. For example, an article about the Miami Heat and an article about the LA Lakers would be both about Basketball, but they wouldn't be a match. However, two different articles about the Miami Heat game the previous night would be a Match. Thanks for the response!
I think the wording on the title might be misleading. My question is not how can residuals be correlated with the fits, but, rather, how can either residuals OR fits be correlated with itself (in the sense of potentially having non-zero covariance terms).
The L2 distance between two Gaussian mixtures is available in closed form. Use this and you should be all set.
I'd say that you need to understand linear algebra very well at the least and probably also some probability for the Bayesian applications. You also need to be comfortable translating math to code, and also be adaptable relative to notation convention (which is not at all consistent). ML is not easy for most computer scientists either.
Do estimation with transformed data.
No, Mann-Whitney are not the appropriate _post hoc_ tests following rejection of a Kruskal-Wallis test. Dunn's test is appropriate, see: http://stats.stackexchange.com/questions/25815/difference-between-post-hoc-test-after-kruskal-wallis-and-bonferroni-corrected-m/95270#95270.
thank you sir. I dont know how to attach linked research paper here so its my request  to download that paper once. In this paper author provided a data in appendix and the five estimated parameters. The four estimated parameters are provided in figure 5. My main problem comes here. I dont know to calculate these parameter. For e.g. the maximum number of flowers is on 232. But he said it on 230 by maximum likelihood estimation. Also i dont understand the meaning of spread of flowering which is 15. Could you please help me to calulate these parameter with the help to appendix data
I've just gone ahead and done this, and can confirm that the differences are as I would expect relative to the ranges over which they are sampled. I've also done some marginalised Kolmogorov-Smirnov tests across individual dimensions, and they also pass and fail in roughly the manner I'd expect them to. This, again, is not a sufficient condition, but it is a necessary condition.
I ended up turning to Paul Allison's book, as I always seem to, and implementing a conditional logistic regression model for this particular problem.
I actually do, so I inserted the relevant portions into my other answer.
Forced by coauthor's quirky use of STATA??  STATA is good software and especially good for Poisson and negative binomial regression.  I beleieve Joe Hilbe wrote some of the code for it.
Sure Peter will explore the pacakge.
@gung well, thank you, I guess I was a bit wordy. It's better now.
@Glen_b That might be true in general but I am not able to see any reason for concave function to not converge to optima within small tolerable value. But I would say that I dont have any practical experience with these and I have just started. :)
Welcome to our site.  Routine questions like this are usually tagged with [tag:self-study] and are treated a little differently from other questions (consult our [help] for details). In particular, we ask that you provide information about your efforts to solve your problem so that we can focus on providing advice to get you unstuck. Please edit your question accordingly.
no. feature selection=model buiding, and should only be done with training set. otherwise estimates provided by training set will be misleading.
Are these questionnaires considered validated in your study population? Are there standard ways to score each (e.g., arithmetic sum of responses, weighted sum, different subscales)?
@Gottfried As I understand it, your research question pertains to theories relevant to social network analysis. However, your data collection procedure is fairly standard. I.e., you can assume independence of observations. This contrasts with many social network studies that sample entire networks to get indices of things like homophily, and have to deal with the dependence in the data. Thus, the analytic approach required for your data does not seem that complicated, relative to the analyses of whole networks using approaches such as Exponential Random Graph Models.
I just got it to work as I added ,`data=ABMtest.df` after `(1|brood)`. Could adding the Feeding as a factor (not as.factor) have cause harm though?
However I am still having an issue:
`> fm2test<-lmer(Feeding~MF.vs.OF+Age.class+tide.h.l+Site+HDp+(1|Brood), data=ABMtest.df)`

`> ms2test<-dredge(fm2test, trace=TRUE, rank="AICc", REML=FALSE)`

`1 : lmer(formula = Feeding ~ (1 | Brood), data = ABMtest.df)
Error in UseMethod("fixef") : 
  no applicable method for 'fixef' applied to an object of class "mer"`
All of your formulas are very, very strange. How, for example, `((xi – u)T(xi – u))` can ever lead to a covariance matrix?
Did you intend to include some kind of more complete reference or link on [2]? Is this Matlab code?
@shadi Correlation is a bivariate relationship in the sense that it tells you the relationship between two sets of variables. Thus, calculating pearson(a,b,c) where I presume a,b,c are different judges is not possible.
@David The data is count data and the explanatory variables chosen are independant and are uncorrelated after checking for collinearity. What I really want to know is can you do a GLM to test differences in Species richness, dominance and diversity? Thanks! Would you use poisson distribution or gaussian?
Thanks for your input, I fully agree with you that writing a good survey is harder than initiall estimated.
The [Wikipedia article on M-estimators](http://en.wikipedia.org/wiki/M-estimator) explicitly allows $\\theta$ to be a vector parameter.
+0, this would serve better as a comment to @Shane's answer.  You might edit your answer to recommend a book that would better sever the stated purpose for those without "substantial prior coursework in statistics".
Actually, @Dilip, you need a Markov chain with states indexed by ordered pairs $(i,i+1)$ and $(i,i-1)$, $i\\in\\mathbb{Z}$.  The transitions are $(i,i+1)\\to(i+1,i+1)$ and $(i,i-1)\\to(i-1,i)$ with probability $p$ and $(i,i+1)\\to(i+1,i)$ and $(i,i-1)\\to(i-1,i-2)$ with probability $1-p$.
Closing at OP's request.  (BTW, in general you can just delete your own question.)
Why is the fact that they are dependent variables relevant? Could you not just run a t-test?
Thanks, not trying to argue ^_^ Take it easy.
How are you planning to integrate friendship links into matrix factorization? Also, it'd be useful to know exactly what kind of data you have available for your task.