tttjjj/elt_bench huggingface dataset:
Calling a predictor a "control" doesn't change anything about the model, & note that the estimated coefficient for the predictor of interest will depend on how you model the control variables; so the LPM is as bad or good in this case as in one where you're interested in all predictors.
+1 in general, but I want to second the suggestion to use age or time-in-school over some coded version of the birth month too. Unless I'm missing something domain specific, I don't see any upside and a lot downside to using the birth month....
@NickCox Your right. I assumed CLRM was a universal abbreviation since it is stated as in few textbooks I picked up. But fixed it.
I edited my answer to respond to your first question. For the second question the answer in your case is "not much".
There are at least five questions on how to compare two groups or two ordinal variables: http://stats.stackexchange.com/q/35382/5739, http://stats.stackexchange.com/q/49318/5739, http://stats.stackexchange.com/q/203/5739, http://stats.stackexchange.com/q/59625/5739, http://stats.stackexchange.com/q/43113/5739, http://stats.stackexchange.com/q/56949/5739. So this is one of the most FAQed things on the site.
At which spatial locations $s$ did you compute the median of the posterior distribution? I believe the principal motivation behind running a model of this sort is to track the distribution of $w(s)$ at *all* points where you are interested in predicting its values, which--in the case of this contour map--would be all 4900 grid nodes.
Can you show how 'the probability of an element' relates to a Pareto? If C isn't even *ordered* then one might as easily put the categories in any order one chose. Certainly you can't even draw a histogram -- only a bar chart. If it's unordered, the categories can be put in any order, so the relative sizes couldn't suggest anything but multinomial.
I think there is a mistake in the last step above (taking the log of the likelihood), it should be:

$$ \\log \\mathcal{L} = \\sum_i \\frac{(f(x_i) - z_i)^2}{2\\sigma^2} - \\log Z $$

This doesn't change the "bottom line" because the log likelihood is linearly related to the RMSE, so minimizing RMSE is equivalent to minimizing log likelihood
If you refer to this: `Following Pedersen, we're going to describe the Gibbs sampler in a completely unsupervised setting where no labels at all are provided as training data.`, then the authors are stating they're going to describe Gibbs sampling as Pederson, probably for didactic reasons. But that doesn't mean that they're going to use it without labels. Actually, you can see after equation 49 they are sampling to obtain that conditional distribution and are distinguishing between label "1" and "0". Otherwise, you wouldn't know what parameters belong to each label.
While that didn't answer the question, the commentary was helpful, and I think pointed me in the rigt direction. You're right. a 0.01% difference is not material and we won't care - or even notice given the already present batch-to-batch variation, but 50% improvement or reduction in performance - thats big news.
Euclidean and cosine aren't that different. On L2 normalized data, they are effecitvely the same (which is why spherical k-means converges - k-means minimizes variance, but if variance = cosines, then you are fine). I would not expect very good results (in particular as k-means is forced partitioning and doesn't handle outliers well), but this seems to be state of the art for text clustering.
I don't think this method would work as there isn't really a pairing between my before and after data. My data is not paired. Am I missing something?
One possibility: Added variable plots
Isn't tumor size the outcome?
If half the coins are biased toward heads and half toward tails, what happens? What if most are biased up a moderate amount, but some are biased down a lot? More generally, if the average proportion is close to 1/2, even though none of the coins are unbiased, it looks to me like your test would have little power against such alternatives - but those are alternatives that appear to be relevant to the original question.
Thank you John & Tai

I guess that covers any potential avenue of my poorly formulated question.

regards

Alfredo
@JWM Thanks for the question. I'm particularly interested in descriptions of database handling procedures and computer programmes for the situation when a database of rare cases is available and it is easy and cheap to get appropriate controls. But an overview of matching algorithms would be nicer.
(+1) Good answer. One thing that I think could be drawn out more explicitly here is that this really has nothing to do with any relationship between the Poisson and the negative binomial, but rather more basic facts about fitting GLMs via maximum likelihood.
@Shadi Did you read the thread I pointed to? You cannot use correlation-based criteria. So, I would better advice you to rely on the ICC unless someone has a better idea to cope with bounded values. For me it's not a problem as you're likely to end-up with similar results than with any other most complicated method. Still I agree with others than it makes sense not to use ANOVA or mixed-effects models with inappropriate link function in certain cases.
See http://stats.stackexchange.com/questions/34613/l1-regression-estimates-median-whereas-l2-regression-estimates-mean for most of the answer.
@Arman It seems that you have unaccepted my answer. Did I make a mistake ?
I still don't quite follow your $n$ & $m$, & the way you are representing the matrices is unusual for me. I'm more familiar w/:$$Y=\\begin{pmatrix}y_1\\\\ \\vdots\\\\ y_N\\end{pmatrix},\\quad X=\\begin{pmatrix}1 &x_{11}&\\cdots&x_{1p}\\\\ \\vdots&\\vdots&\\ddots&\\vdots\\\\ 1 &x_{N1}&\\cdots&x_{Np}\\end{pmatrix},\\quad\\beta=\\begin{pmatrix}\\beta_0\\\\ \\vdots\\\\ \\beta_p\\end{pmatrix},\\quad\\varepsilon=\\begin{pmatrix}\\varepsilon_1\\\\ \\vdots\\\\ \\varepsilon_N\\end{pmatrix}$$
(c) is clearly wrong. (b) doesn't use all the information. So go for (a). But do all three and keep two on hand, just in case someone queries your method.
Do you mean "column" (i.e. variable) rather than "row"? And what does "individual group level" mean? Do you simply mean "state level?" If so, just write "state level."
(+1) Thanks for taking the time to work up all these examples and post them. I think you've covered all the bases. It's curious that group 3 has lower incidence than group 4. Maybe if you smoke a pipe/cigar you tend to be more recreational about your cigarette use than if you smoke only cigarettes.
@Curious Rhe question was about *pseudo*-$R^2$.  The components of the formula that was posted in the question are all in the output.
Wouldn't you just minimise the KL divergence - which is essentially moment matching on $E_{Ga}\\left[\\log(x)\\right]=\\mu$ and $E_{Ga}\\left[\\log(x)\\log(x)\\right]=\\sigma^2+\\mu^2$?
Agreed (+1) - IMO it was just worth qualifying a couple of the subsidiary points.
@onestop: Could you please explain a bit further? Especially the part about affecting the critical values of the test. A larger sample changes the degrees of freedom, since `df = sample size - 1`, but beyond that I’m afraid I don’t really understand what you mean. Thanks in advance!
I've already answered your last comment by pointing you to correlation and regression. FWIW, I am not a statistician, but a geographer. I learned what statistics I know by reading books and analysing data.  So your assumption is inaccurate. Also, please stop running your analogy of tomato plants; your thread is confusing enough already.
Could the two technicians have the same mean and variance yet be measuring differently?
(+1) So @David, in general is there a criterion that describes the quality of contrast?
The linear is fine. The other apparently is quadratic curve, which is practically unfeasible for such a long forecast period, unless there is a prior information on such behaviour or theoretical grounds for it.
@AndyW I understand that problem, but unfortunately, there does not seem to be any place on StacExchange to ask for that kind of thing, and I think that is a real need that should be "filled".
@Glen_b I edited the question, but yet I'm not exactly sure. I found that SAS has an COCHRAN option in TTEST procedure to produces p values for the unequal variance situation by using the Cochran and Cox (1950) approximation. although I don't use SAS.   these all seem confusing, because when I modify the denominator for the z-test to SQRT(SE1^2/n1 + SE2^2/n2) as in approximation t-testsI get very large z values and for the formula for t-test proposed on the attached similar question,not only I don't know how to modify the denominator,but also I even couldn't find it anywhere else.
$Y$ and $E(Y\\mid X)$ are not _unconditionally_ independent -but it doesn't matter. $E(Y\\mid X)$ is a function of $X$. When we take its expectation _conditioning again_ on $X$, it goes out of the expectations operator, as a matter of basic properties of expected values, irrespective of what other variables may be present. In general, $E(Yh(Z)\\mid Z) = h(Z)E(Y|Z)$, irrespective of the _unconditional_ relation between $Y$ and $Z$.
@BGreene - your third paragraph is a tough one for me.  If exploratory analysis shows predictor x1 to be highly correlated with the outcome, wouldn't it defeat the purpose of checking that correlation if one didn't use x1 as at least a candidate predictor in a multivariate model?
If you have access to Frank Harrell's book [Regression Modelling Strategies](http://www.amazon.com/Regression-Modeling-Strategies-Applications-Statistics/dp/1441929185/ref=sr_1_1?s=books&ie=UTF8&qid=1389446151&sr=1-1&keywords=harrell+regression) it has a nice explanation.
The model doesn't have to be linear, but in general I agree that diagnostics could be used to come up with new predictors.
I had considered using some sort of KL divergence measure but I'm not really trying to measure dissimilarity on a whole but instead gain numerical insight into what kind of changes are occurring.

I imagine I could do some sort of 'weighted' KL divergence where I place larger weight on the buckets that I'm more interested in (to see if there is a change in buckets that have larger index for example) but this would only tell me where the change is happening and not if it is positive or negative.
Thank you very much for your answer !
Cool, thanks, nw!
And for the Fisher's test shall I use the "two.sided" option?
Thanks for the link to a nice paper, it makes a very good point!
While you're correct in terms of the graphing, 'cmprsk' uses the subdistribution proportional hazards model, which is...not quite what I'm looking for.
Is the 90/10 requirement "hard"? In lower dimensions you might be able to get close by drawing from a Wishart centered at $I$, computing the correlation matrix, and rejecting samples that aren't within some tolerance. Though I suspect this won't scale well at all...
@Learner I've edited my post to reply to your comment.
The nature of the project might make a big difference in terms of motivating a professional (or perhaps even an organization) to volunteer to help at a charitable rate. KUDOS on realizing that simply plugging your data into a piece of software doesn't make you an expert, and that you need expert help!
Sorry for not answering your question; but I would like to mention that your idea / hypothesis is ideally suited for the [randomized response technique](http://en.wikipedia.org/wiki/Randomized_response), maybe you want to have a look at that (for future studies)?
Modelling economic time-series ain't as easy as you may think; have you read any serious graduate macro or econometric textbooks or papers from top journs? The claim that "economic applications are far more accessible, hence the reason they're included in general purpose time series textbooks" is very much contestable.
The doc on `homals` leaves impression that it is the same or similar to CATPCA that I mentioned in my comment. In SPSS, there is CATPCA (Nonlinear PCA), HOMALS (Multiple Correspondence) and OVERALS (Nonlinear Canonical) which are all optimal quantification techniques.
This may have been answered before.  See if this question provides the answers you need.  [Interpretation of R's lm() output][1]


  [1]: http://stats.stackexchange.com/questions/5135/interpretation-of-rs-lm-output
Absolutely!  As an instructor, what I'm looking for is not a relative ranking of the students -- but rather an absolute measure of comprehension and skills.  I would be perfectly happen (delighted, even), if it turned out that everyone in the class was measured as having 100% comprehension of the subject I'm teaching.  So, the weights on questions are chosen to reflect their importance and how much weight they should have in an overall assessment of understanding of the subject.
@K-1:> if you look at the code/formula of the Tau estimator (type "scale2tau2" in R), it's basically just a weighted mean (with weights set by the median/MAD  and cut of to 0 below a threshold). This should run fast on anything.
Thank you for all your responses. I see that my question was indeed not very clear. What I meant to ask was, Is the missing data completely at random (MCAR) and how do I do this in R. I t
Thanks. I ran my test data through `glmnet` and `elasticnet`, but can't make sense of the results to tell if it would work. `enet` with Lasso fit gives decent coefficients, perhaps, but they seem to treat the factors as numerically scaled (factor 5>>1) rather than as categories?
This is impressive--and especially so, since it's your first answer!  I hope you continue participating here and I look forward to your future contributions to our site.
Welcome to the site. Could you clarify your question a little? What exactly do the different authors recommend? (Formulas would help, references would help, quotes would help).
@Zag, no, there is no such a model in astronomy, the receiving of photons is considered poissonian.
You might be interested in some of [George Forman](http://www.hpl.hp.com/personal/George_Forman/Publications.html)'s work. I think it largely focuses on feature selection for text, but some of the ideas are more generally applicable.
Are there specific conditions under which you do the the adjustments? When are the adjustments necessary and when are they not?
Does the Bahadur-Savage impossibility result fit want you want? (It shows that, despite lots of prior work on non-parametric tests, it's impossible to guarantee having useful power unless one makes some assumptions, even with sane/nice sampling distributions)
Yes so simple :\\
+1 Very interesting question!
Thanks for the edits and the answer.  Very nicely done.
It's a shame that this work has not been published and publicized properly, and the reasons are totally opaque to me. The best resource that I am aware of is http://math.bu.edu/people/sray/talk/jsm_2007.pdf; this guy has moved on to other hotter stuff with bioinformatics computing though. James Berger mentioned the extended BIC in his Wald lectures at the 2007 JSM (http://amstat.org/meetings/jsm/2007/onlineprogram/index.cfm?fuseaction=abstract_details&abstractid=308388); again, I have not seen this stuff published by him.
+1 Because, it is a great idea.
Are the sections supposed to measure different constructs (e.g., linear algebra, real analysis, etc.) or do you consider the total responses correct on all 25 items? If sections correspond to different scales, do the four suspicious items belong to the same section or not? Finally, could you clarify what you mean by "language confound" and "reliability". I interpret the former as a possible effect of [differential item functioning](http://en.wikipedia.org/wiki/Differential_item_functioning), but for reliability I suspect you mix it up with variability (or standard deviation).
(+1) Hopefully the OP will notice that the binomial distribution enters in only in the most nominal way in the proof. Indeed, this is true for any family of distributions formed from sums of exchangeable nonnegative random variables.
Under random sampling, yes; this is simply a confidence interval for a binomial proportion, with the usual caveats about the assumptions of and interpretation of confidence intervals.
It all depends on your data, of course. One example which I encountered in real life: retailers run promotions (like multi-page flyers in people's mailboxes) to increase sales, and a promotion predictor may have levels like "ad on the first page of a flyer", "ad on the second page", "ad on the last page" and so on. However, it is unclear whether the flyer page actually makes a difference, so collapsing all levels into one level or into two levels "ad on first page" and "ad somewhere else" may be better in terms of parsimony and predictive power.
Related: http://stats.stackexchange.com/q/6907/2970
I don't think it is a valid question here; just check out wiki and references there: http://en.wikipedia.org/wiki/Probability_interpretations
I think the assertion that you can expect draws from a chain from an MCMC that has run for a long time to be approximately independent is rather strong (and often completely wrong). In some cases (RWMH for example), the sampler is deliberately not approximately independent (it could be made more independent but that would be counterproductive in that case). And in practice even Gibbs samplers may be quite far from independent even if you were sampling from the stationary distribution.
Ah, thank you, that was exactly what I was looking for.
That link has since moved to http://www.se16.info/hgb/statjoke.htm
It is my understanding that the only thing you care about is that your $\\hat{x}$, i.e. the predicted value of the first stage, is uncorrelated with the error term of the second stage. Your first stage coefficients might be biased or yield predictions outside the unit intervall etc, but this won't induce correlation between the predicted values of your endogenous variable and the error term of the second stage. I have never seen a proof of this though, but I have seen explanations along this line from e.g. Imbens.
@Memming, I have gone through the wikipedia article for the nth time, and that helped me realize my huge values are due to a lacking parenthesis in the code. However, I am still at loss as to why was the prior defined the way it was in the paper ( i.e. it is pretty much larger by 1 than it needs to be..). I will use the formula from wikipedia and hope for the best. Is there a chance you could answer one more question: after each iteration of EM-algorithm, do I need to update the alpha parameters? Or do I keep them fixed?
The Hough transform will work better and exhibit less bias when you measure slope as an angle rather than as a slope (which is the tangent of the angle).  The plot of an individual point in the Hough transform should be sinusoidal, [as illustrated here](http://stats.stackexchange.com/a/33102/919), rather than linear.
@gsk3 - interesting question.  I would have a guess that this "programaphobe" does not have to deal with the complicated spreadsheets?  perhaps he/she would like to live up to their ideology and volunteer to do the hard work with the spreadsheets themselves? :)
Same as it generally does in ordinary ANOVA. Under the usual coding of dummies, the coefficients represent comparisons with the baseline group (the one you leave out). If you change the baseline group, obviously the comparisons with a different group will be different. Since this surprises you, it suggests that you don't fully understand what the coefficients are. I'd supply this as an answer, except I'm pretty sure this is an already-answered question. I'll see if I can find a duplicate for you.
As far as I know, inter-rater reliability is unrelated to the number of raters you have. If you wanted to estimate the amount of inter-rater reliability that would exist in a population of raters, a larger sample should lead to narrower confidence intervals, but that's a different issue. If you just want your raters to be in closer agreement w/ each other, I might focus on polishing your materials &/or some training for your raters.
I think the OP is wondering how to understand A and B being marginally independent but dependent conditional on C, not how to understand A and B being marginally dependent but independent conditional on C.
@NickStauner That is exactly what I wanted to convey. Thanks for making this point even clearer.
Jordan, can't you just check empirically whether the two overlap? For example, putting the histograms side-by-side and checking visually whether there are any points which have non-zero mass under both distributions?
Are you sure Byrne doesn't tell you? Seems odd to mention that in a book and not explain it...
This does have statistical content which stops it being off-topic as a question on how to do something in Stata.
It wouldn't change things. $R^2$ is a measure of strength of relationship. The coefficient of the intercept term is a measure of whether the slope of the line would be different for men and women, regardless of whether the coefficient for male is significant, or whether the coefficient for height is significant.
This is expressible in terms of a [Jacobi Theta function](http://mathworld.wolfram.com/JacobiThetaFunctions.html), $\\frac{1}{x} e^{\\frac{3 \\pi ^2}{8 x^2}} \\left(e^{-\\frac{\\pi ^2}{2 x^2}}\\right)^{3/4} \\sqrt{\\frac{\\pi }{2}} \\vartheta_2\\left(0,e^{-\\frac{\\pi ^2}{2 x^2}}\\right)$.  But what is your question?  It seems to have been cut off after "$n$ i...".
I like this one, could be put as an advise when people  write questions on this site ?
OK, sounds like a good idea. I've done far worse things in the past to lower my reputation...
Then Michael is correct (+1).  The fix is to go back and transform *all* the A values in the same way, all the B values, etc., and re-do the analysis.
The derivation is not difficult — See my answer: http://mathoverflow.net/questions/63496/what-can-be-said-about-an-infinite-linear-chain-of-conjugate-prior-distributions/65203#65203
Here's a hint. There's 81 possible two letter pairs since there's 9 ways to pick the first and 9 to pick the second. That's your denominator. Can you figure the numerator out?
I'd interpret it to be a distribution on 4 variables - the next in the series *univariate, bivariate, trivariate, ...*. However, it's more often spelled *quadrivariate*, which might explain why you had trouble finding it. Googling *quadrivariate* turns up many hits. But for that matter, when I google *quadravariate* the second and fourth hits (that I get, your experience may be different) are consistent with this interpretation, so it's not hard to find. What's the name of the Priestly reference?
*Nonparametric* tests also require assumptions about the probability model: the distinction between parametric and non-parametric does not lie there!
There are time-series uses for the canonical correlation, although the best one requires that you use a multiple-taper method of spectrum estimation. If all of your correlations are coming out as 1, that could be not that something is wrong, but that your series are highly correlated. This can happen easily if they are all seasonal and in-phase and you haven't prewhitened them.
Letting $\\mathbb{A} = (a_{ij}),$ this sum gives the components of $\\mathbb{A}^2$, written in `R` as `a %*% a`.
Try Reinsel's book: http://www.amazon.com/gp/product/0387406190/ref=as_li_ss_tl?ie=UTF8&camp=1789&creative=390957&creativeASIN=0387406190&linkCode=as2&tag=prorobjhyn-20
Further question- Would proc genmod or glimmix work?
Despite using $\\beta_0$, you don't have any intercepts listed. You have *slope* coefficients in these models. In addition, you have the same slope coefficient for both variables in the first model. Are you sure these represent what you are really after? For reference, consider the simplest model encompassing both of these:$$\\text{Sales}=\\beta_{0}+\\beta_1\\text{Intercept}+\\beta_2\\text{Store}+\\beta_3(\\text{Intercept*Store}) +\\beta_{1}(\\text{Intercept*Product})$$
@julienstirnemann - Your point might *partly* apply to some very short, crude, and clumsily-constructed tests, but I agree with the other comments:  this test score sounds as if it can quite safely be treated as an interval-level variable.