tttjjj/elt_bench huggingface dataset:
You may want to take a look at these related questions: [Logistic Regression: Which pseudo R-squared measure is the one to report (Cox & Snell or Nagelkerke)?](http://stats.stackexchange.com/questions/3559/logistic-regression-which-pseudo-r-squared-measure-is-the-one-to-report-cox-s), [Find out pseudo R square value for a Logistic Regression analysis](http://bit.ly/kuSQtK), [How to calculate pseudo-R2 from R's logistic regression?](http://bit.ly/moPLGp).
The correction for multiple comparisons is about your three correlation tests (where $H_0:\\, \\rho=0$ probably) *and* comparisons for your pairwise correlation values? Also, it would be helpful to specify what your measurements are. Indeed, one may propose an alternative way to answer your question provided you add some details about what you ultimately want to study.
Edit - Options - Multiple Imputation...
+1 This is a nice complement to the answers already listed.  We can also note that if you already have a variable with group names (such as A-D), this can be done in the analysis function call without an extra step:  `lm(y ~ as.factor(x))`
A valuable resource for learning about generalized linear mixed models (GLMMs) and how to fit and interpret them is maintained by Ben Bolker and others at http://glmm.wikidot.com/
Yes, in this case you can simply query the language model (LM toolkits generally provide specific methods for efficiently querying the model) to obtain the probabilities required to construct your document vectors.
Is there a discrepancy in your code between AGE and sex? It looks like you are showing tapply() function by AGE, but are referring to it by gender.
Hi Max, Thanks. I came across the timeSlice method (took a bit of searching initially) and was interested in any specific examples of how it is used. I'll go through the help details, but it'd be great if you have any additional information on the same. Thanks, Raj.
(i) I'm trying to estimate their sum.
(ii) The quantity that I'm trying to estimate is a spending on a  commodity.
I assume that your $H_0$ is $\\mu = 0$, correct?
Welcome to the site. You state that you have demonstrated certain thing, can you give more details, or a reference?
If you can't figure out how to model things properly, & the model's to be used for something important; consult a statistician.
Not clear what model you seek here as you don't make it clear what predictors you may or may not be using. But in general if there's a spike in a distribution, there is a corresponding spike in the distribution of any transformation of it; that's inescapable. There are models for zero inflation of various response distributions; here you might want zero inflated beta.
No justification is needed.
The terminology is confusing.  The binomial coeefficients are the terms in a binomial expansion of (a+b)$^n$ and if a and b are positive with n a positive integer all the binomial coefficients are positive!  I assume you mean the regression coefficients in MLM.  In that case I would concur with Sven.
The t is the posterior, not the prior. The prior for the variance is - quite explicitly in what you quoted - inverse gamma.
As long as its the same scale. The reasoning is simple, since dividing a variable with scale $\\theta$ by $\\theta$ gives scale $1$. In the ratios, the $\\theta$s in the numerator and denominator will cancel, making it the same for any other particular value $\\theta$ might be.
Okay, thanks for the useful answer.  Perhaps I should have rephrased my question, but you answered it perfectly :)
The penalty can be interpreted as a prior favoring models of a particular size, though.  If you happen to adopt that prior (which has some information-theoretic justifications), then you can calculate a posterior odds ratio directly from IC values. Also, @RioRaider mentions Akaike weights, which give you the probability that a given model is the best model from the set in terms of K-L divergence. ([ref](http://www.nematodes.org/teaching/tutorials/phylogenetics/Bayesian_Workshop/PDFs/Posada%20and%20Buckley%20Syst%20Biol%202004.pdf)--see p. 800).
@BST Avoid using the term 'intercept' for both the $b$ and $e$ parameters; it's very confusing. Oh and you can't type returns in comments.
Close. What I wanted was perhaps simpler, and that is the straight fixed effects curve. Not so much individual predictions. However this is useful and satisfies Q2. Perhaps a manual command to "curve" will do the trick?
Look in [this](http://stats.stackexchange.com/questions/44279/example-where-a-simple-correlation-coefficient-has-a-sign-opposite-to-that-of-th/44290#44290). Simple regression coef. corresponds to simple correlation coef. Multiple regression coef. - to partial correlation coef.
You'll also want to read this: [interpretation-of-rs-lm-output](http://stats.stackexchange.com/questions/5135/). After having read those, see if you still have any questions left, & if you do, edit your Q to clarify what you still need to know.
@MichalJ.Figurski No it is not. You can make a ROC curve out of a logistic regression model, but that's not a requirement. You can do a ROC curve or of any other model, or you can simply make a ROC curve of your data.
This answer corresponds to everything that has already been said in this post. In particular, see the wikipedia article on [selection bias](http://en.wikipedia.org/wiki/Selection_bias) that was referenced above.
Daniel, have a look, ive done it using this syntax. Thx god!
Apologies, I have no idea how to use the code syntax for this site.
   
    IF  (cape1 = 0) cape_1 = 0.
    IF  (cape1  >=  1 and cape1b = 0) cape_1=1.
    IF  (cape1  >=  1 and cape1b >=  1 ) cape_1=2.
    EXECUTE.
exam revision looking at past papers.
I don't know what `dnorm` is (nomenclature from R?) but assuming it means the _bivariate_ normal density with mean vector $\\mu$ and covariance matrix $\\Sigma$, then the right side of your last displayed expression is an _upper bound_ on $P\\{X \\in S\\}$ since $S$ is a _subset_ of the rectangle $[\\min(S_x), \\max(S_x)]\\times[\\min(S_y),\\max(S_y)]$. To get the exact probability you need to integrate the bivariate density over the exact region $S$. In general, there are no simple ways of expressing such probabilities in terms of elementary functions.
As Hadley suggested, the problem was in the compiled code, in this case the pan() function. I have added tighter error checking, so the issue is solved in mice 2.16.
@BrianFeeny glad to help. If the answer was satisfactory, please consider to click the check mark below the vote up / down button. See [the FAQ on asking questions](http://stats.stackexchange.com/faq#howtoask) for details :)
@Glen_b Correct me if i am wrong, if $\\tau$ = 0, then the pdf would simply be dependent on the random variable $\\lambda$ which gives the pdf: $\\int \\dfrac{1}{e^{\\lambda}}\\ast$$e^{\\dfrac{1}{e^{\\lambda}} t}$ dt. With a non zero $\\tau$, it would be $\\int \\dfrac{1}{e^{\\lambda}+\\tau}\\ast$$e^{(\\lambda+$\\tau$)*t}$ dt
@Srikant You're right.  The concern lies with intermediate situations.  In those cases the correlation is probably not great enough to warrant special measures to deal with correlated regressors.  Moreover, although we speak of "correlation" the relationship likely is not linear, so a lot of care is needed in dealing with it.
Regardless of whether this did what you wanted in one study, it is not the only solution here. For example, if the original measures were weakly correlated, you might end with a measure difficult to interpret.
@jcrudy, [Point Process Theory and Applications](http://www.springer.com/birkhauser/applied+probability+and+statistics/book/978-0-8176-4215-0) by Martin Jacobsen is a probabilistic and general treatment. [Statistical Models Based on Counting Processes](http://www.springer.com/statistics/statistical+theory+and+methods/book/978-0-387-94519-4) is a classic. It is mainly about survival analysis, but has a general treatment. [Stationary Marked Point Processes: An Intuitive Approach](http://www.amazon.com/Stationary-Marked-Point-Processes-Stochastic/dp/0412984318) by Sigman should also be good.
Dear all thank you for your posts - I am confused because the quintile data is, I think, PPP averaged over the quintile.  So the data points aren't really points, exactly.  Are they?  fitting to an average seems like it will give me a totally wrong fit.  Particularly for the 10th 10% of a country, which has to have a long, long tail.  I can show you the data I want to model but I don't know how to add graphs like were added below.
Very nice short video
I think "scoring" is SAS terminology. But it is the way to go. You'd first need to build a logistic model on dataset 1 and then score the dataset 2 using that model: http://support.sas.com/documentation/cdl/en/statug/63347/HTML/default/viewer.htm#statug_logistic_sect050.htm
That makes sense, I'll try it again with a different amount of cases/variables. Thanks for your help!
+1 to your answer for the reference to the Ching, Ng, and Fung paper.  That's a good one to have.  However, after reading through it, it appears that it only covers discrete variables (which is kind of what I expected.)  While I can discretize my continuous variables, I'm still curious -- Are there any models that can handle the raw continuous variables?
I agree. Same thing with movies. But seems like somehow IMDB took care of the bias due to subjectivity, I wonder how :/
As far as the 'causality' part goes, this sounds like testing for [mediation](http://en.wikipedia.org/wiki/Mediation_%28statistics%29), but as the lead of that Wikipedia article says, the possibility of / methods for testing for mediation using empirical data have been challenged. I haven't made a thorough study of this, but i'd number myself among the skeptics.
If you have the original data, you can simple compute the difference between ON and OFF in each sell, and then compute the standard deviation of these difference scores.
@Entropy Because the ambiguity in your question is leading to controversy, please edit it to clarify your meaning of "optimization technique."  Some kind of improvement of this nature is needed so that we can keep this thread open.
I'm sorry, I can't respond to this, since your previous comment about discreteness makes it clear I have no clear idea what you're even discussing.
Course notes... suggests you should include the homework tag.
I'd assume red and non-red cars.
FYI, There are several good R packages such as $Matchit$ (see website  http://gking.harvard.edu/matchit/) that implements nonparametric propensity score matching.
Actually, SVN takes binary blobs such as pdf files etc without problem. I'd suspect git does too.
Maybe this example helps: imagine that I observe the daily return of 10 stocks over 2 years. My hypothesis is that in the first year investors didn't distinguish much between these companies, but in the second year they started to distinguish them. If this is true, I should see that in Year 1 the returns tend to move together very strongly, while in Year 2 we expect more idiosyncratic movements. In other words, the common component (say PC1) should explain much more variance in Year 1 than in Year 2. Now I want something similar, but the universe of stocks in the sample can change over time.
I think article is a good introduction: http://www.jstatsoft.org/v31/i10/paper
I'm not sure that this is directly related to my question. I don't want to make a global test out of many small ones, but to compare two tables as if they were joint distributions. Something like a KS test for tables
I believe this requires n>60 or so - wonder if there is a more general or exact solution?
Troels
@Alexey can u elaborate a bit more
@chl Useful links. Many thanks.
@whuber. This is a least-squares fitting problem.
I am not referring to the median or average inter-center distance, here I want to keep the minimum inter-center distance over a certain threshold (e.g. 60).
Actually, you don't need the book for this (you definitely should read it, but not for your current problem). All you have to do is plot the time intervals between infections and build your control chart based on this variable. Try it, you'll see that it's quite simple.
@RobertSmith I see the term "partitioned matrices" (still not "tensor") but I guess it's similar
@kkp The only ways I can think of right now are: `pairs()` for scatterplot matrices showing all pairwise distribution of the predictors. Package `scatterplot3d` can be used to visualize the scatterplot of 3 variables. See [this answer](http://stats.stackexchange.com/questions/9898/how-to-plot-an-ellipse-from-eigenvalues-and-eigenvectors-in-r/9900#9900) for drawing the ellipse whose axis-ratio equals $\\kappa^{2}$ in the bivariate case. You might also check package `perturb` which offers additional multicollinearity diagnostics, maybe graphics.
"Error: unexpected symbol in" also appears if you try to `return something` instead of `return(something)`. I learned that today ;)
For example, you can find details here: http://www.google.be/url?sa=t&rct=j&q=choice%20bandwidth%20kernel%20density&source=web&cd=3&ved=0CDwQFjAC&url=http%3A%2F%2Fciteseerx.ist.psu.edu%2Fviewdoc%2Fdownload%3Fdoi%3D10.1.1.44.6770%26rep%3Drep1%26type%3Dpdf&ei=psGgT_WDLoag8gO3qcC9CA&usg=AFQjCNGGQORw2z4nqfm6rl9V33M912EU6A&cad=rja
@Aravind: The main difference is the probability you are concerned with. It is 0.5^5 when you look at $P(X1=1,X2=1... X5=1)$ and 0.5 when you look at $P(X5=1)$. In the latter case you have summed over all other $X$s, i.e., $P(X5=1)=\\sum_{x1,x2,x3,x4}P(X1=x1,X2=x2... X5=1)$.
@SpeedBoots Maybe the academic papers you are reading are focused on explaining the past rather than predicting the future.  Furthermore, it could be that there are certain variables that predict a stock's price going up.  The January effect would be an example of one such model.
What a fascinating article @JeromyAnglim. 13 years ago, I did my dissertation on collinearity diagnostics. There was (and maybe still is) a big debate on centering. The article you posted is one side of it, but, at the time anyway, Belsley argued that centering DID reduce problems. The book I used was Conditioning Diagnostics: Collinearity and Weak Data in Regression.

The math gets a bit over my head; and I see Belsley has a new book out; perhaps this has been resolved.
I should add that I am principally interested in estimating the separate variance associated with each measurement instrument.
They are indicators used in SPC (Statistical Process Control). You can easily wiki or google "Statistical Process Control".
It comes down to a basic axiom of probability: p-values are probabilities and probabilities for the outcomes of independent experiments don't add, they *multiply.*  Where multiplication is concerned, logarithms simplify a product to a sum: that's where $\\sum\\log(X_i)$ comes from. (That it has a chi-squared distribution is then an ineluctable mathematical consequence.) Far from begin "convoluted," this is perhaps the simplest and most natural (legitimate) procedure conceivable.
Five to seven -- I'll try to convert it to en-dash.
$S$ is indeed the variance-covariance matrix of the outcomes. However, what makes you think it is diagonal? And if $X$ is $100 \\times 5$, then indeed $S$ (and hence $S^{-1}$) is $100 \\times 100$. And then $X'S^{-1}X$ is $5 \\times 5$. There is nothing special going on here.
+1 Great advice, especially #1.  Concerning #2, be careful not to overestimate what your potential client might know!  (I must have had hundreds of professional conversations in which the client said "I took *x* statistics courses in graduate school but have forgotten everything"; *x* typically is 3 or 4 and the clients were anywhere from zero to 40 years out of school.)
Your title is about what's *possible*. It's *certainly* possible to accept the alternative -- I see people say that they accept the alternative frequently, but that doesn't imply that it's correct. But your body text is not about what's possible, it's about what's correct. Could you make your title match the remainder of your question?
I am currently watching [linear algebra](http://web.mit.edu/18.06/www/videos.shtml) lessons from MIT open-courseware when I get a chance. Also [differential equations](http://ocw.mit.edu/courses/mathematics/18-03-differential-equations-spring-2010/) are on my to-do list. Although neither are directly related to statistics, but each has obvious extensions to a variety of statistical analysis.
Yes, I like the viewer in RStudio. However, as I said in my post, it shows only first 1000 rows. Can I increase this by changing some setting. I tried to change max.print, but that did not affect the data viewer.
It just shifts the whole distribution along by $n_1(n_1+1)/2$, sample test statistic, quantiles and all. If you're doing it R, the solution I offered you there gives you the correct p-value (since the p-value was computed $W$ vs null distribution of $W$, and you want $T_W$ vs null distribution of $T_W$ and both are shifted by the same amount). [I am not sure I understand the difficulty here, it's a bit like saying "I know how to tell if I have cut this piece of wood the right size but if I change from measuring in cm to mm, how do I know if it's the right size now?" ... did I miss something?]
Is there a reason not to include all cases and fit a mixed model using neighborhood as a random effect (in lieu of pairing)? This would remove the variations among neighborhoods from the residual error. In addition, the significance of neighborhoods would indicate the importance of those variations.
Are there exactly five sites?
The same theme is developed, rather differently, in L-moments (start at http://en.wikipedia.org/wiki/L-moment). The second L-moment is essentially a reincarnation of an often invented measure based on comparing using absolute differences rather than squared differences. See also https://projecteuclid.org/download/pdf_1/euclid.ss/1028905831 for an accessible (double sense) historical perspective.
Thank you! I don't know Mann-Whitney U-test. Is it safe to use DeLong's test for my purpose? It seems good that it is non-parametric.
@shabbychef: You are right. Fixed.
Here you go: http://www.encodestatistics.org/ with direct link to the software here: http://www.encodestatistics.org/releases/block_bootstrap-0.8.1.zip
Hint: use the historical definition of e as the limit of [something](http://en.wikipedia.org/wiki/E_(mathematical_constant)). You should end up with a distribution that has a double exponent in it.
@ Michael Thanks for clarifying the concept.
SAS is complicated.  Procedures can overlap on the types of analyses that they do.  But different procedures are developed by different programmers at SAS and they have different ways of specifying things.  To be sure that REG and GENMOD are performing the same exact analysis on your data you need to be very careful that you have done all the specifications correctly for both procedures.
I'd wait on other answers and votes before taking action. I work with time series, but am not a time series guru, so there may well be other options that might help you preserve some of the detail you got. (From your example, you did get more than two readings per month, though not a regular every-two-weeks reading.)
p-value is not a measure of (dis)similarity. The KS statistic does at least measure a kind of dissimilarity, though if sample sizes are a bit different you might take $\\sqrt nD$ rather than $D$. The KS statistic will speak to the question 'least dissimilar', at least by that measure, though of course other measures are possible. The 'clear winner' issue may be less easily settled.
You can push the logarithms through `qqnorm()` too. If there is systematic curvature, lognormal is not quite right, although that doesn't mean that there is a much better candidate. Gamma might be another one to try. My wild guess from the density plot is that lognormal will work better than the gamma.
@aaronlevin, in my experience the field we refer to as "Applied Probability" is much more probability than applied. I like [Applied Probability and Queues](http://www.amazon.com/Applied-Probability-Queues-Stochastic-Modelling/dp/1441918094), with a concise treatment of Markov chains and other fundamental stochastic processes and with many illustrations of probabilistic models of Queues etc. However, I am not sure this is the probability book you are looking for. What kind of work do you do? By "applied" do you actually mean "statistics"?
1) Yes. If the treatment happens through C, then this is the correct specification. 2) if X, A, and B are only used for identification (i.e. they make C a valid treatment), then you can just include them in the regression. That's the case if only C is the "treatment". If all A,B,C are "the treatment", you better stick with the aggregate effect.
Without more detail, this is more like a comment than an answer. Could you add some syntax?
I've edited the question to provide more clarity there.
I used to think that [frequent pattern mining](http://adrem.ua.ac.be/~goethals/software/survey.pdf) has to do with association rules and frequent item sets, two topics that are in close connection with the data mining approach too.
Estimating the model means calculating its coefficients. This is a nonsensical question
if the results look sensible, proceed with caution.
@Wayne, "The problem would be the combinatorial explosion". Indeed, ${30\\choose 7} = 2035800$ ;-). Of course, you could always just create dummies for the combinations that exist... still, this asks a different question, whether the combinations differ, rather than the contribution of each individual to the group. What's important to determine here is which question you want to answer.
In this type of research, the question is always the same. Can we predict the result of the test / procedure by using variables known before we perform the test / procedure? I know its wrong to do it this way. But the papers are all the same. You collect a bunch of variables, do multiple regression and see what becomes "significant", then you publish. That is also the reason why there are so many conflicting results.
Ah, makes sense. Thank you very much also for the pointer to the rms package! Wish I'd known about it a long time ago...
Found another potentially relevant paper. Added to edit.
You appear to be using the term "confidence" in an unusual sense.  Could you provide us a definition, please?
I gave you three possible reasons, each one needs a check
No you can't use correlation to assess the reliability of the measurements or the inter-rater agreement, even for two series of measurement. The correlation computed from two raters will remain the same even if you add some arbitrary value to the 2nd rater's assessments, while the agreement ICC will decrease and correctly reflects that there is a rater-effect.
@Henry, the latter. I've edited the question to (I hope) clarify that.
@ 2) more precisely, in `lme4` you can either specify a diagonal covariance structure (i.e., independent random effects) or unstructured covariance matrices (i.e. all correlations have to be estimated) or partially diagonal, partially unstructured covariance matrices for the random effects. I'd also add a third difference in capabilities that may be more relevant for many longitudinal data situations:  `nlme` let's you specify variance-covariance structures for the residuals (i.e. spatial or temporal autocorrelation or heteroskedasticity),  `lme4` doesn't.
Actually, the TraMineR devs are directing users to Stackexchange, as their mailing-list search is broken.
Thanks! Following this point of view I should also be right in using Spearman's Rho, I guess.
(+1) looks promising ...
It is unclear what "statistically relevant" might mean, but are you perhaps asking the same question as http://stats.stackexchange.com/questions/18603?
Thanks to everyone for the answers.  I'm now clear on what I need to do and I've found a couple of very useful papers.  You're correct,it is logistic regression I want - to get the odds ratios out.  Thanks!!
Last_action_1 seems to be the same as Last_action_2. This should be buy something I think.
Even if you don't call a trimmed mean function, you can get the same results by sorting the values and dropping (e.g. deleting) the tail rows.