tttjjj/elt_bench huggingface dataset:
I don't know, but this is what I'm trying to find out because I'm doing a work sheet in class and I don't know if having Demicals in my answers with a percent is correct!!!! Hellllpppp Me If Anyone finds out I need the info quick plz txt 1-936-332-9209
Hmm... if the variable they use it on is ordered categorical with 5 categories, it would be a somewhat natural choice. It doesn't look like that's the case though. It might just be a general "measure nonlinearity" thing that's grown up in one area, probably because of a single paper doing it and people copying that (and then others copying those papers).
If you want to search for R packages having to do with a certain topic, use http://www.rseek.org.
"The complement", or more accurately "the probability of the complementary event".
Thank you!
A quick question: is there a way to produce r^2 or pseudo-r^2 like Nagelkerke in glm using family=poisson in R? 
Thank you!
@JKP: I had come across this command, actually. However, considering that my outome variable is binary, I am assuming logisic regression would be more appropriate than categorical regression (CATREG) - am I right? Also, the options in CATREG seem rather limited - you can only choose between excluding cases, replacing missing values with mean values or creating an extra category.
A unit increase in $\\ln(\\text{sales})$ is the same as multiplying $\\text{sales}$ by $e$. More typically we'd examine the effect of say a $1\\%$ or $10\\%$ increase in sales ($\\ln(sales) + \\ln(1.01)$ for the log variable, for example) as being more interpretable.
In your $Y = a + b x,$ the new sample mean is $a + b \\mu$ while the new standard deviation is $|b| \\sigma.$ The suggestion is that you would use $Y = 47.5 + 1.5 x $ but not use $Y = 152.5 - 1.5 x.$
What are your covariates and what assumptions are you willing to make about the available student data (e.g. independence from one student to the next, dependent on additive functions of the covariates, independent additive error component for the model).
I added a revision to the content so that it more closely asks what  I am 90% certain @Arumairajan is trying to ask.
The Hausman test is based on the standard errors of your comparison models (see [here](http://stats.stackexchange.com/questions/63049/hausman-test-the-larger-the-sample-the-more-significant-the-hausman-test-statis/63075#63075)). I've never seen this kind of error with xtoverid and it should work because at the bottom of the help file of the command there is an example of how to use it in order to compare FE and RE models.
great, I guess this is the answer ! I did not know Hansen-Hurwitz...
Basically the covariance of the aggregated units is the sum of all the covariances of the micro level units - so when you aggregate more you are adding up more spatial covariance terms. I go through a similar exposition about how *aggregation bias* occurs in [Chapter 2 of my prospectus](https://dl.dropboxusercontent.com/u/3385251/AndyW_Prosp_09252013.pdf). That is about the best hand-wavy reasoning I can give to the spatial auto-correlation off-the-cuff though. (I doubt I understand the implications fully.)
Monotonic *and increasing*, of course.
Causation is just not something you can squeeze from the numbers... so, repeat after me: *causation is not correlation*, *causation is not correlation*...
+1 for tying your answer back to the meaning of the hypotheses being tested.
This is analogous to the distinction between type I & type III Sums of Squares for testing coefficients in linear models. It might help you to read my answer here: [how to interpret type I sequential ANOVA and MANOVA](http://stats.stackexchange.com/questions/20452//20455#20455).
The partition is based completely on the location of the centers, e.g. a point belongs to the cluster with closest center.
In my experience, if you want to understand what someone means when they tell you to do something, the best thing is to ask them what precisely they want, rather than ask other people to guess (at length) what their intentions were.
If instead of using $\\text{age}^2$ you introduce the functionally equivalent $(\\text{age} - \\text{mean age})^2,$ what correlation do you find with $\\text{age}$? :-)
Could you add a couple of links to the couple of articles in question?
How big were the training and validation sample sizes? Maybe the model without an intercept was better just by chance.
+1 for the important distinction about individual vs. group
decision. I should have anticipated your response, given your reply
[here](http://stats.stackexchange.com/a/14178/930) or another response of
yours on the
[medstats](http://groups.google.com/group/medstats/browse_thread/thread/30011781b32ebb1a)
mailing-list. I also found your talk on [Direct Measures of Diagnostic Utility Based on Diagnostic Risk Models](http://www.fda.gov/downloads/MedicalDevices/NewsEvents/WorkshopsConferences/UCM256357.pdf) particularly enlightening in this respect.
It is proved here: http://stats.stackexchange.com/questions/49942/why-is-the-fisher-information-matrix-positive-semidefinite/49961#49961
Thank you for the edit and answer Peter +1!
That's true, I'm still at the problem definition stage. You have any ideas? I was thinking of Markov Decision Process, but I mostly have background in regression, so not sure.
Unfortunately this won't fly for my proposal.  Can I use G-Power and simply calculate for MANOVA?  Or use factor analysis recs of 10 per item?  And what would be the precedent for that? I am having NO LUCK finding anyone with recommendations.
Thanks, Matt! Sample data pasted above.
@ttnphns In this context, "uniform" has a well-established conventional meaning. It corresponds to a Poisson process with constant intensity. It is often known as "CSR" [completely spatially random](http://en.wikipedia.org/wiki/Complete_spatial_randomness).
CV error <-> IS error: yes. Which is better? That will greatly depend on your goal. It may not even be so that either serves you best (e.g. if false negatives are worse to you than false positives). I seem to recall that Friedman et al are rather advocates of CV, but even then you have a ton of criteria for different goals (missclassification, AUC, deviance, ...) You have to figure out which one is the best for your goal.
They are pretty different guidelines.  I suspect the disconnect is whether you think that the test of the overall model matters (the lower N guideline) or the test of the individual slopes that matter (the higher N guideline).
GARCH/ARCH can be used on any time series, but it's not necessarily a good model in many situations. Anytime you think it is a good model, there's absolutely nothing stopping you using it.
I wonder whether looking at correlation coefficients would be useful.  What would the test results tell you?
@feelfree, I believe I fixed my answer. Please give that a try; it does work for me.
Thanks @Pat I think that's exactly what I need.
Constant mean does not imply stationarity.
@andrewH You can up-vote useful answers and comments. The best one that answers your question satisfactorily can be "accepted". i.e. click on that tick mark.
+1 from me--nice explanation.
It would be easier to answer the question with some more information.  There are sixteen individuals but way more than sixteen boxes on the histogram.  Are there lots of sites per individual?  Lots of measurements of methylation for each individual-site?
Opinions differ a little.  A brief "Thank you" at the end of a question is common, but more than that is usually considered an unnecessary distraction.  You can quickly find guidance on the site-wide meta site, such as [the SE FAQ page](http://meta.stackexchange.com/questions/7931/faq-for-stack-exchange-sites).
I understand the definition of plim and the construction of the first fraction, but not how the notes go from the first fraction to the second one.
+1 A better answer than mine as it explicitly takes into consideration the range over which the pdf is non-zero.
No. If you rule out some assignment vectors a priori you no longer have random assignment, strictly speaking.
In line 2, are you sure you want `runif()` for your error term? People typically assume the errors are normal (ie, `rnorm`). It's fine for them to be uniform, but it's worth being clear on that.
@sbg try sorting your `x` variable: `lines(sort(temp$x),predict(mod, list(x=sort(temp$x)))`
+1, thank you very much for your answer. You wrote about the "glaring omission": let's say we find a way to include the information that some objects measured by A is then also measured by B, do you think that then the estimates will be better?
`anova(mod1, mod2, test="Chisq")` and `anova(mod2, test="Chisq")` should give identical results for the interaction term. Note that you don't have to manually calculate a p-value in the first case. Without the actual data / minimal example, it's hard to see what's going on.
Oops - missed editing deadline...

I don't know if this is correct but I've experimented with simply calculating the binomial distribution - i.e. (n! / ((n-k)! k!)) p^k (1-p)^(n-k) where n = number of trials, k = number of failures, p = probability of failure.

In my current problem, the buggy code gave me 3 failures in 9 trials - let's assume this implies a probability of 1/3. Chance of 0 fails in 9 is 2.6%, better still 0 fails in 15 trials is 0.23%. That gives me a feeling for it.
@nico Actually, pattern of the keyboard clicks (without distinguishing which key is hit) is equally interesting.
but was your design a Randomized Complete Block Design?
+1, I got the same. Well done.
use whichever is easiest to implement! They are asymptotically equivalent, so no need to get fancy. For a single coefficient, you can also use the t-test, which allows you to make 1-sided hypothesis tests.
You can do it with 2 time points.
Great answer and more informative than the other one, but not quite what I was looking for.
Oh ok, thanks for your help! I think this may be down to my confusion over how to assign the random effects. `Treat` is rather confusingly named I'm afraid. `Treat` is whether a territory is occupied or not. So I have territory locations on each farm `treat==1` and have selected the same number of random points to act as unoccupied territories `treat==0`. So when I include farm as a random effect in the model, I'm not grouping the data into the three levels but asking the model if the number of occupied or unnoccupied territories differs per farm? (which by design is exactly the same).
I thought there we unreachable parts of AUC as well. But could be wrong.
Just realized I didn't directly answer your question directly.  Yes, if R selected 1 then the remaining selections would come from {2,3,4}, etc.  Confidence level would be calculated for the responses in each individual category.  For example, if we have 50 responses and a four selection forced ranking for each of the ten independent categories how do we calculate confidence and margin of error for the results in each in individual category.  With four statements there are 24 permutations for each category.  This seems similar to a Lickertt scale for each category with a scale from 1 to 24.
so this is basically regression with a 'tube' loss function (0 penalty for points +/- epsilon of the prediction) rather than the quadratic loss function from OLS?
+1 Dikran.  Good answer to a nebulous question.
Honestly I don't understand you. And you seem to try asking some new question unrelated to your current one. Maybe you overhaul it and post as another question.
Adding an L2 penalizer will help control high multilinearity without dropping any variables. That's my preferred technique.
@garciaj ...and come back with more questions!:)
Given that this question already has an accepted answer, this should be more of a comment. You don't really add new information to what was provided before.
Could you, please, point us to a correct calculation? ... in "as if" case?
So please guide how may I proceed? what change in code is to be done Mmm
@TooTone's answer is very good, but one more thing that one needs to keep in mind is that spammers know that naive bayes is a common tool in span detection, so they try to go around that.
Awesome, the examples on 1 and 2 really hit the spot! Thank you so much for your explanation!!
this might help http://stackoverflow.com/questions/2595176/when-to-choose-which-machine-learning-classifier
I  can confirm. It is correct
Knuth discusses this: you worry about the distribution being uneven (detected by a large chi-squared value) and about being *too* even (which is a subtle but potentially important departure from randomness): that's detected by a chi-squared value which is too small.
What definition of "nonnegative definite" are you using?  (The result you are asking for follows immediately from [many definitions](http://en.wikipedia.org/wiki/Positive-definite_matrix#Negative-definite.2C_semidefinite_and_indefinite_matrices).)
Found a matlab-package! **[Kevin Murphy's toolbox](http://www.cs.ubc.ca/~murphyk/Software/HMM/hmm.html) offers a possibility to fit to multiple samples of varying length** and generally offers much more functionality than the functions available in the statistics toolbox.
that's another way of saying that the distribution shown in the plot is symmetric.
Shouldn't be $u_t = \\sum_{j=0}^\\infty \\psi_j \\epsilon_{t-j}$ ? I'm fine with the rest then. Nice explanation.
By "MCA" & "Multiple Correlation Analysis", did you mean *multiple correspondence analysis* (b/c you include the `[correspondence-analysis]` tag)? You certainly wouldn't want *Multinomial* LR, b/c your response is ordered; you'd want *ordinal* LR. Most likely, what you ultimately want is to use [item response theory](http://en.wikipedia.org/wiki/Item_response_theory); however, it is pretty advanced & requires a lot of data. You may need to work w/ a consultant or settle for simpler analyses.
I should clarify that I am actually assuming the functional form of the likelihood is double-exponential I didn't realize that $p(\\textrm{accept}) = L_\\textrm{new}/L_\\textrm{old}$ was the correct calculation; I thought that I needed to calculate $p(L_\\textrm{new})/p(L_\\textrm{old})$. And to estimate $p(L)$ I used `AIC(fitdistr())` to find the bset fit to the distribution of $L$ and so that I could determine the $p(L|\\theta)$ for each value of $L$.
No, $x$ is just one row. Don't concatenate because you will introduce "false" transitions: last state of one line $\\to$ first state of the next line. You have to change the code to loop through the lines of your matrix and count the transitions. At the end, normalize each line of the transition matrix.
@OP: If Wigan have scored a goal after 30 minutes then, for Wigan to win overall, the position in the remaining 60 minutes of the match must be "Wigan score at least as many goals as City" which when added to Wigan's existing lead means "Wigan scores more goals than City".
Where did you see the formula? It's not correct. (There cannot be $y$ on the left hand side while $y$ is integrated over on the rhs).
Do you mean the variable loadings? This is shown under "Rotation:" above or get by `mod$rotation`. But I must miss something, please give some more details.
Giving good advice here would need me to know a lot about what you're modelling & how the model's to be applied. (If that's nuclear reactor faults & preventing them, please consult a statistician.) With that caveat, I'd be inclined in most circumstances to keep all events  over the two years & sample the non-events, perhaps putting 'time' into the model as a covariate. The model based on four months' data loses out on 5/6 of the minority-class events, & data prior to this would have to be *very* irrelevant to offset the loss.
(then accept the answer fo future generations)
"you are saying the predicted outcome is random around a step function, not integer values around a smoothly increasing relationship" --- That is something I had not considered.

In the end, I went with Poisson regression by glm. Its not the perfect choice, but "good enough" for what I needed.
A working example means an example that we can run that also has this error.  This helps us help you more quickly by letting us debug the error on our own systems.  It also makes it more likely for you to find the problem yourself.  If you'd tried to do this, you probably would have tried running on a smaller subset of your data, and it would have worked, and so you would have tried another one, and soon you would have found the problem yourself.
The problem is that stepwise procedures completely invalidate all the assumptions and preconditions for "normal" inferential statistics based on p values, which are then badly biased (downwards towards being "more significant"). So basically, the answer is "yes", with the caveat that one could in principle correct for all these multiple comparisons (but which I have never seen done). I believe strongly that this is the single most important reason why I see so much research in psychology that cannot be replicated - which in turn leads to a huge waste of resources.
But you said you were using a supervised learning method. My comment is about how using the previous work experience as the label (which I assumed you considered the "job that matches the candidate") seems like a problem. However, have you thought about calculating the distance between a candidate's resume and various job postings (with a lot of preprocessing for awful job postings)? Maybe using [LSI](http://en.wikipedia.org/wiki/Latent_semantic_indexing), as you said. You can get a lot of them by crawling some job search pages.
That's a good point. I'm trying to think of a similar example where $k$ is not constant yet maintains the correlation. Nothing seems to come to mind - do you have one in mind?
Only if your profit is based on selling the software---and even then, some people still give the software away as e.g. RStudio whose source code is on github.
For algorithmic performance it almost always is the case that the timing ratio is relevant, not the timing difference.
Because this is a homework/self-study question, you should add the `self-study` tag to your question. Please take a look at http://stats.stackexchange.com/tags/self-study/info
I have provided an example. I must admit that I have not tried to consider alternative algorithms as your second bullet point suggests.
repeated-measures Anova has assumptions (most notably sphericity) that is usually not present for mixed models. I am no expert on SPSS but I would bet this drives the difference.
I recently had good results clustering a corpus of essays with hierarchical clustering using cosine similarity.
RR is meaningless in case control studies because you do not the base rate (which may be different from the rate you observe in your control)
Depends on your data and domain. Sometimes they are, sometimes they aren't. Try different things.
Closely related: [Measuring individual player effectiveness in 2-player per team sports](http://stats.stackexchange.com/q/15776/2970). Also of interest: [How to get started with rating and ranking based on pairwise competition data?](http://stats.stackexchange.com/q/30976/2970). Mild curiosity: [(A comment on) When is logistic regression solved in closed form?](http://stats.stackexchange.com/a/6701/2970). Searching for [Bradley Terry](http://stats.stackexchange.com/search?q=Bradley+Terry) may yield other useful ideas. Cheers.
Thanks for the links Bernd.  The first link I found by myself but was not able to reproduce the example.  The other links might be worthwhile exploring - thanks.
Crossposted here: http://stackoverflow.com/q/21081654/429846 Please don't do that; flag your original question for migration by a moderator to this site.
+1 That's a nice one. The funny thing is this, sometimes people can picture it (make or see a plot of something) and still not understand it. :)
Could you post any code where you think the problem might exist?
I am not sure if I can call them binomial experiments. In my experiment, animals (trials) are not behaving independently. Imagine a sample group of n1 animals, it could end up with x1 animals in the target area simply because some of them decided to live there and the rest follows them. So I guess the estimated proportion p is associated with sample size. In the simplest case: n1=n2, may I still combine them into a single binomial experiment? Thanks
I definitely appreciate the thought rolando, but it's worth adding that I'm likely going to be dealing with fairly messy archival data (in terms of predictor variables) that will be hard to structure for use in a regression.  Hoping this pilot project will not only guide specific hypotheses, but also help give a sense of how to operationalize some of the potential predictors (or convince me that I'll need to work prospectively on this in the future).
Is there another/different tag that I should be using to get this question more attention?
In order to give the best recommendations, it would help us to know how you will proceed after identifying "significant predictors."  Are you trying to *predict* the outcome as exactly as possible; find a *parsimonious* way to predict it (e.g., using a set of up to *k* predictors that will efficiently do so; *explain* what causes the outcome "in reality"; or something else?  Also, how big is your data set?
Nice, thanks for the pointer.
1. "show n" does not work in PSPP,
2. Your code works!
3. "freq var $casenum." does not work as you guessed
+1 for the board game example
When you use a function not in the vanilla distribution of R, please state the package it's from. I presume you mean the `dredge` function in package `MuMin`. Is that correct?
... and this answer is correct in defining studentized residuals from a regression equation.  There is no definition of a corresponding standardized residual.  The regression framework doesn't seem to apply to the question asked.  But this is still a valuable contribution; +1
Closely related threads: [Interpreting PCA scores](http://stats.stackexchange.com/q/4093/930), [What are principal component scores?](http://stats.stackexchange.com/q/222/930)
First of all I have the median of my data (the point I want to plot). Then, after bootstrapping I have the distribution of all the median, which in priciple should be gaussian. So if I want to plot the error bar on the point I have plotted, there are two ways of plotting error bars:
1) measuring the sigma so the errors are median+sigma and median-sigma
2) taking the 16 and 84th percentiles so errors are median-16th and median+84.
Should I divide by a factor of 2 in the second case?
Thanks again Randel! This is a massive help and clears up a lot of things. I've been worrying about this GLM for days!
@HishamC.Hassan Instead of dividing by zero by using Q1 as a reference, you're dividing zero *by* the reference. You'll still get unstable results for the HRs involving Q1. The difference is that's one unstable result, instead of *all* unstable results.
And Characteristic functions are used to prove central limit theorem and for saddlepoint expansions
Then the t-test isn't appropriate.
"To see this, you can look at the higher-order terms in the delta method expansion" that's exactly what I'm going to do tomorrow morning (I don't want to have nightmares tonight :) ). I got interested in small samples because in my application it looks like the asymptotic rate kicks in really slowly, so that (having finite computational power) small sample behaviour becomes important.
I don't understand what you mean by "significant variability among the others". I am also not sure exactly what you mean by "variability" since you say these data are categorical. I can think of various ways to think of this.
Anyone? I posted in the lme4 mailing list as well, but I still haven't got a solution to my problem.
THanks
"*sth*"? Please expand your abbreviations. You don't have a 140-character limit. Please also be consistent with use of affect/effect
Hi @ejs.  You would need to code "sample" the same way as any other categorical variable - dummy coding or effect coding or whatever.
I think it would be better to remove the people who didn't have an accident and then account for the matching of people. If you only have two time points, you could take the difference.