tttjjj/elt_bench huggingface dataset:
+1 Great advice. I have been in an R tutorial class before where the instructor's code was written using an older version of R than the campus lab computers had and it took 30 minutes of debugging to figure out what changed in the point releases and how the course code had to be modified. While some software is perhaps too obedient to backwards compatibility it has never struck me that this would be a criticism of R.
This is the type of data that my example comes from too, except two genotypes and two analysis methods. Try using a mosaic plot, interweaving a third variable into the horizontal axis. Drug a on the vertical axis, and results from the others on the horizontal.  There is a package called productplots in R that might help. It is not as easy to use as I would like.
My ultimate purpose is to be able to calculate coverage. I am making two different bugs calls because I am running the same model with 2 chains for 2 different datasets. I want to do this for say 300 datasets and then calculate coverage (out of 300 times how many times is the true value of p contained within the 95% prediction interval etc). This is the reason I was trying to identify mean or say any other descriptive specific to an output so that in the next loop I could call that.
Sorry, I should have mentioned that you need library(quantmod) for the lag function
Thanks! i will give these a go and let you know how it goes
Actually using the result is not part of my thesis, but one can assign labels to it via some samples of each cluster. Applications could be normal/abnormal packets for future ids or application detection or sth like that.
Please tell us more - what are your IVs? What is your DV? Is there a huge pile at a specific value of the IV?
Any advice on this question?
Oh wow okay that makes total sense - I don't know why I couldn't get there. Thanks!
This is a rather free translation of a saying attributed to Gabriel Lippmann by Henri Poincar\\'e in his Calcul des probabilit\\'es (1896/1912). Original was in French, naturellement. Lippmann won the Nobel Prize for Physics in 1908.
Hi Chris, so you are not reducing the number of items in a likert scale, but rather using less questions/items (who are measured on a likert scale).  In general it sounds like it depends on your measures.  You could check the correlation of the items you intend to take down with the ones you are keeping.  It's actually an interesting how to measure how much to remove - it might be worth to reframe the question that way (if you won't, I might do it later).  Good question :)
... but color me confused because "repeated measures ANOVA" is in the title.
I don't have a strong opinion either way. Agresti has argued that you can re-represent ordinal predictors w/ continuous values that you think are reasonable. If you aren't perfectly correct, there will be some measurement error, but if you have any knowledge of the topic to draw on / aren't totally off, the effect will be minor. If it subsequently seemed normal enough, I might turn them into z-scores. You should read the material that's on the site, these issues have been discussed.
for the uninitiated: http://www.youtube.com/watch?v=zLq2-uZd5LY
Regarding Q1: Are you only referring to observational studies? If not the easiest way to test for a causal effect is to conduct a experiment in which you e.g. apply food-restriction to one group of mice but not to another group and then test if the starving mice live longer.
What does the major exposure of interest interact with? It'd be helpful if you can show the model (variable name and p-value).
My friend's wife constantly nags him to do chores: "Honey, it will only take ten minutes!" From experience he has discovered that a "ten minute" chore will take an hour and an "hour" chore will take up most of the day. So now when she asks him to do something that will only take $x$ minutes, he figures it'll really be $x/10$ hours: that's the bias correction. They're both happier now that they're in agreement :-).  (In many fields, this is called *calibration*.)
Unfortunately the link to the slides is dead and I could find the slides elsewhere.
You might try to edit you answer (you can do so by clicking the "edit" link in lower left) to improve basic grammar (e.g. capitalizing the first word of each sentence, etc.).
how do you construct the indicator variable for membership of group A? Why would you think these approaches are smiliar? On the other hand, you have a highly unbalanced problem (90% A and 10% B if I understood correctly) since the default setting of random forests is simply to take the majority vote that's why A always wins. There are much more A observations. You can try balancing the model by doing randomForest(..., sampsize=c(size,size),...) where size has to be small than the size of your smaller class. Maybe like 60% of it since the objective of random forest is to introduce variation.
It holds for variance not standard deviation
I looked at the package, and found that this cannot be used for GLMMs. Sorry if I didn't specify this clearly, but i'm trying to fit a GLMM with a poisson distribution.
(+1) very thorough answer. I'd only add that it is possible to obtain tests that assume only monotonicity of the dose-response relationship without assigning scores to the ordinal variable using isotonic regression., although simulation or permutation methods are usually needed to obtain the distribution of the test statistic under the null. See Salanti & Ulm 2003 http://dx.doi.org/10.1002/bimj.200390012 .
@cardinal That assumes the RNG is not "high quality."  Actually, come to think of it, the proposed sort-and-select procedure is immune to non-uniformity, whereas my suggested replacement is not.  All you need for the s&s algorithm is an RNG for *any* distribution that (a) is guaranteed independent from one call to the next and (b) has (for all practical purposes) zero chance of producing ties.
Thanks; actually I've run into similar issues when trying to abuse AC1 to my purpose and end up using entirely different approach.
@whuber I am not getting, how R does the standardization? I wanted to it like this: http://stats.stackexchange.com/questions/57804/different-possible-parameter-combinations-to-obtain-a-standardized-generalized-h but it is not working, resp. there are more than just one solution?
I've looked at Compressed Sensing and as a non-statistician I keep asking myself, "Isn't this just inverse random projection?". I know that "just" is an easy word to throw around, but it feels like people are leaving out what seem like obvious connections between random projection (circa 2000) and compressed sensing (circa 2004).
+1, a lot of good thoughts here. 1 quibble though, re #1, I would say we should often have *lower* standards (i.e., *higher* p-values) as preferable. It is often difficult to get enough data to have good power to study something. I've run a number of power analyses for doctors who want to study a rare condition. They say, 'this is really understudied, I have an idea for a new approach, we can probably get 50 patients w/ this over the next two years', & I say 'your power will be 45%', and the project is abandoned. Rare diseases will continue to be understudied if p must be .05 or less.
Because PCA *begins* with either the covariance or correlation matrix as its input, it is hard to see why there's a question about what is calculated first.  Perhaps you might like to learn more about what PCA is and how it works: just click on the [tag:pca] tag beneath your question and read some of our higher-voted threads on the subject.
Here is a method that I developed for Random Forest Regression to Detect Outliers: http://bioinformatics.oxfordjournals.org/content/early/2010/12/05/bioinformatics.btq628
what's an 'on-disk representation'?
A very nice article which might be useful is Hallgren (2012): http://www.tqmp.org/Content/vol08-1/p023/p023.pdf
@MichaelChernick Did you ever read the question? It says: "What BOOK would you recommend for non-statistician SCIENTISTS?" Downvoting something just to "vote" isn't really logic. Besides, if you would have read the book, you would know it takes a pretty scientific look at statistics. Knowing the book (among a truckload of others) and having read the question (in contrast to you) convinces me that my answer validates as correct and may be helpful in regard to the OP's request. Anything else I can explain to you? Tsss...
What is your hypothesized relationship among the variables? Is the Likert scale question to be thought of as a dependent variable? An independent variable? Neither? (Sounds like a DV to me, that is, anxiety depends on consequences).
What are the messages?  I would guess they are related to the nondifferentiability of the first model and the terrible starting values in the second.
(ctd)... To say it 'arrives at time $x$' for a continuous variable, means "what is the limit of that last probability as $dx\\to 0?$. So, what is that limit? Work it out! That *is* the probability it converges to. This feature is intimately related to what makes a continuous pdf continuous.
Thank you Glen_b -- text edited
I agree with @whuber, and dasymetric mapping may be of interest to fmark given the subject material as well. Unfortunately, it is largely separate from the ecological inference literature I cited in my answer (I don't want to pile on more literature more!) What do you think fmark?
This is not a complete answer or anything, but I think it's worthwhile to read "Let's Put Garbage-Can Regressions and Garbage-Can Probits Where They Belong" by Chris Achen. (PDF link: [http://qssi.psu.edu/files/Achen_GarbageCan.pdf](http://qssi.psu.edu/files/Achen_GarbageCan.pdf)) This applies to both Bayesian and Frequentist approaches equally. Just throwing terms into your set-up is not sufficient to "control" for effects, but sadly this is what passes for control in a lot of the literature.
The latter - Looks like i'll have to solve this using convex optimization. Thanks.
Fixation? LOL. Am a PhD student but from Engineering field. Currently working on an interpolation prediction strategy. So happen my data has an exponential trait. Though I managed to get the regression coefficients, then the equation and R squared values, I learn R squared alone is insufficient to test a hypothesis (Null or otherwise).Several write-ups point to that direction.The p-value is said to be an important statistical significant testing parameter.This is my last piece of jigsaw (after learning there is no F statistics for non-liner regression). Will rush to library now.
Consider flaggin answer as correct
Some little examples, especially in the survey context, here: http://filipspagnoli.wordpress.com/category/statistics/lies-and-statistics/
I think its often a question of statistical power and the relative occurrences of each class. In reality, most N-class classification problems will have some classes much better described (in terms of available data) than others due to a lower number of cases for some classes. Personally, I have always gotten better results when reducing a 3 class problem to a binary classification problem.
@chl Thanks - Did not know about that one.
@rbatt: +1 Yes. I am interested in learning more about this topic from a practical perspective and thought this really is a very different topic so I'm posting it as a new question. I am doing some reading but due to my lack of experience, I don't want to use these techniques incorrectly. :) I'll add a cross-reference to the previous question. Thanks for pointing it out.
@DavidJ.Harris By "sometimes present", I mean that the particular training example does not posses the property. It is like if it were equal to zero. In my problem, all my features will be positive numbers in a given range (for example, from 5 to 15 or 100 to 1000).
Do you need to account for scaling, rotation and/or translation?
Sometimes, a statistical model with qualitative response variable is called 'classification model' while a model with quantitative response is called 'regression model'.
A Python implementation of Bayesian Change Point Detection can also be found at [this](https://github.com/hildensia/bayesian_changepoint_detection) Github repo.
Try Ruppert, Wand and Carroll (2003). http://www.amazon.com/Semiparametric-Regression-Statistical-Probabilistic-Mathematics/dp/0521785162
Really, $-\\frac{1}{n}\\pm \\frac{2}{\\sqrt{n}}$? Centered at $-\\frac{1}{n}$?
OK, thanks whuber.
Great question. There are a [bunch of other tail descriptors](http://en.wikipedia.org/wiki/Category:Tails_of_probability_distributions) that appear to be somewhat interchangeable at first glance. In particular, [long-tailed](http://en.wikipedia.org/wiki/Long_Tail) (which is sometimes used interchangably with heavy-, fat-, and right-tail), if you take the first sentence of the wikipedia article on face-value, seems to be a super-set of fat- and heavy- tails (as more rigidly defined on their own pages).
Attempting to interpret the curve's height is problematic, because **the height is meaningless:** density curves depict probability by means of *area* rather than height. (Select any location along the curve. Change the height of the curve there--and only there--to any different value. This does not change *any* areas, so it gives an equivalent density curve.) Examples: http://stats.stackexchange.com/questions/14483/intuitive-explanation-for-density-of-transformed-variable/14490#14490,  http://stats.stackexchange.com/questions/4220/probability-distribution-value-exceeding-1-is-ok/4223#4223.
@MohitVerma : In this terminology ''alias'' refers to the variables that are linearly dependent (i.e. cause perfect multicollinearity). See http://stat.ethz.ch/R-manual/R-patched/library/stats/html/alias.html . I update the answer with this.
The AIC is an estimate of the (relative) Kullback-Leibler information. There is no reason to believe that it is applicable only for nested models (I actually don't know why people think that).
Here's an ad hoc (i.e. non-mathematical) answer: Imagine everyone in the data set is perfectly correlated. Then, you only need to know one observation's relationship to a predictor you figure out the rest. So, you effectively have one observation. Now imagine a less extreme version of that.
I edit the question adding the log likelihood function
Hello @AzulaR. some other information, in case I missed it, I only have a dozen of "clusters" and each one's size is about about 150-200. Also, I only have the "summary" data (i.e. per cluster) not the actual individual patient data. I looked at clogit function in R, I don't think it is suitable for the model you describe. Any further suggestion would be greatly appreciated. Thanks.
@BenBolker Any merit in using bayesglm() without modifying the prior? This also seems to remedy the effect I'm experiencing with glm().
I would consider a finite mixture model with one part being 0 vs not 0 and the other part being among those with more than 0. In `SAS` this can be done fairly straightforwardly with the (new) `PROC FMM`. In `R` the package `flexmix` can be used.
Very helpful, thanks. I'll have to look into these methods.
What format do you want the output to be in? Another SQL table?
Adding the residual plot can be helpful here. But as a general rule, if you see any trend in your residual plot, it means that your model is not good to fit and there are some room to improve your model.
Because this is a homework/self-study question, you should add the `self-study` tag to your question. See http://stats.stackexchange.com/tags/self-study/info
Thank you SOOO much Steffen for the answer.  Will look at the links you recommended and will ask more questions if I have any.  This is VERY useful.
OK, I think your comment was also a good answer. Thanks.
Please explain what you mean by an "expected location."  As to what can possibly go wrong when using Cartesian coordinates for global distance calculations, please visit the [GIS site](http://gis.stackexchange.com/search?tab=votes&q=projection%20distance%20longitude).
Zen's cryptic replies are invitations to do the work myself and are fully half the reason I read C.V.
This text appears to be so garbled that it is not possible to determine what is being asked.
Thank you a11msp, as you've mentioned though, this is not guaranteed to go through the critical points, which is mandatory. My gut tells me that in order for the locally weighted regressive solution to be effective for this problem, the span parameter, or kernel smoother must be dynamic. It must shrink as it gets towards the critical points and grow when in between them. A spline solution would work better, but challenge is how to control the spline minimize distance to non critical points.
I was clearly comparing the AIC in the final output with the one in the trace. Thank you for the explanation.
-1 since the answer is too general and the personal advertising is entirely inappropriate.
It makes sense. The coin-focused question is a bit distracting, though, because answers to such questions typically devolve into discussions of the physics (and sleight-of-hand) of coin flipping. You might be shocked at how contrary the actual situation can be to your strong priors, depending on *how* the coin is flipped. ["It does not make sense to say that the coin has a probability $p$ of heads"](http://www.stat.columbia.edu/~gelman/research/published/diceRev2.pdf).
I'm used to seeing color defined by values on 3 different variables.  Are you talking about color as something that can be defined by a value on a single variable?
1/4 1/8 1/16 ....
I don't use SPSS, sorry.
Another thing the range, the maximum, and the minimum are huge compared to returns per session for instance, the market return per session (mean) 0.10%, the range, maximum, minimum are 12.5%, 8%, -4.5%
I added the nonlinear regression tag, since that what this is. Searching on that tag may find useful posts.
You can just use your dummies as you would in OLS. Include two of the three dummies in your model and then their coefficients are expressed relative to the base-industry (which is the one for which you did not include its dummy). Then you get a column vector of regression coefficients as usual. No need to fit separate models for each dummy. You cannot include all 3 dummies in the model.
The term is defined and explained well on page 5 of the slides that I linked to. You asked what to do and I mentioned the area of econometrics that you may need to study. There is a lot of research and expository material available on this topic, which you can summarize for yourself before asking further, more concrete questions.
@Alexis You can subset a data.frame at any time, for instance: `df <- data.frame(y=rnorm(20), g1=gl(2, 10, labels=letters[1:2]), g2=gl(5, 2, 20)); with(subset(df, g1=="a"), pairwise.t.test(y, g2))`.
@Nick I believe that if we translate the colloquial "characteristic" to the mathematical "real-valued function," we get part of the definition of a random variable. (The missing part, of course, is measurability with respect to a sigma field on the population.) Normally, though, we would translate "characteristic of a sample" into the technical term *statistic*: maybe that's what you are referring to as being a "little off." With these translations, variables do not have "types" at all in Stevens' sense (we can only distinguish discrete from continuous *distributions*)--but some data can.
Right. In Josh Angrist's book Mostly Harmless Econometrics, he refers to the same phenomena as a "bad control." 

He gives the example of regressing wages on education and occupation. This is a horrible regression since it's identified off of differences in education within an occupation (e.g., if the occupation was lawyer, it'd be meaningless to analyze those education differences) and education is what enabled you to be a lawyer! 

I suspected over-matching bias may have been that. Perhaps an epidiomologist will come along to confirm. Thanks for the great Gelman links, I'll check them out.
good point. I've added a blurb and your quick-r link
Got this book and started working through it. It is very practical. In the first 18 pages you implement a complete (basic) recommendation engine.
@rolando, see "Cumulative curves" collection. If you get questions on that, don't hesitate to ask by email.
Are there answers to each question that are objectively right or wrong?
The exercise does not give a starting state, this seems weird to me. Maybe the professor corrected it during the exam, I must admit that even with a starting state I would have trouble having never seen such an exercise in class.
I see your point, and the material of the paper you linked certainly reminds me of the current problem formulation, however I don't think the relation is strong. At least I have not been able to find an answer to my question in the paper, and I don't think that partial observation framework (HMM-like or filtering) is suitable here.
@ClarkW.Griswold: my predict code was: predict(Model4, n.ahead=24)
For this question to be answerable, you need to know something about $Z|\\min(X,Y)\\le c$.
Glad to help. I just noticed that the process for finding the p-value was ambiguous. If it's below the median, subtract its _rank_ from 500, not the value itself.
Sorry @user603 but I am not totally satisfied with your comment. Let us say that the subset of h points contains an outlier. I still do not understand why the determinant will be large... thank you
I am not an expert on SAS or SEM, but if you get t-values, can't you calculate p-values yourself?
Thank you for these references, James. Here, however, the questioner is asking for *explanations* and *advice.* So that your answer can be relevant, would you be able to paraphrase specific "help, advice, or tips" in these references?
I'm not qualified to answer your final question (although it seems suspicious), but perhaps to address the Journals concerns you should also include a OLS model in your paper and show that it performs badly by some metric?
The negative binomial distribution defines a discrete random variable, so there's no curve or area under it; but it can take any non-negative integer value. Your data is clearly not sampled from a negative binomial distribution, & it's safe to say there's no brand-name distribution concentrating probability mass on 200, 306, 411 &c. Perhaps the data could be meaningfully re-expressed - a count of batches or something. Take a step back & explain where they came from & what you want to learn from them.
@Marco I don't understand how one could be any more explicit.  Do you mean restating what I wrote in terms of a formula such as $\\arctan(KL(P,Q))$ or $f(KL(P,Q))$ for $f:\\mathbb{R_+} \\to [0,C]$ with $x \\ge y$ implies $f(x) \\ge f(y)$ for all $x,y \\ge 0$?
Also what is "effect size" in a testing? Is it the test statistic and/or its observation?
Do you have just counts or some additional related time series without `NA` values? It seems that making learning period MSPE could be misleading since the sub-periods' are well described by linear tendencies, but in the missed period a drop down somewhere occurs, and it actually could be any point. Note also that since the forecasts are collinear in trend, their average will introduce two structural breaks instead of seemingly one.
MCMC started the Bayesian revolution of the 1990s with the rediscovery the Metropolis-Hastings algorithm/Gibbs Sampling.
Nice to know that there are ways, I really want to use these ranking information. Kindly let me know if you can remember the name of any of the methods. Thank you.
It sounds like you have unbalanced traning data set. Can you provide frequency of the miss and hit classes in training data?
Find this article in the meantime, The lasso method for variable selection in the cox model, from Tibshirani (Stat. Med. 1997 16: 385-395), http://j.mp/bw0mB9. HTH
Standard normal distribution stands for the case $\\mu=0$, $\\sigma=1$. Please, have a look at the [wikipedia entry](http://en.wikipedia.org/wiki/Normal_distribution).
This one is *much* older than xkcd, LOL!  Did you read J.M.'s contribution (August 9)?  Same joke, different animal.
It would be helpful if you showed code plus output. You can rename the variables.