tttjjj/elt_bench huggingface dataset:
If this question is only about how to do it with MATLAB, it will be better suited for Stack Overflow, provided you add a minimal reproducible example. On the contrary, if you have a statistical question related to the problem at hand, please clarify by editing your post.
@Andy W I thought if I switch the reference with the included binary it will probably turn around the result, to a positive number, showing the increase between the "before reference" to the "before independent". however I am not quite sure if its not too easy just to interpret the coefficient as an percental change, I should probably look into the formulas again - any ideas, whether this interpreation makes sense?
Excellent, thank you very much for this. The weekly data was averaged up from daily samples. How does one discover the radius of convergence?
@Bernd: Thank you for your interest to help me. I am sorry I was not clear and wasted your time. I am trying to calculate Orwin's fail-safe N. The treatment is inexpensive but has saved thousands of lives already.I am trying to decide on the best values to use for the criterion for a trivial log odd's ratio and mean log odds ratio in missing studies to calculate the Number of missing studies needed to bring down odds ratio (because Classic fail-safe method yielded 1014 studies). I am using Comprehensive meta-analysis v2.2
Short answer is no. Think of it this way. PCA works on either the correlation or the covariance matrix. It never sees the means; they were subtracted out in calculating that matrix. So, if a variable with high mean (or low mean) is highly correlated with PC1, all that probably implies is that it reflects a bundle of relatively highly correlated variables. PCA pays no attention to means as such.
I am working with 14 variables, with n = 10 for each correlation. Shouldn't I be able to tell which variables are redundant by looking at the correlation coefficients? And if so, is it possible for two vbls to be redundant with r = 0.8 or less? I thought r would have to be high (>0.9) for them to be redundant.
Thanks Frank, can you give me reference to the article that states " validation is highly unstable if n<20000 "....Thanks Frank.
I would recommend anything by Agresti. He's very high on the clarity/sophistication index. That is, at any given level of mathematical sophistication, Agresti writes clearly, compared to others.
I was just looking at that article; I think maybe it needs to be cited more.  I hate flushing effects down the drain because advanced multiple comparison techniques aren't well known or easy to do.  In contrast, a lmer approach is drop dead simple.  I wonder whether there are serious problems with it that need to be considered.
Classification mainly
I have actually read your article prior to posing my question. I am still confused with regards to the equation where lost sales are calculated:                                           % cannibalization of year to date sales = [(EXP(B1)-1) x (year-to-date sales of B)] / (year-to-date sales of A); where B1 = coefficient from log-log regression..is this correct?
I wonder if quarterly data is right. Why won't you try with t-2, it equals the lag noticed in the authors study.
you may use time series analysis with missing data
+1, great answer.  Looks like I'm going to have to learn R yet.
Have you at least put an effort to solve this problem?
This could be a great question if you ask not for the code but for the maths and the model behind calculation of factor scores in IRT!
If you use R syntax for multiplying vector with matrix, then yes the dimensions are ok. Mathematically, $f(v)$ is the column vector and $A$ is the matrix, so you cannot multiply them in the usual matrix multiplication sense.
@Alexkeil yeah, I'm using a log of length. since my main goal is prediction qualiyy I use different measures like RMSE, MAE, few correlation coefficients
I see, thanks for the explanation, if you have links on examples of how to apply substantive knowledge to determine if a multiple regression variable is worth it to add, please let me know.
**Hint**: Delta method. (Also, look up *variance-stabilizing transforms*, which is part of the motivation.)
May be you can get better answer from [SO](http://stackoverflow.com/) on your second issue
Googling `SPSS Modeler Algorithms` returns version 15 pdf document. C5.0 is found there, but I don't know if it is with boosting option.
A similar reference from ecology: Schielzeth, Holger, and Wolfgang Forstmeier. 2009. “Conclusions Beyond Support: Overconfident Estimates in Mixed Models.” Behavioral Ecology 20 (2) (March 1): 416–420. doi:10.1093/beheco/arn145. http://beheco.oxfordjournals.org/content/20/2/416.
Here is the response I got from my professor: "'Likely high value customer' is not a well defined variable. It is a fuzzy concept which means different things to different people. Unless you quantify what 'likely' means and what 'high value' means, it can not be used in any quantitative analysis. You can define your response to be 'customers with annual revenue exceeding $1,000'. You need to ask yourself the question: if I send this to a team in India, will they define 'likely high value customer' as was intended." Turns out it wasn't a question of statistics, but of methodology.
By "weights" are you referring to coefficients of your independent (explanatory) variables? Not sure what you mean by "attributes". Does your binary regression feature only binary independent variables? An example of your input data and your regression results would help to specify what you want to do.
I don't quite understand your answer, maybe due to terseness. For example, consider the case where the *true* intercept is zero! :)
Big data is not the culprit.  It is how it is used.  When it has information and is applied appropriately it can be very helpful.  Data mining is not all bad.
Breinman's recommendations are to pick a large number of trees, as well as the square root of the number of variables for **m**. See https://cwiki.apache.org/MAHOUT/random-forests.html
and
http://www.springerlink.com/index/U0P06167N6173512.pdf
Indeed. The degree option does not actually do anything. I'll take it out. `rd` always implements local linear regression. If you want higher order polynomials, try `rdrobust lne d, c(0) p(2) kernel(uniform) h(0.2)`.
Since you have not stated what $\\phi(t)$ is, your first displayed equation is, in effect, a _definition_ of $\\phi(t)$, and there is nothing to _show._  Have you been asked to prove that $\\phi(t)$ as _defined_ above is a valid characteristic function corresponding to the distribution of some random variable? If so, think of _mixture_ distributions.
Thanks! I will give this a go, although I have been experimenting with zero/one inflated beta regression too.
@DimitrisRizopoulos The adaptive Metropolis withing Gibbs I mentioned uses a finite mixture of Gaussian distributions as a proposal distribution (as stated in the technical report I posted). If you use the hardcore Metropolis, then you are asking for an answer to the "million dollar question", for which there is no general solution. Typically you have to play with different proposals and different acceptance rates. Very good book, by the way.
I can't really say whether either method is better or not: I suggested this based on what I have read in stats books /sites. One thing (I think) the method adds is that allows you to adjust the 'fences' to whatever you feel is suitable (adjusting the 1.5 factor) ; where just taking the middle third might be more susceptible to including a potentially high outlier (which arguably could be better classified outside of the fence).....but I cant' say that for sure. One other things about using Quartiles, is that the 5-number summary is quite well understood and commonly used.
@MarcClaessen: Right, I am just clarifying for the benefit of OP (whose question sounds like he is quite confused). Note that he seems to be using linear SVM, in which case there is probably only one parameter ($C$) if I am not mistaken. I think your answer is the best here, because others go into unnecessary digressions into what OP is *not* asking (+1).
Glad I could help. If you think thant I answered your question adequately, could you please accept it? Thank you
commoncrawl? http://www.commoncrawl.org/
This appears to be a question just about how to do something in some (unspecified) software package, and not about statistics per se. http://stats.stackexchange.com/help/on-topic
Thank you for the very clear and informative answer! 
Regarding the mentioned paper; in the pdf-version there is a legend under the table you linked to, stating that "values are means +/- SE with ranges in parentheses", so that's what sparked my question.
what do you mean by 'merge' exactly?
Thank you for your reply. I will use the DISTLM rotine followed by dbRDA ordinations, which, in fact, seems more appropriate.
Ordinal regression would be my starting point.
A side-issue here is that using bars is likely to prove confusing. Trivially, downward bars are a bit harder work than upward bars. More fundamentally, bars starting at 1e-3 is arbitrary. More positively, showing point estimates by point symbols and adding error bars would be a lot simpler than showing bars plus error bars. Google "dynamite plot" for more.
I'm not clear what you mean by "live" here. Are you talking about testing data on-line as they come in? (Eg, you want to know when to stop collecting data b/c whether or not you have an effect is clear.) NB, if you have a number per unit (eg, 3 per mile), that is *not* a "proportion", but a **rate**. This should be distributed as a Poisson (likely w/ *overdispersion*), and you would need to include the unit as an *offset* if it varies over your data. Also, I suspect part of your issue is that you have *spacial autocorrelations* if these are counts in contiguous / adjacent locations.
(+1) One of your remarks is somewhat misleading, because the densest 50% region necessarily includes *a* median. The only way "the" median could not be included is when there are an even number of data values and either the upper or lower half is the densest 50%.  But this occurs only because "the" median is conventionally defined as midway between the two middle values. So perhaps a more accurate and useful conclusion would be that a densest 50% region of a dataset always includes a median of the data.
+1 One way to provide intuition concerning your last remark is to point out that under the null hypothesis (of equal breakage rates), we may view the distribution of the three broken samples among the two rows (materials) as being a matter of pure chance, as if a fair coin had been flipped for each breakage to determine which row it would be assigned to. We would see *some* imbalance in the assignments (because $3$ is odd) and even the most extreme imbalance where all breakages occur for a single material would have a chance of $1/8+1/8=1/4$, too big to be considered "significant."
To do power analysis you need to specify the effect size - if $\\frac{b_3}{b_4} = \\frac{1}{2} + 10^{-20}$ you will have much lower power than if  $\\frac{b_3}{b_4} = 5000$. So, you need to decide what is a "small" and "large" effect size in the context of the application, simulate data under those circumstances, then empirically look at how often you reject $H_0$. (btw, I think you meant proportion of $p$-values **less than** $.05$:))
@cardinal - touchee.  Although from the question as posed, we aren't sure what *question* @user3614 is trying to answer.  p-values give the answer to "how likely is the data, given the effects"? - the Bayesian answer I described gives the answer to "how likely is the true value of the effect sizes, given the data I have?".  I  would have a guess that the second answer is more appropriate, requiring a Bayesian approach.
Welcome to the site. Could you  spell out `LDA`? Acronyms vary and, in addition, not everyone who reads this site has English as their first language.
This unfortunately is incorrect.  The duplicate question shows how to combine the SDs into a pooled SD, because their *squares* are variances and that thread concerns combining variances.
Or google tries to help me and thinks I'm looking for tumours because that's what I do more often... "probability near score L1" got me to the paper below.
[This](http://stackoverflow.com/questions/2709753/students-t-distribution-in-javascript-for-google-spreadsheet) may be of interest to you.
Does the question require you to calculate *two standard deviations of the data out from the mean of the data*? Or does it require you to give a *confidence interval for the mean*? Although the confidence interval for the mean involves standard deviations, these are two very different calculations. The second one makes more sense for a stat exam to me, and it is much less likely to go beneath zero. If you do need to calculate the confidence interval for the mean, you will need to calculate the standard error of the mean and use that instead of the population standard deviation.
A recent related discussion: http://stats.stackexchange.com/questions/27589/why-would-someone-use-a-bayesian-approach-with-a-noninformative-improper-prior/27593#27593
Sorry I was not at home.
I'm actually not allowed to post images but the images are similar to those found here [link](http://www.plant-image-analysis.org/software/leaf-color-segmentation)
The question should appear in the text. And the answer is yes as the OP should realise by computing the expectation and using the linearity of this operator.
I just realized that the format of my answer is horrible. Msg me if you want me to send you a better formatted version.
Thank you again! That was a huge help for me.
@Anony-Mousse What you told about recommender systsms or computer vision is not general. Also recommender systems are not supervised necessarily.
Correlation and a strong underlying reason for a link suggest causation until proven otherwise is probably the best you can get.
Can you post the code (or similar example) that you used to get the output?
Did you read the hint by @ttnphns?  PCA pays attention not just to the eigenvectors (which you discuss) but also to the eigenvalues (which you ignore).  This is critical, because the eigenvalues are used to determine which components to retain and which ones to drop.
This question appears to be off-topic because it is about how to use R.
I remember I read sometimes this: In the time when people published books with tables of "random numbers" to be used for simulation, experimentation etc, some of this had marked parts of the tables as suitable for use in small simulations (exhibiting "local randomness") and other parts which should only be used for larger simulations (exhibiting "global randomness") So the concepts seem to point to something valuable!
@Mark: Just google "standardised residuals" and look at the first few hits.
No it does not seem to be working as well as a good book.  To paraphrase Stephen Senn, it is odd that statistics is the only science the people demand be understandable at first look.
@ gung, you are right, I gave some information from the context that this issue is set. Q is just the name of the variable, any similarity with an actual term is due to coincidence. What do you meand "if I have no covariates"? I have two covariates P and Q and their interaction. Did I understand your question?
@whuber Thanks. Now that I know where to look I can see that "quantile rank" is also [used](https://scholar.vt.edu/access/content/group/43c8db00-e78f-4dcd-826c-ac236fb59e24/common/tables/index.html).
@cardinal: Yes, probably. How does this changes the solution?
@IcebergSlim Well besides [being tricky for standard analyses](http://stats.stackexchange.com/questions/13314/is-r2-useful-or-dangerous), it isnt' really that telling/useful in your case since you aren't automatically doing better than the mean. With that said, it still has meaning as a way to compare yourself to the mean of the dependent variable.
And that puts too much emphasis and importance on the month threshold which, looking back, does not seem to be appropriate. I would rather look out for outliers with an intervention detection, and, if one of these corresponds with the "supposed" date of intervention, that would be good, but this doesn't seem to be much the case in practice - these are insights from the in-field team that make me willing to change my approach a bit...
(+1) Do you mean that by "repeated measurements" Montgomery did not mean "repeated measures"? I think it's exactly what you mean, and I agree, but I find that your wording could be a bit more explicit about that.
With regards to data cleaning I am pretty sure I cannot do anything else to my IV's than I already did. It comes down to my DV which is a logit transformation of the proportion of event within a given month. I am thinking that I simply just might remove anything that is more than two-three standard deviations away from the mean, but I am a bit unsure if that is simply just it...?

Hmm, so it is not reasonable to look at "January" as a predictor of an event, or whether a given year had significant influence on my DV?
Jeromy, your blog is a wonderful resource.
I think so. If that wasn't sound a statistician should wish to die young or quit profession soon, in order to escape multiplying type I error in his life course.
As @GiorgioSpedicato suggests,  dynamic time warping (DTW) is the way to go. There is a very nice example in the link suggested by Giorgio Spedicato that clearly explains how to cluster time series, then you could apply any forecasting method in R forecast Package. It looks like you are using weekly data, you could use stlf function in R forecast package which handles high frequency time series.
You haven't mentioned a statistically important issue: the counts within separate bands are likely to be independent (and heteroscedastic) whereas the cumulative counts are strongly interdependent. Thus, any method of fitting the cumulative counts should be able to handle that dependence as well as the heteroscedasticity that is buried within it.
@Eupraxis1981 It may be possible to make a relatively strong claim that the treatment did not **decrease** the scores. This information could be useful in helping to decide whether to continue the intervention or not.
@Chiffa I used SVD instead of baseline predictors at the end
That's nice, but I'd use a dotplot like @Peter suggests above. Here, each group has a different scale and it's difficult to see exactly how two points on the same line compare. To identify the two groups, I'd use different colors or symbols.
The [effects](http://cran.r-project.org/web/packages/effects/index.html) package is from John Fox :-) But F. Harrell's `Hmisc`+`rms` is a must for regression (model and display).
From my present topics that generated with LDA(in R),i wish to generate automaticlly new topic for new documents
Cross-posted to StackOverflow: http://stackoverflow.com/questions/13768019/how-do-i-get-the-p-value-when-using-javas-olsmultiplelinearregression
@Pete as Michelle noted, the items with the highest mean scores will (at least in rasch and one parameter models) have the lowest estimated difficulty. If you want a good measure though, you'll need to keep some of the difficult ones too.
Alright, so, as far as I understand I mixed up the terminology here. Which is not surprising since I'm a beginner. I adapted the title and content of the question. Would you mind checking if it's better now or did I still miss something?
I don't see how another model being a GMM model conflicts with this. Can't you do `hausman` or something like that for two `gmm` models?
Good point. I read it a bit quickly. I guess it depends on what the OP means by "not through randomisation". I've asked for clarification.
(+1) Maximum likelihood estimation in the gamma case is doable; for large sample sizes, the much simpler method of moments should work reasonably well.
Ah, I see now you even mentioned it, sorry.
Also, I don't think there's going to be much you can do to "infer the distribution" of data above 150 when there are no observations that large.
why do you put [var(X)+E(X)2]⋅[var(Y)+E(Y)2] instead of E(X2)E(Y2)???
Citations would be of great help for this issue
Very usefull comment. Thanks for that! Nice idea by the way.
Certainly easy to make a virtual machine representation of Bayesian analysis given the prior is proper so that the possible unknowns (e.g. parameters) can be sampled and then the possible knowns (e.g. data) can be sampled given the unknowns (e.g. parameters) sampled. Stigler points out that it is very general albiet often computationally not feasible yet. 
Fig 5 Stigler 2010 here http://stats.stackexchange.com/questions/39177/periods-in-history-of-statistics/39193#39193
@John Hello, yes I believe it is categorical.  The choice was "left", "middle", or "right".  Was I wrong to use 1,2,3 as an example?
[Marginality principle](http://en.wikipedia.org/wiki/Principle_of_marginality)
?aov for an example of type-I or ?anova for example of type-II & -III.
Thanks for the reply Kyle. I will reread my original statistics textbook on ANOVA. If memory serves me correctly, it was written on papyrus by monks.
Carlos, thanks for the response. Clusters indeed made it difficult to see patterns.
Is there any axiom which is used to justify entropy based inference which has connection with this Grouping property?
For the sake of understanding the connection between this reply and the question, please note that in the question "mean" refers to the average of a set of data whereas in this reply "mean" refers to a parameter of a hypothetical distribution.
The application I had in mind is somewhat different but this looks promising. Thanks!
[Don't cross-post](http://meta.stackexchange.com/q/64068/150510), please. If you don't get a response on Stack Overflow, just ask a moderator for migrating your question here.
In what context? For what purpose? With what kind of SVM? For what kind of data?
@Soakley That's not a sample; that's a subset. For any useful purposes one would want to make inferences about *potential* bomb-blasts; once those are appropriately characterized, they would form the "population" (which would better be thought of as a *process* in space and time, IMHO). The observed incidents are the cases in the sample and fatality is an *attribute* (aka "variable") of the cases.
I wonder what the person who downvoted my answer found objectionable or incorrect in what I said.
can you add more detail in the answer? I know what rattle is, but the OP may not, and links can often be fleeting.
Thank you for your reply. I would be grateful if you could also explain what is meant by parametric and nonparametric in simpler intuitive terms. Thank you
@richiemorrisroe: higher-order factor analysis is done on the basis of **correlated** lower-order factors. As a sidenote, I wouldn't count the fact that some method is routinely used in some psychological disciplines as evidence for its usefulness.
(a) if test A has lower power than test B under some assumed circumstance, that doesn't imply that the cases that A will reject are a subset of those in B (in fact that's not possible at a fixed significance level); they reject different (but heavily overlapping) portions of the sample space; there are always cases each will reject that the other will not. (b) If the assumptions of the parametric test are not met (are they ever actually met?), then parametric tests can have relatively poor power (e.g. t-test vs Mann-Whitney under heavy tails)
I really have no idea. Maybe it is related to this question I answered a while ago: http://stats.stackexchange.com/questions/77904/are-there-any-known-methods-for-picking-out-the-easy-cases-for-classification/77906#77906
How did you compute the densities you used to color the points?
And there is a difference between differencing and detrending.
@AndreSilva Thank you; by far my biggest problem is not knowing the vocabulary, which makes searching for answers difficult!