tttjjj/elt_bench huggingface dataset:
${\\rm Prob}[\\epsilon^2 > \\epsilon] = {\\rm Prob}[\\epsilon^2 > 1] =$ 32%, which is not negligible.
Oh my goodness. Of course. I am a freaking idiot.
The K-S test has exactly the same problem you describe with the Gini coefficient: it "is a summary statistic that erases details of the distributions."  If you want to get beyond that, *draw a picture* of the distributions.  A good one would be a q-q plot.  (The K-S merely tells you the maximum deviation between that plot and a diagonal line.)
[Some precautions apply to the use of exploratory and confirmatory methods on the same data](http://stats.stackexchange.com/a/83545/32036). I'm not sure EDA is usually done visually. Probability models can be useful in exploratory analysis (e.g., parallel analysis). I'll say more in an answer of my own if I can find the time to finish what I've started so far...
I see you have added the `self-study` tag. Thank you for doing so. However, please read the [`self-study`tag wiki info](http://stats.stackexchange.com/tags/self-study/info) carefully. You'll see that you should not expect to get code; if you comply with the suggested format there, you may get something along the line of 'hints and pseudocode'. But that suggested format would include you showing your attempts at and explaining your difficulty with doing the mean square and coverage first.
I suppose it should be taken to mean _unrelated_ to avoid the statistical connotations. The example I remember being brought up was something along the lines of estimating a parameter from a set of observations (e.g. mean shoe size), and another parameter from a complete different set of observations (e.g. mean highway fuel efficiency).
+1.  It's also a truncated triangular and a truncated linear distribution.  (Linear distributions would naturally have a PDF of zero at one endpoint.)
@Glen_b, I believe I can compute these relationships that you mention. What I do have is a table that looks like: Name, Age, Income, Bought_Candy_Bar (Y/N). So, I could get more info from that on correlations/relations between the three "variables". Does that help solving my problem?
Welcome to the site. I fixed some of the spelling and grammar in your post, but it is still a bit unclear. Can you tell us what data you have and what you are trying to find out?
The answers you got to [your previous question](http://stats.stackexchange.com/questions/37399/weighted-geometric-mean-vs-weighted-mean) are general enough to address this one, too.
I would say this may belong here: The issue of (1) probability estimates from RF, (2) whether you can impose a cost function on the model or have to build it into the model, and (3) how to implement cost functions in RF are recurring issues that are not simply related to programming.
Assuming it has happened, the answer is 100%. It is incredibly unlikely that anyone would formulate such a set of conditions before making a particular observation, which is the only context in which a question about probability has a meaningful answer. Thus, a more relevant question would be "what are the chances that somebody who looks for numerical patterns in birth days and months would come up with some relationship among four of their kin that they find interesting?" That, too, is unanswerable, because we cannot determine what patterns would attract your attention.
**One-liner**: `DAG.random <- function(v, e) { A <- matrix(0,v,v);
A[lower.tri(A)] <- sample( c(rep(1,e),rep(0,v*(v-1)/2-e) ) ); A }`
I've tried to slightly reword your question. Please, check that I didn't alter its original meaning. Also, could you give more information on your dataset, in addition to @Peter's remark.
If data are dependent, this should be possible to account for that in the likelihood...
It depends on what the variables are.  Many people do "regular" FA on ordinal variables, although it may cause problems. See [Joreskog & Moustaki](http://www.ssicentral.com/lisrel/techdocs/orfiml.pdf). 
Also see [this thread](http://stats.stackexchange.com/questions/11899/factor-analysis-on-mixed-continuous-ordinal-nominal-data) here on CrossValidated.
the help in R is meant to explain you how to use the function. It's not meant to be a course on statistics. And regarding that, in general I believe the R help pages are among the most complete and handy from all open source packages I know of. And paying packages for that matter. SPSS and SAS give you a lot of mumbo-jumbo with half-truths and complete nonsens as a "guide for interpretation".
You might try clustering, perhaps hierarchical clustering, but I'm not sure statistical testing really answers the question you're interested in. You presumably don't think that across the four and three groups within those divisions the employees are perfectly homogenous, only that the groups more alike within the two main groups than across them. Presumably those levels (grades) are ordered categories on some measure; is that right? (That may affect your analysis.) What is it you want to show they're similar on? Is there some collection of variables you wish to compare?
@byoungb power is generally calculated before you run your experiment so that you can plan your study before you start it!  Now that you have clarified your actual question, you should start a new question!
It's just how stack exchange sites work, no big deal through! 

Read the FAQ
Hi @reqms, welcome. There is the [`R` package `MEWMA`](http://cran.r-project.org/web/packages/MEWMA/). It sounds as if it does exactly what you want. And the [package `MSQC`](http://cran.r-project.org/web/packages/MSQC/) also has a function called `mult.chart` with the option `type="mewma"` that seems to do the trick.
You might also try Statalist for this (http://www.stata.com/statalist/).
@Marco, you're welcome.
So I can centre the moderator without having to do the same for the independent variable (dummy)?
Also, if the wind data sets cover the same time frame and are from sites that are not all that far apart, you'll likely have some contemporaneous correlation.
"Let $X$ have probability mass function $p_j$" means you are *given* the probabilities $p_1, p_2, \\ldots, p_n, \\ldots$; you do not have to calculate them.
I don't know if it is ok to continue our discussion in the comment section, but until someone says otherwise... Yes, I understand your point about CV needs to be done on the outermost level and I am not objecting it. In fact, I just gave the same advice to my colleague yesterday. I am just pointing out that we are often not sure what we are cross-validating. At the same time I wonder if getting  evaluation on the model, rather than the modelling procedure, is what we actually want. A remedy might be one can think that he is using procedure error to estimate model error. Maybe this works.
Beautiful response @Procrastinator.  One other approach is the smearing estimator, which uses all the residuals off of the mean on the log scale to get $n$ predicted values on the original scale and simply averages them.  I'm less up to date on confidence intervals using this approach though, except for using the standard bootstrap percentile method.
If you are interested in the derivation, you can have a look here (https://onlinecourses.science.psu.edu/stat414/node/264)
I will not dispute that such a notation might be commonplace within a particular community, but my point still stands: it is not generally understood and may be subject to multiple different interpretations.  You will therefore reach far more people who can potentially answer your question,and avoid confusing people who are not part of that community, if you would explain what these numbers mean and how they have been obtained.
@Frank the classifier will bypass an existing algorithm which is computationally expensive.  It is an engineering problem (I can't give any more details).  I am getting accuracies which are in the 90s but they want a minimum of 98%.
See [here](http://i.stack.imgur.com/Xhilv.png). Also see [this](http://stats.stackexchange.com/questions/51061/does-the-normal-probability-plot-systematically-underestimate-the-mean) closely related question.
Hi Mark - when conducted under 100000 repetition of the null hypothesis, we do not expect to get this level of difference from 0.05.  Generally we would expect a difference of plus minus something like two times of sqrt(0.05*0.95/100000) from 0.05
@Michael, sorry, that was a bad choice of example, I guess. I am wondering about this in general. The particulars of dose-response relationships shouldn't get in the way. I edited the question to note this.
@Adam People have done the computations for actual birth frequencies and to account for the relative scarcity of 2/29.  It makes little difference.  After all, the chance that one of your 23 people has a 2/29 birthday is roughly only 23/365/4 = 1.6%, too small to appreciably change the chance of a birthday collision.
Yes, you are right. Ok, let's say that I want to extrapolate the cycle based on the past (and let's forget how the cycle is extracted in the first place, and even that it is a ``business`` cycle). What are possible ways to do this?
This syntax defines `Asym`, `xmid` and `scal` as unknowns to be estimated. When the data don't follow a logistic curve, the initial value heuristic (which you can see in the code of `SSlogis`) can fail. There is not much automatic processing that you can do in this case.
The difference between us is only terminology: to me there is no such thing as an interaction term on the intercept; that is just a main effect. Other than that, you are right: the null hypothesis should be that both the main effect and the interaction term are 0
curious that that book has no reviews on Amazon
presumably you could get something about geographic location as well, if you could get the appropriate matching data? You could also use information about popularity of first names over time (google "baby name wizard") to make inferences about age ...
So...22 binary independent variables and one continuous dependent variable? And you want to visualize the results graphically, not numerically as statistics? This probably needs clarification...
+1, lots of good advice here. However, I wonder if the OP's main issue is how to understand the nature of interactions. @user26934, I wrote an answer here: [interaction-in-generalized-linear-model](http://stats.stackexchange.com/questions/49002//49701#49701), that tries to explain interactions; you may find it helpful.
In sample size determination for clinical trials in the medical device and pharmaceutical industries the FDA requires 80% power. This is not a very high standard but failing to show significance is the manufacturer's risk and the FDA is mainly concerned with controlling the type I error because it represents the consumers risk of having an ineffective product approved.  The critical value for a test statistic depends on the sample size and the significance level not the power.  Given the sample size and the significance level you know the criticl value for the test statistic.
If `xxA` and `xxB` are standardized before calling `lm()`, it seems to work. E.g., `dat <- cbind.data.frame(dat, xxAstd=scale(dat$xxA), xxBstd=scale(dat$xxB)); lm.res.scale <- lm(out ~ xxAstd*xxBstd,data=dat); names(newdata) <- c("xxAstd","xxBstd"); newdata$Y <- predict(lm.res.scale,newdata)`. Not sure it's what you seek, though.
Does the question mention testing for a standard normal? Or just general normality?
@StasK: Probably yes, but I am looking for the argument demonstrating this.
I have good (non-statistical) reasons to plot the graph as it is: You directly see the answer to the research question. Furthermore, I am not looking for a error bars for inferential purposes as I know about the within-between problems. But, thanks to pinpointing me back to Mason & Loftus, I must have forgotten that they had a mixed example. I have to think on whether or not it serves my purpose.
I would say that your first try was overfitting on your data and was bad to generalize. And adding noise forced your model to generalize and find other parameters that fitted your data better.
@probabilityislogic I agree with your explanation. Tq for that. I'm using Poisson instead of binomial perhaps for the sake of studying the spatial pattern of "successes" in the original data.
I don't have the book on me, but "Measurement Theory and Practice" by D Hand discusses some issues that sound similar to this.  What in particular do you want to know about the "difference" in ranking.  For example, could you not first create an aggregate ranking for each group, and then take the rank correlation?
@Alexis, interesting note. Does the updated post look more comprehensive?
Is it possible for you describe more concretely what participants were required to do? I'm not quite following what is meant by an event for a step?
I know it means the value of the objective function is scaled. But$|r|$ also include the $w$ factor, So Why can it be scaled like that. Because I can't understand that in Mathematical view
Welcome to the site, Chris. Unfortunately, neither of these ad-hoc procedures are particularly defensible. Notice that in the first case you could not change the shape of your data at all and go from selecting *all* of them to selecting *none* of them (just by multiplying everything by a constant).
Assuming $y$ is binary, $y = b_0 +b_1x_1 + u$ is not a linear probability model (LPM), just a usual linear regression model. $P(y=1\\mid x_1) = b_0 +b_1x_1 + u$ is the specification for the LPM. Also $y = \\Phi(b_0 +b_1x_1 + u)$ is not a probit model - $P(y=1\\mid x_1) = \\Phi_u(b_0 +b_1x_1)$ is. Are these just typos of the question, or you actually estimated the specifications that you write?
Excellent. Thanks a lot for the code and for the visualization tips.
It will be giving you the predicted proportions and you should check this by multiplying the N by the predicted values.
I think this question should be left here; while it is about SAS code, answering the question requires knowledge of statistics. The SAS part is easy (delete or add a word), the statistics part is not.
Alright, thank you so much anyway.
Would it propably be sufficient, to generate a distinct list of all hashtags I have and count for each hashtag how often it occurs in a each user profile with any other hashtag ? It would result in 2 database tables:

id,hashtag,relationid

relationid,hashtag2,count
Your first suggestions worked great. Your second, I wasn't able to achieve what I wanted. Could you give me an example? Thx.
I would probably start with google scholar, and look for evaluation of learning algorithms for the particular ones you are interested in. Many of them are available in PDF form without an academic subscription
Judging from your question, the hyper-parameters of the SVM are known in this case ("generate samples *from* a support vector machine). So, you simply have to generate random data (x1,x2), classify them with your SVM and obtain y.
They are equivalent. The interpretation of $b$ in the first equation is the same as $-b$ in the second equation.
I don't have much background with MCMC but your last sentence doesn't seem oversimplifying. If you look at the effect of auto-correlations on your error estimates they change the value from $\\Delta A^²  = \\frac{\\text{Var} A}{N}$ to $\\Delta A^²  = \\frac{\\text{Var} A}{N}(1+2\\tau)$ where $\\tau$ is the autocorreltion time measured on the same $A$ observables. So it is like having only $\\frac{N}{1+2\\tau}$ 'effective measurement' instead of $N$. Is there still some oversimplification in this statement?
I plotted the integrand, i.e. the pdf of the beta distribution, in Stata - it has a builtin function for the pdf. For large alpha and beta you need to restrict the range of the plot to see it's close to normal.


If I was programming it myself i'd compute its logarithm then exponentiate at the end. That should help with the underflow problems. The beta function in the denominator is defined in terms of gamma functions, equivalent to factorials for integer alpha and beta, and many packages/libraries include lngamma() or lnfactorial() instead/as well as gamma() and factorial() functions.
It depends how the "organization" happened.  If it was at random then the two processes are equivalent.  If organization was on the basis of something that might be relevant (eg 10 clusters of similarly educated people, in a survey that's looking at income levels) then there is definitely less information in the clustered approach.
I strongly agree with this response, but I am also considering that there are some questions we cannot answer with case control studies. So perhaps we can only use the case-control study to inform our analysis of a randomized study design and we may not want to try draw generalizations from case-control a study design.
I could agree if I would have given an answer from my adviser about effect sizes, like you did. It's just some general thoughts about what I'd roughly expect. What striked me was that he just said that I shouldn't do a 2x2 if I didn't expect "good", and with that he means strong significant, results and that I should add more factors. Especially for the interaction effects. To me every result is useful whether it's significant or not. It says something about what you have found. Like you said; there's much more than significant levels. Unfortunately statistics courses usually don't cover that
libsvm is a good choice.
That is the interaction term. The formula is: $y = \\beta_0 + \\beta_1 ft + \\beta_2 temp + \\beta_3 ft*temp$ If you do not need it, replace the * with + in your formula statement.
@Azula The mistake is on your part: you forgot to divide by $\\sqrt{n}$. Those intervals have approximately the correct widths. user26715: it's impossible to tell what you're doing wrong without seeing the code or at least the formulas you think you're using. Please note, though--in response to your title--that indeed these CIs are *provably* correct. All that we're doing here is establishing that your code is *incorrect.* That's not likely to be of interest to anyone else unless some specific statistical question emerges.
For the record, these questions (about interpreting regression output) have been asked in [another thread here](http://stats.stackexchange.com/questions/16567/how-to-interpret-basic-output-from-a-regression-analysis), but the question was so badly formulated it did not gather any good replies.  This is a fundamental question that deserves a "canonical" answer that is elementary yet thorough, clear, and well-explained.
It seems like a lot of what you have will depend on how representative the photos are of the actual moth population in each photographer's area. Do the photographers actively seek out new species, or might they follow their whims to photograph "pretty" or "interesting" moths? Is this hobby widespread or is it big in some states and not others? Are the areas where the host trees grow more or less friendly to casual hikers (weather, elevation-wise)? That is: do the photos rise above a convenience sample or can you model "convenience"?
@whuber They are pre-determined, like mentioned `p=0.6` when attacking and `p=0.7` when defending. I now understand that from the way I give the example it might seem like it depends on troops, but it was just posed like this for illustrative purposes.
Your work looks fine.  It can even be performed in reverse to show that minimizing the $L^1$ norm of the residuals is *uniquely* associated with the Laplace family of distributions in maximum likelihood models.
The statement "hypothesis A is proven true..." is not something statistics can be much help with *in absolute terms*.  Statistics is much more useful at *comparing* different hypothesis, it would be better put in terms of "the data supports hypothesis B compared to hypothesis A" and other comparative statements.
Dear Emil: I am afraid you failed to understand the questions I asked. Your example illustrates the most common definition of "fixed vs random" that I myself provided in my question. Anyway, using your example, can you tell me why it is a bad idea to draw inference about future response from a model where the lot number is a fixed factor?
If you mean the column headed "Value" as returned by a call to `summary`, then it is *a priori* a regression coefficient (for your survival model). Its interpretation depends on the fitted distribution.
The discussion is about definitions. Clearly, if the covariance matrix by definition is required to be non-singular *Macro* provides an example, but this is not a example according to the more liberal definition that @cardinal refers too. One good reason to prefer the more liberal definition is that then *all* linear transformations of normal variables are normal. In particular, in linear regression with normal errors the residuals have a joint normal distribution but the covariance matrix is singular.
Ok thank you very much. I will mark this as resolved and open a new thread asking for a suggested methodology.
Doesn't the function rcopula in the same package do that?  You have to look under the function "Copula" in the documentation.
Velcome to our site!
... And yes, some points of the model validation can be answerd without splitting the data in distinct test and training cases (e.g. model stability in terms of the coefficients). But already model stability wrt. to the predictions should be measured using unknown patients (unknown: never appeared in the process of building the model including any data-driven pre-processing that takes all cases into account). Actually, for a traditional quantitation in chemometrics, the validation has steps that need further independently measured test data: ...
(+1) ... if all groups have the same no. observations.
thank you! The book is really excellent btw.
Chi-square is a lousy way to test for normality, by the way. Use a normal probability plot or (if a $P$-value is something you feel you really need) a dedicated test such as Shapiro-Wilk or Doornik-Hansen.
The intent of your question is not completely clear to me. You define the density of $C$ over the same variable you're taking the integral over. Did you mean to integrate the density with respect to $dx$? If on the other hand you're integrating the random variable rather than its density, is $C$ a constant (but random) value over the range of $x$, or is it a different, independent, uniform value *at every value of $x$ in $(a,b)$*? Some context might help.
Also, the subtitles of the figures say "community: anime" and "community: astrology", which seems to imply that these plots come from different data sets...
Thanks man, but that didn't do it for me. Still haven't found a solution to this.
@Firefeather Don't you mean to say "**do** use" rather than "don't use"??
I could accept your answer if you don't mind to include your last comment and elaborate it more (suggested approach, etc.?). Data can be seen as clustered indeed and I've also tried a multiclass trasformation that I posted [here](http://www.kaggle.com/c/higgs-boson/forums/t/9560/binary-or-multiclass-classification/50395#post50395) but in this case the results are badly worse than those of the binary classification: I arrived to the conclusion that the advantage of binary classification is the feature of the probability confidence scores (based on which a final ranking threshold can be applied)
It's preferable if you ask a question about the concepts that are giving you trouble rather than just pasting in the assignment.
The online [LINEST help page](http://office.microsoft.com/en-us/excel-help/linest-function-HP010069838.aspx) explains how to extract this information (and more).  For instance, the standard error of the slope will be in row 2, column 1 of the LINEST output.  Could you please edit your question to tell us *how* you want to "take these into account"?
@Cardinal Criticizing the statistical methods on that site would be like shooting turtles in a barrel: too easy, messy, time-consuming, and with little value.  Let them have their fun...
Look up 'density estimation'.