tttjjj/elt_bench huggingface dataset:
+1. Good to see an answer that clearly shows why one has to take care, and in particular be explicit, when defining events and their probabilities.
Thank you for your good explanation. Just to test it out, I deleted one of the model specifications and it did increase the degrees of freedom and lower the model fit statistics. Good lesson, thanks for your help!
Thank you so much Glen! You made me learn a lesson here: I should always do the rounding at the final stages to get better and more accurate results! Thanks again!
Actually, as you can see in my data file, I made the observation that each time I run my experiment, my six values are ~ about the same. "Same" for me means 0.2~0.4~0.56 as 10~14~8. May be the same range or order of magnitude. May be I should compare CVs across lines after all. In fact all I need is to express my observation statistically :)
(1) What does a q-q plot show?  (2) What is the reason for testing a Gaussian distribution, anyway? Very few (if any at all) statistical tests or models require data to be so close to Gaussian that they will pass a Goodness of Fit test using 700 iid values.
@Sharon I'm a little bit puzzled about the term 'randomization test' (the only paper I found using this keyword, http://j.mp/9OCB9I, seems rather to focus on power considerations) -- I rest on my idea that you could use cross-validation to assess the classification accuracy in your data, e.g. leave-one-out or k-fold CV for this purpose (available in `kknn` and `class` packages). Now, I don't know on which basis your classes were defined, but it is generally meaningless to use ANOVA to test if the groups differ on the variables that were used to define class membership.
Ah, yes, flagging for deletion  is fine. But if you have no answers there you should probably be able to delete it yourself.
$X$ is a matrix with 10000 rows and 1024 columns. $y$ is a factor vector of length 10000 with three different values. What did you mean by guessing that $y$ has only two columns?
Although I can glean a vague sense of what you are trying to do from this post, I cannot really understand what it is saying.  I do not understand the sense in which "variables" are "sampled," nor is it clear what a "note" is or what a "question type" refers to.  Could you explain these things and make it plain to us, in non-technical language, what you are trying to accomplish?
Welcome to the site! Can you tell us some more about your data and the sorts of comparisons you'd like to make? The more information you provide, the better advice we'll be able to give!
You seem to be confusing vectors, random variables, and paired observations. Mathematically they may look similar, but they function quite differently. In this case it seems you intend your "vectors" really to be paired observations, whence your "$p$" is really an *observed frequency,* not a probability. But because you have *paired* data, you can just as readily estimate the joint density: it's the empirical joint density of $(x,y)$.
Nice answer jbowman, you should also note that Random Forests works by menas of averaging the prediction of many diverse models (CART). Think the jensen inequality: you have the MSE which can be decomposed in its variance and bias parts. A Random Forest always inherits the bias of its trees. It improves accuracy by reducing variance. People are quite worried about reducing bias in Random Forest you can google it. The R implementation actually has its own bias correction but I neither know how it works nor have I seen a substantial improvement when I have used it.
Perhaps you should explain what you mean by “reliability”. If it's the standard error associated with some sample estimate (say the mean score on a scale, the proportion of people who answer this or that, etc.) then your idea is perhaps related to the “bootstrap” (but not quite the same). In any case, it seems to me that splitting the sample randomly makes more sense. I don't really see how differences between demographic groups could be related to “reliability”. If, say, men and women answer differently then it could mean that they are different, not necessarily that the survey is flawed.
Fisher transformation is atanh, nor arctan.
+1 You managed to improve enough on my answer for the OP to give you the check mark.@DilipSarwate.
Look up censoring. Your data will be right-censored. Yes, there are methods that address this. Maximum likelihood in particular can be modified to handle your scenario.
Technically, the minimum sample size for estimating any parameter is *zero* ;-)
Well, I would argue that - while it might be a syntactically correct English - semantically we still have an issue. You appear to be trying to say 'integrate from $-\\infty$ to $\\infty$', but I don't think 'integrate [...] infinitely' clearly suggests that. Perhaps 'integrate [...] over the real line' or 'over the whole line' or some such?
Your question isn't clear to me. Which things are you asking about the validity of? What is the aim of the analysis (i.e. what practical questions are you trying to answer)?
Off the top of my head - `qf()` is giving you the quantiles of an F distribution where the variances are EQUAL. The confidence interval in `var.test()` is drawing a confidence region around the estimated variance ratio. I imagine that uses a non-central F distribution, but have no idea how you derive the non-centrality parameter.
Very helpful, thanks! Sadly this probably won't work in $>1$ dimensions, since it amounts to trying to write $f(\\Sigma)$ as a convolution with functions of rank 1 matrices; i.e., if $f(\\Sigma)=\\int_{\\mathbb{R}^d} g(x)\\exp\\left(-\\frac{1}{2} x^T \\Sigma^{-1} x\\right) dx$, assuming $g$ is symmetric, for some $g_0$ we can rewrite $f(\\Sigma)=\\int_{\\mathbb{R}^d} g_0(xx^T)\\exp\\left(-\\frac{1}{2} \\mathrm{trace}(\\Sigma^{-1} xx^T)\\right) dx=\\int_{\\Omega} g_0(X)h(\\Sigma^{-1}X) dX$ where $\\Omega$ is the set of symmetric rank 1 matrices, and $h(Z)=\\exp(-\\mathrm{trace}(Z)/2)$. That seems impossible...
Computing the SEs in this way gives you standard errors that are consistent only if the null hypothesis is true. In general, equations that give you proper SEs without having to assume the null are quite a bit more complicated. See the recent article by Yuan and Chan (2011) in Psychometrika (http://goo.gl/15YahG).
$X$ is just combination of the LSE estimators of each dimension. It is not the LSE under the squared error in the multivariate case, is it?
Sorry if I was unclear, but right now the issue isn't the color scheme itself. I did however try RColorBrewer earlier today: a great little pacakge! It is spatial data, I'm not sure I want to use Google APIs, but I might have a look.
I think you are trying to work on a problem that is too difficult. If you have a single covariate and a single response finding a breakpoint in a line may not be too difficult but with 196 different variables allowed to change how could you even picture the shape of the response in such a high-dimensional space much less identify breakpoints!
Ordinal IV in modeling/regression is always a problem and you have to choose either to treat it as interval or nominal. But: in your case you have only one IV, and it may solve the problem. Why not to see the IV as the DV and the 4 DVs as IVs, - and use ordinal regression?
John Kruschke just produced two videos where he compares Bayesian and standard statistical methods. He has many examples where the Bayesian method rejects but the standard method doesn't. Maybe not exactly what you were looking for, but anyway... http://youtu.be/YyohWpjl6KU and http://youtu.be/IhlSD-lIQ_Y .
However, I have been trying to change my x-axis as it gives negative values to my covariate that does not take negative values. I have tried to set up the effect using xlevels=list(Cov1=c(0,0.1,0.2,0.3,0.4,0.5)) but it does not change my graph. Do you know how to modify that?
"Nonlinear" pertains to a *relationship* between 2 (or more) variables; saying that a single variable is nonlinear is meaningless.
What's the difficulty? Are you concerned about the curved lines in the residual plot? That's a natural consequence of the discrete data and the model you're fitting.
You condition on events rather than probabilities. In this case the *event* you condition on is an observed proportion, not a probability.
What is your definition of "analytic form"? (What people "accept" as closed-form solutions varies, which is why I ask.) With things like modified Bessel functions (which I wouldn't consider a "closed-form expression") stuck in the denominator, it seems like there may be little hope.
thanks! sorry for the delay
Re the update: *Mathematica* 8 returns 1262 for $p=50$ and 1290 for $p=60$.  Re Normal approximation (@Proc): this cannot be expected to work well in the tails, which is crucial to the calculation.
@AndyW Yes. It all comes down to if we are more likely to have a protractor or a ruler in hands.
A PDF by definition has to satisfy that its integral over all possible x is 1. The intuition is that of all possible events, something has to happen. Now, when you integrate e^-lambda (without the lambda multiplied beforehand) over 0 to infinity (which are all possible outcomes), you will easily find that it evaluates to 1/lambda. If you then multiply by lambda, you get the desired area (=probability) of 1.
"Rearranging" $\\hat\\mu$ does not seem relevant: you need to compute $v(\\hat\\mu)$ (not $v(\\mu)$, which is zero) and $I_1(\\mu)$ in order to test the inequality. How far do you get when you do this?
I often find that simple graphics convey much more information that a (necessarily low-dimensional) measure of effect size. For instance, I find `plot(tab)` already quite enlightening, although something else may, of course, be more appropriate to your research question.
And life's more fun without them? Guess you can only take a metaphor so far...
While you headed and tagged this question as related to propensity scores, your two questions look at odd to me: Binary classifiers are one thing, the use of [tag:propensity-scores] with binary outcome is another topic. Are you asking whether logistic regression is better than, say, neural networks in deriving PSs?
$dy/dx$ refers to the slope of the curve. Calculus tells us the slope of $y=ax^b$ equals $(ba)x^{b-1}$. Obtain your threshold like this: `f <- function(x) coeff[1]*x ^ coeff[2];
fp <- function(x) coeff[1] * coeff[2] * x^(coeff[2]-1);
secant <- (f(max(x)) - f(min(x))) / (max(x) - min(x));
threshold <- uniroot(function(x) fp(x) - secant, c(0,1))$root`. It equals $0.4219369.$ How are the $x$ values in the data chosen? I don't believe the maximum value of $x=1$ is random, but the minimum value of $x=.095$ looks like a measurement. To what extent could these extreme values vary among experiments?
+1 for conveniently using the answer form to add to my question, but I must be brief here. Your auto.arima algo puts the two ES models i.e. ARIMA(021) and ARIMA(022) among the three best models (using c=2). plot(f8<-forecast(m8<-auto.arima(usage,seasonal=FALSE,trace=TRUE,approximation=FALSE), h = 168)). I would expect the algorithm behind ets to highlight the ets(AAN) model 'cause linear ets models are all special cases of ARIMA models. Note the ARIMA(1,1,2) [equivalent to EST(AAdN)] model for which damped forecast is optimal has not been placed among the best by your auto.arima algorithm...
It would help to explain your problem in English and mathematical terms, rather than wholly as an `R` program. There are several reasons for this, including (1) the program might not be doing what you intend, (2) it would help readers avoid having to work through code to understand the question, and (3) in the course of having to explain the circumstances clearly, you might arrive at a solution yourself.
@Blain, frankly, it looks like you are using a poor software implementation and dubious advice. I don't see any reason why a properly coded hot deck procedure should leave any missing values unfilled. I am not sure I am even getting your point 2 on using three DVs (which I assume are dependent variables), and what the best relation of the procedure to regression analysis would be. Generally, I would advise to move on to multiple imputation, and listen to Stef van Burren's advice on it (who wrote a book or two on this).
You might find it useful to know that the [chi-squared distribution](http://en.wikipedia.org/wiki/Chi-squared_distribution) is a Gamma distribution.
Thanks, I realize now that I mistakenly labeled my response variable as a fixed effect. Whoops! I understand that the residuals need to be normally distributed, but I find myself a bit lost by your other explanations and comments (sorry! my brain just doesn't seem to work well with this stuff.) But essentially, my model in R would be something like:
result <- lme(responsevar ~ treatment*sex, random = ~1 | SubjectID)
Does that sound right? Thanks again!
I don't either.  State your objective and then we can figure this out. It may be that you want to see if the distribution of subject responses varies as the values of the factors change.
This a thinly disguised version of a part of [this other question of yours](http://stats.stackexchange.com/q/103384/2592). Please stop posting duplicate questions.
@gung is more than likely spot on, but the solution is probably **not** to delve into objects and rip out whatever you feel but to learn to use *extractor* functions, in this case `coefficients()` or its shorter alias `coef()`, as in `coef(fit)`
@Michael You may want to look at some earlier discussions on whether to treat [Likert scales as continuous or discrete](http://stats.stackexchange.com/q/10/930). Variable scoring sometimes reflect the way we envision the data. I can also recommend De Boeck, P., Wilson, M., and Acton, G.S. (2005). [A Conceptual and Psychometric Framework for Distinguishing Categories and Dimensions](http://www.personalityresearch.org/acton/dimcat.html). *Psychological Review*, 112(1): 129-158.
This is probably off-topic; but it partly depends on your hardware, operating system (espc 64 v 32 bit) and version of R; not to mention the number of columns you are importing.  Try ?memory.limit for a start... See http://cran.r-project.org/web/views/HighPerformanceComputing.html for some options.
I don't understand the model. Can you explain what you're trying to do a little more? Or perhaps post a path diagram?
@whuber This is not necessarily true. Consider the canonical Case-Control study, wherein cases and controls are not assigned by the experimenter. In many circumstances, these studies also struggle with low cell-counts.
Adam1:> once you control (by fixed or random effect) for the correlation between measurement, you will also increase the number of parameters of your model, hence decrease the number of df.
I emailed the presenter at 3 different addresses, and had no response.  Maybe you'll have better luck? mehmet.balcilar(at)emu.edu.tr,
balcilar(at)kimep.kz,
mehmet(at)mbalcilar.net
Are the samples A and B exactly the same size?
@Nick I don't agree - software specific questions have always been on topic, as long as there's a statistical element to the question ("what are the steps to get lambda?" meets that criterion to my eyes, or potentially does). The most popular tag on CV is actually `r`.
The term "SEM" is vague. It could also mean "Search Engine Marketing", for instance, for someone looking for statistical analysis techniques for studying ad click data or evaluating advertising effectiveness. Consider making the title more verbose.
I would be careful saying that a beta is "the" appropriate distribution to use.  It's fairly flexible and it might be appropriate but it doesn't cover all cases.  So while it's a good suggestion and may very well be what they want - you can't really say that it's the appropriate distribution solely on the fact that it's a continuous response between 0 and 1.
No, if it's following my example it shouldn't be called a "census", just a sample that has been analysed to provide estimates of population parameters.  But your original post didn't mention he was calling it a census.
What's your actual problem? This approach doesn't sound like it will lead to a good solution.
what are tokens?
If your aim is actually prediction, hypothesis tests would seem to be the wrong tool altogether. Wouldn't out-of-sample predictive ability be a way to identify a good predictive model, suggesting sample splitting / cross validation and like approaches?
@jorge Mann, Henry B.; Whitney, Donald R. (1947). "[On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aoms/1177730491)". Annals of Mathematical Statistics 18 (1): 50–60. --- the purpose of the original paper was to derive the distribution for two samples of different size, and show its consistency and asymptotic normality as well as giving the exact distribution for small samples.
Perhaps you should add a fourth case right after the approximate normality of $X^2$ pointing out that the distribution is **exactly** normal if and only if $f(X) = aX+b$.
Thanks for the answer! What confuses me: assume I want to check for which ranges of losses the delay adds variability in the quality scores, how would I do that with a linear model? The delay will not always contribute the same amount of variability, but that's also something that I have to test.
Could you add proper links to the talk/papers mentioned above?
As @soufanom has given the intuition behind the chain rule. And the way i see it , you have applied it wrongly i.e in your equation $∑_c p(a|c)p(b|c)p(c)$. Actually according to the graph c is conditional over a and not the other way round and similarly b is conditional over c. I am not sure whether proof exists for chain rule. It has to be derived empirically or to be proven by induction.
The problem seems to be that you have all 3 levels of a dummy variable. Instead, why not a categorical variable called "genes" (or whatever) with three levels: Only A, A + others and Not A. Then the problem is pretty standard. The number of genes is a different issue, I think.
@Zhubarb Interesting point. As I said I am not an experienced statistician. I edited my question, so that the example also shows the extended, detailed output. I think the problem you mention is "solved" by evaluating every class against all others and calculating the Average.
Note to self; mod0<-lme(log(HRMND)~MONTH+L+A+L1+K,data=CODDING,random=~1|ID,na.action=na.exclude) Check AIC to see whether MONTH is important for the model.
I am trying to break something; I know the vales I am getting are statistically correct, but I want to make it easier for the users to understand the data. It's a UI decision.
+1 Even Excel has a log gamma function (`GAMMALN`) for such purposes, suggesting how useful (and basic) taking logarithms is.  I'm sure `GAMMALN` is implemented by means of the asymptotic expansion of $\\Gamma$ (generalizing Stirling's approximation). Concerning (4), it might be a little clearer to write your approximation as $f_{X^{*}}(80) \\times 1$ or even, quite suggestively, $\\int_{80-1/2}^{80+1/2}\\left(f_{X^{*}}(80) + O(x-80)^2\\right) dx.$
LDA extracts the discriminant function - a latent variable that best discriminates your two groups. This function is a linear combination of the input variables. The coefficients in this linear combination are the weights you are speaking about. A weight tells about the discriminative ability of a variable; it tells nothing about a class.
+1 with the request for clarification: the "systematic" part of the residual should be nigh undetectible, otherwise it is an indication that the model is not satisfactory: a systematic effect -- whether real or an artifact of analysis -- has leaked out of the model, as it were, and the model needs to be modified to avoid this.
Hi Alexis, thank you very much for your answer. I will look at the delta method. However I have a question about your answer (please see the update of my previous question).
Closely related: [Standard deviation of binned observations](http://stats.stackexchange.com/questions/60256).
See the edit in the orgiginal posting.
As far as I understand you should limit the range of your output units to -0.9, 0.9. In other words, I have the impression the saturation levels of your output units can go beyond that interval (-1,1), and eventually, if the weights become "too big", the outputs might exceed those limits.
@whuber Repeating, in $\\mathbb{R}$, all $p$-norms are equal so the univariate median minimizes $L_1$ and $L_2$ and all other $L_p$ norms. The geometric median, which is one generalization of the univariate median to higher dimensions, minimizes $L_2$ distance which is also called the Euclidean distance. The geometric median does NOT minimize $L_1$ distance. Take a look at wikipedia or any of your favorite papers/books talking about multivariate medians. Why is there even an argument about that which is clear in the definition of the geometric median accepted by the community?
I'm not very good at the whole nested interpretation that the OP wants... but to test my limited understanding of this lmer model it is: The intercept is the sum of the fixed common intercept plus a random, per-tank intercept. The slope is the sum of a fixed treatment slope and a random, treatment-per-tank slope. The per-tank intercept and slope are not correlated. Is that correct?
glmnet is _not_ a classification method.  And use the correct terminology: you are not seeking classification probabilities but rather event probabilities (risks).
@StasK: I can only delete the question, not close it. Could you  close it but keep it for reference, please?
See also ["Theory-Testing in Psychology and Physics: A Methodological Paradox"](http://mres.gmu.edu/pmwiki/uploads/Main/Meehl1967.pdf). The "null hypothesis" is always false for your field. Even with proper research practices significance tests and hypothesis tests are probably inappropriate.
Voting to close because this is cross-posted at http://math.stackexchange.com/questions/318275/distributions-of-xy-xy-xy-for-x-y-chosen-uniformly-inside-triangle: it's best to keep only one version on SE.
Page 190 of what?
The first option with the same filter command, but grabbing the ninth forecast.
About 3642 random stickers. So the upper bound for the "pack of 5 problem" would be something less than 729 packs.
+1, a lot of good information here, presented concisely and clearly
I just plot the interaction graph based on the Coefficient.
Use the -1SD, 0SD, +1SD for determining high, moderate, Low
A real hash should not give such non-uniform results.  Are you sure the hash algorithm is correctly implemented?
Thanks for the links - tried searching via a query but did not realize looking for top voted questions would also be a good way to search. Please feel free to close as duplicate of the suggested question.
It sounds like it would be profitable for you to modify your question so that it asks for suggestions about approximating the Poisson CDF by means of simple fast calculations. If you choose to do that, then please also provide some indication of (a) the range of $\\lambda$ of interest (Poissons with $\\lambda\\ll 1$ can be particularly challenging) and (b) the range of values at which the approximation needs to be accurate. (*E.g.,* in the tails?  Upper tail only?  Body of the distribution?)
What is the "original pattern" that you seek to preserve or not? Also, what methods of the reduction have you considered already?
Probably because its because its easier to work with squares in equations than a modulus sign
Damien, editing the data was a bad idea, because you have already received several detailed responses that used the data you originally posted. It's unfair of you in effect to pull the rug out from under those who have gone to that work to help you.
I noticed that the vague prior doesn't harm with large sample sizes, but my sample sizes are not always large.  Playing around with changing my prior does seem to affect the results in these cases.  The empirical Bayes approach is intriguing.