tttjjj/elt_bench huggingface dataset:
@user5497 The error of an estimate can be broken down into bias and variance components.  Ordinary logistic regression gives asymptotically unbiased estimates of probability, i.e. in the limit of an infinite dataset, the bias component is zero (assuming the model is "correct").  However the variability of the model due to the sampling of the training data can be high for small datasets.  Regularised models trade-off bias for reduced variability, and hence can achieve better estimates of probability for small datasets (or equivalently datasets with many features) than the standard model.
`paste` is your friend. Just construct a string like `legend.string <- paste("In traffic [Max", max(values),"]")` and then pass that as the `legend` parameter of `legend`
In the output `bias` is simply the difference between the *mean* of the 500 stored bootstrap samples and the original estimate. The `std. error` is the standard deviation of the 500 bootstrap samples and is an estimate of the standard error. The output tells you that your original estimate is *higher* than the mean of the 500 bootstrapped estimates (so not *all* of the bootstrapped MKtaus are lower). The bootstrap is often used to calculate standard errors/confidence intervals without making assumptions about the distribution. Use the `boot.ci` function to calculate the confidence intervals.
@Macro I appreciate your reasoning and agree with it. However, I hope that my answer demonstrates the sense in which this question goes beyond any particular package. (I lump `geoRglm` in with `geoR` because they are typically used together and documented together. `spBayes` uses a different approach.)
Thanks. You helped me a lot!
@whuber's suggestion is discussed in a little more detail [here](http://stats.stackexchange.com/questions/60500/how-to-find-a-good-fit-for-semi-sinusoidal-model-in-r/60504#60504)
@NickCox Regarding the authors documenting code - not necessarily. I remember one book (I think it is Agresti's Categorical Data Analysis but am not sure) where someone else wrote code for the whole book.
(-1): Although Weka is indeed a Data Mining Tool which indeed contains an implementation of NN and DT, the answer is so generic it could be reply to a ton of questions. If you think that Weka is suitable for the special task of rating prediction given extremely highdimensional sparse data, why don't you show an example (or a link to an example). No offense, please take it as a friendly suggestion.
These data would have to be pretty badly skewed to appreciably change the large p-value returned by `ttest`.  Perhaps you are using this merely as an illustration and are looking for a general answer?
(+1) For the edit.
It is equivalent derivation, if and only if. I'm trying to get some correct result, then based on iff the proof is correct.
Are you having a hard time with "correlation coefficient" and "differ significantly" or perhaps just one of those?
Many thanks for your reply Peter. It is much appreciated.
I think it refers to the total number of user, not on how a fraction of the total has evolved.
Effort is commendable -- BUT -- neither PC1 nor PC2 tells you who did best in all subjects.  To do so the PC subject coeffcients would all have to be positive.  PC1 has positive weights for Math and Music but negative for Science and English.  PC2 has positive weights for Math and English but negative for Science and Music. What the PCs tell you is where the largest variance in the dataset lies.  So by weighting the subjects by the coefficients in PC1, and using that to score the students, you get the biggest variance or spread in student behaviors. It can classify types but not performance.
In short, MIC is an excavation of old idea of "plot-all-scatterplots-and-peak-those-with-biggest-white-area", so it mainly produces false positives, has an unreal complexity of $O(M^2)$ (which authors hide behind test-only-some-randomly-selected-pairs heuristic) and by-design misses all three- and more- variable interactions.
(+1) As long as you can compare $X$ and $Y$ (_i.e._ not if they are multidimensional etc.), it has a meaning.
nobody likes my answer?
Sample kurtosis is given by closed-form expressions; there are differing formulas, but I've never seen that which one to use depends on what distribution you think you have. Perhaps you mean is there a closed-form expression for the probability density function of kurtosis when sampling from a Gaussian?
@whuber Rasman's approach is to treat the prepost (ideal versus nonideal) differences as fixed and to determine if there is mediation by group (X versus Y). This approach is simply a two sample T-test between the pair-differences treating the differences between ideal and non-ideal as observed values.
For me, the intuition is that if Loopy BP gets stuck in a feedback loop it stays there forever, whereas if Gibbs hits a local optimum, even if it takes the age of the universe, it will eventually leave.
For SPSS through the GUI interface if you go to `Analyze -> Descriptive Statistics -> QQ-plots` one can get a QQ plot of the observed against various theoretical distributions. Check out the `EXAMINE` command in help for tests of normality like Shapiro-Wilks and KS.
(+1).  Gosh: so if I perform ML fitting of a Normal distribution to my data and limit the possible values of the SD to the range from $10^{-1000}$ to $10^{1000}$ and the mean to the range $\\pm 10^{1000},$ I guess my estimates will no longer be valid ... . :-).  Since those endpoints are beyond the range of IEEE floating point precision, that must mean nobody can trust any of the statistical software that runs on standard computing devices.  It must be time for all of us to pull out that old abacus (it's on the shelf with the slide rule) and go back to doing the calculations by hand.
Technically the 2nd option isn't a split sample, is it? The FY2011-FY2012 is an independent sample that will also contain ~20,000 members (well, there will be some overlap of members with the FY2012-FY2013 dataset so it's not completely independent).
I edited your 2nd paragraph to make it clearer. Please check and correct if needed.
Have you ever regressed anything against a date? Many computer systems start their dates in the distant past, often over 100 or over 2000 years ago. The intercept estimates the value of your data *extrapolated backwards* to that starting time. How certain would you be, say, of the gross domestic product of Iraq in the year 0 CE based on regressing a series of 21st century data?
This is a frequent question here. Generally speaking, you *can't*. Let's say your data look exponential. That doesn't mean they *are* exponential. You can never prove it by looking at the data. As you look at larger and larger samples sizes, you may be able to exclude many alternatives, but there will always be an infinite number of alternatives that will fit the data about as well *even if the data are truly exponential*.
You seem to have read the paper thoroughly. Why don't you write one of the authors or all of them an email.
I bet, that they can give you a better hint on your problem then somebody here because not many people will have read the paper you mention.
Thank you. I like your suggestion of introducing cut offs.
I don't grasp the layout fully. But if you had groups of evaluators judging multiple speakers, you should perform an appropriate inter-rater reliability test (e.g., ICC) for each group. If that test shows sufficiently high agreement among evaluators--and you believe they were competent to judge (presumably; why else ask? still you are assuming the validity of your measures)--compute the mean scores & standard errors for each speaker. You must use judgment at that point to decide how much imprecision you are comfortable with in making your rankings--much as you would when grading on a curve.
I understand your distaste for SPSS and concerns about fishing for results, but that is why I am asking this question - to understand the output. If your opinion is that the multivariate results table offers no valuable input and I should only look at the corrected p values in the within subjects test as a test for significance, I would like to know your reason for thinking so.
+1 Welcome to our site!
For what I understand, hard margin means you don't accept datas in your margin. As a consequence, max(0, calculation) will always return 0.
It seems that you asking 2 different questions, one about AIC, another about installing certain R package. Please ask 2 different questions.
I learn the most from totally wrong answers, Juancentro--especially my own, which fortunately rarely get out in public :-). I had thoughts similar to yours until @ttnphns left a comment.
A quick comment, not an answer since two useful ones are already present: standard deviation is a property of the (distribution of the) random variable(s). Standard error is instead related to a measurement on a specific sample. The two can get confused when blurring the distinction between the universe and your sample.
I wouldn't rely heavily on using considerations of support to select a probability model.  After all, normal distributions--which, like Gammas, have infinite support--are frequently used, with great success, to model distributions that *must* have finite support (such as the sampling distribution of binary responses in a survey question).
Glad to hear it now makes sense.  btw you don't have to post the image - you can write it in latex math notation, e.g. Bayes theorem is written `p(C|F) = \\frac{p(F|C) p(C)}{p(F)}` which when you surround it with `$` signs renders automatically as $p(C|F) = \\frac{p(F|C) p(C)}{p(F)}$.
What exactly is it that you want to know if it's 50-50?
Legato, if you read the tag wiki on [self study](http://stats.stackexchange.com/tags/self-study/info) you will see that you have some responsibilities under that tag before we are supposed to give answers in the form of hints. What have you tried already? Did you get as far as writing down the Poisson probability function? How would you work out the probability of selling exactly 5 products?
Ok a few other questions: 1) Are there any random effects in this model? 2) Does the `packyrs` variable have any missing data? 3) Any chance of a simulated example that exhibits this behavior? It may increase the chance of hunting down this issue if we had some data to tinker with.
Is there a statistical test that can check for the case of a linear contrast ? I check Faraways' book on how to do it in R and i was not able to find something.
Why are you doing MLE if you're dealing with a Bayesian problem?
It is difficult to understand what is being asked here.  My guess is that you are considering uniform random permutations of an array $(x_i)$ of $n_f$ copies of $F$, $n_m$ copies of $M$, and $n_a$ copies of $A$; your random variable $Y$ is the shortest distance between $M$ and $F$ (*i.e*, $\\min\\{|i-j|: x_i=M, x_j=F\\}$); and you want to compute the expectation of $Y$.  But your "toy problem" is confusing because the shortest distance can *only* equal $1$ or $2$, but the probabilities you assign do not sum to unity!  In what respect(s) is my guess incorrect then?
Would be great to share experiences if we ever come to cope with our similar problems. Write to roland dot kofler at gmail if you want to share
The question then is how to get a survfit.object that represents a mixture of two survival curves, rather than two independently occurring curves.
Some care is advised in doing PCA with the categorical variables!  If they are purely categorical (and not ordered), perhaps they can be re-expressed as indicators.
thank you. im meeting my supervisor this Friday. i ll discuss this issue with him. there re pages which shows how to perform it but neither has discussed issues related with it like this.
im sure he is gonna let me drop the variable from analysis. thanks again
I'm not sure what you're asking. If you have thousands, why only relate a few hundred (unless those are important *a priori*)? I can't speak to what users 'care about', doubly so when I have no context at all.
Part of the reason effect sizes were suggested in the first place is to get away from the practice of testing every single statistic for "significance". See [Cohen (1994): The Earth is round (p < .05)](http://scholar.google.com.au/scholar?cluster=10018019703886757098&hl=en&as_sdt=0,5)
I have 1000 time series observations and it's my HAC Newey West that is giving me significance across the board (you said they were "biased down"? Or did you mean SEs were biased down and not the test statistic?)
@FrankHarell Maybe so but other objectives than reproducibility might be just as important including getting results ASAP, especially if you are not going to analyze data every week. Even the fact that more people know it is a valid reason to choose it over something else (or for employers to demand it). How “reproducible” is something implemented with an R script if you are the only one knowing R and you leave your job? Bottom line is that the mere fact that Excel is so popular shows it's providing something to its users.
Even if you do succeed in categorizing these numbers into five groups, that cannot (of itself) tell you *anything* about whether they are "terrible," "excellent," or anything in between: those are *value judgments* that cannot be determined solely from a bunch of numbers.
Never heard of this algorithm before, but it sounds pretty interesting. Could you provide a link to a explanatory source ? Thank you !
Hang on! I can't figure out how to get it to come up, BUT am i correct in saying that if my total after rotation is 4.886 and I have 20 items then my percent variance is 4.886/20*100!!!
Check the `party` package in R. The `ctree` function builds trees using permutation tests and thus splits only on statistically significant splits, so you won't need to worry neither for significance and neither for overfitting
@PeterFlom It's not homework but a question that I'm self-studying. I added the tag though.
I'd guess the LRT for Weibull shape parameter less than one would have good power against long tailed alternatives.
@Aniko: I was just using 'w' because that's what code that I grabbed had. I should have renamed it 'r2' to represent r-squared during the looping, and then added a new variable 'transform' for the 'w' in the second to the last line in the original code. When I gave up trying to get a return value from 0-1, I resorted to <code>x1 * w * stdDev + mean<code> but the code is not used in the context of overriding Random.Sample() anymore. How do I get <code>z<code> given the Marsaglia polar method?
You're right, there are. The important point to make is that we must look beyond just getting a result. The first step is to assess the situation to determine what can be accomplished and whether our clients' actions and decisions will be sensitive to these issues of data comparability and quality. A statistician who proposes procedures too early in this process, before the question and data are sufficiently well understood, potentially sidesteps all these considerations and may ultimately lose all involvement when the client runs off and implements those procedures themselves.
I just updated my answer to briefly mention the existence of the equivalent in GSL and to link to the part of the manual that relates to it.
The suggestion of zero-inflated negative binomial regression is a good one.
You have, it seems, a discrete distribution with four substantial modes (very common values) and four other distinct values which each occur once only. I'd go upstream and find more about how the data were produced. This doesn't look like any textbook distribution.
(+1) Keeping it simple has several merits, especially from a pedagogical point of view.
The Pearson correlation coefficient doesn't require the variables to have a certain distribution. The only connection I see between the case you are describing and correlation is that, if the variables are uncorrelated, it does not mean they are independent (which is simply the general case).
We could probably use some additional context here - what are you studying? What's your sample size? Is there a theoretical reason to believe this interaction exists? Is this exploratory or confirmatory analysis?  Because I can think of a great many contexts in which I would say that p = 0.08 is plenty of evidence that that interaction exists.
@andrea Thanks :). I've tuned the language a bit.
[This one](http://stats.stackexchange.com/a/60324/805) *was* a good example of what you were concerned about - it was originally just the first sentence - pretty much just a couple of links  (but it's been edited since).
yes, RFs include a random sampling of variables (typically $\\sqrt{p}$) when growing a tree and each tree is based on a boostraped sample of the individuals; variable importance is computed on so-called out-of-bag samples (those not used for building the decision tree) using a permutation technique. The algorithm is repeated for m trees (default m=500) and results are averaged to compensate uncertainty at the tree level (boosting).
Related, though perhaps pedantic, is the acknowledgement that the normal distribution, because it has support on the whole real line, is not a great representation of something like "height of trees," since heights of trees are bounded below by 0. A distribution with support $x \\in [0, \\infty)$ might be more appropriate.
Does this answer your question? http://stats.stackexchange.com/questions/6181/can-the-multiple-linear-correlation-coefficient-be-negative  If not, then please provide more information: this is the "SPSS output" of what procedure?
(+1) Worth reading, yes, entertaining, yes. But a textbook? Not exactly... I take the subtitle to mean it's intended as a *companion* to a textbook. Andrew Gelman's review of it is worth reading too: http://www.stat.columbia.edu/~gelman/research/published/angristpischke2.pdf
My point is that in many cases, the purpose of regression modeling is to figure out what's happening even when we don't know all the causes of a particular outcome. But as it seems unclear, I will remove that question.
@114, The correlation does have an effect on the outcome of the regression, I was just showing that the range of possibilities was quite large.
So long as you use a formula that adjusts for the number of ties, the actual number is immaterial.
Biggest problem: this approach doesn't generalize well. If you have two factors, you still have to leave something out.
you have received an excellent answer by Dan. Please consider upvoting and/or accepting it if you are happy with his explanation. The model of this site depends on the contribution of everyone. Writing a good answer takes a good amount of time which is time that is devoted to help others. Accepting/upvoting only costs you a click with the mouse :)
Looks like @probabilityislogic beat me to it.
A negative binomial (or poisson) distribution can produce large sequences of zeros with a low enough rate parameter. How do you know all of your zeros are "real" (i.e. not the result of a Bernoulli trial prior to, say, a Poisson process)?
@Andy The OP's data is a data set with multiple observations per period, over many periods, without knowing whether in each period the same cross-sections are observed or not. That the OP may also have two or more observation on some cross-section in the same period, makes matters more complicated -but the place for the OP to begin digging the literature is the pseudo-panel one.
That moment when, a year later, you realize that **the** Michael Betancourt posted an answer to your question...
@Dougal Thank you for your comment. Differences between MLE and log. regression are (rounded): 0.0951  0.0106  0.0016 -0.0005 -0.0006 -0.0003 -0.0002 -0.0001 -0.0001  (0 for rest 12 points). Total sum for MLE is 0.1279 and for l.r. 0.0225
@Pinocchio: that is most certainly not a silly question, but unfortunately, there is no good answer. I edited my answer to give some more information. Hope this helps.
Dear juampa, I kindly request you to give a reply
I'd keep x2 just since it is already ordinal, you want to keep an ordinal relationship.
I am not sure what the problem is with the output. It seemed like all the variables were highly significant in all variations.
Looks like no other takers.
The three studies have $50,000$ people in total. It wouldn't make sense for each study to have $50,000$ people. In this case, the $100,000$ people screened would still be the source population?
The alternative's that the null ain't. I don't know that the L-B test was designed for maximum power against any specific alternative; the ones of interest are those where there *is* auto-correlation. The 2nd paper cited illustrates power analysis for AR(1) alternatives with a variety of white noise distributions.
Hi Jonas, I have edited your mathematics a little. Please check that it still says what you want. If you want $\\hat{\\theta}$ instead of $\\theta_h$, you get that by typing `$\\hat{\\theta}$`.
Okay, that's great, thank you :)
Thanks for your nice reply!I am also wondering What's the difference between your anova(fit1ml, fit2ml) and my anova(fit1, L=c(-1, -1, 1, 1))?
@Misakov I updated my answer.
:cardinal I guess I felt that your comment was important and made it clearer as to what was being assumed. If you think the idea of "second order stationary" adds clarity please help me add it to my answer in a way that sheds light in simple straightforward English.
+1 for proportion is empirical, and it is often a good estimate of a probability which is theoretical !
Hello Whuber.  Your example of ESDA is a great example, thank you!  If you (or others) can suggest other examples of when formal procedures are less relevant - this would be most helpful.
I think glmnet is a great tool and would give reasonable results in this circumstance.  However, you can't set the penalty to Zero.  Glmnet uses a coordinate descent algorithm, but it still will have diverging coefficient values just the same (and even has some exception code to handle this if you look at the source code).  You can however just include a very small penalty.  But why not cross validate and pick a good lambda?
http://stackoverflow.com/questions/1408897/connecting-gnu-r-to-postgresql
@shabby I agree that it can be seen as an EDA problem, but it can also be framed as a fuzzy ranking question and it's identical to finding an [efficient frontier](http://en.wikipedia.org/wiki/Efficient_Frontier) in financial analysis.  This leads to three completely different solution techniques.
...I just tried to add `X = apply(X, 2, function(x){ x-mean(x) })` at the beginning of my code - and it did the trick (also with matrices)! Could this be all?
I am not sure of this but I read that ridge regression cannot set the norm of a coefficient to zero [here on stats.stack](http://stats.stackexchange.com/questions/866/when-should-i-use-lasso-vs-ridge) In such a case your argument wont hold, right?
I constructed an index, consisting of x1 and x2. Denote it as x3. People in seminars are now concerned whether the index x3 is endogenous to y - mainly because x1 may drive y. However, neither x1 and the index x3 are strongly correlated (correlation is 0.08), nor is x1 strongly correlated with y. Can we thus say that x1 does not cause endogeneity of x3 with respect to y?
@user1893354 Answer revised.  Let me know if that helps your intuition.
@Giuseppe That paper is referring to the gradient descent within boosting, not the "typical" (more familiar to statisticians) gradient descent in which all of the covariates are simultaneously modified. I have tried to clarify in my answer.
that is a nice package that you have written, but I am interested in estimating the sample SE, and these functions appear to give variance estimates for the meta analysis effect sizes, whereas I would like to infer the variance of the population (e.g. scaled to the original data). Could you provide an example of how the functions in the `compute.es` package could be used to replicate the equations and function that I have written above?
Between 210~750. It will change because this a `gui` but the size is near these values.
Please, do NOT [cross-post](http://dsp.stackexchange.com/questions/6048/knnsearch-using-examples).