tttjjj/elt_bench huggingface dataset:
For making your data generally and systematically available in citable form to complement the paper, the folk at the [Dataverse Network](http://thedata.org) can probably help, often at no cost. (btw I'm not one of them).
What do you mean by 'trivariate'? It would also help if you could to the extent possible use equations to illustrate your situation. Do you mean that you observe $Y$ where $Y = X + Z$, $X \\sim Exp(\\lambda)$ and $Z \\sim N(\\mu,\\sigma^2)$?
@matcheek This is the SVM reality -- you can try randomly reducing the dataset and optimizing on it, to at least get an approx. where to look. Or just change the classifier ;-)
ok so you're saying that if I associate low residuals with high predicted values, heteroscedasticity can arise even if residuals are normally distributed? I think I've got it, though i should really think more about it.. anyhow thank you!
What is your response variable? Is it a continuous variable that just happens to fall between 0 & 1? Is it the proportion of successes from some number of Bernoulli trials? Is it a continuous proportion?
Try searching with Google Scholar (!) or a similar academic search engine. Also, the Weka book might have something in the 3rd (latest) edition.
You might be better off sending this question to statalist.
It would seem that they were given a set of counts that were binned, & there is some question about whether the counts are the same across bins. I'm reading between the lines somewhat here, but I can make an educated guess what analysis they may have hoped students would do.
If the url is wrong, email the author! he is the only one who can do something with it
I answered in the question you link in, check this: http://stats.stackexchange.com/questions/26024/moving-average-model-error-terms/74826#74826
Is my question not interesting ? answerless ? or anything. I would be happy with any element.
This does belong on StackOverflow. There's no statistics question here at all. Only a general programming question.
I'm actually taking a NLP course in the Fall.
I’m sorry, I was talking about a logistic regression.
Thank you, now I'm working with two different approach regarding the channels and I'll see if it'll affect the classifier's accuracy :)
Well, those two statements contradict each other. You have to be careful that regression is only looking at linear relationships; there is probably a nonlinear relationship going on.  You are not ready to report yet.
Any type of regression.
_But what questions are you actually trying to answer?_ The first question was whether such a regression would violate an OLS assumption, like someone with more statistical experience suggested. Your answer confirms my hunch, namely that it does not. Secondly, we wanted to get an quick overview of which variables will likely be useful in the regressions that followed, which for example try to find out whether being good in quantitative topics tends to help get good grades in law and other ideas like this.
Are the subjects grouped according to how many times items are presented (so in group 1, everyone gets 3 presentations per day for 4 days?), or are the number of presentations different for each subject (randomly?) per day?
Thanks, simple but works!
@whuber I sagree with you that the most important thing is goodness of fit. With that being said, goodness fit is relative to how other models fit. Additionally, the Shapiro-Wilk test is quite conservative and if your data is not passing the test then you are likely to have poor fit and the set of linear regression models you run. But I agree with you, goodness of fit is paramount.
*How to lie with statistics* makes a nice, cheap companion to this. Every undergrad taking Stats 101 should be encouraged (or forced) to read both.
Hello, thanks for your answer. The point is not which analysis to perform. Indeed from the paper I read, I know that I must use the losgistic regression, non the normal regression. To answer to your quesstion, yes 6 and 7 are considered as similar.
In your model you are implicitly assuming that there is a **constant variance** across time. However, in reality, the variance of stock returns is not constant over time. In other words, the problem is you have a misspecified model.
I have posted the question in a new page. Thanks a lot! http://stats.stackexchange.com/questions/53380/variance-of-powers-of-a-random-variable
Those two techniques might both end up working out -- I had no idea there were so many methods for calculating confidence intervals.  In particular, thanks for the pointer to BlakerCI.  Though an ideal method might somehow use the greater population's background probability $p$, it's good to have solutions I know I won't screw up with crummy choices for beta distribution parameters.
I think you are all confused, rms stands for your favorite GNU advocate, [Richard Matthew Stallman](http://en.wikipedia.org/wiki/Richard_Stallman)
Wow, thanks for taking the time to write out such a complete explanation.
Thanks for the reply! Looking forward to seeing the new chapter soon. For now, I'm considering sample variance of $\\frac{p_A - p_B}{p_A}$, and terminating the test when it approaches zero.
@dieter-menne: Would clinical chemistry be medical enough? You should be able to find a lot of $c_{method B}$ over $c_{method A}$ plots there. Paper is 2d, paired data is 2d, sounds like a nicely fitting situation to me... That would even allow for a further (symbol shape or color) dimension indicating who got treatment A first and who started with treatment B (so you can check whether there is really no difference between those two groups)
Thanks a lot jbowman! Your answer was really helpful. I took time to reply back as it took me sometime to understand your argument.
@StasK Why simulate, when there are only $2^{15}$ subsets to work with?  In fact, by using the FFT to expand $\\prod_{z \\in \\{0.17, \\ldots, 0.69\\}} (1-z+z t)$ in powers of $t$, the computation is practically instantaneous.
Alright, I gather that the "background" variable relationship is in facts a main effect adjustment plus maybe *some* interactions with the basic skills IVs. Thanks for all the references you added.
+1 Log-linear sounds good given the plentiful requirements! :)  I'd also recommend Agresti's text on categorical data analysis.
Ticket sales are probably all nonnegative, and they may be integer, so you should expect residuals that are skewed (simulate negbin data, fit them with the correct model and look at the residuals). That does not need to mean that your models are incorrect.
It sounds as if your question is "What might I do with all this information?"  Which is really not well suited to this Q&A site.  There is no correct answer.  Your best bet is to explore topics in statistics, especially in time series analysis.  And you can do a fair amount of that by reading threads on this site.  If you settle on a course of analysis and find that you're getting stuck somewhere, this will then be a good place to ask a question about it.
This is the variance given $ X $.  I think its errors in variables that's wanted.
@MichaelChernick: Thank you. One reason for the good result could be the commonality of words in tweets and general language usage. Is there a way to say something like "it worked on a sample size of X, so with a confidence of 95%, it will work on the others"?
This has been posted on Statalist too. Neither forum has rules against cross-posting to the other; nevertheless anyone tempted to reply on Cross Validated should check out the Statalist archives to avoid repeating points already made. You do not need to join or register; you can go straight to stata.com/statalist/archive/2013-04/msg00032.html to see discussion.
@Toby If it could be relevant, why is the assumption of a normal distribution not mentioned in the question?
@whuber: ok, I edited the above to consider the absolute value (I also derived the mean of the absolute value but <a href="http://www.ceremade.dauphine.fr/~xian/meanabs.pdf">it is rather ungainly</a>...)
I still have one question: which post-hoc test is the most appropriate in my case? Tukey or Scheffe? Thanks!
Out of curiosity, where do these numbers come from? Most datasets consist either of measurements--which are rarely made to more than a few significant figures--or of ratios of such measurements. Apart from `x6`, these appear to be neither kind, making the extended precision rather surprising. It makes me wonder whether we might be looking at data that have been derived, via calculations, from other data. As with historical evidence, so is it with statistical evidence: usually the closer you can get to the primary source--the *original* data--the better off you will be.
@Marcinthebox Thx. See the update please.
@whuber: Could you point out which are the other answers in the thread?
@mpiktas  If $b_i$ is the first bit in a run of ones,  why should be $b_{i+1}=0$? ( I think you intended to write $(Y_i=1)=(X_{i-1}=0, X_i=1, X_{i+1}=0)$)
Have you tried simple linear regression, one model for each of the target variables? With that many input dimensions it could work. The sum-to-one constraint could be dealt with by normalizing the outputs of the four models. Not sure if the result would make sense though. Anyway, I'd give it a shot!
Pick the model that suits your purpose, considering the relative costs of false positives and false negatives. AUC can be a misleading measure of the quality of a model, particularly in cases like this where you don't care about all the possible predictor-variable values (in this case, you apparently don't care about values that give FPR > 0.2). It's good that you plotted the curves; they indicate that for your purpose at low FPR the green model has superior sensitivity, as you thought.
Note that use of Bayes theorom does not automatically warrant calling it bayesian statistics. One problem is that at this moment you just have estimates of the underlying probability and no feeling for the uncertainty of your estimation. +1 for the detailled writeup.
This doesn't answer your question directly, but there are alternatives to boosted trees called "bump-hunting" approaches ([example](http://www.scs.gmu.edu/~jgentle/csi991/members/mainpaper.pdf)), which automatically identify maxima as part of the model (see especially sections 5 and 15 of the linked paper).  There are even boosted bump-hunting methods, which may get better accuracy.
There are many options, I think. One obvious pattern would be events that increase or decrease in probability with their own occurrences. Hard to think of some examples. Perhaps ants arriving in your kitchen/ picnic. Time it takes for the first ant to arrive is probably a lot longer than the second or third, and of course more ant arrivals mean more likely future ant arrivals (given that their trails/ communication with one another). Not sure if that would count.
But NLP cant handle sound files.
@Pedrofigueira That's okay, I will try to rephrase it so that it is well-posed.  What you have said is correct, I am looking to test if the variation in performance is entirely due to the parameters.
But the [lognormal distribution](http://en.wikipedia.org/wiki/Lognormal_distribution) does not have a moment-generating function.
Hmm. This is the way to go, as all the answers are pointing out. Appreciate your advise @Peter.
The probability can have any value from $0$ to $1$. The value $1/25$ given in the answers are based on the $5$ orientations being _equally likely_ for all the screws and the choice of orientation of each of the screws being _independent_ of the other choices. Neither assumption might hold in an actual situation. In fine carpentry, the three screws might be placed in the _same_ orientation, or two in one orientation and the third in an orthogonal orientation.
The results you give for the iid case hold for any continuous distribution. Furthermore, the sequence of events $A_i := \\cap_{j < i} \\{O_i < O_j\\}$ corresponding to whether the $i$th observation is a *record* are, in fact, independent. This is a famous result due to Renyi. There is also some recent work on statistical properties of rankings. One example, which isn't immediately related to your problem, but has some vague similarities is P. Hall and H. Miller (2010), [Modeling the variability of rankings](http://projecteuclid.org/euclid.aos/1278861456), *Ann. Stat.*, vol. 38, no. 5, 2652-2677.
That's fine--but you might be emphasizing the wrong things. It sounds like there's nothing special about your dataset apart from having nonnegative values (and even that characteristic would vanish the instant the data are recentered, which would not change the solution you seek). Thus, unless I am missing something, the question is "how does one go about clustering data?"
Measure of what?
Are you sure that offset(y) works with glm.nb?  This could why predict isnt working.
(This reply came here after a duplicate question, framed somewhat differently, was merged.)
It seems like this DrWho has asked and gotten answers for this exact question a number of times: http://stats.stackexchange.com/questions/11917/what-is-the-best-statistical-method-to-compare-case-fatality-rates-of-17-years-fo, http://stats.stackexchange.com/questions/8358/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-cas , http://stats.stackexchange.com/questions/8187/how-to-assess-effect-of-intervention-in-one-state-versus-another-using-annual-dea
It would be helpful to know why the previous answers were not satisfactory.
Please mark homework problems with the `homework` tag. Also, since these are homework problems, please also indicate what you've tried and what, specifically, you are stuck on. Welcome to the site, by the way!
@ schenectady while i sympathesise with your viewpoint, for me at least, the R code provides a bridge that i can use to greater my understanding of the equations and maths concerned. That being said, i wholeheartedly agree with the need for problems, statistics and maths in general is something that can only be learned by doing.
Canonical loadings are easy to compute after the analysis (this is probably why many packages don't calculate them for you). Loadings and cross-loadings are the correlations between the variables and the canonical variates.
@Frank I often like restricted cubic splines. The only negative is one of interpretation, which is a little tricky and often turns my clients off. Adding a polynomial term (after centering) seems to be more interpretable
(+1) Michael, I have enjoyed reading your papers and answers here and elsewhere on CV. However, I am still struggling to understand what is so bad about F/N-P "hybrid"/"mishmash". Let's say I report p as either <0.05, <0.01, or <0.001, and it it's >0.05 declare it not significant. This is, quoting from your "Bad statistical practice..." paper, *"a hybrid approach that neither controls error rates nor allows assessment of the strength of evidence"* -- why? Error rate is controlled at 0.05. Strength of evidence is assessed, albeit in a suboptimal, quantized way. What is the big deal?
(Don't get me wrong; I have huge respect about Baayen's work (he also gives great feedback as a referee), anything he does though doesn't immediately apply to my/your work.)
(The link associated with Neymann-Chuprow allocation makes no mention of it.)  This is a nice start.  It would be helpful to add a few remarks about the need to collect data to test the model specification: often, optimal designs are inadequate for this (crucial) purpose.
Sorry for the typo, it should be correlation coefficient, not correction coefficient.
Should you clarify you answer and provide some details.
Jaynes first example: Not one statistician in his right mind will ever use an F-test and a T-test on that dataset. Apart from that, he compares a two-tailed test to P(b>a), which is not the same hypothesis tested. So his example is not fair, which he essentially admits later on. Next to that, you can't compare "the frameworks". What are we talking about then? ML, REML, LS, penalized methods,...? intervals for coefficients, statistics, predictions,...? You can as well ask whether Lutheran service is equivalent or superior to Shiite services. They talk about the same God.
I guess I need to rewite this, I assume I have been confusing in my question.  Any advice would be appreciated.  Thanks in advance.
Also How will i compute the partial differentiation with E(exp(-theta*Ta)) with E being a part of the equation. If you could provide me a head start on how to differentiate I would be able to proceed.
I am familiar with ridge and lasso penalties to cope with a large number of predictors and have read some techniques like support vector machines are good at coping with the curse of dimensionality but I'm looking for ways to spot samples or features which may be throwing me off. With mixed data and so many features though I'm not sure of equivalents of cooks distance or metrics/ways to define "outliers" and that.
My bins will be used to represent certain ranges in my data. Assuming my data is grades in a classroom, 1st bin will contain the lowest grades and 5th bin the highest. I would like to decide on the bin capacity by looking at the distribution of data. However, I don't know what to look at.
Because this exposition wanders and never really seems to get to a single clear question, it is hard to see what you are asking. Most of it reads like an exploration of the (well-known) consequences of floating-point imprecision. And what kind of simulation are you running where imprecision on the order of $10^{-16}$ is going to make any difference? Would it be fair to sum up your question by characterizing it as "How can I obtain a Cholesky-like decomposition for a degenerate correlation matrix using (only) double-precision arithmetic"?
@ssdecontrol That was the closest reasonable explanation so far, thanks. But it doesn't fit well though.                         For instance: m0 <- c(0.227, 0.0665, 0.0645, 0.0595, 0.353, 0.08, 0.08, 0.081)                                                  m0/1000000
[1] 0.0000002270 0.0000000665 0.0000000645 0.0000000595 0.0000003530 0.0000000800 0.0000000800 0.0000000810               If you I do apply this reasoning using previous election result then, for the firs cell I should see something like: 0.3006/1000000 = 0.0000003006; everything seems to me too much arbitrary.
Characterizing this problem as a search for outliers, chris, does it injustice: you are actually tackling a *quality control* problem. The principal distinctions are (1) you have an ongoing stream of data rather than a static dataset to analyze and (2) you intend to specify periodic actions to take as a result of each analysis: that is, whether to intervene (and attempt to improve the process) or not (and let the process run as is). Understanding that this is the nature of your problem shows that the huge literature on quality control is relevant, providing a rich assortment of solutions.
If you compare `m0`,`m1`, and `m2` as you suggest you get the warning: `Fitted objects with different fixed effects. REML comparisons are not meaningful.`  To compare fixed effects you have to use regular maximum likelihood instead of REML.  See edit.
Look on the [NHANES](http://www.cdc.gov/nchs/nhanes.htm) site, @Elvis, or go to the [CDC](http://www.cdc.gov/growthcharts/).  (Weight, height, head size, and BMI by age up to 20 years are available at the CDC.  Unfortunately, hand size, leg size, and--especially--cerebrum weight will not be available.)
While @Glen_b gave you an excellent answer, I would like to note that if your $\\sigma_i^2$ (or the proportionality constants $\\tau_i$, in his notation) are not very accurate, then you may end up being worse off with your regression than you would have assuming equal variances. Econometricians have given this topic a lot of thought in the 1970s-1980s.
Thanks for the suggestion!  I actually stumbled on the package myself a couple days ago.  From my cursory investigation, it looks like the gemtc package is basically an R interface into a package that does Bayesian data analysis like rjags.  Because I'm not really familiar with Bayesian methods, I've been reluctant to spend the time to figure out how to use the package.  Do you know of any tutorials besides the package documentation that illustrate how to use the package?
What has that got to do with uncertain class labels?
$g(x)$ could be assimilated as a vaccine, and $f(x)$ how this vaccine is effective. $f(x)$ is modeled to evaluate a group. To accelerate the simulation I do not analyze people by people, even group by group, be cause there is a thousand of million of people. As $f(x)$ depends of number of dead people, I calculate the average of them to use in $f(x)$. In general, this gives me a good perspective of how is the situation. So, I would like to know how the error of doing this simulation a hundred times or a thousand times should be. How reliable is this value.
That's exactly what I was looking for! Now I just have to figure out his math...
Vic writes that "sensitivity and specificity are both similarly important", which I interpret as the cost of the different misclassifications are identical. In this case a simple cost function like the Youden Index is appropriate (but it isn't the case in general, indeed).
If you want the actual average, removing outliers will (nearly always) bias your answer. You'll need to define 'decently accurate'
Any nonlinear correlation that is positive. The context may be: when $\\mu-Mo_1$ can be a measure of error in the second generally non-normal distribution (subscript #1)? I'd appreciate also an incomplete result, like for linear positive correlation.
You might find Winship and Morgan's [overview article](http://www.wjh.harvard.edu/~cwinship/cfa_papers/theestimationof.pdf) helpful.
Is your main research Q more about length of time until cancer or about the causes of the three different outcomes?  If the latter, you might try a multinomial logistic regression.
Thanks for the answer; this is actually something I have done and I find it very useful. In this question though I'm looking more for something that looks inside the network rather than treats it as a black box.
@MrBenderV You said "the same letter should have the same number". And in the comment "the letter c shouldn't always give 214". Can you clarify?
This question is very similar to this other one. http://stats.stackexchange.com/questions/53345/why-must-one-trade-off-between-bias-and-variance/53347#53347 Sorry about the auto-citation. Gung's answer is very nice
Thanks for the suggestion. I ran this analysis and saw similar effects as the original analysis (an overall effect - which in running the one-way can be indirectly understood as an interaction).
No I'd look at both VIF and condition index - also see this similar question: http://stats.stackexchange.com/questions/4099/vif-condition-index-and-eigenvalues. Keep in mind if you're building a predicitve model, a little multicollinearity isn't necessarily bad since coefficient estimates are unbiased even if SEs are inflated.
"How could I know which line has bigger incline?" - seems like he's asking how to recover the coefficients (from 2 different models?) given p-value, standard error, and so I'd guess "sample volume" is sample size
@whuber Apologies, updated above.  I was using k as the number of successes.  So it reads (for example): Chance of drawing less than 3 successful outcomes.
Bear in mind that inferences on transform data cannot be validly applied to non-transformed data because $f(\\sigma^{2}_{X}) \\ne \\sigma^{2}_{f(X)}$. So if you find statistical evidence that $f(X) \\ne f(Y)$ you **did not** just find statistical evidence that $X \\ne Y$.
Modelling a binary (dichotomous) yes-no response as normal makes no sense to me. I don't know how you would even imagine doing that. With continuous predictors and a binary response a logit model would usually be the first thing to think of.
great answer. that was very clear. thanks.
Welcome to this site yimon. This is a very general question, probably too general to give a sensible answer. Can you narrow it down? For example, if you have a specific research project in mind and are trying to decide between these two, then it may help a lot if you describe that research project.