tttjjj/elt_bench huggingface dataset:
@Michael, have you ever witnessed a productive back and forth that begins with "why was I downvoted"? I sure haven't. That's why I (and several other members here) discourage even starting the conversation, regardless of whether or not you think it's justified. It's pointless and generally leads to extended off-topic discussion.
I believe the problem with the argument given is that the events considered are not independent. For example, consider a single block. If I tell you that (a) there is no run of eight that starts at position 6, (b) there *is* a run starting at position 7 and (c) there is no run starting at position 8, what does that tell you about the probability of a run starting at positions, say, 9 through 15?
(+1) Indeed, this is precisely the example I keep in my head when thinking about Jensen's inequality so that I don't accidentally get the inequality in the wrong direction.
Do you have the "correct" data to CV with?
You have to decide whether ($1/3$ $1$st, $1/3$ $2$nd, $1/3$ $3$rd) is better or worse than ($1/6$ $1$st, $2/3$ $2$nd, $1/6$ $3$rd), or ($1/2$ $1$st, $1/2$ $3$rd). We can't make that decision for you. If you have a linear evaluation of each place, then you could try evaluating bots by their expected values, and you can test significance after estimating the standard deviations of those values.
@IngridCharvet  If as your comment above says the points removed were those with values greater than a threshold, this is a key point.  I suggest that you edit your question to make this clear and, if possible, indicate why it was decided to remove those points.  This may help you to obtain answers addressing the particular case you are interested in.
Are you expecting the errors to be additive ($y=ab^{x_1}c^{x_2}+\\epsilon$) or multiplicative ($y=ab^{x_1}c^{x_2}\\epsilon$)?
Thanks. The parameterization I am using is also from Gelman.  I'll give this paper a re-read to see if I get any ideas.
@MarcClaesen [If your primary modeling goal is predictive accuracy](http://appliedpredictivemodeling.com/), theoretical properties of individual models is far less important. In this case blending is often an excellent approach: the strengths of one model can make up for the weaknesses of another.
1.(a) Sometimes when the variables are measured on comparable scales the directions of largest variance are thought to be more interesting.
+1 Nice question; interesting link. (Since it's based off residuals from a fitted model I'd probably have called it a bootstrap test rather than a permutation test, but that's of no great import)
@Michael Everything is not ok, because *assuming the calculations were correctly done and the model assumptions truly are applicable,* nothing will change the evidence that the coefficient of $X_1^2$ is positive, which is in striking contrast to the expected negative value. It's hard to say much more until additional information is offered about what a "Poisson model" really is in this context and some details about the data are furnished.
Well, I chose any value for $\\theta$ because MA(1) processes are always stationary. But I didn't think of the consequences of non-invertibility.
Please consider formatting your question in a more neat manner. Your title is atrocious! See the [advanced markdown help](http://stats.stackexchange.com/editing-help), including how to use latex to format mathematical notation in your question.
I think that's already taken care of in the cph call (and works fine in other cases with other data, just not here...)
@Glen_b These are from a lecture I attended before. The results were simply listed without any explanation given. Do you happen to know anything, please? BTW, are you really 13, please?
Thank you so much! I'm new to R, and your write-up was v. helpful.
According to Wikiquote it is misattributed to Joseph Stalin; the origin is Kurt Tucholsky: http://en.wikiquote.org/wiki/Joseph_Stalin#Misattributed
The filenames like the regular ones you can see of bittorent sites and such (Minecraft 1.7.2 Cracked,Homefront (2013) DVDRip XviD-MAX, and so on ). I guess piracy is main source of files for my users.
Thanks @HongOoi. I haven't understood everything concerning the regressions though. SO make a regression, meaning that we look for the function minimizing the sum of squared residuals. something like $respond = a*predictor + c$ and $respond = a*predictor + b*predictor^2 + c$ for the linear and quadratic models. Correct? Then, what is the likelhood function?
But the normalizer is $B$ , which is already in the form of $\\sum A$, which I understand. How did he get the term in the numerator, i.e. $A$? That is the part that is confusing me
Gosh, I wish you'd paste the *actual* output from the model. You've really butchered this so trying to reconstruct the issue is difficult, to say the least!
Analytical answers are preferable, but given its not homework, I am curious why you don't use brute force Monte Carlo simulation.
It depends in part on the potential consequences of either a type 1 or 2 error, particularly in a safety context
I don't know the answer but In think this is a great question. It is hard for me to think of better approaches than the ones you have mentioned. i do think there are differences between the approach to univariate kernel estimation compared to multivariate. This book by David Scott might be helpful regarding the multivariate kernel approach, though I am not sure he discusses peak hunting.  http://www.amazon.com/Multivariate-Density-Estimation-Visualization-Probability/dp/0471547700/ref=sr_1_1?s=books&ie=UTF8&qid=1344016374&sr=1-1&keywords=multivariate+density+estimation
For an example of a highly skewed dataset where the t-test nevertheless *is* applicable (because of the large amount of data), please see http://stats.stackexchange.com/questions/110418/is-bootstrapping-appropriate-for-this-continuous-data.  These two cases together show there is no cut-and-dried answer to the question: you have to consider both the distribution of the data and the amount of data when deciding whether the t-test will be meaningful and accurate.
The decision should not be based on the outcome of a formal hypothesis test, which answers entirely the wrong question. Your question of interest here is about how much impact the non-normality in your data will have on your inference, and hypothesis tests don't speak to that at all. Indeed, they're most likely to reject when you have a nice large sample size... in which case, for things like ANOVA, say, the non-normality may barely matter. If your sample sizes are small, you may have highly non-normal data (and a big impact on your inference) but little power to reject it.
Principal Components Analysis?
Are you in 3d space then?  Or only looking at the surface of the brain?  I'm sure that you could probably generalize the "Coneley estimator" to 3d space, but I haven't seen it done.  If you can pull it off you should publish it somewhere.  Note that the code takes geographic coordinates as input, which are modified into distance based on trigonometry.
I answered a generalization of this question at http://stats.stackexchange.com/questions/3961, but I suspect for your special case their *might* exist better solutions.
I have more than 1000 samples. Is this indicating that my sample is bad?
I know in reality it is not. However, that's the way i is coded in the data I have.
It is OK to use standard CCA with data which partly or all are binary variables. Since a binary variable has only two levels, if behaves identically whether it is seen as numeric or categorical. At least with analyses based on correlations - such as CCA.
This is a topic called power analysis. The sample size needed depends on how tight a confidence interval around the estimate you need. It also depends on the nature of the variable "perceived satisfaction" and its reliability.
@Leo: Yes. Continuous-time stochastic processes provide an example. The canonical example is Brownian motion, where the sample space $\\Omega$ is taken to be $\\mathcal{C}$, the space of all continuous real-valued functions.
Hint: Fix a value for $x$, say $x = 1$.  Put in limits $\\theta=0$ and $\\theta=\\infty$ on the integral on the first line, and ask yourself the question: When $\\theta = 0.5$, what is the value of the integrand $f(1|\\theta=0.5)\\pi(0.5)$? Similarly, what is the value of $f(1|\\theta=0.9)\\pi(0.9)$? Then determine whether the limits on the integral on the _second_ line are $0$ and $\\infty$, or something else.
Added code for what my data looks like.
Actually, $\\ln x,\\mu$ and $\\sigma$ have the same unit.
@Tal: Fully agree! I think the whole area on optimal separation in minimax testing is starting from this idea ... and it is still so confused for a lot of statistician. For those interested see the paper of donoho http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.aos/1085408492 (and the references in the paper ! since things are much older than donoho's paper)
@ EpiGrad: Stratification score (a retrospective balancing score) is the probability of disease (outcome or "case-control status") as described in a recent paper [1]. Is stratification score better than propensity score in a case-control study? [1]: http://www.ncbi.nlm.nih.gov/pubmed/21402731
I throw a fair coin and record its outcome with a binary indicator ($X_1$).  You throw a fair coin and similarly record its outcome ($X_2$).  Is it not obvious those two random variables differ?  Nature will not guarantee they always produce the same result, that's for sure!  Yet they have identical distributions and are independent.
As far as I understood, zero-inflated binomial regression is for count data with excessive zeros. This model is for binary data (yes/no) and the class distribution is not necessarily very skwew. 
From what I have read, I think a mixture of binomial models is never identifiable, but I am not sure. Also, I am wondering, if the above is true, whether there are workarounds for that (extra constraints or something).
@kwak: why not post your comment as an answer?
(I can see a reason to set the derivatives equal, but if you're doing it for a different reason, that would be important to know). Oh, and is this for some subject?
Is this regression or time-series (you have the time series tag)?
You are looking for [sparklines](http://en.wikipedia.org/wiki/Sparkline). Searching in R for `??sparkline` shows that the packages `sparkTable` and `YaleToolkit` could be helpful.
Thanks for immediate response...Yes I need help with SPSS specifically. I did alot of reading and i am getting more and more confused.. While doing analysis for interactive effect between MSG and TFA should I exclude the ST and MSG+TFA dataset and then do the model? I tried the multiple linear regression and repeated measures, but I still cant figure out where to look for the interactive effect..
@Glen_b I didn't see any reference I can use for this article. Do you have any idea of this?
It wouldn't hurt to ask questions about the papers on the topic that you found and couldn't understand, in case you want help with those too.
**Request**: The proof I've given above is my own. I worked on it out of enjoyment, since the problem intrigued me. However, I make no claim to novelty. Indeed, I cannot imagine that a similar proof using a similar technique doesn't already exist in the literature. If anyone knows of a reference, **please** post it as a comment here. I would be **very** interested to know of one.
@IrishStat: In the future, can you use the formatting options? (the correct key to insert links is ctrl-l, not ctrl-c).
Welcome to the site! Please give your questions more informative title (so that it can gather some attention) and update your profile instead of signing the question.
I don't see what ellipsoid you are talking about.
How would plm help?
Without knowing local variation in wages ($W_{it}$), it seems the best you can do is integrate over $i$ in $D_{it}$ and model the time series on equal footing. You cannot infer local information from global information without assuming some model of how the global predictor is distributed locally.
So is this basically similar to the F-Test in regression?
Many thanks for your comment - even if incomplete this is clearly useful. I was also unsure how to handle the rotations - maybe you've seen it already but these are demonstrated in a 2x2 matrix in the thread I linked too in the [original post](https://stats.stackexchange.com/questions/9898/how-to-plot-an-ellipse-from-eigenvalues-and-eigenvectors-in-r), although in a different example that I quoted. I will take a look at implementing what you suggest today and seeing if I can resolve this remaining question. Thanks again.
Quick note: this improper prior is known as Haldane's prior after John Burdon Sanderson Haldane, a British-born geneticist and evolutionary biologist.
@January, "not compatible" is too strong. [Bayesian Adaptive Methods for Clinical Trials](http://www.amazon.com/Bayesian-Adaptive-Methods-Clinical-Biostatistics/dp/1439825483) has a lot of information about computing Type I errors under Bayesian adaptive designs.
+1 That's an impressive blog post.
This frames the situation reasonably well, but I don't understand where the "groups" come in. All I can find in the question itself is a reference to $50$ *tissues* and the question seems to be asking about the distribution of concordance scores. It gives us little to go on concerning what the "expectation" of that distribution would be, but it might be fair to interpret the ultimate phrase in terms of a permutation test of outcomes A and B. Would your "groups" perhaps be synonymous with "outcomes"? If so, the data might not permit identification of the groups if only the scores are known.
if your data is non parametric you can use excel to rank the data however is correct then use R to run Spearman
I would say it is often intuitive, but unfortunately hidden by a thick coating of arithmetics and mathematical notation... :)
If you can fit your dataset in memory of a single machine, I wouldn't call this a "Big Data" problem, specially if you do this in the title of your question
I don't know r, but from reading it, it seems like you're just using a general-purpose nonlinear optimizer over the vectors (which could do gradient descent or something else), right? (Similar to [`scipy.optimize.minimize`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html#scipy.optimize.minimize)?) I guess that works (+1), but I was hoping for something more insightful and possibly faster than just plugging it into a generic black-box solver.
Thanks for modifications Jeromy.
The scores were standardized before adding them.
Welcome to the site, @stephen. Note that we are not here to do your analyses for you. In particular, 'plez send me the codez' questions elicit a good bit of annoyance. See if you can edit your Q to clarify substantive questions you have that don't amount to asking for someone to analyze your data for you. It may help you to read through our [help page](http://stats.stackexchange.com/help), which contains guidelines for questions. Otherwise, this question may end up being closed.
@Glen_b  Not a textbook question. My textbook simply states that the strictly increasing CDF assumption can be relaxed by using the quantile function. It was not a textbook exercise but a general query. I will nevertheless use the tag from now on.
AIC implements Occam's razor?
What's wrong with random sampling?
Point taken. I changed the question to express it in terms of the means.
@user4701 Why are you not supposed to follow the 7-point-mean,t-test approach? Could you cite the paper?
Please explain the abbreviations CFA and EFA
I think it's pretty relevant here since the question deals with linear regression and the request for advice is much more relevant here than in math.stackexchange (you know, mathematicians are comfortable with tensors but in a more abstract setting so they don't have to struggle with this kind of calculation.)
Yes, the statisticians know what to think of p-values. We (the biologists) usually lag a few decades behind when it comes to statistics :-) randomization is a good idea, I will test it.
(+1) Would there be any advantage to using multivariate regression (like Stata's `mvreg`) since the RHS variables are all the same? I think it should be a bit faster. Also, `sureg` carries out a $\\chi^2$-based test, while `mvreg` does an F-test in this case.
This is an interesting & creative proposal, but I'm skeptical of it. In both your version & in ordinal logistic regression as suggested, eg, by others on this thread will require the proportional odds assumption. However, OLR will allow the thresholds / cutpoints b/t the categories to vary more flexibly, I believe, whereas they will be determined by the binomial parameters $p$ & $n$ in your scheme. This assumption would have to be verified against the data, & I suspect, would lead to problems. (BTW, the -1 did not come from me.)
@whuber ? I have read it I'm asking if i can use that pattern 1 -1because I'm testing the residuals of a linear regressions
Basically what you have is a distribution that is the sum of 2 component distributions. If these were both normal, this would be trivial. Do you know what the other dist is? In general it would help if you could provide more details about your situation, your data & your goals; reading this blog post might help: [How to ask a statistics question](http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/).
I have edited my answer to answer these questions
You *can't* compute it exactly from the information given - it can be any value between the two values either side that you do know, though the additional information (such as IQR) might in some cases restrict it further. If you make additional assumptions you may be able to say more.
So, does the formula not imply that the graphical model has a time component. The map is a deterministic system and evolves to generate a time series for each concept/node. Is this the same mechanism for markov chain ( I am unaware of the details about Markov chain). To me both look like a state diagram.I shall be really grateful for a detailed insight.
It would be helpful to hear a bit more about the nature of your data and the sort of test you wish to carry out; it sounds interesting.
The method is chi-square test of **independence**, not that of **goodness of fit**, as I have no idea what $D$ I should test for. Ordering the categories doesn't quite work either. There are 67 categories in my data, trying each ordering for these categories is computationally unfeasible. However, thanks for the effort for answering (the diagrams are awesome!)
If you are trying to predict, I would use the model even if there is heteroscedasticity. You can use a holdout sample to see how well your model predicts. By contrast, if you are trying to provide evidence for a theory about how your variables are related, I would be a bit more careful. The standard errors will appear larger than they are. See if you can make the data more homoscedastic with some transformation. In network analysis, you sometimes have some very skewed distributions  and a log transformation can help. Good luck.
+ Thanks, but here's what bothers me about that. Suppose $S=N=1$. Then $F\\sim\\beta(2,1)$ whose mode is 1 and mean is 2/3. That's counterintuitive because it could be that a large fraction of the stones might all have different colors, so I really just picked one of those colors at random.
"we could only interpret β as a influence of number of kCals in weekly diet on in fasting blood glucose if we were willing to assume that α+βX is the true model": Not at all! Such is the importance of avoiding causal language. Even if the trend were curvilinear, the interpretation of $\\beta$ *is* a population averaged parameter which estimates the first order trend of the kCal and glucose association. $E(u|X)=0$ even under curvillinearity for fixed $X$. As we know, 0 covariance does *not* imply independence.
Part of your question seems to have been lost while you were editing. PLease edit to clarify your question.
@Mustafa, I previously thought the weights are known, so more details about your question are also helpful. I edited my answer, and provided the link about how to calculate the variance $\\sigma^2$.
... However this might well not be the case for different reasons such as biases in the sequencing process due to the chemistry involved, actual genomic changes in the specimen as compared to the reference, limitations in the algorithm used for mapping these short fragment to the reference... So one thing that we want to investigate is the reference coverage bias introduced at different steps of the process versus some properties of these short DNA fragments, the random x variable. A notorious example is the G+C nucleotide content of the fragment.
I doubt there's a closed form for this.  It does have a curious characterization, however: when $X_k$ are iid Poisson($\\lambda/2$) variates, $k=1,2,\\ldots$, then $1-F_{min}(\\lambda)$ is the chance that all $X_k \\le k$.
One thing to note here, is that it doesn't matter if the DV is normally distributed, only if the residuals are (see here: [what-if-residuals-are-normally-distributed-but-y-is-not](http://stats.stackexchange.com/questions/12262/)), & even then, with a large sample size the central limit theorem will cover you. You are likely to be fine.
You should always try to give some context to your question!
In your last line, do you really mean "... the $\\theta$s to generate a sample of $p(\\phi|y)$" (switching the $\\phi$ and $\\theta$)?  It would go better with eq. 5.4 above...
Additional answers: http://stats.stackexchange.com/search?q=random+diehard.
After some additional thoughts, I changed the configuration of the 2 time series but I still have doubt 

https://www.dropbox.com/s/lvkku670tgg1b3a/ex_2.xls
I could run the code in Mathematica Environment, but since I'm not familiar with this topic, can you tell me how can I plot the impulse response of this filter? I just could plot the amplitude response.
+1 For estimating the distribution parameters, the sample sizes are large enough to overcome @Cardinal's (good) objections.  (You do need to account for the serial correlation in deriving standard errors or confidence intervals for the parameters, however.)  From the look of the illustration, though, a Weibull is a bad fit.
This is a reasonable (and sometimes effective) approach. However, it carries two risks that would be worth discussing. First, it can take a huge number of trials unless the threshold `e` is made adequately large. Second, the resulting sample might look nothing at all like a binomial distribution!  A modification of your approach that uses the methods suggested in comments by @Glen_b would resolve the second issue.
Maybe the section labeled "Predictions and/or confidence (or prediction) intervals on predictions" on this [page](http://glmm.wikidot.com/faq) might give you some hints.
Ah good idea, thanks.
@gui11aum That's a good point, than you.  I was hoping to circumvent a slightly more technical demonstration of the falseness of the second claim, but it looks like in so doing I resorted to a fallacious argument.  I'll edit that.
Closely related: http://stats.stackexchange.com/questions/74537, http://stats.stackexchange.com/questions/298
Refer to this presentation for Lewandowski alogorithm. http://www.youtube.com/watch?v=qKt2P1De6U8
+1 `R` computes the full distribution of the counts of "instances from class A" in the sample accurately even for such large parameters, as in `dhyper(0:10, 1000, 11000000-1000, 30000)`.
I forgot to mention that this feature doesn't normally distributed
@Macro Thank you for your comments. I asked the Q here since to me it felt more like a statistics Q than programming. May be I did not word it right then? In my case, it is the number of bee colonies in an apiary that are infected that I used as weights and the proportion of the number of infected colonies as the independent variable. In this sense, my usage of number of colonies tested as `weights` in a `glm` would be right, or?