tttjjj/elt_bench huggingface dataset:
(+1) Nice question. I've seen work on similar problems from a few years ago; I'll try to find it and dig it out of my notes. It would be a very nice addition if you could add some detail to explain how you can randomly sample an outlink of a given vertex without knowing the corresponding outdegree! Such additional detail may help readers provide better guidance. Welcome to the site. Cheers.
Rainfall?  Sister!  (I'm analyzing rainfall patterns from ~100 stations in the desert southwest of the U.S.).  On a more appropriate note, there are at least two different physical mechanisms.  One is rainfall generated by frontal systems moving across the area, e.g., cold fronts from Alaska or warm fronts from Hawaii, and the other is thunderstorm activity.  Depending on where the station is located, the larger rainfall events may be primarily generated by one or the other, e.g., just east of San Diego there's little thunderstorm activity, near Phoenix big rains are almost all thunderstorms.
Joel, I've noticed you've asked several questions (and gotten several answers with upvotes) but never accepted an answer. Consider upvoting and/or accepting answers you've found helpful.
"*Unfortunately your mission is doomed to failure since you are restricted to R and SPSS.*" -- this comment, unfortunately goes too far; **any** Turing-complete language can implement any algorithm written in any other, even if it is only programmable in machine code by flipping switches & is entirely implemented in Lego. There's no calculation that can be implemented in one that can't be done in another. Unless you're claiming magical properties for AUTOBOX, I believe what you probably mean to say something like "already available as functions in the vanilla distribution" as the difference.
+1 The viewpoint added in the recent edit is especially appreciated
@Tomas Someone edited my answer and changed R to WinBUGS. From what I recall, the book provides R code.
People who have done a lot of numerical analysis with matrices routinely compute their [condition numbers](http://en.wikipedia.org/wiki/Condition_number) to check for potential instability in operations like inversion.
Once you have computed (6), it is a mere function of $\\theta$ you can optimize.
Sorry, but it is not clear what exactly your question is, and as stated it is too vague and un-answerable. There are different aspects of financial assets that are modelled depending on what you want to know, and what the characteristics of the particular financial asset are.
I don't think his comment is that it is not possible, but rather that it might not make a ton of sense. The best thing to do would likely be to compare the 20 that COMPLETED to the 11 that DID NOT COMPLETE. This is, after all, likely the question of interest, is it not? To see if the 20 that COMPLETED differ from those who COMPLETED and DID NOT COMPLETE is unusual, since the second group would include the same individuals as the first.
@Alecos Notice that the PDF of the Beta$(\\alpha,\\beta)$ distribution, $x^\\alpha(1-x)^{\\beta-1}\\frac{dx}{x},$ can be rescaled by $1/\\beta$ to yield $(\\frac{x}{\\beta})^\\alpha(1-\\frac{x}{\\beta})^{\\beta-1}\\frac{dx}{x}$ and that for large $\\beta$, $(1-\\frac{x}{\\beta})^{\\beta-1}\\approx\\exp(-x)$, giving the PDF for a scaled Gamma$(\\alpha)$ distribution. By matching expectations you get a remarkably good approximation; for $\\alpha\\le 10$, the absolute difference in CDFs is comparable to $1/\\beta$. So maybe there is some merit in an approximation if one finds Gamma integrals easy to compute.
It is known in the MCMC literature that when a Markov chain is geometrically ergodic, it has exponentially fast alpha-mixing decay. I am unclear how X_{n} could converge rapidly to the target distribution and yet maintain high correlation between successive samples. Are there any simple examples? Thanks for any inputs!
Are the species different in means or SDs?  If so, if you leave the difference between species in the analyses, you will reduce power.
You can ignore all constants of proportionality in *any* probability density function because at the end you know it has to have unit total area. For instance, it suffices to know the PDF of the standard normal distribution is proportional to $\\exp(-x^2/2)$; finding the constant (equal to $1/\\sqrt{2\\pi}$) is just an exercise in integration. That's all that's going on here. Answers to the rest of your questions are available here by searching on keywords like [conjugate prior](http://stats.stackexchange.com/search?q=conjugate+prior).
You came fairly close--perhaps by chance. The true answer is $\\frac{1}{108} (871+960 \\log(2)+405 \\log(3))$ = $18.345919\\ldots$. Your code has two problems: (1) the iterations are not mutually independent; and (2) to obtain reasonable precision, it ought to be coded to be faster. Why not do the simulation directly, as in `n <- 10^7;
distance <- sqrt((runif(n,0,30)-runif(n,0,30))^2 + (runif(n,0,40)-runif(n,0,40))^2)`. That will get you about four significant figures (in less time), as you can check by computing the standard error `sd(distance) / sqrt(n)`.
I've changed your second RHS term to $by_{t-2}$, is that correct?
It is no surprise that the variance exceeds the range, nor is it meaningful of itself. The variance is a *squared* unit of measure. So, for instance, you could measure clay (with a range of 0 to 100) as a percent and have a variance of 10 percent *squared* yet the same data for clay in parts per million would have a variance of one billion ppm *squared.* If you use a sufficiently small unit of measure, the variance will *always* exceed the range. Also, kriging logs is *quite* different than kriging values, because weighted averages of logs correspond to *geometric* means, not arithmetic means.
@Tee Kay $\\epsilon$ is fairly standard notation for random additive components indexed by time $t$, as suggested by how this post calls it the "error term" and describes it as possibly having a normal distribution.  For instance, [the Wikipedia article on ARMA](http://en.wikipedia.org/wiki/Autoregressive%E2%80%93moving-average_model) uses the same notation.
@user1134241 You mentioned "empirically observed", I assumed that you have data from this stochastic sequence. If you want to estimate P(X$_i$=x|X$_i$$_-$$_1$=y) for each index i-1 where X$_i$$_-$$_1$=y, count the number of times X$_i$ = x and divide it by the number of times X$_i$$_-$$_1$ = y (regardless of what X$_i$ equals). That is an estimate because the observed finite sequence is just a sample of a portion of a sequence of the stochastic process.
This comment can represent, briefly and inadequately,  the point of view that this will often prove futile if not dangerous. The success or failure of a model is best regarded as the result of a team effort by the predictors (and their particular functional forms, interaction terms, etc., etc.) and is to be judged as such. Naturally, most of us are interested in relative importance of predictors and it is not nonsense, but attempts to quantify it exactly need to be accompanied with full statements of the technical and philosophical limitations on such an exercise.
@Yekver, usually logistic regression is solved using [IRWLS](http://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares). It involves solving least squares problem in each iteration and it is guaranteed to converge. Why did you not use this method?
@Glen_b - yes, a Pareto distribution.
A deposit in my bank account would be great as well! Just kidding, let me have a look, no promises.
I did print the loglog of the distribution. It's a flat line. How does that help though? What would you look at? The reference for the paper is great, thanks.
That's an amazing answer! Thanks a lot
Easiest is to calculate the probability that out of seven cards none of them are spades and take that away from 1.
1. The test statistics is likely to be Pearson's chi-squared, which is always positive. 2. Not sure what the question means. :(
If the values are proportions, then the variances are derived directly from the means. Offhand, I am not sure of the difference (if there is one) between prop.test and t.test, except for the interface. You could, of course, examine the code.
possibly better on cross validated?
Could you please explain it a bit more? I still do not understand what's the problem...
If the transformation can be different at each point, there's no reason to expect continuity. I'm not sure how one even defines a slope in that context.
I did not downvote, Michael, but I am inclined to convert this reply into a somewhat mysterious comment to the original question. I am puzzled why you are invoking a "linear model" and "a partial $R^2$" when the question appears to concern a 2 x 2 contingency table and, by virtue of its [tag:homework] tag, it begs us for *elementary* and *clearly explained* responses.
Everyone is getting the same treatment in all clusters. We don't have any active control clusters - the control has already been established as the 15 point increase I mentioned in point 3.
Take a look here as well http://stats.stackexchange.com/questions/58941/ncvtest-from-r-and-interpretation
I do not have a copy of The Bayesian Choice (or any of @xian's other books, which probably reveals me to be a non-Bayesian :)), but I don't think there is a substantive difference. I for one have always used the subscript.
(+1) Welcome to the site EduEyras. Curiously enough, that question was posted by the same user as this one! :)
I think you can put the continuous variable into the covariates box.
@aix I think your intimation is correct; I would need an indexable skip list, not just a sorted doubly-linked list.  The idea is to have a data structure that permits the insertion of one element, the deletion of one element, and finding the median in expected O(log(n)) time (or better).
Is this a homework or self-study question? Wording of first paragraph hints at such. If so, please flag as such and study our guidelines on self-study questions.
xIs this based on the silhouette measure of cohesion? If yes its values lie between 0-1.  The diagram produced in SPSS is far clearer than this one shown.
+1, the point about $0.07\\approx 0.03$ is an excellent one.
Can I migrate this back to stackexange? I am getting 0 response here.
It is not necessarily the case that Likert scaled items are Euclidean; change the coding from 1-2-3-4-5 to 1-4-5-6-7 and you change the distance, and that's OK if they are only ordinal.
If you have more assumptions on your data, for example a model of how it changes over time, then you can include that information in a ML or MAP classifier.
Thanks for the paper, looks very useful and interesting
Thank you very much! ^_^
without providing additional background information such as outputs from proc arima, your question is unanswerable. You are asking to interpret a chart with no chart provided. I would recommend you to read the following blog by Peter Flom on how to ask a statistics question. http://www.statisticalanalysisconsulting.com/how-to-ask-a-statistics-question/
@user777 thank you,it was very helpful, it's a great opportunity for me to finally start learning Latex!
@I'm satisfied with the solution for the example problem, but the question is how to find such ordering in $R^{1000}$? It is not said that there will exist one attribute that You can use to order vectors. It might be some a linear combination of all attributes or such transformation might even not exist. The question is how to find a transformation that will minimize inversions on the training data?
MANOVA compares the *mean* of the variables X, Y between G, P taking into account correlation of X, Y. It is a different think to compare the *distributions* of X, Y between groups G, P. What exactly is your purpose?
Sorry, but I'm having real trouble understanding this. Maybe some example numbers would help?
Umm, the hinge loss coefficient? I've mostly found it called C, some people use gamma but that confuses it with the radial basis coefficient. Basically the coefficient associated with the slack variables in the objective function.
Re 2nd paragraph: two different tests can be of the same null hypothesis but have different power against different alternatives. Even if you're not a Puritan, you can't allow yourself too much latitude in choosing/devising a test against an alternative suggested by looking at the data. So I think that's wrong.
The effort involved in your approach is far greater (and unless special care is taken, it's more likely to suffer numerical stability issues).
@jank, good point. The first reason is that I am not entirely sure how to do that. Second the data I have is going to get very sparse (in reality I have many birds and not a lot of observations per bird) if I analyse based on multiple features. Having said that, I would be very keen if you could demonstrate/explain how this can be done.
I'm not sure I follow what you mean `"treat it as a normal real value and round to the nearest integer category on prediction"`, but it sounds like an incorrect approach. Categorical valued data is treated similar to boolean data. It is a discrete variable and should be treated as discrete, not real value. Just assign integers $1,...,K$ as labels to your $K$ categories, then use the counts to estimate parameters of your [categorical distribution](http://en.wikipedia.org/wiki/Categorical_distribution) just as you used the counts to estimate the Bernoulli's.
Should be a community wiki
I'm very tired but I think that $a^T\\eta \\sim \\mathcal{N}(a^T\\mu_2, a\\Sigma_2a^T)$ hence $\\eta_\\star \\sim \\mathcal{N}(a^T\\mu_2, \\Sigma_1+a\\Sigma_2a^T)$
Thank you so much Glen! Your advice is always so complete and insightful! Thank you so much!
@NRH : I am very happy for your answers and references. The types of ncRNAs are those annotated in [Ensembl](http://www.ensembl.org/) database: snRNAs, snoRNAs, misc_RNAs, rRNAs and miRNAs; for homology searches. The objetive is identify putative ncRNAs in a Tunicate genome, not only by homology approaches, also by structural alignments. We hope report a in silico list of candidates to the different types of ncRNAs families, that may be classified based on [Rfam] annotation. Also, I want to use 4 different Blast strategies, suggested to search ncRNAs, for the homology stage.
That helps a little, thanks. Can you edit your Q to give an explanation of the background, situation, questions, goals, etc.? The blog post linked above will help w/ what I mean.
awesome, thanks for the research tips!
This Q has reappeared on the 'active questions' page as it was 'poked' by the 'Community' background process, one of whose tasks is "Randomly poke old unanswered questions every hour so they get some attention". I don't think this question *deserves* any more attention. It's community wiki, it received only one up-vote, got a reasonable answer on Aug 26 but the user who asked it was last seen on Aug 24, so it seems unlikely that user is ever going to accept an answer. I suggest it should be closed by a moderator.
The excellent article you allude to is @Article{wai06fin,
  author = 		 {Wainer, Howard},
  title = 		 {Finding what is not there through the unfortunate
binning of results: {The} {Mendel} effect},
  journal = 	 {Chance},
  year = 		 2006,
  volume =		 19,
  number =		 1,
  pages =		 {49-56},
  annote =		 {can find bins that yield either positive or negative
association;especially pertinent when effects are small;``With four
parameters, I can fit an elephant; with five, I can make it wiggle its
trunk.'' - John von Neumann}
}
@whuber I am asking about correlation arising from the least squares estimation procedure. If they are slight and inconsequential, I would like to know why.
Although the argument is correct, in practice (within most sizable populations) the distribution of children's ages will be close enough to uniform that reasonable estimates of age can be made based on height. Maybe, then, it would be more constructive to say "In short, *yes*, provided you look up additional information or make some assumptions," and then describe how to use that information to achieve the desired results.
I think this is pretty close to what I'm looking for. Thanks! Embarrassingly, I have that example tagged in the first edition.
A monte-carlo simulation is not a bad idea (because it can also obtain an estimate of the entire distribution), but to estimate the mean ratio to better than three significant figures requires over $10^6$ iterations and unfortunately your code won't scale well at those sizes. Consider rewriting it to take advantage of `R`'s built-in vectorization of calculations: that will speed it up by an order of magnitude at `ns=10^6` and progressively more for larger numbers of iterations.
There are many many examples out there already, which you can easily find by googling.
Even though you assert that they're uncorrelated, nevertheless, what does the cholesky decomposition (or SVD) of X'X, or the little-R part of QR decomposition of X look like? In particular, what are the diagonals of the cholesky factor, or of the sigma-matrix or of the R at elements between 140 and 150 or so?
@Drey $\\phi$ being the CDF of $\\mathcal{N}$
@Macro, no I understand that people have looked at LD for multiple testing correction (and FDR is part of that). My question is are there specific ways to identify the false positives getting low p-values in independent tests due to LD? Or approaches other than penalized regression to identify only the strongest markers within strongly correlated marker sets?
It is unclear what the sum formula has to do with the clustering or even what "clustering" is.  Here's one interpretation.  Suppose the cluster sizes are $n_1, \\ldots, n_k$.  An obvious way to generate random partitions with these sizes is to permute all $N$ items and assign the first $n_1$ to the first cluster, the next $n_2$ to the next cluster, and so on.  Given its obviousness, I presume you have rejected this as a solution, but why is it not a solution?
If you however want to get an estimate of the hazard or the survival for a particular value of the covariate vector, then you do need an estimate of the baseline hazard or survival. The Nelson-Aalen estimate usually makes the job...
Essentially all forecasting methods do better in the short-term.... The only exceptions are whenever the long-term state or value is inevitable, and here predictive success is a property of the phenomenon, rather than a mark for the method. So, I don't expect to be around in 2113, but I'm very uncertain about 2033.
I suppose it depends on (a) what you end up wanting to do with these quantities and (b) what you mean by a "matrix inversion". Oftentimes you avoid matrix inversion by simply solving the right system of linear equations using a convenient decomposition. Here, worst case, you can use the SVD by writing $\\Sigma_{xx}^{-1} = U D^{-1} U'$ where $\\Sigma_{xx} = U D U'$, so only (explicit) inversion of a diagonal matrix is needed.
Very interesting question and application. I could be missing something but it seems to me this model is over-parameterized. The correlated errors $ϵ_{i1},ϵ_{i2}$ can effectively be factored into a an "unshared" component and a "shared" component, the latter of which has the same function as $δ_{0i}$. You'll have to either delete $\\delta_{0i}$,  make the $\\epsilon$'s iid errors, or impose constraints like $r^2_{12} = .5$ for identifiability - are you doing that on purpose to decouple environmental/genetic components to sibling correlation?
For the history page this is fine. But for the current busyness i am looking for something that would relate to the real world. The color coding red for busy and green for free is nice but it is too basic.

I am not sure but for an example i can say - there can be 3 intervals say - busy, free and intermediate. So maybe for busy status i could show a cheetah running, for a free i could show a tortoise and intermediate maybe a deer or something.
I am really liking that book. I wish it included R-code as well.
@whuber OK, how about "if the assumptions are met and you apply both a z-test and a t-test, sometimes the z-test will reject the null when the t-test fails to do so". In practice, of course, with large n, the 2 distributions are virtually identical.
This question is answered in many threads: please [search our site](http://stats.stackexchange.com/search?q=multiple+regression+sign). I have selected one of these as the apparent duplicate.
Here's a hint for the general question: consider $a=10$ with probability one, so a constant variable, and $b$ uniform on, say, [0,1]. Note however that although this answers your general question, it does not answer your main question since Z and X are dependent.
Maybe google 'power analysis'?
Yes, I understand that, but a standard Newey-West estimator doesn't. It assumes that you have a time series model. Proximity in time, which is proxied by proximity in location in the model matrix, is how it models the correlation structure. If there is no correlation structure, then there should be no non-zero off-diagonal terms in the variance covariance matrix. The Newey-West estimator will converge to this result.
As your new illustration shows, when you are too aggressive in the filtering you lose obvious trends, too: your *smooth* is so strong it misses almost all of the last third of the data. Regardless, this definitely is not a decomposition into the qualitative notional components described by the Wikipedia article you reference. Another concern with the filtering approach (however it is conducted) is the statistical one: how do you *test* whether there is a "trend" and whether it is up or down?
Read http://stats.stackexchange.com/q/79043/3277. PCA and SVD are not comparable at all. PCA is a data analytic method. SVD is a mathematical operation. PCA is often done via SVD (BTW PCA does not necessarily remove means). Some other analytical methods, similar to PCA - Correspondence analysis, Principal Coordinate analysis, Procrustes rotation etc. - use SVD also.
Yes. (This is an unusual use of the chi-square test, but I think your approach works for your purpose.)
If your counts are typically high, the usual diagnostic approaches to checking the model's variance assumption apply (i.e. to look at one of the types of standardized residuals for GLMs and do a display suitable for checking that those have constant variance much as one would with linear models).
If the fitted line does not have a negative slope, the best you can do is a zero slope, which will pass through the point $(x,y)$, thereby uniquely determining it.
@Peter Ellis: I still not sure what dnagirl wanted, so I just wrote according the current situation. If she needed not only p1, p4... that's another case, then we can rewrite the commands :-)
+1. Thank you for your answer; it is a little over my head, but I will be looking into it as a possible alternative. I have moved forward with these models using a Gaussian link as the simplest method available that I can reasonably explain. :)
for the seasonal predictions you may use the latest updates in `forecast` library http://www.r-bloggers.com/major-changes-to-the-forecast-package/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+RBloggers+%28R+bloggers%29 many thanks to @Rob J. Hyndman as always. And try the improved `auto.arima` Your error says you have a unit root problem, seasonal differencing will help hear (strong seasonal part equivalent to seasonal unit root). And also a good non-parametric forecast could be done by `stlf()`
Thank for the clear answer, exactly what I was looking for.
In that case you will have to modify your data frame prior to using glm. This question should be reworded and posted to StackOverflow where programming questions are asked
@onestop I did test the scale by weighing myself repeatedly, and it seemed quite accurate. I suspect part of the problem is that I multiply two measurements together. That would frequently lead to both measurements erroring in the same direction, and thus, I believe, the multiplication would magnify the error. If it did this in one direction one day and the opposite direction the opposite day the change would end up being quite big.
@cardinal They're up now. thanks