tttjjj/elt_bench huggingface dataset:
*Standard errors* (due to MC sampling) will shrink, for the same reason that adding more respondents to a survey will shrink margin of error. But continuing to draw values from the posterior won't change the *posterior variance.* I think you're hung up on the idea that stationary means that there is a single, "correct" value. Bayesian inference is much more interested in parameters' variability, which is to say the posterior *distribution*.
Circle overlap is allowed, but the data points covered by both circles would not be double-counted.

Thanks for the pointer to neighborhood/focal operation on raster datasets.  I will search for something along those lines.
@Andrie. I have tried those but they do not work.
Hi Alexa. Welcome to the site. Please consider adding some more detail and context to your question via an edit. This should make your interests and objectives more clear to other users and hopefully encourage more positive interaction.
@user11852: Thanks for response.
$t$ usually stands for your test statistic, while $p$ is your p-value which reflects the probability of observing a result as least as extreme under the null. If p is low (i.e., below a given threshold, say $\\alpha=0.05$, defined before seeing the data), then when conclude that such an extreme result is unlikely to be due to random fluctuations alone.
2 does not make much sense, from the context it should be like 8 to 12 clusters. So probably I'll have a look at some other methods, thanks for your help and advice!
For some additional comments, pursuant to @Behacad's objections, please see http://stats.stackexchange.com/questions/23200/are-all-attributes-data-points-inherently-nominal/23266#23266.
@probabilityislogic I'm trying to optimize the response time for a website, and I've identified 5 factors believed to have an influence on that response time (such as thread pool size, connection pool size, etc). For each experimental run, I do a load-test and measure the average response times.
Calculating robust standard errors doesn't change the heteroscedasticity of anything.
@Leo 1. I think OP can just use all data to get a classifier which would lead to LDA. 2. OP can just recompute LDA with those vectors.  Paul, Is this right? I think it's a simple and interesting idea.
On the contrary, this is the *crux* of the question.  It's perfectly fine in statistics to posit a null that says the effect is -10%.  Your experiment will be able to reject it if it produces strong enough evidence to the contrary.  Note, though, that (barring extraordinary computational and conceptual machinations) you can test only a single such hypothesis per experiment.  Note, too, that it's the rare experimenter who knows so precisely what the effect size will be (but still feels a need to test it!).
Depending on your sample size, you could use bootstrapping to obtain CIs.
+1 This is mostly correct.  I'm not sure that the second paragraph is necessary, at least as-is.
Yes: First close the rod into a circle.  Then break it (uniformly, randomly, independently) at $n+1$ breakpoints.  This introduces a helpful symmetry :-).
Don't you answer your own question there? There is a common trend, we agree on that. Differentiating allows to look past the trend: How are fluctuations around the trend? In this case, the correlation between x and y happens with lag 0 and 8. The effect at lag 8 is also visible in the autocorrelation of ddf$dy. You would not know that without differentiating.
3. The native place of the patient and the detials of prior treatment were recorded at admission it self. This work was done with the help of an International Organizations. Migrated cases were taken as belonging to the state of their origin. 4. I would be extremely happy to clarify any other point.
I think my discussion is less equivocal than you think. (1) the t-test can be *arbitrarily* bad compared to the MW, the converse *does not hold*; (2) the MW is *very nearly as good as the t*, when its assumptions hold exactly; (3) the case where the $t$ has a (moderate) advantage - lighter tails - is rarer than the case where it can have a large disadvantage (heavy tails); (4) with small samples *you can't check how heavy tailed it is*, so just you don't know quite how badly you're screwed; (5) finally, there are a few side cases where it's not especially clear which is better.
Does [this paper](http://projecteuclid.org/download/pdf_1/euclid.aoms/1177728272) help at all?
Your question title is about measurement error; your final question is about bias. The two are not the same. You also ask simultaneously: what is the difference, and is there a difference? That said, here goes. The central idea of bias is that a procedure is wrong on average, yielding a mean that is higher or lower than the true or correct value, known somehow. That could be true of a measurement procedure, but it can apply where measurement error is not the question. That's what I understand from statistics; psychometrics may have a different notion, but I doubt it.
Thank you for a very thoughtful and instructive answer!
@miura: KKT stands for Karush-Kuhn-Tucker. The KKT conditions are certain equations that solutions to (sufficiently regular) optimization problems must fulfill ([wikipedia article](http://en.wikipedia.org/wiki/Karush%E2%80%93Kuhn%E2%80%93Tucker_conditions)).
Don't you want `operator|nozzle` random?
I presume this is homework? What have you tried so far?
Hi Natasha, welcome to the site. I added a link to your reference to the google book site, and removed the signature line. Generally we do not need signature lines for posts here, as you can leave this information in your profile. Thank you for the reference.
It's not clear, but you might be looking for *zero-inflated* models (which you can search about).
Dimitriy, thank you so much for the replicate. Could you please explain a bit on what you mean by "This makes sense since the residual for such observations will be exactly zero." in "reg PHARVIS ILLDAYS i.hh if inrange(hh,1,100), cluster(hh)"
@Mike, not exactly disallowed, frown upon is the better word.
Could you please tell me how you are normalizing the 2D-histogram and using to get the marginals? I haven't managed to this correctly. I've in fact opened up [another question](http://stats.stackexchange.com/questions/24568/normalizing-a-2d-histogram-and-getting-the-marginals).
@Ashok most often connection you are searching for arises from the description of convex sets with a probability measure on its extreme points (Choquet Theory).
See the related questions on the right hand side of the screen. Here is a duplicate question, [For a classification problem if class variable has unequal distribution which technique we should use?](http://stats.stackexchange.com/q/2131/1036)
Hi Analyst, in the project did you combine the results of multiple methods based on the same set of variable or not?
Thanks but it seems that your method works when we do know the value of the boundaries. In my case, I don't know their value, all I know is that the incoming data will fill a square when time tends to infinity  and I don't know how to exactly decide about when a point is not in the square since any given point may end up converging to the square in infinite time. thanks.
+1 Cause I like the analogy of the maps. I'll use it in the future!
AND .... where's the code?
If you've written your own its it's almost bound to 'fail' once you get past the limits of your knowledge (that you've embedded within the design). Have a look at http://www.random.org/analysis/ which list some of the checking methods, and obviously http://stackoverflow.com/q/2130621/717355. Philosophically, If the programme is deterministic it can't be random anyway ;-) It's definitely a worthwhile subject to study (and don't forget password hacking crypo discussions).
@David Khireche The code simply samples from a KDE in the usual way, then the consistency. $h$ is the bandwidth involved in this estimation/simulation, it does not represent a perturbation at all. You might want to read a bit on KDE if you are interested on this approach.
(ctd) Goodness of fit tests exist, which if they reject the null indicate that the sample isn't really consistent with a Poisson (though true Poisson data will once in a while do the same - in fact you choose how often that happens). Or you can use diagnostic displays which will show how close to consistent with a Poisson your data appear to be. A crucial question: why are you trying to do this?
The AIC (or the BIC) is used when you want to compare models. For example, model (1): Y regress on x1; model (2): Y regress on x1 and x2. Which model of the two will you choose? AIC and BIC try to give an "objective" answer: you pick the model that gives you the smaller AIC or BIC. For details, you can refer to wiki page: http://en.wikipedia.org/wiki/Akaike_information_criterion  When you know more about linear regression, you can learn how to deduce AIC/BIC by yourself.
@idoda looks like I made a mistake. Nonlinear training time goes in the order of hours for more than 500k instances on some data I have. For 100k instances you can expect 10-15 minutes per model (which is still **much** more than linear).
@nbrent I have written such a piece of software. It is called AUTOBOX and is available from http://www.autobox.com.
Then I think you can pool them.
@Glen_b thanks for the answer.
You may be interested in this recent article published in PARE, *Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon*, http://j.mp/biLWrA.
@eng The revelation about the meaning of $P$ and $T$ helps immensely.
Your equation for the correlation is not correct. It should be the covariance divided by the two standard deviations (not the variances). Your end result is correct though.
Generally when testing hypotheses sample size does not dictate the best test.  What is known about the data determines whether not parametric or nonparametric tests are preferred.  In some situations there exist uniformly most powerful tests.  Those tests are best for every sample size.  What chnages in small samples is that even the most powerful test has low power.
A starting point may be to simply regress $y$ on $x$ and $x^2$ including a constant. You could then apply robust regression techniques to this approach if is necessary.
Main concern against using K-Means, as you mentioned spherical clusters, is that the clusters are bound to be biased to similar variance. This might be troubling since this type of data is very likely to have uneven clusters.
Training a classifier is exactly as usual.
And there is no code for *manual* analysis of the clustering results, otherwise it would not be called manual. Again, don't use them blindly or as-is. Any clustering *will* have some errors that you need to fix.
@Procrastinator I'm trying to help a scientist communicate to me her beliefs about the gen. shape of a distribution. Many domain-experts find it easier to introspect their own beliefs in terms of percentiles. However, on the flip-side, many domain-experts are quite comfortable reading PDFs. What I'm trying to do is allow experts to input their speculation about the general shape of the distribution in a form they find easy to introspect, but check it against a form they know how to read: in this case, a PDF. A sort of write/read feedback loop.
@user946850: No, it's not homework. I am conducting my thesis on **Evolutionary Algorithm**. I want to use standard deviation as a measure of population diversity. Just looking for more efficient solution.
I think that's the point, though.  That is, the theories (Blackwell's theorem?) generally apply to asymptotic limits (i.e. in your case, the time to wait for the 2nd set of 55), whereas your simulation is just for the first one, so the theory won't apply directly.  Maybe you can simulate to see if the second set of 55's comes after 35 or 36 trials.  But as I wrote, I'm not an expert on these issues, so it will be interesting to see what others have to say!
Thank You. This is helpful
@Macro: I added simplified format of how I am formatting the data. My understanding of "collinear" is that there are independent variables that correlate to each other.
Bertin's graphical ideas were often extremely good, but reading his prose is often like swimming through mud. I don't blame the translations, either. I don't find that term at all evocative and I am not surprised it never caught on. (I have looked through Bertin several times and never thought it worth noting, let alone repeating.) But, but, but: If there is no good term, it's your chance to suggest a better one in your dissertation, at least for your purposes.
Looking at the Mekko website, they advertise "32 chart types". I guess you will need to be a little more specific as to which type exactly you want.
@IrishStat: Yes, I even have a comment in my code pointing to the fact that you said it was a bad model, I just thought some R code might help people by giving a basis to experiment with.
I'm more interested in a consolidated table or logic tree of information on when to use each.
It depends on the variances, if they vary accordingly. However, if sphericity doesn't hold (see [Mauchly's test](http://en.wikipedia.org/wiki/Mauchly%27s_sphericity_test)) you can correct the degrees of freedom or, perhaps better, use a linear mixed model.
In my personal experience (usually Windows/Linux combinations) there has always been an overhead. This may well be due to external reasons. For instance, in the context of the set-up in the question, I think the hypervisor and the underlying OS itself will need a lot of resources, so you won't have access to those. As you mention, I/O can be quite problematic as well. I consider this to be a big enough drawback to avoid using a virtual machine when dealing with big data operations on a daily basis.
Scratch the above. I *do* have a way of knowing if the implementation is working and even of quantifying how well it is working. It's a bit complicated and time consuming but I can (should be able to) do it.
This, too, is a trick that might not be what you are looking for. Also, I wasn't sure whether it was better to edit my previous answer or post this as a separate answer. However, based on http://meta.stackexchange.com/questions/28471/two-answers-one-question I decided separate was better as these are two different approaches.
If there is any way to do the analysis without dropping the day 8? I heard about the r software? what do you think? Do I need another software if I would like to carry the analysis without dropping the day 8?
(+1) In response to the votes to close as off topic: Apparently the basis for those votes is that the question appears to ask a purely software-related question ("how to plot such-and-such in R"), a question that indeed ought to appear on SO.  Note, however, that buried in the current reply are *statistical formulas* to create the plotting points.  This suggests there is statistical interest to the question, so I am reluctant to vote for migration.  A *good* reply here would highlight and explain this statistical point.
Nobody could explain me?
This does not answer the question. It is an useful comment on Bonferroni but should be IMHO (in scope of this question) a comment and not an answer.
Can you provide a link to the "online resource" in question? Note that the formula you list is just the OLS estimator for a regular (non-errors in variables) simple regression model.
Hint: What is $E(X_1+X_2+\\cdots+X_n\\,|\\,\\bar X)$?  How do $E(X_i\\,|\\,\\bar X)$ and $E(X_j\\,|\\,\\bar X)$ differ? (I suspect your intuition is really concerning $E(X_i\\,|\\,X_j,\\bar X)$.)
It would be good if you provide the source of your quote. Still, I found it by googling (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2964116/), but even after looking up the context, the quote does not make sense to me. I do not think this word means what the authors think it means.
@JMS Gelman et. al. 2003 made this point, and I have tried to summarize it, they seem to consider the noninformative priors for background / theoretical / heuristic purposes
@Samarithan Thanks for clear description. I am trying to using some regression on the data for travel time prediction.
@Articuno: The `e1071` package in R provides an interface to `libsvm` as well as functions for one-against-all multiclass SVMs based on libsvm.  For example `model <- svm(Species ~ ., data = iris, cost = 100, gamma = 1)`
1) The posterior in this region is much higher than the posterior evaluated at the earlier parts of your chain. 2) From your traceplot, gamma_tau and and delta_tau are zero, which means there are no species random effects, which is causing an identification problem. 3) I'm not an expert in JAGs syntax, but I think that this would tighter your prior, but also move it closer to zero.  Try a mean preserving tightening.
thanks, it's really helpful to be able to picture it
But nltk NB doesn't consider multiple occurrences of a token. From the literature I posted, "When classifying a test document, the Bernoulli model uses binary occurrence information, ignoring the number of occurrences, whereas the multinomial model keeps track of multiple occurrences." I'm able to calculate the conditional prob that nltk gets for d5 from that example using the Bernoulli model, but not the using the Multinomial model. It's binning features, but it's binning each single DISTINCT feature, not all features like duplicate words.
Except there is nothing special about it being an nth degree polynomial other than it lets you imagine that in some way everything is still being described by x.  You'd get just as far with n - 1 random variables as predictors so long as no two variables were perfectly correlated.
Absolutely +1 to what David Harris said. Do not rely on the default values. Running in a simple example -- using the mtcars data, trying to predict mpg based on cyl, hp, wt -- if I use the default lambdas (I don't specify lambda, nlambda, or lambda.min.ratio) I get non-trivial discrepancies. (Intercept 38.00658324 vs 38.75179, for instance.) When I put in 100 or 1000 lambdas the discrepancies between glm and glmnet shrink by 5+ orders of magnitude.
Why does heteroscedasticity in the *independent* variables bother you?  Are you using a mixed model in which they are treated as random variables?
@AndyW: Thanks.
@rolando2: agreed that if the OP has many systems, average them and then state how far each machine is from the average. I'd recommend also calculating the standard deviation for each attribute, so that readers can determine if System 32 is really out of line on Attribute A, or not.
Your question is unclear. Please explain what you're discussing, expand abbreviations on first use, and otherwise make it as obvious as you can to use what is going on. Are you talking about a Markov decision process?
@Glen_b: Thanks! Should  $T(X)$ be equal to the p-value based on $U(X)$, for any sample $X$? If I understand correctly, I found it on page 5 of [this slides](http://www.biostat.jhsph.edu/~iruczins/teaching/140.615/notes/n.pnp.pdf) So the benefit of using permutation test is to compute the p-value of the original test statistic $U$ without knowing the distribution of $X$ under null? Therefore, the distribution of $T(X)$ can be not necessarily uniform?
My students who have gone on to take positions like the one you seek almost universally report that the most useful statistical skill they learned was...how to use Excel effectively :-).
Hey gung, I understand, I will try to rephrase, but I think this thread can be helpful to everyone. Most people have their favorite package that works surprisingly well, and I have not been able to find any good threads on this.
What you want to estimate is not a sample size, but the size of a population (total number of unique items returned by the web sercice).
I guess the English terms are [regression analysis](http://en.wikipedia.org/wiki/Regression_analysis) and [analysis of variance (ANOVA)](http://en.wikipedia.org/wiki/Analysis_of_variance).
How many replications did you ask for?  In its simplest form the lower and upper bounds of the bootstrap confidence interval are the the 2.5$^{\\mathrm{th}}$ and 97.5$^{\\mathrm{th}}$ percentiles. So with only 200 replication the lower bound would the 5$^{\\mathrm{th}}$ smallest and upper bound would be the 195$^{\\mathrm{th}}$ largest value. So with only a few replications those bounds can become unstable. The solution would then be to just increase the number of replications. 20,000 is a number that often works well for me.
(1) What do you mean by affecting a gene? Linkage? (2) ANOVA needs an approximately normally distributed dependent variable. How do you derive your dependent variable from the information on *tg*?
+1 This CDF has a "closed form inverse" using the "ProductLog" ([Lambert W function](http://mathworld.wolfram.com/LambertW-Function.html)).  Wolfram Alpha will [compute it](http://www.wolframalpha.com/input/?i=ProductLog[1]) and lists some of its properties.
:). I meant what exempler cases we use the other ways to calculate ranks, or we don't want ties in ranks?
There are some good ideas here: http://stats.stackexchange.com/questions/56322/graph-for-relationship-between-two-ordinal-variables
Good point! I flagged my question and suggested a move to Cross Validated.
It is not a distribution curve in the traditional way, but I think kurtosis is a good start point. Maybe I can modify it to fit my situation. Thanks very much!
Thank you and would be highly obliged if you could pass on the modified script which you have written.
The components $\\psi$ given by the spectral decomposition of the covariance function: $C(x_i,x_j) = \\Sigma_k \\lambda_k \\psi_k(x_i)\\psi_k(x_j)$ are themselves linear combinations of the original modes of variation in the sample. I guess that simply examining the correlations between the projections/scores $A$, $A_l = \\Sigma_i \\{Y_l(x_i) - \\mu(x_i) \\} \\psi_k$,  associated with a specific $\\psi$ (what you call weights I think) and each original variable will give you the answer you want. So... a bar chart for each component? :D Simple and understandable from all?
Have you take a look at Collombier's or Tinson's books ?
Variance is additive [outside](http://onlinestatbook.com/chapter4/variance_sum_law2.html) of portfolio theory, too.
Nothing the matter with noobie questions. As proof, this one has been asked and answered in many forms here; *e.g.*, http://stats.stackexchange.com/questions/7790/how-to-determine-statistical-validity-of-results/7807#7807. (Although it is correct that the binomial distribution governs *individual* outcomes, the *proportions* observed in the experiment will be very accurately described by a normal distribution unless those proportions are extreme (close to $0$ or $1$).) As far as effect size goes: if you aren't interested in the difference in proportions, how *do* you want to express it?
bill them to collect more data.
@Sara, I'd explain fundamentals of DA right in my answer if I had time. Sorry. _When_ assumptions of DA are met (i.e. normal distribution etc.) DA usually classifies objects somewhat more accurately than LR.
My guess is that it will probably depend on the "constant" in the $O(n^{-\\frac{1}{2}})$ error, which will vary between models.
This paper may be of help: http://cowles.econ.yale.edu/P/cp/p07b/p0777.pdf
In short, it usually works by calling a series of not-too-fast functions that tries to find and execute complex expressions that may exist in a formula (like `(a+b)^3+c*(e+f)+g:h+0`) and makes a new `data.frame` ready to be used by the modelling function. In case you just want to pass predictors and decision this is a fully redundant and is a significant overhead with large sets due to numerous copying.
Colours are fine, but why not use different symbols as well or instead? Different symbols e.g. $\\diamond \\circ +$ express qualitative contrasts well. On colour, remember that many people have difficulties distinguishing colours, especially red and green.
@zhubarb, thank you very much! Your explanation and advice are very useful!
@Alan From the [definition of the identity matrix](http://en.wikipedia.org/wiki/Identity_matrix). $\\boldsymbol{\\iota}$ is the identity matrix if $\\forall$ matrices $\\mathbf{a}$, $\\boldsymbol{\\iota}\\mathbf{a} = \\mathbf{a}$.