tttjjj/elt_bench huggingface dataset:
A well constructed multivariate MH proposal may greatly outperform Gibbs sampling, even when sampling from the conditionals is possible (e.g. high dimensional multivariate normal, HMC beats Gibbs by a wide margin when variables are highly correlated). This is because Gibbs sampling doesn't allow the variables to evolve jointly. It's sort of analogous to optimizing a function by iteratively optimizing the individual arguments - you may do better if you optimize wrt all of the arguments jointly rather than each one in succession, even though it is *easier* to do the latter.
Given your question is about life/death type of information, I would have thought taking a representative sample is way more important than bayes vs frequentist differences.  Random sampling does not gaurantee you this in the specific case.  This is part of that prior information I was talking about, where you control for known causes/differences between the people who survive and the people who die (age is usually an obvious example).
1. The equation appears to have no relationship to $\\hat{k}(\\theta)$ at all. Is it missing something?  2. Because one can *define* $a(n,\\theta)=\\frac{\\partial l(X_1, \\dots , X_n)}{\\partial \\theta} / (\\hat{\\theta}-\\theta)$, the equation appears to add no information whatsoever.
@JEquihua: Thanks! In my dataset, I had 8 clusters, and 400 sentences that I labelled blindly, that is, without knowing from which cluster they came. So for every cluster I have a distribution of labels. That's what the plots show. I made the plot with a custom Python script, using the Python Imaging Library.
Just for the future, this question might be best asked as three separate questions.
I don't think the best that can be done is to ignore the counts of events 5-8. Sometimes these will make your estimates impossible, for example.
Actually I need to use the collected data as stated above to do some forecasting. I need to show if a particular area where the data were recorded are apt to be used as a site for installation of wind turbines. So i needed a model which will allow me to predict this with the least error possible and I found Weibull distribution is used mostly for this purpose. But I can't find a way how to apply Weibull distribution to my recorded data. Regards:
Arvind
By the way, I have seen a similar post  [here](http://stats.stackexchange.com/questions/35971/is-sampling-relevant-in-the-time-of-big-data?rq=1). That discussion answered my questions.
Essentially, you just want to extend the single changepoint method to multiple changes.  The easiest way to do this (not necessarily the best) is to implement Binary Segmentation.  That is essentially a smart `while` loop which starts with the entire data, runs the `change.point` function then splits the data at the changepoint if one is identified.  Then you run `change.point` again on the sub series.  Repeat the above until no more changepoints are found.  This is an approximate segmentation though compared to PELT which is exact.
@AndreSilva I'm not using any software.I'm just a programmer. Currently I need to use multidimensional concept to create Sudoku game.But I am unable to understand multidimensional concept. This is place where there are many visualization experts. So, I thought it would be wise to ask question here.
No.  It appears to me that you are conflating "random sample" with "responses".  The two are identical only if the response rate is 100%,
What is the regression equation? Is it  Y=aX +bZ +c? Or is one of the other variables the dependent variable? Are X, Y, Z each 1 dimensional or 3 dimensional?
"Crucial" is a category; severity is a continuous dimension, as is kurtosis (or skew, for that matter).  Consider [this related question](http://stats.stackexchange.com/q/2492/32036).  Note also that [skew may be less problematic than kurtosis for issues of variance](http://stats.stackexchange.com/a/30053/32036) such as correlations.  Last, for the sake of distinguishing between kurtosis and skew, there is a specific test of kurtosis that ignores skew: [the Anscombe-Glynn test](http://biomet.oxfordjournals.org/content/70/1/227.short).
Is your model actually a scaled Brownian motion with drift? If so, averaging the price process over $n$ observations to get a second series yields another Gaussian process and is determined by its mean and covariance, which you can readily calculate via the mean and covariance of the underlying price process. If your model is some (nonlinear) *function* of a Brownian motion, then things get more complicated.
+1 You can obtain the distribution with an (efficient) dynamic program.  Let $p(n,j,k,m)$ be the probability of $j$ isolated actors and $k$ non-isolated actors choosing from $m$ actions.  Adding one more actor gives the recursion $p(n+1,j,k,m)$ = $p(n,j,k,m)(k/m)$ + $p(n,j+1,k-1)((j+1)/m)$ + $p(n,j-1,k)(m-j-k+1)/m$.  Of course $p(n,1,0,m) = 1$, letting us start the recursion.  Computations simplify when you base them on $m^{n-1}/(m-1)^{[j+k-1]}p(n,j,k,m)$: the values are integers independent of $m$.  ($x^{[j]}$ is the factorial power $x(x-1)\\cdots(x-j+1)$.)
It is also informative to read the Discussion and Rejoinder to this paper. In the discussion, Peter McCullagh wrote that he disagrees with a substantial portion of what Gelman wrote. My point is not to favor one or the other, but to note that there is substantial disagreement among experts and not to put too much weight on one paper.
A unit root in the MA part makes the series non-invertible; see [here](http://stats.stackexchange.com/questions/50682/what-is-the-intuition-of-invertible-process-in-time-series).
@damned: Yes it is. Are you looking for a exact solution does not matter exponential?
@Glen_b	Physics would lead to transforming predictors, however, then their distributions differ from normal, and R2 is less, regression is worse. I maybe would prefer to transform the response for better prediction (its distr. becomes closer to normal after transf.). So I want to understand _statistical_ rules for regression.
I think you need to first state more precisely what it is you are trying to test, then someone here (maybe me) can come up with some test of that.
You might want to take a look at the [gene shaving](http://genomebiology.com/2000/1/2/research/0003/) method.
(+1) Especially for the extension.
@tomka, I changed `fit2` to `fit` in the above example as per your correction. Thanks!
To my knowledge you can interpret the coefficients generated through a lasso procedure the same as you would those generated through a standard logistic regression.
About the model: the point of measurement modelling is to build a model of how the unobserved thing to be measured affects the observed items, usually with the assumption it affects them each separately (local independence). In your case the model will be some set of logistic regressions where the 'independent' variable   is the thing to be measured and the dependent variable is an item.  It might be worth working through the first link in my first comment link to get the idea.
True score as explained by Peter is measured score + error. The error here represents "construct bias". The measurement error could increase  observed score and hence, we can ascertain true score by plus or minus "error. Further, I think that the statistics  does not recognize the construct bias. It simply takes cognizance of the moderator factors the effect of which can be neutralized by the appropriate (or larger) sample size.
I cannot comment yet but I think in the accepted answer shouldn't the decision line and margins be: abline(b/w[2],-w[1]/w[2]) abline((b+1)/w[2],-w[1]/w[2],lty=2) abline((b-1)/w[2],-w[1]/w[2],lty=2) since abline takes input as intercept and slope respectively? w.x-b=0 **which implies:** w1.x1+w2.x2=b **which implies(since x2 is the "y")** abline(b/w[2], -w[1]/w[2]) Sorry this is an old post but I thought I'd ask.
Anyhow, did you understand the roloe of the parameter $q$ in my answer?
@kwak Check out Tibshirani's webpage, http://www-stat.stanford.edu/~tibs/lasso.html and the `lars` R package; other methods include coordinate descent (see JSS 2010 33(1), http://bit.ly/bDNUFo), and the Python `scikit.learn` package features both approaches, http://bit.ly/bfhnZz.
You might want to read up on the difference between 1 and 2 tailed tests, if this topic is not familiar to you.
thanks for this. appreciate it.
@NeilG No, the text doesn't discuss hyperpriors.
Hi there.  The webinar that I referenced concludes with an ROC example that probably fits your needs.  I'll attach the code extract to the original answer.
Can you clarify the reason you want to do this manually in Excel, rather than use a statistical package (even a free one, like R)?
will edit, I saw the error in my answer while writing.
You're on the same track pursued by Gretton *et al* at http://eprints.pascal-network.org/archive/00004335/01/NIPS2007-Gretton_%5B0%5D.pdf.  Also see their slides at http://www.kyb.mpg.de/fileadmin/user_upload/files/publications/attachments/ismTalk_%5B0%5D.pdf.
Did you tried it? What results did you obtain?
Each binary variable for each website gets 1, if there is a property in the website. The websites should be compared based on these 80 binary variables (or the 4 main variables than are sums of binaries)
+1 Nice examples and reference.
If you prefer I could replace the gamma mode example with a more commonly used MLE, but I don't see how it doesn't fit the bill for a descriptive parametric statistic. I see your point regarding the sample mean, that wasn't a great example, but it's not wrong either: the sample mean is defined as $\\frac{1}{n}\\sum x_i$: there aren't any distributional assumptions in there.

I'll qualify my "linear regression" example as ordinary least squares, since that's what I meant, and I believe it qualifies as a parametric technique. Regression trees are definitely a non-parametric technique.
@Perri Ma , You have to put a "dot"after the sign. It should be like this 
status~.  or  this data$status~.
I hope this helps
ANOVA can be seen as "syntactic sugar" for a special subgroup of linear regression models. ANOVA is regularly used by researchers who are not statisticians by training. They are now "institutionalized" and its hard to convert them back to using the more general representation ;-)
@Glenn: Yes. But you don't even have to transform to $[0,1]$ for this to happen.
Thank you for the thought experiment/simulation, it was instructive to see that there would be a correlation/sampling bias.
(1) Out of curiosity, how is it possible to take a weighted average and not know the weights? (2) You mean the weights separately sum to $1$, not $0$, right? (3) It sounds like you know the weighted averages: is this correct? Again, what is your interest in the variances? Why is that relevant to inferring the weights? I think I'm missing something important about your question.
I answered this question in the first half of my answer here: http://stats.stackexchange.com/questions/55692/back-transformation-of-an-mlr-model/58077#58077
@ Ashok - yes.  I would consider what I wrote to be based on a terse version of one definition of a tangent space.  Of course since the cotangent space is dual to the tangent space, one could equally argue that $d\\theta$ are the true dual basis.  In any event as long as the $p_{\\theta}$ don't vanish, I think you're good.
For example, look at this posts:  http://stats.stackexchange.com/questions/22680/can-the-interaction-term-of-two-insignificant-coefficients-be-significant?rq=1         http://stats.stackexchange.com/questions/29520/interaction-term-change-the-significance-of-the-main-effect-variable?rq=1          http://stats.stackexchange.com/questions/32702/how-to-interpret-an-insignificant-main-effect-that-became-significant-at-the-pre?rq=1                                             http://stats.stackexchange.com/questions/42950/interactions-make-terms-significant-in-regression-when-they-should-not-be?rq=1
*Every* part of the correct formula is covered by my discussion above. Can you more clearly explain what you don't get and I can point to where it's already covered by my answer. I can try expanding my explanation further.
Depending on the amount of data, number of variables, shape of relationships and so on, a lot of methods could be used. I strongly recommend a lot of graphical exploration first.
+1 for mentioning "reservoir samples", which caused me to learn something new.
Also called the matching problem, the envelope-matching problem, the envelope-letter problem, the forgetful(/absent-minded) secretary problem (among other names). e.g. [see here](http://www.math.uah.edu/stat/urn/Matching.html)
What have you tried so far - where are you getting stuck?  Is this a homework question, if so it is useful for us to know, so that we give you an explanation you can learn from.
If you are going to have the question here, you need to delete the version on math.SE.
@qoheleth They're *two* numbers per row separated by a space each number with a [decimal comma](http://en.wikipedia.org/wiki/Decimal_mark#Countries_using_Arabic_numerals_with_decimal_comma). So "1.52" is written "1,52". Many countries use them.
Swapping $X$ and $Y$ in the first equation in the Wikipedia article on [Law of Total Expectation](http://en.wikipedia.org/wiki/Law_of_total_expectation) gives the equivalence of (2.9) and (2.11). Read that article for proofs.  (2.12) is immediate, on the understanding that $f$ is to be chosen in order to minimize EPE.
You're on your way towards re-inventing Ripley's K function (http://www.stat.iastate.edu/preprint/articles/2001-18.pdf ).  Note, however, that much information is lost in this approach.  Suppose, for example, that 15 of the distances are zero and the other 15 equal 10,000.  Is that 15 clusters of two words or one cluster of 16 words and 14 isolated words?
I've voted to close as "opinion-based", such things are poor fit for [se] sites in general. I would say No too. Very powerful computers haven't rendered stats obsolete (through simulation etc). In fact they've done opposite. "Big" data will more than likely do the same & stimulate new areas of endeavour within the subject.
@Dmitrij Celov. I've edited the question to clear-up the D-F explanation. Sorry for the confusion.
(+1) brilliant.
I thought that interaction between the covariates and the factors indicates violation of the homogeneity of regressions assumption (another assumption also mentioned in the first link I gave)
The intercept should change (coefficient in most output), but not the random intercept! if the random intercept is only the random intercept for the base reference group --> that implies that a random effects model as many random intercepts as there are levels in the categorical variable. But there is only one random intercept.
well, there do appear to be some MCMC java libraries (which is needed for Bayesian inference), HYDRA seems to be popular. Alternatively, there is JAGS, BUGS which are popular non-Java options. This assumes knowledge of how to use MCMC in Bayesian inference though. Personally, I had difficult time going from Bayesian theory to Bayesian practice, but after understanding (and taking more time than I care to mention), I wrote an [ebook](https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers) to help others. It uses Python and PyMC. Might this help?
Welcome to the site. I removed your signature, as it is automatically added. What is it you are trying to do? It seems like reaction times are the dependent variable and age an independent variable, is that right? Do you  have other IVs?
I think M&W is actually 2007.
Yes, sorry, I mean a variable I am statistically controlling for.
Observations from same subjects are correlated, it is confirmed. I already did the analysis and got the expected results. What I need is an estimate of the mean, which is identical to the one I would get with independent observations (correlation doesn't affect the mean), and an adjusted variance. I got this with the log link, and didn't with logit, I need to know why, the maths. Why the log link gives me the right outcome and logit doesn't ?
@whuber: sorry, I was still editing
Unfortunately, not always - sometimes the data is something like:

7.967261,	8.032799,	7.7836
why the data is missing? if the fact that the data is missing tells something about this observation, then you're in trouble
Thank you - I have read somewhere that just using t-tests when comparing multiple strategies (well, they write about therapies or sth. mostly) could give more false positives, so I want to use Tukey's.
The [help](http://stats.stackexchange.com/help/on-topic) on what's on topic says "*Questions about obtaining particular datasets are off-topic (they are too specialized)*"
I wasn't sure exactly what to make of your last line, so I didn't edit that. Maybe you could edit to clarify. That said, at a glance, I'm not sure the reason behind the downvote, either. Cheers.
Ahh i cant upvote this answer 15 rep required. Once i get 15 i will revist this answer and upvote :)
Wonderful, Wonderful(!), answer!  Thank you Drury, I'll go through it a few more times to see what I can introduce to my teachings.
Please could you post the outputs of str(xx) and head(xx) so that we see the essential of your data ?
I'm starting to think that if I _do_ ignore the standard errors for each individual's slope or intercept and just use the raw slope or intercept estimate as a predictor variable in a new model, there might be bias but it will be conservative bias. After all, why should the fact that we chose to be diligent and collect several replicates for each individual and record the order they were collected result in a _less_ reliable model than if we had collected one measurement per individual like everyone else does?
Upvoted for mentioning "multinomial logit model"
Logistic regression seems ok, but it has the additional constraint of row and column total being fixed. This may not be the case with Poisson data. I believe the answers won't differ much.
@whuber I got no sense of any problem with your comment which seemed matter of fact; sometimes these things are hard to gauge in text.
@loganecolss - this is not a paradox as the notion of bias only exists "locally" - that is, with respect to a given statistical model.  The "paradox" exists for a person who: 1) knows the "true model", and 2) decides not to use it.  That person is an idiot in my book.  If you don't know the "true model" then there isn't a problem - unless you've found a good model and decided not to use it...
Welcome to CV.stackexchange. Maybe you could demonstrate with the given example on question how this new version of `gausspr` function would provide the posterior covariance matrix returned by kernlab.
Agreed, and since the blue curve is so different from other two, I imagine the smoothing parameter choice is difficult to get right. What smooths the smaller curves might mush the taller curve entirely.
Thanks! i did want to do an MANOVA, not an anova, i was just unsure in how far you have to include this into your results (aka my results section in my paper), whether it is enough to report on the manova main effect and then ignore whatever is found in the anova.
+1 for some good comments but I think more can be said and some things I havein mind are positive and constructive.
@BrianBorchers, that's what JohnK meant by "consistent."
thank you alll.... the comments are very helpful..
Thank you for your response. I have an additional thought/question about the randomization. Only the obersvations/subjects has been randomly selected, could this be a problem when evaluating the model? for example time here cant be randomized and which group observations belongs isnt either randomized. Dont know if it makes sense, thankful for any response.
Also I still think that this is a very good method.
We welcome questions like this, @Erin, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. To better understand the process, you should read the [wiki](http://stats.stackexchange.com/tags/self-study/info) for the `[self-study]` tag.
You're making your life hard. You're doing exactly the same as what @danas.zoukas suggested, except you're averaging across multiple examples. First, you're depriving yourself of all the analytical tools that come with metrics such as precision. Second, the probability doesn't really exist - the observations do.
You probably know much more about the mechanisms generating the “number of hits” than written here – that mechanism may be a key point in building the right statistical model. Times series often show autocorrelation or periodicities, modelling should include these if they are present. “Significant decrease” may then be defined as deviation from what was predicted, but it's important to have a good prediction model.
You don't need an explicit prior on $x_t, t=1,\\dots,T$, only on $x_0$, as your second expression in the state space model in effect, writing heuristically, constructs that prior from the priors on $x_0$, $B$, and $R$.  Your parameters are $\\Phi = (x_0, A, B, R, Q, c, \\mu)$.  You'll still get a joint posterior for the latent $x_t$ out, though.
Thanks for the detailed explanation. This really helped!
Might be of interest: http://cseweb.ucsd.edu/~saul/papers/logpca_aistats03.pdf
Something like a k-sample Kolmogorov-Smirnov? Or are you interested in allocating any rejection to the three causes?
Hi, this was not was I was looking for. I should have been more clear. Maybe taking the problem upside will make it more understandable. Is it possible to generate a sample with mean of 95, standard deviation of 25 (sample values has to be positive and can't exceed 100)? But thank you to take time for this.
This is very helpful. Unfortunately I am not very familiar with linear combination or general polynomial approach. Can you through some light on these? Which statistical technique would allow me to accomplish this?
A related post [link](http://stats.stackexchange.com/questions/3653/adding-coefficients-to-obtain-interaction-effects-what-to-do-with-ses) has some discussion on this, but I would very much appreciate more detailed advice, e.g. in the form of code.
ANOVA can correct for multiple factors, or across the "family" of factors, but this usually requires an additional step that can be overlooked resulting in the conditions that you describe.
@amoba and @whuber: You probably know this, but I’d like to clarify it anyway. Note that it’s possible for the ANOVA test to be significant even if none of the Tukey’s HSD tests are. Simple R example with a balanced data set with three groups: `set.seed(249); 
group = rep(1:3, each=2);
y = group + rnorm(6);
mod = aov(y~factor(group));
summary(mod);
TukeyHSD(mod);
plot(y~group)`
@JDav Thanks, I was a bit confused about whether those papers had widespread application or depended on particular data sets or types of data. I came across them while trying to figure out whether violating IIA is always a problem. (I think you edited out an earlier comment which seemed to say something useful about how the data sets examined in both those papers probably have particular characteristics explaining why MNP and MNL give similar results?)
The beta distribution takes on the values 0 and 1 with probability 0, and this data set comes from a distribution with a large point mass at both of these points, so beta regression probably isn't appropriate.
I think this answer needs some qualification, because under one straightforward interpretation of the question it's obviously not correct. Because all the regression coefficients are linear combinations of the dependent values, this makes them linear combinations of normally distributed data, whence they have *normal* distributions, not Student-t distributions.  Are you perhaps discussing *estimates* of the slope and intercept and distributions of *test statistics* rather than sampling distributions?
@vinux I am trying to generate the assumptions from raw data sets. So I guess causality test would be used for this?