tttjjj/elt_bench huggingface dataset:
Not to my understanding. If you model the sites as "random" you are essentially giving up on estimation of distribution of the site-results. At the very least you are saying that you really do not believe that the site-effect warrants any assumptions about the nature of its parent distribution. My understanding is that the fixed effects are what return useful estimates. I have raised the fact that this is (very) confusing terminology with persons more expert than I and have gotten general agreement.
@ralph-winters If I understand correctly they are all independent.  Some will work better with others, but they all have an affect on the Percent who purchased.  Image and title usually have the biggest impact though.
@Lucas, without knowing your specific problem I wonder whether you need KL-convergence of the transition kernels in the MCMC algorithm or whether the MCMC algorithm is run as part of the computation or optimization of the KL-divergence under consideration.
When two sets of observations appear to have the same distribution, they can be viewed as two independent sets of iid draws from that common distribution, whence the unions of these *observations* will appear to have the same distribution.  (A union of *distributions* makes no sense; the concept you may be referring to is a *mixture*.)
RMSLE is useful in scenarios with outliers/noisy data.
AFAIK, [HDBSCAN](http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6086201) is not from the original DBSCAN authors. But instead by some other chinese guys... Or maybe you were talking about another paper ?
@whuber, I had the formula in wikipedia page in mind. There the difference between estimate of the regression coefficient and its true value is expressed as sum of two terms. The bias is expectation of this sum. The expectation of one term is always zero, since it involves the regression error. The expectation of another term is zero if there is no correlation between the omitted variables and the variables in the regression. I modified the answer to reflect your comment.
@Quartz Thanks for the heads up.  This used to display properly, so evidently there has been some change in the processing of the $\\TeX$ markup.  I found a workaround by enclosing all single quotes within braces.  (This change has probably broken a few dozen posts on this site.)
@AdamO I have Diggle et al. (Second edition) in front of me and section 13.4.2 specifically addresses complete case analysis and states that it is biased.
Thank you Peter! In another question I've read that it's quite daunting to check the the normality of residuals. Someone else suggested that normality is not needed with modest sample size.
There are only 32 combinations possible; you could write them all out and just count up the ones that have three heads in them.  You could save some effort by noting that all combinations with a tail in the third place cannot have a sequence of three heads, so you actually only have to write out 16 combinations (the ones with a head in the third place) and remember that the other 16 don't have any sequences of three heads.
my reading of @deemar's question is that the data is not time series. It is just data collected in the past.
@Andrew To train a classifier you need to label examples beforehand manually. Given this and the fact that addresses as you said tend to follow predefined patterns, why not using [regular expressions](http://en.wikipedia.org/wiki/Regular_expression) instead ?
(+1) A recent tutorial with BUGS/R can be found here: [The Use of R and WinBUGS in Fitting Item Response Theory Models](http://idecide.mskcc.org/stats/multi/irt_winbugs.php).
Hi @whuber. Actually, with my answer I wanted exactly to address this misconception, that's why I stated, probably too quickly, _the future will not depend on the past_. What would you suggest to make it stronger and more complete? What other possible procedures are you referring to?
Any sample data or a reproducible example (cf. your last question)?
In another way my question is how can I estimate through the dlm package an arima(1) with intercept different from zero?
Nevermind, answer is Z_(ir) is a row vector, not a column vector,  Apologies
Could you give us some background that explains why you would expect the two plots to be similar?
@VincentWen I fixed some formatting to make it easier to follow and I realized I didn't add in the square roots, so now this should have the correct coverage (i.e., 95%)
stan - Yes, different people use the term differently - some use it to mean spread, others CV, and likely still others something else - hence my caution near the start of my post where I say "If by dispersion you mean...". Thanks for the link; it's good to have examples.
@robin: "matrix" in this context is standard analytical chemistry terminology; it refers to the medium where the entities to be analyzed for (the "analytes") can be found. For instance, if you're analyzing for the concentration of lead in tap water, lead is the analyte, and water is the matrix.
Because you have only 2 classes, only 1 predictive "principal component" can exist, and that being a linear combination of the features, the regressional model. You could prefer logistic regression or linear regression. You could then drop features with small coefficients from the model if you want to have a concise model. There is no need for preliminary PCA or PLS at all, for me (unless there are collinearity/singularity problems).
+1 @StasK for a very good answer to a question that is very difficult to deal with.  I think there is a tinge of unnecessary sarcasm though and you might consider editing your answer so as to be a little more respectful of the OP.
I agree to some extent but "necessary" and "any kind of research" is going a bit far. A lot of disciplines don't often deal with dirty data and many leading researchers have never needed to touch a programming language.
Strictly speaking, $P(X_2 = x) = 0$ for every $x$, since $X_2$ is continuous. You can compute the density of $X_2$, which can be greater than one since it is not a probability.
For better understanding please provide details of your question and the R codes you tried.
That's an interesting paper. Thanks
Excellent. Can you amplify your answer about UMP just a little, as how/why the non-centrality distribution enter?
The term "standard error" is attributed to G.U. Yule (1897) by H.A. David in http://www.stat.iastate.edu/preprint/articles/2011-10.pdf Presumably it echoes "standard deviation" introduced earlier by K. Pearson (1894). These are dates for the terms, not the ideas.
When getting started with Bayesian inference, you can't really go wrong starting from Gelman's *Bayesian Data Analysis*. It won't help from the chemistry side, but it will help with understanding the underlying approach to this kind of inferential problem.
I really don't know. At least you now know that your LPM is severly misspecified and that your logit model suffers from perfect predictions. I would start reading into what perferct prediction does, alternative ways of dealing with that, etc.
ok, got this: http://www.stata.com/statalist/archive/2006-12/msg00564.html ...
$X'X$ is now one-dimensional. Calculate explicitly the $det$ and then calculate explicitly the OLS estimator $(Z'Z)^{-1}Z'y$ (using perhaps sum-symbols rather than vector products). You will see that eventually the $x_{n+1}$ and the $y_{n+1}$ cancel out, for the determination of $\\hat \\beta_1$, _if you insert the obtained equation for $\\hat \\nu$ in the equation for $\\hat \\beta_1$_. Now I have revealed everything!
@MarcClaesen, I am sorry, but I don't quite understand what you suggest has anything to do with my question... mind to elaborate?
For example, I want to calculate the probability of being a paid user for users with these conditions: x1>10, x2>12, 300>x3>375, 4>x4>6, x5>2, x6=1. Since I can only calculate probability for exact conditions (like x1=8,x2=10,x3=250,x4=6,x5=2,x6=0) I don't know how to calculate ranges for numeric variables, that's why I said cumulative probabilities.
sorry for my ignorance, who is Gujarati?
That's exactly what I was looking for
It's not clear what you want exactly. "Population" is the entire data from which you sample, so I think that is probably not the right term. You can certainly have populations that have bimodal distributions.  If you are looking for a test for number of peaks, there is the Hartigan dip test.
(-1) nowhere in the question does it state that his data is normal. OP is asking, if knows the underlying family of distributions (Poisson, Normal, Gamma, or whatever) should he base his estimator on ML even if it is biased, or just use the generic (nonparametric) unbiased estimator anyways. For the Poisson distribution, there is a lot more than just a scaling factor separating the MLE of the variance and the unbiased estimator the OP gave.
It is a question of combining survey answers from multiple surveys written at different times, where the creators have changed their wording from year to year (though they all target the same population).
Are you at all familiar with prior distributions and posterior distributions? If not, read through an example of doing [Bayesian inference on binomial data (i.e., given an observed sequence of coin flips, infer the probability of heads)](http://goo.gl/Mdl9D), then it's a straight forward generalization to multinomial data (i.e., given some rolls of a biased $K$-sided die, infer the probability of observing each side). The latter problem, which involves "placing a Dirichlet prior" on the probability vector, is the same inference one does with a Bayesian HMM.
I am afraid you need to ask another question. A quick search shows something like How to formulate a good null hypothesis has not been  asked yet. This will allow me to deploy Tukey. Ask about history of that golden rule of stats (p-value > 0.05) and I will refer you to Hall. In many situations hypothesis testing is less than appropriate regardless of the size of your data.
Do you have 3, 30, or 100 branches? This isn't really clear. Your data are from 3 years, do you have the same data (eg sales) on each unit (eg a bakery) for every day over that time? If so, you should use time-series methods to address the serial correlation, not multiple regression.
+1 I also recommend Quantum GIS from the freely available, opensource softwares.
@Cardinal Given that the OP has twice used the word "mixture" and did not equate "$p * X_1 + (1-p) * X_2$" with a $\\mathcal{N}(p U_1 + (1-p)U_2, p^2 \\sigma_1^2 + (1-p)^2 \\sigma_2^2)$ distribution, it would be erroneous to interpret this as a linear combination, don't you think?
Thanks, I have re-formulated the question with more specifics of my data.
I would say the canonical reference on the ``nlme`` package is [Pinheiro & Bates (2000)](http://www.amazon.com/Mixed-Effects-Models-S-PLUS-Statistics-Computing/dp/1441903178/). They are, after all, the authors of that package. And in that book, they give plenty of examples illustrating the use residual correlation structures.
@RamSharma: I just added edits above.
No. You need to compute the weighted sum of inner products *in feature space* of all support vectors and the test instance. To compute the inner products, you have to use the kernel function (you didn't specify which one you use). If it is RBF, it is implemented like this in libsvm: $K(\\mathbf{x}_i, \\mathbf{z}) = \\exp(-\\gamma \\|\\mathbf{x}_i-\\mathbf{z}\\|^2)$, with kernel parameter $\\gamma$.
From `?pdClasses`: "Users may define their own `pdMat` classes by specifying a constructor function and, at a minimum, methods for the functions `pdConstruct`, `pdMatrix` and `coef`. For examples of these functions, see the methods for classes pdSymm and pdDiag." This question may be more appropriate for http://stackoverflow.com/.
The information seems pretty garbled to me.  E.g., "The conditional probability" that what?  And toxic or nontoxic substances are going to have their own attributes: the attributes would not be assigned by some sampling process.
Could you add a plot of your regression against the original data? (by the way, you can also directly use `model$residuals` or `resid(model)` to find the residuals)
@Stefan Maybe, maybe not. If you transform your outcome, then your inferences based on the transformed relationships do not necessarily apply to the inverse transformations after you have performed your analysis; this is because $\\text{Var}(f(x) \\ne f(\\text{Var}(x))$. So if you analyze $\\ln Y =\\beta_{0} + \\beta_{X}X + \\varepsilon$, finding a significant $\\beta_{X}$ does not necessarily translate into a significant $e^{\\beta_{X}}$, nor does CI$\\beta_{X}$ necessarily correspond to $e^{\\text{CI}\\beta_{X}}$.
Are you wanting to calculate some *empirical estimate* of the entropy of a source?
I think that stating that the google data represent 79% of your data points is enough...
"*the prices are slightly negatively skewed but seem to be normally distributed*" -- those two statements appear to be contradictory.
To truncate, use T(-,-), but read the users manual for info on censuring and truncationq
+1 though a couple of points that might bear a mention - OLS regression assumes constant variance, but we wouldn't usually expect to see constant variance in count data with changing mean. Non-constant variance is often more of an issue with normal theory procedures (like t-tests and tests of regression coefficients) than normality. It might be better to consider using GLMs (ones suited to counts, at least) for either a plain comparison of means or for a regression-like model.
Understood ! Thank you very much, again !
I have computed the KL divergence using the integrate() function of R and I have obtained the same result as yours.
I also tried to produce a graphs as the link below http://www.metafor-project.org/doku.php/plots:forest_plot_with_subgroups however, suddenly, the program didn't show the graphs in the output anymore after I used the command dev.off(). The graphs was saved on my computer, but I would prefer to see the graph in the output (I use R Studio) before I save it. How can I change it back?
No this is right. From here though - is there a way to graph each of this dynamically into a tree in R? If not, is there another tool I could use?
I did the obvious thing and tried it. It does not seem to make a difference.
Could you provide some model statements?
Let me state my objection a little more clearly, Peter: when you remove and/or adjust pseudorandom values in an *ad hoc* way in order to approximate some desired property, such as lack of clustering, it is difficult to assure that the resulting sequences have *any* desirable properties. With your method, for instance, could you even tell us what the first moment of the resulting process would be?  (That is, can you even assure us that the intensity is uniform?) What about the second moment? Usually these constitute the minimum information needed to use the sequences effectively for inference.
The calculated entropy refers to the pictures above. The JPG-file has a size of about 5MB, the TrueCrypt-Container about 100MB. Even if I take a 5MB piece of the TrueCrypt-Container, it is equally distributed - much more equal than the JPG-file.
You answer gives many details about the entropy I haven't heard, thank you for this! Maybe some details too much, I am not into statistics too much... I have just tried to "use" statistics for a while. One question is still left:
What is the reason why a distinction can be made with the frequency analysis (e.g. chi square), but not with the entropy?
It means you need to find good values for the hyperparameters. I fail to see what the authors mean by it being less generic for imbalanced data sets, since hyperparameters should always be tuned when using SVM.
Is there are any particular reason you're suppressing the intercept by including `+ 0` in the formula?
For now flexmix is working very nicely, thank you very much for the hint
What do you mean by predictive quasi-normal log-likelihood? quasi-likelihood certainly has its roots in estimation, but for prediction, maximum quasi-likelihood predictions are the same as conventional maximum likelihood. Plus I don't understand how a prediction would give you an estimate of correlation...
I'll leave the question. I still don't know whether the model is identifiable but I am quite convinced that `nlmefit` is not suited for this model because it is not linear in $\\gamma_{i}$ and $a_{i}$.
I appreciate your help, though. I'm paying a tutor and trying my best, but I just don't get these formulas. The ones that are built into Excel and the calculator I can do, but ones like this I am lost.
Actually, my objective is not a prediction but testing the impact of independent variables. The dependent variable is election win/lose.
I'm saying that rather than write a function to calculate the A-D statistic from scratch, you can modify the code used in `ad.test`. (Type `ad.test` to see it.)
My initial thought was 'I don't think there is an answer to this question.' Similar to what is written in one of the replies. The idea of a confidence interval about some smooth curve, and then getting narrower as n increases, seems like a cool idea, though.
There's no point in a truncated normal where there's positive probability for that point. Either you don't mean 'probability' or you don't mean 'truncated'.
I installed R and all necessary packages. would you please provide some example for orm function? I didn't find by searching. For my regression model, what the code should be?
Except I have feline overlords to obey. :)
Why sample at all? Even with n in the millions, running a chi-square on the full data should not be a problem.
@cardinal: Many thanks, I did not see that in the documentation. Very much appreciate the answer, but can you point me toward a link, etc. My search for gsl_ran_gamma was pretty unsatisfying. Absolutely not questioning the answer, but just so I don't ask something so simple again.
@Abe, updated and hopefully clarified in the process. There was one particularly bad typo in the previous version. Sorry about that.
Your calculations are correct. If you use `R`s internal function `prop.test`, you'll get the same result: `prop.test(x=c(19,15), n=c(34,34), correct=FALSE)`
Actually it's a bit of both. I now have the rough data tough, so I guess it's more on how to analyze the information (esp. considering how to get the information is off-topic - I'm sorry! Thought it could fit on one question).
https://www.google.be/#output=search&sclient=psy-ab&q=sum+of+normal+distributions&oq=sum+of+n&gs_l=hp.3.0.0l4.1131.4145.0.6153.10.6.2.2.2.0.210.739.3j2j1.6.0...0.0...1c.1.12.psy-ab.IgJqdx9M6MI&pbx=1&bav=on.2,or.r_qf.&bvm=bv.45960087,d.d2k&fp=e3c46b2e87b6b124&biw=1366&bih=640
The last paragraph of the question hints that something else is being asked for.  I have asked for clarification.
If you have several response variables (Category $A$ have five samples it does not mean five variable)  than it is multivariate regression model. And do not think "shall I generate a regression model for each group" because it highly influenced precision of the model. You must have one data file under the columns response variable and classified variables. In response variable distinct columns can come and in classified variables( I think you have two, category and Samples)  e.g first response variable belong to category $A=1$ and first sample=1, second value category $A=1$ second sample=2.
Thanks! This is perfect!
The question makes sense, but in general it depends on your sampling procedure. Polling firms have to develop very robust sampling procedures precisely to ensure the confidence and margin of error of their results. In the absence of them, you need to make some possibly unwarranted assumptions, e.g. that your sampling is perfectly uniform (the probability of getting a "yes" is the same as the proportion of "yes"es in the general population).
@mike You will get the same results from using a linear model approach as you will for just using 
    summary(aov())
The advantage with using lm() is you can visualize the results.

ANOVA is good for testing the hypothesis that two or more means are equal to each other versus the alternative hypothesis that at least one of the means is different from the others.  In this case you appear to be looking to test the null hypothesis the score for coke1 is the same between the two sexes.  ANOVA, also called ANCOVA, also allows for such tests after accounting for a covariate, such as coke0.
But when I do something in OpenBugs, for example, I just need to write {T∼dbeta(3,7); X∼dbin(T,n)} n=15. And this program through Gibbs Sampling automatically sample many times X|T and T|X. It means that they use some numerical technique for sampling T|X cause I can put any distributions and I'll get correct results. How can I do this, sample from T|X, in R, for example?
Look at ACF and PACF in the correlograms in addition to the Ljung-Box. Google what you should be looking for. This is a good way of visually seeing if there's any seasonality or serial correlation in your data.
I think there's a lot to be said for this position. Given that many researchers mostly want to know about patterns in their data, I've often wondered if we could reasonably set aside much of statistics and simply use plots of the data. (Of course, this assumes the plots would be done skillfully and insightfully, and hypothesis tests wouldn't be as bad if we could say that about them.)
@nico @Ralph Maybe this has to do with a distinction between local and global information (contour) processing, but honestly I cannot really "see" such a difference in terms of apparent dispersion. It seems to me (but this has already been said) that binning (or window span in kernel density estimate) helps to spot multiple mode if any, not the overall shape.
What was the SD for the effect size?
"That wouldn't be OK because you know that there has to be some [correlation]." To be fair, we know (almost) all null hypotheses are incorrect, but that doesn't stop us hypothesis testing in general. In this case, the null hypothesis of zero correlation would correspond to a very weird physical situation, so is unlikely to be the thing you want to test.
+1 This is correct.  Just for fun, how does Excel's answer differ from R's when you use 5000 as an argument instead of 5?
Well, since the inverse gaussian is defined only for $x > 0$, the numbers can't actually have an inverse gaussian distribution...  did your data come from a process that might have rounded off small numbers to zero?  It helps to know where the zeroes come from when answering this question.
Thanks for the reply! The problem is that I can't take an individual score out as the group agreed one on score as a group. I basically want to know whether the individual agreed with the score that group agreed on by consensus.
@Aaron You and Wikipedia are both right. The modes of the two *marginals* of a Dirichlet$[n,n,n]$ for $n\\gt 1$ are at $(n-1)/(3n-2) \\ne 1/3$.  The (very interesting) error made here is that the components of the mode of a $d$-dimensional distribution are not necessarily the modes of its marginals.
The question is too vague in my opinion. Metric for what? Metrics are rarely computed in isolation but are computed to serve some particular objective. So, you have metrics that evaluate model fit because the goal here is to assess how well different models fit a particular data set. Similarly, you have metrics for evaluating speed of an algorithm because the goal is to design faster algorithms etc
I went to these book website, and found some related R scripts, which are related to some part of book contents. But I could not find any scripts that are related to chapters 11 and 12.
Two quick notes on your notes. 1. The C-vM distance is precisely the L^2 cousin of the Kolmogorov (L^infinity) and (univariate) K-R (L^1) distances, and hence interpolates between them.  2. One advantage I didn't mention of the K-R and B-L distances is that they generalize more naturally to higher dimensional spaces.