tttjjj/elt_bench huggingface dataset:
The issue here lies in the distinction between *actual* coverage and *nominal* coverage.  You seem to assume the two are equal, but for most discrete distributions they are not.  Worse, the actual coverage depends on the proportion one is trying to estimate!  Before considering the question further, you might want to research the two methods @Michael Lew mentions.
You should say what questions you wish to answer from this data.
@bill_080 As you can see, it is more complex in practice.
@Gaetan Aside from my response, I gave my +1 to the question because I think it is very relevant for debating about statistical practice and project management.
Why do you want time series as opposed to a multi-level model or even just summing the 3 days? Are you interested in day-of-week effects?
Because I'm not worried about a sample size, I am looking at a population?
The nice scatterplot clearly reveals that it is the second pc that discriminates the two clusters.
I want to assure my understanding of the first point. By a single bag you mean a single Random Forest classifier, right?
Ok, Firstly:  If you use SPSS you would need the SP1, SP2, SP3 columns but not the FCT column, in order to run a repeated measures One-way anova.  You would then look for a main effect of 'sport' - this will tell you, for example, whether RT  get better with SP3 (30mins jogging) compared with SP2 (10mins of jogging) and SP(0mins of jogging).
@DilipSarwate I'm happy to take the answer as a humongous sum. I am passing these expected values for $H/S$ and $H$ as part of a payoff function in an evolutionary game, which I will simulate. I am extending a game for which there is a simple(ish) solution (not short and sweet, but not a humongous sum). Only in that case, there weren't any discontinuous functions like those in $H$.
Many thanks for the extremely detailed response.
If the "particular knowledge" is a 0-1 thing (they know this thing or they don't), it sounds like you might want a McNemar test.
+1.  But the "little generalization" is *too* general: given any target set of values $t_i$ (summing to unity), you can find such an $f$ for which each $s_i$ is transformed into $t_i$, effectively losing all information about the original $s$'s.  Normally, then, we restrict $f$ to be *positive,* *continuous,* and *strictly monotonically increasing* so that at least the order of the original sequence is preserved.
Since this is being done repeatedly during an optimization routine, perhaps the data in the regression are (eventually) changing slowly. This suggests an algorithm adapted to your situation: start with some form of robust regression, but when taking small steps during the optimization, simply assume in the next step that any previous outlier will remain an outlier. Use OLS on the data, then check whether the presumptive outliers are still outlying. If not, restart with the robust procedure, but if so--which might happen often--you will have saved a lot of computation.
Feature space just refers to the collections of features that are used to characterize your data.  For example, if your data is about people, your feature space might be (Gender, Height, Weight, Age).  In a SVM, we might want to consider a different set of characteristics to describe the data, such as (Gender, Height, Weight, Age^2, Height / Weight) etc; this is the mapping to another feature space
So, what are some possible approaches for the non-independent case?
OK, with `gamm()`, if my description of the problem is what you want to do - you can ignore the gap at that site. Make sure you nest the AR or ARMA correlation within site. And be prepared to wait around for a while for the model to fit... ;-)
@gung The algorithm mimicks the manual dealing of a deck of cards, more a psychological thing I guess :-) I will try it the way you suggest too.
Do you have any data on how often such things happen?
I added math markup to your answer. Hope this made it clearer...
do you know the sample sizes... the range of them?  Small sample sizes wreak havoc on the mean... moreso than the median.
Peter, you need to take a crash course in [multivariate normal distribution](https://en.wikipedia.org/wiki/Multivariate_normal_distribution) and do [some exercises](http://www.amazon.com/Matrix-Algebra-Econometric-Exercises-Vol/dp/0521537460) to understand the derivations involved. This was discussed on CV [before](http://stats.stackexchange.com/q/30588/5739), and you can also find [helpful handouts](http://www.math.uiuc.edu/~r-ash/Stat/StatLec21-25.pdf) from various multivariate statistics classes.
+1 for your answer, but I don't know what H(x,y)=C(F(x),G(y)) tells me? But I can use any distribution for the margins right? I am not getting, why then the margins are uniform? Since they also can be normal or so?
I would say this question has to do with so called "all relevant feature selection" algorithm, not RF *per se*, so that you might want to think of a more informative title for your question. BTW, you didn't tell us what the supplied R code is supposed to show, although it looks like it has to do with the significant $t$-values (LM) vs. selected features (Boruta).
There may be a single objectively valid good answer to this question, so I hesitate to turn it into CW.
If you have a reason for not wanting to use average before and after scores, what is that reason?  Do you think you will get more power by using a Fisher Exact Test to detect higher rate of significant multiple Mann-Whitney tests?  I think it is likely that such a two stage approach has lower power than just doing a t-test on the before and after averages.  In any case, if you want to "use all the data" have you considered doing an AVOVA with subjects as blocks?
That has to be it. Thanks!
True, I hadn't thought of that, but that's a nice point. It's also a big plus for the R help files (another kind of text), since all the examples there use simple data sets that are included with R, or with the library by default.
Yes, that is what I think as well. Do you know of any reference on this?
Welcome to the site, @universalis. Are you interested in ways of thinking about how to visualize these, or do you just need help w/ MATLAB code? Note that the latter would be off-topic for CV (see our [help page](http://stats.stackexchange.com/help)), but on-topic on [Stack Overflow](http://stackoverflow.com/). If the former, please edit to clarify; if the latter, flag your Q & we'll migrate it for you (*please don't cross-post, though*).
Hi Whuber, with follow-up, I indeed mean which additional statistical analyses I need to conduct. When I plot the interaction, I can see that the interaction is driven by one steep line. How can I test the significance of that slope (in a multilevel design). I think I need to conduct a single slope analysis in order to interpret my 3-way interaction but I could be wrong. With splitting the file, I meant running the analyses twice, for example once per group, and than testing the 2-way interactions. Do you know what I mean?
Granularity refers to the size of the clusters, smaller being more granular, and larger being coarser (less granular). Bigger inflation values will lead to smaller clusters (more granular). Indeed, weakly connected groups may cluster together at lower inflation values and separate at higher inflation values. It is best to just experiment.
It seems more natural to use a single error term, but if you have to separate error sources it can make sense to have two. :) In (1) you do indeed get $E\\hat{a}=a+q$ if $\\hat{a}$ is the usual OLS estimator, so $\\hat{a}_{corrected}=\\hat{a}-q$ seems reasonable if $q$ is known. In (2) you still have unbiased consistent estimators, but OLS is not necessarily very efficient. Do you mean $X_i$ when you say "observation $i$" in (3)?
Thank you for your answer Glen.
Doesn't sound like a familiar or standard problem to me.
@Alex: Sorry, wrong link, I meant [Bayesian Core](http://www.amazon.com/gp/product/1441922865/ref=as_li_ss_tl?ie=UTF8&tag=chrprobboo-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=1441922865). Note that we also posted [solutions to all problems on arXiv](http://xianblog.wordpress.com/2009/10/25/solution-manual-to-bayesian-core-on-line/). So posting a complete solution here would not hurt!
I dimly remember the world map of the facebook network, which was done in R. I think the author described his process in some detail in his blog. I suppose using that approach would generate a map that is informative even with 4 million nodes.
Thanks Dr. Mike .. I understand now the error in my approach . what do you suugest should be the best approach
I'm saying a Poisson distribution is probably a bad model for this particular example.
It depends on whether you are trying to indicate to the devil the variation in values or uncertainty in the estimate of the mean
Welcome to our site, Dan!  This recommendation is called a "majority filter"; as such, it is offered as a substitute for the median filter--but that's exactly the kind of approach the OP is trying to avoid. Your suggestion of a CRF is intriguing: would you care to expand on how it would be applied here?
Thanks, I've decided to opt for bootstrapping the confidence intervals instead.
@Michael, Yes that works too but you throw away information. I use it when i need to combine sparse levels of a nominal variable - when the ultimate modeling will be done without a tree type approach (say logistic regression or SVM and many sparse dummy variables causes issues)
Please note that although that paper describes an approach towards observational studies I have my doubts as to the reliability of randomization for biological experiments due to my question and the answer [here](http://stats.stackexchange.com/questions/74350/is-randomization-reliable-with-small-samples)
Excellent answer, thank you. Some of it is a bit over my head, but I think I'll be able to remedy that myself :)
The summary in the [Wikipedia article](http://en.wikipedia.org/wiki/Multivariate_t-distribution) seems quite clear about this. Does that resolve your question?
try looking into cohen's kappa and interrater reliability.
Given your description *you are* trying to evaluate causal effects, there is no difference in your situation to others you have seen referenced in literature.
I disagree in part with your initial paragraph because the questions asks about the concept of Bayes theorem. The Frequentist-Bayesian debate *is relevant* to this part of the question. The Kolmogorov axioms do not give Bayes theorem the same conceptual importance as the "probability as extended logic" axioms do.
`we plug in sample values for k of the n variables` Please tell more clearly what it is. Do you want to say that you already have some variables with the wanted covariances and now you want to generate more variables following with each other and with the old ones acoording to a full, extended cov/ matrix?
What is the outcome you are measuring in each group? Is it time to some event?
Regarding $h(x)$, you have a mistake (although it's a very common confusion). You write, "interval->0, 'instantaneous' death probability". A correct statement would be 'instantaneous death *rate*'. This cannot be a probability because it is a probability divided by $dt$; moreover, it could be >1.
Where do they use `glm()` then `lm()` in the chapter you link to. Seems to me the `glm()` is all that is required and used there, but I may have missed something. You can try generalised least squares (`gls()` in **nlme**) which allows weights to be estimated to control for the type of heteroscedasticity you mention; see `?varFunc` and follow the links from there. IIRC `varFixed()` will do what you want.
okay gonna take this as an answer then
(+1) It wouldn't take much to make this rigorous by appealing to the Intermediate Value Theorem for the integral approximations (which would replace the $x_0$ in the integrands with some value $x^\\prime$ in the interval $[x_0, x_0+\\epsilon]$) and then invoking the continuity of $p_x$ to take the limit as $\\epsilon\\to 0$.
Best practices for this design have already been discussed on this site, see http://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs
I suppose that you want a decision tree because you can easily represent IPv4 space as such. You want something that when a lot (say > 80) of child nodes are characterized as spammers then the whole set is to be considered as a spam generator, right? Is your database static? Do you take in account that as time passes the spamability of an IP address may change due to measures taken by the network managers? Have you considered querying databases such as Spamhaus to help your decision tool?
Welcome to the site, @prospero. I have edited your question, & used CV's formatting options to make it more readable & clearer. However, my background isn't in machine learning, & I had to guess in places at what you were trying to say. Please make sure that it still reflects what you want to know. Also, there is nothing in the question about 'optimal sensor placing' such as is listed in the title, so you may want to edit your question or the title to make them more consistent.
I think any valid theory of inference is good to know.  While unbiasedness can be a good property, bias is not necessarily bad.  When emphasis is put on UMVUEs there can be a tendency to attribute "optimality" to it.  But there may not be any very good estimators in the class of unbiased estimators.  Accuracy is important and it involves both bias and variance.  What is better about the MLE is that there are conditions under which it can be shown to be asymptotically efficient.
Note that $\\sum_i(y_i-\\bar y)^2=\\sum_iy_i^2-n\\bar y^2$; this will help to compute the expectation of $\\sum_i(y_i-\\bar y)^2$ which should equal $(n-1)\\sigma^2$. Then, computing $\\mathbb{E}(\\hat\\sigma^2)$ is trivial.
I ended up going down this route, thanks for the tip.
@Henry You're right: James' formula estimates $|A\\cap B|$ by $\\frac{| \\tilde{A} \\cap \\tilde{B} |}{| \\tilde{A} | \\times |  \\tilde{B} |}|A|\\times |B|$, which is your formula.  It's interesting that it attains its lack of bias by occasionally making estimates that are mathematically impossible!
The main point of the spread-vs-level plot is to *look* at it, because what you see provides information well beyond a single statistic (the suggested power transformation).  What does yours look like?  (`R` code to construct this plot appears in an answer at http://stats.stackexchange.com/a/74594/919, which explains how to interpret it.)
Don't confuse dropping a variable with having no effect, Joe! For example, let y ~ a+d+e, b=a/2+2d+e/2, c=a/2-d+e/2, with d, e, (and the implicit error) all small compared to the variation in a.  Then a may have a profound effect on y, but (due to its association with b and c) is unnecessary in the model (and won't even be significant in the full model y ~ a+b+c); the best model would be y ~ b+c (because this equals a+d+e).
@Mike, precision and recall are common evaluation metrics in, e.g., information retrieval where ROC, or in particular specificity is awkward to use because you already expect a high number of false positives.
This is a clever idea.  Note, however, that "just as well" in Fourier space is *not* going to be the same as a good fit in the original space: those are two different criteria of goodness of fit.  One is also led to wonder about the potential complications of fitting *complex*-valued functions: suddenly we have a two (real) dimensional problem rather than a one-dimensional one.  I'm not trying to suggest these are show-stoppers, but they are concerns that would have to be investigated by anyone working through this approach.
@Dilip Perhaps.  Or maybe there are strong correlations and the data were rounded.  But all this is beside the point: it's clear that this interpretation of the data is not appropriate.  The asker wants to fit a Gaussian to a 2D grid, as you suggested in your comment to the question.  (There are nevertheless some technical issues to settle, such as whether the grid values are probabilities for equal-area cells or values of a PDF at the cell centers or something else and how the values were measured or computed--that is, their error structure.)
I think you want to say "loses meaning when..." instead of "uses meaning when...".
I'm not sure if something like the "gene set enrichment analysis" can be used for DNA methylation as well, but if yes that may help you to aggregate collections of genes into metagenes.
It's all easy to find on Wikipedia through the obvious searches :-).  As far as correlation goes, the problem is that variables can be strongly correlated but have totally different distributions; in fact, there exist sequences of distributions that get arbitrarily "far apart" (in an intuitive sense) while the distribution of their correlation coefficient gets arbitrarily close to $1$ in large datasets.  This makes the correlation coefficient a poor discriminator of distribution.
FYI, this article from Tuerlinckx et al. 2006 provides a nice review of GLMMs: http://bit.ly/9k8vJs. Also, about the use of GEE as an alternative marginal model, it was proposed by Feddag, Grama, and Mesbah (2003) to estimate parameters of the Rasch Model (one-parameter IRT model), or loglinear multidimensional IRT models as defined by Kelderman and Rijkes (1994). It is currently available in the Stata `raschtest` package.
good point. derp.

http://bit.ly/zhuenC
You need to become familiar with [PIE](http://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle).
What is your goal?  How is the clustering going to be used?
@Glen_b: I think I was wrong, it is $I(\\hat{\\theta}_n)$ not $I_n(\\hat{\\theta}_n)$  which is still an MLE of $I(θ_0)$, and therefore is consistent, i.e. converges to $I(θ_0)$ in probability.
Thank you very much Lucas! One question: "Because if we have visited it once, we will most likely visit it again, and again, and again.". What ensures that we are going to revisit a state after we did once?. If our chain has infinitely many states, how can we be sure that we are going back to one states after we have been once there?
@gung My guess: because then the usual quantiles people are interested in will correspond to observations rather than needing interpolation.
How do you train the CNN? Layerwise as stacked RBMs or as stacked Autoencoders or something else?
ANOVA is regression.  I don't understand your question.  Maybe you're confused why the two outputs differ?  Can you provide the code?
this is false!  it's the probability of data equal to **or more extreme than** the observed values, under the null. $P(D|H)$ is the likelihood, not the p-value.
@Melon, sure. First, I have edited "extremes" to "rare events", which is more appropriate I believe. What I was referring to, in particular, is that you can use the *exponential change of measure* technique if you have a light tail (that is, not a heavy tail) and you need other tools, and get different kinds of results, if the tail is heavy. A  reference is Chapter XIII in [Applied Probability and Queues](http://www.amazon.com/Applied-Probability-Queues-Stochastic-Modelling/dp/0387002111).
Welcome to the site, @JohnForbes. One of our goals for CV is to develop a permanent repository of actionable information about stats in the form of answers to questions. As such, one thing we worry about is link rot. Would you mind editing your answer to include a standard citation, in case those links go dead? Since you're new here, you may want to read our [about page](http://stats.stackexchange.com/about) & our [FAQ](http://stats.stackexchange.com/faq), which contain info like this.
I also recommend this paper: http://www.cs.stanford.edu/people/ang/papers/nips01-discriminativegenerative.pdf
If you have a binomial-GLiMM model, the traditional $R^2$ does not apply. Are you referring to one of the so-called *pseudo-$R^2$s?
@Dapz Please accept my sincerest apologies for not replying to you until now - I made the mistake of assuming nobody would answer. Thank-you for your answer below - I have upvoted it since I can see it is most useful, however I need to spend a little time considering it. Thank-you for your time, and I will reply to your post below soon.
Sam, if you can compute all the other probabilities, then surely you can exploit the fact that they all must sum to unity!
Maybe it sounds like ARIMA, but it ain't. ARIMA is easier to estimate because the forecast refers to previous real Y which is fixed. In my problem you refer to previous forecasted Y which changes anytime the parameters of the model change.
Any idea what causes the clustering?
I would suggest choice 1, since even the lme4 authors are unsure of how well quasi-likelihood models work in the specific approach to GLMMs taken by `lme4` ...
(mean $-$ median) / SD has the interesting and at first sight surprising property that it lies in $[-1,1]$, something periodically rediscovered. Although necessarily no single measure can capture all that might be desired, it deserves a little more use as a descriptive measure. It will occasionally have the "wrong" sign, as will any such measure. L-moments deserve more use for describing distribution shape.
You need to ask a specific question.
@Hans All I can say is that it works for me :-(
As you see in the right sidebar, there are several discussions adressing this question.
We agree that there would exist a contradiction with Rao Blackwell, if you had Lehmann Scheffe without sufficiency. Still we are not sure what exactly is wrong with our reasoning.
How many cells are in the table? (Since the sample size is so tiny, you should have no problem posting the entire table here for our examination.)
+1 for emphasising the distinction between Likert items and Likert scales.
@MichaelChernick I do computer science research, and sometimes I need to do user controlled experiments, but generally, I can only get small samples because users are hard to get. Mainly, I take measures of task durations, how people use the tools (these sometimes implement different treatments), and users frequently answer surveys/questionnaires with Likert scales. So, in light of what I do, which statistics do I need, and which books do you recommend? Do these books you listed above are suitable for my work? Or there are others more suitable? Thanks
Thanks for the quick response. I've read though the link you suggested several times-- agreed, it's a fantastic resource. I suspect that I'm over thinking my question. My concern is that x and y are very tightly correlated with plotted against each other, but the "flattened" plot looks like a bunch of random points. Is this just a result of having the z data being so uncorrelated?
Thank you very much for your answer and for editing my post too. Your answer is very helpful, Greatly appreciated !
this answer could be improved with a little detail on why each test is considered good/better than others.
@reisner Welcome to our site and thanks for answering.  You might want to look through some other questions of interest to you (troll through the ['tags'](http://stats.stackexchange.com/tags) section for ideas) to see the ways in which people try to clarify and improve questions before answering them.  It's easier to provide a great answer when you're sure about what is being asked.
Hi. A multilevel model is quite right. Thank you. Until today I had only been able to find models in which the "level 2" predictor is the group identity, i.e. Yij = B0j + eij; B0j = C00 + u0j. But now I found that I can also use a level-2 continuous predictor, i.e. Yij = B0j + eij; B0j = C00 + C1*groupX + u0j. So, thank you...
" I clustered related predictors together " can you explain better what steps that involves? "I was expecting to get a much better result from using the robust principal components" Can you explain how you measure results?
I think I fixed it - replaced the yhats with ybars. Is that more correct? I always screw up yhat.