tttjjj/elt_bench huggingface dataset:
In online surveys, often the same respondent fills in twice or thrice (for some reason). Do you gather any personal ID info to identify the same person?
Use the group LASSO: [Yuan & Lin (2006), "Model selection and estimation in regression with grouped variables", *JRSS B*, **68**,1](http://pages.stat.wisc.edu/~myuan/papers/glasso.final.pdf)
Agreed here, but I was thinking that - if indeed there are only two rows - let one axis be "violent" let the other axis be "non-violent" and plot each column in this 2-d space as a scatter plot. With only two components, there's no dimensionality in this dataset to reduce. Maybe centering/scaling each row with improve the plot.
@DavidJ.Harris "and the user tries to pick the optimal depth based on cross-validation." you mean e.g consider different allowed maximal depths in seperate experiments and then based on results (precision, recall, F) of course crossvalidated (e.g k-fold) decide what maximal depth is reasonable?
Hi Joel, thanks for your answer.  Does it mean that the two methods I described are equivalent?
"Can I do that" as can I do that in the software I use or can I do that and still being "statistically kosher?"
I am looking for the difference between the means. Regarding the sample size, it may limit the external validity of the study, but I am happy if for this small study I can determine somewhere a statistical difference between the means of X, Y and Z, that is, that a difference was not by chance.
I think in this context (as, in my opinion, in most contexts) a backward selection procedure would not make sense. If the fish.species and anemone.species variables account for some variation, the backward selection procedure will favor keeping them, but if they don't account for much variation at all, then by definition it won't matter much whether you keep them or leave them in the model anyway! So you can't really go too wrong with just leaving them in regardless of what the backward selection procedure decides.
there is a definition of Q2 on this page http://www.moleculardescriptors.eu/tutorials/T5_moleculardescriptors_models.pdf
@Chris In addition to Roland's answer, which is a good one, making calculations like this are useful to make sure you understand exactly what the software output means.
Please edit your question to make it readable by itself without reference to other links (which can rot).  It's unfair to ask your readers to do research just to understand what you are asking.  The situation may suggest to some people that you don't care enough about this question to write it out fully, causing them to vote to close it.  For guidance on writing good questions, please consult our [help].
**Hint** What's the chance of not finding one of those individuals in a sample of 1?  How about the chance of not finding one of them in another sample of 1 from the remaining 9999?  Now put the two answers together: what was the chance of not finding any individual in the two samples?  Iterate...
No disrespect, but that strikes me as entirely subjective. I have used latex for since the late 80s and R since the 90s yet I see no good reason to move away from embedding eps files. Everybody's mileage will differ here, and Thomas should maybe look at Task Views for particular problem domains.
I wouldn't proceed quite that way, but I can edit something similar in.
Of possible interest: [What is the difference between fixed effect, random effect and mixed effect models?](http://stats.stackexchange.com/q/4700/930)
If you're looking for someone to explain what the intercept, standard error, z, and Pr(>|z|) mean and how they function, I think you're best off looking in a textbook or doing a web search for a good intro treatment of logistic regression.  (try David Garson's at N. Carolina State U., or UCLA's stats page).  Contributors to this site may not want to write all that up.
The regression with "group" would need some random person intercept or other trick to be similar to a paired t test.
This answer is extremely misleading. There are very predictable non-stationary series, because the cause of non-stationarity may come from the deterministic part. What matters is the power of the deterministic component to the power of the stochastic component in the whole. For example $y_t = e^{y_{t-1}}+gaussian error$ can be filtered very easily although the series explodes and non-stationary by any definition.
Thanks, Matthias.
Great explanation, I hope this is taken by people to encourage them to describe the details of whatever standard scores they are using in their write-ups.
Just in case you do not know it, you can look at all of the residuals by using `model$residuals`.
This material is covered quite well in the book [Econometric Modelling with Time Series](http://www.cambridge.org/us/academic/subjects/economics/econometrics-statistics-and-mathematical-economics/econometric-modelling-time-series-specification-estimation-and-testing) by Martin, Hurn, and Harris. Specifically, see Chapter 15 and pages 567-571. I may provide a full answer later.
Could you provide a little bit more details on what are your particular needs: academic (scientific, PhD), practical (model building, engineering, programming), level of disaggregation (macro, micro, panel data), field of application (microeconomics, macroeconomics, finance, physical sciences), may be some other details you feel are relevant.
@NickCox There's no real difference between 0.666666667 and 2/3. But using 2/3 is more expensive because the computer will have perform additional calculation. Most people will just hardcode it that way.
Are you assuming that the two "correctnesses" are independent of each other?
I admit to being baffled as to what you hope to see in an answer, because it seems like you have correctly answered your own question already.  As your comment indicates, merely replace "1 mm" and "0.5 kg/ha" in the example by "1 rainfall SD," and "0.5 crop SDs" respectively.
@DWin Maybe you're right and I haven't grasped the underlying principle well enough. Like I said, I'm a statistics newbie! I know that two variables can in fact be jointly normally distributed, and would just like a way to check whether this is true (at least with a certain level of confidence/probability). And PS: just a small note - it's a she, not a he.
+1 This is more or less what I was trying to get at in my comments above.
@Kostas - I'm not sure what more you need; I've tried to put the explanation in terms of your problem, but I can't tell without more detail from you where the weak spots are.
There is no problem in principle about a histogram with some, or even many, bars of zero height. Such bars merely represent values not observed in the data, and such gaps are part of what you want to show. Alternatively, consider quantile plots as discussed at e.g. http://www.stata-journal.com/sjpdf.html?articlenum=gr0018
Thanks for you reply! But I must say that I don't know any topic that you are talking about. I was wondering if a QQplot will become handful for this. What do you think? From the alternatives that you give, which you think that I should use? the goodness fit? Where I can find information and/or the tests that you are talking about?  (of the Smooth Test for godness fit) Also, did you know if someone has R code for the blog code? (I dont know Matlab or SAS). And, many thanks for your answer!
No need to apologize at all: we appreciate you sharing your experience.  Welcome to our site!
+1 for beta-binomial.  That's a great tool to have in one's toolbox.
I think it was definitely a good idea to redirect to an ANOVA calculation to find out the overall effect of stats but I'm not sure finessing the p-values is good.  The regular `anova` command will give you F's.
I added the post notice because this answer is unintelligible without following the links.  Please at least indicate what you are recommending. *Much* better would be to include the reasons for your recommendations.
Do you have multiple participants or just a single participant? How many blocks do you have?
So the answer is yes. I still do not understand exactly why. This might be because I do not know how the serial correlation is tested. 

It seems like you know a lot about serial correlation. Do you know how to test for serial correlation when using pooled OLS?
I can't see any point in doing this other than as a diagnostic tool to identify problematic points or missing non-linear effects, as suggested by the above comments.  I think the two dot points in the OP ("thoughts on the subject...") are pretty sound.
Why don't you post your proof up until the point where you get confused
It might help to change how you think about chances a little. Rather than thinking in terms of probabilities, which are numbers between $0$ and $1$, logistic regression invites (nay, forces) you to think in terms of log odds (which can be any real number).  When you do that, logistic regression looks *remarkably* like OLS multivariate regression.  Perhaps then your present question almost answers itself from this viewpoint?
John Tukey describes median polish, with many examples and exercises, in his EDA book.  It's easily carried out by hand (although a spreadsheet helps for doing the subtractions correctly :-).  It's important that there not be too many missing data (empty cells) in the array, for then median polish can fail to converge and can be pretty biased.  Otherwise, there are no other requirements.  The proof of the pudding is in the analysis of the residuals.  Tukey describes several very clever ways to extract additional information (e.g., interactions, transformations) from them.
@ttnphns Those points appear to be correctly generated and distributed.  They look more concentrated in the center because the sphere is thicker in the center.
Is work additive, such as the amount of some product? Then you can use linear regression to estimate the contribution from each worker, and sort workers by their coefficients. If it isn't additive then you might want something more complicated.
It may be close enough to a prior question to be a duplicate of [How to conduct conditional Cox regression for matched case-control study?](http://stats.stackexchange.com/q/2748/1036). It has a very nice example of a frailty models for matched data in Stata.
If this is related to some subject, could you add the `self-study` tag please?
Note that linear in such contexts usually means "linear in parameters". d(t) in your example formulations doesn't seem to contain parameters at all.
SVMs can perform multiclass classification. The method is very similar to softmax regression (see "On the algorithmic implementation of multiclass kernel-based vector machines").
It is much easier to sort two lists of 50 data points than one list of 100 data points.  Therefore, splitting into descending classes using some easy method such as a stretch function and then sorting each class alone will ordinarily be much faster.  This is shown by looking at the sort time for various algorithms and then playing with the number of points sorted and then multiplying to get the total time based upon number of groups.  This is because all sorting algorithms scale nonlinearly with number of points sorted.  No algorithm that I know of benefits from lumping points together.
Look at http://stats.stackexchange.com/questions/173/time-series-for-count-data-with-counts-20/ . It's not a duplicate, but the problem is similar at a first sight.
great answer. I got the concept. thank you so much.
I [answered](http://stats.stackexchange.com/a/35460/13669) too.
Yes to the first, No to the second. A number of threads on this site with the 'data-transformation' tag should be helpful, e.g., http://stats.stackexchange.com/questions/18480/interpretation-of-log-transformed-predictor (see Charlie and jthetzel's answers).
Are you asking, then, how to find some subset of the data that have a specified mean, SD (and size)?  Or some multisubset?  (In general this is not possible, but you could find such "samples" whose means and SDs are as close as possible to the specified ones.)  One wonders what the purpose of such an exercise would be.
Hi there.  Does the formula you've suggested include *all* previous values with exponentially diminishing weights?  So the *entire* time-series would be included, not just the most recent $N$ points?  Referring to the $N=4$ example posed in the question, are you suggesting setting $\\alpha=0.25$ or $\\alpha=0.5$ to approximate the 4-point moving average?
@AlecosPapadopoulos I am looking at conditions in a six month period and returns in the following month. I'm looking at a single potential signal which can take either positive or negative values and with a mean of about zero. Values close to zero must be considered to provide no information about the future.
+1 That's a very interesting take on the question. In that case, can Euclidean be considered one such metric? As the dimensionality is reduced, the points are projected into a lower dimensional space but that could mean the notion of distance can be lost. I am having a hard time to see how distances can be preserved when using reductions like this.
I have used the same data set and analyzed it in SPSS. The results are similar to SAS. However if we can standardized the data as in the book, the first value at c=0 is similar in R and SPSS (running a multiple linear regression). However the results are quite different when we move further as indicated above. I too found no clue of the difference between outputs for same data.
Good wrap-up, Chris. Thank you for your efforts!
Alright, thanks a lot. I already thought so.
A while ago I posted [this question regarding power & graphical representation](http://stats.stackexchange.com/questions/14140/how-to-best-display-graphically-type-ii-beta-error-power-and-sample-size) that might be of some help
@Glen_b said it well.  Dichotomies are bad, continuous estimation is good.
Ah okay, and a_counter is also shared state so allows current thread to know which values it's getting.  Yes, I think this could work, especially if I fine tuned the size of the block of numbers that each thread takes before passing on the flag.  Would be nice not to have so much synchronization, although I think you're right that contention would be not too bad.
The learning tag has no wiki, but I believe it's orientated more towards learning of statistics knowledge.
2. Even if the values are not exactly 1 or 0, that's still ok. There is some kind of a Bernoulli assumption there, but it still works practically. 6. You can never be sure not to be stuck in a local minimum with neural networks. Also, you probably don't want to get into a local minima wrt your training set, because that would most certainly imply overfitting.
Welcome to the site.
(+1) but I added an answer with just an illustration of what you just said, as I have a geometric mind :)
(1) Why wouldn't you want to include a variable that directly calculates the independent variable? This seems like the most desirable predictor, if only you can find such a predictor.
-1 Absolutely incorrect workflow for large p small n, FS overfitting is guaranteed.
There is a difference between "0.4 times lower" (incorrect) and "0.4 times as high" (correct).  E.g., if baseline odds are 0.5, then "0.4 times lower" = .5 - .4*.5 = .3, whereas "0.4 times as high" = .5*.4 = .2.
Do you mean chi-square goodness of fit test? That sounds like what you are looking for.
This does seem strange, for the reason you give.  Perhaps you could give us the paragraph before as well in case there's something we're missing?  One thing that makes the bias not a big deal is that it becomes pretty unimportant as the sample size gets bigger, and is probably not material compared to all the other problems eg model mis-specification that we normally have - but this is not the reason given in your source.
You should use Colin's answer, still your idea of making Monte Carlo simulation is also correct.
You're saying clogit isn't a two tailed test?
I think you want the `image.plot()` command to add a color bar. Also, `filled.contour()` generates a similar plot with a color bar added by default.
Welcome. Google for `Multiple regressin Venn's diagram'. You'll learn that, for example, if $R^2$ is the observed R-squared of multiple regression and $B$ is the squared _part_ correlation between the DV and the pridictor B, and, likewise, $C$ is the squared _part_ correlation between the DV and the pridictor C, then the the part of the variance of the DV that B and C _share_ is $R^2-B-C$.
Thanks, signal decomposition looks like it might be interesting for working with this type of data.
I intend to use this [formula](https://upload.wikimedia.org/math/c/b/b/cbb518be041b820958181a932a5cd4ff.png). Th joint and marginal probabilities can be estimated by a kernel function (e.g. Gaussian or epanechnikov)
@TonyMorland do you think something like this *would* work in *practice*?
Thanks, the double "frequency" is my concern, too.
It seems like the assumption that the logs follow normal distributions must be doubted.
I think you're talking about the case when we specify the law $X_1$, then the conditional law $(X_2 \\mid X_1)$, then the conditional law $(X_3 \\mid X_2, X_1)$, and so on; here that does not solve the question at all.
DaRob: Do you have any tutorials to point towards for this model?
An interaction doesn't mean to multiply parameters, it's about multiplying regressors. Thus, the model looks a bit strange. Repeated measures anova with heterogeneous variances can be modelled in R e.g. using package 'nlme'.
This is amazing! Thank you.
Note that the ligne REPEATED is useless here. What did you expect about this line ? Could you describe your model in mathematical terms ?
Thanks @wolf. I'm inclined to accept this answer. If the first definition is standard, how would you define mutual information?
+1 Thank You for your explanation. So, incorporating what mbq had suggested, i.e., computing `P=density(A, from=0, to=max_value)` and `Q=density(D, from=0, to=max_value)`, can you suggest what I should pass on as arguments to `KLdiv`? Is it (`P` and `Q`) or (`diff(P$x)*P$y` and `diff(Q$x)*Q$y)`?
Thanks. Both answers are acceptable but I guess the other one is more useful for third-party reader. :-)
@chl I have updated question with details
I don't follow at all, because you have no need to make inferences about the population mean: you have stated you know it!
You should if possible supply a minimal (i.e. as small as possible) example with the problem you have.
Okay thanks. When I got two different answer I wasn't sure if I did it wrong, or if I was just barking up the wrong tree. Sorry for writing it wrong, I had a feeling I would do that that's why I put the link.
+1 just for the 1st mention on this site of a Gilligan's Island character.
The two halves of the equation The left part of the sum Vs. the right side. The a(sum((w_ij^2) for i=1,2,..I) where a is a regulatory factor and w is the weight of NN fro i to j
Except for some very specific contexts (ones where you know a lot about the process and the sd can or should only be in a limited range of values), there's no such rule.
No, I won't be implementing a Gibbs sampler with this particular conditional posterior, but I was asked about it, and wanted to see what it would come out to be. I'm not turning in this work, I just wanted to see what the steps would be to get this posterior in $\\mu, \\sigma^2$ repectively.
Great answer! I had made this realization a little after posting. And since the variance of our sampling is inversely proportional to $n$ (the number of samples), our confidence is also improved (at least, theoretically).
thanks for spotting those errors Nick! still working on my citation skills in latex.
@jacqueswainer: I'm not exactly "testing" the classifier on the training data... I just appreciate being able to plot the results. As I stated in my comment to topepo, I've *never* gotten a perfect prediction rate for training data. There's always been *some* error, and I appreciate visually representing the data. In essence, I suppose my plots above are really just visual confusion tables, showing how "heavy" the clusters are at each real vs. predicted intersected point. I've just never seen 100% accuracy ever, so it seemed odd to me.
I'm totally unfamiliar to the program you use, but theoretically it is correct to compute silhouette statistic for a separate cluster - the statistic is basically computed for each object and serve as an index how "well" or "poorly" that specific object was clustered
I think rather in this example it would most typically be mean = 4.1 CI = 2.6 to 6.2.  I.e. the confidence interval is usual expressed as total amounts rather than differences from the mean (at least that's what I've seen)
No problem. Good luck!
I like these replies.  These are very fundamental and fruitful topics.  You also may want to look up the terms *partial correlation* and *statistical control*.
Perfect, thank you very much! Well deserved bounty ;-)
Correct me if I'm wrong, but isn't that the same as using the difference scores (i.e. residuals) in a linear regression as the features with the only difference being that now I'm using a more sophisticated model (i.e. GAM)?
Could you provide more specific information about the structure of the dataset and the purpose of the analysis? From your present description is sounds like the data consist of a few counts of occurrences of traits.  Instead of being the multivariate tableau required for PCA that would just be a collection of univariate frequencies.
Do you care to tell us how many independent samples (cases) you have?
When you say, "I have tried to divide the output by 100," can you explain more clearly what you mean?
usually, in GLM models, the predictor ("independent") variables are just supposed to be some known constants, there are __NO__ distributional assumptions about them! So there is nothing wrong in using count data as predictors.
Good job answering your own question-- this explanation makes sense.  Feel free to accept your own answer.
Thanks for your advice, seems like I didn't approach $n<<p$ problems like that, interesting.
The paper has indeed been accepted, just a few months ago. You can download it here https://gtas.unican.es/pub/378. This is actually a quite rare problem (which may explain its lack of popularity), but we still managed to find some interesting applications. Have a look at the experiments at the end of the paper if you like.
"... you're making more and more discoveries...": A known phenomenon with FDR is that you can gain power by adding false null hypotheses, or removing true nulls [1] (referred to as "cheating with FDR"). This is not necessarily your case, just a suspicion of mine.

[1] Finner, Helmut, and M. Roters. “On the False Discovery Rate and Expected Type I Errors.” Biometrical Journal 43, no. 8 (2001): 985–1005. doi:10.1002/1521-4036(200112)43:8<985::AID-BIMJ985>3.0.CO;2-4.
And why do you want to replace A and B by a combination of the two?
Or for a similar sort of idea, [Hinton and Neal (1998)](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.33.2557)
First of all, thanks for your edits. I will provide a citation of an article which accactly does what I want to accomplish.
As always, a really excellent answer Gavin. Thank you! That clarifies things a lot, and I'll be using PCA then. Given that the intertidal community is 3 dimensional, the percent cover actually went 100% in some cases when the the organisms grew over each other. This is not the closed compositional form that you are talking about though, right?
Thanks so much. Now that I see it, I feel dumb.
It seems to me that the use of words is not independent, & so the probability of any particular *combination* of words will need to be estimated separately from data, rather than built up from estimated components.
no experience with amos but i think the factor should have a variance specified / estimated (or set to one)
I have now added a specific example that is maybe clearer. And yes, by `clusters of variables` I do mean clusters of variables rather than clusters of samples.
Why do you want to compare the correlation matrices, and not apply a suitable tool to the 600x50 matrix with the population membership as the dependent variable? What do you actually want to find out?