tttjjj/elt_bench huggingface dataset:
@EdS I think this was what A-L Boulesteix used in a benchmark study (should find the notes I took during an informal talk she gave to my colleagues and me, though -- so let's take it for granted)
you might want to read the counter example at the begining to [this](http://stats.stackexchange.com/a/56404/603) answer
Hi Laura B, welcome to CV.  Please read the [faq](http://stats.stackexchange.com/faq) especially the early part relating to asking questions. Please add the `self-study` tag to your question (since it's clearly course-related). Please also read the [self-study tag wiki info](http://stats.stackexchange.com/tags/self-study/info). If you do those things, you'll notice that the sort of question you have asked places some onus (and restrictions) on both the asker and any answerers. Please try to edit your question to fit with those pieces of information ... (ctd)
@Glen_b to be honest, I'm not completely sure what you mean. I thought my approach with the interaction was something like that ?...A line for one treatment group and a line for the other, and where they cross is the point at which both treatments have the same outcome, but when I ran this model on the data, this point would be in negative time-to-treat (if I have interpreted the output correctly)
[Examples of types of average](http://en.wikipedia.org/wiki/Average#Types)
nice, I see it is a new entry to Wikipedia.  Thanks.
"bets"? are you talking about a standardized coefficient?
The original paper is freely available at http://sci2s.ugr.es/keel/pdf/algorithm/articulo/shapiro1965.pdf.  It is linked from the first hit in a Google search for [scholar: shapiro wilk test](https://www.google.com/search?q=scholar%3A+shapiro+wilk+test).
I`d like to mark this as answered (thank you!) but I can`t find how to do that.
Thanks for your answer! So basically the motivation behind the concept is "perfect sequences gives a decomposition of a graph, which is handy because we can factor a density function over those cliques"?
The start of a solution is outlined at http://stats.stackexchange.com/questions/32699, where the question (implicitly) suggests computing the *raw* moments of the mixture and combining them (according to the definitions) to obtain the variances and covariances.
A guess: The heatmap corresponds to the ellipsoidal contours of a multivariate Gaussian density obtained by fitting the data (probably by maximum likelihood or similar). The white contour lines correspond to the conditional (Gaussian) density obtained from conditioning on the given height, so, e.g., the bottom line is the density for weights given a height of 140 (cm, I'd guess). Do you have access to the underlying data?
+1  This is a nice complement to my answer.  I tried to keep my answer simple and intuitive.  However, it is certainly true that there are more complexities than I discussed, and that they can be very important.  I should have acknowledged that in my final paragraph.  Thanks for keeping me honest.
Cool! I didn't think of splitting up trick.
@Nicole: it may be possible to use `arima`, but it appears to me that all it saves of the original data is the residuals (innovations), though it wants to predict from the end of the original data forwards. R has LOTS of packages, though, so there may be something that works. I'd suggest looking in the CRAN website's (cran.r-project.org) Task Views to get a feel for what's out there.
And if that's not a suitable answer ... you might ponder what essential piece of information you omitted.
This is probably more appropriate for the R section of Stackoverflow.  Thanks for providing the data; however, it looks like your data column names might be slightly inconsistent.  you have p1 <- MS1/(MS1+M2), but I'm not seeing MS1 in the dataframe.  Should that be M2?
It certainly seems consistent with your thesis that the original (untransformed) data are close to lognormal in the body of the distribution but with an approximately Pareto tail. These images could perhaps be added to the end of your question as a followup.
@Jase, well I used the term in a sense the OP asked, i.e. find meaningful statistical relationship.
I have 8 categories and not ordered. In fact I have 2 categorical variables and multiple numerical variables and I want to train a logistic regression model. I want to test the independence between all my variables (only predictors), and remove the ones that are dependent, meaning that they are redundant in my model.
What is the question you would like to answer with this regression?
I meant to write 'skew', not 'whew'. I have corrected the text.
Good question. It should be suitable for a beginner but it is not going to be easy. It depends on how big is your motivation ;)
@walkytalky You mention “other things that don't properly apply”, but let me give a counter example. The [phi coefficient](http://en.wikipedia.org/wiki/Phi_coefficient) and [Cramér's V](http://en.wikipedia.org/wiki/Cram%C3%A9r%27s_V) rely on the determinant of such a contingency table:
$$\\phi=\\frac{n_{11}n_{00}-n_{10}n_{01}}{\\sqrt{n_{1\\bullet}n_{0\\bullet}n_{\\bullet0}n_{\\bullet1}}}=
\\frac{\\text{det}\\begin{bmatrix}
n_{11} & n_{10}\\\\n_{01} & n_{00}
\\end{bmatrix}}{\\sqrt{n_{1\\bullet}n_{0\\bullet}n_{\\bullet0}n_{\\bullet1}}}$$
@rosser no worries!  I had the same question the first time I used `caret` in R to cross validate a model.
Thanks, that is great. I have been exploring a little, and I think my `m2` has an extra hierarchy of `Subject` nested within `Subject`, hence the unexpected output.
@Christos I can only think you are missing the point that it is $8$ calls in total across both of the first two hours, rather than $8$ calls in the second hour only
How can you hope to compare teams in different leagues? Suppose the best elementary school team is undefeated. In what sense do you want to say they are better than a mediocre professional team? If the two played, I would bet on the professional team to crush the kids.
I've removed the tag "logistic", which seems quite irrelevant here. If I missed the point, please edit the posting to explain otherwise. (The use of a binary predictor is immaterial.)
Why would you want groupings in the first place? It rarely makes sense to categorize continuous data; it may help you to read my answer here: [how-to-choose-between-anova-and-ancova-in-a-designed-experiment](http://stats.stackexchange.com/questions/24077//24080#24080), especially below "update". I readily acknowledge that the context differs from yours, but the idea is that discretizing data isn't generally a good thing to do.
I added the time series tag, so some of our experts on time series may take a look.
(+1) Unfortunately, I don't have Feller vol. 2 handy and am curious to see his proof. Slow variation is our friend; the simplest way I can think to make it work is to fix $\\epsilon > 0$ and choose judiciously $\\newcommand{\\one}{\\delta_1}\\newcommand{\\two}{\\delta_2} \\one = \\one(\\epsilon) > 0$ and $\\two = \\two(\\epsilon) > 0$. Then, we "split" on them as follows, $$\\{X_1 > (1+\\one)x, X_2 > -\\one x\\} \\cup \\{-\\one x < X_1 < (1+\\one)x, X_2 > (1+\\one)x \\} \\subset \\{X_1 + X_2 > x\\} \\subset \\{X_1> \\two x, X_2 \\leq \\two x\\} \\cup \\{X_1 \\leq \\two x, X_2 > \\two x\\} \\cup \\{X_1 > \\two x, X_2 > \\two x\\} \\>.$$
Related to your question [Predicting response for two groups in multiple regression](http://stats.stackexchange.com/q/36019/930) (under a different account). Could you please indicate to one of the moderators (by flagging your post here) which account you want to keep and [register](http://stats.stackexchange.com/faq#login)?
$\\int f(x)^2 dx$ is also $\\mathbb{E}[f(x)]$ where the expectation is with respect to $X \\sim f(x)$. Does this help?
Each terminal node (aka leaf) will have a different mean and equation.  You could use something like R's caret() package and just use your tree (i.e. your model) to predict() for your new variables.
*"continuous variables can vary according to groups"* - could you please write the model? I'm not sure I understood your example, but I think it could lead to ANCOVA with interaction, and the random effect could then still be applied only to the parameter of categorical variable - and then this example would not lead to the answer *NO* to my question. Maybe if you sketch the model, we will see it right away.
If you post a piece of your data it will be easier to give you an answer. If possible, please post the output from `dput(x)`, where `x` are the temperatures recorded at one sensor. If the data set is large you could post just some of the observations.
Since your question is very broad -- "how does one interpret  a binomial regression?" -- I would suggest picking up an introductory text on the topic. Agresti's *An Introduction to Categorical Data Analysis* is very approachable.
the PCM and generalised PCM can also be estimated using the `ltm` package, if you encounter a poor fit using the Rasch PCM
Minute comment: polytomous is much better Greek than polychotomous. Both forms are common, but poly- matches dicho-, not di-.
Is there a reason for using hierarchical clustering?
For an example of computing standard errors of estimates for linear functions of a trinomially distributed variable, please see http://stats.stackexchange.com/q/18603.  Although the reply there does not fully answer your question, it exemplifies techniques useful for obtaining an answer.
@user603 done. Sadly Gomez et al (2007) is not available without subscription.
@subhash c. davar: It is in turn incorrect to say "multivariate correlation" is not included in statistics. Very likely it is not what Katie wishes to do, but that's a different detail.
A simpler case of: http://stats.stackexchange.com/questions/26020/geometric-distribution-without-replacement (I am happy to draw up to X+1 balls, or more...)
Always check wikipedia first. If you enter Tukey HSD it immediately directs to the page about Tukey HSD where is a section how to calculate p-value. This is how I found it, I had no previous knowledge about this statistic.
Use `tbats()`. It is ideally suited to this problem.
I am using 3 independent variables( Recent Terrorists attacks,Past trauma, Major Life stressors) 3 dependent variables ( PTSD, Depression & Anxiety)and 2 moderating variables(Self Efficacy and Social Support). All variables have different Likert Point Scales. If I take average of each variable separately and then run regression analysis, is it Ok. What will be accuracy level of this regression analysis?
Ali-Jena, I am largely ignorant of those methods. I suggested the GECM because there were so many similarities in your description of your data and problem to an analysis in which I found the GECM to be useful. That said, there are differences also, so you mileage may vary. :)
I guess you mean "median survival time" rather than "mean survival time". The median survival time is easily found to be $t_{\\textrm{med}} = exp(\\mu)$.
There is no one answer to your question. It all depends in the estimated impact and the error variance from the model. If you post your data and the assumed point of change, I will try and help you.
+1 The division into three concepts is memorable.  The thorough discussion, with examples, makes a good answer.  The language is nicely matched to the target audience.
Yes this is how I was thinking of the problem. If the candidate function is the average, then $Y$ having a (scaled) binomial distribution would have the desired property and you've given such a distribution for $max$. Either a crucial detail is missing from the problem description, or our point is not being fully understood.
Consider the relationships between SD, variance, and total sum of squares about the mean.
Nice comment! Very sincere. I was thinking if you could restate the problem like this. You have "bear events" at days $t_1,t_2,\\dots,t_n$, and stocks prices at the end of those days, and you want to say which stock is more probable to fall early when those kind of events happen again in the future. So, it is a problem of prediction.
Dropping the linear version of the variable because it is not significant is not recommended, see here: [Does it make sense to add a quadratic term but not the linear term to a model?](http://stats.stackexchange.com/questions/28730/) In addition, I discuss the interpretation of such variables in my answer on that thread. IMHO, this question should be considered a duplicate of that one. Please read through it. If you have any remaining questions afterwards, edit your question here to clarify what you still need to understand.
How did you get from 39 questions to 13 variables?
1. Does the graph show the aggregated / cumulative rate ? Or the rate per half-day-interval ? 2. 200.000 runs per test means 100000 per group, is this correct ? 3. I guess that the test has been stopped at the beginning of the sharp decline, is this correct ?
Thank you very much, that's a great help! I'll try exploring the glmer model a bit more with interactions - I'll be adding in data from a different participant group soon, so there'll be a bit more to explore. But it's very reassuring to know I'm not making a complete mess of the glmer from the beginning, thank you.
I agree, but as I am new here I couldn't create that tag.
@whuber Please see in the question where Gary King explicitly says that he is not talking about statistical power, but about "the power that can be extracted from knowledge of the substance".
"*the raw data is not normally distributed*" - why would this matter? What assumes raw data is normal?
@Macro and in the functions what is n1, n2, p1, p2,y, k ?
Perhaps your question is answered [here](http://stats.stackexchange.com/questions/60383/bonferroni-adjustment-in-spss-what-does-it-do).
@Tomas, I fear you are incorrect.  The code (at line 5) explicitly declares `h[N]` to have a Poisson distribution with mean `lambda[N]`. As the MCMC sample count rises, if there were no posterior variation in `lambda[N]`, this will converge to a `lambda[N]` distribution, not a constant.  Due to the variation in `lambda[N]`, the distribution of `h[N]` will in fact converge to a mixture of Poissons.
Thanks for your answer. Here's another link to the [paper](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.31.9325&rep=rep1&type=pdf)
You find the equation on page 14 (2480) as part of a longer equation. My idea was to multiply it out and regress on z as an exogeneous variable (which hasn't an own equation in the VAR).
@cardinal: We could add an initial phase uncertianty, but that would complicate the model. Given that there is a reasonable physical argument for knowing the initial phase, we decided to stick to the case of one unknown parameter for our first paper on the topic.
As per our [FAQ](http://stats.stackexchange.com/faq), we don't provide direct answers to homework problems (at least not for a period of time), so that you can learn by working it out yourself. Here's a hint: what kind of model can handle a response variable (ie the result) that is 0 or 1? Would this model allow you to statistically control for a possible confounder?
In addition, would you know of one test giving one p-value that can test the assumption that symptoms differ from each other in variance explained? I have 14 values (between 0.1% and 19.8%) of explained variance, but don't have them per subject (so I cannot use tests like Hotelling's T).
This is unfortunately a very broad question.  If you search for some resources on time series analysis you will get some suggestions on the broad approach to answering this sort of thing, but I'm afraid you're unlikely to get a good answer to such a broad question on this forum.  Time series analysis addresses exactly what you are interested in: predictions; assessment of the relative importance of possible factors (I'm afraid you'd have to identify at least the candidates yourself); issues such as seasonality; and quantifying the uncertainty of predictions.
@buruzaemon: Yes
Two-way comparisons with paired samples are not restricted to matching. Siblings, twins, pre/post measurement, or responses to a question asked to both wife and husband might all be tested using, e.g., a t-test for paired samples. Could you clarify what your situation actually look like?
@whuber, Can you please rename tag "glm" into "generalized linear model". Because there is confusion among statisticians and site visitors over GLM abbreviation. Some people take it as "generalized..." and some for "general...". For "general linear model" we have a good tag "linear model" already.
Evidently, Kolmogorov entropy is the best approach, in a philosophical sense, if you think of "abstract pattern simplicity" and you're not trying to predict how simple it will result to a human mind. It simply states the entropy as the "length of the shortest program which can produce that pattern". Of course, you still need to specify the computer language, but you can still rely on an abstract Turing machine to play the trick.
Thanks for a pointer to some sample code, I'll check it out!
N=44, not perfectly balanced accross factor levels. The goal is not predictions at this stage, just finding the best way to model sample data variance.
It would be helpful if you added the formula in case this link goes dead.
How about a two-sample t-test?
@Henry and yes vol. is for volume and if the category is not present in any hour it is 0 (though there are no such cases in the actual data :).
Hum, yes, this looks fine. In that case you could think to non parametric methods like the bootstrap. Regarding the fact you have replicates, once the data are "normalized" you should be able to pool them (under the assumption they are identical up to the measurement errors). However I am sure you can’t be the first to do that. Did you try scholargoogling "microarray expression profile" ?!
@Jana, you might also want to read the [faq](http://stats.stackexchange.com/faq). It gives a pretty good summary how this site works.
Thanks Alecos--very helpful. But what if I want $x_i - y_i$ rather than $y_i - x_i$ on the LHS? If I run such a regression in e.g. Stat--will I need to premultiply the estimates by -1 in order to correctly interpret them?
Do you not like the idea of blocking from a theoretical or practical point of view?
you might also look up the Workshop on Algorithms for Modern Massive Data Sets (MMDS). It's young, but draws a pretty impressive set of speakers at the interfaces of statistics, engineering and computer science as well as between academia and industry.
Yes, I had looked there. perhaps I am missing something, but I do not feel that the Wikipedia article gives an adequate answer to my questions.
@Momo I did what you said and what should I do now?
Can you solve this for the special case where $\\mu_1=\\mu_2=0$ and $\\sigma_1=\\sigma_2=1$?  (Hint: what distribution does $Y-\\rho X$ have?)  If you can, you're done, because the general case easily reduces to this special case upon standardizing the variables.
Did you take a look at the [task view for experimental design](http://cran.r-project.org/web/views/ExperimentalDesign.html). If so, can you specify how this does not provide the desired information?
I would generally agree but these experiments are very much related.
@user20287 No, you should not renumber if you are deleting the (missing/wrong) values. Remember that you have to let the model *know* that you have some data points missing. If you read the vignette you'll see examples of how missing values are handled. Also, welcome to StatExchange. You have made your comment an answer. You can delete that and use comments for clarifications.
The chapter is so-so. Solid effort, but hard topic to summarize. Lots of unsupported claims published on various methods. I do think stratified undersampling in random forests is a good start from a machine learning perspective. Code is in the book's package.
Thanks. I think it would be handy to have built-in functions (with min and max paraemters ala the unif family). It's a bit ugly to have to add function definitions into scripts just to use the discrete uniform distributions the way you would use other standard distributions. The builtin functions also deal with error handling (for example - if parameters aren't integers) and are optimized for speed.
Thanks, this looks like a useful resource.  However, it looks like this isn't quite written at the level that I want.  I already have an abundance of resources on how to draw appropriate conclusions from data.  What I'm missing is an understanding of the underlying math.  For example, I know in general that ML estimation finds the parameter values that maximize the likelihood of observing the data, but I don't really understand how one finds those parameter values or why different methods of ML work.
I am not sure why you think this anomalous.
@David Reilly: I am amazed at your your commendable and authoritative discussion. One Journal from UK and 1 journal from India refused the paper because I did time series analysis and forecasting to compare what is expected and what was achieved with the new treatment.
You can use many methods for feature selection, but I don't think this method reduces the multiplicity issue or protecting you from someone annoyingly noting that "Only the pre-specified model is one for which the p-value make sense/are accurate". In addition (1) shrinkage methods are really focused on prediction (2) cross-validation methods (deletion-substitution-addition) don't convince but YMMV (3) Allen-Cady is a bit outdated, but again YMMV
Times when Lasso would be better: Many of the features are meaningless (Lasso does more feature selection than SVM or RandomForest).  The relationships really are close to linear (RandomForest will struggle to approximate a linear releationship and SVMs will at least get all squiggly)  There aren't any strong interactions (RF and SVM will waste resources looking for things that don't exist)...
@Ming-ChikKao I think is not a problem because the size of the sample is 800 (800 values) so I don't think it is small for this purpose, no? Ok, so I can use Jarque as "second" test, What do you think about Anderson-Darling test for normality as first step?
I wonder if you may be confusing *dummy variable* (typically a synonym for 0/1-coded *indicator variable*), with *categorical variable*. Dummy variables genneraly have exactly two levels. There are alternative ways and analytic reasons to code nominal/dichotomous/binary variables than 0/1, but that's usually the simplest.
I disagree with your statement about equivalence of SEM on covariances and raw data. If you need to perform more advanced estimation and testing, including any of: Satorra-Benter scaled/adjusted tests, ADF/WLS estimation, Bollen-Stine bootstrap, you need the raw data. In other words, the raw data will likely allow a wider range of estimation techniques. With the covariance/correlation matrix, you are stuck with the normal theory based methods and GLS, and that will hardly work well with 80 variables, which are probably on Likert scale, too.
I can use any software. As I point out Matlab does this, but I would like to learn the numerical method behind this method; ex how to do Brownian bridge on multivariate distrubution
@Isaac Sorry, I completely misremembered what is going on here.  I should reread my own answers!  I defined mu1 and mu2 in the first sentence: if you know M1, M2, and the covariances, that determines V which determines mu1 and mu2.
Also, do you know of any source which discusses this in more detail?
This is by far the most useful answer. Thanks for your help. Again most examples of these methods are done in SAS or SPSS, what I need is an algorithm to implement myself.
Please also have a look at this video which explains this very well using the German tank problem. [Link to the video](http://www.youtube.com/watch?v=XaAtkCzdjLE)
Your question lacks some context. Supplying it might help attract an answer. As it stands, it's not clear what your question really is. As currently posed, the answer is that $Z_1 = Z_2$ (almost surely) if and only if $Y = 0$ almost surely.
Thanks........-