tttjjj/elt_bench huggingface dataset:
Logistic Regression is not a requirement. What do you recommend?  Other attributes are (a) number of questions posted on the forum (b) number of tags attached to the question (c) number of days on the forum (d) number of days since first post on the forum (e) number of times the OP has commented on his post
@Scortchi Someone might argue that it isn't usually thought of in those terms. I'd say that I wouldn't think of it as an estimator\\*, but that it is a sample statistic. \\*(at least not one I'd care to use for anything in general; in particular instances perhaps)
Apologies, it looks like CV doesn't make this distinction between mixed models and mixed effects models. (I'm not sure why both tags are needed on CV then?)
Did you really mean to say that the logit "transforms the binomial distribution in a normal distribution"??
In the case of ordinary regression this sequential approach (when correctly carried out, which regresses $y'$ on the *residuals* of $x_1$ rather than $x_1$ itself) is *identical* to multiple regression (see the explanation at http://stats.stackexchange.com/a/46508). Thus it is questionable whether there really is any reduction in computation. You might as well do logistic regression (or any GLM) with all the variables at the outset. That ought to yield--perhaps with appropriate coaxing--all the information, and the same information, you seek to obtain through the sequential method.
The Spearman correlation will capture the linear correlations that the Pearson correlation captures between the non-ranked transformed variables, along with some nonlinear correlations. Also, correlation matrices only capture pairwise interactions. It seems to me that the loading you obtain from PCA are your primary interest, so I would just stick to the Spearman correlation. It is not clear to me why you want to capture higher order interactions between your variables.
Also: http://stats.stackexchange.com/questions/31641/do-inputs-to-a-neural-network-need-to-be-in-1-1
See the genetic sequences at http://anson.ucdavis.edu/~mueller/changetrees.pdf (at the end).  A dataset of 56 cases where a slope exhibits a discontinuous jump is given at http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#statug_mcmc_sect048.htm .  Googling "multivariate changepoint problem dataset -time" yields promising hits.
That is the type of book I am looking for. I think gung's suggestion is very interesting but more targeted at general career planning. If Michael Chernick or other experts in biostatistics have further key books this advice would be much appreciated!
Okay, so since pot size does not affect plant height, there should not be an interaction; I also see why we can't call cultivar a confounder when testing for equality of means among pot sizes.  Say that after testing for interaction I find none; then I go ahead with calculating the CI under the 2x2 factorial design with the point estimate $$\\frac{W_1 (\\bar X_{AL} - \\bar X_{BL}) + W_2 (\\bar X_{AS} - \\bar X_{BS})}{W_1 + W_2}, \\quad W_1 = \\frac{n_{AL}n_{BL}}{n_{AL}+n_{BL}}, W_2 = \\frac{n_{AS}n_{BS}}{n_{AS}+n_{BS}}.$$ Does this estimate control for the cultivar effect?
+1 - this has nothing to do with *Cox* regression as opposed to any other form - it's a question about the nature of the variables in the model.
Although I suppose I could specify 2nd- or 3rd- order adjacency for the county $W$; this would perhaps allow me to use the $\\rho$ estimated from the state model.
Cochran Mantel Haenzel test.
Without using any jargon (and especially, without using the word 'significant'), what are you actually trying to achieve?
I use R-2.14.2 on Windows and the last update of lme4.
It doesnt look like it will fit in a polynomial model? Or will it? How to find the best non-linear model which will fit here?
I think this makes great sense. Excellent work. Thank you very much!
$(E[Y])^2 = (\\frac{1}{2}\\int_0^1 1-x dx)^2 = (\\frac{1}{2}[x-\\frac{x^2}{2}]_0^1)^2 = ([1-\\frac{1}{2}]*\\frac{1}{2})^2 = (\\frac{1}{4})^2 = \\frac{1}{16}$

So, $Var(Y) = \\frac{1}{6} - \\frac{1}{16} = \\frac{5}{48}$. As a result, we can compare the MSE for the estimators $\\hat X_1 = E[X|Y]$ and $\\hat X_2 = Y$ and see that $\\hat X_1 < \\hat X_2$ which makes sense given that the conditional expectation is the best predictor.
could i e.g. simply use this code for the williams test and replace the correlation coefficients with the partial correlation coefficients? http://imaging.mrc-cbu.cam.ac.uk/statswiki/FAQ/Will
Captain Holt's reaction to that movie pretty much sums it all: http://www.mit.edu/~jjenny/brooklyn99-statistical-analysis-cc.gif
Thanks a lot Sir. I will go through it.
StackOverflow is a QA site for programming questions. You question seems more theoretical and statistical, and thus off-topic here, but on-topic on CrossValidated StackExchange.
I think you meant "The median is more resistant to such errors than the mean". I agree with your comments though, and I believe the US census typically reports median's for many categories in official reports (not just the age) for basically all of the same reasons. Income is maybe even a better example than age to illustrate such points.
Thank @Ben, I tried the copula package. 
`library(copula);
tmp <- normalCopula(0.5, dim=3);
x <- rCopula(10000, normalCopula(0.5, dim=3));
y <- cbind(qbinom(x[,1:3], 1, 0.5))`

`cor(x)` is 0.5,
`cor(y)` is 0.33; how does the 0.33 come up?
For univariate regression, it is difficult to think of any real advantage of using $R^2$, because--as the OP notes--that actually loses information about the direction of the relationship afforded by the sign of $r$. Maybe this question should be approached by expanding consideration to the case of multiple regression, where $r$ has no clear analog but $R^2$ remains interpretable. Another way to address these issues is in terms of ANOVA, which seeks to decompose measures of variation into additive components: that practically forces $R^2$ upon us and justifies your answer.
Both assume continuous variables, so that the probability of ties within a sample is $0$. With the given data, there will be a lot of ties.
I would perhaps say compute a new variable in the current dataset using the random variable functions (it will that way be the same number of cases), then use `varstocases` to reshape the dataframe and then one can use the regular independent samples t-test command. I don't quite get the point of the exercise though.
What a great answer @whuber just gave.
@probabilityislogic, your point is well taken. However, my intent was to address terminology in order to (hopefully) reduce confusion and (possible) misinterpretation. A density function supported on a compact set is pretty much the antithesis of "fat-tailed".
Again, I suspect (but obviously don't know for sure) that everything's fine. Orth. polynomials are good for numerical stability and hypothesis testing, but (as you're figuring out) the actual parameter values can be harder to interpret. The current version of lme4-devel (I just posted a version that should pass tests, it might take ~24 hours to rebuild on r-forge, unless you can build from SVN yourself) should give you matching predictions between raw/ortho polynomials. An alternative is to center and scale continuous predictors à la Schielzeth 2010 *Methods in Ecology & Evolution* ...
I used to do a little of this, but long ago and I've forgotten details. But it's a large and complex field. Amazon lists a bunch of books if you enter FMRI data analysis.
it is easy to find out on help pages or any very basic manual that you *don't* need to explicitly include intercepts. So `m1 <- lmer(dependent ~ pred1 + ... + (pred1|group) + ...` is sufficient.
when you have more specific and technical questions, [cogsci.SE](http://cogsci.stackexchange.com/) would be a good place to ask them.
Your not wrong, but your definition of what a fixed effect is is not what I would think of when someone says fixed effect. Here is what I think of when someone says fixed effect http://en.wikipedia.org/wiki/Difference_in_differences , or this http://www.stata.com/support/faqs/stat/xtreg2.html (particularly equation 3 on the Stata page)
You are right. SVD is a method to obtain a solution to the PCA problem.
(+1) Nice answer. I'd add `alpha=0.5` to the first plot (to `geom_density()`) so the overlapping parts aren't hidden.
It's not clear to me that this is a statistical question at all.
@guest43434: In other words, $E(c) = c$ where $c$ is a constant.
@djhurio $Y={X_1...X_N}$ is a population with N individuals, $y={x_1...x_n}$ is a sample of the population with n observations. $\\mu$ is the mean, I should had used a subscript to separate them. $\\bar y=\\sum x_i/n$ is the mean of the sample.
@Mariana Do you mean "general linear model" (http://en.wikipedia.org/wiki/General_linear_model ) or "generalized linear model" (http://en.wikipedia.org/wiki/Generalized_linear_model )?  The second one handles binomial dependent variables.
Both are useful: they address different questions. The test of proportions evaluates whether the data indicate a difference in proportions of yeses between the `low` and `high` groups. The CIs provide intervals in which the true proportions are likely to lie. You should consider including estimates of those proportions in your summary.
Excellent.  Link seems down, try this one.  [http://faculty.vassar.edu/lowry/prop1.html](http://faculty.vassar.edu/lowry/prop1.html)
Getting 40 significant results out of 40 tests suggests those tests aren't telling you a whole lot: they are either the wrong tests to use or you're using them to point out something very obvious.
As the outliers occur in seperate variables with only one per variable (in the image), the mean is probably not affected to much. In a condition with 20 outliers, it depends on how they are distributed. Are they concentrated in specific variables? If so, are these varables still normally distributed?
What are you trying to forecast?
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.37.3325
A closely [related question](http://math.stackexchange.com/q/85448/26851) has been answered on math.SE.
thanks for answering
Thanks for suggestions!
Regarding the discreteness issue: I choose some bins for the histogram 1-5, 5-10... and etc.
+1 to this clear advice. In addition, the scatter plot is a simple key. If a linear model seems applicable then zeros in the predictor are not intrinsically invalid. It may be that zeros and non-zeros are better handled via an indicator (dummy) variable.
Almost certainly, though it could also be done with a chi-square test.
"*Could someone get me started on this?*" -- You have not sufficiently met your obligations for self-study problems. Look at the first sentence after (1). What does it ask for? Can you generate a single random sample (of some given size)? If you can, then you need to go back and read the tag wiki info for the self study tag and edit your question to explain what you have tried. If you can't, this is the wrong place to get basic instruction in elementary R-coding. In that case I'd start with reading some of the many sets of introductory documentation that is available on CRAN.
What you've done is create a factor that is the product of `level3` and `level2` and (based on your comment that it's taking 20+ minutes to run), this product factor has _many_ levels, many of which probably only have a couple of observations in them. So, the model is having a hard time estimating this random effect variance. Is this what you meant to do?
A *Mathematica* simulation with $10^7$ iterations produced $0.119983,$ which is likely to be correct through the first four digits.  The code is `n = 10^7;
Total[Boole[Equal @@ (#[[Ordering[#, -2]]])] & /@ x =  RandomInteger[{1, 100}, {n, 25}]] / n`
Thanks for completing my answer ;-) (+1)
Changed 'replicated' to 'repeated' as these are not the same in statistics.
@probabilityislogic, while your response is interesting, maybe it could be refocused to better address the OP's question. Also, ***no*** software for computing least-squares solutions performs matrix inversion, much less a determinant. Ever. Unless it's inverting a $1 \\times 1$ matrix.
If you can get your hands on the book/PDF `Understanding Complex Datasets:
Data Mining with Matrix Decompositions`, I would highly recommend that for interpreting SVD + other methods. I read it too long ago to recall the content though.
Thanks Rio. I will take a look at this website.
As far as I remember, the output of Naive Bayes is sum-normalized to fake probabilities. Given this and [this piece of documentation](http://weka.wikispaces.com/Making+predictions), the prediction column represent the sum-normalized estimated probability that e.g. instance 1 belongs to class 1 (with label 0) or instance 7 belongs to class 2 (with label 1). Can you please post the output if you use -distribution (see again linked documentation) ?
I'm not looking for the name of the function. I'm looking for the name of the *algorithm* that the function implements.
Levene's test is based on ANOVA. Why not approach it in the same way as pairwise means comparisons in ANOVA?
@ bistaumanga: linearity is understood with respect to the parameters i.e. $\\beta'$s and not the $X$'s. Even if you have terms like $X_1^2$ or $X_1.X_2^2$ you can simply let $Z_1=X_1^2$ and $Z_2=X_1.X_2^2$ and your model remains linear  i.e. $$\\ln\\frac{p}{1-p}=\\beta_0+\\beta_1Z_1+\\beta_2Z_2.$$
Why don't you try ? :-/
You simply must read Mosteller & Wallace, *Applied Bayesian and Classical Inference, The Case of the Federalist Papers.*  It is a classic, it's readable, and if by page 12 you aren't thoroughly convinced it has plenty to show you, just return it to the library or bookseller.
I am still getting 0.4 in matlab. I have followed the syntax. Can anyone confirm in matlab?
A quick [google search](http://www.google.com/search?sourceid=chrome&ie=UTF-8&q=Dickey%2C+Hasza+and+Fuller) brings up several resources that might meet your request. Including the [SAS/ETS users guide](http://books.google.com/books?id=SPIXvo-N_wkC&pg=PA145#v=onepage&q&f=false).
1) If you're doing a Box-Cox transformation and then fitting to a Weibull, the data itself isn't distributed Weibull, 2) What is your time stamp?  Wind speeds are highly correlated over short time intervals, and also have very strong patterns within the day, typically.  3) A small goodness-of-fit p-value basically means either "this distribution is a poor fit to the data" or "I have a lot of data, so no off-the-shelf distribution will pass a GoF test", with some mixture of the two for intermediate sample sizes.
**My guess**: You are calculating (1) on an *uncentered* and *unscaled* matrix, whereas the call to `ridge` (as per the MATLAB documentation) indicates that (as is standard): *[By default, b is computed after centering and scaling the predictors to have mean 0 and standard deviation 1. The model does not include a constant term, and X should not contain a column of 1s.](http://www.mathworks.com/help/toolbox/stats/ridge.html)*.
What do you mean by the fourth constraint, "The magnitude of a is .."
@atiretoo Yes, it also happens when i run the code. I dont know why. Maybe anyone else knows? jose
No, the context would not help.  Data splitting for your sample size is not reliable.  I.e. you get different model and different test data performance if you split again, and the mean squared error of your performance metric is high.
This is a pretty big topic once you get into it. My suggestion is that after you read that book read this. It is very rigorous. http://www.amazon.com/Probability-Theory-The-Logic-Science/dp/0521592712
@StumpyJoePete Why don't you make an answer ... provide an  introductory paper or book, some description and why it is suitable and you are done :).
Another school of thought, differing from your opening remark, is that proprietary statistical software *must* document its procedures in order to be trustworthy.  One expects such details to be available *especially* from commercial vendors.  The question isn't asking for code: there is no potential for compromising a software license.
This problem does not appear to have enough information to be be answerable. Is it a real problem that you face? If so, please clarify what you mean by "best."
@MichaelMcGowan: if one starts with a prior point mass at $p=0.5$ there will be a posterior point mass as well. Whether or not this prior makes sense depends on the problem...
Just out of curiosity, is your statement of the question taken verbatim from the assignment?
+1 - The user experience site [had the same question](http://ux.stackexchange.com/q/36408/10639), and this basically sums up all of the responses over there.
+1, and nice of you to share the article! It might be indeed worth writing the code. The article is quite mathematical though (not easy to understand by data analysts like me). Can you point me to the main places there, particularly corresponding to the 4 stages of the algo you describe?
--ci 0.95 command could help??
Based purely on their merits as answers, I would downvote both regardless of anything else - glen_b, this answer is a comment, if anything, to the main question. I don't see how this answers the OP's question, in any way. Regarding @Peter's answer - *"That there is no relation between slopes and init_diversity, after controlling for seed_population and the interaction"* - no, that is not the null hypothesis that is being tested.
@whuber: I updated my post.
@longtalker you can plot the between subjects error bars if you like. Visualization here is a matter of art and tradition more than a matter of what is 'right'. I typically like the plots I see to reflect the salient aspects of the test they are related to. But, there is room for latitude here.
Closely related http://stats.stackexchange.com/questions/2772/what-is-the-difference-between-pca-and-asymptotic-pca
Do you have additional infos (descriptive features) or just the id of the objects ? How does your problem compare to this one: [Rank ordering and/or classification problem](http://stats.stackexchange.com/questions/9003/rank-ordering-and-or-classification-problem)
A [reproducible](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example) example always help.
@joriki This approach is sensitive to the sum, not to the actual numbers and their differences. Firstly, for first example 73% is too low, it should be more than 90%. But this sentence is not so mathematically strict. Secondly, what's more important, if there's one cluster with `100` outcomes, and ten with `10` outcomes each, then first will be measured as 50% confidence, when in reality in such situation first cluster is almost always correct. So, more "noise" - more confidence that not-noisy cluster is correct.
I additionally suggest to move (!= double post) this question to stackoverflow. Although I am seriously not happy with such a move (Crossvalidated is dedicated for ML !), one cannot ignore the fact that more ML people are hanging around on stackoverflow (as far as I see).
I have some notes on the topic here: http://cgibbons.us/courses/ps239/Selection.pdf
You need to formulate a hypothesis.  "Samples are different" is not a hypothesis. If you said "these heights are different from the heights of XXXX" you could test it. XXX could be taken from some general population for which you know the average height. Or you could just take a number (e.g. "these heights are not equal to 175 cm".
That's exactly what i'm doing, thanks!
@Adam In the on-line help for Anova(), you will find a working example with within- and between-subjects factors; the main caveats of Anova() is that you need to pass idata/idesign parameters in addition to your lm object. Also see this thread on the r-help mailing list, http://j.mp/9lphud.
Rushdi, it then reads `Although it’s unlikely that you know sigma when the population mean is not known, you may be able to determine sigma from ... a pilot test/simulation`. Is that not clear? You can know sigma from 1) literature, 2) your pilot study, 3) special simulation processes. It is noteworthy that if you use some (previous study's) _sample_ sigma it is better to pick the formula's Z from Student's rather than normal distribution.
Have you noticed that to convert $x-\\theta$ to $x$ you only need to add $\\theta$?
Hi, couldn't you index both series? Let's say the first observation in each series is 100. Then you have them on the same scale, and you can compare them.
+1 This is a much better answer than my attempt. Thanks.
Ripley's K is a global measure as well. But I agree with your first point, and one may be interested in simulating local autocorrelation as well as global autocorrelation. The ambiguity of the OP's question makes it difficult to give a more thorough or directly pertinent answer.
I think its always better to have 50% of goods and 50% bads as per the rule of thumb. The out of og the model must not be biased on the sample.
You seem to be taking for granted that **"any control variable that has no effect is usually left out of the model"** - why is that? Unless you have a specific reason for doing so, it seems that any variables that must be "controlled for" (based on substantive knowledge) should be left in the model regardless of their significance.
Thank you. It has been undeleted and is now available again.
The use of "censored" in this answer is non-standard. When censoring at a boundary occurs, although the actual values are not recorded, the fact that they fall with a specific interval beyond the boundary *is* recorded (and therefore they constitute "observations," albeit unquantified ones).  This causes the distribution to have positive atoms outside the boundary, which is not the case for the lognormal, $\\chi^2$, or Gamma distributions and does not seem to be the subject of the question.
@Kat, what does it mean that "in some cases... it is a time that resembles infinity"? Are there cases where the searcher never finds the target before you get tired of waiting? If so, and you remove those before analysis, you will end up with biased results. You should probably work with survival analysis, possibly with a statistical consultant.
Not exactly, you said that it is used exclusively for identifying causal effects of a treatment.  My examples were meant to show that it is commonly used in studies such as case control studies where causal inference is not the goal.  The idea is to correct the inference because of lack of randomization rather than to prove a cause and effect relationship.  Don Rubin and Judea Pearl have formalized the concept of causal inference.  In many statistical problems we have data that show that variables X and Y are correlation but that alone does not prove that X causes Y or vice versa.
Thank you for the welcome! No, answers can be in any software.
Computing dummy variables works, but I was trying to avoid that since it makes the PCA result even harder to understand. Thanks ttnphns for the hint with the CatPCA, I will look into that. I hoped there is a way to calculate a PCA for all these mixed data types combined.
Perhaps I could be of more help . Please post data for 148 days for say 5 of your companies and I will try and be specific about my recommended analysis.
What is it that you want to test?
I would probably take a look at the [rms](http://cran.r-project.org/web/packages/rms/index.html) package and its `cph()` command.
Sure, there are plenty of ways. It's important to specify more precisely what you mean by "multivariate gaussian" and non "homogeneous." After all, you can take any finite number of locations and generate multivariate gaussian values according to any positive semidefinite covariance matrix associated with those locations. Ultimately this kind of construction, by its very generality, is not very useful: what is your *real* question? Why do you need to generate such data?
Oh so with these aggregate numbers, is there nothing useful that can be said? It feels like there's gotta be at least some information in those aggregate statistics..
See also the `INLA` package and papers by Havard Rue.