tttjjj/elt_bench huggingface dataset:
These are all really useful comments, thanks. The data are actually not 30 x 0/1 but Likert scales. It is a scale assessing activities of daily living. The problem is that it is designed for dementing patients and we're using it on depressed patients. The ceiling effect is caused by its insensitivity to the more subtle instrumental difficulties (financial management, shopping, complex tasks) that *recovered* depressed people have.
This sounds like it is the principle the L Method (see my answer) evaluates.
Many, many answers to this question are available by searching our site for the tags: [regression model-selection](http://stats.stackexchange.com/questions/tagged/model-selection+regression?sort=votes&pageSize=50)
I agree. I'm just saying that Bayesian inference is not *a priori* concerned with unbiasedness properties of estimators. If such properties are useful for the substantive inference demanded by the applied problem, then of course it would make sense to try to get guarantees about unbiasedness. But if an inference procedure performs well without any particular good or bad properties related to its bias, a Bayesian will still use it while a frequentist may disregard it on *a priori* grounds that it doesn't provide a particular guarantee about unbiasedness.
When you say "I'm looking at what percentage were discharged from the hospital at different time points after surgery" you're asking a question about point estimation. When you say "What is the most appropriate statistical test for this?" you're asking an entirely different question. You should be clear whether your interest is in estimation of values or testing for differences.
... in any case F1 shares the problems of accuracy @FrankHarrell alludes to: these  stem from counting fractions of test cases of hard classifcations. To get one of Frank's proper scoring rules, you'd need to switch to probabilistic output of the SVM, and then e.g. use Brier's score (mean squared error) instead of accuracy. I guess you could also derive a MSE-type version of F1. Such measures should indeed be better for the tuning step. For communicating the final performance, you may also need the typical ways (e.g. accuracy, F1) of expressing performance for your community.
@user2157668: Can you please explain what you mean by "YC1CO etc are individuals and each has many observations (millions)"? Millions of observations -- are these data points or features? What is the original dimensionality of the dataset? How many data points are there in the dataset? If each individual contributes multiple data points, then how do you project individuals on the PC1-PC2 plane?
Are you trying to model P(object in image | image, tags) or just P(object in image | tags)? I ask as even those these seem like related tasks, they would likely require very different approaches.
See [Levene's test](http://en.wikipedia.org/wiki/Levene%27s_test). Note that the assumption of homoscedasticity was not discredited; it may or may not have been met (& you could argue that it'll never be met, exactly, in the real world).
Of possible interest: [Inter-rater reliability for ordinal or interval data](http://stats.stackexchange.com/q/3539/930).
I thought the whole point of using ranks (Spearman) was to increase robustness.  Also, this could be a good exploratory tool to identify suspicious observations.  The next step is to use substantive knowledge to investigate those points before a call can be made on modelling things separately, exclude them or use other robust methods.  For some robust suggestions, I higly suggest Randy Wilcox "Fundamentals of Modern Statistical Methods".
Logarithms are appropriate in such circumstances and, if not enough, [Stirling approximation](http://en.wikipedia.org/wiki/Stirling's_approximation) gets you close enough to the actual value to evaluate the magnitude of the quantity.
A non-leap-year has 365/7=52.14 weeks. So 52 weeks is less than one year. To estimate seasonal patterns, you normally need several years of data, although if you make strong assumptions you can do it with less.
this doesn't address the issue of evaluating the final model, decision trees are generally poor predictors
Your source resorts to the kind of speculation I was referring to (somewhat elliptically) at the beginning of my answer. The internet is full of people who make the same assertion, but because it is equally easy geometrically to find an *arithmetic mean* as a geometric mean, in the end this property (of having a "geometric" construction) does not appear to explain anything. It would be very interesting to find an authority who can track down the actual historical uses of "geometric" and "arithmetic" to help us understand how these terms really arose.
I am using factor, so that it will be a 0,1 dummy variable when used in the regression.
I assume you mean when is this maximized as a function of the data. This happens when half of the $X$s are $0$ and the other half are $1$. Since this seems like homework, I'll leave it to you so figure out why that is.
A better question might be "Why does R do stepwise regression without penalization?" (as does every other statistical package).  As documented at length on this site, stepwise regression is a mindfield of disasters.  What motivated you to want to use statistical significance or AIC to choose your model for you?
Thanks for the pointer--I hadn't thought about just sending my test data into JAGS, but that should do it.
[Here are some recommendations I made a while back](http://stats.stackexchange.com/questions/1324/parametric-techniques-for-n-related-samples/1326#1326). I still think Gelman and Hill is a good one - it builds its way up to mixed models from a pretty basic level.  I still refer to it pretty frequently.
Thants it!The data give the road distances (in km) between 21 cities in Europe. The data are taken from a table in The Cambridge Encyclopaedia. Thanks a lot!
(+1) For completeness.
I am not sure how to scale the degrees of freedom. If you are multiplying a chi-square by 1/2, does it mean that the resulting variable has HALF the degrees of freedom?
Everyone decides on the analysis before they conduct the study, don't they ;)
It would probably help if you were to describe what procedure and data underly the plot.
Hi Vincent, could you explain me your "difference" function please? what's the "x" variable? thanks
can you give reference for whitening ?
I think it's a great one (I think it goes well with - "no models are true, some  are useful")
I know this doesn't answer your question, but robust standard errors (Huber-White or otherwise, check out the `sandwich` package for those) won't do? Sure, the estimation will be less efficient but at least you'll be sure the variance-covariance matrix will be consistently estimated.
A side note: the `igraph` package in `R` has a lot of graph measures for community analysis...
+1 Thank you for pointing me in this direction. Am I right in understanding that due to the presence of a few large values (potential outliers), it makes more sense in looking at the log scale?
5k is very specific. What is your population & your sample? What is the nature of the inference you want to make?
@shabbychef You are assuming the ARIMA filter which will convert an autocorreleated series to a white noise series. Unfortunately a first difference filter an inject structure see the Slutzky Effect http://en.wikipedia.org/wiki/Eugen_Slutsky which teaches us that one can create an autocorrelated series by differencing a white noise series. Assuming a filter,any filter can be quite dangerous. ARIMA model identification http://en.wikipedia.org/wiki/Box%E2%80%93Jenkins is the procedure to follow in order to yield an equation and an error process that is Gaussian. See my comments in my answer .
The help files are helpful. Please read them. In this case, it says that x may be a univariate or multivariate time series.
Owing to the unusual request and the subtleties of randomization, could you please say more about the purpose of this procedure (and, especially, what the repetition needs to accomplish)?
The effect of X is indeed very large (high beta, and a very high significance level). However, both variable A and X are already mediating variables. So in the first model I have the control variables, in the second model the independent variable and in model 3 the mediators (variable X and A). I verified the baron and kenny method with several more advanced mediation tests and came up with the same results. If I have instead of variable X another mediating variable, then variable A is also a mediator. Only in combination with variable X, variable A loses its (significant) power in influence.
Possible duplicate of [Likert scale and yes/no calculation of one score](http://stats.stackexchange.com/q/45374/32036) too.
@forecaster: Do you have any source for any of your assertions? Specially in your answer to question 1. As it stands, this answer appears unnecessarily prejudiced.
+1 because you introduce a new cue (power analysis) but as you are not actually answering the question - of course I cannot mark it as an answer. but still - very much appreciated! 
Nonetheless - your statement with "1000 dentists" confuses me a bit, as I state that there are no more than 500 in that town.
Ok thanks guy, this tells me I need to be careful which model (decision tree vs linear or GLM) I use.
This is a tricky problem, and I don't see a way to circumvent the absence of a ground truth. A ground truth could even be some kind of model of how the data is generated, it does not necessarily need to be what you call a 'truth document'. Anyway, without such a thing I do not see any approach that can be deemed objectively best. The points raised by Anony-Mousse are all very valid and good to be aware of.
No, I'm not missing the problem. The effects have a size. You can utilize them to estimate power. I've simplified the example I gave. Feel free to complicate it for your purposes. BTW, are you thinking structural equation modelling or semantic modelling? I've never heard of the semantic equation modelling terminology you're using.
Note first that your title and your post don't match. Variance of lifetime has dimensions time$^2$ and mean lifetime has dimensions of time, so the sum of the two quantities is dimensionally meaningless. More generally, I don't think you have given enough information on the set-up to make this a tractable problem.
Good to know Richard. You should be able to at least accept my answer, welcome to the site. Although to note in the future, the content of this question would be better suited for Stackoverflow than here. See the FAQ of here for more explicit details of what is on or off topic.
Thanks for your knowledgeable comment. I will explore the cpower function.
@Michael: I think A and B as distinct methods that give different boundary decisions. Each one has an adjustable parameter (what in 1D is a threshold), the parameters are independent, and give (for each) a family of classifiers. I'll try to plot a diagram to try to clarify, hold on.
80% is just threshold, because I may have a few customers who live very far away, and I don't want them to be part of the trade area. I've also tried using higher % thresholds, which produced very large and jagged trade areas. I am trying to capture geographic area around the store that represents the closest 80%, while allowing important customers who are distant to enlarge that area somewhat.
This doesn't make sense to me. You created the model so you already know what is in it. Perhaps you shouldn't have created a model? Or do you want a different model?
If you mean '*pseudo-likelihood*', see the first sentence [here](http://en.wikipedia.org/wiki/Quasi-maximum_likelihood)
See also this question  (asked with specific reference to designining clinical trials): http://stats.stackexchange.com/questions/2770/good-text-on-clinical-trials/
Indeed this is the same proof as the one provided by Paul Rubin on or-exchange.com. Multiplying each correlation by a constant has a very significant drawback though - it is a very very small stress test. It is possible to apply a much larger stress test by adding a constant to each matrix, not multiplying. However, it seems the analytical solutions for that are a little less obvious.
Your example describes another problem regarding my models - So how do you cope with that?
So, what specific estimates don't match? Your factors all have more than 2 levels, so the ANOVA table is going to show effects that take up more than 1 degree of freedom (pooled effects, in a way), whereas all parameter estimates shown in a regression output are going to be 1-df effects. In other words, in order to get the same F test and p-value in regression as in ANOVA for the effect of a factor with more than 2 levels, you would need to do a nested-model comparison between a regression model including all terms, and one excluding all dummy codes affiliated with the factor in question...
Discussion on Gelman's blog: http://andrewgelman.com/2012/11/16808/
Recall the definition of Brownian motion. There are only two requirements to be met. Check them for each of the process and Bob's your uncle.
@dlj: You seem to misunderstand nonparametric analysis. It does not require knowledge or assumptions about population parameters, but it's fine if they exist. I don't think it would ever be meaningful to say data are nonparametric.
Do **not** cross-post the same question on multiple sites, it is **not** permitted.  The question on Stack Overflow is now closed.  This is the more appropriate site as this is not about programming but asking about sampling without replacement.
You can use any method to check for lack of fit; see the help pages for `glm`, or search regarding the methods. If you are really interested in this type of model, a good book on various regression methods is [Long](http://www.amazon.com/Regression-Categorical-Dependent-Quantitative-Techniques/dp/0803973748/ref=sr_1_1?ie=UTF8&qid=1351527967&sr=8-1&keywords=categorical+and+limited+dependent)
Great response, thanks!  Any pointers on your last comment?  What do you mean by inference that is harder or easier, and what variable structures affect it?
If you want something introductory you can't hope to have a problem you couldn't reasonably do some other way. The restriction of NOT giving you something that starts showing algorithms eliminates pretty much every paper I know (and that must be a couple of hundred papers). How can a paper about an algorithm *not discuss the algorithm*? That would mean only speaking about it in general terms ... which wasn't what you wanted, remember?
@DavidJ.Harris You can estimate $\\phi$ in a linear model as well (well, calculate it directly from $a$ and $b$ in the linear model), see the post the OP linked to.
The binning you describe does not appear consistent with making mere summaries.  Why do you really want to make these bins?
@probabilityislogic thanks for the input. I will use your input in my actual code...any idea on how can augment this code to run over hadoop cluster?
Fisher's $p$ value and the Type I error rate $\\alpha$ are incompatible according to the following paper: [Hubbard, Bayarri (2012): Confusion over measures of evidence ($p$'s) versus errors ($\\alpha$'s) in classical statistical testing](http://isites.harvard.edu/fs/docs/icb.topic1387400.files/HubbardBayarri2003.pdf). Also, have a look at [this post](http://stats.stackexchange.com/questions/33453/comparing-and-contrasting-p-values-significance-levels-and-type-i-error?rq=1) here on the site.
The Wikipedia page does not really provide guidance on how cross-over designs should be analyzed.  It states that most cross-over designs are fully crossed and that beyond that the analysis is guided by the clinical trial protocol.  Mixed linear models with random effects are also introduced as an option (which may be viable).  I'll take a look at the books.
@whuber thanks for the comment. Post edited. In an effort not to run afoul of guidelines, was trying not to write too much lest it seem a discussion.
Right: "rough" in this instance includes (among other things) the fact the bounds do not evaluate lack of fit. Thus, to be realistic, one would need to *widen* these (fiducial) bounds even further. But at least they begin to indicate just how much uncertainty there is in the IC50 estimate in the absence of any information about the likely form of the response function.
$\\log(Y) - \\log(B)$ = $\\beta_0 + \\beta_1 (\\log(A)-\\log(B)) + \\beta_2(\\log(A)-\\log(C)) + \\varepsilon$ implies $\\log(Y) = \\beta_0 + (\\beta_1+\\beta_2)\\log(A) + (1-\\beta_1)\\log(B) - \\beta_2\\log(C) + \\varepsilon$, forcing a nontrivial linear relation among the coefficients of $\\log(A)$, $\\log(B)$, and $\\log(C)$ (they must sum to unity).  Naturally, $R^2$ will increase when you relax that relation.  @celenius: this answers your first question.
Depending on how you interpret it, this is a very broad question or a coding question (which is not appropriate on this site). Regarding the latter, a Google search for “SVM R” seem to return useful results.
Also, I have tried using JAGS and it also runs into numerical problems with this prior. I'm estimating quite a few parameters. Basically, I'm estimating task performance at different intervals in the first step. In the second step, I'm using these parameters to do non-linear regression.
Welcome to the site, @user383287. This isn't really an answer to the OP's question. It is more of a comment. Please only use the "Your Answer" field to provide answers. I recognize it's frustrating, but you will be able to comment anywhere when your reputation >50. Alternatively, you could try to expand it to make it more of an answer. Since you are new here, you may want to read our [tour page](http://stats.stackexchange.com/about), which contains information for new users.
hmmm, interesting point. Fact is there are tons of explanatory variables that could be causing the data to change as it does, especially in a hurricane evacuation situation, but data is very scarce in the world of hurricane evacuation. If this is the case that could explain a lot. Thank you very much with for your input. I also noticed that when I increase the sample size the p-values of some of the variables decrease to below .001 (with a sample size of 483) but that is only for 2-3, the others remain above 0.2.
@John, this is off-topic to my question, nevertheless: believe me, this is OK, I did a lot experiments with that. My first lm model is entirely equivalent to `logInd ~ Year*Group`, only the coefficients are in different shape, nothing more. Depends on your taste and what shape of coefficients you like, nothing more. There's no exclusion of "Year main effect" in my 1st model as you write... `logInd ~ Year*Group` does exactly the same, the `Year` coefficient then is not the main effect, but the Group1:Year.
I guess I should take the first suggestion as a vote of confidence.
This question is quite tricky, so I start with just a tiny comment: Bland-Altman-plots are designed to study the agreement between two different *methods*, not different *raters*. In daily work this is essential because the raters will change. (Or put differently, the raters you are using can be considered as a random draw of all possible raters whereas in Bland-Altman settings, the methods are fixed).
@Jonathan's concern is a valid one.  And if you don't have groups that are randomly sampled, or at least otherwise representative of the larger populations of interest, then doing the analysis with just 10 in one group is pretty pointless.
+1, I like.  @Thomas: Given an "innocent until proven guilty" system, you could think of type I as a primary error to avoid (imprisoning the innocent) and type II as a secondary error (guilty go free).
Are matchups independent?
No, I do not think it depends on reference, the reference would give me an idea to look up which tests you use exactly. Panel co-integration and unit-root testing is a bit murky. A lot depends on what kind of problem you are trying to solve. I suggest looking up papers of Pedroni, in my opinion he gives the most coherent picture of how to proceed with unit-root testing and cointegration.
@Srikant Vadali - sample data could be press releases, news stories, etc .. the textual data would be free-form, likely obtained from rss feeds or similar. Market data for a given company is what I'm looking to analyze/correlate. So maybe Blogger Bill writes a story about an upcoming VMware feature release, and VMW jumps 10%. (Oversimplified, I know)
No, it's not necessary for your independent variables to be normally distributed.
Good ideas for a tough situation.
@ocram You are correct, but it is quite sensible to take $X\\log X=0$, because that is the limiting value as $X$ goes to $0$ from above.
@whuber - the variance is indeed 0.25. Thanks for the help. The answer looks simple, but I could not find it online. Can you please tell me where I may get hold of the argument as i plan to refer to it in the model document. Thanks again.
Logistic regression DOES have an error term. Logistic regression predictions are in the form of a probability, not 1 or 0.
I see. Please update the original post to include this information, some biological background would be helpful. For instance, if the counts are the aggregate frequency of a certain mRNA sequence as determined by a tool like mass spectrometry then you will need to use a count model (i.e. Poisson) to test for differences in frequency using the denominator as an offset.
Isn't the p-value just less than some number, which is based on the number of simulations you did?  E.g. if you did 1000 sims, the p-value given is 0, but it's just <0.001. So you treat it as 0.001.  (Or whatever).
Hi @Michael - Can you give any more detail here? It may not be clear to some how you can incorporate a functional form for the error variance into a regression model. Did you have something specific in mind?
(+1) To clarify, multilevel modeling software handle missing data on the dependent-variable side, but typically not on the predictors side (though this seems to be irrelevant here--just wanted to make sure this didn't go unnoticed in case the OP has missing data on predictor variables as well).
Note that  $X|X>x$ is a truncated distribution truncated below "$x$".Specially it is shifted exponential distribution and shifted exponential does not have **memoryless property**.
+1.  The issue is not ANCOVA v regression (they are the same) but how you build a model.  The colleague's proposed approach sounds like a variant on stepwise regression, and will have all the associated problems as Peter Flom points out.  See http://stats.stackexchange.com/questions/13686/what-are-modern-easily-used-alternatives-to-stepwise-regression .
Perhaps you could adapt the concept of [compactness](http://en.wikipedia.org/wiki/Compactness_measure_of_a_shape) for you purpose.
Hello Glen, In Prism, they use Cov to refer to the normalized covariance matrix, where each value is between -1 and 1. I do not really understand why it is (might be) important
No, not that the *means* are similar, but that the *variances* are. Although, when means are very far apart, variances are likely to differ; but the you can use e.g. Satterthwaite to fix that.
"*Is the number of possible sd's under the z-distribution endless?*" - yes, but they become increasingly unlikely if the null hypothesis is true; the probability of being more than two standard deviations away from the mean is roughly 5%. The mean of the residuals that I gave is zero under the null. The standard deviation is a bit less than 1.
The advantage to you of me not just telling you the answer is that you get to discover you already had all the information (probably in your head). You actually know more than you realize. So not only are you now right, but you discover you already understood something that maybe it didn't seem like you did. You seem to be doing okay.
John, are you asking about works discussing, say, which *models* may have multiple local maxima or is your question about *optimization methods* and the risk that they might return maxima that are global but not local?