tttjjj/elt_bench huggingface dataset:
My data is such that to ensure at least one example of the lowest occurring level in every variable, a single bootstrap replicate size will exhaust my computer memory resources (64GB RAM and twice that swap file). This is because I have discrete-choice data, many predictors, mostly categorical and mostly with some categorical levels that have relatively low frequency. Would it introduce bias to cheat and ignore upcoming variables that have all levels included as the sample is built by stratifying on the earlier variables? I could then randomly decrease replicate size to something manageable.
Hi Jeremy, thanks! I ran an EFA on the overall group and came up with a good fitting, theoretically sound, 2 factor model (4 items and 9 items). However, when conducting a CFA at the group level, the depressed group (n=95) fits well, whilst non-depressed group (n=85) fails to converge through floor effects of three of the items, all of which loaded onto the second factor. I rescoped the CFA dropping these three items, again getting a good fit overall and for the depressed group, but low fit metrics in the non-depressed group, which produces errors when I do a MGCFA. Any advice how to proceed?
I figured it out by myself. The three restrictions for the Verbeek's problem are: b1 - ln(2)/ln(3) * b2 = 0; b3 - ln(4)/ln(5) * b4 = 0; b2 - ln(3)/ln(4) * b3 = 0. Similar solutions exist also when dealing directly with x rather than ln(x).
Aaron K. Han, Non-parametric analysis of a generalized regression model: The maximum rank correlation estimator, Journal of Econometrics, Volume 35, Issues 2–3, July 1987, Pages 303-316, ISSN 0304-4076, http://dx.doi.org/10.1016/0304-4076(87)90030-3.
(http://www.sciencedirect.com/science/article/pii/0304407687900303)
@Benny I'd be interested in more information about the nested model approach. On the face of it, constraining a parameter to zero or treating it as known when it is unknown seems like it would alter the standard errors of other parameters, but I'd be keen to read more about the approach.
@whuber. I mean to say why don't we need unbiasedness condition in simple kriging? Why are we using that in ordinary kriging? Also my ultimate question is when to use which. As in my case, ordinary kriging performed much better than simple kriging, even though my data was generated from a gaussian distribution with mean 0,which is assumed by simple kriging
@DilipSarwate I mean that there were 12 'succesess'
Are you asking whether *these two clouds* have different spreads (which is literally what the question says) or are these clouds *samples* of two populations and you want to know whether the *population* variances are different?  Also, could you explain what an "error ellipsoid" might be and how that would be related to the "spreads" or variances?
Leaving a part for testing has been explained in the post. Also, as explained cross-validation is needed for comparing the models. Thus, it is not possible to apply it for feature selection unless we use the nested-cross-validation idea. However, the data set is so small and it is difficult to apply the nested-cross-validation.
@Gaba_p [Likelihood ratio](http://en.wikipedia.org/wiki/Likelihood-ratio_test) is typically used for comparing two different models.
Should this be community wiki? (Is there a "right" answer?)
@stan, I recognize your confusion. Terminology in stats is often confusing & unhelpful. In this case, "fixed" is not quite the same as the *fixed* in 'fixed effects & random effects' (although they are related). Here, we're not talking about effects--we're talking about the $X$ data, ie your predictor / explanatory variables. The easiest way to understand the idea of your $X$ data being fixed is to think of a planned experiment. Before you have done anything, when you're designing the experiment, you decide what the levels of your explanatory will be, you don't discover them along the way.
I've posted on data example. Would you like to answer specifically to this problem. Thanks
The question asks two things: *derive* and identify.  Your software has done a beautiful job of the latter, but it reveals nothing about the derivation.  Can it?
This is often fudged: people report P-value but flag that strictly it applies only to normally distributed data. In practice, a better procedure would be to bootstrap and get a confidence interval. In most research, correlations measure or indicate strength of relationships. If there is doubt about correlations being not zero, then perhaps the sample size is too small. Note that other assumptions, especially  independence of observations, are often dubious but more rarely discussed. Testing the corresponding regression coefficient downplays this problem and is often closer to the science.
A regression with one response and a one binary predictor is in essence equivalent to Student's t test comparing two means. This is often regarded as too puzzling to be mentioned in introductory texts and too obvious to be underlined in more advanced texts. (In practice, not everything in your favourite software to do either procedure may match exactly.)
Won't you have severe estimation problems in with such a logistic regression? And why go to that work when you can just look at a single count in a four-cell table and check whether it's very small compared to another cell? I appreciate that you are trying to respond to the question and frame this in terms of regression, but the risk here might be that you appear to describe a procedure that hides the extremely simple nature of the analysis that you are proposing. When I read words like "prove," "logical," "necessity," and "causes" in the question I *expect* the results to be misinterpreted.
+1 nice answer!
Since your data don't really seem to be paired, the `mcnemar-test` tag doesn't seem to fit.
..Since that's not of interest, you could simply not code it (cf: not code gender), which then enables you to say something about the general population irrespective of whether someone is a smoker or not (cf: male or female). This implies that you are acknowledging that there may be an effect of this factor, but that you are happy to blur both of its levels (smoker&non-smoker, or male&female) into a single sample that is homogenous from this point of view, and that is thus representative of an equally-homogenous underlying population.
If you standardize all the variables, then whichever has the highest beta will be "most important" in some sense. But be careful - standardized coefficients are dependent on the sd of the variables in the particular data set you are using.

In general, I don't think it's a good idea to try to do what you are doing (although other statisticians disagree with me)
+1 for being the first to point out the important point that $\\mu$ and $\\sigma$ should not be estimates.
"Best" for what purpose?  How do you intend to use the correlation estimate once you have it?  What are the potential consequences of making an estimation error due to the influence of a few outlying data?  Are you trying to estimate Pearson correlation or do you just want a quantitative assessment of a strength of (linear?) relationship between two variables?
This latter idea is called RANSAC and could probably be used by itself, especially if the number of circles per image is small.
@Thomas It still looks like a bad question to me.  It attempts to measure two concepts (how to read a control chart and the distinction between "assignable" and "chance" causes), which is one mistake, and it punishes the thoughtful test-taker who knows that much more information is necessary to interpret the OOC points than is given here, which is the more egregious mistake.
@gung I was curious whether you actually recognized it as a textbook question. I know you're aware of the [meta dialog about the self-study tag](http://meta.stats.stackexchange.com/questions/1172): my reading of it is that we should unilaterally apply this tag only when the routine nature of the question is very clear. To me "routine" means, *inter alia*, that questions very much (but not exactly) like it have been asked and answered here and the application of those answers is straightforward. I am not aware of seeing any question remotely like this one, but I can't remember them all!
The utility function would have to be arbitrary and information-losing.
@Corib: note that you can try a much simpler approach, when  for each subject you average X on "yes" days and on "no" days, obtaining two values for each subject, and then run a paired t-test on those values. If that works (i.e. shows significance difference), why bother with anything more complicated?
And the other basic plots to see would be the autocorrelation function plot for the residuals of each of your regressions; and the cross correlation function plot of the two.  These things are possible in Python but you'd find it much easier to do in R.
Thank you Gilbert!
I'm not sure if my answer will clear things up, but I just calculate the $\\chi^2$ per trial based on $(x-\\bar{x})/\\sigma)^2$, where $x$ is the result, $\\bar{x}$ the expected result and $\\sigma$ is the uncertainty.
That is fine, too. But I don't know how to compute exactly. Please enlighten me ;-)
Meta-analysis (pooling) is almost always better but I'm not sure what you are asking exactly. The method I think you are asking about is the naive method, in which there is no weighting but is dangerous (see Simpson's Paradox).
(+1) It is no surprise that the (unpaired) t-test yields essentially the same p-value, 0.000349.  Despite what the reviewer said, the t-test *is* applicable to these data.  The reason is that the *sampling distributions* of the means are approximately normal, even though the distributions of the data are not.  Moreover, as you can see from the results, the t-test actually is more conservative than the permutation test.  (This means that a significant result with the t-test indicates the permutation test is likely to be significant, too.)
I think you can accept the answer by clicking the checkmark if you are satisfied.
Well the original question was about R though.
Basically, but as I mention in the metropolis code: you want to perform your calculations in log space. Likelihood calculations tend to operate on very small values, so you generally get much better results adding logarithms and exponentiating your results than multiplying raw values together.
of course, you can conduct a 1 sided test. Or you can conduct a 2-sided test and, if the direction of the effect suggests a benefit due to the drug, you can infer that the drug is helpful. It's a big mistake to think that, when you reject the null hypothesis from a 2-sided test, all that you know is that, "the effect is at least different from 0". In practice, you can make two types of decisions based on significant findings from a 2 sided test.
For many of the data sets I work with, subject-matter expertise is not well-developed either. There are many studies and guesses, but no consensus  it seems. So although the advice is sound to me, I find it hard to use in many situations I encounter.
yes, the distribution of the statistic is certainly not of a standard form, and as far as I know there is no central limit theorem equivalent for sample variance.  I've seen derivations for the variance of the variance of a RV, what I suppose I need is a sample variance of the sample variance...
Yes this is correct. Except that all degrees are the same. So the second dataset has about 5000 rows (students), and breaks down the number of advanced degrees per student per school in the five columns. I'll reword to be consistent with the nomenclature of "advanced degrees". Thanks.
@rolando2 Perhaps my point was ambiguous. I'm referring to designed experiments. Say experiment 1 manipulates factor A, and experiment 2 A and B. Assuming a balanced design, both factors are orthogonal. Assuming both factors explain variance, the variance explained by factor A in experiment 2 will be less than in experiment 1, where the level of factor B is held constant. Thus, when comparing factorial experiments with one-factor experiments, I think partial eta squared is more similar across factorial and one-factor experiments, especially if there is no interaction effect.
It outputs a variable between $(0,1)$ but that doesn't make it a probability?
Get the raw data.  The analysis works on the raw data the went into generating the means.  Otherwise, you'll get a result that looks suspiciously like your original ANOVA.
Do you have the individual data or only those summary statistics?
Please edit the question to be more precise. Do by `multivariable` you mean multiple independent variables ("multiple regression") or multiple dependent variables ("multivariate regression" or "MAN(C)OVA")?
Yes, it is necessary, if not, adds the sum of squares of each factor in the residual term.
Thank you for your reply. Unfortunately I can't seem to find the articles freely available anywhere so that I can take a look at the methods. Regarding your suggestion for the cross-validation: my goal is to speed things up a bit right now; would selecting in a k-fold cross validation a very small k still be OK? Take into account  the fact that I have 74 positive instances and 496 negative instances.
@Matt As Shane wrote; connect with RCurl, get the output and parse it with XML. We would be able to give you the code if you'd give more details about this web service.
Your addendum is the most useful piece of information I ran into on stats.SE, so far. All my warmest thanks for this. I simply reproduce here the reference you gave: http://research-repository.st-andrews.ac.uk/bitstream/10023/1591/1/Endres2003-IEEETransInfTheory49-NewMetric.pdf Endres and Schindelin, A new metric for probability distributions, *IEEE Trans. on Info. Thy.*, vol. 49, no. 3, Jul. 2003, pp. 1858-1860.
Well done! The purpose of a GUI is not to learn the language, but to make learning the language unnecessary. (edit:) I didn't know that some GUI's had the features described by G. Kay Kerns. I suppose that would help you in the early stages of learning the language.
Although some implementations of FA can cope with complete multicollinearity (singularity), that would be a palliative trick. _Theoretically_ FA requires nonsingularity. It assumes that _each variable_ has its _independent_ part, which is also independent of the common factor(s). (That part is [called unique factor](http://stats.stackexchange.com/a/95106/3277)). This assumption cannot hold under singularity.
Thank you! working beautifully
This is your third basic question in 30 minutes.  If you want help, show us what you know and what you have tried.
I hope you don't mind me referring you to these links: [a-little-book-of-r-for-time-series](http://a-little-book-of-r-for-time-series.readthedocs.org/en/latest/src/timeseries.html) and [Rob Hyndman's online book](http://otexts.com/fpp/). They both document how to forecast using Holt-Winter's in R.
Hey Alecos, in view of the lack of covariance between the estimators of the slope and intercept, would you recommend this model instead of the standard OLS one or do you think it makes no difference?
It looks like the same consideration would apply for those who are used to say that the (true) parameter has a 95% chance of lying between xx and xx, when speaking of/interpreting confidence intervals.
Thank you for your response. I already did that but my dilemma is that if I should use AICc for selecting the variables in my models rather than the default AIC used in the stepGAIC function in GAMLSS.
In the first instance, I'd run tests within A and B separately. However, ultimately, I'd want to compare the results of A with the results of B. So, perhaps *stratifying variable*? I just need a single, understandable term to summarize my study design in tabular format.
I'm not a physicist and I don't know the necessary equations, but I imagine it will involve a numerical integral over the density. R's `integrate()` function may be useful for that.
n/w = Network. Sorry for using the abbreviation. I have transmitted 1MB of data to the peer system. like that I computed the bandwidth
For the density, you might consider something like the sample estimate described [here](http://stats.stackexchange.com/questions/73623/kernel-density-estimation-is-a-convolution-of-what/73626#73626) for example (and might be considered as the limit of a histogram with increasingly-narrow bins); it's an average, and I believe its expectation is the density. In respect of the cdf you might like to consider whether you can do something with the empirical cdf to make it in the form of an average. Both ideas would seem to work with samples from a joint distribution.
Final nitpicking, usually the notation is $X\\sim F$ where X is a random variable, $F$ is the distribution function and $\\sim$ means distributed as. To note that two random variables have the same distribution notation $X=^dZ$ is used. The notation though is a minor matter, the understanding is much more significant, keep up the good work!
@gung number of successes; we require P(X >= x)
Thank you for this answer. I suppose that there is theoretical reason enough to include the organizational controls. As to the second part of the question, perhaps an insignificant incremental F test would demonstrate the lack of explanatory power in the set of indicator variables? I will follow that path for the time being.
I am note sure to know how to implement this part of my model in Jags (I guess it would be the same in WinBugs). Eventually, I need to specify a distribution for X and It seems to me that one are limitated to distribution proposed by the software. Do you agree with that ? I am aware of the one/zero trick but it seems unsatistactory in my case.
In repeated measures, between-subject factor and RM factor are crossed: every individual comes through all the RM levels. In clustered sample, between-subject factor is nested within cluster factor: clusters are comprised of different individuals.
Thanks.  I may be able to get mass categories (i.e. 0.01 - 0.1 g for Red and 0.001 to 0.01 g for Blue) for a wider range of beads than I have accurate mass information for.  I'll likely have the same problem of no information for some bead types (i.e. maybe there is no mass information for Green beads), and it means reaching out to the cantankerous bead-mass-expert community for insights...
I'd say yes. But can you give some context?
This is not an answer, but do note that the $t$-distribution approaches the normal distribution as the degrees-of-freedom parameter $\\nu$ grows larger. Past $\\nu \\geq 30$, there is no appreciable difference, particularly in most hypothesis-testing frameworks. The limiting behavior is "from above" in the sense that the if $T \\sim t_{\\nu}$ and $Z \\sim \\mathcal{N}(0,1)$, then $|T|$ is [stochastically larger](http://en.wikipedia.org/wiki/Stochastic_ordering) than $|Z|$.
Hi Matt,   
  
In decision tree model, I am using CART algorithm, Could you give some more insights about tool/algorithm where I can set weights to value of dependent variable(churn).

 Thanks
$X_i$ are IID sequence, I have just simplified question too much.
I don't know of a specific reference that says both of these things. Any book on regression will say the first and any book on trees will say the second. E.g. [Breiman et al.](http://www.powells.com/biblio/62-9780412048418-1)
Thanks for your answer. However, I would highly appreciate if you please let me know any reference how to calculate the confidence interval (the concepts/formula) for time series prediction.
thank you for the great answer! Given your previous hints, I managed to work through it myself as well...
@johanna. That is simple. You have a value x and a standard deviation y where x and y are in the same units then x/y puts x into units of standard deviations. So if in the original units the standard deviation is 0.23 then X/0.23 expresses the quantity x in standard deviation units. If you want to see all the inteval endpoints and estimates in the original units just multiple by the standard deviation (in my example you multiple by 0.23).  Is the standard deviation given in the study article?
@mrrrau Yes, see my edit.
+1 A whole bunch of good points.
@ezbentley, that really depends on what you mean by significant. That word is not really very meaningful in most contexts because it has been contaminated by the Fisher-Neyman-Pearson hybrid. If you obtained a very small P-value then it is fair to say that the true mean is probably not zero, but it is important to say what the observed mean was, and indicate its variability (SEM or confidence interval), and don't forget to say what the sample size was. A P-value is not a substitute for specification of the observed effect size.
I'm not sure if there is a standard term in my field, so I will probably use `relative error` as it sounds a little more self-explanatory.
If this is [tag:homework], please tag it accordingly. Someone will probably help you provided you indicate where you get in trouble.
How is the rank calculated?
@PeterFlom. Thank you. So as far as I understand there is no practical significance for the number of degrees of freedom in the context that I've described above. Is that right?
+1 Nice job by both you and John.
A summary of what *MASS* says is [here](http://stats.stackexchange.com/questions/70619/dispersion-in-summary-glm/77594#77594).
(+1) Thanks for this, especially Friedman's paper.
That's the link for the full report.
@GaBorgulya, thanks for the edit, did not know about `smallmatrix`, so did not try to edit, since usual solution of breaking the formula in several lines would not have worked here.
The good folks over at http://math.stackexchange.com/ might be able to provide a better understanding to the problem you're describing, which might in turn suggest good ways to implement it :)
You write as if samples do not cost anything. In many fields of endeavor it would be impossible to achieve your preferences and utterly ridiculous to suggest taking tens of thousands of samples. Besides, whether they are your preferences, or those of a Nobel Prize winner, or of a random person on the street, ought to make no difference in an *objective* answer to a statistical question.
Anyone who is aware of a freely available resource that is not listed already is encouraged to edit this to include that information.
As I understand it: when Simpson's Paradox occurs, it tells you that the boundaries of the sub-populations *may* be completely arbitrary and therefore leading to no meaningful information. It's up to you to justify the boundaries that are made.
Do you necessarily need overall precision / accuracy? You could just report them by class. That contains more information.
E commerce is a part of CPI calculation ,however small, but that's not whats important here. my hypothesis is that E commerce has a an effect on inflation through another channel, which is that as offline prices(physical stores) go up, consumers tend to switch to lower prices which can be found online. This would result in reduced inflation because then offline stores know they have to reduce prices if they want to attract customers again. I just want some opinions on whether it is possible to test this hypothesis. I hope this comment cleared thing up a bit, @NathanCalverley
Important point about logging the population size in the offset.  Everybody forgets this at least once.
I didn't think of that! Yes I am currently treating it as a factor but switching to a continuous variable should do the trick - accounting for all three levels of genotype when calculating significance. Thank you!!
Discreteness (and test score bounds) aside, those data suggest they come from distributions that are not so terribly far from normality; QQ plots are pretty straight, for example. Being quite robust to mild departures from normal distributions, I would expect the t-test would perform quite well (both with respect to level and power) in such circumstances.