tttjjj/elt_bench huggingface dataset:
Note that the AIC is based on an an asymptotic argument, originally one relating to regression models. If I recall correctly (and I may not, it's been a long while) the $-2$ is there basically because of the $-\\frac{1}{2}$ in the exponent of an asymptotic normal distribution. It's not really necessary, and I've seen some people work with [AIC half that of the usual](http://www.csse.monash.edu.au/~dschmidt/ModelSectionTutorial1_SchmidtMakalic_2008.pdf) (see p31). The adjustment term at the end of the AIC is there to (asymptotically) adjust for the bias introduced by maximizing the likelihood.
That's a good point.  If you assume that it's a widget with no defined value (like money) does that change what is possible?  I'm thinking of a widget that lacks a good secondary market that can be used to verify prices, I guess like an non-liquid luxury item.
Thanks for this informative answer. But I don't have sufficient information to plug in those numbers to make use of the equation.
We welcome questions like this, @Jamie, but we treat them differently. Please tell us what you understand thus far, what you've tried & where you are stuck, & we'll try to provide hints to get you unstuck. To better understand the process, you should read the wiki for the `[self-study]` tag.
thank you , that sounds interesting! I appreciate your help
Short answer: Yes. Slightly longer answer: In both cases you need a (mild) integrability condition to be satisfied for $X$ and $Y$. If you have any familiarity with measure theory, it might help you to review the *definition* of conditional expectation. It may appear quite abstract and a bit obtuse on first encounter, but it's worth understanding. (There *is* intuition lurking in the definition.)
Explain why you are using data splitting.  This is typically very unstable and inefficient unless $N>20000$.  To your question I would consider the two measures you listed, plus $R^{2}$, rank correlation between $\\hat{Y}$ and $Y$, and a high-resolution nonparametric calibration curve.
Thanks Leeor. Read a bit into this and it looks like what I'll try out. However, out of curiosity, what do you mean "interpretable machine learning classification algorithm"?
Thank you for your answer. I'll adapt the code to my simulation study in order to get some results.
Thanks for your answer! I agree with your above suggestion and soon I will formulate in a new question what I want to know.
What loss function(s) do you have in mind?
@Deah It sounds like you need to read more about factor analysis. At a **really basic level**, extract the specified number of factors in the test with a rotation and see whether the items have highest loadings on factors as specified by the test. There's a lot more you could do of course (e.g., look at tests of how many factors should be specified; look at confirmatory factor analysis; etc.).
Are the two scores measuring the same attributes, so they are repeated measures of the same thing? What are your entities?
Your actual population is infinite. It is not the people who came to the bank it is the theoretical class of bank customers. You are just dividing your sample from that population. For this you want Fisher's Exact Test.
from this model, $g_i$ and $\\delta_i$ are not identifiable.  For $g_i + \\delta_i=(g_i+c) + (\\delta_i-c)$ and the adjustment leaves the variances unchanged.  This model would be identifiable if $g_i=g$ (i.e. same for all i) or if it was the same across clusters (as in multilevel models).
That's a good thought, but it throws away the information coming from the ordering of disability levels.  Is there a reason you are not suggesting, say, an [ordered logit](http://en.wikipedia.org/wiki/Ordered_logit) model?
You are doing it right, but I would say, rather, "The null hypothesis is rejected" and, yes, I'd use b instead of r. Oh, and I'd say p < .001 instead of p = .000. That's a quirk of SPSS output; but p is never exactly 0.
Could you please tell us more about the "weighted data set"? What and how did you weight? In a mixture model one usually specifies the density of a random variable as being a (finite) weighted sum of component densities. The latter are fully specified densities, so it is not clear to me what changes for the "weighted data set".
see also: http://www.stat.columbia.edu/~liam/teaching/neurostat-spr12/papers/brown-et-al/time-rescaling.pdf
Some example code would help a lot.
A small effect. In a correlation table the pearson correlation between ROA and Debt to Asset ratio is .107 with sig. value of 0.165. But in a coefficient table the unstandardized coefficient of -.022 with sig. value of .425.
Is validation accuracy of 100% common in your field of work.  It is basically unheard of in my field of work.  If you found something that is very unusual, perhaps you might check your work with a second statistical software package.
I will add the ClassCenter plot to my question above. I also thought that the different scales of my predictors might be the problem, but it seems like scaling is not necessary when using random forest. I just quickly tried the legend command but it didn’t work, but I think it’s probably me making a mistake. Will give it another try tomorrow. Thanks again!
As illustrated in [a related thread](http://stats.stackexchange.com/questions/41438/covariance-of-inid-order-statistics), there is no nice simple expression in general: the situation can get quite complicated.
But the cox model doesn't assume the hazard is constant, so the models aren't nearly the same. The Cox model has the baseline hazard that's equivalent to the K-M/nonparametric empirical hazard, I wouldn't expect the two to be the same
You're welcome. AFT models take into account censoring time too. The are survival models and can be taken almost in the same scenarios than cox models , but without the assumption of proportional hazard ratios. On the con side , they are parametric models so you have to choose the right probability distribution for time. I usually go for lognormal or loglogistic distributions. The your model will estimate the mean and scale of the distribution that better fit your data.
Of course, percentile intervals are easy to obtain. I just wanted to check a general alternative for the others.
This question appears to be off-topic because it is about statistics, and not academia
I don't understand what T does if X already gives you the sequence of observations...
If I understand correctly, the problem they tackle is a bit different (and more complicated), since they seem to assume any classifiers working on any inputs, whereas I use the original ROC curve with a single scalar input and the trivial threshold classifiers. Good finding anyways, thanks!
I interpreted as the probability that a draw from a Uniform sample U(1,3) would be less than c.
@chl: (continued) If I'm not mistaken we need ***n-plicates***, for example, in the case of RM ANOVA: when we have factors let say: "donor", "concentration", "substance" we can estimate not only a distinct factor but their combination (but I have a doubt...)
Are these TS AR, ARMA, ARIMA, etc?  there is a function arima in the stats package that will help you set up a time series.  Here is a link to a description:.  http://stat.ethz.ch/R-manual/R-patched/library/stats/html/arima.html  You'll have to do some work on the time series like look at the ACF and PACF functions to decide what type of time series you're working with.  Building different models and looking at the $\\sigma^2$ and AIC output for the model will let you know how good a model you've built (the lower the AIC the better).  AIC will be an object of the time series you build in ARIMA.
I cannot download the paper (maybe because I am on phone) but are you sure you aren't just suffering from rounding error?
Thank you for your answer. I've attempted to clarify above.
@cardinal +1, I've upvoted the question earlier, but I just want to thank for a very good answer, examples and links to the articles.
+1 for your first point. That's not only good advice for data recording and transfer, but ideally should result in feedback concerning experimental design or monitoring.
In case you are still interested in that: can you describe how exactly you did the cross-validation? How was the CV error computed? One has to be careful to do it correctly for PCA, and I suspect you might have done it wrong.
Reading more about RF, I just realized my calculation in (2) is incorrect, as random features are not just selected once for a tree, but for _every_ splitpoint within a tree
Thanks for your reply. Sorry, if it is a little unclear but the first sentence in the problem-section should sum it up pretty good. I would love to create a low amount of representative data points. Is there another information I can provide you with?
I just add a real example, which should be helpful. thanks
you have to start your testing procedure with a trend and a drift. then your remove the trend, and finally the drift.
But is there a reason, in general, to expect them to be correlated? I can see both sides of this argument.
I do not understand why negative values are "completely meaningless," because you are looking at *differences* between measurements. That means they include measurement variability (and perhaps other sources of variability as well) in addition to the true difference in values. Even when one measures a quantity known to be non-negative, it is not surprising to obtain negative values and when one computes a difference of two such measurements, negative values should be expected. (Radiation readings are a good example of this, because they need to subtract background, too.)
What made you consider PCA specifically as oppose to discriminant analysis?
+1 to ttnphns. @Beka, if you want to read more about suppressor variables, I wrote a little bit about it here: [x-and-y-are-not-correlated-but-x-is-significant-predictor-of-y-in-multiple-regression](http://stats.stackexchange.com/questions/33888/34016#34016); another (long, but) great resource is [this](http://luna.cas.usf.edu/~mbrannic/files/regression/Partial.html).
Are you sure that the output is an actual probability? Anyway, I don't know how multinomial logistic regression works, so I'll have to look into that. But isn't there an (algorithmic) way to explain how a perceptron with two or more output nodes is constructed? Are they chained together somehow?
I can't answer regarding your proposed solution, but estimating the average treatment effect, ignoring interaction, has been fairly well established, methodologically, in causal inference. The main options that I'm familiar with are propensity scores, inverse probability weighting, g-methods, and instrumental variables. It may be easier to use one of those methods for your average effect, rather than creating your own solution. Unless you are specifically interested in creating a solution.
Sounds doable as an RE model with crossed random effects (rater and rated).
A day later, I now... don't really recall. :S I grabbed my stats book and agree that $a + b - 2$  is reasonable (though, it doesn't have Fieller's Theorem).
@Legend A test of difference of means may be inappropriate, but that is not the fault of @robin, as pointed out in the second half of his response, which is apt: the test to use is determined by which characteristic of a suite of results signals an "improper" experiment.  You could conduct an F-test for a difference of standard deviations; you could conduct multiple-outlier tests; you could conduct a Kruskal-Wallis test; etc., depending on what kind of differences you're looking for.
@RasmusBååth So you put a prior distribution on the model parameters and the posterior distribution of the correlation is your Bayesian correlation, if you want.
I don't know of a single tree algorithm that works with non-huge datasets.  That's why random forests, bagging, and boosting are so popular.  But you lose the interpretability of the tree with these.  In general there is a interpretability-reliability tradeoff.  You might as well fit a rich regression model with appropriate attention to number of parameters examined.  Regression models are interpretable and you can make them reliable.  A good start is usually a flexible additive regression model, allowing many of the continuous predictors to have nonlinear effects (using e.g. splines).
For this to become an answerable question, we need specifics about what you intend to do with this language and the criteria you need to apply to compare languages.  I'm afraid that "advanced" is not a specific enough description of what you're looking for.
@sajni: you should definitively read the documentation...
"'par':	Initial values for the parameters to be optimized over."
I think repeated measures anova is fine. Multiple testing is not needed, bec. you only have two conditions. If you have many repeated measures, you could use a random intercept model or a random slope model instead.
@IrishStat I was assuming that your software could identify the breakpoints but does not estimate linear functions of time.  So you would solve part of the problem for him. Also and Arima fir in each interval could be better than the regression lines.  That would depend on the data.  Am I right?
The standard method of CCA works with product moment correlation coefficient matrix. For largest mgnitude CC it constructs two composite variables z1(n) and z2(n) by linear combination of two matixes (with n rows and m1 and m2 variables) such that abs(correlation(z1,z2)) is maximized. This objective function may be maximized directly even if correlation(z1,z2) is not product moment but defined differently.
I've never been a big fan of discretising continuous data, but that's an interesting idea.
See my edit to answer your question, @Michael
You seem to have 2 questions here: (1) How does the computer / software know whether your data were randomly sampled or not? & (2) Why is random sampling important?
It's nice to see such an argument based on the concept of hazard rate (+1). But since you're not giving a mathematically rigorous proof, wouldn't it be simpler--and more in keeping with the spirit of this approach--just to notice that (conditional on the system still working at time $T$) the chance of failure by time $T+\\Delta T$ for sufficiently short increments $\\Delta T$ is essentially the chance of a single failure, which will be the sum of the $h_i$? That makes the result intuitive and immediate.
Thanks!  It appears that one should run this test before doing pairwise comparisons.
Just a note, reliability is intimately connected to the slope parameters in IRT (the larger the slope, the more the item correlates with your trait...and with lots of these the reliability overall tends to increase overall). Therefore if you are constraining them by design you aren't really looking for low reliability items. This is why I'm pretty sure you want item fit indices rather than reliability type measures.
@ttnphns sorry, typo on my part, i copied the wrong line from excel. the numbers are quoted as percentages, 50/23000 = 0.21% = 0.0021.
What do you mean, simple stratification? I already know the proportions, I just want to know if they are significantly different between age groups.
Well put! I'll mark this as the answer!
I edited my answer, hope it answers your questions.
How did the subjects end up consuming different quantities of the chemical?  Calling something a "natural experiment" implies that there is an exogenous source of variation in the treatment variable.  Do you have reason to believe that true in this case?
It will help people answer your question if you could provide more detail on why you want to do this.
@Zhubarb Thanks for the correct! I must have been undercaffeinated. ;)
A permutation test might be appropriate here.
There are different techniques for different distributions. Some techniques can be used for many distributions. To answer the question as asked, it's a 'takes a book to answer' question. Please try to narrow the scope of the question a little.
Hi John, thanks for the response and I now realize that my original plan is not too accurate. The reason I do not want to just plot MAD for my error bars is because this provides similarly sized error bars for each bin. Since my first bin has less than 100 samples and my 5th has over 2000,shouldn't the error bars for the last bin be much smaller than the first due to the larger sample size?
@chl Just one disorder
Excellent answer. I'd underline that identifying a good functional form for the relationship between variables is widely underestimated in importance, while being right about the distribution of errors is widely overestimated.
Often I see loess being treated as more of a black box. Unfortunately, it's not true. There is not other way than to look at the scatter plot and the superimposed loess curve and check if it does a good job of describing the patterns in the data. **Iteration and residual checks are key in loess fitting**.
Thanks for your reply Stef. I was worried that this was going to be an orphan question. So I do know how to aggregate the variables and the process involves _just_ categorisation and a linear combination. I have tried out option 1 as you suggest. What has worried me is that the direction of the effect with complete cases only has reversed after imputation. How would I try the 'on the fly' passive imputation? Am I right that passive variables are not part of the imputation model, and so how would this make a difference?
I guess its a bit counter intuitive to me that the fit for age is being explained by passenger class. Neither age, nor class-age nor sex-age is statistically significant
The remark about maintaining old packages goes just as well for a user written macro or an old proc that sas hasn't updated.
Statistically rank-based tests are robust against outlier. But an outlier is an outlier, on the operational level the analyst should still examine that case. So, I'd say we still need to "partially" worry about outliers.
I'm not sure, I guess you will need to put in place more machinery for that, since the distance between two sets doesn't tell you anything, by itself, about probability that they come from the same population... I'm not very knowledgeable about that. I'm not aware of any best way to do it, though.
Oh lovely. Thank you so much.
good answer.  I have a small quibble: the use of the word "subjective" and "objective" is not quite appropriate, because no methods are "objective".  I'd say more the frequentist and "objective" Bayesians simply derive their probability distributions by using certain rules or standards.  So rather than tayloring for the specific case at hand, a frequentist/objective Bayesian will apply "default" choices (thus hiding their subjectivity).
Why are you transforming the data?
@septagram `zoo` handles regular series as well, see `?zooreg`, and it works like `ts`. Sorry, I should have been more explicit.
http://en.wikipedia.org/wiki/Support_vector_machine
please do a simple proof-reading before posting the question. Putting caps and spaces between words is language independent. I am writing this purely because, every question you ask is poorly formatted.
In the paper, the authors re-analyses different data sets where a power-law has been fitted. The authors point out that in a number of cases the power-law model isn't that great and an alternative distribution would be better.
+1 Great answer/summation.  You're on a roll today.
Without getting into too many details, I don't believe that model would work for this data set.  What I have are the counts of disease along with some categorial factors for each year.  (i.e. Asian, Men, 2005, 10 out of 500 had disease.)
Can you be more specific on the number of rounds and sessions in your experiment?
But small amounts of imbalance are not a problem.
It should indeed be a uniform distribution, my point being that with a small number of observations, even a uniformly distributed random variable could look quite bumpy. What about the random split? Could you try that?
Well, my variable was a non-negative integer before I combined observations and substituted the number of events with an AVERAGE number of events, which is obviously mostly non-integer. I don't need to predict the number of events in the every particular case since there are some random effects that I can't consider in my model, I just need an average number for a certain set of conditions (predictors).
@Michael, spatial autocorrelation means that the correlation between points depends on their spatial locations. I think this answer would be more useful if you could explain why using a smooth function estimate, with the spatial locations as inputs, accounts for this. On the surface, it seems that the smooth function approach models the _mean_ while spatial autocorrelation refers to the _covariance_ structure. I know there is a relationship between the covariance function of a smooth process and smooth function estimation but, without making that connection, this answer seems incomplete.
See the geepack package for `geese`
Thanks, the last point is very interesting. Could you suggest some references on adding continuous value to adjust batch effect in linear regression?
related: [what-methods-can-be-used-to-specify-priors-from-data](http://stats.stackexchange.com/questions/5542/what-methods-can-be-used-to-specify-priors-from-data)
Thank you for taking the time to answer the question complete with the examples.
@Damian, I will look into those post hoc tests you suggest, thanks. But I guess I am wondering why the use of post hoc tests on adjusted means is not more common. Semi-rhetorical question: Do most people (non statisticians, but users of statistics) know that ANCOVA post hoc tests are often on original means?
No experience with HMM here. But it seems you are plotting the LL of the current parameter values over the progression of the training algorithm. This has nothing to do with concavity of the LL over the parameter space. Hope this makes sense.
See equation (11) in http://www.stuart.iit.edu/shared/shared_stuartfaculty/whitepapers/thomopoulos_some.pdf (but watch out for the awful typesetting).
The proposed routine may make sense only when you select from a pre-specified set of regressors. But if I choose say a quadratic trend, 5- and 9-knots B-splines, and may be a CART, I am not sure how to apply this proposal.
What is the distribution when $Z = -1$? When $Z = 1$? Then, integrate $Z$ out by averaging over those two situations and you will have a solution - this is related to the law of total probability (a.k.a. smoothing). Is this homework, by the way?
Draw a picture.  (And note that when you multiply two sides of an equality by a negative value, the direction of the inequality changes.)
@MichaelChernick: Thank you for your comments. Let's say we don't know that Chicago is more democratic, and that's what we want to know. You find that 2/5 people in the state are democrats. You get a confidence interval, and then find that in Chicago the proportion is higher than that confidence interval, would it be valid to say that Chicago is in fact more democratic than the state at large? Keeping in mind that this is all from the same poll. I'm not trying to predict anything as such, but merely trying to tell if the subgroup is in fact significantly different from the group overall.
Sadly, I don't have the original studies.  All I have is reported means and CIs of log ratios.  Some are symmetric and calculated parametrically.  Others are bootstrapped, and hence, asymmetric.  I'm essentially summarizing a number of different meta-analyses into a single measure.  I do like the idea of the transformation to inspect the properties on another scale.  That might be a nice way to go.
OK I added a description. 
(I have a coin with three sides S1, S2, S3). I flip the coin n times. The coin has a probability p=1/3 to be flipped to the side S1, and similarly with S2 and S3. X1 counts the number of times the coin is flipped to S1 (X2 and X3 are defined similarly).