tttjjj/elt_bench huggingface dataset:
Are you sure it's "stepwise validation"? It sounds like your software is doing model selection using either a stepwise procedure or using cross-validated accuracy. In which case, this could result in different models, but your question isn't really answerable. (In my limited knowledge, anyhow.)
After you've observed it, it's not.  The point is that if $\\rho \\neq 1$, $\\text{E}X_{n+1}$ is not always $\\leq$ or $\\geq$ $X_n$.
@BGreene: no rambling, I did ask back. The interesting question is: how much are the overall d.f./complexity reduced by setting coefficients to zero in a *data-driven* way. Anyways, we're drifting into a different story...
Any model with $y$, $x_1$ and $x_2$ will give you a conditional expectation of $y|X_1=x_2,X_2=x_2$.  If you swap the values of x1 and x2 in a model with a tensor interaction, you don't get the same answer unless the interaction is totally smooth.  In a linear model, the two are equivalent.  Note that I've never read this anywhere authoritative as a justification for omitting main effects in tensors, but it represents my parsing of that inscrutable Simon Wood advice that you reference.  If wrong somehow, please let me know.  In the `by` case, you want main effects unless you're sure they =0
I don't have much to add, except to give an example as you ask. I find corr(observed, expected) to be useful as a descriptive measure for generalized linear models even though in general that is not what is maximized in fitting, or even an equivalent. There is an argument at  Zheng, B. and A. Agresti. 2000. Summarizing the predictive power of a generalized linear model. _Statistics in Medicine_ 19: 1771-1781. The art with "figures of merit" is to use them for illumination, not support.
See [markov-chain](http://stats.stackexchange.com/search?q=markov+chain) questions/answers on this site.
I would not recommend studying mixed effects models until fixed effects models are mastered and random effects are needed.
You should probably amend your `xtreg` code with the `, fe` option.
Nice complement to the theoretical definitions used above!
@AlexH. I agree that the stated advantage I lated added is *theoretical*. Constructing a simulated empirical situation where raw residuals are misleading, and studentized residuals provide a more accurate picture of the conditional distributions would be a good addition.
@Will: I see. So what'd be the objective in predicting the unrecorded data for the pre 2000 era?
@MichaelChernick, I think of huber-white as a robust method with minimum possible assumptions. I think by adding more relevant information to the model we can gain more efficiency and I think GEE with exchangeable varcov can do that. But on the other hand, H-W method is robust in terms of model misspecification whereas GEE is not and that might blow up everything. I think my challenge is to find enough evidence to choose between efficiency and robustness. In my study we have many latent variables and there is a high chance of model misspecification and this might be a reason against GEE.
@user5497 1. Naive Bayes does not deliver the exact probability, but a heavily skewed score due to the independence-asssumption, LogRes however, does. 2. The output of svm can be calibrated to approximate a probability (e.g. see http://stats.stackexchange.com/questions/10387/what-do-real-values-refer-to-in-supervised-classification )
Please let me know if this answered your question or if you have any further concerns.
Honestly, its a pain to do. If I were you, I'd change my example slightly - rather than asserting that the *overall* risk in the population is 0.15, I'd say the baseline risk in the non-diseased is, say, 0.15 or 0.10, then determine the increase in risk I want in the diseased and let the overall risk fall where it may, rather than trying to set it. It's considerably easier to code, though you will possibly not have numbers that are quite as clean at the end.
Also, how are the messages initialized? Assuming a uniform distribution perhaps? Can the initialization affect the value reached upon convergence ?
@Adam The capacity for this product is 50 bottles. Maybe my description of the problem is not clear enough. I will try to edit it a bit to give more context. Basically I am looking for ideas how to solve the general problem and I am very grateful for all the comments so far. The specific dataset is given just as example what the data looks like. For other vending machines I can provide data for much longer period.
With the inclusion of all the non-donators, it seems that 80-90% of the data will be 0. Is the $t$-test still sufficiently robust if there's such a strong deviation from normality? It would also be interesting to know if the data is essentially categorical because people typically donate 5, 10, 15 dollars, ... That would mean a lot of ties, and Mann-Whitney wouldn't be appropriate.
I would guess that there's a more elegant solution, but that's the first one I came up with.
@Kaelin: Unfortunately I can't think of a text which discusses this in particular.  I can tell you that the L1 distance is preferred since it's less sensitive to outliers.  It's also related to distances between empirical distributions in probability theory ( L1 is twice the "total variation distance": http://en.wikipedia.org/wiki/Total_variation_distance ).
(oops, editing timed out) ...  What I really needed was a way to calculate predicted values for the entire model so that I could create a single representative plot of the overall effect.  I don't want to communicate between-school differences, I want to average them out and present the sample-wide trends.
I am not following. How do additional dimensions matter? Are you interested in a partial correlation between $X$ and $Y$ given the values of another two variables?
(ctd) ... I'll look again at whether I can back it out from that approach, though. I think I see how to do that now.
That shows there is only a little difference when the population is big.  It is worth noting that there is a large difference when it is small: with 600 [phyper(35, 240, 360, 100)] gives about 0.157 while an even more extreme example of 110 [phyper(35, 44, 66, 100)] gives about 0.001
(+1) Good to know, thanks.
@BrianBorchers - no, I am assuming I don't know the system input.
You're right, this is off-topic.  You can obtain the income data from the [Census Bureau](http://www.census.gov/compendia/statab/cats/income_expenditures_poverty_wealth.html).
I don't use SPSS so can't help. But note that this wouldn't qualify as a multivariate analysis in most usages. In older literature, analyses were often called multivariate if they used many variables, but in modern usage "multivariate" implies many response variables. Your analysis is multiple, not multivariate.
Thanks very much for your helpful answer.
BMIs are measured before the onset but **not on regular intervals**. The data is from PHC. So the BMI readings are taken whenever each patients visit the hospitals. I was doing a normal linear regression with onset age as dependent variable and BMI closest to the onset of disease as independent variable. But I was wondering if I can use all the other measurements of BMI. Please suggest the best technique to model this.
If you had 6000 data points, identifying fairly general kinds of association should be possible. If you had 600 observations and made some weak-to-moderate assumptions about the general forms of association, you might get somewhere. If you had 60 observations and made fairly strong assumptions about particular forms of association it might be doable. On six observations with large error? I doubt there's any way you're going to achieve much; even something as explicit as monotonic association is going to be nigh impossible to show if the errors are large.
@MånsT Well all formulation seem to revolve around a scaled distribution: $\\sqrt(n)((\\sum_(i=1)^n X_n)/n - \\mu) \\rightarrow N(0, \\sigma^2)$. How do we get to the form in the original question from there?
What part of this formula is a riddle to you? This looks like very very basic stuff.
I need both since they are capturing different information. One is the destination country and one is the home country. It's possible for the destination country's currency to be worth less and for the home currency to depreciate as well, which would have offsetting effects. In theory, $\\beta>0$ and $\\eta<0$  Alternatively, I can use the ratio of the two, which should be roughly equivalent.
good tip! We don't know the ref.
similar question http://stats.stackexchange.com/questions/2717/clustering-with-a-distance-matrix I vote to delete my answer since I have found similar question that already has an answer.
You are looking for linear regression models with _interval-censored covariates_. See [here](http://www.ncbi.nlm.nih.gov/pubmed/12529872), for example.
@probabilityislogic - I like the intuition of what you said, but I don't think it is exactly a restatement because $X - Y$ isn't exactly a rescaling of the SD: the SD forgets the sign. So independence of mean and SD follows from independence of $X+Y$, $X-Y$ (when $n = 2$), but not the other way around.  That may have been what you meant by "basically".  Anyway, it's good stuff.
It would help if you could post some sample data, &/or sample code. Also, could you post the silhouette figures? (Nb, if SE won't let you, post them to imgur.com & put a link here, then someone can add them for you.)
@Rover Having 6 pairwise tests that all pass with p > 0.05 already means you now have p > 0.26, that's a rather significant change.
All the more reason to reject the hypothesis that the sun has exploded, then. :-)
+1, However, I suspect by: "None of the coefficients of my independent variable would result in my dependent variable crossing 0 (0%) and 1 (100%)", the OP means 'w/i the [0, 1] interval of possible X values'. This is, of course, still wrong, but your #3 won't address the way the OP presumably thinks about this issue.
THanks a lot for this perfect answer!
Do you need to know this for arbitrary, specific combinations of counties, or for only a small list of specific counties, or just know, in general, the probability of x number of counties being affected by a hurricane in a single year?
I was not relying on a Markovian assumption.  I thought that the regression tree makes cuts to reduce the squared error within the groups.  The best way to do this is to predict at the mean of the group, which would imply residuals centered at 0.  That's why I asked if these were weighted or if some of the points were not visible on the plot.  I am intrigued, I'll think about it some more, but do throw a hint!
Well, I know ML mostly from University course, Coursea, lecture videos on the internet and a few papers I have read on specific topics. I know there are lots of sample datasets everywhere, but I am looking for some with information how different ML algorithms performed on them, so I can validate my own implementations.
@KatharinaHilken Welcome to the site! Some tips: your posts are automatically signed with your username, so there is no need to do it manually; and if some answer suits your needs, you can "accept it" by pressing green tick mark next to it -- you and the answer author will get a rep bonus and other users can see that your problem was solved.
Also to Chl. The specification doesn't include constant term, so I think it is OK to add all dummies.
@whuber: As I mentioned in a comment to one of the answers, there cannot be a linear trend unless it is constant, as it would cause the probability of the trait to go over 100% or under 0%.  A hyperbolic trend is _a priori_ possible, but linear is not.  The inverse log trend comes from theory rather than looking at the data; I might do some work to see if I can get more specific predictions from that model and compare to the data.
Why are you discarding variables? (http://stats.stackexchange.com/questions/18214/why-is-variable-selection-necessary) Why do you want to know the level of relation between the response and each predictor?
Why are you `not sure if the test is a valid measure for my research`?
@ttnphns ironically, i was actually looking at this page moments before your reply. The link doesn't talk about use of bayesian estimation for mean and variance of class predictors.
@xpda Mathematical solutions will provide insight into what matters and what doesn't.  For instance, they will show that you don't necessarily need to model huge populations.  They will also indicate the role played by variability, which is harder to handle analytically and comes to the fore in a simulation.
Thks a lot Frank and Peter for your help !
This question could benefit from clarification. What do you actually want? You have a proof of $E[E[Y|X]] = E[Y]$, but it is not possible for us to know what "densities and stuff" means. It sounds to me like you want to understand without taking the time to understand (the 'quick' way). And what is the problem with Example 2? The line by line argument is "use $E[E[Y|X]] = E[Y]$". Do you have conceptual problems with the concepts of expectations and conditional expectations? Or is there a particular step in the arguments that you don't understand?
@Srikant I know all this. CV is a means for finding "best". What is the definition of "best"?
For chemical concentrations, your reflex--and default approach--should be to analyze their logarithms.  This is (in part) because the logarithms are what enter linearly in equations of chemical equilibria rather than the concentrations themselves.
One solution might be to build in an interaction term with a dummy variable. So create a dummy variable $D$, 1 if negative and zero otherwise. Then your model becomes $\\ln(\\lambda) = \\alpha + \\beta_1 x + \\beta_2 (\\mathit{D})$. The interpretation of $\\beta$_1 does not change other than it is controlled for any extra effect of negative values.
What have you done to try to solve this? Is it homework or self study? If so, please tag it as such. (BTW, the failure rate is the failure rate and you're taking samples to try to figure out what it is. So, you don't actually increase or decrease the failure rate by taking samples, you do it by making the cars better or worse.)
I agree, most people probably don't care about such a small difference in practice.  This is mostly a mental exercise for me to understand tests of significance in contexts different to the ones I'm used to.
@DilipSarwate: From the Wikipedia link, a random seed needs not be random, or do I miss something?
[This one](http://www.ncbi.nlm.nih.gov/pubmed/15171807) might interest you (also see [this](http://andrewgelman.com/2013/06/09/heterogeneity-of-variance-in-experimental-studies-a-challenge-to-conventional-interpretations/)). Then there's [this](http://beheco.oxfordjournals.org/content/17/4/688.full). That relates to testing equality of variance and testing normality respectively before a two-sample t-test.
@whubber: Good point. I did my best to do that.
This question suggests you may have different ideas of what "representative" means. The use of t-tests implies you just want the sample to have a comparable mean to the population. The reference to correlation implies you are looking at a second-order comparison between the sample and the population. A chi-squared test examines the full distribution of the sample in comparing it to the population. So, the first thing you could do is reflect on what precisely you mean by "representative," then come back here and share that with us. Could you also say *why* you're checking "representativeness"?
Generalized estimating equation
Anyone know an answer to this?
Hi Chris and welcome to the site! I have a question: Are the datasets independent from each other?
By database, do you mean SQL, Postgres, Mongo-like structures? ([Induction](http://inductionapp.com) for Mac has such functionalities.) Or are you after any program that accept CSV files and allows to drag and drop columns onto graphical templates, or à la [GGobi](http://www.ggobi.org)?
Several relevant threads already e.g. http://stats.stackexchange.com/questions/13086/is-there-a-boxplot-variant-for-poisson-distributed-data Short answer includes transform first then! histograms; quantile plots of various kinds; strip plots of various kinds.
There is a bit of generality here. The same proof works for an arbitrary (finite) "state space" for $X_i$, for example. As far as $m$ balls go, my hunch is that you are in for an intractable problem, even for $m=2$. You may find some information in the literature on [covering codes](https://en.wikipedia.org/wiki/Covering_code).
All Wilson-Hilferty does is allow you to use the Normal dist'n to test hypotheses that you would normally (!) use the $\\chi^2$ dist'n to test.  However, with modern software, you can get the $\\chi^2$ p-values, etc., as easily as the Normal ones, so there's not much value to it any more.  It does not allow you to test the parameters of a $\\chi^2$ variate, so the answers to your questions are "No" and "It can't", respectively.
$Var(X + Y) = Var(X) + Var(Y) + 2 Cov(X, Y)$ (http://en.wikipedia.org/wiki/Variance)
Err, not quite.  When you condition *on* something you put it on the right hand side of the bar.
Thanks, that explains why I wasn't able to find anything like this where people were discussing "polynomial regression". I've updated my question with the results from fitting that model!
this is great to know... perhaps I was asking how `sp` works.
If they are not fungible then is there really a common p you are estimating?  How much variability is there in the p's from the individual subjects? Is it greater than you would expect by chance?  Perhaps a meta-analysis approach might be useful.
Thanks for the recommendation. While it looks like *rrvglm()* is along the lines of what I need, VGAM only implements a bivariate logit/probit model and it's not clear to me how I could use constraints to coax a multinomial logit model into a multivariate logit.
Another option is to regard $\\psi = \\alpha + \\beta$ as a measure of precision, and again use $\\mu = \\frac{\\alpha}{\\alpha + \\beta}$ as an the mean. This is done all the time with Dirichlet processes, and the beta distribution is a special case. So  maybe toss a gamma or log-normal prior on $\\psi$ and uniform on $\\mu$.
I use leave-one-out cross-validation for setting the regularisation parameter of ridge regression models, which can be performed very cheaply (see         http://dx.doi.org/10.1016/j.neunet.2007.05.005 ).  However, IIRC, the regularisation parameter for ridge regression typically corresponds to a parameter of the covariance function of the GP, however the exact relationship depends on how it is coded.
This is exactly what i ended up doing
You could compare skewness and kurtosis of the empirical distribution for both variables to see which one is closer to the normal (assuming both are unimodal). QQ-plots would also help in investigating which one is "more normal".
Should we add Hadley Wickham for making GoG possible in R?
In order to get useful and focused answers, please provide some context to explain what these data mean and *why* you are fitting a curve. If indeed you do not know what they mean, at least explain your objective.
FWIW I did also redo this with the monthly data and stats models and made some more headway but the main question still stands.. It seems to me that the positive correlation indicated at 4 and the negative at 2, although not above the significance shaded area are still, "good enough" still having trouble with the stats models parameters to get a prediction, and no idea of how to incorporate survey data...(probably with a different method??)
@ttnphns: Thanks you! I've visited you website & it is a great resource for SPSS. But unfortunately I don't use SPSS too much. But I'll surely use it if I don't get anything in R.@chl: Thank you! But it's a huge list. I read somewhere that Daisy, in cluster package, has functionality of Gower similarity.
Basically classification introduces a problem - if class A has probability 51% then our best prediction is class A. Thus the balance of probabilities is lost in the predictions - they are biased towards class A. If we try and fix the bias by using a probabilistic prediction (predict class A 51% of the time, entirely randomly) then we increase prediction error. I don't see any way around that (but I'm no expert, merely an enthusiastic amateur!)
The procedure I already gave can be applied in large dimension, as I said, using a gaussian assumption.  If the dimension is really large with respect to the sample size (i.e. p>>n) then you can  make some sparcity assumption (assume that the parameters of your gaussian distribution lie in a low dimensional space for example)  and use a thresholding estimation procedure for the estimation of the parameters...
@this.is.not.a.nick Sadly you didn't post any data. I suspect you're miss-using these boxplots. Have a look at the answers to [this question](http://stats.stackexchange.com/q/13086/603)
As an example, one piece of software which has so far peaked my interest is here:  http://www.pqsystems.com/products/SPC/CHARTrunnerLean/CHARTrunnerLeanTour.php
Thanks for the input!  Unfortunately my data cannot fairly be approximated as normal.  I may have to post another question to discuss my shady binomial confidence intervals, but for now, I just want to combine oracle-given CI's.
Some of the techniques in this syllabus will be relavant: https://www.coursera.org/nlp/wiki/view?page=syllabus
I added the citation so it is easy to find - there is also a preprint here: http://eprints.soton.ac.uk/147383/
Actually, this is not correct. You give the chance for four people all having *specific* birthdays (e.g., twice Jan 14 and twice Apr 11). This is not the same as "two out of four persons share a birthday, and the other two share one, too", where we do *not* specify in advance *which* days these are. Person A is free in "choosing" his birthday. Person B has the same birthday as A (chance of 1/365). C is again free, and D is determined by C (again chance of 1/365). Overall answer to your problem: (1/365)^2, not ^4. (Of course, the OP may have yet another problem in mind.)
Are the various results all bouncing around your cutoff for significance (ie, some p=.04, some p=.06)?
Difficult to answer your question because it is so broad. Two points, though (i) What difference do you want to test? Difference in the means? A t-test is still possible in this case. (ii) Why normalize and lose the valuable variance information?
@Andre, simple code in R says that $y=1/x$ is not a solution: `x <- runif(100,0.5,0.7); y<-1/x; summary(lm(x~y+I(x*y)-1))`
Hi Sarah,If I were you I will try to do "data mining", which means, I will do the cluster analysis using all possible continuous variables. Then you might find the (dis)similar characteristic (between)within clusters. After that, you might have better idea what you want to do for the next step. Good Luck
Relevant questions on this site: http://stats.stackexchange.com/questions/2492/is-normality-testing-essentially-useless and http://stats.stackexchange.com/questions/61715/choosing-a-statistical-test-based-on-the-outcome-of-another-e-g-normality
"believes they have been overestimated" sounds like a way of saying "considers they look uncomfortably large". That aside "abs mean deviation" or even "abs median deviation" needs to be fixed; mentioning "abs" first implies it's the last operation, which is quite wrong.
Read **[this](http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Definition)** and **[this](http://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Properties)**
*ANCOVAs allow for both continuous and categorical variables. You can also have $n$-way versions, not just two-way.
Your thoughtful comments deserve very detailed discussion. My experience confirms that people in many fields are very concerned to show that there's a single  correct analysis of a given data set, which is what they did.
The regression is to determine if there is a difference.  Regression assumes that the residuals from the model are normally distributed. Methods for checking the assumptions are included in R, SAS and SPSS; they are probably in MINITAB, but I have never even seen MINITAB used, so I don't know