tttjjj/elt_bench huggingface dataset:
It depends on how you think ADHD will varies over time/place and on what you are trying to study. It could be everyone born in 1985, or everyone born in 1985 in a specific place where you selected the cohort. Or it could be everyone born between 1980-90, if you think that 1985 is representative of an entire decade in terms of risk of ADHD diagnosis. The source population is that population about which you are trying to make some inference. The cohort is your sample. Whether or not the sample actually reflects the experience of your source population determines the validity of your conclusions.
(+1) Is it n-1 or is it indeed n-2 in the denominator of $\\hat\\sigma$ ?
Choosing correct prior distributions given some prior information is, to my knowledge, still an unsolved problem. One way forward would be to pick the maximum entropy prior, which for a given mean and covariance, is a normal distribution (google maximum entropy distributions, or pick up a copy of E.T.Jaynes 'Probability Theory - The Logic of Science' for more info). However, the presence of the histogram complicates this a bit, as it certainly has useful extra information in it. I'm not sure how you can best incorporate that.
Ben, your question would be even better if you included some equations of the model(s) you'd like to fit before you get to a coding attempt.
Just make sure that you don't run into any election paradoxes with switching candidate pools.
(+1 but I think the OP is after some pure R solution.) It works great with the [rjags](http://cran.r-project.org/web/packages/rjags/index.html) package, but we still need to specify our model in BUGS syntax in an external file.
My only concern is how well would I know if I am overfitting, for example the error keeps on decreasing in the training set as I increase the size of nodes in the layer which is obvious.
Can you explain which part of my answer suggests I thought your original variance was constant, rather than the coefficient of variation?
Clearly $X$ and $\\log X$ are dependent, so the interpretation of the coefficient of either has to be more nuanced than this when both are included in the model. But it's easy enough to differentiate the right-hand side in this case.
The [car package](http://cran.r-project.org/web/packages/car/) has the `linearHypothesis` function that enables you to test the claim that all betas sum to 1 easily. If `mod` is your model, then use the following command: `linearHypothesis(mod, "1 = beta_a + beta_l + beta_f + beta_o")` where `beta` are the names of your variables $\\ln(A)$ etc. as entered in the regression command. Have a look at the help page of `linearHypothesis` for more examples.
Against what null hypothesis? That - conditionally on the number of each value, they're just randomly placed - or something else?
Thanks for the great introduce of pandas, I didn't know about these packages and with your post I started to read about it. I see you have used rows for percentiles how can I switch that?
Are you asking what anova or regression residuals are, or what I mean by "fairly"?  (The latter is pretty open to arbitrary interpretation, I admit, but I don't think there are definitive, effective, objective ways to test for normality.)
Excellent. One of the great points in Anscombe's article is that the values of $R^{2}$, $\\alpha$, $\\beta$, and the $p$-value for $\\beta$ are the same on each graph. I have supplemented his quartet in my classes with an omitted variables graph with these properties, also. :)
I think you'll get better answers if you give some scientific context to your data, and state what scientific questions you hope the interaction P values will answer.
Can you paste your code for `model2`? I'm not familiar with your field, but I'm wondering why the binomial distribution is appropriate for clutch size (= number of eggs laid in one egg-laying period -- right?), instead of Poisson.
(+1) I think this is said implicitly in condition (1) but you also require that $\\mu$ is constant, right?
Sorry, if this isnt very helpful, originally I had an answer and saw it was wrong because of what I said above
Your example illustrates the value of robust statistics.  The *sample median* estimates the location parameter of a Cauchy distribution well.  One could argue that the weakest link in using a t-test with 30 samples is the t-test, not the 30 samples.
@Macro could you please help me understand the concept of "standardized" variables.
I expect in this case you're meant to do a bivariate change of variable. Don't forget the Jacobian.
If the data have no ties, gamma = tau. If the data have ties, you should think what to do with them - leave out? ignore? take into account - how? Hence you arrive at gamma or tau-a, tau-b, tau-c, etc. Please search this site for `Gamma`, `Kendall`.
Thanks a lot for your response :)
+1 That's a good discussion you found.  However, it seems to fall short of really addressing the question, which is whether somehow the pooled statistic could be corrected to give the desired test size and--perhaps--yield greater power.  To resolve this issue, I have provided a separate reply.
@TrynnaDoStat,  The main concern is the variability of the estimates, with the multicolinearity, a small change in a single data point can wildly swing the coefficient estimates (without the bias).  By biasing towards 0 there is not much change in the estimates of the coefficients (because that rubber band is pulling them towards 0) with a minor change in a single data point, reducing the variability.
In my case profitability is a categorical variable. Because there were predefined answer options in the questionnaire. The answer options for profitability were like, "increasing", "decreasing", "stable" etc. Actually you are right that there is no reason for 1,2 or 2,1.
Lasso is unrelated to pca, as it is not a rotationally invariant algorithm.  Ridge regression is rotationally invariate as the penalty depends on the length of the beta vector (which is unchanged under rotations).  However if you use the lasso with squared error loss, and the predictors are uncorrelated, then lasso will drop the variables with smallest correlation and shrink the coefficients of the non-zero variables (compared to least squares).
This is an interesting paper.  Could you add to your answer with some more references and include (for instance) the paper's title and author?  Are there any particular academic research labs working in this area that would be relevant?
@user11852, sorry if I wasn't clear.  I meant that in an analog to centering and scaling (which counts only for the expectation) one could also presume there to be parallel methods that account for the "linear combinations" and thus find value in removing them from the set and dealing with the remnants.  Kind of a conceptual cousin of boosting.
Martin, this is a terrible model: DEM errors do not behave this way. They have very long-range spatial correlations. If this were not the case, DEMs would have a salt-and-pepper look to them, which they don't. I won't deny that plenty of people may have published peer-reviewed papers using such a model, but it's still wrong and not defensible for use in any realistic study. If you truly want to understand how DEM errors affect slope, then you must (1) use a reasonably correlated error model and (2) account for positional error. For more about this, see http://gis.stackexchange.com/a/1573.
1. 9/10 null p-values above 0.5 *is* suspicious. 2. Multicollinearity doesn't bias results towards insignificance 3. Too small for what? 4. Non-linearity? Under the null, the relationship is flat, and that's what we care about here. 5. It is *impossible* for binary random variables to be over-dispersed, and they are the default in logistic regression - not "binomial" regression.
The quantile is a rank-based value. One you probably know is the median: You sort your observations according to size, and the median is the observation in the middle (say, you have the income of 100 people, then the median income is the 50th-highest income). Now generalizing, the 1%-quantile is the probability at 1% of the observations. The advantage of a quantile is that it is non-parametric, you don't have to make assumptions about the underlying distribution, and that it is less biased than the quantile-function of a symmetrical distribution when you have skewed data.
@ Glen_b : Yes the observations are in the same order
This question has a lot of overlap with your two other questions at http://stats.stackexchange.com/questions/11079/problem-with-anova-repeated-measures-error-model-is-singular and http://stats.stackexchange.com/questions/11113/wrong-results-using-anova-with-repeated-measures Please follow whuber's suggestions to improve them. Has your first question (`Error()` problem) been sufficiently answered? How is this question here different from your second one?
I did search statistics website but it is very generic:                  213 Info & Communication Technology

2131 IT strategy and planning prfsnals
2132 Software professionals
@J.M. I believe that the t-test assumes a normal distribution. For my data nothing is assumed about the distribution.
This is a basic filtering question. It should be moved to Math.SE or Cross Validated.
@Glen_b has answered the question I asked, but I've realized that I didn't frame the question in a way that would get at what I'm really trying to figure out. I will try again in a separate question, unless somebody tells me that goes against site norms.
Thanks. It sounds as if different countries just appear in the model through their means or overall level of various predictors. If that's so, the model as such does not know how many countries are represented. Apart from that, why say that fewer countries means more bias? Bias depends on selection of countries, not the number of countries, as I see it.
1) use the `set.seed` function before starting the model. 3) the `caret` and `cvTools` packages both automate some aspects of cross-validation and both have vignettes/manuals with examples. 4) I'm not quite sure what's going on there.
"Crucially, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say if the data really are lognormal or gamma rather than normal." So outliers are defined wrt some model, while high leverage points and influential points aren't?
Thanks for your answer. This is also my intuition. But I am more interested in cases where updating the variables independently is more efficient (weak correlations, poor proposal distribution, I guess), ideally with proof.
Thanks Glen_b that's very intuitive. It hadn't crossed my mind that's what the confidence interval is accounting for.
I think everyone knows what the range is, but there are two possible areas in which it can be applied: to a *sample* or to a *distribution*.  In the latter case the range is not often used because the range is often infinite.  Thus I assumed you were asking about a sample.  Regardless, in either case the ratio of the range to a central statistic cannot measure dispersion.
Hi there. I have run the analysis and I have one more questions. I understand how to interpret the table for the within subject effects. However, I am confused regarding the effect of GROUP in the table Tests of Between-subject effects. I found that the effect of GROUP is significant for FA; however, how will I know if there was a differences in FA1 between GROUP 1 & 2, or if this difference is in FA2 between GROUP 1 & 2? Do I need to carry out post-hoc analyses to know which one is involved? Thanks!
@yoyo That sounds great.  I guess even my conjecturing , guesses and resulting suggestions helped!  I am glad for that. What does the acronym RF stand for?
You might benefit from exploring [the highest-voted questions](http://stats.stackexchange.com/questions/tagged/p-value?sort=votes&pageSize=50) related to the [tag:p-value] tag.
Isn't this an "economist" joke? Saying that the economist Jargon of "the long run" we never actually get to "the long run"
Thanks gung. I'm believe I am using the correct code in R. So this might either be a statistical question, ie whether I am trying to do something that is simply not possible. Or the problem might be due to an issue that R has with this covariance matrix (and the question is posted on R-help https://stat.ethz.ch/pipermail/r-help/2012-August/321409.html).
Are you sure it is applicable for the time series data? And which parts of the plot constituted separate samples that you were comparing?
What about exercise solutions (a big part of the original question)?
+1 Well done! Examples are much more convincing then invoking hypothetical possibilities.  But if you ever get a chance, please consider sharing the thought process you went through in constructing this counterexample.
@Wayne Thanks a ton for the Yule paper.I have been quoting this since 1970 and have never actually read it. It is well known in some, apparently small, circles.
May we presume that "ICA" refers to [independent components analysis](http://en.wikipedia.org/wiki/Independent_component_analysis) and not one of a [hundred other possible meanings of this acronym](http://en.wikipedia.org/wiki/ICA)?  But where you mention a "vector basis," to which space are you referring?
I added an example I hope will make my question more clear.
Thanks, sorry for the late acceptance, $e_i$ and $\\eta_i$ are positive for all $i$. The normalization is weird indeed.
@Zen, to provide some background: I'm just beginning Bayesian analysis and I'm attempting to grasp the *mechanics* of the prior/likelihood manipulations. The above problem is entirely hypothetical, and, as such, there is no "sense" to the above supports. The prior/likelihood was selected purely on the basis that I hadn't yet encountered this combination; however, in hindsight, perhaps I should have chosen a more appropriate combination.
@ErnestA thats another strange thing, the answer is no if one is considering the outcome as continuous and yes if one is considering the _observed outcome_ which is discrete. And remember it is based on the outcomes observed in the sample; the most probably ones may not be in the sample. But if you are really interested, I think have to do some _reading_ and that pdf will suffice.
that is a nice article, but it would help to  summarize some of the information in your answer
@jbow I don't care more about range than variance. I want to know whether I may have underestimated the estimated OLS relationships because I had to exclude this group of data points with significantly-lower-than-average-scores. I'm quite certainly using the wrong terms, judging by what turns up when I google. I'll rephrase and gladly use correct terms (I first tended towards "constricted variance" but that led to only 85 hits).
A few things: (3 up) an interaction does not mean your independent variables are correlated, these are just different things; (2 up) if model 3 is significantly better than model 2, then yes, this suggests the interaction is significant (since the interaction is the only thing that differs between them);  (1 up) you want to avoid just fishing for significant effects unless you are thinking of your study as a pilot that you will use to plan a subsequent confirmatory study (in this case I think you're OK); I gather you ran this study to look at all three, thus go with model 3.
Adding the `self-study` tag (which you should add) will require you to identify a tag to remove. I'd suggest you don't need *both* `power` and `type-ii-errors` as tags.
why use loop? df <- within(df,y<-10^6/x) is much nicer and more efficient. Also OP asks for line, so you need to supply type="l" in your plot function.
But, you didn't say that in question. You just mentioned you added the data.
Perhaps moving window type clustering (for example SaTScan, http://www.satscan.org/ ) would meet your requirements. That specific program is for spatial/temporal data, so not really meant for higher dimensions, but maybe will give you some ideas or a place to start.
I'll report back if I get an affirmative :)  Their paper [k-mean alignment for curve clustering](http://mox.polimi.it/it/progetti/pubblicazioni/quaderni/13-2008.pdf) has some more implementation details that also might be useful to someone wanting to do this themselves.
Thanks for improving this @user (+1). I'll delete the erroneous comments now.
@whuber which last bracket? "total variance" means the variance of the total dataset.
It seems that for pitch or some other feature of sound the sound of a word could represent a short time series for the individuals.  If the sample is not too short you could perhaps fit harmonic regression models for individuals and look to see if amplitude and frequency vary more between members from different towns than for comparing members from different towns.
@stlandroidfan - I don't understand what you're finding confusing.  I've added a figure; does this help?
+1  It seems to be poorly known that the Wilcoxon/Mann-Whitney test is a test of medians only when there is purely a shift in distribution.  This can be hard to get across to non-statisticians: in some fields, the M-W has become so popular that people assume it's *always* applicable.  That's what "nonparametric" means, right? ;-)
Your question seems to require a lengthy essay style response.  Do you think you could rephrase this into a few more focal questions?
Thank you, can come in handy!
Thanks for your (always) enlightened remarks! You provide a more thorough response than mine wrt. OP (I must admit I'm a little bit biased toward epidemiology). [I'll +1 ASAP]
To me, the interesting part of this question concerns what assumptions would be needed to justify the binomial distribution in the first place!
Mapping would be a natural form of visualization. The zip codes tend to be associated with separate polygonal regions (whose coordinates can be found online or through third-party providers). Regardless, two comments seem particularly worth sharing here: (1) restricting yourself to Excel seems like it places unnecessary (and rather severe) limitations on what can be accomplished: is this restriction really necessary? and (2) the choice of visualization depends on *why* you want to do it: what information is it intended to present to whom?
Good evening @whuber, I was thinking of cases where you are trying to process a large number of data points, thus the sampling cost would be high. Investigation objectives I assume would be to see if there is a correlation between the length of time and the rate of change. Regarding the random times, I thought that would help reduce the correlation between each group of 3 rates. I am new to this though so its quite possible that I'm mistaken.
The data points are really vectors with sequence counts from 25.000 genes. The edgeR package was designed for this particular case.
About the glass, I think that just the boundary between phases lies in the half of its height.
Furthermore, it is clear why the error in the vertical position is $\\frac{\\sigma^2}{n}$ - we know that the line has to pass through $\\bar{y}$ at $x=\\bar{x}$. Now $\\bar{y}$ contains the average of $n$ iid errors, and hence will have SE equal to $\\frac{\\sigma^2}{n}$. Wow! Thanks a lot for your diagram and clear explanation, I really appreciate.
Define what you mean by the "beta coefficient".
You still haven't declared data or parameters, so it's impossible to say if your model is correctly specified. What is `M`? What kind of parameters are $\\Theta$ and $\\pi$? Do they have bounds? What are their dimensions? The stan manual is really, really useful for learning about how the properly build  a model in stan. But do you want your answer to be about stan or about BUGS?
Why would any algorithm be able to give you a prediction in this case? You have only one predictor variable, & it's missing, so there's nothing to go on. I suppose if you wanted you could use the marginal mean of your output as a naive prediction if you wanted to, but that's unrelated to the model by definition.
(+1) concise, yet very clear explanation.
But I get another question: Is it the same for the classifiers like SVM?
Cooooooooooooooooooool!
Welcome to the site! Is one of the predictors categorical and the other two continuous, or the other way around? How many factor levels do the categorical variables have? Do you have an OLS problem or something more exotic? With this info, we may be able to point you to good ways to graph the interaction, which always helps in understanding.
It is right after equation no 22 in the pdf
I've flagged your post on math.se and I'm waiting for moderators decision there. In any case, there should be only one copy of this question on SE sites.
What makes an observation positive for each class? I assume you mean by "binary distributed" that in each class there is some probability of having a positive outcome, and the outcome is thus bernoulli distributed? Is there some (perhaps relevant?) reason for the strange naming of the number of observations for each class? Apart from these weirdnesses, this feels quite a bit like a homework assignment... If so: please tag it as such, and take the time to type in the assignment properly.
I put a comment in the SO post: Beware of the type of the variables! You have encountered an int/float problem. In `sum(x) / len(x)` you divide ints, not floats. So `sum([1,5,7]) / len([1,5,7]) = 13 / 3 = 4`, according to integer division (whereas you want `13. / 3. = 4.33...`). To fix it rewrite this line as `float(sum(x)) / float(len(x))` (one float suffices, as Python converts it automatically).
@Ehab, there are several problems with your question, which should be addressed. 1. Why do you mention GLS, `Arima` uses maximum likelihood. 2. What is your model? You mention the error, but what is the variable you are trying to model? 3. You say that the forecast is larger than expected, how do you know that?
Aleco, impressive approach!
Ahh, of course, I raised K to the wrong power instead of 1, thank you, can't believe I didn't see that!
After reading the paper: An introduction to exponential random graph (p^{\\star}) models for social network. I find that there exists one assumption that the number of the observed graphs should be the same, for example n. My scenario differs from ERGMs in that the number of the observed graphs is not the same. For example. The underlying graph has 30 nodes. One observed graph has 14 nodes, the other 10, ...Based on the observed graphs, the model that I try to seek is to reconstruct the 30 nodes underlying graphs from those observed graphs with different node numbers. Thanks again.!
if you take the difference, then you get log(price/carat)
From my way of thinking you both should get upvotes which is what I am doing.
Thank you very much...and, sorry for a late reply for your answer... :) I will recheck everything again...
Thanks a lot.  I saw these earlier and they look quite simple, much easier than the suggested compAnalysis() and experimentalComparison() functions I was suggested to use.  Do you know why those might have been of any use?
Can I use the "predict" function on a new dataset, and retain the same error structure?  How does the gls command know what order the observations are in?
Start by thinking about centers of mass.
The repeated measures analysis is flawed because subjects who have a second clinical event 2 days after their index clinical event will have censored data. You don't know if they would have gone on to experience a hospitalization on day 3 had they not experienced the second clinical event. Even if timepoints are well defined, there's no reason not to use a parametric survival model like a Weibull or a piecewise exponential which can handle censoring, time varying covariates, and provide postestimated risks at specific cumulative time intervals.
It is usually a mistake to seek an arbitrary cutoff when the true relationship with disease probability is smooth.  Also, testing the null hypothesis that the ROC area is 0.5 is a quite boring hypothesis.  For most predictions you care how good the prediction is, not whether it is random.