tttjjj/elt_bench huggingface dataset:
Where have you obtained the expression? There is no $C$ on the right-hand-side, so for sure this cannot be in general true. In other words: more context required.
OK, that's different. Then you want to try to figure out why the effect sizes are so different.
Actually it is included, I forgot. Age of drivers are grouped into several groups, too.
**Hints:** Is the company trying to make decisions about the size-18 customers who have sent in their heights or about all future customers who order size 18?  What is the sample?  What quantitative information do you have about the sample?  What is the population?  Do you know its mean height?  If so, compute it; if not, consider how you could estimate the mean from a sample.  Similarly, do you know the SD of heights in the population?  If not, how would you estimate it?
Aren't [random-intercept-only models anti-conservative?](http://idiom.ucsd.edu/~rlevy/papers/barr-etal-2013-jml.pdf) IE., lmer(out ~ within + (within|subj) is better?
Hi Peter 

We're looking for an explanation rather than prediction. Specifically, we believe that mood and illness perceptions will account for more of the variance in subjective memory than objective memory will. 

The IQ estimate is in fact a measure of premorbid functioning based on the observation that, even in the presence of brain injury, certain skills are relatively spared. Studies have shown that given healthy adults an IQ test, and the test of premorbid function - correlations increase when age, gender, years of ed are taken into account.
Can you give the link to the chapter of the book you mean, and state roughly where it is to be found?
I changed the title and tags to be a bit more informative. If I misunderstood something, please let me know and I will change back.
@Peter Flom, can you just explain why is it so ?
The question you *actually* ask has already been posed at http://stats.stackexchange.com/questions/949/when-is-logistic-regression-solved-in-closed-form.  The question it seems you *wanted* to ask is addressed by the answers here.
You mean, notepad++ with the use of npptor :)
It is nearly always a bad idea to include an interaction without the lower order terms, and I don't this is an exception to the rule. Why force the 2 way interactions to be 0?
If Y is dichotomous (e.g. sick/not sick) you want bivariate logistic. Using "controls" and "cases" in this way is going to cause a lot of confusion, I suggest changing the terms.
What do you mean by growth/recession. Why not just compute the 14 day percent change in your indicator and the relative difference between the two? Fancy names tend to make things more confusing.
@Peteris Ah I should have mentioned that these are rare events (I think that's the term?) i.e I have less than 10 in about 6k frames.
Many thanks for all these references, including R packages.
@StasK When I saw this question I thought this sounds like a great question for Macro.  But I guess he is retired for a while.  These are especially the time when we would miss him.
Alright. Because I am dealing with means, I can safely use the T distribution. Got it. Thank you very much!
Thanks!  That analysis was long ago and I am no longer doing it, but similar things may come up again; and others do search these threads a lot, so your comment is not wasted.
Two questions: (1) Is there an illustrative plot you could include and (2) Can you give some sense as to why you want/need the $\\alpha$-stability? At first I thought you were trying to maintain closure under linear transformations, but apparently not since introducing the mixture will break that.
I'm a little confused -- you you have data on all of your explanatory variables?  It seems like not, if you're specifying a distribution for them.  Or are you doing something Bayesian, and sampling from a multivariate prior?  If so, do your unobserved variables correlate with each other, or with your observed variables?
I stand corrected. What I meant was the "Point-Bieserial" correlation coefficient.
By the Bayesian viewpoint I refer to the fact that, under normality, the Kalman filter can be seen as a recipe to update the distribution of the state; you start with a prior distribution, reflecting your knowledge (or a difuse prior, if you know nothing) and update that distribution  each time you process a new observation.
What are the important features of your lists? (e.g. ordering, size, etc.). For the basic case where you'd like to measure the degree of overlap between 2 lists, something like Jaccard's Similarity would make sense.
Good explanation. Translation for non-U.S. readers: counties are parts of states which are parts of the U.S.A., one country. Thus "state" is not here a synonym for "country".
Yes, I will add more variables to my model. However, I was planning on doing it one at a time, ie considering each time crimes VS income, crime VS age, crime VS ethnicity, etc. Doing everything at the same time may be faster, but will be harder to visualize (plus, I don't need relations between the other variables, just with the crimes).
I need to learn about Transformation of variables, leverages, box-cox, box-tidwell etc..
I'd recommend lavaan as well, stupid I didn't think of it earlier.
"We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet." - Anne H. Milley, director of technology product marketing at SAS
B_Miner,  a level shift is a change in the intercept. A pulse is a one time outlier.                Wayne, It is clear that there are two level shifts in the plot and in good statistical analysis.  The pulses are subtle, but are there.  Every model is debatable, but the point being is that you should be looking for them.
Hastie et al. is also available from here - http://www-stat.stanford.edu/~tibs/ElemStatLearn/
I'll quote @jthetzel from a recent comment on this site: "A good question, but one that most here studied in semester-long university courses, and some have spent careers studying." It's kind of like sitting down with a person and saying, "Can you teach me Swahili this afternoon?" Not that Gung doesn't make good points in his answer. It's just a vast territory.
Thanks, whuber, very much appreciated. Just to be 100 % sure, you say that $$f(\\alpha_0 +\\alpha_1 z_1 + \\cdots + \\alpha_d z_d)=f(\\beta_0 + \\beta_1 x_1 + \\cdots + \\beta_d x_d),$$ since unique minimum is assumed, $ \\alpha_0 +\\alpha_1 z_1 + \\cdots + \\alpha_d z_d=\\beta_0 + \\beta_1 x_1 + \\cdots + \\beta_d x_d$ and you then identify the different coefficients "term-wise" (coefficients in front of the $x_1$s must be the same, etc)?
@Angelo Computing correlation requires pairing data.  The correlation measures the degree to which a change in number of smokers changes the number of heart attacks in a linear fashion.
The magic words are "sequential" or "online" clustering. [This paper](http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.61.2931) looks promising.
@Ashok: I asked the question as it was at the time. It has since changed.
@QuantIbex I think I have, I'm quite busy atm, but will edit the question asap.
For MSD, if MSD = t_{alpha,2n-2}*sd*sqrt(2/n), is SE = MSD*n/(t_{alpha,n}*sqrt(2)) correct?
Please clarify 'estimation problem'. What does the model do? What are the inputs and what are the outputs?
@AlecosPapadopoulos $\\sigma_{Y|X}^2$ above should be $\\sigma^2_Y$.
I think you are right. I resorted to doing a simple ANOVA and move along.
@jerad thank you but I know that. Just what to do next. How to solve the second part of the question? Where to put the data and how to make a plot of prior, likelihood and posterior distribution.
I think for practical purposes what I said about above and below the diagonal can be ignored.  My problem isn't really a classification problem, but in terms of reducing the matrix to a rectangular matrix for other purposes your approach may be useful.
It is wonderful(!) to see you on this site Xi'an :)
If data are multivariate normal they have marginal univariate normality automatically.
I would definitely avoid using attach in all circumstances, but I think that's the right idea.  You can use `with()` to attach a data frame for a single command: `with(dat, weighted.mean(x, weight, na.rm = TRUE))`.  Also, many functions like weighted.mean include a `data=` option that essentially internalizes this.
thank you for making it very clear.
(+1) Any thought about its performance compared to [NetCDF](http://www.unidata.ucar.edu/software/netcdf/)?
I'm not seeing how this data falls into the "proportion is not frequency" class...? if you watch something for N minutes, then everything is going to add up to N. You can't have negative proportions like in that post.
Thank you very much. I came up with the same revelation right before you posted. I came up with lambda = 2.000261 as the mean. Thanks for the confirmation.
A discrete gamma won't be confined to a finite range, but a discrete beta will do the trick.  (A discrete gamma generalizes the geometric distribution, which itself is just a discrete exponential.)
The question asks *why* it is preferred to "make time series stationary." This doesn't really answer that question.
Height *really* matters. (There is a threshold speed proportional to the square root of length in the water, beyond which it takes much more additional energy to create faster speeds.)
I would observe that, given $|X_n -1| < \\epsilon < 1$, there is an $\\eta$ (what function of $\\epsilon$ is it?) such that $|1/X_n-1| < \\eta$ and that as $\\epsilon \\to 0$, $\\eta \\to$ (what), and continue on from there.
Yes, that is what I really want , thank you very much for your help
I think you may find this interesting: http://homepage.tudelft.nl/19j49/Matlab_Toolbox_for_Dimensionality_Reduction.html
@Glen_ b Q3 + 1.5IQR is the typical rule of thumb taught at this level for identifying outliers in the upper tail. Whether or not to remove them is another matter.  Are you arguing that the distribution is right skewed because the mean is larger? Why ignore the fact that Q1 is further from Q2 than Q3 is?
This is helpful - thank you for the link!
Thanks @Irishstat. You are right. I forgot bout moving Holidays and their lead and lag effects.
But why throw away good information? Over-weight collapses BMIs of 26 and 40 into the same group, although they're very different...
@Cyan What do you mean by $\\alpha_0$, $\\beta_0$ and $\\lambda_0$?
@whuber. thank you Your editing was very good.
$plim$ is the probability limit (you can find a nice explanation of it on the [mathematics site](http://math.stackexchange.com/questions/224959/can-someone-explain-what-plim-is) of SE). The proof for the omitted variable bias formula should be in any econometrics textbook, like Dougherty (2007) "Introduction to Econometrics" page 202, or in these [lecture slides](http://areweb.berkeley.edu/courses/EEP118/current/section/D5/Handout5.pdf) on page 2.
(1) To what end?  (2) did you really mean for your plot to look like that? Perhaps you can include an image of what you think plot produced by that code was supposed to look like.
Alex, You can use stochastic and deterministic modeling techniques to handle this. It sounds like you could use trend variable, level shift and then another trend variable. Post your data set and we can take a look.
I can't really grasp the difference between conditional probability and Bayes Theorem. I looked it up many times but I can't seem to understand.
In the deterministic case, I'm thinking of having two non-stochastic vectors of finite dimension and I use any norm to measure how far away they are from each other. Then divide this difference by the norm of either one to get a relative difference. I know I can't think of distributions in the same manner (finite dimensional vectors) but is there an analogous relative divergence.
Why is the first statement _correct in essence?_ I thought that the population mean is supposed to be a constant whose value may be unknown to the statistician. How can this constant have a normal _distribution_? And what does the value of the _population_ mean have to do with the size of the sample?
Here is [my blog post](http://vzemlys.wordpress.com/2010/01/06/time-series-data-aggregation-using-r-and-reshape/) with example of using `melt` and `cast`. There conversion from wide to long format is done at one stage. There really isn't anything more special.
I asked a very related question earlier this year (see: http://stats.stackexchange.com/questions/52274/choosing-a-predictive-model-after-k-fold-cross-validation). The answers from that thread suggest that the best approach is to use the 10-fold CV to choose the size of the regularization penalty (i.e. the C_1,C_2 in the elastic net), and then train your model using this penalty on *all* the data.
A lower *p-value* does not mean that the observed effect is stronger.
@JoshHemann Hi Josh, thanks for your help. My final goal is primarily on visualize results from classification and clustering, though I realize to do so, a major method is to use feature compression methods, e.g. PCA or LSI to find the most important features and reduce the dimension to 2D in order to plot. But I do want to know if there is any other way to visualize classificaiton/clustering rather than feature compression. Cheers.
Frank, can you explain it `Spearman's ρ is related to the probability of majority concordance among random triplets of observations` in more details, not very mathematically hard, if possible? Thanks.
You don't have to choose. Good plotting wrappers for correlation matrices, such as http://www.statmethods.net/advgraphs/correlograms.html ,  will happily give you the option to plot the scatters in one of the diagonals, and the coefficients in the.
7 firms and 5 years is a small dataset by most standards. You don't give full details of the model you are trying to fit -- which here is much more important than your use of Stata or of particular Stata commands. But without those details it appears only that your data don't allow discrimination of some 2008 effect from some specialized firm effect. There isn't likely to be a work-around beyond more data or a simpler model.
This is a way of updating a *cumulative* mean but not a moving mean: that's why no buffer is needed.
fantastic thanks!
(-1) Getting "two different answers to the same problem" often just means you're bashing away mindlessly without thinking about which technique is appropriate for your analytical aims. It does not mean that one or (as you state) both techniques are not sensible, but only that at least one might not be appropriate for the problem or the data. Furthermore, in many cases you can anticipate that covariance-based PCA and correlation-based PCA *should* give different answers. After all, they are measuring different aspects of the data. Doing both by default would not make sense.
@Sami: You're welcome :-)
Thanks for the answer, I am a bit confused as to what to consider as a sample: I either have a sample of one (the observation) and the background list as the population OR my sample is the background list and I am comparing that to the whole (unsampled) population which according to the null hypothesis should have the same mean. Once this is decided, the test goes to different directions I guess.
[Optimal classifier selection and negative bias in error rate estimation: An empirical study on high-dimensional prediction](http://bit.ly/wbGHSf) is an interesting paper discussing over-optimistic prediction errors when selecting a 'best' model among several ones, including those not reported in the final paper, by [AL Boulesteix](http://bit.ly/wgsewg). (When checking the reference, I came across this internal report that I haven't time to read, [Correcting the optimally selected resampling-based error rate:
A smooth analytical alternative to nested cross-validation](http://bit.ly/wNTueQ).)
"if seniors" ... I'm not sure age confers any special value; "and exports here" ... and you probably mean 'experts', but you don't need an expert for basic material
A more compact version of the same answer: `dpois(1:3, 1:3/10)`.
Not that I know of; I was just pointing out that there do exist distributions like lognormal but with the heavy tails StasK mentioned. I couldn't reply to his comment, as I don't have the rep on stats.SE yet.
You will find a similar question and answers here : http://stats.stackexchange.com/questions/33888/x-and-y-are-not-correlated-but-x-is-significant-predictor-of-y-in-multiple-regr/33897#33897
Sorry, my partner is the expert on the deconvolution part. The R package "deamer" has a deconvolution function which takes a sample of U and a sample of C and produces a density for S. The function calls in his code look like `deamerSE2(U.sample, error=C.sample, from=from, to=to, grid.length=100000)`. From that I assumed we don't need distribution assumptions.
"Simple linear regression" does not entail an "enter" method so far as I know. Are you doing something stepwise? How are the data held? In storage types that are exactly comparable? Regardless of that, this is perilously close to "I get slightly different results in different programs". Can you provide reproducible examples with specified datasets and exact commands so that people can replicate results in either program, or indeed any other program?
I really don't get what you mean.
Have you tried drawing a state diagram? Also, this sounds an awful lot like a hw question: if it is, you should add the tag "self-study."
What are "primary sampling units"?
@Pat: This is not uncommon in dynamic state estimation, e.g. the Unscented Kalman Filter (UKF) approximates (using Moment Matching) a d-dimensional Normal distribution by a discrete probability distribution (consisting of 2*d+1 components) with same mean and covariance. Sometimes, the quality can be improved by approximating the original distribution with a more complex discrete distribution. I am interested in a good distance Measure which can be used to perform this kind of optimization.
Thats great, thanks!
thanks Peter! I do assess these IVs for multicollinearity (without respect to the outcome of interest) as one of my initial steps. I like your idea of random deletion though! Thanks again, and I will look into partial least squares as well!
Welcome to the site, @user34906. At present this question is not answerable, can you narrow it down? There are some resources for / guides to asking questions on CV on our [help page](http://stats.stackexchange.com/help).
The question isn't very clear, I suspect the answer is simply to use multiple regression. Can you post the code for the two regression, including the second one on the errors from the previous model?
Your intuition that _the Moving Average Model doesn't seem to have anything to do with the "normal" Moving Average_ is sound. See for example: [Why are MA(q) time series models called “moving averages”?](http://stats.stackexchange.com/questions/58242/why-are-maq-time-series-models-called-moving-averages)
My guess is the convention varies by discipline. If you want to report the estimate plus or minus one standard deviation and that is what your colleagues use and accept, go ahead. I would not use such an interval because it's coverage is far less than what I find to be appropriate. It would only contain the true value about 68% of the time.
They don't prioritise it for publication, I think they just want to highlight it. I'll edit my answer with an example.
Crossposted on SO as well -- please don't do that.
Shapiro-Wilk W test for normal data : p-value=0.0000000 < 0.05. So is that mean the residuals are not normally distributed?
Thank you so much for this. I had some follow-up questions if you don't mind. Since the comment box doesn't allow me to list them clearly, I included them in the post.
In a recent [comment to a related question](http://gis.stackexchange.com/questions/54727/maximum-circle-inside-an-irregular-polygon-from-a-random-point#comment72507_54727), I provided links to some survey papers on this subject that are available online.  The links focus on disguising Census data, but many of them address the questions of disguising data in general.
+1 for the idea, although I don't think it's really appropriate.  This answer implicitly assumes (as does the question, I will grant) that "likely to recycle" is a binary characteristic.  It almost surely lies along a continuum, though: some people are more or less likely to recycle than others and being able to distinguish people along this continuum could be (very) useful. This argues for a regression-based approach rather than discriminant analysis or classification methods.
May I add the two [Brockwell and Davis](http://www.stat.columbia.edu/~rdavis/books.html) books?
Your comments and the reference are very useful and appreciated.
@Miranda do you have any data on call content? This would seem to be the most direct way to test the hypothesis about legislation affecting call volume - if you observed a certain category of calls related to the legislation increase.
+1 Important to mention. I actually came back here several times to make this point but each time ended up responding to something else without making the intended change.
Also, when using a seasonal time series with the forecast package, you should set up a ts or msts object. Type ?ts or ?msts to see the documentation entry. Also you know about S3 objects, right? If not, the R Inferno has a quick intoduction. Search in Google for "R Inferno".
Thanks, @Amigable — good catch! :)
Thank you I found your answer really useful.  Regards Cody
@P Sellaz Did you also proved this result for the score statistic?
Type `?mahalanobis` in `R` and look the documentation.
Because this is a 1D, not multidimensional, integral, numerical methods (as suggested by @probabilityislogic) tend to be much faster and more accurate than Monte Carlo methods.
How is it related to taking PCA transformation of $J$ most important components?
In literature I know the majority convention by far is the reverse of that here, i.e. to exchange axes so that the curve is convex down. Inequality is a keyword here especially for finding more detailed work, especially on summarising this curve, e.g. in the study of incomes in economics.
I think it should be `aggr` instead of `df.aggr`. Other then that, this works great.
Because this is your experiment you will have the raw data used to compute the percentages.  If the denominators vary, it would be wise to devise a visualization that reveals the sizes of the denominators, because they are closely related to the uncertainties in the percentages.  Since this possibility raises genuine questions of data analysis and visualization, your question appears to go beyond the mere "what's the code for this" queries that otherwise would be closed as off topic here. If, however, you just want code, then [SO] is the place to post your question.
Yes! Thank you for your answer!  Of course the errorest() function would not know I was expecting a probability unless the family and link were set somewhere.  Thanks again for your clear answer.
There is a huge literature on this. One good book is [MacKinnon](http://www.amazon.com/Introduction-Statistical-Mediation-Multivariate-Applications/dp/0805864296/ref=sr_1_4?ie=UTF8&qid=1369263467&sr=8-4&keywords=moderation+mediation)
This looks like exactly what I need. Can you explain how you got this expansion? I tried in a lot of ways and was unable to do that ...
(1) Do you mean 'Rate' rather than 'Rage'? (2) can you give context? Where did you see it? What was being discussed?
First edition at http://cran.r-project.org/web/packages/IPSUR/vignettes/IPSUR.pdf and second edition at http://ipsur.org/
This is almost the same as the question at http://stats.stackexchange.com/q/14500/919 which concerns the same phenomenon in multivariate regression generally.  The answers there apply here, too.
@Tal - a couple of queries. When you say " .. y based on x1 and x2, using a OLS regression." , you mean your create a linear model and _estimate parameters using OLS_. Am I right? and @chl's question -- do you want to predict the lower and upper bounds for the prediction interval?
What might interest you is Rosenthal's $N$, which is a statistic that tells you how many null-result studies would have to have been file-drawed for the observed significance to disappear.