tttjjj/elt_bench huggingface dataset:
for sure.  In few cases would this be reasonable in general, but in many it might.  One wrinkle that I left out of my original post is that I'm using tensor product interactions, so the effect of one variable on another will depend on the values of both.  This mitigates that concern somewhat, but doesn't change the assumption that the selected-out variable only has a moderating effect, and not a level effect.
What happens if you've already drawn all the $2$s by the time you draw your first ace?
Thank you Max, your answer is both helpful and insightful. Yours, Tal
Due to the linearity of the expected value, and the i.i.d (=exchangeability) assumption, the expected value you want is equivalent to $5E(X_i\\mid \\sum_{i=1}^nX_i)$ for any $i$.
It would be helpful if you posted the code so that we could see how the calculations are done. Also, the fact that we don't understand how the program gets the "outlier limit" value is a huge problem. To remove outliers from the calculation, you have to understand what the program considers outliers.
Yes, perhaps my terminology is incorrect and certainly self-selection is a problem in all online research. However, 10k is truely the universe for this population. So how do I calculate MoE? Or can I?
Big ooops!!! It is indeed simple. I didn't know it was possible to change span of smooth.
It works well with this code : `xyplot(y1+y2+y3~time|subject,dat,type='smooth',span=.3)`

My question is perhaps not very useful after all. Perhaps it is better to delete it? Let me know.
Readers may note also this previous, related question: [what-test-is-this-for-endogenous-variables](http://stats.stackexchange.com/questions/61054/).
Good point about rescaling-- I wasn't sure how accurate that would be since technically the arc lengths are different because of small deviations. I'm not sure what you mean about 'gradient of movement' though?  I've included a close-up shot of the turn...
Thanks for the answer. But I want to make use of the known covariance matrices $\\boldsymbol{\\Sigma}_{\\mathbf{X},i}$ and $\\boldsymbol{\\Sigma}_{\\mathbf{Y},j}$ in testing. Any thoughts?
(+1) I suppose the answer depends on what, precisely, Tim means. If he means that the outcome of the process is allocated randomly to, e.g., categories, A, B, and C, then my answer is correct. If he means that the outcome is chosen uniformly from integer values between $0$ and $n$, then your answer is correct. (FWIW, I think he means the latter.)
Depends what you mean by 'structural'. The Weibull distribution is certainly more common in time-to-event analysis. It has a simpler cumulative distribution function, and it's the only distribution that has both the proportional hazards and accelerated failure time properties.
It seems from the question, the entropy/volume connection is established, similarly we know of the volume/surface area connection. So the point here is to connect the trace of the FI to entropy. Or am I missing the point of your question?
See [here](http://en.wikipedia.org/wiki/Law_of_total_variance).
When you say "pooled cross section", is this what you actually mean or do you mean that you have panel data? This can make a difference for the robustness of your estimation strategy. See the discussion [here](http://stats.stackexchange.com/questions/80287/data-setup-for-differences-in-differences/80290#80290).
This question is potentially interesting but some clarification would be welcome.  Are you concerned about one tail or both?  How do you measure "fatness"?  (Are you willing to shift and rescale the two distributions to make the comparison, for instance?)  How do you measure deviations in "fatness"?  If you contemplate a hypothesis test, then what will the alternative hypothesis be, precisely?
In my description - it does move to higher energy states with certain small probabilities, the issue here is whether or not it can stay even if the neighbor energy is lower.
If they don't affect it positively, you can remove them safely. They can be irrelevant but not harm easily, just by having a very low magnitude.
I really appreciate you providing a code snippet, and I do have great respect for R, but I use it so infrequently that I forget the basics.  I eventually figured out that I need a library(strucchange) statement, but I'm still running into various odd errors with the format of clock.  Can you make this a complete example, even if with only a tiny amount of uninteresting data?
Wow, I didn't even think about the question from this perspective.  Thanks!
Are you ignoring outliers? Yes. Are you ignoring impacts before and after holidays? Yes. Are you ignoring possible changes in the day of the week patterns? Yes. Are you ignoring possible changes in trend (or any trend at all????)? Is there any day of the month importance as has been found with cash data? Take a look here as well...http://stats.stackexchange.com/questions/58657/time-series-forecasting-with-daily-data-arima-with-regressor/58673#58673 I see that you posted that question I linked to....so you tried TBATS and now you are onto Fourier? Did you try my recommendations?
Check out Mike Lawrence's [answer](http://stats.stackexchange.com/a/6829/26743) to the linked question -- he also references a paper that deals with cross-study comparisons.
'x1 <- rnorm(100)
x2 <- rnorm(100)
x3 <- rnorm(100)
x4 <- rnorm(100)
x5 <- rnorm(100)
x6 <- rnorm(100)
x7 <- rnorm(100)
x8 <- rnorm(100)
x9 <- rnorm(100)
x10 <- rnorm(100)
x11 <- x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10
dv <- rnorm(100)
junk <- data.frame(cbind(x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11,dv))
cor(junk)
library(perturb)
m1 <- lm(dv ~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9 + x10 + x11)
colldiag(m1)'
In the theorem, $X$ is a _column vector_ of random variables while $A$ is a fixed $n\\times n$ matrix of constants, that is, it is not random at all. Thus, $AX$ is a column vector whose covariance matrix is, by definition, $$\\operatorname{cov}(AX) = A\\operatorname{cov}(X,X^T)A^T = A \\operatorname{cov}(X)A^T$$ in your notation.
I would suggest to formulate a general question regarding selection of items or aggregation of items into unidimensional scales.
Wilcoxon is to see which *column* is bigger, not which individual rows are bigger. If you want to compare individual rows you will need more data; right now, each row would only have two numbers in the columns you are interested in.
But should he do a $t$-test or a $z$-test?
(ctd) .... -- assuming I didn't make any errors. That's not to suggest that MoM is a particularly desirable way to estimate the parameters in this case, just that it's quite straightforward. If ML is not too hard, that's probably what I'd do. If there's any prior information on the parameters, then a Bayesian approach is the obvious way to proceed.
to clarify...IDF refers to Inverse Document Frequency, correct?
I will also be happy when it is well done :-D
Perhaps Section 4 would be of interest to you, Tal. Section 8 explains model selection and Appendix A lists all the distributions that can model the variable relationship.
Related, but for small number of sets (for reference): http://stats.stackexchange.com/questions/4211/how-to-visualize-3d-contingency-matrix
@whuber Thanks. I'll summarize it and come back asap!
The manual entry for a command in Stata (the .pdf or the book, _not_ the help-file, i.e. what you get when you type `help something`) usually contains a "methods and formulas" section.
Sounds like homework..?
What is the exact question you want to answer?  Is it, (1) Do men or women, on average, sell more items (over all categories or within each category)? or (2) Do men or women, on average, sell a higher dollar total (over all categories or within each category)? Or do you have another question in mind?  Also, you probably have unequal numbers of men and women vendors and that inequality probably varies by category. Unless you deal with these inequalities, your analysis might simply reflect that, rather than what you are really interested in.
Note that these are not probability distributions, but really 16 separate (presumably independent?) distributions per grid.  This question has abstracted so much from your original one that although answers may be suggestive, it is not clear how applicable they would be. For instance, the answers to your question about octrees should depend on how those octrees were constructed (the answers ought to be very different for fixed octrees and adapted octrees, for instances).
Do you have multiple response variables, or one response variable measured at different times? It sounds like the latter.
I'm giving you the tick, because I'll probably implement this.
All function do regression alright but there is no obvious way to handle repeated measures using these functions
It depends on how you define the term "better". If you define it as the model that provides less residual sum of squares, then the answer is yes. This is because, this test compares the reduction in the residual sum of squares.
thanks for your reply, if i did regress log(debt) on log(income), but was interested in the debt to income ratio, how would i interpret my coefficients. could i do to a regression of debt/income to income in levels?
@Glen_b: Do you think iid sample points  are acceptable sometimes?
You can use a [copula](http://en.wikipedia.org/wiki/Copula_%28probability_theory%29) to construct the joint distribution.
Maybe I should edit my answer and delete my "example" - this is not the serious part of my answer. My answer mainly was about the meaning of "the" Bayesian approach. What do you call the Bayesian approach ? This approach requires the choice of a subjective prior or it uses an automatic way to select a noninformative prior ? In the second case it is important to mention the work of Bernardo. Secondly you have not defined the "superiority" relation between intervals: when do you say an interval is superior to another one ?
Since this question appears to be mostly about interpreting R's output, it may belong on [Stack Overflow](http://stackoverflow.com/) rather than here. OTOH, however, IIRC, the reason the package authors decided not to display the p-values has to do w/ the difficulty of determining the right number of degrees of freedom to use in assessing the t-values. This is a very subtle statistical issue, & requires some nuanced explanations, so it might be worth leaving this Q here to give CVers a chance to attempt to explain it first.
1. IMHO time spent with R is well invested if you plan to do anything serious.
2. Also consider KNIME http://www.knime.org/ as RapidMiner alternative.
Interesting thought...I am not sure but sounds that it might work! Did you have in mind a specific pacgage like mgcv, or robustgam or rgam? Thank you VERY MUCH for your immediate response. I sincerely appreciate it.
Can you elaborate further? If want to predict the classes you have to use supervised classification.
Well, the point is that, since the solution (see link above) is expressed in terms of the first 4 moments of the population, the solution also assumes the existence of those moments.
No; multivariate regression means a multiple response (target or outcome or dependent variable). Having multiple predictors (independent variables) does not itself make a regression multivariate.
Cool,Thanks very much
Alas, that seems not to be the case, @Rob.  When the means are zero, then yes, the mean absolute value is a measure of dispersion.  But consider what happens when the shift is large enough to make all values (actual and predicted) positive.  Now the mean absolute values coincide with the means, giving us no information about the dispersions (absent special parametric assumptions).  Notice that no analog of the decomposition of mean square into squared mean *plus* variance exists for the $L^1$ (or other $L^p$) measures of dispersion.
@whuber: Your solution involving the binomial distribution was clever.  I briefly considered something in that direction, but I didn't get all the way to a solution because I returned to kram's assertion that they were to assume a Gaussian distribution.  With a lack of data, I felt forced to make an assumption about the standard deviation of something or other.
(+1) I was looking for a good (and short) interpretation of BND and found at last :)
I am trying to formally prove that the data do not provide any information about $\\theta$.
@Behacad I saw several macros/syntax for SPSS to do parallel analysis, e.g. here is one https://people.ok.ubc.ca/brioconn/nfactors/nfactors.html
Nice find, Krysta.  A tip for future situations: checking the df in your model outputs is always a good diagnostic; doing so in this case would have made this particular problem easy to spot.
possible duplicate of [Bootstrapping hierarchical/multilevel data (resampling clusters)](http://stats.stackexchange.com/questions/46821/bootstrapping-hierarchical-multilevel-data-resampling-clusters)
My suggestion would be to construct a confidence interval for either the difference in time, or the difference in log-time or the difference in *speed* (1/time), whichever is the most relevant for you. If the difference you use has a CI that includes zero, you can't distinguish the time-difference from just random noise and should probably avoid calling one of the programs faster. Such a CI could be constructed via a bootstrap if the sample size isn't too small, or via permutation/rank based methods, or via parametric methods (e.g. from a GLM type model)
Some details given in this excellent answer on the differences between BUGS and R distribution syntax... http://stats.stackexchange.com/a/5564/13267
(+1) This is a nice approach.  Thanks for sharing it.
Also, we have quite a bit of historical data (full surveys and some simulated ones) so I may have more options for cross-validation.  The simulated ones would be a rough guide, as I'm not very confident of the quality of that data.  So one method I had in mind was to delete some of the specific variables in a historical full survey, apply the matching, then cross-validate.
Contributions are usually focused, so they could be counted as times a participant has submitted a contribution or the size/complexity of the contribution. The records allow both.
So far your question contains only a statement; you should ask a specific question. Could you explain what your approach to estimating it is and precisely what difficulties you're having?
You can see what to do with your gender coefficient here: http://stats.stackexchange.com/a/43683/7071.
@Glen_b, that's what I had in mind. I've never done one, though. Frank Harrell's suggestion of OLR is a clever way to work around the problem as well.
+1. Good approach, good advice. But I would caution against using a polynomial model: it is contraindicated by all understanding of how materials vary within sediments. A first cut would be a piecewise *exponential* model using a robust fitting procedure.  (Allowing for a changepoint helps represent multiple discrete strata of sediments.)
Do you mean *components*, rather than factors?
@love-stats $(0.04)^5$ is the probability that the ticket held by the OP is drawn on all $5$ draws, and so if the question was "What is the probability that your ticket is _not_ drawn on at least one of the five draws?" the answer would be $1 - (0.04)^5$ which is _very close_ to $1$; it is _very highly_ likely that an event of probability $0.04$ will _not occur_ at least once on $5$ trials.
Thanks, I think the Friedman test is interesting, but I can't quite figure out how it is doing that adjustment for Type I error in the post-hoc.  The comments say it is a "Wilcoxon-Nemenyi-McDonald-Thompson test" but I've never heard of that before could you explain it?
It's not sufficiently clear from your description whether samples are paired or not. If there's something making one winter observation more like one particular summer observation than a randomly chosen summer observation, that would be pairing. One such example would be if you had a set of measurements taken in different locations, both in summer and winter. In that case the observations would be paired by their shared location.
Any changepoint method (search the [tag:change-point] tag) will do the trick when applied to the cumulative sums of absolute values of the data.  The nice thing about such an approach is that it is not purely *ad hoc*: it will enjoy all the properties of the changepoint detector you choose.
I think you'd have to summarize the arguments that Abel and Koch and Urbach make, if you want people here to critique them. Otherwise, only people who have the book will be able to comment. The link you provided only shows things like programs used in the book
Thanks whuber. Great answer. So if I center the theoretical extremum on 0 (it is actually a minimum) I am fine with ommitting the linear term. This acutally leads to a highly significant quadratic predictor (without the linear one).
As @Scortchi alluded, the method you proposed is not based on any good statistical principles. If you really need to do model selection (which is not obvious why this is helpful; weak variables don't hurt models), then you should combine model selection with shrinkage using a method such as elastic net.
see this related question: http://stats.stackexchange.com/questions/1848/bayesian-rating-system-with-multiple-categories-for-each-rating
Then you may want to use the top-down strategy. That is, compare `null.1 <- lmer(meas ~ 1 + (1 | animal), REML=FALSE)` to `full.1 <- lmer(meas ~ location + (1 | animal), REML=FALSE)` to show the location fixed effect is significant, then compare `full.1` to `null.2` after refitting `full.1` with `REML`. Then you can avoid testing the `location` random effect twice.
Is this question more appropriate for Mathematics?
+1, this is really clever. Is this published somewhere? Is there an 'official' reference for this?
What are you trying to achieve (i.e. what is the actual problem you seek to ask of your data)?
Setting a diagonal element to infinity makes no sense: perhaps you meant zero? Regardless, there is no basis for adjusting the mean: the mean, as a parameter separate from the variance matrix, remains whatever it was regardless of what changes you make to the matrix.
Rick Durrett's probability text has this as a fun problem. In the third edition, it's on page 83.
Wikipedia calls it loglinear analysis: http://en.wikipedia.org/wiki/Loglinear_analysis
+1 for Friedman test, I use it often with the Nemenyi post-hoc test and plot it as critical distance diagram. (I have an R script for this, I'm searching for it right now)
-1 for not reading the documentation of `?cutree`. **read the documentation, goddamnit**. Somebody put a lot of effort into writing the code and documentation for you, and you don't read it. Fail.
@rosser A few thoughts. #1. There should indeed be a lit review, and yes, that should be step one. But a proper Bayesian analysis that controls for confounding properly needs a full, quantitative lit review of *every* variable to be included in the model. That's no small task. #2. Depending on Moore's law is a bad idea. First, recent gains have been made mostly in multi-core/GPU systems. That needs software written for it, and problems that gain from parallel processing. A single GLM model done with MCMC might not be that. Cont...
I see your further comments but don't have any insights right now.
You will need to provide some context to your problem, otherwise we'll have to migrate to stack exchange or close with the recommendation you try `?stl` and `?decompose`.
My question is: how do i interpret the estimate b. Does this mean that a 1 percentage point increase in X, increases the growth rate by 0.07 percentage points?
The semi-colon ; is to be preferred to the vertical bar | as the later is usually reserved to conditioning in conditional probability. From a classical perspective, $\\theta$ is an unknown parameter driving the distribution of the observations, not a random variable. If you instead follow a Bayesian approach, then the conditioning sign does make sense as $\\theta$ becomes a random variable.
In fitting the nonlinear curves have you modelled the unstransformed response or the log of the response, or something else?
@mpiktas In R, it is more natural to make a list, set its `names` parameter and later either just use it, `attach` it or convert it into an environment with `list2env` and `eval` inside it. With no loops, parse or other ugly stuff.
(1) Ok, I see the distinction between a fixed parameter from prior knowledge, versus a parameter estimated from the data, and counting the latter case in the AIC k parameter makes sense. At the same time, I see why as a practical matter, it is the differences in the number of parameters that are most important when comparing models. (2a) I can see how variability in measurement error may be unusual. But I do think that that is the situation I am in. (2b) So, yes, this is part of KaleidaGraph, each data point can have its own associated standard deviation. Thanks for the diligent responses!
(+1) I can only agree with you. About R specifically, it seems very difficult to ensure that (a) some computations will remain reproducible after updating a package (which happens to me recently), and (b) no conflict with dependencies will emerge one day (it was the case e.g., for `lme4`).
What definition/construction of a Wiener process are you using? From that you should know much more about it than just the marginal distribution at a fixed time.
@AlefSin why don't you make an answer ;) ? @anyone: Can someone verify that the procedure performed by the OP is correct ? @evdstat: (+1) a really interesting question
Well, why exactly are you using a mixed model? If you can explain that to your audience, this could go a long way in terms of providing intuition for the audience.
+1 for extra information, but now this looks like a new question. Incidentally, if you want to delete the previous one now I think that would be okay, it looks like you have now covered the additional information it had.
Not really clear what you seek here, not least is this homework? One thing not in the formulas is that really big datasets can bring bigger problems of heterogeneity, data quality and missing values. The arguments are visible in discussions of the relative merits of national censuses compared with more tightly controlled sample surveys.
@Erik - 1) so what about transforming the 1st statement as `res.var <- sumsq[i]/chisquare[i]` and specifying prior or res.var? 2) so maybe you need something like `dgamma(alfa, beta)`? I don't know, but my thoughts would go that way... This is philosophically wery difficult to say... It is a similar problem as in [this question](http://stats.stackexchange.com/q/14719/5509)
Thanks! Still it is not clear to me why Euclidean distance can not be used as similarity measure instead of cosine of angle between two vectors and vice versa?
@Glen_b: But in both books, the distributions are not necessarily continuous.
Quasi just operates off the mean and variance specifications; it's effectively like ML for whatever distribution corresponds to the variance function (... possibly times a scaling factor - for example quasi Poisson is effectively like fitting a Poisson with an unknown scaling factor). I'd think if you aren't familiar enough  with GLMs to know how quasi models work, you probably don't have enough background to do what you're trying to do. I'd go with some good basic books on GLMs first.
@Ralph The whole point of the comments following the question is that although the distribution of the data might "resemble" a uniform distribution, it is extremely unlikely that the data were truly obtained from this distribution.  That's why we use quantitative methods in statistics: they keep us from fooling ourselves (and others) with suggestive patterns or resemblances that are not justifiable.
I think the blur is due to the Word scaling algorithm. You may have better results resizing in an image editing program before inserting so it is displayed at 100% by Word.
@MichaelMayer It is not wrong then to drop the 1 that confuses me for the sake of a simpler alternative?